Beyond Concept Bottleneck Models: How to Make Black Boxes Intervenable?

Authors: sonia  laguna
ričards  marcinkevičs
moritz  vandenhirtz
julia e vogt
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 14:09:07

Abstract: Recently, interpretable machine learning has re-explored concept bottleneck models (CBM). An advantage of this model class is the user's ability to intervene on predicted concept values, affecting the downstream output. In this work, we introduce a method to perform such concept-based interventions on *pretrained* neural networks, which are not interpretable by design, only given a small validation set with concept labels. Furthermore, we formalise the notion of *intervenability* as a measure of the effectiveness of concept-based interventions and leverage this definition to fine-tune black boxes. Empirically, we explore the intervenability of black-box classifiers on synthetic tabular and natural image benchmarks. We focus on backbone architectures of varying complexity, from simple, fully connected neural nets to Stable Diffusion. We demonstrate that the proposed fine-tuning improves intervention effectiveness and often yields better-calibrated predictions. To showcase the practical utility of our techniques, we apply them to deep chest X-ray classifiers and show that fine-tuned black boxes are more intervenable than CBMs. Lastly, we establish that our methods are still effective under vision-language-model-based concept annotations, alleviating the need for a human-annotated validation set.

Main paper: 
 introduction: 
 Interpretable and explainable machine learning (Doshi-Velez & Kim, 2017; Molnar, 2022) have seen a renewed interest in concept-based predictive models and approaches to post hoc explanation, such as concept bottlenecks (Lampert et al., 2009; Kumar et al., 2009; Koh et al., 2020) , contextual semantic interpretable bottlenecks (Marcos et al., 2020) , concept whitening layers (Chen et al., 2020) , and concept activation vectors (B. Kim et al., 2018) . Moving beyond interpretations defined in the high-dimensional, unwieldy input space, these techniques relate the model's inputs and outputs via additional high-level human-understandable attributes, also referred to as concepts. Typically, neural network models are supervised to predict these attributes in a dedicated bottleneck layer, or post hoc explanations are derived to measure the model's sensitivity to concept variables. This work focuses specifically on the concept bottleneck models, as revisited by Koh et al. (2020) . In brief, a CBM is a neural network consisting of successive concept and target prediction modules, where the final output depends on the input solely through the predicted concepts. Such models are trained on labelled data, in addition, annotated by attributes. At inference time, a human user may interact with the CBM by editing the predicted concept values, which, as a result, affects the downstream target prediction. This act of model editing is known as an intervention. The user's ability to intervene is a compelling advantage of CBMs over other interpretable model classes, in that the former allows for human-model interaction.Figure 1 : A simplified, intuitive example: an image of a grizzly bear is wrongly identified as an otter. Our method allows performing a concept-based intervention and flip the predicted class. In order of appearance from left to right and top to bottom, the depicted concepts and classes are "fierce", "timid", "muscle", "walks", "otter", and "grizzly bear".In contrast to previous works (Yuksekgonul et al., 2023; Oikarinen et al., 2023) , we focus on instancespecific interventions, i.e. performed individually for each data point. To this end, we explore two questions: (i) Given a small validation set with concept labels, how can we perform instance-specific interventions directly on a pretrained black-box model? (ii) How can we fine-tune the black-box model to improve the effectiveness of interventions performed on it?Such instance-specific interventions can be relevant in high-stakes decisions. Our specific motivation is healthcare. For instance, consider computer-aided diagnosis, where a doctor may make decisions assisted by a predictive model. In this setting, the doctor handles patients on a case-by-case basis and may benefit from instance-specific interactions with the black box. While, in principle, a specialist may just override predictions, in many cases, concept and target variables are linked via nonlinear relationships potentially unknown to the user. Figure 1 contains a simplified, intuitive example of an instance-specific conceptbased intervention for natural images. Additional and more comprehensive examples can be found in Appendix A.Contributions This work contributes to the research on concept bottleneck models and conceptbased explanations. (1) We devise a simple procedure that, given a set of concepts and a small labelled validation set, allows performing concept-based instance-specific interventions (Figure 1 ) on a pretrained black-box neural network by editing its activations at an intermediate layer. Notably, concept labels are not required in the large training set, and the network's architecture does not need to be adjusted. (2) We formalise intervenability as a measure of the effectiveness of interventions performed on the model. Utilising intervenability as a loss, we introduce a novel fine-tuning procedure. This fine-tuning strategy is designed to improve the effectiveness of concept-based interventions. It preserves the original model's architecture and representations to be used in downstream tasks.(3) We evaluate the proposed procedures alongside several baselines on the synthetic tabular, natural image, and medical imaging data. We demonstrate that in practice, for studied classification problems, we can improve the predictive performance of pretrained black-box models via concept-based interventions. We investigate fully connected and more complex backbone architectures. We show that the effectiveness of interventions improves considerably when explicitly fine-tuning for intervenability. Lastly, we observe that our methods are successful in datasets where concept labels are acquired using vision-language models (VLM), alleviating the need for a human annotation. 
related work: 
 The use of high-level attributes in predictive models has been well-explored in computer vision (Lampert et al., 2009; Kumar et al., 2009) . Recent efforts have focused on explicitly incorporating concepts in neural networks (Koh et al., 2020; Marcos et al., 2020) , producing high-level post hoc explanations by quantifying the network's sensitivity to the attributes (B. Kim et al., 2018) , probing (Alain & Bengio, 2016; Belinkov, 2022) and de-correlating and aligning the network's latent space with concept variables (Chen et al., 2020) . Other works (Xie et al., 2020) have studied the use of auxiliary external attributes in out-of-distribution settings. To alleviate the assumption of being given interpretable concepts, some have explored concept discovery prior to post hoc explanation (Ghorbani et al., 2019; Yeh et al., 2020) . Another relevant line of work investigated concept-based counterfactual explanations (CCE) (Abid et al., 2022; S. Kim et al., 2023) .Concept bottleneck models (Koh et al., 2020) have sparked a renewed interest in concept-based classification methods. Many related works have described the inherent limitations of this model class and attempted to address them (Margeloiu et al., 2021; Mahinpei et al., 2021; Marconato et al., 2022; Havasi et al., 2022; Sawada & Nakamura, 2022; Marcinkevičs et al., 2024) . Another line of research has investigated modelling uncertainty and probabilistic extensions of the CBMs (Collins et al., 2023; E. Kim et al., 2023) . Most related to the current paper are the techniques for converting pretrained black-box neural networks into CBMs post hoc (Yuksekgonul et al., 2023; Oikarinen et al., 2023) by keeping the network's backbone and projecting its activations into the concept space. Additionally, these works explore automated concept discovery using VLMs.As mentioned, CBMs allow for concept-based instance-specific interventions. Several follow-up works have studied interventions in further detail. Chauhan et al. (2023) and Sheth et al. (2022) introduce adaptive intervention policies to further improve the predictive performance of the CBMs at the test time. In a similar vein, Steinmann et al. (2023) propose learning to detect mistakes in the predicted concepts and, thus, learning intervention strategies. Shin et al. (2023) empirically investigate different intervention procedures across various settings. 
methods: 
 In this section, we define a measure for the effectiveness of concept-based interventions and present a technique for intervening on black-box neural networks. Furthermore, we propose a fine-tuning procedure to improve the effectiveness of such interventions. Additional remarks beyond the current scope are included in Appendix C.In the remainder of this paper, we will adhere to the following notation. Let x ∈ X , y ∈ Y, and c ∈ C be the covariates, targets, and concepts. A CBM f θ , parameterised by θ, is given by f θ (x) = g ψ (h ϕ (x)), where h ϕ : X → C maps inputs to predicted concepts, i.e. ĉ = h ϕ (x), and g ψ : C → Y predicts the target based on ĉ, i.e. ŷ = g ψ (ĉ). CBMs are trained on data points (x, c, y) and are supervised by the concept and target prediction losses. At test time, if the user chooses to replace ĉ with another c ′ ∈ C, i.e. intervene, the final prediction is given by ŷ′ = g ψ (c ′ ).Next to CBMs, we will consider a black-box neural network f θ : X → Y parameterised by θ and a slice ⟨g ψ , h ϕ ⟩ (Leino et al., 2018) , defining a layer, s.t. f θ (x) = g ψ (h ϕ (x)). We will assume that the black box has been trained end-to-end on the labelled data {(x i , y i )} i . Lastly, for all techniques, we will assume being given a small validation set {(x i , c i , y i )} i .Given a black-box model f θ and a data point (x, y), a human user might desire to influence the prediction ŷ = f θ (x) made by the model via high-level and understandable concept values c ′ , e.g. think of a doctor trying to interact with a chest X-ray classifier (f θ ) by annotating their findings (c ′ ) in a radiograph (x), where findings correspond to the clinical concepts, such as the presence of edema or fracture. To facilitate such interactions, we propose a simple recipe for concept-based instancespecific interventions (detailed in Figure 2 ) that can be applied to any black-box neural network model. Intuitively, using the given validation data and concept values, our procedure edits the network's representations z = h ϕ (x), where z ∈ Z, to align more closely with c ′ and, thus, affects the downstream prediction. Below, we explain this procedure step-by-step. Pseudocode implementation can be found as part of Algorithm B.1 in Appendix B. Step 1: Probing To align the network's activation vectors with concepts, the preliminary step is to train a probing function (Alain & Bengio, 2016; B. Kim et al., 2018; Belinkov, 2022) , or a probe for short, mapping the intermediate representations to concepts. Namely, using the given annotated validation data {(x i , c i , y i )} i , we train a multivariate probe q ξ to predict the concepts c i from the representations z i = h ϕ (x i ): min ξ i L c (q ξ (z i ) , c i ), where L c is the concept prediction loss. Note that, herein, an essential design choice explored in our experiments is the (non)linearity of the probe. Consequently, the probing function can be used to interpret the activations in the intermediate layer and edit them.Step 2: Editing Representations Recall that we are given a data point (x, y) and concept values c ′ for which an intervention needs to be performed. Note that this c ′ ∈ C could correspond to the ground-truth concept values or reflect the beliefs of the human subject intervening on the model. Intuitively, we seek an activation vector z ′ , which is similar to z = h ϕ (x) and consistent with c ′ according to the previously learnt probing function q ξ : arg min z ′ d (z, z ′ ) , s.t. q ξ (z ′ ) = c ′ , where d is an appropriate distance function applied to the activation vectors from the intermediate layer.Throughout main experiments (Section 4), we utilise the Euclidean distance, which is frequently applied to neural network representations, e.g. see works by Moradi Fard et al. ( 2020) and Jia et al. (2021) . In Appendix F.8, we additionally explore the cosine distance. Instead of the constrained problem above, we resort to minimising a relaxed objective:EQUATIONwhere, similarly to the counterfactual explanations (Wachter et al., 2017; Mothilal et al., 2020) , hyperparameter λ > 0 controls the tradeoff between the intervention's validity, i.e. the "consistency" of z ′ with the given concept values c ′ according to the probe, and proximity to the original activation vector z. In practice, we optimise z ′ for batched interventions using Adam (Kingma & Ba, 2015) . Appendix F.2 explores the effect of λ on the post-intervention distribution of representations.Step 3: Updating Output The edited z ′ can be consequently fed into g ψ to compute the updated output ŷ′ = g ψ (z ′ ), which could be then returned and displayed to the human subject. For example, if c ′ are the ground-truth concept values, we would ideally expect a decrease in the prediction error for the given data point (x, y).Concept bottlenecks (Koh et al., 2020) and their extensions are often evaluated empirically by plotting test-set performance or error attained after intervening on concept subsets of varying sizes. Ideally, the model's test-set performance should improve when given more ground-truth attribute values. Below, we formalise this notion of intervention effectiveness, referred to as intervenability, for the concept bottleneck and black-box models.For a trained CBM f θ (x) = g ψ (h ϕ (x)) = g ψ (ĉ), where ĉ are the predicted concept values, we define the intervenability as follows:EQUATIONwhere D is the joint distribution over the covariates x, concepts c, and targets y, L y is the target prediction loss, e.g. the mean squared error (MSE) or cross-entropy (CE), and π denotes a distribution over edited concept values c ′ . Observe that Equation 2 generalises the standard evaluation strategy of intervening on a random concept subset and setting it to the ground-truth values, as proposed in the original work by Koh et al. (2020) . Here, the effectiveness of interventions is quantified by the gap between the regular prediction loss and the loss attained after the intervention: the larger the gap between these values, the stronger the effect interventions have. The intervenability measure is loosely related to permutation-based variable importance and model reliance (Fisher et al., 2019) . We provide a discussion of this relationship in Appendix C.Note that the definition in Equation 2can also accommodate more sophisticated intervention strategies, for example, similar to those studied by Shin et al. (2023) and Sheth et al. (2022) . An intervention strategy can be specified via the distribution π, which can be conditioned on x, ĉ, c, ŷ, or even y: π (c ′ |x, ĉ, c, ŷ, y). The set of conditioning variables may vary across application scenarios.For brevity, we will use π as a shorthand notation for this distribution. Lastly, notice that, in practice, when performing human-or application-grounded evaluation (Doshi-Velez & Kim, 2017) , sampling from π may be replaced with the interventions by a human. Algorithms E.1 and E.2 provide concrete examples of the strategies utilised in our experiments.Leveraging the intervention procedure described in Section 3.1, analogous to Equation 2, the intervenability for a black-box neural network f θ at the intermediate layer given by ⟨g ψ , h ϕ ⟩ isE (x,c,y)∼D, c ′ ∼π [L y (f θ (x) , y) -L y (g ψ (z ′ ) , y)] ,where z ′ ∈ arg min z λL c (q ξ (z) , c ′ ) + d (z, z) .(3)Recall that q ξ is the probe trained to predict c based on the activations h ϕ (x) (step 1, Section 3.1). Furthermore, in the first line of Equation 3, edited representations z ′ are a function of c ′ , as defined by the second line, which corresponds to step 2 of the intervention procedure (Equation 1).Since the intervenability measure defined in Equation 3 is differentiable, a neural network can be fine-tuned by explicitly maximising it using, for example, mini-batch gradient descent. We expect fine-tuning for intervenability to reinforce the model's reliance on the high-level attributes and have a regularising effect. In this section, we provide a detailed description of the fine-tuning procedure (Algorithm B.1, Appendix B), and, afterwards, we demonstrate its practical utility empirically.Naïvely optimising intervenability from Equation 3 may decrease the predictive performance. Therefore, to fine-tune an already trained black-box model f θ , we combine the intervenability term with the target prediction loss, which amounts to the following optimisation problem:min ψ,z ′ E (x,c,y)∼D, c ′ ∼π L y g ψ (z ′ ) , y , s.t. z ′ ∈ arg min z λL c (q ξ (z) , c ′ ) + d (z, z) . (4)Notably, Equation 4 can be generalised by introducing a weight for the intervenability term:EQUATIONwhere β ∈ (0, 1] is the aforementioned weight. Note that for β = 1, the optimisation simplifies to Equation 4. For simplicity, we treat the probe's parameters ξ as fixed. However, since the outer optimisation problem is defined w.r.t. ϕ, ideally, the probe would need to be optimised as the third, inner-most level. By contrast, in the simplified setting under β = 1 (Equation 4), the parameters of h ϕ do not need to be optimised, and, hence, the probing function can be left fixed, as activations z are not affected by the fine-tuning. We consider this case to (i) computationally simplify the problem, avoiding trilevel optimisation, and (ii) keep the network's representations unchanged after fine-tuning for purposes of transfer learning for other downstream tasks. In practice, fine-tuning is performed by intervening on batches of data points. Since interventions can be executed online using a GPU (within seconds), our approach is computationally feasible.i,k = 1 {[h(Xi,:)] k ≥m k } , where m k = median [h (X l,: )] k N l=1 , for 1 ≤ i ≤ N and 1 ≤ k ≤ K + J.4. Let c i = u i,1:K and r i = u i,(K+1):(K+J) for 1 ≤ i ≤ N . 
experimental setup: 
 Datasets We evaluate the proposed methods on synthetic and real-world benchmarks summarised in Table D .1 (Appendix D). Across all experiments, fine-tuning has been performed exclusively on the validation data, and evaluation-on the test set. Further details can be found in Appendix D.For controlled experiments, we have adapted the nonlinear synthetic tabular dataset from Marcinkevičs et al. (2024) . We consider two data-generating mechanisms shown in bottleneck, and incomplete. The first scenario directly matches the inference graph of the vanilla CBM. The incomplete is a scenario with incomplete concepts, where c does not fully explain the association between x and y, with unexplained variance modelled via a residual connection.Another benchmark we consider is the Animals with Attributes 2 (AwA2) natural image dataset (Lampert et al., 2009; Xian et al., 2019) . It includes animal images accompanied by 85 binary attributes and species labels. To further corroborate our findings, we perform experiments on the Caltech-UCSD Birds-200-2011 (CUB) dataset (Wah et al., 2011) (Appendix D.3) , adapted for the CBM setting as described by Koh et al. (2020) . We report the CUB results in Appendix F.4.To investigate settings without human-annotated concept values, we evaluate our method on CIFAR-10 ( Krizhevsky et al., 2009) and the large-scale ImageNet (Russakovsky et al., 2015) natural image datasets. Following the previous literature (Oikarinen et al., 2023) , we use concepts generated by GPT-3. Concept labels are produced based on CLIP (Radford et al., 2021) similarities between each image and verbal descriptions. We utilise 143 attributes for CIFAR-10 and 100 for ImageNet.ImageNet results are reported in Appendix F.5.Finally, we explore a practical setting of chest radiograph classification. Namely, we test the techniques on public MIMIC-CXR (Johnson et al., 2019) and CheXpert (Irvin et al., 2019) datasets from the Beth Israel Deaconess Medical Center, Boston, MA, and Stanford Hospital. Both datasets have 14 binary attributes extracted from radiologist reports. In our analysis, the Finding/No Finding attribute is the target variable, and the remaining labels are the concepts, similar to Chauhan et al. (2023) . For simplicity, we retain a single X-ray per patient, excluding data with uncertain labels. The results on CheXpert are similar to those on MIMIC-CXR and can be found in Appendix F.6.Baselines & Methods Below, we briefly outline the neural network models and fine-tuning techniques compared. All methods were implemented using PyTorch (v 1.12.1) (Paszke et al., 2019) . Appendix E provides additional details. The code is available in a repository at https://github.com/sonialagunac/Beyond-CBM.Firstly, we train a standard neural network (BLACK BOX) without concept knowledge, i.e. on the dataset of tuples {(x i , y i )} i . We utilise our technique for intervening post hoc by training a probe to predict concepts and editing the network's activations (Equation 1, Section 3.1). All experiments reported in Section 5 use a linear probe, while the nonlinearity is explored in Appendix F. As an interpretable baseline, we consider the vanilla concept bottleneck model (CBM) by Koh et al. (2020) . Across all experiments, we restrict ourselves to the joint bottleneck version, which minimises the weighted sum of the target and concept prediction losses:min ϕ,ψ E (x,c,y)∼D [L y (f θ (x) , y) + αL c (h ϕ (x) , c)],where α > 0 is a hyperparameter controlling the tradeoff between the two loss terms. Finally, as the primary method of interest, we apply our fine-tuning for intervenability technique (FINE-TUNED, I; Equation 4, Section 3.3) on the annotated validation set {(x i , c i , y i )} i .In addition, as a common-sense baseline, we fine-tune the black box by training a probe to predict the concepts from intermediate representations (FINE-TUNED, MT). This amounts to multitask (MT) learning with hard weight sharing (Ruder, 2017) . Specifically, the model is fine-tuned by minimising the following MT loss: min ϕ,ψ,ξ E (x,c,y)∼D [L y (f θ (x) , y) + αL c (q ξ (h ϕ (x)) , c)]. Interventions on this model are performed using the three-step approach introduced in Section 3.1.As another baseline, we fine-tune the black box by appending concepts to the network's activations (FINE-TUNED, A). At test time, unknown concept values are set to 0.5. To prevent overfitting and handle missingness, randomly chosen concept variables are masked during training. The objective is given by minψ E (x,c,y)∼D [L y (g ψ ([h ϕ (x) , c]) , y)],where g takes as input concatenated activation and concept vectors. Note that, for this baseline, the parameters ϕ remain fixed during fine-tuning.Last but not least, as a strong baseline resembling the approaches by Yuksekgonul et al. (2023) and Oikarinen et al. (2023) , we train a CBM post hoc (POST HOC CBM) using sequential optimisation. Our implementation follows the original methods by Yuksekgonul et al. (2023) and Oikarinen et al. (2023) , while adjusting some design choices to make the techniques more readily comparable. The optimisation comprises two steps:(i) ξ = arg min ξ E (x,c,y)∼D [L c (q ξ (h ϕ (x)) , c)],(ii) min ψ E (x,c,y)∼D [L y (g ψ (q ξ(h ϕ (x))), y)]. Additionally, we explore the impact of residual modelling (Yuksekgonul et al., 2023) in Appendix F.9. The architectures of individual modules were kept as similar as possible for a fair comparison across all techniques.Evaluation To compare the methods, we conduct interventions and analyse model performance under varying concept subset sizes. We report the areas under the receiver operating characteristic (AUROC) and precision-recall curves (AUPR) (Davis & Goadrich, 2006 ) since these performance measures provide a well-rounded summary over varying cutoff points and it might be challenging to choose a single cutoff in high-stakes decision areas. We utilise the Brier score (Brier, 1950) to gauge the accuracy of probabilistic predictions and, in addition, evaluate calibration. 
results: 
 Results on Synthetic Data Figures 3(f) and 4( Importantly, under an incomplete concept set, black-box classifiers are superior to the ante hoc CBM because not all concepts relevant to the target prediction are given. Moreover, fine-tuning for intervenability improves intervention effectiveness while maintaining the performance gap. This experiment suggests the superiority of our method in settings where the concept set does not capture all label-relevant information. Other fine-tuning strategies (FINE-TUNED, MT and FINE-TUNED, A) are either less effective or harmful, leading to a lower increase in AUROC and AUPR than attained by the untuned black box. Lastly, CBMs trained post hoc perform well in the simple bottleneck scenario, being only slightly less intervenable than FINE-TUNED, I. However, for the incomplete setting, interventions hurt the performance of the post hoc CBM. This behaviour may be related to the leakage (Havasi et al., 2022) and is not mitigated by residual modelling explored in Appendix F.9.To study the influence of the validation set size (N val ) on probing and fine-tuning, we perform ablations under the bottleneck scenario (Figure 3 ). For a fair comparison w.r.t. sample efficiency, here, we train a CBM on the dataset of the same size. While the effectiveness of interventions on FINE-TUNED, I is slightly hampered by smaller validation sets, the decrease is moderate. We observe impactful interventions with validation set sizes as small as 0.5% of the original one (Figure 3(a) ). Across all settings, our method remains superior to baselines. Importantly, our fine-tuning approach has a better sample efficiency than post hoc CBM, which exhibits a considerable dropoff in intervention effectiveness. Likewise, the performance of the CBMs decreases, suggesting their limited utility under smaller dataset sizes. Analogous results w.r.t. AUPR are reported in Appendix F.1.In Table 1 , we report the test-set performance of the models without interventions (under the bottleneck mechanism). For the concept prediction, CBM outperforms black-box models, even after fine-tuning with the MT loss. However, without interventions, all models attain comparable AUROCs and AUPRs at the target prediction. Interestingly, FINE-TUNED, I results in better-calibrated probabilistic predictions with lower Brier scores than those made by the original black box and after applying other fine-tuning strategies. As evidenced by Figure F.9(a) (Appendix F.7), fine-tuning has a regularising effect, reducing the false overconfidence observed in neural networks (Guo et al., 2017) .In the supplementary material, we report several additional findings. Figure F.2 (Appendix F.1) contains further ablations for the intervention procedure on the influence of the hyperparameters, intervention strategies, and probe. In addition, Appendix F.2 explores the effect of interventions on the distribution of representations. Lastly, in Appendix F.10, we show that the performance of the CBM on this dataset is not sensitive to the choice of the optimisation procedure (Koh et al., 2020) .Results on AwA2 Additionally, we explore the AwA2 dataset in Figure 4 (b). This is a simple classification benchmark with class-wide concepts helpful for predicting the target. Hence, CBMs trained ante and post hoc are highly performant and intervenable. Nevertheless, untuned black-box models also benefit from concept-based interventions. In agreement with our findings on the synthetic dataset and in contrast to the other fine-tuning methods, ours enhances the performance of blackbox models. Notably, black boxes fine-tuned for intervenability even surpass CBMs. Overall, the simplicity of this dataset leads to the generally high AUROCs and AUPRs across all methods.To further investigate the impact of hyperparameters on the interventions, we have performed ablation studies on untuned black boxes. These results are reported in Figures F.4 Figure F.4(b) compares two intervention strategies: randomly selecting concepts (random) and prioritising the most uncertain ones (uncertainty) (Shin et al., 2023) to intervene on (Algorithms E.1 and E.2, Appendix E). The strategy has an impact on the performance increase, with the uncertaintybased approach yielding a steeper improvement. Finally, Figure F.4(c) compares linear and nonlinear probes. Here, intervening via a nonlinear function leads to a higher performance increase.To show the efficacy of our methods across different backbone architectures, in Appendix F.3, we also explore AwA2 with the Inception (Szegedy et al., 2015) backbone (note that Figure 4 (b) reports the results on the ResNet-18 (He et al., 2016) ). Finally, Table 1 contains evaluation metrics at test time without interventions for target and concept prediction. We observe comparable performance across methods, which are all successful due to the large dataset size and the task simplicity. To demonstrate that our approaches are effective without human-annotated concepts (Yuksekgonul et al., 2023; Oikarinen et al., 2023) , we present the results on CIFAR-10 in Figure 4 (c). Here, concept labels were generated using a VLM. In addition, we explore the ImageNet in Appendix F.5. The results have been obtained using the backbone architecture of Stable Difusion (Rombach et al., 2022) . We do not include the CBM, as we cannot retrain such a large backbone due to computational constraints. By contrast, our method allows fine-tuning the pretrained network, thus being helpful where a CBM is impractical. As in the previous experiments, black boxes are, in principle, intervenable, and our fine-tuning approach outperforms other baselines.Application to Chest X-ray Classification To showcase the practicality of our approach, we present empirical findings on two chest X-ray datasets, MIMIC-CXR and CheXpert, primarily focusing on the former. Figure 4 (d) shows intervention curves across ten independent initialisations. Interestingly, untuned black-box neural networks are not intervenable. By contrast, after fine-tuning for intervenability, the model's predictive performance and effectiveness of interventions improve visibly and even surpass those of the CBM. Given the challenging nature of these datasets, black-box model predictions may not be as strongly reliant on the considered attributes. Moreover, CBMs do not necessarily outperform black-box networks, unlike in simpler benchmarking datasets. Finally, post hoc CBMs (even with residual modelling) exhibit a behaviour similar to the synthetic dataset with incomplete concepts: interventions have only a slight positive, no, or adverse effect on performance. Analogous findings for CheXpert can be found in Appendix F.6.i,k = 1 {[h(Xi,:)] k ≥m k } , where m k = median [h (X l,: )] k N l=1 , for 1 ≤ i ≤ N and 1 ≤ k ≤ K. 6. Let y i = 1 {g(ci)≥my} , where m y = median {g (c i )} N l=1 , for 1 ≤ i ≤ N .Incomplete Last but not least, to simulate the incomplete concept set scenario, where a part of concepts are latent, we slightly adjust the procedure from the bottleneck setting above:1. Follow steps 1-3 from the bottleneck procedure.2. Let h : R p → R K+J and g : R K+J → R be randomly initialised multilayer perceptrons with ReLU nonlinearities, where J is the number of unobserved concept variables.i = 1 {g(ui)≥my} , where m y = median {g (u i )} N l=1 , for 1 ≤ i ≤ N .Note that, in steps 3-5 above, u i corresponds to the concatenation of c i and r i . Across all experiments, we set J = 90. 
discussion & conclusion: 
 This work introduces a technique for performing instance-specific concept-based interventions on any pretrained neural network post hoc. We formalise a novel measure of intervenability as the effectiveness of concept-based interventions and propose a method leveraging it to fine-tune black-box models. In contrast to CBMs (Koh et al., 2020) , our method circumvents the need for concept labels during training, which can be a challenge in practical applications. Unlike recent works on converting black boxes into CBMs post hoc (Yuksekgonul et al., 2023; Oikarinen et al., 2023) , which generally do not explore instance-specific interventions, we propose an effective intervention method that is faithful to the original architecture and representations. Lastly, we introduce and study several other common-sense fine-tuning baselines that perform worse than the proposed method, highlighting the need for the explicit maximisation of intervenability.The utility of our method is highlighted empirically on synthetic tabular and natural image data. We show that, given a small annotated validation set, black-box models trained without explicit concept knowledge are intervenable. Moreover, our fine-tuning method improves the effectiveness of the interventions, with overall better results than alternative techniques. In addition, our approach is effective in scenarios where the concept labels are generated using VLMs. Thus, we can alleviate the need for costly human annotation while maintaining improved intervention effectiveness. Lastly, we apply the techniques in a more realistic setting of chest X-ray classification, where black boxes are not directly intervenable. The proposed fine-tuning procedure alleviates this limitation, while the other post hoc techniques are ineffective or even harmful.Limitations & Future Work Our work opens many avenues for future research and improvements.Firstly, the variant of the fine-tuning procedure considered in this paper does not affect the neural network's representations. However, it would be interesting to investigate the more general formulation wherein all model and probe parameters are fine-tuned end-to-end. According to our empirical findings, the choice of intervention strategy, hyperparameters, and probing function can influence the effectiveness of interventions. A deeper experimental investigation of these aspects is warranted. Furthermore, we only considered a single fixed intervention strategy throughout fine-tuning, whereas further improvement could come from learning an optimal strategy alongside fine-tuned weights.Beyond the current setting, we would like to apply our intervenability measure to evaluate and compare other large pretrained discriminative and also generative models. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: All results are reported as the summary statistics across ten independent experiments (seeds). The standard deviations are provided in all tables. In the plots, all the lines correspond to medians and confidence bands are given by interquartile ranges. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We provided information on which type of GPUs we used and the time of development of our method in the implementation details in Appendix E.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: The code of ethics was strictly followed.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: This paper presents work whose goal is to advance the field of Machine Learning, and we do not believe there are potential societal consequences of our work. Therefore, we feel that no extra statement needs to be specifically highlighted here.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: To our best understanding, our proposed method does not have generative capabilities nor any risk of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All external datasets used are publicly available and are accordingly cited and acknowledged, with their corresponding licences properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: We provide an in-depth explanation of the introduced methods as well as the algorithms used and procedures to generate new data.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This work does not include experiments nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This work does not include experiments nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: This appendix provides additional schematics and intuitive examples, clarifying instance-specific concept-based interventions. Algorithm B.1 contains the detailed pseudocode for fine-tuning for intervenability described in Section 3.3. Recall that the black-box model f θ is fine-tuned using a combination of the target prediction loss and intervenability defined in Equation 3. The implementation below applies to the special case of β = 1, which leads to the simplified loss. Importantly, in this case, the parameters ϕ are treated as fixed, and the probing function q ξ does not need to be fine-tuned alongside the model. Lastly, note that, in Algorithm B.1, interventions are performed for whole batches of data points x b using the procedure described in Section 3.1. ξ ′ N i=1 L c (q ξ ′ (h ϕ (x i ) , c i )) ▷ Step 1:Probingfor e = 0 to E -1 do Randomly split {1, ..., N } into mini-batches of size M given by B for b ∈ B do z b ← h ϕ (x b ) ŷb ← g ψ (z b ) ĉb ← q ξ (z b ) Sample c ′ b ∼ π Initialise z ′ b = z b , z ′ b,old = z b + ε I e, and e I = 0 ▷ Step 2: Editing Representationswhile z ′ b -z ′ b,old 1 ≥ ε I and e I < E I do z ′ b,old ← z ′ b 12 z ′ b ← z ′ b -η I ∇ z ′ b [d(z b , z ′ b ) + λL c (q ξ (z ′ b ) , c ′ b )] ▷ Equation 1 e I ← e I + 1 end ŷ′ b ← g ψ (z ′ b ) ▷ Step 3: Updating Output ψ ← ψ -η∇ ψ L y (ŷ ′ b , y b ) ▷ Equation 4 end end return f θThis appendix contains extended remarks and discussion beyond the scope of the main text.Design Choices for the Intervention Procedure The intervention procedure entails a few design choices, including the (non)linearity of the probing function, the distance function in the objective from Equation 1, and the tradeoff between consistency and proximity determined by λ from Equation 1. We have explored some of these choices empirically in our ablation experiments (see Figure F.4 and Appendix F) . Naturally, interventions performed on black-box models using our method are meaningful in so far as the activations of the neural network are correlated with the given high-level attributes and the probing function q ξ can be trained to predict these attribute values accurately. Otherwise, edited representations and updated predictions are likely to be spurious and may harm the model's performance.Should All Models Be Intervenable? Intervenability (Equation 3), in combination with the probing function, can be used to evaluate the interpretability of a black-box predictive model and help understand whether (i) learnt representations capture information about given human-understandable attributes and whether (ii) the network utilises these attributes and can be interacted with. However, a black-box model does not always need to be intervenable. For instance, when the given concept set is not predictive of the target variable, the black box trained using supervised learning should not and probably would not rely on the concepts. On the other hand, if the model's representations are nearly perfectly correlated with the attributes, providing the ground truth should not significantly impact the target prediction loss. Lastly, the model's intervenability may depend on the chosen intervention strategy, which may not always lead to the expected decrease in the loss.Intervenability & Variable Importance As mentioned in Section 3.2, intervenability (Equation 2) measures the effectiveness of interventions performed on a model by quantifying a gap between the expected target prediction loss with and without performing concept-based interventions. Equation 2 is reminiscent of the model reliance (MR) (Fisher et al., 2019) used for quantifying variable importance.Informally, for a predictive model f , MR measures the importance of some feature of interest and is defined asM R (f ) • • = expected loss of f under noise expected loss of f without noise . (C.1)Above, the noise augments the inputs of f and must render the feature of interest uninformative of the target variable. One practical instance of the model reliance is permutation-based variable importance (Breiman, 2001; Molnar, 2022) .The intervenability measure in Equation 2 can be summarised informally as the difference between the expected loss of g ψ without interventions and the loss under interventions. Suppose intervention strategy π is specified so that it augments a single concept in ĉ with noise (Equation C.1). In that case, intervenability can be used to quantify the reliance of g ψ on the concept variable of interest in ĉ. The main difference is that Equation C.1 is given by the ratio of the expected losses, whereas intervenability looks at the difference of expectations.We can draw a relationship between the concept-based interventions (Equation 3) and conceptual counterfactual explanations (CCE) studied by Abid et al. (2022) and S. Kim et al. (2023) . In brief, interventions aim to "inject" concepts c ′ provided by the user into the network's representation to affect and improve the downstream prediction. By contrast, CCEs seek to identify a sparse set of concept variables that could be leveraged to flip the label predicted by the classifier f θ . Thus, the problem tackled in the current work is different from and complementary to CCE.More formally, following the notation from Section 1, a conceptual counterfactual explanation (Abid et al., 2022) is given byarg min w L y g ψ h ϕ (x) + w C , y ′ + α ∥w∥ 1 + β ∥w∥ 2 , s.t. w min ≤ w ≤ w max , (C.2)where C is the concept bank, y ′ is the given target value (in classification, the opposite to the predicted ŷ), α, β > 0 are penalty weights, and w min , w max defines the desired range for weights w. Note that further detailed constraints are imposed via the definition of w min , w max in the original work by Abid et al. (2022) .Observe that the optimisation problem in Equation C.2 is defined w.r.t. the flipped label y ′ and does not incorporate user-specified concepts c ′ as opposed to interventions in Equation 1. Thus, CCEs aim to identify the concept variables that need to be "added" to flip the label output by the classifier. In contrast, interventions seek to perturb representations consistently with the given concept values.Below, we present further details about the datasets and preprocessing involved in the experiments (Section 4). The synthetic data can be generated using our code. AwA2, CUB, CIFAR-10, ImageNet, CheXpert, and MIMIC-CXR datasets are publicly available. Table D .1 provides a brief summary. (Koh et al., 2020) . In the incomplete scenario, we are given incomplete concepts, i.e. c does not fully explain the variance in y. Here, unexplained variance is modelled as a latent variable r via the path x → r → y.Unless mentioned otherwise, we mainly focus on the simplest scenario shown in Figure D .1(a). Below, we outline each generative process in detail. Throughout this appendix, let N , p, and K denote the number of independent data points {(x i , c i , y i )} N i=1 , covariates, and concepts, respectively. Across all experiments, we set N = 50,000, p = 1,500, and K = 30. This dataset was divided according to the 60%-20%-20% train-validation-test split. Bottleneck In this setting, the covariates x i generate binary-valued concepts c i ∈ {0, 1} K , and the binary-valued target y i depends on the covariates exclusively via the concepts. The generative process is as follows:1. Randomly sample µ ∈ R p s.t. µ j ∼ Uniform (-5, 5) for 1 ≤ j ≤ p.2. Generate a random symmetric, positive-definite matrix Σ ∈ R p×p .3. Randomly sample a design matrix X ∈ R N ×p s.t. X i,: ∼ N p (µ, Σ).2 4. Let h : R p → R K and g : R K → R be randomly initialised multilayer perceptrons with ReLU nonlinearities.Animals with Attributes 23 dataset (Lampert et al., 2009; Xian et al., 2019) serves as a natural image benchmark in our experiments. It comprises 37,322 images of 50 animal classes (species), each associated with 85 binary attributes utilised as concepts. An apparent limitation of this dataset is that the concept labels are shared across whole classes, similar to the Caltech-UCSD Birds experiment from the original work by Koh et al. (2020) . Thus, AwA2 offers a simplified setting for transfer learning across different classes and is designed to address attribute-based classification and zero-shot learning challenges. In our evaluation, we used all the images in the dataset without any specialised preprocessing or preselection. All images were rescaled to 224 × 224 pixels. This dataset was divided according to the 60%-20%-20% train-validation-test split.Caltech-UCSD Birds-200-20114 dataset (Wah et al., 2011) is another natural image benchmark explored in the original work on CBMs by Koh et al. (2020) . It consists of 11,788 bird photographs from 200 species (classes) and originally includes 312 concepts, such as wing colour, beak shape, etc. We have followed the preprocessing routine proposed by Koh et al. (2020) and keep the original train-validation-test split to avoid spurious mixing of photographers in the data. Particularly, the final dataset includes only the 112 most prevalent binary attributes. We have included image augmentations during training, such as random horizontal flips, adjustments of the brightness and saturation, and normalisation. Similar to AwA2, CUB concepts are shared across all instances of individual classes. No additional specialised preprocessing was performed on the images, which were rescaled to a resolution of 224 × 224 pixels.CIFAR-105 (Krizhevsky et al., 2009) is a benchmarking natural image dataset. It includes 60,000 32×32 colour images in 10 classes, with 6,000 images per class. There are 50,000 training and 10,000 test images. To generate the validation set, we randomly hold out 10,000 images from the training data to remain faithful to the original test set. Following the setup by Oikarinen et al. (2023) , we generate 143 concept labels as described in Section 4 using VLMs by comparing the similarities between each instance and the concept text embedding with thus of not the concept. We apply random horizontal flips, adjustments to brightness and saturation, resize the images to a resolution of 128×128 pixels, and apply normalisation.ImageNet6 dataset (Russakovsky et al., 2015) is a large-scale natural image benchmark. It includes 1,000 object classes and contains 1,281,167 training, 50,000 validation, and 100,000 unlabelled test images. In our experiments, we allocate half of the validation as the test set. We apply random horizontal flips, adjustments to brightness and saturation, resize images to a resolution of 128×128 pixels, and apply normalisation.We adapt the 4,751 original concept variables introduced using GPT-3 by Oikarinen et al. (2023) . To ensure that concepts are predictive of the target variable and can be inspected manually, we retain 100 attributes. Specifically, we keep those with the highest correlations with the final target while prioritising their diversity. To this end, we cluster concepts into 25 groups using the k-means algorithm and sample 4 attributes from each cluster based on Cramér's V (Cramér, 1999) between the concept variable and y. Final concept labels were generated using CLIP as for CIFAR-10(Section 4).As mentioned, we conducted an empirical evaluation on two real-world chest X-ray datasets: CheXpert (Irvin et al., 2019) and MIMIC-CXR (Johnson et al., 2019) . The former includes over 220,000 chest radiographs from 65,240 patients at the Stanford Hospital.7 These images are accompanied by 14 binary attributes extracted from radiologist reports using the CheXpert labeller (Irvin et al., 2019) , a model trained to predict these attributes. MIMIC-CXR is another publicly available dataset containing chest radiographs in DICOM format, paired with free-text radiology reports.8 It comprises more than 370,000 images associated with 227,835 radiographic studies conducted at the Beth Israel Deaconess Medical Center, Boston, MA, involving 65,379 patients. Similar to CheXpert, the same labeller was employed to extract the same set of 14 binary labels from the text reports. Notably, some concepts may be labelled as uncertain. Similar to Chauhan et al. (2023) , we designate the Finding/No Finding attribute as the target variable for classification and utilise the remaining labels as concepts. In particular, the concepts are atelectasis, cardiomegaly, consolidation, edema, enlarged cardiomediastinum, fracture, lung lesion, lung opacity, pleural effusion, pleural other, pneumonia, pneumothorax, and support devices. In our implementation, we remove all the samples that contain uncertain labels and discard multiple visits of the same patient, keeping only the last acquired recording per subject for both datasets. All images were cropped to a square aspect ratio and rescaled to 224 × 224 pixels. Additionally, augmentations were applied during training, namely, random affine transformations, including rotation up to 5 degrees, translation up to 5% of the image's width and height, and shearing with a maximum angle of 5 degrees. We also include a random horizontal flip augmentation to introduce variation in the orientation of recordings within the dataset. Both chest radiograph datasets are divided according to the 80%-10%-10% train-validation-test split.This section provides implementation details, such as network architectures and intervention and fine-tuning procedure hyperparameter configurations. All models and procedures were implemented using PyTorch (v 1.12.1) (Paszke et al., 2019) and scikit-learn (v 1.0.2) (Pedregosa et al., 2011) . We run the reported experiments on a cluster of GeForce RTX 2080 GPUs with a single CPU worker. The span of time elapsed to run each method is dependent on the dataset and architecture. On the synthetic tabular data, on average, it takes approx. 3 hours to train a concept bottleneck or black-box model. For the Animals with Attributes 2 and chest X-ray datasets, it takes up to 10 hours to train black boxes and CBMs. However, when using a pretrained backbone, e.g. Stable Diffusion, only fine-tuning is required, the run-time of which ranges from 10 minutes to 1 hour for all considered datasets.Network & Probe Architectures For the synthetic tabular data, we utilise a fully connected neural network (FCNN) as the black-box model. Its architecture is summarised in Table E .1 in PyTorch-like pseudocode. For this classifier, probing functions are trained, and interventions are performed on the activations of the third layer, i.e. the output after line 2 in Table E .1. For AwA, CUB, MIMIC-CXR, and CheXpert, we use the ResNet-18 (He et al., 2016) with random initialisation followed by four fully connected layers and the sigmoid or softmax activation. Probing and interventions are performed on the activations of the second layer after the ResNet-18 backbone. Furthermore, we provide results for AwA with the Inception (Szegedy et al., 2015) backbone. For CIFAR-10 and ImageNet, we showcase the scalability of our methods on the pretrained Stable Diffusion Rombach et al. ( 2022) backbone followed by four fully connected layers and the sigmoid or softmax activation. For the CBMs, to facilitate fair comparison, we use the same architectures with the exception that the layers mentioned above were converted into bottlenecks with appropriate dimensionality and activation functions. Similar settings are used for post hoc CBMs with the addition of a linear layer mapping backbone representations to the concepts.For fine-tuning, we utilise a single fully connected layer with an appropriate activation function as a linear probe and a multilayer perceptron with a single hidden layer as a nonlinear function. For evaluation on the test set (Table 1 ), we fit a logistic regression classifier from scikit-learn as a linear probe. The logistic regression is only used for evaluation purposes and not interventions. Interventions Unless mentioned otherwise, interventions on black-box models were performed using linear probes, the random-subset intervention strategy, and under λ = 0.8 (Equation 1). Recall that Figures F.4 and F.2 provide ablation results on the influence of the latter hyperparameter. Despite some variability, the analysis shows that higher values of λ expectedly lead to more effective interventions. The choice of λ for our experiments was meant to represent the "average case", and no tuning was performed for this hyperparameter.Similarly, we have mainly used a linear probing function and the simple random-subset intervention strategy to provide proof-of-concept results without extensive optimisation of the intervention strategy or the need for nonlinear probing. Thus, our primary focus was on demonstrating the intervenability of black-box models and showcasing the effectiveness of the fine-tuning method rather than an exhaustive hyperparameter search.In ablation studies, we compare two intervention strategies (Figure F.4) inspired by Shin et al. (2023) : (i) random-subset and (ii) uncertainty-based. Herein, we provide a more formal definition of these procedures described as pseudocode in Algorithms E.1-E.2. Recall that given a data point (x, c, y) and predicted values ĉ and ŷ, an intervention strategy defines a distribution over intervened concept values c ′ . Random-subset strategy (Algorithm E.1) replaces predicted values with the ground truth for several concept variables (k) chosen uniformly at random. By contrast, the uncertainty-based strategy (Algorithm E.2) samples concept variables to be replaced with the ground-truth values without replacement with initial probabilities proportional to the concept prediction uncertainties, denoted by σ. In our experiments, the components of ĉ are the outputs of the sigmoid function, and the uncertainties are computed as σ i = 1/ (|ĉ i -0.5| + ε) (Shin et al., 2023) for 1 ≤ i ≤ K, where ε > 0 is small.Input: A data point (x, c, y), predicted concept values ĉ, the number of concept variables to be intervened on1 ≤ k ≤ K Output :Intervened concept values c ′ c ′ ← ĉ Sample I uniformly at random from {S ⊆ {1, . . . , K} : |S| = k} c ′ I ← c I return c ′ Algorithm E.2: Uncertainty-based Intervention Strategy Input: A data point (x, c, y), predicted concept values ĉ, the number of concept variables to be intervened on 1 ≤ k ≤ K Output :Intervened concept values c ′ σ j ← 1/ (|ĉ j -0.5| + ε) for 1 ≤ j ≤ K, where ε > 0 is small σ ← (σ 1 • • • σ K ) c ′ ← ĉ Sample k indices I = {i j } k j=1 s.t. each i j is sampled without replacement from {1, . . . , K} with initial probabilities given by (σ + ε) / Kε + K i=1 σ i , where ε > 0 is small c ′ I ← c I return c ′ Fine-tuning for Intervenability The fine-tuning procedure outlined in Section 3.3 and detailed in Algorithm B.1 necessitates intervening on the representations throughout the optimisation. During fine-tuning, we utilise the random-subset intervention strategy, i.e. interventions are performed on a subset of the concept variables by providing the ground-truth values. More concretely, interventions are performed on 50% of the concept variables chosen uniformly at random.The baseline methods described in Section 4 incorporate concept information in distinct ways. On the one hand, the multitask learning approach, FINE-TUNED, MT, utilises the entire batch of concepts at each iteration during fine-tuning. For this procedure, we set α = 1.0 (recall that α controls the tradeoff between the target and concept prediction loss terms). On the other hand, the FINE-TUNED, A approach, which appends the concepts to the network's activations, does not use the complete concept set for each batch. In particular, before appending, concept values are randomly masked and set to 0.5 with a probability of 0.5. This practical trick is reminiscent of the dropout (Srivastava et al., 2014) and is meant to help the model remain intervenable and handle missing concept values.Hyperparameters Below, we list key hyperparameter configurations; the remaining details are documented in our code. For the synthetic data, CBMs and black-box classifiers are trained for 150 and 100 epochs, respectively, with a learning rate of 10 -4 and a batch size of 64. Across all other experiments, CBMs are trained for 350 epochs and black-box models for 300 epochs with a learning rate of 10 -4 halved midway through training and a batch size of 64. CBMs are trained using the joint optimisation procedure (Koh et al., 2020) under α = 1.0, where α controls the tradeoff between the concept and target prediction losses. All probes were trained on the validation data for 150 epochs with a learning rate of 10 -2 using the stochastic gradient descent (SGD) optimiser. Finally, all fine-tuning procedures were run for 150 epochs with a learning rate of 10 -4 and a batch size of 64 using the Adam optimiser. ImageNet is an exception to the above configurations due to its large size. The black-box models in this dataset were trained for 60 epochs, and the probes and fine-tuning procedures for 20 epochs. At test time, interventions were performed on batches of 512 data points.This section contains supplementary results and ablation experiments not included in the main body of the text. In some cases (Abid et al., 2022) , it may be deemed desirable that intervened representations z ′ (Equation 1) remain plausible, i.e. their distribution should be close to that of the original representations z. and AUPR. Figure F.4(b) compares two intervention strategies: randomly selecting a concept subset (random) and prioritising the most uncertain concepts (uncertainty) (Shin et al., 2023) to intervene on (Algorithms E.1 and E.2, Appendix E). The intervention strategy has a clear impact on the performance increase, with the uncertainty-based approach yielding a steeper improvement. Finally, Figure F.4(c) compares linear and nonlinear probes. Here, intervening via a nonlinear function leads to a significantly higher performance increase.Finally, to show the scalability of our methods to different backbone architectures, we report in Figure F .5 results with the Inception (Szegedy et al., 2015) backbone. As can be seen, the intervention procedure and our fine-tuning method remain successful regardless of the backbone architectures. In line with the previous literature (Koh et al., 2020) , we assess our approach on the CUB dataset with the results summarised in Figure F.6. This dataset is similar to the AwA2, as the concepts are shared across whole classes. Thus, concepts and classes feature a strong and simple correlation structure. Expectedly, the CBM performs very well due to its inductive bias in relying on the concept variables.As in the previous simpler scenarios, untuned black boxes are, in principle, intervenable. However, the proposed fine-tuning strategy considerably improves the effectiveness of interventions. On this dataset, the performance (without interventions) of the post hoc CBM and the model fine-tuned for intervenability is visibly lower than that of the untuned black box. We attribute this to the poor association between the concepts and the representations learnt by the black box. Interestingly, post hoc CBMs do not perform as successfully as the models fine-tuned for intervenability. Generally, the behaviour of the models on this dataset falls in line with the findings described in the main body of the text and supports our conclusions. To further support the findings on CIFAR-10 using CLIP-based concept annotations, we explore the intervention effectiveness of our method in the large-scale ImageNet dataset. In Figure F.7 we show how the proposed fine-tuning method improves the intervention effectiveness when compared with the studied baselines. Note that the CBM is not computed due to the constraint of retraining the Stable Diffusion backbone from scratch for the concept bottleneck adaptation. To further showcase the practicality of our approach, we present empirical findings on the CheXpert dataset, which are complementary to the MIMIC-CXR results included in Section 5. The fine-tuning approach introduced leads to better-calibrated predictions (Table 1 ), possibly, due to the regularising effect of intervenability. In this section, we further support this finding by visualising calibration curves for the binary classification tasks, namely, for the synthetic tabular data and chest radiograph datasets. Figure F.9 shows calibration curves for the fine-tuned model, untuned black box, and CBM averaged across ten seeds. We have omitted fine-tuning baselines in the interest of legibility since their predictions were comparably ill-calibrated as for the black box. The fine-tuned model consistently and considerably outperforms both the untuned black box and the CBM in all three binary classification tasks, as its curve is the closest to the diagonal, which corresponds to perfect calibration. These results suggest that the intervention procedure is, indeed, sensitive to the choice of the distance function, and we hypothesise that the distance should be chosen to suit the latent space of the neural network considered. Encouragingly, the proposed fine-tuning procedure is equally effective under the cosine distance. Similar to the Euclidean case, it greatly improves the model's intervenability.F.9 Additional Baselines: Post Hoc CBMs with Residual ModellingAs an extension of post hoc CBMs, we consider residual modelling as described by Yuksekgonul et al. (2023) . We refer to this baseline as POST HOC CBM-H, i.e. hybrid post hoc CBM. This variant adds a residual connection between the network's representations and the final output. In particular, after sequential optimisation of the post hoc CBM parameters (Section 4), a residual predictor r ζ is trained and added to the model's output:min ζ E (x,c,y)∼D [L y (g ψ (q ξ(h ϕ (x))) + r ζ (h ϕ (x)) , y)].Similar to Yuksekgonul et al. (2023) , we utilise a linear residual predictor in our experiments. 2020): independent, sequential, and joint. We show in both synthetic and MIMIC-CXR datasets that the results are comparable, and therefore, throughout the manuscript, we have focused solely on the joint optimisation procedure. We chose two datasets for this ablation to span the simpler synthetic tabular scenario and the more realistic chest X-ray classification, where the concepts and final target may have a more complex dependency. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We provide the links and citations of all the used public datasets and the code to generate our synthetic dataset. Additionally, the code used to run the experiments and our method is provided in an anonymised repository. Guidelines:• The answer NA means that paper does not include experiments requiring code. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.Xi,: refers to the i-th row of the design matrix, i.e. the covariate vector xihttps://cvml.ista.ac.at/AwA2/https://www.vision.caltech.edu/datasets/cub_200_2011/https://www.cs.toronto.edu/~kriz/cifar.htmlhttps://www.image-net.org/update-mar-11-2021.phphttps://stanfordmlgroup.github.io/competitions/chexpert/https://physionet.org/content/mimic-cxr/2.0.0/ 


Decision: All reviewers find the proposed approach to make a pre-trained black box model interpretable very interesting.
All reviewers clearly vote for accepting this paper.

Reviews: review: paper_summary: paper_summary: The paper proposes a method to make any pre-trained black box model intervenable, given a small validation dataset with concept.

This is done by the following procedure:
- Extract the output of a layer of a black-box $g_{	\psi} (z)$ and train a probing network $q_{\xi}(z)$ to extract the concepts from that layer using the validation dataset.
- For intervening i.e changing concepts $c$ to $c'$: Given $z$ find $z'$ that minimizes $L^c(q_{\xi}(z'),c')+d(z,z')$ where $d$ is a distance measure. $z'$ is optimized using gradient descent.
- Train the black-box network to be intervenable by finetuning, minimize $L^y(g_{\psi}(z'),y)$ 
- To edit the representations,  replace $z'$ with $z$.

The paper demonstrated the effectiveness of their approach by comparing it to the following baselines:
- A black-box with representation editing and no finetuning.
- CBM
- A finetune black-box that predicts the concepts and the output (Multitask-learning)
- A finetune black-box by appending the black box activations with the concepts.
- Post-hoc CBMs

The approach was evaluated on the following datasets:
- A synthetic tabular dataset with a complete and incomplete concept dataset.
- Vision datasets with labels: AWA2 and CUB
- Vision datasets with no labels: Cifar10 and imageNet (Here labels are obtained from an LLM in a label-free style).

Results that the proposed method outperforms others in cases where data is limited i.e the % of validation dataset is low or when the number of interventions is generally low.
strength_weakness: strength_weakness: ### Significance -- Fair

- There are two main advantages of using CBMs, interpretability and intervenability. While the proposed method allows for intervenability, this method does not add interpretability to black box model.

- One could argue that since you need a validation set with concept anyhow, one should just put a CBM in the original architecture giving both interpretability and intervenability. Empirical validation in the paper did show that the proposed method did better than CBM however baseline CBM used in the evaluation was not trained for intervenability as done by Mateo, et al. [1] which could dramatically improve the performance of regular CBMs. So its difficult to conclude that the proposed method is in fact superior to CBMs in any way.



[1] Espinosa Zarlenga, Mateo, et al. "Learning to receive help: Intervention-aware concept embedding models." Advances in Neural Information Processing Systems 36 (2023).
questions: questions: - Which layer of is used for interventions, does it matter? 
- For the multi-task baseline, Fine-Tune MT how do you intervene on test time since changing the multitask label does not affect the output of the classifier?
- For the appending baseline Fine-Tune A do you throw away the last layer of the original network (i.e the layer after the one used for interventions) and create a new one to handle the new size?
limitations: limitations: Yes.
paper_summary: paper_summary: The paper proposes a novel method for test-time concept-based intervention on black-box models. Starting from the interactive intervention procedure of concept-bottleneck models, the authors devise a novel technique for reproducing it on any pre-trained black-box model. It consists of training a concept probe on the latent space of the model and using it to modify the representation of a given concept and checking the response of the model to this intervention. Also, the authors define a measure of intervenability which can be optimized to fine-tune a model architecture, while making it more prone to concept-intervention. The proposed approach is tested over several benchmarks.
strength_weakness: strength_weakness: ## Major issues
- **Presentation**: the paper is not very clear in a few passages. Also, some comments to the results in the experimental sections are excessive or not completely true, and they must be changed/softened. See minor issues for indications of the individual sentences to edit.
- **Equation 4**: presenting Eq. 4 and then only considering the special case of $\beta=1$ is not correct, as it completely neglects a term. This is not acceptable even if it implies the training of the probe, a reader expects to see an ablation study on that in the experiment. I solicit the authors to present it otherwise (e.g., presenting first the actual optimized equation) or provide an ablation study on that. 

## Minor issues
- In the abstract, “Recently, interpretable machine learning has re-explored concept bottleneck models (CBM)”. The sentence is not very clear, it seems in this way that concept bottleneck models have already been proposed in the past. You also say it in the introduction. If it is the case, and it has been proposed earlier, you should provide a citation.
- “While, in principle, a specialist may just override predictions, in many cases, concept and target variables are linked via nonlinear relationships potentially unknown to the user.” This sentence is not super clear and does not seem to support the paper point. I would consider rephrasing or removing it. 
- Figure 1: The concepts reported are not clear. Another example or the same example with textual concepts instead of icons would be probably easier to read.
- In the Method, “we will assume being given a small validation set”, I would explicitly add “equipped with concept annotations”. 
- “think of a doctor trying to interact with a chest X-ray classifier $(f_\theta)$ by annotating their findings”, which findings? Not very clear, why don’t you directly provide an example of concept for the X-ray task?
- “Note that, herein, an essential design choice explored in our experiments is the (non)linearity of the probe”. Not clear from this whether you consider both a linear and a non-linear, or only one of the two without looking at the experiments.
- You should probably cite also TCAV [1] in 3.1 when talking about probing. The CAV is probably the most renowned concept probe.
- I think you should better explain and point out the fact that if you directly optimize equation 3 you could decrease the performance of the model, since you would maximize the error over the non-intervened network. Instead, you says “ Equation 3 is differentiable, a neural network can be fine-tuned by explicitly maximising”. And, in equation 4, you change the optimization problem optimizing both losses
- Fine-tuned, A: it is not very clear the reason of including this model, nor the explanation. Please rewrite and add further motivations.
- Commenting Figures F.4 and Figure 4 this close is confusing. If possible, position the appendix figures in a different order to avoid confusing the reader. 
- “Finally, post hoc CBMs (even with residual modelling) exhibit a behaviour similar to the synthetic dataset with incomplete concepts: interventions have no or even an adverse effect on performance.” This claim is not supported by the results presented in figure 4(d) bottom: the AUPR is increasing in a non-negligible manner. Please rephrase the sentence. A similar consideration holds also for the Discussion section.

[1] Kim, Been, et al. "Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)." International conference on machine learning. PMLR, 2018.
questions: questions: 1. Table 1 report results where the black-box model is normally not that good for the target performance (among the worst results in general). This is something that I would not have expected, since both the bottleneck and the fine-tuning pose further constraints that normally impact the model performance (as also reported by CBM and many other concept-based approaches). How do you explain it?
2. Why is the CBM more computationally expensive in the experiments with VLM-based Concept? I think you could use a fixed CLIP to predict the concepts and simply train the task predictor on that. If I am right, I think the following sentence is unfair, and I suggest removing it “By contrast, our method allows fine-tuning the pretrained network, thus being helpful where a CBM is impractical”.
3. In Cher X-ray classification the black-box models are not intervenable. This is a surprising result, which partly contrast with the claim that you propose a method to make “interventions on any pretrained neural network post hoc. Could you elaborate further on that?
limitations: limitations: The authors report mostly possible expansions of their work, rather than actual limitations and possible societal implications of their approach. Indeed, even though the proposed method improves the interpretability of the model, it may impact its security. An attacker, for instance, could misuse the proposed technique for detecting model biases towards some specific concepts and creating realistic adversarial attacks that could go undetected.
paper_summary: paper_summary: This work introduces a method for performing concept-based interventions on pretrained black-box neural networks, requiring only a small validation set with concept labels. The paper formalizes the notion of intervenability as a measure of the effectiveness of these interventions and experimentally assesses the intervenability of black-box classifiers across several benchmarks and architectures.
strength_weakness: strength_weakness: *   **Clarity / Quality:** I could not find the precise training/validation procedure used for the baseline "CBM val", which represents one of the most critical baselines to compare with. I double checked both the paper but I could not find a detailed description. I also checked the code, but I could not find the configuration file (and the rest of the training/validation flow) for this baseline. The reason why I'm curious is that CBMs can also be viewed as "classification heads" fine-tuned on top of pre-trained models using a small validation set (e.g., a ResNet pre-trained on ImageNet used as a backbone of a CBM trained on CUB). The main result of this paper seems to consist in making interventions in CBMs effective with a smaller validation set. That is why the way this baseline is constructed is relevant to assess the quality and impact of this work. I could imagine at least two ways to construct such a baseline using a pre-trained backbone $g: X \to H$ (e.g., a ResNet trained on ImageNet) and a classification head $f: H \to Y$ (e.g., trained on CUB using downstream task labels but not concept labels): 
 1. Fine-tune a single layer $f^{(c)}$ of $f$ to align the activations of this layer on ground-truth concept annotations of the validation set. Freeze all the other layers of the model $f$.
 2. Fine-tune all parameters before the layer $f^{(c)}$, while freezing all the layers following $f^{(c)}$.
 3. Opposite of option #2.
 4. Train a new model $f'$ using only the ground-truth concept annotations of the validation set.

 I could not understand from the paper/code to which option the baseline "CBM val" corresponded to and whether the fine-tuning approach was used to train this baseline as well.
questions: questions: ### **Major**

* **"CBM val" Baseline:** Could you clarify in detail how the baseline "CBM val" is trained and validated?

*   **“Since interventions can be executed online using a GPU (within seconds), our approach is computationally feasible”:** This is not a formal justification for why the proposed approach is computationally tractable. The time complexity can be estimated in a more formal way.
    

### **Minor**

*   **“This work focuses specifically”**: This paragraph is already quite technical. While clear, it could be moved to a "preliminary" section after the introduction and rephrased here to make the introduction easier for non-technical readers. Also, the notation is presented again in the methods section (which is fine as it is a technical section).
    
*   **Related Work:** Consider moving the related works section after the experiments. This way, you could use the experimental results to discuss the relations of your contributions with related works.
    
*   **Figure 2, Step 3:** The arrow in step 3 might go downwards (as in the figure in the appendix).
    
*   **“For a trained CBM fθ (x) = gψ (hφ (x)) = gψ (ˆ c)”**: Consider explaining again here what c^\\hat{c}c^ represents as this was mentioned in the introduction, but it was not mentioned in the presentation of the notation.
    
*   **“we define the intervenability as follows”**: Consider explaining the idea behind the equations (e.g., "Here, the effectiveness of interventions is quantified by" for Eq 2) before showing the equation to ease reading.
    
*   **“where D is the joint distribution over the covariates …”**: Consider describing the variables involved in an equation before showing the equation to avoid the reader going back and forth.
    
*   **“To alleviate the need for human-annotated concept values”**: This sentence might be confusing as it is not clear whether it is the proposed approach or label-free CBMs that alleviate the need for human-annotated concepts (or both). I'd rephrase it with something like "To evaluate the effectiveness of the proposed approach in the absence of human-annotated samples ..."
    
*   **Results section:** Organizing results by datasets leads to repetitions of some observations. I'd suggest organizing paragraphs by key findings (which might sometimes relate to a single dataset, but it is ok).
    
*   **Table 1:** Consider dropping "0" at the beginning of each value (e.g., ".987") or express numbers in base 100 (e.g., 98.7) to save space. Consider highlighting the best models in bold.
    
*   **CIFAR 10 results in Table 1:** Why not use CIFAR-100 (where concept labels are available as a ground truth) instead of CIFAR-10?
    

* **Discussion & Conclusion:** The largest gap with respect to baselines is on AwA and MIMIC. Do you have an intuition why this is the case? The discussion could benefit from this analysis.
limitations: limitations: Authors adequately discuss limitations and future works in a dedicated section.

Comments: title: Thanks
Answer to Official Comment Reviewer ajS6
comment: I would like to thank the authors for their response, very interesting work, hope to see this paper in the conference.
I thank the authors for taking the time to fix the issues and answer my questions. I think the paper is ready to be accepted now.
I agree on both points!
Congrats again for your work!
Thank you for your nice feedback and for your interest in our work!

As a follow up to the two raised points:

- **Computational cost**: We agree with your point and will include the discussion in the appendix, thank you! Regarding the most critical factors, we believe all, the convergence parameter, number of concepts, number of datapoints, value of lambda, and the dimensionality of z play a relevant role. Due to the nature of the gradient-based optimization that runs until convergence, it is challenging to provide a formal estimation of the cost or ranking. However, one could try to approach the problem empirically by providing wall-time results of the optimization under controlled synthetic experiments varying the mentioned parameters to provide a better understanding of their relevance, which we will include in the appendix of the revised paper. We will, in addition, consider characterising more formally the complexity of a *single* update step in this gradient-based procedure, which should give additional intuition on the complexity.


- **CIFAR**: We believe that the original concepts coming from CIFAR-100, i.e. the classes, and those from [7]  in the VLM scenario differ significantly. More precisely, the original work in [7] introduces a total of 824 concepts in CIFAR-100 while the original annotated labels are only 100, and cover different categories. We assume in this discussion that the “concepts” the reviewer refers to in CIFAR-100 are the super-classes. For this reason, using the VLM-based concepts to intervene on a model trained with the original CIFAR-100 concepts would not be possible. However, we envision a setup where keeping the original concept categories, the CIFAR-100 dataset can be re-annotated using CLIP, this way the experiment proposed by the reviewer could be conducted. Based on our experience with VLM annotated data, we would expect it to have more noise and, therefore, potentially, less effective interventions than under the ground-truth annotations.
Thank you for your effort and for taking the time to explain your work even further!

I am aware that the following comments/questions are not critical nor the focus of this work. I am asking out of curiosity.

**[nit] Computational cost**: Thank you for providing a more detailed discussion on this. I think the paper would benefit if you could report this discussion in the appendix and take it a bit further by ranking the most critical factors and estimate their relationship with the computational cost (e.g., linear, quadratic, etc).

**[nit] CIFAR**: I understand the motivation for choosing CIFAR without using concept annotations. I'm wondering though whether the concept annotations generated using the VLM are aligned in some way to the original concept labels in CIFAR-100. And in case VLM and CIFAR-100 concept labels overlap, I'm curious to understand whether: (i) the VLM's annotations are actually matching the ground truth, and (ii) the interventions would be more effective using VLM's annotations or ground-truth annotations.