Addressing Leakage in Concept Bottleneck Models
Authors: Marton Havasi
Sonali Parbhoo
Finale Doshi-Velez
Venue: NeurIPS 2022 Accept (2022)
Keywords: interpretable models
concept bottleneck model
leakage

Abstract: Concept bottleneck models (CBMs) enhance the interpretability of their predictions by first predicting high-level concepts given features, and subsequently predicting outcomes on the basis of these concepts.  Recently, it was demonstrated that training the label predictor directly on the probabilities produced by the concept predictor as opposed to the ground-truth concepts, improves label predictions. However, this results in corruptions in the concept predictions that impact the concept accuracy as well as our ability to intervene on the concepts -- a key proposed benefit of CBMs. In this work, we investigate and address two issues with CBMs that cause this disparity in performance: having an insufficient concept set and using inexpressive concept predictor. With our modifications, CBMs become competitive in terms of predictive performance, with models that otherwise leak additional information in the concept probabilities, while having dramatically increased concept accuracy and intervention accuracy.

Addressing Leakage in Concept Bottleneck Models
Marton Havasi
Sonali Parbhoo
School of Engineering and Applied Sciences
Department of Electrical Engineering
Harvard University
Imperial College London
mhavasi Saavvss@@eess..
harvard. .
s. parphoo@@mperial........... ac uk
Finale Doshi-Veles
School of Engineering and Applied Sciences Harvard University
finale@seas hww.saa..aaaee..e. edu
Abstract
Concept bottleneck models (CBMs) enhance the interpretability of their predictions by first predicting high-level concepts given features, and subsequently predicting outcomes on the basis these concepts. Recently, it was demonstrated that training the label predictor directly on the probabilities produced by the concept predictor as opposed to the ground-trute concepts, improves label predictions. However, this results in corruptions in the concept predictions that impact the concept accuracy well as our ability to intervene on the concepts a key proposed benefit of CBMs. In this work, investigate and address two issues with CBMs that cause this disparity in performance: having an insufficient concept set and using inexpressive concept predictor. With our modifications, CBMs become competitive in terms of predictive performance, with models that otherwise leak unintended information the concept probabilities, while having dramatically increased concept accuracy and intervention accuracy.
Introduction
Building interpretable machine learning models is important in several high-stake domains where humans must typically interact with the models to make meaningful decisions. Among such models, Concept bottleneck models (CBMs) (Chen al./ 20200; Koh et al. 2020 Yeh et al. 2020 Lage & DDshhi- Velez, 2020 Wang et al., 2017) propose explicitly aligning the intermediate layers of neural network with some pre-defined expert concepts by first training a predictor of concepts, and subsequently using this to predict the label (Figure I left). CBMs are examples of interpretable models that are well-suited for human-AI joint decision making. A human supervisor can interpret and understand the label predictions by inspecting the concept predictions. Moreover, when the supervisor notices a misprediction among the concepts, they may intervene by correcting the concept prediction and, in turn, the model is able to update its final prediction accounting for the revised concept.
Typically when training CBM, the concept predictor and the label predictor are trained independently on ground-truth concepts. Unfortunately, these models can underperform in predictive performance compared to a black-box predictor. Instead, the most predictive CBMs use soft concepts, meaning that the inputs to the label predictor are the concept probabilities (the numerical values between 0 and I) as predicted by the concept predictor (Koh et al., (2220). Soft concepts rely on this information rich representation, which in contrast with hard concepts, where label predictor only accepts binary concepts.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Input
Concepts K
Label
Carnivore: 0.9
Two-legged:00 0.9
T-Rex: 0.9
y
Claws:
0.8
Intervention
y
x
y
Updated prediction
Carnivore:
0.9
Two-legged: 0.0
Cat: 1.0
T-Rex: 0.0
Claws: 0.8
Figure 1: Left: The concept bottleneck model (CBM) makes its prediction in two stages. First, the concept predictor (parametrized by 6) estimates the probability of each binary concept ci...ck cK in the input Then, the label predictor (parametrized by Φ) predicts the label y. When the model mispredicts a concept, one can intervene by setting the concept probability to 0 or 1 and obtain an updated prediction. Middle: In side-channel CBM, model set of latent concepts z that inferred from data. The concept predictor predicts both and z (po(c, = 3)=)), and the label predictor predicts the label (po(ylc, c, z) 2)). An amortization network trained rapidly infer z from (py, (z|c)) when needed. Right: In an autoregressive concept predictor, concept a. is predicted given the input a, and all previous concepts c1 ... Ck-1 (pe (Ck I c1)- This enables the concept predictor to model correlations in the concepts.
While soft concepts may improve predictive performance, this improvement comes at cost. The issue is that soft concepts allow the concept predictor convey unintended information about the label that would otherwise not be available to the label predictor (Mahinpei et al. |221] Margeloiu et al. 202T). For example, in an animal classifier, when concept predictor recognizes an image as cat, it may predict a slightly higher likelihood to the concept 'fur' than the concept 'tail', and when it recognizes the image as a dog, it may assign higher likelihood to the concept 'tail' than the concept "fur'. Even though cats and dogs are indistinguishable based on these two concepts alone, the label predictor may still compare the two soft concept probabilities and make an accurate classification. A flexible predictor, such deep neural network, can quickly learn such encoding patterns and improve its prediction using the information leaked by the concept predictor. Thus, the problem is dubbed leakage.
Leakage compromises the interpretability and trustworthiness of the model. The concept predictor no longer needs to faithfully predict the concepts to obtain high-quality label predictions. Instead, it only needs to encode the class label in the soft concept probabilities. human supervisor can never be sure if the model is predicting certain concept because it is likely to be present, because is encoding for something else. Morcover, since interventions remove this extra information from the concept predictions, predictive performance conditional on interventions suffers significantly.
How can one avoid leakage? While hard concepts ensure that information not leaked, these models suffer from poor performance in comparison to their soft counterparts. In this work, we focus on the performance disparity between soft and hard CBMs. Specifically, we attribute this disparity to (i) having an insufficient concept set and (ii) having inflexible concept predictor. The former issue occurs as result of the Markovian assumption under which CBMs operate: all the information about the label in the input must be captured by the concepts. If this assumption is not met, the performance of the hard CBM is limited by the information available in the concepts. The latter issue stems from the fact that CBMs predict concepts independently and cannot capture correlations between concepts. These correlations are important since some combinations of concepts may be unlikely or impossible (eg. the concepts 'fur' and 'scales' are mutually exclusive). These two issues hinder the performance of hard CBMs, while soft CBMs are able to mitigate them with leakage, at the cost of interpretability and intervenability.
We introduce two technical contributions that allow hard CBMs to match the predictive performance of soft CBMs without compromising their resilience to leakage. First, we circumvent the Markovian assumption by introducing side-channel to learn a set of latent concepts alongside the known con- cepts that are needed for accurate predictions. Second, we introduce an autoregressive configuration

---

for the concept predictor that significantly improves their flexibility and it allows them to capture correlations in the concepts.
On several synthetic and real applications, we demonstrate that our side-channel models improve the performance of hard CBMs in terms of accuracy, without affecting concept predictions intervenability. In addition, our autoregressive concept predictor significantly improves both concept accuracy and intervention accuracy compared soft CBMs and non-autoregressive hard CBMs.
Summary of contributions:
our experiments, we demonstrate that leakage detrimental to interpretability and intervenability in soft CBMs.
Hard CBMs are resilient to leakage, but their performance is hindered by not meeting the Markovian assumption and having an inflexible concept predictor. We address these shortcomings:
We circumvent the Markovian assumption by introducing side-channel to learn set of latent concepts.
We increase the flexibility of the concept predictor by adapting autoregressive architecture.
Background: Concept Bottleneck Models
Ji=1 consisting of N triplets of the form: input E Rd concept vector (0. 1 K of K binary concepts, and class label E N (we consider classification problems, but our results extend to regression problems where y E RR). Formally, a CBM comprised of a concept predictor, parametrized by 0, which is trained to predict the concept vector from the input pe (c|æ), and a label predictor, parametrized by é subsequently trained to predict the label from the concepts Pa (ylc) (Koh et al. 2020 (Figure left).
Consider a dataset D (x((1, c( "
Interprefability and Interventions There are two key benefits CBMs that make them well- suited for human-AI joint decision making. First, they explain their predictions by showing which high-level concepts the concept predictor was able find in the data. Second, human operator may intervene on these concepts to see how changing the concept values affects the final prediction. This to indicate their presence or absence. The
done by setting the probabilities of the concepts label predictor is able to adjust its prediction to account for the intervention (Figure Ilfft).
Hard CBMs In hard CBMs. the label predictor only accepts binary concepts inputs. The concept predictor and the label predictor are trained independently on the ground truth data: = arg max, Ep [log [log pe([x)) and & = arg max, Ep [log (v(())) (here, we omit regularizers for brevity). To make predictions on a new input point, one must marginalize out the possible concepts: (o)(v(x) = (clp)( Po (ylc)].
Soft CBMs Soft CBMs, on the other hand, pass the concept probabilities (the value [0, 1) directly to the label predictor. When encountering a new input v, a soft concept model first calculates the output concept predictor g(x) where a (g(x))) = po po(ck = 1|x) for k 1... K. These probabilities then forwarded to the label predictor, whose output is f(c) po (vl c). Combined, they produce the final prediction (g(()) = Pes (y)xx)
Three different training strategies have been proposed to train soft CBMs (Koh et al. 2020). First, independent training, where the concept predictor and the label predictor are trained independently i = arg max, Ep [log [o]xr) and = arg max, Ep [log Po (y) c)]. Second, sequential training, arg max, Ep [log po(c)>) and : arg max Ep [log pes (y]=)) Finally.join training, where the concept predictor and the label predictor are trained jointly 0, J arg maxes Ep [} log Po (clar) + log (2]=)) Here, 0 < À is scalar hy perpaaraeeterr controlling the tradeoff between the two terms. High prioritizes accurate concept predictions, while À close 0 focuses on accurate label predictions.
where the label predictor is trained after the concept predictor

---

Leakage Leakage occurs in soft CBMs when trained sequentially or jointly (Mahinpei et al. 202I). The label predictor learns to utilize the additional, unintended information in the soft concept probabilities output by the concept predictor.
In the animal classification example, when trained jointly, the concept predictor can learn to assign slightly higher likelihood to the concept 'fur' than the concept 'tail' when it recognizes the image as a cat and predict higher likelihood the concept 'tail' than the concept 'fur' when it recognizes the image as a dog. Despite that cats and dogs are indistinguishable based on these two concepts, the label predictor can compare the two soft concept probabilities and make an accurate classification. The label predictor is no longer limited by set of the pre-defined concepts: the concept predictor is able leak further information about the label to improve the predictive performance. This decreases the overall loss: the concept log-likelihood log (c(z)) decreases slightly due to the slight perturbation, but the label likelikoooooo Pog (y|ar) increases significantly since the label predictor now has access to the leaked information. This is further exacerbated by using À < 1, which prioritizes the label lig-liieldhood over the concept log- eigliikllhoodd in the training objective.
Leakage damages both interpretability and intervenability. Interpretability suffers because the explanation of the label is based not only on the concepts, but also on the behavior of the specific concept predictor. The human operator would need an in-depth knowledge of the concept predictor to understand why a small change in the likelihood of 'fur' leads to a significant change in the classification. Interventions are also adversely affected. When one intervenes on the 'fur' concept by setting its probability to L, it can decrease the predictive accuracy because this removes all information encoded in the soft probability value.
Hard CBMs avoid leakage by only passing binary concepts to the label predictor. The concept predictor and the label predictor learn to faithfully predict the concepts from the inputs and the labels from the concepts respectively.
3 Related works
Concept Activation Vector and Whitening Methods Several methods exist to interpret the layers of neural network and what they learn. Among Chese,,Kim et al. (2018);
: Zhou et al. (2018): Ghoobbaii ct al. (2019) use concept activation vector methods to analyse networks that have been pre-trained. The concept activation vectors are chosen to align with pre-defined or automatically-disoove
concepts. These vectors are based on the assumption that the latent space can be analysed posthoc in a specific form. Other methods such as concept whitening (Chen et al.) (020) directly alter a given layer of a network by adding a module that aligns the latent space to certain axes of interest to understand the computations leading to that layer. Unlike our work, none of the former methods address the fundamental issue of leakage in CBMs that significantly impacts the interpretability and intervenability of these models.
Soft Concept Bottleneck Models Soft CBMs (Chen et al. 2020 Koh et al., 2020) propose passing concept probabilities (value in (0, 1) to the label predictor to significantly improve prediction accuracy in comparison to CBMs that learn hard concepts. Though performant, several works show that slightly distorting the predictive concept probabilities enables the concept predictor to 'leak' the final label to the label predictor (Mahinpei et al., 2021 Margeloiu et al.) 2021). We propose tools that allow us to diagnose and address these issues to directly improve the performance of hard CBMs, thereby overcoming the issue of information leakage in their soft counterparts.
Information Bottleneck Methods IB methods take a joint distribution p(X, Y) and compress the inputs X into a latent representation Z that is most informative about the target Y (Tishby et al. 2000). This latent representation may thought of as a type of concept. Several methods have been proposed to use such a concept representation 2 to compute the effect of an intervention (Parbhoo cct al. 2020; Chicharro et al. 2020). Both the traditional IB and the deep variational IB formulations assume specific Markov structures for the generative model (Alemi et al., 20166; Wieczorek & Roth 2020). In contrast, our work uses side-channels to learn informative concepts that are predictive of the target while compressing the inputs, and is independent of such Markovian assumptions.
Causal Concept Effect Models Gooal et al. 2019 ; Feder et al. (021) explain coupling a CBM with causal model and computing the Causal Concept Effect to speculate about what might happen

---

to the prediction had concept been different. Unlike this work, we examine only the statistical setting where can intervene correct modify an incorrect concept prediction, but not to speculate about counterfactuals since the model does not capture causal relationships.
Concept completeness Yeh et al. (2000) study when a set of concepts sufficient for explaining the prediction in deep neural network. They propose accuracy-baa securccy-based completeness score. Their score, however, does not imply that the concepts capture all information in the input i.e. the Markovian assumption. Our information-bite basod completeness score indicates precisely when the assumption is met.
4 Addressing Two Shortcomings of Concept Bottleneck Models
In this section, we explain and address the performance disparity between soft CBMs and hard CBMs. We present two sources of information that CBMs unable capture: information that is not available in the concepts and information in the correlations in the concepts. Soft CBMs can lessen the impact through leakage, but hard CBMs cannot. We propose modifications hard CBMs that allow the use information from these two sources while retaining their resilience to leakage.
4.1 The Markovian Assumption
First, we consider the case when the concepts are insufficient for predicting the final label, but has more information that can help. We refer to this as the lack of Markovian assumption. When the Markovian assumption holds, the chain + conditionally independent of x given c: I(y; æ/c) = O, where I denotes the mutual information.
y is Markovian, which defined as y being
When the assumption holds, must contain all information about y that is present in z. Otherwise, c is insufficient for predicting y, therefore, a CBM that relies on hard concepts will underperform compared black box x -> predictor. In what follows, describe how CBMs can still be applied with the help of a side-channel and define a concept completeness score to diagnose when the Markovian assumption may not hold.
4.1.1 Using Side-channel
Along with the known concepts model L unknown concepts 2 E (0, 13 CBM (Figure middle). This setup allows the model to capture information about y present but not present in via the latent concepts.
d in a side-channel
Training the side-channel model is analogous to training a hard CBM. We optimize the predictive log-lik lig-likelihood using stochastic gradient ascent:
0, Ô arg max ED [log)o(c(>)) log (x)()))) 2)](
0.6
A caveat is that the gradient of T paflle) pa (y) c, =)] need to be estimated using gradient estimator:
We use the reinforce estimator (Williams) 1992) because it able to give an unbiased estimate the gradients, and it works well in a low dimensional settings such as
V ag (a)l)) [p,(slc, 2)] = Epo(s(n) [po(llc, 2) V, log po(2)=)]
(2)
The gradient is then estimated using M Monte-Carlo samples:
V poostte (pslycc R
Σ
Polule, z (m) Vo log po (m())()))
M
At prediction time, we can decide whether to the side-channel for prediction or rely purely the known concepts. If the side-channel is used, the predictions are calculated using M Monte-Carlo samples:
Poo(y Rd
Σ
(m) (m)))
(4)
M
(10) (m))
5

---

In applications where the label prediction must be fully explained by the concepts, we have to marginalize out the side-channel. This requires estimating Po(2))) One can obtain an estimate using po(=(c) Re Deeee' [po (2|2")); however, this requires iterating over the whole dataset each time a prediction made.
Our solution is to train a small (1 hidden layer) predictor that can rapidly estimate pa (2(2(c) from c. We call this the amortization network, since it amortizes the cost of inferring from c. We denote its parameters 7, and train it using maximum-likelitee our experiments, the amortization network is trained over 10-20 epochs depending on the dataset. Equipped with we can make predictions without using the side-channe::
training: 7 = arg måx, Ep [log py, (z|c)]. In
Epo(ela) [po(ulc)] =
1
Σ
Polule!m), (0())
(5)
M
((m) (m() (p)p),
A second benefit marginalizing the side-channel with the amortization network is that it can help us understand how much the model relies on the side-channel for prediction. To formalize this concept, we introduce completeness score.
The side-channel model can be used to estimate what fraction of the relevant information is present in c. This is useful for diagnosing when a side-channel is needed and to understand how much the model relies it. If the model heavily relies on the side channel, it indicates that key concepts are missing from c. We define
A Completeness Score
1(y:e)
(6)
T(y; e,
0 S C < 1, because its a ratio of non-negative quantities and the nominator is less-than-or-equee to the denominator since I(y; c, x) 1(y; =lc) 2 0 (Cover 1999p. When C' = we have I(y; c) I[y; r), so I(y; - [(y: = I(y; z/c) 0 thus the Markovian assumption holds, and e captures all relevant information. When = 0 ([(y: c) i.e. and y are independent), none of the information in e is relevant for predicting y. With the help of the amortization network, it is possible to estimate C:
H(y) + ED [log Expp, Pp c, =]]]
C 2v H(y) + Ep log (espptts))) (wlc,
Interpretability of the Side-channel The latent concepts captured by the side channel can be inspected for interpretability. They may align with human understandable concepts (in which case, they can be added to c), but is possible that they are uninterpretable. The side-channel model should still be used to avoid the detrimental effects of leakage that we have agency on deciding whether to use the side channel information at prediction time or not. Including the side-channel information is ideal for applications where accuracy is high-priority and partial explanations are sufficient (Equation 4b. For applications where the human operator must know the full set of concepts contributing the prediction, we can make predictions without it (by marginalizing 2 with the help of the amortization network as shown in Equation
concept accuracy and interventions. Note
4.2 The Expressivity of the Concept Predictor
The second source of information that CBMs are unable to capture is correlations in the concepts. An issue with the usual formulation of a CBM is that the concept predictor predicts the concepts independently (conditional on the input). This may give poor performance when the concepts are correlated. Consider two concepts that are mutually exclusive. A predictor that predicts them independently does not have the flexibility to express their mutual exclusivity and inadvertently predicts non-zero probability of both concepts being present. While soft CBM can learn during training that the concept probabilities correspond to mutually exclusive concepts, a hard CBM cannot.
We propose a modification to the CBM architecture that allows hard CBMs to capture correlations in the concepts. Our solution is to use an autoregressive architecture for concept prediction similarly to classifier chains in multilabel classification (Read al. 2011 Dembezynski al. 2010). To predict

---

the k-th concept, use not only the input x, but also the already predicted concepts c1:k (Figure right).
The log-likelihood, which is needed for training, can be computed by iterating over the concepts: log po(clz)) = Ek=1 log po (c(c |z, -1:k-1). To sample concept set, which is needed for prediction, we must sample the concepts iteratively: Cj ~ po (r, c1-k-1) for l 1...K. K.
Lastly, interventions require a modified sampling algorithm. In autoregressive models, we cannot eas- ily generate Monte-Carlo samples from the concept distribution conditional on of interventions. Instead, to estimate po. [ū] L.. { }r) (where (G)HEzC {) denotes the set of intervened concepts), we propose normalized importance sampling algorithm that generates M weighted concept samples for prediction ( (a)z) R >m=1 (ileP(())e!"))) Our proposal distribution c(1) c.. g(c)
> Wm
matches the interventions { when the concepts are intervened on and selects a random concept conditional on the previous concepts when they are not intervened on:
[8(ck, (x) [po(culo, (I:-1)
k E I
g(cx)
(8)
k $ I
where 5 denotes Kronecker's s delta function. To obtain an importance weighted estimate, we must compute the weight of each of the M samples Wmm- (afemm)a,((a.) This is done by iterating over the K concepts:
((ctm))
K II
po
po
(m)) [2,
1
(m)
KEI k $ I
Wm
k=1
since the probability mass ratio (c) (m)) ( when lo I. Finally, we obtain the impor-
tance weighted estimate (ūla, ((:z)) Em
W Pm c! Note that due this re-weighting Σ 1'm
scheme, interventions are able to affect concept predictions of earlier concepts. We also show this experimentally in Appendix E Normalization is important to ensure the predictive distribution is proper. Psudocode for the interventions is shown in Appendix
5 Results and Discussion
We present empirical results showcasing the efficacy of our proposed modifications to concept bottleneck models (CBM).
5.1 Datasets and Models
We used two datasets in our experiment. First, an Early Warning Score (EWS) prediction task based on electronic health records data that simulates a potential use case. Second, to showcase the versatility of CBMs and compare prior work, we apply our method an image-based bird recognition task. Our code is available at https:: /tttps://githu.... com /tak/ http//taa/aaddddee hwt.//dddresniiggg leakage
MIMIC-III EWS (Johnson al. 2016) The task is predict an EWS for patients at hospital that informs the medical professionals about abnormalities in the vital signs that often precede deterioration and cardiac arrests (Subbe et al. 2001p. The concepts ('Mild hypotension', 'Severe fever' etc.) provide justification for the score. The EWS ranges from 0 to 15, and it is based on the patient's vitals. Deviations from the norm increase the score depending on severity. For example, 'severe hypotension' adds +3 to the EWS, while "mild fever' adds +1. The dataset does not contain concept annotations, so we defined them synthetically (Appendix (A). The dataset contains records from 17,289 patients over combined N=796.250 time steps (split into 530,802 training and 265,448 test examples, while ensuring that no patient appears in both sets). It has 13 input features, = 22 concepts and 16 label classes (EWS from 0 to 15).
For MIMIC-II EWS, the concept predictor is a two-layer feed-forward neural network with a hidden layer of size 100, and the label predictor is a two-layer network with hidden layer of size 50. In the
C1 is shorthand notation for Ck-1
7

---

MIMIC-III
CUB-2011
polple [+]:) accuract
10075
100%
Table I: Concept and label accuracies (%) in soft and hard CBMs. The best result is bolded in each column. Hard autoregressive models (Hard AR)
8076
90%
6006
excel at concept accuracy and they are on-par with
40%
80%
soft joint models in label accuracy.
20%
MIMIC-III BWS
CUB-2011
Contept groups intervered
Concept groupe intervened
Model
(o(c)))
Peo(a(a)
polei))
pes(y]a)
Independant
0.1
Independent
38. 38.7+00
45.9+0.1
48.1#0.
72.8+0.8
Seqpentisl
Join 0:31
Hard AR (Thba work)
Sequerrial
38.7+0.
47.6+0.0
73.9+0.1
Soft Joon 1
Sol. Junt 0.001
Soft Joint == 1
38.72000
48.0220.0
47.7:20.2
75.4:0.11
Solt Joint == 0.1
35.610.11
48.010..0
47.910.4
82.710.2
Figure 2: Label accuracy after intervening on
Joint
82.1+0.3
Hard
38.7200
46.6+20.0
68.0:002
79.5:20.3
subset of concepts. Soft joint CBMs deteriorate quickly, while hard CBMs, including our proposed autoregressive model (Hard AR), improve with interventions.
Hard AR (This work)
41.0:0.0
48.51.0.0
81.410.1
81.710.2
autoregressive case, a small, two-layer network (hidden layer size 20) predicts each concept. The input to these networks the previously predicted concepts, concatenated with the 100 hidden units of the concept predictor. We use M = 200 Monte-Carlo samples for prediction. For training hyperparameters, see Appendix
CaLDech-UCSD Birds 2011 (Wah et al. 2011) For an accurate comparison to Koh et al. (020))
replicated their results with the same preprocessing steps. This dataset contains N 1=11, N=11,788 5,9994 training and 5,794 test) images of 200 bird species native to North America, along with 312 binary attributes (eg.: 'wing-color' and 'beak-shape') ). Following Koh et al. (2020), of attributes to = 112 binary concepts, only including attributes that are present for at least 10% of the dataset. To de-noise the attributes, consolidate the concepts within each species by majority voting (where if the majority of ravens have black wings, we ensure that all ravens in the dataset are annotated as having black wings (Koh al. 2020)).
narrow down this list
For CUB-2014 we fine-tuned a pre-trained Inception v3 network (Szegedy et al. 2016) as the concept predictor. Following Koh et al. (2020), the label predictor for the soft models is a simple linear predictor, while for hard models, we use a two-layer neural network with a hidden layer of size 200. In the autoregressive case, small, two-layer network (hidden layer size 50) predicts each concept. The input to these networks are the previously predicted concepts concatenated with the output of the Inception network. We use M 200 Monte-Cate samples for prediction. For training hyperparameters, see Appendix C
5.2 Autoregressive Concept Predictions
Our first set of experiments quantitatively assess the impact of leakage as well as examine the effectiveness of autoregressive concept predictors. We lead with these experiments to justify the use of the autoregressive architecture in our subsequent side-channel experiments. It is important to note that both datasets have complete concept sets (they meet the Markovian assumption =1) so any performance disparity is due to the inflexibility of the concept predictor.
Leakage makes soft CBMs unsuitable for tasks where interpretability or intervenability are required. Soft CBMs underperform in concept accuracy, but they are often able to predict the label correctly (Table [T. This is only possible if the concepts are leaking information about the label in an uninterpretable way. Interventions, where the human operator sets concepts to their true values, are ineffective on sequential and joint soft CBMs (Figure 2). This should improve their predictions and yet they still mispredict the label even when all concepts are supplied, because interventions remove the leaked information from the soft concept probabilities.
Autoregressive predictors close the gap between soft and hard CBMs in label accuracy while having dramatically increased concept accuracy. In Table I we see that soft CBMs perform well in label accuracy, but leakage is detrimental to their concept accuracy. The hard CBM performs better in concept accuracy, but its performance is limited by the inflexibility of the concept predictor. The autoregressive model (Hard AR, this work) offers the best concept accuracy with competitive
8

---

CUB-2011
MIMIC-II EWS
CUB-2011
0976
100%
Shidll oocuracs
45.095
6076
Completeres (C)
10%
0.8
00%
40.0%5
23,0%5
0.00
0.6
40%
0.4
40%
10
20
Coroapt groups during training
Concapt groups doring traning
0%
20
Scft: jiin A- 1. w/o dide- chonehl
Iiad A22 w/ eite ccoreee
Concept groupe during trainin
Concept groups during training
1ard All iddecctonnl
Hard Alt sitechorral
(dde-chaerd not uod prediction)
Completeran (4)
Polgre) accurncy
The side-channel explains the gap between the hard and soft models. It captures the information that is Icaked in the soft joint CBM but not captured by the hard CBM. Right: The completeness score and label accuracy of a hard autoregressive CBM with side-channel. The x-axis denotes the number of concept groups available during training. When the completeness score reaches I, the Markovian assumption holds by definition.
Figure 3: Left:
The predictive performance of soft joint CBMs and hard CBMs.
label accuracy: in the case of MIMIC-III EWS, Hard AR above soft models in both concept and label accuracy, while in CUB-2011, Hard AR has the best concept accuracy with label accuracy within 1% of joint models. (Note that concept accuracy refers to predicting all concepts correctly simultancously as opposed to averaging the accuracies over the different concepts.)
Hard autoregressive models excel at interventions while soft CBMs fail to adjust their predic- tions. On Figure we see that the performance of soft sequential and soft joint models deteriorate quickly due to leakage, while soft independent and hard models improve with interventions. The best performing model the Hard AR (this work) on both datasets.
5.3 Using a Side-channel
Next, we examine the side-channel model as well as our proposed completeness score. In these experiments, trained CBMs with only subset of the available concepts to simulate the scenario where the Markovian assumption does not hold. We used = 20 latent concepts for MIMIC-III and L 100 latent concepts for CUB-2011 (we settled on these values because larger L yielded no significant improvement in predictive performance). Note that we use autoregressive concept predictors in hard CBMs, any performance disparity is due to having an insufficient concept
In MIMIC-II EWS, the concepts were grouped depending on which of the five vital signs they derived from (blood-preee pressure, temperature, respiratory rate, heart rate, oxygen saturation). In CUB-2011, a group of concepts refer to related attributes, such as 'wing-color'.
The side-channel closes the gap between soft and hard CBMs when the Markovian assumption is not met. In Figure we see that the performance of the side-channel matches the predictive performance the joint model when accounting for the latent concepts and it performs equivalently to the no side-channel model when the latent concepts are marginalized with the help of the amortization network. Therefore, the side-channel models precisely the information that is not present in the concepts, i.c. the information that is leaked in the soft joint model.
The completeness score indicates when the Markovian assumption is fulfilled. The complete- ness score proves be useful diagnostic tool to show when the available concept set limits the performance of hard CBMs. Figure 3 (right) depicts the completeness score C depending on the number of concepts used during training. As the concept set grows, C tends towards 1, and reaches once the concept set contains all information about the final label, and the Markovian assumption met. Interestingly, completeness reaches 1 before Pa (ylc) accuracy peaks in CUB-2011. This possible when a concept contains information to predict y, but this information is not present in (e.g. the color of the belly not visible on images where it could be the deciding factor). Such concepts do not contribute to completeness, but they increase label accuracy when the concepts given.

---

MIMIC-III
100%
100.0%
Boufil (o]z) accuracy
5076
80%
90.0%
Table The computation costs of different mod- els. The times are recorded on VI00 GPU.
70%
80.0%
6076
These times should be considered approximate as we were unable to account for factors such as
50%
the utilization of the assigned compute node.
Concapt groups intervened
Concapt groupe intervened
concopes
Daraset
Model
Timelepuch
Training epochs
Hard AR ado-dhandd traned on concgll
Soft Indkpenddeuseepeemillnnn
20
Figure Interventions remain effective when trained with side-channel on partial concepts.
MIMIC-II EWS
Hard
173.4
27.0
20
side
306.7
The color wheel denotes the number of concepts
Soft
60
groups used during training: violet corresponds
K3.2
wt side channel
83.7
to single concept group and red corresponds to almost all concept groups used. The performance identical for all models in MIMIC-III EWS and the lines completely overlap.
The side-channel is fully compatible with interventions There have been concerns that a side- channel model could encode the label information in the side-channel, thus rendering interventions ineffective (Koh et al. 2020). We found that this is not the case. 4 shows that interventions are as effective when trained with a subset of concepts as when trained with the complete concept set. In the case of MIMIC-III EWS, the intervention accuracy of the side-channel models completely overlap with the hard CBM trained on all concepts. In CUB-2011 I, the interventions are less effective with 1-2 concept groups, but they quickly catch up to the model trained with the complete concept set.
Computational costs. The benefits of autoregressive models and side-channel models come an increased computational cost. Table[]showw the computational cost of each model. Notably, the more complex image dataset, CUB-201 takes more epochs converge. The cost of the autoregressive architecture and the side-channel is more significant when the predictive model is relatively simple, which is the in MIMIC-II EWS.
6 Conclusions
Leakage is detrimental to model interpretability and intervenability in soft concept bottleneck mod- els. Hard CBMs are resilient to leakage. but they lag behind their soft counterparts in predictive performance. We exhibited and addressed two causes of performance disparity between soft and hard CBMs. Firstly, not meeting the Markovian assumption, which can be diagnosed using our proposed completeness score and addressed using side-channel model. Secondly, the inflexibility of the concept predictor, which can be mitigated by adapting an autoregressive architecture.
With our modifications, CBMs no longer suffer from leakage and have significantly improved concept accuracy and intervention accuracy. This makes them more interpretable and more suitable for human-Al joint decision decision-making making tasks.
In future work, hope to explore methods to present the side-channel information in human- interpretable way. This would allow us to discover concepts that are needed to complete the concept set for a given task.
10

---

Acknowledgements
This work was funded by grant from the National Institute of Mental Health (grant no. R0IMH123804). The funders had no role in the design and conduct of the study; collection, manage- ment, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.
References
Alemi, A. A., Fischer, I., Dillon, J. V., and Murphy, K. Deep variational information bottleneck. arXiv preprini 112:0:161..00410, 2016.
Chen, Z.. Bei, Y., and Rudin, C. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):7227722, 2020.
Chicharro, D., Besserve, M., and Panzeri. S. Causal learning with sufficient statistics: an information bottleneck approach. arXiv preprint arXIv:2010.05375 2020.
Cover, T. M. Elements of information theory. John Wiley Sons, 1999.
Dembezynski, K., Cheng, W., and Hüllermeier, Bayes optimal multilabel classification via probabilisticc classifier chains. In ICML, 2010.
Feder, A., Oved, N., Shalit, U., and Reichart, R. Causalm: Causal model explanation through counterfactual language models. Computational Linguistics, 47(2):333-386, 2021.
Ghorbani, A., Wexler, J., Zou, J. Y., and Kim, B. Towards automatic concept-based -bsee explanations.
Advances in Neural Information Processing Systems, 32, 2019.
Goyal, Y., Feder, A., Shalit, U., and Kim, B. Explaining classifiers with causal concept effect (cace). arXiv preprint 1107:0907. 2019.
Johnson, A. E., Pollard, T. J., Shen, L., Li-Wei, H. L., Feng, M.. Ghassemi, M., Moody, B., Szolovits, P. Celi, A., and Mark, G. Mimic-iii, freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 2668-2677. PMLR, 2018.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6600 1412.698, 2014.
Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., and Liang, P. Concept bottleneck models. In International Conference on Machine Learning, pp. 5338-5348. PMLR, 2020.
Lage, and Boshi-Velez. Learning interpretable concept-based models with human feedback. arXiv preprint arXiv:2012.02898. 2020.
Mahinpei, A., Clark, J., Lage, I., Deshi-Velez, F., and Pan, W. Promises and pitfalls of black-box concept learning models. arXiv preprint arXiv:2106. 13314, 2021.
Margeloiu, A., Ashman, M., Bhatt, U.,, Chen, Y., Jamnik, M., and Weller, A. Do concept bottleneck models learn as intended? arXiv preprint arXiv:2105.04289 2021.
Parbhoo, S., Wieser, M., Wieczorek, A., and Roth, V. Information bottleneck for estimating treatment effects with systematically missing covariates. Entropy, 22(4):389, 2020.
Read, J., Pfahringer, B., Holmes, G., and Frank, E. Classifier chains for multi-label classification. Machine learning, 85(3):333-355 2011.
Subbe, C. P., Kruger, M., Rutherford, P., and Gemmel, L. Validation of a modified early warning score in medical admissions. Qim, 94(10):521-266 2001.

---

Szegedy, C., Vanhoucke, V., loffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Tishby, N., Pereira, F. C., and Bialek, W. The information bottleneck method. arXiv preprint physics/0004057, 2000.
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-uesd 200ss2200-2011 dataset. 2011.
Wang. T., Rudin, C., Doshi-Velez, Liu, Y., Klampil, E., and MacNeille, P. A bayesian framework for learning rule sets for interpretable classification. The Journal of Machine Learning Research, 18(1):2357-2393, 2017.
Wieczorek, A. and Roth, V. On the difference between the information bottleneck and the deep information bottleneck. Entropy, 22(2):131, 2020.
Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, (13:229-2555 1992.
Yeh, C.-K., Kim, B., Arik, S., Li, C.-L., Pfister, T., and Ravikumar, P. On completeness-awant concept-based explanations in deep neural networks. Advances Neural Information Processing Systems, 33:20554 2020.
Zhou, B., Sun, Y., Bau, D., and Torralba, A. Interpretable basis decomposition for visual explanation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 119-134, 2018
12


Reviews:
[{"review_id": "hp6Wu7A_IGF", "reviewer": "Reviewer_YNur", "summary": "The authors present a novel concept bottleneck model (CBM) aimed at\nretaining the interpretability and suitability to be combined with\nhuman interventions of hard CBMs (as compared to soft CBM), while\naddressing two major limitations of hard CBM that they identify as\npotential causes affecting their predictive accuracy: the inability to\ndeal with correlated concepts, and the problem of having an incomplete\nconcept set.", "questions": "How crucial is the marginalization on concepts for the performance in\nhard CBM? how would an approach maximizing on concepts rather than\nmarginalizing perform? ", "limitations": "The authors highlight the increased training cost due to the\nautoregressive and side-channel components, which is however not\ndramatic. One issue that would be worth discussing is the impact on\ninterpretability of the marginalization on concepts that is performed\non hard CBM, and how the marginalization on side concepts further affects it.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The work is definitely relevant, given the importance that\ninterpretable neural networks and concept-based models have in\nboosting trustworthiness and reliability of deep networks. The\nproposed architecture is novel and all contributions are appropriately\nmotivated, with each contribution addressing one of the shortcomings\nof existing hard CBMs. \n\nThe paper is sound in terms of justifications for the different\ncomponents introduced (the side channel, the autoregressive model) and\nthe strategies used for training, prediction and for dealing with\nintervention, as well as for the introduction of the information\ntheoretical completeness score.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fxiuYwy2pS", "reviewer": "Reviewer_skST", "summary": "This paper considers the problem of leakage for concept bottleneck models (CBM). CBMs are models that predict concepts from features and then predict labels from concepts. When concept probabilities contain information about the label, leakage happens, which decreases intervention performance: when users intervene by changing concepts. Hard CBMs use binary concepts and avoid leakage but have lower performance. This paper proposes two methods for improving the performance of hard CBMs: (1) autoregressive where a concept is predicted using features and all previous concepts, (2) side channel where the label is predicted from concepts and directly from features. In experiments, the authors found that (1) autoregressive improves performance, especially intervention performance when a few concepts are changed, (2) side-channel improves performance when a few concepts are available.\n", "questions": "See my suggestions in the weaknesses section above.", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- CBMs are important models that support intervention. Leakage is a major issue with CBMs.\n- Experimental results are mostly positive.\n\nWeaknesses:\n- Clarity: while I understand the paper now, I did get confused on my first read. The many ideas being presented got a bit mixed up. I think a reorganization will improve the clarity of the paper. \n\n- Novelty: The application of autoregressive and side channeling to CBMs is novel, but these are fairly common modeling techniques, so the novelty is not high.\n\n- Experiment 1 showed that autoregressive hard CBMs have better intervention performance when a small number of concepts (e.g. 5 out of 20) are changed. I think this was because the changed concepts propagate to other concepts, so it only works well for changing concepts that are around the beginning. This may also make it more difficult for users: changing one concept makes other concepts change. I think these should be tested in experiments and discussed.\n\n- Experiment 2 showed that side channeling improved hard CBM when a small number of concepts are available. With a side channel, the model can predict the label directly from features, so this improvement is not surprising. The original CBM paper discussed side channeling as a potential future work, but pointed out a drawback: \"cannot cleanly intervene on a single concept\". So I think this means the intervention performance will be lower. This should also be tested in experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "d28SoFnL8s7", "reviewer": "Reviewer_L4TG", "summary": "This work attends to the problem of “leakage” in the soft concept bottleneck models (CBMs): the soft concepts can convey unintentional information about the label, which compromises the interpretability and intervenability of a CBM -- the main purpose of such models -- even if it may improve the label prediction accuracy. The paper suggests two techniques: (1) a side channel to learn latent concepts to complement the insufficient (known) concepts under the Markovian assumption and (2) an autoregressive model to learn correlation among concepts. The authors conduct experiments on MIMIC-III and CUB-2011 to show the effectiveness of their proposed approaches including whether indeed adding the side-channel improves the label prediction and adding the autoregressive model improves the concept prediction accuracy.\n", "questions": "* Am I correct in understanding that the authors did not provide any result on a model that employs both side channel and autoregressive models altogether?\n* Autoregressive modeling for concept prediction seems quite random and only adds for improving predictive performance. How is it relevant to addressing the leakage issue?\n* It’s not clearly explained why there is a need for the path from c to z and how to train this amortization network `gamma`.\n", "limitations": "* There are a few places that appear to be written at the last minute and especially the main technical parts are not described entirely clearly.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The main idea and presentation of the paper is good overall.\n* The experimental results on the autoregressive model (Hard AR) performing well on the concept prediction as well as the label prediction (also when intervened) are nice.\n* The adaptive use of the side-channel from the amortization network shows its applicability for different scenarios.\n\nWeaknesses:\n* Perhaps it’s unfair but to me the pitch of this work (and thus the title) is a bit misleading: this paper is actually focused on closing the gap between soft and hard CBMs (in terms of the performance) rather than directly addressing the leakage issue in soft CBMs. To elaborate, adding a side channel does not prevent leakage; it helps improve predictive performance of a hard CBM which already/inherently prevents the leakage issue. If you say the leakage issue is being addressed, one would expect something like improving based upon a soft CBM, not hard one. Perhaps simply “improving hard CBMs” for example or “Addressing Two Shortcomings of Concept Bottleneck Models (the title of Section 4)” would suit better.\n* Modeling correlations between concepts doesn’t quite fit the main motivation of preventing leakage; also, it’s not convincing yet why modeling correlations between concepts should be a key.\n* The paper is not addressing potential limitations of the proposed approach at all, except they say it’s costly for a large data set at the end of Section 5. I think this part needs a lot more addressing on the computational overhead (for amortization network, autoregressive model, MC sampling, etc.) to make this work fair.\n* There is a potential room for improvements in modeling the correlations among concepts. However, the proposed idea of using an autoregressive model seems quite under-explored.\n* The two main ideas of using the side channel and autoregressive model are separate.\n* The leakage is already addressed in prior work [Mahinpei et al., 2021, Margeloiu et al. 2021] and therefore the demonstration of it itself doesn’t seem to add much new insights.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept