# Measuring and Addressing Information Leakage in Concept Bottleneck Models

Raffael Schon, Baptiste Abeloos, Stéphane Herbin

► To cite this version:

Raffael Schon, Baptiste Abeloos, Stéphane Herbin. Measuring and Addressing Information Leakage in Concept Bottleneck Models. 2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Oct 2025, Honolulu, United States. pp.635-643, 10.1109/ICCVW69036.2025.00071. hal-05526532

HAL Id: hal-05526532

https://hal.science/hal-05526532v1

Submitted on 25 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Measuring and Addressing Information Leakage in Concept Bottleneck Models

Raffael Schon¹*

Baptiste Abeloos²

Stéphane Herbin²

¹Université Paris-Saclay

²DTIS, ONERA, Université Paris-Saclay, 91120 Palaiseau, France

## Abstract

Concept Bottleneck Models (CBMs) are transparent models that use a human-interpretable bottleneck layer to improve interpretability in tasks such as image classification. However, model transparency does not guarantee that decisions are supported by relevant information. When this is not the case, we speak of information leakage.

To provide a quantitative measure of this property, we introduce a metric called Irrelevant Concept Contribution (ICC), the first metric that directly quantifies information leakage for concept bottleneck models. Using this metric, we propose a comprehensive comparison of bottleneck architectures, training strategies, and decision processes. In particular, we demonstrate that state-of-the-art performance in image classification can be achieved using only relevant concepts that contribute positively to the decision process on commonly used datasets. This makes the decision process relevant by design and in an explicit way.

## 1. Introduction

Concept Bottleneck Models (CBMs) [8] are interpretable-by-design models that analyze the input data using human-understandable concepts before making a final decision. This two-step decision logic enhances transparency by aligning the model's reasoning with human understanding concepts in a specific layer. CBMs operate by first predicting a set of active concepts, which are then directly used for the prediction.

In [8, 12], authors state that CBMs address three key desiderata: interpretability – identifying which concepts are essential for predicting the targets; predictability – ensuring that targets can be accurately predicted based solely on the learned concepts; and intervenability – enabling the modification of concepts to enhance predictive performance.

Recent works [11, 12] investigated a phenomenon termed information leakage arising in CBMs that states that concept activations contain unintended information which is used for downstream predictions. Crucially, the usage of this unintended information is not necessarily noticeable to humans.

Information leakage has so far been measured in CBMs only in an indirect manner. The authors of [16] measure it by pruning the concept to target weight matrix to the top 5 contributing concepts and observing the change in model decision. In [12], authors compare CBMs trained with different strategies using a concept oracle model. Importantly, they find that CBMs outperform the concept oracle significantly, implying that extra information is learned and encoded in the concept activations.

We suggest that information leakage studies could benefit from a direct and quantitative method of evaluation. To this end, we propose defining information leakage as the contribution of *irrelevant* information in the decision process. Based on this definition, and leveraging external knowledge of relevant concepts for each class prediction, we can assess the extent to which a CBM is affected by information leakage by analyzing how both relevant and irrelevant concepts contribute to class predictions.

Using this insight, we introduce a new metric to quantitatively measure information leakage, termed the Irrelevant Concept Contribution (ICC) metric. ICC utilizes knowledge of concept-class associations to measure the influence of concepts that should not contribute to the decision-making process, providing a clearer evaluation of unintended information leakage.

Our contributions are summarized as follows:

* We introduce the Irrelevant Concept Contribution (ICC) metric as a novel quantitative measure of information leakage.

* We compare several bottleneck architectures and evaluate the impact of different training strategies based on information leakage measures and model accuracy.

<footnote>*E-mail: raffael.schon@onera.fr</footnote>
* We demonstrate that reducing the influence of irrelevant concepts in the decision process can be achieved without compromising accuracy obtained with state-of-the-art models. We apply the method to public datasets such as CIFAR10, CIFAR100 and CUB

* Our approach demonstrates that constraining the CBM decision layer to relevant concepts and positive contributions inherently eliminates information leakage while maintaining performance, thereby enhancing the model's interpretability and ensuring the relevance of its decisions.

## 2. Related Works

**Interpretability Techniques.** The application of neural networks in critical areas such as healthcare, biomedicine, and legislation, created a need for explainability techniques. Two main approaches to explain neural network decisions have been developed: *post-hoc* methods and *interpretable-by-design* models. Post-hoc methods are applied after the model has been trained and aim to estimate the importance of input features using mainly gradient-based or perturbation-based techniques.

However, this approach has many limitations with regards to effectiveness of these methods for achieving true interpretability presented in [1, 2]. To address this issue, *interpretable-by-design* approaches have been developed [2, 15]. They feature decision-making processes that are inherently more understandable to humans. Concept Bottleneck Models (CBMs), which incorporate an explicit human-interpretable layer called the Concept Bottleneck Layer (CBL), also belong to this framework [8]. Notably, training a concept-based model (CBM) requires concept labels in addition to target labels. Moreover, CBMs can be trained using any of three strategies: independent, sequential, or joint training. Since obtaining additional concept labels for a dataset is a laborious process, a lot of recent research interest [13, 16, 19, 20] has been focused on using Large Language Models to obtain concept annotations. Besides training on binary concept multilabels, other works experimented using CLIP [14] similarity scores as proxy labels [13].

**Information Leakage.** The phenomenon that Concept Bottleneck Models fail to learn concepts as intended is shown in [12]. Authors show using post-hoc gradient-based methods that the learned concepts often lacked semantic meaningfulness in the input space, concluding that only independently trained CBMs may fulfill all three CBM desiderata (interpretability, intervenability, and predictability). Based on this work in [11], the authors term this failure to learn concepts information leakage and explore its mechanisms. However, no quantification nor precise definition of this behavior is provided.

<figure><img src="image_2.png" /><figcaption>Figure 1. Illustration of four cases showing information flow between concepts in a CBM and an output class. Concept activations (dots) and architecture weights (arrows) are colored green for positive and red for negative values. Concepts relevant to predicting the class “cat” are highlighted in blue, illustrating expected (Cases 1 and 3) and unexpected (Cases 2 and 4) contributions to the prediction.</figcaption></figure>

**Measuring Information Leakage.** Early work compared independent, sequential, and joint CBMs (each limited to *k* concepts) against a concept oracle. For small *k*, sequential and joint CBMs outperformed the oracle—saliency-map analyses revealed low alignment with oracle activations, indicating unintended information capture. Mahipai et al. [11] showed that even CBMs trained on incomplete or “fragile” concept sets maintained strong down-
stream accuracy, attributing leakage to soft sigmoid-based
encodings that PCA visualizations confirmed were captur-
ing broader input variance; decorrelation methods such as
Concept Whitening [4] failed to remove inter-concept pre-
dictability. More recently, Sun et al. [17] quantify leakage
by measuring accuracy drops when top-activating concepts
are removed at inference, while Srivastava et al. [16] prune
to the top-5 weights per class and measure the resulting de-
cision changes by the model. Importantly none of the pre-
sented approaches provide a direct method to measure in-
formation leakage.

**Mitigating Information Leakage.** Following the iden-
tification of information leakage and its implications for
CBM interpretability, multiple efforts sought practical so-
lutions to mitigate its effects [5, 16, 17]. In [5], the
authors proposed using hard concept representations with
side-channel information to compensate for the accuracy
drop typically associated with hard concepts. Fusing con-
cept and class prediction processes by explicitly supervis-
ing concept learning with class labels was suggested by
[17], thus directly tying concept predictions to downstream
tasks. The authors of [16] used a sparse final layer connect-
ing from the CBL to the target predictions, with the sparsity
being controlled by their NEC metric.

Collectively, the listed approaches provide different
practical modifications in order to tackle information leak-
age, however, the lack of a direct method for measuring it
makes an overall systematic evaluation of the provided ap-
proaches difficult.

3. Interpretability of CBM

**3.1. Training Concept Bottleneck Models**

CBMs are trained using supervised learning for both the
concepts and the prediction target (i.e., a class in the
context of image classification). Therefore, CBMs re-
quire supervision at two levels: the concept level and
the target label level. Formally, the dataset has the form
{⟨x⁽i⁾, y⁽i⁾, c⁽i⁾)⟩ₙᵢ=₁, where n is the number of examples,
x ∈ ℝᵈ is the input, y ∈ {1...k} the target, c ∈ ℝᵐ is
the set of concept labels associated with x, and m the num-
ber of concepts incorporated in the CBM. As a two-step
process, a CBM can be decomposed in (f ◦ g)(x) where
g : ℝᵈ → ℝᵐ corresponds to the concept detection from
the input image, and f : ℝᵐ → ℝᵏ corresponds to the tar-
get prediction from the concepts. The Concept Bottleneck
Layer refers to the intermediate layer in the network where
human-interpretable concepts are predicted g(x) = ĉ be-
fore the final output. Typically, the final output is predicted
with a single linear layer f, resulting in a CBM of the form
f(g(x)) = g(x) · W, where W is the weights matrix of the
linear layer, and Wᵢⱼ links a concept j to a class i.

Let's denote the models learned during training as $f$ and $g$. As outlined in [8], the CBM can be trained using three different strategies:

1. **Independent Training** Independent training learns $\hat{f}$ and $\hat{g}$ independently. First, $\hat{f}$ is obtained by minimizing the target loss using the concept labels $c$ as inputs, while $\hat{g}$ is obtained by minimizing the discrepancy between the predicted and concept labels. Note that even though $\hat{f}$ is trained using concept labels $c$, at test time it takes the predicted concepts $\hat{g}(x)$ as input.

2. **Sequential Training** In sequential training, the model $\hat{g}$ is first learned by minimizing the concept prediction loss. Then, $\hat{f}$ is trained using the predicted concept values, rather than the ground-truth concept labels as in independent training.

3. **Joint Training** In joint training, both $\hat{f}$ and $\hat{g}$ are learned simultaneously by minimizing a weighted sum of the target loss and the concept losses. In particular, the optimization problem is given as a one-step process, where a coefficient $\lambda > 0$ controls the contribution of the concept prediction term in the loss.

**3.2. Relevance in CBMs**

CBMs are a prime example of interpretable-by-design ap-
proaches in XAI, as demonstrated by their built-in *Concept
Bottleneck Layer* (CBL) and their two-step decision pro-
cess. During the construction of such an architecture, a set
of concepts to be represented in the CBL is chosen based on
the target classes. Crucially, this introduces additional in-
formation inside a CBM that can be leveraged to distinguish
how concepts contribute towards class predictions into rele-
vant and irrelevant contributions. In this context, the idea of
relevance merely refers to whether a concept can be mean-
ingfully involved in a reasoning to predict the class. For in-
stance, activation of the concept *whiskers* would be clearly
relevant for the class *cat*, but irrelevant for the class *plane*.
From this perspective, one of the interpretability features
of CBMs — the ability to intervene — can be understood
as the removal or addition of relevant or irrelevant concept
activations that can predict the class.

Importantly, although *relevance* has not been explicitly
named in the CBM literature so far, it is a cornerstone fea-
ture of CBMs. The class predictions of a CBM need to be
formed in an understandable and meaningful manner. For
example, if a CBM bases its decisions primarily on irrele-
vant concepts, its overall behavior and ability to generalize
cannot be trusted. From this standpoint, the issue of infor-
mation leakage is of particular concern, as unintended infor-
mation accidentally encoded in the CBL is clearly irrelevant
information.
## 3.3. Information Leakage

### 3.3.1. Defining Information Leakage

As outlined in the introduction, our proposal is to expand the definition of information leakage to encompass the impact of irrelevant information on decision-making processes. We propose leveraging the commonly available prior information about the sets of concepts considered relevant for predicting each class. Using this information, we can calculate the relevant or irrelevant contribution of each individual concept to each class.

Let's denote by $A_y$ the index set of concepts or visual attributes that are relevant for predicting class $y$. This set originates from prior or external knowledge about the class. It is characterized by the fact that its elements can meaningfully contribute to predicting the class $y$. In practice, $A_y$ can also be constructed from existing attributes or concepts ($c_j = 1$) associated to the class $y$ provided in a labeled dataset as follows:

$$A_y = \{ j \mid \exists i \in \{1, \dots, n\} : y^{(i)} = y \wedge c_j^{(i)} = 1 \}$$

In CBMs where classification is based on a linear layer, the contribution of a concept $j$ to the prediction of class $i$ can be defined as the product of the concept's activation $g(x)_j$ on image $x$ and the weight $W_{i,j}$ learned during training.

$$\sigma_{ij}(x) = g(x)_j \cdot W_{i,j} \quad (1)$$

In an ideal classifier with no information leakage, only the relevant concepts ($j \in A_i$) would contribute to the classification of class $i$. The decision process would be unaffected by irrelevant concepts ($\sigma_{ij} = 0$). However, in practice, many biases are introduced during training, which decreases the quality of the decision-making process and the model's ability to generalize.

Depending on the choice of the concept activation function and the architecture weights, $\sigma_{ij}$ can be either positive, negative, or null, leading to different interpretations of concept contribution to the classification process. The simple scenario where the activation function and the weights are restricted to a positive sign is described in 3.3.2, while the more general scenario with no constraint on the sign of the activation function and the weights is described in 3.3.3.

### 3.3.2. Positive Concept Contribution Scenario

To better understand how concepts contribute to decision-making, we first study a constrained decision process. In this configuration, we restrict $g(x)$ and $W$ to positive values: $g : \mathbb{R}^d \to \mathbb{R}^m$ and $W \in \mathbb{R}^{m \times k}$. This choice implies that the concepts can only positively impact class predictions. The interpretation of this scenario is that a class is predicted if the appropriate concepts are detected. Using this set, we can separate how concepts can contribute

towards a decision into two different cases: expected contribution with regards to the external knowledge, and unexpected contribution, as illustrated in Table 1 with Case 1 and Case 2.

Cases 1 and 2 are also illustrated in Figure 1 for the prediction of a cat and the set of relevant concepts for that class containing the attributes *Fur* and *Whiskers*. Case 1 is an example of expected behavior, in which two relevant concepts contribute positively to the class prediction. In contrast, in Case 2, one of the two concepts (*Yarn*) contributes positively to the prediction despite not being intrinsically relevant to the class. This behavior is indicative of a statistical bias, relying on unintended information. We propose to measure and quantify information leakage as the sum of all unintended contributions to the decision process. We term this metric the Irrelevant Concept Contribution (ICC+), and define it as follows:

$$ICC_{+}(x) = \sum_{i=1}^{k} \sum_{j \notin A_i} \sigma_{ij}(x) \quad (2)$$

This metric is computed across all possible classes, not just the predicted one, to evaluate the model's overall behavior. This metric is also appropriate if the quantity is used as a penalty term in the loss function to account for unintended contributions, as detailed in Section 3.4.2.

Since the metric is the sum of positive terms, it naturally increases with the number of concepts included in the architecture. We propose a normalized version of the metric that focuses on the proportion of unintended concept contributions relative to the total contribution across all concepts. We refer to this metric as the Irrelevant Concept Contribution Ratio (ICCR), which is defined as follows:

$$ICCR(x) = \frac{\sum_{i=1}^{k} \sum_{j \notin A_i} \sigma_{ij}(x)}{\sum_{i=1}^{k} \sum_{j=1}^{m} \sigma_{ij}(x)} \quad (3)$$

### 3.3.3. Unconstrained Scenario

Let's now assume that there are no restrictions on the sign of the weights and activations, i.e. that $g : \mathbb{R}^d \to \mathbb{R}^m$ and $W \in \mathbb{R}^{m \times k}$. In this configuration, each concept can contribute positively or negatively to a decision. Depending on whether the contribution is intended, we can now identify four cases instead of two, as illustrated in Table 1. A negative contribution is considered *discriminative* because it indicates that the concept negatively affects class prediction. However, it can still be relevant for characterizing the class. Note that, as described in Eq. 1, the sign of a concept's contribution depends on the sign of its activation and the sign of the associated weight. Consequently, there are eight possible combinations, each corresponding to a distinct interpretation of the contribution.
<table><caption>Table 1. Table illustrating expected and unexpected contributions of concepts j to the prediction of class i, based on the sign of the contribution σ<sub>ij</sub> and the set of relevant concepts A<sub>i</sub>.</caption><thead><tr><th>Case</th><th>Concepts</th><th>Sign of σ<sub>ij</sub></th><th>Interpretation</th></tr></thead><tbody><tr><td>1</td><td>j ∈ A<sub>i</sub></td><td>+</td><td>Relevant</td></tr><tr><td>2</td><td>j ∉ A<sub>i</sub></td><td>+</td><td>Unintended</td></tr><tr><td>3</td><td>j ∈ A<sub>i</sub></td><td>-</td><td>Discriminative</td></tr><tr><td>4</td><td>j ∉ A<sub>i</sub></td><td>-</td><td>Unintended</td></tr></tbody></table>

Fig. 1 illustrates the four cases detailed in Table 1. Figures (a) and (b) illustrate Case 1 and Case 2, respectively, as already described in Section 3.3.2. Figure (c) illustrates Case 3, where a missing relevant concept provides a negative contribution to the decision. If we expect to detect Whiskers on a cat, the absence of whiskers would negatively impact the detection score of a cat prediction. This negative contribution can be interpreted as discriminative for the class cat. The cat could still be detected if other attributes, such as Fur are identified. Figure (d) illustrates Case 4, where the decision is negatively affected by concepts that are irrelevant for the class prediction. If the model learned during the training that a cat is never in the same image as a jet engine, the detection of the concept jet engine would negatively affect the prediction of the class cat. Since there is no causal relationship between a jet engine and a cat, this behavior can be interpreted as a form of statistical bias and corresponds to an unintended contribution in the prediction, representing a case of information leakage.

In this unconstrained scenario with positive and negative contributions, we therefore define the Irrelevant Concept Contribution (ICC) as follows:

$$ICC(x) = \sum_{i=1}^{k} \sum_{j \notin A_i} |\sigma_{ij}(x)| \quad (4)$$

This metric quantifies the information leakage, taking into account the four cases of Table 1. Here again, the metric can be normalized in order to be able to compare ICC scores across models, which may not use the same sets of concepts. We introduce the Normalized Irrelevant Concept Contribution metric (NICC) as:

$$NICC(x) = ICC(x) / \left( \sum_{i=1}^{k} \sum_{j \notin A_i} 1 \right) \quad (5)$$

Note that this normalization differs from the one used in the unconstrained configuration (see Eq. 3) because negative contributions are now possible, some of which are acceptable. Therefore, the metric can no longer be interpreted as a ratio of unintended contributions. Additionally,

the NICC can only be used as a relative metric to compare different models, however it does not have an inherent standalone interpretation.

## 3.4. Addressing Information Leakage

Defining information leakage as unintended concept contribution not only enables its measurement but also provides a foundation for mitigating it. Ideally, we aim to penalize or eliminate such leakage during training. To this end, we propose two complementary methods for addressing information leakage in CBMs. The first imposes a *hard* architectural constraint, effectively preventing leakage by design. The second introduces a *soft* constraint through an additional loss term that penalizes ICC, thereby significantly reducing leakage during training.

### 3.4.1. Relevant Network Concept Bottleneck Model

In order to eliminate information leakage by construction, we applied a hard architectural constraint to the CBM architecture. In this architecture, the final layer connects each class node exclusively to its relevant concept nodes, removing unintended contributions by design. In order to construct this final partially connected layer, we leveraged the association information between concepts and classes in the sets A<sub>i</sub> and constructed a binary mask M, where M<sub>j,i</sub> = 1 if j ∈ A<sub>i</sub>, and 0 otherwise. We then used a standard fully connected layer as the final layer and, during the forward pass, performed an element-wise multiplication with the binary mask. The resulting CBM architecture has the following form: f(g(x)) = g(x) · (W ⊙ M). For the activation function of the CBL, we chose the sigmoid function, as it offers better interpretability properties than other activation functions by expressing the likelihood of a concept being present. We term this architecture the Relevant Network Concept Bottleneck Model (RelevNet CBM).

### 3.4.2. Irrelevant Concept Contribution Loss

To encourage the model to learn predictions based on relevant concepts, we included ICC as an additional term in the loss function, thereby applying a soft constraint on the model during the training. To avoid confusion, if the ICC metric has been used as a loss term, we will refer to it as the ICC Loss (ICCL) from now on. The integration of ICCL into the training objective depends on the specific training strategy employed for the CBM, as detailed in Section 3.1, and is controlled by the hyperparameter λ<sub>iccl</sub>. The training strategies that incorporate ICCL are detailed as follows:

#### 1. Independent Training

$$\hat{f} = \arg \min_{f} \sum_{i}^{n} \left[ L_{Y}(f(c^{(i)}), y^{(i)}) + \lambda_{iccl} ICCL(c^{(i)}) \right] \quad (6)$$
$$ \hat{g} = \arg\min_{g} \sum_{i}^{n} \sum_{j}^{m} L_{C_j}(g_j(x^{(i)}), c_j^{(i)}) \quad (7) $$

## 2. Sequential Training

$$ \hat{f} = \arg\min_{f} \sum_{i}^{n} \left[ L_{Y}(f(\hat{g}(x^{(i)})), y^{(i)}) + \lambda_{iccl} ICCL(\hat{g}(x^{(i)})) \right], \quad (8) $$

where $\hat{g}$ is learned in the same way as in (7).

## 3. Joint Training

$$ (\hat{f}, \hat{g}) = \arg\min_{f,g} \sum_{i}^{n} \left[ L_{Y}(f(g(x^{(i)})), y^{(i)}) + \sum_{j}^{m} \lambda L_{C_j}(g(x^{(i)}), c^{(i)}) + \lambda_{iccl} ICCL(g(x^{(i)})) \right] \quad (9) $$

# 4. Experiments

## 4.1. Effect of Training Strategy and Activation Function on Information Leakage

To investigate how information leakage and prediction accuracy vary across different architectures, we evaluated the performance of several concept bottleneck model (CBM) designs and training strategies. These experiments were performed on the CUB dataset [18], a standard benchmark in the CBM literature due to its provision of concept labels for image classification. We first compared different activation functions for the Concept Bottleneck Layer. For each activation function, the network was trained using joint, sequential, and independent training strategies. The activation functions evaluated include sigmoid, binary step with a straight-through estimator [3], ReLU, and Batch Normalization [7]. We also compared two approaches for class prediction: one constrained to positive concept contributions only, and the other with unconstrained (both positive and negative) contributions.

### 4.1.1. Experimental Setup

As in [16], we used a ResNet-18 [6] pretrained and fine-tuned on CUB as the backbone for the concept predictor, followed by a linear layer with the chosen activation function as the Concept Bottleneck Layer. A subsequent linear layer followed by a softmax function predicts the image class at the output of the network. The architecture can be summarized as $f(g(x)) = g(x) \cdot W$, where $g(x)$ denotes the predicted concepts of the image $x$, $W$ are the linear layer

weights of the class prediction, and $f(\hat{c})$ gives the class prediction from the detected concepts $\hat{c}$. During training, the backbone remains frozen. All experiments utilize the AdamW optimizer [10] with a learning rate of 0.001, a cosine learning rate scheduler [9], and $\lambda = 1$. For joint training, models are trained for 180 epochs, while for sequential and independent training, the concept predictor $g(x)$ and class predictor $f(c)$ are trained for 50 and 80 epochs respectively. Cross-entropy loss $L_Y$ is used for the classification task, while the concept loss $L_C$ depends on the activation: binary cross-entropy for sigmoid, and mean squared error for the other activation functions.

### 4.1.2. Results

Figures 2 and 3 present the experimental results, displaying test accuracy on the y-axis and information leakage (ICCR or NICC) on the x-axis. We observe that, on average, sequential training yields higher accuracies across activation functions, in contrast to claims made in [8], where joint training achieved better accuracies. Consistent with [13], independent training results in the lowest information leakage, yet with a substantial accuracy trade-off. Surprisingly, the choice of concept representation (hard for binary, soft for others) does not yield consistent results across strategies: CBMs with binary activations achieve high scores for sequential training but lower scores for joint and independent training, differing from conclusions in [11]. Notably, the architecture RelevNet provides the most favorable trade-off between accuracy and information leakage under sequential training. Contrary to our expectations, constraining RelevNet to only positive concept contributions achieves similar accuracy to unconstrained models. This is a significant step, as it combines predictive accuracy with the explanatory benefits of restricting contributions to relevant and positive concepts, thereby further enhancing interpretability and enabling more effective human intervention

## 4.2. Effect of ICCL on Accuracy

We now evaluate the impact of introducing ICCL - a soft constraint for CBMs during training - on both information leakage and classification accuracy.

### 4.2.1. Experimental Setup

We use the same experimental setup described in Section 4.1.1, and trained a model with the sigmoid activation function using sequential training. This choice is motivated by this setup achieving the highest accuracy on our previous experiment. ICCL has been incorporated into the sequential training pipeline as outlined in Section 3.4.2. We used different values of the hyperparameter $\lambda_{iccl}$ depending on whether the model is positively constrained or unconstrained, as ICCL values tend to be higher in the positive scenario. Specifically, we set $\lambda_{iccl} = 0.1$ for unconstrained
models and $\lambda_{iccl}$ = 0.001 for positively constrained models.

### 4.2.2. Results

<table><caption>Table 2. Impact of ICCL controlled with $\lambda_{iccl}$ = 0.001 on test accuracy and ICCR on the CUB dataset when output classes are predicted based on all concepts in the CBM. Concepts are detected using a sigmoid activation function and are constrained to have a positive contribution in the output decision.</caption><thead><tr><th rowspan="2">Activation</th><th colspan="2">Pos. contribution</th></tr><tr><th>Test Acc</th><th>ICCR</th></tr></thead><tbody><tr><td>Sigmoid</td><td>74.99%</td><td>53.08%</td></tr><tr><td>Sigmoid (ICCL)</td><td>73.35%</td><td>44.07%</td></tr></tbody></table>

<table><caption>Table 3. Impact of ICCL controlled with $\lambda_{iccl}$ = 0.1 on test accuracy and NICC on the CUB dataset when output classes are predicted based on all concepts in the CBM. Concepts are detected using a sigmoid activation function.</caption><thead><tr><th rowspan="2">Activation</th><th colspan="2">Unc. contribution</th></tr><tr><th>Test Acc</th><th>NICC</th></tr></thead><tbody><tr><td>Sigmoid</td><td>75.16%</td><td>3.30 × 10<sup>-2</sup></td></tr><tr><td>Sigmoid (ICCL)</td><td>69.19%</td><td>3.25 × 10<sup>-5</sup></td></tr></tbody></table>

Table 2 and Table 3 show the impact of ICCL on test accuracy and ICCR (or NICC) in the positive and unconstrained scenarios, respectively. We can see that ICCL significantly reduces information leakage as measured by both ICCR and NICC in the two scenarios. In the positive scenario, ICCR drops from 53% to 44% with a decrease of 1.64% in classification accuracy. In the unconstrained scenario, NICC decreases by three orders of magnitude with a cost in accuracy close to 6%. The results obtained in Table 2 and Table 3 reflect the choice of the hyperparameter $\lambda_{iccl}$. While ICCL can be seen as a penalization of information leakage during the training, the trade-off between information leakage and accuracy is not observed when predictions are based solely on relevant concepts by design, as is the case with RelevNet.

## 4.3. Benchmarking RelevNet and Measuring Information Leakage in CBMs

In this section, we compare RelevNet with other state-of-the-art Concept Bottleneck Models across several datasets, with a particular focus on measuring information leakage using the NICC metric.

### 4.3.1. Experimental Setup

A key challenge in evaluating CBMs across diverse datasets lies in the availability of well-annotated, binary multi-label

concept datasets. Although several recent CLIP-based annotation methods exist [13, 20], they can not be compared directly with CBMs such as RelevNet that require binary annotations at the concept bottleneck layer. To ensure direct comparability, we adopted the concept sets and annotations provided by [16] across all datasets. For all methods, we retained their original parameter budgets, concept detectors, and training configurations as described in [16].

To assess information leakage, we extended the codebases of both VLG-CBM [16] and LF-CBM [13] to measure the NICC. Each model was (re)trained according to its original scripts, and NICC was computed on the test sets alongside standard accuracy. For LF-CBM, due to the lack of binary concept labels, we used their GPT-3-generated concept sets to obtain the relevant concepts for each class.

### 4.3.2. Results

Table 4 presents the classification accuracy and information leakage, measured using NICC, for VLG-CBM, LF-CBM, and RelevNet. We show that RelevNet achieves competitive or superior accuracy compared to VLG-CBM and LF-CBM, while restricting decisions to positive concept contributions only and eliminating information leakage by design. The measurement of NICC also supports prior claims in [16] that VLG-CBM exhibits lower information leakage than LF-CBM.

While NICC offers a quantitative measure of information leakage, its interpretation can be challenging, and we plan to work on a more interpretable metric in future work. We also plan to address a broader range of classification problems with an expanded number of classes and concept sets.

## 5. Conclusion

In this work, we introduce the Irrelevant Concept Contribution (ICC), a metric for quantifying information leakage in Concept Bottleneck Models, along with a method to enforce decision-making based on relevant information during training. We evaluate and compare various bottleneck architectures and training strategies, considering both positive and discriminative contributions in the decision-making process. We provide a quantitative evaluation and show and that reducing irrelevant concept influence does not need to compromise accuracy. We believe this study marks a significant step toward improving the trustworthiness of Concept Bottleneck Models.

## 6. Acknowledgments

This work was supported by the European Union's Horizon Europe research and innovation programme under the Marie Skłodowska-Curie COFUND grant agreement No 101127936 (DeMythif.AI), and by the European Defence Fund Project KOIOS (EDF-2021-DIGIT-R-FL-KOIOS),
<table><caption>Table 4. Comparison of RelevNet with state-of-the-art models based on classification accuracy and information leakage measured by NICC.</caption><thead><tr><th rowspan="2">Dataset</th><th colspan="2">CIFAR10</th><th colspan="2">CIFAR100</th><th colspan="2">CUB</th></tr><tr><th>Accuracy</th><th>NICC</th><th>Accuracy</th><th>NICC</th><th>Accuracy</th><th>NICC</th></tr></thead><tbody><tr><td>VLG-CBM</td><td>88.63%</td><td>1.97e-2</td><td>66.48%</td><td>2.47e-3</td><td><strong>75.82%</strong></td><td>1.47e-3</td></tr><tr><td>LF-CBM</td><td>86.40%</td><td>2.78e-2</td><td>65.13%</td><td>3.47e-3</td><td>74.31%</td><td>2.32e-2</td></tr><tr><td>RelevNet</td><td><strong>89.18%</strong></td><td><strong>0</strong></td><td><strong>69.87%</strong></td><td><strong>0</strong></td><td>74.66%</td><td><strong>0</strong></td></tr></tbody></table>

<figure><img src="image_3.png" /><figcaption>Figure 2. Scatterplots showing how different training strategies affect test accuracy and ICCR on the CUB dataset. All networks are constrained to positive concept contributions. The y-axis reports test accuracy; the x-axis reports ICCR.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Figure 3. Scatterplots showing how different training strategies affect test accuracy and NICC on the CUB dataset. The y-axis reports test accuracy; the x-axis reports NICC (log scale). While RelevNet's NICC is zero by design, and since the figure is in log scale, we assign it a small nonzero value in the figure for visualization purposes.</figcaption></figure>

which co-funded the work of the second and third authors. KOIOS is a project funded by the European Union. The views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the European Union or the granting authorities. Neither the European Union nor the granting authority can be held responsible for them.
## References

[1] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In *Advances in Neural Information Processing Systems*. Curran Associates, Inc., 2018. 2

[2] David Alvarez-Melis and Tommi S. Jaakkola. Towards robust interpretability with self-explaining neural networks. *Advances in Neural Information Processing Systems*, abs/1806.07538, 2018. 2

[3] Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. *CoRR*, abs/1308.3432, 2013. 6

[4] Zhi Chen, Yuxin Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. *Nature Machine Intelligence*, 2(12):772-782, 2020. 3

[5] Marius Havasi, Shamsi Parbhoo, and Finale Doshi-Velez. Addressing leakage in concept bottleneck models. *Advances in Neural Information Processing Systems*, 35:23386-23397, 2022. 3

[6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 770-778, 2016. 6

[7] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In *Proceedings of the 32nd International Conference on Machine Learning (ICML)*, pages 448-456, 2015. 6

[8] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. *Proceedings of the 37th International Conference on Machine Learning*, page 5338-5348, 2020. 1, 2, 3, 6

[9] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In *International Conference on Learning Representations*, 2017. 6

[10] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In *International Conference on Learning Representations*, 2019. 6

[11] Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models, 2021. arXiv preprint. 1, 2, 6

[12] Andrei Margeloiu, Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian Weller. Do concept bottleneck models learn as intended? *International Conference on Learning Representations 2021 Workshop on Responsible AI*, 2021. 1, 2

[13] Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. In *International Conference on Learning Representations*, 2023. 2, 6, 7

[14] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual

models from natural language supervision. *arXiv preprint arXiv:2103.00020*, 2021. 2

[15] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. *Nature Machine Intelligence*, 1(5):206-215, 2019. 2

[16] Divyansh Srivastava, Ge Yan, and Tsui-Wei Weng. Vlg-cbm: Training concept bottleneck models with vision-language guidance. *Conference on Neural Information Processing Systems*, 2024. 1, 2, 3, 6, 7

[17] Ao Sun, Yuanyuan Yuan, Pingchuan Ma, and Shuai Wang. Eliminating information leakage in hard concept bottleneck models with supervised, hierarchical concept learning. *arXiv preprint arXiv:2402.05945*, 2024. 3

[18] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, Pasadena, CA, USA, 2011. 6

[19] An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, 2023. 2

[20] Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2023. 2, 7