There Was Never a Bottleneck in Concept Bottleneck Models
Authors: Antonio Almudévar
José Miguel Hernández-Lobato
Alfonso Ortega
Venue: ICLR 2026 Poster (2026)
Keywords: concept bottleneck models
information bottleneck
representation learning
variational inference
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Deep learning representations are often difficult to interpret, which can hinder their deployment in sensitive applications. Concept Bottleneck Models (CBMs) have emerged as a promising approach to mitigate this issue by learning representations that support target task performance while ensuring that each component predicts a concrete concept from a predefined set. In this work, we argue that CBMs do not impose a true bottleneck: the fact that a component can predict a concept does not guarantee that it encodes only information about that concept. This shortcoming raises concerns regarding interpretability and the validity of intervention procedures. To overcome this limitation, we propose Minimal Concept Bottleneck Models (MCBMs), which incorporate an Information Bottleneck (IB) objective to constrain each representation component to retain only the information relevant to its corresponding concept. This IB is implemented via a variational regularization term added to the training loss. As a result, MCBMs yield more interpretable representations, support principled concept-level interventions, and remain consistent with probability-theoretic foundations.

Published as a conference paper at ICLR 2026
THERE WAS NEVER A BOTTLENECK IN CONCEPT BOTTLENECK MODELS
Antonio Almudévar
José Miguel Hernández-Looato
Alfonso Ortega
ViVoLab, 13A University of Zaragoza a al gundevar@unizarr
Machine Learning Group University of Cambdridge
ViVoLab, 13A
University of Zaragoza o ortega@uni Taregauunizarccom .as
jmh233 cam. ac uk
ABSTRACT
Deep learning representations are often difficult to interpret, which can hinder their deployment sensitive applications. Concept Bottleneck Models (CBMs) have emerged as a promising approach to mitigate this issue by learning representa- tions that support target task performance while ensuring that each component predicts concrete concept from predefined set. In this work, we argue that CBMs do not impose a true bottleneck: the fact that a component can predict a concept does not guarantee that it encodes only information about that concept. This shortcoming raises concerns regarding interpretability and the validity intervention procedures. To overcome this limitation, propose Minimal Concept Bottleneck Models (MCBMs), which incorporate an Information Bottleneck (IB) objective constrain each representation component to retain only the information relevant to its corresponding concept. This IB is implemented via a variational regularization term added the training loss. As result, MCBMs yield more interpretable representations, support principled concept-level interventions. and remain consistent with nobability-theereee foundations.
INTRODUCTION
Most machine learning models operate by learning data pppresentatiossccompeee input that retain the essential information needed to solve a given task (Bengio et al., (2013). However, these representations often encode information in ways that are not easily interpretable by humans. This lack of interpretability becomes especially problematic in sensitive domains such as healthcare (Ahmad et al., 2018 Xie al.] 20200 Jin et al,, 2022), finance (Brigo et al.) 202T; Liu al. 2023), and autonomous driving (Kim & Canny 2017 [uu et al. 2024b). To address this issuc, Concept Bottleneck Models (CBMs) have been proposed, which enforce representations to be defined in terms of set of human-understandaale concepts (Koh et al. 2020).
versions of the
Given an input x and target y, Vanilla Models (VMs) are trained with a single goal: the representation derived from should encode all information needed to accurately predict y. In many settings, additional side information-offen referred to as concepts-is available, denoted by = {cz } Concept Bottleneck Models (CBMs) leverage this by extending VMs with second objective: each concept çj must be recoverable from a designated component 2j E By enforcing this additional constraint, CBMs are purported provide: (i) enhanced interpretability of the learned representation space, and (ii) the ability to perform targeted interventions on specific concepts by manipulating 2j and propagating the resulting changes to the model's predictions.
However, CBMs are prone to phenomenon known as information leakage (Margeloiu et al. 2021 Mahinpei al.) 202T), where the representation encodes input information that cannot be butri to the predefined concepts c. We refer to this additional information nuisances n.. Information leakage raises two main concerns: (i) it undermines interpretability, since 2, cannot fully explained by its corresponding concept cj; and (ii) it compromises the validity of seerveetionsmmodifyiiee 2 may alter not only the associated concept Cj. but also other unintended information encoded in žj.
We argue that information leakage stems from fundamental limitation in the current formulation of CBMs: the absence of an explicit Information Bottleneck (IB) (Tishby et al. 2000) that actively constrains 2j to exclude information unrelated to cj. While the second objective in CBMs encourages each 2j to retain ej in its entirety, it does not enforce that 2j captures only information about cj. In the worst-case scenario, 2, could encode the entire input x and still satisfy this objective.

---

Published as a conference paper at ICLR 2026
To address this issue, propose Minimal Concept Bottlencck Models (MCBMs), which incorporate IB into each 2j. This ensures that 2j not only retains all the information about its associated concept Cj, but also ex- cludes any information unrelated to cj. as summarized in Table | The name reflects that 2j is trained to be minimal sufficient statistic of cj (Fisher, 1922 I935). in contrast to traditional CBMs where 2, is optimized to be merely a sufficient statistic of cy. As illustrated in Figure this design yields disentangled representations that directly address the two shortcomings of CBMs: (i) it improves interpretability, since 2j can fully explained by its corresponding concept ey; and (ii) it enables valid interventions- moddfying 2j affects only the associated concept Cy.
Table Differences between VMs, CBMs and MCBMs.
VMs
CBMs
MCBMs
Does žj encode all e;?
Does 2, encode only oj?
X
quanti- ties, showing that the IB can be implemented via variational loss. In Section 3, we review alternative approaches to address information leakage. In Section 1 we present experiments demonstrating that MCBMs enforce true bottleneck, thereby enhancing interpretability and intervenability compared to existing alternatives. Finally, in Section we show that the assumptions made in CBMs to enable interventions are theoretically flawed.
In Section 2
we connect the data generative process to MCBMs through information-
C2
C3
Vanilla Encoder
CBM Encoder
MCBM Encoder
#1
22
22
21
F2
Figure 1: In Vanilla Models, concepts and nuisances may be arbitrarily entangled in the representation, and variable 2j may capture only part of concept (depicted as paler colors). In CBMs, each z; encodes all information about its corresponding concept Cj. but may also capture some information about nuisances (e.g., 21) or other concepts (e.g., 22). In contrast, MCBMs enforce that each representation variable zj encodes all-and only the information about its corresponding concept.
FROM DATA GENERATIVE PROCESS TO MCBMS
2.1 DATA GENERATIVE PROCESS
For this scenario, we consider inputs x E X, targets y E V, concepts c= {c E and nuisances n E N, such that p(æ, y, C, n) = p(æ) c, n n)p(y/kx)p(, n). i.e., the inputs æ are described by the concepts and the nuisances n, and the targets difference between e and n is that the former are observed, while the latter are not or, in other words, labels on are provided to us. We assume access to training set (x)(), y(d) c()) (i). N
fully described by the input æ. The only
Ti=1, which defines
the empirical distribution p(æ: y, c) Σ (a (0)) (y y ( 8 (c cf The graphical
model corresponding to this generative process is shown in Figure 2a for the case of two concepts.
2.2 VANILLA MODELS
In machine learning, the most commonly studied problem is that of predicting y from æ, which serves as a foundation for more specialized tasks. We refer to models trained to address this problem as Vanilla Models (VMs). These models typically operate by first extracting an intermediate representation E Z from the input æ via an encoder (e(z x) parameterized by 0. Subsequently, a prediction ÿ E y produced from z using task head ( 2) parameterized by Φ. Since is intended to facilitate accurate prediction of y, the mutual information between z and y, denoted [[Z; Y), should be maximized. Appendix B. we formally show that:
max [(Z; Y) 8,0
max Ep(x.)) [Epo(tta) [log go (ū]=)]
(1)
2
2

---

Published as a conference paper at ICLR 2026
Figure 2b shows the graphical model of a Vanilla Model with wwo-dimensional representation
Black edges represent the encoder po(2 z), while green edges indicate the task head q6 x). The encoder is typically chosen to be deterministie, i... pa æ) = 5 (z fz))) where fo X Z is a adopt a stochastic formulation where po(= x) = N (z; fo(z)), ( T). as summarized in Table 2 The choice of task head ( z) depends the structure of the output space V/ also detailed in Table 2
mapping. However, for tractability reasons (see Section 2.4 we
The objective in Equation I corresponds to minimizing the cross-entropy loss when y is binary or multiclass, and to minimizing the mean squared error between y and (2(2) when y is continuous.
2.3 CONCEPT BOTTLENECK MODELS
Vanilla Models generally lack interpretability with respect to the known concepts c, as the encoder fe is often opaque and difficult to analyze. Moreover, these models tend to entangle the concepts in such way that it becomes intractable determine how individual concepts influence specific components of the latent representation z, and consequently the predictions J. Concept Bottleneck Models (CBMs) have been introduced to address this limitation. In a CBM, each concept Cj is predicted from a dedicated latent representation 2j via concept head g(é; I 2z). Consequently, 2j must encode all information about --tthat is, 2j must a sufficient representation for cj. This requirement can be formalized as maximizing the mutual information [Zz; Cj), for which the following identity- pranvi in Appendix B.2 is employed:
max [Zz;; Z,
max E Epta..,) 0,6
[pp(ttlac) [To 90 (C,le,))
(2)
As illustrated in Figure 2c
CBMs extend Vanilla Models by incorporating a concept head q(è,
2,).
depicted with blue arrows. These models jointly optimize the objectives in Equations T and 2 As detailed in Table 2 the form of the concept head go(C, 22) depends on the nature of the concept space C. The objective in Equation corresponds minimizing the cross-entropy loss when ej binary or multiclass, and the mean squared error between Cj and (zz)) when is continuous.
How are Interventions Performed in CBMs? As explained in Section | key advantage often attributed to CBMs is their ability to support concept-level interventions. Suppose we aim to estimate P( ej = a. 2). In CBMs, this intervention is performed through the latent representation 2y:
p = o., x) /f P(ile so)p(e,,e, = o))(zzy) dz, dzy
However, the conditional distribution p(zy cj) is not defined tthrre is no directed path from oj to 2j in Figure 2c Intuitively, because 2j may encode information about beyond cj. it cannot be fully determined by Cj alone. This raises a key question: how can interventions be performed in CBMs if p(z, (₂) is unknown? To make interventions feasible, CBMs typically impose two constraints:
(i) Concepts 6 E c, are assumed to be binary. If concept originally multiclass with categories, it is converted into k binary concepts (One-vs-Rest (Rifkin & Klautau| (004)).
(ii) The concept head is defined as (c(c;) 22)) = o22)), where a denotes the sigmoid function.
Since a is invertible, this setup permits defining p(2; eg) 2 (c;). However, this ill-defined at the binary extremes, as (1) a (0) = xo. To address this, in practice, p(z; Cj = 0) and p(2, c= = 1) are set as the 5th and 95th percentiles of the empirical distribution of zj. respectively
(oo et al. 2020 This workaround, however, introduces two crucial issues discussed in Section
Table 2: Distributions considered in this work. So X + E is typically a large neural network while 9% Z + V., Sf : Z - C and g C - Z are comparatively lightweight networks (see continuous latent representation.
Appendix F.I. Throughout this work, we model
po (2(2
qs (0(())) 2)
go(d)12;)
(o(2))c))
Binary
Bernoulli (z)
Bernoulli 95(2g)
Multiclass
Categoric
(2((2)
Categoric (8((2))
Continuous
N of))
N (g)8(2) (2)()))0))
N
g8 (g8()))
N (g5(c))
3

---

Published as a conference paper at ICLR 2026
2.4 MINIMAL CONCEPT BOTTLENECK MODELS
As discussed in Section I CBMs lack an explicit mechanism for enforcing a bottleneck, which often undermines their intended advantages. To address this limitation, we introduce Minimal Concept Botteneck Modely (MCBMs), which explicitly impose an Information Bottleneck. This ensures that each retains all information about its associated concept Cj. while excluding information unrelated to Cj. In other words, 2j becomes a minimal sufficient representation of cj. To achieve this, we introduce a representation head ( (e) that predicts 2j from Cj. This encourages 2j to discard information unrelated to c,. as doing so improves the predictive accuracy of (2g (;). Formally, this objective corresponds to minimizing the conditional mutual information I(Z X C2).
). We note that
if 2y is a representation of æ, then the following propositions are equivalent: (i) [(5 XIC)) =
(ii) the Markov Chain X ++ Cj ++ Zj is satisfied and (iii) p(zy(e)) = p(z,lar). To minimize
T(Zy; X | Cj), we leverage the following identity, which proven in Appendix B.3
min (Z2 x(C)) = min Bp(a..,) [DRE (po (=0(2)l=))l00())111 0,0
(4)
2,
Figure 2d illustrates how MCBMs extend CBMs by introducing the representation head g(ē, ()), depicted with red arrows. These models are trained to jointly optimize the objectives given in Equations [ 2 and 4 Although the KL Divergence in Equation 4 does not admit a closed-form solution in general, we show in Appendix B.4 that, under the distributional assumptions listed in it reduces to the mean squared error between (z( and (cg).
Table 2
How are Interventions Performed in MCBMs? In contrast to CBMs, MCBMs explicitly constrain 2j to contain only information about cj. As a result, modifying 2j corresponds intervening solely on ej. This is reflected in Figure 2d where MCBMs introduce a directed path from ej to zj through the intermediate variable 2z. This structure permits the computation of:
p(zzle))= - p(:; p(zz|2)) (2)))cc))) d2,
(5)
which simplifies to p(zz |c,) = [=)= (zy I (e)) when p(2) 22) = ö(2; - 2,). i.e.. once the objective in Equation (iiiioo///sssss is optimized and 2j encodes exclusively cj. As shown in Table 2 we define the encoder distribution as (z( (₃)
where mean function g (c;) is chosen according to
the following rules:
(i) For binary concepts, (g,),) = A if ey 1, and gg(C,) = - otherwise.
(ii) For categorical concepts, g (c,) = X one hotchot(c;). This mirrors the Prototypical Learning Snell et al. 2017) approach where class-dependent prototypes {})())))} are fixed.
(iii) For continuous concepts. gg(c)) = X Cj.
Here, À is a scaling constant that controls the norm of the latent representation, fixed to À = 3 in all experiments. Regarding the variance term, set: (i) a2 0 in the case of CBMs to obtain deterministic encoder in line with their original formulation, and (ii) a = Og = 1 for MCBMs.
Practical Considerations for Optimizing MCBMs To optimize MCBMs, we combine the three previously introduced objectives, incorporating two key considerations. First, approximate the expectations over p(z, y) and p(z,c, cÿ) using the empirical data distribution, replacing integrals with summations over the dataset. Second, to enable tradient-based optimization through the stochastic encoder, we apply the reparameterization trick (Kingma 2013): TEpa( x)) [log go (o(y)2) =)] wo
Σ, Log 96 (ylf (I, ())) and Epo(zz1m) [log 4o (Ggl24) ~ Σ, (eylfb, (zz, (2), where cz = fo(x)= + oLLe (due to the choice of po(x(<)) in Table 2. e ~ N (0, and fa., (z, e) corresponds to the element of I' (æx e). Combining the considerations above yields the final objective for training MCBMs, as shown in Equation 6 where 3 and y are hyperparameters. The first term corresponds to the objective used in Vanilla Models, the second term is introduced in CBMs, and the third is specific to MCBMs. Detailed training algorithms for the various cases listed in Table 2 are provided in Appendix C
log 96 (8),6 (()))(())) +B
log a (cylfes (2((),c(o)
max
Rm
(6)
|| DEL (Do (2)) go (2)lo(1))
4

---

Published as a conference paper at ICLR 2026
F1
y
3
(a) Data Generative Process
(b) Vanilla Model (VM)
C1
21
C1
&1
21
y
ès
(c) Concept Bottleneck Model (CBM)
(d) Minimal Concept Bottleneck Model (MCBM)
Figure 2: Graphical models of the different systems described for concepts and two-dimensional representations. Appendix shows the analogous figure for m concepts and m-dimensional repre- sentations. Inputs are defined by some concepts ] and nuisances næ; and targets y are defined by (gray arrows). Vanilla models obtain the representations { from through the encoder po(2|æ> (cyan arrows) and solve the task ý sequentially through the task head qo z)) (green arrows). Concept Bottleneck Models make prediction êj of each concept cj from one representation 2j through the concept head (z))) (blue arrows). Minimal CBMs make a prediction &, of each representation Zj from one concept Cj through the represenlation head qo (0(2glc5 (c)) (red arrows).
3 RELATED WORK
Information Leakage in Concept Bottleneck Models Information leakage occurs when the learned representation z encodes information outside the concept set reducing both interpretability and intervenability (Margeloiu et al. 202T Mahinpei et al. (02T). This arises when the Markovian assumption fails-i.e., i.e., when the target y is not fully determined by the concept
equivalently,
p(yle (c) * ppyle c, æ), which is typically the case in real-world scenarios (Havasi et al. 2022). Con- cept Embedding Models (CEMs) (Espinosa Zarlenga et al., 2022) were proposed to mitigate the accuracy-imteppretabllty trade-off in CBMs. However, this trade-off fundamentally limited by the chosen concept set, and CEMs may even be less interpretable than standard CBMs, their more entropic representations tend to amplify information leakage, a critique formalized in (Parisini et al., 2025). Havasi et al. (202)) also introduced Hard Concept Bottleneck Models (HCBMs), which predict from binarized concept predictions êj rather than from z (see Appendix DD. thereby imposing an ad-hoc Information Bottleneck. Extensions such Autoregressive CBMs (ARCBMs) and Stochastic CBMs (SCBMs)(Haaat et al. 2022; Vandenhiriz et al. 2024) incorporate dependencies between concepts. Energy-based CBMs (Xu al. (024a)) replace the task head with an energy function that scores concept-label compatibility, enabling richer and more structured concept relationships. Beyond architectural changes, prior work has characterized leakage using information thheoretic quan- tities (Parisini et al. 2025 Makonnen et al. (2025). To the best of our knowledge, this is the first work that leverages Information Theory to (i) formally identify the underlying design Maw-CBMs require each 2j to predict сj but do not restrict 2₁ from encoding additional nuisance information and (ii) introduce a principled, variational IB objective that directly constrains each latent variable to retain only coneept-relevant information.
Information Bottleneck in Representation Learning The Information Bottleneck (IB) (Tishyy et al. (0000) provides a principled way to balance preserving information about a factor with compress- ing the representation. In this framework, representation is sufficient if it retains all the information
5

---

Published as a conference paper at ICLR 2026
about the variable of interest, and minimal if it contains only that information (Achille & Soatto 2018a b. Shwartz Ziv & LeCum 2024). Computation of these quantities is intractable, so variational methods have been developed to derive tractable evidence bounds (Alemi et al. 2016 Fischer 2020).
4 EXPERIMENTS
In this section, we present series of experiments designed empirically demonstrate that CBMs fail to impose an effective bottleneck, even in simple settings. We also examine the consequences of this limitation. In contrast, we show that MCBMs successfully enforce a bottleneck, thereby mitigating the issues that arise in its absence. Further implementation details and training hyperparameters for each experiment- are provided in Appendix
encoder architectures
4.1 Do CBMS AND MCBMs LEAK INFORMATION?
Some works assume that the task is fully defined by the concepts c. However, is unrealistic to ex- pect a finite set of uumman-nnderstaddalle concepts to completely describe an arbitrarily complex task. In practice, certain nuisances that describe the input æ also influence y. Specifically, we decom- pose the nuisances n {n, n.g). where ny captures nuisances that, together with the concepts, describe y. while n comprises those independent of y (see Figure 3a). As discussed throughout this work, CBMs provide no incentive to remove infor- mation unrelated to the concepts. Moreover, since the objective of most models is to solve y. they are incentivized to preserve not only but also
ngy
f2
nyy
y
Fyy
c
c
(a) Data
(b) Representation
Figure 3: Some nuisances ny E n affect the task y while others ny E n do not. None of them should affect the representation z since it must be fully described by the concepts c.
Ny. While this may improve task performance, it comes at the expense of interpretability and valid interventions. For instance, if 2j is intended to represent the concept cj c, one might assume that modifying 2j corresponds solely to intervening on cj. However, if 23 also encodes information about ny (or even ng we show later), then modifying &j also affects the nuisances, invalidating any causal conclusions. By introducing the Information Bottleneck in Equation 4 we explicitly constrain the model to remove nuisances from zy, thereby restoring the validity of causal analyses involving cj- We note, however, that this necessarily reduces task performance: if e is incomplete and solving y requires information from n (i.e., ny 7 0), excluding from will lower predictive performance.
We next examine whether ny and ny are present in the representations across different CBM variants and datasets. For this purpose, we define the following task-concept configurations: (i) MPI3D (Gonda et al. 2019), where is the object shape, nh, the horizontal axis, Tig the vertical axis, and the remamiin generative factors; (ii) Shapes3D (Kim & Mnih 2018), where y is the shape, ny the floor color and wall color, ny the orientation, and c the remaining factors, (iii) CIFAR-10 (Krizhevsky al. 2009), where y is the standard classification task, e consists of 64 of the 143 attributes extracted by Oikarinen et al. (2023) using GPT-3 (Brown et al. (020)) and ny the remaining attributes, since all nuisances are correlated with y: (iv) CUB (Wah et al. 201 where y the bird species, includes concepts from twelve randomly selected attribute groups, and ny the attributes from the remaining 15 groups; and (v) AwA2 (Xian et al.) 2017). where y is the animal class, c consists of 20 of the 85 natively support multiclass concepts, the other baselines in our study are limited to binary concepts. To ensure fairness, factors with & classes are therefore represented as & binary concepts.
attributes, and n, the remaining attributes. While MCBMs
Task-related information leakage To measure the presence of a nuisance factor n, E ny in we estimate I(N;; Z Che information about ny contained in z beyond what is explained by c. Since this quantity is intractable, we approximate it as 1 i(N;; Z (((NIC) (NrIC, Z), where iI C)= - -IN log (c(t)) and (N C, -
E log h (c(k), (())). Here, hG, and he are MI. classifiers trained to predict ny from e and (c, z), respectively. In Table we
report the average value of i(N;:2IC)
H(N,) across all n, E Nyy which we call Uncertainty Reduction Ratio (URR). From these results, conclude that: (i) CBMs tend to reduce nuisance information compared to VMs, though not consistently; (ii) CEMs and ECBMs generally preserve the largest amount of nuisance information, often exceeding even VMs; (iii) ARCBMs and HCBMs show no systematic

---

Published as a conference paper at ICLR 2026
advantage over CBMs in terms of nuisance removal; and (iv) MCBMs provide the strongest reduction of nuisance information, particularly as y increases, which enforces a stricter bottleneck.
Table 3: Average value of URR for task-related nuisances Rly.
MPI3D
Shapes3D
CIFAR-10
CUB
AwA2
Vanilla
35.0 + 1.9
45.5 + 6.4
19.8
3.8 + 1.0
1.5 + 0.3
CBM
28.1
2.9
0.7
0.8
1,4 0.3
CEM
3.9
0.8
1.1
1.5
ECBM
23.2
3.7
1..1
4.5
1.3
ARCBM
1.7
18.4
1..2
3.9
1.3
SCBM
24.3
1..8
1.7
3.8
1.2
MCBM (low ?)
10.7
0.1
0.5
3.9
1.0
MCBM (medium 7)
6.7
0.2 0.3
1..
0.9 0.4 0.8
MCBM (high 7)
0.0 0.0
0.0 + 0.0
17.6 0.5
2.4 1.0
0.7 + 0.4
Tark-unde information leakage Neural net-
Table 4: Average value of URR for task- nui- sances.
works typically discard input information, as their
layers non inneererteb (Tishby & Zaslavsky| [0155] Tschannen et al. 2019p. Moreover, since n2 is ir- relevant for predicting y. there is no incentive to retain it in the representation z. One might there- fore expect be free of ngg. However, prior work has shown that neural representations often preserve information directly related to the task (Achille & Soatto| 201 Sal Arjovsky et al. (019). To exam- ine this, we analyze whether ng is present in for MPI3D and Shapes3D- the only settings where ny non-empty. Table 4 reports URR values for the nuisance variables in ny in. We find that: (i) as in Table 3 CEMs and ECBMs retain more
MPI3D
Shapes3D
Vanilla
11.3 0.1
42.7 + 9.1
CBM
7.4
20.6 3.3
CEM
40.9 4.2
ECBM
6.2
46.4
ARCBM
8.7
26.6
SCBM
0.3
2.7
MCBM (1
0.0
0.0
MCBM 7)
0.0
0.0
MCBM (h 7)
0.0
0.0
0.0 0.0
nuisance information than even VMs; and (ii) MCBMs are the only models that consistently climinate nuisances across all values of as expected: with no incentive to preserve ny and an explicit penalty for doing so, such information is naturally discarded.
Are concepts removed to greater extent in MCBMs? One might worry that removing nuisance information could also inadvertently eliminate informa- tion about the concepts. To test this, we report concept prediction accuracy for CIFAR-10 10 and CUB in Table 5 (note that all models reach 100% accuracy MPI3D and Shapes3D). We can observe that model consistently out- performs the others- some preserve more concept information in CIFAR-10, while others perform slightly better in CUB; and (ii) increasing ? in MCBMs gradually reduces concept accuracy, as stronger regularization may suppress features correlated with the concepts. These results indicate that MCBMs effectively remove nuisance information while largely preserving concept-relevant content.
Table 5: Average concepts accuracy
CIFAR-10
CUB
AwA2
CBM
84.8 1 0.2
96.3
98.1 0.1
CEM
84.8 + 0.2
9.1
98.0 0.1
ECBM
0.2
9..1
9.1
ARCBM
8..
9.1
9.1
SCBM
84.3+
9..5
98.4
MCBM (17)
84.9 0.1
9.2
9.2
MCBM (m 7)
84.9 + 0.2
9.11
97.9 + 0.1
MCBM (h
+.1
95.8 0.3
+.2
How does this affect task performance? As previously discussed, when ny is non-empty, restrict- ing to contain only information about e should reduce performance on y: if e alone is insufficient to solve y. then z cannot achieve perfect accuracy. In Table we observe the following: (i) CBMs, CEMS and ECBMs reach task accuracy comparable to (or even higher than) VMs, indicating that they also rely nyy to predict y: (ii) ARCBMs and SCBMs achieve lower task accuracy, they impose a bottleneck after the representations (see Appendix D); and (iii) MCBMs show decreasing task accuracy as y increases. reflecting the stricter bottleneck applied to the representations, Importantly, this reduction in task accuracy should not be viewed negatively: since models achieve similar concept accuracy (see Table 5 it indicates that predictions rely less nuisance information.
7

---

Published as a conference paper at ICLR 2026
Table 6: Task accuracy
MPI3D
Sh
CIFAR-I0
CUB
AwA2
Vanilla
100.0 0.0
100.0 0.0
72.1
77.4
91.6
CBM
99.9 0.0
100.0 0.0
7..6
91.1
CEM
1.0
100.0
73.1
0.3
9..7
ECBM
9.0
1.0
7.4
0.5
9.3
ARCBM
2.6
2.47
0.3
0.8
0.3
SCBM
24.2 0.5
0.3
6.0
7.4
9.1
MCBM (1 (17)
9.1
0.0
0.5
78.3
0.6
9..
MCBM (m))
46.0 0.6
7
77.4 7.440.4
9.21
MCBM ()
24.9 0.3
3.21
70.5 0.6
73.5 + 1.5
88.7
4.2 Do MCBMS YIELD MORE INTERPRETABLE REPRESENTATIONS
One of the key properties often attributed CBMs is that their internal representations align with cumaa-interpretabbl concepts (Debole et al. (025). While this generally holds, we show that MCBMs yield representations that are even more interpretable.
support this claim, we employ two metries: (i) Centered Kernel Alignment (CKA) (Cristianini et al. 2001 Cortes et al. 2012 Kornblith et al. 2019). which measures the similarity between learned re representations and concept labels (encoded onc-hot vectors); (ii) Disentanglement (Eastwood & Williams 2018), which assesses whether each dimension of the representation 2; captures at most one concept cj: and (iii) Oracle Information Score (OIS) (Zarlenga et al. (223), which dextends beyond standard disentanglement by accounting for concept dependencies, requiring that correlations between 2j and 23. do not exceed those between Cj and cę- These metrics capture interpretability from complementary perspectives:
(i) CKA indicates whether the information is organized in a concept-aware fashion; and (ii) Disentanglement and OIS reflect the extent to which individual concepts are independently encoded in the representations. As shown in Figure 4 As shown in Figure 4 (i) CBMs, CEMs, AR-CBMs, and HCBMs generally achieve higher CKA with concepts than Vanilla Models, but they do not consistently improve Disentanglenen or OIS; (ii) ECBMs do not, in general, produce more interpretable representations than VMs; and (iii) MCBMs consistently improve CKA, Disentanglement, and OIS, with larger gains as y increases, reflecting the removal of additional nuisances.
These results indicate that explicitly removing nuisances leads to more interpretable representations. To analyze this systematically, Table reports the rank correlations between the metrics in Figure 4 and the URR values in Table 3 across datasets. We observe that CKA and OIS are strong and consistent predictors of nuisance information, whereas disentanglement is noticcably weaker. This is particularly helpful for tuning "y: since ny is unobserved in real scenarios, URR cannot be computed in Table As workaround, these metrics- computed solely from and serve practical proxies for selecting the trade-off between task accuracy and leakage.
Vanila
1.87
2.88
18.81
39.
19.94
42.58
20.86
14.52
23.61
26.00
1921
22.00
CBM
78.05
71.02
27.21
62.05
86.32
26.67
15.87
63.47
25.80
27.18
10.51
21.79
27.82
9.30
9.66
72.43
67.91
26.05
84:20
37.05
66.66
27.27
10.07
ECBM
25.02
0.36
20.03
ARCBM
85.77
$7.78
23.23
85.91
90.34
34.72
34.05
$1.92
24.99
30.25
19.35
28.49
24.91
9.41
9.62
SCBM
86.91
67.64
22.14
69.72
58.04
12.90
50.98
11.22
34.15
16.47
26.71
25.16
9.04
9.89
MCBM
(medium dilm
MCBM
59 63
99.96
38.69
80.77
92.26
77.23
93.00
67.36
61.92
87.89
7.08
3.05
17.59
8.89
9.59
[high
MCBM
99.30 98
99.30
41.21
83.75
91.57
85.55
98.
76
70.49
$7.05
91.83
6.21
2,53
19.82
8.1
9.48
MPDD
3tapo830
C998-10
QuB
Awdi
MP130
ShapseiD
GF48-10
CUB
MP130
Shapo:30
CRA8-10
Qm
Figure 4: CKA (left, 1), Disentanglement (middle, 1) and OIS (right, d)
8

---

Published as a conference paper at ICLR 2026
Table 7: Rank Correlation (p-value) between each metric and URR for By across datasets
MPI3D
Shapes3D
CIFAR-10
CUB
AwA2
CKA
.95 (9.550000 (<.001)
-99
-.71
-.77 (.02)
-.75 (.04)
Disentanglement
-.81 (.01)
-.57 (.13)
-.43 (.29)
-669 (.05)
-999 (<.00))
OIS
.90 (.002)
.81 (.01)
.86 (.006)
.89 (.002)
.66 (.07)
4.3
Do MCBMs ENABLE MORE RELIABLE INTERVENTIONS?
Another key property often attributed to CBMs is their capacity support interventions. Although standard CBM intervention methods suffer from theoretical limitations (see Section 5. they can still be applied in practice. Accordingly, for CBMs follow the procedure in Section 2.3 for ARCBMs and SCBMs we use the procedure described in Appendix D and for MCBMs we apply the method in Section 2.4 To evaluate intervention reliability across models, we adopt the standardized protocol of Koh et al. (2020), which tracks how prediction error changes as the number of intervened concepts increases. Following Shin al. (2023). ) we examine the effect of interventions using two complementary policies on CIFAR- 10, CUB, and AwA2: (i) intervening on the concepts with the lowest predicted confidence (results in Figure 5): and (ii) intervening on randomly selected concepts (results in Appendix D.. For MPI3D and Shapes3D, interventions yield only minor changes in performance, which aligns with Table showing that their concepts are relatively weak predictors of the target. Across datasets where concepts informative, both intervention policies lead to consistent and converging conclusions: (i) CBMs may even increase error when multiple concepts are intervened effect of nuisance information leaking into the representation due to the absence of proper bottleneck; (ii) ARCBMs and SCBMs mitigate this by applying an Information Bottleneck after the representations, which improves intervention effectiveness while leaving interpretability unchanged (Figure 4) (iii) in MCBMs, intervention gains remain mostly insensitive to when only small or moderate number of concepts are intervened evidenced by the nearly parallel curves across values but increase with higher when more concepts are intervened, as reflected in steeper negative slopes; (iv) at low and medium 7, MCBMs deliver the strongest intervention performance, especially when fewer than 100% of the concepts are intervened, clearly outperforming ARCBMs and SCBMs; and (v) for 1wA2-wheee all models exhibit low levels of nuisance retention (Table intervention performance nearly identical across methods.
CBM
ARCBM
SCBM
MCBM (low y) MCBM (med.
(high
40
12-
31
18
6
22
9
0-
(a) CIFAR-10
(b) CUB
(c) AwA2
Figure 5: Error (y-axis) versus percentage concepts intervened (x-axis) across different models.
OTHER FUNDAMENTAL THEORETICAL FLAWS OF CBMs
As briefly discussed Section 2.3 CBMs- MCBMs-do not provide a principled mechanism to estimate p(z; c;). To enable interventions, two assumptions are typically introduced: (i) multi- class concepts are handled using a One-vs One-vs-Rest scheme, and (ii) interventions are implemented via the sigmoid inverse function, i.e., p(zy c,) z 5 (z, hoc but also theoretically incorrect, as we explain and illustrate with toy experiments below.
(c))). These assumptions not only ad
One-vs-Rest Limitations One-vs-Rest strategies exhibit several limitations: (i) individual binary classifiers tend to be biased toward the negative class, and (ii) the predicted probabilities across clas- sifiers are typically uncalibrated (Bishop) 2006). To illustrate these issues, we design an experiment where the concepts are defined based on four-class spiral dataset with imbalanced class distributions. We train: (i) a CBM with One-vs-Rec binarization, and (ii) an MCBM modeling concepts directly as multiclass variables.

---

Published as a conference paper at ICLR 2026
The results reveal three major shortcomings of CBMs trained with the Ane-vs-Rest strategy. First, Figure (left) shows that they overpredict the most frequent class, especially when the true class is rare and spatially close to the dominant. Second, Figure [ (middle) shows that CBMs produce poorly calibrated predictions, lacking the smooth likelihood transitions observed in MCBMs. Finally, Figure [ (right) shows that CBMs often assign near-one likelihoods to mul- tiple classes simultaneously, whereas MCBMs confine high secondary likelihoods to regions of overlap, producing more reliable uncertainty estimates. Importantly, these effects may not be captured by standard metrics like accuracy. yet they represent fundamental flaws from proba- bilistic perspective.
Figure 6: CBM (top) versus MCBM (bottom): class boundaries (left), highest predicted likelihood (middle), and second -hiondt predicted likelihood (right).
Sigmoid Inverse Function
The intervention
procedure p(z; Cj) 2 8 (2j - -c-
0.5
pizl
pizic= 1)
does not satisfy Bayes' rule, i.e., p(z. l5)
(c-1)
gole; sxlp(=s) For example, illustrated in
pirlc- o)
p(G.)
(c=
7 when the prior p(2y) is bimodal-4 common case for representations arising from two classes- (3) differs markedly from
Figure
0.2
(cy). Ignoring the prior therefore not only violates probability theory but also yields poor practical approximations. This issue is diffi- cult overcome CBMs, as the prior p(z,) is unknown and challenging to estimate.
0.0
-3
0
Figure 7: Density (y-axis) vs. z (x-axis)
6 CONCLUSIONS, LIMITATIONS AND FUTURE DIRECTIONS
In this paper, we argue chat-contrary common belief-Concept Bottleneck Models (CBMs) do not enforce a true bottleneck: although representations are encouraged to retain concept-related information, they are not constrained to discard nuisance information. This limitation undermines interpretability and provides no theoretical guarantees for intervention procedures. To address this, we propose Minimal Concept Bottleneck Models (MCBMs), which introduce an Information Bottleneck in the representation space via additional loss term derived through variational approximations. Beyond enforcing a proper bottleneck, our formulation ensures that interventions remain consistent with Bayesian principles. Empirically, we show that CBMs and their variants fail to remove non- concept information from the representation, even when irrelevant to the target task. In contrast, MCBMs effectively eliminate such information while preserving concept-relevant content, yielding more interpretable representations and principled interventions. Finally, we highlight fundamental flaws in the intervention process of CBMs, which MCBMs overcome due to their principled design.
Regarding the limitations of this work, we highlight the following: (i) MCBMs introduce a new hyperparameter, 7. which must be tuned balance predictive accuracy and interpretability; (ii) the representation head of adds a small number of parameters, though this overhead is negligible relative to the backbone; and (iii) while MCBMs consistently reduce nuisance information, complete removal remains challenging for high-variance datasets, suggesting that performance depends on the expressive capacity of the backbone architecture.
As for future directions, promising avenues include: (i) extending the model with auxiliary la- tent variable 2m+1 appended to z to explicitly capture task-relevant nuisance factors ng. allowing 21..... 2m remain strictly interpretable while still retaining task-useful information in the full rep- resentation x; and (ii) studying how the choice of prior distributions (2) (t)) affects representation quality and evaluation metrics such as disentanglement, alignment, and concept leakage.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 101007666, MCIN/AEE/10.110 13039.1000/1110000 100011033 under Grant PID20024-559480B--8 and the Govermentt of Aragón (Grant Group T36 23R).
CODE AVAILABILITY
An implementation accompanying this work is available at https://github.cm com/ antonioa mudevarrmminima _cbm
REFERENCES
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. Journal of Machine Learning Research, 19(50):1-34, 2018a.
Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE transactions on pattern analysis and machine intelligence, 40 (12):2897-2905, 8b.
Muhammad Aurangzeb Ahmad, Carly Eckert, and Ankur Teredesai. Interpretable machine learning in healthcare. In Proceedings of the 2018 ACM international conference on bioinformatics, computational biology, and health informatics, pp. 559-560, 2018.
Alexander A Alemi, Ian Fischer, Joshua Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410 2016.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.0288: 02289,, 2019.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new
perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1828, 2013.
Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006. ISBN 978-0- 387-31073-2.
Damiano Brigo, Xiaoshan Huang, Andrea Pallavicini, and Haitz Saez de Ocariz Borde. Interpretability in deep learning for finance: case study for the heston model. arXiv preprint 1rXiv:2104.09477 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. The Journal of Machine Learning Research, 13(1):795-83) 8288, 2012.
Nello Cristianini, John Shawe-Taylor, Andre Elisseeff, and Jaz Kandola. On kernel-target alignment. Advances in neural information processing systems, 14, 2001.
Nicola Debole, Pietro Barbiero, Francesco Giannini, Andrea Passerini, Stefano Teso, and Emanuele Marconato. If concept bottlenecks are the question, are foundation models the answer? arXiv preprint wwxXiv25004.17744, 2025.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations. In International conference on learning representations, 2018.
Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, et al. Concept embedding models: Beyond the cesuracy--xplainablli.. trade-off. Advances in Neural Information Processing Systems, 22221400-21413 2022.

---

Published as a conference paper at ICLR 2026
Ian Fischer. The conditional entropy bottleneck. Entropy. 22(9):999, 2020.
Ronald A Fisher. On the mathematical foundations of theoretical statistics. Philosophical transactions of the Royal Society of London. Series A, containing papers of mathematical or physical character,
2226594-6044:309--688 1922.
Ronald A Fisher. The logic of inductive inference. Journal of the royal statistical society, 98(1):
-9-82,10 1935.
Muhammad Walced Gondal, Manuel Wuthrich, Djordje Miladinovie, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation the real world: new disentanglement dataset. Advances in Neural Information Processing Systems, 32, 2019.
Marton Havasi, Sonali Parbhoo, and Finale Doshi-Veles Addressing leakage in concept bottleneck models. Advances Neural Information Processing Systems, 25:23386-23397, 2022.
Di Jin, Elena Sergeeva, Wei-Hung Weng, Geeticka Chauhan, and Peter Szolovits. Explainable deep learning in healthcare: A methodological survey from an attribution view. WIREs Mechanisms of Disease, ( 2022.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International conference on machine learning. pp. 2649-2658. PMLR, 2018.
Jinkyu Kim and John Canny. Interpretable learning for self-driving cars by visualizing causal attention. In Proceedings of the IEEE international conference computer vision, pp. 2942-2950, 2017.
Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
Pang Wei Koh, Thao Nguyen. Yew Siang Tang, Stephen Mussmann, Emma Pierson. Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, pp. 5338-5344 PMLR, 2020.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference machine learning, pp. pp. 3519-3529. PMLR, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Shun Liu, Kexin Wu, Chufeng Jiang. Bin Huang, and Danqing Ma. Financial time-series forecasting: Towards synergizing performance and interpretability within hybrid machine learning approach. arXiv preprini arXiv:2401.00534. 2023.
Anita Mahinpei, Justin Clark, Isaac Lage, Finale Vesli-Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. arXiv preprint arXiv:2106. 13314, 2021.
Mikael Makonnen, Moritz Vandenhirtz, Sonia Laguna, and Julia E Vogt. Measuring leakage in concept- bancept-based methods: An information theoretic approach. arXiv preprint arXiv:2504 09459, 2025.
Andrei Margeloiu, Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian Weller. Do concept bottleneck models learn as intended? arXiv preprint arXiv:2105.04289 2021.
Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. arXiv preprint arXiv:2304.06/29) 2023.
Enrico Parisini, Tapabrata Chakraborti, Chris Harbron, Ben D MacArthur, and Christopher RS Banerji. Leakage and interpretability in concept-based models. arXiv preprint 1TXiv:2504.14094 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. PmLR, 2021.
12

---

Published as a conference paper at ICLR 2026
Ryan Rifkin and Aldebaro Klautau, In defense of one-vs-all classification. Journal of machine learning research, 1(1am):11-114,,200 2004.
Sungbin Shin, Yohan Jo, Sungsoo Ahn, and Namhoon Lee. A closer look at the intervention procedure of concept bottleneck models. In International Conference on Machine Learning, pp. 31504-31520. PMLR, 2023.
Ravid Shwartz Ziv and Yann LeCun. To compress or not to compress- seltrsss-seee learning and information theory: A review. Entropy, 26(3):252, 2024.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pp. 1-5. Icce, 2015.
Naftali Tishby. Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physicx/0004057, 2000.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv:1907.13625, 2019.
Moritz Vandenhirtz, Sonia Laguna, Ričards Marcinkevičs, and Julia Vogt. Stochastic concept bottleneck models. Advances in Neural Information Processing Systems, 37:51787-51810, 2024.
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning-the good, the bad and the ugly. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4582-4590 2017.
Yao Xie, Melody Chen, David Kao, Ge Gao, and Xiang' Anthony' Chen. Chexplain: enabling physi- cians explore and understand data-driven, ai-enabled medical imaging analysis. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1-13, 2020.
Xinyue Xu, Yi Qin, Lu Mi, Hao Wang, and Xiaomeng Li. Energy-based concept bottleneck models: Unifying prediction, concept intervention, and probabilistic interpretations. arXiv preprint arXiv:2401. 14142, 2024a.
Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. JEEE Robotics and Automation Letters, 2024b.
Mateo Espinosa Zarlenga, Pietro Barbiero, Zohreh Shams, Dmitry Kazhdan, Umang Bhatt, Adrian Weller, and Mateja Jamnik. Towards robust metrics for concept representation evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11791-11799, 2023.
13

---

Published as a conference paper at ICLR 2026
A COMPLETE GRAPHICAL MODELS
C2
Zm
(a) Data Generative Process
(b) Vanilla Model
1
Q
81
1
82
9
62
2m
Cm
2
(c) Concept Bottleneck Models
(d) Minimal Concept Bottleneck Models (ours)
Figure 8: Graphical models of the different systems described. We consider m concepts and m- defined by some concepts {cy) and nuisances n:; and targets y are defined by (gray arrows). Vanilla models obtain the representations {=/)/= from through the encoder Pe (2(zz (cyan arrows) and solve the task ý sequentially through the task head qo (g) 2z) (green arrows). Concept Bottleneck Models make prediction èG of each concept oj from one representation z, through the concept head q(,,lz,) (blue arrows). Minimal CBMs make prediction 2, of each representation 2j from one concept Cj through the representation head q(2)
dimensional representations, Inputs
(red arrows).
14

---

Published as a conference paper at ICLR 2026
B DETAILS OF DERIVATIONS
B.1 PROOF OF EQUATION
I(Z:Y) /ʃ p(z, y) log p(y|z) p(y)
dy d2
//] p(r, y)po (2|2) log
plyl2)
dr dy dz
(8)
ply)
p(r, (=) log p(ylz) 9 (0]22)
dr dy dz
p(v) 90
Ep(x)) [po(=|)) [log q (0]=)] + po) [D (0(0l=)llo0(01331 H(Y)
(10)
2 Ep(2,)) [ppo(s1)) [log d [+]= + HY)
(11)
Thus, since H(Y is independent and Φ, we have that:
max ((2:)) = max
maxEppaa pp(a.)) (ppo(=la) [log go x)]]
(12)
0,0
B.2
PROOF OF EQUATION 2
I(z;;C)) /ʃ p(2), c)) log p((z)zz) p(c,)
de, dz,
(13)
[.] p(a, p(z,c))po(z)laz) log p(czlzz) dr de, dzz
(14)
p(e,)
p(2sc)))o(sglz)) log p(cylzz) g(dg)1g) dr de, dz,
(15)
plc,) ((Cg11z)
2 Ep(a,e)) (po(z))=) [log (0, [21)]] EPo(2)) [De (t
+ HCC))
(16)
2 Ep(n.eg) Epo(2s1 [log ((g1e,)]]] + HC,))
(17)
Given the fact that H (C₂) is constant, we have that:
max
1Z33 Cj) max Ep(a,es) [pp((z)))) [log (((G)lz))]
(18)
2j
B.3 PROOF OF EQUATION
I(2;;X(C;) /Iʃ p(z, ())p((z))x) log p(zz(z)
dx de, dzj
(19)
p(2zlc,)
p(a, ((),c))oo(ss(tt)) log p(=)(=) ((2g((3) dr de, dzj
(20)
p(2ylo)) {(2glc;)
Bp(r,c)) [De (pe(2) Ep(e)) WDx (#(2)1e)ll((3)ee))
(21)
< Bp(r.))) DDL (2((2
(22)
Thus, we have that:
min ((2z;X(C,) min Ep(r,es) DKL (pe (po()))e)ll(()(ee))
(23)
2
15

---

Published as a conference paper at ICLR 2026
B.4 KL DIVERGENCE BETWEEN TWO GAUSSIAN DISTRIBUTIONS
We are given the conditional distributions:
p=(2)1x)= N(((()),,
( = -V(ol(c)),
and aim to minimize the expected KL divergence:
min By(,,,,) [PLL (po(z))o) go (3)1o))))
0.6
The KL divergence between two multivariate Gaussians with diagonal covariances given by: DRI (Nup Ep) || (ug E,)) 2
(r(En' Ex) + (Pg pn) ETp (mg - 1p)) d + log det 2ig det Ex,
Applying this to our case:
Up [o(x) My - (c))
Σp = 13 - = I
d is the dimension of Zj-
Plugging in, we obtain:
do DRL. 2
- sf()))2 -
+ dlog
Note that all terms except the squared distance are constant with respect to 0 and ø. Therefore:
1
Bo(a,,,) [DEL (po(2yl<) || (o(3)100))
= 20 E P, [1]fo(z) (1 + const.
Conclusion: Minimizing the expected KL divergence
min 0.6
E p(z,e)) [DKL (po(z)læ) (o(3glo)))
equivalentt (up to scaling factor) minimizing the expected mean squared error:
min E p(a,es)) [Ilf(()); 3.(c))11
0.0
16

---

Published as a conference paper at ICLR 2026
C TRAINING ALGORITHM OF MCBMS
Algorithm Training Algorithm for MCBMs
Input: Dataset D = (x(*), (h) (( N
k=11 latent norm X, learning rate batch size B Output: Parameters e (encoder), (class-head, mesk-heads, representation-heaee
1: Initialize parameters 0, & and representations heads:
2: for all 1....,, do
3: if' cj is binary then
4: 9 < if oj = else
5: else if oj is multiclass then
6:
9j + X one_hot (cy)
7: else
8:
gj + X-
9:
end if
10:
end for
while not converged do
Sample mini-batch (x((), ((k,, c
jk=1 D
for all (*)), (8), (/ in batch do
14:
Encode: Compute 148 t- So (z!(e) Sample noise ~ (0,
(k)
15:
Reparameterization trick with only one sample
16:
Reparameterize: z (k) Mg
(k)
+ ox Oe
17:
Task prediction: (k)) gh (z (1)
Similar to VMs
Task loss: ((k) y (k) [k) 1²
y is continuous else CE (k) g())
19:
for all 1,...,,.. n do
20:
Concept j prediction: 2) + I (z)
Similar to CBMs
Concept loss: (. 100 //01/
if ej continuous else CE (c/)) (k)
2(A) t 9j
22:
Representation j prediction:
D Novelty in MCBMs
Representation j loss: (A) x (K)
2
23:
24:
end for
Total loss: C(M) + c/ Σッ= (1h y Σラ= (())
end for
27: Update 0, ó using gradient descent:
-
|
1
0+0-
ot v
B
28: end while
17

---

Published as a conference paper at ICLR 2026
D HARD CONCEPT BOTTLENECK MODELS
As discussed in Section 3 one of the most successful approaches mitigating information leakage the family of Hard Concept Bottleneck Models (HCBMs) (Havasi et al. 2022). Unlike MCBMs, which enforce Information Bottleneck (IB) directly the representation level HCBMs operate by pruning the information contained in z before producing the task prediction ý. In doing so, they encourage depend solely on c, thereby yielding interventions that are more reliable in practice. This approach differs fundamentally from MCBMs: whereas Equation 4 in MCBMs minimizes over Zy. thereby removing nuisances directly from the representation 2j, HCBMs optimize the prediction level, focusing on Y. Formally, they are trained to minimize:
min I(;; xIC) = min EE p(z,,;)) [DEL (po. ( (()))))((())) 8.6
(24)
That the optimization is performed directly over the predictions ÿ. To implement this, defines:
(ū /I] d0 (gle") p (c)))) (élx) po (z/x) del dé dz
(25)
where 4o (y (²) denotes the new lask head, q (é) z) the concept head, and po (z x:) the encoder,
as defined in Section 2 The distribution
p (ctle)) = d (c) e (é, - 0.5))
(26)
referred as the binarizing head, applies the Heaviside step function θ produce binary version
èj E {0, 1 of čy. In this way, HCBMs enforce an ad-hoc Information Bottleneck by predicting ý from binarized concept representations. This process schematized Figure
How are Interventions Performed in HCBMs? Because HCBMs introduce bottleneck immedi- ately before predicting ý. the intervention process is more straightforward than in standard CBMs. More specifically, interventions are carried out according to:
p pli|c = x) = /f 90 dj. oi )p(efle, = ()) (x) dēž dē
(27)
Here, p(clle, = a) is approximated as 8(c) a), while the other distributions are available in closed form. Although this procedure provides stronger guarantees than the intervention mechanisms in standard CBMs, two main issues remain:
(i) The optimization is performed over the predictions ý instead of the representation z. As result, the representations themselves are not necessarily interpretable, as evidenced in Figure which limits one of the core motivations for adopting CBMs in the first place.
(ii) HCBMs still require binarizing multiclass concepts, which introduces theoretical limitations and practical drawbacks, as further discussed in Section 5
n
é
ai
è
Figure 9: Graphical models of HCBMs with two concepts and two-dimensional representations. Hard Concept Bottleneck Models obtain binarized version ět of each predicted concept è, through the binarizing head p(ch 2,) (fuchsia arrows). Unlike the models in Figure 2 HCBMs predict the task output from the binarized concepts è using the new task head qo(g) 2") (green arrows).
18

---

Published as a conference paper at ICLR 2026
E OTHER INTERVENTION PROCEDURES
further examine how performance evolves intervene increasing number of randomly selected concepts, with the resulting curves shown in Figure I0 The overall behavior is consistent with the trends observed in Figure highlighting similar model characteristics. As expected, the results exhibit somewhat higher variance across seeds, since each run intervenes on different subset of concepts.
As detailed in Section 4.3
CBM
ARCBM
SCBM
MCBM (low
MCBM (med. y) MCBM (high y)
40
27
22
9
50
75
100
25
50
75
100
25
50
75
100
(a) CIFAR-10 10
(b) CUB
(c) AwA2
Figure 10: Error (y-axis) versus percentage of concepts intervened (x-axis) across different models for randomly selected concepts.
F EXPERIMENTS DETAILS
F.1 HYPERPARAMETERS FOR SECTION
Table 8: Hyperparameters for Section For all datasets, the concept head J is implemented the identity function in CBMs, and multilayer perceptron (MLP) with three hidden layers in MCBMs. We emphasize that CBMs are, by design, restricted to use invertible Ie to enable intervention procedures.
MPI3D
Shapes3D
CIFAR-10
CUB
AwA2
fo architecture
ResNet20
ResNet20
conv. layers
Inception
ResNet-50
fe pretraining
None
None
None
ImageNet
ImageNet
9% hidden layers
64
64
64
256
256
9 hidden layers
None
None
None
None
None
9 hidden layers)
low y
I
0.1
0.05
0.05
medium Y
3
0.3
0.1
0.1
high* y
0.5
0.3
0.3
number of epochs
50
200
250
120
batch size
128
128
128
128
128
optimizer
SGD
SGD
Adam
SGD
SGD
learning rate
x 10
6 x 10
x 10
2 x 10
2 X 10-2
momentum weight decay
0.9
0.9
0.
0.9
0.9
x 10
4 x 10-5
10
x 10-5 Step
4 x 10-5
scheduler
Step
Step
Step
Step
step size (epochs)
20
20
80
100
50
scheduler
0.1
0.1
0.1
0.1
0.1
Beyond the choice of y, MCBMs introduce an additional design decision: the architecture of (c;)). ). In all experiments, we implement this module as small MLP with single hidden layer of size 3, adding only 8 parameters per concept. Since concept sets typically contain at most 200 concepts, this corresponds roughly 1600 additional parameters- neglligibe compared to the size of standard neural encoders fo.
19

---

Published as a conference paper at ICLR 2026
F.2 DATASETS
MPI3D This is synthetic dataset with controlled variation actoss seven generative factors: object shape, object color, object size, camera height, background color, horizontal axis, and vertical axis. In our setup, y corresponds to the object shape, ny to the horizonial axis, ny to the vertical axis, and e to the remaining generative factors. To ensure consistency in the mapping between concepts and task nuisances and the target, we filter the dataset such that any combination of elements in {c, nyy corresponds to a unique value of All invalid combinations are removed accordingly.
Shapes3D
This synthetic dataset consists of 3D-rendered objects placed in room, with variation across six known generative factors: floor color, wall color, object color, scale, shape, and orientation. In our setup. y corresponds to the shape, ny includes floor color and wall color, ny corresponds to orientation, and e comprises the remaining factors. We follow the same filtering strategy as in MPI3D to construct this configuration: we retain only those samples for which each combination of {c, n uniquely determines y, removing all invalid configurations.
CIFAR-10 CIFAR-10 10 is a widely used image classification benchmark consisting of 60,000 natural images of size 32 x 32, divided into 10 classes (c.g., airplanes, automobiles, birds, cats, etc.). The dataset is split into 50,000 training and 10,000 test images, with balanced class distributions. To reduce the need for manual concept annotations, the concepts the methodology of (Vandenhirtz et al. (024).. A total of 143 attributes are extracted using GPT-3 (Brown al. 2020): 64 form the concept set c, while the rest define the nuisance set nyy Binary values are obtained with the CLIP model (Radford et al.) 2021) by comparing the similarity of each image the embedding of an attribute and to its negative counterpart.
synthetically derived following
CUB The Caltech- UCCDD Birds (CUB) dataset contains 11,788 images of 200 bird species, anno- tated with part locations, bounding boxes, and 312 binary attributes. Following the approach of Koh jet al. (2020), we retain only the attributes that are present at least 10 species (based on majority voting), resulting in a filtered set of 1 112 attributes. These attributes are grouped into 27 semantic clusters, where each group is defined by a common prefix in the attribute names. In our setup, the task variable y is to classify the bird species. The concept set consists of the attributes belonging to 12 randomly selected groups (per run), while the nuisance set ny includes the attributes from the remaining 15 groups. Since most attributes exhibit some correlation with the classification task, set ng to the empty set.
AwA2 The Animals with Attributes A2) dataset (Xian et al. 2017) contains 37,322 images of 50 animal classes annotated with 85 human-defined attributes describing appearance, behavior, and habitat. In our setup, the task variable y is the animal class. To construct the concept and nuisance partitions, we retain a subset of 20 attributes covering fundamental appearance and morphology features as the concept set while the remaining 65 attributes form the nuisance set Ny. Since nearly all attributes exhibit some degree of correlation with the class label, we set ny to the empty set. Following the preprocessing in Xian et al. 2017), we binarize continuous attributes using a threshold of 0.5.
20


Reviews:
[{"review_id": "ocHOfPlMXB", "reviewer": "Reviewer_dC8m", "summary": "- This paper introduces Minimal Concept Bottleneck Models (MCBMs), which address the key limitation of Concept Bottleneck Models (CBMs); Information leakage. Although CBMs ensure that each latent variable $z_j​$ can predict its corresponding concept $c_j$​, they allow $z_j$​ to retain nuisance information from the input $x$.\n- MCBMs formalize this issue as $I(Z_j;X|C_j) > 0$ and introduce an explicit Information Bottleneck (IB) objective to enforce minimal sufficient representations. By adding a variational regularization term, MCBMs constrain each $z_j$​ to encode only concept-relevant information.\n- Experiments show that MCBMs yield more disentangled and interpretable representations, reduce nuisance leakage, and enable consistent, concept-level interventions while maintaining comparable concept accuracy to standard CBMs.", "questions": "- (Related to weakness 1) The model's performance varies extremely with $\\gamma$, from 100% to 24.9%. Please provide a concrete procedure for setting $\\gamma$ on a real dataset where ground-truth labels for $n_y$ are unavailable. \n- (Related to weakness 2) We observed a poor accuracy/nuisance-removal trade-off on CIFAR-10 and CUB. Could you provide insight into why this occurs? \n- (Related to weakness 2) To assess the method's generality, have the authors also conducted experiments on other standard concept benchmarks, such as AWA2 or OAI?\n- Unlike [1] and [2], where interventions on the CUB dataset reduced task error ([1], Fig. 4; [2], Fig. 5), this paper shows the opposite trends. Could the authors explain what experimental differences might cause this discrepancy?\n\n\n\nReference\n- [1] Koh et al., Concept Bottleneck Models, ICML 2020.\n- [2] Shin et al., A Closer Look at the Intervention Procedure of Concept Bottleneck Models, ICML 2023.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Clear formulation of problems: While the issue of information leakage in CBMs is known, this paper clearly formalizes it from an information-theoretic perspective $(I(Z_j;X|C_j) > 0)$ and identifies the root cause as the lack of a minimal sufficient statistic.\n- Principled approach: The application of the classic Information Bottleneck (IB) framework to solve this problem is logical and well-founded.\n- Proposal of an information leakage metric: The paper introduces URR, a new metric to provide a quantitative way to approximate the information leakage $I(Z_j;X|C_j)$.", "weaknesses": "- Impractical Hyperparameter ($\\gamma$): The model's behavior is dictated by $\\gamma$, which requires ground-truth $n_y$ labels for tuning. The authors provide no practical guidelines for setting $\\gamma$ on real-world datasets where such labels are unavailable.\n- Performance degradation on real datasets: The strong bottleneck effect from synthetic data does not translate to CIFAR-10 and CUB. On these real-world datasets, MCBM achieves only a marginal reduction in nuisance leakage (URR). This minimal gain results in a disproportionately large drop in task accuracy, raising questions about the method's practical utility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LJUtK2wvy2", "reviewer": "Reviewer_xCkz", "summary": "This paper revisits the theoretical foundations of Concept Bottleneck Models (CBMs) and argues that there is, in fact, no true “bottleneck” in existing CBMs.\nThe authors show that each latent variable $z_j$ may encode nuisance information unrelated to its corresponding concept $c_j$, thus undermining interpretability and invalidating standard intervention assumptions.\nTo address this, the paper proposes the Minimal Concept Bottleneck Model (MCBM), which introduces a per-concept Information Bottleneck (IB) constraint to enforce that each $z_j$ becomes a minimal sufficient statistic of $c_j$.\nExperiments on several datasets (CIFAR-10, CUB, Shapes3D, MPI3D) demonstrate improved disentanglement and more stable concept-level interventions compared to prior CBM variants.", "questions": "1. In Lines 40–41, the authors state that “each concept $c_j$ must be recoverable from a designated component $z_j \\in z$.”  \n   In the original CBM paper, it is only assumed that an intermediate layer is resized to the number of concepts and trained with a concept loss, but not that each $z_j$ uniquely represents $c_j$.  \n   Could the authors please **cite the exact section or equation** in Koh et al. (2020) that formalizes this one-to-one correspondence?\n\n2. The paper assumes the factorization  \n   $\n   p(x, y, c, n) = p(x \\mid c,n)\\,p(y \\mid x)\n   $\n   Could the authors clarify the **causal assumptions** under which this holds?  \n   Specifically:  \n   (i) Do you assume $y \\perp (c,n)\\mid x$?  \n   (ii) If there exist direct dependencies of $y$ on $c$ or $n$ not mediated by $x$, how does this decomposition remain valid?  \n   A detailed justification or citation would strengthen the theoretical basis.\n\n3. In Appendix E.2, after aggregating CUB attributes into 27 semantic groups, why are **12 groups randomly selected for $c$** and the remaining **20 groups** used for $n_y$?  \n   How was this number (12 vs 20) chosen, and is it consistent across runs or random seeds?\n\n4. This partition is not semantically meaningful but rather a controlled perturbation design.  \n   Because many task-relevant attributes are discarded into $n_y$, even a fully faithful model cannot recover $y$.  \n   Under such an incomplete concept set, can MCBM still be regarded as a valid model that achieves both interpretability and predictive ability?\n\n5. Energy-based Concept Bottleneck Models also jointly model $p(y|x)$ and $p(y|c,x)$.  \n   Why were such approaches not included as baselines, given that they share similar modeling goals and provide a natural comparison?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Clear theoretical motivation:\nThe paper provides a precise and mathematically grounded critique of CBMs, identifying the lack of causal interpretability in the conventional bottleneck assumption.\n2. Principled formulation:\nThe proposed information bottleneck constraint is elegant and conceptually well-justified; it directly connects to the goal of removing nuisance information while retaining task-relevant concepts.\n3. Strong theoretical discussion:\nThe analysis of $p(z_j|c_j)$ and the derivation of the KL regularization term are rigorous and highlight a meaningful direction for improving causal interpretability in concept-based models.", "weaknesses": "1. The paper does not clearly cite any existing literature to support the statement that “each concept $c_j$ must be recoverable from a designated component $z_j \\in z$.”\nThe original CBM (Koh et al., 2020) only enforces a per-concept prediction loss, not a one-to-one structural correspondence between $c_j$ and $z_j$.\n2. From a performance standpoint, MCBM is a “more interpretable but weaker” model.\nThe paper does not explicitly position its goal as interpretability improvement rather than predictive performance.\nIf the objective is purely interpretability, it remains unclear how encoding only a subset of concepts leads to better explanations, especially since MCBM explicitly removes part of the concept information.\n3. The paper does not discuss how much task accuracy loss is acceptable or provide any principle for balancing the IB weight $\\gamma$.\nThis trade-off is central to the model’s practical usability but left unanalyzed.\n4. The random split of the CUB attribute groups into c vs $n_y$ introduces noise and may compromise the validity of the disentanglement and CKA metrics.\nBecause semantically correlated features can be split across c and $n_y$, the reported “high disentanglement” might reflect random partitioning rather than true semantic separation.\n5. Intervention evaluation (Section 4.3) lacks robustness:\nno variance or confidence intervals are reported, and the curves for different $\\gamma$ values are nearly parallel, making the claim of “intervention gain invariance” unconvincing.\nMoreover, there is no comparison with baseline settings such as random or group-wise interventions.\nThe CUB intervention results in Figure 5 also differ substantially from those reported in CEM (Fig. 6 of their paper), raising reproducibility concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OrYKOl9rll", "reviewer": "Reviewer_5MFs", "summary": "This work presents Minimal Concept Bottleneck Models (MCBMs), an advancement upon the traditional Concept Bottleneck Model (CBM) framework. The proposed method integrates an Information Bottleneck (IB) objective to mitigate \"concept leakage,\" thereby constraining the intermediate representations to encode only concept-specific information. This approach yields a significant advantage by establishing a sound theoretical basis that provides formal guarantees for the efficacy of concept-level interventions.", "questions": "1) With how many concepts MCBM is supposed to properly work?\n2) Is the training timing significantly increased while using MCBMs or it requires more epochs to converge?\n3) concept leakage is generally associated with an increasing performances (as the nuisances can help the development of reasoning shortcuts to solve the problem). How do you motivate MCBMs has very close or even higher task accuracy on CUB than other methods suffering from leakage?\n4) Apart from concept leakage, MCBMs have also consequences on other aspects, e.g. in terms of disentanglement and OIS between concepts? See e.g. this recent paper [3] for a discussion on different metrics to evaluate concept quality.\n\n\n[3] Debole, Nicola, et al. \"If Concept Bottlenecks are the Question, are Foundation Models the Answer?.\" arXiv preprint arXiv:2504.19774 (2025).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- the paper is sound and propose an interesting analysis of CBMs.\n- the problem of concept leakage in CBMs is very relevant and the results on experiments are encouraging.", "weaknesses": "- \"Formally, given a task (...), Vanilla Models (VMs) are trained (...)\" here for how it is presented it seems the VMs work from c to y, while in Sec. 2 it is clear it is intended from x to y. Thus I'd rephrase like: \"their representations z should capture the information necessary to predict y given x accurately.\" and also rephrase at the beginning of the sentence: \"given an input x, a task y and a set of concepts c\". As a fussiness, I'd change \"Formally\" with \"Specifically\" as task, concepts are not precise terms, and to make this sentence formal I'd expect to know what are the space y, c, and x belong. \n- at the beginning of Sec. 2, y is not in bold, but immediately after it becomes bold.\n- In Sec. 2 some symbols are not defined or even just dropped without being commented, e.g. theta, phi, f_theta ...\n- I find a bit odd to have Section 5 discussing problem of intervention in CBMs that is instead addressed by MCBM. Indeed, I think it can strengthen the motivation to propose MCBM. But I guess this is done to don't interrupt the flow of the model.\n- no limitations of MCBMs are discussed.\n- no discussion about future work is reported.\n- not clear what bold denotes in table 5 and 6.\n- Other metrics to measure the concept leakage has been used in prior work (e.g. [1,2]). I think it would be useful to take into account also other metrics and not just rely on URR, as this is the main advantage MCBMs propose. \n\n\n[1] Marton Havasi et al. Addressing leakage in concept bottleneck models. In NeurIPS, 2022.\n[2] Anita Mahinpei et al. Promises and pitfalls of black-box concept learning models. In Workshop on Theoretic\nFoundation, Criticism, and Application Trend of Explainable AI @ ICML, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EsbNTCIwRd", "reviewer": "Reviewer_6hx5", "summary": "This paper addresses the limitations of standard Concept Bottleneck Models (CBMs) in ensuring true concept-level interpretability. The authors argue that while CBMs predict human-defined concepts, their internal representations may still encode extraneous information, undermining interpretability and intervention validity. To address this, they propose Minimal Concept Bottleneck Models (MCBMs), which introduce an Information Bottleneck (IB) objective to ensure each representation component retains only concept-relevant information.", "questions": "See Weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-organized, and the arguments regarding CBM limitations are presented clearly, with intuitive figures that effectively contrast CBMs and MCBMs.\n- The paper provides formal justification for the proposed Information Bottleneck-based extension, connecting interpretability, information leakage, and probabilistic soundness.\n- Experiments across multiple datasets and model variants (e.g., CBM, CEM, AR-CBM, SCBM) offer a broad empirical perspective, strengthening the claim that MCBMs improve both alignment and disentanglement.", "weaknesses": "- While the paper’s theoretical insights are valuable, the overall novelty appears modest. The Information Bottleneck principle is a well-established framework, and similar approaches applying it to CBMs seem to have been explored in [1] and [2]. Given this overlap, it would be helpful if the authors could better clarify the significance of their contribution—specifically, what distinguishes their approach from prior works.\n- In Table 5, the accuracy reported for SCBM in [3] is 86.00, which appears higher than the 84.8/84.9 achieved by MCBM. Could the authors clarify what accounts for this difference?\n- Introducing additional layers based on the Information Bottleneck framework would likely increase inference cost and memory usage. Could the authors quantify how much overhead this introduces?\n\n**References** \\\n[1] Parisini, E., Chakraborti, T., Harbron, C., MacArthur, B. D., & Banerji, C. R. (2025). Leakage and interpretability in concept-based models. arXiv preprint arXiv:2504.14094. \\\n[2] Makonnen, M., Vandenhirtz, M., Laguna, S., & Vogt, J. E. (2025). Measuring leakage in concept-based methods: An information theoretic approach. arXiv preprint arXiv:2504.09459. \\\n[3] Vandenhirtz, M., Laguna, S., Marcinkevičs, R., & Vogt, J. (2024). Stochastic concept bottleneck models. Advances in Neural Information Processing Systems, 37, 51787-51810.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)