A Probabilistic Hard Concept Bottleneck for Steerable Generative Models
Authors: María Martínez-García
Ricardo Vazquez Alvarez
Alejandro Lancho
Pablo M. Olmos
Isabel Valera
Venue: ICLR 2026 Poster (2026)
Keywords: generative models
interpretability
steerability
concept bottleneck
hard concepts
probabilistic models
Primary area: interpretability and explainable AI
TL;DR: We introduce the Variational Hard Concept Bottleneck (VHCB) for CBGMs, improving steerability by mitigating concept leakage and enabling generation from specific concept configurations. We also propose a systematic evaluation framework for CBGMs.

Abstract: Concept Bottleneck Generative Models (CBGMs) incorporate a human-interpretable concept bottleneck layer, which makes them interpretable and steerable. However, designing such a layer for generative models poses the same challenges as for concept bottleneck models in a supervised context, if not greater ones.  Deterministic mappings from the model inner representations to soft concepts in existing CBGMs: (i) limit steerable generation to modifying concepts in existing inputs; and, more  importantly, (ii) are susceptible to *concept leakage*,  which hinders  their steerability. To address these limitations, we first introduce the Variational Hard Concept Bottleneck (VHCB) layer. The VHCB maps probabilistic estimates of binary latent variables to hard concepts, which have been shown to mitigate  leakage. Remarkably, its probabilistic  formulation enables direct generation from a specified set of concepts.  Second, we propose a systematic evaluation framework for assessing the steerability of CBGMs  across various  tasks (e.g., activating and deactivating concepts). Our framework which allows us to empirically demonstrate that the VHCB layer consistently improves steerability.

Published as a conference paper at ICLR 2026
A PROBABILISTIC HARD CONCEPT BOTTLENECK FOR STEERABLE GENERATIVE MODELS
María Martinez-Garcia"
Ricardo Vazquez Alvarez
Dept. of Computer Science
Dept. of Signal Theory and Communications Universidad Carlos III de Madrid
Saarland University
Alejandro Lancho
Pablo M. Olmos
Dept. of Signal Theory and Communications Universidad Carlos III de Madrid
Dept. of Signal Theory and Communications
Universidad Carlos III de Madrid
Isabel Valera
Dept. of Computer Science
Saarland University
ABSTRACT
Concept Bottleneck Generative Models (CBGMs) incorporate a human- interpretable concept bottleneck layer, which makes them interpretable and steer- able. However, designing such a layer for generative models poses the same chal- lenges for concept bottleneck models in a supervised context, if not greater ones. Deterministic mappings from the model inner representations to soft con- cepts in existing CBGMs: (i) limit steerable generation modifying concepts in existing inputs; and, more importantly, (ii) are susceptible to concept leakage, which hinders their steerability. To address these limitations, we first introduce the Variational Hard Concept Bottleneck (VHCB) layer. The VHCB maps probabilis- tie estimates of binary latent variables to hard concepts, which have been shown mitigate leakage. Remarkably, its probabilistic formulation enables direct genera- tion from a specified set of concepts. Second, we propose a systematic evaluation framework for assessing the steerability of CBGMs across various tasks (e.g., ac- tivating and deactivating concepts), which allows us to empirically demonstrate that the VHCB layer consistently improves steerability.
INTRODUCTION
In recent years, a new line of research has emerged to improve the interpretability of Generative Models (GMs) by leveraging Concept Bottleneck Models (CBMs) (Koh et al., 2020). A CBM is an inherently interpretable model consisting of concept predictor, which maps inputs to human- understandable concepts, and label predictor, which maps these concepts to task outputs. CBMs offer two key advantages: (i) predictions are explicitly grounded in concepts, and (ii) users can in- tervene concepts steer the output. However, CBMs face key challenges that limit steerabil- ity: concept incompleteness, when the concept set fails to capture all task-relevant information, and concept leakage, when soft concept probabilities unintentionally encode task information. Incom- pleteness can be addressed with side channel that encodes task information not explained by the concepts (Havasi al., 2022; Espinosa Zarlenga al., 2022: Sawada Nakamura, 2022; Yuksek- gonul et al., 2023), while leakage can be reduced by using hard concept representations (Margeloiu al., 2021; Havasi et al., 2022: Lockhart al., 2022: Vandenhirtz. et al., 2024; Sun et al., 2024).
Concept Bottleneck Generative Models (CBGMs) extend CBMs from classification to generative tasks (Ismail al., 2024; Kulkarni al., 2025). this framework, Concept Bottleneck (CB) layer is introduced at an intermediate location of the generative model, mapping the inner representation of the generator to a set of human-understanddale concepts. These methods aim to make generative
"mart nez-garci alcs. uni saarl land. de

---

Published as a conference paper at ICLR 2026
models inherently interpretable and, most importantly, steerable. By manipulating concepts in the latent space it is possible to control generation, either by (i) generating new data according to specific concept configurations (steerable generation), (ii) intervening concepts in existing observations (steerable conditional generation). Current CBGM methods rely on soft concepts and deterministic CB layers, remaining vulnerable concept leakage and restricting steerability to concept interven- tions, as they do not model generative process over the concept space.
Contributions. We address these limitations by introducing the Variational Hard Concept Bot- tleneck (VHCB) layer, defining the first probabilistic CBGM with hard concepts. The VHCB is based on a binary Variational Autoencoder (VAE), producing probabilistic estimates of binary latent variables that are directly mapped to hard concepts (Section 3). Moreover, this probabilistic formu- lation enables direct generation from specified concept configurations while still supporting concept inference and steerability through concept interventions. In addition, we introduce a systematic evaluation framework for CBGMs across various inference and steerable generation tasks (Section 4). This comprehensive evaluation allows us to analyze accuracy in concept prediction, alignment between intended and actual outcomes of concept manipulation, distributional shifts on non-target concepts during intervention, and the impact of correlations and biases present in the training data in the behavior of the model (Section 5).
CBGM FRAMEWORK
In CBGMs, a CB module is inserted at an intermediate location of generative model to map inner representations to a set of human- citerriittrpretablle concepts (Ismail et al., 2024; Kulkarni et al., 2025). In general, CBGMs consist of three components, as shown in Figure 1. The pre- pre-bottleneck module (u -> w) maps (noisy) latent input u (usually Gaussian distributed) to pre-bottleneck embedding w. The concept bottleneck module (w - (c, s) + ii)) maps w to set of predefined concepts e and an unsupervised embedding s. she embedding s acts as a side channel to capture information not represented by the concepts, addressing the inherent incompleteness of the concept set in generative tasks, since generation depends on factors that cannot be fully described by human- cumaanniterpreetbble concepts (e.g., textures, lighting, or object position). The CB then maps (c, s) to post-concept embedding w. Finally, the post-bottleneck module (á + x) generates the final output æ. In this setting, one can adopt either an in-hoc approach, training the base generative model (pre- and post- bottleneck modules) together with the CB layer, or a post-hoc approach, where the CB layer is inserted into a pretrained generative model.
Concept label sources. We consider a set of K human-interpretabb binary concepts y E 1) indicating the presence of specific features in observations æ. For example, in the image shown in Figure 1, 'smiling' and "black hair' would be active, while 'has mustache' and 'wearing sun- glasses' would be inactive. Concept labels can be obtained from three sources: (i) annotated datasets (æ;, y, M (ii) pretrained supervised classifiers that predict from æ, or (iii) pretrained zero-shot classifiers that infer concepts without task-specific training (Kulkarni et al., 2025).
Desiderata. The goal is to design generative models that produce high-quality outputs while re- maining both interpretable and steerable. Generation quality requires the model to produce diverse data that accurately reflects the training distribution. Interpretability requires accurate inference of human-understandable concepts. Steerability requires precise control over the generative process: in steerable generation, the model should produce samples reflecting specific concept configurations; in conditional steerable generation, it should be possible to modify target concepts in existing ob- servations while keeping remaining concepts unchanged. Additionally, disentanglement between the unsupervised representation 8 and the concepts e is necessary to ensure that modifying the side channel does not unintentionally alter concept information in the generated outputs.
2.1 STATE OF THE ART
Two different approaches have been proposed for implementing the CB module in CBGMs. Is- mail et al. (2024) follow a in-hoc approach. They extended the Concept Embedding Model (CEM) (Espinosa Zarlenga et al., 2022) layer by mapping the mrebbotleneck embedding into unsu- pervised embedding and a set of two embeddings per concept, representing the concept's 's active and inactive states. These embeddings then combined according the likelihood of each concept
2

---

Published as a conference paper at ICLR 2026
Variational Hard Concept Bottleneck
concepts
ec
Ze
Concept Bottleneck Generative Model
2y(c/ur w)
Pre Bottleneck Module
Concept Bottleneck
Post Botzleneck Module
Encoder Iv (w)
Decoder fo(z)
76
BCC
modell
nokay ingott
pre-bottlenokk embedding
yoat-bottlenook embedding
Qu(a m)
outyut
Zc
representation
(b)
Figure/ 1: Block diagram of (a) the general architecture of CBGMs, and (b) the VHCB layer. Note that the Corre-Ccrg Code (ECC) in the VHCB layer is a deterministic transformation that enables effeettve inference.
Cs being active produce the post-concept embedding it The CEM layer and the base generative model are trained jointly.
In contrast, Kulkarni et al. (2025) proposed the Concept Bottleneck Autoencoder (CB-AE), CB module that can be introduced into pretrained generative models. The encoder maps w into a latent space that includes both concept predictions, with each binary concept represented by two logits, and an unsupervised embedding. The decoder produces the post-concept embedding w which must accurately reconstruct w preserve the pretrained model's generative performance. Training uses synthetic images generated by the pretrained model, with concept labels obtained from pretrained or zero-shot classifiers such as Contrastive Language-Ina Image Pre-training (CLIP) (Radford et al., 2021). The CB-AE is optimized using
L = f (w, w)) + Lr, (xx, 2)) + L. c) + Li, (Vtarget Fims) Lig (9 target Cint);
(1)
where Cr₁ and Lr, are Mean Squared Error (MSE) reconstruction losses on the embeddings (w, w)) and images (z, @). respectively. Ce is cross-entropy loss aligning predicted concepts e with labels y. Intervention losses L;, and L;, enforce that concept interventions produce the desired changes: Li, compares target concepts Vrarget. with those in the intervened image Viot while Li, aligns the concepts Cint. predicted from the intervened embedding Win with Itarget- Kulkarni et al. (2025) also propose lightweight version of the CB-AE, referred to as Concept Controller (CC). this model, only the encoder is trained using reconstruction loss, and interventions are performed via an optimization based procedure that applies small perturbations to w to induce target concept. However, there is guarantee that these interventions actually utilize the inferred latent concepts, since concept inference and intervention are independent processes. For this reason, we do not consider the CC in analysis.
Limitations. Both methods rely on soft concept representations, which are susceptible to concept leakage, where concept probabilities unintentionally encode task-related information (Havasi al., 2022: Margeloiu et al., 2021). This reduces control over generation, one of the primary objectives of CBGMs, models may exploit information shorteuts rather than effectively capture concept in- formation, causing concept interventions to fail to produce the intended effect in the final output. In the case of the CB-AE, this issue is partially mitigated through training-tim interventions (see (1)), but at the expense of increased training complexity. Moreover, since both methods are de- terministic, they cannot model a generative process over the concept space. As a result, steerable generation is restricted to modifying concepts existing inputs, while direct sampling from the concept bottleneck or from concept distributions is not supported.
VARIATIONAL HARD CONCEPT BOTTLENECK LAYER
To address the limitations existing CBGMs, we propose the Variational Hard Concept Bottleneck (VHCB) layer. The VHCB builds on binary VAE, in which the binary latent variables that govern the generative process are directly mapped to hard concepts, mitigating concept leakage (Margeloiu et al., 2021; Havasi et al., 2022; Lockhart et al., 2022; Vandenhirtz al., 2024: Sun et al., 2024). As a result, training can rely on a straightforward reconstruction- based objective without requiring additional intervention terms. Its probabilistic formulation further enables direct generation from
3

---

Published as a conference paper at ICLR 2026
specified concept configurations, while still supporting concept inference and controlled manipula- tion of generated images through concept interventions. Moreover, the binary latent space yields compact and interpretable concept representations that are easy to manipulate.
We build the VHCB by extending the Coded Discrete Variational Autoencoder (Coded DVAE) from Bartfnez-Garcif et al. (2025), ttate-of-the-art (SOTA) binary VAE, incorporate two binary la- tent representations: a vector of concepts c, whose components are aligned with predefined hard concepts; and an unsupervised embedding which as side channel to address concept incom- pleteness (Ismail et al., 2024; Kulkarni et al., 2025). For each observation æ, concept supervision introduced during training using conditional concept distributions p(y x) as informative priors for These conditionals p(y(æ) are obtained from either a supervised or zero-shot pretrained classifier, as described in Section 2. We consider the Coded DVAE as base model as it offers (i) binary latent space with a factorized prior, suitable for modeling binary concepts, (ii) probabilistic estimates of binary latent variables that can be directly mapped hard concepts, and (iii) effective inference mechanisms that allow for robust modeling of those concepts. Remarkably, despite the discrete latent space, training remains stable through a reparameterization trick based on overlap- ping smoothing transformations.
3.1 MODEL DESCRIPTION
Consider a dataset D = (w, L., ys) where w; is the inner rep- resentation of a generative model producing image æ; and y; are the associated K binary concept labels. The VHCB maps w to two binary latent representations: a concept vector c {0,1)K, whose components are aligned with K predefined hard concepts, and an un- supervised representation & E {0, 1] which serves as a side channel to capture generative factors not represented by accounting for con- cept incompleteness (Ismail et al., 2024; Kulkarni et al., 2025). We model and a binary latent variables with independent Bernoulli components. By defining the side channel as a binary factorized la- tent space, the model can learn potentially interpretable unsupervised representations, enabling the potential discovery of new concepts.
a2
Prior work showed that protecting binary latent variables during gen- eration by introducing structured redundancy with Error-Correcti Codes (ECCs) improves inference in binary VAEs (Martfnez-Garcca et al., 2025). An ECC defines a deterministic transformation that in- creases the dimensionality of the original vector by adding redundancy. In our setting, we protect and using two separate ECCs, defining the transformations -> Ve, with E 10, and K' > K; and U.. with v, E 1) and L' > L. These transformations in- crease the Hamming distance between the binary vectors allowing error correction during inference by selecting the nearest valid codeword in the K' space for and the L'-bit space for s. Follow- ing Martínez-García et al. (2025), we employ uniform repetition codes, where each bit is repeated multiple times to construct the codewords.
Figure 2: Graphical model of the VHCB.
Following the Coded DVAE framework, after encoding, a smoothing transformation p(zlv) = 11, p(zy((( (2)) is applied both u, and v. enable differentiable sampling. Specifically, this trans- formation overlapping exponential distributions:
((=-1)
e 9=
plz|u- 1)
p(z|u 0)
Zz
Z,3
where E 1), E [0. 10. Ze (1-e B)/B and 3 acts an inverse temperature parameter. The transformation is applied to both v. and v., yielding z. and ze, which are then concatenated to form the final embedding that is fed into the decoder Neural Network (NN). The likelihood of the data is conditioned on this smooth auxiliary variable z as (o(w|z) = p(fo(z)), where fa(.)
denotes the decoder NN.
Variational Family. In this setting, the posterior over the hard concepts factorizes as (, 2,(w) = Vn (n/c/w (((c/))p(z(ee (x), while the posterior over the unsupervised representation factorizes analogously
4

---

Published as a conference paper at ICLR 2026
as 9ry (s, _5|s) (n(s(w)p(z2l1) In both cases, assume independent Bernoulli posteriors, consistent with the Coded DVAE formulation. The marginal posteriors qz. j = 1...., K of the bits in and obtained in two stages: first, the encoder NN produces the marginals of the coded bits, an) (w/w) i Ber 93.m (w)) using the encoder NN (n((). Next, these marginals are refined by leveraging the structure of the repetition code through soft majority voting procedure, yielding improved marginals for the bits in c after correcting errors made by the encoder NN. More details on this process can be found in Martinez-Garcia
al. (2025).
Training, The VHCB can be trained either in-hoc, jointly with the base generative model, or post- hoc, using pretrained model. We focus on the post-hoc setting, as it is more practical: it is more computationally efficient, reduces data requirements, and enables steerable generation with models that already produce high-quality outputs. Following Kulkarni al. (2025), we generate samples a; from the pretrained model during training and store their intermediate representations w,. Each generated image æ; is then passed through concept classifiers to obtain p(y, ja:;), pairing every w, with the concepts of its generated image. The model is trained by optimizing the following loss
C = [ log po (wlz) DSKL (9n(clw), p(y)x)) - DDLL (gn (s) w (lp((s) + MSE(x, )))
embeckting recoostraction
concept loss
side channel regularization
image rec.
(3)
where x = c zg] is the concatenation of the soft latent variables passed to the decoder NN. The first term enforces an accurate reconstruction of the pre-bottleneck embedding w. The concept loss aligns the concept posterior 9n (c/w)) with [x) using the symmetric Kullback-Leiler (KL) diver- gence (Jesfreys divergence), defined as DskL (p, q) A DKL (pilg) + DKL (4llp) (Polyanskiy & Wu, 2025). It preserves distributional modes and prevents the posterior from collapsing to the mean, thereby improving control in concept interventions. We empirically validate this choice through an ablation study available in Appendix D. The side channel regularization term constrains the unsu- pervised latent s to remain close its prior. Finally, following Kulkarni et al. (2025), MSE loss between the original images æ and their reconstructions & is included. This term is added in the post-hoc setting to preserve the generative quality of the pretrained model, while in in-hoc training it would be replaced by the loss of the underlying generative model (Ismail et al., 2024).
Steerable generation. With the VHCB, we can directly sample e and from its latent space to gen- erate an embedding v conditioned on a specified concept distribution. This differs from prior SOTA methods (Ismail et al., 2024; Kulkarni et al., 2025), which only enable steerability through interven- tions on existing model outputs æ. Operating directly on hard concepts mitigates leakage and makes interventions transparent and intuitive: controlling concept reduces to toggling its corresponding bit (1) or off (0), allowing straightforward manipulation of the generative process. Note that at test time the binary latents and can be sampled from a uniform prior or fixed to a given choice, and the smoothing transformation in (2) is applied to obtain z. Interventions on generated outputs x are carried out by setting the bit in c corresponding to the target concept to the desired value.
SYSTEMATIC EVALUATION FOR CBGMs
We aim to design CBGMs that are interpretable, steerable, and capable of producing high-quality outputs. Building on the desiderata outlined in Section 2, we introduce systematic evaluation framework for CBGMs. We define set of tasks and metrics that allow to assess performance and analyze important aspects such as concept entanglement, cascading effects of interventions, and the impact of correlations or biases present in the training data inherited from the pretrained generator.
4.1 METRICS
We evaluate model performance using set of similarity and divergence metrics, selected based on whether concepts are represented as hard variables e E {0, 1] or soft class probabilities p(c/w)) Note that a concept model can be based on hard concepts (those used to generate the datum æ:) and still provide soft class probabilities q(c/w). This is the case of the VHCB.
Metries for hard concepts. For hard binary predictions, we evaluate performance using accuracy, defined as aco(y. c) Σx= [gy ez], where K is the vector dimensionality, yj is the ground-
K
5

---

Published as a conference paper at ICLR 2026
truth label, Cj is the predicted concept value, and [[:] is the indicator function. This metric quantifies the fraction of entries where the predicted vector matches the ground-truth vector.
Metrics for soft concepts. Let p be soft concept vector, where each entry Py represents the ground-truth class probability p(y = 1|x), and let q be the corresponding vector of predicted concept probabilities, q(c, = 1/w)). In this case, we consider the following metries:
Cosine Similarity. Quantifies the similarity of two vectors by calculating the cosine of the angle between them, which is given by sim(p, q) - Tpplz Ilqllaa
p.g
Total Variation Distance (TV). Treating soft binary predictions Bernoulli distributions allows us to compute statistical distances such as the TVV, which quantifies the total change in probability mass between two distributions (Polyanskiy & Wu, 2025). In this case, it is computed per concept and averaged over all K concepts as TV K Σ PPj
qzl-
Metries for image generation. To assess the quality of generated images, we use the standard Fréchet Inception Distance (FID), which quantifies the similarity between the distributions of gener- ated and real images in a learned feature space (Heusel et al., 2017). The FII) captures both fidelity and diversity of the generated images.
4.2 TASKS
As discussed in Section 2, our goal is design CBGMs that provide accurate concept inference for interpretabilityy, enable steerable generation through concepts, and preserve high generative quality. In this section, we present set of tasks systematically evaluate these properties.
Robust concept inference. A primary aspect to evaluate is the model's ability to accurately map pre-bottleneck embeddings w concepts e. Additionally, we need to assess the disentanglement between and the unsupervised representation s, which acts a side channel to address concept incompleteness, ensuring that concept information is captured solely by c.
Concept prediction. We assess the model's ability to accurately predict concepts by comparing the concept labels y ~ p(y]az) of generated images æ with the latent concepts c predicted by the CB layer from their corresponding pre-bottleneck embeddings w. In the post-hoc setting, ground-truth labels are obtained from images generated by the base generative model (without the CB layer), and the associated pre-bottleneck embeddings w are mapped concepts c. This ensures that the evaluation reflects how well the model predicts the concepts that are inherently represented in by the base generator.
Disentanglement. To assess whether the unsupervised representation encodes concept infor- mation, we generate two independent samples from the CBGM, æ and æ2, with corresponding concept labels y1 y/- latent concept vectors C1, C22 and unsupervised latents 81, 82. We then swap the unsupervised latents to form (C1, 82) and 81), and produce new outputs a' and æ/y with concept labels y' and y/2. If s and e are disentangled, concept labels should remain unchanged. We quantify this by computing accuracy between y, and y! and, when soft concept labels are available, by evaluating soft similarity metrics between p(yl (z;) and p(y)
Steerable generation. central feature of CBGMs the ability to steer generation through con- cepts. To evaluate this, we define tasks that include generation from specified concept configurations and targeted concept interventions in existing generated data. Although CBGMs assume concepts to be independent, they often exhibit statistical relationships (spurious not) that generative models may capture or even amplify. As a result, random sampling or arbitrary interventions can yield out- of-distribution concept configurations that the model cannot process correctly, limiting steerability. For example, an intervention that activates two mutually exclusive concepts simultaneously. such 'blond hair' and 'black hair', will fail to generate both. Our evaluation framework accounts for this, allowing us to distinguish errors caused by limitations of the CB layer from those arising from out-of-distribution concept configurations.
Direct concept-based generation. In probabilistic CBGMs, it is possible sample (c, s) from the CB's latent space and generate images conditioned on a specified concept configuration c, which can also be out-of-distribution The sampled pair is mapped post-concept embedding

---

Published as a conference paper at ICLR 2026
W. which is then decoded to produce image ~ p(z ( are then estimated with a classifier as y ~ p(y|z). Finally, we measure the accuracy between y and to evaluate concept aligmment.
c). Associated concept labels
Single concept intervention. In this case, we evaluate the ability to manipulate individual con- cepts in existing data. First, a datum zo is generated with the CBGM, with associated con- cept vector Co. and concept labels yo p(yo]zo) The target concept in co is then in- tervened (activated or deactivated) to produce Ctarget- A new output Dint is generated as Zint pn (n ins 80, Co Crarget), with corresponding labels Vint p(y Fint): Comparing Vint with Crarget and p(yul20)) with piy (inne) allows us to assess (i) whether the intervention accurately reflected in Zine (Target accuracy) and (ii) whether non-target concepts remain unchanged (Non Target accuracy). To ensure comprehensive evaluation, interventions are tested in both directions.
Minimum Hamming distance intervention. The evaluation is the same in the single concept setting, but in this case we target concept patterns that are frequent in the dataset to avoid out-of- distribution configurations. For each sample zo with latent concept vector Co. we select Ctargtt as the closest training concept configuration in Hamming distance, defined as the number of positions at which two vectors of the same length differ. This approach modifies the fewest possible concepts while staying in-distribution, helping to distinguish errors due to CB layer limitations from those caused by out-of-distribution configurations.
5 EXPERIMENTS
In this section, we present experimental results for the proposed VHCB layer and the baseline CB-AE, evaluated the tasks defined above. As indicated in Section 3, we focus on the post-hoc setting. Experiments are conducted with StyleGAN2 (Karras et al., 2020), pretrained on CelebA-HQ (256x256) (Lee al., 2020) and CUB-200-200 (256x256) (Wah et al., 2011), and Denoising Diffu- sion Probabilistic Model (DDPM) (Ho et al., 2020) pretrained on CelebA-HO (256x256). Following Kulkarni et al. (2025), for CelebA-HQ we evaluate the model in (i) large imbalanced regime with all 40 concepts (all set), and (ii) a small balanced regime with the 8 most balanced concepts (bal- anced set). addition, we introduce (iii) generator can reliably activate (low correlation set), allowing to isolate the effect of concept cor- relations. For CUB-200-2011, we adopt the same setup as Kulkarni et al. (2025), using 10 balanced concepts. Results are reported using two pseudo-label sources: supervised ResNet18 classifiers and CLIP zero-shot classifiers. Main results are presented for StyleGAN2 trained on CelebA-HQ and CUB-200-2011, 1, with additional experiments and extended DDPM results reported in the Appendix.
correlattoo subset of balanced concepts that the
Model configuration. For both the CB-AE and VHCB, we use 4-layer Multilayer Perceptrons (MLPs) for the encoder and decoder. In the CB-AE, following Kulkarni et al. (2025), the unsuper- vised latent is a continuous vector 8 E IR 40 In contrast, the VHCB employs a low-dimensional binary latent E {0, 1]. Additional implementation details are provided in the Appendix.
Automated evaluation. We use independent ResNet50 concept classifiers, not involved in training, obtain concept labels y p(y)(z) for the generated images. These concept labels act as ground truth for evaluating model performance.
Evaluating single concept interventions. For each target concept, we identify Ik pre-bottleneck latents w that generate images æ with P(Mharget 1|x) < 0.5 (concept inactive) and 1K latents with p(ytageet 1|x> > 0.5 (concept active). These are then used to do single-concept interventions in both directions, with the same latent set applied across all models to ensure fair comparison.
5.1 CONCEPT INFERENCE
As shown in Table 1, the proposed VHCB consistently outperforms the deterministic CB-AE across all concept inference metrics, under both ResNet18 and CLIP supervision. Although both models show reduced performance with CLIP due to the noisier zero-shot labels, VHCB remains superior, demonstrating more robust mapping from w c. The gains are even larger for disentanglementt VHCB achieves higher accuracy and cosine similarity while maintaining lower TV, indicating that
7

---

Published as a conference paper at ICLR 2026
Table I: Concept inference and disentanglement between c and 8.
Evaluation on Ik random samples generated by StyleGAN2 pretrained on CelebA-HO.
Concept Inference
Disentanglement
Concept Set
Pseudo Label Clf.
Model
Acc. (f)
Cosine Sim. (1)
TV(L) Acc.(t) Cosine Sim. (t) TV())
VHCB
all 0.855
0.804
0.148
0.927
0.917
0.076
CB-AE
all 0.857
0.763
0.161
0.901
0.853
0.101
ResNet18
VHCB low corr. 0.791
0.831
0.192
0.874
0.947
0.110
CB-AE low
0.779
0.739
0.238
0.701
0.803
0.229
VHCB
all 0.623
0.613
0.337
0.921
0.901
0.083
CB-AE all 0.565
0,469
0,403
0.875
0.775
0.101
CLIP
VHČB low corr.
0.599
0.730
0.299
00.54
0.943
0.125
CB-AE low corr.
0.607
0.639
0.346
0.667
0.782
0.248
$
2
$
2
S
Ci
C2
Figure 3: Qualitative evaluation of the disentanglement between s and c with the VHCB layer and StyleGAN2 models pretrained CelebA-HQ.
changes in have little effect on generated concepts. The qualitative results shown in Figure further indicate that modifying does not induce semantic changes in the generated images. This stronger separation results from the compact side channel (5 bits compared to 40 dimensions in CB-AE) of and robust inference mechanisms introduced in VHCB, enforcing cleaner division between supervised and unsupervised factors.
5.2 STEERABLE GENERATION
Direct generation. The probabilistic formulation of the VHCB enables direct generation from spec- ified concept configurations. Table 2 reports metrics showing that generated images actually reflect concepts in C. Since random sampling of concept configurations can produce nut-of-distribution concept configurations, we also sample concept patterns according to their empirical frequency in training data. This further improves accuracy and highlights how biases the base generative model can limit the steering capacity of the CBGM. Figure 4 shows examples of generated images with specific concept configurations for both CelebA-HQ and CUB-200-2011.
Table Steerable Generation. Evaluation of gen- eration from specific concept configurations using a StyleGAN2 pretrained on CelebA-HQ. Random samples concept sets uniformly at random, while Patterns samples them according to their empirical frequency in the training data.
CokbA-HQ
Bange 1, M.MottOQpen 1. Young O. Suillng Mustache 0)
CUB-- CUUB-200-0011
Blackppeepete WhtteTnsirparrs BaakkWigg
BlackBil
Random
Patterns
Pseudo Concept Set
Target Acc. (t)
Target Acc. (t)
Model
Label Clf.
all low corr.
0.551
0.873
ResNer18
0.715
0.814
VHCB
Figure 4: Steerable generated examples
all 0.533
0.830
CLIP
ing the VHCB and a StyleGAN2 pretrained CelebA-HQ and CUB-200-2011.
low corr.
0.632
0.687
8

---

Published as a conference paper at ICLR 2026
Table 3: Test-time interventions. Evaluation of single-concept activation (i + a), deactivation (a + i), and interventions guided by training concept patterns (minimum Hamming distance) using a StyleGAN2 pretrained on CelebA-HQ.
Single (i + a)
Single (a -> i)
Hamming Dist.
Pseudo Label Clf.
Concept Set
Target Acc. (1)
Non Target
Target
Non Target
Target
Non Target
Model
Acc. (1)
Acc. (1)
Acc. (t)
Acc. (t)
Acc. (t)
VHCB
all 0.170
0.903
0.550
0.902
0.660
0,938
CB-AE
all
0.105
0.924
0.453
0.924
0.542
0.955
ResNet18
VHCB
low corr.
0.769
0.762
0.765
0.781
0,634
0,845
CB-AE
low corr.
0.420
0.825
0.554
0.822
0.418
0.880
VHCB
all 0.131
0.890
0.536
0.885
0.595
0.923
CB-AB
all 0.085
0.922
0.311
0.922
0.466
0,949
CLIP
VHCB
low cor..
0.567
0.691
0.532
0.694
0.534
0.729
CB-AE
low
0.317
0.829
0.404
0.834
0.496
0.869
Table 4: DDPM results, obtained with a DDPM pretrained on CelebA-HQ.
Concept Inf.
Single (i + a)
Single (a - i)
Pseudo
Target
Kon-Target
Target
Non- Mon-Target Acc (t)
Model
Concept Set
Acc (1)
TV (4)
Label Clf.
Acc (f)
Acc (t)
Acc (t)
all
0.773
0.254
0,131
0.905 0.910
0.546 0.571
0.898
ResNet18
VHCB
low corr.
0,621
0.383
0.187
0.857
Ce Mouth Open
CelebA-HQ Bangs
All
Low
Low
(a 8)
(
i)
(1 a)
( - a)
[a (e++<
CUB-200-2011 Black Upperparts
CUB-200-2011 White Underparts
CUB-200-2011 Black Wing
(a + 1)
(i a)
(a + 8)
(i - a)
(a - i)
(i
Figure 5: Single-concept interventions with the VHCB layer and StyleGAN2 models pretrained on CelebA-HQ and CUB-200-2011.
Single concept interventions. We next evaluate the models' ability to manipulate outputs via single-concept interventions. with results reported in Figure 5 and Table 3. The VHCB consis- tently outperforms the CB-AE in target accuracy, learning a robust mapping from latent concepts to post-bottleneck embeddings without explicit intervention losses, indicating that hard concept representations enhance steerability. We observe that the CB-AE shows slightly higher non-target accuracy across settings, but this does not indicate more robust interventions. Instead, it reflects a weaker ability to modify the target concept: in many cases, the target is unchanged (lower target ac- curacy), leaving the output the same and artificially inflating non non-target accuracy, which is computed over all Ik interventions. In contrast, the VHCB achieves better compromise, with large gains in target accuracy (average increase ~466) and only a minor reduction in non-target accuracy (aver- age drop ~7%). For example, the low-correlation with ResNet1 8, target accuracy rises from

---

Published as a conference paper at ICLR 2026
0.42 to 0.77, while non-target accuracy decreases slightly from 0.83 to 0.76. We provide qualitative examples of this behavior in the Appendix, for both VHCB and CB-AE.
We also find that stecrability is strongly influenced by the concept distribution in the dataset. In CelebA-HQ. most concepts are underrepresented, making deactivation easier than activation, as re- flected by higher success rates for deactivating concepts. The effects of imbalance and model biases are evident when comparing the full set of 40 unbalanced concepts to balanced or low-correlation subsets, where performance metrics consistently improve for both models (results for the balanced set are provided in the Appendix). Thus, the composition of the concept set has a significant impact on CBGM performance. We observe the same effect in the DDPM results in Table 4. Additional results illustrating the effects of concept incompleteness and biases are provided in the Appendix.
Hamming distance interventions. Arbitrary single-coneept interventions can produce out-of- distribution concept configurations that the model cannot generate correctly, resulting in failed inter- ventions. To mitigate this, we restrict interventions to target one of the 100 most probable concept patterns observed in the dataset, selected via minimum Hamming distance. This approach consis- tently improves intervention success rates in both models, with the VHCB still yielding superior metrics, as shown in Table 3. These results further support that steerability depends not only on the expressiveness of the CB layer but also on biases inherent in the pretrained generative model.
CONCLUSION
In this work, we introduced the VHCB layer, the first probabilistic CBGM with hard concepts. The VHCB enables improved steerability by (i) mitigating concept leakage by considering hard concepts, and (ii) enabling direct generation from specified concept configurations thanks to its probabilistic formulation, while still supporting concept inference and steerability through concept interventions. We also proposed a systematic evaluation framework for CBGMs, through which showed that the VHCB consistently enhances steerability. This framework further revealed the impact of correlations and biases training data, which are not captured under our assumption of independent concepts. Future work includes modeling concept relationships in the latent space and fine-tuning base models to mitigate spurious biases exposed through steerability analyses.
7 ACKNOWLEDGMENTS
We would like to thank Sofia Molotkova for her support in early stages of the project and every member of the Probabilistic Machine Learning group (UdS) for their invaluable feedback, well as the anonymous reviewers whose feedback helped us improve this manuscript. This work has been funded by the European Union (ERC-2021-STG.. SAML., 101040177, PI Valera) and (MSCA-2024- DNS, MLCARE, 101226456, PI Olmos). However, the views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible.
Alejandro Lancho and Ricardo Vazquez have received funding from the Comunidad de Madrid's 2023 Cesar Nombela program under Grant Agreement No. 2023-11/COM-29066 ally. Alejandro Lancho has received funding from the Ministerio de Ciencia, Innovación Uni- versidades, Spain, under Grant Agreement No. PID2023-1488560A 1488560A-1060 (funded by MICIÚ/AFI/ and by ERDF/UE).
Addition-
10.13099/5011000
Pablo M. Olmos has also been supported by the Comunidad de Madrid IND2024/TIC-3472.. IDEA- CM project (TEC-2024/COM-89) the ELLIS Unit Madrid (European Laboratory for Learning and Intelligent Systems), the 2024 Leonardo Grant for Scientific Research and Cultural Creation from the BBVA Foundation, and from the Ministerio de Ciencia, Innovación Universidades. Spain, un- der Grant Agreements No. PID2024- 157856NB-I00 PARTESIAN,PID2002 1022022-1231820BB... EPi- CENTER (funded by MICIU/AEI/ 10. 10039/00 10001 1033 and by ERDF/UE).
10

---

Published as a conference paper at ICLR 2026
8 REPRODUCIBILITY STATEMENT
We have made a strong effort to ensure the reproducibility of our results by providing detailed descriptions the implementation, training procedures, and evaluation framework. In particular, we explain how the method integrates with different generative architectures, including StyleGAN2 and DDPM, specify the pretrained models used in our experiments, detail the training procedures for each architecture, and describe the concept sets employed for the various model configurations (with a brief overview in the main text and more extensive discussion in the Appendix). Additionally, we provide comprchensive description of the evaluation framework, including implementation details and dataset-specific considcrations.
REFERENCES
Tameem Adel, Zoubin Ghahramani, and Adrian Weller. Discovering Interpretable Representations for Both Deep Generative and Discriminative Models. In International Conference on Machine Learning, pp. 50-59, PMLR, 2018.
Guillaume Alain and Yoshua Bengio. Understanding Intermediate Layers using Linear Classifier Probes. International Conference on Learning Representations, 2017.
Yonatan Belinkov. Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics, 48(1):207-219, 2022.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation Learning: Review and New Perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798- 1828, 2013.
Marc-André Carbonneau, Julian Zaidi, Jonathan Boilard, and Ghyslain Gagnon. Measuring Disen- tanglement: A Review of Metrics. IEEE Transactions Neural Networks and Learning Systems, 35(7):8747-8) 8761, 2022.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. Advances in Neural Information Processing Systems, 29, 2016.
Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772-782, 2020.
Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Gian- nini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, et al. Concept Embedding Models: Beyond the Accuracy-Explainabllty Trade-Off. Advances in Neural Information Processing Systems, 35:21400-21444 2022.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. GANspace: Discovering
Interpretable GAN Controls. Advances in Neural Information Processing Systems, 33:9841- 9850, 2020.
Marton Havasi, Sonali Parbhoo, and Finale Veshi-Velez. Addressing Leakage in Concept Bottleneck Models. Advances in Neural Information Processing Systems, 35:23386-23397, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. Ad- vances in Neural Information Processing Systems, 30, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE. Learning Basic Visual Concepts with Constrained Variational Framework. In International Conference on Learning Representations, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models, Advances in Neural Information Processing Systems, 33:6840-683 2020.

---

Published as a conference paper at ICLR 2026
Aya Abdelsalam Ismail, Julius Adebayo, Hector Corrada Bravo, Stephen Ra, and Kyunghyun Cho. Concept Bottleneck Generative Models. In International Conference on Learning Representa- tions, 2024.
Insu Jeon, Wonkwang Lee. Myeongjang Pyeon, and Gunhee Kim. IB-GAN: Disentangled Represen- tation Learning with Information Bottleneck Generative Adversarial Networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7926-7934, 2021.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and Improving the Image Quality of StyleGAN. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 81 10-18117 1198, 2020.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept Bottleneck Models, In International Conference on Machine Learning, pp- 5338-5348. PMLR, 2020.
Akshay Kulkarni, Ge Yan, Chung-En Sun, Tuomas Oikarinen, and Tsui-Wei Weng. Interpretable Generative Models through Post-hoc Concept Bottlenecks. In IEEE/CVF Conference on Com- puler Vision and Pattern Recognition, 2025.
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. MaskGAN: Towards Diverse and Inter- active Facial Image Manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5549-5558, 2020.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging Common Assumptions in the Unsupervised Learn- ing of Disentangled Representations. In International Conference on Machine Learning, pp. 4114-4124. PMLR, 2019.
Joshua Lockhart, Nicolas Marchesotti, Daniele Magazzeni, and Manuela Veloso. Towards learn- ing to explain with concept bottleneck models: mitigating information leakage. arXiv preprint arXiv:2211.03656, 2022.
Emanuele Marconato, Andrea Passerini, and Stefano Teso GlanceNets: Interpretable, Leak-proof Concept-based Models. Advances in Neural Information Processing Systems, 35:21212-21227, 2022.
Andrei Margeloiu, Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian Weller. Do Concept Bottleneck Models Learn as Intended? arXiv preprint arXiv:2105.04289, 2021.
María Martinez-Gareia. Grace Villacrés, David Mitchell, and Pablo M. Olmos. Improved Variational Inference in discrete VAEs using Error Correcting Codes. In Proceedings of the Forty-first Con- ference on Uncertainty in Artificial Intelligence, volume 286 of Proceedings of Machine Learning Research, pp. 1973-3012 PMLR, 21-25 Jul 2025.
Brooks Paige, Jan-Willem Van De Meent, Alban Desmaison, Noah Goodman, Pushmeet Kohli, Frank Wood, Philip Torr, et al. Learning Disentangled Representations with Semi-Supervised Deep Generative Models. Advances in Neural Information Processing Systems, 30, 2017.
Yury Polyanskiy and Yihong Wu. Information Theory: From Coding to Learning. Cambridge University Press, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models from Natural Language Supervision. In International Conference Machine Learning. pp. 8748-8763. PMLR, 2021.
Yoshihide Sawada and Keigo Nakamura. Concept Bottleneck Model With Additional Unsupervised Concepts. IEEE Access, 0144758-417666 2022.
Ivaxi Sheth and Samira Ebrahimi Kahou. Auxiliary Losses for Learning Generalizable Concept- based Models. Advances in Neural Information Processing Systems, 36:26966 26990, 2023.
12

---

Published as a conference paper at ICLR 2026
Ao Sun, Yuanyuan Yuan, Pingchuan Ma, and Shuai Wang. Eliminating Information Leakage in Hard Concept Bottleneck Models with Supervised, Hierarchical Concept Learning. arXiv preprint arXiv:2402.05595, 2024.
Moritz Vandenhirtz, Sonia Laguna, Ričards Mareinkevičs, and Julia Vogt. Stochastic Concept Bot- tleneck Models. Advances in Neural Information Processing Systems, 37:51777-51810, 2024.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-200 2011-001, California Institute of Technology. 2011.
Tao Yang. Yuwang Wang, Yan Lu, and Nanning Zheng. Disdiff: Unsupervised Disentanglement of Diffusion Probabilistic Models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 19130-69156, 2023.
Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc Concept Bottleneck Models. In Inter- national Conference on Learning Representations, 2023.
13

---

Published as a conference paper at ICLR 2026
A EXTENDED BACKGROUND
Interprefability in Generative Models. A central research direction focuses on learning disentan- gled representations, where independent factors of variation are captured along latent dimensions to enable controlled generation (Bengio et al., 2013; Chen et al., 2016; Higgins et al., 2017: Paige et al., 2017; Jeon et al., 2021; Yang et al., 2023). Although these factors can sometimes be interpretable, disentanglement does not guarantee alignment with human-interpretabee 2019). Also post-hoc approaches have been proposed, such as Adel et al. (2018), where an exist- ing latent space is transformed using invertible transformations to encode interpretable concepts. However, interventions in this setting are not direct as more than one latent dimension can map to the same concept information. Another method aims to identify interpretable control directions in Generative Adversarial Networks (GANs) by applying Principal Component Analysis (PCA) to in- ternal representations of the model (Härkönen al., 2020). A different line of work, mainly applied in Natural Language Processing (NLP), trains classifiers to predict concepts from model representa- tions (Alain & Bengio, 2017: Belinkov, 2022). While this reveals that internal states contain concept information, it does not imply that the model relies on such concepts for its predictions.
concepts (Locatello et al.,
Concept Bottleneck Models (CBMs). A CBM (Koh et al., 2020) an inherently interpretable model consisting of two modules: concept predictor, which maps inputs to concepts, and a label predictor, which maps these concepts to task outputs. These modules can be trained independently, sequentially, or jointly, and concepts may be represented as discrete labels, class probabilities, or embeddings (Koh et al., 2020; Havasi et al., 2022; Espinosa Zarlenga et al., 2022). CBMs offer two key advantages: (i) predictions high-level concepts, and (ii) users can intervene on these concepts to alter the final output. How- ever, they face main challenges: concept incompleteness and concept leakage. Incompleteness arises when the predefined concept set fails to capture all task-relevant information, which can be mitigated through a side channel or by learning representations that encode additional information not explained by the concepts (Havasi et al., 2022; Espinosa Zarlenga et al., 2022; Sawada & Naka- mura, 2022; Yuksekgonul al., 2023). Leakage occurs when soft concept probabilities unintention- ally encode task labels. undermining interpretability and trustworthiness (Margeloiu et al., 2021; Havasi al., 2022). This can be alleviated by using binary concept representations or enforcing uncorrelated concept representations (Chen et al., 2020; Margeloiu et al., 2021; Havasi et al., 2022; Espinosa Zarlenga et al., 2022; Marconato et al., 2022; Sheth Ebrahimi Kahou, 2023; Vanden- hirtz et al., 2024). In our work, adopt hard binary concepts leveraging the Coded DVAE, which prevent leakage while providing a low-dimensional and casily interpretable latent representation.
explicitly grounded in interpretable
A.1 RELATED WORK: INTERPRETABILITY IN GENERATIVE MODELS THROUGH CBMS
Concupl Embodding Model
Context NN
Context
Context NN side chounel
Concept Bottleneck Autoencoeer
9
embeediing..o of
embedding o
embedding of
ombodding op
comorge loyeta
Prob. NN o.
Prob. NN ex
w
Encoder
Decoder u
embedding
embedding
umanperrsiod nege ceprtcentation
(a)
(b)
Figure 6: Diagrams of SOTA CBGM architectures: (a) the Concept Embedding Model (CEM)
(Ismail al., 2024) and the (b) Concept Bottleneck Autoencoder (CB-AE) (Kulkarni et al., 2025).
Concept Embedding Model (CEM). Ismail et al. (2024) were the first to introduce CBMs into generative models, aiming to make them both interpretable and stecrable through a aodel-agnostic concept bottleneck layer. Building the CEM layer (Espinosa Zarlenga et al., 2022), the pre-
14

---

Published as a conference paper at ICLR 2026
bottleneck representation w is mapped into two embeddings per concept, representing its active and inactive states, which are mixed based on the likelihood of the concept yi being active. To capture task-relevant information not covered by predefined concepts, the authors introduce unknown concept network that learns an additional unsupervised embedding. In this framework, the genera- tive model and bottleneck layer must be trained jointly, making the approach reliant annotated data and computationally expensive.
Post-Hoc Concept Bottleneck Autoencoder (CB-AE) and CC. To reduce the dependence la- beled data and the computational cost of CBGMs, Kulkarni et al. (2025) proposed the CB-AE, a concept-basee based autoencoder that can be integrated into pretrained generative models. The encoder maps the pre-bottleneck latent w into a concept space that includes both concept predictions, with each binary concept represented by two logits, and an unsupervised embedding that captures in- formation not covered by the concepts. The decoder reconstructs the latent it, which is passed to the post-bottleneck module to generate the final output æ. In this setup, depicted in Figure only the CB-AE is trained, while the generative model remains fixed. To this end, the CB-AE leverages synthetic images generated by the pretrained model, with pseudo-labels y given by pretrained or zero-shot classifiers such CLIP (Radford al., 2021). The model is trained by minimizing the following objective
(4) where Lr, and fr, are MSE reconstruction losses between generative latents (w, ib) and images (x, i).. L. is cross-entropy loss aligning predicted concepts with pscudo-labels y. The inter- vention losses L., and Lig are cross-entropy losses that enforce that interventions in the concept space yield the desired changes in both the generative latent Witt and the generated image Dimt' Li, compares the target concepts Vtarget with the pseudo-labels of the intervened image Yint while Liz aligns the predictions Cints obtained by mapping Wine to the concept space, with Vrarget In this case, interventions are performed by swapping the concept logits.
L= L (w, i)) + (x, 2) + L.. + Ls, (Mtarget Mint) + Lig (Ftarget Cint),
Alternatively, the authors propose optimization-basel interventions where small perturbations applied to the latent w to induce a target concept, an approach referred to as CC. In this setting, only the encoder trained with reconstruction losses, but there is no guarantee that the model relies on latent concept information to carry out the interventions.
B STYLEGAN2
The main experimental results of this work were obtained by integrating our proposed VHCB layer into pretrained StyleGAN2 models (Karras et al., 2020). This family of generative models is par- ticularly well suited to the CBGM framework, as image generation is controlled by a single latent representation that map concepts and that the generator then transforms into output. In StyleGAN2, we place the VHCB layer between the Mapping Network and the Synthesis Network, as illustrated in Figure 7, following the approach of Ismail al. (2024) and Kulkarni et (2025). These prior works showed that this the optimal location in this setting, since it enables high-quality image generation while allowing the CB layer to capture semantic rather than style information.
StyleG Generator
VHCB layer
output
model
26
Mapping Network
w
Latent to Concepts
Concepts Latent
Synthesis Network
noiay input
Figure 7: Diagram showing the integration of our VHCB layer into the StyleGAN2 architecture.
Datasets and pretrained models. We one trained on CelebA-HQ at 256 x 256 resolution (Lee et al., 2020), and one trained on CUB 200- 2011 at 256 256 resolution (Wah al., 201 2 Following Kulkarni al. (2025), for CelebA-HQ
publicly available pretrained StyleGAN2 models:
15

---

Published as a conference paper at ICLR 2026
train and evaluate the VHCB layer in (i) large imbalanced regime with all 40 concepts (all set), and (ii) a small balanced regime with the 8 most balanced concepts (balanced set). In addition, we introduce (iii) low-correlation subset of 8 balanced concepts that the generator can reliably activate (low correlation set), allowing us to isolate the effect of concept correlations. For CUB, we adopt the same setup Kulkarni al. (2025), using 10 balanced concepts. The specific concept sets as follows:
CelebA-HQ
Complete set (all, 40 concepts): '5 o Clock Shadow", 'Arched Byebrows', 'Attrac- tive', 'Bags Under Eyes', 'Bald', 'Bangs', 'Big Lips', 'Big Nose'. 'Black Hair', 'Blond Hair'. 'Blurry, 'Brown Hair', "Bushy Eycbrows', 'Chubby', 'Double Chin', 'Eyeglasses', 'Goatee', 'Gray Hair', 'Heavy Makeup'. 'High Cheekbones', 'Male', 'Mouth Slightly Open', 'Mustache', 'Narrow Eyes'. 'No Beard', 'Oval Face', 'Pale Skin', 'Pointy Nose', 'Receding Hairline', 'Rosy Cheeks'. 'Siddebuns', 'Smilling, "Straightt Hair', "Wavy Hair', 'Wearing Earrings', 'Wearing Hat', 'Wearing Lipstick', "Wearing Necklace', 'Wearing Necktie', and 'Young'.
Balanced set (8 concepts):
'Attractive', "Wearing Lipstick', 'Mouth Slightly Open', 'Smiling', 'High Cheekbones', 'Heavy Makeup'. 'Male', and 'Arched Eyebrows'.
Low correlation set (8 concepts): 'Pale Skin', 'Bangs', 'Big Lips', 'Mouth Slightly Open', 'Chubby', 'Young', 'Smiling', and 'Mustache'.
CUB-200-2011
Balanced set (10 concepts): 'Small size 5, to 9 inches', 'Perching like shape", 'Solid breast pattern', 'Black bill color', 'Bill length shorter than head", 'Black wing color', 'Solid belly pattern', 'All purpose bill shape', 'Black upperparts color', and 'White underparts color'.
Implementation and training details. For both the CB-AE and VHCB, we use 4-layer MLPs for the encoder and decoder. In the CB-AE, following Kulkarni et al. (2025), the unsupervised latent contrast, the VHCB employs low-dimensional binary latent
continuous vector 8 E 12:00 {0, protected by uniform repetition with code rate of R 5/50. Concept variables are encoded with code rates R E {8/240, 10/300, 40/ 10/800} depending on the number of concepts. Models are trained for 50 epochs using Adam with learning rate of 0.0002, with concept labels provided either by supervised ResNet18 classifiers or CLIP zero-shot classifiers. The VHCB used batch size 32 and 1000 steps per epoch, and weighted the concept loss by 20 in the training objective. For the CB-AE, we halved the batch size to 16 due to memory limits and doubled the steps match the number of training samples. For CelebA-HQ with the balanced concept set, we used the model weights provided by the authors :; all other configurations were trained from scratch. Thresholds for computing the concept losses in the CB-AE were set to 0.9 for supervised classifiers, 0.6 for CelebA-HQ with CLIP, and none for CUB-200-201 1, following the original code specifications.
C DENOISING DIFFUSION PROBABILISTIC MODEL
To assess the generality of the proposed VHCB across generative architectures, we integrate it into DDPM (Ho al., 2020). Following the setup of Kulkarni et (2025), we use a publicly available pretrained DDPM from Google, trained on CelebA-Ho A-H at 256 x 256 resolution, which employs U-Net as the denoising network.* This enables a direct comparison with the baseline method from Kulkarni et al. (2025) under identical conditions.
We have considered two different implementations. Section C. following Kulkarni al. (2025), we attach the VHCB module to the bottleneck layer of the U-Net architecture. In Section C.2, we attach the output of the VHCB module to all levels of the DDPM's U-Net architecture in the upsampling path.
16

---

Published as a conference paper at ICLR 2026
U-Net
3
Dowssampling Block
Upsompling Black
it prodicted time
generoted imagee
at tome I
Doensnmpiing
Upasmpling
time embedding at time É
Downsampling
Upasmpling
Bottlmnck
VHCB layer
Latent to Concepts
Concepta to Latent
1
Figure 8: Integration of the VHCB module into the pretrained DDPM architecture. A noisy image X, at diffusion timestep is processed by the U-Net, which predicts the corresponding noise éG- The VHCB module is inserted at the bottleneck of the U-Net and maps intermediate features to binary concept vector via an encoder, which is then used for concept prediction, steerability, and reconstruction via the decoder. The objective is to ensure that semantic information is captured and can be manipulated through c, without disrupting the denoising trajectory.
C.I VHCB AT BOTTLENECK
Our aim here is to replicate the original post-hoc CBM configuration and verify whether the trends observed with StyleGAN2 also hold in diffusion models. schematic the model with the VHCB module is shown in Figure 8.
Implementation and training details. We follow the same dataset and concept setup as in the StyleGAN2 experiments (see Section B), but restrict analysis to CelebA-HQ. We first generate a dataset of 50,000 images using the pretrained DDPM, storing both the latent noise vectors and the resulting images. Concept pseudo-labels are obtained using a bank of ResNet-18 classifiers (one per concept), and probabilities are thresholded at 0.5 to produce binary targets.
The VHCB trained by replaying the diffusion process: the stored noise latents are fed into the fixed pretrained DDPM, now augmented with our module. We restrict training to timesteps f e [0, 400], since later steps add excessive noise and degrade the semantic structure in the bottleneck representation (Kulkarni et al., 2025). At each step, the U-Net predicts the noise at timestep t, and the VHCB receives the corresponding bottleneck features.
The loss is the same supervised objective as in equation (3), consisting of (i) binary cross-entropy loss for concept prediction, (ii) KL divergence regularizer for the side channel, and (iii) recon- struction term. In the DDPM setting, the reconstruction is implemented as consistency constraint: the mean squared error between the predicted noise of the pretrained U-Net and that of the U-Net with the VHCB attached. This ensures that the VHCB learns anncett-alinne representations with- out interfering with the denoising process.
We use the same VHCB architecture described in Section B. However, here we only consider code rates R E (8/240, 40/20000000000 40/800} depending on the number of concepts. Models are trained for 50 epochs using Adam with learning rate of 0.0002, batch size of 32, with concept labels provided either by supervised ResNet18 classifiers or CLIP zero-shot classifier. A weight of 3 = 20 was again applied to the concept loss term increase its influence during training.
17

---

Published as a conference paper at ICLR 2026
U-Net
Dowssampling Block
Upsampling Bluck
predicted modse time 8
genereted image
Downsumpling Block
Upeampling
Block
Upaazspling
time embedding od time /
2D Coca Layer
Upsacppling
Block
Block
2D Coco Layer
Bottlenekk
Upascspling
VHCB layer
Concepts
Lasent
Figure 9: DDPM architecture with VHCB at all levels, featuring separate decoder-like network that adapts the VHCB bottleneck feature resolution to match each decoder output layer in the U- Net's upsampling path (shown in green at the right side of the figure). The ① symbol indicates element-wise addition of the corresponding outputs.
C.2 DDPM EXTENSION: VHCB AT ALL LEVELS
Following the same procedure as the original DDPM implementation, where the VHCB layer was placed only after the bottleneck, we now attach the output of the VHCB module all levels of the DDPM's U-Net architecture in the upsampling path.
Implementation and training details. This extension introduces dedicated upsampling path (de- coder like architecture) that takes the VHCB's reconstructed bottleneck representation and progres- sively transforms through series of upsampling and 2D convolutional layers. At each stage, the network expands and refines the bottleneck features so they match the required spatial resolution of each decoder output layer in the upsampling path of the U-Net. These adapted bottleneck features are then added element-wiss wise to the outputs of the respective decoder layers. A schematic overview of this extension provided in Figure 9.
The training procedure for this extension follows the same setup the original implementation described in Section C.I. with a few additional considerations. To reduce overall training time, the VHCB layer was initialized using the weights from its previously trained model (VHCB-at- bottleneck). An early stopping strategy was also employed, becoming active only after the first 20 epochs and terminates training if no improvement is observed within patience window of epochs. The maximum number of epochs is set to 50, matching the previous training configuration. The consistency term, previously defined as the mean squared error between the predicted noise of the pretrained U-Net and that of the U-Net with the VHCB attached, is now computed between the predicted noise of the U-Net without the VHCB extension and the predicted noise obtained when the bottleneck bottleneck-adjusted representation is injected into each decoder output in the upsampling path. The same learning rate and batch size were also used.
D ABLATION STUDIES
We have conducted some ablation studies to gain intuition on the behavior of the model under different configurations, and empirically validate some design choices.
Ablation study on the concept loss. To empirically validate the choice of the symmetric KL di- vergence concept loss, have conducted an ablation study comparing thethe performance
18

---

Published as a conference paper at ICLR 2026
Table Ablation on the concept loss. StyleGAN2 CclebA-HQ.
Concept
Pseudo Label
Concept Loss
Target Acc. (1)
Target Acc. (1)
Target Acc. (1) Hamming Distance
FID (L)
Set
Clf.
(i (i- a)
(a
ResNet18
BCE
0.165
0.555
0.676
0.543
ResNet18
KL
0.168
0.524
0.669
0.546
ResNet18
Symm. KL
0.170
0.550
0.660
0.551
all
CLIP
BCE
00.177
0,364
0.621
'0.327
CLIP
KL
0.128
0.508
0.603
0.530
CLIP
Symm. KL
0.131
0.536
0.594
0.533
ResNet18
BCE
0.395
0.564
0.527
0.632
ResNet18
KI.
0.314
0.748
0.726
0.663
ResNet18
Symm. KL
0.375
0.814
0.594
0.670
balanced
CLIP
BCE
0.405
0.654
0.721
0.613
CLIP
KL
0.306
0.745
0.767
0.602
CLIP
Symm. KL
0.346
0.682
0.423
0.607
Table 6: Ablation on the side channel regularization. Concept inference and steerability results on StyleGAN2 CelebA-HQ. using supervised ResNet18 classifiers for concept supervision during training.
Concept Set
Size
Concept Ace. (t)
Target Acc. (f)
Target Acc. (f)
Target Acc. (1)
Weight
(inference)
(i + a)
(a > i)
(Hamming Dist.)
0.853
0.173
0.558
0.666
0.855
0.170
0.550
0.660
all
10
5
0,856
0.173
0.562
0.656
20
0.853
0.173
0.544
0.673
0.850
0.174
0.550
0.659
0.773
0.358
0.784
0.627
0.752
0.375
0,814
0.594
balanced
10
5
0.754
0.362
0,822
0,609
20
0.763
0.364
0.817
0.596
40
0.763
0.337
0.799
0.561
10
0.766
0.363
0.785
0.606
0.770
0.356
0.788
0.514
balanced
0.774
0.351
0.822
0.509
20
10
0.774
0.350
0.818
0.499
40
10
0.774
0.326
0.811
0.500
steerability and generation models trained with Binary Cross Entropy (BCE), forward KL, and symmetric KI. concept loss. These results complement provided intuition in Section 3.1, by illustrating that models trained with BCE tend to perform slightly better when deactivating concepts, models trained with forward KL generally achieve slightly better control when activating concepts, and models trained with symmetric KL provide the best balance, offering strong interventional per- formance in both directions.
Ablation on the side channel regularization. To determine the impact of the side channel on the model's overall behavior, conducted an ablation on the side-channel regularization term in the loss function (Eq. (3)) by varying the unsupervised KL weight for various side-channel dimensions and concept sets. The obtained results show that this parameter does not have any significant effect in any of the reported metrics when the concept set is large (i.e., 40 concepts in CelebA-HQ). When considering small concept sets, both steerability and FID slightly deteriorate as side-channel regularization increases. This can be explained by the fact that the proposed CBGM generative model optimized to minimize reconstruction loss. When only a small subset of concepts is used and the side channel is over-regularized the expressivity of the model is constrained, which in turn limits both the quality generated samples (measured by FID) and, consequently, the steerability.
19

---

Published as a conference paper at ICLR 2026
Table Ablation on the side channel regularization. Generation results on StyleGAN2 CelebA- HQ, using supervised ResNet18 classifiers for concept supervision during training.
Concept
KL Weight
Size
Target Ace. (t)
FID (1)
Set
(generation)
(forward)
0
5
0.548
6.855
5
0.551
7.248
all
10
5
0.550
7.546
0.549
7.590
40
5
0.543
7.743
5
0.659
11.130
5
0.670
11.019
balanced
10
5
0.609
11.870
20
5
0.596
12.374
0.561
12.687
0.669
10.946
0.773
8.661
halanced
10
10
0.663
10.309
20
10
0.664
10.481
40
10
0.665
9.216
Table 8: Ablation on the side channel regularization. Disentanglement results on StyleGAN2 CelebA-HQ, using supervised ResNet18 classifiers for concept supervision during training.
Concept Set
KL
Size
Acc. (1) (after swap)
Acc. (1)
c (t)
DCI a (t)
DCI [c, s] (t)
Weight
(after swap)
0.163
0.095
0.161
0.984
0.838
0.156
0.103
0.154
0.983
0,822
all
10
0.162
0.107
0,160
0,982
0.796
20
5
0.149
0.098
0.148
0.980
0.750
40
5
0.153
0.087
0.151
0.980
0.677
0
0.206
0.096
0.202
0.994
0.805
0.207
0.043
0.201
0.999
0.798
balanced
10
0.198
0.086
0.192
0.997
0.813
0.194
0.076
0.188
0.998
0.755
40
0.180
0.073
0.176
0,994
0.712
10
0.260
0.091
0.235
0.989
0.819
10
0.249
0.090
0.240
0,990
0.819
balanced
10
10
0.230
0.071
0.213
0,983
0.742
0.228
0.074
0.214
0.983
0,742
40
10
0.212
0.084
0.210
0,983
0.696
Table 9: Effeet of removing the unsupervised latent 8. Disentanglement results StyleGAN2
CelebA-HQ, using supervised ResNet18 classifiers for concept supervision during training.
Concept
Size
Target Ace, (t)
Target Acc. (f)
Target Acc. (t)
Target Acc. (t)
FID (L)
Set
(i + a) (a - i)
(Hamming Dist.)
(generation)
(forward)
0.170
0.550
0.660
0.551
7.248
all
0.191
0.538
0.664
0,539
7.921
0.375
0.810
0.594
0.670
11.016
balanced
0.329
0.711
0.420
0.625
20.950
0.769
0.765
0.634
0.715
11.589
low corr.
0
0.604
0.837
0.765
0.688
19.501
We also analyzed the disentanglement between and s for different levels of regularization of the side channel. For this, use the Disentanglement, Completeness, and Informativenes (DCI) metric
20

---

Published as a conference paper at ICLR 2026
(Carbonneau et al., 2022) and replicate the disentanglement experiment using the autoencoder chitecture of the VHCB layer. The procedure follows the same steps as the original disentanglement experiment explained in Section 4.2, with the only difference that, after swapping the unsupervised vectors 81 and 82, we project the reconstructed latent representation back into the VHCB latent space. If e and 8 indeed disentangled, both components should be accurately recovered after the swap. In the case of the DCI we observe that: (i) the DCI for s is always significantly lower that the DCI for c, indicating that concepts are not directly encoded in the unsupervised latent; (ii) the DCI for с and for [c. s) is comparable, suggesting that does not contain relevant concept information. Similar to the steerability results, we observe a slight degradation in these metries, likely caused by the limited expressivity of the model. Nevertheless, the disentanglement experiment (accurucy after swap) shows that the concept information can still be reliably recovered even after modifying the side channel, regardless of the degree of regularization applied to it.
Effect of removing the unuspervised latent 8. The unsupervised latent 8 (side channel) used to handle concept incompleteness. As discussed in Section 3.1, generation depends factors that cannot be fully captured by human-interpretable concepts (e.g., texture, lighting, or object position), making the predefined concept inherently incomplete. Removing the side channel produces the following effects: (i) for large concept sets, we observe that removing the side channel does not have significant impact in performance, following the results obtained in the ablation of the weight of the side channel regularization; (ii) for reduced concept sets, we observe a significant drop in generation quality, with FID nearly doubling, due the limited expressiveness of the latent space; (iii) removing the side channel significantly reduces the generative model's expressivity and, consequently, also (slightly) its steerability.
E ADDITIONAL EXPERIMENTAL RESULTS
In this section, we extend the experiments from the main paper. Specifically, we report results using pretrained StyleGAN2 models for the three CelebA-HQ concept sets, as well results for the CUB- 200-2011 dataset. We also expand the experimental evaluation by including results obtained with a pretrained DDPM.
Intervening concepts. In the VHCB, test-time interventions are performed by setting the latent variable of the target concept the desired value (0 for inactive, 1 for active). For the CB-AE, interventions are applied by assigning the maximum logit (out of the two representing each concept) to the position corresponding to the target value. Metrics for single-concept interventions are caleu- lated by systematically intervening on every concept in the set, with the aim of changing the target concept while keeping all other concepts unchanged.
Evaluating single concept interventions. For each target concept, we identify I pre-bottleneck latents w that generate images with P(yarget (x) < 0.5 (target concept inactive) and 1k latents with P(Murge = 1|x) > 0.5 (target concept active). These latents are then used to do single-concepi interventions both directions, with the same latent set applied across all models to ensure fair comparison. Because some concepts are underrepresented in the dataset, the pretrained generative model may fail to reliably generate images where those concepts are active. As a result, in the large imbalanced CelebA-HQ regime, Ik active latents could only be obtained for subset of the 40 concepts. Specifically, active latents were obtained for the following concepts: 'Arched Eyebrows', 'Attractive', 'Bangs', 'Big Lips', 'Big Nose", 'Chubby', 'Goatee', 'Heavy Makeup', 'High Cheeck- bones', 'Male', 'Mouth Slightly Open', 'Mustache', 'No Beard', 'Pale Skin', 'Smiling', 'Wearing Lipstick', and 'Young'.
E.1 STYLEGAN2 ON CELEBA-HQ
Concept inference. We observe the results generalize across the 3 considered concept subsets. As shown in Table 10, our proposed VHCB consistently improves over the deterministic CB-AE across concept inference metrics, under both ResNet18 and CLIP pseudo-label supervision. Notably, the VHCB improves the soft metrics, suggesting that its posterior is properly aligned the pretrained classifiers. While performance decreases for both models with CLIP supervision due to the noisier zero-shot labels, VHCB remains clearly superior, indicating a more robust mapping from w to c. The gains are even larger for disentanglement: VHCB achieves substantially higher accuracy and
21

---

Published as a conference paper at ICLR 2026
Table 10: Concept inference and disentanglement between and 8. StylcGAN2, Ce
Concept Inference
Disentanglement
Pseudo Label
Concept Set
Model
Acc. (t)
Cosine Sim.
TV
Acc. (f)
Cosine Sim.
TV (L)
Clf.
VHCB
all 0.855
0.804
0.148
0.927
0.917
0.076
CB-AB
all 0.857
0.763
0.161
0.901
0.853
0.101
VHCB balanced
0.757
0.883
0.217
0.867
0.937
0.111
ResNet18
CB-AE balanced
0.761
0.872
0.234
0.776
0.811
0.192
VHCB low corr.
0.791
0.831
0.192
0.874
0.947
0.110
CB-AE low corr.
0.779
0.739
0.238
0.701
0.803
0.229
VHCB all 0.623
0.613
0.337
0.921
0.901
0.083
CB-AB all 0.565
0,469
0.403
0.875
0.775
0.101
VHCB balanced 0.558
0.713
0.353
0.824
0.918
0.138
CLIP
CB-AE balanced 0.552
0.601
0.411
0.709
0.735
0,248
VHCB low corr.
0.599
0.730
0.299
0.854
0.943
0.125
CB-AE low corr.
0.607
0.639
0.346
0.667
0.782
0.248
8
$2
9
22
8
82
81
82
Ci
C2
Ci
C2
Figure 10: Qualitative evaluation of the disentanglement between s and c with the VHCB layer and StyleGAN2 models pretrained CelebA-HQ.
cosine similarity with lower TV, showing that modifying has minimal effect concepts. The qualitative results shown in Figure 10 further indicate that modifying s does not induce semantic changes in the generated images. This stronger separation arises from the compact side channel of VHCB (only 5 bits compared to 40 dimensions in CB-AE), which restricts concept leakage into and enforces cleaner division between supervised and unsupervised factors.
Single concept intervention. We next evaluate the models' ability to manipulate outputs via single- concept interventions, with results reported in Figure 11 and Tables 11 and 12 (intervening on every concept in each set). The VHCB consistently outperforms the CB-AE in terms of target accuracy, learning a robust mapping between latent concepts and post-bottleneck embeddings i without requiring explicit intervention losses. We observe that the CB-AE shows slightly higher non-tut non-target accuracy across settings, this does not indicate more robust interventions. Instead, it reflects weaker ability to modify the target concept: in many cases, the target is unchanged (lower target accuracy), leaving the output the same and artificially inflating non-target accuracy, which is com- puted over all Ik interventions. In contrast, the VHCB achieves better compromise, with large gains in target accuracy (average increase ~46%) and only a minor reduction in non-target accuracy (average drop ~7%). For example, the now-coreelation set with ResNet18, target accuracy rises from 0.42 to 0.77. while non-target accuracy decreases slightly from 0.83 to 0.76. Similarly, with CLIP, target accuracy improves from 0.32 0.57, with non-target accuracy dropping from 0.83
22

---

Published as a conference paper at ICLR 2026
Table 11: Single concept interventions (activation). StyleGAN2, CelebA-HQ.
Single (i a)
Concept Set
Target
Non Target
Non Target
Non Target
Pseudo Label Clf.
Model
Acc. (1)
Acc. (1)
Cosine Sim. (f)
TV (L)
VHCB
all
0.170
0.903
0.868
0.098
CB-AE
all 0.105
0.924
0.910
0.078
VHCB
balanced
0.376
0.812
0.869
0.160
ResNet18
CB-AE
balanced
0.283
0.846
0.898
0.135
VHCB
low corr. 0.769
0,762
0,853
0.187
CB-AE
low corr.
0.420
0.825
0.921
0.136
VHCB
0.131
0.890
0.830
0.112
CB-AE
all 0.085
0.922
0.906
0.079
VHČB balanced
0.346
0.706
0.740
0.245
CLIP
CB-AE
balanced 0.206
0.839
0.884
0.135
VHCB low corr.
0.567
0.691
0.813
0.234
CB-AE
low 0.317
0.829
0.921
0.130
Table 12: Single concept interventions (deactivation). StyleGAN2, CelebA-HQ
Single (a +
Concept
Target
Non Target
Non Target
Non Target
Pseudo Label Clf.
Model
Set
Acc. (1)
Acc. (1)
Cosine Sim. (1)
TV (4)
VHCB
all 0.550
0.902
0.868
0.098
CB-AE
all 0.453
0.924
0,910
0.078
VHCB
balanced
0.810
0.758
0.859
0.190
ResNet18
CB-AE
balanced
0.833
0.777
0,881
0.178
"VHCB" low corr.
0.765
0.781
0.867
'0.177
CB-AE
low corr. 0.554
0.822
0.918
0.138
VHCB
all 0.536
0.885
0.821
0.116
CB-AE
all 0.311
0.922
0,906
0.080
VHCB balanced
0.682
0.627
0.732
0.296
CLIP
CB-AE
balanced
0.509
0.779
0,868
0.177
VHCB low corr. 0.532
0.694
0.821
0.233
CB-AE
low corr. 0.404
0.834
0.918
0.130
to 0.69. The same pattern is observed in the complete and balanced sets, showing that the VHCB achieves better balance between modifying target concepts and preserving non-target concepts.
We also observe that steerability is strongly influenced by concept distribution in the dataset: in CelebA-HQ most concepts are underrepresented, making deactivation easier than activation. The effect of imbalance and model biases is also evident when moving from the full set of 40 unbal- anced concepts to balanced and low-correlation subsets, with metrics consistently improving in both models. Therefore, the definition of the concept set has also large influence in the CBGM perfor- mance.
Hamming distance intervention. To mitigate the effect of out-of-distribbtion concept configura- tions that arbitrary interventions yield during evaluation, we restrict interventions to target one of the 100 most probable concept patterns observed in the dataset, selected via minimum Hamming distance. Results are reported in Table 13. This approach consistently improves intervention suc- cess rates in both models, and all the concept sets, with the VHCB still yielding superior metrics, as shown in Table 13. These results further support that steerability depends not only on the expres- siveness of the CB layer but also biases inherent in the pretrained generative model.
Impact of concept incompleteness and model biases. With large concept sets, correlations (spu- rious not) between concepts can limit steerability, some configurations fall out of distribution.
23

---

Published as a conference paper at ICLR 2026
Smiling
All
Balanced
Low corr.
(i+a) a)
(a+1) i)
(1+ o)
(a + f)
Mouth Open
All
Balanced
Low corr.
(a + 1)
(1+a) a)
(a () t)
(++ a)
(a ++
(i a)
Figure 11: Examples of single concept interventions in CelebA-HQ with VHCB layers trained on different concept sets.
Bangs
Hair
CB-AE
VHCB
CB-AE
VHCB
(i a)
(f +
(i - a)
(i 1++0 a)
Figure 12: Examples of single concept interventions CelebA-HQ with VHCB and CB-AE layers.
However, concept information is generally better isolated. In smaller concept sets, low correlations make interventions easier, but the model may inadvertently capture information from unmodeled concepts due to dataset biases. For example, in CelebA-HQ, hair color and 'smiling' exhibit spuri- ous correlations (this effect can be observed in some interventions in Figure 11). Introducing hair color concepts (e.g., "blond hair', "black hair') allows to better isolate the concept 'smiling'. How- ever, single concept interventions this setting become harder, since activating mutually exclusive configurations. Therefore, the definition of the concept set has
hair colors yields also a large influence in the CBGM performance. To further illustrate this effect, Table 13 presents examples of interventions aimed activating the 'Mustache' concept, which is strongly correlated with 'Male' and 'No Beard', When modeling the complete set concepts, most interventions fail, 'No Beard" (which was already active) and 'Mustache' mutually exclusive, and due to the
24

---

Published as a conference paper at ICLR 2026
Table 13: Hamming distance-based interventions. StyleGAN2, CelebA-HQ.
Hamming Distance
Concept Set
Target Acc. (1)
Non Target
Non Target
Non Target
Pseudo Label Clf.
Model
Acc. (f)
Cosine Sim. (t)
TV (L)
VHCB
all
0.660
0.938
0.863
0.085
CB-AE
0.542
0.955
0,910
0.066
VHCB
balanced
0.594
0.868
0.892
0.129
ResNet18
CB-AE
balanced
0.374
0.837
0.896
0.145
VHCBB Tow corr.
0.634
0.845
'0.895"
0.149
CB-AE
low corr.
0.418
0.880
0.936
0.110
VHCB
all 0,595
0.923
0.827
0.099
CB-AE
all 0.466
0.949
0.902
0.070
VVHCB balanced
0.423
0.719
0.753
0.239
CLIP
CB-AE
balanced
0.252
0.854
0.896
0.130
VVHCB" Tow corr.
06,344
0.729
'08888"
'0,223
CB-AE
low
0.496
0.869
0.933
0.113
Mustache
Low corr.
i
4
All
g
Figure 13: Examples of interventions activating the concept mustache, we can see the effect of the concept incompleteness and the effect of the biases present in the data, captured by the generative model.
data biases, 'Female' and 'Mustache' simultaneously active is also out-of-distribution. However, when using reduced concept that excludes 'No Beard' and 'Male', interventions 'Mustache' succeed. In cases where the mustache is added to a female image, the model changes the gender 'Male' to accommodate the intervention. Additionally, we observe the model captures spurious correlations associated with the 'Mustache' concept, such as dark hair and higher likelihood of wearing sunglasses.
Image generation. The probabilistic formulation of the VHCB enables direct generation from spec- ified concept configurations. Table 14 reports metrics showing that generated images actually reflect concepts in Since random sampling of concept configurations can produce out-of- concept configurations, we also sample concept patterns according to their empirical frequency in the training data. This further improves accuracy and highlights how biases in the base generative model can limit the steering capacity the CBGM. As with interventions, we observe a general improvement in generation accuracy when considering balanced and low-correlation concept sets.
25

---

Published as a conference paper at ICLR 2026
Bangs 1, MouthOpen = 1, Young = 0, Smiling
Mustache =
Bangs 0, Ooet
0, Young =
1, Smiling 0, Mustache = 0
Figure 14: Examples of generation from specific concept configurations in CelebA-HQ. The results
were obtained with pretrained StyleGAN2.
Table 14: Generation. StyleGAN2, CelebA-HQ
Random
Patterns
Pseudo Label CIf.
Concept Set
Model
Target Acc. (f)
Target Acc. (t)
all balanced
0.551
0.873
ResNet18
0.670
0.778
low corr.
0.715
0.814
VHCB
all balanced
0.533
0.830
CLIP
0.607
0.679
low corr.
0.632
0.687
Table 15: Image quality. We report FID30k using a StyleGAN2 pretrained on CelebA-HQ.
Forward
Random Generation
Concept Set
Pseudo Label Clf.
Model
FID (4)
FID (L)
VHCB
all
7.248
16.095
CB-AE
11.645
VHHCB
balanced
11.016
19,636
ResNet18
CB-AE
balanced
9.169
VHCB low corr.
11.589
34.925
CB-AE low corr.
13.590
VHCB
all 6.388
19.601
CB-AE
12,070
VHCB balanced
10,345
19.163"
CLIP
CB-AE balanced 11.439
VHCB low com.
9,666
24.037
CB-AE low corr.
8.083
Table 15, we report FID scores for images generated using the full CBGM (Forward) and by directly sampling from the VHCB latent space (Random Generation). In the forward setting, the VHCB achieves a substantial improvement over the baseline the complete concept set. However, this gap narrows when considering reduced concept sets, with the baseline outperforming VHCB in two cases. In these experiments, the baseline uses side channel dimension 40 (Kulkarni al., 2025), while our model uses of dimension 5. The difference in latent dimensionality
26

---

Published as a conference paper at ICLR 2026
Table 16: Concept inference and disentanglement between cand 8. StyleGAN2, CUB-200-2011.
Concept Inference
Disentanglement
Pseudo Label Clf.
Model
Acc. (1)
Cosine Sim. (1)
TV (L)
Acc. (1)
Cosine Sim. (1)
TV (L)
VHCB CB-AE
0.743
0.914
0.167
0.864
0.957
0.100
ResNet18
0.720
0.919
0.181
0.962
0.991
0.027
VHCB CB-AE
0.554
0,812
0.246
0.792
0,943
0.127
CLIP
0.570
0.543
0.355
0.612
0.867
0.216
Table 17: Single concept interventions (activation). StyleGAN2, CUB-200-2011.
Single (i --aa)
Pseudo Label
Non Target Cosine Sim. (t)
Model
Target Acc. (f)
Non Target Acc. (1)
Non Target TV (L)
Clf.
VHCB CB-AE
0.392
0.754
0.896
0.164
ResNet18
0.112
0.669
0,895
0.184
VHCB CB-AR
0.372
0.665
0.859
0.196
CLIP
0.226
0.905
0.905
0.157
affects model expressivity and, consequently, generative performance. However, we note that the VHCB achieves comparable performance with a much more compact latent space. In contrast, FID increases substantially when generating from random latent samples, out-of-distribution concept configurations reduce the similarity between the distributions of generated and real images in the learned feature space.
E.2 STYLEGAN2 ON 2UB-200-2011
We also evaluate the different CB layers in the CUB-200-2011 dataset, evaluating the tasks did in CelebA-HQ. This is a more challenging dataset, as the annotations are more noisy, as some of the attributes cannot be seen in the pictures, and the size of the dataset is smaller, which does affect the CB layer itself (as it is trained with generated data) but it does affect the performance of the classifiers employed for training (in the supervised case) and for automatic evaluation.
Concept Inference. In this case, we again observe the robustness of the VHCB layer. Metrics in Table 16 show that, while both models perform similarly when trained with supervised classifiers, the VHCB shows a substantially smaller drop in performance when trained with zero-shot classifiers, which tend be more noisy.
Interventions. We next evaluate the models' ability to manipulate model outputs via concept inter- ventions. Tables 17 and 18 contain the metrics for single concept interventions, Table 19 contains the results for Hamming-distance interventions, and Figure 15 provides qualitative examples of the interventions. Although in this case the models perform more closely, we observe that the VHCB has better capability to activate concepts.
Generation. Table reports metrics showing that generated images actually reflect concepts in c, and Figure 16 shows examples of images generated with fixed concepts. Since random sampling of concept configurations can produce distrribtttiitooonn concept configurations, we also sample concept patterns according their empirical frequency in the training data. This further improves accuracy and highlights how biases the base generative model can limit the steering capacity of the CBGM.
27

---

Published as a conference paper at ICLR 2026
Table 18: Single concept interventions (deactivation). StyleGAN2, CUB-200-2011.
Single (a
Pseudo Label
Non Target
Model
Target Acc. (1)
Non Target Ace. (1)
Non Target TV (L)
Cif.
Cosine Sim. (1)
VHCB CB-AE
0.707
0.715
0.895
0.176
ResNet18
0.886
0.586
0.901
0.192
VHCB
0.513
0.660
0.885
0.190
CLIP
CB-AE
0.430
0.711
0.921
0.156
Black Upperparts
White Underparts
Black Wing
(a + 1)
(i ++00
(a - i)
(i-+0) a)
(a + i)
(i ( u)
Figure 15: Examples of single concept intervention in CUB-200-2011. Results obtained with StyleGAN2 pretrained on CUB.
Table 19: Hamming distance-based interventions. StyleGAN2, CUB-200-2011.
Hamming Distance
Pseudo Label
Non Target
Model
Target Acc. (f)
Non Target Acc. (1)
Non Target TV (L)
Clf.
Cosine Sim. (t)
VHCB CB-AE
0.642
0.764
0,884
0.171
ResNet18
0.734
0.697
0,888
0.187
VHCB CB-AE
0.342
0.675
0.874 0.913
0. 087
CLIP
0.308
0.722
0.154
Black pperrupsss 0, WhiteUnderparts 1, BlackWing 0, BlackBill
pperpuppsss 1, WhiteUnderpartss 1, Black BlackBill
Figure 16: Examples of generation from specific concept configurations in CUB. Results obtained with a StyleGAN2 pretrained on CUB.
28

---

Published as a conference paper at ICLR 2026
Table 20: Generation. StyleGAN2 CUB.
Random
Patterns
Pseudo Label Cif.
Model
Target Aec. (t)
Target Acc. (1)
VHCB
0.669
0.750
ResNet18
CB-AE
VHCB
0.516
0.550
CLIP
CB-AE
Table 21: Image quality. StyleGAN2 CUB.
Forward
Random Generation
Pseudo Label Clf.
Model
FID (L)
FID (L)
VHCB CB-AE
14.864
16.882
ResNet18
27.243
VHCB CB-AE
16,952
24.762
CLIP
24.597
E.3 DDPM ON CELEBA-HQ
We next evaluate our approach on a diffusion- generator, specifically on the pretrained DDPMs deseribed in Section C. We replicate the evaluation protocol from our StyleGAN2 experiments, using the same concept sets and pseudo-label classifiers. Results are reported in Tables 22 and 23 for direct comparison with StyleGAN2. Unlike StyleGAN2, the post-hoe CB-AE baseline from Kulkarni al. (2025) is not available for this setting, but this does not affect our objective here: our aim is to verify whether the same overall trends observed with StyleGAN2 also hold for DDPMs, rather than to benchmark against all baselines.
Concept inference. Table 22 shows that the VHCB captures concept information reliably in the diffusion setting. With ResNer18 supervision, we observe competitive accuracy on the balanced and low-correlation sets, while disentanglement metrics remain consistent across concept sets. As in the StyleGAN2 case, CLIP supervision yields lower overall scores due to noisier labels, but the trends remain stable. Importantly, the side channel retains only limited concept information, confirming that the compact binary bottleneck continues to mitigate leakage in DDPMs. Injecting the VHCB model at all levels the U-Net's upsampling path provides only slight improvements in these metrics compared to the case where the VHCB module is only injected at the bottleneck of the U-Net architecture.
Concept interventions. The steerability of the VHCB within DDPMs is summarized in Table 23. We evaluate activation (i + a), deactivation (a -> i), and interventions restricted to training con- cept patterns (minimum Hamming distance). As in the StyleGAN2 setting, we observe substantial improvements in target accuracy when deactivating concepts, with non-target attributes largely pre- served. However, unlike in StyleGAN2, Hamming distance interventions do not improve target accuracy: while they achieve excellent preservation of non-target concepts (above 0.98 across all settings), they fail to reliably enforce the target concept. This contrast suggests that enforeing con- cept changes in DDPMs is harder than in GANs, likely because the denoising trajectory resists out-of-distribution edits. Moreover, this may reflect a limitation of introducing the VHCB only at the bottleneck of the U-Net backbone. Injecting the output of the VHCB module at all levels of the U-Net architecture's 's upsampling path is a first step to sidestep this issue. Indeed, quantitative met- rics across concepts generally improve in this case. However, believe that further research must be performed properly characterize the architectural trade-offs, identify the most effective injection strategy, and ensure stable and consistent improvements.
29

---

Published as a conference paper at ICLR 2026
Table 22: Concept inference and disentanglement between e and 8.
DDPM, CelebA-HQ.
Concept Inference
Disentanglement
Pseudo Label Clf.
Concept Set
Model
Acc. (t)
Cosine Sim. (t)
TV (4)
Acc. (t)
Cosine Sim. (t) TV (4)
all 0.733
0.595
0.254
0.733
0.595
0.255
ResNet18
balanced
0.546
0.627
0,402
0.545
0.633
0,402
VHCB
corr.
0.621
0.333
0.383
0.622
0.338
0.383
(hottleneck)
all 0.684
0.431
0.320
0.680
0.431
0.321
CLIP
balanced 0.577
0.224
0,430
0.569
0.231
0,434
low corr.
0.339
0.481
0.569
0.345
0.483
0.567
0.753
0,547
0.242
0.750
0,548
0.243
ResNet18
balanced 0,567
0.601
0.404
0,568
0.600
0.405
VHCB (all levels)
low corr.
0.563
0.480
0.410
0.557
0,482
0.412
all 0.688
0,437
0.308
0.688
0,437
0,308
CLIP
balanced 0.545
0.667
0.393
0.552
0.672
0.391
low 0.547
0.668
0.393
0.552
0.672
0.391
Table 23: Test-time interventions. Evaluation of single-concept activation (1-> a). deactivation
(a i), and interventions guided by training concept patterns (minimum Hamming distance). DDPM, CelebA-HQ.
Single (i +)
Single (a
i)
Hamming Dist.
Pseudo Concept Set
Target Acc. (t)
Non Target
Target Acc. (t)
Non Target
Target Acc. (f)
Non Target
Model
Label Clf.
Acc. (f)
Acc. (f)
Acc. (f)
all balanced
0.131
0.905
0.546
0.898
0.047
0.993
ResNet18
0.406
0.625
0,250
0.491
0.044
0,983
VHCB
low corr.
0.187
0.910
0.571
0.857
0.041
0.990
(bottleneck)
all 0.131
0,905
0.546
0.898
0.041
0.992
CLIP
balanced 0.343
0.607
0.250
0.500
0.037
0.980
low 0.219
0.906
0.571
0.847
0.046
0.991
all 0.150
0.904
0.568
0.895
0.332
0,952
ResNet18
balanced
0.562
0.598
0.313
0.455
0.378
0.851
VHCB (all levels)
low corr.
0.531
0.584
0.464
0.469
0,205
0.877
all 0.118
0.901
0.545
0.899
0.296
0.953
CLIP
balanced 0.281
0.651
0,250
0,500
0.197
0.842
low corr.
0.437
0.616
0.357
0.489
0.213
0.844
Qualitative Results. Examples of test-time interventions on the CelebA-HQ dataset are presented in the following figures. Each example consists of triplet of images displayed side by side using the model configuration of concept code rate 40/800 and an unsupervised binary latent with code rate 5/50. The left image shows the output of the baseline DDPM model without any intervention or VHCB layer. The center image illustrates the result of applying the VHCB intervention at the bottleneck layer only (as depicted in Figure 8). The right image shows the effect of injecting the VHCB at all levels of the upsampling path of the U-Net (Figure 9). We demonstrate interventions on several semantic concepts: Smiling (Figure 17). Male (Figure 18), and Mouth Slightly Open (Figure 19) (each from the inactive to the active state). These images were taken following the same evaluation scheme: intervening on concepts starting only at timestep denoising diffusion implicit model (DDIM) Scheduler for 50 total number of timesteps. Comparing both DDPM variants with the VHCB, there is no significant impact for the VHCB when injected in all levels when compared the injected at bottleneck only model. Note that the quality of the images depends on the pretrained model as previously stated, where the training data used were mre-generated images from the original model.
400 and using the
30

---

Published as a conference paper at ICLR 2026
Figure 17: Examples of single concept intervention in CelebA-HQ dataset for the smiling attribute, from inactive to active. Results obtained with DDPM pretrained on CelebA-HQ. VHCB model configuration: concept code rate of R 40/800 and unsupervised latent code rate of R 5/50. Each image triplet shows: baseline DDPM (left), VHCB at bottleneck (center), and VHCB injected across all upsampling layers (right).
Figure 18: Examples of single concept intervention in CelebA-HQ dataset for the male attribute,
from inactive to active. Results obtained with DDPM pretrained on CelebA-HQ. VHCB model
configuration: concept code rate of R 40/800 and unsupervised latent code rate of R 5/50. Each image triplet shows: baseline DDPM (left), VHCB at bottleneck (center), and VHCB injected across all upsampling layers (right).
31

---

Published as a conference paper at ICLR 2026
Figure 19; Examples of single concept intervention in CelebA-HQ dataset for the mouth slightly open attribute, from inactive to active. Results obtained with a DDPM pretrained on CelebA-HQ. VHCB model configuration: concept code rate of R 40/800 and unsupervised latent code rate of = 5/50. Each image triplet shows: baseline DDPM (left), VHCB bottleneck (center), and VHCB injected across all upsampling layers (right).
Table 24: Generation. DDPM, CelebA-HO.
Random
Patterns
Pseudo Label Clf.
Concept Set
Target Acc. (f)
Target Acc. (t)
Model
all balanced.
0.506
0.711
ResNet18
0.490
0.490
VHCB
low corr.
0.503
0.580
(bortleneck)
all 0.505
0.705
CLIP
balanced. 0.491
0.545
low corr.
0.502
0.530
all 0.506
0.752
ResNet18
balanced.
0.498
0.561
VHCB (all levels)
low corr.
0,511
0,533
all 0.503
0.761
CLIP
halanced, 0.499
0,563
low corr.
0.510
0.552
Image generation. Results are summarized in Table 24, As with interventions, generated images generally reflect the intended concepts, confirming that the VHCB enables controllable generation within DDPMs. Sampling concept patterns according to their empirical frequency in the training data typically improves target accuracy compared purely random sampling, particularly for the full concept set. For the balanced and low-correlation subsets, improvements are smaller and occa- sionally absent.
We attribute this variability not to limitations of the VHCB itself, but to how effectively concept information can be propagated through the U-Net denoiser over the many iterative steps of the diffusion process. As discussed earlier, injecting the VHCB at all levels of the upsampling path provides a stronger and more distributed conditioning mechanism, which is particularly relevant given the multi-step nature of DDPMs. The generation results reflect this: the multilevel variant yields slightly higher target accuracy across all concept subsets, although the improvements remain modest overall.
32

---

Published as a conference paper at ICLR 2026
Taken together, these findings indicate that while the proposed framework enables effective concept- based control, achieving stronger and more consistent gains will likely require a deeper exploration of how concept representations are integrated and preserved throughout the denoising trajectory in diffusion architectures.
33


Reviews:
[{"review_id": "yzRfxkOsJh", "reviewer": "Reviewer_AUUQ", "summary": "This paper proposes the Variational Hard Concept Bottleneck (VHCB) layer that uses binary latent variables protected by error-correcting codes to map internal representations to hard human-interpretable concepts within pretrained generative models. Also, the authors introduce a systematic evaluation framework for assessing the steerability of CBGMs across various tasks, and demonstrate that VHCB consistently enhances steerability empirically. In general, the paper is clearly written, the metrics are sensible, and the authors also points out the impact of correlations and biases in training data, which is helpful for future work.", "questions": "1. Can you report a simple leakage diagnostic to empirically support the claim that hard concepts + compact s reduce leakage?\n2. What would happen if you insert VHCB at multiple U-Net locations on DDPMs?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors introduce the limitations of existing approaches (i.e. relying on soft concept representations and susceptible to concept leakage), and the proposed VHCB layer is thus well motivated and is a clear conceptual advance.\n2. The systematic evaluation framework for CBGMs proposed by this paper is thorough and comprehensive, which can be useful for many purposes.\n3. The paper presents compelling empirical results that VHCB yields large gains in accuracy, and also transparently analyzes failures due to dataset correlations/bias.", "weaknesses": "1. The main comparison in this paper is VHCB vs. CB-AE. It would be helpful to compare the proposed method to several more baselines.\n2. The authors use independent concept classifiers to obtain labels, which could introduce label bias that propagates into training and evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LPEdEIAHHj", "reviewer": "Reviewer_5vRH", "summary": "The paper introduces the Variational Hard Concept Bottleneck (VHCB) layer, a probabilistic extension of Concept Bottleneck Generative Models (CBGMs). Instead of previous approaches, it uses hard binary concept representations (binary latent variables with probabilistic smoothing) to reduce concept leakage and a probabilistic latent formulation to enable direct sampling from concept configurations. The approach is validated on StyleGAN2 and DDPM trained on CelebA-HQ and CUB-200-2011 datasets.", "questions": "- Why a symmetric KL instead of standard cross-entropy or forward KL? \n- It is hard to disentangle whether leakage prevention comes from binary concepts or the KL regularization. Can the authors speak more to this? Can you train a simplified version of your model in a toy setting like in the work of GlancNets so that you can tease these issues apart? \n- I would suggest that the authors also look into measure of disentanglement or independence between latent factors to concretize their claims.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Addresses an important and previously challenging problem in the concept bottleneck literature. \n- The side channel regularization is quite interesting. This 'independence regularization' term is quite different from those in the literature. It acts as a capacity control and seeks to prevent concept leakage into the unsupervised channel. \n- The evaluation, in the work, compared to the baselines, is quite thorough. \n- The paper also has a nice trick for reducing potentially incoherent steering on concepts. They restrict interventions to target one of the 100 most probable concept patterns observed in the dataset, selected via minimum Hamming distance. This ensures that interventions are more coherent. Of course, this is also limits the ability of the model to extrapolate.", "weaknesses": "- The probabilistic formulation is based on existing binary VAEs (from Martínez-García et al., 2025). This works applies it to CBGMs directly. \n- The paper introduces a binary side channel regularized toward a factorized prior via a KL penalty to mitigate concept incompleteness. This mechanism does not guarantee independence from the concept latents; it only limits capacity. The authors could strengthen their claims by including ablations on the KL weight, measuring mutual information between, and or visualizing variations induced by the unsupervised channel to confirm it captures residual, non-semantic factors.\n- The current disentanglement evaluation, in the paper, is also not quite appropriate. It involves: sampling two independent latent pairs, swapping the unsupervised latents  to form mixed pairs, and regenerating images and comparing whether the concepts remain unchanged. This test examines whether observable concept labels change, but not whether these latents themselves are statistically independent. You could have apparent disentanglement (concepts unchanged) while they still encodes correlated concept information in latent space. If those classifiers are not sensitive to subtle variations, you can get artificially high “disentanglement” scores. I would recommend the authors consider other concrete measures of disentanglement. For example, the Glancenets paper that the authors cite has a measure, DCI, and toy experiments that they can use to stress test their approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1p20HdcwBc", "reviewer": "Reviewer_Te4N", "summary": "This paper focuses on concept bottleneck generative models (CBGM) which are interpretable and steerable generative models. Existing CBGMs are deterministic models which limit steering to concepts predicted in existing inputs and may suffer from concept leakage. To overcome these limitations, this paper proposes a Variational Hard Concept Bottleneck (VHCB) layer which allows direct generation from sampled concepts and could reduce concept leakage. They evaluate the proposed method compared to the previous SOTA method and show improvements in steerability.", "questions": "Please address my questions/comments from the weaknesses section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The paper is fairly well-written and easy to follow with good illustrations.\n\n* The paper focuses on the relatively under-explored topic of concept bottleneck generative models.\n\n* The proposed VHCB layer seems to be novel and it makes intuitive sense to move from deterministic CB layers (like in CB-AE or CBGM) to probabilistic ones like VHCB, especially given that the generative task itself is not deterministic.\n\n* I also appreciate the extensive experiments, especially the disentanglement experiments which were not reported in prior work.", "weaknesses": "* It is unclear to me how the error correcting codes help mitigate concept leakage. Rather it seems that increasing the dimensionality of the concepts (and the unsupervised embedding) would increase the chances of concept leakage. It would be good to elaborate on why (and which part of VHCB) actually helps to mitigate concept leakage.\n    * Another related concern is that concept leakage might actually stem from the unsupervised embedding itself. It would be good to discuss this in the paper and comment on whether this can be removed from concept bottleneck generative models.\n\n* Based on Table 10, it seems VHCB sometimes achieves better and sometimes worse FID scores than CB-AE, but there is no explanation/intuition offered for this.\n    * Related to this, the FID scores are significantly higher when generating from random concepts which is attributed to some concept combinations potentially being out of distribution. I wonder if this can be improved by generating images from random concept samplings during training and applying a loss on the concept predictions (or an image generation loss to improve the quality of those images) similar to the intervention losses in CB-AE.\n    * The higher FID scores when generating from random concepts also seems to be a major limitation because that was proposed as one of the main contributions of this work (which was not possible in the deterministic CBGM or CB-AE).\n\n* Minor issues (typos):\n    * L113: \"Blottleneck\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)