Graph Concept Bottleneck Models
Authors: Haotian Xu
Tsui-Wei Weng
Lam M. Nguyen
Tengfei Ma
Venue: Accepted by TMLR ()

Abstract: Concept Bottleneck Models (CBMs) have emerged as a prominent framework for interpretable deep learning, providing human-understandable intermediate concepts that enable transparent reasoning and direct intervention. However, existing CBMs typically assume conditional independence among concepts given the label, overlooking the intrinsic dependencies and correlations that often exist among them. In practice, concepts are rarely isolated: modifying one concept may inherently influence others. Ignoring these relationships can lead to  oversimplified representations and weaken interpretability. To address this limitation, we introduce **Graph CBMs**, a novel variant of CBMs that explicitly models the relational structure among concepts through a latent concept graph. Our approach can be seamlessly integrated into existing CBMs as a lightweight, plug-and-play module, enriching their reasoning capability without sacrificing interpretability. Experimental results on multiple real-world image classification benchmarks demonstrate that Graph CBMs (1) achieve higher predictive accuracy while revealing meaningful concept structures, (2) enable more effective and robust concept-level interventions, and (3) maintain stable performance across diverse architectures and training setups.

Published in Transactions on Machine Learning Research (02/2026)
Graph Concept Bottleneck Models
Haotian Xu
haoliann.mu@stomnbboookk. eduu
Deportment of Applied Mathematics 8 Statistics State University of New York Stony Brook
Tsui- Wei Weng
lueng edu
Hokcroğia Data Science Institate University of Colifornia, San Diego
Lam M. Nguyen
Hamagrgen.mmttddimmmcmmm
Thomas J. Watson Research Center
IBM Research
Tengfei Ma
tengfei.maaSstongnnne edu
Department of Biomedical Informatics
State University of New York Stony Brook
Reviewed on OpenReview: https: // openreview. net/ ht////ormmm id a4ozUY
Abstract
Concept Bottleneck Models (CBMs) have emerged as a prominent framework for interpretable doep learning, providing human-understandabll intermodiate concepts that enable transparent reasoning and direct intervention. However, existing CBMs typically assume conditional independence among concepts given the label, overlooking the intrinsic dependencies and correlations that often exist among them. In practice, concepts are rarely isolated: modifying one concept may inherently influence others. Ignoring these relationships can lead to oversimplified pppresentations and weaken interpretability. To address this limitation, we introduce Graph CBMs, a novel variant of CBMs that explicitly models the relational structure among concepts through latent concept graph. Our approach can be scamlessly integrated into existing CBMs as a lightweight, plug-and-play module, enriching their reasoning capability without sacrificing interpretability. Experimental results on multiple real-world image classification benchmarks demonstrate that Graph CBMs (1) achieve higher predictive accuracy while revealing meaningful concept structures, (2) enable more effective and robust concept-level interventions, and (3) maintain stable performance across diverse architectures and training setups. We provide our implementation details at this URL
1 Introduction
Deep neural networks have demonstrated remarkable performance and efficiency across a wide range of tasks in Computer Vision (He et al.] 2016 Dosovitskiy et al.] 2021 Tolstikhin et al., 2021. Natural Language
Processing (Vaswani et al. 2017 Devlin al.] 2019: Brown et al. 2020 Raffel et al. 2020). and Graph Learning (Kipf & Welling 2017 You et al.] 2020b Rong et al. 2020). Despite their success, these models are often considered black baxes due to their lack of interpretability. In safety-critical domains such medical applications, this opacity raises concerns, as there is growing demand for transparent and reliable decision-making processes.
To address this issue, Concept Bottleneck Models (CBMs) Koh et al. 2020 have been introduced to enhance interpretability by mapping hidden representations to a human-underssandablle concept space. In CBMs, each neuron the bottleneck layer corresponds to specific concept with an associated confidence score. Unlike end-to-end models that map raw inputs (e.g., pixel values) directly to output labels, CBMs make

---

Published in Transactions on Machine Learning Research (02/2026)
predictions based the activation scores in the concept space. This formulation enables human intervention the concept activation vector to correct predictions without modifying model parameters.
However, existing CBMs overlook a key aspect: the correlations among concepts. In reality, certain concepts often co-oceur, c.g., "fur" and "whiskers" typically suggest the presence of "tail" or "paws," " while "wings" and "beak" " imply "feathers." Such patterns are well-established in cognitive science; for instance, Kourtzi & Connor [201]] show that visual neurons respond more strongly when related concepts appear together, suggesting the brain encodes concept relevance implicitly. Yet recent CBM approaches (Espinosa Zarlenga ett al. 2022 Kim et al. 2023 Yuksekgonul et al, 2023) fail to model these natural correlations. Ignoring such relationships undermines model trustworthiness, as co-varying concepts can provide critical contextual cues to aid human understanding (Bar) 2004 Oliva & Torralba 2007). This omission weakens interpretability and trust, especially in domains like healthcare, where co-occurring symptoms provide crucial context. For instance, during diagnosis, our model can suggest likely symptoms and their interrelations, while allowing doctors intervene and make the final decision.
Motivated by these insights, we introduce Graph CBMs, framework designed to uncover intrinsic correlations among concepts through learnable graph structure and complement to existing CBMs with enhanced performance and interpretability. We hypothesize the existence of unified, input-independent concept graph that encodes prior semantic knowledge, where individual input samples activate only subset. of relevant concepts. We represent this graph structure using a Graph Neural Network (GNN), and learn it via self-supervised contrastive learning. In our approach, the subgraph of activated concepts for an input image is treated as an augmented view, forming positive samples in the contrastive loss. This encourages the learned graph to capture semantically grounded relations and enables the graph to be integrated plug-in module across various CBM variants.
Our key contributions are summarized as follows:
We propose Graph CBMs, which incorporate a lcarnable concept graph to model interactions among concepts, thereby enhancing both prediction performance and interpretability.
We develop a self-supervised learning framework that automatically constructs latent concept graphs. Empirical results demonstrate its effectiveness on both classification and intervention (on average 1%~2% improvement across 8 datasets).
Our approach shows strong generalization across settings, with or without concept annotations, and works robustly across different backbones. Moreover, Graph CBMs remain effective even under concept interventions.
2
Related Work
Concept Bottleneck Models (CBMs) improve interpretability by introducing intermediate concept. layer that predicts human-understandalbl concepts before performing the final linear classification (Koh et al. 2020), in which the final classification usually ignores the hidden correlation among concepts. Subsequent studies have addressed several practical limitations, such as limited concept sets and restricted predictor expressiveness (Havasi et al.; 2022), and have proposed probabilistic or energy-based formulations that jointly model the input concept label triplet (Xu et al., 2024). Recent extensions focus on modeling richer dependencies among concepts or enabling efficient model editing: Vandenhirtz et al. (024 capture concept dependencies via learned multivariate normal distribution
concept logits; Hu et al. 2025) introduce approxinations that support efficient concept or data edits without retraining: and Prasse et al. (2025) leverage foundation-modee segmentations to construct concept banks for data- efficient visual concept discovery. C2BM (De Felice et al., 2025) embeds a causal graph within the concept layer, enforcing structural-causal constraints derived from prior knowledge or causal discovery. In contrast, our proposed Graph CBMs (G-CBMs) enable flexible, task-adaptive structure learning when causal information is limited or uncertain.
2

---

Published in Transactions on Machine Learning Research (02/2026)
To reduce the reliance on costly concept annotations, a parallel line of work explores automatic or weakly supervised concept discovery. Label-Free CBMs (Oikarinen et al. 2023) employ large pretrained models
and neuron-level interpretability tools such as CLiP-Dissect to extract candidate concepts, combined with
sparse prediction layers to balance interpretability and accuracy. PCBM (Yuksckgonul ct al.
2023) minimizes
annotation costs by training only the final linear classifier on concept activalions. Yang el a. (2023) leverage
submodular optimization to select diverse and discriminative concepts, while Yan et al. 2023 further enhance
concept selection under limited supervision.
A related family of methods, Concept Embedding Models (CEMs), generalize CBMs by representing
concepts in continuous embedding space (Espinosa Zarlenga et al. 2022, Kim et al. 2023 While CEMs capture richer semantic relationships among concepts, they often treat these embeddings as conditionally independent and lack explicit structural modeling.
Unlike prior approaches that rely on fixod or heuristic assumptions about concept independence, our Graph CBMs explicitly model the latent relationships among concepts through a learned graph structure. This formulation enhances predictive accuracy and interpretability, allowing information to propagate across related concepts. As demonstrated in Tables D and 2 Graph CBMs can be seamlessly integrated into both supervised and label-free CBM variants, serving as a plug-in module that injects relational reasoning capabilities into the concept space.
Graph Structure Learning (GSL) aims to infer useful graphs from raw data. Early methods emerged in signal processing (Dong et al. 2019) and probabilistic graphical models (Yu et al.) 2019). In the deep
learning era, GST is often integrated with graph neural networks (GNNs) (Frncceshhi al. 2019 Kipf et al.,
2018 Shang et al. 2020 Kazi ct al. 2022 Ma ct al., 2023 For example, NRI (Kipf et al. 2018 learns latent
interaction graphs in dynamic systems using latent variables, while Franceschi et al. (2019) treats GSL
a bi-level optimization problem over discrete edge distributions. Recent advances also explore contrastive learning for hypergraphs (Wei et al.) 2022). Our approach aligns with this line by learning a deterministic latent graph end-to-end through contrastive supervision, tailored to model concept relationships.
3 Graph Concept Bottleneck Models
Graph CBMs, as shown in Figure 1 aim to introduce latent graph structures into the concept space, enabling models to capture dependencies among concepts that are typically treated as independent in conventional CBMs. Specifically, Graph CBMs construct a latent concept graph where each node represents concept and each edge encodes the learned relational strength between concept pairs. This graph is jointly learned with the concept and label predictors, allowing information to propagate among related concepts before classification. The resulting framework preserves the interpretable bottleneck of CBMs while enriching it with relational reasoning. Learning such latent topologics- often referred to as graph structure learning has proven beneficial across domains such as time series forecasting (Shang et al. 2020). physical system modeling (Kipf et al. 2018), and computer vision (azi et al. 2022), even when explicit graph structures already exist (Franceschi et al. 2019). Similarly, in human perception, interactions among fine-grained visual concepts are known to enhance recognition and generalization (Bar) 2004; Oliva & Torralba| 2007 Kourtzi Connor 2011). Building these insights, we extend the CBM framework by modeling concepts nodes and their interactions odges in learned latent graph. This concept graph not only supports more expressive and accurate predictions but also enhances interpretability by revealing how concept relationships influence model decisions. In summary, Graph CBMs integrate relational inductive bias into the bottleneck layer, enabling structured concopt reasoning within an interpretable deep learning paradigm.
3.1 Preliminaries
Concept Bottleneck Models (CBMs): Given a set of images V= = U1, U2...., of concepts t,, .2........ lg, each image 2/ is associated with a label y E y. A CBM consists of two components: a mapping fi from the image space to the concept score space, and mapping fa (linear for most of CBMs) from the concept score space to the label space. The image is first projected into the concept.
.....,,, Un and a predefined set
3

---

Published in Transactions on Machine Learning Research (02/2026)
Module 1: Encoding & Parameterization
Module 2: GNN Message Passing
Module 3: Regularization & Prediction
E
R
2y E R
Zp
E
IR
kg E
Rd
R
2g
Image Encoder
embadding contrast
2/ E Re loxd
Projection
E R
Text Encoder
1000
activation contrast
zy
size abnomality
Concept Graph
O
E TR
R
R
Figure 1: Overview of Graph CBM. ! Extract embeddings via pretrained encoders and initialize the graph structure. @ Update the concept embeddings and concept activations through message passing. @
Use
different gramularity contrastive losses to qualify the latent graph and make final prodictions.
space via = fi(;;; E R producing concept scores. These scores are used to predict the final label: 3. - f2(c) Each element of (1 S k) reflects the relevance between image e and concept 4
Our Setting: Given an image i and a concept set T, our goal to identify a subset of activated concepts that both describe and help classify the image effectively. Following the label-froc CBM sctting (Oikarinen et al.) (023). we assume access to pretrained multimodal encoders: Eg V + Rd and E, T R which embed images and concepts into shared latent space. We denote the concept embedding matrix ExTT) E #xd For each image vi the initial concept score vector is computed via the matrix-vector multiplication:
2r E R where 2r, = (v;). If ground-truth concept annotations are available, concept scores can alternatively be obtained via a learnable MLP: c; = ML.P(z..).
We define the latent concept graph as G AV, where the node set V corresponds to the concepts, and the adjacency malrix A encodes structural relations among them. In Figure [ we show our procedures for learning the latent concept and will explain it in more detail the following sections.
3.2 Module 1: Encoding and Parameterization
Module 1 prepares the inputs for Graph-CBM model and consists of two components: (1) encoding images and concepts using pretrained encoders, and (2) parameterizing the initial latent concept graph, which will be refined in later stages.
3.2.1 Encoding Images and Concepts
There are two primary CBM training settings: the label-free and concept-superuised paradigms. In the label-free setting, concept annotations are unavailable, and concept sets must be defined heuristically at the dataset or instance level (Yang et al.] 2023 Oikarinen et al. 2023
In both settings, images passed through pretrained vision encoder to obtain image embeddings. For label-free CBMs, the concept descriptions are also processed through pretrained language encoder to extract their embeddings. Importantly, both encoders remain necessarily frozen during CBM training to avoid converting the pipeline into fully end-to-end model, thereby preserving the interpretability of the concept space.
3.2.2 Parameterizing the Initial Graph
The purpose of the latent concept graph to model the underlying relationships among concepts. Since the concept set is defined at the dataset level, the same latent graph structure E R a k adjacency matrix over concept vetrices, is shared across all data instances. To enable finer control and interpretability, decompose the graph node representation into two components: the semantic embedding set Vanb {2r,))
4

---

Published in Transactions on Machine Learning Research (02/2026)
and the activation vector Vact C.- Thus, the full graph is denoted as G2 (Vemb, Vact, A will be the same across instances, whereas Vanb and Vact will become instance-dependent during training.
Vomb E _exdd captures the intrinsic meaning of each concept node, while Vact E [ reflects the extent to which each concept is activated per instance. The adjacency matrix A encodes initial structural relations between concept nodes. We initialize A either by computing similarities among node embeddings or randomly. Empirically, we find that random initialization achieves comparable performance, so we adopt it for simplicity. During training, A is optimized jointly with other model parameters and receives gradients from the task objective CE as well the regularizers, following exactly the same optimization procedure other neural parameters (e.g., MLP weights).
Although A is shared across instances in a dataset, the distinction brought by Vact will trigger different nodes in the latent concept graph (we consider a concept node being activated the corresponding entry value in Vact positive), resulting input-dependent graph structure in end.
3.3 Module 2: GNN Message Passing
With the latent concept graph, we can derive better representations of images through concopt node embeddings and activations in the graph. To enhance the expressive capacity of structure learning, we adopt a multi-layer approach, where each layer is associated with its own learnable adjacency matrix. Specifically, we define the graph at layer I as G (Vimb " where I denotes the layer index. The final latent concept graph is constructed as the union of edge sets across all layers, i.., U. In practice, use 3 layers for structure learning.
At each layer I, we follow a standard message passing framework (Kipf & Welling 2017 to aggregate neighborhood information and propagate it to the concept nodes. Since concept nodes at layer are characterized by both embeddings (Vel)) and activations (Vilz)) we perform two distinct message passing steps at each layer.
We apply the renormalization trick introduced in KKipf & Welling 2017 to the adjacency matrix: Renormalize(A") = D AD- where A' A' + I, Du E,A4,,, Here, I denotes the identity ma- trix, and is the degree matrix of A'.
The update equations for emboddings and activations at layer are given by:
V Vemb- a (D- (レ
[
volbb)
Vi = (D-
(1)
Here, @ denotes element-wise multiplication. The product Vi-1 @ Vili captures the degree to which cach concept node is activated in the latent concept graph at layer l. o(.) is a non-linear activation function.
3.4 Module 3: Regularization & Prediction
To obtain supervision on the intrinsic structure among concepts rather than on the mapping from concepts to labels, We address this by using contrastive learning, treating each image and its activated concept graph as a positive pair, since datasets do not provide ground-truth relational information between concepts. By imposing self-supervised contrastive objectives, the model is able to learn label-agnostic structures while also providing supervision in label-free settings.
After message passing, we apply mean pooling extract graph-level representation for each input image: Ay. MeennPooling( (Vemb) Rd, where is the number of layers. As previously discussed, the concept graph acts augmentation technique for input images. Under the assumption of intrinsic concept relationships, we introduce a contrastive regularization term based on the normalized temperature-scaled cross-cntropy loss (NT-Xeet) (Chen et al.) 2020). Moreover, relying solely on the subeddinggwise. rontrastive loss yields suboptimal performance (see Tables g and 4 Inspired by Ribeiro et al./ (2017); ; [unn et al. (2019h; You et al. (2020a):: Wang et al.] (2022), we design a second contrastive loss a different granularity to further regularize the updated concept scores &. We treat é, Vm as the positive pair for image e in R (the concept space). We project be into the same space using ML. layer f3 Rdd IR* yielding غ. fa(z,, Φ) as the anchor
5

---

Published in Transactions on Machine Learning Research (02/2026)
point in the concept space.
gaim(Ze)Fuall D/T
Cemb = - llg
Lact = - -lg
(2)
chiml
1=1
j=
where sim(-.) denotes the cosine similarity, and temperature hyperparameter (sct to r = 0.3 in our experiments). In each mini-batchi the positive pair consists of the image embedding 2v, (or 2,,) and its corresponding concept graph embedding Zg. (or tz), while the negative pairs are the graph embeddings of all other images.
Specifically, the embedding- contrastive loss Lemb is computed in the latent representation space R while the activation-level contrastive loss Lact is computed in the concept activation space. R after the mapping fa Rd + R= Thus, the two contrastive losses operate at different layers and regulate the latent graph structure from complementary perspectives. Lemb enforces alignment and uniformity between the image and graph-level embeddings, while Leet encourages concept activation vectors serve as effective hidden reprosentations of the input images (Wang & Isola 2020
In addition, hypothesize that the performance degradation observed when using CBMs is partly due to information loss during the projection from the gigh-dimensional image latent space to a lower-dimensional concopt space. This is supported by the fact that directly classifying the image latent features (with only a single trainable linear prediction head) achieves 81.19% accuracy on CUB, whereas the intervening concept bottleneck reduces the usable signal. Consequently, making the concept space more faithful to or better aligned with the geometry of the image feature space can benefit downstream classification. This motivation underlies the second contrastive loss, and our empirical results in Section corroborate this intuition.
Combining both with a prediction loss and sparsity regularization term, the final loss becomes:
L = CE v.) + a(Cems Lact) + ((()).
(3)
where CE(-,) ) is the cross-entropy loss, (1) denotes L1 regularization, and a, 8 are weighting hyperparameters. The prediction Os is computed by passing & through another MTP f2 IR. - )) i.e., Di = (2,))
The above formulation is primarily designed for the label-free setting. When concept supervision is available, the model directly learns concept representations from ground-truth annotations. In such cases, we no longer require a language encoder to obtain concept embeddings, and the contrastive loss reduces to Lact, alongside a sparsity regularization term. The final loss function in the concopt-supervised sctting becomes:
C = CEE), mi) + BCE(2, (4) + aCnnt + +-B (AA),
(4)
where BCE(.,)) is the binary cross-entropy loss. Since concept annotations provide structural cues, the BCE loss guides the latent graph to align with these ground-truth relationships. Simultaneously, Lact regularizes the graph by treating é as a meaningful latent representation of image F,. Together, they enable the model to learn high-quality latent graph structure.
Base experiments
CUB
Flower102
HAM10000
Cifse-10
Cifar:10
LF-CBM
73.90% 28).)
81.77% (+0.59%)
66.76% (+0.43%)
86. 40%
(1%0.1%%)
65.16% (#0..11%)
Graph-(LF-CBM)
75.59% (=0.18%)
86.00% (t..98%)
87.47% (1.0.61%)
86.54%
(+0.15%)
65.96% (#00 (=90.1656)
PCBM
74.69 % (1:0.16%)
79.01% % (11.1
77.61% (=0.60%)
95.71% (+00.0 (40.07%)
80.02% (+0.39%)
Graph-PCBM
77.95% (=0.69%)
89.25% (#0.69%)
78.50% (+00.5%%)
95.95% (+..09%%)
80.86% (#00.26%)
Table 1: Graph CBM can better capture image information. We report the mean and standard deviation from 10 random runs.

---

Published in Transactions on Machine Learning Research (02/2026)
4 Experiments & Results
4.1 Setup
Datasets: For evaluating Graph CBMs, we choose various real-world datasets ranging from common objects
dermoscopic images. We introduce 1) common objects: CUB (Wah et al. 2011). Flower102 (Nilsback &
Zisserman 2008), AwA2 (Xian et al. 2018), and CIFAR (Kizhevsky et al. 2009) and 2) medicial domains:
HAM10000 [schandl 2018) and ChestXpert (Irvin et al. 2019 Dataset details can be found in appendix A
Since many datasets do not contain concept annotations, we will use LLMs to generate and filter the concept set for each dalaset. In CUB and CIFAR datasets, we use the same concept sets provided by (Oikarinen) jet al. 2023). Furthermore, we found that concepts can be redundant and noisy in CIFAR datasets; thus, we apply the submodular filtoring algorithm (Yang et al. 2023) to reduce the number of concepts in (G-)PCBM experiments. For Flower and HAM10000, we use the concept candidates in (Yang al. 2023).
Baseline: We choose the oaate-of-theart CBM models that do not need concept annotations during training, i.., LF-CBM (Oikarinen et al. 2023) and Post-hoc CBMs (PCBM) YYkksckgonul et al. 2023) For multi- modality encoders, we choose the standard CLIP (Radford et al. 2021 For a fair comparison with Label-free CBMs, we use CLIP(ViT-B/16) and CLIP(RN50) as the image encoder and the backhone (ResNet-18 trained on CUB from imgelsmob as the backbone for CUB). In the concept-supervieed setting, standard CBMs (Koh ct al. 2020) and CEMs (Espinosa Zarlenga al.) 2022 serve as our baseline, and we still use CLIP(ViT-B/ID to extract features CUB (112 concepts) and Aw (85 concepts) dataset. For ChestXpert (11 concepts), use Bio ViL (Bannur et al. 2023) as the image encoder, BioViL is pre-trained on large X-ray datasets. We use the Adam optimizer and cosine scheduler during training. More configuration details and efficiency concerns are offered in Appendix
4.2 Latent Graphs Enhance Model Performance
Graph-LF-CBM and Graph-PCBM can substantially outperform their counterparts without graphs under label-free settings. Observing the solid enhancement in Table [ of having latent concept graph across different datasets, we can conclude the effectiveness of our proposed mothod. To better undersland the benefit of having a latent concept graph, we draw T-SNE plots for os (2:) in Figure 6 Appendix E showing that models with latent concept graphs can better cluster concept activations to corresponding label groups. In Table I0 Appendix G our method surpasses the baseline on large-scale datasets, which further validates the effectiveness of using latent concept graphs.
Graph CBMs can generalize their improvements to concept-supervised settings. In Table 2 adding the latent graph can increase the label prediction performance for all the datasets from different domains, and match or even surpass the baseline's capability on concept prediction. It is important to note that the latent graph is primarily designed to leverage interactions among concepts to improve label prediction and enable more effective interventions, rather than directly optimize concept-level predictions. Furthermore, our models still demonstrate notable improvements in some cases, particularly, G-CEM not. only boosts label prediction accuracy over standard CEM but also narrows the gap in concept prediction performance compared to CBM-based approaches.
4.3 Comparison with SOTAs
In Figure compare proposed method with state-of-the-art models to demonstrate the effectiveness of constructing latent concept graph. We evaluate our method under both label-free and concept-superrised settings using the CUB dalaset, a common benchmark for concept-based modeling. For the label-free setting, we include additional strong baselines such as BotCL (Wang et al.) 2023). CDM (Panousis et al.) 2002),, LaBo (Yang et al.) 2023), and Res-CBM (Shang et al.] (004).. For the concept-supervised setting, we compare against ProbCBM (Kim al. 2023), HardAR (Havasi et al. 2022), E-CBM (Xu al.) (2024.. and S-CBM /Vandenhirtz et al. 2024h.
7

---

Published in Transactions on Machine Learning Research (02/2026)
CUR
Aw
Chees Upert
Method
Label
Concept
Label
Concept
Concupt
CRM
78.46%
(#0.32%)
70.40% (=0.09%)
95.24% (#0.07%)
97.48% (#0.005)) (50.005%)
66.4055
(#:0.86%)
83.41% (=0.444%)
Grapb CDM
80.03%
(.10. 16%)
68.33% (.10. 21%)
95.34% (110 08%))
97. 07.05% (1100. (.10.07%)
66.827 (06.82%
(L0.44%%
83.20% (11.02%)
CEM
80.86% (2%).12%)
61.34% (1%0.21%)
95.21% (4:0. 480.)
98.16% (4:0.30%)
68%.
(1.0. (...3)
17.90%
(..0.43%)
Graph-CEM
81.11% (#00. 2616)
61.68% (=0022%%)
96.449% (tb0 4.5%)
98.62% (+0.11%%)
66.99% (+:0. 62%5)
78.27%
(=0.139)
Table 2: Comparison between label prediction and concept prediction. We report the average accuracy (for label prediction) and roc-auc (for concept prediction) from 10 random seed experiments.
Under the label-free setting, LaBo and Res- CBs perform well but rely on large concept sets (e.g., 10,000 for CUB). In contrast, our G-PCBM achieves higher accuracy using only 200 concepts, fewer training epochs, and a comparable backbone (c.g., ViT-L/14 improves G-PCBM from 77.1% 82.3%). For other reference, Yaan et. al., 2023) reports 63.9% with the same setup. This demonstrates the efficiency and effectiveness of leveraging latent graphs in CBMs.
In the concept-supervised setting, our models, G-CBM and G-CEM, perform competitively with the best current methods. HardAR and E-CBM both aim to model latent concept correlat ions but follow fundamentally different approaches. HardAR focuses on autoregressively generating concept scores, assuming sequential dependencies. Our method, by contrast, employs graph-based modeling, which is inherently permutation- invariant Maron et al. (2019) Bronstein al. (2021) | and better suited to capturing rich, non-linear concept interactions. Furthermore,
latent graphs can seamlessly plug into HardAR's framework to further enhance its relational reasoning through message passing.
Compared to E-CBM, which encodes concept dependencies implicitly via energy-based modeling, our approach offers explicit graph structures that are casier to visualize and interpret. This transparency is especially valuable for downstream tasks such as intervention (see Section 4.4 The learned graph not only improves model performance during training but also enables more effective post-hoc analysis and decision manipulation.
Importantly, our method is orthogonal to existing CBM designs and can be flexibly integrated into variety of architectures. To support this claim, we present case study with CDM (Panousis al.) 2023 in Appendix H. showing how latent, graphs can complement other CBM variants. Moreover, unlike models such as HardAR, E-CBM, or S-CBM that require datasets with explicit concept annotations, our latent graph framework is versatile and can be applicd in both concept-supervised and label-frec CBM settings, broadening its applicability across domains.
Labil-Frea
B. Concsp--S.periiee
82,300
81,000
80.000
82.
B1,400
80,900
81,200
80.0
78,500
77.5
77.5
75,600
74,200
73 800
73,500
72.5
725
70.0
87.5
67.6
BatCl
PCBM
LF-CBM
CDM
LaBn
Re--CBMG-LF-CBB G-PCBM
VIT-L/14
CBM
PrecBBM HardAR
CEM
E-CBM
S-CBM
G-CBM
G-CEM
Figure 2: We compare our models with current SOTA results for corresponding training and datasct settings, i.e., label-free and coppepe-s We report label accuracy in both subfigures.

---

Published in Transactions on Machine Learning Research (02/2026)
100
Labol-froo Intervantion
100
Concopt-Superviis Intervonton:
CBM
Acturaty (50)
E-OBM
80
G-CBM
65
16.0
Accumity (8)
90
0.1
0.3
0.4
0.5
Concops-Su Suppriiiid Intervention: Ches
100
CUB (G-PCBM)
CUB (PCBM)
Acauracy (7)
60-
G-CDM
CBM
76,97
87,44
92.04
80
Flowe1022 (G- (C-CCMM
Flawr102 (PCBM)
60
71,52
73,08
HAM10000 (PCBM)
HAM1 0000 (G-PCBM)
Ratio
Ratoo
Figure 3: This figure shows the effectiveness using latent graphs when intervening concepts. Subfigure A compares the after-intervention performance between PCBM and G-PCBM across three datasets under the fabel-free setting: subfigures B and select concept-supervised datasets. Full ratio intervention comparisons between G-CBM and F-CBM are in appendix
4.4 Intervention Dynamics
A core advantage of CBMs lies in their intervenability: allowing users to adjust concept activations to correct false predictions and improve trust. Beyond enhancing label prediction post-training, we evaluate whether latent graphs further strengthen CBMs during concept interventions. Notably, the benefit of incorporating latent graphs becomes more pronounced in this setting, as relational cues help guide more effective interventions. Following the UCP policy (Shin et al., 2023), we select relevant concepts and apply simple Lazy Intervention strategy (Appendix D to modify activation values.
In both label-free and concopt-supervised settings, incorporating a latent graph improves intervention performance. As defined in Equation [ the message passing mechanism allows changes in the intervened activation vector c, to propagate to neighboring concepts via the latent graph, amplifying the effect to calculate the final ē, Thus, models with graphs implicitly intervene more broadly, leading more robust corrections, shown in Figure A.
In the concept-supervised setting (Figure 3 B & C), we compare performance on the CUB and ChestXpert datasots. Unlike E-CBM (Xu et al. 2024], which requires iterative energy calculations and gradients for intervention, G-CBM achieves strong results via a single forward pass. This efficiency, combined with high accuracy, highlights the benefit of explicitly modeling concept relations. We hypothesize that concept supervision encourages meaningful correlations, which the latent graph further captures and utilizes during intervention. The improved performance across datasets supports the positive impact of latent structure in enhancing interpretability and trust.
4.5
How Contrastive Terms Affect Model Performance
Using both Lemb and Last is crucial for Graph LF-CBM. Table 3 examines different combinations of contrastive losses on the CUB dataset. We find that using only Lemb marginally improves performance, while incorporating fact leads to further gains. However, on HAM10000 and CIFAR-100, applying either loss alone fails to outperform the LF-CBM bascline. Notably, using both Lemb and Lact consistently achieves the best accuracy across tasks. When only Lemb is used, the learned graph becomes overly dense due to insensitivity to the 4 regularization edge weights. Conversely, relying solely on Last tends to yield overly sparse graphs. Thus, combining both objectives enables effective control of graph complexity and improves representation quality.
With target supervision, contrastive regularization further enhances the expressivity of Graph PCBM. Contrastive learning provides a self-supervised signal by treating concepts as alternative views of the same input. We investigate whether contrastive regularization still benefits Graph PCBM under target supervision. As shown in Table applying both Lemb and Lact improves performance across CUB,
9

---

Published in Transactions on Machine Learning Research (02/2026)
Regularizer
Existence
Lemb
Lact
Dataset
Performance
CUB
73.90%
(+0065%)
75.38% (=0.12%)
74.11% (+0.17%)
75.59%
(+0.18%)
HAM10000
66.76%
(+0.411%)
62.87% (=1.3%%)
65.10% (+0.47%)
67.47%
(+0.61%)
Flower102
84.77%
(+0.44%)
85.99%
(+0.68%)
76.78%
(+0.49%)
86.00%
(+0.80%)
CIFAR10
80.76%
(+%.
72.42%
(+1 (11))
84.08% (+0.22%)
84.65%
(+0.10%)
CIFAR100
65.16% (+00.14%) 59.49%
(=0.34%)
62.97% (+0.21%)
65.96% (=0.17%)
Table 3: Multi-level contrastive learning is crucial for self-supervised concept graph qualities. If we train Graph LF-CBMs with one excluding contrast loss, the model can fail to yield good latent structure; while considcring both contrast losses, models can gain more benefits.
Method (Graph PCBM)
CUB
HAM10000
CIFAR-100
w/o (Cemb Lact)
76.89% (+0.65%)
77.66% (+0.41%)
75.67%
(+0.35%)
(Lemb + Loct)
77.95% (1.069%)
78.50% (+00.5%%
80.86% (+0.26%)
Table 4: Adding contrast loss to target supervision can help model performance. Having the two contrast. regularizers will lead to higher accuracy for label prediction.
HAM10000, and CIFAR. On CIFAR-100 particular, we observe a performance gain of over 5%. Without contrast losses, the model relies entirely on target supervision and & regularization, making the learned graph highly sensitive task-specific supervision signals and regularization strength. Contrast regularization thus provides stable, auxiliary training signal that improves generalization and structure learning.
To further investigate the underlying source of effec- liveness in the proposed methodology, additional ablation study on the CUB dataset under the label-froe setting. The goal of this experiment is to disentangle whether the observed improvement is primarily driven by (i) the use of contrastive learning or (ii) the introduction of the GNN massagg-passing module. Specifically,
conduct
Contrastive Impact
PCBM Graph-POBM
70
compare: (1) the effect of adding Lemb and/or /or Lact to PCBM, and (2) the same set of losses applied to Graph-PCBM. The results are presented in Figure 4
17.4%
From this comparison, we observe that the dominant.
Less
Figure 4: Contrastive learning shows positive impact mainly when the idea of graph is implemented.
source of improvement arises from enabling message passing over concepts. Classical CBMs treat concepts independent units, whereas Graph-CBM allows
concepts to interact through learned graph. We argue that this aligns better with the semantic structure of many domains. For example, in fine-grained recognition tasks, certain concepts reinforce or suppress others, and modeling such dependencies improves both predictive performance and intervention.
To summarize the analyses this section, we emphasize that model capacity is not the main driver of improvement. The GNN introduces only O(k2) additional parameters, does not fine-tune or alter the image encoder, and the learned adjacency matrices sparse. Therefore, the performance gain cannot. be
10

---

Published in Transactions on Machine Learning Research (02/2026)
attributed to increased representational capacity. Instead, the benefit appears to stem primarily from adopting GNN-based message passing rather than from contrastive learning alone. This reinforces our hypothesis that modeling intrinsic concept relations via a latent graph is beneficial.
4.6
How to Interpret The Concept Graph
(122) a raucous call - (295) loud, rasping voice
224 12
(133) red spot on beak (208) whitish throat
(151) small mouth (299) olive-green back
355
239
(152) small songbird - (164) a small, green body
(241) black wings with white spots (337) white and black coloration
(245) blue wings - (300) orange and black wings
(250) brownish wings with white bars (347) white stripes the wings
(253) dark brown stripes on the back (295) loud, rasping voice
(260) gray upperparts - (263) grayish-white underparts
(260) gray upperparts (349) white underparts with streaks
G-CEM
Cardiac and Mediastinal Abnormalities
Cardiac and Mediastinal Abnormaiities
Lung Parenchymal Diseases
Lung Parenchymal Diseases
Pleural and Lung Structural Issues
Puural Lung Structural Issues
9
10
Figure 5: Top: Illustrating part of the learned latent graph under the label-free setting on CUB, alongside representative connected concept pairs. Bottom: G-CBM and G-CEM capture similar and trustworthy concept interactions for the ChestXpert dataset.
Latent graph can perform comparably to recover ground-truth concept graph in both settings. We use ChatGPT (Ouyang et al. 2022 to generate 50 highly correlated concept pairs to build 'real-world' concept graph for label-free setting. Substituting the learned graph with this LLM--deived structure yields a CUB accuracy of 77.69%, while our learnable graph achieves 77.14%, demonstrating similar efficacy. Interestingly, LLMs excel at capturing surface-level similarities (c.g., color or body parts), while our method also discovers non-obvious correlations (e.g. "a loud, harsh cry" "a raucous call"). Tn Figure we also obscrve that concepts sharing the same label tend to be connected, regardless of architecture, validating that learned graphs reflect semantically meaningful groupings. In Appendix [ we can further extract salient subgraphs with fewer edges that maintain (or even improve) label accuracy and interpretability. Figures 10 & 11 show that key concept relations are retained, reinforcing the utility of our latent graph formulation. Interestingly, latent graphs behave distinctly to different training regimes. As shown in Figure 13 label-free models learn sparser graphs, while concept-supervised ones favor denser connections. More analyses are in Appendix K
Latent graphs improve robustness under concept attacks. We conduct an experiment on the CUB dataset in Appendix F where we randomly mask perturb a fixed number of concepts in three different.
11

---

Published in Transactions on Machine Learning Research (02/2026)
groups: (1) connected concepts, (2) isolated concepts, and (3) random selections from the entire concept set. As shown in Figure corrupting connected concepts results in significantly less performance degradation compared to isolated or randomly selected ones. This suggests that the latent graph enables the model to recover corrupted concept information by aggregating signals from clean neighboring nodes, thereby enhancing resilience against noise or missing data.
5 Limitations & Conclusions
Graph CBMs currently capture only latent concept interactions, without modeling hierarchies or complex relationships. Incorporating external knowledge about concept hierarchy could enhance interpretability, and future work will explore hierarchical structures beyond pairwise interactions.
In this paper, we introduced Graph CBMs, a simple yet effective framework that incorporates graph structures to model concept correlations and enhance interpretability Our method orthogonal to existing CBMs and can be seamlessly integrated to boost performance while supporting targeted concept interventions. Notably, the learned latent graphs can match the effectiveness of ground-truth concept graphs. Future directions include leveraging external knowledge graphs for graph distribution modeling and extending latent graph techniques to interpret individual neurons within CBMs (Oikarinen & Weeg 2024).
References
Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Percz-Garcia, Maximilian Isse, Daniel C Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, et al. Learning to exploit temporal structure for biomedical vision-language processing. In Proceedings of the BEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15016 15016-15027, 2023.
Moshe Bar. Visual objects in context. Nature Reviews Neuroscience, 5(8):617 629, 2004.
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint orXi::2104.13478 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neurol information processing systems, 33:1877-1900 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive Icarning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Giovanni De Felice, Arianna Casanova Flores, Francesco De Santis, Silvia Santini, Johannes Schneider, Pietro Barbiero, and Alberto Termine. Causally reliable concept bottleneck models. anXiv preprint arXiv:2503.0453 2025.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE Conference on Compuler Vision and Pallern Recognilion, pp. 248 255, 2009. doi: 10.1109 10.1109/CVPR.2D0 CCt.CC/.........22220
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (cds.), Proceedings of the 2019 Conference of the North American Chapler of the Association for Compulational Linguisties: Human Tanguage Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653 / 11/N1991111 1423. URL
Xiaowen Dong, Dorina Thanou, Michael Rabbat, and Pascal Frossard. Learning graphs from data: A signal representation perspective. IEF. Signal Processing Magazine, 36(3):44 63, 2019.
12

---

Published in Transactions on Machine Learning Research (02/2026)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https:://peereviiex htt.:/owww....eeoooooooooaoooooo
Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelan- gelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, Pietro Lio, and Mateja Jamnik. Concept embedding models: Beyond the accuracy-explainabliiy trade-off. Advances in Newral Information Processing Systems, 35, 2022.
Luca Franceschi, Mathias Nicpert, Massimiliano Pontil, and Xiao Hc. Learning discrete structures for graph neural networks. In International conference on machine learning, pp. 1972-1982. PMLR, 2019.
Marton Havasi, Sonali Parbhoo, and Finale Doshi-Veler Addressing leakage in concept bottleneck models. Advances in Neural Information Processing Systems, 35:23356-2333 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770 778, 2016.
Lijie Hu, Chenyang Ren, Zhengyu Hu, Hongbin Lin, Cheng-Long Wang, Zhen Tan, Weimin Lyu, Jingfeng Zhang, Hui Xiong, and Di Wang. Editable concept bottleneck models. In Forty-second International Conference on Machine Learning, 2025. URL https::/openrreviee http://///t/////ggoooooeeeeee//a http:///ww......oommmmmoooooooon
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-lluus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 590 597, 2019.
Ances Kazi, Luca Cosmo, Seyed-Ahmad Ahmadi, Nassir Navab, and Michael M Bronstein. Differentiable graph module (dgm) for graph convolutional networks. IEFF Transactions on Pallern Analysis and Machine Intelligence, 45(2):1606-1617, 2022.
Eunji Kim, Dahuin Jung, Sangha Park, Siwon Kim, and Sungroh Yoon. Probabilistic concept bottleneck models. International conference on machine learning, 2023.
Thomas Kipf, Ethan Fetaya, Kuan-Chich Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International conference machine learning, pp.
PMLR, 2018.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations, 2017. URL SJU
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, pp. 5338-5348. PMLR, 2020.
Zoe Kourtzi and Charles E Connor. Neural representations for object perception: structure, category, and adaptive coding. Annual review of neuroscience, 34(1):45-67, 2011.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alejandro López-Cifuentes, Marcos Escudero- Veruderr-Viñolo, Jesús Bescós, and Álvaro Martfa-Martín, Semantic-aware scene recognition. Pattern Recognition, pp. 107256, 2020.
Tengfei Ma, Trong Nghia Hoang, and Jie Chen. Federated learning of models pre-trained on different features with consensus graphs. In Uncerlainty in Artificial Intelligence, pp. 1336 1346. PMLR, 2023.
TorchVision maintainers and contributors. Torchvision: Pytorch's computer vision library. https://github con/pyytocch/visionn 2016.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. In International Conference on Learning Regresentalions, 2019. URL htttps://openneeeeee.eee////// id=Syx72j09tm
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
Tuomas Oikarinen and Tsui-Wei Weng. Linear explanations for individual neurons. In Proceedings of the 41st International Conference on Machine Learning, pp. 38639 38662, 2024.
Tuomas Oikarinen, Subhro Das, Lam M. Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. In The Eleventh International Conference on Learning Representations, 2023. URL https; : //openreview geppenreview.net/ononuee .mtt://orumm7dd11147777eeee
Aude Oliva and Antonio Torralba. The role of context in object recognition. Trends in cognitive sciences, 11 (12):520 52,55777772220 2007.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow inst.uuctinss with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 27730-27744 Curran Associates, Inc., 2022. URL https://pproceeenngsss www.:gg.....rrrrr..... www.:uuw.....pppppereeeeeeeceeee files/ pdf
Konstantinos P. Panousis, Dino Ienco, and Diego Marcos. Sparse lincar concept discovery models. In 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pp. 2759 2763, 2023. doi: 10.1100/CCCVW6003832333.. 00029..
Katharina Prasse, Patrick Knab, Sascha Marton, Christian Bartelt, and Margret Keuper. DCBM: Data- efficient visual concept bottleneck models. In International Conference on Machine Learning, 2025. URL
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Πly Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8748 8763. PMLR, 18 24 Jul 2021. URL https://proceedaggs.oo
press/v139/zatford211. html
Colin RAffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 (140)):1-66, 2020.
Leonardo FR Ribeiro, Pedro HP Saverese, and Daniel R Figueiredo. struc2vec: Learning node representations from structural identity. In Proceedings of the 23rd ACM SIGKDD international conference knowledge discovery and dala minnng, pp. 385 394, 2017.
Yu Rong, Tingyang Xu, Junzhou Huang, Wenbing Huang, Hong Cheng, Yao Ma, Yiqi Wang, Tyler Derr, Lingfei Wu, and Tengfei Ma. Deep graph learning: Foundations, advances and applications. In Proceedings of the 26th ACM SIGKDD international conference knowledge discovery & data mining, pp. 3555 3556, 2020.
Chao Shang, Jie Chen, and Jinbo Bi. Discrete graph structure learning for forecasting multiple time series. In International Conference on Learning Representations, 2020.
Chenming Shang, Shiji Zhou, Hengyuan Zhang, Xinzhe Ni, Yujiu Yang, and Yuwang Wang. Incremental residual concept bottleneck models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation (CVPR), pp- 11030-11040, June 2024.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Sungbin Shin, Yohan Jo, Sungsoo Ahn, and Namhoon Lec. A closer look at the intervention procedure of concept bottleneck models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 31504-31520. PMLR, 23-29 Jul 2023. URL https://proceeddnggs wwiiggsmmmr...mm www..p.res//v0000 20i003asinn3a.....m html
Fan- Yun Sun, Jordan Hoffman, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In International Conference on Learning Representations, 2019.
Nya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mip architecture for vision. Advances in neural information processing systems; 14:24261-24277 2021.
Philipp Tschandl. The HAM10000 dataset, large collection of multi-source dermatoscopic images of common pigmented skin lesions, 2018. URL http:s///00ll.rrr10001110/0////1
Moritz Vandenhirlz, Sonia Laguna, Ričards Marcinkevids, and Julia E Vogt. Stochastic concept bottleneck models. In eightt Conference Newral Information Processing Systems, 2024. URL https //openreview .wtt:/oorurl.d--11t1555555
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongic. The caltoch-uesd birds-200-2011 datasct. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.
Bowen Wang, Liangzhi Li, Yuta Nakashima, and Hajime Nagahara. Learning bottleneck concepts in image classification. In Proceedings of the seee/cof conference on computer vision and patlern recognition, pp- 10962 10971, 2023.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International conference on machine learning, pp. 9929 9939. PMLR, 2020.
Ziyang Wang, Huoyu Liu, Wei Wei, Yue Hu, Xian-Ling Mao, Shaojian He, Rui Fang, and Dangyang Chen. Multi-level contrastive learning framework for sequential recommendation. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM '22, pp. 2098 2107, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450392365, doi: (0.1145/3511808.355744 http://di.oorr///// URL https://doi.coo org/10 10 11155//511808835574444
Tianxin Wei, Yuning You, Tianlong Chen, Yang Shen, Jingrui He, and Zhangyang Wang. Augmentations in hypergraph contrastive learning: Fabricated and generative. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 1909-1922. Curran Associates, Inc., 2022. URL http:://pproceeeaggsso www.p:ss...iiip.....cpp..rpeee files/ http:////////oecceeaaaaaaaaaaaa/
pdf
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. IEFE transactions on pullern analysis and machine intelligence, 41(9):2251 2265, 2018.
Xinyue Xu, Yi Qin, Lu Mi, Hao Wang, and Xiaomeng Li. Energy-based concept bottleneck models: Unifying prediction, concept intervention, and probabilistic interpretations. In The Tweffth International Conference on Learning Representations, 2024. URL https:://operrevierr http:///pppaaooooee.eeeeeerrraap
An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue Hc, Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. In Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), pp. 8090-3100, October 2023.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Yue Yang, Artemis Panagoponlou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in bottle: Language model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 19187-19199
2023.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Bal- can, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 5812 5823. Curran Associates, Inc., 2020a. URL S1220000ee12223110004441444aaa43 Paperrr--
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in neural information processing systems, 33:5812-588 -8232, 2020b.
Yue Yu, Jie Chen, Tian Gao, and Mo Yu. DAG-GNN: DAG structure learning with graph neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedingss of Machine Learning Research, pp. 7154 7163. PMLR, 09-15 Jun 2019. URL https::/pproceddingg mlr..mm..... htt..pprees/v77yy1l...hht html
Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. In The Eleventh International Conference Learning Representations, 2023. URL sttp:::/oppnreeviiswnn///ooo nA5AZ8CEyow
16

---

Published in Transactions on Machine Learning Research (02/2026)
Appendix
Table of Contents
Dataset Introduction
17
B Configuration and Running Environments
18
C Data Processing for ChestXpert
19
D Lazy Intervention
19
E Having a Latent Concept Graph Enriches concept activations Expressivity
20
Latent Graphs Offer Robustness under Label-Free Settings
21
Performance on other larger-scale datasets
H Latent Graphs Benefit Various CBM Backbones
Salient Subgraph for Concept-Supervised Settings
22
G-CBM Intervention at full ratio
24
K Graph Complexity
24
Visualization of Learnable Graph
25
A Dataset Introduction
Caltech-UCSD 200s=-000-001,, CUB (Wah et al. 2011 D: CUB is the most widely-used dataset for fino-grained visual categorization tasks. It contains 11,788 images of 200 subcategorics belonging birds, we follow the samne data processing done in the Label-free-CBM (Oikarinen al., 2023) setting to select 5,990 images for the training set and 5,790 images for the validation set. For the concept supervised setting, process the same way (Koh et al. 2020) and (Espinosa Zarlenga ct al. 2022). selecting 112 attributes as the concepts and use the same data splits.
Oxford 102 Flower (Nilsback & Zisserman, 2008 is an image classification dataset comprising 102 flower categories. The flowers chosen to be flowers commonly occur in the United Kingdom. Each class consists of between 40 and 258 images. We follow the torchvision dataset setting (maintainers
& contributors 2016) and only use the training set for training and the validation set for testing.
CIFAR-10 & CIFAR-100 (Krizhevsky et al. 2009 The CIFAR-10/100 dataset (Canadian Institute for Advanced Research, 10 classes) is subset of the Tiny Images dataset and consists of 60,000 32x32 color images. CIFAR-10 labels images with one of 10 mutually exclusive classes, and CIFAR-100 has 100 classes grouped into 20 superclasses. There are 6000 images per class with 5000 training and 1000 testing images per class in CIFAR-10, while CIFAR-100 divides the dataset into 500 training images and 100 testing images per class.
HAM10000 [Tschand] 2018): A dataset of 10000 training images for detecting pigmented skin lesions. contains 7 labels and a representative collection of all important diagnostic categories in pigmentod lesions HAM10000 provides 10,000 training images and 1,511 1 testing images.
17

---

Published in Transactions on Machine Learning Research (02/2026)
AwA2 (Xian et al. 2018 is zero-shot learning dataset containing 37,322 images and 50 animal classes. Unlike the CUB dataset (Wah et al. 2011) in which concepts are defined at the instance level, images under the same label inside AwA2 (Xian et al. 2018) will share the same concepts. We use all 85 attributes as concepts.
ChestXpert (Irvin et al. 2019 is large dataset of chest X-rays and competition for automated chest X-ray interpretation, consisting of 224,316 chest radiographs of 65,240 palients, which features 14 uncertainty observations and radiologist-labeled reference standard evaluation sets. ChestXpert (Irvin et al. 2019) does not provide binary label classification, so we cluster 11 out of 14 observations into categories. The detailed data processing can be found in appendix
The number of concepts for datasets is as follows: 370 for CUB, 108 for Flower, 48 for HAM10000, 30 for CIFAR-10, and 50 for SIFAR-100. For (G-)LF-CBM experiments, we use the original number concepts without any filtering (143 for CIFAR-10 & 892 for CIFAR-100).
B Configuration and Running Environments
dataset
training epochs
Learning Rate
CUB
100
1c-3
0.1
0.2
F0ower102
500
1e-3
0.1
0.05
HAM10000
100
1e-3
0.1
0.05
CIFAR-10
100
1c-3
0.1
0.1
CIFAR-100
100
1e-3
0.1
0.05
Table 5: Training Configuration for Graph LF-CBM
dataset
training epochs
Learning Rate
CUB
100
1e-3
0.1
0.2
Flower102
100
1e-3
0.1
0.01
HAM10000
100
1c-3
0.1
0.05
CIFAR-10
50
1e-3
0.1
0.05
CIFAR-100
30
1e-3
0.1 0.05
Table 6: Training Configuration for Graph PCBM
dataset
training epochs
Learning Rate
CUB 100
le-3
0.05
0.0
AwA2 50
le-3
0.01
0.0
Chest Xpert
50
le-3
0.03
0.0
Table 7: Training Configuration for Graph CBM
dataset
training epochs
Learning Rate
a
B
CUB
100
le-3
0.07
0.0
AwA2
50
1c-3
0.07
0.0
ChestXpert
50
1e-3
0.07
0.0
Table 8: Training Configuration for Graph CEM
We run all the experiments on a single GPU (NVIDIA A100). The GPU memory for Graph LF-CBMs and Graph PCBMs less than 10GB with a batch size of 512 for all datasets. The full training run takes from minutes 2.5 hours depending on the dataset size and the number of training epochs. In practice, Graph LF-CBMs trained on CIFAR-100 take 2.5 hours, while Graph PCBMs trained Flower need less than 3
18

---

Published in Transactions on Machine Learning Research (02/2026)
minutes for execution. Graph CBMs and Graph CEMs average take 8~-1 mins to finish training the CUB and ChestXpert datasets, while they spend about 3~5 mins on the AWA2 dataset. We also offer time measurements in Table g since the latent graph introduces more computational units, the model will be unavoidable to spend more time on training.
Model PCBM
Training
Inference
5.7it/s
2.09it/s
G-PCBM
9.43it/s
2.12it/ss
CBM
6. 6.12it/s
2.90it/s
G-CBM
7.89it/s
2.57it/s
Table Time Measurement between our proposals and baselines
C Data Processing for ChestXpert
CheXpert (Irvin al. 2019) uses "No Finding" to indicate the abnormality of patients' chest radiographs, and it is highly unbalanced. We will then cluster CheXpert's concepts into different labels, and will use the frontal and lateral X-ray images for cach patient. The concepts and labels are classified in this way:
Group 1: Cardiac and Mediastinal Abnormalities:
Enlarged Cardiomediastinum
Cardiomegaly
Edema (related to heart conditions)
Group 2: Lung Parenchymal Diseases:
Lung Opacity
Lung Lesion
Consolidation
Pneumonia
Group 3: Pleural and Lung Structural Issues:
Atelectasis (collapse of lung lissue)
Pneumothorax (air in pleural space)
Pleural Effusion
Pleural Other
If one patient meets multiple abnormal conditions (multi-labeled), will select the most significant abnormality. We choose not to include the Fracture observation, as it forms cluster itself and shares no commonalities with other observations.
D Lazy Intervention
Lazy Intervention: We define two sets of concept scores
R = {c: M&)) = y:), W (G h(()) # ydh.
where R and W are sets of concept scores making right and wrong predictions. We can further partition them using class labels, so R = R', and so does W. We then define the difference set,
D (mean(R') mean(W") 1 m).
19

---

Published in Transactions on Machine Learning Research (02/2026)
is the number of classes. Each d' D can be viewed prototype of the intervention vector for j-th class. The intervention procedure will be
Intervention fa + d' Ve, W", 1
1 S S.
When the dataset contains concept annotations, the R records positively classified concepts, and the W records falsely classified concepts. At the intervention step, Lazy intervention will only intervene on falsely classified concepts.
E
Having a Latent Concept Graph Enriches concept activations Expressivity
t-SNE 2D Visualization for FLOWER102 (G-PCBM)
80
L-SNE Feature
20
-20
40
-60-
1-SNE Feature
t-SNE 2D Visualization for FLOWER102 (PCBM)
80
1:SNE Faature
-80
-60
-40
-20
t:SNE Feature 1
Figure 6: T-SNE visualization of concept activations distribution. Top shows better clustering, while bottom has a more chaotic spread. Nodes with the same color are from the same label category.
We want to understand the essential benefit brought by the latent graph when the model needs to make predictions. In order to offer a visualized explanation and evidence, we choose to draw the T-SNE plots for models trained on the Flower102 dataset. (as this dataset contains sufficient many distinct labels and has a relatively balanced sample distribution). As shown in Figure 6 G-PCBM more likely admits isolated and independent concept score vector clusters, differentiating from PCBM which mixes some label groups spread up. Consequently, G-PCBM succeeds in gaining more expressivity for concept activations and boosting prediction performance substantially.
20

---

Published in Transactions on Machine Learning Research (02/2026)
F
Latent Graphs Offer Robustness under Fabel-Free Settings
Performanca
Attacks
Noly Altacks
Alk on Baseline
75
Allk Ramdom
Allk on Connected
77
Allk lsokated
65
177
80
75
50
Baseline
Random
Allk Connected
Isolated
73 0.1
0.4
40
Masking
Nolky Altack Ratio
Figure 7: Attacking different types of concepts (nodes) in the CUB dataset can result in different scales of performance degradation.
In this section, we further investigate the effect of latent graphs on concepts (nodes), and we design a simple experiment to test such impacts. We first partition concepts into connected concepts (node degree 2 1) and isolated concepts, and we also keep the whole concept set as a baseline group for random attacks; then, we will randomly mask perturb the same number of concepts in these groups separately; lastly, let model predict those corrupted concepts. We choose the CUB the testing dataset for this experiment [
As shown in Figure 7 attacking connected concepts does not harm the model performance as much as attacking isolated concepts or random concepts. We hypothesize that the latent graph structure provides much better robustness towards connected concepts, while the isolated ones cannot benefit from it. When masking or perturbing a concept, there will be information missing for the final prediction layer. Nevertheless, the latent graph structure can aggregate neighborhood information recover the masked or perturbed concept so that the final prediction layer is still able to make rcasonable prediction.
G Performance on other larger-scale datasets
Places365
ImageNet
Method
Acc
Acc
PCBM
55.16% (10..11%)
77. 77.49% (10.10%)
Graph-PCBM
55.25% (10.08%)
0.1
0.17
78.48% (1.0.1%%
0.1 0.17
Table 10: We report the average accuracies from 4 different random seed experiments along with the standard deviation.
In this section, investigate the performance enhancement in prediction and intervention brought by latent graphs large-scale datasets. We choose Places365 [López-Cifuentes et al. 2020), scene recognition dataset composed of 10 million images comprising 434 scene classes, and ImageNet (Deng et al. 2009) which
contains 14, 14,197,122 annotated images according to the WordNet hierarchy. We train G-PCBMs on these two datasots with 10 epochs, 1024 batch size, learning rate at 0.01, and Adam optimizer.
In Table 10 the G-PCBM has latent graph to help capture the intrinsic concept correlation and improve label prediction in both large-scale datasets. We continue present the positive impacts of involving a concept latent graph inside the model on ImageNet a study for intervention. Figure also indicates
We select checkpoints with similar amount of connected concepts and isolated concepts.
21

---

Published in Transactions on Machine Learning Research (02/2026)
Comparison of PCBM and G-PCBM Intervention
95.0
Method
94.61%
94.92%
PCBM
93.89% =
G-PCBM
92.80%
92.5
92.05 %
91.08% -
90.0
89.53%
Accuracy (%)
87.5
86.42%
85.0
82.5
80.0
78.48%
77.49%
77.5
75.0
0.0
0.25
0.5
0.75
1.0
Intervention Ratio
Figure 8: In large-scale dataset, having latent graphs can still promote models to interact with intervention at different ratios.
the effectiveness of latent graphs in intervening large-scale datasets. particular, G-PCBM significantly improves intervention performance in low intervention ratios. This also aligus with results we conclude the other datasets.
Latent Graphs Benefit Various CBM Backbones
In this section, we validate the effectiveness the latent graph on other CBM backbones. We choose CDM Panoousi et al. 2023) as the case study to show the benefits of latent graphs. The most important unit in CDM (Panousis et al. 2023) the concept presence indicator which modeled from a Bernoulli distribution, and the motivation behind the indicator variable is to sparsify the required concepts for label prediction. In Figure Ø applying latent graph to CDM make grcat enhancement in terms of label prediction, and latent graph can help to recover those filtered concept information to further boost model performance. The phenomenon is similar to what we have discussed in F as masking attacks also sparsify concepts, and the latent graph can prevent the degradation caused by such attacks or operations. Along with the performance enhancements in Tables D and 2 learning latent graphs offers a robust generalization ability across different model architectures and training settings, which encourages the expectation of latent graphs effect on other methodologics like LaBo, HardAR, and E-CBM.
Salient Subgraph for Concept-Supervised Settings
The concepl-supervieed setting lavors dense graphs, but il also easily leads to redundant connectivity for the latent graph, which will harm the concept graph interpretability and inference efficiency. We follow a heuristic strategy find the salient subgraph inside the original concept graph: we mask one edge a time and check the performance; if the performance remains the same or goes up, we delete that edge from the original subgraph. By doing this, we improve our label prediction marginally without, sacrificing concept.
22

---

Published in Transactions on Machine Learning Research (02/2026)
Latent Graph CDM
77.16%
77.14%
I
Percentile Accuracy (%)
76
73.53%
70
CDM
G-CDM
G-PCBM
Figure Adding a latent graph to the CDM can siguificantly improve model performance.
Figure 10: Both graphs G-CBM concept graphs for the CUB dataset. Right: original concept graph. Left: salient subgraph structure.
Cardiac and Mediastinal Abnormalities
Lung Parenchymal Diseases
Pleural and Lung Structural
CBM
Abnormalities
Lung Parenchymal Diseases
Pl curral and Lung Structural Issues
Figure 11: Both graphs are G-CBM concept graphs for the ChestXpert dataset. Right: original concept.
graph. Left: salient subgraph structure.
23

---

Published in Transactions on Machine Learning Research (02/2026)
prediction (CUB label accuracy: 80.03% 80.40%; concept AUC: 83.20% + 83.20%: # of edges: 679 239). Extracting the salient subgraph makes the concept graph more easily interpretable: for example in Figure I we can filter out the redundant edges and draw more sparse concept graph. In the salient subgraph, concepts belonging to the same label class are more likely to connect. This also demonstrates that our latent graph effectively captures the hidden concept correlation embedded in the concept supervision.
We provide a quote from ChatGPT to show the reasonable connection between two node pairs: (node 6 (Pneumonia) and node 10 (Pleural Other)):
A. Pneumonia can directly affect the pleura, leading to conditions like pleural effusion, empyema, or pleurisy. These pleural complications often arise due to inflammation or infection spreading from the lungs the pleural space, making the relationship between pneumonia and pleural disease significant in both diagnosis and trealment.
And (node (Enlarged Cardiomcdiastinumm) node 10 (Pleural Other)):
Cardiac vascular causes: Enlarged cardiomediastinum often results from cardiac enlargement (e.g,, heart failure or pericardial effusion) vascular abnormalities (like aortic aneurysms). Some of these conditions can also cause pleural changes. For example:
1) Heart failure can lead to pleural effusion (fluid in the pleural space), which may manifest as a "Pleural Other' abnormality.
2) Aortic aneurysm dissection may affect surrounding pleural structures due to proximity, causing pleural thickening effusions.
Malignancies: Tumors in the mediastinum (c.g., lymphomas metastatic discase) can enlarge the cardionediastinum and simultaneously invade or affect the pleura, leading to pleural abnormalities.
Infections and inflammatory conditions: Severe infections like tuberculosis mediastini- tis can affect both the mediastinum and pleura, causing enlargement of the cardiomediastinum and pleural changes.
J G-CBM Intervention at full ratio
We compare our G-CBMs with E-CBMs (Xu et al. 2024 and standard CBMs (Koh et al. 2020) at full interventions. In Figure I2 G-CBMs can improve the performance significantly compared to the standard CBMs, and it can match up with E-CBMs as well. E-CBMs require energy calculations and gradient backpropagations to update the label probability, so the full intervention can offer sufficient information for E-CBMs to reach a good interrintervention performance. However, E-CBMs cannot handle low-ratio intervention well shown in Figure B, while our G-CBMs can be effective at different intervention ratios coherently. Plus, one-step intervention makes G-CBMs more efficient.
K
Graph Complexity
Graph CBMs favor sparse graphs under the label-free settings. Without concept supervision, Graph CBMs prefer sparse graphs in general. As shown in Figure 1 the model gains performance improvement continue making the learnable graph sparse. Dense graphs result in over-smoothin concept scores, while sparse graphs can better preserve each concept value and propagate concept relation reasonably. We compare different graph complexity effects on the CUB datasct by varying the hyperparamcter / (large / indicates more sparsity in the graph) under the Graph PCBM setting. We observe that our Graph PCBMs favor sparse graphs. The reason might be due to the way one collects concepts. If the model is trained under the label- free setting, one relics on sophisticated concept generator like LLMs; however, LLMs will casily provide lots of redundant and repeated concepts. On the other hand, the dataset contains concept annotations, concept sets usually smaller in terms of the number of concepts and sufficiently informative. Moreover, concept.
24

---

Published in Transactions on Machine Learning Research (02/2026)
Full Intervention
100.0
98.000
97.5
96.910
92.5
Accuracy (%)
91.220
90.0
87.5
85.0
82.5
80.0
CBM
E-CBM
G-CBM
Figure 12: Comparison among models when full interventions.
annotations can act an implicit concept correlation supervision, and models are expected to capture those hidden correlations in nature.
L Visualization of Learnable Graph
25

---

Published in Transactions on Machine Learning Research (02/2026)
Performance via Graph Complexity
Model
G-PCBM G-CBM
80.0
79.5
79.0
Accuracy
78.5
78.0
77.5
77.0
76.5
0.00
0.05
0.10
0.15
0.20
0.25
0.30
Beta
Figure 13: Performance changes as we set different B value to control the latent graph complexity.
26

---

Published in Transactions on Machine Learning Research (02/2026)
15
10
36
225
103
337
12
245
288
335
239
347
250
307
246
287
30
232
235
2
313
Figure 14: Zoom-in the densest component in the latent concept graph.
27

---

Published in Transactions on Machine Learning Research (02/2026)
Figure 15: The overview of the CUB's concept graph in label-free settings.
28


Reviews:
[{"review_id": "GNgtii9gZ5", "reviewer": "Reviewer_1nzf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes graph-based concept-bottleneck models wherein the pretrained image embeddings are passed through a graph neural network to obtain a concept vector interpretable through textual concepts. The paper introduces a plug-and-play graph-based extension of CBMs and optimization objectives to learn concept-relationships and a final concept-vector. The key benefit of this approach is the ability to enable robust concept level interventions while retaining higher predictive accuracy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Partially, no. While the results of the paper do support the claims, additional experiments/information may be necessary to attribute the improvements to graph-based concept-model.", "audience_yn": "Yes", "audience_explanation": "Yes, multi-modal models are of popular interest and this paper proposes an improvement to concept-bottleneck models that enables robust intervention.", "requested_changes": "1. The paper requires few changes for enhanced readability/clarity. Here are a few suggestions:\n* On page-4: \"For each image $v_i$, the initial concept score vector is computed via the ~~dot product~~ matrix-vector product\"\n* On page-4, could you please clarify the shape of adjacency matrix $\\mathcal{A}$: is it $k$ or $k+1$? \n* On page-5, *\"The two contrastive losses operate at different layers and ...\"*: as per Eq (2), it appears that the two losses are computed based on the node-embeddings at the last-layer. Please clarify. \n2. Perhaps I missed it, but could you please describe the details of how Graph-CBM extends LF-CBM and PCBM? More specifically, I am curious to better understand the source of gains shown in Table 1. Are they coming from Graph-CBM having a higher capacity? Or, is it due to the losses introduced? Or, both? \n3. Table 4 is good. However, for reference, it may be good to also show the classification performance based on just CE-loss (i.e., no contrastive-terms): both, with the GNN and with a simple linear-projector at the end of embeddings.", "broader_impact_concerns": "NA.", "additional_comments": "NA.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GrAu5B6aph", "reviewer": "Reviewer_vaNy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a method to extend concept bottleneck models (CBM) to not work with independent concepts but a concept graph. The method employs message passing along the edges in the context graphs and can be integrated in various CBMs. The authors report superior performance over conventional CBMs and present results on the robustness of graph CBMs.\n\nStrengths:\n- The method builds on the reasonable insight that often \"concepts\" are not independent\n- The authors provide an extensive evaluation, spanning multiple aspects/perspectives\n\nWeaknesses:\n- The method still remains somewhat vague. For example, figure 1 does not comprehensively describe the method and the accompanying text does not clear up all the missing details. For example, how the concept graph is generated is unclear and requires a detailed description since it appears to be integral part of the approach.\n- A lot of design choices seem somewhat arbitrary and are only accompanied by hand wavy explanations. For, example it is unclear why supervised approaches are susceptible to spurious correlations while unsupervised methods are not.", "claims_and_evidence_yn": "No", "claims_explanation": "> To capture intrinsic interactions within the concept space, concept graphs must remain independent of label supervision, as labels can introduce spurious correlations that obscure genuine concept dependencies.\n\nIt is not clear why an unsupervised approach is not suffering from the same limitations\n\nThe \"second contrastive\" lacks justification and proper ablation. Just switching terms on and off (table 3) does not suffice since multiple losses require delicate balancing.\n\nThe claim\n>  inherently permutationinvariant \n\nis not well discussed explained.\n\nWhile there is a slight outpeformance of unsupervised SOTA (with missing error bars), the proposed method does not outperform prior work under supervision, despite the claim in the abstract etc.", "audience_yn": "No", "audience_explanation": "At the current stage the paper remains very vague and inaccurate s.t. I doubt that others would be greatly interested.", "requested_changes": "Rework method description. Due to the many parts, I suggest a self-contained main diagram detailing the model. Otherwise, address comments in other fields..", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VAYnkJPZhx", "reviewer": "Reviewer_2tRj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an interesting variant of Concept Bottleneck Models (CBMs) that explicitly relates different concepts through a learnable graph structure, referred to as Graph CBM. The approach is flexible and can be incorporated into different methods, improving both performance and interpretability.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors evaluate the proposed method on 8 different datasets (5 with unlabeled concepts and 3 with labeled concepts), including four different combinations of the graph structure with other methods. Overall, the proposed structure demonstrates improved performance.", "audience_yn": "Yes", "audience_explanation": "Yes, the idea is very interesting, and improving model interpretability is an important and highly relevant topic.", "requested_changes": "Overall, the paper is well written. A small request:\n- Figure 13 is difficult to read. Consider zooming in or reorganizing it similarly to Figure 4.", "broader_impact_concerns": "None.", "additional_comments": "- The grouping of concepts, which promotes sparsity, seems to be key to the reported improvements in interpretability and performance. The use of a graph structure, along with the heuristic edge removal strategy described in Appendix I, is an elegant approach. However, the paper does not discuss alternative methods for inducing sparsity in CBMs. Could the authors comment on whether simpler approaches, such as weight pruning or auxiliary sparsity losses/regularization applied to a standard CBM, might achieve comparable performance gains? Including a discussion of such alternatives in the related work could strengthen the proposed approach.\n\n\n- Interpretable models offer the opportunity to analyze performance under errors or hallucinations. Could the authors comment on this aspect? In particular, how does the model behave under out-of-distribution inputs? For example, if a cat image is presented to a model trained on the CUB dataset, would it be possible to clearly observe that the model cannot recognize the input, or would it still assign it to some of the bird classes/concepts?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)