Concept Flow Models: Anchoring Concept-Based Reasoning with Hierarchical Bottlenecks
Authors: Ya Wang
Adrian Paschke
Venue: Accepted by TMLR ()

Abstract: Concept Bottleneck Models (CBMs) enhance interpretability by projecting learned features into a human-understandable concept space. Recent approaches leverage vision-language models to generate concept embeddings, reducing the need for manual concept annotations. However, these models suffer from a critical limitation: as the number of concepts approaches the embedding dimension, information leakage increases, enabling the model to exploit spurious or semantically irrelevant correlations and undermining interpretability. In this work, we propose Concept Flow Models (CFMs), which replace the flat bottleneck with a hierarchical, concept-driven decision tree. Each internal node in the hierarchy focuses on a localized subset of discriminative concepts, progressively narrowing the prediction scope. Our framework automatically constructs decision hierarchies from visual embeddings, distributes semantic concepts at each hierarchy level, and trains differentiable concept weights through probabilistic tree traversal. Extensive experiments on diverse benchmarks demonstrate that CFMs match the predictive performance of flat CBMs, while substantially reducing effective concept usage and information leakage. Furthermore, CFMs yield stepwise decision flows that enable transparent and auditable model reasoning.

Published in Transactions on Machine Learning Research (02/2026)
Concept Flow Models: Anchoring Baccept-Based Reasoning with Hierarchical Bottlenecks
Ya Wang Fraunhofer Institute for Open Commanication Systems Freie Universitäl Berlin
yo. jo.uang@@foouus.haaaoooe..
Adrian Paschke
udrian.ppschiee fronnsoffrufrr...moo de
Fraunkofer Institute for Open Communicction Systems Freie Universität Berlin
Reviewed on OpenReview: https: openrvieew..........mmmmmo
Abstract
Concept Bottleneck Models (CBMs) enhance interpretability by projecting learned features into a human-understnn understandable concept space. Recent approaches leverage vision-language models generate concept embeddings, reducing the need for manual concept annotations. However, these models suffer from a critical limitation: as the number of concepts approaches the embedding dimension, information leakage increases, enabling the model to exploit spurious or semantically irrelevant correlations and undermining interpretability. In this work, we propose Concept Flow Models (CFMs), which replace the flat bottleneck with a hierarchical, concept-driven decision tree. Each internal node in the hierarchy focuses on a localized subset of discriminative concepts, progressively narrowing the prediction scope. Our framework constructs decision hicrarchics from visual embeddings, distributes semantic concepts at each hierarchy level, and trains differentiable concept weights through probabilistic tree traversal. Extensive experiments diverse benchmarks demonstrata that CFMs match the predictive performance of flat CBMs, while substantially mitigating information Ieakage by reducing effective concept usage. Furthermore, CFMs yicld stepwise decision flows that enable transparent and auditable model reasoning with hierarchical class structures.
1 Introduction
Deep Neural Networks have revolutionized many fields, yet their opaque decision-making poses significant challenges to reliability, interpretability, and traceability, especially in safety-critical applications (Adadi & Berrada 2018 Rudin 2019). Concept Bottleneck Models (CBMs) (Koh et al. 2020) offer a step toward concepts, which are then used to predict
interpretability by first predicting set of human-understandee the final label. The original CBMs (see Fig. [ typically relied on feature extractors and manual concept annotations to supervise concept prediction during training. Despite improving interpretability, CBMs face two primary issues that limit their broader applicability: first, their reliance on manually labeled concepts restricts scalability; second, they prone to information leakage (Margeloiu al.) 2021 Mahinpei al.) 2021 Havasi et al. 2022 Parisini et al. 2025), phenomenon where learned concept representations encode unintended information about the task label or other concepts beyond their predefined semantics. This enables the model to achieve high accuracy by exploiting spurious correlations in concept activations rather than their semantic content, thereby compromising interpretability.
To address these limitations, post-hoe CBMs (Yuksckgonul et al. 2022 leverage pro-trained vision-language models (VLMs) embed a of task-specific linguistic concept descriptions. By comparing these concept embeddings with image feature embeddings, the model obtains concept activinions, which are subsequently

---

Published in Transactions on Machine Learning Research (02/2026)
weighted via linear layer to generate the final prediction. In the remainder of this paper, refer to this VL.M-based variant as "CBM" for brevity, unless otherwise specified. While this approach removes the need for explicit concept learning and costly manual annotation, the problem of information leakage persists: pre-trained VLM emboddings inherently encode rich information beyond their intended concept semantics, and the linear layer may exploit or misuse these embeddings to achieve high accuracy. For example, when classifying images of cats, the model may rely on vehicle-rclated concepts such as "body paint" or "doors", which are unrelated to animals but spuriously correlate with the target class. Previous studies (Yan al.) 2023 Shang et al., 2024) have shown that the number of concept embeddings approaches the feature embedding dimension, even random concepts-those entirely unrelated to the task-can allow the model to maintain high accuracy, thereby severely undermining interpretability.
2
2
Original CBM
Post-hoc CBM
CFM (Ours)
Figure 1: Illustration of concept-based model structures: Original Concept Bottleneck Models (left), Post-hoc Concept Bottleneck Models (center), and our proposed Concept Flow Models (right). Rounded rectangles represent concepts, trapezoids represent the feature extractor, circles denote classes, & indicates the feature embedding, and C represents the concept matrix.
Recent works have sought to mitigate information leakage by discovering and selecting more compact concept Shang et al. 2024) or by enforcing sparsity Oikarinen et al. 2023 These approaches aim to reduce the number of concepts onto which feature embeddings are projected during prediction, as fewer active concepts limit the model's capacity to exploit semantically irrelevant concepts, thereby reducing the risk of information leakage. To quantify this, Srivastava et al. (2024) introduced the Number of Effective Concepts (NEC), which measures the average number of concepts used to predict each class. However, as task complexity grows, the required number of concepts inevitably increases, heightening the risk of leakage. This raises the question: instead of projecting feature embeddings onto the entire concept space, we partition the concept space into subspaces and route predictions only through the relevant ones, thereby isolating unrelated concepts from the decision path? Building on this intuition, we propose Concept Flow Models (CFMs) (see Fig. 1. which replace the flat bottleneck with hierarchical bottlneccks where each class prediction follows a unique path from root to leaf. This design decomposes prediction into a sequence of localized, concept-driven subproblems, offering three advantages: reduced concept dimensionality at each node-eade step projects features onto a small, node-specific subset of concepts, limiting exposure to unrelated information; reduced total concept usage per class class-each prediction aggregates concepts only along its decision path; and traceable reasoning the tree structure yields step-by-sten decision pathways, enhancing interpretability and enabling systematic auditing.
sets Yan et al., 2023 Yang et al. 2023 Schrodi et al. 2024
the concept-to-label linear layer Yuksekgonul et al. 2022:
Our contributions are summarized as follows:
We propose the Concept Flow Model (CFM), a concept-based model with hierarchical bottlenecks that decomposes decisions into step-by-step, interpretable paths. To support this, we develop a pipeline that constructs decision hierarchies using pretrained vision-language models, selects discriminative concepts without requiring manual concept annotation, and enables end-to-end model training.
We provide both theoretical and empirical analysis demonstrating that CFMs mitigate accuracy gains from random, task-irrelevant. concepts and increase reliance on meaningful semantic ones, while
2

---

Published in Transactions on Machine Learning Research (02/2026)
matching the accuracy of flat CBMs under the same concept budget and structurally reducing the number of effective concepts per prediction, thereby reducing information leakage.
2
Related work
The paradigm of predicting intermediate concepts before final label prediction originated in early inter- pretability rescarch (Kumar ct al., 2009; Lampert ct al. 2009), where attribute-basod classifiers improved transparency at the cost of accuracy. KKoh et al. (2020) formalized Concept Bottleneck Models (CBMs), demonstrating that saccept-based prediction can enable duman--n-theed
interaction while maintaining
competitive accuracy. Subsequent extensions improved CBM utility through concept editing, interactive
mechanisms, and enhanced concept reprosentations (Chauhan et al.) 2023 Steinmann et al. 2023 Hu et al.
2024 Espinosa Zarlenga et. al.
2022:
2023). Another direction models cross-concepi. dependencies within the
bottleneck layer. Havasi et al. (2022) introduce autoregressive concept predictors that sequentially predict each concept conditioned on previous ones, capturing inter-concept correlations to improve concept accuracy and mitigate leakage. Vandenhirtz et al. (024] propose stochastic formulations that propagate interventions through learned covariance structures, improving intervention effectiveness. DDminnii et al. (2024) learn explicit causal graphs over concepts, enabling do-interventions and counterfactual reasoning for enhanced causal transparency. These approaches model dependencies within flat concept layer, whereas our method achieves concept sparsity structurally by distributing concepts across class hierarchy, complementary mechanism that could potentially be combined with dependency modcling.
Despite these advances, they still rely on human-amnotated concept sets, which limits scalability in domain- specific applications. To overcome this limitation, Yuksekgonul et al. (2022) leveraged multimodal models to generate concept embeddings from natural-language prompts, utilizing ConceptNet (Speer ct al. 2017)
and predefined concept sets to reduce manual annotation effort. Later works (Oikarinen al.] 2023 Yang
2023 improved scalability by generating domain-specific concepts with large language models, ena enabling et al. open-vocabulary concept acquisition. However, these approaches face critical challenge (Yan et al. 2023 Shang et al. 2024 Srivastava et al. 2024): as concept count grows toward the embedding dimension, even random concepts can induce linear separability, allowing models to preserve accuracy through spurious correlations rather than semantic grounding. Recent efforts focus on selecting or discovering high-quality concepts using Elastic Net regularization (Yuksckgonul et al.] 2022), submodular selection (Yang et al., 2023), incremental discovery (Shang et al.) 2024), dictionary learning (Yan et al.) 2023) and open-domain object detectors (Srivastava et al.) 2024). Nonetheless, the required concept count inevitably grows with task complexity. Our method introduces a human-aligned decision hierarchy to decompose the task, which structurally reduces per-node concept demand and enforces hierarchical sparsity constraints, thereby mitigating information leakage and enhancing the utility of semantic concepts in classification.
Another line of work enhances interpretability through hierarchical structures. Some models dynamically route inputs through tree-like architectures (Murdock et al. 2016 Mullapudi et al.] 2018 Cai et al.] 2021). )
but rely on impure leaves and yield stochastic, uninterpretable paths. Others (Deng et al. 2014 Redmon & 2019) use predefined hicrarchies like WordNet [Miller et al.) 1990 which introduce biases when concept ual similarity misaligns with visual discriminativeness. Neural-Backer Decision Trees (NBDTs) (Wan et al. 2020 train networks with path probabilities and extract hierarchies by clustering final-layer weights, assigning node labels using WordNet. While our method shares a similar training objective to NBDT, it differs by incorporating concept-level semantics directly into the decision nodes and learning the tree hierarchy end-to-end, enabling inherently interpretable, concept-driven inference paths. Recent. work has explored combining decision trees with concept-based models for different purposes. [Rodiguez/ et al. 2024) replace the CBM label predictor with differentiable soft decision tree to enhance decision transparency, though each node receives the full concept vectors and all leaves contribute probabilistically to predictions. Ragkousis & Parbhoo (2024) use decision trees to inspect and quantify information Icakage by comparing hard versus soft concepl ippresentations, enabling localized leakage control but requiring ground-truth concept annotations. Other approaches introduce hierarchies within concept representations rather than over classes: [Sun et al. (2024) augment percoptual concepts with descriptive attributes using class-specific subsets and an intervention matrix, while Pittino et al. (2023) organize concepts into multiple categories whose features are concatenated for final classification, allowing all concept types to jointly influence
Farhadi 2017 Brust & Denzler
3

---

Published in Transactions on Machine Learning Research (02/2026)
predictions. In contrast, method constructs hicrarchy over classes, distributes concepts to internal nodes such that each prediction uses only path-specific subsets, and operates without manual concept annotation, providing structural isolation that complements these approaches.
3 Method
Dog
LUM
Dog
PaN
Cat
Image
Text
Airplane
Vahicle
Airplane
Truck
Truck
Pruning
Car
08%
Decision
Concept Contributions
99%
Root (1.00)
*
Cat 188
Explanation
ft3,0
Animal
90%
Image Encoder
36%
Airplane
Animal (99%)
Truck
90%
40%
2.8%%
Dog (97.02%)
Figure 2: Building Concept Flow Models: Our framework consists of three phases: (1) Hierarchy Extraction & Semantic Annotation: Construct decision hierarchy from CLIP embeddings, prune low- separation nodes, and label internal nodes using LLMs. (2) Concept Generation & Allocation: Generate LLM-based candidate concepts, select discriminative ones, and allocate them to hierarchical bottlenecks. (3) Differentiable Concept Flow Training: Train the model end-to-end with hierarchical path traversal loss, enabling traceable predictions through concept-weighted transitions.
As illustrated in Fig our approach differs from Concept Bottleneck Models, which expose all concepts in a flat bottleneck layer. Instead, we extract a semantically coherent tree hierarchy from CLIP embeddings (.2) and allocate LLM-generated concepts its internal nodes (See train them with concept-driven probabilistic flows (Sec 3.4,, thereby constraining each class prediction to rely primarily concepts along the most probable decision path, while reducing the influence of unrelated concepts and yielding traccable, interpretable predictions.
3.1 Problem setup
We consider a multi-class classification task over a dataset D = ((x, })} where E X denotes input images and y E y (1...., [.] represents class labels. The core challenge lies constructing an interpretable decision tree hierarchy nodes encode hierarchical relationships by representing localized subspaces of the overall concept space. Unlike CBMs that compress decisions through single layer with uncoordinated concepts, discriminative concepts across multiple hicrarchical nodes. Each internal node v E V L maintains: (1) set. of human-interpretablee concepts, (2) learned importance weights for these concepts, and (3) transition probabilities to child nodes based on concept relevance. The tree constrains each class to follow a unique docision path from the root the corresponding leaf node, thus minimizing the inffuence of concepts assigned nodes outside this path. The learning objective minimizes the cross-entropy loss through smcept-weighted hierarchy traversal:
(V, E) where leaf nodes L V correspond to target classes V, while internal
distribute
4

---

Published in Transactions on Machine Learning Research (02/2026)
min B( (y(2))
(1)
where 0 denotes all concept importance weights in the hicrarchy, and poly)ar) represents the probability of reaching the ground-truth class y through concept-driven transitions (detailed in Sec. 3.4
3.2 Tree hierarchy extraction
Our approach builds on a pretrained VLM, CLIP (Radford al. 2021 which comprises an image encoder 2 X - He and text encoder 4r T + E. projecting inputs from both modalities into shared d-dimensional semantic space. The cross-modal aligmment of CLIP ensures that semantically related images and texts are embedded closely together, resulting in a structured representation space that supports both hierarchical clustering and concept-based reasoning (Bhalla et al.) 2024 Stein et al., 2024p. We leverage this property as a unified basis for constructing the concept hierarchy and performing concept selection. To construct a semantic hicrarchy, we first compute class-level emboddings f4; by averaging the CLIP image embeddings of all instances belonging to class i. We then apply agglomerative clustering with Ward's linkage (Ward Jr. 1963) to the resulting centroids M -(E),...,/1 yielding binary tree represented by a linkage matrix
([141-1xx4 The hierarchy is built bottom-up manner by ileratively merging node pairs (up, U2) with the smallest merge distance in Z. Overly deep trees can cause an exponential growth in node count, fragmenting the concept space into small subspaces and yiclding weakly separated nodes near the leaves. We mitigate this with distance-based pruning, removing nodes whose merge distances fall below the I-th percentile of all distances in Z, producing a multi-branch hierarchy with fewer, broader internal nodes. Alternatively, node separability can be measured by the classification accuracy of a Linear Discriminant Analysis (LDA) fitted on its samples, pruning nodes whose scores fall below a threshold
Semantic annotation: We annotate internal nodes via breadth-first traversal using L.L prompts. For each non-leaf node, we generate descriptive labels by prompting an LLM with the leaf classes in its subtree, producing hierarchical summaries. Since tree structures vary with VLM backbones and pruming thresholds, manual review may be applied to ensure the selected backbone is well-suited to the target domain, different VLMs are trained on different data and exhibit varying strengths across tasks. In practice, widely-used VLMs (e.g,, CLIP ViT-B/32) produce scaantically coherent hierarchies for common domains such as general object or action recognition; manual intervention is primarily needed for specialized domains where off- offthh VLMs may lack sufficient coverage (see Appendix A.1
3.3 Concept selection and distribution
Following prior works YYan et al., 2023 Yang et al., 2023). we leverage LLMs to generate candidate concepts for each class, enhancing visual and hierarchical grounding by including image samples and hierarchical context in the prompt. For each internal node v, aggregate all candidate concepts from the leaf classes under that node to form its concept set. We preprocess this aggregated set by discarding concepts that contain class names and eliminating semantically redundant concepts (sce Appendix A.2 and A.3 The resulting filtered concepts E, = (ex,1.....,,e
are embedded via the text encoder + into a matrix C, E xR To evaluate concept relevance for node v, we compute the cosine similarity between all training image features Z (X) IR." xxd and the concept embeddings C.:
S. = = E Run..
(2)
where 2 and Č. denote the no-normalized rows of Z and Co, respectively. For large-scale datascts, computing S, over all n samples can be prohibitive. Since the goal is concept selection rather than model training, a representative subset suffices. We therefore apply stratified sampling by grouping samples by class and randomly drawing a balancod subset from cach class to bound the total samples (c-g., Rmma 15,000) while ensuring sufficient samples per class (e.g., 30) for reliable estimation. We then perform concept selection for each internal node using Tasso-regularizeed multi-class logistic regression. Let W R denote the
5

---

Published in Transactions on Machine Learning Research (02/2026)
weight matrix and ya Ilys Z] the class indicator. We optimize:
I min EE Cyaa, WT 8:) +
W
(3)
where 8; the i-th row Sp. Each concept en.j receives an importance score 1, = Σ [Wijls and the top r concepts with highest 1 are selected to form the final set Cg. To determine r, we adopt hybrid allocation strategy based on the node child count and the average concept share per node B (sce Appendix A.4
max(a [Ch(v)] + (1 - a) [B], Tmiu).
(4)
where E [0, 1] and miin are adjusted based on tree balance, preventing any node from being underrepresented or overburdened with concepts. For simplicity, we omit the superscript in C. hereafter.
3.4 Model architecture and training
CFM replaces the flat bottleneck layer with hicrarchy of bottleneck nodes, cach equipped with selected concepts and trainable weights. This enables a differentiable decision process that propagates class probabilities along the concept-driven hierarchy. Each internal node v is parameterized by two matrices:
C. E R Wo E R
55)
where C2 represents concept embeddings pre-selected for node v, and W,, denoles trainable weights for these concepts, which model the local branching decisions each node.
Temperaturo-scaled transitions: Each node reccives normalized image embedding 3 = (x)/ (x) l2 input. It. first projects the embedding onto the pre-selected concept matrix C. RR (as defined in Section 3.3 obtaining concept activations. These activations are then transformed by the learned local concept weight matrix W, E R where m = [Ch(v) is the number of child nodes, yiclding the transition probabilities to the child nodes. To control the sharpness of branching decisions, each node also learns a temperature parameter T, E R++ The transition probability from node to one of its children by computed as:
(3CJ) W.
Po-te, (x) = softmax
(6)
T,
In contrast to CBMs, which connect input embeddings all concepts globally, CFMs associate each node with local subset of concepts, thereby enabling parallel computation across nodes with no significant increase in computational overhead (see detailed complexity analysis in Appendix C.Ip.
Path probability aggregation: The class probability corresponds to the probability of reaching leaf node E L from the root node vo, computed recursively along the path ()),
(
Po-my (x) (o)) -b,
Pog Pog-st
II
(7)
where by + is a learnable bias term calibrating path-specific confidence. Final predictions use:
((x) = arg max Pen (t)))
(8)
TeT.
Training objective: We freeze the CLIP image encoder & and optimize the tree parameters (W., Tu, by by minimizing the negative likeliiel of ground-truth
class probabilities:
L
log Po, (x)-
[D]
(R.N)E
where (x)) is the probability of the hierarchical decision path from the root vo the leaf node corresponding to the true class y-

---

Published in Transactions on Machine Learning Research (02/2026)
4 Theoretical Analysis of Concept Usage in CFMs
We provide a thcoretical analysis of CFMs, focusing on how they utilize concepts differently from flat CBMs. Our analysis examines two scenarios: when models use random concepls, which are unrelated to the prediction task (either sampled i.id. from the unit sphere in theory or drawn from a general English dictionary in experiments); and when both models use curated semantic concepts, which are semantically related to the target task (e.g., "has wings" for birds). Let (I, y) ~ D be the data distribution, and let & (x) E R denote the CLIP image embedding. We assume total concept budget of R vectors for both CBM and CFM.
CBM. A flat CBM uses a single concept matrix C TR Rxd to compute logits = a followed by classification arg (sX)(
CFM. A CFM constructs a tree with m internal nodes. Each node e is assigned a subsct of r' concepts from a total of available concepts, where the expected number of concepts per node satisfies r' R/m. Each node also contains local branching weights We. During inference, each input traverses root-to-leaf path U2) of expected length L = E log, m, where b denotes the average branching factor of the r (Ix) (() tree.
4.1 Information-Leakage Barrier of Hierarchical Bottlenecks
Proposition 4.1 (Leakage Resistance). Let the image embeddings Z = ((), C Rd x [L]]] be linearly separable across ||| closses. Using random concept vectors:
(i) CBM. Seporability is still preserved with high probability once the total concept count solisfies R = -(d)..
(ii) CFM. To keep every internal node separable, each node must receive r' = (i(d) concepts, giving a total requirement R = ([md) to ensure global separability, i.e., a factor m larger than the flat CBM.
Interpretation. A flat bottleneck leaks all d dimensions at once, so (2(d) random projections suffice to reconstruct the decision boundary to preserve accuracy. In hierarchical structure, however, each of the m internal nodes handles a different local classification subproblem. Since random projections do not capture specific properties of the data, a small set of random concepts cannot reliably separate subproblems. Thus, each node requires its own of ((d) random concepts, resulting in a total of (mmd) concepts to ensure global separability. Moreover, as m increases under fixed concept. budget R each node receives fewer concepts, reducing concept sharing compared to CBM, which shares all R concepts in single layer. Distributing concepts across more nodes naturally enforces concept isolation: each predicted class relies only concepts along its decision path, increasing concept sparsity and mitigating information leakage).
4.2 Semantic Efficiency of Hierarchical Decision Paths
Proposition 4.2 (Semantic Concept Sparsity via Hierarchical Structure). Assume that for each internal node there crists small concept subset of size r' that lincarly separates its child clusters with high probability. If the global budget R is distributed so that each node receives at least this r', then:
(i) The resulting CFM can achieve comparable training accuracy to a flat CBM that uses all R concepts al once.
(i) The number of concepts contributing a single prediction is at most er', where r' R/m and the voot-to-leaf poth length. For a balanced tree, ss log, m, so the expected concept usage per prediction is O( log, m),, compared to O(R) for the flat model.
Intorprotation. Assuming that the tree hicrarchy reflects the natural semantic hicrarchy of the data, each internal node only needs to separate its own (smaller) set of child clusters, which simplifies the local separation task and allows each node to use only modest subset of concepts (r' < R) instead of all R concepts. For
7

---

Published in Transactions on Machine Learning Research (02/2026)
single prediction, only the concepts along the root-to-leaf decision path are used. Thus, the expected number of concepts per prediction is at most fr'. In practice, this is much less than R since irrelevant. concepts (outside the decision path) are structurally isolated. Mathematically, the upper bound O R log, m) shows that the ratio log, less than for any m > and > 1, and satisfies limm- hierarchy becomes more granular (i.e., as m increases), the number of used concepts per prediction decreases rapidly, promoting greater concept sparsity and reducing the risk of information leakage.
log, 0. Thus, as the m
m
Why accuracy need not drop. When the hierarchy is semantically coherent e.g., constructed via hierarchical clustering of CLIP embeddings, with nodes of low separability pruned-each internal node handles a simpler "few-way" sub-problem. The target class only consults concepts within its decision path; other concepts structurally do not activate for the prediction. handful curated semantic concepts can shared by all Icaf classes under the same node, supporting accurate decisions, In contrast, random concepts lack semantic alignment and cannot be effectively reused by its associated classes.
Semantic Utility Metric. To complement the above analysis, introduce the Semantic Improvement over Random Concepts (SIR) metric to quantify the role of semantic concepts in mitigating information leakage. SIR measures the relative accuracy gain when replacing random concepts with curated semantic ones under the same concept budget:
A aCCsem &CCrand
(x (x100%),
(10)
aCCrand
where aCCsem and aCCrand denote model accuracy with semantic and random concept sets, respectively, A high A indicates strong reliance on task-relevant features, while low or zero value suggests that predictions can be replicated with uninformative concepts increptt-indictinng potential leakage.
5 Experiments
To empirically validate our theoretical claims in Propositions 4.1 and 4.2 we conduct experiments examining the two scenarios analyzed in our theory: using non-semantic random concepts and curated semantic concepts. We further perform ablation studies to assess the contribution of individual components within CFMs, visualize and interpret the decision- decision-making process, and discuss broader implications and limitations.
5.1
Setup
Datasets. We conduct our evaluations on five image classification benchmarks: CIFAR-10, CIFAR- 100 (Kiizhevsky et al.] 2009). UCF-101 (Soomro et al. 2012). CUB-200 (Wah et al.) 2011 and Tinylm- ageNet (Le & Yang 2015). These datasets span a range of semantic granularity and domain complexity. Specifically, CIFAR-10 contains 10 general object classes, CIFAR-100 and UCF-101 include 100 classes each (with CIFAR-100 focused on common objects and UCF-101 on action scenes), and both CUB-200 and TinylmageNet contain 200 fine-grained categories, with the former targeting bird species classification and the latter composed of small-scale versions of ImageNet classes. Additional experiments ImageNet-1KK are provided in Appendix B.S.
General settings. Unless otherwise specified, all models use the CLIP VIT-B backbone outputting 512-dimensional embeddings. We train with Adam (Ir=0.01, batch size=128) using ReduceLRRnPlateau (learning rate halved on validation loss plateaus) and early stopping with a 20% validation split. Training terminates upon validation stagnation, retaining the best checkpoint. For controlled experiments, CFMs are pruned to four internal nodes with layers in depth (see Appendix B.2 for details).
5.2 Scenario 1: How do CFM and CBM perform under random, non-semantic concepts?
Settings. We systematicalyy compare CFM and CBM on CIFAR-10 using random (non-semantic) concepts, and include a linear probe model (Radford et al., 2021) as an upper bound of the training accuracy (trained directly on CLIP image embeddings without interpretability constraints). Random concepts are generated
8

---

Published in Transactions on Machine Learning Research (02/2026)
A. CFM vs. CBM (Budget R)
В. CFM vs. CBM (Budget R m vs. R)
Test Accuracy (%)
10
50
Linear Probe (total)
Linear Probe
(per CBM
CBM
20
20
100
200
300
400
500
100
200
300
400
300
Number of Total Concepts
Number of Bottleneck Concepts
CFM: Pruning = 60)
CBM: Sparsity 60)
Test Accuracy (%)
70
50
50
30
Number Nodes
Regularization
Figure 3: Information Leakage with Random Concepts. (A) Accuracy of CFM and CBM as total concept budget increases (same R for both). (B) Accuracy when CFM receives concepts per node (R x m total), compared to CBM with R. (C) Effect the number internal nodes on CFM accuracy R = 60. (D) Impact of Elastic Net regularization on CBM accuracy R 60. All experiments are repeated three times with different random seeds. Detailed numerical results are provided in Appendix B.4
Our experiments examine: (A) accuracy as the total concept budget increases, with both models constrained to the same R;; (B) accuracy when CFM receives R concepts per node (i.e., R x 4 concepts in total), compared to CBM with R; (C) varying the number of internal nodes in CFM under fixed R 60; and (D) varying the sparsity regularization strength X in CBM (implemented via. Elastic Net from Zou & Hastie (2005) with L1 ratio 0.9) under fixed R = 60.
by sampling dictionary phrases (see Appendix A.2
Results. In (A), CBM accuracy rises quickly with R, reaching 90% with only 64 concepts and approaching the linear probe baseline as R approaches the embedding dimension, consistent with Proposit ion 4.1 i). Tn contrast, CFM achieves only 75% under the same budget, reflecting its structural resistance to information leakage from random concepts. In (B), when CFM's total budget is increased to R x 4, its accuracy curve closely tracks that of CBM, consistent with Proposition 4.1 fii,, which states that achieving global separability in CFM requires scaling the total concept count with the number of internal nodes. In (C), reducing the number of internal nodes by pruning relaxes the structural sparsity constraints, increasing accuracy more concepts shared per node and information isolation is weakened. The complete unpruned binary tree represents the case of most strict concept isolation. In (D), increasing sparsity regularization in CBM generally reduces accuracy by limiting the number of effective concepts, analogous to the structural sparsity imposed by CFM, although sparsity regularization camnol structurally prevent misuse of concepts from other classes. Notably, sparsity regularization can similarly be applied to CFM to control concept usage (see Appendix B.3). Overall, these results confirm that CBMs with a flat structure are prone information lcakage, while CFMs provide a structural barrier that limits accuracy gains from random concepts, consistent with our theoretical analysis.
9

---

Published in Transactions on Machine Learning Research (02/2026)
5.3 Scenario 2: How do CFM and CBM perform under curated semantic concepts?
Settings. We compare CFM and CBM using curated semantic concepts, with the linear probe again serving as a non-interpretable baseline. For CBMs, we include two representative models: PCBM (Yuksekgomul ctt al. 2022), which retricves predefined concepts from ConceptNet (since the original PCBM lacks concept selection, we implement it by ranking concepts based on ConceptNet edge weights and selecting the top-r most relevant), and Labo (Yang et al. 2023), which uses LLMs to generate and select task-specific concepts. To quantify the utility of semantic concepts, we further compare each semantic model to its random counterpart: CBM (Random) and CFM (Random), where semantic concepts comparison, CFM uses the same candidate concept set as Labo, and all models are trained without sparsity regularization. We conduct experiments on all five datascts, keeping the total concept budget identical across models. Specifically, the total concept budget is set to two concepts per class for each dataset, except for CIFAR-10, which uses a fixed budget of 60 concepts. This yields 60, 200, 202, and 400 concepts for CIFAR-10, CIFAR-100, UCF-101, and B0TB-200/TiinImmne threshold and concept allocation ratio obtain exactly four internal nodes with suitable concept distribution. Hyperparameters for Labo and PCBM are also optimized each dataset (see Appendix B.2 experiments comparing CFM and PCBM with sparsity regularization are provided in Appendix B.6
replaced with random ones. For fair
respectively. For CFMs, we finc-tune the pruning
Additional
Metrics. In addition to accuracy, we report NEC (the Number of Effective Concepts) proposed by Srivastava et al. (2024), which quantifies the average number of concepts used to predict, each class. We also report the Semantic Improvement over Random Concepts (SIR), which measures the accuracy gain achieved by semantic concepts relative to random concepts.
Results. Consistent with Proposition 4.2 CFMs achieve comparable accuracy to CBMs (PCBM and Labo) under the same concept budget, while using far fewer effective concepts (see Table D For example, on CIFAR-10, CFMs reach 91.59% accuracy with only 7 effective concepts, compared to Labo (90.93%) and PCBM (91.1%%), both using around 57. On TinylmageNet, CFMs achieve 70.79% accuracy with 92 effective concepts, outperformigg Labo (69.86%) and achieving comparable results to PCBM (71.07%), both using over 360. This efficiency sterns from structural constraints that limit each prediction to the concepts along its root-to-leaf path, resulting in much lower average concept usage per prediction. As a result, CFMs show substantially higher SIR across all datasets, for instance, 19.78 on CIFAR-10 versus 1.94 for Labo, with similar gains TinyImageNet and other benchmarks. This structural sparsity constraint also reduces information leakage, predictions depend on a smaller, path-specific subset of concepts rather than the entire pool. Additionally, CFMs' much lower accuracy with random concepts (e.g., 76.46% on CIFAR-10 vs. CBM's 89.20%) highlights their reliance on mcaningful semantic concepts. When the concept budget is limited, CFMs consistently match CBM accuracy by prioritizing semantic utility.
Table 1: Comparison of various methods across five benchmark datasets. We report the number of effective concepts (NRC), classification accuracy (Acc), and semantic improvement over random concepts (STR). Results are reported mean over runs. Bold values highlight the best in each column (excluding linear
probe). Full results with standard deviations are provided in Appendix B.5
CIFAR-10
C1F
UCP101
CUB200
TinyImageNet
Method
NEC
Ace
SIR
NEC
Acc
SIR
NEC
Acc
SIR
NEC
Acc
SIR
NEC
Ace
SIR
Linear Probe
N/A
94.59
N/A
N/A
77.03
N/A
N/A
95.00
N/A
N/A
71.68
N/A
N/A
75.25
N/A
CBM (Random)
56.80
89.20
184.20
68.50
0
187.31
79.15
344.19
52.23
0
362.34
68.78
(Random)
76.46
54.24
62.00
44.99
72.76
104.61
47.08
89.83
PCBM
56.80
91.15
2.19
183.99
68.98
0.70
N/A
N/A
N/A
N/A
N/A
N/A
362.01
71.07
3.32
Labo
57.73
90.93
1.91
184.33
70.98
3.62
190.60
81.62
6.91
357.92 957.92
65.13
21.70
369. 369.36
69.
1.57
CPM (Ours)
6.90
91.59
19.78
55.02
72.29
16.61
45.80
84.60
16.27
109.34
65.68
39.50
92.12
70.79
9.09
5.4 Ablation Study
Effect of tree hierarchy on performance. To validate the effectiveness of our ULIP-induced hierarchy generation strategy, we compare it against two alternative approaches: (1) Random, which generate the hierarchy by randomly clustering the classes, and (2) NBDT (Weights), following Wan et al. (2020), which generate the hierarchy by clustering the final-layer weights (each of the final-layer weight matrix
10

---

Published in Transactions on Machine Learning Research (02/2026)
corresponds to class representative) in the trained linear probe model (see Appendix A.1 As shown in Table 2 our approach achieves higher accuracy on four out of five datasets, with particularly strong gains on UCF-101 (+3.16% vs. Weights) and CIFAR-10 (+1.26%). By contrast, the random baseline consistently ranks last, highlighting the importance of semantic coherence in hicrarchy construction. The CLIP-induce hierarchy underperforms on CUB-200, likely due to the fine-grained nature of bird-species classification, which challenges CLIP's general-purpose embeddings.
Figure f shows that, when the total concept budget is fixed, increasing tree depth reduces accuracy. This is because the number of internal nodes grows exponentially with depth, so each node receives fewer concepts, weakening local separability and accumulating errors along the decision path (sce Table for further evidence). Pruning the internal nodes mitigates this effect by allocating more concepts per node and improving accuracy. The number internal nodes thus acts as a structural sparsity constraint: deeper trees promote interpretability through sparser, more traceable decision paths, but risk degrading accuracy if concepts or the CLIP backbone are not sufficiently strong. In practice, tree pruning serves as sparsity hyperparameter that requires tuning.
Table 2: Accuracy (%) comparison among three hierarchy- construction strategies: Random, NBDT (Weights), and our CLIP-induced approach. Each method builds tree hierarchies differently before concept selection. Bold values denote the highest accuracy per datasct.
90
CIFARIO
UCF101
Accuracy (%)
80
CUB200
60
50
Method CIFAR-10 CIFAR-100
UCF101
CUB200 TinyImageNet
30
20
Random
90.82
69.97
81.53
60.44
60.35
1
2
3
4
5
6
7
8
10
11
NBDT
91.00
71.13
82.28
66.24
70.84
Depth
Ours
92.26
72.16
85.44
65.79
71.24
Figure 4: Impact of tree depth pruning
Evaluation of concept selection strategies. We compare our Lasso regression-based concept selection method against four baselines (see more detail in Appendix A.5]: random selection, CLIP similarity maximization, concept embedding orthogonality, and submodular optimization (Yang et al. 2023 As Table 3 shows, our method outperforms all alternatives, achieving 92.26% on CIFAR-10 (vs. 87.18% for submodular) and 65.79% on CUB-200 (vs. 61.63% for orthogonality). The poor performance of similarity- (83.79% on CIFAR-10) suggests that naively prioritizing concept-image relevance leads to redundant concepts, while orthogonality over-constrains the concept space. Submodular optimization, which greedily selects discriminative and diverse concepts, outperforms both approaches. Nonetheless, our method better balances diversity and discriminative strength, yielding compact yet semantically complementary concept sets.
selection
! ablates CFM's core components. Removing the concept matrix (w/o Concept Matris), which approximates the NBDT design, nearly matches Linear Probe's accuracy (94.67% vs. 94.75% on CIFAR-10), indicating the hicrarchy itsclf contributes minimally accuracy loss the primary limitation stems from insufficient concepts across nodes in deeper hierarchies. Path calibration contributes measurably to performance (81.72% vs. 85.44% on UCF101 without it). Node temperature scaling contributes most to fine-grained tasks (62.62% + 65.79% on CUB-200), moderating error propagation in deep hicrarchies. Adding a hierarchical auxiliary loss slightly degrades performance (91.13% vs. 92..99..22. 92.26%) consistent with the observations by Wan et al. (0200) that over-supervision at each node disrupts path probability learning (see Appendix A.6).
Impact of model architecture design choices. Table
5.5 Visualization and interpretation of concept-driven decision paths
We provide qualitative examples comparing the decision paths and activated concepts of CFMs and CBMs. The top part of Fig. 5) shows an example of a "cat" from 10 The CLIP-induced hierarchy in CFM first separates categories into " "Animal" and "Vehicle", and then further narrows the scope to "Domesticated" animals. At each stage, CFM uses the current node's selected concopts to determine the transition probabilitics to child nodes (i.c., the next concept subspaces), effectively routing the prediction and structurally isolating concepts outside decision paths. For the cat sample, CFM uses "facial markings" and "fur texture" to identify
11

---

Published in Transactions on Machine Learning Research (02/2026)
Table 4: Ablation study the model architecture.
Table 3: Comparison of concept selection methods across three benchmark datascts. Bold values high- light the highest accuracy (Accuracy in %).
Accuracy is reported in %. Bold values indicate the accuracy of our model per dataset.
Method Variant
CIFAR-10
UCF101
CUB200
Method
CIFAR-10 UCF101 CUB200
CFM (Ours)
92.26
85.44
65.79
Random
81.25
81.79
57.54
Linear Probe
94.75
94.52
72.40
Similarity
83.79
80.85
52.08
Orthogonality
79.31
77.51
61.63
w/o Concept Matrix
94.67
93.73
72.14
Submodular
87.18
83.29
59.71
w/o Path Calibration
91.72
81.72
64.84
w/o Node Temperature
91.82
84.65
62.62
Lasso (Ours)
92.26
85.44
65.79
Hierarchical Loss
91.13
83.78
65.38
Aiplann
Domesticated (100%)
Automobile
focol markingg ncar oyes (0 92) wature
focia whicker pacs (3)
servccry rountled fees whilewr
Vehicle
wark pointed secknssrl
CIFAR10
will rump patch
facial marking (0.18)
anout curvelure (L..))
Frog
Animal
100%
fecial markings near eyes (12)
visible ear lufts (GB) facial whisker pada 20, (0.178)
Dome comeeticatte Dog 1%%
votible body (0. (51)
fur-ooverad
coloration
Cat 99%
sourdded paw shupe 198)
hortzontal oltt shaped pupls in 1523
doors 160)
Songhirde & Passecines (100%)
Varied Perching (100%)
Belted Kingfisher (30.8%)
vocal sac
0.20
large stout bill
crown and nape
0.19
002800
wing feather
0.14
orange
0.11
bright throat
0.11
black wings
0.20
blue-gray head
0.11
0.00
0.05
0.10
0.35
0.20
0.00
0.10
0.20
0.15
0.20
Figure 5: CFM decision path visualizations. Top: CFM and CBM explanations for a CIFAR-10 "cat" sample. Bottom: CFM decision path visualizations for selected samples from CUB-200. All models are trained using the CLIP-ViT-L/14 backbone, and candidate concepts are generated via GPT-4.1-mmni The top activated concepts per node for CFM and the top 9 activated concepts for CBM listed.
the image as an animal, then leverages "whisker pads" and "pointed ears" to rule out frogs, birds and deer, and finally relies "sensory whiskers" and "rounded facial whisker pads" to distinguish cat from similar classes such as dog or horse. In contrast, CBM lacks such structured reasoning and treats all concepts as flat set, which may occasionally lead to the misuse of irrelevant concepts for prediction. For instance, in the same cat example, CBM activates concepts like "vehicle body paint" " and "side doors", which CFM assigns to the vehicle concept space and are unrelated to animals. This illustrates how CFM structurally isolates irrelevant. concepts, thereby reducing information leakage.
The bottom part of Fig, shows a CFM decision path example from CUB-200. The prediction follows the LLM-annotated route Songbirds & Passerines Colorful & Varied Perching + Belted Kingfisher, with activated concepts at each node reflecting visual attributes of the image. (e.g., "vocal sac", "blue-gras grat head""., These stepwise explanations illustrate how CFM progressively narrows the prediction space, offering additional intuition and traceability for model decisions. However, CFM can still activate concepts not actually present in the image (e.g., "orange streaks" for this Belted Kingfisher), even when concept- seneepttinnatiionn prompts are enhanced with sample images from the training set.. This phenomenon, also noted in prior work (Yang et al. 2023; Oikarinen et al.] 2002), likely arises from CLIP's reliance on global image-text text alignment without explicit region- levect-level grounding. Leveraging region-aware VLMss grounded object detectors (Zhong
12

---

Published in Transactions on Machine Learning Research (02/2026)
et al. 2022
Chen et al. 2024 may help mitigate this issue. We provide further qualitative analyses and extended examples in Appendix 18.7
5.6 Discussion
CFM reconciles accuracy and interpretability by enforcing structured concept utilization through scmantically coherent, hierarchical bottlenecks. In this framework, each internal node assigned a subset of discriminative concepts, which are only shared by the leaf classes under that node. This structural constraint localizes concept usage, ensuring that irrelevant concepts are isolated from class prodictions outside the decision path and thereby effectively mitigating information leakage. As a result, CFM achieves a significantly lower number of effective concepts per prediction and higher semantic improvement ratios (SIR) compared to flat CBMs. By anchoring decisions at human- innnaan-iterrretbble nodes, prioritizing task-specific concepts, and aggregating concept-woighted routing, CFM performs stepwise, somantically grounded inference.
Broader impact. Interpretable models should align reasoning processes with human cognitive frameworks. While traditional CBMs explain predictions through disconnected concept activations, CFM constructs human-aligned decision hierarchics from semantic embeddings, mirroring structural human reasoning patterns. By distributing discriminative concepts across LLn-annotated nodes (Sec. 3.2 and training interpretable hicrarchical decision flows (Sec. 3.40, CFMs provide auditable sequential reasoning paths rather than isolated concept contributions. This structured approach achieves improved semantic utility over CBMs while matching their accuracy (Table D]. proving interpretability frameworks can adopt human-like reasoning hierarchies without sacrificing performance- critical advancement for high-stakes applications requiring transparent. decision-making.
Limitations and mitigations. Three challenges persist: 1) Interpretability: CFM's interpretability fundamentally relies on the quality of visual-language alignment provided by VLMs and the relevance of LLn-genertted concepts. As discussed previously, some concepts are not faithfully grounded in the image and may reflect generic semantic knowledge rather than concrete visual evidence, which can limit. the utility of explanations. This issue is exacerbated in specialized domains (e.g., medical or satellite imagery) that differ from web-crawled training data. This highlights the importance of backbone selection and candidate concept generation prerequisites for both interpretability and accuracy. 2) Fired Tree Hierarchy: CFM employs a fixed tree structure, assuming that class relationships are inherently hierarchical. While this is a common inductive bias in taxonomic classification, it is a strong assumption that may not hold for all tasks, particularly those lacking clear semantic hierarchies or requiring multiple overlapping category structures. Extending CFM to support alternative or dynamic relational structures could enhance its generality and applicability. 3) Generalization: While CFM leverages CLIP's vision-language alignment, its reliance on such backbones currently limits its generalizability. Integrating learnable projection layers OOikarinen el al.) 2023 may enable CFMs to be extended to non con-vvsion-laae vision-language language architectures and other domains.
6 Conclusion
We proposed the Concept Flow Model (CFM), a hierarchical bottleneck architecture that enforces structural sparsity concept-based models by restricting each prediction to path-specific subset of coucepts. Our results show that CFMs can match the accuracy of flat CBMs under the same concept budget, while substantially reducing the number of effective concepts per prediction and mitigating information leakage-particulanyy by limiting accuracy gains from random, task-irrelevant concepts and incrcasing reliance on mcaningful semantic ones. These findings indicate that hierarchical bottlenecs may provide a promising direction for enhancing interpretability without sacrificing predictive performance. Future work will explore adaptive or dynamic hicrarchical structures to incrcase flexibility and generality, investigate CFM's potential as structured context for VLMs to reduce hallucinations, and address limitations in visual groundedness for more reliable interpretability.
13

---

Published in Transactions on Machine Learning Research (02/2026)
References
Amina Adadi and Mohammed Berrada. Peeking inside the black-box: survey explainable artificial intelligence (xai). IEEE access, (552138-52100, 2018.
Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio Calmon, and Himabindu Lakkaraju. Interpreting clip with sparse linear concept embeddings (splice). Advances in Newral Information Processing Systems, 37: 84298 84328, 2024.
Clemens-Alexander Brust and Joachim Denzler. Integrating domain knowledge: using hierarchies to improve deep classifiers. In Asian conference on pattern recognition, pp. 3 16. Springer, 2019.
Shaofeng Cai, Yao Shu, and Wei Wang. Dynamic routing networks. In proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 3588-33888 3598, 2021.
Kushal Chauhan, Rishabh Tiwari, Jan Freyberg, Pradeep Shenoy, and Krishnamurthy Dvijotham. Interactive concept bottleneck models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 0944-5955, 2023.
Hong- You Chen, Zhengfeng Lai, Haotian Zhang, Xinze Wang, Marcin Eichner, Keen You, Meng Cao, Bowen Zhang, Yinfei Yang, and Zhe Gan. Contrastive localized language-image pre-training. Xiv preprinl arXiv:2410.0274 0274,, 2024.
Jia Deng, Nan Ding, Yangqing Jia, Andrea Frome, Kevin Murphy, Samy Bengio, Yuan Li, Hartmut Neven, and Hartwig Adam. Large-scale object classification using label relation graphs. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pp. 48 64. Springer, 2014.
Gabriele Dominici, Pietro Barbiero, Mateo Espinosa Zarlenga, Alberto Termine, Martin Gjoreski, Giuseppe Marra, and Mare Langheinrich. Causal concept graph models: Beyond causal opacity in deep learning. arXiv preprint arXiv:2405. 16507, 2024.
Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelan- gelo Diligenti, Zohreh Shams, Frederie Precioso, Stefano Melacci, Adrian Weller, Pietro Lio, and Mateja Jammik. Concept embedding models: Beyond the aecuracy-explainaaliity trade-off. In Advances in Neural Information Processing Systems (NeurIPS) 35, pp. 21400-21413, 2022.
Mateo Espinosa Zarlenga, Katherine M. Collins, Krishnamurthy Dvijotham, Adrian Weller, Zohreh Shams, and Mateja Jamnik. Learning to receive help: Intervention-awan concept embedding models. In Advances in Neural Information Processing Systems (NeurIPS) 36, 2023.
Marton Havasi, Sonali Parbhoo, and Finale Velei- Addressing leakage in concept. bottleneck models. Advances in Neural Information Processing Systems, 35:23356 23397, 2022.
Lijie Hu, Chenyang Ren, Zhengyu Hu, Hongbin Lin, Cheng-Long Wang. Hui Xiong, Jingfeng Zhang, and Di Wang. Editable concept bottleneck models. Xr preprint Xxi::2405.15476 2024.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Picrson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, pp. 5338-5348. PMLR, 2020.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Neeraj Kumar, Alexander C Berg, Peter N Belhumeur, and Shree K Nayar. Attribute and simile classifiers for face verification. In 2009 IEEE 12th international conference on computer vision, pp. 365 372. IEEE, 2009.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In 2009 IEFE conference on computer vision and pattern recognition, pp. 551-958. TEFF, 2009.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi- Velez, and Weiwei Pan. Promises and pitfalls of black-box concept. learning models. ai preprint urXi::2066 13314, 2021.
Andrei Margeloiu, Malthew Ashman, Umang Bhatt, Yanzhi Chen, Maleja Jamnik, and Adrian Weller. Do concept bottleneck models learn as intended? arXiv preprint arXiv:2105. 04289, 2021.
George A Miller, Richard Beckwith, Christiane Fellbaum, Derek Gross, and Katherine Miller. Introduction to wordnet: An on-line lexical database. International journal of lepicography, 3(4):333 1990.
Ravi Teja Mullapudi, William R Mark, Noam Shazeer, and Kayvon Fatahalian. Hydranets: Specialized dynamic architcctures for efficient inference. In Proceedings of the IEEE conference on computer vision and poltern recognition, pp. 8080 8089, 2018.
Calvin Murdock, Zhen Li, Howard Zhou, and Tom Duerig. Blockout: Dynamic model selection for hierarchical deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2583-2591, 2016.
Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. arXiv preprint orXic:2304.0011 06129, 2023.
Enrico Parisini, Tapabrata Chakraborti, Chris Harbron, Ben D MacArthur, and Christopher RS Banerji. Leakage and interpretability in concept-based models. arXie preprint urXiv:2504.1409 2025.
Federico Pittino, Vesna Dimitrievska, and Rudolf Heer. Hierarchical concept bottleneck models for vision and their application to explainable fine classification and tracking. Engineering Applications of Artificial Intelligence, 118:105674, 2023. doi:
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748 8763. PMLR, 2021.
Angelos Ragkousis and Sonali Parbhoo. based leakage inspection and control in concept bottleneck models. arXiv preprint arXiv:2410. 063522, 2024.
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7263-7271, 2017.
David M. Rodríguez, Manuel P. Cuéllar, and Diego P. Morales. On the fusion of soft-decision-trees and concept-based models. Applied Soft Computing, 160:111632, 2024. doi: 10.1066/aaso0222211166...
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instcad. Nature machine intelligence, 1(5):206-215, 2019.
Simon Schrodi, Julian Schur, Max Argus, and Thomas Brox. Concept bottleneck models without predefined concepts. arXiv preprint arXiv:2407.00 039221, 2024.
Chenming Shang, Shiji Zhou, Hengyuan Zhang, Xinzhe Ni, Yujiu Yang, and Yuwang Wang. Incremental residual concept bottleneck models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11030 11040, 2024.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. A dataset of 101 human action classes from vidoos in the wild. Center for Research in Computer Vision, 2(11):1-7, 2012.
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilinuual graph of general knowledge. In Proceedings of the AAAÍ conference on artificial intelligence, volume 31, 2017.
Divyansh Srivastava, Ge Yan, and Lily Weng. Vlg-cbm: Training concept bottleneck models with vision- language guidance. Advances in Neural Information Processing Systems, 37:79057-7999 -79994, 2024.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Adam Stein, Aaditya Naik, Yinjun Wu, Mayur Naik, and Eric Wong. Towards compositionality in concept learning. ar arxiv preprint orXiv:22000 1853,, 2024.
David Steinmann, Wolfgang Stammer, Felix Friedrich, and Kristian Kersting. Learning to intervene on concept bottlenccks. arviv preprint arXic:2308.1944 2023.
Ao Sun, Yuanyuan Yuan, Pingchuan Ma, and Shuai Wang. Eliminating information leakage in hard concept bottleneck models with supervised, hierarchical concept learning. ar preprinl urXiv:2402.050) 2024.
Moritz Vandenhirtz, Sonia Laguna, Ričards Marcinkevits, and Julia E. Vogt. Stochastic concept bottleneck models. In Advances in Neural Information Processing Systems (NeurIPS) 2024, 2024.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 datasct. 2011.
Alvin Wan, Lisa Dunlap, Daniel Ho, Jihan Yin, Scott Lee, Henry Jin, Suzanne Petryk, Sarah Adel Bargal, and Joseph E Gonzalez. Nbdt: Neural-backer decision trees. arXiv preprint arXiv:2004.00221 2020.
Joe H Ward Jr. Hierarchical grouping to optimize objective function. Journal of the American statistical association, 58(301):236 244, 1963.
An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp.
2023.
Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in bottle: Language model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 19187-19197,
2023.
Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. ar preprinl arXiv:2205.15400 2022.
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Nocl Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16793 16793-16803, 2022.
Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B: Statisticul Methodology, (7))3330 320, 2005.
A Implementation Details
A.1 Tree hierarchy extraction
Tree generation. We implement multiple strategies to generate the decision tree structure T:
We first compute class centroids by averaging CLIP image embeddings of training instances per class. Then, we apply hicrarchical clustering (Ward's linkage (Ward Jr 1963) on these centroids to obtain the linkage matrix Z that defines the merge structure.
Data-induced tree:
Weights-induced tree: Instead of computing class centroids from data, we use the learned linear classifier weights from a trained linear probe as the class representatives, and perform hierarchical clustering on them.
Random tree: We generate entirely random vectors (matching the dimensionality of the class centroids) and perform hierarchical clustering on these vectors to create purely random hierarchy.
16

---

Published in Transactions on Machine Learning Research (02/2026)
After obtaining Z, build the tree bottom-up. Each merge in Z creates new internal node by combining two child nodes. To avoid excessive merging of semantically distinct classes, we prune internal nodes whose merge distances fall below a pruning threshold т, defined as a specific percentile (e.g., the 30th percentile) of all distances in Z. This pruning step produces a multi-branch hicrarchy with internal nodes representing semantically coherent groups. For controlled experiments comparing with other CBMs, we prune the tree hierarchy using a threshold т, set to 0.6 for CIFAR-IO, 0.96 for CIFAR-100, 0.98 for CUB-200 and Tiny ImageNet, and 0.96 for UCF101, in order produce a tree with exactly four internal nodes and maximum depth of four.
The tree is implemented using TreeNode structure, maintaining references to children, parent, node classes, and computed centroids. We ensure structural consistency by post-order traversal pruning and update sibling relationships and levels for downstream processing.
Tree annotation. We annotate internal nodes to improve semantic interpretabilily. This process involves:
Prompt construction: For each non-leaf node, collect the set of leaf class names in its subtree. We design a prompt that asks large language model (LLM) to summarize the set into concise cluster names. The prompt format is as follows:
You are an experienced taxonomist analyzing the dataset pestaset_name]) Given cul__trrsrrs clusters, each containing the following classes:
(Cluster 0: [class0, classl, ...], Cluster 1: [...], ....)
Your task to identify common characteristica and assign ahort, clear, exclusive name to each cluster.
Return the result as JSON: { 'cluster_O'; 'name0', 'cluster_1'; 'namel',
LLM querying: We use either GPT-4.1 from OpenAl (by default) or alternative LLMs (c.g., DeepSeek) to generate names. Responses are parsed to ensure valid JSON format and semantic consistency. We attempt multiple retries (set by default) if the output parsing fails.
Node naming: Each internal node assigns its generated name to its immediate children, thereby incrementally annotating the hierarchy during breadth-first traversal.
A.2 Candidate concept generation
Across the experiments, we generate three types of concepts: random concepts for non-semantic baselines, existing concepts from Labo (Yang et al.) 2023 to enable controlled comparisons between CFM and prior CBMs, and I.I.-generated concepts for visualization and interpretation of CFMs.
Random concepts. To establish a random baseline, we generate a large pool of purely random phrases independent the dataset or hierarchy structure.
The random concept generation process procoods as follows:
Word list collection: We first download an English word list containing approximately 500,000 unique words from public repository
Phrase construction: We randomly sample phrases by concatenating between 1 and 5 randomly selected words. Fach phrase forms simple noun-like structure (e.g., river bright stone, glowing arc monument), introducing linguistic randomness.
Storage: We generate a fixed number (e.g., 30,000) of such random phrases and store them in a JSON file for reuse across experiments. Each phrase is indexed by unique integer key.
Phrase sampling during training: During training or evaluation, random phrases are drawn from the pre-generated pool to simulate random concept sets.
17

---

Published in Transactions on Machine Learning Research (02/2026)
This random phrase generation pipeline provides controlled method to assess the model's robustness against semantically meaningless features, serving as baseline when evaluating the effectiveness of selected concepts from the existing or LLM-generated concept pool.
Existing concepts. We reuse pre-existing MMM-gennrated concepts to ensure fair baseline model comparison. Specifically, we leverage the concepts provided by LaBo (Yang et al. 2023). which are publicly available
The concept loading process proceeds follows:
Concept file retrieval: For each dataset, we access the corresponding JSON file containing class- to-concept mappings, where each class is associated with list of LLM-generated visual concepts.
Concept extraction: For each internal node, we ilerate through its leaf classes and collect all concepts associated with the classes from the pre-loaded mappings.
This procedure enables experiments using high-quality, curated concept sets without requiring costly LLM queries.
LLM-generated concepts We generate candidate concepts per leaf class and only aggregate them internal nodes afterwards. Concretely, we first run a per-class concept discovery stage and then construct nodo-level candidate pools by unioning concepts from descendant leaves.
Per-class discovery (multimodal prompts). For each leaf class c, sample up to max max_images__pp class training images and group them into batches of size images_pezz perpppppe_ppoopttt (discarding groups smaller than min sarages_per_grort image batch (as base61-encoded PNGs) before a textual prompt that includes: the target class name, its parent-chain, its sibling classes, and the set of concepts already discovered for c (to discourage repetition). The prompt also enforces constraints on the number of parent-consistent and traits and a maximum phrase length.
Each request to the LLM attaches the
subling-distingiise
LLM querying and parsing. We query an OpenAI chat model (default: GPT-4.1-ming configurable via model __alllnnammmaaaeeeee with constant backoff and at most max_
The model returns short. noun phrases; we parse the response line-wise, strip numbering, enforce max_wordss perrd__pr____ _as___ee____oeee concept limit, and keep only novel items relative to the per-class memory set. We iterate image batches until reaching max_ pox__conepts____e _pertts__pr_______ sactseereersssseeeee class or exhausting data, with an api_call_dee dellly between calls.
Caching and resumption. Concepts discovered for each class are saved incrementally to JSON file (<dataset>> _bbackbone>_______eeeeeaaaaaee ___1___m_____________ concepts..jeeeeeeo When resume is enabled, previously completed classes are skipped and generation continues for the remaining classes.
Bottom-up aggregation at internal nodes. After per-class discovery, we traverse internal nodes in a bottom-up order and construct each node's candidate pool by aggregating the concepts of its descendant leaf classes. We also retain the originating class index for each concept to preserve the linkage between concepts and classes. No additional LLM calls are made this stage.
This two-stage design grounds concepts in real images for cach class while leveraging hicrarchical context: (parents and siblings) to promote class-specificity and sibling contrast. The subsequent bottom-up aggrega- tion yields node-local candidate pools without extra LLM cost, supporting fine- fine-grained grained classification and interpretable, hicrarchy-aware decisions.
A.3 Concept preprocessing
Before using LLM-generated candidate concepts for concept selection, we apply two preprocessing steps to ensure quality and semantic diversity:
18

---

Published in Transactions on Machine Learning Research (02/2026)
Removal of concepts containing class names. To prevent data leakage and trivial shortcuts, filter out candidate concepts that contain any class names substrings thereof. For each candidate concept e, we perform a case-insensitiv regular expression search against all class names. Concepts matching any class name pattern are discarded.
Removal of semantically redundant concepts. similar concepts within each node. Specifically, we:
To encourage concept diversity, we climinate highly
Encode candidate concepts into CLIP text embeddings using the backbone model.
Normalize all embeddings and compute pairwise cosine similarities.
For each concept, remove subsequent concepts whose cosine similarity exceeds a predefined threshold 0 (We set 0 = 0.9 for all experiments).
This redundancy removal is performed in a greedy, batch-efficient manner to minimize memory usage. We prioritize keeping carlier concepts when conflicts arise and explicitly free unused tensors reduce CPU memory overhead.
Overall, this two-stage cleaning process ensures that the remaining candidate concepts for each internal node are both distinct and informative, leading to better semantic coverage and improving interpretability in later concept selection stages.
A.4 Concept distribution
To determine the concept budget Tv for each internal node v, we employ a hybrid allocation strategy that combines fixed base share with an adjustment based on the node's child count:
Tv = max a [Child(v) + (1 - a)) [B], rmin),
v Vint
(11)
Here, B denotes the mean (base) share when N concepts are distributed across m internal nodes. The coefficients E [0, 1] and (1 (1-a) a) balance the influence of the chigh-weighted term and the uniform bascline, respectively. Tmin enforces minimum number of concepts allocated per node. Further implementation details are provided in the supplementary code.
Hyper-parameters.
a: weight of the child-count term (default 0.2).
mmin: minimum number of concepts per node (default 1).
We fix Tmin for all experiments. The child-weighted coefficient a is tuned from 0 to 1 in increments of 0.1. The final settings are chosen as follows: a = 0.6 for CIFAR-10, CIFAR-100, and Tiny ImageNet; a = 0.9 for UCF-101; and a = 0.8 for CUB-200.
A.5 Concept selection
After generating and preprocessing candidate concepts for each internal node, we apply a concept selection step to choose compact subset of informative concepts. The selection process is organized follows:
Multi-class data preparation. For each internal node, we collect training examples corresponding to the child classes of that node. We construct multi-class prediction task where the goal to predict the child class from instance features, enabling quantitative evaluation of concept quality.
19

---

Published in Transactions on Machine Learning Research (02/2026)
Selection strategies. We implement five different strategies to select subset of concepts:
Lasso selection:
We represent each sample as a vector of cosine similarities to candidate concept. embeddings. We then apply fi-penalized logistic regression (Lasso) with the following settings for all experiments: penalty='11',
multimclaas='oree class='ov'', and max iter=1000. The resulting sparse model identifies a compact set of concepts that are predictive of child classes. We select. the top-r ranked concepts, where is the desired number of concepts per node.
Similarity selection: For each concept, we compute its average cosine similarity to all training samples. Concepts with the highest average similarity scores are selected.
Orthogonality selection: We use facility location-based submodular optimization approach to select the top-r concepts that are maximally diverse (i.e., mutually orthogonal in embedding space).
Random selection: Concepts are randomly shuffled and the top-r concepts are selected a baseline.
We use a mutual informattion-augmeenter submodular optimization method, balancing concept relevance and diversity. This approach follows the procedure outlined LaBo (Yang et al., 2023) and operates by selecting concepts that maximize a mixture of informativeness and diversity scores.
Submodular selection:
A.6 Hierarchical loss
We add a hierarchical auxiliary loss to supervise routing decisions at internal nodes of the tree. At each internal node, the model predicts probability distribution over its children based on the imput embedding. We apply cross-entropy loss between the predicted distribution and the ground-truth child node, determinod by the path leading the correct leaf class. The hierarchical loss is accumulated along the path from the root to the leaf and added to the main classification loss during training. This auxiliary supervision encourages more accurate intermediate decisions but can slightly degrade overall performance, consistent with obscrvations from prior work (Wan et al. 2020
20

---

Published in Transactions on Machine Learning Research (02/2026)
B Experiment details
B.1 Dataset statistics
Table sumnariies the key statistics of the datasets used in our experiments, including the number of classes and the number of samples in the training, validation, and lest sets. For datasets without predefined validation splits, we randomly split the training set into 80% for training and 20% for validation.
Table 5: Datasct statistics.
Datasct
# Classes
Train
Validation
Test
CIFAR-10
10
40,000
10,000
10,000
CIFAR-100
100
40,000
10,000
10,000
CUB-200-2000
200
4,795
1,199
5,794
Tiny ImageNet
200
80,000
20,000
10,000
UCF101 (processed)
101
7,639
1,898
3,783
CIFAR-10 and CIFAR-100 have predefined 50,000 training and 10,000 test images. We split the original training set into 80% training and 20% validation.
CUB-200-2011 consists of 11,788 images across 200 bird species, with a standard split of 5,994 training and 5,794 test images. We split the training set into 80% training and 20% validation.
Tiny ImageNet provides 100,000 training images and 10,000 test images across 200 classes. We split the original training set into 80% training and 20% validation.
UCF101 refers to the processed version released by Yang et al. (023 based mid-frames from videos, available at https://drive http://rvvve.oogllee...o.ooo.o/e
We split the training set into 80% training and 20% validation.
References:
CIFAR-10 and CIFAR-100:
CUB-200-2011: http:/0001..ttrr/////.....so.o..
Tiny ImageNet: https///ww..agggaa...//////aa//p
UCF101
(processed
version):
B.2
Experiment settings
Our experiments are conducted on a system equipped with Intel Xeon E5-2640 v4 CPUs (20 cores, 40 threads) operating at 2.40 GHz, 503 GiB of RAM, and an NVIDIA RTX A6000 GPU with approximately 49 GiB of VRAM. All data are processed using the CL.P VIT-B/32 (Radford al. 2021) backbone (unless otherwise specified) and cached to accelerate model training. Two worker threads are used for data processing, For all models, train only the linear layers based on the cached, featurized data. Constructing Concept Flow Model, including hierarchy generation, concept selection, and training, takes between 0.1 and 5 hours depending on the datasct. All hyperparameter settings are provided in Sec. A with additional details for Linear Probe, PCBM, and Labo in the supplementary code and YAML configuration files. We do not explicitly include NBDT et al. 2020) as a baseline for comparison, as was not designed to support concept-level explanations. However, our ablation study Sec. 5.4 which removes the concept matrix, results model that is approximately equivalent NBDT and achieves performance comparable the non-interpretable linear probe baseline.
21

---

Published in Transactions on Machine Learning Research (02/2026)
Training setup. All models are trained using the Adam optimizer with initial learning rate of 0.01. We use mini-hatch size of 128 (unless otherwise specified) and train for up to 200 epochs with early stopping based on validation loss. If the validation loss does not improve by at least 0.0001 for 5 consecutive epochs, training is terminated early. The learning rate reduced by a factor of 0.5 the validation loss plateaus for epochs, following a ReduceLLROn
Plateau Plateea schedule.
All experiments are implemented in PyTorrc and executed on CUDA-enabled devices. Random seeds are fixed across data splits, concept processing and model initialization to ensure reproducibility. Concept generation, selection, training, and evaluation pipelines are fully automated and configurable via YAML configuration files.
B.3
Experiment with Elastic Net regularization
Test Accuracy (96
Test (%)
60
40
40
20
Probe
20
Linear Probe
CBM
CBM
(per
(total)
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0,4
0.6
0.8
1.0
Sparsity
Sparsity
(a) Accuracy vs. per-node sparsity
(b) Accuracy total sparsity
Figure 6: Impact of Elastic Net regularization-indues sparsity accuracy. (Left) Accuracy trends for the first experimental group, where CFM (4000 total concepts, 1000 per node) is initialized with four times more random concepts than CBM (1000 total concepts). As the number of effective concepts per node (CFM) or layer (CBM) progressively reduced, both models exhibit similar accuracy trends. (Right) Accuracy trends for the second experimental group, where CBM and CFM are initialized with the same total number of random concepts (4000).
Srivastava et al. 2024 Yuksekgonul et al., 2022) have applied Elastic Net regularization ZZu & Hastie 2005 which combines and le penalties the concept-to-class linear layer in concept- based models to control the number of effective concepts and improve robustness. However, noted by Srriasttval et al. 2024 tuning the regularization strength to target specific number of effective concepts can be labor-intensive and dattaset-dependdent. For this reason, omit Elastic Net regularization in our controlled experiments (Sec. 5.2 and instead directly control the number of concepts.
Previous works
Nonethcless, we conduct additional experiments hore to validate that Elastic Net regularization can achieve similar effects. Specifically, we Irain CBMs and CFMs on CIFAR-10 using purely random concepts, applying Elastic Net regularization induce the sparsity of the linear layers.
Following Yuksekgomul et al. (022). we define the regularization objective
À
?(w),
N.K
where N. is the number of concepts, K is the number of classes, À controls regularization strength, and 9(w) is the Elastic Net penalty:
?(w) = //|l/]] + (1 - w
We fix the mixing ratio a 0.9 to emphasize sparsity through l regularization. We consider two experimental groups: (1) CBM is initialized with 1000 concepts, and CFM is initialized with 1000 concepts per internal
22

---

Published in Transactions on Machine Learning Research (02/2026)
node (resulting in 4000 total concepts across 4 nodes); (2) CBM is initialized with 4000 concepts, while CFM maintains 1000 concepts per node (4000 total concepts). In both setups, measure sparsity as the proportion of near-zero weights ([wy) < 10 in the concept weight matrices. We vary the regularization strength across 20 logarithmically spaced values between Amin 0.01 and Amax = 20.0 to achieve a range of larget sparsity levels for comparison.
Figure (left) shows the results for the first experimental group. As the number of effective concepts decreases (i.e., sparsity increases), the accuracy of both CBM and CFM declines in similar manner, mirroring the trends observed when directly reducing/inan 'incressing the number of concepts (see Fig. 3). In contrast, Figure B (right) presents the second experimental group, where CBM and CFM are initialized with the same total number of concepts (4000). In this setting, CFM experiences an earlier and sharper accuracy drop as sparsity increases. This is because the smaller number of random concepts per node CFM limits each node's discriminative capacity, making harder to maintain high classification accuracy compared CBM.
For convenience, omit Elastic Net regularization in the main paper's controlled experiments to enable precise control over concept count and facilitate comparison with Labo and PCBM. However, in practical applications, recommend using Elastic Net regularization in CFM, particularly when node contains large number of concepts (e.g., CUB dataset with 100 concepts per node).
B.4 Detailed Experimental Results for Scenario
Table 6: Detailed results for Figure
All values are accuracy (%) reported as mean + std over three runs.
(a) Fixed total budget R
(b) Fixed bottleneck concepts
CFM
CBM
R
CFM
CBM
8
30.11 + 3.77
63.46 + 3.95
2
30.11 + 3.77
30.79 + 2.98
16
41.90 + 5.15
72.02 + 2.01
4
41.90 + 5.15
42.20 + 4.98
32
60.76 0.48
84.01 0.55
8
60.76 0.48
56.14 + 0.06
64
1.68
89.78 + 0.26
16
75.95 3.68
76.86 + 2.73
128
91.26 0.77
32
0.78
85.49 = 0.86
256
38.94
92.25 +
64
88.94
89.68 0.06
512
90.39 1.41
92.63 0.06
128
90.39
91.04 = 0.07
256
90.63 - 1.86
92.25 = 0.12
512
90.33 + 1.51
92.63 + 0.06
(c) CFM pruning (R = 60)
(d) CBM sparsity (R = 60)
Inner Nodes
Accuracy
Accuracy
88.41 0.78
0.1
84.53 0.90
2
85.95 + 0.88
0.2
81.37 + 1.46
3
80.76 + 1.41
0.3
75.84 + 1.33
4
71.86 + 3.96
0.4
74.00 + 0.29
5
71.76 3.30
0.5
70.02 + 1.31
67.33 4.03
0.6
56.36 - 13.07
61.97
0.7
56.02 9.35
61.88 2.65
0.8
37.29 4.73
54.05 + 1.56
0.9
38.75 9.42
23

---

Published in Transactions on Machine Learning Research (02/2026)
B.5 Detailed Experimental Results for Scenario 2
Table 7: Full experimental results with standard deviations over runs. NEC: number of effective concepts;
Acc: classification accuracy (%); SIR: semantic improvement over random concepts (%)
Dataset
Method
NEC
Acc (%)
SIR (%)
Linear Probe
N/A
94.59110.16
N/A
CBM (Random)
56.80110.82
89.20111.18
0
CFM (Random)
6.90++0.00
76.46+0.82
0
CIFAR-10
PCBM
56.80=0.52
91.15++0110
2.19
Labo
57.73:10.15
90.93:10.12
1.94
CFM (Ours)
6.9010.10
9..5
19.78
Linear Probe
N/A
77.53+0.34
N/A
CBM (Random)
184.20=:0.72
68.50+11.2
0
CFM (Random)
54.24+#1.11
62.00+1.11
CIFAR-100
PCBM
183.99110.13
68.98111.69
0.70
Labo
184.33110.51
70.9810.67
3.62
CFM (Ours)
55.02+88.3
72.29=0.02
16.61
Linear Probe
N/A
95.00110.38
N/A
CBM (Random)
187.31+1.99
19.15+1.38
0
CFM (Random)
44.99+1.79
72.76+22.23
0
UCF101
PCBM
N/A
N/A
N/A
Labo
190.60::0.88
84.62:10.44
6.91
CFM (Ours)
45.80+1.25
84.60+0.21
16.27
Linear Probe
N/A
71.68+0.60
N/A
CBM (Random)
344.19- 19-1199-8... .49
52.2342.00
0
CFM (Random)
10461112.447
47.08.12.22
0
CUB200
PCBM
N/A
N/A
N/A
Labo
357.92:10.7
13.00.3
24.70
CFM (Ours)
109.34+13.13
65.68-10.51
39.50
Linear Probe
N/A
75.25
N/A
CBM (Random)
362.31113.21
68.78--0.71
0
CFM (Random)
89.83:10.16
61.89-10.59
0
ImagenteeNet
PCBM
362.01+0..43
71.07+0.31
3.32
Labo
369.36+00.57
09.86+00.3
1.57
CFM (Ours)
92.12+4.71
70.79=1.28
9.09
B.6 Comparison of CFM and PCBM with Sparsity Regularization
We conduct additional experiments comparing CFM and PCBM with and without sparsity regularization. While the main experiments in Section 5 disable sparsity regularization to isolate the effect CFM's structural design from rogularization-innueee sparsity, we demonstrate here that CFM's advantages persist, when sparsity regularization applied to both methods.
Experimental Setup. We compare PCBM and CFM on CIFAR-10 under two conditions: (1) without sparsity regularization, (2) with sparsity regularization at matched NEC levels 5 and 3. Following the original PCBM implementation (Yuksekgonul et al.] 22022). we set the L1 ratio 0.99 and tune the regularization strength À to achieve the target NEC values. This methodology follows VL.G-CMM (Srivastaval 2024), which emphasizes that fair comparison requires matching effective concept counts across methods.
et al.
Results. Table presents the results. Without sparsity regularization, CFM achieves comparable accuracy PCBM (91.68% vs. 91.36%) while using substantially fewer effective concepts (7 vs. 57). When sparsity regularization is applied to match NEC values, CFM significantly outperforms PCBM: NEC=5, CFM
24

---

Published in Transactions on Machine Learning Research (02/2026)
achieves 90.66% accuracy compared to PCBM's 85.85% (+4.81%); at NEC=3, CFM maintains 90.61% accuracy while PCBM drops to 84.41% (+6.20%).
Table 8: Comparison of PCBM and CFM with and without sparsity regularization on CTFAR-10. For experiments with regularization, is tuned achieve matched NEC values. Results demonstrate that CFM's structural sparsity complements sparsity regularization effectively.
Method
Sparsity Reg.
NEC
Accuracy (%)
PCBM
x
57
91.36
CFM X
7
91.68
PCBM
5
85.85
0.8
CFM
/
5
90.66
0.02
PCBM
3
84.41
1.3
CFM
90.61
0.1
These results demonstrate three key findings: (1) without sparsity regularization, CFM achieves comparable accuracy with substantially fewer effective concepts, validating Proposition 1.2 (2) with sparsity regularization at matched NEC levels, CFM significantly outperforms PCBM, indicating that structural sparsity provides benefits beyond what regularization alone can achieve; and (3) CFM's structural sparsity and sparsity regularization can be effectively combined. CFM requires much smaller À values to achieve the same NEC, suggesting that its hierarchical structure already enforces substantial concept isolation.
B.7 Qualitative Analysis of Decision Paths
To complement the quantitative evaluation in Section 5 we present qualitative analyses of CFM decision paths. We select four samples across three datasets illustrate both successful and unsuccessful predictions: three correctly classified cases demonstrating different levels of concept-image correspondence, and one failure case to examine CFM's diagnostic capabilities. All models use the CLIP ViT-L/14 backbone with candidate concepts generated by GPT-4.1-mini.
Case-by-Case Observations. Table 9 illustrates how CFM produces traceable decision paths, where each prediction follows unique root-to-leaf trajectory through localized concept subsets.
(a) Bowling. With only two transitions, the model routes from "Root" " to "Sports and Athlotic Events" (99.99%) using motion-related concepts such as "ball trajectory on lane" (0.25), then to "bowling" (99.91%) via task-specific concepts including "ten-pin setup" (0.30) and "bowler's side-on stance" (0.21). However, "colored cue ball markings" (0.16) which refers to billiards rather than bowling illustrates that not all activated concepts are semantically correct, even in successful predictions.
(b) Painted Bunting. The three-level hierarchy progressively refines the prediction: generic bird features ("songbird vocal sac presence", "songbird wing feather pattern") at the first level, visual characteristics ("large stout bill", "glossy black back feathers") at the second, and species-level attributes at the leaf. The final transition activates two concepts, i.e., "iridescent green wing coverts" and "iridescent golden-green wing coverts", both referencing green coloration. While painted buntings do have greenish back feathers, the most visually prominent features in Figure 7b (blue head, red breast) are not captured by the selected concepts.
(c) Parking Meter (Failure Case). This misclassification illustrates how CFM's decision path can expose prediction uncortaint. At the second transition, the model shows only 69.24% confidence for "Everyday Objects & Apparel" versus 30.76% for "Structures & Vehicles", which is a notably uncertain split. The concepts activated throughout the path ("rows of stop knobs", "tactical utility belt", "graduated fluid chamber") show little semantic correspondence the input image. The resulting path probability of 68 68.69% is clearly lower than the correctly classified samples before. TinylmageNet's low resolution (64x64) may further limit CLIP's ability to extract discriminative visual features.
25

---

Published in Transactions on Machine Learning Research (02/2026)
(a) Bowling (01)-101): Correct
(b) Painted Bunting (CUB-200) Correct
(e) Reel (TinyImageNet): Misclessified Parking Meter
(d) Clean and Jerk (UCF-101): Correct
Figure 7: Input samples for qualitative analysis. We analyze CFM decision paths for four cases across three datasets. Detailed decision paths with concept activations are provided in Table 9
(d) Clean and Jerk: The decision path transitions through "Personal Actions and Instrument Play" + "Physical Activities" + "clean and jerk". Upper-level concepts exhibit grounding issues: "standing musician posture" and "child-sized cleaning tool" do not correspond to the image. The latter may arise from spurious linguistic correlation with "clean" in the class name. In contrast, leaf-level concepts such as "weight plates visible barbell" (0.28) and "weightlifting platform mat." (0.19) directly match salient visual elements. In this case, accurate prediction occurred despite poorly aligned upper-level concepts, suggesting that leaf-level concept quality may be more critical for final classification.
Observations on Interpretability. These examples highlight both the utility and limitations of CFM's decision paths.
Traceability. CFM's stepwise structure makes prediction reasoning explicit. In the parking meter failure case, the uncertain routing at the second transition (69.24% 30.76%) and the semantically misaligned concepts provide interpretable signals of an unreliable prediction. This contrasts with flat CBMs, where all concepts contribute to single linear combination, making it harder to identify which semantic distinctions caused an error.
Concept alignment limitations. Even correctly classified samples, some activated concopts are semantically incorrect ("colored cue ball markings" for bowling) or fail to capture visually salient features (green wing coverts for bird with prominent blue and red coloring). These observations align with limitations of
26

---

Published in Transactions on Machine Learning Research (02/2026)
Table 9: Decision paths and concept activations for samples in Figure 7 Each row shows transition, the branch probability, and the top activated concepts with weights (up to per transition). Unlike flat CBMs that expose all concepts to every class prediction, CFM restricts each transition localized concept subset. Path Prob. denotes the cumulative product of transition probabilities.
Sample
Transition
Prob.
Top Activated Concepts (weight)
ball trajectory on lane (0.25)
(a) Bowling
Root + Sports and Athletic Events
99.99%
spherical ball kicked by player (0.19)
(UCF-101)
rider's legs bent at knees (0.18)
Path Prob: 99.90%
ten-pin sctup (0.30)
Sports and Athletic Events + howling
99.91 %
bowler's side-on stance (0.21)
colored ene ball markings (0.16)
songbird vocal sac presence (0.20)
Root + Songbirdss & Passerines
100.00%
songbird wing leather pattern (0.14)
(b) Painted Bunting (CUB-200)
bright chestnut throat (0.14)
large stout bill (0.22)
Songhirds & Passerines
Path Prob: 98.96%
Colorful and Varied Perching Birds
98.99% glossy black back feathers (0.21)
glossy black wings and back (0.18)
Colorful Varied Perching Birds
99.97% iridescent green wing coverts (U.16)
+ painted bunting
irideseent golden-green wing coverts (U.16)
rows stop knobs (0.41)
Root + Objects & Structures
100.00% series of tall vertical piers (0.39)
(c) Parking Meter (TinymmagNee) Pred: reel
waist-level fit (0.36)
Objects & Structures
tactical utility belt (0.22)
Everyday Objects & Apparel (va. Structures & Vehicles 30.76%)
69.24%
graduated fluid chamber (0.18)
Path Prob: 68.69%
side-tied bottom strings (0.17)
designed for manual operation (0.43)
Everyday Objects & Apparel reel
99.21 %
stick extending lengthwise (0.31)
tapered cooking depth (0.24)
standing musician posture (0.16)
Root +
Personal Actions and Instrument Play
98.85%
padded weightlifting bench support (0.15)
child-sized cleaning tool (0.15)
(d) Clean and (UCF-101)
waist-level prop manipulation (0.15)
Personal Actions and Instrument Play Physical Activities
100.00%
climbing-specffic grip technique (0.15)
Path Prob: 98.77%
elimbing shoes with rubher soles (0.14)
weight plates visible on barbell (0.28)
Physical Activities + clean and jerk
99.92%
weightlifting platform mat (0.19)
weightlifting barbell with weighted plates (0.18)
CLiP-based concept aligument noted prior work (Oikarinen et al. 2023 Yang et al. 2023 indicating that concept pool quality remains important for explanation fidelity regardless of the model architecture.
B.8 ImageNet-1K Experiments
To validate the scalability of CFM to large-scale datasets, we conduct experiments on ImageNet- eKK,,, which contains 1000 classes and over 1.2 million training images.
Scalability via Stratified Sampling. The computational bottleneck ariscs from computing the similarity matrix S, E Bxxx and solving Lasso regression over all n samples during concept selection. For ImageNet-1K, processing all training samples is prohibitive. We address this via stratified sampling: grouping samples by class and drawing a balancod subset with Rlaax 15,000 samples. Since concept selection requires representative samples rather than exhaustive coverage, this preserves class structure while remaining tractable regardless of dataset size.
27

---

Published in Transactions on Machine Learning Research (02/2026)
Experimental Setup. We use CLIP ViT-B/32 the backbone (512-dimensional embeddings) with total budget of 1000 concepts (one per class). CFM is configured with 5 internal nodes and 3 levels. All other settings follow Section 5.1. We report mean and standard deviation over 3 runs with different random seeds.
Results. Table 10 summarizes the results. CFM achieves competitive accuracy (72.88%) with significantly fewer effective concepts (339 vs. ~860). With 1000 concepts exceeding the 512-dimensional embedding, flat CBMs exhibit severe information leakage. CBM (Random), PCBM, and Labo all fall within 0.25% of Linear Probe, with near-zero SIR. confirming that random concepts suffice when the concept. count exceeds the embedding dimension. In contrast, CFM limits per-node concepts below the embedding dimension through its hierarchical structure, yielding substantially higher SIR. (4.70% and validating hicrarchical bottlenecks' resistance to information leakage at scale.
Table 10: Comparison of methods on ImageNet-IK. We report the number of effective concepts (NEC), classification accuracy (Acc), and semantic improvement over random concepts (SIR). Results mean + std over 3 runs. Bold values indicate best performance among concept-based methods.
Method
NEC
Acc (%)
SIR (%)
Linear Probe
N/A
73.03 + 0.63
N/A
CBM (Random)
853.92 + 16.70
72.82
0.43
CFM (Random)
260.10 1.62
69.61 0.23
PCBM
860.44 8.18
73.08
0.76
0.37 + 1.51
Labo
868.31 0.69
72.86
0.55
0.06 + 0.42
CFM (Ours)
339.18 = 1.12
72.88
0.56
4.70
0.58
C Theoretical Analysis
C.1 Computational Complexity Analysis
We compare CBMs and CFMs under the same concept budget R and assume modern GPUs can parallelize independent nodes at the same tree level; different levels are computed sequentially (root - leaves).
Notation. d: CLIP embedding dimension; k: number of classes (lcaf nodes); number of training samples; R: total number of concepts; m: number of internal nodes in CFM; b: average branching factor; bmaxi maximum branching factor; t: average root-to-leaf path length; r' = R/m: average number of concepts per internal node. In a rooted tree with & leaves and m internal nodes, the sum of the numbers of children over all nodes equals the total number of edges, which + m
C.1.1 Forward Pass Complexity
CBM Complexity.
Concept projection: TT with C Rxd, O(Rd)
Classification: sW with e liik O(Rk)
Total: O(Rd+ + Rk)
CFM Sequential Complexity.
Concept projections across all internal nodes: Σ, Ccll d= Rd ((Rd)
Local branching multiplications: Σ. r' |Ch(u)] R (k: (k+ QR +
28

---

Published in Transactions on Machine Learning Research (02/2026)
Wise softmax + probability propagation: touch each edge once O m)
Rk + R + k m) ORd + Rk Total: a(R
+ m)
m
CFM Parallel Complexity (level-wise parallelism). Assuming perfect parallelization across nodes the same depth, the per-lovel cost is O Rd m²) (concept projections plus local branching). Over e levels and assembling leaf scores:
m
o(e(m Rd + m)) + ((bmmax)
For >> b, this is well-approximated by Oee Rd). m
C.1.2 Comparison Under Different Scenarios
Inference (all-class logits). It is standard to compute all class logits once and take arg max.
CBM: O(Rd + Rk)
CFM (parallel); of Rd
Rb
+ (ebmax)
Training (negative log- log-likelihood). With loss - Pm CFM only evaluates the unique path to the ground-truth leaf y:
CFM (per sample):
o(E(1) Rtd m
(bb)
(EEb) (lower-order),
CFM (datasct): One en Rd mb) m)).
m
CBM (all-class logits) remains ((n(Rd= Rk)).
C.1.3 When is CFM faster?
A practical rule from the above expressions:
Rd
Rb
Ibmax
< Rd + Rk.
For typical regimes with d > this simplifies to Rd is not tiny, CFM often has an advantage.
f < 3. Thus, once m > and k Rd + Rk, i.e. m
The path aggregation work in CFM O(k + m) (each edge touched once), while its parallel time contribution is (fbmmax). Both much smaller than projection pegjection/muuiippiccaaat terms. The main limiter for speedups in practice is that many small matrix multiplies may utilize GPUs less efficiently than few large ones.
Bottlenecks.
Example. R=512, d=768, 1000 m=50, b=33 L= loga 100]=5:
CBM: Rd + Rk 393,216 51, 200 = 444,416
CFM (parallel): { Rd
+
n2h m R (7,864 31) ~ 39,475 (the extra lbmax is negligible here)
Thcoretical speedup = 11.3x (in practice smaller due to kernel efficiency).
Early stopping (inference). CFM can roduce effective work by pruning unlikely branches:
Confidence thresholding: stop expanding once path probability exceeds
Beam search: keep top-B children per node
These heuristics reduce the effective depth/ hothhbbbanchingggg explored and thus the constant factors in the CFM terms.
29


Reviews:
[{"review_id": "PhYPihpWFf", "reviewer": "Reviewer_ZAYm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## **Summary**\nThis paper explores how to mitigate concept leakage in concept-based interpretable models when the concept set is large (particularly when it is larger than the embedding dimensionality). It achieves this by introducing the Concept Flow Model (CFM), a hierarchical concept-based architecture that splits the predictive process into a hierarchy of prediction sub-problems, each mapping concept subsets to intermediate semantic nodes and generating paths that lead to one or more leaf nodes representing output task classes. CFM operates in three steps. First, it uses hierarchical clustering of class-level embeddings to produce a tree whose leaf nodes are task classes and whose intermediate nodes are given human-like interpretations with the aid of a Foundation model. Second, it uses a label-free pipeline to produce concept candidates and splits those concepts across intermediate nodes in the hierarchy by solving a Lasso selection problem at each node to select a sparse set of informative subconcepts. Finally, it learns transition probabilities between intermediate nodes, allowing one to assign probabilities to each path in the tree and to learn the model’s parameters via MLE. This paper shows, both **theoretically** and **empirically**, that CFMs avoid leaking information of concepts outside their selected predictive path and, by doing so, lead to high-performing models that, compared to existing label-free baselines, (1) have a better effective concept size, (2) can achieve better or competitive task accuracies, and (3) can generate human-interpretable hierarchical explanations for their downstream predictions.\n\n## **Strengths And Weaknesses**\n\nThank you so much for submitting this work! I enjoyed reading this paper. Below are what I believe are this paper’s main strengths, followed by what I think are some of its weaknesses:\n\n### Strengths\n\n1. **[Significance, Critical]** Overall, I think the paper’s framing of how to address information leakage via hierarchical modeling, and the learning algorithm that follows, is very interesting and theoretically sound. More importantly, I think the proposed mechanism for generating, naming, and learning transition probabilities in CFM’s intermediate nodes can be easily extended and applied in other fields and directions beyond information leakage. Therefore, I believe this work has the potential to attract not just the concept-based interpretability community but also the neurosymbolic and reasoning communities. \n2. **[Quality, Critical]** The presence of both non-trivial theoretical and empirical results makes this paper’s evaluation very strong. In particular, I appreciate how the experiments were carefully divided and explained, how individual components of the architecture were tested via ablations, and how the paper presents qualitative results at the end. Moreover, I want to mention that I appreciate this paper including a limitations section and an extensive set of appendices that facilitate the reproducibility of the results.\n3. **[Clarity, Major]** The writing is very clear and easy to read. I have some issues with the specific sections that lack breaks in long paragraphs (as described below). Still, aside from that, the paper was very easy to follow (even the theoretical parts, which were written extremely well).\n4. **[Originality, Minor]** The splitting of prediction into n-ary trees that yield different predictive paths using different concept subsets is certainly novel and an interesting way of approaching the issues that arise from leakage when one deals with large concept sets. The only reason I do not mark this as a critical strength is that several related ideas have been explored in the past (and should, at the very least, be discussed in the paper, as I discuss below).\n\n### Weaknesses\n\nIn contrast, I believe the following are some of this work’s limitations:\n\n1. **[Quality, Critical]** As discussed below in my requested changes, I have some trouble with the main results table (Table 1). In particular, I am not entirely convinced that it is fair to compare against existing methods without using those methods as defined by the authors (e.g., using Labo or PCBM without sparsity regularizers). Please see below for specific questions on this matter.\n2. **[Quality, Critical]** Against good conventional scientific practices, there are no error bars included for any of the empirical results. This makes it difficult to judge the significance of any observed differences.\n3. **[Quality, Major]** The qualitative study in the paper lacks depth (there is only a single example across the entire paper, including the appendix).\n4. **[Clarity, Minor]** Some of the writing could be improved by splitting long paragraphs into more processable bits of text. See below for specific feedback on this issue.\n5. **[Originality, Minor]** The idea of exploring non-flat predictive concept-to-label models is not entirely novel. In fact, this has been explored by several previous works (e.g., [1-7] to name only a few). Because of this, this direction is not entirely novel. That being said, I mark this weakness as minor for two reasons: (1) novelty, as long as findings are sound and interesting, is not one of TMLR’s key criteria; and (2) the appoach used is relatively different to those used in the past, although similar ideas have been explored also with leakage as a motivating factor [7].\n\n### References\n\n1. Havasi et al. \"Addressing leakage in concept bottleneck models.\" NeurIPS (2022)\n2. Xu et al. \"Energy-based concept bottleneck models: Unifying prediction, concept intervention, and probabilistic interpretations.\" ICLR (2024).\n3. Raman et al. \"Understanding inter-concept relationships in concept-based models.\" ICML (2024).\n4. Vandenhirtz et al. \"Stochastic concept bottleneck models.\" NeurIPS (2024)\n5. Dominici et al. \"Causal concept graph models: Beyond causal opacity in deep learning.\" ICLR (2025).\n6. Barbiero et al. \"Relational concept bottleneck models.\" NeurIPS (2024).\n7. Ragkousis et al. \"Tree-based leakage inspection and control in concept bottleneck models.\" *arXiv:2410.06352* (2024).", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper discusses both theoretical and empirical results. The theoretical results appear sound and are empirically verified in the first section of the experimentation. The rest of the empirical results then provide clear evidence that the proposed pipeline can learn to accurately model a downstream task of interest while utilizing a relatively small effective concept size and producing helpful hierarchical explanations for its predictions. Moreover, the experimental section includes ablations that provide clear evidence that each component in the proposed method serves its intended purpose within the architecture. See my summary above for further details.", "audience_yn": "Yes", "audience_explanation": "As described in my summary, I believe this paper will be of interest for the XAI/Interpretability communities (due to its use of a concept-unsupervised pipeline for learning hierarchical explanations) and the neurosymbolic/reasoning communities (due to the generality of the proposed algorithm for learning a human-understandable hierarchy of intermediate reasoning steps for a downstream task of interest). Therefore, I believe a large community within TMLR's audience will find this paper interesting.", "requested_changes": "Below, I list some potential changes for each section and their importance in securing my recommendation for acceptance. If time is limited, please focus on all **critical** requests, followed by all **major** requests. If I misunderstood something at any point, just let me know, as this is always possible (apologies in advance if that’s the case!).\n\n### Section 1 (Introduction)\n\n- **[Major, Attribution]** Information leakage was first discussed by Mahinpei et al. [1] and by Margeloiu et al. [2]. At the very least, I would strongly suggest citing these works when information leakage is first introduced.\n- **[Minor, Clarity]** The first paragraph of the introduction is very large (it spans more than half a page). I would suggest breaking it up into more manageable chunks.\n\n### Section 2 (Related Work)\n\n- **[Major, Missing References]** The following are missing references that I would consider including in this related work, as they are highly relevant to subsections of your related work and are not discussed anywhere:\n    - Espinosa Zarlenga et al. \"Concept embedding models: Beyond the accuracy-explainability trade-off.\" NeurIPS (2022)\n    - Havasi et al. \"Addressing leakage in concept bottleneck models.\" NeurIPS (2022).\n    - Espinosa Zarlenga et al. \"Learning to receive help: Intervention-aware concept embedding models.\" NeurIPS (2023)\n    - Rodriguez et al. \"On the fusion of soft-decision-trees and concept-based models.\" Applied Soft Computing (2024)\n    - Ruiz Luyten et al. \"A theoretical design of concept sets: improving the predictability of concept bottleneck models.\" NeurIPS (2024)\n    - Ragkousis et al. \"Tree-based leakage inspection and control in concept bottleneck models.\" arXiv (2024).\n    - Vandenhirtz et al. \"Stochastic concept bottleneck models.\" NeurIPS (2024)\n    - Parisini et al. \"Leakage and interpretability in concept-based models.\" arXiv (2025).\n    - Dominici et al. \"Causal concept graph models: Beyond causal opacity in deep learning.\" ICLR (2025).\n- **[Minor, Clarity]** As in Section 1, this entire section is a single paragraph. For clarity and readability, I suggest breaking it into multiple paragraphs.\n\n### Section 3 (Method)\n\n- **[Critical, Clarity]** I would strongly suggest not using $c$ to refer to task classes when you have also “concepts” all over the place in the methodology itself. This opens up the door for extremely confusing notation. Instead, I would recommend using something like $l$ (for label) to refer to task labels (rather than for a path as it is currently used) and $c$ to refer to intermediate concepts. Also, on this note, notice that $k$ is usually used for the cardinality of the concept space (rather than the cardinality of the task space), so using $K$ for the number of task classes also invites confusion for those of us used to different notation in the concept-based literature.\n- **[Major, Quality]** In Section 3.2, it would be helpful if, in this section, rather than in an appendix, you clarify how sensitive the method is to different values of $\\{t, \\tau, r_\\text{min}, \\alpha\\}$ (and whether that selection matters). Ideally, this is backed with some further experiments down the line (in the experimentation section or an appendix). I can see a discussion of some of these hypers in the appendices, but I could not find any discussion of the method's sensitivity to these values.\n- **[Major, Clarity]** Given that this is a relatively important component of your pipeline, and one that may become a serious practical constraint on the usability of your proposed method, I would recommend discussing the “manual review”  mentioned at the end of Section 3.2 within the main body of the paper rather than in an appendix.\n- **[Major, Quality]** How well does this method scale for large datasets, given that computing Equation 2 could be quite computationally heavy (if not intractable) for very large datasets? I would recommend that the authors discuss ways to mitigate this potential issue in Section 3.3, when the concept generation/filtering mechanism is defined.\n- **[Major, Clarity]** The variable $B$ is used in Equation 4 but not defined anywhere up to this point in the manuscript. Please define this variable before or immediately after using it.\n- **[Minor, Notation]** Note that there are inconsistencies in how this paper names the children of a node (both $\\text{Ch}(v)$ and $\\text{Child}(v)$ are used). I would suggest sticking with one convention only.\n\n### Section 5 (Experiments)\n\n- **[Critical, Quality]** I can see an argument for why, in some experiments, you may want to evaluate against PCBMs and Labos without sparsity regularization. However, when comparing it against your method on the purely-accuracy-based experiments (i.e., in Section 5.3), I would argue that you should compare with, at least, the original variant of both of these approaches as proposed by the authors (which includes the sparsity regularization). I can see that some of these experiments are in Appendix B.3. However, I would argue that it is important to show the results of using sparsity regularization for Labo and PCBM in Table 1 as that is how the authors proposed those methods (otherwise calling them Labo/PCBM in the table itself is a bit misleading and may give the wrong impression). After doing this, and as a follow-up experiment, and this is where fairness comes in place, you can then evaluate what happens when one includes or excludes regularization to all methods (to see if really going over the decision-tree path is worth it over simpler methods such as sparsity regularization). This is important as I am not entirely convinced by the argument in Appendix B.3 that fine-tuning for regularization is somehow harder than having to fine-tune for the hyperparameters that are now present in CFM but are not in other methods (e.g., $t$, $\\tau$, $\\alpha$, $r_\\text{min}$, etc.). For example, why is it harder to select $\\lambda$ in the sparsity regularization than it is to select $\\alpha$ for CFM? I am just not entirely convinced that they are inherently different problems. Therefore, I would greatly appreciate it if the authors could further comment on this.\n- **[Critical, Quality]** I would greatly appreciate it if error bars could be included for all empirical results to follow good scientific practice. This is important because some results are close enough to be due to noise.\n- **[Major, Quality]** Regarding the many claims in the paper of CFM reducing information leakage: would it be possible to measure this (to see if the claim holds) against other more information-theoretic metrics for leakage (e.g., the ones in [3] or [4]) than the proposed SIR metric?\n- **[Minor, Clarity]** I would suggest splitting the first paragraph of Section 5.4 into two. One part can discuss the results of Table 2, and the other can discuss Figure 4. Right now, these two experiments are discussed in the same paragraph, with little to no connection between them. A similar comment applies to the paragraph in Section 5.5.\n- **[Nit/Minor, Typo]** There is a space missing in \"PCBM(Yuksekgonul et al., 2022)” and in “Labo(Yang et al., 2023)” in Section 5.3.\n\n### General Questions\n\n- **[Critical, Quality]**  Do you have more qualitative examples showing hierarchical explanations for downstream tasks? Having some extra examples could help clarify how helpful this method's explanations are in practical situations (rather than showing a simple example where it might've gotten \"lucky\" with its explanation).\n- **[Major, Quality]** Could you please provide a comparison somewhere in the text (at least conceptual, so not necessarily empirical) of CFMs vis-a-vis methods that also consider cross-concept relationships when making their predictions, such as Autoregressive CBMs [5], Stochastic CBMs [6], or Concept Graph Models [7]?\n\n## References\n\n1. Mahinpei et al. \"Promises and pitfalls of black-box concept learning models.\" arXiv (2021).\n2. Margeloiu et al. \"Do concept bottleneck models learn as intended?\" arXiv (2021).\n3. Zarlenga Espinosa, et al. \"Towards robust metrics for concept representation evaluation.\" AAAI (2023).\n4. Parisini et al. \"Leakage and interpretability in concept-based models.\" arXiv (2025).\n5. Havasi et al. \"Addressing leakage in concept bottleneck models.\" NeurIPS (2022)\n6. Vandenhirtz et al. \"Stochastic concept bottleneck models.\" NeurIPS (2024)\n7. Dominici et al. \"Causal concept graph models: Beyond causal opacity in deep learning.\" ICLR (2025).", "broader_impact_concerns": "I do not believe there are any broader ethical implications from this work that would merit a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "i2YSycHNFS", "reviewer": "Reviewer_5oea", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "TLDR: Well-known idea of hierarchy, applied to narrow domains on small size datasets with limited baselines; technically accurate but unclear how it advances the community.\n\nAuthors propose Concept Flow Models (CFMs), a hierarchical version of Concept Bottleneck Models (CBMs). Instead of exposing one big flat set of concepts to the classifier, they build a CLIP-based class hierarchy and make the model follow a root -> leaf path. At each node, only a small, node-specific set of concepts is used. The hierarchy is built automatically via agglomerative clustering of CLIP embeddings, pruned if nodes are not well separated, and then labeled with an LLM. For each node they select concepts using L1/Lasso from LLM-generated candidates. During inference, images are projected to these node-local concepts, node scores are turned into transition probabilities (temperature-scaled), and final class scores are aggregated along the path. Experiments on CIFAR-10/100, CUB-200, Tiny-ImageNet-200, and UCF101 show that this hierarchical bottleneck can match flat CBMs but with fewer effective concepts.\n\nStrengths:\n\n1. Paper targets an actual weakness of flat CBMs: when you expose all concepts at once, leakage becomes easy. Here they try to make leakage harder by only exposing concepts that are needed at each node. The CLIP-induced hierarchy -> node-local concepts -> path-based prediction pipeline is consistent with this goal.\n\n2. There is some theoretical justification: decomposing the task into multiple node problems makes it more expensive (in terms of irrelevant/random info) to recreate the same leakage as in the flat setting. So the structure is not only heuristic, there is a rationale.\n\n3. Authors compare their L1/Lasso-based node selector against other, more naive selection schemes (similarity-based, orthogonality-based, random, submodular) and show cleaner node-level concept sets. So the design choice is supported by experiments.\n\nWeakness:\n\n1. Evaluation is only on small/medium vision datasets. No ImageNet-1K, no modern long-tailed fine-grained sets, no domains where CLIP is weak (medical, satellite). So we don’t know if the hierarchical bottleneck still helps when classes explode or when text–vision alignment is worse.\n\n2. Two very relevant hierarchical / leakage-related CBM baselines [1, 2] are missing. Both operate in almost the same problem space (CBM + hierarchy + leakage/intervention). Not comparing against them makes it harder to judge how much CFM actually moves the needle.\n\n3. Strong dependence on CLIP and LLM quality. If the LLM proposes bad / non-visual concepts, or CLIP embeddings don’t match the label space, interpretability degrades. This will be worse in domains that are not web-image-like. Paper mentions this but doesn’t evaluate such a case.\n\nSuggestion for improvement:\n\n1. Show that the learned root -> leaf concept paths are useful beyond the classifier. For example, feed them as structured context to a VLM (similar to COCO-Tree style prompting) and check if hallucinations drop or VQA accuracy improves. That would make the hierarchy look like a general-purpose “video/image concept organizer,” not only an internal CBM trick.\n\n\n\n\nReferences:\n\n1. Sun, Ao, et al. \"Eliminating information leakage in hard concept bottleneck models with supervised, hierarchical concept learning.\" arXiv preprint arXiv:2402.05945 (2024).\n2. Pittino, Federico, Vesna Dimitrievska, and Rudolf Heer. \"Hierarchical concept bottleneck models for vision and their application to explainable fine classification and tracking.\" Engineering Applications of Artificial Intelligence 118 (2023): 105674.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, but with gaps. For the presented scope, the empirical evidence supports the main claim: CFMs can match flat CBMs while reducing effective concept usage. For gaps, please check the weakness section.", "audience_yn": "Yes", "audience_explanation": "Yes. It’s relevant for TMLR readers working on interpretability and concept bottleneck models.", "requested_changes": "Please address the weakness section.", "broader_impact_concerns": "No broad impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "45qgnJQiuH", "reviewer": "Reviewer_qzy6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes Concept Flow Models (CFMs) which try to learn a meaningful feature space of human-understandable concepts, rendering the underyling decision processes interpretable. Unlike Concept Bottleneck Models (CBMs), which use a \"flat\" concept layer, CFMs assume a hierarchical concept structure that they model with a decision tree. The approach is supposed to increase interpretability by providing a sequence of increasingly fine-grained concepts as well as reduce concept leakage due sparsity of the tree structure.\n\n**Pros:**\n- well-written and good structure\n- nice complexity analysis; nice theoretical analysis and new metric, given the paper's notion of \"leakage\"\n- reasonable number of experimental settings and datasets\n\n\n**Cons:**\n- claims about the \"automatic pipeline\" downplay the required \"manual review\" used to ensure reasonable concept learning\n- I'm still a bit confused about what \"leakage\" actually means here; it seems to be measured and limited via *capacity* rather than more directly focusing on meaningful concepts; would be nice with a more explicit definition of leakage, also discussing previous works like [1]\n- empirical results seem to report only one run per setting rather than reporting standard errors or confidence intervals over a number of replicates\n\n[1] Havasi, M., Parbhoo, S., & Doshi-Velez, F. (2022). Addressing leakage in concept bottleneck models. Advances in Neural Information Processing Systems, 35, 23386-23397.", "claims_and_evidence_yn": "No", "claims_explanation": "The claims would be supported if the authors addressed the points mentioned in **Cons** above.", "audience_yn": "Yes", "audience_explanation": "Many independent lines of work boil down to learning structured concept spaces, and the paper here provides some new perspectives and techniques for the problem.", "requested_changes": "**Critical**: Address each of the **Cons** mentioned above, namely:\n1. be more transparent in the abstract and intro about the limitations mentioned later in the paper, especially the \"manual review\" step, and the reliance on a sparse hierarchical underlying concept structure (which is a common and perhaps reasonable inductive bias, but still a strong and not-easily-testable assumption)\n2. be a bit more explicit about what \"leakage\" means, for example with an (informal) definition and/or reference to previous literature; I do see that the authors attempted this, including referencing Srivastava et al. (2024) and the NEC metric, but the surrounding discussion (and e.g., Props 4.1 and 4.2) seems to me more related to complexity of the model class rather than other notions of concept leakage I've seen in the }literature; perhaps the authors can explain/clear this up for me, and nothing needs to be added to the paper?\n3. the results should include some kind of measure of uncertainty, like standard errors or confidence intervals", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)