Title: Bayesian Concept Bottleneck Models with LLM Priors
Venue: NeurIPS 2025
Accepted: yes
Keywords: Bayesian inference

concept bottleneck models

explainable AI

large language models
Abstract: Concept Bottleneck Models (CBMs) have been proposed as a compromise between white-box and black-box models, aiming to achieve interpretability without sacrificing accuracy. The standard training procedure for CBMs is to predefine a candidate set of human-interpretable concepts, extract their values from the training data, and identify a sparse subset as inputs to a transparent prediction model. However, such approaches are often hampered by the tradeoff between exploring a sufficiently large set of concepts versus controlling the cost of obtaining concept extractions, resulting in a large interpretability-accuracy tradeoff. This work investigates a novel approach that sidesteps these challenges: BC-LLM iteratively searches over a potentially infinite set of concepts within a Bayesian framework, in which Large Language Models (LLMs) serve as both a concept extraction mechanism and prior. Even though LLMs can be miscalibrated and hallucinate, we prove that BC-LLM can provide rigorous statistical inference and uncertainty quantification. Across image, text, and tabular datasets, BC-LLM outperforms interpretable baselines and even black-box models in certain settings, converges more rapidly towards relevant concepts, and is more robust to out-of-distribution samples.


Peer reviews:
comments: SUMMARY:
To have interpretable model, concept bottleneck models require expensive human expert annotation on the data. This paper aims to reduce this cost by letting LLMs generate the concept and annotate the dataset instead. The authors propose a split-sample update to reduce the impact of LLM hallucination. Experiment results show the proposed method performs better than interpretable baseline models.

STRENGTHS AND WEAKNESSES:
Strength
- The problem this paper aims to solve is quite interesting.
- The idea of using LLMs within a rigorous Bayesian framework to achieve reliability is quite neat.

Weakness
- No thorough analysis is conducted on the LLM extracted concept. It is unclear how interpretable or trustworthy these concepts actually are.
- In the iterative procedure of generating new concepts, in each iteration, all training data are used. This can be highly inefficient once the data set gets larger and/or the iteration becomes large.
- Calibration is measured by Brier, which seems to me is an odd choice. ECE is a more standard measurement of calibration in the literature?
- The writing could be improved, I find the paper hard to follow. Also, the notation is a bit messy. A vector is represented by bold symbol and upper arrow at the same time.

QUESTIONS:
- I still don't fully understand how to use the proposed method to make prediction on a new data point. I think we need to first let the LLM annotate the new data point based on learnt concepts, but then how do you make prediction? Also, how do you account for the error that might occur during the LLM annotation?

- Citation is missing in Line 234 in appendix.

LIMITATIONS:
See weakness above.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel method that incorporates Large Language Models (LLMs) into Concept-Based Models (CBMs) using a Bayesian framework, with inference performed via Metropolis-Hastings sampling. The authors suggest that LLMs can act as informative priors, effectively guiding concept discovery, particularly in scenarios with limited prior information. Experiments on medical applications and bird image data suggest that  the proposed framework yields improved performance compared to the considered baselines.\n\nSTRENGTHS AND WEAKNESSES:\nThe proposed methodology—approaching concept refinement from a Bayesian perspective and leveraging LLMs as informative priors rather than mere knowledge sources—is both novel and well-motivated. This framing is particularly compelling given the challenge of applying Concept-Based Models (CBMs) in real-world settings where concept annotations are scarce. \n\nThe paper begins by listing well-known limitations of  CBMs—such as restricted context due to training data, computational cost, the trade-off between interpretability and accuracy and hallucinations. Even though these are indeed issues of CBMs,  I find that the authors do not convincingly show how the proposed method overcomes these issues. In terms of accuracy/interpretability, the authors base their experimental evaluation on a highly irregular regime that may be more favorable for the proposed framework but it does not provide adequate comparison to SoTA CBM approaches. At the same time, the interpretability evaluation of the approach in the context of augmenting a tabular model with clinical notes was only evaluated by \"four clinicians\", rendering the interpretation of the results ambiguous.\n\nThe computational cost is again something that is not clear. Yes, some CBM methods do use thousands of concepts but there are recent methods that consider sparsity inducing methods to effectively select the concepts; at the same time, I'm confident that using GPT4o-mini for querying for relevant concepts will be more costly (see also the limitations section). \n\nIn this context, the authors employ GPT-4o-mini—a vision-language model rather than a pure LLM—which introduces ambiguity regarding the method's dependence on prior visual/textual knowledge. The claim that BC-LLM outperforms other LLM-based CBMs in low-knowledge regimes is not convincingly demonstrated here (and leakage that was mentioned in the context of standard CBMs can also be an issue here). \n\nThe authors mention that \" we prove that BC-LLM can provide rigorous statistical inference and uncertainty quantification\" and this claim is spread throughout the manuscript; however, it is weakly supported by the experimental evaluation and it's only captured by the Brier metric and the entropy in the OOD setting but without substantial insights.\n\nOne other major concern lies in the design of the prompts used (appendix C), and since the quality and effectiveness of concept extraction can be highly sensitive to prompt phrasing, a more thorough analysis—such as a prompt ablation study—would substantially strengthen the experimental section.\n\nOverall,  this work aligns with a growing trend of leveraging LLMs for enhancing interpretability, particularly within the context of Concept-Based Models (CBMs). While the authors aim to improve concept quality through a statistical framework, the fundamental limitations of using LLMs remain largely unaddressed. The method relies heavily on the LLM at multiple stages of the pipeline, making the resulting interpretations highly sensitive to factors such as the specific model used, decoding parameters (e.g., temperature), and the nature of the task itself. As a result, the robustness and consistency of the interpretations may vary significantly, raising concerns about their reliability and generalizability. At the same time, the experimental evaluation, in my view, does not demonstrate substantial improvement or provide the necessary insights to justify the original claims of the authors.\n\nQUESTIONS:\nEven though the authors consider MC approaches, there are some other works in the literature concerning variational methods for concept selection. Wouldn't it also make sense to prompt the LLM once for concepts and then consider variable selection methods for the final subset?\n\nI kindly ask the authors to provide the wall time measurements for the considered experiments. \n\nDid the authors compare their approach to standard CBM benchmark settings such as CUB with the full 200 classes and not the 37 \"families\"? Did the authors also consider, SUN, Places365, ImageNet, or any other image dataset?\n\nCan the authors elaborate on the OOD setup? Did the authors consider all the remaining 50% of the data (the testing split) for their computations? Can the authors also provide an accuracy metric for OOD?\n\nIn p.7 line 308 the authors mention \"The difference when applying the Bunting bird CBM to an actual bunting bird versus an OOD sample\". What is a Bunting bird CBM? The authors consider a single CBM with multiple prediction targets correct?\n\nLIMITATIONS:\nNo, there isn't a dedicated limitations subsection. \n\nEven if some potential issues of the proposed framework are hinted at various parts of the manuscript, it should be clear for the readers what are the limitations of this work.  For example, one limitation seems to be complexity. Even though the authors discuss the complexity in terms of O() notation and then mention that \", BC-LLM was quick to run, as one iteration takes less than a minute for text data and a few minutes for image data on a normal laptop\". Another limitation can be the impact of the design of the input prompt or the parameters used for the LLM/VLM. None of these are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes BC-LLM, a Bayesian framework that enhances Concept Bottleneck Models (CBMs) by leveraging Large Language Models (LLMs) to explore an open-ended space of interpretable concepts. Unlike standard CBMs that rely on a fixed concept set, BC-LLM uses LLMs as both concept extractors and priors, enabling scalable and flexible concept discovery with uncertainty quantification. Despite potential LLM hallucinations, the authors provide theoretical guarantees. Experiments across image, text, and tabular domains show that BC-LLM outperforms interpretable and black-box baselines, converges faster to relevant concepts, and is more robust to distribution shifts.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Presents a novel approach to building Concept Bottleneck Models (CBMs) by leveraging Large Language Models within a Bayesian framework.\n\n\n2. Demonstrates versatility by applying the method across multiple modalities, including image, text, and tabular data.\n\n\n3. Offers a more lightweight yet effective CBM by selectively identifying and using only the most relevant concepts, reducing annotation cost and complexity.\n\nWeaknesses:\n\n1. Although the method is evaluated across different modalities, the dataset diversity—particularly for complex, real-world image domains, is limited. This raises concerns about the method's robustness in settings where human-defined concepts are ambiguous or less clearly represented.\n\nQUESTIONS:\n1. Generalizability to Other Image Domains:\nCan this approach be applied to different image classification datasets beyond those used in the paper?\n\n\n2. Applicability to Domain-Specific Tasks:\nHow would this method perform on tasks that require highly domain-specific knowledge, where LLMs may lack sufficient prior understanding, such as satellite imagery or industrial/factory imaging? Is BC-LLM still applicable or effective in these cases?\n\n3. Prompt Design for Concept Annotation:\nCould you provide the exact prompt used in Step 3 for concept annotation in the bird image classification task (Appendix)? How are concept annotations generated across different modalities?\n\n\n4. Entropy in OOD Settings:\nIn Table 1, the ResNet model shows the highest entropy in the out-of-distribution (OOD) setting. Could you clarify why this might be the case?\n\n5. Related Work Suggestion:\nYou may want to cite the paper \"Constructing Concept-based Models to Mitigate Spurious Correlations with Minimal Human Effort\" (ECCV 2024), which also uses LLMs for concept candidate generation, filtering, and annotation. It seems closely related and could help contextualize your contribution.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a BC LLM, i.e., a bayesian approach for concept bottleneck models. The key insight here is to use LLMs as priors and concept annotators while doing inference via a split-sample metropolis-within-Gibbs scheme. The paper also presents a theorem that suggests that the markov chain defined by running this split-sample scheme then the BC-LLM will asymptotically recover the true set of concepts that optimize the log-likelihood.  The BC-LLM is initialized with top concepts from an LLM. The concepts are then refined via a query replacement scheme. The paper then shows the benefits of the BC-LLM for bird-classification, clinicals, and for a tabular model.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths \n\n- Interesting Theorem to justify selection: the theorem offers a consistency guarantee, BC-LLM will asymptotically recover the true set of concepts that optimize the log-likelihood.\n\n- The result is inspired by a well-established literature on Bayesian model selection and is nontrivial to extend to an infinite concept space proposed by LLMs. The split-sample scheme is nice, and practically effective.\n\n- Evaluation is thorough, diverse, and realistic: covering simulated and real-world clinical text, vision (CUB birds), and multi-modal EHR data.\n\n## Weaknesses\n- Unclear implication of assumption H.1. It would be great to include a high-level summary of the justification for this. \n- What if all the initial concepts suggested by the LLM are not related to the task? As with all metropolis hastings like schemes, the sampling complexity here might be high. \n- Scalability, to large number of concepts, is a challenge, but the authors already mention this. \n- Some of the figures are small and could be bigger.\n\n- Ablation against LLM+CBM and Boosting LLM+CBM are fair and convincing. Concept interpretability was validated through expert clinician evaluation, adding real-world credibility.\n\nQUESTIONS:\n- How can one recover from poor-quality concept suggests at the start? \n- What is the cost for additional K? Specifically, how do you expect the computational cost to change with increasing concept size. \n- Can the LLM hallucinate concepts valid for training, but not relevant for test? How should the uncertainty adapt in this case?\n\nLIMITATIONS:\n- Scalability to larger concepts as the paper mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}