Bayesian Concept Bottleneck Models with LLM Priors
Authors: Jean Feng
Avni Kothari
Lucas Zier
Chandan Singh
Yan Shuo Tan
Venue: NeurIPS 2025 poster ()
Keywords: Bayesian inference
concept bottleneck models
explainable AI
large language models
Primary area: probabilistic_methods
TL;DR: BC-LLM is a novel procedure that integrates Large Language Models into a Bayesian framework for concept discovery to achieve better predictive performance, converge faster to relevant concepts, and provide rigorous uncertainty quantification.

Abstract: Concept Bottleneck Models (CBMs) have been proposed as a compromise between white-box and black-box models, aiming to achieve interpretability without sacrificing accuracy. The standard training procedure for CBMs is to predefine a candidate set of human-interpretable concepts, extract their values from the training data, and identify a sparse subset as inputs to a transparent prediction model. However, such approaches are often hampered by the tradeoff between exploring a sufficiently large set of concepts versus controlling the cost of obtaining concept extractions, resulting in a large interpretability-accuracy tradeoff. This work investigates a novel approach that sidesteps these challenges: BC-LLM iteratively searches over a potentially infinite set of concepts within a Bayesian framework, in which Large Language Models (LLMs) serve as both a concept extraction mechanism and prior. Even though LLMs can be miscalibrated and hallucinate, we prove that BC-LLM can provide rigorous statistical inference and uncertainty quantification. Across image, text, and tabular datasets, BC-LLM outperforms interpretable baselines and even black-box models in certain settings, converges more rapidly towards relevant concepts, and is more robust to out-of-distribution samples.

Bayesian Concept Bottleneck Models with LLM Priors
Jean Feng, Avni Kothari, Luke Zier University of California, San Francisco
Chandan Singh Microsoft Research
Yan Shuo Tan National University of Singapore
Abstract
Concept Bottleneck Models (CBMs) have been proposed compromise between white-box and black-box models, aiming to achieve interpretabilityy without sacrific- ing accuracy. The standard training procedure for CBMs is to predefine candidate set of human- iema-ninterpretable concepts, extract their values from the training data, and identify sparse subset inputs transparent prediction model. However, such approaches are often hampered by the tradeoff between exploring a sufficiently large set of concepts versus controlling the cost of obtaining concept extractions, resulting in a large interpretability-accrrre novel approach that sidesteps these challenges: BC-LLM iteratively searches over a potentially infinite set of concepts within a Bayesian framework, in which Large Language Models (LLMs) serve as both concept extraction mechanism and prior. Even though LLMs can be miscalibrated and hallucinate, we prove that BC-LLM can provide rigorous statistical inference and uncertainty quantification. Across image, text, and tabular datasets, BC-LLM outperforms interpretable baselines and even black-box models in certain settings, converges more rapidly towards relevant concepts, and is more robust to -utto--distribution samples.
tradeoff. This work investigates a
Introduction
Although machine learning (ML) algorithms have demonstrated remarkable predictive performance, many lack the interpretability and transparency necessary for human experts audit their accuracy, fairness, and safety [I 2). This has limited their adoption in high-stakes applications such as medicine 3 and settings where regulatory agencies require algorithms to explainable [4]. Recent works have explored Concept Bottleneck Models (CBMs) [5] 6 as potential solution: these methods leverage black-box algorithms to extract small number of interpretable concepts, which are subsequently processed by a fully transparent tabular model to predict target label. Thus, in principle, CBMs are much safer for use in high-stakes applications because humans can audit and, necessary, modify the extracted concepts to fix predictions.
Nevertheless, CBMs have yet fully realize this promise. Early CBM methods were difficult to scale because they relied on human experts to specify and annotate concepts on training data. Recent works have proposed using LLMs to suggest and even annotate concepts 7 10 as LLMs are cheaper, often have sufficient world knowledge to hypothesize useful concepts, and can provide (relatively) high-quality concept annotations [9] 10] This introduces new problems such as LLM hallucinations and inconsistencies but, more critically, does not resolve the fundamental problem that standard CBMs sacrifice a significant amount of accuracy to gain interpretability [[]. Recent proposals try to address this by encoding concepts using "soft" continuous values, such as through embedding similarity [ rather than "hard" binary values. However, "soft" CBMs can leak information about the label that would otherwise not have been available [2H[6];; this risk is particularly severe when
Code for running BC-LLM and reproducing results in the paper are available at :tpps:///ttttin com/ jjfeng/bc- 11m
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

concept extraction is jointly trained with the final label predictor [1] Because the definitions of soft concepts are ambiguous, humans can no longer easily interpret, audit, or intervene on such CBMs.
The aim of this work is to minimize the accuracy gap without sacrificing the interpretability of the extracted concepts. To this end, we revisit known pain point of CBMs: Current training procedures require pool of candidate concepts to be identified a priori, but ensuring this pool contains all relevant concepts difficult, partieularly in domains that are yet be scientifically well-understood and where there are an infinite number of potentially relevant concepts [17 10]. For instance, for the task of predicting hospital readmission, there are an infinite number of factors affecting a patient's risk. Even a single concept like "smoking status" has an infinite number of refinements, such as the correlated concept of "whether a patient has quit smoking," " the broader concept of "substance use," and more. As shown in prior work, when the candidate pool misses important concepts, the resulting CBMs may be inaccurate and even mislead users as to which concepts are truly relevant [8]
Rather than specifying concepts upfront, we investigate iterative refinement of concepts in a CBM with the assistance of an LLM. This updates classical modeling paradigm for the LLM era: prediction models were traditionally designed through the collaboration between domain experts and data scientists, in which they iteratively refine/engineer features for simple tabular models. LLMS can help significantly accelerate this co-design process and scale up the number of iterations, but they are also imperfect query engines that can hallucinate, suggest bad candidate concepts, incorrectly annotate concepts, and may not even be self celf-consittnt in their prior beliefs [19]. To set up guardrails, we "wrap" the LLM within a Bayesian posterior inference procedure to explore concepts with the help of an LLM in a statistically principled manner. This approach, which we refer to as Bayesian Concept bottleneck models with LLM priors (BC-LLLM), offers the following benefits:
BC-LLM finds relevant concepts, even in settings where there is little to no prior knowledge:
Unlike prior works that soley rely on prespecified set of candidate concepts, BC-LLM explores potentially infinite set of concepts data-adaptively and converge towards the true ones.
tradeoff' in real-world datasets: Exper- iments across multiple datasets and modalities (text, images, and tabular data) show that BC-LLM selects human-interpen iutaaniiterprettabee concepts while outperforming comparator methods, often even black- box models. Moreover, when assisting a hospital's 's data science team to revise an existing tabular MI. model, clinicians found BC-LLM to be substantially more interpretable and actionable.
BC-LLM substantially improves the interpretabilit-acccree
BC-LLM corrects LLM mistakes in a statistically rigorous manner to provide calibrated uncer- tainty quantification: Mistakes in the LLM's s prior are fully allowed and even expected, as they are corrected in the Metropolis accept/reject step. Moreover, BC-LLM provides uncertainty quantification the selected concepts, which improves performance in both in-distribution and out-of-distributio (OOD) settings. We prove that BC-LLM converges to the correct concepts, asymptotically, even when the LLM prior is poor or fails to be self-consistent. The Bayesian framework also naturally protects against overfitting and overconfidence in selected concepts. BC-LLM is cost-effective: BC-LLM selectively annotates ((TT) concepts, where K is the number of concepts in the CBM and T' is the number of outer loops; we often use only K x 5 and T 2 5. In contrast, existing methods annotate hundreds or thousands of prespecified concepts g
2 Related work
Prior CBM methods train on observations with hand-annotated concepts 55 6 20 and, more recently,
concepts annotated by an LLM or vision language model (VLM) 21
10 These methods are
known to be highly sensitive to the selected set of concepts [8] More recent works have suggested
using LLMs to prespecify the list of candidate concepts instead 7 g 22 26 but this requires the
LLM to have accurate prior knowledge about the supervised learning task. We note that many of
these works also make use of "soft concepts" and therefore suffer from information leakage [12 16].
To find relevant concepts in more data-driven manner, recent works have proposed taking an iterative approach 17 27+29.. These methods employ a boosting-like approach, where the LLM is provided of misclassified examples and asked to add/revise concepts to improve performance. However,
For instance, human would fill in O when asked "does this bird have gray breast" if there is no gray breast, but a "soft" CBMs may extract 0.2 if it sees a blue breast or gray head. This provides additional hints for downstream prediction.

---

0: happfranes cinace rote"
0. Keystrraa adtaction
Drop concupt and hopphrase mode conceot.
R. Prop candicarte concapts (uaad the top seepfreac prspors candiae
prodict
ihot raptayy arosiig
Poteett
in preeisun
cheel puél..
LL Exsart. Ciivir paliss dimaking
vopfessea
Kesphrawe
Scaf
comongta puppreeen the soaliing eanogppl and
Provide probabilty each candabine"
Cancitbetee
0P
Choud pain
0.3
"3 pallent chrst Pr
Publen 2 70 is acreted
7Bysa--ai, Matformin
metformin and insuli..
Accenthmiont condidaat concenns the existing canciaate concapte san the dara, samp hoon the the samplet conoept. reploce dropped one with some protetii a.
Aersstat sush stouerattons with cansiakde
Red concepts to dilce the GBV lsarning soodure"
propone
ponworr
Emusing!
Agp Slove
Obaet
LLM
Patini: 1
CS partent achweest medication??
CK Doee the pattent have disbele??
Figure 1: BC-LLM is initialized by having the L.M hypothesize the top concepts based on keyphrases extracted from each observation (Step 0). The concepts are then iteratively refined by dropping concept (Step 1), querying the LLM for candidate replacements (Step 2), annotating each observation with the candidate concepts using the LLM (Step 3), and determining which, if any, of the candidate concepts to accept (Step 4).
the LLM often struggles to identify relevant patterns from the presented examples, so the resulting concept proposals are far from optimal.
An alternative strategy is to conduct post-hoc distillation of a black-box model into interpretable concepts (3), which can be used to learn an interpretable prediction model. Nevertheless, prior work has shown that the generated explanations often not faithful to the black-box model, leading to poorer performance from the interpretable model [35].
Another related line of work uses LLMs to describe conceptual differences between pairs of datasets 36:38 or to directly describe a black-box function using natural-languaal concepts [3]44] How- ever, existing methods are primarily suited for simple classification tasks, as they essentially learn CBM with a single concept rather than a set of concepts.
Finally, recent works have considered combining LLMs with Bayesian techniques 43 45
Some
have suggested that In-Context Learning (ICL) can be viewed as LLMs conducting Bayesian inference 46 47), though recent work has proven that ICL is not fully consistent with Bayesian inference [48] In contrast, this work studies how we can iteratively discover new concepts to include using LLMs using a Bayesian inference procedure, accounting for imperfections in LLM reasoning.
3 Method: BC-LLM
Consider dataset DD = (211 //) -1).-... (Inr yn)]; where xy are model inputs and y/ are labels. Let X = (I1, ................................ xn.) and == (yI,.... ,,).. 3n).
A concept e defines a binary- or real-valued function de of model input, e.g. "Does the doctor note describe an elderly patient?" where 1=yes and O=no. A CBM selects a fixed number of K concepts с = (CI,..., ex) and fits parametric model for using the extractions (Øe, (I), &er (x)). For simplicity of presentation, our discussion will focus on logistic regression (LR)-based CBMss although other choices are possible. In this case, the data
is modeled by p pY = 1 = x, 0 c) = σ [KK_ k= 0. (z) + Bu where o is the sigmoid function and 0 = (0p.... ... 0xx are the parameters. We refer to as the "support" of the model. Throughout, we use capital and lowercase letters to differentiate between random variables and realized values, respectively. For instance, C' refers to a random concept whereas e refers to a particular one. We generally used boldface to denote vectors and regular font to denote scalars, with the exception of the model input of each example being denoted with regular font.
3.1 Review: Bayesian variable selection
Learning CBMs shares many similarities with the more standard problem of learning sparse models in high-dimensional settings, where one must select among a finite number of features that have already been tabulated [1). Here, we review the Bayesian solution for this more standard setting and then discuss how it may be extended to the CBM setting.
3

---

Given priors on and 0, the goal of Bayesian inference for sparse models in high-dimensional settings to sample from the posterior p(e, cly, X), which allows for uncertainty quantification of the true support and coefficients. Posterior samples also describe the uncertainty at a new point To via the posterior predictive distribution p(yo ro. X, c) = Iʃ p(yn) zoo 0, c) cly, X)dlddc, which can be viewed as combining the posterior samples into an ensemble model. Factorizing the posterior per p(0, cly, X) = p(0|c, c, y, x)(cc X), the first term describes the posterior for the model parameters and the second describes the posterior over selected variables. As the former can be readily obtained by classical Bayesian inference for low-dimensional settings (e.g., posterior inference for LR coefficients), our discussion will focus the latter.
Inference for p(cly. X) is typically achieved through Gibbs sampling. For each outer loop, Gibbs rotates through indices k = I..... K, during which it replaces the kth concept in the current iterate e by drawing from the posterior distribution for C& conditional on the other concepts c- When one is unable to sample from the conditional posterior, Metropolis-within-Giiin wwthin- can be used instead [52]. Rather than immediately replacing the kthh concept, proposes candidate concept é given the other concepts per some distribution (Cxx; C_ c-*) and accepts the candidate concept with probability a. This acceptance probability a, delineated in Algorithm [ depends on the relative posterior probabilities of the candidate and existing concepts and is carefully designed so that the posterior distribution is stationary with respect to the sampling procedure.
We extend Bayesian variable selection to learn CBMs. where the key difference is that the number of poten- tial concepts is effectively infinite, which makes it dif- ficult to formulate an appropriate prior over concepts pl (C) and proposal transition kernel QCCk: C-x). In- tuitively, seems that LLMs can help perform both these tasks, but how to do so correctly and efficiently surprisingly subtle. To motivate our choice, we first describe two naïve options:
Algorithm Metropolis-wiitinn-Ciii
1: Initialize concept set (CI, .Ck)
2: List of concept sets L = D
3: for = 1,2...... T do
4: for k = 1, 2.... K do
5:
Ck + MH-M MMH-UPDATECS k) // Update
k-th concept
Update 1: Propose with the LLM's "prior" Given the working concepts C_ C-k, we can prompt the LLM to propose new concepts, without revealing any data, e.g. "Propose an additional concept for predict- ing readmissions, given the dtready-existing concepts c_ This can be viewed as treating the proposal distribution Q(Cx; C_ = c_k) as sampling from the LLM's conditional prior distribution p(Cx/cc_a Consequently, the MH acceptance ratio simplifies to pivl(e_x.(x)..) which allows us to override proposed M)))e.
6:
Append the current to
7:
return L
8:
function MH-UPDATE(C, k)
9: Propose concept & ~ C(C=;C_A)
10:
min {
plie-A.ealy
picly, X)lyy,X)Qffff..
11:
if Accept with probability a then return
12:
else return
concepts that lead poor model performance. With- out access to data however, the LLM is likely to propose many irrelevant concepts from its prior, resulting in low acceptance probabilities and slow sampling. So while Update I may provide valid inference, it is inefficient and costly.
Update 2: Propose with the LLM's "posterior" Leveraging LLMs' ability to perform ICL 46
another idea is to include all the data when prompting the L.MM to propose new concepts. That is, we ask the LLM itself to "conduct posterior inference" and interpret the LLM's proposal distribution Q(Cx; C[Cx;CC-- = c_.)) as sampling from a conditional posterior distribution p(Ca (c_ y X) for some implicitly defined concept prior. This approach is risky; while LLMs may be able to approximate posterior inference for small datasets, this approximation breaks down for larger sample sizes [48). no longer override poor concepts with data the MH acceptance ratio simplifies to 1, i.. the proposed concept is always accepted. So while Update 2 may generate concepts more efficiently, the sampling procedure may not converge to good or even valid posterior distribution.
Furthermore,
These update options are at the end of two extremes, trading off between efficient sampling versus unbiased posterior inference. Our task then is to find a better balance.
'If' one believes the LLM's prior knowledge about the outcome of interest may be misleading, we can mask this information, e.g. "Propose an additional concept for predicting some label Y, given the already-existing concepts c-

---

3.2 The Split-sample update: Propose with the LLM's partial posterior
Rather than relying on the LLM's prior or posterior, key idea in BC-LLM take the middle road by sampling from an LLM's partial posterior, which we will show enjoys the best of both worlds. The proposal, which we refer as Split split-sample Metropolis update (Algorithm 2). uses the LLM to propose candidate concepts based on only a portion of the data and fixes mistakes in the LLM's proposal using the remaining data in the accept-reject step. This addresses the drawbacks of the two naïve updating procedures in Section 3.1] as it simultaneously uses the data to generate more efficient proposals while overriding the LLM as needed.
More specifically, the split-sample update first randomly splits the data to construct subset S of size [wn] and its complement S", for some fraction w (e.g. half). The LLM is prompted to generate candidate concepts by combining its prior knowledge with information from the data in S. We can then interpret the LLM's proposal distribution (Ca; C- C-&) as the (conditional) partial poste- rior distribution p(Cx/c-R. Vs, X). Then in the accept-reject step, the MH acceptance probability simplifies to the partial Bayes factor plyselys.(e x.).))
(53 This compares the likelihood of the held-out data S with respect to the candidate and existing concepts-muud like sample-spliting in frequentist settings-and and thus provides opportunity to correct inconsistencies between the LLM's proposal and the actual posterior distribution.
p(yse c.
Algorithm 2 Split-sample Metropo-
Algorithm Multiple-try split-sample Metropolis update
lis update
1:
function MULTI-SS-MH-
k)
1: function SS-MH-UPDATEIC k)
2: Sample subset S of size [wm].
2: Sample subset S of size [wm]. candidate
3: Propose a( ..... &
C
3: Propose
4: Sample in E {1, ... M) with probabilities x Win
p(ys= (ys, (c_,, ),
Vs.
5: wo - p(ys= ys, c, X)Q((a
min p(yyellys.ee
pyyse lxs.c,))
6: a + min Que we-..--.............. Eosms Mom 3m
5: if Accept with probability
then return &
7: if Accept with probability a then return (1h)
6: else return
8: else return Cj
In practice, rather than sampling a single candidate concept, we use a version of multiple-try Metropolis-Hasting (Algorithm 3) [54]. Here, the LLM proposes batch of candidates. We sample a candidate from this batch based on their posterior probabilities and consider the sampled candidate for acceptance. This is more cost-effective as it lets us batch concept annotations, described later.
Assuming that the LLM's partial posterior across all iterations is consistent with some prior distribu- tion p( ()), we can prove that the Markov chain defined by this procedure has the posterior p(cly, X) as stationary distribution (see Proposition H.I in the Appendix). Since LLMs are known to be imperfect Bayesian inference engines [8], this assumption is at best only approximately satisfied. Nevertheless, we can prove that the procedure still converges to the true concepts even when this assumption does not hold:
Theorem 3.1. Suppose the data is HID. Let L(c) = max E(log 0).)) and C* =
argmax, L(c). For sample size n, let II, denote the set of stationary distributions of the Markov chuin defined by running Algorithm with with SS-MH-UPDATE or MULTI-SS-MHUUPDAA instead of MH-UPDATE, Assuming a Gaussian prior for O and under regularity conditions (see Assumption
G.1 then (T(")) 11,} + 1 in probability -
3.3 Putting together: BC-LLM
We now put all the pieces together. Mathematically, BC-LLM runs Metropolis-within-COO (Algo- rithm with a split-sample update (Algorithm 2 or 3p. We translate this into a step-by-step recipe (Fig After initializing BC-LLM in Step 0, each iteration of Metropolis-within-CGiin performs split-sample update to output CBM. The CBMs form a posterior distribution, as well as an ensemble prediction model. Example prompts and implementation details are in the Appendix.
(Steps to 4)
5

---

Step O: Initialization and keyphrase extraction We first use an LLM to extract "keyphrases" from each observation 2, which will be used by the LLM to brainstorm candidate concepts. Here, "keyphrases" refer to short phrases: keyphrases for images describe what's in the image (e.g. blue eyes), while keyphrases for text data can be direct quotes or summarizations (e.g. diabetes) [55]. Keyphrases can be thought of as rough concepts, as they do not define a function but can help an LLM come up with a formal concept question (e.g. "diabetes" is a keyphrase whereas "Does the patient currently have diabetes?" is concept). This brainstorming phase does not restrict the set of keyphrases the LLM can extract to allow for a wide exploration of concepts. The tradeoff is that the extracted keyphrases may be an incomplete summary of each observation. To initialize the CBM sampling procedure, these keyphrases are used to fit a (penalized) LR model for the target Y. similar to fitting bag-of- ofowords (BoW) model. An LLM then analyzes the most predictive keyphrases to generate an initial set of K concepts and extracts these concepts from each observation.
Drop concept & fit keyphrase model Each iteration proposes candidate concepts based on some data subset S to replace the kth concept for E (1..... ; K}. Given the context limits of LLMs, having an LLM review each observation in S individually is ineffective. Instead, we generate a summary by highlighting the top keyphrases in S for predicting Y. on top of the existing K - concepts C-k- For continuous outcomes, this can be accomplished by finding the top keyphrases associated with the residuals of a model that predicts Y given c-k- For bi- nary/categorical outcomes, we instead fit a (multinomial) LR model of Y, where the inputs keyphrases encoded using BoW and annotated concepts C-k. Let the coefficients in this "keyphrase model" f for keyphrases and concepts be denoted fw and Be... respectively. We fit it by solving 1
Step 1:
ming .ww n E.=. (v. :. + (J ) where E is the logistic loss, 3w is pe- nalized with some penalty function J, and penalty parameter À is selected by cross-validation. We found the ridge penalty to work well in practice but other penalties may be used instead.
Step 2: Query the LLM for candidate concepts Given the summary of the top keyphrases generated from Step 1, the LLM is asked to propose M candidates for the k-th concept. In addition, we ask the LLM to provide the value of the partial posterior probability for each candidate concept. For instance, if the top keyphrases include "chronic kidney disease" and "acute kidney injury," the LLM may propose concept like "Does the patient have history of kidney disease?" with an assigned partial posterior probability of 0.4.
Step 3: Concept annotation Given candidate concepts, we use the L.M to extract their values using
a zero-shot approach, i.e. (x) for c E {e()) ..... ok 2(M)). Because multiple-try try Metropolis- -iithin Gibbs proposes a batch of candidate concepts, we can batch their extractions into a single LLM query for each observation. This makes BC-LLM more cost/query-
than ordinary Metropolis- within- Gibbs, which outputs only single concept per iteration and thus cannot be batched. Note that letting the L.M occasionally output probabilities when it unsure can be helpful, though it is critical to avoid information leakage (e.g. no joint training of concept extraction and label prediction).
Step 4: Accept/reject step The final step to compute the sampling probabilities for the candidate concepts and the acceptance ratio, which involve comparing the proposal probabilities for the existing and candidate concept sets e as well as their split-sample posteriors p(ys= lys, c, X) J p(ys= 0 X) (0ly: c, de0 To estimate the integral, one approach is to sample from the posterior on 0. but this can be computationally expensive since one would need to implement posterior sampling in inner and outer loops. Alternatively, in the case of LR-based CBMMs one can use a Laplace-like approximation [56] (see Appendix). As the posterior distribution of LR parameters are asymptotically normal, this approximation becomes increasingly accurate as -> x..
Computational cost. BC-LLM performs O(nTK) LLM queries, where T is the number of outer loops. K is typically chosen to be small, because a large K yields less interpretable CBMs. Although standard Bayesian procedures typically draw thousands of posterior samples, we found that a small T was quite effective in practice when combined with a greedy warm-start procedure; our experiments all use T 5. Generally speaking, a small T suffices for achieving high prediction accuracy, larger T (c.g., 10) is helpful for quantifying the uncertainty of relevant concepts. Critically, running BC-LLM is significantly more cost-effective than standard CBMs, which require O(n)) queries where W is the number of pre-specified concepts and must be large (typically hundreds or thousands) to avoid missing relevant concepts g. In our experiments, BC-I.LM was quick to run: each iteration completes within one to two minutes for the experiments in Section 4

---

Table 1: Performance of CBMs and black-box models (ResNet) for classifying bird species.
In-distribution AUC (t)
OOD
Method
Accuracy (1)
Brier ())
Entropy (1)
BC-LLM
0.680 (0.614, 0.747)
0.874 (0.840, 0.907)
0,428 00.357.0.500
0.865 (0.693, 1,036)
LLM+CBM
0.640 (0.573, 0.707)
0.810 (0.768, 0.853)
0.452 (0.377. 0.528)
0.663 (0.474.
Boosting LLM+CBM
0.538 (0.463, 0.614)
0.722 (0.673, 0.772)
0.577 (0.499, 0.654)
0.842 (0.630, 1.054)
Human+CBM
0.658 (0.591. 0.725)
0.835 (0.791, 0.879)
0.499
10.414,0.5844
0.758 (0.558, 0,959)
LLM+CBM (No keyphrases)
0.555 (0.488, 0.623)
0.759 (0.713, 0.805)
0.651 (0.548, 0.754)
0.626 (0.495, 0.757)
ResNet
0.664 (0.613, 0.716)
0.853 (0.821, 0.885)
0.457 (0.398, 0.516)
0.914 (0.748, 1.079)
4 Experiments
We evaluated BC-LLM in across three domains and modalities: classifying birds in images (Sec- tion [.). simulated outcomes from clinical notes (Section (4.2). and readmission risk in real-world clinical data (Section 4.3) The experiments below used GPT-4o-mini [57] and Section 4.3 used pro- tected health information-compll includes results using other LLMs, implementation details, and experimental settings.
version of GPT-4o for real-world clinical notes. The Appendix
Baselines. We compared against CBMs fit using the standard approach where humans select and annotate concepts (Human+CBM) (5), CBMs where an LLM brainstorms concepts after seeing which keyphrases are most associated with the label but without any iteration (LLM+CBM) [58 10, and CBMs fit using boosting algorithm that iteratively adds concepts by having an LLM analyze observations with large residuals (Boosting LLM+CBM) [17 27]. We include black-box and semi-inte -nteerpretable models assess the tradeoff between interpretability and accuracy. For the bird clasccaassiiiottion dataset, we also compare to a CBM that uses LLM-suggested concepts rather than keyword- -bsed concepts; use the 370 LLg-suggested concepts from 8 and extract the values of these concepts with an LLM CBLM+CBM (No keyphrases)).
Evaluation metrics. Methods were evaluated with respect to predictive performance as measured by AUC and/or accuracy and uncertainty quantification as measured by Brier score. When true concepts are known, we also quantify concept selection rates using "concept precision," as defined by K P(Cr E c² y, X), and "concept recall," defined by Ex=p(ce E Č Xy, where C is the posterior distribution over concept sets from BC-LLMM or single set of concepts learned by non-Bayesian procedure. Concept matches were verified manually (see Appendix for details).
4.1 CBMs for classifying bird images
We first evaluate how well BC-LLM can learn concepts that differentiate between bird species within the same family in the standard CUB-birds image dataset [59]. Grouping the 200 bird species into their respective families, this resulted in 37 prediction tasks. We considered family-based prediction tasks because there likely exists a parsimonious CBM within each bird family, in contrast to the task of predicting between all 200 species. To assess how well BC-LLM performs in settings with limited prior knowledge, we do not tell the LLM that we are predicting bird species. That is, we replace the labels with one-hot encodings and simply tell the LLM to find concepts for predicting Y. Consequently, the LLM must search over a larger space of concepts.
The data is split 50/50 between training and testing. The number of concepts K learned for each task was set to the number of classes, but no smaller than 4 and no greater than 10. For the black- box comparator,
fine-tuned the last layer of ResNet50 pre-trained on ImageNetV2 [60] 61). HuMan+CBM was trained on the 312 numan-annotater features available in the CUB-birds dataset.
BC-LLM achieved higher accuracy and better calibration on average, compared to all the other fully interpretable CBMs (Table D. Notably, soft CBMs no longer performed as well once the concepts were constrained to be fully interpretable. BC-LLM also outperformed CBMs trained on human collated and annotated features, potentially because it was able to consider additional concepts. More critically, BC-LLMM outperformed ResNet50 in both accuracy and calibration, as ResNer50 easily overtrained on the small bird datasets. In contrast, BC-LLM could identify relevant concepts for simple parametric model. leading to faster model convergence.
7

---

beity
contain beatherd
conppet (94/
and beily 10 941
Dons Fil kectoe be
grecn mggio
Done the bidli inags hove aoitis boly??
Cose the bidd image hove buil
and beky
slede wings 30 32]
Dow the bidin tnii teach
mannge Lecy
Dow brd inage hove winger
(0.24)
distnetive
the kestrre bely"
doaard buathrr?
parmmpparoge 11
Droe ih lan tio tnsinniniioe heide
Kirantblue feathers 10.22
uitii (3
Dovs Pr
Cose the dirc in image houe baly?
heur?
Uae the ler Tey light love leak vergs?
Example of BC-LLM classifying Bunting birds. (Left) Learned concepts trained on 1/3 versus 3/3 of the training data, respectively. Labels are shortened concept questions, generally of the form "Does the image depict...?" Proportion of posterior samples with the concept are shown clustered hierarchically solely for visualization purposes. Highlighted
Figure 2:
in parentheses. Concepts labels are distinguishing bird features. (Right) Application of BC-LLM to an actual bunting bird (top) versus dog pretending to be one (bottom).
AVC
2r
(bcomiort 9..0
currenty
0.30
donornal ohee sige (0.16)
as nenarmote (0.20)
benne hogghtrlbeddo (0:20)
a former moker (000
0.20
smoking
nacsea teekrng queiy (0.0)
the sareen / Hiwely fechoe (044)
0.10
Mechod
cervicel pa (0.40
ceaeine un
Pas48
Docitinn LLM CCM
fistory mire
(0.24)
coneurty austol 500
2o0 in Bug 60 (i-- (00
haeeer cocunn
emaking Nitrey (0.3))
a partne (22)
8-2
-60
sorld suppert (4:44
Mam Mam
Figure 3: MIMIC results: (Left) Comparison of BC-LLM and existing methods in terms of perfor- mance and recovery of true concepts with 95% CI. (Right) Dendrograms of concepts learned by BC-LLM with 100 and 800 observations (left and right, respectively). Labels shortened concept questions generally of the form "Does the note mention the patient...?" Highlighted labels correspond the true color-coded concepts in Section 4.2
We also assessed BC-LLM's robustness in OOD settings, the Bayesian framing should improve uncertainty quantification (Table []. For each model, we used it to classify birds for a species was trained Compared to other CBM methods, BC-LLM yielding higher entropy across the output classes. We note that ResNet model had the highest entropy, though it has the advantage of not being limited extracting a small set of interpretable concepts. To illustrate the benefit of using fully interpretable CBMs in OOD settings, Figure 2 difference when applying the Bunting bird CBM an actual bunting bird versus OOD sample. Whereas the CBM extracted zeros and ones for the actual bunting bird, it was unsure how to extract the concept "Does this image show vibrant feathers?" ?" for the image of dog wearing rainbow wings, The LLM thus outputted a probability of 0.5, citing that "the image contains vibrant colors but the wings do not actually contain feathers." This shows how having an LLM output its reasoning when answering concrete concept questions can reveal situations that need human intervention.
more appropriately unsure,
(right) shows the
Finally, visualize the posterior distribution over concepts using hierarchical clustering (Fig 2 left). Continuing with the Bunting bird example, we see that BC-LLM converges towards distinguishing concepts training data increases and away from vague or non-specific concepts (e.g. "sharp beaks" apply to all bunting birds). Moreover, the number of unique concepts decreases with the amount of training data, because it has converged to concepts that truly distinguish the different species.
4.2 CBMs for classifying clinical notes with simulated outcomes
To objectively measure how well BC-LLM can recover true concepts, we conducted a simulation study where the true concepts are known. To assess BC-LLM in settings where the LLM has no prior
8

---

knowledge about the target, we simulated Y using a LR model with five Social Determinants of Health as inputs, which were annotated from real-world patient notes MIMIC-IV 62 63
1. Does the note mention the patient consuming alcohol in the present or the past?
2. Does the note mention the patient smoking in the present or the past?
3. Does the note mention the patient using recrcational drugs in the present or the past?
4. Does the note imply the patient is unemployed?
5. Does the note imply the patient is retired?
CBMs with K=6 were trained on 100 to 800 observations. Human (Dracle) +CBM uses true concepts.
BC-LLM outperforms the comparator methods both in predictive performance and calibration (Fig- ure 3a) and, with only 400 observations, performs as well Human (Oracle) +CBM. The performance gap between BC-LLMM and the comparator methods widens with more training observations because BC-LLMM can iteratively refine its concepts. Although Boosting LLM+CBM is also an iterative pro- cedure, it struggles to hypothesize relevant concepts because it reviews only a few misclassified observations each iteration and cannot revise concepts added in previous iterations. Additionally, BC-LLM was better at recovering the true concepts compared to existing methods.
Visualizing the posterior distribution using hierarchical clustering (Fig 3 right), the learned posterior notably contracts towards the true concepts with increasing data as the number of training observations increases from 100 to 800. Moreover, BC-LLM remains appropriately unsure about certain concepts that are highly correlated. For instance. it states many variations of the smoking concept that are essentially impossible to distinguish between, which is both expected and desired.
4.3 Augmenting a tabular model with clinical notes
Here we demonstrate how BC-LLM can help ML developers integrate different data modalities to improve model performance. The data science team at Zuckerberg San Francisco General Hospital has both tabular data and unstructured notes from the electronic health record (EHR). The team has so far fit model using the tabular data predict readmission risk for heart failure patients and wants to assess clinical notes contain additional concepts that are predictive. To address this, we extend BC-LLM to take as inputs (i) the risk prediction from the original tabular model and (ii) K 4 concepts extracted from the clinical notes, which can be viewed as revising the existing model by adding features. The CBMs were trained 1000 patients and evaluated on 500 held-out patients.
The original tabular model, trained on lab and flowsheet values, achieved an AUC of 0.60. BC-LLM revised this model to perform substantially better, achieving an AUC of 0.64 (Fig 4 left). Moreover, it outperforms comparator methods with respect to both AUC and brier score.
To assess the interpretability of the learned concepts, five clinicians were asked to rate concepts in the CBMs as well as top features in the original tabular model by how predictive they were, from ==low to 3=high clinical relevance 4 Concepts from BC-LLM received an average rating of 2.5, whereas concepts from the other methods were rated <2 (Fig 4 right). When unblinded the concepts, clinicians commented on a number of advantages to using BC-LLM. First, many of the learned concepts were known to be causally relevant but were difficult to engineer solely from tabular data. For example, information for determining "substance dependence" is rarely documented in the tabular data but often noted in clinical notes. Second, BC-LLM also suggested new features to engineer that the data science team had not previously considered. Most interestingly, some of the learned concepts pointed to interventions that the hospital could try to implement, such as ensuring follow-up appointments were scheduled and opting for medication regimes that were easier to follow. In contrast, the top features in the original tabular model were difficult to interpret and not actionable.
Discussion
BC-LLM is a new approach to learning CBMs that iteratively proposes concepts using an LLM within Bayesian framework, which allows for rigorous uncertainty quantification despite LLMs
'Tabular features were rewritten as questions so they could not be distinguished from learned concepts. See Appendix for annotation instructions.

---

Prccant hopptal advinsion
falecaice laring Hode
Method
AUC (95% CT)
Brier (98% CT)
(abosacc dipe souace busi
BC-LLM
0.64
0.14
mubile meciaaioo prourite
Boles slilia ane
LLM-CBM
0,59 00.5 0.5)
0.29 (0.25.0.3))
LLM
Boosting LLM-CBM
0.14
Papertesian (1.47)
Bag. wagrosswords
0.52 0146.0.00 0.50)
0.29 (9.75,0.34)
the put pubbi ond L Fo0 aoteor infecced Gustee schooie 20.27
BGAI
Acerage Sating
Figure 4: Learning to augment a readmission risk prediction model for heart failure patients. Den- drogram labels are shortened questions of the format "Does the note mention the patient having...?". Highlighted concepts received scores from clinicians being highly predictive (scores 2.5+). Aver- age clinician ratings for concepts/features from the different methods are shown on the right.
being prone error and hallucinations. Because it explores and suggests concepts in data-adaptive manner, BC-LLM is particularly well-suited for settings with limited prior knowledge about which concepts relevant where the number of potentially relevant concepts is infinite. The method compatible with various data modalities (text, images, and tabular data) and can be extended beyond the settings of binary and multiclass classification. The empirical results show that BC-LLM outperforms existing methods, even black-box models in certain settings.
Future work. BC-LLM is currently designed for learning highly interpretable CBMs where the number of concepts K is typically no more than 20. While BC-LLM can be applied to learn even more concepts, future directions can consider further speeding up posterior inference, such through mini-batching of observations or concepts.
Impact Statement. The interpretability provided by BC-LLM additionally enables better understanding of the underlying algorithm and facilitates human oversight, which may improve the safety of Al algorithms and Al-based pacision-making The use of LLMSs incurs significant computational cost. and corresponding environmental impacts; however, BC-LLM much more computationally efficient than comparable LLM-based CBMs, potentially reducing overall environmental impacts.
Acknowledgments
This work was funded through a Patient-Centered Outcomes Research Institute@ (PCORI@) Award MME-202CC-226119, The views presented in this work solely the responsibility of the author(s) and do not necessarily represent the views of the PCORIO, its Board of Governors or Methodology Committee. The authors thank the UCSF AI Tiger Team, Academic Research Services, Research Information Technology, and the Chancellor's Task Force for Generative Al for their technical support related to the use of Versa API gateway. The authors thank Patrick Vossler, Yifan Mai, Seth Goldman, Andrew Bishara, James Marks, Julian Hong, Aaron Korbllihh, Nicholas Petrick, and Gene Pennello for their invaluable feedback the work.
10

---

References
[1] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pages 214-226. ACM, 2012.
[2] Cynthia Rudin. Please stop explaining black box models for high stakes decisions. arXiv preprint arXiv:1811.10154 2018
[3] Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(0):1930-1940, 2023.
[4] Bertalan Meskó and Eric Topol. The imperative for regulatory oversight of large language models (or generative ai) in healthcare. NPJ digital medicine, 6(1):120, 2023.
[5] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pages 5338 5348. PMLR, 2020.
[6] Eunji Kim, Dahuin Jung, Sangha Park, Siwon Kim, and Sungroh Yoon. Probabilistic concept bottleneck models. Proc. Int. Conf. Mach. Learn., June 2023,
[7] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint arXiv:2210.07188, 2022.
[8] Tuomas Oikarinen. Subhro Das, Lam M Nguyen, and Teu Weng. Label-frec concept bottleneck models. International Conference on Learning Representations, April 2023.
[9] Denis Jered McInerney, Geoffrey Young, Jan- van Meent, and Byron Wallace. CHiLL: Zero-shot custom interpretable feature extraction from clinical notes with large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, December 2023.
[10] Vinamra Benara, Chandan Singh, John Xavier Morris, Richard Antonello, Ion Stoica, Alexander Huth, and Jianfeng Gao. Crafting interpretable embeddings for language neuroscience by asking L.MMs questions. In The Thihtt-eighth Annual Conference Neural Information Processing Systems, November 2024.
[11] Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. arXiv preprint arXiv:2205. 15480, 2022.
[12] Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Veles Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. arXiv preprint arXiv:2106.13314 2021.
[13] Andrei Margeloiu, Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian Weller. Do concept bottleneck models learn as intended? arXiv preprint arXiv:2105.04288 2021.
[14] Marton Havasi, Sonali Parbhoo, and Finale Doshi-Velez. Addressing leakage in concept bottleneck models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[15] Angelos Ragkousis and Sonali Parbhoo. Tree-based leakage inspection and control in concept bottleneck models. arXiv preprint arXiv:2410.066352, 2024.
[16] Mikael Makonnen, Moritz Vandenhirtz, Sonia Laguna, and Julia E Vogt. Measuring leakage in concept-based methods: An information theoretic approach. In ICLR 2025 Workshop: XAI4Science: From Understanding Model Behavior to Discovering New Scientific Knowledge. 2025.
[17] Josh Magnus Ludan, Qing Lyu, Yue Yang. Liam Dugan, Mark Yatskar, and Chris Callison- Burch. Interpretable-by-dess text classification with iteratively generated concept bottleneck. arXiv preprint arXiv:2310.19660 2023.

---

[18] Vikram V Ramaswany, Sunnie S Y Kim, Ruth Fong, and Olga Russakovsky. Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10932-10941, 2023.
[19] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. International Conference on Learning Representations, 2023.
[20] Mert Yuksekgonul, Maggie Wang, and James Zou, Post-hoc concept bottleneck models. The Eleventh International Conference on Learning Representations, September 2023.
[21] Ajay Patel, Delip Rao, Ansh Kothary, Kathleen McKeown, and Chris Callison-Burde Learning interpretable style embeddings via prompting LLMs. Proc. Conf. Empir. Methods Nat. Lang. Process., May 2023.
[22] Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Bure and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[23] Konstantinos Panagiotis Panousis, Dino lenco, and Diego Marcos. Sparse linear concept discovery models. In Proceedings of the ieee/cvf international conference on computer vision, pages 2767-2771, 2023.
[24] Chandan Singh, Richard Antonello, Sihang Guo, Gavin Mischler, Jianfeng Gao, Nima Mesgarani, and Alexander G Huth. Evaluating scientific theories predictive models in language neuroscience. bioRxiv, pages 2025-08, 2025.
[25] Haotian Xu, Tsui-Wei Weng. Lam M Nguyen, and Tengfei Ma. Graph concept bottleneck models. arXiv preprint arXiv:2508. 142555 2025.
[26] Michelle S Lam, Janice Teoh, James A Landay, Jeffrey Heer, and Michael S Bernstein. Concept induction: Analyzing unstructured text with high-level concepts using Iloom. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pages 1-28, 2024.
[27] Chenxi Liu, Yongqiang Chen, Tongliang Liu, Mingming Gong, James Cheng, Bo Han, and Kun Zhang. Discovery of the hidden world with large language models. February 2024.
[28] Yiqun Sun, Qiang Huang, Yixuan Tang, Anthony KH Tung, and Jun Yu. general framework for producing interpretable semantic text embeddings. arXiv preprint arXiv:2410.03433, 2024.
[29] Jeeyung Kim, Ze Wang, and Qiang Qiu. Constructing sancept-based models to mitigate spurious correlations with minimal human effort. In European Conference on Computer Vision, pages 137-153. Springer, 2024.
[30] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). Proceedings of the 35 th International Conference Machine Learning, 2018
[31] Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-ave aware concept-bast explanations in deep neural networks. Advances in neural information processing systems, 33:20554-20566 2020.
[32] Ruochen Zhao, Shafiq Joty, Yongjie Wang, and Tan Wang. Explaining language models' predic- tions with high-impact concepts. Findings of the Association for Computational Linguistics: FACL 2024, May 2024.
[33] Yibo Jiang, Bryon Aragam, and Victor Veitch. Uncovering meanings of embeddings via partial orthogonality. In Thirty-seventh Conference on Neural Information Processing Systems, November 2023.
12

---

[34] Chandan Singh, Armin Askari, Rich Caruana, and Jianfeng Gao. Augmenting interpretable models with large language models during training. Nature Communications, 14(1):7913, 2023.
[35] Finale Doshi Veliz and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv [sLat.ML,. February 2017.
[36] Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. Goal driven discovery of distributional differences via language descriptions. Conference on Neural Information Processing Systems, November 2023.
[37] Ruiqi Zhong, Charlic Snell, Dan Klein, and Jacob Steinhardt. Describing differences between text distributions with natural language. In International Conference on Machine Learning, pages 27099 27116. PMLR, 2022.
[38] Lisa Dunlap, Yuhui Zhang, Xiaohan Wang, Ruiqi Zhong, Trevor Darrell, Jacob Steinhardt, Joseph E Gonzalez, and Serena Yeung-Levy, Describing differences in image sets with natural language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24199-24208, 2024.
[39] Laura Kopf, Nils Feldhus, Kirill Bykov, Philine Lou Bommer, Anna Hedström, Marina M-C Höhne, and Oliver Eberle. Capturing polysemanticity with prism: A multi-concept feature description framework. arXiv preprint arXiv:2506. 15538, 2025.
[40] Chandan Singh, Aliyah R Hsu, Richard Antonello. Shailee Jain, Alexander G Huth, Bin Yu, and Jianfeng Gao. Explaining black box text modules in natural language with language models. arXiv preprint arXiv:2305.cm 09863, 2023.
[41] Steven Bills, Nick Cammarata, Dan Mossing, William Saunders, Jeff Wu, Henk Tillman, Leo Gao, Gabriel Goh, llya Sutskever, and Jan Leike. Language models can explain neurons in lan- guage models. https: http///pppaaaacccc.co/p blob. core. wwndows..wo.ooooe... net www.erroeeeeeaeeeeerrrreeeeeeeee paper/index html 2023.
[42] Nhat Hoang-Xuan, Minh Vu, and My T Thai. LIm-assisted concept discovery: Automatically identifying and explaining neuron functions. arXiv preprint arXiv:2406.08577 08572, 2024.
[43] Adam X Yang, Maxime Robeyns, Thomas Coste, Jun Wang, Haitham Bou-Ammar, and Lau- rence Aitchison. Bayesian reward models for Ilm alignment. arXiv preprint arXiv:2402.13210, 2024.
[44] Deyuan Liu, Zecheng Wang. Bingning Wang, Weipeng Chen, Chunshan Li, Zhiying Tu, Dianhui Chu, Bo Li, and Dianbo Sui. Checkpoint merging via bayesian optimization in I1m pretraining. arXiv preprint arXiv:2403. 19390, 2024.
[45] Tennison Liu, Nicolás Astorga, Nabeel Seedat, and Mihaela van der Schaar. Large language models to enhance bayesian optimization. arXiv preprint arXiv:2402.0392// 2024.
[46] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning implicit bayesian inference. In International Conference on Learning Representations, October 2021.
[47] Naimeng Ye, Hanming Yang, Andrew Siah, and Hongseok Namkoong. Pre-training and in-context learning is bayesian inference la de finetti. arXiv preprint arXiv:2408.033 033300 2024.
[48] Fabian Falck, Ziyu Wang, and Chris Holmes. Is in-context learning in large language models bayesian? martingale perspective. arXiv preprint arXiv:2406.0077: 2024.
[49] Hemant Ishwaran, Udaya B Kogalur, Eugene H Blackstone, and Michael S Lauer. Random survival forests. The annals of applied statistics, 2(3):841-860, 2008
[50] Ismaël Castillo, Johannes Schmidt-Hieber, and Aad van der Vaart. BAYESIAN LINEAR REGRESSION WITH SPARSE PRIORS. The Annals of Statistics, 43(5): 2018, 2015.
[51] Niloy Biswas, Lester Mackey, and Xiao-Li Meng. Scalable spike-and-slab. In International Conference Machine Learning, pages 2021-2044 PMLR, 2022.
13

---

[52] W R Gilks, N G Best, and K K C Tan. Adaptive rejection metropolis sampling within gibbs sampling. J. Stat. Soc. Ser C. Appl. Stat., 44(4):455, 1995.
[53] Anthony O'Hagan. Fractional bayes factors for model comparison. Journal of the Royal Statistical Society: Series B (Methodological), 57(1)99-1118, 1995.
[54] Jun Liu, Faming Liang, and Wing Hung Wong. The multiple-try method and local optimization in metropolis sampling. Journal of the American Statistical Association, 95(449):121-134, 2000.
[55] Eirini Papagiannopoulou and Grigorios Tsoumakas. A review of keyphrase extraction. Wiley Interdiscip. Rev. Data Min. Knowl. Discov., 10(2);e1339, March 2020.
[56] Steven M Lewis and Adrian Raftery. Estimating bayes factors via posterior simulation with the Laplace- Metropolls estimator. J. Am. Stat. Assoc., 92(438):648 655, June 1997.
[57] OpenAl. GPT-4 technical report. arXiv:2303.0m 088774, 2023.
[58] Chau Minh Pham, Alexander Hoyle, Simeng Sun, and Mohit lyyer. Topicgpt: A prompt-based topic modeling framework. arXiv preprint arXiv:231 2023.
[59] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-uesd birds 200. 2010.
[60] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
[61] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389-5400. PMLR, 2019.
[62] Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. Mimic-iv, freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
[63] Hiba Ahsan, E Ohnuki, Avijit Mitra, and H You. MIMIC-SBDH: A dataset for social and behavioral determinants of health. MLHC, 149:391-413. August 2021.
[64] Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition. IEEE, June 2018.
[65] imagenette: A smaller subset of 10 easily classified classes from imagenet, and a little more french.
[66] David A. Levin, Yuval Peres, and Elizabeth L. Wilmer. Markov chains and mixing times. American Mathematical Society, 2006.
[67] Roman Vershynin. Righh-Dimensiona Probability: An Introduction with Applications in Data Science, volume 47. Cambridge University Press, 2018.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We carefully ensure the claims made in our abstract and introduction and clearly defined and justified through our methodology (Section and experimental results (Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the
paper.
14

---

The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No NA answer this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The discussion section mentions that the current method is primarily suitable for learning smaller, highly interpretable CBMs. Future work can further scale up the procedure.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification,
asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Assumptions are stated in Section g and full proofs for theorems can be found the Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)? Answer: [Yes)
15

---

Justification:
Experiment details are found in Section 4 and reproducible code is found at our
github.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-soure models, it may be that access to the model is limited in some way (c.g.. to registered users). but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]
Justification: A link to the source code is provided, with specification of all data, dependencies. instructions to reproduce results. The only experiment that cannot be reproduced is Section 4.3 due to the use of PHI data. Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/ public/
wwwiissssoopopiiyoy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https://nig //tp:s/////ppeeeeeeeee cttpiiss.cpppulieee public/ Sriides/CodeSubiisee www.iisssooooicyy.. for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reprodueible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted. Experimental sefting/details
16

---

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Experimental details can be found in the Appendix and Section Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
supplemental material. Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
The full details be provided either with the code, in appendix, Experiment statistical significance
Answer: [Yes]
Justification: Confidence intervals for performance measures are provided in all figures/tables in Section
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical sigmificance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors). should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1 I-sigma error bars, but one should state it. The authors should preferably report 2-sigma eπrоr bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). If error bars are reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experi- ments?
Answer: [Yes]
Justification: The main manuscript discusses compute time and resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud
provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: All ethical guidelines were followed.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
17

---

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts are discussed in the Discussion section at the end of the paper. Guidelines:
The answer NA means that there is societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to par- ticular applications, let alone deployments. However, if there a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: Our research does not release any new data or models
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assests in the paper are cited in the references section.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwwpetss.ppppwstttaaam thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's
creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new method introduced is detailed in Section 3 and is implemented in our repository with instructions for use.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Instructions given to clinicians for rating concepts are provided in the Appendix. Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: IRB approval was obtained for analyzing real-world clinical notes from the hospital, as mentioned the Appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, non- standard component of the core methods in this research? Note that if the L.M is used only for
19

---

writing, editing, or formatting purposes and does not impact the core methodology, scientific nggroousess, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
Our method detailed in Section uses L.LSs propose and refine concepts for
CBMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer our LLM policy should or should not be described.
htt.://pp.....oo..oeeeeee/////// for what
Appendix
A Multiple-Try Try Cettrpolis-Hastiinee
In this section, we provide intuition for the Multiple-Thy Try Metropolis- Hetropolis-Hastingss partial posterior method (Algorithm 3) To draw connections with the original Multiple-Try Metropolis Hasttigg method [54]. we first work in a completely abstract setting.
Given state space Ω2 stationary distribution and proposal transition kernel T xr define weights w0x y) = π (T;; æ). Note that under regular Seetroooiis-Hsttnng the acceptance ratio for proposing the state given state z is min min(w((y (w) (w, v), 1].
Classical version.
We briefly describe [54] 's original method. Suppose the current state is x.
1. Draw 21, 22, 2M ~ 2.
2. Sample y from distribution with probabilities
w( x.) P(Y = 23} Em w (1 x.)
(I)
Let m denote the sampled index.
3. Sample 27...., - ~ T(Z:y) y), and set 2 x.
4. Accept y with probability
M w(2).)) EE w(2
Blz, (iz,v = min
(2)
One can show that this Markov chain satisfies the detailed balance equations when marginalizing out the intermediate states 21, .2, RM, 2i......, RM (see Theorem in (54)). We would like, however, to minimize the number of proposals necessary, so natural ask whether the algorithm can be
modified such that the proposals needed for the backward transitions, .i...., -M can be omitted. This is indeed possible.
Modified version.
1. Draw 21, 22..... . 2M a. Sample from a distribution with probabilities
u(2j, xx)
P(Y = = EM w(2,, x:)
(3)
Let m denote the sampled index, z = (21, 22, ... 2M), x" (21...., 2m-1, 2n .MM))... and set z(y:)) P(Y y/21....,22..
3. Accept y with probability
p(y - 22 x/
alz, y: z.) = min
p(æ + - y)
(4)
x I M
i=1 T(2:8)q x0
min
T(x)
M
20

---

Proposition A.1. The modified Multiple- Try MH sampler is reversible.
Proof. Let (yy x) denote the actual transition matrix. We want to show
<<)) = - (x)))
(5)
To see this, we compute
(x)) v) KT (T)x) /
M (II [
M
M
=k min (()T(<;; v) II 2 x)
II (() 2)
dz-1
2=
= =x))
Connections between modified and classical versions. version of Multiple-Try MH can be expanded as follows:
The acceptance probability in the modified
T T(y) IIM T(z:8)
w(x.y)
T(y) 1=11 T(=N))((1;2))
ンm (
min
π (x) i=1 a(2):))(1;;)) 1 " min
T(x) 1== T(2);
Eisi w((22a)))
Σ Eipn wo(z x)) + w(y x)
M
T(=:v)
min
II s=2 T(an)))
Σ Eiem w(2s, v) w(2 y)
(6)
T--, is invariant in the first argument (as in our application to BC-LLM), then this formula exactly equal B(x, yt z"), the acceptance probability for the classical version, but if were to use the same points for both the non-realized forward and backward proposals.
Equivalence with MULTI-SS-MH-UU UPDATE. Finally, we show that the modified version of Multiple-Try MH described in the previous sections is equivalent to what is implemented MULTI- SS-MH-UPDATE, for fixed S. To see this, we make the replacements:
+ (c c- for i = 1, I,...,M y (c C-
T( (- X)
ys,
T(-
Plugging these into
(6). the acceptance ratio becomes
Q(cxic X) ΣM ,((
min
Q(a(m) k.N.s. s. X) Emen p((c(m), c- +y, X) 11
QCcri Vs. X) EmEr plys |ys (ctm (m)
C... X) c.k.yss
min
QQ(( : : X) EosmsM ply s ys, (e(T)) ,c,)),
where the equality comes from (44) and we set c C₁ for convenience of notation. Observe that this is exactly the formula in Line of Algorithm
Meanwhile, plugging into (3) and using we see that the sampling weights used in the modified version of Multiple-Try MH equivalent to those in Line 3 of Algorithm This completes the proof the equivalence.
B Implementation details for BC-LLM
Here discuss how the hyperparameters for BC-LLM should be selected:
21

---

Fraction of data used for partial posterior: Choosing a small s may lead to the LLM proposing less relevant concepts, but tends lead more diverse proposals. In contrast, large w tends to lead to less diverse proposals because the L.M is encouraged to propose concepts that are relevant the dataset 'D which may not necessarily generalize. experiments, we found that = 0.5 provided good results.
Number of candidate concepts M: More candidate proposals per iteration can allow for more efficient exploration of concepts. The number of candidates is limited by the number of concepts that the LLM can reliably extract in a single batch. There tends to also be diminishing returns, the first few candidates generated by the LLM based the top keyphrases tend to be highest quality and most relevant. We found that setting M 10 provided good performance.
Warm-start and Burn-in: Since Gibbs sampling can be slow converge, we precede it with warm-start, which we obtain by updating concepts greedily. That is, we select the concept that p(7/c_/ C- Ys; X), instead of sampling from the distribution. In experiments, we run warm-start for one epoch and stored the last 20 iterates as posterior samples; the rest of the samples treated burn-in.
maximizes argmax
Number of iterations 7": Although Monte Carlo procedures for Bayesian inference typically have thousands of samples, this is cost-prohibitive when an LLM must query each observation for new concept per iteration. In experiments, we found that even setting 7 as low as 4 to be quite effective. Generally speaking, if the goal is solely prediction accuracy, a small may suffice. On the other hand, if the goal is uncertainty quantification for the relevant concepts, one may prefer T on the higher end (e.g. 10) for more complete exploration of concepts.
C Example prompts
Here provide example prompts that may with BC-LLM. The prompt should vary with how much prior knowledge one has about the prediction target as well as how much information one would like to reveal to the LLM about the prediction target.
C.1 Example prompt for keyphrase extraction (Step 0)
Example prompt for extracting keyphrases from images, where we reveal limited information to the LLM about the prediction target:
Given image, your task is brainstorm set of descriptors that will help classify the image to its corresponding label Y. For the provided image, list many descriptors about it. For each descriptor, also list as many descriptors that mean the same thing or generalizations of the descriptor. All descriptors, synonyms, and gemeralizations cannot be more than two words. Output characteristics that this image possess, such as "round wings" or "red head." " Do not output general categories descriptors like "wing color" or "head shape." Output at least 10 keyphrases.
Example prompt for extracting keyphrases from clinical notes, where do not reveal information about the prediction target but we provide ideas on patient characteristics that one should extract:
Here is a clinical note: note
Output a list of descriptors that summarizes the patient case (such as aspects on demographics, diagnoses, social determinants of health, etc). For each descriptor, also list as many descriptors that mean the same thing or generalizations of the descriptor. All descriptors, synonyms, and generalizations cannot more than two words. Output as a JSDN in the following format...
22

---

C.2 Example prompt for proposing concepts (Step 2)
Example prompt for having the LLM brainstorm candidate concepts based on top keyphrases in the keyphrase model, where we do not reveal information about the prediction target and only reveal that the dataset contains patient notes:
The goal to come up with a concept bottleneck model (CBM) that only extracts 3 meta-concepts from patient notes to predict some outcome with maximum accuracy. meta-concept is a binary feature extractor defined by a yes/no question. have meta-concepts so far:
2.
To come up with the 3rd meta-concept, I have done the following: I first fit CBM on the existing meta-concepts. Then to figure out how improve this 3-concept CBM, I first asked an LLM to extract list of concepts that are present in each note, and then fit linear regression model the extracted concepts to predict the residuals of the 3-concept CBM. These
the top extracted concepts in the resulting residual model, in descending order of importance:
{ top keyphrases from keyphrase model
Given the residual model, create cohesive candidates for the 3rd meta-concept. Be systematic and consider all the listed concepts in the residual model. Start from the most to the least predictive concept. For each concept, check if it matches an existing meta-concept or create new candidate meta-concept. Work down the list, iterating through each concept. Clearly state each candidate meta-concept as a yes/no question.
Suggestions for generating candidate meta-concepts: Do not propose meta-concepts that are simply a union of two different concepts (e.g. "Does the note mention this patient experiencing stomach pain or being female?" is not allowed), questions with answers that are almost always yes (e.g. the answer to "Does the note mention this patient being sick?" is almost always yes), questions where the yes/no options are not clearly defined (e.g. 'Does the note mention this patient experiencing difficulty?" is not clearly defined because difficulty may mean financial difficulty, physical difficulties, etc). Do not propose meta-concepts where you would expect over 95% agreement or disagreement with the 4 existing meta-concepts (e.g. 'Does the note mention the patient having high blood pressure?" overlaps too much with 'Does the note mention the patient having hypertension?"".
Finally, summarize all the generated candidate concepts JSON.
C33 Example prompt for annotations for candidate concepts (Step 3)
A simple concept extraction procedure is to obtain binary or categorical concept extractions. We use prompts like this:
will be given clinical note. will give you a series of questions. Your task is answer each question with 1 for yes or for no. If the answer to the question is not clearly yes no, you may answer with the probability that the answer is a yes. Respond with JSON that includes your answer to all of the questions. Questions:
23

---

1. Does the note mention the patient having social support? Does the note mention the patient having stable blood pressure?
Does the note mention the patient not experiencing respiratory distress?
4. Does the note mention the patient experiencing substance abuse dependence?
5. Does the note mention the patient being uninsured?
clinical note: ( |ote )
Note that in cases where the LLM was unsure about the concept's value, it was allowed to return probability, which prior works have found be helpful [6]. In general, the LLM outputted I's and O's. Probabilities were outputted occasionally, which as highlighted in the main manuscript, can be audited and checked by human.
D Additional Results
D.1 MIMIC
To assess robustness of BC-LLM, we include the following variations. First, we test robustness of BC-LLM to prompt phrasing. In particular, we feed in prompt that is more vague than that used in the main manuseript during Step of BC-LLM by replacing the original sentence "Dutput a list of descriptors that summarizes the patient case (such as aspects on demographics, diagnoses, SDOH, etc) with "Output a list of descriptors that summarizes the patient case. Second, we test the robustness of the results to the choice of LLM by rerunning the same experiment using Cohere's Command-R LLM, rather than GPT-4o-mini.
Results are shown in Figure 5 First, we see that BC-LLM is quite robust to prompt phrasing, as the results with this simpler prompt are similar to the more detailed prompt. Second, we see that performance of all the methods-BC-LLLM and the comparator methods-an are lower using Command-R. Nevertheless, we see that the performance rankings between the methods are similar, with BC-LLM performing better for nearly all metrics.
AUC
Brier
Recall
Precision
0.7
0.25
0.75
Method
0.20
BC.LLM Coherec
0.8
0.15
0.4
LLM LLM+CBM Cahere
Cohere
Human|Drace C/DN
250
500
750
250
500
750
0.00
250
500
750
250
500
750
Bag on-rrdds Cohere
Mam train obs
Num train ebs
Num train obs
Num train obs
Figure 5: Additional results running BC-LLM and comparator methods on the MIMIC dataset, evaluated in terms of performance and recovery of true concepts. Error bars indicate the 95% CI.
D.2 CUB-Birds dataset
Table 2 reports results from additional sets of experiments on the CUB-Birds dataset. First, to evaluate robustness of BC-LLM to the choice of LLM, we report the performance of BC-LLM and comparator methods when Anthropic's Claude 3.5 Haiku is used instead, with all other settings the same as that discussed in Section Second, to facilitate comparisons to other "hard concept" CBM results on the CUB-Birds dataset, we evaluate BC-LLMM on dataset's original task of classifying all 200 bird species, setting K = 100. Because the size of this dataset and the number of concepts is substantially larger, we only ran the greedy phase of BC-LLM. Nevertheless, BC-LLM outperforms the comparator methods.

---

Method
Accuracy (1)
AUC (1)
Brier (1)
Using Claude 3.5 Haiku
BC-LLM
0.665 (0.593, 0.736)
0.849 (0.811, 0.888)
0.443 (0.367, 0.519)
LLM+CBM
0.583 (0.517, 0.648)
0.771 (0.722, 0.821)
0.507 (0.434, 0.580)
Boosting LLM+CBM
0.640 (0.568, 0.712)
0.811 (0.769, 0.852)
0.552 (0.455, 0.649)
Classtfying between all 200 bird species
BC-LLM
0.558 (0.514, 0.600)
0.992 (0.990, 0.994)
0.593 (0.562, 0.625)
LLM+CBM
0.516 (0.474, 0.558)
0.988 (0.984, 0.991)
0.606 (0.567, 0.646)
Boosting LLM+CBM
0.005 (0.003, 0.007)
0.500 (0.500, 0.500)
1.012 (1.011, 1.012)
Table 2: Additional results for the CUB-Birds dataset
D.3
Other image datasets
To evaluate BC-LLM on other real-world image datasets, Table includes two additional imaging datasets:
Functional Map of the World (fMoW) [64] RGB satellite images classified into 62 building/land use categories. Models were trained on satellite images from the US, with 100 images sampled from each category. CBMs were trained to have 60 concepts. Models were evaluated in terms of in-distribution performance (i.e. images from the USA) as well as ofitt--distributiin (OOD)
performance when evaluated on satellite images from China.
Imagenette 665] 10 classes from ImageNet. CBMs were trained to have 10 concepts.
Method
Accuracy (1)
AUC (1)
Brier (4)
[MoW: USA images
BC-LLM
0.357 (0.340, 0.375)
0.904 (0.898, 0.910)
0.780 (0.766, 0.792)
LLM+CBM
0.311 (0.295, 0.327)
0.878 (0.871, 0.885)
0.892 (0.872, 0.914)
Boosting LLM+CBM
0.118 (0.105, 0.130)
0.757 (0.748, 0.764)
1.029 (1.018, 1.039)
MMoW: OOD-Chine images 0.265 (0.246, 0.285)
BC-LLM
0.840 (0.830, 0.849)
0.853 (0.839, 0.868)
LLM+CBM
0.223 (0.205, 0.242)
0.791 (0.780, 0.801)
1.023 (1.00, 1.045)
Boosting LLM+CBM
0.084 (0.072, 0.098)
0.707 (0.693, 0.722)
1.065 (1.054, 1.077)
Imagenette
BC-LLM
0.987 (0.980, 0.993)
0.999 (0.997, 1.000)
0,072 (0.062, 0.085)
LLM+CBM
0.902 (0.883, 0.919)
0.988 (0.986. 0.990)
0.125 (0.107, 0.142)
Boosting LLM+CBM
0.639 (0.610, 0.670)
0.899 (0.888, 0.909)
0.448 (0.422, 0.479)
Table 3: Results on additional imaging datasets
E Experiment details
E.1 Evaluating correct concept matches
Quantifying a "correct" concept match requires going beyond exact string matching, as there are many possible refinements for single concept. For instance, for the concept "Does the note mention the patient smoke at present?", there are rephrasings ("Does this note mention if the patient currently smokes?"), polar opposites ("Does the note not mention the currently smoking?"). or closely overlapping concepts ("Does the note mention the patient's 's smoking history?"). To quantify a learned concept was correct, we used the following rule: if the absolute value of the Pearson correlation for L.M annotations of two concepts exceeds 50%, the two concepts are sufficiently similar. We manually confirmed that any pair of learned and true concepts that passed this rule were always closely related.
25

---

E.2 MIMIC
The entire dataset consists of 7043 observations of which we trained on 100, 200, 400, and 800 randomly selected observations and evaluated 500 held-out observations. We used the "Chief Complaint" and "Social History" sections of the notes. The label was generated per LR model, in which
Pr(Y = 1[x) Pr(Y 0|x)
log
=4 (DDes the note imply the patient is unemployed?)
+ 4 (Does the note imply the patient is retired?}
1 (Does the note mention the patient consuming alcohol in the present the past?}
4 1 (Does the note mention the patient smoking in the present or the past?)
1 (Does the note mention the patient using recre- ational drugs in the present or the past?)
CBM summarization and BC-LLM were run to fit CBMs with K = concepts. BC-LLM was for T 5 epochs. Boosting LLM+CBM ran for 10 iterations, in which most candidate was added each iteration.
E.3 CUB-Birds dataset
To train bird classifier for R subtypes, LLM+CBM summarization and BC-LLM
fit CBMs with K = min 10, max(4, R)) concepts. BC-LLM was run for a maximum = 5 epochs or 25 iterations, whichever was reached first. Boosting LLM+CBM ran for 10 iterations. in which most new candidate was added each iteration. For the black-box comparator, we used a ResNer50 model pre-trained ImageNetV2, which ensures ResNet's training set does not overlap with the CUB-Birds dataset.
E,4
Clinical notes and tabular data from the Zuckerberg San Francisco General Hospital
Data in this experiment contained PHI, for which IRB approval obtained. To predict readmission risk, the models were trained to analyze the sections "Brief history leading to hospitalization" and the "Summarry of hospitalizatton" in the discharge summary for each patient. LLM+CBM summarizati and BC-LLM were run to fit CBMs with K = 4 concepts. BC-LLM was run for T = 5 epochs. Boosting LLM+CBM ran for 10 iterations, in which most I new candidate added each iteration.
E.5 Clinician survey
To conduct the survey for clinical relevance of features and concepts, the following instructions were given clinicians:
Please rate the features learned by BC-LLM and its comparator methods. In the attached CSV, have listed 25 or features from various methods, where we mask which method has learned which feature. Please fill in the column "How clinically relevant is the candidate concept with determining a patient's readmission risk? Enter values 1: low, 2:medium, 3: high." the full range of scores, rather than assigning all the features the same score.
Please try
Table shows shuffled to obfuscate which method had learned which concepts.
4soow the table that clinicians were asked to fill in. The ordering of features was randomly
F Laplace Approximation of Split-Sample Posterior
In this section, we describe how to perform Laplace approximation the split-sample posterior s= yy, c, X) when the likelihood model is logistic and the prior the coefficient vector 0 is standard normal 22T). For simplicity, we omit discussing the constant term fg. Fixing the list of concepts we let & denote the x K matrix whose (i, j)-th entry is given by Φ. = dey (I.).
26

---

Table 4: Clinicians were asked to fill in the following table with their ratings of the clinical relevance of each feature/concept for predicting patient's 's readmission risk.
Candidate concept
Score (1-3)
Does the patient have multiple medications prescribed?
Does the patient have a history of hypertension (HTN)?
Is this patient experiencing chest pain?
What is the patient's 'ss weight?
Does the patient have heart failure with reduced ejection fraction?
What the patient's lab results for BNP?
Does the patient have substance dependence?
Is the patient stable at discharge?
Does the patient have history of falls?
How many emergency department encounters does this patient have?
What are the patient's lab results for Lactate Dehydrogenase?
Does the patient have uncontrolled diabetes?
Does the patient have a history of frequent hospital admissions?
How many encounters does this patient have?
Does the patient have a history of diabetes mellitus type (DM2)?
Does the patient have complex medical history?
Does the patient have a history of sepsis?
What are the patient's ' lab results for Creatinine?
Does the patient have a history of drug or substance use disorder?
What is the patient's 's value for Expiratory Positive Airway Pressure (EPAP)?
Does the patient have had any missed outpatient appointments?
What are the patient's 's lab results for Glucose?
Are the patient's 's outpatient follow-up appointments scheduled?
Given any subset of example indices T C [n]) write the joint likelihood for these examples as
plyr|0,c.X) exp
Σ
yy Φ. - log(1 exp((' &..))
(8)
NET
Multiplying by the prior for 0, the joint conditional distribution for yr and 0 is then:
p(O,yT/e, x) = (2mm?) 1/2 exp
Σ
y2:0" Φ. - log(1 + exp((0 2.1)
|
(9)
2-
IET
To form the Laplace approximation to first denote
(0) = - log p0gp((0,) y X)
Σ
0 Φ. + log(1 exp(O 2.11)
10
(10)
2-
EE
and its Hessian by
( = (2gr(0) = Σ
T. T +y-
(11)
o,
ET
Let
MAP arginin gr
(12)
be the maximum a posteriori (MAP) estimate. We can then perform a quadratic approximation of the exponent around 0MAP. get
p(0, 10,yy/cc c, X) POOMAPPTTyyIC X) X)exp (- (00AP))))) "
(O((AAP)))) - (MAPPT))
(13)
27

---

Plugging in 9
evaluated at 0MAP,T and integrating out with respect to 0 gives
MAPP 12
p R exp
11 y: MAR. P, -log(1
+
277²
(14)
FET
(2x) (K 1 det (H7 (0MA,,77)))
We finish by applying the above formulas with respect to T = [n] and 7' = S and taking ratios:
plys-lys.c.3s p(ylc,X)
p(ys|c,)) X)
Σ
(v.KiAP log(1 exp(O $..))
exp
(15)
(v MAP, Dr log(1 + exp(O
iES
[|0MAP
S 100MPPl ( det((((OMA)))
1/2
eexp
2,2
Remark F.I. The MAP estimate OMAP
be computed by call to scikit-learn's
SeggiticRegressioon () model, setting C C= 2
Remark F22 the current implementation of our procedure, we use a slightly different method perform Laplace approximation, which instead approximates the integral p(ys lys, c, X) p 0, c, X)p(Olys. c, X)d0.
G
Proof of Theorem 3.1
Recall the definition
L(c) == max (lgx
(16)
This quantifies how relevant the concepts in с are to the prediction of the response Y. Note that the definition does not require that the model class is correctly specified (i.e. there does not have to be
some and 0 so that (Y = 1|X x, 0, c) a (Ek= k== 02 (I) + Bb)). Before presenting the proof, we first detail the assumptions we make. Assumption G.1.
E {0. is a binary response;
ii. The data D generated i.i.d. from some distribution Ps iii. The prior on 0 is (0,>~");;
iv. C* == argmax, I(c) comprises all permutations of single vector "
v. There exists some A > 0 such that L(c ") argmaxoge- L(c) 2 A;
vi. Let C₁
x.. Q(c; yc, exp(n' for some 0 < < 1;
0}. Assume C, finite set of size most
vii. There exists > 0 such that for each there exists f1,2...., K} such that oj = y, X) 2 7) almost surely for all y, X with
argmax, L[(c, c.))), of & c-k. and Q(c c. large enough;
viii. There exists some such that for any CK all distinct, we have ||0:1la S R, where
0- = argmaxg
(lo
(17)
ix. Let p. - xx(x) (dh, (X), dog (X),.. Deg (X)) for X ~ μ. There exists some Amin such that for any E K, all eigenvalues of the second moment matrix E[ are bounded from below by Amin- Here, C,, denotes the K-fold product of C, but with any vectors containing duplicate concepts removed;
x. We modify the algorithm so that the random subset S can only take one of exp(n' possibili- ties S, {S1, ... SB..).
Note that for simplicity, we avoid discussing the constant term lg. Most components in Assumption G.1 are either standard regularity conditions ((), (i),, (iv), (v), (vi), (vii), (ix)) that are often made
28

---

in statistical analysis or are hyperparameter choices ((ii) and (x)). The key new assumption is (vii), which asserts that at each step, there is least a fixed nonzero probability of the LLM proposing one of the K "best" concepts conditional on the data. This helps to ensure at every outer loop of the sampler (i.e. for every cycle through all the positions of the concept vector), enough mass flows to the optimal set of concepts. We believe this is a reasonable assumption even in regimes where the LLM has limited knowledge, as long as the degree of knowledge is non-zero.
Proof of Theorem 3.7 clear how to generalize it to MULLT-SS--II-UPDATES
We prove the theorem only for SS-MH-UPDATE for simplicity, but it will be
Step 1: Taking a subsequence. Note that the Markov chain in question corresponds to the list T. returned line of Algorithm I Denote this using (ē = Because of the cyclic nature of the updates, this chain is not time invariant and is hence difficult to work with. We argue that it suffices to study the temporal projection (c ") where C₁ = črK for t = 0, 1, This is because every stationary distribution for (c )) is also stationary for (c ")
Step 2: Reduction to connected components. Since (č ') satisfies the detailed balance equations, it is a symmetric Markov chain, i.e. it has an equivalent representation as a flow on a network, in which the vertices are states and the edges correspond nonzero transition probabilities. This property is inherited by (c ")==1 Consider a connected component A of the network. The Markov chain, when restricted A. is aperiodie, since there must exist self-loop. Furthermore, it is finite according to Assumption G.1 vi). As such, (c ')E= has a unique stationary distribution TA supported on A [66]. Indeed, all stationary distributions of (c f) are convex combinations of such distributions. Hence, it suffices to show the desired property for TAA
Step 3: Concentration. In order to analyze тA, we need quantitative control over the transition probabilities via concentration. For given dataset D of size n, we condition on the event guaranteed by Proposition G.3
Step 4: Convergence. We again focus on a single connected component A and the goal is to show that with high probability, T A) 2 1 - 5 for some 6 - 0 uniformly. We may ignore the rest of the network. Let a = |A| and b = e n Al. Since itself is finite, we can enumerate its states starting with those in C* A.A. The stationary distribution can be written as vector = (V1sb, Vo+1sa), while the Markov chain itself can be represented as a transition matrix P. which we write in block form as
P I1 P12 P21
P
(18)
P22
The stationarity equation gives us
Vo+1:a (-22)) V1.8P12.
(19)
from which we obtain
P PPillop
Vo+11alla = V1. - P22) 1222)2112 - 1- P22llopp
(20)
so long as the denominator on the right hand side is nonzero. To see this, notice that any state j
for b + 1 S j S a corresponds to a vector ē (0) & C*. However, using Assumption G.1 vii), there
is a sequence (é (0) è (1) ...... č (K (K-1)) such that for k = I,..., K, we have (i) é k
[R]
= č_ (k-1)
(ii) c (k-1) ys. X 2 n, (iii) L(e (+) 2 L(e (k-1)) with equality holding iff ē (k) = c (k-1).
Combining (iii) with (88. see that all acceptance ratios along this sequence are equal to 1, which
means that the transition probability from state j to states in C* is at least
K
(k-1), ys. X) 2 K "n.."""...
(21)
k=1
The sum of entries in row j - in P22 is the total probability of transition from state j to other states
within A'C' and hence has value less than 1 - n7 This holds for any row. Applying Lemma G.2 we thus get [PPzelopp < - =,,,, which allows us to bound the denominator in (2)). Next, every
entry in P12 is the probability of transition from a state in C* to a state in A'CC". Using Step 3 and
Assumption G.T [v) the acceptance probability a of such a transition satisfies
log o < max log logply p= e, X) plys lys, 7, ") CEAYC.
(22)
< -n + O(
29

---

We thus have
P122llpp & V(a - exp
(-n + O( ()).
(23)
which allows to bound the numerator in denominator gives
20
Combining this with the carlier bound on the
l|V+t1:el22 V(a- b)by exp (-nA(1 O(n
(24)
We then compute
TA nA) ||
Vo-
Va b|[va+1:el
(25)
v Ma - b)21 exp (-nA(1+0(n
whose error term we can bound as
bia b)3n K exp (-nA(1+0(nn)) 2 CC, cxp (-nA(1 + 0(n-())
I eexp(n' exp [=n((++0nn"
(26)
S exp(O(n
Finally, we note that all hidden constants can be chosen to be independent of n.
Lemma G.2. Let M be a m X n matrix whose entries are non-negative, and suppose ΣE= Mis S B fori ..,,..., m. Then |M S B.
Proof. Let D be the m x m diagonal matrix whose entries are given by Da = Σ}=1 Mij. Then D- is stochastic matrix, which has operator most 1, while | S maxisisa Eラ= Mi S B. Putting these together, we get
|M |ID 'M < |D||op DoM||DDMMooo S,8,
(27)
as we wanted.
Proposition G.3 (Concentration of split-sample posterior). Assuming the conditions in Assumption then any e' e/2, we have
sup eEC,, SES,
[(1 - w)) logp(ys-lys,c. X) I(c) O(n
(28)
with probability converging to
Proof. Let fix for now. We first write
4 (0) = logp(Y = wilX = x,, 0,
0 Φ - log (1 (1+ exp(0" exp(0" 2..))
(29)
Let T(0) = Ex.., [(()]] and set 0" = argmax I(0). For convenience, denote + = 4e. Note that has entries bounded between and 1 and hence has a second moment matrix with max- imum eigenvalue bounded above by K. Furthermore, it is sub-Gaussian random vector with bounded sub-Gaussian norm. Using these properties, we may find a universal value R' such that E
R') E [44" Consider the ball B(2R) of radius 2R. For any 0 E B(2R).
30

---

the Hessian of the population log likelihood hence satisfies
H = (
exp( T 0)
-E
(1 + exp 4 ()) DΦL
exp($T0) 0)
-E
DΦ " 1(1 R')
(1 (1+ expe
exp(v"') 0)
(30)
inf
(1 exx((xv 0))² F. [
VER(T))
RRRRR
2(1 + 22
Amnn 2R' R
2(1 + 22 I.
Using this lower bound on the curvature, we see that
T(0) 2 Amin 1t2 2
T(0')
sup (EBB(2R)
2(1 + 22R'R)2
(31)
Next, we use the fact that n 1/2 Di= (1:(0) - I(0)) is a sub-Gaussian process with increments bounded according to
1/2 (1) 112 (4(0')-((0)
KK 0'112-
(32)
8/
We can thus use Talagrand's 's comparison inequality 67 to get, for any t> 0, that
1/2 sup 0e B(BR)
Σ (4(0)) ((0))-(())
SRVK Rt
(33)
with probability at least e n Ei 400) is concave, with
Take 1/2-dd for any é' e/2. Note that (0)
(1(0') 2 1(00) O(n
(34)
while
(0) < sup 8E0B(28)
i(0) + O(n
sup @EAB(2R)
(35)
< I(00) + Amin R2 2 2n' n
+ O(n
2(1 + 22 RyZ
As such, whenever is large enough, we see that i, also attains its global maximum in B(2R) and that this maximum value satisfies
sup i.(()) - i(00) = Onn (n)). ØERK
(36)
Recall further that I(0") L(c). We next want to integrate out p(yl0, c, X) = exp(nin with respect the prior. Using (36), get the upper bound
plylc,X)
ply O, X)
(xp(-11013/277111
(2/7?))K/2
(37)
I exp(nl((c) + (m)
31

---

To get a lower bound, we use the fact that the Hessian of the population likelihood also has an lower bound
H = H lap ž KI.
(38)
For any 0 < < R, this gives lower bound of
inf (E in)
L( L( VKr²
(39)
where B(0", r) is the ball of radius centered at 0" Using 33 again, we get
inf (0) 2 L( K O(n=
(40)
BEB(0 .))
The prior density is bounded from below by (2 exp(-21P //2) B(2R). We therefore have
/ plylo, X)
(2TT2)K/2 K/2 cxpl cxp(-1101
(41)
K (o1(B()) exp( 22-2 //2) exp (n(L(0)) - VKr² -(n
(2m72)K/2
The maximum is achieved when we set 1/4 we get
12n) Combining
37 and and taking logarithms,
loy c, X) nL(c) + O(n
(42)
Performing the same calculation with ys instead of y and taking differences, we get
[(1 w)n pypplysolss.co
1
L(c) O(n Onn
(43)
Finally, we take a union bound over all E Cn, and S E S,, noting that all hidden constants can be chosen be independent
H Stationary Distribution Under Consistent Proposals
In this section, we motivate the multiple-try partial posterior Metropolis update (Algorithm 3 in more detail. Notably, have the following proposition.
Proposition H.1. Suppose the L.LM proposes from the conditional partial posterior distribution,
i.e. Q(Cr; C_ c_k) = p(Cglo_R, ss X). Then the Markov chain defined by running Gibbs
sampling (Algorithm 7 with SS-MH-UPDATE or COLTI-SS-MH-UUPDE instead of MH-UPDATE has the posterior p( C as stationary distribution.
Proof. We first show this for SS-MH-UPDATE. suffices to show that the acceptance probability in 2 is equal to that of Metropolis-Hastingg filter with p( (Clyy, X) as the target.
Line 4 of Algorithm
We compute:
c((c,c-x)ly,x)- plys ys, (c. X)pys (e.Dypsss.c.......... (c.c..))
ply[X)
(44)
p(yss (c,ccL)) X PsC
p(y(X)
Hence,
p((c X)Q(c cs, p((E,c=)) X, C- C=kl,,
p((c,c_x)) ()p((()((C C b)y,, X)
(45)
plys- lys, (c 8). X)
as we wanted.
To show the statement for MULT--SS-MH-UPDAEE we use its equivalence, for a fixed S, the modified Multiple-Try Metropolis-Hastine method described in Appendix [A] In Appendix A we prove that this method satisfies the detailed balance equations for the posterior. Since the detailed balance equations are linear in the transition probabilities. we can marginalize this over S to show that the posterior is stationary with respect for MULTI-SS at random.
which draws S uniformly
32


Reviews:
[{"review_id": "LeR3OV66Pz", "reviewer": "Reviewer_JKHs", "summary": "This paper presents a BC LLM, i.e., a bayesian approach for concept bottleneck models. The key insight here is to use LLMs as priors and concept annotators while doing inference via a split-sample metropolis-within-Gibbs scheme. The paper also presents a theorem that suggests that the markov chain defined by running this split-sample scheme then the BC-LLM will asymptotically recover the true set of concepts that optimize the log-likelihood.  The BC-LLM is initialized with top concepts from an LLM. The concepts are then refined via a query replacement scheme. The paper then shows the benefits of the BC-LLM for bird-classification, clinicals, and for a tabular model.", "questions": "- How can one recover from poor-quality concept suggests at the start? \n- What is the cost for additional K? Specifically, how do you expect the computational cost to change with increasing concept size. \n- Can the LLM hallucinate concepts valid for training, but not relevant for test? How should the uncertainty adapt in this case?", "limitations": "- Scalability to larger concepts as the paper mentioned.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths \n\n- Interesting Theorem to justify selection: the theorem offers a consistency guarantee, BC-LLM will asymptotically recover the true set of concepts that optimize the log-likelihood.\n\n- The result is inspired by a well-established literature on Bayesian model selection and is nontrivial to extend to an infinite concept space proposed by LLMs. The split-sample scheme is nice, and practically effective.\n\n- Evaluation is thorough, diverse, and realistic: covering simulated and real-world clinical text, vision (CUB birds), and multi-modal EHR data.\n\n## Weaknesses\n- Unclear implication of assumption H.1. It would be great to include a high-level summary of the justification for this. \n- What if all the initial concepts suggested by the LLM are not related to the task? As with all metropolis hastings like schemes, the sampling complexity here might be high. \n- Scalability, to large number of concepts, is a challenge, but the authors already mention this. \n- Some of the figures are small and could be bigger.\n\n- Ablation against LLM+CBM and Boosting LLM+CBM are fair and convincing. Concept interpretability was validated through expert clinician evaluation, adding real-world credibility.", "was_revised": true, "final_justification": "I have read the rebuttal from the authors, and their response. I believe this is a worthy contribution. They have successfully addressed my concerns regarding initial concept suggestion quality, and scale.", "rebuttal": "We thank the reviewer for the thoughtful comments. We are heartened to hear that you found the Bayesian framework to be interesting and nontrivial and the experiments to be convincing. This work was inspired by our real-world collaboration with the local hospital to build risk prediction models, where there is a major need for methods that learn fully interpretable clinical risk prediction models for unstructured data. We believe BC-LLM, by bringing together the world knowledge of LLMs and the rigor of Bayesian inference, has the potential to address this critical gap. Thus we sincerely appreciate the reviewer's feedback on how the manuscript can be further improved and address the reviewer's points below:\n\n> Unclear implication of assumption H.1. It would be great to include a high-level summary of the justification for this.\n\nAssumption G.1. is used to prove Theorem 3.1, which states that MH sampling with a potentially misspecified LLM prior and proposal distributions can still be robust and converge to a desired target distribution. This result is highly unique, because most works on Bayesian analysis analyze an explicitly defined posterior and do not consider the effect of misspecification in the sampling process on the target distribution. Thus additional assumptions are required. Most components in Assumption G.1 are either standard regularity conditions ((i), (ii), (iv), (v), (vi), (viii), (ix)) that are often made in statistical analysis or are hyperparameter choices ((iii) and (x)). The key new assumption is (vii), which asserts that at each step, there is at least a fixed nonzero probability of the LLM proposing one of the K “best” concepts conditional on the data. This helps to ensure at every outer loop of the sampler (i.e. for every cycle through all the positions of the concept vector), enough mass flows to the optimal set of concepts. We believe this is a reasonable assumption even in regimes where the LLM has limited knowledge, as long as the degree of knowledge is non-zero.\n\n> What if all the initial concepts suggested by the LLM are not related to the task? As with all metropolis hastings like schemes, the sampling complexity here might be high.\n\nThank you for this question! First, we would like to highlight that this work was motivated by this very concern you have raised. Most existing methods ask LLMs for a set of candidate concepts or rely on a pre-existing concept list, but this is not effective when LLMs or humans have limited prior knowledge of which concepts are truly relevant. BC-LLM addresses this by iteratively refining the set of concepts during posterior inference.\n\nSecond, we agree that if an MH sampling procedure is initialized poorly, sampling complexity will be high. Because of this very concern, the initialization and warm-up steps of BC-LLM are specifically designed to accelerate convergence. For initialization, the LLM explores a large variety of concepts by brainstorming keyphrases that describe each observation and then selects concepts based on the keyphrases that are most predictive. For warm-up, the LLM greedily chooses concepts that improve prediction performance, again based on keyphrases that are most associated with the residual. Together, these two steps ensure the MH sampling procedure starts closer to the posterior.\n\n> Scalability to a large number of concepts (K) is a challenge… How do you expect the computational cost to change with increasing concept size?\n\nBecause the primary goal of this manuscript is to introduce a Bayesian formalization for learning CBMs, there is still room for improvement when it comes to the scalability of BC-LLM. Currently, the runtime of BC-LLM is linear in the number of concepts K, because each iteration of Gibbs refines/replaces a single concept. A straightforward way to speed this up is by refining J>1 concepts in each iteration, which reduces the runtime by a factor of J. Our preliminary experiments show that BC-LLM achieves similar results when doing this for the greedy warm-up procedure. However, extending the Gibbs sampling procedure to refine/replace multiple concepts in a single iteration while preserving statistical guarantees is beyond the scope of this paper, and we leave it to future work.\n\n> Can the LLM hallucinate concepts valid for training, but not relevant for test? How should the uncertainty adapt in this case?\n\nDuring the posterior sampling procedure, it is certainly possible for the LLM to propose irrelevant concepts (e.g., because it fits the training data well). Nevertheless, for a sufficiently large dataset, irrelevant concepts are likely to be rejected since they don’t fit the held-out data well. For small datasets where there is insufficient sample size to distinguish between relevant and irrelevant concepts, irrelevant concepts are more likely to be accepted, but the posterior sample will also reflect this increased uncertainty by including a larger variety of concepts. Indeed, we observe such behavior in Figure 3 (right): the posterior samples for the smaller dataset reflect high uncertainty (many unique concepts) while the posterior samples for the larger dataset reflect low uncertainty (few unique concepts)."}, {"review_id": "HCeyyhGqKS", "reviewer": "Reviewer_EwWN", "summary": "This paper proposes BC-LLM, a Bayesian framework that enhances Concept Bottleneck Models (CBMs) by leveraging Large Language Models (LLMs) to explore an open-ended space of interpretable concepts. Unlike standard CBMs that rely on a fixed concept set, BC-LLM uses LLMs as both concept extractors and priors, enabling scalable and flexible concept discovery with uncertainty quantification. Despite potential LLM hallucinations, the authors provide theoretical guarantees. Experiments across image, text, and tabular domains show that BC-LLM outperforms interpretable and black-box baselines, converges faster to relevant concepts, and is more robust to distribution shifts.", "questions": "1. Generalizability to Other Image Domains:\nCan this approach be applied to different image classification datasets beyond those used in the paper?\n\n\n2. Applicability to Domain-Specific Tasks:\nHow would this method perform on tasks that require highly domain-specific knowledge, where LLMs may lack sufficient prior understanding, such as satellite imagery or industrial/factory imaging? Is BC-LLM still applicable or effective in these cases?\n\n3. Prompt Design for Concept Annotation:\nCould you provide the exact prompt used in Step 3 for concept annotation in the bird image classification task (Appendix)? How are concept annotations generated across different modalities?\n\n\n4. Entropy in OOD Settings:\nIn Table 1, the ResNet model shows the highest entropy in the out-of-distribution (OOD) setting. Could you clarify why this might be the case?\n\n5. Related Work Suggestion:\nYou may want to cite the paper \"Constructing Concept-based Models to Mitigate Spurious Correlations with Minimal Human Effort\" (ECCV 2024), which also uses LLMs for concept candidate generation, filtering, and annotation. It seems closely related and could help contextualize your contribution.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. Presents a novel approach to building Concept Bottleneck Models (CBMs) by leveraging Large Language Models within a Bayesian framework.\n\n\n2. Demonstrates versatility by applying the method across multiple modalities, including image, text, and tabular data.\n\n\n3. Offers a more lightweight yet effective CBM by selectively identifying and using only the most relevant concepts, reducing annotation cost and complexity.\n\nWeaknesses:\n\n1. Although the method is evaluated across different modalities, the dataset diversity—particularly for complex, real-world image domains, is limited. This raises concerns about the method's robustness in settings where human-defined concepts are ambiguous or less clearly represented.", "was_revised": true, "final_justification": "The reviewer addressed some of my concerns during the rebuttal. However, I still find that the approach is limited in domains that require highly specific knowledge not captured by LLMs. Therefore, I will maintain my original score.", "rebuttal": "We thank the reviewer for their thoughtful comments and for recognizing the novelty of this Bayesian framework for learning CBMs. This work was inspired by our real-world collaboration with the local hospital to build risk prediction models, where there is a major need for methods that learn fully interpretable clinical risk prediction models for unstructured data. We believe BC-LLM, by bringing together the world knowledge of LLMs and the rigor of Bayesian inference, has the potential to address this critical gap. Thus we sincerely appreciate the reviewer's feedback on how the manuscript can be further improved and address the reviewer's points below:\n\n> Generalizability to Other Image Domains and Applicability to Domain-Specific Tasks: Can this approach be applied to different image classification datasets beyond those used in the paper? How would this method perform on tasks that require highly domain-specific knowledge, where LLMs may lack sufficient prior understanding, such as satellite imagery or industrial/factory imaging?\n\nThank you for these two questions about the robustness of BC-LLM to other real-world image datasets\\! To illustrate the generalizability of BC-LLM to other image domains, we have run experiments on two additional datasets:\n\n* *Functional Map of the World* (fMoW) (Christie et al., 2018): RGB satellite images classified into 62 building/land use categories. Models were trained on satellite images from the US, with 100 images sampled from each category. CBMs were trained to have 60 concepts. Models were evaluated in terms of in-distribution performance (i.e. images from the USA) as well as out-of-distribution performance when evaluated on satellite images from China. As the reviewer mentioned, these satellite images may require more domain expertise than the CUB dataset included in the original submission.\n* *Imagenette* (https://github.com/fastai/imagenette): 10 classes from ImageNet. CBMs were trained to have 10 concepts. Note that we do not run on all of ImageNet, since BC-LLM is not intended for learning CBMs with a very large number of classes.\n\nThe table below reports the accuracy of BC-LLM as well as comparators (we exclude human+CBM because there are no available human-generated concepts for these datasets). As we see, BC-LLM outperforms the CBM methods and even outperforms ResNet on the fMoW dataset.\n\n|  | fMoW (USA images) | fMoW (OOD-China images) | Imagenette |\n| :---- | :---- | :---- | :---- |\n| BC-LLM | **0.36** | **0.27** | **0.99** |\n| LLM+CBM | 0.31 | 0.22 | 0.90 |\n| Boosting LLM+CBM | 0.12 | 0.08 | 0.64 |\n| ResNet | 0.30 | 0.17 | **0.99** |\n\nWhile these results are promising, we acknowledge that for highly specific problem domains, BC-LLM may require some tailoring. For instance, rather than using a general LLM as we do in the experiments, one may instead need to use a domain-specific LLM (e.g. a medical LLM). Furthermore, if there is no LLM that can accurately extract values for domain-specific concepts in a zero-shot manner, one may need to train/tune a model specifically to extract domain-specific concepts.\n\n> Prompt Design for Concept Annotation: Could you provide the exact prompt used in Step 3 for concept annotation in the bird image classification task (Appendix)? How are concept annotations generated across different modalities?\n\nHere is the prompt used for concept annotation (Step 3\\) in the bird image classification task. The image is uploaded to the LLM through the API alongside the prompt\n\n*\"You will be given an image. I will give you a series of questions. Your task is to answer each question with 0 or 1\\. You may output a probability if you are unsure, but avoid this in general. Summarize the response with a JSON that includes your answer to all of the questions. Questions:*\n\n1. *\\<CONCEPT QUESTION 1\\>*\n\n2. *\\<CONCEPT QUESTION 2\\>*\n\n3. *…*\n\n*Output your answer in JSON\"*\n\nWe fill in the prompt with the concept questions for which we want annotations.\n\n> Entropy in OOD Settings: In Table 1, the ResNet model shows the highest entropy in the out-of-distribution (OOD) setting. Could you clarify why this might be the case?\n\nResNet can achieve higher entropy than CBM models because CBMs are highly constrained: CBMs have only a small number of input features by design, and the values of these input features are further constrained to be interpretable (e.g., a probability). As a consequence, a CBM may not be able to fully represent the additional uncertainty in OOD settings. Nevertheless, because a CBM is easy to audit by humans, it is also easier to debug than a ResNet model. By simply observing the extracted concept values, one is likely able to detect whether an observation is OOD. Furthermore, as shown in the new experiment with fMoW, we find that BC-LLM can outperform ResNet in terms of OOD performance in certain cases.\n\n> Related Work Suggestion: You may want to cite the paper \"Constructing Concept-based Models to Mitigate Spurious Correlations with Minimal Human Effort\" (ECCV 2024), which also uses LLMs for concept candidate generation, filtering, and annotation. It seems closely related and could help contextualize your contribution.\n\nThank you for this suggestion – we have added a discussion on it to our related works section. The paper highlights studying correlated concepts and is very similar to the “LLM+CBM” baseline we compare against. The iterative Bayesian framing we propose provides an extra layer of uncertainty quantification and enables finding more concise models than the method proposed in the suggested paper."}, {"review_id": "DrIzVsvNaE", "reviewer": "Reviewer_igan", "summary": "The paper introduces a novel method that incorporates Large Language Models (LLMs) into Concept-Based Models (CBMs) using a Bayesian framework, with inference performed via Metropolis-Hastings sampling. The authors suggest that LLMs can act as informative priors, effectively guiding concept discovery, particularly in scenarios with limited prior information. Experiments on medical applications and bird image data suggest that  the proposed framework yields improved performance compared to the considered baselines.", "questions": "Even though the authors consider MC approaches, there are some other works in the literature concerning variational methods for concept selection. Wouldn't it also make sense to prompt the LLM once for concepts and then consider variable selection methods for the final subset?\n\nI kindly ask the authors to provide the wall time measurements for the considered experiments. \n\nDid the authors compare their approach to standard CBM benchmark settings such as CUB with the full 200 classes and not the 37 \"families\"? Did the authors also consider, SUN, Places365, ImageNet, or any other image dataset?\n\nCan the authors elaborate on the OOD setup? Did the authors consider all the remaining 50% of the data (the testing split) for their computations? Can the authors also provide an accuracy metric for OOD?\n\nIn p.7 line 308 the authors mention \"The difference when applying the Bunting bird CBM to an actual bunting bird versus an OOD sample\". What is a Bunting bird CBM? The authors consider a single CBM with multiple prediction targets correct?", "limitations": "No, there isn't a dedicated limitations subsection. \n\nEven if some potential issues of the proposed framework are hinted at various parts of the manuscript, it should be clear for the readers what are the limitations of this work.  For example, one limitation seems to be complexity. Even though the authors discuss the complexity in terms of O() notation and then mention that \", BC-LLM was quick to run, as one iteration takes less than a minute for text data and a few minutes for image data on a normal laptop\". Another limitation can be the impact of the design of the input prompt or the parameters used for the LLM/VLM. None of these are discussed.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The proposed methodology—approaching concept refinement from a Bayesian perspective and leveraging LLMs as informative priors rather than mere knowledge sources—is both novel and well-motivated. This framing is particularly compelling given the challenge of applying Concept-Based Models (CBMs) in real-world settings where concept annotations are scarce. \n\nThe paper begins by listing well-known limitations of  CBMs—such as restricted context due to training data, computational cost, the trade-off between interpretability and accuracy and hallucinations. Even though these are indeed issues of CBMs,  I find that the authors do not convincingly show how the proposed method overcomes these issues. In terms of accuracy/interpretability, the authors base their experimental evaluation on a highly irregular regime that may be more favorable for the proposed framework but it does not provide adequate comparison to SoTA CBM approaches. At the same time, the interpretability evaluation of the approach in the context of augmenting a tabular model with clinical notes was only evaluated by \"four clinicians\", rendering the interpretation of the results ambiguous.\n\nThe computational cost is again something that is not clear. Yes, some CBM methods do use thousands of concepts but there are recent methods that consider sparsity inducing methods to effectively select the concepts; at the same time, I'm confident that using GPT4o-mini for querying for relevant concepts will be more costly (see also the limitations section). \n\nIn this context, the authors employ GPT-4o-mini—a vision-language model rather than a pure LLM—which introduces ambiguity regarding the method's dependence on prior visual/textual knowledge. The claim that BC-LLM outperforms other LLM-based CBMs in low-knowledge regimes is not convincingly demonstrated here (and leakage that was mentioned in the context of standard CBMs can also be an issue here). \n\nThe authors mention that \" we prove that BC-LLM can provide rigorous statistical inference and uncertainty quantification\" and this claim is spread throughout the manuscript; however, it is weakly supported by the experimental evaluation and it's only captured by the Brier metric and the entropy in the OOD setting but without substantial insights.\n\nOne other major concern lies in the design of the prompts used (appendix C), and since the quality and effectiveness of concept extraction can be highly sensitive to prompt phrasing, a more thorough analysis—such as a prompt ablation study—would substantially strengthen the experimental section.\n\nOverall,  this work aligns with a growing trend of leveraging LLMs for enhancing interpretability, particularly within the context of Concept-Based Models (CBMs). While the authors aim to improve concept quality through a statistical framework, the fundamental limitations of using LLMs remain largely unaddressed. The method relies heavily on the LLM at multiple stages of the pipeline, making the resulting interpretations highly sensitive to factors such as the specific model used, decoding parameters (e.g., temperature), and the nature of the task itself. As a result, the robustness and consistency of the interpretations may vary significantly, raising concerns about their reliability and generalizability. At the same time, the experimental evaluation, in my view, does not demonstrate substantial improvement or provide the necessary insights to justify the original claims of the authors.", "was_revised": true, "final_justification": "Taking into consideration the discussion with the authors and their responses to my and the other reviewers' concern, I raise my score from 2 to 3. Even though I find the use of a Bayesian approach appropriate and novel in the context that it is used, I still find that the manuscript is missing some important information/experiments/discussion to fully justify the contribution. These pertain to the comparison with other CBM approaches and the highly irregular experimental regime as noted in the initial review. Even though I can understand the benefit in the \"low-knowledge\" regime, I still found that there were some conflicting statements, e.g., using a more targeted LLM for those settings, that could potentially invalidate the contribution of the approach (given that a domain appropriate LLM could yield similar or better results to the proposed approach). \n\nThe authors did mention that some additional ablation studies and experiments will be included in the revised manuscript but I cannot assess the results without having them. Thus, I still believe that the approach is missing some key elements towards publication, that is why I cannot recommend acceptance at this time.", "rebuttal": "We thank the reviewer for their detailed comments and for recognizing that our method is both novel and well-motivated. Before addressing each of the reviewer’s points, we believe it would be fruitful to highlight that the goals of this work significantly differ from prior work. The goal is to learn a fully-interpretable non-leaky CBM in the “low-knowledge regime” where it is difficult to list potentially relevant concepts upfront. This is a major unaddressed need in healthcare, as clinical providers often want fully interpretable, faithful models that can predict (often noisy) outcomes using the unstructured data and have limited insight into which candidate concepts are good to include. In fact, BC-LLM is inspired by our real-world collaboration with Anonymous hospital to build a risk prediction model, which is the experiment in Section 4.3 of the paper. As illustrated in this experiment, existing CBM methods do poorly because they are not designed for the low-knowledge regime. In contrast, BC-LLM does much better.\n\n**Choice of experiments**: The reason we do not copy existing experiments from other CBM works is that they are not designed to test the ability of a CBM to learn concepts in low-knowledge regimes. Existing works test the setting where the LLM already has good knowledge of the prediction task and is able to list a comprehensive list of candidate concepts. In contrast, our experiments are designed to simulate the low-knowledge regime as follows:\n\n* In the MIMIC example, we have real-world clinical notes and simulate the label Y based on a hand-annotated set of concepts. Because the label Y is simulated and we only tell the LLM the outcome is “some label Y”, the LLM is unlikely to know what set of concepts we actually used in the simulation. Furthermore, using this semi-synthetic setup, we can evaluate if BC-LLM converges to the known true concepts.\n* In the readmission risk prediction example based on real-world clinical notes, we use a dataset that is private to the Anonymous hospital, thereby eliminating the possibility of an LLM having seen the data. Furthermore, readmission is known to be very difficult to predict, and factors influencing readmission are known to be highly variable across different hospitals. The factors picked up by BC-LLM are indeed *uniquely* relevant to Anonymous hospital, where a substantial subgroup of patients exhibit substance use.\n* For the CUB example, we designed the task to be slightly different from prior setups to minimize the chance of the LLM already knowing the answer. To do this, we do not reveal to the LLM that the task is to predict bird species and change the prediction task. Rather than predicting one of 200 bird species, the task is to predict a bird species within its bird family (a set of closely related bird species, e.g. a Bunting bird family is composed of three different Bunting bird species). Furthermore, predicting within bird families is better aligned with the fact that (i) our primary goal is to learn highly interpretable CBMs with fewer concepts and (ii) there likely exists a smaller CBM for distinguishing bird species within a family. In contrast, a CBM with a large number of concepts is no longer easily interpretable.\n\nAdditional experiments, including OOD performance: We have added the following\n\n* *Original CUB-200 dataset*\n* *Functional Map of the World* (fMoW) (Christie et al., 2018): RGB satellite images classified into 62 building/land use categories. Models were trained on satellite images from the US, with 100 images sampled from each category. CBMs were trained to have 60 concepts. Models were evaluated in terms of in-distribution performance (i.e. images from the USA) as well as out-of-distribution performance when evaluated on satellite images from China. As another reviewer mentioned, these satellite images may require more domain expertise than the CUB dataset included in the original submission.\n* *Imagenette*: 10 classes from ImageNet. CBMs were trained to have 10 concepts.\n\nThe table below reports the accuracy of BC-LLM as well as comparators. As we see, BC-LLM outperforms the CBM methods and even outperforms ResNet on the fMoW dataset.\n\n|  | fMoW (USA images) | fMoW (OOD-China images) | Imagenette | CUB-200 (prediction for all 200 bird species) |\n| :---- | :---- | :---- | :---- | :---- |\n| BC-LLM | **0.36** | **0.27** | **0.99** | 0.56 |\n| LLM+CBM | 0.31 | 0.22 | 0.90 | 0.47 |\n| Boosting LLM+CBM | 0.12 | 0.08 | 0.64 | 0.005 |\n| ResNet | 0.30 | 0.17 | **0.99** | **0.74** |\n\n**Adequate comparison to SoTA CBM approaches**: To our knowledge, the paper includes all major non-leaky CBM comparators: (A) Fitting a CBM on human-selected and annotated concepts (referred to as “Human+CBM”), (B) Using an LLM to suggest and annotate concepts and then fitting a variable selection method on these concepts (referred to as “LLM+CBM (no keyphrases)”), (C) Using an LLM to brainstorm keyphrases that describe each observation, fitting a sparse model on these keyphrases, and having the LLM suggest concepts based on the top keyphrases (“LLM+CBM”), and (D) Using an LLM to suggest concepts in a boosting procedure (“Boosting LLM \\+ CBM”). Note that we do not include CBM methods that rely on “soft” (i.e. continuous-valued) concept extractors, as they can suffer from information leakage and thus are not fully interpretable.\n\n**Interpretability evaluation**: We appreciate these concerns and will ask for input from more clinicians for the revised manuscript. That said, we would like to highlight that the average relevance score for BC-LLM is \\>1 standard error higher than that for the other methods. Getting input from clinicians is rare for AI/ML conference papers, and having input from four clinicians is substantially more than most. We will include error bars to Fig 4 in the revised manuscript.\n\n**Rigorous statistical inference and uncertainty quantification**: BC-LLM is supported by rigorous theoretical analyses, faster convergence to the correct concepts, lower Brier score, and higher entropy in OOD settings. We have also demonstrated better performance in OOD settings in the additional experimental results shown above.\n\n**Computational cost**: The average wall times for BC-LLM are: 1 minute per epoch for Section 4.1 (short clinical notes), 3 minutes per epoch Sec 4.2 (CUB), and 1.2 minutes per epoch for Section 4.3 (long clinical notes). Each BC-LLM experiment was run for 5 epochs. The cost for BC-LLM is around \\\\$0.0015 per request for CUB. On average, this translates to around \\\\$3 per experiment. The cost for MIMIC is around \\\\$0.0005 per request, which translates to around \\\\$1 per experiment. We believe these costs are highly reasonable. We note that this is all without requiring human annotations for the concepts, which is significantly more costly. Furthermore, the concepts in BC-LLM are not extracted through embedding similarity-based measures, which are known to be leaky and more error-prone.\n\n**Ablation studies to study sensitivity to LLM dependence**: We have investigated sensitivity to the choice of the LLM in Section D of our appendix, where we show results from using BC-LLM with Cohere’s Command-R and Claude 3.5 Haiku instead. We consistently find that BC-LLM outperforms comparator methods. We used a temperature of zero to minimize stochasticity and default parameters otherwise. We used very basic prompts (Section C of our appendix) to reflect the “low-knowledge” regime. The revised manuscript will explore how performance changes if we conduct further prompt engineering.\n\n**Related works**: We have additionally added some discussion on variational methods such as [2]\n\nWe hope our responses clarify any points of confusion and demonstrate the novelty and importance of this work. If possible, we kindly ask the reviewer to reassess their score.\n\n\\[1\\] Ragkousis et al. \"Tree-Based Leakage Inspection and Control in Concept Bottleneck Models\" arXiv:2410.06352\n\n\\[2 \\] Panousis et al. \"Coarse-to-Fine Concept Bottleneck Models\" NeurIPS 2024"}, {"review_id": "su6pHttKXq", "reviewer": "Reviewer_EHr8", "summary": "To have interpretable model, concept bottleneck models require expensive human expert annotation on the data. This paper aims to reduce this cost by letting LLMs generate the concept and annotate the dataset instead. The authors propose a split-sample update to reduce the impact of LLM hallucination. Experiment results show the proposed method performs better than interpretable baseline models.", "questions": "- I still don't fully understand how to use the proposed method to make prediction on a new data point. I think we need to first let the LLM annotate the new data point based on learnt concepts, but then how do you make prediction? Also, how do you account for the error that might occur during the LLM annotation?\n\n- Citation is missing in Line 234 in appendix.", "limitations": "See weakness above.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n- The problem this paper aims to solve is quite interesting.\n- The idea of using LLMs within a rigorous Bayesian framework to achieve reliability is quite neat.\n\nWeakness\n- No thorough analysis is conducted on the LLM extracted concept. It is unclear how interpretable or trustworthy these concepts actually are.\n- In the iterative procedure of generating new concepts, in each iteration, all training data are used. This can be highly inefficient once the data set gets larger and/or the iteration becomes large.\n- Calibration is measured by Brier, which seems to me is an odd choice. ECE is a more standard measurement of calibration in the literature?\n- The writing could be improved, I find the paper hard to follow. Also, the notation is a bit messy. A vector is represented by bold symbol and upper arrow at the same time.", "was_revised": true, "final_justification": "My concerns regarding the interpretability is successfully addressed by the rebuttal and therefore I raise my score to 5 (accept).", "rebuttal": "We thank the reviewer for their thoughtful comments and are heartened to hear that you find the method quite neat\\! This work was inspired by our real-world collaboration with the local hospital to build risk prediction models, where there is a major need for methods that learn fully interpretable clinical risk prediction models for unstructured data. We believe BC-LLM, by bringing together the world knowledge of LLMs and the rigor of Bayesian inference, has the potential to address this critical gap. Thus we sincerely appreciate the reviewer's feedback on how the manuscript can be further improved and address the reviewer's points below:\n\n> No thorough analysis is conducted on the LLM extracted concept. It is unclear how interpretable or trustworthy these concepts actually are.\n\nThank you for this question\\! We can use the semi-synthetic MIMIC experiment to check how trustworthy the extracted concepts are, because we have the human annotations for the true concepts. In particular, we have human annotations for “Does the note mention the patient consuming alcohol in the present or the past?”, “Does the note mention the patient smoking in the present or the past?”, “Does the note mention the patient using recreational drugs in the present or the past?”, “Does the note imply the patient is unemployed or retired?”. BC-LLM converged to similar concepts with the following agreement rates with the human annotations:\n\n* For the alcohol concept, BC-LLM learned concept questions like “Does the note mention this patient having a history of alcohol use disorder?” and “Does the note indicate that the patient consumes alcohol?”. The agreement rate of LLM extractions with human annotations is 91.5%.\n* For the tobacco concept, BC-LLM learned concept questions like “Does the note mention the patient’s smoking history?” and “Does the note mention the patient as currently smoking?”. The agreement rate of LLM extractions with human annotations is 85.9%.\n* For the recreational drugs concept, BC-LLM learned concept questions like “Does the note mention the patient having a history of cocaine use?”. The agreement rate is 95.1%.\n* For the unemployment/retired concept, BC-LLM learned concept questions like \"Does the note mention the patient being retired?\" and \"Does the note mention the patient as unemployed?\". The agreement rate is 73%.\n\nSo in this experiment, we found that the LLM extracted concepts are quite trustworthy. In manual reviews for other experiments, we also found the extracted concepts to be quite accurate. That said, in settings where the LLM cannot extract certain concepts accurately, we expect BC-LLM to converge towards concepts that it can accurately extract *and* are predictive of the label.\n\n> In the iterative procedure of generating new concepts, in each iteration, all training data are used. This can be highly inefficient once the data set gets larger and/or the iteration becomes large.\n\nBecause the primary goal of this manuscript is to introduce a Bayesian formalization for learning CBMs, there is still room for improvement when it comes to the scalability of BC-LLM. Currently, the runtime BC-LLM is linear in the number of observations since it uses all the training data in each iteration. A straightforward way to speed this up is by using mini-batches instead. Our preliminary experiments show that BC-LLM achieves similar results when doing this for the greedy warm-up procedure. We are now speeding up the Gibbs sampling procedure with mini-batching while preserving statistical guarantees, using methods that estimate the MH acceptance ratio (see e.g. Seita 2018 and Bardenet 2014). We believe this is beyond the scope of this paper and thus leave it to future work.\n\n> Calibration is measured by Brier, which seems to me is an odd choice. ECE is a more standard measurement of calibration in the literature?\n\nBoth Brier and ECE are standard measurements of calibration. We had selected Brier because it is also a proper scoring rule, so the best possible model will also attain the lowest Brier score. ECE tends to be very noisy for smaller dataset sizes since the ECE is highly sensitive to binning strategy; in addition, inaccurate models can also attain low ECE values.\n\nIn the case of the semi-synthetic MIMIC experiment, the true probabilities are known, so the Brier score–defined here as the mean squared error between the true and predicted probabilities–is exactly what we want. As shown in the results, BC-LLM achieves a lower Brier score than other methods.\n\nFor the experiments where the true probability is not known, all the methods have similar ECE values (using 5 bins). For CUB, we find that the BC-LLM attains an ECE of 0.25 with a 95% CI of (0.17, 0.32). This overlaps with the comparators LLM+CBM (ECE=0.23), Boosting LLM+CBM (ECE=0.24), Human+CBM (ECE=0.26), LLM+CBM (no keyphrases) (ECE=0.31), ResNet (ECE=0.26). For the readmission risk prediction experiment in Section 4.3, we find that BC-LLM attains an ECE of 0.184 with a 95% CI of (0.176, 0.193), which again overlaps with comparators LLM+CBM (ECE=0.182), Boosting LLM+CBM (ECE=0.180), and Bag-of-Words (ECE=0.177).\n\n> I still don't fully understand how to use the proposed method to make prediction on a new data point. I think we need to first let the LLM annotate the new data point based on learnt concepts, but then how do you make prediction? Also, how do you account for the error that might occur during the LLM annotation?\n\nCorrect – To make a prediction on a new data point, the LLM annotates it with the learnt concepts. Each CBM in the ensemble (i.e., the posterior samples) then outputs a prediction per its fitted parameters. The final prediction is an average across the ensemble.\n\nBC-LLM does not additionally adjust for annotation error. Instead, it relies on the Bayesian inference procedure to implicitly do this adjustment. That is, concepts that the LLM cannot annotate accurately will have a low MH acceptance rate because these concepts will not be predictive of the target in held-out data. Instead, BC-LLM will converge towards concepts that both (i) the LLM is able to correctly annotate and (ii) are predictive of the target.\n\n> The writing could be improved.\n\nThanks for this feedback. We will revise the notation to use only bold symbols for vectors, add a notation table for clarity, and clean up writing wherever possible to make the paper easier to follow."}]

Author rebuttal:

Meta-review: This paper addresses the problem of model interpretability through human-interpretable concepts via secondary models (like concept bottleneck models, CBMs). Unlike most CBMs, which trade-off interpretability and accuracy, this paper proposes to use LLMs as priors within a Bayesian formulation to sample and refine concepts through posterior sampling ideas. The authors show that their approach asymptotically recovers the correct concepts, provides notions of uncertainty quantification, comprises interesting empirical results comparing with standard CMBs, and features clinician validation in a clinical example.

**Strengths**
* The Bayesian framework for interpretation with the prior information contained in an LLM is a new and interesting perspective to the interpretability problem, agreed upon by all reviewers.
* Their split-sample scheme is a simple yet effective strategy to provide guarantees.
* The approach is applicable in a variety of modalities (image, text, and tabular data).
* Validation by clinicians on clinical example adds credibility. 

**Weaknesses**
* Some assumptions were not clearly flashed out.
* Reviewers questioned aspects of computational costs of the approach, and their comparisons to other CBM methods.
* The choice of datasets lacked some diversity (they focused on "low-knowledge" regimes) and was not very standard (as opposed to "high SNR" regimes like Place365, etc).

**Discussion summary and conclusion**

The discussions between authors and reviewers was productive. Most weaknesses were addressed and resolved (e.g. by clarifying assumptions, providing further experimental results on new datasets and with further clinical evaluation, and further details on computational cost). Two reviewers provide strong support (5), one borderline accept (4) and one borderline reject (3). With the understanding that the authors will include the new results and promised details in their revised version of the manuscript, and based on the novelty of their proposed Bayesian approach, I recommend acceptance.
Decision: Accept (poster)