CHiLL: Zero-shot Custom Interpretable Feature Extraction from Clinical Notes with Large Language Models
Authors: Denis Jered McInerney
Geoffrey Young
Jan-Willem van de Meent
Byron C Wallace
Venue: EMNLP 2023 Findings (2023)
Keywords: Interpretability
Large Language Models
Healthcare
Electronic Health Records
Feature Extraction
Zero-shot
TL;DR: We allow domain experts to describe high-level features in natural language and extract them using LLMs without any training or labels for use in simple interpretable models for downstream tasks.

Abstract: We propose CHiLL (Crafting High-Level Latents), an approach for natural-language specification of features for linear models. CHiLL prompts LLMs with expert-crafted queries to generate interpretable features from health records. The resulting noisy labels are then used to train a simple linear classifier. Generating features based on queries to an LLM can empower physicians to use their domain expertise to craft features that are clinically meaningful for a downstream task of interest, without having to manually extract these from raw EHR. We are motivated by a real-world risk prediction task, but as a reproducible proxy, we use MIMIC-III and MIMIC-CXR data and standard predictive tasks (e.g., 30-day readmission) to evaluate this approach. We find that linear models using automatically extracted features are comparably performant to models using reference features, and provide greater interpretability than linear models using “Bag-of-Words” features. We verify that learned feature weights align well with clinical expectations.

CHiLL: Zero-shot Custom Interpretable Feature Extraction from Clinical Notes with Large Language Models
Denis Jered McInerney Northeastern University
Geoffrey Young
Brigham and Women's 'ss Hospital gsyoungêbwh. harvard. edu
edu
Jan-Willem van de Meent
Byron C. Wallace
University of Amsterdam
Northeastern University b. wwweeeeeeteeetteeeeeeeeeeeeeeeee edu
w. vandemeent@uva. nl
Abstract
poom diniiii solee
ply 112) ((8x)
We propose CHiLL (Crafting High-Level La- tents), an approach for natural-lann lanuuallannguge speci- fication of features for linear models. CHiLL prompts LLMs with expert-crafted queries to generate interpretable features from health records. The resulting noisy labels are then used to train a simple linear classifier. Gener- ating features based on queries to an LLM can empower physicians
Auteentas and lerpyy adicass bpx
pww.ettteeo Fanc rivore
IP 1200
[
x
ctronic
Raw patient notes
Prompt LLM (Flan-T55
Linear modal high- loval features
their domain exper- tise to craft features that are clinically meaning- ful for downstream task of interest, without having to manually extract these from raw EHR. We are motivated by real-world risk predic- tion task, but as a reprodueible proxy, we use MIMIC-III and MIMIC-CXR data and standard predictive tasks (e.g.. 30-day readmission) to evaluate this approach. We find that linear mod- els using automatically extracted features are comparably performant to models using refer- ence features, and provide greater interpretabil- ity than linear models using "Bag-of of- Words" features. We verify that learned feature weights align well with clinical expectations.
Figure I: We propose to allow domain experts specify high-level features for simple linear predictive models in natural language, and then extract these zero-shot (without supervision) using large language models.
pretable features is that one often has to manually extract them from patient records. This can be particularly difficult when features are high-level and must be inferred from unstructured (free-text) fields within EHR.
In this work we investigate the potential of zero- shot extraction for definition and abstraction of interpretable features from unstructured EHR to use inputs for simple (linear) models (Figure 1). Recent work (Agrawal al., 2022) has shown that LLMs are capable zero-shot extractors of clin- ical information, i.e., they can extract structured information from records without explicitly being trained to do so. These can be high-level features specified in natural language, which allows prac- titioners to define features that they hypothesize may be relevant to downstream task. Such fea- tures, even if noisy, likely to be more inter- pretable than low-level features such as Bag-of- Words (BoW) representations. The motivation of this work is to evaluate the viability of training small linear models over high-level features auto- matically extracted via withuut explicit supervision frrmio unstructured data in EHR.
1 Introduction
LLMs have greatly advanced few- and zero-shot ca- pabilities in NLP, reducing the need for annotation. This is especially exciting for the medical domain, in which supervision is often scant and expensive. However, given the high-stakes nature of clinical work and the challenges associated with developing models (e.g.. long-tail data distributions, weakly informative supervision), predictions can rarely be trusted blindly. Clinicians therefore tend to favor simple models with interpretable predictors over opaque LLMs that rely on dense learned represen- tations. Risk prediction tools are often lincar mod- els with handcrafted features. Such models have the advantage of associating features with weights; these can be inspected to ensure clinical tenability and may avoid undesired fragilities of large neural models. However, a downside to relying on inter-
A secondary concern that this paper seeks to ad- dress is any reliance on closed-souree LLMs for healthcare, which is undesirable for a number of

---

reasons. First, such models are inherently opaque, precluding interpretability analysis which is espe- cially important healthcare. Second, EHR data sensitive, and so submitting this to an API (e.g., as provided by OpenAl) is potentially problematic. In this work we show that despite the specialized do- main, Flan-T5 (Chung et al., 2022; Wei et al., 2022) Bariants-whice can fit single GPU and be run locally-can perform zero-shot feature extraction from EHR with reasonable accuracy.
approach is use linear model defined over Bag- of-Words (BoW) representations:
BoW (y= - 1|x² - =ow BoW).
p
(2)
This approach operates over (transformations of) token counts, and therefore the learned has a natural correspondence words in the vocabulary. Linear models that operate over tens of thousands of word predictors occupy an intermediate space between the interpretability afforded by simpler, smaller models defined over high-level features and neural models which use opaque representations. For some clinical tasks, however, BoW with large vocabularies can be competitive with respect downstream performance.
Our contributions are as follows. (1) We propose and evaluate a method for extracting high-level in- terpretable features from clinical texts using Flan- T5 given corresponding prompts, and we evaluate the how well these extracted features align with ground truth. (2) We demonstrate that we can ex- ploit LLM calibration to improve performance, al- lowing models that use inferred features to perform comparably to those using reference features. (3) We show that the resulting linear models' weights align with clinician-an aneiottted notions of how fea- tures should impact a prediction. (4) We investigate the data- and feature- efficiency of our approach and find that it can achieve similar results with much less data and utilizes features efficiently.
In this paper, we use instruction-tun tuned LLMs perform zero-shot inference of intermediate, high- level, features f E RN using N expert-specified prompt templates 11.,,, lx:
fhbinary [argmax, (2LM(2)(, Uyesl
(3) where fa denotes the prompt obtained by popu- lating the template in with x, represents the next token after the prompt and Eyes represents the index of the token "yes" in the vocabulary. We then use a relations between inferred features, probing the degree to which such predictors provide useful and reliable interpretability, and modifying trained lin- ear models directly based on expert judgement.
Our promising initial results suggest several av- enues for further exploration, e.g.. modeling cor-
simple linear model (with weights w E I over these predicted features to predict the target:
ply - 1) = ow pbinnry).
(4)
Predictions from the LLM for the high-level bi- nary features will be imperfect, and are naturally associated with confidence under the LLM: The probability of the token "yes" normalized by the mass assigned to either "yess" or "no". As sim- ple means of incorporating uncertainty in extracted features, we can use this continuous value in place of binary feature indicators. For feature n (elicited using template tx), the feature value front is:
2 Methods
We consider binary classification of patients on the basis of free text from their EHR data. A now stan- dard approach to such tasks would entail adopting a neural language encoder to induce a fixed- length d-dimensional distributed representation of an input x e.g.. the [CLS] embedding in BERT (Devlin et al., 2018; Alsentzer et al., 2019) models- and feeding this forward linear classification head yield logit:
LLM(= = Uyes (x()))
LLM(2 Uyes 4, (x)) LLM(2 Uno 4m (x))
(5)
pΦu - 1 11)) oww SuTTEEpss Eiccs) (z))
(1)
Inspired by previous work (Zhang et al., 2020),
where y E {0. 1}, RRVV and w E Rd (For BERT, d = 768.) A drawback of this approach is that it is not amenable inspection. The prediction is made on the basis of a dense learned representa- tion from pre-trained network, and unclear which patient attributes give rise to the prediction. A simpler and (at least arguably) more interpretable
we split text into chunks of a particular maximum length and take the maximum (per-feature) of the feature values of the chunks as the final features.
3 Evaluation
To evaluate the proposed approach, we consider four tasks on publicly available MIMIC data to

---

permit reproducibility. We start with standard clinical predictive tasks in MIMIC-III (Johnson et al. 2016b,a; Goldberger et al. 2000, Section 3.1): Readmission, mortality, and phenotype prediction. For these we treat ICD codes as proxies for high- level features. This allows us to evaluate the accu- racy of the zero-shot extraction component, and to compare the performance of linear model defined over the true ICD codes compared to the same model when operating over inferred features.
In-hospital Mortality prediction This task in- volves predicting if patient will pass during a hospital stay with notes from the first 48 hours. We adapt preprocessing code from (Zhang et al., 2020).
Phenotype prediction Zhang el al. (2020) also de- rive various phenotypes using ICD codes, and use these to define phenotype prediction task which we also consider. (We again adapt their preprocess- ing code for this task.)
We then consider X-ray report classification using the MIMIC-CXR dataset (Johnson et al. 2019b,a,c; Johnson et al.; Goldberger et al. 2000; Section 3.2). In this setting, we elicited queries for intermediate feature descriptions from radiolo- gist co-author. Queries take the form of questions that radiologist might a priori believe to be rele- vant to the report classification categories defined in CheXpert (e.g., "Does this patient have a normal cardiac silhouette??" see Appendix Table for all examples). This demonstrates the flexibility of the approach-and and the promise of allowing domain experts to specify features in natural language but we are limited in our ability to evaluate the extraction performance directly in this case.
3.2 Chest X-ray report classification
We next consider chest X-ray report classifica- tion. This allows us to draw upon the exper- tise of the radiologist co-author. We use the MIMIC-CXR dataset (Johnson al., 2019b,a,c; Johnson et al.; Goldberger et al.. 2000) with CheX- pert labels (Irvin al., 2019) again ensure reproducibility- and evaluate performance on a 12-label classification task.
Given that these labels can be automatically de- rived, the predictive task considered here is not in and of itself practically useful, but it is illustrative of how domain experts (here, a radiologist) can craft bespoke features for a given task. We use outputs of the CheXpert automated report labeler as "labels". We omit two downstream labels from our Consoliionniiiitiinn and Pleural Effusion- because included these intermediate features instead under the advisement of our radiologist co- author, and it does not make sense to include labels for which there exists an exactly corresponding fea- ture.* 4 (The names of the 12 labels we predict are shown in Table 2.)
3.1 Clinical predictive tasks on MIMIC-III
For the three standard clinical prediction tasks we consider, we use ICD codes as proxies for high- level, interpretable features that one might want to infer. While this task is somewhat artificial, it allows us to evaluate how well the inferred features agree with the "truc" features (i.c., ICD-code in- dicators). We use the top 10 most common ICD codes in the training set as features. 2 We ask the LLM: "Does this mean the patient has <code>?", where we replace <code> with the long descrip- tion of the ICD code. To illustrate the flexibility of the approach, we also consider two custom fea- tures which one might expect to be informative: (I) Does the patient have chronic illness? (2) Is the condition life- tifetthreatening?
To infer intermediate features, we asked the ra- diologist co-author to provide natural binary ques- tions they would ask to categorize radiology reports into the given classes (Appendix B). We refer to these questions as queries, and use them as prompts for the LLM to extract corresponding indicators (or probabilities) for each answer. Because these are novel predictors specified by domain expert, we do not have "reference" features compare to (as we did above where we treated ICD codes as high- level features). However, in Section 4.3 a domain expert assesses the degree to which learned coeffi- cients for features align with clinical judgement.
Readmission prediction For 30-day readmission, we follow the task definition, setup, and data pre- processing outlined in (Huang et al., 2019).
'In practice one could course use ICD codes directly as predictors: but this serves as an exemplary task using publicly available data to show potential of the proposed approach in way that also allows to evaluate models that have access to "ground-trute"
"We discard the images this setting and only use text. "We did use these extra two labels in training. but this only affects BERT, which was trained in a 'multi-task" way with a shared encoder used (and updated) across all labels. We think it unlikely that this would impact performance much.
high-level features, i.e., the ICD codes. 'In preliminary experiments we found that including >> 10 ICD codes as predictors in lincar models did not appreciably improve AUROC for the three tasks considered.

---

Continuous
Binary
AUROC
Score
0.4
0.6
0.2
Mypertantion
0.4
Heart Fellure
Atrial libillation
Acite respiratory fullurs
unerascierrst cereeeeeesss
...Lse anticaa iettocauuaantn coaaauaantss
Kidnwy laltura
Diabeles metttue Khoeriig portriideet ...Gr. kidney diseano
Hypertennt peeeteetton
Caronary Amer ceccer deeeeees.s.... aed
Heart Fagura
Asriar fioiiatioon
Acude respiratory kidrey fallure
...usa antlopngue glggu
Diabeles Hlypenipe Sypperpiieenn .Crr kidney disesse
mellin
Coronary
Figure 2: Feature extraction performance for the readmission prediction task. AUROC for continuous ICD code features (left) and Fl for binary ICD code features (right), compared to reference ICD codes.
3.3 Experimental details
Continuous
Binary
1.0
1.0
92
To extract features, we use FLAN-TS-XXL (Roberts et al., 2022) with fpl6 enabled (for speed) on a Quadro RTX 8000 with 46G of memory. We use a maximum chunk size of 512 (as de- scribed in section 2) and use maximum of chunks. To fit logistic regression models, we use the sklearn (Pedregosa et al., 2011) package's SGDClassif sierr with the logistic loss and the the default settings (this includes adding an intercept and an /2 penalty). We show the full prompt tem- plate used for getting the features appendix A.
0.8
0.4
34
0.6
.21
19
0.4
AUROC
Score
Preciston
Figure 3: Feature inference performance labeled Chest X-ray data. Histograms are over features. For continuous feature AUROCs, we omit features that do not have any positive labels in the 50 labeled examples. For Fl of binary features, we omit features that corre- spond to ill-defined precision (no positive predictions) and/or recall (no positive labels) scores are set to O.
4 Results
We aim to address the following research questions. Feature extraction (4.1) How accurate are zero- shot extracted features, as compared reference (manually extracted) predictors?
the readmission task in Figure
For the radiology task, we have no reference features to use for evaluation. Therefore, our radi- ologist co-author annotated test example reports with a set of features applicable to each report. This allows us to create binary labels for each feature and report, in turn allowing evaluation with the same metrics used above for binary and continuous feature encodings. There are 105 features- many present individually-sot so we report the feature scores in violin plots (Figure 3).
Downstream classification performance (4.2) How does classification based on exper our ap- proach compare to black box and simple models on the ultimate downstream classification?
Interpretabilitty (4.3) Do the inferred features per- mit intuitive interpretation, and do the resultant coefficients align with clinical expectations?
Data and feature efficiency (4.4) Do these fea- tures offer additional benefits terms of data and/or feature efficiency
Zero-shot (feature) extraction performs reason- ably well here in general, as suggested by prior work (Agrawal et al. 2022: although they used GPT- 3, not Flan-T5). Scores are well above chance for MIMIC-II features. Given the zero-shot setting, they are not par with supervised approaches for ICD code prediction (Mullenbach al., 2018), but this may be partially due to limitation of the ICD code reference features: ICD codes are known be noisy (Kim et al., 2021; Boag et al., 2022), so even correctly inferred features may not align well
4.1 Feature extraction
We first measure the accuracy of automatically in- ferred high-level features. In the case of ICD-code proxy features, evaluate this directly using the actual ICD codes as reference labels with which calculate precision, recall, and Fl for binary features, and AUROC for our continuous features (although we cannot do this for the two custom features considered). We present these metrics for

---

ChestXray Classification
IDF) features with varying vocabulary sizes. We report AUROCs for phenotyping and X-ray report classification, macro-averaged over labels.
MIMIC-II
1.0
To evaluate the degree to which zero-shot feature inference degrades performance, we also report results for logistic regression model defined over reference ICD codes ("Ground Truth") for MIMIC tasks. And finally, we compare against direct zero- shot prediction of the downstream classification using FLAN-T5-2 T5-XLL (Roberts et al., 2022).
AUROC
0.7
0.5
intered
Ground
Zem Shot Gream Prediction
Figure 4 demonstrates that across all tasks, the model variants with continuous intermediate fea- tures have significant signal (i.e., AUROC is sig- nificantly above chance 0.5). Our method also performs comparably to or better than "Ground Truth" features for all tasks except Phenotyping. where the ICD code features (real more-so than inferred") have an advantage because the pheno- types considered were derived from ICD codes.? 7 We also see that the addition of just two custom queries does improve performance to varying de- grees for MIMIC-III tasks relative to models that solely employ ICD-code queries, indicating that there is indeed a benefit that can be derived from employing natural language queries to predict in- termediate features.
thanctyping
Regdmasion fradicionn
Mortasdy Pradiction
Figure 4: Downstream classification performance. We compare variants that use inferred continuous features with models that use "ground truth" features (here, ICD codes), and models that perform zero-shot prediction directly. We also show the performance of TF-IDF and BERT models (dashed horizontal lines) and 95% Cls.
with the labels. In contrast, our radiologist's direct annotation of the Chest X-ray report features reveal much higher AUROCs for continuous features. A novel aspect of this work, in contrast to (Agrawal al., 2022), is that in the case of the chest X- ray task the features are extracted using custom queries, specified as natural language questions provided by a domain expert (radiologist). While far from perfect, our results suggest that modern large instruction-tunce LMs are capable of infer- ring arbitrary clinically salient high-level features from EHR with some fidelity.
That said, making downstream predictions using these features performs worse than BERT and TF- IDF (30k) models. This not surprising given that the number of features for our method in the range of 10-105 as compared with 30k for TF-IDF and 100k+8 for ClinicalBERT..
Zero-shot prediction of the downstream task per- forms worse than (supervised) linear models on top of inferred features on Readmission prediction and Chest X-ray report classification, equivalently on Phenotyping, and slightly better on Mortality pre- diction. However, such predictions are completely blackbox (see Section 5 for discussion around inter- pretability of zero-shot extracted features.) Finally, we also find that using binary features instead of continuous features degrades performance signifi- cantly (Table 1); calibrating (un)certainty helps.
4.2 Downstream classification performance
Classification performance in and of itself is not our primary objective; rather, our aim is design a method which allows for flexible construction of small and interpretable models using high-level fea- tures, with minimal expert annotation effort. But to contextualize the predictive performance of the models evaluated we also consider several base- lines that offer varying degrees of "interpretability.. Specifically, we evaluate fine-tuned variants of (a)
Clinical BERT (Alsentzer et al., 2019), and (b)
When manually inspecting some example radi- ology reports, it became apparent that downstream labels are often verbalized in the radiology report.
logistic regression model defined over BoW (TF-
Custom
Pheno.
Readmis.
Mort.
CXR Class.
Expanded set of results in Apppendix Table C.I. 'Inferred codes not expected to fully align with noisy ICD codes (see section 4.1).
-054
-025
-052
-050
-023
-088
-060
Performance not 100% because only a subset the
Table I: Difference in AUROC between using Binary and Continuous features.
ICD codes used for phenotyping were used as features.
"aprroximately, after the embedding layer

---

TF-IF (30k)
Ground Truth
Inferred Continuous w/ Custom
trach
Hypertensive chronic kidney discese
Coronary atherescienccis unspecified type vessel Hypertensive chronic kichey disease
0th
Acute respiratory fallure const
no
Acute respitatory custom custom_has,ae hhr_ncoonic
6snd
dialysi
verices
hd
Caronary athenosclerosts of unspecified type of vessel
Long (current) anticaguulants
Congestive heart
const
hemodialysis
Acute kidney failure
Other and unspecified hyperlipidemia
rehab
Congestive heart fallure Atrial fibnilationn
Unspecified essential hypertension
comfort
Acute kidney
subdural
Long-term (surrent) use of anticogulants
tracheostomy
metformin
Diebetes mellitus without mentio complication
Unspecified essential hypertension Other and unspedifted hypertipidemia
Diabates mellitus without mention of complication Atrial fibnilation
line
overdose
cu Jitthrerethee
Figure 5: Linear model coefficients for readmission prediction. Blue and red indicate features that support or refute the label, respectively, and "const" refers to the values of the intercept. Though many feature weights (e.g. "Hypertensive Chronic Kidney Disease") make a lot of sense in the context of readmission prediction. "Coronary atherosclerosis [..]]"" which we see in the inferred features, does not make sense as a top feature. We suspect this is due imperfect feature inference as it not top feature when using the ground truth features.
P@]
P@5
P@
10
P@ 20
AUC
Continuous Feature Coefficients (Atelectasis)
No Finding
1.00
0.40
0.20
0.10
0.62
bulging fissures
Enlar. Cardiom.
1.00
0.20
0.10
0.05
1.00
stags antler sign
decreased lung volumes
Cardiomegaly
1.00
0.80
0.60
0.35
0.61
alveolar filling process
Edema
0.00
0.20
0.30
0.20
0.62
X convex left atrial appendage
collapse lung
Pneumonia
0,00
0.20
0.20
0.10
0.64
reticular opacity
Atelectasis
0.00
0.20
0.20
0.10
0.69
dense op air filling pus water blood thickening of the fissures
Pneumothorax
1.00
0.60
0.30
0.15
0.84
const
Fracture
1,00
0.40
0.30
0.15
0.74
Indistinct basilar opacity Increase subcarinal angle X Lapical lucency prominent right atrial contour peribronchiolar cuffing
Lung Lesion
0.00
0.40
0.25
Lung Opacity
0.00
0.00
0.00
0.00
0.29
Pleural Other
0.00
0.20
0.40
0.20
0.73
Support Devices
0.00
0.40
0.30
0.30
0.49
Average
0.42
0.35
0.27
0.16
0.67
Figure 6: Linear model coefficients for predicting At- electasis the Chest X-ray Dataset using the top 15 continuous features. "const" represents constant bias feature and is therefore not annotated. V and X denote post-hoc annotations of the feature coefficient as align- ing not aligning with clinical expectations, respec- tively. Most features (both with negative and positive coefficients) do align.
Table Precisions and AUCs of learned feature rank- ings the Chest X-ray classification task, evaluated against a priori relevancy judgements per class pro- vided by our radiologist collaborator. The model was not trained rank features but nevertheless implicitly learned feature importance that aligns with intuition.
This makes sense given that the CheXpert labeler needs to extract these from the report, and interme- diate features are therefore not explicitly modeled, This gives TF-IDF and BERT models particu- lar advantage over inferred feature models that is unlikely exist for natural tasks (which are not defined by an automated labeler, and where inter- mediate features will probably be important).
they judged each feature to be predictive. Conse- quently, we have label for each of the 105 features that indicates whether the domain expert believed the feature to be likely supportive of, or likely not relevant to, each of the 12 classes defined in the chest X-ray task. We use these "feature labels" to measure the degree to which the learned weights agree with the domain knowledge that they are intended to encode. In particular, feature by coefficient and compute precision at k and AUC for all 12 classes (Table 2). For all la- bels except Lung Opacity and Support Devices, the rank of features in terms relevance consistently agrees with expert judgement (AUC > .5).
rank each
4.3 Interpretability
Given our primary motivation to offer mmtrpretabllity-ppeiiie
via small linear models over small number of high-level features-we next investigate how well learned coefficients align with domain expert judgement.
As further evidence of the greater interpretability afforded by the proposed approach, we report the top features for readmission prediction Figure 5.
The radiologist who specified features for the chest X-ray dataset also indicated for which tssk(s)

---

HISTORY: Intubsted, evaluate ET tube.
Targets
Clinical Judgement:
Edemaa Should be a target!
(CheXpert Labeler)
The ET tube is 3.5 om above the carina. The NG lube upp is the film, at least the stomach. Right IJ Cordis tip is in the proximal SVC. The heart size is moderately enlarged. There ill-defined vasculature and alveolar infiltrate, right greater than left. This markedly increased compered to the film from two hours prior and likely represents fluid overload.
FINDINGS:
Cardiomegaly
Continuous Features
Top Decision-Impacting Features (Edema)
Lung Opacity
endotrecheal tube
const
Support Devices
tube
alveolar filling process
enlarged heart
alveolar fluid
Model
enlarged cardiac silhouette
peripheral lucency
Predictions
alveolar fluid
enlarged heart
Cardiomegaly
alvealar filling pracess
upper lobe pulmonary venous engorgement
Edemna Lung Opacity
esophègeal tube
penhilar opacity
Support Devices
nasogastric tube
gastric tube
gestric tube
esophageal tube rounded left heart border
enlarged card aardeisaiati silhouette
Figure 7: Qualitative example of features and feature influence for predicting Edema in the case of ostensible "false positive". We selected this example because on cursory inspection Edema would seem to be applicable here. We presented this to our radiologist co-author who confirmed that this report should in fact be labeled positive for Edema. We show the report and the downstream reference and predicted labels on the left. In the middle, we report top raw feature values, and on the right we show the top scores, i.c., feature values times the corresponding coefficients Edema. Bolded features were confirmed by the radiologist as aligning with clinical judgement.
Here we compare the top-ranked features in linear models using TF-IDI (left), reference ICD codes (the "ground truth" high-level features here; center), and inferred high-level features (right). The top positive inferred high-level features align with the top positive reference ICD code features.
not spend time searching for, or reporting the asso- ciated atelectasis, which in this case is secondary feature of far lower independent importance.
Figure 6 shows an example illustrating how a clinician might inspect a classification. Because the predicted "edema" label disagreed with the CheX- pert labeler, our radiologist collaborator reviewed this case to determine which features led to this ostensible "false positive". While inspecting the features for source of error, they determined that the top positive features ("alveolar filling process", "alveolar fluid") accurately described the report and could support Edema, but more commonly indicate pncumonia or other causes. Subsequently, while reviewing the report, the radiologist concluded that the model prediction of edema was actually cor- rect; the CheXpert label was false negative. In this case, the inferred features did correctly influ- ence the 'edema/no edema' classification.
For high-level features, coefficient magnitude mass is concentrated on the very top features, whereas in the TF-IDF case mass is more uniformly distributed. This held across many of the tasks and labels (see Appendix Table C.2), and is likely an artifact of having many more TF-IDF features. renders such models difficult to interpret. We also see that the custom "has__hromee chronic" feature is among the top features and the "is_ "ss_lffethreateningg feature is at the bottom, aligning with intuition.
We also consider the coefficients for chest X-ray report classification, enlisting our radiologist co- author to annotate these. It is important to do this analysis post-hoc because the initial feature annota- tions used for the metrics in Table 2 are not neces- sarily exhaustive. Figure 6 reports assessments for the linear model coefficients for "atelectasis". Most of the feature influences agree with expectations. Interestingly, this analysis indicates the pres- ence of certain known reporting biases present in the reports. For example, the feature "apical lu- cency" specifically indicates possible pneumoth- orax, a cause of passive atelectasis, and so ratio- nally should support the 'atelectasis' label, but is weighted to refute the label. We speculate that this reflects "satisfaction of search' bias, and other closely aligned reporting biases; pneumothorax such a critically important condition that radiolo- gists reporting pneumothorax will in many cases
Appendix Figure C.2 shows an example where a feature influenced the model incorrectly, even though the model made the correct prediction. These examples illustrate two of many ways in which our approach might facilitate model interpre- tation and debugging.
4.4 Data- and feature- efficiency
Finally, we consider how our model fares compared to baselines in terms of data efficiency. Specifically, construct learning curves by incrcasing the per- cent of the available data used train models. In Figure 8, we see that the performance of small models plateaus with relatively minimal supervi- sion. At low levels of supervision such models are competitive with-and and in some cases better

---

Phenotyping
Readmission Prediction
Chest X-Ray Feature Ablation
Pecure
Prune Randomly 110 runs)
Frune by Coeflicient
SERT
Ground Trtth
Interee Continuuus
Inforee Continuo is Custom
TE-IF (314)
0.80
0.70
0.60
Festure Typc
Tha: Dara Perreeagee jef 24157)
Thnn Eats Precestage 24245)
Infereed Sinury Laaoom Inferred Continuuus Custam
Mortally Prediction
Chest Xray Classification
20
40
60
80
302
20
40
60
20
100
Number of Featlnnes
Number Featuress
Figure 9: Feature ablation for Chest X-ray classifica- tion. After training, we explore how pruning features randomly (left) or based on coefficient magnitudes (av- eraged over all classes) affects performance.
ALROC
144
24
109
30%
1008
18:
36
10%
1309
D31a
Porcostage 126813
Dota Porcentage 200001
enced the output for a given instance (e.g.. Figure 7). This is in stark contrast to deep neural net- work models (like Clinical BERT; Alsentzer et al. 2019), which operate over dense learned embed- dings. However, in healthcare high-level features must often be manually extracted from unstruc- tured EHR. In this work proposed to allow do- main experts define high-level predictors with natural language queries that are used to infer fea- ture values via LLMs. We then define small linear models over the resulting features.
Figure 8: Learning curves for different methods.
than larger models (e.g.. based on Clinical BERT representations), which benefit more from addi- tional supervision. We again emphasize, however, that using more complex representations comes at the expense of interpretability; this is true even for TF-IDF (see Figure 5).
We also explore the distribution of information among the features by pruning features from our continuous model after training in Figure 9. If we prune features randomly, we see a curve that indi- cates that we have not saturated our performance, and adding additional features will likely increase performance further. In fact, while annotating 50 examples on the test set, our radiologist co-author noted many features that were not included that would likely increase performance.
The promise of this approach is that it enables one to capitalize on LLMs while still building mod- cls that provide interpretability. Our approach al- lows domain experts to craft features that align with questions clinician might seek answer in order to make a diagnosis and then use the coefficients from a trained linear model to determine which features inform predictions. Using such "abstract" features in place of word frequency predictors con- fers greater interpretability (Figure 5).
we prune features with the smallest-man magglnuuee coefficients first, we see that-in in contrast to the model using binary teatures-the continuous fea- ture model has a much more rounded curve, indi- cating that there are large number of features that are actively contributing to performance increase. We also note that use dramatically fewer in- ferred features compared to TF-IDE. Indeed, as we report in Appendix Table C.1, if we limit TF-IDF to a vocabulary size of 100, using continuous inferred features outperforms TF-IDF on all tasks.
Conclusions
We have proposed using large language models (LLMs) infer high-level features from unstruc- tured Electronic Health Records (EHRs) to be used as predictors in small linear models. These high- level features can be specified in arbitrary natural language queries composed by domain experts, and extracted using LLMs without explicit supervision.
On three clinical predictive tasks, showed that this approach yields performance comparable to that achieved using "reference" high-level fea- tures (here, ICD codes). On a fourth task (X-ray re- port classification). we enlisted a radiologist collab- orator to provide high-level features in natural lan-
5 Discussion
Small linear models over well-defined features of- fer transparency in that one can read off coeffi- cients for predictors and inspect which most influ-

---

guage. We showed model using these features as automatically extracted via an LLM realized strong predictive performance, and-more importantly- provided varieties of model transparency which we confirmed aligned with clinical judgement, and which provided insights to the domain expert.
the degree to which the LLM inferences align with the domain experts' intent regarding the respective features.
This suggests that one might want to design queries for high-level features that, whenever pos- sible, are easily verifiable upon inspection of the EHR. Asking if patient is risk of death is not directly verifiable, because it is necessarily pre- diction; asking the patient has an "enlarged heart" (for example) probably is. However, even using verifiable features does not guarantee that clini- cians will hastily confirm or remain unbiased by unverified features in practice.
This work demonstrates the promise of using LLMs as high-level feature extractors for small linear models, which may admit varieties of inter- pretability. However, this is complicated by the issues discussed in the Limitations section, below. We believe that more research into this hybrid ap- proach is therefore warranted.
Limitations
Can we manually intervene when a model learns counter-intuitie dependency? Because we exploit features that have a direct relationship to domain expert knowledge, it might be possible draw upon their judgement to intervene when a model learns dependence on a predictor in way that does not align with expectations (e.g. Figures and C.2). This may happen because either: (1) the feature itself was incorrectly inferred. (2) there exists some spurious correlation in the data that does not hold in general. A less likely possibility is that (3) the expert prior simply incorrect. Future work might study if pruning this feature "correct" the model improves generalization.
While the initial results reported above are encour- aging, they also raise several questions as how one should design queries to yield intermediate features that are interpretable, informative, and ver- ifiable. One limitation of using LLMs to gener- ate high-level features is that inference for these features remains opaque. This inherent limitation suggests few immediate research questions for future work.
The slippery "interpretability" of inferred fea- tures; Is there trade-off between interpretabil- ity and predietive power? Not every query in- formative of the downstream prediction task will necessarily aid interpretability, For example, when predicting mortality, the response to the query "Is the patient at risk of death?" will likely corre- late strongly with the downstream task (as can be seen in Figure 4, which reports performance for zero-shot prediction). But this query essentially paraphrases the original downstream task, and so does little aid interpretability. It instead simply shifts the problem explaining what elements of the EHR give rise to the predicted "fcature". This suggests that expert queries should be written to elicit features that correlate with, but are distinct from, the prediction task.
Ethics Statement
Our approach raises ethical concerns because the extracted features may be incorrect. Though this work does show promise in terms of the interpre- tations aligning with clinical expectations, is far from ready to deploy because there is a danger that clinicians will make incorrect assumptions even if warned of the model's potential for making mis- takes both in producing features. Just because the explanation of the model makes sense clinically does not mean the underlying features for an in- stance are factual. Models like this may also play into clinician bias given that the model trained data produced by clinicians. Therefore, we present this work only for scientific exploration, and it is not intended to be used in a deployed system in its current form.
How do we know whether predicted features are faithful to their queries? A related compli- cation to this approach is that zero-shot feature extraction will not be perfect. While results for ICD code proxies indicate good correlation, will in general not be able easily verify whether the indicator that extracted by an LLM is in fact predicting the feature intended. The interpretabil- ity of the resultant model will therefore depend on
Acknowledgements
We acknowledge partial funding for this work by National Library of Medicine of the National In- stitutes of Health (NIH) under award numbers

---

R01LM013772 and ROILM013891. The work was also supported in part by the National Science Foundation (NSF) grant 1901 117. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NTH or the NSF.
Alistair Johnson, Matt Lungren, Yifan Peng, Zhiyong Lu, Roger Mark, Seth Berkowitz, and Steven Horng. Mimic-xxt-jpg-chst radiographs with structured la- bels.
Alistair E. W. Johnson, Tom J. Pollard, Seth J. Berkowitz, Nathanicl R. Greenbaum, Matthew P. Lungren, Chih-ying Deng. Roger G. Mark, and Steven Horng. 2019b. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data, 6(1):117.
References
Monica Agrawal, Stefan Hegselmann, Hunter Lang. Yoon Kim, and David Sontag. 2022. Large language models are zero-shot clinical information extractors. In Proceedings of Empirical Methods in Natural Lan- guage Processing (EMNLP).
Alistair E. W. Johnson, Tom J. Pollard, and Roger G. Mark. 2016a. MIMIC-III clinical database (version 1.4).
Alistair EW Johnson, Tom J Pollard, Nathaniel R Green- baum, Matthew P Lungren, Chih-ying Deng, Yifan Peng. Zhiyong Lu, Roger G Mark, Seth J Berkowitz. and Steven Horng. 2019c. Mimic-cxr-jpg, a large publicly available database of labeled chest radio- graphs arXiv preprint 190.:::901000000
Emily Alsentzer, John Murphy, William Boag, Wei- Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Nat- ural Language Processing Workshop, pages 72-78, Minneapolis, Minnesota, USA. Association for Com- putational Linguistics.
Alistair E.W. Johnson, Tom J. Pollard, Lu Shen, Li- wei H. Lehman, Mengling Feng. Mohammad Ghas- semi, Benjamin Moody, Peter Szolovits, Leo An- thony Celi, and Roger G. Mark. 2016b. MIMIC-III, freely accessible critical care database. Scientific Data, 3(1):100335.
William Boag, Mercy Oladipo, and Peter Szolovits. 2022. EHR Safari: Data is Contextual. In Proceed- ings of the 7th Machine Learning for Healthcare Conference, pages 391-408. PMLR.
Youngwoo Kim, Cheng Li, Bingyang Ye, Amir Tah- masebi, and Javed Aslam. 2021. Supervised Learn- ing in the Presence of Noise: Application in ICD-10 Code Classification. urXiv:2103.07800 [cs].
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling citructiio-finetuun language models. arXiv preprint 22 10. 11416.
James Mullenbach, Sarah Wiegreffe. Jon Duke, Jimeng Sun, and Jacob Eisenstein. 2018. Explainable predic- tion of medical codes from clinical text. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume (Long Papers), pages 1101-1111, New Orleans, Louisiana. Association for Computational Linguis- tics.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lec, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 1rXiv:1810. 1810.04805.
Ary L. Goldherger, Luis AN Amaral, Leon Glass, Jef- frey M Hausdorff, Plamen Ch Ivanov, Roger G Mark. Joseph E Mietus, George B Moody, Chung-Kaue Kang Peng, and H Eugene Stanley. 2000. Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. circulation, 101(23):e15-ee220.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- esnay. 2011. Seikit-lear:: Machine learning in Python. Journal of Machine Learning Research, 12:2825-288 2830.
Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019. Clinicalbert: Modeling clinical notes and pre- dicting hospital readmission. arXiv:1904.05 1904.0532..
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sha- ran Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowyez. Alex Salcianu, Mare van Zee, Jacob Austin, Se- bastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdh- ery, Jasmijn Bastings. Jannis Bulian, Xavier Gar- cia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-lleus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. 2019. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 590-597.
A Johnson, T Pollard, R Mark, S Berkowitz, and S Horng. 2019a. Mimic-cxr database (version 2.0.0). physionet.

---

Maitin-See Maitin-Shepart Shepard, Noah Fiedel, Mark Omernick, Bren- nan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. 2022. Scal- ing up models and data with t5x and seqio. arXiv preprint 2rXiv:2203. 2atn:2223.11899 17189.
Jason Wei, Maarten Paul Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew Mingbo Dai, and Quoe V. Le. 2022. Finetuned language models are zero-shot learners.
Haoran Zhang, Amy X. Lu, Mohamed Abdalla, Matthew McDermott, and Marzyeh Ghassemi. 2020. Hurtful words: Quantifying biases in clinical con- textual word embeddings. In Proceedings of the ACM Conference on Health, Inference, and Learn- ing, CHIL 20, page 110-120, New York, NY, USA. Association for Computing Machinery.

---

A
Full Prompt
Table A.I: The full prompts incorporate the input text the questions, described in section 3 and appendix Table
B according the following templates.
MIMIC Tasks
Read the following text from clinical note:
<input>
<question>
Chest X-ray Classification Task
Read the following Chest X-ray report:
<input>
<question>
Hand-crafted Feature Queries for Chest X-ray Dataset
Table B.2: Features grouped by the labels they may support. (Features may be written under more than one label.)
No Finding
normal
Is this patient normal?
clear lungs
Does this patient have clear lungs?
normal cardiac silhouette
Does this patient have normal cardiac silhouette?
sharp costophrenic angles
Does this patient have sharp costophrenic angles?
Enlarged Cardiomediastinum
enlarged cardiomediastinal silhouette
Does this patient have an enlarged cardiomediasti- nal silhouette?
Cardiomegaly
enlarged cardiomediastinal silhouette
Does this patient have an enlarged cardiomediasti- nal silhouette?
increased cardiothoracic ratio
Does this patient have an increased cardiothoracie ratio?
prominent right atrial contour
Does this patient have prominent right atrial con- tour?
enlarged heart
Does this patient have an enlarged heart? Does this patient have globular cardiac silhou- citc?
globular cardiac silhouette
rounded left heart border uplifted cardiac apex double density splaying of carina increase subcarinal angle
Does this patient have a rounded left heart border?
Does this patient have an uplifted cardiac apex? Does this patient have double density?
Is there splaying of the carina?
Does this patient have an increased subcarinal an- gle?
convex left atrial appendage
Does this patient have convex left atrial ap- pendage?
third mogul sign superior displacement of left mainstem bronchus
Does this patient have a third mogul sign?
Does this patient have superior displacement of left mainstem bronchus?

---

rounded cardiac apex
Does this patient have rounded cardiac apex?
shmoo sign
Does this patient have shmoo sign?
hoffman rigler sign
Does this patient have a Hoffman-Rigler sign?
Edema
upper lobe pulmonary venous engorgement
Does this patient have upper lobe pulmonary ve- nous engorgement?
stags antler sign
Does this patient have stag's antler sign?
bilateral opacity
Does this patient have bilateral opacity?
symmetric perihilar opacity
Does this patient have a symmetric perihilar opac- ity?
bat wing opacity enlarged cardiac silhouette
Does this patient have a bat wing opacity? Does this patient have an enlarged cardiac silhou- ette?
increased cardiothoracie ratio
Does this patient have an increased cardiothoracic ratio?
peribronchiolar cuffing
Does this patient have peribronchiolar cuffing?
hazy perihilar opacity
Does this patient have hazy perihilar opacity?
septal thickening
Does this patient have septal thickening?
kerley b lines
Does this patient have Kerley B lines? Is there a thickening of the fissures?
thickening of the fissures
fluid in the fissures
Does this patient have fluid in the fissures? Does this patient have pleural effusion?
pleural effusion
Pneumonia
reticular opacity
Does this patient have a reticular opacity?
hazy opacity
Does this patient have a hazy opacity?
consolidation
Does this patient have consolidation?
segmental opacity
Does this patient have a segmental opacity?
lobar opacity
Does this patient have lobar opacity?
bulging fissures
Does this patient have bulging fissures?
cavitation
Does this patient have cavitation?
Atelectasis
subsegmental lincar basilar opacity
Does this patient have a subsegmental linear basi- lar opacity?
linear basilar opacity subsegmental crescentic basilar opacity
Does this patient have a linear basilar opacity?
Does this patient have a subsegmental crescentic basilar opacity?
crescentic basilar opacity
Does this patient have crescentic basilar opacity?
decreased lung volumes
Does this patient have decreased lung volumes?
indistinct basilar opacity
Does this patient have indistinct basilar opacities?
Pneumothorax
apical lucency
Does this patient have an apical lucency? Does this patient have peripheral lucency? Is there collapse of lung? Does this patient have visible visceral pleura?
peripheral lucency
collapse of lung
visible visceral pleura
Fracture
lucency in a rib or clavicle
Does this patient have a lucency in a rib or clavi- clc?
deformity of rib or clavicle
Does this patient have a deformity of rib or clav- icle?
wedge deformity of vertebra
Does this patient have wedge deformity of vertebra?

---

step off of rib clavicle
Does this patient have step-off of a rib clavi- cle?
Lung Lesion
rounded opacity
Does this patient have rounded opacity?
pulmonary calcification
Does this patient have pulmonary calcification?
cavitation
Does this patient have cavitation?
abnormal high density
Does this patient have abnormal high density?
opacity with indistinct borders
Does this patient have an opacity with indistinct borders?
spiculated opacity
Does this patient have a spiculated opacity?
hazy opacity
Does this patient have a hazy opacity?
Lung Opacity
alveolar blood
Is there alveolar blood?
alveolar fluid
Is there alveolar fluid?
Pleural Other
pleural empyema
Is there pleural empyema?
hemothorax
Does the patient have hemothorax? Is there pleural tumor?
pleural tumor
pleural thickening
Is there pleural tumor?
calcificied pleural plaques
Does this patient have calcified pleural plaque?
foreign body in pleural space
there a foreign body in the pleural space?
Support Devices
surgical clip
Does this patient have a surgical clip?
metallic density
Does this patient have metallic density?
curvilinear density
Does this patient have curvilincar density?
foreign body
Does this patient have foreign body?
metal lead
Does this patient have metal lead?
wire
Does this patient have wire?
tube
Does this patient have tube?
stent
Does this patient have a stent?
endotracheal tube
Does this patient have an endotracheal tube?
chest tube central venous catheters
Does this patient have a chest tube?
Does this patient have central venous catheter?
picc line
Does this patient have PICC line?
tracheal stent
Does this patient have tracheal stent?
coronary stent
Does this patient have coronary stent?
aortic stent
Does this patient have an aortic stent?
arterial stent
Does this patient have an arterial stent?
pacemaker
Does this patient have a pacemaker?
icd
Does this patient have an ICD?
aortic balloon pump
Does this patient have an aortic balloon pump?
Ivad
Does this patient have an Ivad?
clamshell closure device
Does this patient have clamshell closure device?
SVC stent
Does this patient have an SVC stent?
IVC stent
Does this patient have an IVC stent?
IVC filter
Does this patient have an IVC filter?
Other Features
alveolar hemorrhage
Is there alveolar hemorrhage?
dense op air filling with pus water blood
Does this patient have dense opacity that sug- gests the airspace filling with pus, water, or blood? Does this patient have an esophageal tube?
esophageal tube

---

g-tube gastric tube loculated pleural effusion nasogastric tube
Does this patient have g-tube?
Does this patient have a gastric tube?
there a loculated pleural effusion?
Does this patient have a nasogastric tube?

---

Phenotyping
Readmission
Mortality
Chest X-ray
BERT
0.820 + 0.024
0.661 0.019
0.874 + 0.025
0,991 0.008
TF-IDF (30)
0.624 + 0.031
0.573 0.020
0.692 + 0.034
0.760 0.033
TF-IDE (100)
0.697 + 0.030
0.613 0.020
0.787 + 0.031
0.881 0.027
TF-IDF (1k)
0.813 + 0.025
0.668 0.019
0.863 1 0.025
0,952 0.014
TF-IDF (30K)
0.830 + 0.024
0.94
0.872 0.872 0.024
0.952 0.013
Ground Truth
0.798 + 0.027
0.019
4
Inferred Binary
0.651 + 0.031
0.589 0.020
0.699
Inferred Binary w/ Custom
0.669 +
0.603 0.020
0.039
0.828 0.034
Inferred Continuous
0.705 + 0.029
0.614 + 0.020
0.032+0.032
Inferred Continuous w/ Custom
0.719 + 0.029
0.626 0.020
0.806 + 0.029
0.889 = 0.026
7cro-Shar Shot Downstream
0.711 + 0.030
0.579 0.020
0.816 + 0.030
0.800 0.030
Table C.I: Expanded results of Downstream Classification Performance with 95% Confidence Intervals.
Phenotype
Readmission
Mortality
Chest X-ray
TF-IDF (30)
3.05
3.41
1.00
1.98
TF-IDF (100)
3.84
4.54
2.67
1.79
TF-IDF (Ik)
4.63
6,84
6,24
1.86
TF-IDF (30k)
9.50
10.30
10.27
4.17
Ground Truth
1.30
2.39
2.00
Inferred Binary
1.96
2.39
1.85
Inferred Binary w/ Custom
2.03
2.56
1.50
3.98
Inferred Continuous
1.65
2.31
1.97
Inferred Continuous w/ Custom
1.75
2.51
1.13
3.08
Table C.2: How is the magnitude mass distributed across coefficients? We report the entropy of the distribution described by the coefficient magnitudes: (ssottmax((c)) where e represents the vector of coefficients for a model and | takes the absolute value of each element. (Entropy is averaged over labels in the case of Phenotype and Chest X-ray tasks.) This measure gives some insight into how uniform the coefficient magnitudes are. TF-IDF (30k) has much higher entropy across the board indicating that many more features have high magnitude in this model.
Extra Figures and Tables
Alignment Clinical Exoectations
1.0
0.8
AUROC
0.2
0.0
Entarged Condiometiitto andiomediatttuu medeaatttumm
Anding
Cardooegaah
Edoma
Prnumaniia
Atolecteds
meomathhrra
Practure
Lung Lesion
Lung Oparity
Picural other
Support Devices
Figure C. 1: Using a set of features annotated as supporting particular label, we plot the AUROC obtained by using the coefficient values (from the continuous features model) to predict this feature list for each label.

---

Table C.3: AUC per Phenotyping label.
BERT
TF-IDF (30k)
Ground Truth
Inferred Continuous
Inferred Continuous w/ Custom 0.676 0.713
Zero-Shot
Downstream
Conduction Disorders
0.856 0.809
0.847
0.712
0.678
0.758
Pncumonia (Except That
0.834
0.744
0.709
0.733
Caused By Tuberculosis Or
Sexually Transmitted Disease)
Disorders Of Lipid Metabolism
0.785
0.763
1.000
0.705
0.723
0.639
Acute Myocardial Infarction
0.867
0.862
0.785
0.788
0.796
0.805
Other Lower Respiratory Dis-
0.696
0.719
0.684
0.580
0.630
0.649
case
Pleurisy; Pncumothorax; Pul-
0.663
0.722
0.583
0.593
0.627
0.661
monary Collapse
Any-Chronic
0.874
0.867
0.951
0.779
0.784
0.659
Diabetes Mellitus With Compli-
0.863
0.867
0.652
0.774
0.776
0,831
cations
Cardiae Dysthythmiass
0.846
0.838
0,905
0.799
0.793
Any-Acute
0.819
0.834
0.819
0.726
0,515
Coronary Athcrosclerosis And
0.870
0.868
0.964
0.820
0.838
0,801
Other Heart Disease
Hypertension With Complica-
0.873
0.870
0.864
0.749
0.769
0.670
tions And Secondary Hyperten-
sion
Septicemia (Except In Labor)
0.874
0.883
0.748
0.732
0.728
0.782
Respiratory Failure; Insufli-
0.882
0.884
0.985
0.787
0.784
0.780
ciency; Arrest (Adult)
Any-Disease
0.903
0.897
0.934
0.812
0.826
0.632
Congestive Hcart Failure; Non-
0.837
0.830
0.979
0.767
0.770
0.704
hypertensive
Chronic Obstructive Pulmonary
0.777
0.789
0.670
0.658
0.670
0.716
Discase And Bronchiectasis
Complications OF Surgical Pro-
0.712
0.756
0.589
0.569
0.581
0.701
oodures Or Medical Care
Acute And Unspecilied Renal
0.830
0.848
0.956
0.703
0.714
0.646
Failure
Shock
0.873
0.872
0.756
0.715
0.726
0.694
Other Upper Respiratory Dis-
0.831
0.833
0.642
0.686
0.698
0.737
ease
Fluid And Electrolyte Disorders
0.734
0.760
0.691
0.648
0.650
0.618
Other Liver Discases
0.828
0.855
0.661
0.641
0.669
0.730
Gastrointestinal Hemorrhage
0.855
0.887
0.583
0.628
0.636
0.779
Diabetes Mellitus Without Com-
0.700
0.723
0.971
0.719
0.720
0.638
plication
Acute Cerebrovascular Disease
0.907
0.936
0.677
0.628
0.709
0.841
Essential Hypertension
0.729
0.720
0.996
0.627
0.634
0.588
Chronic Kidney Disease
0.857
0.869
0.830
0.736
0.771
0.817
Table C.4: AUC per CheXpert label.
BERT
TF-IDF (30k)
Inferred Continuous w/ Custom
Zero-Shot
Downstream
Atelectasis
0.997
0.968
0.888
0,944
Cardiomegaly
0.997
0.957
0.913
0,876
Edema
0.997
0.961
0.910
0,954
Enlarged Cardiom.
0.988
0.904
0.764
0,592
Support Devices
0.996
0.965
0.925
0,826
Pneumothorax
0.990
0.950
0.911
0,868
Fracture
0.999
0.987
0.948
0.977
Pncumonia
0.982
0.921
0.795
0.799
Pleural Other
0.995
0.965
0.804
0.566
No Finding
0.987
0.944
0.906
0.230
Lung Opacity
0.994
0.944
0.899
0,858
Lung Lesion
0.976
0.965
0.862
0.804

---

images shows a small consolidation at the right base, most consistent with pneumonia. There consolidations. There is no evidence interstitial edema. There are no pleural effusions. The heart the upper limita of norma. The mediastinal
Targets
Cardiomegaly
Top Decision-Impacting Features (Pneumonia)
Pneumonia
Top Continuous Features
Predictions
consolidation
opacity with ind indistinct borders
dense
Cardiomegally
wire
of opacity
lobar opacity
Pneumonia
alveolar filling process
alveolar filling process consolidation
dense op filling pus water blood
dense op air filling with pus water blood rounded cardiac apex
There are sternotomy wires place.
normal cardiac silhoustte
dense op
IMPRESSION: Consolidation right base most consistent with pneumonia.
opacity with indistinct borders rounded opacity
normal cardiac silhouette
Figure C.2: Qualitative Example of Features and Feature Influence for Pneumonia. (Similar to Figure 7.) This correctly predict Pneumonia, but this happens in spite of consolidation being incorrectly identified by the linear model as not supporting Pneumonia. These plots make clear that this is not a mistake of the feature extractor at inference time because consolidation among the top Features. The mistake comes from the linear model having negative coefficient for consolidation.


Reviews:
[{"review_id": "gqbm67DNLE", "reviewer": "Reviewer_y8JT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. The core contribution here is the proposal to use LLMs to extract high-level, interpretable features from EHR to be used as features in a simple linear model that relates these to a “downstream” label of interest (e.g., 30-day readmission). This is to permit interpretability: Once trained, the model coefficients (each associated with a single high-level feature) can be inspected, on a global and per-instance level. Therefore, the primary purpose of the linear model is to relate high-level features to the output variable of interest. As a point of comparison, we do evaluate using LLMs to directly perform the “downstream” inference (Figure 4, black bars). But note that this is entirely blackbox. We are not *primarily* interested in the classification performance in and of itself; rather, we are proposing a new hybrid method that combines strengths of LLMs with those of linear models. We will improve the presentation of the work to further clarify the contribution and aims.\n\n2. Table 2 does not report prediction performance. Rather it shows, for each task, how well the top features of the model align with expert intuition (i.e., the features that a clinician selected before seeing any model results or examples that should most contribute to a prediction), and it is impressive because it is completely unsupervised and purely auxiliary to the task on which the model was trained. The fact that this is clearly better than random is meaningful. We will revise the text to make sure this is communicated clearly in future versions of the work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea to use LLM to generate examples for training is somewhat interesting; \n2. The experiments are comprehensive and illustrate the effectiveness of this idea. ", "reasons_to_reject": "1. I may doubt the necessity of the linear classifier. What if we just let the LLM give the prediction according to the classified results from the templates? Maybe we could give the model several examples to see the few-shot classification performance; \n2. From Table2, it seems that the large language models are not so good at prediction these diseases. AUC > 0.5 does not seem to be a good guarantee that the returned features are reasonable. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes to use LLM to extract information from the raw patient notes, which could serve as the interpretable features for training a simple classifier. This kind of method could achieve compatible results with other methods, while maintaining the interpretability which might be useful for real applications. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hZx9fe2s0f", "reviewer": "Reviewer_DqbU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Though LLMs and linear models are existing techniques, to our knowledge, no prior work has used LLMs to extract binary or continuous features to be used as input for linear models. This is the core contribution on offer in this work. It is also novel that these features are specified by clinicians in natural language and extracted completely zero-shot. \n\n2. With regards to concerns about privacy, we specifically choose FLAN-T5 as the LLM; variants of this model can be run locally within hospitals in a way that fully complies with privacy regulations and policy. (Indeed, we ran a variant at a hospital for this work.) We agree that one should not feed such data to APIs (e.g., OpenAI), but that is not what we are proposing here.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1, The task is an interesting task, especially in the medical domain.\n2, They provide comprehensive experiment results and discussion.\n3, This paper is well-written and easy to follow.", "reasons_to_reject": "1, I do concern about the novelty of this work. To my best knowledge, most of the methods in this work are existing techniques.\n2, I am wondering if this approach would raise the privacy issue. In the real-world setting, sensitive data would be strictly constrained to be fed into LLMs.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors utilize the power of LLM for feature extraction in the medical domain. Plus, they investigate how the alignment of simple models' weights affects the prediction. They conduct experiments on medical datasets and provide discussion based on the results.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8RJ4kyanAD", "reviewer": "Reviewer_J7J2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Though it is true that TF-IDF and BERT do perform better than our proposed approach at task prediction, it is worth reiterating that (as the reviewer notes) our primary objective is to introduce a new hybrid method that capitalizes on the zero-shot strengths of LLMs and the interpretability of linear models. We are therefore primarily interested in realizing interpretability in a flexible way,  rather than strictly maximizing predictive performance. In fact, the only trained portion of our model is a simple linear model over a very small number of features; it is not surprising that this fares worse than models defined over TF-IDF and BERT representations (which comprise >300x and >30,000x as many features respectively). However, the argument is that the proposed approach permits clinicians to specify arbitrary high-level features and then define a simple model over these; this is not possible with TF-IDF and BERT approaches. Furthermore, even putting aside the issue of allowing domain experts to craft novel features, these representations afford comparatively poor interpretability. \n\nIn particular, regarding the reviewer’s interpretability concerns, we do not dispute that TF-IDF representations do contain individual features that are interpretable (e.g., compared to BERT or other dense neural representations) and our approach has some that are not (because feature extraction is not perfect). However, we argue that our features are interpretable more often than not by showing that learned features do align with the intuitions of clinicians before even seeing the model results: Table 2 shows most AUCs are much greater than 0.5. Figure 5 also demonstrates that TF-IDF one-word features are much harder to interpret than the hand-crafted features that our approach is capable of producing. TF-IDF, naively implemented, defines thousands of features; learning coefficients for each complicates any interpretation of model output.\n\nPer the reviewer’s suggestion, we will revise Figure 5 to explicitly mention the uninterpretable feature and explain that this is a product of the imperfect feature extraction because the ground truth features do not show this feature at the top.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "From the point of my view, the main contribution of this study is that the author and their clinical collaborator derived a detailed, hand-crafted prompt set to extract features from Chest X-ray Dataset (Supplementary A, B). I believe these prompts can significantly help the community to establish a structured dataset from unstructured medical text.", "reasons_to_reject": "The main drawback of this study is that the authors did not prove their model is better than the TF-IDF-based BoW model or other traditional models.\n\nOf note, Medical text classification basically does not require a model to understand the language. The task performance basically relies on the existence of several key-word. Therefore, we can observe that the BERT-based model did not obtain significantly better performance than TF-IDF (BoW) model, and the performance of proposed model even obtained worse performance than the TF-IDF model (Figure 4, 8). Although the author claimed performance is not their primary objective, it is disappointing that such a computationally expensive model only obtained a worse performance than a model proposed several decades ago. Meanwhile, there is also a topic model (i.e., LDA) or neural network-based topic model that can extract interpretable features and be applied to downstream tasks. It will be better if the author can include them as baselines.\n\nMeanwhile, the author does not prove the proposed model is more interpretable. In Figure 5, the author claimed that the proposed model is interpretable because coefficient magnitude mass is concentrated on the very top of features, while the TF-IDF masses are distributed uniformly and are hard to interpret. However, we can find that the most positive high-level feature is hypertensive chronic kidney disease, and this finding is also obvious in the TF-IDF model. We can find that ESRD (end stage renal disease), dialysis, and hemodialysis are also on the top of the TF-IDF mass distribution. If we directly use TF-IDF to analyze the data, we can also obtain the conclusion that chronic kidney disease is the most important factor in readmission prediction. Meanwhile, the proposed model claims that the coronary atherosclerotic is a protective factor of readmission, which is unintuitive, but the author does not explain it. Therefore, it seems not inappropriate to claim that the TF-IDF is hard to interpret and the proposed model is more interpretable.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "I'm not sure whether my understanding is correct because the methodology section of this study is a little bit hard to follow. It seems this study proposed a two-phase model. In the first phase, the author adopted a large language model (LLM, Flan-T5-XXL) to extract a series of preset features in a zero-shot manner. Then,  they trained a simple linear model to generate labels via the extracted features in a supervised manner. As the simple linear model is intrinsically interpretable, the classification result can be regarded as interpretable.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes CHiLL (Crafting High-Level Latents), an approach for natural-language specification of features for linear models. The authors conduct experiments on medical datasets and discuss the impact of weight alignment in simple models.

Reviewers appreciate the interesting task addressed in the paper and commend the comprehensive experiments and clear writing style. They find the experiments illustrative and the task relevant, especially in the medical domain. However, concerns regarding the novelty of the work, privacy implications, and the necessity of the linear classifier are raised.

In light of the reviewers' comments, the paper's contributions are evaluated. The paper's strengths lie in its attempt to bridge the gap between LLMs and interpretable features in the medical domain. However, the lack of clear evidence demonstrating the superiority of the proposed approach over traditional methods like TF-IDF and the absence of explanations for certain model decisions weaken the paper's overall impact.

Despite concerns raised by reviewers, the paper receives an average score above the acceptance board-line, indicating its potential value. Therefore, considering the overall positive sentiment and the potential significance of the paper's contributions, the decision should lean toward acceptance. However, the authors are encouraged to address the reviewers' suggestions, particularly by providing more robust evidence of the proposed method's superiority and further clarifying the interpretability aspect.
Decision: Accept-Findings