From Narratives to Numbers: Valid Inference Using Language Model Predictions from Verbal Autopsies
Authors: Shuxian Fan
Adam Visokay
Kentaro Hoffman
Stephen Salerno
Li Liu
Jeffrey T. Leek
Tyler McCormick
Venue: COLM (2024)
Keywords: multiclass, inference, transportability, public health, verbal autopsy
TL;DR: We show how to perform valid statistical inference when using NLP predictions instead of directly observed data.

Abstract: In settings where most deaths occur outside the healthcare system, verbal autopsies (VAs) are a common tool to monitor trends in causes of death (COD). VAs are interviews with a surviving caregiver or relative that are used to predict the decedent’s COD. Turning VAs into actionable insights for researchers and policymakers requires two steps (i) predicting likely COD using the VA interview and (ii) performing inference with predicted CODs (e.g. modeling the breakdown of causes by demographic factors using a sample of deaths). In this paper, we develop a method for valid inference using outcomes (in our case COD) predicted from free-form text using state-of-the-art NLP techniques. This method, which we call multiPPI++, extends recent work in “prediction-powered inference” to multinomial classification. We leverage a suite of NLP techniques for COD prediction and, through empirical analysis of VA data, we demonstrate the effectiveness of our approach in handling transportability issues. multiPPI++ recovers ground truth estimates, regardless of which NLP model produced predictions and regardless of whether they were produced by a more accurate predictor like GPT-4-32k or a less accurate predictor like KNN. Our findings demonstrate the practical importance of inference correction for public health decision-making and suggests that if inference tasks are the end goal, having a small amount of contextually relevant, high quality labeled data is essential regardless of the NLP algorithm.

Published as a conference paper at COLM 2024
From Narratives to Numbers: Valid Inference Using Language Model Predictions From Verbal Autopsy Narratives
Shuxian Fan +1
Adam Visokay *1 Kentaro Hoffman
Stephen Salerno²
Li Liu Jeffrey I. Leek? Tyler H. McCormick'
University of Washington 2 Fred Hutchinson Cancer Center
'Johns Hopkins University
Abstract
settings where most deaths occur outside the healthcare system, verbal autopsies (VAs) are common tool to monitor trends in causes of death (COD). VAs are interviews with a surviving caregiver or relative that are used to predict the decedent's COD. Turning VAs into actionable insights for researchers and policymakers requires two steps (i) predicting likely COD using the VA interview and (ii) performing inference with predicted CODs (e.g. modeling the breakdown of causes by demographic factors using a sample of deaths). In this paper, we develop a method for valid inference using outcomes (in our case COD) predicted from free-form text using state-of-the-art NLP techniques. This method, which we call mul multipPI++, extends recent work in "prediction- powered inference" to multinomial classification. We leverage suite NLP techniques for COD prediction and, through empirical analysis of VA data, demonstrate the effectiveness of our approach in handling transportability issues. multipPI++ recovers ground truth estimates, regardless of which NLP model produced predic- tions and regardless of whether they were produced by more accurate predictor like GPT-4-32k or a less accurate predictor like KNN. Our find- ings demonstrate the practical importance of inference correction for public health Decision-making and suggests that if inference tasks the end goal, having small amount of contextually relevant, high quality labeled data is essential regardless of the NLP algorithm.
1 Introduction
Verbal Autopsies (VAs) are a pivotal tool in public health research, particularly in contexts where access to formal medical records is limited (Soleman et al., 2006; Thomas et al., 2018). VA serves as a method to ascertain causes of death (COD) by collecting information from relatives witnesses in circumstances necessitated by resource constraints, remote locations, or inadequate healthcare infrastructure (Baqui et al., 2006; Byass et al., 2012; Wahab et al., 2017; Blanco et al., 2021). VA interviews include structured yes/no no questionnaire and open text narrative where the interviewees provide additional information about the illness or death in their own words.
After the interview, COD is assigned either by clinician review or, more commonly, by using statistical algorithms. Finally, since financial and logistical constraints prevent VAs from being performed on all deaths, researchers and policymakers use statistical tools to summarize patterns in COD (e.g. the breakdown of infectious disease deaths by age, race, sex, etc).
We propose a method for valid inference using the open text VA narratives, summarized in Figure 1. To do this, we address two pressing challenges. First, we use text narratives
Denotes first authorship
1

---

Published as a conference paper at COLM 2024
exclusively to classify COD. Structured VA interviews are long (typically around an hour) and emotionally taxing for respondents (Surek-Clark, 2020). Narratives provide an opportu- nity for respondents to describe the circumstances around the death of someone they knew well in their own words, without needing to translate clinical jargon or answer irrelevant questions. Using narratives alone could also dramatically reduce the length of the interview, both minimizing impact on the respondent and allowing enumerators to collect more VAs. Recent work (Danso al., 2013; Blanco et al., 2021; Manaka et al., 2022; Cejudo et al., 2023) has begun exploring the potential of natural language processing (NLP) for COD classifica- tion in VA. Our work furthers this discussion by leveraging state-of-the-arr NLP tools. Also, unlike prior NLP work which focuses on predictive performance (Naradowsky et al., 2012; Liu et al., 2019; Ye et al., 2021; Peskoff & Stewart, 2023), our goal is to utilize these predictions in statistical inference to understand patterns in COD.
Second, we propose method for inference that corrects for misclassification in CODs pre- dicted by NLP. Training and evaluating COD prediction algorithms is extremely challenging (Murray et al., 2011; Fottrell & Byass, 2010; Murray et al., 2014; Clark et al., 2018; Li et al., 2020), in part because building training sets with reliably labeled CODs is taxing. Clinicians could review VAs and assign causes, but this diverts critical resources from patient care and, since most deaths occur outside healthcare facilities, a physical autopsy is impossible. Hence, is appealing to create a larger training set by combining the limited number of labeled VAs one domain with other labeled VAs from different domains (e-g. locations). However, the diversity of cultural practices, linguistic nuances, and variations in interview techniques (not mention potential biological factors) means that an algorithm trained on labeled VAs in one context may perform poorly in another. Our inferential methods must account for additional uncertainty that arises when most COD labels are predicted, not known. We also need to adjust for transportability so that VAs labeled in other contexts can be incorporated effectively into the training set. To do this, we extend prediction-powe inference (PPI++; Angelopoulos et al., 2023a;b) to settings of multinomial classification, showcasing its adaptability to a broader range of prediction problems. We propose an estimator, called multipPI** used to draw valid statistical inference using predictions from any arbitrary black-box machine learning model given access to a small subset of ground truth labels for multinomial data.
The Workflow for Valid Inference Using multiPPI++ for VA Narratives.
Labeled Dataset
Unlabelad Dataset
Varcal Austney
Virtall sumvery
scleniffc intores:
Sceaniiii Invar
Coos: of Dealh
Cens of Desth
O
Pruasted Cezsaral Dadh
Prodicae Duose of Duath
Correcton
Para noter
+
?
Paranietero
Figure 1: Overview of multippI+ correction. Ground truth labels and predicted labels are used separately to perform the same inference task in Domain A. We use the difference between these estimates correction factor in Domain B where ground truth labels are not available.
2

---

Published as a conference paper at COLM 2024
2 Population Health Metrics Research Consortium Narratives
The Population Health Metrics Research Consortium (PHMRC) dataset was collected in 2005 to provide a high quality, standardized dataset that includes ground truth COD labels from traditional autopsies, structured questionnaire responses, and free-text open narrative responses (Murray et al., 2011). is one of a very small number of VA datasets with clinically validated causes. These data were curated from six sites across four countries: (Andhra Pradesh and Utfar Pradesh, India; Bohol, Phillippines; Dar es Salaam and Pemba Island, Tanzania; and Mexico City, Mexico). An example PHMRC narrative follows: "the deceased had been burnt and had løst mental balance and died within 1.5 hours of the accident" (Ground Truth from Traditional Autopsy: Fires).
We focus our analysis on adult deaths, excluding those under 12 years of age, in the de- identified narratives released in Flaxman et al. (2018). CODs are categorized into five broader groups: non-communicable diseases, communicable diseases, external causes, maternal causes, and AIDS or tuberculosis (AIDs-TB), based on International Classification of Diseases Tenth Revision (ICD-10) codes (McCormick et al., 2016). See Appendix A.1 for the full mapping from ICD-10 codes to COD labels.
Under these five broad classes, there still remains considerable heterogeneity in the COD distribution between sites. Figure 2 illustrates this phenomenon. While "non-communicable" diseases (e.g,, various cancers, COPD, stroke) are the most common COD across all six sites, the second most common cause is site-specific. Namely, two sites identify communicable diseases (e.g., malaria, pneumonia) as the second most common COD, while external causes (e.g., suicide, homicide, traffic accidents) are reported in three sites and AIDs-TB in one site. In addition, the difference in rates between the first and second most common CODs varies considerably, with, for example, a 5:4 ratio in Pemba versus a 6:1 ratio in Mexico.
These observations suggest that while information from one site may provide some insights into another, caution is warranted when extrapolating NLP model performance from one location to all others. In practice, this lack of transportability between sites poses significant public health challenge and is well-documented in the VA literature on predicting COD (e-g., see McCormick et al., 2016). However, without a means of addressing this issue for downstream statistical inference, it may be necessary to establish COD data collection systems each new site of interest, an endeavor that would likely prove prohibitively expensive. We address this explicitly in Section 3.2.
Frequency Distribution of COD by Site
Mecico (m-1308)
Andhra Pradesh (n-1144)
Utter Pradesh (1-1)00)
ins-commniiable
Mew.ommunicato
Non-commaniatle
bitenal
botmrl
Ecamal
Add--1
Commoncaate
Adh-b
Communicable
Nateral
Matmra
Mitenaa
Ad6-1
200
400
660
800
100
200
300
400
500
600
100
200
300
420
500
603
Reeperic
Prequency
Dar Saleam (n- 1384)
Bohol (n-11000)
Pemba (n-260)
ios-conmunicblee
Nen-com munccormaniiaae
Non comcommmnicabl
Add--b
Commusicable
Edteral
conmucablee
biennl-
communccate
Eoumal
Matemaa
Naternal
Motmmal
Ado/bb
Adv-tb
200
-00
600
120
200
430
820
CC0 Wnegencc
CDD Peequency
000 Hragunccy
Figure 2: While non-communicable disease is the most common COD in each site, relative prevalence of each COD varies considerably.
3

---

Published as a conference paper at COLM 2024
3 Proposed Analytic Workflow
3.1 NLP for VA Narratives
The first aim of this work is to assess the predictive performance of contemporary NLP methods for classifying COD from free-text narratives, rather than structured interviews. To predict COD from the free-text narratives, we analyze a range of NLP techniques, including various methods based on bag-of-words (BoW) representation and more sophisticated pre-trained models such as Bidirectional Encoder Representations from Transformers (BERT; Devlin et al., 2018) and large language models (LLMs) such as the recently popular Genera- tive Pre-trained Transformer (GPT model; Lai et al., 2023).
To generate BoW representations of VA narratives, we employed scikit-learn' CountVectorizer, LabelEncoder, and TridfVectorizer (Kramer & Kramer, 2016; Pedregosa et al., 2011), as well as nl tk's word_tokenin processing, the initial token count of 556, 713 was reduced to 218, 804. We then applied three supervised classifiers to predict COD using the BoW representations and ground truth cause of death labels: Support Vector Machine (SVM), K Nearest Neighbors (KNN), and Naive Bayes (NB)SVM was implemented with third-degree linear kernel and C = 1.0, while KNN used cosine distance and 9 nearest neighbors, chosen by incrementing from K = 3 until accuracy saturation. The BERT BASE encoder was configured with five transformer layers, to include two input layers of size 256, main layer of size 768, an intermediate layer of size 512, and a fine-tuning output layer of size 5. As the VA narratives in our data were de-identified, we were able to further assess the predictive performance of OpenAl's GPT models (GPT-3.5, GPT-3.5-turbo, GPT-4, and GPT-4-22K Biswas, 2023). After exploratory study, we selected GPT-4-32K for its larger context window. We used zero-shot prompt (see Figure 3) with a temperature set to zero for all tasks. For all of the methods, we split the data into an 80% training set and 20% testing set, where we held out the true COD labels in the testing set to assess each method's ''s predictive accuracy and Fl-score. For BERT BASE, training was conducted over two epochs.
(Millstein, 2020) for data pre-processing, After
We faced a series of challenges in engineering the LM prompt, namely, coercing the LM to consistently return outputs constrained to our exact list of output classes |communicable, non-communicable, external, maternal, aids-tb). Earlier iterations of the prompt yielded responses such "the patient died due to external causes." " While perhaps semantically correct, validating outputs such as this at scale is intractable and would require additional steps (regex, fuzzy string matching, etc.). Instead, we found that including our desired output classes between <option> html tags and explicitly asking the LM to return only choice from the list of options in the prompt improved results considerably. Future work may include few shot prompting with select domain-specific examples or retrieval augmented generation from a larger database of narratives.
3.2 Valid Statistical Inference with multippI++ Correction
The second aim of this work addresses the challenge of conducting valid downstream inference on covariates associated with predicted COD labels. As long as the NLP model is not always 100% accurate, uncritical inference using NLP predictions will always contain bias, which we must correct for before trusting the results. Note that while the PHMRC dataset is complete with ground truth COD labels from traditional, physical autopsies, this is often not practical due to resource constraints. Particularly in vulnerable areas without healthcare infrastructure, COD is abstracted solely from VA without access to means of verification. Recent works have shown that reifying algorithnicall--deivedd outcomes (here, COD) for downstream inference can lead to biased coefficient estimates and anti-conservative inference (Wang et al., 2020; Angelopoulos et al., 2023a;b; Miao et al., 2023; Egami et al., 2023; Ogburn et al., 2021; Hoffman et al., 2024).
To address this, we employ a recent technique known as prediction-powee powered inference (PPI; Angelopoulos et al., 2023a) and its extension, PPI++ (Angelopoulos et al., 2023b) and derive its appropriate form in the context of multiclass logistic regression. These methods operate by using small subset of ground truth COD labels to rectify the coefficient estimates and
4

---

Published as a conference paper at COLM 2024
GPT-4 Zero-Shot Prompt Used for COD Prediction
<narrative>
INPUT
Each narrative gets plugged in here
</narrative>
<labels>
aids-Ib:
Patient died resulting from HIV-AIDs Tuberculosis.
communicable: Patient dled communicable disease such pnoumonia, diarrhea or dysentery.
externll: Patient died from external causes such as fires, drowning, road traffic, falls, paisonous animals, suicide, homicide, or other injurles.
maternal: Patient
pregnancy childbirth
Context
including from severe bleeding, sepsis, pre-eclampsia and eclampsia.
non-communicable: Patient died from a non-communicable disease such as cirrhosis, epilepsy. myocardial infarction, copd, failure, cancer, diabetes, stroks, malaria, asthma.
unclassified: narrative does not contain enough information to predict cause death. </labels>
<options>
aids-tb,
communicable,
external,
Explicitly require output in this format
maternal,
non-communicable,
unclassified
<loptions>
Which label from options applies applles narrative?
If you are not sure, return your best guess. Limit your response to one options exactly as appears the list.
Instructions
Figure The zero-shot prompt explicitly tags the VA narrative, provides minimal context for each COD label, lists an explicit set of COD options to coerce a constrained output, and provides direct instructions pointing the <narrative-l.aaee>xptttoo tags.
standard errors from regression based on predicted COD labels. To that end, assume that our data are comprised of n labeled (I) observations, L {{Y(())))))) 8/ 1,1 X1 Z,di; = 1,..., I...,, N}, where Y denotes the true cause of death, Y/ denotes the cause of death predicted from an NLP function, fZ),, X denotes the covariates interest, and denotes the VA text narratives. In statistical inference, objective function of the form:
and N unlabeled (1) observations, U {(Y, Xu,i Zu,);
can frame estimating statistical parameters, 8, as minimizing
f Ground Truth
arg min IF X, Y)].
(1)
BERR
where la x (X, Y) is defined the negative log conditional likelihood of the multinomial logistic regression. In practice, replacing the observed COD, Y, with yf yields what we refer to as Naive estimate:
y- IR is convex loss function in 0 E Rdd In our case, the loss function
Naice = arg min E[4 Y...
(2)
BER
The Naive estimate relies heavily on the assumption that the NLP model's predictions closely resemble the true CODs.
PPI++, and our extension, multipPI++, seeks to utilize all available information from the unlabeled and labeled data, by minimizing the following objective function:

---

Published as a conference paper at COLM 2024
x
@PPI++
arg min OcR
Ella (X, Yi) + A (El,, Y0)1 (lI p/11)}
(3)
where A E [0, 1] tuning parameter which controls how much weight is placed on using the real labels versus the predicted labels. Note that this estimator combines the ground truth estimator from the labeled subset, which is statistically valid but may lack sufficient sample size, with the Naive estimator, which provides additional information and data from the unlabeled subset.
We extend the application of the PPI++ approach from its logistic regression example to the multinomial case. Angelopoulos et al. (2023b) mention this is possible, but do not provide details of the derivation, and their model has overparameterization issues. We instead derive different parameterization that avoids identifiability issues, and we complete the derivation of PPI++ correction for multiclass logistic regression. We refer to this variant as "mul tiPPI++", where, for the multinomial classification problem with K classes and outcomes Y; E {O....,K, 1}, our loss function takes the form
4(X, 1x = n - + log
Σ
ex
We present the multiPPI+ adjustment procedures in Algorithm 1. See Appendix A.2 and A.3 for complete derivations.
4 Inference with COD Predicted from VA Narratives
4.1 Experimental Setup
To study the performance of the proposed methods, we designed an experiment in which synthetically removed ground truth COD labels from the PHMRC dataset and replaced them with predicted COD from the NLP methods detailed above. Specifically, for each of the the six sites in the PHMRC dataset, we trained the NLP methods on the other five sites and used the resulting models to predict COD in the sixth site (see Figure 4).
We compared the predicted outcomes the withheld, true outcomes for the sixth site in terms of predicted accuracy and F1-score. We then carried forward these predictions to a downstream inferential model in which we regressed site-specific COD on the age of the decedent, to illustrate the performance of the mul tiPPI++ estimator. For each site, we retained 20% of the labeled outcomes and replaced the remaining 80% with the NLP pre- dictions, to mirror the labeled and unlabeled subsets of the data necessary for the mul mu approach. We compared the results the proposed estimator to two additional estimators: (i) the 'grounn truth' estimator, which utilized the full set of true COD labels, and (ii) the 'Naive' estimator, which used only the predicted COD, treating them as if they were ob- served. Note that, in many practical settings, the ground truth estimator is not feasible, but is included here as a point of reference. Further,
due to the variability in data between sites and the fact that the NLP model was not trained on this particular site, we anticipate that the regression coefficients estimated using the NLP-predicted COD will differ from those estimated using the true COD.
We exemplify one representative set of experiments in which the Uttar Pradesh site is excluded from the NLP model training and used the validation site assess the NLP models' predictive performance and the downstream inferential results.
4.2 NLP Prediction Results
First, using the "leave one site out" procedure described in Figure 4, we trained each NLP model to predict COD from narratives on five sites, with the sixth site reserved as validation set to use for assessing the methods. As evidenced by the COD distribution

---

Published as a conference paper at COLM 2024
"Leave One Out" Synthetic VA Transportability Experiment
1306)
Fully Labeled
100% true COD
Andhra Pradesh 1144)
Train
Dar es Salaam (n 1584)
Uttar Pradesh (n 1380)
Nalve
80% NLP predicted
COD and 20% true COD
Rohol In = 1109)
MultiPPI++ Adjusted
Naive with MultiPP conrection
Pemba (n 260)
Figure The classic and BERT models are trained with VA narratives from five other sites and used to predict COD from Uttar Pradesh narratives. The zero shot GPT-4 model predicts COD without any site specific narrative fine-tuning, Ground truth, Naive and mul +iPI++ corrected inference is performed using these Uttar Pradesh predictions.
displayed in Figure 2, we are dealing with considerable class imbalance within and between sites. We address this in sensitivity analysis (see Figures 29-54 in A.6) where we train each model on stratified sample with an 80/20 split for each class to ensure an equal proportion of training examples for each class. The results from this analysis are not substantively different from our main results where we do not account for the class imbalance in our modeling. Alternative resampling methods such as SMOTE, ADASYN or Tomek Links (Chawla et al., 2002; He et al., 2008; Tomek, 1976) could be applied to correct for the imbalanced distribution of COD. However, even in the absence of such considerations, show that multiPPI++ still obtains valid estimates in downstream inference using predictions from imperfect NLP modeling. This extensibility of our approach is one of the advantages of the multiPPI++.
We focus our subsequent analysis one representative site, Uttar Pradesh, but share results from the other sites in the Appendix. These results for Uttar Pradesh summarized in Table and Figure 5, which contain the four models with highest accuracy. All models scored between 0.60-0.75 accurate, with GPT-4 yielding the highest accuracy (0.75). Fl-scores were more uniform, with all but KNN getting Fl-scores between 0.71-0.74. KNN scored 0.66. These results seem to point GPT-4 either being slightly better or nearly equivalent to the other NLP methods we tested. GPT-4's predictions matched the performance of the best examples from the VA literature, despite using only using only the VA text narratives and not the structured questionnaire (James et al., 2011; Murray et al., 2014; Serina et al., 2015).
Metrics
BoW w/ SVM BoW w/ KNN
BoW w/ NB
BERT
GPT-4
Accuracy
0.68
0.63
0.60
0.55
0.75
F1-Score
0.74
0.66
0.72
0.71
0.73
Table 1: GPT-4 is far more accurate than competing NLP models and is among the best in terms of F1-score. Using only text narratives, GPT-4 performs on par with the best predictions in the VA literature, which are produced with much richer data in the form of structured questionnaires.
A unique feature of LLMs compared to other prediction algorithms is that they have the capacity, and in fact the tendency, output class labels that do not exist in the ground labels. In our analysis, GPT-4 predicted 1503 out of 6763 COD labels as "unclassified" " Upon manual review, these 1503 narratives contained no useful information for COD inference. The modal "unclassified" narrative was "the interviewee thanked the interviewer" (n=178). While the accuracies appear similar, GPT-4 correctly identified a flaw in the data cleaning in the PHMRC dataset, which suggest that the accuracies among the other methods may be overinflated. For each of the combination of six sites and five NLP model, we generated predictions and used this to generate regression coefficients.
7

---

Published as a conference paper at COLM 2024
For the sake of brevity, present the four models with the best performance for the Uttar Pradesh site, NB, KNN, SVM, and GPT-4. Figure displays point estimates and 95% confidence intervals for the ground truth, Naive and multiPPI++ corrected inference.
Uttar Pradesh NLP Prediction Confusion Matrices
Pickiia Likers
Mabernck
Connouncaae
Hebomni
Avpe
0
AAW-
0
0
246
17
21
12
191
Labes
Labze
171
106
7
125
Narenal
0
633
Hon commmsiiiee
15
35
15
14
561
(a) BoW with Naive Bayes
(b) BoW with K Nearest Neighbors
Predicted Aibel
conzamioail
Mitemna
puterna
1
54
Commolaabo
Communiation
4
25
9
184
18
The LodE
203
3
The
Maberna
1
6
69
54
1
2
2
114
10
7
603
3
(c) BoW with Support Vector Machine
(d) GPT-4
Figure 5: For VA narratives from Uttar Pradesh, most of the COD misclassifications are assigned non-communicable COD label. Naive Bayes mostly predicts non non-communicable and achieves 0.60 accuracy in-part because non-communicable is overwhelmingly the most common ground truth COD.
Inferential Model Results
In the case of NB and SVM, which had some of the lower Fl-scores, there is a large discrep- ancy between the ground truth and the predicted values, demonstrating that replacing 80% of the causes of death with predicted causes death yields significantly different regression coefficients for the age of the decedent. multipPI++ correction rectifies estimates back to baseline, demonstrating its ability to account for (i) model inaccuracy and (ii) transporta- bility bias. Interestingly, we observe that the Naive regression coefficients from KNN and GPT-4 are not as distant from the ground truth as the other two methods, even though KNN had the worst Fl-score. Figure 7 shows the association between F1 F1-score and mul tiPPI++ X, which also fail to show any significant association. This implies that F1-score alone may not be good predictor of the quality of downstream statistical inference.

---

Published as a conference paper at COLM 2024
As for the widths of the confidence intervals, we observe that ground truth naturally had the thinnest, as had the largest sample size and used real data points. On the other hand, the Naive estimate and multiPPI++ correction show similar confidence interval widths. Since the widths of the confidence interval for multipPI++ is the sum of the widths of the Naive interval plus the width of the correction term, this seems to indicate that performing the multiPPI+ correction does not seem to add significant amount of uncertainty. This demonstrates that it can correct for transportability bias without much additional informa- tion cost. This is further evidenced by the seeming lack of association between the tuning parameter A, which weights the use of the predicted outcomes the downstream inferential model and the accuracy of the predictions. As À is function of both the outcome and the associated features, we conclude that better predicted outcomes carry no additional useful information for parameter estimation.
Figure 6 demonstrates that multiPPI++ improves the accuracy of statistical estimates using post-facto correction term estimated using relatively small amounts of labeled data. As this is applied after model predictions are made, multiPPl++ does not improve or degrade prediction accuracy. This procedure provides correction for the biased statistical inference parameters and the deflated uncertainty intervals around them, given NLP produced estimates of likely COD from VA narrative predictions. This is nuanced yet important distinction as these downstream statistical inference parameters are what describe the relationship between covariates, such as age, and cause of death distributions. This same intuition extends to other domains where the end goal is to use inference to guide decision making where NLP predictions are used in place of ground truth labels.
Inferential Experiment Results
Pradesh
Uttar Pradesh
0.150
Ground Thush
0,150
Ground Trith
Nalve
Natue
Comrection
mutiPPii+ Comectionn
0.100
0.100
Lgg O2th Batl
0.675
0.000
0.050
0.025
0,025
-0.025
-0.025
Communicable
Exteral
Manermal
Moncommunicate
Communicable
External
Maternal
Mon-commun
Causi Death Label
Cause Deach Lubel
(a) BoW with Naive Bayes
(b) BoW with K-Nearest-Neighbors
Uttor Prodesh SVM
Uttar Pradesh got4
0.100
Enmure Trush
0.150
Ground Truth
Neiver
Naive
multirtte Corraction
mutuP714 Comrection
0.100
0.100
Log Did6 fadto
0.030
0.050
0.000
0,000
000025
Cause Death
Label
(c) BoW with Support Vector Machine
(d) GPT-4-32k
Figure 6: Ground truth, Naive estimation, and multiPPI++ correction of estimation of log odds ratios for multinomial regression for age in Uttar Pradesh. multippI++ correction both recovers point estimates and reflects the increased uncertainty corresponding to the use of predicted COD rather than known COD with wider confidence intervals.

---

Published as a conference paper at COLM 2024
No Apparent Association Between Fi-scores and multiPP++ A's
Addin mradch Dar Sa ar Fembo
Figure 7: There appears strong global relationship between mul À and Fl-score, though associations do emerge when subsetting by site, hinting at a Yule-Simpson effect.
Discussion
This paper uses variety of NLP models to predict COD using VAs and, subsequently, develops statistical method for valid inference using these predicted CODs. This research not only contributes the advancement of predictive modeling public health, but also underscores the significance of parameter interpretation in guiding decisions within diverse and dynamic health scenarios.
Based on our research, there are several points that we believe are worthy of further investigation. First, we noticed that better predictive accuracy the COD classifications did not necessarily translate to improved parameter estimation when used as generative model for downstream statistical inference, at least not under the accuracy ranges examined in this paper. In situations where the user is ultimately interested in accurate parameter estimation, as is often the case in public health, care must be taken to differentiate between the most accurate model for prediction versus the most accurate model for parameter estimation, as it is possible that very cheap models such as KNN can be as useful as more expensive, newer models like GPT-4. For example, the cost to train and predict COD with BoW representations and KNN, NB and SVM was trivial, whereas performing the same task with GPT-4 cost $3000USD.
Second, the extreme class imbalance in COD drives many of our prediction results. While it not possible in practice to directly balance across CODs, there are broader design questions that could substantially improve inference. How, for example, should researcher spend their very limited budget to label relevant VAs? And are there other ways of categorizing deaths that provide richer information than the five categories we use? Third, there is evidence that NLP techniques such as large language models perform worse in non-English languages Navigli et al. (2023); Lai et al. (2023); Hirschberg & Manning (2015), which is major consideration since VAs are common in non-English settings. This could introduce bias to potential VA inference in other languages (Surek-Clark, 2020). This is limitation of our simulation study, since does not account for differential translation quality across languages. Future work might instead leverage NLP models specifically pre- pre-trained for multiple languages or medical text (Peskoff et al., 2021; Singhal et al., 2023).
Acknowledgements
We thank the University of Washington eScience institute for granting us Microsoft Azure computing credits used to produce all of our NLP predictions. We also thank Aidan Andronicos, Robert Fatland, Musashi Hinck, Eddie Hock, Jesse Peirce, Denis Peskoff, Nels Schimek, Sander Schulhoff, and Brandon Stewart for their help.
10

---

Published as a conference paper at COLM 2024
References
Anastasios N. Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I. Jordan, and Tijana Zrnic. Prediction-powered inference. Science, 382(6671) November 2023a. ISSN 1095-9203, doi: 10.1126/ http://cccnncccci00000 URL adi6000.
Anastasios N. Angelopoulos, John C. Duchi, and Tijana Zrnic. Ppi++: Efficient prediction- powered inference, 2023b. URL //arxi//
http://aaxxiv.oorra///////// org/abs/2311 231.000/3
A. H. Baqui, G. L. Darmstadt, E. K. Williams, V. Kumar, T. U. Kiran, D. Panwar, V. K. Srivastava, R. Ahuja, R. E. Black, and M. Santoshama. Rates, timing and causes of neonatal deaths in rural India: Implications for neonatal health programmes. Bulletin of the World Health Organization, 84(9):706-713, 2006. ISSN 00429686. doi: 10.2471/BLT.05.02444
Som S. Biswas. Role of chat gpt in public health. Annals of Biomedical Engineering, 51()): 868-869, March 2023. ISSN 1573-9686. doi: 10.1007//00499-022--3372211 URL http: doi. org/10.
Alberto Blanco, Alicia Perez, Arantza Casillas, and Daniel Cobos. Extracting cause of death from verbal autopsy with deep learning interpretable methods. IEEE Journal of Biomedical and Health Informatics, 25 apr 2021. ISSN 21682208. doi: 10.1109/JBHI.202 3005769. URL https: orgg wwrgg/bbsracc..o /bbbbbbccrcccccccueee//191100000 /dtc///ettt122025/////
Peter Byass, Daniel Chandramohan, Samuel J. Clark, Lucia D'Ambruoso, Edward Fot- trell, Wendy J. Graham, Abraham J. Herbst, Abraham Hodgson, Sennen Hounton, Kathleen Kahn, Anand Krishnan, Jordana Leitao, Frank Odhiambo, Osman A. Sankoh, and Stephen M. Tollman. Strengthening standardised interpretation of verbal autopsy data: the new InterVA-4 tool. Global Health Action, 5:1-8, 2012. ISSN 16549880. doi: 10.3402 GHAOO V510.1988.
Ander Cejudo, Arantza Casillas, Alicia Pérez, Maite Oronoz, and Daniel Cobos. Cause of death estimation from verbal autopsies: Is the open response redundant or synergistic? Artificial Intelligence in Medicine, 143:102622, 2023.
Samuel J Clark, Zehang Li, and Tyler H McCormick. Quantifying the contributions of training data and algorithm logic to the performance of automated cause-assignment algorithms for verbal autopsy. Xiv preprint XXio:1803.07141, 2018.
Samuel Danso, Eric Atwell, and Owen Johnson. Linguistic and statistically derived features for cause of death prediction from verbal autopsy text. In Language Processing and Knowl- edge in the Web: 25th International Conference, GSCL 2013, Darmstadt, Germany, September 25-27, 2013. Proceedings, pp. 47-60. Springer, 2013.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, 11100.08804 2018. URL http: /tap:///rxivv wrw..ooorg/ooooo www//rg////////// aa/aa/ss//111/ /110/ 04805.
Naoki Egami, Musashi Jacobs- Harukawa, Brandon M Stewart, and Hanying Wei. Using large language model annotations for valid downstream statistical inference in social science: Design-based semi-supervised learning. Xiv preprint wxXio2330.004746 2023.
Abraham D. Flaxman, Lisa Harman, Jonathan Joseph, Jonathan Brown, and Christopher J.L. Murray. A de-identified database of 11, 979 verbal autopsy open-ended responses. Gates Open Research, 2:18, April 2018. ISSN 2572-4754. doi: 10. /0.12688/gatsspeenrss.. patesopenres.12812.11 URL
http://dx. doi. org/rog////ooo 12w.8/ 2/88//gatesopenres. 12812.1.
E. Fottrell and P. Byass. Verbal autopsy: Methods in transition. Epidemiologic Reviews, 32(1))38-55, March 2010. ISSN 1478-6729.
doi: 10.1093 /etp//eeee///mo/..../ URL http:
htt..oo...org//..118////eeeeeeee //93//pprre/mmx0003....33
Julia Hirschberg and Christopher D. Manning. Advances in natural language processing. Science, (494234)2262422226666 July 2015. ISSN 1095-9203. doi: 10.1126/ wwweeeccceeaaaaaaaee URL
http://dxx.di..rgg/// 102.11115550000 /et///cinnce..ee// aaa8685.
11

---

Published as a conference paper at COLM 2024
Kentaro Hoffman, Stephen Salerno, Awan Afiaz, Jeffrey T Leek, and Tyler H McCormick. Do we really even need data? arXio preprint ar (rii:2440100770 088702, 2024.
Spencer L James, Abraham D Flaxman, Christopher J Murray, and Consortium Population Health Metrics Research. Performance of the tariff method: validation of simple additive algorithm for analysis of verbal autopsies. Population Health Metrics, 9(1):31, 2011.
Oliver Kramer and Oliver Kramer. Scikit-learn. Machine learning for Evolution Strategies, pp. 45-53, 2016.
Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernon- court, Trung Bui, and Thien Huu Nguyen. Chatgpt beyond english: Towards a com- prehensive evaluation of large language models in multilingual learning. arXiv preprint arXie:2304.05613, 2023.
Zehang Richard Li, Tyler H McComick, and Samuel J Clark. Using bayesian latent gaussian graphical models to infer symptom associations in verbal autopsies. Bayesian Analysis, 15 (3):781, 2020.
Ran Liu, Joseph L. Greenstein, Sridevi V. Sarma, and Raimond L. Winslow. Natural language processing of clinical notes for improved early prediction of septic shock in the icu. In 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 6103-6108, 2019. doi: 10.1109/ 10.1109/EMBC MAA.C.C.019.88771111
Thokozile Manaka, Terence van Zyl, and Deepak Kar. Improving cause-of-death classifica- tion from verbal autopsy reports. In Southern Africun Conference for Artificial Intelligence Research, pp. 46-59. Springer, 2022.
Tyler H. McCormick, Zehang Richard Li, Clara Calvert, Amelia C. Crampin, Kathleen Kahn, and Samuel J. Clark. Probabilistic cause-of-death assignment using verbal autopsies. Journal of the American Statistical Association, 111(515):0336-144)) July 2016. ISSN 1537- 274X. doi: (0.1000/016214592211611101 2016. 1152191.
URL http: //tx: wog.i.oor/100// 1080/01 621459.
Jiacheng Miao, Xinran Miao, Yixuan Wu, Jiwei Zhao, and Qiongshi Lu. Assumption-lean and data-adaptive post-prediction inference. Xrv preprint ar arXio:2311.14220 2023.
Frank Millstein. Natural language processing with python: natural language processing using NLTK. Frank Millstein, 2020.
Christopher JL Murray, Alan D Lopez, Kenji Shibuya, and Rafael Lozano. Verbal au- topsy: advancing science, facilitating application. Population Health Metrics, 9(1,, July 2011. ISSN 1478-7954. doi: 10.1186 10.1186/1478-7954 1478-7954-9-18. URL http://ddd doi. org/10. 1186/ 1478-7954-9-11
Christopher JL Murray, Rafael Lozano, Abraham D Flaxman, Peter Serina, David Phillips, Andrea Stewart, Spencer L James, Alireza Vahdatpour, Charles Atkinson, Michael K Freeman, Summer Lockett Ohno, Robert Black, Said Mohammed Ali, Abdullah H Baqui, Lalit Dandona, Emily Dantzer, Gary I. Darmstadt, Vinita Das, Usha Dhingra, Arup Dutta, Wafaie Fawzi, Sara Gómez, Bernardo Hernández, Rohina Joshi, Henry D Kalter, Aarti Kumar, Vishwajeet Kumar, Marilla Lucero, Saurabh Mehta, Bruce Neal, Devarsetty Praveen, Zul Premji, Dolores Ramírez- Samírez-Villalooos, Hazel Remolador, Ian Riley, Minerva Romero, Mwanaidi Said, Diozele Sanvictores, Sunil Sazawal, Veronica Tallo, and Alan D Lopez. Using verbal autopsy to measure causes of death: the comparative performance of existing methods. BMC Medicine, 12(1), January 2014. ISSN 1741-7015. doi: 10.1186/ 1741-7015-12-5 URL hwtphhtpp//..cee http//dd.oo.ooog/////// org/10 wttp./...rgg/...11//771111-10000 1186/1741-7711--12--
Jason Naradowsky, Sebastian Riedel, and David Smith. Improving NLP through marginal- ization of hidden syntactic structure. In Jun'ichi Tsujii, James Henderson, and Mar- ius Paşca (eds.), Proceedings of the 2012 Joint Conference on Empirical Methods in Nat- ural Language Processing and Computational Natural Language Learning, pp. 810-820, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https: //ac wwattholgg..org/122--11
12

---

Published as a conference paper at COLM 2024
Roberto Navigli, Simone Conia, and Björn Ross. Biases in large language models: Origins, inventory, and discussion. Journal of Data and Information Quality, 15(2):1-21, June 2023. ISSN 1936-1963. doi: 0..1145/3597385 URL
1145/3993377.
Elizabeth L Ogburn, Kara E Rudolph, Rachel Morello-Frosch, Amber Khan, and Joan A Casey, A warning about using predicted values from regression models for epidemiologic inquiry. American Journal of Epidemiology, 190(6):1122-11477 2021.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 22-2825-2830 2011.
Denis Peskoff and Brandon Stewart. Credible without credit: Domain experts assess gener- ative language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), PP. 427-438, Toronto, Canada, July 2023. Associa- tion for Computational Linguistics. doi: 10. 10.18653/v1 /tt/://ww1/222../shorrrt.. URL https: //aclanthology. org//ogg/33 acl.aaorrt.s..so........
Denis Peskoff, Viktor Hangya, Jordan Boyd-Graber, and Alexander Fraser. Adapting entities across languages and cultures. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, Pp. 3725-3750 Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/ /v11/vvvv/// 1tt///22...innns.../e. findings--emnlp. 315. URL htttps://cclanthhlogg/ org/2021. (o1221.finningg. findings-emnnnn 315.
Peter Serina, Ian Riley, Andrea Stewart, Spencer L. James, Abraham D. Flaxman, Rafael Lozano, Bernardo Hernandez, Meghan D. Mooney, Richard Luning, Robert Black, Ramesh Ahuja, Nurul Alam, Sayed Saidul Alam, Said Mohammed Ali, Charles Atkinson, Ab- dulla H. Baqui, Hafizur R. Chowdhury, Lalit Dandona, Rakhi Dandona, Emily Dantzer, Gary L. Darmstadt, Vinita Das, Usha Dhingra, Arup Dutta, Wafaie Fawzi, Michael Freeman, Sara Gomez, Hebe N. Gouda, Rohina Joshi, Henry D. Kalter, Aarti Kumar, Vish- wajeet Kumar, Marilla Lucero, Seri Maraga, Saurabh Mehta, Bruce Neal, Summer Lockett Ohno, David Phillips, Kelsey Pierce, Rajendra I' Prasad, Devarsatee Praveen, Zul Premji, Do- lores Mamirez-Villaloooss Patricia Rarau, Hazel Remolador, Minerva Romero, Mwanaidi Said, Diozele Sanvictores, Sunil Sazawal, Peter K. Streatfield, Veronica Tallo, Alireza Vadhatpour, Miriam Vano, Christopher J. L. Murray, and Alan D. Lopez. Improving performance of the tariff method for assigning causes of death to verbal autopsies. BMC Medicine, 13(1), December 2015. ISSN 1741-7015, doi: 10. 0011166/1111600000 9-2216-015-005779. URL http://dx. wotp://ooo..og//////// 1186/1 1211161111--122-2222
Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansfield, Dina Demner-Fushman, Blaise Agüera y Arcas, Dale Webster, Greg S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan. Large language models encode clinical knowledge. Nature, 620 (7972):172-180, July 2023. ISSN 1476-4687. doi: (0008//41186-002-6699999 h/tp://dx. doi.doi.oooooooooo org/10. 0038/s41586-000 021155666-02---9--2222
URL
Nadia Soleman, Daniel Chandramohan, and Kenji Shibuya. Verbal autopsy: current prac- tices and challenges. Bulletin of the World Health Organization, 84(3):239-245, 2006.
Clarissa Surek-Clark. 301 Verbal Autopsy Interview Standardization Study: A report from the field. In The Anthropologicaal Demography of Health, Oxford University Press, 10 2020. ISBN 9780198862437. doi: 10.1093, 000///o//// /000/9788998245339 10. (tt.:osss/000010000000
00038662237.003.001111 000.0001..
URL https://doi. org/
Lisa LisaeM Thomas, Lucia D' Ambruoso, and Dina Balabanova. Verbal autopsy in health policy and systems: literature review. BMJ global health, 3(2):e000639, 2018.
13

---

Published as a conference paper at COLM 2024
Abdul Wahab, Ifta Choiriyyah, and Siswanto Agus Wilopo. Determining the cause of death: Mortality surveillance using verbal autopsy in Indonesia. American Journal of Tropical Medicine and Hygiene, 97(5):141111468, 2017. ISSN 00029637. doi: 10.4269/ AJTMH.16-08155.
Siruo Wang, Tyler H McCormick, and Jeffrey T Leek. Methods for correcting inference based on outcomes predicted by machine learning. Proceedings of the National Academy of Sciences, 117749):30266-90275 2020.
Zihuiwen Ye, Pengfei Liu, Jinlan Fu, and Graham Neubig. Towards more fine-grained and reliable NLP performance prediction. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 3703-3714, Online, April 2021. Association for Computational Linguistics. doi: aclanthology wtrg/2221..g eacl-malnnnn.. 324.
URL https://
Haibo He, Yang Bai, Edwardo A. Garcia, and Shutao Li. ADASYN: Adaptive synthetic sampling approach for imbalanced learning. In 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), pp. 1322-1328, 2008. doi: 10.1109 (..110/IJCNNN.2000 /II/CNNNN.000043366666
Ivan Tomek. Two Modifications of CNN. In IEEE Transactions on Systems, Man, and Cybernet- ics, 1976. doi: 10.1109 AL110//SMMC.177.433000
N. V. Chawla, K. W. Bowyer, L. Hall, and W. P. Kegelmeyer. SMOTE: Synthetic Minority Over-sampling Technique. In Journal of Artificial Intelligence Research, volume 16, pp. 321-357, 2002. doi: https://do.oorg//1111/////aaaa/
14

---

Published as a conference paper at COLM 2024
A
Appendix
A.1
ICD-10 COD Classification
Mapping 34 PHMRC All-Cause Mortality Labels to Five Broad COD Labels
ICD-10 Code
All-Cause Mortality Label
Broad COD Label
K74
cirrhosis
non-communicable non-communicable communicable
G40
epilepsy
J18
pneumonia
J44
copd
non-communicable
121
acute myocardial infarction
non-communicable
X00-X09
fires
external
N17-N19
renal failure
non-communicable
C34
lung cancer
non-communicable
O00-O99
maternal
maternal
W65-W74
drowning
external
100-199
other cardiovascular diseases aids
non-communicable
B20-B24
aids-tb
Varies
other non-communicable diseases
non-communicable
W00-W19
falls
external
V01-V99
road traffic
external
E10-E14
diabetes
non-communicable
A00-B99
other infectious diseases
communicable
A15-A19
tb
aids-tb
X60-X84
suicide
external
Varies
other injuries
external
C53
cervical cancer
non-communicable
160-169
stroke
non-communicable
B50-B54
malaria
communicable
J45
asthma
non-communicable
C18-C20
colorectal cancer
non-communicable
X85-Y09
homicide
external
A09
diarrhea diarrhea/dysentery
/dysentery
communicable
C50
breast cancer
non-communicable
C81-C96
leukemia/ leukemia/lympphee /lymphomas
non-communicable
X40-X49
poisonings
external
C61
prostate cancer
non-communicable
C15
esophageal cancer stomach cancer bite of venomous animal
non-communicable
C16
non-communicable
T63
external
A.2
PPI and PPI++ overview
We formalize our problem setting while reviewing the prediction-powertte inference PPI and PPI++ approach (Angelopoulos et al., 2023a;b). The goal of PPI is to estimate a d- dimensional parameter 0 with labeled data points (X Yn) P, E (1...., and N unlabeled data points Xyi IP x, E (1,...,Ny NJ. Additionally, there is black-box model f that uses the features to predict the outcomes based on labeled and unlabeled data, denoted as f/- = ( Y/ and Y/f (Yf...., &fx) The prediction rule f assumed to be independent of the observed data. While PPI performs inference on a correction term to incorporate these black-box predictions; PPI++ improves the methodology to be more computationally efficient and develops a modification that adapts to the accuracy of the prediction rule f by introducing additional parameter À that interpolates between classical and prediction-poweee inference.
The starting point of PPI and PPI++ approach is to define the population losses:
I(0)) (X, Y)), and
15

---

Published as a conference paper at COLM 2024
(e) EXlo(X, Pf.
It is recognized that
(0) = (X,, 8f1 Ella Xu, Y.)1 = (0).
Therefore the "rectified" loss is defined as:
L PP (0) (n( (0) -I,
where
(0)
lo (Xii Yii), Σ i=1
Li (8) =
1
Σ
14( X 11)),, and
11
i=1
Σ le Xui Pf
The rectified loss leads to the prediction-powered point estimate:
@PPI arg migmin/ PP) (8).
OER
PPI constructs the confidence set C. CPP/ {0 R. E C (0)} for true parameter 0 by forming (1 - -c-ofddieeece sets, C. (0), for the gradient of the loss VL.PP(()) This approach works well for fixed but suffers from computational issue when forming the confidence set for every Rd PPI++ derives the asymptotic normality about Ò. This result allows the construction of (1 - a) -confidence intervals CPP = =8P-P1 = a/2 x /n), where 03 is a consistent estimate of the asymptotic variance of 0.PP and zq is the q-quantile of standard normal distribution. Similar confidence intervals centered around @PPI can be constructed when the inference performed on the whole 8* vector rather than single coordinate fj,
which results in far more efficient algorithm.
Moreover, PPI++ generalizes PPI to allow the inference to adapt to the accuracy of the supplied predictions, yielding estimations that are never worse than the classical inference. This approach is called "power tuning" by incorporating tuning parameter À E [0, 1] in the rectified loss, leading to the following estimator:
0 arg min 1 (0),
where
PPPI+ (8) (0) A(L (0) I.f (0)).
PPI++ recovers the original PPI approach when A = 1 and reduces to classical inference
when A = O. One can adaptively choose a data-dependent tuning parameter Å to maximize the estimation and inferential efficiency.
A.3 multiPPI++
In K-class multinomial classification problem with outcomes Y; E (0,..., K- variates X; E IR,, the parameter of interests d(K 1)-dimensional vector whose k-th block 0g E R of components represent parameters for class k E (1,..., 1} excluding the reference class to avoid overparameterization. The loss function in classical inference takes the form
1} and co-
4x, = -
Σ
x Oy, + +log
Σ exx
16

---

Published as a conference paper at COLM 2024
Algorithm 1: multipPI++ powe inference for Multinomial Classification
Input: labeled K-category COD data {(X,, Yn) unlabeled data { /i=1/=1/ NLP
model f, significance level (0,1), coefficient index E [d(K - 1)]
1. Optimally select tuning parameter À // set tuning parameter
2. 6n arg argmin (8)
// multipPI++ estimator
A BERi(K-1)
3. = Nin (-i= 4 (X) pm
+ + V (X x X where
400,x) = log [ ex 9/
0g E Rd empirical Hessian
4. Σ ( V where
VF = N'Cov N+ ( (V X,)) and
VA Cov. (x7 + (AY/ Xn) // covariance estimator
Output:
Prediction-powent point estimates 0 and confidence interval
CT (m + a/2 [ij//1 for coordinate
Defining
Σ Eerloe
(00,x) log
and the multipPI++ loss is given by
(0) (0) + MLL (0) - Li (0))
- 1 11
Σ (X1i0yn 10 X(:)
À
(X e - (0,
Σ (7.0 -
N
Angelopoulos et al. (2023b) shows that the parameter can be optimally tuned to minimize the asymptotic variance of the ceddiction-powereee estimate, leading to better estimation and inference than both the classical and PPI strategies. Particularly, in finite samples, we can obtain the plug-in estimate
Tr (A-1 Opp:
(Covn( (Von(Vlonn Opmp' VI
Opp! + Covn (V/ Vlggn)) A (ppt)
Opp!
X
2(1 + N)
Tr
A
Oppi Cov VI Oppi 1A,1
0ppl)
where 0ppr = gpeT++ is obtained by taking fixed À E [0,1].
We summarize the multippI++ adjusted inference procedures for multinomial logistic re- gression coefficients on predicted CODs by an NLP model f against covariate X, under significance level (0, in Algorithm 1.
17

---

Published as a conference paper at COLM 2024
A.4
Parameter Estimates Across All Sites: Full Data 80/20 Split
mexico KNN
Ground Truth
Naïve
0.04
MultiPPI++ Correction
0.02
Log Odds Ratio
0.00
-0.02
-0.04
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 8: Full Data 80/20 Split: Site/Mode Modeel
18

---

Published as a conference paper at COLM 2024
mexico NB
0.10
Ground Truth
Naïve
+ultiPPI++ Correction
0.08
0.06
Log Odds Ratio
0.04
0.02
0.00
-0.02
-0.04
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 9: Full Data 80/20 Split: Site/Modi
19

---

Published as a conference paper at COLM 2024
mexico bert
0.10
Ground Truth
Naïve
0.08
+ultiPPI++ Correction
0.06
Log Odds Ratio
0.04
0.02
0.00
-0.02
-0.04
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 10: Full Data 80/20 Split: Site/ Site/Model: 3
20

---

Published as a conference paper at COLM 2024
mexico gpt4 _zs
0.06
Ground Truth
Naïve
+ultiPPI++ Correction
0.04
0.02
Log Odds Ratio
0.00
-0.02
-0.04
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 11: Full Data 80/20 Split:
21

---

Published as a conference paper at COLM 2024
ap KNN
Ground Truth
0.06
Naïve
+ultiPPI++ Correction
0.04
Log Odds Ratio
0.02
0.00
-0.02
-0.04
-0.06
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 12: Full Data 80/20 Split:
5
22

---

Published as a conference paper at COLM 2024
ap SVM
Ground Truth
0.06
Naïve
multiPPI++ Correction
0.04
0.02
Log Odds Ratio
0.00
-0.02
-0.04
-0.06
-0.08
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 13: Full Data 80/20 Split:
6
23

---

Published as a conference paper at COLM 2024
ap NB
Ground Truth
0.075
Naïve
multiPPI++ Correction
0.050
0.025
Log Odds Ratio
0.000
-0.025
-0.050-
-0.075
-0.100
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 14: Full Data 80/20 Split:
7
24

---

Published as a conference paper at COLM 2024
ap gpt4 zs
Ground Truth
0.04
Naïve
+ultiPPI++ Correction
0.02
Log Odds Ratio
0.00
-0.02
-0.04
-0.06
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 15: Full Data 80/20 Split:
8
25

---

Published as a conference paper at COLM 2024
up KNN
0.08
Ground Truth
Naïve
+ultiPPI++ Correction
0.06
Log Odds Ratio
0.04
0.02
0.00
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 16: Full Data 80/20 Split:
26

---

Published as a conference paper at COLM 2024
up SVM
Ground Truth
0.12
Naïve
multiPPI++ Correction
0.10
0.08
Log Odds Ratio
0.06
0.04
0.02
0.00
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 17: Full Data 80/20 Split: Site/Mode 10
27

---

Published as a conference paper at COLM 2024
up NB
0.16
Ground Truth
Naïve
0.14
multiPPI++ Correction
0.12
Log Odds Ratio
0.10
0.08
0.06
0.04
0.02
0.00
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 18: Full Data 80/20 Split: Site/Modi 11
28

---

Published as a conference paper at COLM 2024
up gpt4 zs
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
0.00
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 19: Full Data 80/20 Split: Site/Model 12
29

---

Published as a conference paper at COLM 2024
dar KNN
0.08
Ground Truth
Naïve
0.06
+ultiPPI++ Correction
0.04
Log Odds Ratio
0.02
0.00
-0.02
-0.04
-0.06
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 20: Full Data 80/20 Split: Site/Model 13
30

---

Published as a conference paper at COLM 2024
dar SVM
Ground Truth
Naïve
0.06
+ultiPPI++ Correction
0.04
Log Odds Ratio
0.02
0.00
-0.02
-0.04
-0.06
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 21: Full Data 80/20 Split: Site/Model 14
31

---

Published as a conference paper at COLM 2024
dar NB
Ground Truth
0.08
Naïve
+ultiPPI++ Correction
0.06
0.04
Log Odds Ratio
0.02
0.00
-0.02
-0.04
-0.06
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 22: Full Data 80/20 Split: Site/Model 15
32

---

Published as a conference paper at COLM 2024
dar gpt4 zs
Ground Truth
Naïve
0.04
+ultiPPI++ Correction
0.02
Log Odds Ratio
0.00
-0.02
-0.04
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 23: Full Data 80/20 Split: Site/Mode 16
33

---

Published as a conference paper at COLM 2024
bohol KNN
Ground Truth
0.08
Naïve
+ultiPPI++ Correction
0.06
Log Odds Ratio
0.04
0.02
0.00
-0.02
-0.04
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 24: Full Data 80/20 Split: Site/Model 17
34

---

Published as a conference paper at COLM 2024
bohol SVM
Ground Truth
0.08
Naïve
+ultiPPI++ Correction
0.06
Log Odds Ratio
0.04
0.02
0.00
-0.02
-0.04
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 25: Full Data 80/20 Split: Site/Model 18
35

---

Published as a conference paper at COLM 2024
bohol NB
0.150
Ground Truth
Naïve
0.125
multiPPI++ Correction
0.100
0.075
Log Odds Ratio
0.050
0.025
0.000
-0.025
-0.050
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 26: Full Data 80/20 Split: Site/Model 19
36

---

Published as a conference paper at COLM 2024
bohol gpt4 zs
Ground Truth
Naïve
0.06
+ultiPPI++ Correction
0.04
Log Odds Ratio
0.02
0.00
-0.02
-0.04
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 27: Full Data 80/20 Split: Site/Mode) 20
37

---

Published as a conference paper at COLM 2024
pemba gpt4 _zs
Ground Truth
0.125
Naïve
multiPPI++ Correction
0.100
0.075
Log Odds Ratio
0.050
0.025
0.000
-0.025
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 28: Full Data 80/20 Split: Site/Modi 21
38

---

Published as a conference paper at COLM 2024
A.5 Sensitivity Analysis
In a sensitivity analysis, we varied our data splitting strategy, which revealed nuanced insights regarding the behavior of the classifier. Notably, when maintaining the split proportions (80/20) within each COD, the PPI++ classifier exhibited minimal utilization of the labeled data, indicated by the relatively small values of Å. Conversely, when employing an 80 80/20 split on the entire dataset, this lead to a significant information loss for the minority classes, resulting larger À values for certain splits. Since an (80/20) on the whole dataset more closely resembles what one would see in reality on prospective study while an (80/20) split by COD is resembles retrospective study, this illustrates the importance of purposefully choosing an appropriate data splitting strategy to match the desired analysis.
39

---

Published as a conference paper at COLM 2024
A.6
Parameter Estimates Across All Sites: Stratified 80/20 Split
pemba KNN (A = 0.04)
0.08
Ground Truth
Naïve
MultiPPI++ Correction
0.07
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External
Maternal
Non-communicable
Cause of Death Label
Figure 29; Stratified 80/20 Split: Mite/MMoel 22
40

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 30: Stratified 80/20 Split: Site/Mode Model 22
41

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 31: Stratified 80/20 Split: Site/Mode Model 22
42

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 32: Stratified 80/20 Split: Site/Mode Model 22
43

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 33: Stratified 80/20 Split: Site/Mode Model 22
44

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 34: Stratified 80/20 Split: Site/Mode Model 22
45

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 35: Stratified 80/20 Split: Site/Mode Model 22
46

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 36: Stratified 80/20 Split: Site/Mode Model 22
47

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 37: Stratified 80/20 Split: Site/Mode Model 22
48

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 38: Stratified 80/20 Split: Site/Mode Model 22
49

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 39: Stratified 80/20 Split: Site/Mode Model 22
50

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 40: Stratified 80/20 Split: Site/Mode Model 22
51

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 41: Stratified 80/20 Split: Site/Mode Model 22
52

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 42: Stratified 80/20 Split: Site/Mode Model 22
53

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 43: Stratified 80/20 Split: Site/Mode Model 22
54

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 44: Stratified 80/20 Split: Site/Mode Model 22
55

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 45: Stratified 80/20 Split: Site/Mode Model 22
56

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 46: Stratified 80/20 Split: Site/Mode Model 22
57

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 47: Stratified 80/20 Split: Site/Mode Model 22
58

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 48: Stratified 80/20 Split: Site/Mode Model 22
59

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 49: Stratified 80/20 Split: Site/Mode Model 22
60

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 50: Stratified 80/20 Split: Site/Mode Model 22
61

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 51: Stratified 80/20 Split: Site/Mode Model 22
62

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 52: Stratified 80/20 Split: Site/Mode Model 22
63

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 53: Stratified 80/20 Split: Site/Mode Model 22
64

---

Published as a conference paper at COLM 2024
pemba KNN (λ 0.04)
0.08
Ground Truth
Naïve
0.07
multiPPI++ Correction
0.06
Log Odds Ratio
0.05
0.04
0.03
0.02
0.01
Communicable
External Cause of Death Label
Maternal
Non-communicable
Figure 54: Stratified 80/20 Split: Site/Mode Model 22
65

---

Published as a conference paper at COLM 2024
B Ethics Statement
As this paper covers the study of the lives and deaths of real human beings, the ethical aspect of this investigation was considered paramount importance. First, ensure that any potential personally identifialle information was not mishandled, multiple layers of data security were employed, the primary means being the use of secure Microsoft Azure cluster for the running of local models. Data concerns are always paramount. We are fortunate to be working with de-identified data, however, when working with more sensitive data there are number of open-source LLMs which can be run locally (Llama-3, meditron, etc) to ensure safe and ethical research practices. Additionally, many institutions, public and private, buy access to local OpenAl servers. Our of these methods is illustrative, aiming to lay foundational understanding of how to perform valid inference with NLP predictions. When communicating with outside organizations such OpenAl's GPT models, ensured that all data set were from the publicly available PHMRC data, which has been previously deidentified. Finally, in accordance with the importance of studying a sensitive topic such as VAs deserve, we have done our utmost to ensure that our study's objectives and tools utilized, the best of our abilities, gives fair, and respectful analysis of subject matter in question so that its results may be used for the furtherment of public health goals.
C Reproducibility Statement
To ensure the reproducibility of results, have published our code repository with our entire pipeline NLP training, predicting, inference and visualizations to Github. Please direct questions about data modeling via email to avisokay or fansx at uw. edu. All data is available in the data folder in the Github repository.
66


Reviews:
[{"review_id": "bCTAAcdMov", "reviewer": "Reviewer_EkdW", "summary": "This paper develops an approach to perform *inference* on categorical predictions from a machine learning model applied to unstructured text (in contrast with prior work performing *prediction*) for the task of cause-of-death classification from narrative text.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for this feedback and recognize the importance of demonstrating the versatility and robustness of our contribution. We will highlight that our primary goal was to develop and validate an initial framework that could be further generalized and adapted to various tasks within the domain of cause-of-death classification. In particular, in our discussion about future work, we will add more details about how we plan to validate the approach with external data, as well as extend to other application areas such as  in analyzing clinical text.\n\nWe agree that the ethical considerations related to our research need to be more clearly articulated. We will revise the ethics statement to (1) highlight the potential for biases in human-provided narratives (e.g, inaccuracies, incomplete information, or cultural, social, or personal perspectives of the individuals providing the narratives, (2) model-induced biases (e.g, bias from the methods which may embed biases present in the training data, and (3) misclassification (e.g., in particular from class imbalance, which we address in the appendix using an output-stratified bootstrap sample for our leave-one-out experiment. (See figure 10, for example)). Further, we will clearly state that in practice, ensuring the privacy and confidentiality of the individuals whose narratives are used is critical. We will reiterate that all of the data used in this study were anonymized, but other sources of narratives may preclude the use of pre-trained LLMs such as ChatGPT due to ethical concerns surrounding data use and privacy. Here, we wanted to highlight the performance of a range of NLP techniques, and were fortunate to have a rich, de-identified dataset to do so.\n\nRegarding your question about models using logistic / cross-entropy loss, yes, both our method and PPI++ in general are designed to be agnostic to how the prediction rule was trained and uses data-adaptive weights to incorporate information from ‘black box’ predictions into the overall procedure. We will make this more explicit in the main text. \n\nLastly, we agree that including threshold-free metrics such as the ROC curve, ROC-AUC, PR curve, and PR-AUC would provide a more comprehensive evaluation of how our various NLP models perform. We will gladly revise Section 4.2 to include these metrics, as well as provide a more robust discussion on the performance of the method in light of these changes. Thank you for all of these valuable suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper considers an important task with the potential to significantly reduce human effort and improve decision-making accuracy. The paper is well-written, clear, does not over-claim, and is both theoretically and empirically principled.", "reasons_to_reject": "The considered task, while important, is quite specific. A drawback of this paper is the focus on a single task, without empirical benchmarking of the proposed inference approach on multiple tasks.\n\nThe ethics statement is not specific enough for the nature of research being performed. The statement should warn readers of the variety of risks of using the proposed approach in practice, especially since the data includes human-provided narratives which might be biased or incorrect along with the biases embedded in machine learning models.", "questions_to_authors": "Can multiPPI++ be generalized to any model that uses a logistic / cross-entropy loss (eg. a frozen LLM with a logistic head)?\n\nCould threshold-free metrics be reported in Section 4.2. Namely: the ROC curve and the ROC-AUC, and the precision-recall (PR) curve and the PR-AUC?", "extra_scores": {}}, {"review_id": "O6WXIee9Io", "reviewer": "Reviewer_hhrX", "summary": "This paper applies an array of multi-class classification approaches to cause of death prediction from verbal interview transcriptions with the bereaved family (a KNN, NB, SVM BOW approach), a fine-tuned BERT model and GPT-4. \nThe authors further claim to propose the adaptation of PPI++ to multiclass logistic regression, in order to perform domain adaptation for a model trained on domain A (with labelled samples available) to domain B (with only unlabelled elements). \nThe authors independently evaluate the approaches in a classical fully-labelled setting. Without any class-imbalance mitigation strategy, most of the BOW models and BERT overfit on the majority class prediction. Only GPT-4 doesn't suffer from this problem and mainly only confuses Communicable with non-communicable causes. No error analysis is provided to truly understand why or whether prompt adaptation strategies could solve the problem. \nAs for multiPPI++, which is used to enable domain transfer, only the distributions of corrected  confidence estimates on the loss ar4e provided, this absolutely doesn't evaluate whether it helps for domain transfer. \n\nNormally multinomial classification means the same thing as multiclass classification, which Angelopoulos et al. (2023 b) https://arxiv.org/pdf/2311.01453, already define on page 7 section 3.1. \nDo the authors mean multilabel multi-class classification?  But then is the only adaptation computing the mean of the first terms across labels ? PPI++ involves many prerequisites w.r.t. the loss function, does your modification preserve these prerequisites? Particularly the stochastic smoothness? Maybe it's obvious for someone more mathematically adepts, but it feels like this is the sort of thing that's extremely important to assess the validity of what's proposed. \nI feel like your modification assumes the multiple outcomes are independent from each other, is this a reasonable assumption? I am not convinced it's that simple...", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 2 for their detailed review and helpful comments. Regarding domain transfer, a primary aim of this work is to illustrate how predicted outcomes from an NLP model can be adjusted to yield accurate statistical parameter estimates even when a domain transfer has occurred. Additionally, an alternative view of this problem would be to account for ‘covariate shift,’ i.e., when the underlying distribution of our predictors differs across datasets. There has been discussion of this in Angelopoulos et al. (2023), which can readily be adapted to this work. \n\nIn section A2, we will direct readers to the specific section of Angelopolulos et al rather than quoting their description. \n\nWe will add ICD-10 Codes to the table in A.1. \n\nWe compute f1-score using the standard sklearn implementation and show confusion matrices in Figure 5. \n\nWhile we include our prompt in Figure 3, we agree that we do not sufficiently discuss our strategy. We will elaborate on the prompt engineering and associated error analysis in that section. \n\nRegarding class imbalance, in a sensitivity analysis (see Figures 29-54 in A.6), we used one approach to correct for class imbalance by training the model on stratified data, with an 80/20 split for each class to achieve balance w.r.t COD. The results were not substantively different from our main results, so we showcase an example where the NLP model is not perfect, but multiPPI++ still obtains valid statistical estimates. To clarify our attention to this important concern, we will incorporate more discussion, as well as comparisons to state of the art methods (e.g., SMOTE). \n\nAs for the performance, Figure 6 demonstrates that multiPPI++ improves the accuracy of statistical estimates using a post-facto correction term. MultiPPI++ does not change prediction accuracy, it only affects the accuracy of the statistical estimates and appropriately widens the confidence intervals to account for the additional uncertainty. We will take care to emphasize this point more clearly. \n\nRegarding multinomial classification, Agelopoulos et al. 2023 provide a formula. We take that formula, reparameterize it, and develop an optimization algorithm. We will clarify that this does not affect the statistical properties of the estimates (the stochastic smoothness condition is satisfied, as $\\nabla \\theta$ is locally Lipschitz near $\\theta^*$ for the problem.)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The task explored is interesting from the perspective of biomedical informatics, as it's something that's not typically not addressed much.\nIn theory the proposed extension would be a very interesting contribution (if done right, with the right assumptions).", "reasons_to_reject": "The experimental validation of the various approaches without multiPPI++ is very incomplete, we need a much more detailed report, with a complete error analysis and discussions on prompting strategies. Class-imbalance must also be addressed, otherwise the behaviour of the simple baselines is really to be expected. \n\nThe proposed extension of PPI++ seems to be made under questionable simplifying assumptions that do not seem realistic (independence of selected subset of outcomes). The requirements of PPI++ regarding the modified loss aren't discussed and there is nothing that indicated whether they still hold (even if it's supposed to be obvious, some readers might need to explicitly see it stated with at least an informal justification. \n\nI feel like the paper needs to focus on one contribution: the application task (with a proper evaluation, including domain transfer ability) or the more fundamental extension of multiPPI++ (with proper proofs and mathematical justifications!).", "questions_to_authors": "A.1 ICD-10 COD Classification > Better to include the actual ICD-10 codes \n\n\"We then applied three\nsupervised classifiers to predict COD using the BoW representations and ground truth cause of death\nlabels: Support Vector Machine (SVM), K Nearest Neighbors (KNN), and Naive Bayes (NB). SVM\nwas implemented with a third-degree linear kernel and C = 1.0, while KNN used cosine distance and\n9 nearest neighbors, chosen by incrementing from K = 3 until accuracy saturation. The BERTBASE\nencoder was configured with five transformer layers, to include two input layers of size 256, a main\nlayer of size 768, an intermediate layer of size 512, and a fine-tuning output layer of size 5. As the VA\nnarratives in our data were de-identified, we were able to further assess the predictive performance\nof OpenAI’s GPT models (GPT-3.5, GPT-3.5-turbo, GPT-4, and GPT-4-32K Biswas, 2023)\" > Was there any parameter estimation performed? did you have an a priori rationale to select these specific parameters? \n\n\n\nF1-Score > Can you detail how you are computing the F1 score? In multinomial classification you would compute individual F1 scores for each class, the mean of class-wise F1-scores wouldn't be particularly meaningful... \nEven then, we would perhaps like additional information (e.g. what the distribution of the of the scores is. In that sense the confusion matrices are much more informative. \n\n\nBesides for GPT-4, it looks like all the models are overfitting and predicting the majority class! Something must be done to tackle class-imbalance, that's likely the reason why most of the approaches perform poorly (including BERT!)\nWhy isn't the confusion matrix from BERT also represented? Are you trying to hide issues that you may have had? \n\n\n\"Figure 6 displays point estimates and 95% confidence intervals for the ground truth, Naive\nand multiPPI++ corrected inference\" While it's a nice visualization, what meaningful information does this bring for the analysis? \nNot really commented, what's the meaning of those difference? Does it have any significance in terms of domain expertise? \n\nDoes adding multiPPI++ improve or degrade performance? Why don't you report of the task performance for all three scenarios? You can train with the predicted label augmentation and evaluate only on full labels. \nIf this impreoves performance it's certainly a good indicator of the effectiveness, idependently of whether the uncertainty estimates of multiPPI++ better correspond to the true uncertainty estimates. \n\n\nMost of the text in A2 is a verbatim copy of the text in (Angelopoulos et al., 2023a;b), if you are quoting, you should clearly say so, or just telle readers to read the appropriate section in the original papers. \nEven if it's in the appendinx, this is wrongful attribution.  \n\n\nIf multiPPI is a contribution of the paper, then it should be described adequately in the main body of the paper. Given the evaluation provided, I don't see any demonstration of the concrete usefullness of multippi++ (no because there isn't any, but because this is not evaluated). \n\n\nI am missing an error analysis, particularly for models that do not overfit on the majority class, as we could for example understand why communicable and non-communicable are so often confused by GPT-4. \n\n\nWhy is algoritm 1 in the annex if it's cited in the body of the text, that's a way of cheating the page limit in my opinion. \n\nAs far as I see in (Angelopoulos et al., 2023a;b) there is a specific \"stochastic smoothness\" condition on the loss function, for this method to work, besides the iid requirement. Did you actually verify any of these requirements to determine if the method is applicable at all?", "extra_scores": {}}, {"review_id": "DzounAdWe1", "reviewer": "Reviewer_wuA5", "summary": "The authors present a benchmark for predicting the Cause of Death from Verbal Autopsies and introduce a method, called multiPPI++, to identify quantitative factors responsible for the cause of death, such as age. \n\nThe experimental setup is constrained by geographically-dependent distributions and investigates valid inference with a mix of ground truth and predicted labels. \n\nWhile experiments are sound and multiPPI++ are interesting, the conclusion and contributions do not appear significant enough. The contributions are focused on an NLP-agnostic statistical method rather than advancing LLMs (as a field).", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for this feedback and agree that we can do more to emphasize why this work is an important contribution to advancing the field of LLMs. While it is true that we are not proposing new NLP algorithms, NLP methods are used in an extremely wide range of settings, often with the goal of performing downstream inference. Our multiPPI++ method enables valid inference when outcomes are predicted using an NLP model and, thus, enhances the utility of NLP predictions. We believe this is a contribution to the field as it is an important step in the process of leveraging NLP to make sound, data-informed decisions.\n\nWe thank the reviewer for bringing this to our attention and agree that these claims are not sufficiently supported in our initial submission. We will elaborate on how our parameterization overcomes the issues which arise in Angelopoulos 2023b. We will also include citations to the leading prediction accuracies in the existing literature and provide a direct comparison to our NLP prediction accuracies using text narratives. These clarifications will specify and substantiate our claims you identified above. \n\nWe thank the reviewer for their valuable comments highlighting all of the above ways we can improve the clarity, scope, and motivation of this work. In addition to what we have outlined above, we will include a Related Works section to our discussion to emphasize what motivates our work and showing how our contribution is situated in the language modeling literature.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The problem tackled is interesting and addressed with a consistent experimental setup in which results are correctly analyzed.\n\nThe exploratory data analysis helps anchor the challenges of geographically-dependent Cause of Death distributions.\n\nThe use of Language Models, in comparison with older NLP techniques, to study inference and prediction of the Cause of Death from Verbal Autopsy Narratives, represents an opportunity to investigate an important topic.", "reasons_to_reject": "Originality is limited. The crux of the novelty in the proposed method is to adapt a published method called PPI++ to settings of multinomial classification. \nThe paper is not well suited for a conference on Language Models but rather adapted for a statistics venue.\n\nSome claims are either too vague, not corroborated or not explained: \n- \"Angelopoulos et al. (2023b) [...] 's model has overparametrization issues.\"\n- \"GPT-4’s predictions matched the performance of the best examples from the VA literature.\"\n\nThe presentation could be adapted to improve the clarity of 1) the motivation and 2) the method. \nAlso, the paper could benefit from a dedicated section focusing on Related Works.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iXL73Hkdli", "reviewer": "Reviewer_VDYf", "summary": "This paper presents a methodology for deriving statistically sound inferences from causes of death predicted using verbal autopsy (VA) narratives and modern NLP techniques. Their approach, termed multiPPI++, is an extension of prediction-powered inference adapted for multinomial classification scenarios commonly encountered in public health research. Despite the variability in model accuracies, the multiPPI++ method consistently reconciles predicted causes of death with a small subset of verified ground truth data Results suggest that inference correction is important in health-based decision making", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for raising these concerns and providing constructive feedback. We agree that in our review of prior work we should be more thorough in citing the confluence of VA and NLP. We intend to include more specific examples such as Manaka et al. 2022a and 2022b. These authors use BERT and ELMo for feature extraction from VA narratives to enhance COD classification tasks but do not comprehensively address the COD class imbalance challenge. We will note that this is a difficult problem in this domain that our method helps to address. \n\nRegarding the gap between BoW methods and GPT-4, it is our intent to show that our method multiPPI++ is model-agnostic, and applicable across a range of NLP models. This is an important point because practitioners could feasibly use any number of NLP methods, sophisticated or archaic, and our method remains robust and adaptable to models of varying performance.\n\nEchoing our response to Reviewer 2 regarding class imbalance and overfitting, we point out that we conducted a sensitivity analysis (see Figures 29-54 in A.6), where we used one approach to correct for class imbalance by training the model on stratified data, with an 80/20 split for each class to achieve balance w.r.t COD. The results were not substantively different from our main results, so we showcase an example where the NLP model is not perfect, but multiPPI++ still obtains valid statistical estimates. To clarify our attention to this important concern, we will incorporate more discussion, as well as comparisons to state of the art methods (e.g., SMOTE). \n\nTo expand on the multiPPI++ complexity and tradeoffs, we will add clarification to emphasize that MultiPPI++ does not change NLP prediction performance, it only affects the accuracy of the downstream statistical estimates and appropriately widens associated confidence intervals to account for the additional originating from the NLP model. We will take care to emphasize this point more clearly. \n\nData concerns are always paramount. We are fortunate to be working with de-identified data, however, when working with more sensitive data there are a number of open-source LLMs which can be run locally (Llama-3, meditron, etc) to ensure safe and ethical research practices. Additionally, many institutions, public and private, buy access to local OpenAI servers. Our use of these methods is illustrative, aiming to lay a foundational understanding of how to perform valid inference with NLP predictions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-\tThe multiPPI++ method is somewhat novel approach to multinomial classification and may be appropriate for the complex data environments common in public health\n-\tThe empirical validation is well supported by the very extensive supplemental material, although much of the detail in that (overly large) section does not make its way back into the paper itself\n-\tThis is an important area of work, especially for low-resource settings where most deaths occur outside hospitals an few autopsies are performed.", "reasons_to_reject": "-\tThere has been a lot of work over the past decade on modern NLP methods used to assess verbal autopsies, and relatively little of that was cited. Rather, the literature review somewhat superficially touches on each of the component topics separately (e.g., NLP in the whole, verbal autopsies in the whole) but not their previous confluence. Indeed, despite the mere fact that there is some novelty (as mentioned above), this is broadly a slight evolutionary step. \n-\tThe empirical validation in the paper is somewhat light, and the rationale could be improved. In particular, the comparison of archaic methods like KNN and SVM to GPT-4 provides a massive gulf that one would expect to be filled by modern pre-trained language models that can be fine-tuned and run on-premises. \n-\tLittle attention is paid to typical issues with data of this type, especially overfitting or other forms of poor generalization. \n-\tThere is little justification for the complexity of the multiPPI++ process, nor are tradeoffs (e.g., between complexity and performance improvement) explored. Better ablations may have elucidated this.", "questions_to_authors": "-\tIs there a concern about sending healthcare data to OpenAI’s servers, that could limit the use of this tool broadly? You had some flexibility given your dataset, but this wouldn’t this contradict data legislation for actual healthcare data?", "extra_scores": {"ethics_concerns_details": "The issue of sending THIS dataset to OpenAI is addressed in the Ethics section, but this simply would not be legal on ACTUAL healthcare data across many jurisdictions"}}]

Author rebuttal:

Meta-review: This work focusses on inferring numerical attributes of interest from free-text, ultimately to identify patterns from verbal autopsies. This is a shift, then, from the usual emphasis on *prediction* and toward *inference* over free-text. The work is clearly presented and there are clear contributions here.

While some reviewers raised concerns regarding evaluation, there was also broad consensus that this work constitutes an interesting direction and operationalizes this in service of an important problem in public health. The approach may be adopted for a range of related inference problems in public health.

[At least one review was discounted during the decision process due to quality]
Decision: Accept