End-To-End Causal Effect Estimation from Unstructured Natural Language Data
Authors: Nikita Dhawan
Leonardo Cotta
Karen Ullrich
Rahul Krishnan
Chris J. Maddison
Venue: NeurIPS 2024 poster (2024)
Keywords: treatment effect estimation
unstructured reports
real-world effects
Primary area: natural_language_processing
TL;DR: We propose a family of treatment effect estimators built with large language models that leverage the large quantity, diversity and flexibility of observational, unstructured text data in the form of reported experiences.

Abstract: Knowing the effect of an intervention is critical for human decision-making, but current approaches for causal effect estimation rely on manual data collection and structuring, regardless of the causal assumptions. This increases both the cost and time-to-completion for studies. We show how large, diverse observational text data can be mined with large language models (LLMs) to produce inexpensive causal effect estimates under appropriate causal assumptions. We introduce _NATURAL_, a novel family of causal effect estimators built with LLMs that operate over datasets of unstructured text. Our estimators use LLM conditional distributions (over variables of interest, given the text data) to assist in the computation of classical estimators of causal effect. We overcome a number of technical challenges to realize this idea, such as automating data curation and using LLMs to impute missing information. We prepare six (two synthetic and four real) observational datasets, paired with corresponding ground truth in the form of randomized trials, which we used to systematically evaluate each step of our pipeline. NATURAL estimators demonstrate remarkable performance, yielding causal effect estimates that fall within 3 percentage points of their ground truth counterparts, including on real-world Phase 3/4 clinical trials. Our results suggest that unstructured text data is a rich source of causal effect information, and NATURAL is a first step towards an automated pipeline to tap this resource.

End-To-End Causal Effect Estimation from Unstructured Natural Language Data
Nikita Dhawan'
Leonardo Cotta
University of Toronto, Vector Institute nikita@es, toronto. edu
Vector Institute
leonardo.cottennte htt.:a/o..octtteetttttrisssitaaa
Karen Ullrich
Rahul G. Krishnan
Meta Al
University of Toronto, Vector Institute
karenuSmeta.com com
rahulgk0cs toronto. edu
Chris J. Maddison
University of Toronto, Vector Institute cmaddis@cs. toronto..eeeeee edu
Abstract
Knowing the effect of an intervention is critical for human recision-makinge but current approaches for causal effect estimation rely on manual data collection and structuring, regardless of the causal assumptions. This increases both the cost and time-to-completie for studies. We show how large, diverse observational text data can be mined with large language models (LLMs) to produce inexpensive causal effect estimates under appropriate causal assumptions. We introduce NATURAL, a novel family of causal effect estimators built with L.LMs that operate over datasets of unstructured text. Our estimators use LLM conditional distributions (over variables of interest, given the text data) assist in the computation of classical estimators of causal effect. We overcome number of technical challenges realize this idea, such as automating data curation and using LLMs to impute missing information. We prepare six (two semi-synthetic and four real) observational datasets, paired with corresponding ground truth the form of randomized trials, which used to systematically evaluate each step of our pipeline. NATURAL estimators demonstrate remarkable performance, yielding causal effect estimates that fall within 3 percentage points of their ground truth counterparts, including real-world Phase 3/4 clinical trials. Our results suggest that unstructured text data is rich source of causal effect information, and NATURAL is first step towards an automated pipeline to tap this resource.
1 Introduction
Estimating the causal effects of interventions time consuming and costly, but the resulting outcomes are precious. Health agencies around the world often require randomized controlled trial (RCT) data to approve medical interventions. Clinical trials are key contributors large R&D costs for drug developers (30]. Natural experiments are another source of rich interventional data, but they may not always exist or have enough data relevant to given causal hypothesis 12]
When treatment randomization is infeasible, observational data can be used to identify average treatment effects (ATEs) 488, under common assumptions, egg., unobserved confounding. Such
Work done during internship at Meta AI.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Financial cost
Study cobort(s)
Treatment assignment
Observed data
Example estimator & causal graph
Completely randomized experiment
Difference-in-Mese
$$$
Observational structured experiment
IPW
$$
Ohservational $
teied NATURAL
natural language
halped! haven't had single
experiment
Figure I:
When compared to experimental and other observational studies, NATURAL has lower costs and provides greater diversity in cohort selection, for causal effect estimation.
data is abundant but even when the necessary assumptions are satisfied, it must be structured (i.e., the outcomes, treatments, and relevant covariates must be defined, recorded, and tabulated) before it becomes amenable to computational analyses.
Yet, unstructured observational data presents unique opportunities for cheaper, more accessible, and potentially even better [32 effect estimation. For example, thousands of people living with diabetes choose to share their experiences with treatments online patient forums. Some of their posts contain rich descriptions of daily lives, the drugs they have been prescribed, the treatment responses and side effects, well pre-treatment information like age and sex. Their posts contain their lived experiences including evidence of an outcome in an observational experiment, albeit in an unstructured form. Other potential sources of rich unstructured, observational data include newspaper classifieds, police reports, social media, and clinical reports. Despite being collected for a myriad of purposes, researchers have often turned to such data test hypotheses since: (i) unstructured data does not require restrictive data collection designs, e.g., measurement choice, and can admit many different post-hoc analyses; (ii) the reported outcomes may reflect what matters to subjects better than standard outcome measures; (iii) value may be recovered from outcomes that would otherwise be lost; (iv) there may be more unstructured data available underserved marginalized populations. Figure I contrasts our setting with previous works using randomized or structured observational data.
This work asks simple question: How we large language models automate treatment effect estimation using freely available text data? We introduce NATURAL, a family of cext-conditioned estimators that addresses this by performing NATural language analysis Understand ReAL effects.
At high level, the steps required to compute NATURAL estimators are as follows. Given an observational study design and a dataset of natural language reports, filter for reports that are likely to conform to the experimental design. Then, using a large language model (LLM), extract the conditional distribution of structured variables of interest (outcome, treatment, covariates) given the report. Finally, use the conditionals to compute estimators the ATE, using classical strategies such inverse propensity score weighting and outcome imputation.
NATURAL is data-driven pipeline. It leverages and relies on the LLM in a manner that mimics the learning task it was trained for: providing parametric approximations to conditional distributions. As in all observational studies, the validity NATURAL also depends on prior causal knowledge about the task. Expert knowledge is required to define appropriate covariates and confirm that they satisfy the necessary assumptions for effect estimation. However, we anticipate that NATURAL estimators could be developed under other structural assumptions (e.g. instrumental variables) as well.
The core contributions of our work are:
We derived NATURAL ATE estimators based on classical estimators of the ATE, like inverse propensity score weighting and outcome imputation. NATURAL estimators operate on entirely unstructured data under two novel data-access assumptions.
We implemented NATURAL estimators using L.M--hased pipeline.
We developed six observational datasets to systematically evaluate parts of this pipeline: two semi-synthetic datasets constructed using marketing data, and four clinical datasets curated from public (pre-December 2022) migraine and diabetes subreddits from the Pushshift collection [7]. For each dataset, treated the ATE from corresponding real-world completely randomized experiment (CRE) as ground truth. Remarkably, our predicted ATES all fell within percentage points of the ground truth ATEs, potential cost savings of many millions of dollars.

---

1.1 Related Work
Leveraging natural language data 42 to support causal claims is pervasive in applied research Our work falls under the broad umbrella of accelerating the identification of real-world evidence (RWE) 40]. For instance, in the context of healtheare. RWE supports not only drug repurposing, but also post-market safety evaluations - its most common application. NATURAL expands the boundaries of how quickly one can obtain and validate such real-world evidence from observational data 41.
43 13].
The use of natural language data in causal inference comes in different flavors: using text to measure confounders (23), ii) using text to measure causal effect outcomes [15]. or iii) producing interpretable causal features from text [15 6 e.g., what words are more likely to explain the cause of an event. NATURAL distinguishes itself from these lines of research in two ways: i) NATURAL does not require any curated task-specific training data (it zero-shot), and ii) NATURAL is not interested in how the text itself, i.e., its words. relate to the causal problem -thtt is, we are only leveraging the model's ability to predict the distribution of a specified variable conditional on the input text. We highlight that our work lies distinct from research at the intersection of text and causality that combines text and numerical or tabular data [1), the latter of which may be unavailable or incomplete in settings involving neglected diseases, unrecorded abortions, or illicit drug use. Other work has also studied topic modelling approaches [2] or the ability of language models to infer latent variables (that are implied but not explicitly identified in text data) (36 13 Rather, we require the precise specification of covariates to condition on - we view this as being crucial to creating a more direct way for an end user to verify the validity of information extracted with approach.
Prior works have also leveraged LLMs black-box fashion for causal tasks by querying the model for causal statements. In the context of causal discovery, users directly ask for the existence of cause-and-effee relationships, e.g.." "Does changing the age of an abalone causes a change in its length?" 25 33
6.. Due to the large amount of training data, it is possible that the model learns to apply causal model described in the training data and answer causal questions with it [35 47). The issue with this approach is i) the user is limited to the causal models observed in training, ii) the user is not aware of which causal model they are using, and iii) the queries tend to present high prompt sensitivity (28). Finally, we note that recent work created benchmark and showed how LLMs struggle to distinguish pairwise correlation from causation (20). while another shows that checking causal relationships in a pairwise manner can lead to invalid causal graphs [45).
2 Preliminaries
We are interested in estimating the causal effect of a treatment relative to either another treatment or treatment in population of interest. More precisely, consider treatments / E (0, 1} and the corresponding potential outcomes (1)) and (0) under each treatment. We wish to compute the quantity == (1) Y())), often referred as Average Treatment Effect (ATE). Sometimes, (0) may correspond to no treatment (control). Throughout this work, we assume binary treatments and outcomes in the Neyman-Ruil Rubin causal model. We provide a full list of notation in appendix A
A Completely Randomized Experiment (CRE) with n participants requires no prior causal knowl- edge. In a CRE, the treatment assignment vector (T)) is a random permutation of ni ones and n n1 zeros sampled independently of the outcomes. In this case, the difference-in-ment n T,Y((1) - Dn (1 - T.)Y, provides with an unbiased estimate of 7. DT.
Despite the indisputable necessity of CREs in high-stakes settings, it is often expensive and/or infeasible have complete control over the treatment assignment. Instead, observational data is more readily available. Observational data often contains spurious correlations between the observed treatment T' and the observed outcome Y (L) + (1 T)Y (0) through common cause (confounder). Typically, this confounding is formalized as a variable X, which we assume to be discrete throughout this work, representing covariates associated with each individual. Given i.i.d. samples {(,, T, Yo) from the target population, standard causal inference techniques can correct for confounding bias and provide consistent estimates of 7 under Assumptions I and
Assumption (Strong Ignorability.) The potential outcomes are independent of trealment assign- ments conditional on covariates, i.e.. (0), (1)) 1L T.X.
3

---

Assumption (Positivity.) For every treatment and covariate set z, 0 << P(T = = <1.
Following are two classical estimators of the ATE from observational data, each of which rely on and We refer the reader to Ding [1] for further details.
X satisfying Assumptions
Inverse Propensity Score Weighting (IPW). The propensity score is the conditional probability of receiving treatment given the observed features, i.e., e(x) = P(T = 1|X æ). The IPW estimator given by
T.Y.
(1 (1-1T) T.
NPw =
((.))
1- eXx))
where ē(œ) ) an approximation of P(T = x= When é(z) is the true propensity score, fiew is an unbiased estimator of When ē(x:) is estimated as empirical probability, new is consistent.
Outcome Imputation (OI). Outcome Imputation learns model to impute outcomes from features and treatment and then marginalizes away the features to estimate with
for
Σ 71
7X,, 1) O),
where f(x, f) approximates P(Y = =|| x, T = t). Note that if far t) is an unbiased estimation
of this quantity, for is unbiased estimator of T.
3 NATURAL estimators of the ATE
Both CRE and observational studies require direct access tabulated data (X,, T,, Y,) for every individual i. Our NATURAL estimators the other hand estimate the ATE from observational. unstructured natural language data in the form of language reports RG In addition to Assumptions | and 2 NATURAL estimators require the following assumptions to guarantee their consistency.
Assumption (Natural language report data.) The target population is described by an observa- tional gentr-ggnnraiingg process P(X,T, Y, R) of data (X,T,Y) Y), which satisfies Assumptions Aand 2 and is jointly distributed with a random natural language string R, called a report. We assume access i..dd sample of reports ( from the marginal of this process.
Assumption 4 (Access to the true observational conditional over (X,T.Y). We can either r) of the true data- tentt-ggnerating process, or (ūi) we can sample from P(X- a: R = r) and compute P(T t, Y y R = r, X 2).
(i) compute the conditional P(X x T t, R
Intuitively, these assumptions give NATURAL indirect access to (X, T. Y) through R. They can be weak or strong, depending on the definition of the reports R.. On the one hand, if reports are copies of the observational data, i.e., R = (X, T, Y). then Assumption is trivial to satisfy. On the other hand, if reports are all the constant, empty string, R. e, then Assumption 4 guaranttess that we have full access to the true observational joint density function over (X, Y). which a strong assumption. In other words, it requires way simulate trial outcomes unconditionally (without any data). We consider how we might satisfy these assumptions in practice in the next section. Here, we assume that they hold and develop a series of consistent estimators of the ATE.
NATURAL Full. Given (R, |2 and P(X- x, T = 1, Y = R = r), we can construct an idealized version of NATURAL. Let start by noting that the law of total expectation gives
TY 11-77 T e(X)
TY
(1 TTY
1 e(X) ]1
T Ex.r.
ER EX,TYVI
(3)
1 1- eX)]
(((X)
Monte Carlo estimate over reports is given by
FN-Full = M Σ = P(x=2,T=
yR)
ty
(1 t)y
(x)
- EN.
im 2.4.y
which further approximates éN (x)) from the given conditional. We used eq. below. We note that fx-Full above is derived from IPW, but can also be derived from OI, as shown appendix B

---

The estimator F--Full above relies on enumerating all possible values of (X.T,Y) making it compu- tationally expensive for sigh-dimensionll X. Below, we present two hybrid versions of our method which combine sampling of some variables and computation of conditional probabilities of others.
NATURAL IPW. To construct hybrid estimator, augment the data R& )== by sampling from (R4 independently for each report R, This gives us a dataset {{R,, J;) drawn i.i.d. from Then, our hybrid estimator is derived from the form of IPW follows:
P(X, R) by Assumption 4
TY (1 -1-TM- e(()
= ER.x
ET.YIR,
1 1- 1-e(x)
(5)
N-IPP Σ Σ
= = Ri
ty
(1-t)y
(6)
TI
(X,)
- 6N.IPW (Xi)
where (x)) is consistently estimated in the following manner:
êN.Pw(2) (x) E 1=1 P(T = 1 X.)I(XX ER.x P(T = X =x =e(x).
e(x).
1
ER,x [IX= -
(7)
NATURAL OI. Similarly inspired by the Ol estimator in equation 2 we have for {0, 13,
ET, [P(Y = X, = x, =x
P(Y = -
(8)
ER.X. [[(X = =x, - ()
Thus, for our hybrid Ol estimator, we augment the data { {R,),_, by sampling from P(X,T|R) independently for each report R; This gives a dataset {(R;, X,, T:) drawn i.d. from P(R, X, T) by Assumption Then, our consistent outcome predictor is given by
ox, E - P(Y = 1 Ra, X., Tx) x, T:
(9)
Σ I(X; x. =;
and the final estimator is given by:
No1 No,iXX, - 1) (X,
(10)
1=1
NATURAL Monte Carlo. Further in the direction of sampling more variables, we can obtain samples R2) and compute a Monte Carlo estimate, fN- The set of samples (x,,T,, Yt) from constitute tabular dataset which can be plugged into standard ATE estimator
{(X,, T, Y)), j
like IPW or Ol, as described in section 2 We refer to these sample-only estimators as N-MC IPW
and N-MC OI, respectively.
Inclusion Criteria conditioned ATE We are sometimes interested ATES over populations defined by constraints on pre-treatment covariates X,, known inclusion criteria and denoted by 7. This conditional ATE r rI) E[) (0) | E I] and satisfies the following identity.
P(X E 7(R) P(X E 1)
TY (1-THY
T(T) En
E EX,T,
X R 1- e())
X e
(11)
e(X)
The conditional version of NATURAL IPW can be estimated by filtering out reports where P(X E [[[2;) = O, sampling X1; ~ P(X Ri, X 1) i.i.d., and finally weighting datapoints by the relative likelihood of matching the inclusion criteria given the report:
P(X E ((R) E P(XETR)
m
TY (1-TYY T-e(X,,)
+(I) +()= Σ
ETYIX
(12)
e(Xi;)
where the inner expectation is estimated similar to eq. 6 complete derivation for eqs. and (12) and related discussion are included in appendix In practice we took = 1.
5

---

observational language data
initial covariate extraction
filtered by inclusion criteria
final covariate extraction
conditional distributions inferred
relevance filtered
tried drug and 11 helped. stngle
tried halpwet havan' had single
Is this irepart relaaannt to ithe problem !setting LLN PROMPT
Extract
Given these finclusion criteria
Which itretteen??
ithese
attributes
Whicc routcome?
ifrom this report
eextrcc
these...
LLM PROMPT
PROMPT
LLN PROMPT
Figure 2: Our pipeline leverages LLMs to curate data that can be plugged into natural language
conditioned estimators for average treatment effects.
4 Implementing NATURAL estimators with Large Language Models
LLMs are trained on vast datasets of real-world data, e.g., 3 which likely contain records of data generated by processes that are consistent with Assumption 3 Because LLMs can learn well- calibrated conditionals [22), our hypothesis is that LLMs be prompted to approximate the conditionals required by Assumption for real-world causal effect questions of interest. Our LLM implementation of NATURAL estimators is built on this hypothesis to try to satisfy Assumptions 3 and 4 (Assumptions 1 and 2 must be guaranteed by a domain expert). We defer exact prompts for LLM inference appendix D full worked example to appendix [ and discussion of the limitations of our approach to the next section. Figure summarizes our pipeline.
Filtering to match Assumption 3 For our real-data clinical settings, our first goal is to produce a dataset of i.i.d. reports R, that are very likely to be jointly distributed with the random variables (X,, T, T, Y)) of specific observational study of interest. Given a study of interest and a dataset of real-world reports that are potentially relevant to the study, we pass it through a sequence of filters with increasing detail and strictness:
Initial filter. Inspired by other work with social media data rules to filter out uninformative reports: posts that were removed, are too short, have "bot" in the author's s name, have no mention of any keyword related to the study, etc.
38
we first use deterministic
Filter by relevance. We prompt an LLM to determine whether each report contains information that would make relevant to the study. We remove reports that are deemed irrelevant. (iii Filter by treatment-outcome We ensure that each report pertains specifically to the treatments and outcomes of interest. We do so by prompting an L.LM extract only treatment and outcome information, and retaining only the posts that are deemed to both mention one of the treatments in question and also contain outcome information.
(ii)
(iv) Filter known covariates by inclusion criteria. For ATEs conditioned inclusion criteria, as in our real-world datasets, we included a filter to enforce these criteria. Managing inclusion criteria is complicated by the fact that many reports R, contain covariates that are required to verify inclusion. So, in this filtering step, our goal was to ensure that the final set of reports have non-zero probability of matching the inclusion criteria. We begin by prompting an LLM to extract the full set covariates X,, following constraints on the possible values each attribute can take, but allow the L.LM extract Unknown if is impossible for the LLM to determine the value of a covariate. We then remove reports, any of the non-Unknown covariates are determined to fail their inclusion criteria. We found the JSON-mode made available for generation by certain LLM APIS, to suffice for this task; however more involved strategies for constrained generation are also possible 46 50).
partial information about
Sampling from and computing conditional probabilities to match Assumption 4 Given a set of reports ) that pass the filtering stage above, next steps LLMs extract the samples and conditionals (X, T, R), required to compute NATURAL estimators. For each R, we:
(v) Extract covariates, both known and unknown. We run a final covariate extraction by prompting an LLM to determine the full set of covariates X, from the report RG subject to the constraint that X; satisfics the inclusion criteria. In contrast (iv) we ask the LLM to

---

KLPP1PT
10
P- P
RMSE(E m1)
10
101
2
reports
reports
Figure 3: For Hillstrom (left) and Retail Hero (right), the KL divergence between estimated joint and propensity distributions and their true counterparts reduces with increasing number of posts (top), as does the RMSE between the NATURAL Full estimate and true ATE (bottom).
guess the values of Unknown covariates. We verified that this second extraction agreed exactly with the first extraction (iv) on the known covariates (i.e., the ones that were not extracted as Unknown in the first extraction). We contrast the empirical distributions of these known and covariates for our experiments in appendix F.
unknown/guese
(vi) Infer conditionals. Given { RR, X, X; from the previous steps, we compute the probabilities PL = f, Y = y R. X₂)) by prompting an LLM that makes log peg--obbbbilities accessible. Specifically, we ask LLM to answer questions about T,Y Y given access X, and we score every possible answer T = t, Y y using the LLM log-probann wog--robabilities We exponen- tiate and renormalize these scores across the space of possible realizations to obtain valid probability distribution.
(vii) Weight reports according to inclusion criteria match. Similar to item (vi we compute PI (X; RR: to obtain the weights in eq. 12), by prompting LLM with descriptions of the inclusion criteria that must be satisfied and each report Rg It may possible to skip this weighting step under additional structural assumptions the data. These assumptions well as experimental results without the weighting are included in appendix G
Nevertheless, while our empirical results are remarkably consistent with the correctness of our pipeline, we cannot formally guarantee that it satisfies Assumptions and The final outcome of this pipeline is a dataset { R, X; 1 and set of conditionals P(T ft Y y R; X;) that can be plugged into the hybrid NATURAL estimators in section to predict ATEs. Therefore, we see this as first implementation of NATURAL estimators, which we anticipate can be improved.
3
Limitations and Broader Impact
In addition the limitations that NATURAL shares with every observational study, i.e., the validity of the practitioner's s causal assumptions, it comes with an extra dependence on how well one can approximate the desired conditional distributions. While more and more capable LLMs are being continually developed, the extent to which they satisfy NATURAL's 's assumptions is nearly impossible to formally test. Indeed, while pretraining tends to produce calibrated LLM predictions (22]. post- training techniques can compromise calibration 34 Therefore, we emphasize that NATURAL was not developed recommend therapeuties directly to end-users
to directly inform high-stakes public policies. Instead, we envision NATURAL as a powerful tool to help approximate ATES at scale and prioritize confirmatory CREs. We strongly recommend that all predictions made by NATURAL estimators be validated experimentally before being used to inform high-stakes decision- making. Apart from its dependence on LLM capabilities, NATURAL is also limited by the nature of observational, unstructured natural language data:
Network Interference. In practice, acquiring i.i.d. reports can be challenging. For instance, social network users might talk to each other and influence their treatment choices. This is well-known issue in causal inference and statistical sciences general. Existing solutions rely on a known network structure to sample individuals or correct for their neighbors' treatments [10 26 [7]. Outcome Measurement. Since NATURAL deals with self-reports, subjects need to be able to report the outcomes of interest. For example, this cannot be applied if the outcome is measured with an expensive, inaccessible test. Therefore, the study design implemented with NATURAL must account for the accessibility of endpoints to users.
7

---

Selection Bias. Results might be biased towards individuals' choice of reporting an outcome given their experience with the treatment. Luckily, outcome missingness a widely studied problem in causality research, see e.g., how to test [9 or how to mitigate [1 it. Note, however, that solutions will often accumulate assumptions on top of NATURAL and should always be critically evaluated by practitioners. Finally, apart from individuals" choice of reporting, selection bias might also arise from which individuals participate in online forums, i.e., our framework is only capable of estimating local ATEs external validity is not guaranteed priori. We demonstrate this challenge section by simulating systematic bias in semi-synthetic settings.
6 Empirical Evaluation
Evaluating an end-to-end pipeline for causal inference from unstructured real-world text data to ATEs presents challenges regarding access to data, ground truth ATEs and insightful intermediate metrics. We used two semi-synthetic datasets where we augmented randomized data to mimic real-world observations, while continuing to have access to ground truth evaluation. In addition, we study four real datasets, curated from publicly available Reddit posts from the PushShift dataset, as described in section 4 These six datasets allowed us to systematically evaluate NATURAL.
Semi-synthetic Datasets. Causal effect estimation is typically evalu- ated using synthetic datasets with one or more relationships between the ob- served covariates, treatment and out- come being contrived. We instead syn- thesized unstructured observational text data from real randomized tab- ular datasets, using an LLM. Specifi- cally, we (i) introduced confounding bias by sampling datapoints according to an artificial propensity score, (ii) randomly dropped covariates, (iii) de- scribed covariates, treatment and out- come in shuffled orderings, (iv) simu- lated realism by sampling persona from the the Big Five personality traits [27 for each datapoint and finally, (v) prompted the LLM to generate a realistic report describing the provided information in the style of someone with the given traits (see appendix D for the full prompt). We used two standard, publicly available randomized datasets: Hillstrom [9] and Retail Hero [49]. and plan to scripts to generate
Table I: The NATURAL IPW ATE outperforms other ver- sions of the method as well as trained baselines on semi- synthetic datasets, as measured by RMSE.
Hillstrom
Retail Hero
(%6
(%)
RMSE
Selection-hissed N-IPW
-3.49 + 1.46
9.58
10.66 + 2.24
7.67
Uncorvected
1.86
0,28
3.08
N-Fall
4.26
0.86
2.02
1.80 à: 1.38
2.08
N-MC OI
6.17 1.61
1.61
4.94 1: 2.17
2.70
N-MC IPW
1.85
2.01
N-OI
4.58
0.61
2.00
1.43
1.72
N-IPW
5.28 = 1.00
1.32
3.83 =: 1.29
1.39
Bag-uf- Wards
7.37
2.61
2.08
2.42
Sentence Encoder
0.00
0.00
6.09
1:97 =: 1.62
2.10
IPW (Structured)
6.38 0.26
3.09
0.30
Ground
6.09
10
8.32 [
data. Step (i) above is in similar vein as Keith et al. [24), in that our subsampling strategy does not modify the marginal distribution over covariates and the ATE remains identifiable from observational data.
Real-worke Datasets. To study how our framework may be deployed to test hypotheses using real data from online forums; we considered two medical conditions for which there exist abundant Reddit posts in the PushShift collection 77., with individuals' personal experiences: the effect of diabetes medications (e.g. Semaglutide) on weight loss and the tolerability of migraine treatments. For each condition, we picked two clinical trials which performed head-to-head comparisons of two treatments that we expected to find references to in relevant subreddits. Moreover, to mitigate selection bias we selected pairs of similar treatments, e.g., comparable availability, where we believe the probability of a user reporting their experience is approximately equal in both. As we will discuss, our results suggest external validity as well, meaning that the probability of user reporting their experience with the treatments seems to be (approximately) equal to the prior probability of user reporting any experience. We limited our data collection to posts that were written before December 2022 and made publicly available in the PushShift archives. We curated four datasets for comparison between different treatments, each of which has a ground truth RCT: Semaglutide vs. Tirzepatide [18] and Semaglutide vs. Liraglutide (8 for their effect on weight loss and Erenumab vs. Topiramate (37 and Onabotulinuntoxin. vs. Topiramate (39] for their tolerability. We used the first of these to validate implementation choices NATURAL (like filtering, imputations, prompt specifications) and
8

---

Table 2: Using real data, best performing NATURAL estimators fall within 3 percentage points of their corresponding ground truth clinical trial ATEs. Possible ATE values lie between - 100 and 100.
Tuned
out
Semaglutide Tizzpatide (5 weight loes 2 5%)
Semaglutide Liraglatide (% weight loss 2 10%)
Fremunsh Topiramate
Omabotalinamtostn. Topiramate
1% discontinned due to AE)
(% discuntineed due to AE)
ATE (%)
RMSE
ATE (%)
RMSE
AIE (%)
RMSE:
ATE (9)
RMSE
Uncorrected
333.56 0.77
43.67
83.37
0.43
88.87
29.07 +: 0.48
2.87
21.55
19.49
N-MCOI
5.80 # 1.03
4.28
-8.23 d: 0.94
6.54
25.62 # 0.51
2.72
46.20 =: 1.94
5.56
N-MC IPW
5.02 + 0.85
4.81
7..10 + 0.94
T.66
26.65 + 1.44
3.19
46.02
1.02
0.86
N-OI
+:
0.38
-6.37
1.06
2:16
1.77
1.92
3.39
N-IPW
1.26
-12.54
2.33
25.64
2.68
42.53
2.07
2.57
Ground Truth
10.11 NCC00967919,
18
-14.7
[NCT03191396.
28.3
NNC106225599
XY]
41.00 (NC702191579 1V
the other three as held-out test settings, see appendix C We include further details for all our datasets in appendix E. including the definitions of covariates and outcomes.
Next, we investigate several questions about the performance of NATURAL empiricaly. We used GPT-4 Turbo for sampling and LLAMA2-70B for computing conditional probabilities.
How well does NATURAL estimate observational distributions from self-reported data? Our semi-synthetic datasets give access the true joint distributions P(X, Y) and true propensity scores P(T = 1 X The top row of fig. 3 shows the KL divergence between these distributions and those estimated by NATURAL Full, for Hillstrom (left) and Retail Hero (right). We find that these KL divergences decrease steadily as the number of reports used in the estimation increases. The bottom row shows corresponding root-mean- qquares error (RMSE) between NATURAL and the true ATE. This corroborates the insight that as the joint distribution and propensity scores are estimated more accurately, the predicted ATE gets closer its true value. In particular, we observe a clear correlation between the quality of estimated propensity scores and estimated ATEs.
How do NATURAL methods compare to one another and to trained baselines? We present estimated ATE and its RMSE semi-synthetic datasets in table T. Further, we evaluate two trained baselines, which use a Bag-of- Words model and a sentence encoder respectively, to train representations of text data with their labels. Here, for each attribute in the set of covariates, treatments, and outcomes, we train a MLP model with 5-fold cross validation to predict that attribute. We then use these predicted attributes as tabular dataset of samples that be plugged into any causal inference estimator. We find that our methods are competitive with or outperform these baselines, despite not being trained with any labels. In particular, the sentence encoder baseline collapsed an ATE of zero for Hillstrom, having learned the constant predictor for outcomes. IPW (Structured) is an oracle estimator, which assumes full access structured data. Selection- N-IPW demonstrates the challenge of ATE estimation in the presence of bias which was systematically simulated as function of covariates "channel" and "zip code" " for Hillstrom and "age" for Retail Hero.
Table 2 compares NATURAL methods to estimate the ATES in real-world clinical settings using self-reported data from the PushShift collection of Reddit posts. Remarkably, our predicted ATES (a) depict the same direction of effect, and (b) fall within 3 percentage points of their corresponding ground truth clinical trial ATEs. For both semi-synthetic and real data experiments, NATURAL IPW outperforms other versions across datasets, except for the Semaglutide vs. Liraglutide setting, where NATURAL OI performed the best. Both N-MC versions perform similarly on all datasets.
This result is significant. Clinical trials can take on the order of years and costs in the tens to hundreds of millions of dollars. Going from the raw language observational data ATE in our framework takes on the order of days and costs at most a few hundred dollars of compute. For problems in medicine, economics, sociology, and political science where randomization is infeasible expensive, NATURAL provides a tractable way to leverage observational data to rank potential experiments prior to conducting them.
How do different choices in the NATURAL pipeline effect ATE prediction? We assess the impact of key choices in our pipeline described section 4 by ablating them one-by-one. We investigated and selected these choices on the Semaglutide vs. Tirzepatide experiment. Fig- ure (left) compares the RMSE of predicted ATES when data is not filtered according to in- clusion criteria and LLM imputations are replaced with samples from uniform distribution.

---

12.5
RMSE(A.r")
RMSE(
7.5
5.0
Unilern
fhaiing- + L.M impatsiuns
ELAMAZ-78
2
Inclusion weighting Unitorn imputations
2.5
ELAMAA-13B
weighting LLM imporsions
2
22
2
200
reports
reports
Figure 4: Ablation study on Semaglutide vs. Tirzepatide, to tease apart the effect of data filtering and imputation (left) well LLM scale for conditionals (right) NATURAL performance.
shows that both inclusion-based filtering and imputations from pretrained LLM are cru- cial for the performance of NATURAL. We also compared performance of our method when the conditional probabilities in eq. (6) are evaluated using models of different scales in fig. (right), and found that performance improves at larger scales and with greater quantity of data.
duration days stan weight
Uniform
How well do different estimates of propensity score balance covari- ates? A property of accurate propen- sity score estimates is that they bal- ance covariates across treatment co- horts (see Ding [TT] for details and proofs), ie. the average treatment effect on each covariate, corrected using propensity scores, is close to zero. Figure [ visualizes this quan- tity for different covariates of the Semaglutide vs. Tirzepatide experi- ment and shows that propensity scores estimated using LLAMA2 conditional distributions balance the covariates far better than a uniform distribution does, with the 70B model consistently estimating the treatment effect each covariate as close to zero. Similar visualizations for the test settings are shown in fig. 10 of appendix F2
ILAMMA-718
LLAMA2-13B
Covariates
start HbAlc cc
LLAMA2-70R
bmii
set
age
-1.0
-1.25
11.00
-0.75
-0.50
-0.25
0.00
0.25
Bulance across treatment groups
Figure 5: NATURAL propensity scores balance the Semaglu- tide vs. Tirzepatide covariates better than uniform scores.
Conclusion
In this work, we introduced NATURAL, family of text-conditioned estimators. to automate treatment effect estimation using free-form text data. We demonstrated NATURAL's efficacy with six semi- synthetic and real datasets for systematic evaluation of its pipeline. We exposed the ability of LLMs to extract meaningful conditional distributions over structured variables and how their combination with classical causal estimators can predict real-world causal effects with remarkable accuracy. Given this promising performance, exciting directions for future work include (i) incorporating automatic prompt tuning methods into the pipeline, (ii) extending our methods to real-valued (X, T, Y), (iii) exploring whether our assumptions can be weakened, (iv) exploring other domains in applied research, e.g., social sciences, (v) performing a more extensive evaluation of NATURAL on different study designs to better understand what type of treatments, outcomes, and reports show better worse practical performance with NATURAL or (vi) deploying the pipeline test hypotheses at even larger scales.
NATURAL estimators have numerous use cases with potentially far-reaching impact. As long as patients have access treatments and report their experiences, NATURAL can be used compare two treatments in new indications or new populations. Therefore, our pipeline can in principle support efforts prioritize trials for repurposed drugs or supplements in ander-served diseases or populations. Further, a crucial step after drug approval is post- marketinn surveillance for side effects (positive or negative) that may not have been measured or may have been too rare to identify in a smaller trial. NATURAL can leverage the diversity of available language data to detect these effects. While our motivations largely stem from the challenges of drug development, our NATURAL estimators are applicable to any effect estimation setting for which there exists relevant natural language data.
10

---

Acknowledgements
Our work was directly inspired by work done by Noah MacCallum, George Hosu, Sina Hartung. and Zain Memon at Eureka Health. Their project, Social Treatment Insights, explored the use of LLMs with social media data draw medical insights. We thank them for the spark that led this project and for the suggestion to study weight loss treatments. We would also like to thank Dexter Ju for useful practical suggestions on filtering social medial posts, Amol Verma and Fahad Razak for pointers migraine-related clinical keywords, and David Lopez-Paz, Patrick Forré, Roger Grosse and Sheldon Huang for feedback on an initial draft of the paper. Resources used in preparing this research were provided in part by the Province of Ontario. the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), CGPPN-2021-03445.
References
[1] Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kul- shreshtha, A., Nemade, G., Lu, Y., et al. (2020). Towards a human-like open-domain chatbot. arXiv preprint 1XXiv:001100977.
[2] Ahrens, M., Ashwin, J., Calliess, J.-P., and Nguyen, V. (2021). Bayesian topic regression for causal inference. arXiv preprint 10Xiv:2/09.05317
[3] AI Meta (2024). Llama 3 model card.
[4] Antonucci, A., Piqué, G., and Zaffalon, M. (2023). Zero-shot causal graph extrapolation from text via llms. arXiv preprint arXiv:2312,146700
[5] Arsenyan, V. and Shahnazaryan, D. (2023). Large language models for biomedical causal graph construction. arXiv preprint arXiv:2301.12247
[6] Ban, T., Chen, L., Wang, X., and Chen, H. (2023). From query tools causal architects: Harnessing large language models for advanced causal discovery from data. arXiv preprint 2tt.:2330..99000
[7] Baumgartner, J., Zannettou, S., Keegan, B., Squire, M., and Blackburn, (2020). The pushshift reddit dataset. In Proceedings of the international AAAI conference on web and social media, volume 14, pages 830-839.
[8] Capehor,, M., Catarig, A.-M., Furberg. J., Janez, A., Price, H., Tadayon, S., Vergès, B., and Marre, M. (2020). Efficacy and safety of weekly semaglutide 1.0 mg once-daily liraglutide 1.2 mg as add-on to 1-3 oral antidiabetic drugs in subjects with type 2 diabetes (sustain 10). Diabetes & metabolism, 46(2):100-10)
[9] Chen, J. M., Malinsky, D., and Bhattacharya, R. (2023). Causal inference with outcome- dependent missingness and self-censoring. In Evans, R. and Shpitser, I., editors, Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, volume 216 Proceedings of Machine Learning Research, pages
PMLR.
[10] Cotta, L., Bevilacqua, B., Ahmed, N., and Ribeiro, B. (2023). Causal lifting and link prediction. Proceedings of the Royal Society A, 479(2276)20230121.
[11] Ding, P. (2023). A first course in causal inference. arXiv preprint arXiv:2305.18793
[12] Dunning, T. (2012). Natural Experiments in the Social Sciences: A Design-Based Approach. Strategies for Social Inquiry. Cambridge University Press.
[13] Egami, N., Fong, C. J., Grimmer, J., Roberts, M. E., and Stewart, B. M. (2022). How make causal inferences using texts. Science Advances, 844))ea
[14] Falavarjani, S. M., Hosseini, H., Noorian, Z., and Bagheri, E. (2017). Estimating the effect of exercising on users' online behavior. In Proceedings of the International AAAI Conference on Web and Social Media, volume 11, pages 734-738.

---

[15] Feder, A., Keith, K. A., Manzoor, E., Pryzant, R., Sridhar, D., Wood-Doughhs Z., Eisenstein, J., Grimmer, J., Reichart, R., Roberts. M. E., et al. (2022). Causal inference natural language processing: Estimation, prediction, interpretation and beyond. Transactions of the Association for Computational Linguistics, 10: 138--1588.
[16] Flury, B. K. and Riedwyl, H. (1986). Standard distance in univariate and multivariate analysis. The American Statistician, 40(3):249-251.
[17] Forastiere, L., Airoldi, E. M., and Mcalli, F. (2021). Identification and estimation of treatment and interference effects in observational studies on networks. Journal of the American Statistical Association, 116(53)):011-11.
[18] Frías, J. P., Davies, M. J., Rosenstock, J., Pérez Manghi, F. C., Fernández Landó, L., Bergman, B. K., Liu, B., Cui, X., and Brown, K. (2021). Tirzepatide versus semaglutide weekly in patients with type 2 diabetes. New England Journal of Medicine, 385(6):503-515.
[19] Hillstrom, K. (2008). The MineThatDData E-Mail Analytics And Data Mining Challenge. Challenge Dataset.
[20] Jin, Z., Liu, Lyu, Z., Poff, S., Sachan, M., Mihalcea, R., Diab, M., and Schölkopf, B. (2023). Can large language models infer causation from correlation? arXiv preprint arXiv:2306.058: 05836.
[21] Jiralerspong, T., Chen, X., More, Y., Shah, V., and Bengio, Y. (2024). Efficient causal graph discovery using large language models. arXiv preprint arXiv:2402.01207.
[22] Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-
Dodds, Z., DasSarma, N., Tran-Johnson, E., Johnston, S., El-Showk, S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D., Hernandez, D., Jacobson, J., Kernion, J., Kravec, S., Lovitt, L., Ndousse, K., Olsson, C., Ringer, S., Amodei, D., Brown, T., Clark, J., Joseph, N., Mann, B., McCandlish, S., Olah, C., and Kaplan, J. (2022). Language models (mostly) know what they know.
[23] Keith, K., Jensen, D., and O'Connor, B. (2020). Text and causal inference: review of using text to remove confounding from causal estimates. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J., editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online. Association for Computational Linguistics.
[24] Keith, K. A., Feldman, S., Jurgens, D., Bragg, J., and Bhattacharya, R. (2023). Ret rejection sampling for causal estimation evaluation. arXiv preprint arXiv:2307.1517.
[25] KIcıman, E., Ness, R., Sharma, A., and Tan, C. (2023). Causal reasoning and large language models: Opening a new frontier for causality. arXiv preprint a0X0002305.00000.
[26] Leung, M. P. (2022). Causal inference under approximate neighborhood interference. Econo- metrica, 90(1):267-293.
[27] Lim, A. (2023). Big five personality traits:
The 5-factor model of personality. simply psychology.
[28] Long, S., Schuster, T.,, Piché, A., de Montreal, U., Research, S.,, al. (2023). Can large language models build causal graphs? arXiv preprint NXXX::2003.02299
[29] Lu, S. and Ding. P. (2023). Flexible sensitivity analysis for causal inference in observational studies subject to unmeasured confounding. arXiv preprint arXiv:230S.176A
[30]
J., Sussex. J., and Towse, A. (2012). The R&D cost of a new medicine.
[31] Miao, W., Liu, L., Tchetgen, E. T., and Geng, Z. (2015). Identification, doubly robust estimation, and semiparametric efficiency theory of nonignorable missing data with shadow variable. arXiv preprint arXiv: 1.509.0255.
[32] Mueller, S. and Pearl, J. (2023). Personalized decision making-a conceptual introduction. Journal of Causal Inference, 11(1):2022050.
12

---

[33] Naik, N., Khandelwal, A., Joshi, M., Atre, M., Wright, H., Kannan, K., Hill, S., Mamidipudi, G., Srinivasa, G., Bifulco, C., et al. (2023). Applying large language models for causal structure learning in non small cell lung cancer. arXiv preprint arXiv:231
[34] OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L.., Akkaya, I., Aleman, F. L., Almeida,
D., Altenschmidt, J.. Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapiro, G., Berner, C.. Bogdonooff. L.. Boiko, O.. Boyd. M., Brakman, A.-L., Brockman, G., Brooks, T., Brundage. M., Button, K., Cai. T., Campbell, R., Cann, A., Carey, B., Carlson, C., Carmichael, R., Chan, B., Chang, C., Chantzis, F., Chen, D., Chen, S., Chen, R., Chen, J., Chen, M., Chess, B., Cho, C., Chu, C., Chung. H. W., Cummings, D., Currier, J., Dai, Y., Decareaux, C., Degry, T. Deutsch, N., Deville, D.. Dhar, A., Dohan, D., Dowling, S., Dunning, S., Ecoffet, A., Eleti, A., Elounou,, T., Farhi, D., Fedus, L., Felix, N., Fishman, S. P., Forte, J., Fulford, I., Gao, L.. Georges, E., Gibson, C., Goel, V., Gogineni, T., Goh, G., Gontijo-Lopes, R., Gordon, J., Grafstein, M., Gray, S., Greene, R., Gross, J., Gu, S. S., Guo, Y., Hallacy, C., Han, J., Harris, J., He, Y., Heaton, M., Heidecke, J., Hesse, C., Hickey, A., Hickey, W., Hoeschele, P., Houghton, B., Hsu, K., Hu, S., Hu, X., Huizinga, J., Jain, S., Jain, S., Jang. J., Jiang, A., Jiang, R., Jin, H., Jin, D.,, Jomoto, S., Jonn, B., Jun, H., Kaftan, T., Lukasz Kaiser, Kamali, A., Kanitscheider, I., Keskar, N. S., Khan, T., Kilpatrick, L., Kim, J. W. Kim, C., Kim, Y., Kirchner, J. H., Kiros, J., Knight, M. Kokotajlo, D., Lukasz Kondraciuk, Kondrich, A., Konstantinidis, A., Kosic, K., Krueger, G., Kuo, V., Lampe, M.. Lan, I., Lee, T., Leike, J., Leung, J., Levy. D., Li, C. M.. Lim, R., Lin, M., Lin, S., Litwin, M., Lopez, T., Lowe, R., Lue, P. Makanju, A., Malfacini, K., Manning, S., Markov, T., Markovski, Y., Martin, B., Mayer, K., Mayne, A., McGrew, B., McKinney, S. M., McLeavey, C., McMillan, P., McNeil, J., Medina, D., Mehta, A., Menick, J.. Metz, L.,, Mishchenko, A., Mishkin, P., Monaco, V., Morikawa, F.,, Mossing, D., Mu, T., Murati, M., Murk O., Mély, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo, R., Noh, H., Ouyang. I.., O' Keefe, C., Pachocki, J.,, Paino, A., Palermo, J.. Pantuliamo, A.. Parascandolo. G., Parish, J., Parparita, E., Passos, A., Pavlov, M., Peng. A., Perelman, A., de Avila Belbute Peres, F., Petrov, M., de Oliveira Pinto, H. P., Michael, Pokorny, Pokrass, M., Pong, V. H., Powell, T., Power, A., Power, B., Prochl, E., Puri, R., Radford, A., Rae, J.. Ramesh, A., Raymond, C., Real, F., Rimbach, K., Ross, C., Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M., Sanders, T., Santurkar, S., Sastry, G., Schmidt, H., Schnurr, D., Schulman, J., Selsam, D., Sheppard, K., Sherbakov, T., Shich, J., Shoker, S., Shyam, P., Sidor, S., Sigler, E., Simens, M., Sitkin, J., Slama, K., Sohl, I., Sokolowsky, B., Song, Y., Staudacher, N., Such, F. P., Summers, N., Sutskever, I., Tang, J., Tezak, N., Thompson, M. B., Tillet, P., Tootoonchian, A., Tseng, E., Tuggle, P., Turley, N., Tworek, J., Uribe, J. F. C., Vallone, A., Vijayvergiya, A., Voss, C., Wainwright, C., Wang, J. J., Wang. A., Wang, B., Ward, J., Wei, J., Weinmann, C., Welihinda, A., Welinder, P., Weng, J.. Weng. L.. Wiethoff, M., Willner, D., Winter, C., Wolrich, S., Wong, H., Workman, L., Wu, S., Wu, J., Wu, M., Xiao, K., Xu, T., Yoo, S., Yu, K., Yuan, Q. Zaremba, W., Zellers, R., Zhang, C., Zhang, M., Zhao, S., Zheng, T., Zhuang, J., Zhuk, W. and Zoph. B. (2024). Gpt-4 technical report.
[35] Pearl, J. (2023). Judea pearl, ai, and causality: What role do statisticians play? [Online; accessed May-2024).
[36] Pryzant, R., Card, D., Jurafsky, D., Veitch, V., and Sridhar, D. (2020). Causal effects of linguistic properties. arXiv preprint arXiv:2010. 129/9.
[37] Reuter, U., Ehrlich, M., Gendolla, A., Heinze, A., Klatt, J., Wen, S., Hours-Zesiger, P., Nickisch, J.. Sieder, C., Hentschke, C., et al. (2022). Erenumab versus topiramate for the prevention of migraine-a randomised, double-blind, active-controlled phase 4 trial. Cephalalgia, 188
[38] Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M.. Liu, Y., Xu, J., Ott, M., Shuster, K. Smith, E. M., al. (2020). Recipes for building an open-domain chatbot. arXiv preprint arXiv:2004.13637
[39] Rothrock, J. F., Adams, A. M., Lipton, R. B., Silberstein, S. D., Jo, E., Zhao, X., Blumenfeld, A. M., and Group, F. S. I. (2019). Forward study: evaluating the comparative effectiveness of onabotulinuntoxina and topiramate for headache prevention in adults with chronic migraine. Headache: The Journal of Head and Face Pain, 59(10)
[40] Schurman, B. (2019). The framework for fda's real-world evidence program.
13

---

[41] Sheldrick, R. C. (2023). Randomized trials vs real-world evidence: how can both inform pecision-making? Jama, 329(16):1352-1353
[42] Sridhar, D. and Blei, D. (2022). Causal inference from text: A commentary. Science advances.
[43] Sridhar, D. and Getoor, L. (2019). Estimating causal effects of tone in online debates. arXiv preprint 196.047906.
[44] Tu, R., Ma, C., and Zhang, C. (2023). Causal-discovery performance of chatgpt in the context of neuropathic pain diagnosis. arXiv preprint wrxii:2301.13812
[45] Vashishtha, A., Reddy, A. G., Kumar, A., Bachu, S., Balasubramanian, V. N. and Sharma, A. (2023). Causal inference using Ilm-guided discovery. arXiv preprint arXiv:2310.5511..
[46] Willard, B. T. and Louf, R. (2023). Efficient guided generation for Ilms. arXiv preprint arXiv:2307.09702.
[47] Willig, M., Zecevic, M., Dhami, D. S., and Kersting, K. (2023). Causal parrots: Large language models may talk causality but are not causal. preprint, 8.
[48] Winship, C. and Morgan, S. L. (1999). The estimation of causal effects from observational data. Annual review of sociology, 25(1 (659-7666
[49] X5 (2019). X5 Retail Hero: Uplift Modeling for Promotional Campaign. Challenge Dataset.
[50] Zheng, L., Yin, L., Xie, Z., Huang, J., Sun, C., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. (2023), Efficiently programming large language models using sglang.
14

---

Appendix
A Notation
R
Random variable corresponding to unstructured natural language text from a social media post (or report).
X
Random variable corresponding to features of an individual in causal inference dataset.
T
Random variable corresponding treatment intervention assigned to an individ- ual in a causal inference dataset.
Y
Random variable corresponding to outcome observed for an individual in causal inference dataset.
Possible instance of X from its support X.
1
Possible instance of T from its support T = (0, 1} (binary treatments).
y
Possible instance of Y from its support {0, 1} (binary outcomes).
r
Possible instance R from its support R.
((t)
Random variable corresponding to potential outcome observed for an individual after receiving treatment
e(X)
Propensity score function for binary treatments, equal to P(T = 1[X).
X;
Sampled value of X for individual i.
T;
Sampled value of T for individual i.
Y
Sampled value of for individual i.
R
Sampled report R for individual i.
Average treatment effect (ATE) given by (1) Y()),, where the expectation is over some defined population of individuals.
Total number of individuals.
711
Total number of individuals that assigned treatment 7 1.
710
Total number of individuals that are assigned treatment 7
15

---

B Deriving NATURAL Full from Outcome Imputation estimator
In section we derived the idcalized version of our method, NATURAL Full using the form of the IPW estimator. In fact, any unbiased estimator would lead to same form eq. (3), which can then be used derive NATURAL Full using reports R and the law of iterated expectations. Here, we show this using the Outcome Imputation estimator. For discrete X and binary T and Y, we have
= -= - -
- 1
(outcome imputation)
(13)
(expectation of discrete X)
SEX
(14)
P(X=x))(T=1|X+ PY=11,==1,==0 -, x)
= Σ P(X= 8)
(expanding conditionals)
x:)
SEX
(15)
P(Y = 1,,=
P(Y
0, (-
(definition of e(x))
c(z)
ZEN
(16)
=ΣΣ, ZEN VEX
P(Y = V. -
P(Y = v, x = x
(since y = {0, 1))
((x)
e(x)
(17)
P(Y -t,== -1
P(Y y, t, z)
=EE
(since TT 10,
c(c)
c(x)
xEX NEY IET
(18)
ty
(1-
Σ (2,1,)
P(Y = y. y, x)
(rearranging terms)
[e(x)
1 - e(x)
(19)
TY
(1 (1-
EXTY
(20)
e(X)
e(X)
which is equivalent eq.
16

---

C Worked Example: Semaglutide vs. Tirzepatide
To make the NATURAL pipeline and its implementation more concrete, we now work through an end-to-end example using the Semaglutide vs. Tirzepatide dataset. We used this setting to develop our evaluation setup and constructed the pipeline as a function of the experiment design. While is infeasible exhaustively describe the entire decision space, major decisions that were found to impact ATE prediction are (i) filtering strategy, (ii) prompt tuning for extraction, and (iii) covariates discretization. The strategies we tried are included in the pipeline description below, with the specific decisions made for each highlighted in this color.
Pipeline for Semaglutide vs. Tirzepatide. Given the experimental design of the clinical trial in Frías et al. [88] (NCT03987919), we defined our experiment follows:
1. Treatments:
Semaglutide and Tirzepatide
2. Outcome: Percentage of participants who lost 5% more of their initial weight
3. Covariates: Age, Sex, BMI, Start weight, Start HbAlc, Duration
4. Inclusion criteria:
(a) The user must be diagnosed with Type 2 Diabetes with starting HbAlc between 7% and
10.5%.
(b) They must already be on a regime of the treatment called Metformin.
(c) They must have BMI of 25 kg/m² or more.
(d) Since different treatment dosages can have varying effects, we also included dosage as an inclusion criterion, i.e. we aimed include only posts that reported taking Img for Semaglutide and 5mg for Tirzepatide, as in the clinical trial.
Decisions in the pipeline that are direct function of an experimental design like the one above highlighted in this color in the following description.
Next, we implemented the entire pipeline as follows:
1. Initial filter. We identified nine subreddits relevant this problem setting: r/Mounjaro, r/Ozem- pic, r/fasting, r/intemiteenffattnngg, r/keto, r/loseit, r/Semaglutide, r/SuperMorbidlyybeer r/- PlusSize. From each subreddit, we downloaded all submissions and comments posted upto December 2022 from the PushShift collection, so as to only use publicly available data. This resulted in a dataset of 577,733 submissions and comments. An initial deterministic, task- agnostic and rule-based filter removed any submission or comment if its content was not string, if it had no score, the content was " "[deleted] " or [[reoove]]" ", if it was a comment with fewer than ten space-separated strings (presumably, words), if the author's name contained the string "bot", if there were no spaces in the first 2048 characters, and if less than 50% of all characters were alphabetic. This reduced the dataset size to 380,276. We then formatted this data into dictionary-like datapoints with fields: subreddit, title, date created, post/comment ments written by the author as replies to their own post may contain additional relevant in- formation when combined with with original post and other replics. We then passed these through a task-dependent drring-maching filters. For this dataset in particular, we listed strings used commonly to refer the treatments, ["ozempic", "mounjaro", "semaglutide "tirzepatide", "wegovy", "rybelsus", "zepbound"], included common misspellings generated with GPT-4 and Perplexity] and filtered datapoints that did not contain any of these strings. Similarly, we listed keywords relevant to the outcome of inter- est, ["kg", "kilo", "Ib", "pound", "weigh", "drop", "loss", "lost", "gain", "hb", "aic", "hemoglobin", "haemoglobin", "glucose", "sugar"] and filtered out datapoints that did not contain any of these strings. This filtered dataset now contained 50,654 datapoints.
author replies. We indcluded the last field because com-
Filter by relevance. Next, we wrote a problem setting description and prompted GPT-3.5 Turbo to determine whether the posts, along with auxiliary information from the formatted dictionaries described above, were relevant the described setting. The description and instructions for this particular dataset are shown in prompt [ We manually labeled handful of datapoints as Yes or No and included these as incontext examples to improve the LLM's generations. We removed datapoints that were deemed irrelevant, resulting in a "relevant" dataset of 21,229 datapoints. Filter by treatment-outeomee, To further filter the data to points that refer specifically to the treatments and outcome of interest, we prompted GPT-3.5- Turbo to extract only information
17

---

required to ascertain the treatment and outcome, as shown in prompt Since the outcome for this dataset, achievement of target weight loss of 5% or more, may reported in several ways, we attempted to cover all those possibilities. Specifically, we prompted the LLM to extract the user's starting weight, end weight, change in weight and percentage of change in weight. Several combinations of these attributes allow us to programmatically infer the final outcome. We also extracted the units in which weight was reported, converting all extractions to be in lbs. We filtered out any datapoint for which the extracted treatment was not one of the treatments considered for this task or for which it was not possible to infer the outcome using the above-mentioned extracted information. This finally gave us natural language dataset of 4619 relevant reports, each of which contained treatment and outcome information pertaining to the defined problem setting. 4. Filter known covariates by inclusion criteria. To evaluate against the clinical trial, we further filtered the dataset according to its inclusion criteria. At a high level, different strategies for this filtering trade-off how strictly we match the criteria with how many datapoints remain in the filtered set. Striet filtering to match every criteria exactly resulted in very few reports. Less strict filtering to match subset of the criteria resulted in predicted ATEs that varied depending on which criteria we chose to satisfy, with no apparent task-agnostic principle to determine crucial criteria. Finally, we used instructions shown in prompt 4 extract covariates and aimed for set of reports with non-zero probability of satisfying these criteria. As described in item (iv) of section f and motivated in appendix G we extracted all covariates including ones related to the inclusion criteria and removed datapoints whose extractions were not Unknown and failed to satisfy the criteria above. This resulted in dataset of 1265 reports.
5. Extract known and unknown covariates. Treating these 1265 reports as the final dataset from which to estimate an ATE, we extracted the set of covariates given in our experiment definition. We also included the duration of treatment as covariate since this information is often reported and is likely to influence the outcome. This extraction step was conditioned on inclusion criteria being satisfied, description of which was included in the extraction prompt, as in prompt 5 We tuned the prompts for extracting attributes, which include general instructions for the task and specific questions for each attribute. This was done by inspecting handful of reports and corresponding extractions and then modifying prompts to correct any observed errors.
6. Infer conditionals. We inferred conditional distributions from LLAMA2-700 70B for different ver- sions of NATURAL, with the strategy described in item (vi) of section 4 and LLM inputs of the form shown in prompt Here, "conditioning on covariates" was implemented by adding questions about the covariates and their sampled answers to the input. For instance, for sex, the question "What is the reported sex of the user?" was followed by its previously extracted answer (Male or Female). The scoring strategy required enumerating possible op- tions for treatments and outcomes for each input, which were ("Semaglutide like Ozempic or Wegovy or Rybelsus", "Tirzepatide like Mounjaro or Zepbound"] and ["No", "Yes"], respectively.
7. Weight reports according to inclusion criteria match. We also used LLAMA2- 70B to compute the weighting terms described in item (vii)of of section 4 Coneretely, we constructed prompt, like prompt 7 describing the inclusion criteria listed in item 4of the experiment design above, followed by a report, R. and an instruction asking the LLM to determine whether all the described criteria are met. We then scored the possible answers, ["No", "Yes"], exponentiated and renormalized them to obtain P(X E I R.). We marginalized over reports to compute the denominator, P(X E 7), in the weight. The contribution of each report to the ATE estimates was weighted by this relative likelihood matching the in inclusion criteria of the experiment given the report. 8. Finally, given all the required extractions and conditional probabilities, we required discrete covariates to plug them into our NATURAL estimators. Hence, we converted any continuous covariates into discrete categories. These categories for each dataset are shown in table for all our datasets. Different choices of discretization led to slightly diffcrent ATE predictions. We found it most helpful to discretize continuous numerical covariates into intervals such that the number of datapoints were roughly balanced across intervals. This avoided covariate strata with too many or too few datapoints and resulted in ATE predictions from all NATURAL estimators that were sufficiently close to the ground truth.
Adapting the pipeline to test trials. The decisions above in this color directly or indirectly influenced the ATE and we made our choices with access to the ground truth for Semaglutide vs. Tirzepatide. Hence, we call this a "tuned" setting. We fixed these decisions for the three test settings. Note that no other aspect of the pipeline depends the ATE, All the choices in this color are
18

---

function of the experimental design. Hence, the pipeline can be easily adapted to any new setting, given its experimental design and without knowledge of the ATE.
D LLM Prompts
Prompt 1: Semi-synthetic report generation (Hillstrom)
You user who used website for online purchases in the past year and want share your background and experience with the purchases social media.
## Attributes
The following attributes that you have, along with their descriptions. (features)
## Personality Traits
The following dictionary describes your personality with levels (High or Low) of the Big Five personality traits.
(traits)
## Your Instructions
Write social media post in first-person. accurately describing the information provided. Write this post in the tone and style of with the given personality traits, without simply listing them. Only return the post that you can broadcast on social media and nothing more.
Post
19

---

Prompt Relevance filtering (Weight Loss)
You are an expert researcher looking around reddit for posta/ comments describing the effect of a treatment on weight loss or blood sugar level experienced by the author.
Problem Setting
You are interested self relf-reported effects treatment on user who took the treatment themselves. You want to be able to ansver some or all of the following questions from the text of the post comment:
1. Which treatment did the user take?
2. What change did they observe in their weight due this treatment, and during what duration did they observe this change?
3. What change did they observe in their blood sugar, aka HbA1c levels, due to this treatment, and during what duration did they observe this change?
4. What other attributes they report, e.g. age, sex, country of residence, diabetes diagnosis, other treatments they have tried, or side effects?
Your Instructions
will show you post or comment, contextual information about Based on the given problem setting and contextual information, you need to judge whether it is relevant to the problem setting described above or not. Answer Yes if the post is relevant and No otherwise; nothing else. Here few examples:
{incontext examples}
Subreddit
This post was found on the subreddit r/(subreddit).
## Title
This post titled: {title)
Date Created
This post was created on (date_created).
## Post {post)
The author also replied with the following in the thread:
> {replies}
Answer Yes comment is relevant and otherwise, nothing more. ## Your Answer
20

---

Prompt Treatment-outeome filtering (Weight Loss)
You are a medical assistant, helping a doctor structure posta about weight loss treatments found on Reddit. Your task is to use the self-repors to interpret accurate information about the following fields and store them in JSON dictionary.
Your Instructions
I will provide post along with its subreddit name, title and date of creation. You must return valid JSON dictionary containing the following keys along with the corresponding accurate information: "start_weight weightt; Numerical value for the user's starting weight, before starting the treatment described, sometimes referred to as SW.
"end_weight": Numerical value for the user's current or final weight, at the end of the treatment regime, sometimes referred to CW. "weight_unit", unit" Units which weight is reported: "kg" "Ib". "weightt_change". Numerical value for net change the user's weight. Use postive sign to indicate weight gain and negative sign for weight loss. Leave blank if it is not possible to infer the change in weight. serrrcettte_wwight_ttttteee
Numerical value for percentage reduction in user's weight relative to their start weight. Use postive sign to indicate weight gain and negative sign for weight loss. Leave blank if it is not possible to infer the percentage.
"drug_type" Treatment taken by the user: "Semaglutide", "Tirzepatide" "Other". Semaglutide includes Ozempic, Wegovy Rybelaus. Tirzepatide includes Mounjaro or Zepbound.
Assign valid value to each key above. If you can't find the required information in the post, assign the value "Unknown". Remember ONLY return valid JSON with of above keys and their accurate values.
Prompt 4: Covariate extraction (Weight Loss)
medical assistant aiding physician, your role involves examining Reddit posts discussing weight loss treatments and interpreting self-reported information accurately. This data needs to be translated into well-structured JSON dictionary, with the most suitable option chosen from the choices provided.
Your Instructions
Assume user shares a post along with related data. Your job will be to create dictionary comprising of the following keys well their matching accurate data:
(covariate descriptions)
Please ensure you fill all the fields and that you choose valid value for each key from the provided options. Unfilled fields instances where certainty impossible, make your best educated guess, or provide the "Unknown" value. Note that your completed task should ONLY yield a JSON containing ALL the listed keys alongside their accurate values.
allowed. In
Here are few examples:
(incontext examples}
Input (report}
Output
21

---

Prompt Covariate imputation (Weight Loss)
You are a medical assistant tasked with creating a profile a patient who is taking a weight loss treatment, and presenting it as a JSON dictionary with prespecified keys. Fill in suitable values for ALL the information provided about the patient.
keys. You
Your Instructions
A patient has Type 2 Diabetes, is knoun to have taken Metformin for the last 3 months and has BMI greater than 25 kg per meter squared. Dosage for Semaglutide, Ozempic, Wegovy and Rybelaus is img. Dosage for Tirzepatide, Mounjaro and Zepbound is 5mg.
Create possible profile for this patient with the following fields and represent it as dictionary:
{covariate descriptions)
Please ensure you fill all the fields with a valid value. Unfilled fields or values like "Unknown" are not alloved. Note that your completed task should ONLY yield JSON containing ALL the listed keys alongside their accurate values.
Here is an entry that the patient wrote about themselves, which be useful for your task.
Input {report}
## Output
Prompt 6: Conditional distribution inference (Weight Loss)
You medical assistant aiding physician. I going to ask you few multiple choice questions about some posts just found online. Please, answer accordingly.
## Your Instructions
I will give you post about individual's experience with treatment and ita effect their weight, and few questions with their correct answers, folloved by additional multiple choice questions and options choose from. Pick the right answer.
## Social Media Post (report}
## Questions and their correct answers
Q: (question about covariate X1} A: {X1 sample).
{question about covariate X2] (X2 sample}.
## Questions
Which treatment did the user take? Options: a) {to} b) {t1}
{to}
Q: Did the user lose 5 more percent of their initial weight? Options: a) {yo) b) {y1)
[yo}
22

---

Prompt Inclusion weights (Weight Loss)
You are a medical assistant aiding a physician. Based on the following social media post about an individual's experience with a diabetes treatment and its effect on their weight, evaluate whether the person meets ALL of the following criteria:
1. Type 2 Diabetes: Diagnosed with Type Diabetes and has an HbAlc between
7\% and 1011.
2. Metformin: Has been taking Metformin for at least the past 3 months.
3. BMI: Has BMI greater than 25 kg/m"2.
Medication Dosage: If taking Semaglutide (e.g., Ozempic, Negovy, Rybelsus), the dosage is img; OR if taking Tirzepatide (e.g., Mounjaro, Zepbound), the dosage is 5mg.
After analyzing the poat, determine whether the individual meets ALL of the above criteria.
Social Media Post (report)
Question
Q: Does the user satisfy the given inclusion criteria? Options: a) No b) Yes
A: (No/Yes}
23

---

E Dataset Details
We provide further details about the treatments, outcomes and covariates, along with inclusion criteria and discrete categories used in our experiments, for each dataset in tables 3 and
Table 3: Treatments, outcomes and synthetic confounders (where applicable) for each dataset.
Dataset
Treatment
Outcome
Synthetic confounder
Hillstrom
email communication
website visit
newbie
Retail Hero
SMS communication
purchase
age
Semaglutide vs. Tirzepatide
corresponding drug
weight loss weight loss of 10% more
or more
Semagfutide vs. Liraglutide
corresponding drug
NA
Erenumab vs. Topiramate
corresponding drug
discontinuation due to adverse effects discontinuation due to adverse effects
NA
Onabotulinumtoxink Topiramate
corsesponding drug.
NA
Table 4: Covariate descriptions, corresponding discrete categories and inclusion criteria enforced for each dataset. Intervals for continuous numerical variables were determined from the extracted values such that each discrete category is roughly balanced in terms of its number of datapoints.
Covariate
Description
Discrrtt categories
Inclusion criteria
Hillstrom
ercemey
nimber of morths since last purchase
[I 4,5 6---. 8,9 12]
histury
dollar value of previous parchase purchase men's merchandise puschise
100, 100 200..... > 1000)
mens
|Tuee [Tlue
merchandise type of residence now customer channel used for purchases
NA
sid_
Suburba aree Rural Urean arca] [Tru: Falsc]
nowhic
chainel
[Phose Web. Malichhllnn
Retail Hero
arg. purchase
arg: purchase value per tnasaction
263, 396, 611.> 612)
avp: prodicc quanitty
avg. munieer of products boughe arg. number of points received total number of trmoactions so for age of user
7.>
erg. points received
15 5. 51
NA
num transactions
[s 6,9 - 15, 16 - 27, 28]
16 45. 45)
Semaglutide vs. Tirzeuatide
age
age of user sex of user body mass index of user initial glycated haemoglobin value weight lbs number days treatment was taken for
[S 45, > 45]
sxx
[Male.eemne]]]
(2 diabete- True)
bmi
1. 5. 28.51
(7 start HhAle < 10.59)
Ibbiec
[x 7.5.> 7.5] 2211,
& (metiomin-=Trum True)
(bmi 25)
dunsion (days)
[ 90, > 901
Semaghutide vs. Liraglutide
of body mass
45. 45]
[omale
diahetes-True) (7 S stut HhAle 11)
28.5]
start weight
start Hhvlic
initial glyested lnsemoglhhin volue inital weight in lbs number of days treatment was taken for
[S 220. > 220)
[S 7
& (meffominlothrr- Tru))
dumatoon (days)
[ 120, > 120)
Erenumah vs. Topiramate
age
ago of sour ses of user country of residence
32, > 32]
se%
[Male.Femlll]
(18 S age S 65)
country
[United States, Canada...)
(pregnntt=Tahee
baselline MMD
initial number of monthly migraine days number days treatment taken for
[% 6,> 6] 30, 30)
(buseline MMD > 4)
duration (daps)
Onabutalinomtoonnk vs. Topicumute
age
age of user ses of user country residence
[ 25, > 25] [MaaleFeemale]] (United States. Canada...) 15. 15) 30, 30)
se%
(18 < age (5) & (bsseline MMD 15)
baselise MMD
initiel nundorr montlly migmine number days treaiment taken for
duration (days)
24

---

Further Experimental Results
F.1
Known and Unknown/Imputed covariates for real data experiments
We refer the reader to figs. to for empirical distributions of covariates extracted by an LLM in its first extraction as well as those imputed in its second imputataion conditioned on inclusion criteria.
100
100
0.2
505
Figure 6: Distributions of "known" (top) vs. "unknown" and imputed (bottom) covariates for Semaglu- tide Tirzepatide.
100
3.50
0.35
0.50
3.75
1.50
60
2600
405
55
130
153
200
Figure 7: Distributions of "known" (top) "unknown" and imputed (bottom) covariates for Semaglu- tide vs. Liraglutide.
F.2 Balancing property of propensity scores
We refer the reader to Figure 10 for visualizations of the propensity score corrected average treatment effect covariates for all test clinical settings. For each setting, our estimated propensity score balances each covariate, far better than a uniform propensity distribution would.
Since covariates may take values different scales, computed the standard mean difference (SMD) across cohorts for each covariate X (t) [16), given by:
(1)1)-X())
SMD
(21)
VO.5 * (var (1) var
where (1) (0) estimates the average treatment effect on (6), using propensity score weighting, and var(-) denotes sample variance.
25

---

11
4600
5100
8300
courtry
Sundin ddys
Figure Distributions of "known" (top) vs "unknown" and imputed (bottom) covariates for Erenumab Vs. Topiramate.
HEMerrOn wwwurcaddoo tore auree
20
6.0
0.2
6.6
0.5
3.0
1.0
200
100
100
Figure 9: Distributions of "known" (top) vs "unknown" and imputed (bottom) covariates for Onabo- tulinumtoxinA vs. Topiramate.
Tunncios Balt
Hersim bay
Cocca tendia
uan wine
Navtine MaN0L
LLMMA:
groge
(a) Semaglutide vs. Liraglutide
(b) Erenumab vs. Topiramate
(c) OnabottlinnmtoxxinA vs. Topira-
mate
Figure 10: Propensity scores estimated with LLAMA2-70B balance covariates of the real clinical datasets, far better than uniform scores do.
E.3 Sensitivity Analysis
While we rely on domain expertise to define the confounder set for each setting such that necessary causal assumptions are satisfied, unobserved confoundedness remains challenge. We followed strategies in Lu and Ding (29] to analyze the sensitivity of our ATE estimates to the degree of unobserved confoundedness, Specifically, the idea is introduce sensitivity parameters,
(o) = X] and (x) = -
1
(x(X)
E (
26

---

which quantify the degree of unobserved confound- edness. In our case, sensitivity parameters are the density ratio between likelihood of each potential outcome in the treated vs. untreated group. The ATE estimate is non-increasing in the sensitivity pa- rameters. So, for positive ATEs, we are looking for
Table 5: NATURAL IPW for Semaglutide vs. Tirzepatide is robust to the degree of unob- served confoundedness shown below.
1,00
1.05
1.10
1.15
1.20
1.25
-2.88
the largest sensitivity parameters that maintain the
1.05
8.51
5.58
2,91
0.47
-1.76
-3.82
1.10
7.58
1.97
-0.46
+2.70
-477
positivity of the ATE, which tells us the degree of
1.04
unobserved confoundedness that estimator is ro- bust to. Table 5 shows that the direction (sign) of NATURAL IPW ATE estimates for the Semaglutide Tirzepatide dataset change from positive to negative at large values of sensitivity parameters, implying that they are robust to large degrees of unobserved confoundedness.
1.20
5.71
2.77
0.10
-2.33
-4.57
-6.62
1.25
4.77
1.84
-0.83
-3.27
-5.50
-7.56
We further investigated the importance of each confounder as suggested by the leave-one-covariate out approach in Section 4 of Lu and Ding [29]. by dropping that covariate as if is an unobserved confounder and measuring the corresponding worst-case (over all possible values of the remaining covariates) sensitivity parameters with the remaining covariates. Figure IT shows contour lines depicting NATURAL IPW ATE estimates at different values of sensitivity parameters for all our real-world datasets. Orange regions denote sensitivity parameter values for which the sign of the estimate ATE does not change, while blue regions denote the values that flip the direction of the estimate. Stars show sensitivity parameters for each covariate in the different settings. Hence, this estimator is sensitive to the covariate set and all covariates are important in the case of Semaglutide vs. Tirzepatide, while most important for Erenumab not sensitive to covariates the case of Semaglutide vs. Liraglutide and OnabotulinumtoxinA vs. Topiramate, implying that the estimated causal effects could only be explained away by an unobserved confounder that is stronger than all observed confounders.
Topiramate. The direction of ATE estimates
11-
1.2
200
Semaglutide vs. Tirzepatide
(b) Semaglutide vs. Liraglutide
(c) Erenumab vs. Topiramate
(d) OnabotulinumtoxinA vs. Topiramate
Figure 11: NATURAL IPW ATE estimates for different values of sensitivity parameters in each real setting. The orange (blue) region corresponds to ATES with the same (flipped) sign our estimates.
27

---

G Inclusion Criteria conditioned Estimator
We are interested an ATE conditioned on inclusion criteria denoted I,
T(I) E - (0) X E 1}.
(22)
Let (X, be function such that
-()) Ex,r, (x,,x((XXTTT)) X eT.
(23)
For example, rI] can estimated by the IPW estimator,
TY
11-T)Y
7XX,T,Y)
e(X)
1- eX))
because the P(T 1X = 2, x 1 P(T 1|x x) for all E J. Throughout this
section, we operate under Assumptions 3 and and assume that the LLM gives us access to the true data-generating conditionals.
The law of total expectation gives us an estimator that can operate on samples of reports Rt:
-()) = (x,,, [XE]
ERIXEL (x,TyI XET,
P(RR = E [EExT(((XXTT)) XEI,
DP(R= - P(X 1(
Ex.,yl((x,T,)) X E ET, rr)
P(XEI)
[P (I))) ExTxI((X,T.)) XEI.R En P(XEI)
To summarize, we have the identities:
T()- = ExxeI (xxxeEExxxxxxxXX E
(24)
[(X E IRR)
= ER P(X (xX,TT(XX,TX)) X E 1, R
(25)
We prompted LLLAM2--70B with descriptions of the inclusion criteria and each report to estimate P(X,EJJ P(X.E7) similar to other conditional distributions described in section 4 and marginalized over reports for the denominator.
is also possible to avoid this weight computation at the expense of additional structural assumptions on the data. We discuss these conditions below and show corresponding results for NATURAL estimators in table
Let X RD and let us make the following assumption on the inclusion criteria:
Assumption (Inclusion eriteria specification) The inclusion criterion I defines a box, i.e., is specified separately for each covariate dimension I d E (1....,) and the set of covariates satisfying every inclusion criteria is given by the product of individual criteria over the covariate
dimensions, ie., {X 1 = 1X E [d] where X is the d-th dimension of X = (X4)
Recall from section 4 that inclusion-basee based filtering leaves us with reports whose covariates are either "known" and satisfy their criteria or Unknown. We also have the value of the known covariates. Let K (0, P be the binary vector of variables Kd that indicate whether the covariate X is found be "known" for a random report R. Let X (xd Kd 1) be the vector of length Σ, Kd holding the values of the known covariates. For ease of notation, define the event that the known covariates satisfy their criteria and the event that the unknown covariates satisfy their criteria:
{X* E Tk} = (x" E 7d Yd: Kd = =1} (x1-K E -- E ld vd: Kd = 0}
(26)
(27)
28

---

Notice that fx*=x***0 E E Ex--x]
[X E 1}. Thus, after the filtering steps we have (Ri, Ki )
(28)
with the guarantee that the knowns satisfy their inclusion criteria, {x Ki E [k.). Formally, assuming that the LLM computes the true conditional distribution of the data data- generatingg process (Assumption
this gives us data sampled i.i.d. from P(R = т. K = X = x* x E I*). Note, that we are assuming the existence of an additional ground-truth random variable process that describes whether covariate is knowable from report. Here, we show how to estimate rT) from this dataset of filtered reports using importance sampling, under the following assumption:
the Sata-generating
Assumption (Satislaction of by Unknown covariates) Satisfaction of inclusion criteria by un- known covariates is conditionally independent of the report and the known covariates given satisfac- tion of inclusion criteria by known covariates, i.e,, for all r, k x²:
1,, x xx,''_KE1'-K = P(R= = r, k. X² =x E 1
(29)
One can derive the following identity in a similar fashion as eq.
25
P(R, K, X (XE)) P(R, K, XK XK E (K)
(() XERKEE
BxTYF X K,
(30)
From assumption 6 the fraction above simplifies leaving with the following estimator
1 Σ Ex.,, Ex.,,xI7(X,, T. X, E E.E,I,R K.
(1)
(31)
which can be computed from the information available at the end of filtering. In practice, we do not condition the L.M on K; in the final inference step [vi) which amounts additional conditional independence assumption:
Assumption (Conditional independence of knowable covariates) K (T,Y) (X,
Equation 31 above can now be more efficiently estimated by prompting the L.M to extract covariates under the constraints of the inclusion criteria for each report our filtered dataset, and then following the remaining steps in the pipeline to an ATE estimate.
Table 6: ATE estimates on real datapoints that are filtered according inclusion criteria but not weighted by the relative likelihood that they meet the inclusion criteria of experiment given the report. Best performing NATURAL estimators fall within 3 percentage points of their corresponding ground truth clinical trial ATEs, Possible ATE values lic between -00 and 100.
Tuned
Held-out
Semaglutide Tirzepatiee (% weight loss 2 5%)
Semaglutide Liraglutide (% weight loss 2 10%)
Erenumah Topiramate
Omabotulllammtoztn Topirnmate
(% discontineed due AE)
(5 discontineed duc to AF)
ATE (%)
(%)
(%)
ATE ATB(S%
Uncerreeted
33.56 0.77
43.67
83.57
0.43
68.87
20.07 # 0.48
2.87
21.55
1.22
19.49
N-MC Of
5.43 + 1.01
4.79
-7.71
0.91
7.05
23.91 + 1.63
4.68
46.21
#: 1.99
5.56
N-MC IPW
5.23 4: 0.93
4.97
-7.43
0.93
7.33
25.29
4: 1.72
3.47
46.23
1.03
5.57
N-OI
4.36 1. 2.05
6.09
-15.90
1.14
1.65
31.21 1:
3.38
44.91
1.46
4.17
N-IPW
Ground Truth
10.11
NCC009679199 13
144.7
NCC003991966
28.3
NCT03628539
17
41,00
1NC702191579
10
29

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how answer these questions. For each question in the checklist:
You should answer [Yes]. [No] [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant
information is Not Available.
Please provide a short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No]" ", it perfectly acceptable to answer "[No] providedd proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We substantiate our claims through empirical evidence in section aand explicitly state the assumptions under which they are expected to hold in section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: We discuss the limitations and broader impacts of our work in detail in section We further state the assumptions made by our work in section 3
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the
paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
30

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't I acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: While this is not theoretical paper, we provide the full set of assumptions we operate under and method derivations in section 3 with strategies to satisfy these assumptions in section 4 and additional details and derivations about more nuanced steps of our method in appendix G
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Experimental details to reproduce our datasets and empirical results are provided in O with further details like prompts used to query language models in appendix D
section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that appropriate the research performed.
31

---

While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]
Justification: We plan to release code for reproducing all experimental results, along with scripts for producing each dataset, with the final version of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
code..
Please see the NeurIPS code and data submission guidelines (htttps:/nips.cc//p))) cc/public/
Puiides/CodeSubmin sww.sssiiconnc for more details.
While encourage the release of code and data, understand that this might not be possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: pottps://nitt //nips cc/pulicc/ quides/CodeSun saaubmissiooPoccyyy for more details.
The authors should provide instructions data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended. but including URLs data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details for datasets, tuning of the method and impact of different choices are discussed in section 6 with additional details throughout the appendix..
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, Experiment Statistical Significance
supplemental material.
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report standard error for all our main results in tables and
Guidelines:
The answer NA means that the paper does not include experiments.
32

---

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical sigmificance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given
experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors). It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experi- ments?
Answer: [Yes]
Justification: We use pretrained models for inference only and discuss the efficiency considerations for different versions of our method section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud
provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We conform to all aspects of the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there a special consideration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We dedicate section to the limitations and broader impacts of our work in section 5
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied to par- ticular applications, let alone deployments. However, if there a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We do not release any large models. We do however plan to release scripts to reproduce our dataset, ensuring they rely on publicly available data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all original owners of code, data, and model used in this work.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of
that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should
be provided. For popular datasets. papprswii laperats. thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's
creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
34

---

Researchers should communicate the details of the dataset/code/moodl as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human
subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Sub- jects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human
subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
35


Reviews:
[{"review_id": "3AJtwZPIbe", "reviewer": "Reviewer_oMqj", "summary": "The paper seeks to use LLMs for a workflow that extracts structured variables\nfrom free text, filters the dataset, and then computes a causal estimate using\nthe imputed variables. The paper applies this methodology to two synthetic and\nfour real-world datasets and shows that it produces estimates that are\ncomparable to those from known ground truth or RCT estimates.", "questions": "Line 267-268: What is the point of sampling a persona from Big Five personality\ntraits? How does this simulate realism?\n\nLine 147-149 says \"if all reports are the constant, empty string ... we have\nfull access to the true observational joint density.\" Am I correct that this is\na necessary assumption for the overall method to work in theory, but the actual\nimplementation (steps (v) and (vi) in lines 206-220) would not work if all\nreports are the constant, empty string? This may be obvious, but it could be\nworth clarifying that when you say \"we cannot formally guarantee that [our\nmethod] satisfies Assumptions 3 and 4\" you mean the form of Assumption 4 that\ndoes not involve all reports being empty strings.\n\nIn Tables 1 and 2, which LLMs are used? LLAMA2-70B? Overall, it would be\nhelpful to more clearly label which LLMs are used when. In Line 204, the paper\nsays that the GPT-4 API sufficed for filtering; was GPT-4 used anywhere else?\nHow was the decision made of which LLM to use where?\n\nIt might be helpful to have a figure or appendix that walks through every step\n(the paper does this for many steps, just not all in one place) in which an\nLLM is queried and discusses what assumptions are being made.", "limitations": "Overall, the paper does a good job of not overclaiming. However, see weakness 1\nabove. I would also suggest the authors discuss the challenges of using RCT\nestimates to validate observational studies -- there are many reasons why those\nestimates might not align (selection bias, dropout, etc).", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This is a very ambitious paper that pulls together a complicated workflow for\nestimating causal effects from real-world natural language datasets. Just\nshowing that this can be done is a significant contribution.\n\nThe paper is clearly written and the main points are easy to follow.\n\nWhile I am somewhat skeptical of any practical applicability of this work, the\npaper is reasonable conservative in highlighting its limitations.", "weaknesses": "The paper at almost every turn relies on prompting an LLM and implicitly\nassuming that the model returns data from a desired distribution. In general,\nthis will introduce measurement error which could systematically bias the\noverall method's estimates. See for example [1] and [2] below.\n\nThe synthetic data evaluation is fairly limited. It would be nice to at least\nunderstand how performance varies as sample size increases from a few hundred\nexamples to many thousands (especially as the IPW oracle baselines are not\nparticularly good). This would also allow you to stress-test the different\nassumptions being made. If you have synthetic data in which reports for certain\npatients are systematically truncated or censored, or where different groups\nwrite reports in different languages or styles, how does that affect the\ndifferent steps (i-vi)?\n\n[1] Ashwin, Julian, Aditya Chhabra, and Vijayendra Rao. \"Using Large Language\nModels for Qualitative Analysis can Introduce Serious Bias.\" Development\nResearch (2023).\n\n[2] Egami, Naoki, et al. \"Using imperfect surrogates for downstream inference:\nDesign-based supervised learning for social science applications of large\nlanguage models.\" Advances in Neural Information Processing Systems 36\n(2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and positive feedback! Below, we address the questions in your review and describe corresponding changes to improve our paper.\n\n> (W1) The paper at almost every turn relies on prompting an LLM and implicitly assuming that the model returns data from a desired distribution. In general, this will introduce measurement error which could systematically bias the overall method's estimates.\n\nWe are able to measure this error in synthetic settings, which we quantify as KL divergence between the true and predicted distributions and visualize in Figure 3. With increasing amounts of data, this KL divergence reduces, as does the ATE error.\nNevertheless, we agree that measurement error due to an LLM’s output distribution is a potential limitation of our work, as described in Section 5. We discussed selection and other possible biases in more detail in Appendix E of our submission due to space constraints. We will include this discussion in the main paper in the final version. \n\n> (W2) The synthetic data evaluation is fairly limited. It would be nice to at least understand how performance varies as sample size increases from a few hundred examples to many thousands (especially as the IPW oracle baselines are not particularly good). This would also allow you to stress-test the different assumptions being made. If you have synthetic data in which reports for certain patients are systematically truncated or censored, or where different groups write reports in different languages or styles, how does that affect the different steps (i-vi)?\n\n- We would like to clarify that the oracle estimates in Table 1 are denoted by “IPW (Structured)”. We will update the discussion of results in Section 6 to clarify this.\n- We constructed synthetic datasets of 2000 reports each and found these IPW (Structured) oracle estimates to be very close to the ground truth ATE, as sample size was increased to 1024 or higher. If the reviewer thinks it is important, we are happy to generate more synthetic data and show performance at higher sample sizes.\n- Simulating systematic biases in synthetic settings is a great suggestion and important future work! For example, one could simulate reporting bias by subsampling the data based on a subset of the covariates. We ran this during the rebuttal period for the Hillstrom dataset using two covariates (channel and zip code). Indeed this results in a NATURAL IPW ATE of $-3.49$ which has a large error of $9.58$. We will include this result in Table 1 as an example of NATURAL’s current limitation in handling reporting bias, which we hope to address in future work. \n- We are also open to further discussion on how to investigate data that is biased in other ways, such as being “truncated”, if the reviewer has concrete suggestions.\n\n> (Q1) What is the point of sampling a persona from Big Five personality traits? How does this simulate realism?\n\nThe Big Five personality traits (Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism) helped simulate diversity in writing style, tone and report lengths, and hence made the synthetic data more realistic. \nWe observed the generated reports were diverse in their verbosity, style and tone, as one might expect in real-world online forums.\n\n> (Q2) Am I correct that this is a necessary assumption for the overall method to work in theory, but the actual implementation (steps (v) and (vi) in lines 206-220) would not work if all reports are the constant, empty string?\n\nYour understanding is correct! If the reports are all empty strings, then Assumption 4 requires that the LLM is able to simulate trial outcomes unconditionally (without any data). This would be a very strong assumption that is likely to be violated. We will clarify this in Section 3 of the paper. In practice, we filter our data to look for the most informative reports such that they contain some information about $(X,T,Y)$, as described in Section 4.\n\n> (Q3) In Tables 1 and 2, which LLMs are used? LLAMA2-70B? Overall, it would be helpful to more clearly label which LLMs are used when. In Line 204, the paper says that the GPT-4 API sufficed for filtering; was GPT-4 used anywhere else? \n\nThank you for pointing this out! We used GPT-3.5 Turbo for filtering steps, GPT-4 Turbo for sampling and LLAMA2-70B for computing conditional probabilities, for all datasets. We will explicitly state these details in Section 6.\n\n> (Q4) How was the decision made of which LLM to use where?\n\n- We tried different LLMs and chose the best performing ones available at the time, by examining a handful of examples at different points in the pipeline. We also took into account the costs associated with these models. \n- The filtering steps were relatively easy tasks, but were executed on larger amounts of data; hence we opted for the cheaper GPT-3.5 Turbo model. The sampling steps were tasks involving more reasoning and were executed on smaller datasets that had already been filtered; hence we opted for the best performing model at the time: GPT-4 Turbo. The LLM conditional probabilities required access to log-probabilities, and we opted for LLAMA2-70B as it was a top open-source model providing this functionality. An ablation on the scale of the model used for this step is included in Figure 4 (right).\n\n> (Q5) It might be helpful to have a figure or appendix that walks through every step (the paper does this for many steps, just not all in one place) in which an LLM is queried and discusses what assumptions are being made.\n\nThis is a great suggestion; thank you! We will add a section in the appendix that walks through a complete worked example of executing our pipeline with the Semaglutide vs. Tirzepatide dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WWcwg94A3I", "reviewer": "Reviewer_7S1Z", "summary": "Estimating causal effects is costly and time-consuming. The authors propose  to use large language models (LLMs) to mine unstructured text data for causal effect estimation. This paper introduces NATURAL, a family of causal effect estimators using LLMs to process unstructured text data. This seems to be a good application of LLMs for the task of causality.", "questions": "-  How do the authors make sure that all of the confounders are accounted ? I can imagine there to be many confounders mentioned in the text.\n\n- There could be a bias in social media text, I thought people would report negative cases more often than positive, did you observe any such biases?", "limitations": "Mentioned", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-  Determine the effects of treatments can be expensive and time-consuming so the paper introduces a novel task of effect estimation using unstructured natural language data.\n\n- This method is adaptable to various domains where textual data is abundant.\n\n- This seems to be a more practical application of LLM for the task of causality.", "weaknesses": "- LLMs might inherit biases present in their training data or hallucinate. \n\n - Social media users might not reflect the demographics or behaviors of the broader population.\n\n- Finetuning the model to one domain may not lead to good generalisation in others, during fine-tuning does the LLM learn inductive biases regarding the \"topic\" or task?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful review! Here, we will clarify and address the points brought up in your review.\n\n> (W1) LLMs might inherit biases present in their training data or hallucinate.\n\nAs described in Section 5, this is an important limitation in the use of LLMs, in general as well as in the case of NATURAL. In our experiments, we mitigate errors or hallucinations via prompt tuning. We believe NATURAL will benefit from progress in the field of research with LLMs, but the specific challenge is outside the scope of our work.\n\n> (W2) Social media users might not reflect the demographics or behaviors of the broader population.\n\n- It is true that there exists selection bias in the data on social media forums and this is an important limitation that we’re hoping to explore in future work. We discuss this and other biases in Appendix E of the submission and plan to expand Section 5 with this discussion in the final version of our paper.\n- Related to this is the treatment of inclusion criteria that define the population over which we are interested in estimating ATEs. In Appendix G of the paper, we made certain structural assumptions that allowed us to impute unobserved covariates restricted to the inclusion criteria. An alternate approach that we also discussed in Appendix G is to weight the estimated potential outcomes for each datapoint by the relative likelihood that they meet the inclusion criteria of the experiment given the report: $\\frac{P(X \\in I | R)}{P(X \\in I)}$ where $I$ defines the inclusion criteria. We revisited this alternate approach during the rebuttal period and explored using the LLM to estimate $P(X_i \\in I | R_i)$ and renormalizing to estimate  $\\frac{P(X_i \\in I | R_i)}{P(X_i \\in I)}$. These weights can be estimated with an LLM similar to other conditional distributions described in the paper and involves averaging over reports for the denominator. \n- We show these new results in Table 2 of the attached PDF, which are close to our original results. We believe this alternate weighting approach is a simpler, more intuitive way to understand the treatment of inclusion criteria that also removes the need for additional assumptions. We will include both original and new results in the final draft.\n\n> (W3) Finetuning the model to one domain may not lead to good generalisation in others, during fine-tuning does the LLM learn inductive biases regarding the \"topic\" or task?\n\nTo clarify our use of LLMs, none of our methods require any fine-tuning. Instead we use LLMs purely for inference. We will clarify this point in Section 4 of the paper.\n\n> (Q1) How do the authors make sure that all of the confounders are accounted ? I can imagine there to be many confounders mentioned in the text.\n\n- We rely on domain expertise to define the confounders for each setting such that necessary causal assumptions are satisfied. In our case, we used baseline characteristics from real-world RCT specifications which are designed by experts. Together with Assumption 4, which guarantees access to the conditional distribution $P(T = t, Y = y, X = x|R = r)$, these assumptions ensure that our estimators are consistent in theory.\n- Intuitively, we break this problem into two parts: (1) defining the confounder set $X$ such that Assumption 1 (Strong Ignorability or Unconfoundedness) is satisfied, and (2) correctly sampling these $X$ from $R$, which is guaranteed by Assumption 4 (in theory) and implemented with the help of LLMs (in practice).\n- Still, we think you bring up a really important point. So, we conducted a sensitivity analysis for the Semaglutide vs. Tirzepatide dataset. Specifically, we follow the sensitivity analysis strategy of Lu and Ding (2023). Briefly, the idea is to introduce sensitivity parameters ($\\varepsilon_0(X) = \\frac{\\mathbb{E}[Y(0) | T=1,X]}{\\mathbb{E}[Y(0) | T=0,X]}$ and $\\varepsilon_1(X) = \\frac{\\mathbb{E}[Y(1) | T=1,X]}{\\mathbb{E}[Y(1) | T=0,X]}$) that quantify the degree of unobserved confoundedness. In our case, sensitivity parameters are the density ratio between the likelihood of each potential outcome in the treated vs. untreated group. The ATE estimate is non-increasing in the sensitivity parameters. So, for positive ATEs, we are looking for the largest sensitivity parameters that maintain the positivity of the ATE, which tells us the degree of unobserved confoundedness that an estimator is robust to. Table 1 of the attached PDF shows that the estimated ATE changes from positive to negative at large values of sensitivity parameters, which means that it is robust to large degrees of unobserved confoundedness.\n- We further investigate the importance of each confounder as suggested by Section 4 of Lu and Ding (2023), by dropping that covariate as if it is an unobserved confounder and measuring the corresponding sensitivity parameters with the remaining covariates. Figure 1 of the attached PDF follows Figure 1 of Lu and Ding (2023) and shows that in the worst-case (over all possible values of the remaining covariates), our estimator is sensitive to the covariate set considered and each covariate is important to ATE estimation.   \n- We believe this sensitivity analysis significantly improves our paper and will include it in the final version of our paper. Thanks for your feedback!\n\n[1] Lu, S., & Ding, P. (2023). Flexible sensitivity analysis for causal inference in observational studies subject to unmeasured confounding. arXiv preprint arXiv:2305.17643.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2H9J8JWqMO", "reviewer": "Reviewer_oSFK", "summary": "The paper introduces a family of causal effect estimators named NATURAL, designed to use LLMs for mining causal effect estimates from observational text data. The authors address the challenge of automating data curation and using LLMs to impute missing information, presenting a method that conditions on structured variables to assist in computation of causal effect estimators. The authors develop six datasets (two synthetic and four real) to evaluate their method, and demonstrate that NATURAL estimators produce causal effect estimates within close range of ground truth from randomized trials.", "questions": "1. Authors explain that as part of their process, they “filter for reports that are likely to conform to the experimental design”. Could this introduce bias?\n\n2. All of the “classical strategies” used assume no-hidden confounding. Could you imagine how someone would use an LLM with a more complex identification strategy (say IV, or RDD)? You claim at the end of the intro that you can anticipate someone doing so, I wonder if you can elaborate.", "limitations": "The authors have addressed the limitations of their work, acknowledging the dependency on the quality and calibration of LLMs and the potential inaccuracies in approximating the required conditional distributions. They emphasize that NATURAL estimators should not be used for high-stakes decision-making without experimental validation and stress the need for domain expert involvement to ensure causal assumptions are met.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Clear and thoughtful use of LLMs to estimate causal effects from text data. The authors present a cool methodology, properly formalized, for using LLMs to extract and impute structured variables from observational text data. \n\n2. Development and evaluation of six datasets, including real-world clinical trial data.\n\n3. Strong results showing causal effect estimates are within a close range of ground truth values.", "weaknesses": "1. The original claims made in the abstract+intro seems to be too grandiose, but in fact it is about the use of LLMs to classify variables of interest from the text. I would change the framing of the paper a bit to better reflect that. It would make the claims of the paper less objectionable and would better demonstrate the usefulness of NATURAL.\n\n2. There is a vast literature in the social sciences on recovering interpretable variables directly from text for estimating causal effects from observational data. Most of these approaches rely on probabilistic approaches like topic models, but recently there’s growing interest in doing so automatically with LLMs. The authors should better address this literature, and compare their approach to relevant methods if possible.\n\n3. The approach is really cool, but like any ML-pipeline approach it surely introduces errors at each step of the process (see for example Egami et al @ NeurIPS 2023). Each error will then bias the causal effect estimation. From the results of the paper I’m convinced empirically your approach works well, but would I be able to understand whether it produces accurate estimates on a new dataset? A deeper analysis and perhaps a way to propagate error/uncertainty to the estimation would be very helpful here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for such a positive and encouraging review! Below, we address all the questions raised in your review.\n\n> (W1) The original claims made in the abstract+intro seems to be too grandiose, but in fact it is about the use of LLMs to classify variables of interest from the text. I would change the framing of the paper a bit to better reflect that. It would make the claims of the paper less objectionable and would better demonstrate the usefulness of NATURAL.\n\nThank you for this feedback! The two take-homes that we tried to emphasize in the abstract and introduction were (1) unstructured text data can be used as a *sole* source of rich causal effect information and (2) LLMs can be used to extract this information. We also believe it is important to clearly distinguish our data access setting from the classical settings of causal inference (see response to W2 below). Still, we’re more than happy to make edits to get the tone right, if the reviewer can make some specific suggestions or point out specific sentences that are objectionable. \n\n> (W2) There is a vast literature in the social sciences on recovering interpretable variables directly from text for estimating causal effects from observational data. Most of these approaches rely on probabilistic approaches like topic models, but recently there’s growing interest in doing so automatically with LLMs. The authors should better address this literature, and compare their approach to relevant methods if possible.\n\n- Thank you for pointing out this section of the literature! We would greatly appreciate any specific references from the reviewer. To reiterate the setting considered for NATURAL, we look at data sources that *only* contain natural language. This contrasts our setting from works like Falavarjani et al. (2017), which rely on a combination of text and numerical or tabular data. The ability to use only text data is important and motivated by settings where tabular data is unavailable or incomplete, e.g. neglected diseases, unrecorded abortions in certain countries, or illicit drug use. To our knowledge, such a setting with text as the sole source of information has not been considered yet. \n- Another example of related work that we found is Ahrens et al. (2021), which estimates latent topics or variables from text, and is relevant to the broad problem we consider. NATURAL is distinct in two ways. First, similar to above, Ahrens et al. rely on tabulated numerical outcomes as well as text data, while NATURAL operates using only text data. Second, NATURAL relies on domain expertise to provide a study design and in particular, define the treatment, outcome and covariate variables to be extracted from text, whereas Ahrens et al. discover these variables in the text. Extending NATURAL to remove dependence on domain expertise may be interesting future work, but would require further assumptions on LLM capabilities. \n- We will include the above citations and discussion in the final version of our paper.\n\n[1] Falavarjani, S. M., Hosseini, H., Noorian, Z., & Bagheri, E. (2017, May). Estimating the effect of exercising on users’ online behavior. In Proceedings of the International AAAI Conference on Web and Social Media (Vol. 11, No. 1, pp. 734-738).\n\n[2] Ahrens, M., Ashwin, J., Calliess, J. P., & Nguyen, V. (2021). Bayesian topic regression for causal inference. arXiv preprint arXiv:2109.05317.\n \n> (W3) The approach is really cool, but like any ML-pipeline approach it surely introduces errors at each step of the process (see for example Egami et al @ NeurIPS 2023). Each error will then bias the causal effect estimation. From the results of the paper I’m convinced empirically your approach works well, but would I be able to understand whether it produces accurate estimates on a new dataset? A deeper analysis and perhaps a way to propagate error/uncertainty to the estimation would be very helpful here.\n\nYour intuition is correct in that different errors can arise due to the assumptions required by NATURAL, some of which may not be formally testable yet. While this is important future work that we are actively pursuing, there are some approaches we can already use to mitigate and/or understand such errors.\n- We try to minimize errors in the steps using LLMs via prompt tuning on a handful of examples. We will describe this tuning in full detail in the appendix via a worked example that goes through all the steps of NATURAL for the Semaglutide vs. Tirzepatide dataset.\n- For methods that involve propensity score estimation, we use the balancing score property of propensity scores as a sanity-check. Figure 5 of our paper confirms that propensity scores estimated by NATURAL do indeed balance covariates across treatment groups. \n- We will also add a sensitivity analysis to gain more insight into the satisfaction of our Assumption 1 about Strong Ignorability or Unconfoundedness (see response to Q1 of Reviewer oSFK and attached PDF). \n\n> (Q1) Authors explain that as part of their process, they “filter for reports that are likely to conform to the experimental design”. Could this introduce bias?\n\n- The purpose of filtering for reports that conform to the experiment design is to estimate local ATEs within a specific population. This also makes our estimators comparable to the treatment effect from real-world RCTs that enforce inclusion criteria for participants. In practice, we use LLMs for this step, which can definitely introduce bias and is a limitation of our work, as discussed in Section 5.\n- In general, we do not expect one estimated ATE to transport to a different experiment design. Given another experiment design, one would have to execute the NATURAL pipeline with that design and estimate a new ATE. One advantage of NATURAL is that doing this for different designs is feasible in a time- and cost-effective manner.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mGqPTipy5Y", "reviewer": "Reviewer_p9SY", "summary": "This work aims to the causal effect estimation from unstructured observational text data, proposing a pipeline based on LLM to extract the treatment, covariate and outcome from the unstructured data by LLM. It's a interesting exploration for causla inference and the pipeline process of the inference method is quite standard.", "questions": "1.there are some claims that are not well expalined, leading to confuse people. For exampel, \"value may be recouped from outcomes that would otherwise be lost\" this claim sounds good but i don't what does it mean here.\n\n2.the overall idea of this work to utilize LLM to explore the causal effect estimation from unstructured data is good, i like it. But the writing can not match the good idea. For example, line 54-59, i have no idea about what the authors try to express, The transition from sentence to sentence is too stiff \n\n3.The binary and discrete assumption about the outcome Y and covariates X is kind of too strict. Typically, one can assume the teatment assignment is binary but outcome and covariates are continous. It's not clear why this work adopt such assumption. if so, the applying scale of the proposed method would be limited. also, the assumption 4 is also too strong.\n\n4.in the  outcome predictor eq. (9), why it is P(Y=1|R_i,X_i,T_i) instead of P(Y=t|R_i,X_i,T_i)? I don't understand. When t=0, the outcome predictor still ouput the expectation over P(Y=1|xxx)?\n\n5.the description how to extrct the covariates and their corresponding values is not clear.  How to determine the feature values? For each report, the set of covarivates is the same, right? Can you present a specific example? what does each feature value mean in the real-world? The readability of the method section is poor.\n\n6.In the evaluation table 1, the \"IPW (Structured) \" is a baseline or ground truth? there is no any discussion in the main body about this baseline and i am so confused about the discussion below because of it, given that the \"IPW (Structured) \" performs the best.\n\n7.how does the ground truth of ate comes from? Can you clarify this point?\n\nby the way, i like the idea of this work, but the presentation of this paper is really poor. thus if the above questions can be addressed well, i will consider to increase the score.", "limitations": "authors have adequately addressed the limitations.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "(1) The problem solution for treatment effect estiamtion is quite novel.\n\n(2) The proposed LLM pipeline can address the limitation of the current effect estimation method, i.e., can not deal with the unstructured data.", "weaknesses": "(1) some details are missing, leading to the poor readability.\n\n(2) Some equations are not correct (based on my understanding).\n\n(3) The presnetation of the whole methodology is not good enough, alought I am not an expert of LLM, I know how LLM works. However, i can not imagine how the proposed pipeline work and how it estimate ATE very clearly. \n\n(4) The evaluation part is kind of weak, given that baselines are too limited and simple.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and questions! Below, we will clarify all the questions in the review.\n\n> (W1) some details are missing, leading to the poor readability.\n\nCould the reviewer point to specific details that they found unclear? We are more than happy to provide clarification. \n\n> (W2) Some equations are not correct (based on my understanding).\n\nSimilar to above, if the reviewer can point out particular equations that they found to be incorrect, we are happy to clarify them.\n\n> (W3) The evaluation part is kind of weak, given that baselines are too limited and simple.\n\n- Since we propose a novel setting of constructing causal effect estimators from unstructured text data, there aren’t a lot of existing baselines. We constructed fair comparisons and included ablation studies for our experiments. In synthetic settings, we were also able to train baselines that require additional structured data (which NATURAL does not assume access to) and presented those results in Table 1. In real-world settings, we further teased apart the effect of different choices we made in the method implementation, as shown in Figure 4.\n- If the reviewer has specific suggestions for baselines that are relevant to our setting, we would be happy to include them. \n\n> (Q1) there are some claims that are not well expalined, leading to confuse people. For exampel, \"value may be recouped from outcomes that would otherwise be lost\" this claim sounds good but i don't what does it mean here.\n\nWe apologize for any confusion and expand on the referenced phrase here. \n- Sometimes there are outcomes that individuals might observe themselves, that are not captured in structured, tabulated observational data, but that are captured in unstructured text data on the internet. \n- Had these outcomes been tabulated and recorded, they could have contributed in valuable ways to decision-making. \n- NATURAL recovers the value of these outcomes for decision-making by extracting them from unstructured text data. \n\n\nWe are happy to change this sentence to “value may be *recovered* from outcomes that would otherwise be lost” in the paper.\n\n> (Q2) the overall idea of this work to utilize LLM to explore the causal effect estimation from unstructured data is good, i like it. But the writing can not match the good idea. For example, line 54-59, i have no idea about what the authors try to express, The transition from sentence to sentence is too stiff\n\nThe referred lines are intended to explain the different information NATURAL depends on. These are: (1) LLMs to approximate relevant conditional distributions, and (2) expert knowledge to ensure necessary causal assumptions are satisfied by the study design.\n\n> (Q3) The binary and discrete assumption about the outcome Y and covariates X is kind of too strict. Typically, one can assume the teatment assignment is binary but outcome and covariates are continous. It's not clear why this work adopt such assumption. if so, the applying scale of the proposed method would be limited. \n\n- Our NATURAL Full estimator requires discrete and finite variables since we sum over them to approximate a conditional expectation with conditional probabilities. If we choose to use samples instead of conditionals, NATURAL can further be extended to continuous variables.\n- While we have binary outcomes in our experiments, the method and implementations extend to non-binary discrete outcomes easily. \n- It is true that outcomes and covariates in the real world are often continuous. In order to keep NATURAL applicable in these settings, we discretize these variables. We will explain the exact discretization through a complete walkthrough of the Semaglutide vs. Tirzepatide dataset in the appendix. \n\n> also, the assumption 4 is also too strong.\n\nAssumption 4 is weak or strong depending on how much information is in the reports. Our data filtration pipeline aims to maximize the information in the reports so that Assumption 4 may be matched. We discuss this in more detail in Section 3 of the paper as well as our response to Q1 from reviewer 7S1Z.\n\n> (Q4) in the outcome predictor eq. (9), why it is P(Y=1|R_i,X_i,T_i) instead of P(Y=t|R_i,X_i,T_i)? I don't understand. When t=0, the outcome predictor still ouput the expectation over P(Y=1|xxx)?\n\n- We don’t believe there is a mistake in eq. (9). It is a Monte Carlo approximation of the first term in eq. (2). Eq. (2), in turn, is the standard form of the Outcome Imputation estimator (see Ding (2023)).\n- To clarify our notation, $Y$ is a random variable denoting outcomes and $T$ is a random variable denoting treatments. Therefore, ${Y=t}$ is not an event that any of our estimators would consider.\n\n[1] Ding, P. (2024). A first course in causal inference. CRC Press.\n\n> (Q5) the description how to extrct the covariates and their corresponding values is not clear. How to determine the feature values? For each report, the set of covarivates is the same, right? Can you present a specific example? what does each feature value mean in the real-world? The readability of the method section is poor.\n\n- While the definition of the covariate variables and their descriptions are the same for reports in a given dataset, the values these variables take will vary for each report. For instance, if a covariate of interest is “age”, its value in the real world may be a different number for different reports. \n- We will include more details and exact descriptions of each variable for each dataset in the appendix of our final paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their time and the effort they took to provide valuable feedback! 

Overall, all of the reviewers appreciated the significance and novelty of our work, with oMqj calling it “ambitious work” and with oSFK calling it a “cool” and “properly formalized” methodology. Most reviewers also appreciated the clarity of our presentation. 

Reviewer feedback has helped us identify a few ways to improve the paper. Based on the reviewers’ suggestions, we would like to include the following edits. We expand on these points in more detailed reviewer feedback in the comments below.

1. **Complete walkthrough of NATURAL for Semaglutide vs. Tirzepatide (oMqj)**: We will add a section working through the entire NATURAL pipeline for the Semaglutide vs. Tirzepatide dataset, in the appendix. This will include a step-by-step explanation of how to implement our method as a function of the experiment design, as well as the strategies we used to minimize error in each step.

2. **Expanded section on limitations (oMqj and oSFK)**: We agree that it is very important to detail the limitations of NATURAL. We stated these in our paper and included an extended discussion of these in the appendix of our original submission, due to space constraints. We will consolidate all discussions addressing limitations in an expanded Section 5 of the final version of our paper.

3. **Sensitivity analysis to address the Strong Ignorability assumption (7S1Z)**: We have run an analysis of the sensitivity of our ATE estimates to the degree of unobserved confoundedness, following the strategy in Lu and Ding (2023). In the attached PDF, Table 1 shows that the direction (sign) of our estimates is robust to large changes in the degree of unobserved confoundedness. Further, Figure 1 shows that each covariate we accounted for is important to our ATE estimation, via a leave-one-covariate-out approach. We expand on the interpretation of these analyses in our comments to reviewer 7S1Z below.

4. **Intuitive way to treat inclusion criteria and partially address selection bias (oMqj and 7S1Z)**: Selection effects are important considerations for our work, which includes factors like inclusion criteria and reporting biases. Reporting bias is outside of the scope of our current paper, and we address it in the expanded limitations section (see appendix and above). When it comes to inclusion criteria, in Appendix G of the paper we made certain structural assumptions that allowed us to impute unobserved covariates restricted to the inclusion criteria. During the rebuttal period, we implemented an alternate approach, which is also described in Appendix G in the original paper, that allows us to remove these additional assumptions by estimating an additional probability with the LLM. We show these new results in Table 2 of the attached PDF, which are close to our original results and which we will include alongside the original results in the final draft. We describe this in more detail in our response to reviewer 7S1Z.

We look forward to engaging further during the discussion period to clarify anything else.

[1] Lu, S., & Ding, P. (2023). Flexible sensitivity analysis for causal inference in observational studies subject to unmeasured confounding. arXiv preprint arXiv:2305.17643.

Meta-review: The paper develops an innovative, causal-aware way to estimate causal effects with the help of large language models. Overall, the reviewers including myself found the idea fresh and interesting. Here, I noted the term 'causal-aware' in the sense that the method does not have a theoretical grounding as many other papers in the causal inference track but offers a clever way together with strong empirics. 

The reviewers had some comments on the presentation (which can hopefully addressed in the camera-ready version). The question regarding more baseline is a difficult one, given that the task is fairly novel, so that simply baselines are not available (which is often the case in causal inference). 

Overall the reviewers find the contribution valuable and the authors have made a significant effort in addressing clarifying questions and concerns from reviewers. I do believe all additional experiments should be incorporated in the main paper. As a result, I think that the paper is ready to be accepted.
Decision: Accept (poster)