End-To-End Causal Effect Estimation from Unstructured Natural Language Data

Authors: nikita  dhawan
leonardo  cotta
karen  ullrich
meta  ai
rahul g krishnan
chris j maddison
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 02:54:43

Abstract: Knowing the effect of an intervention is critical for human decision-making, but current approaches for causal effect estimation rely on manual data collection and structuring, regardless of the causal assumptions. This increases both the cost and time-to-completion for studies. We show how large, diverse observational text data can be mined with large language models (LLMs) to produce inexpensive causal effect estimates under appropriate causal assumptions. We introduce _NATURAL_, a novel family of causal effect estimators built with LLMs that operate over datasets of unstructured text. Our estimators use LLM conditional distributions (over variables of interest, given the text data) to assist in the computation of classical estimators of causal effect. We overcome a number of technical challenges to realize this idea, such as automating data curation and using LLMs to impute missing information. We prepare six (two synthetic and four real) observational datasets, paired with corresponding ground truth in the form of randomized trials, which we used to systematically evaluate each step of our pipeline. NATURAL estimators demonstrate remarkable performance, yielding causal effect estimates that fall within 3 percentage points of their ground truth counterparts, including on real-world Phase 3/4 clinical trials. Our results suggest that unstructured text data is a rich source of causal effect information, and NATURAL is a first step towards an automated pipeline to tap this resource.

Main paper: 
 introduction: 
 Estimating the causal effects of interventions is time consuming and costly, but the resulting outcomes are precious. Health agencies around the world often require randomized controlled trial (RCT) data to approve medical interventions. Clinical trials are key contributors to large R&D costs for drug developers [30] . Natural experiments are another source of rich interventional data, but they may not always exist or have enough data relevant to a given causal hypothesis [12] .When treatment randomization is infeasible, observational data can be used to identify average treatment effects (ATEs) [48] , under common assumptions, e.g., no unobserved confounding. Such data is abundant but even when the necessary assumptions are satisfied, it must be structured (i.e., the outcomes, treatments, and relevant covariates must be defined, recorded, and tabulated) before it becomes amenable to computational analyses.Yet, unstructured observational data presents unique opportunities for cheaper, more accessible, and potentially even better [32] effect estimation. For example, thousands of people living with diabetes choose to share their experiences with treatments on online patient forums. Some of their posts contain rich descriptions of daily lives, the drugs they have been prescribed, the treatment responses and side effects, as well as pre-treatment information like age and sex. Their posts contain their lived experiences including evidence of an outcome in an observational experiment, albeit in an unstructured form. Other potential sources of rich unstructured, observational data include newspaper classifieds, police reports, social media, and clinical reports. Despite being collected for a myriad of purposes, researchers have often turned to such data to test hypotheses since: (i) unstructured data does not require restrictive data collection designs, e.g., measurement choice, and can admit many different post-hoc analyses; (ii) the reported outcomes may reflect what matters to subjects better than standard outcome measures; (iii) value may be recovered from outcomes that would otherwise be lost; (iv) there may be more unstructured data available on underserved or marginalized populations. Figure 1 contrasts our setting with previous works using randomized or structured observational data. This work asks a simple question: How can we use large language models to automate treatment effect estimation using freely available text data? We introduce NATURAL, a family of text-conditioned estimators that addresses this by performing NATural language analysis to Understand ReAL effects.At a high level, the steps required to compute NATURAL estimators are as follows. Given an observational study design and a dataset of natural language reports, filter for reports that are likely to conform to the experimental design. Then, using a large language model (LLM), extract the conditional distribution of structured variables of interest (outcome, treatment, covariates) given the report. Finally, use the conditionals to compute estimators of the ATE, using classical strategies such as inverse propensity score weighting and outcome imputation.NATURAL is a data-driven pipeline. It leverages and relies on the LLM in a manner that mimics the learning task it was trained for: providing parametric approximations to conditional distributions. As in all observational studies, the validity of NATURAL also depends on prior causal knowledge about the task. Expert knowledge is required to define appropriate covariates and confirm that they satisfy the necessary assumptions for effect estimation. However, we anticipate that NATURAL estimators could be developed under other structural assumptions (e.g. instrumental variables) as well.The core contributions of our work are:• We derived NATURAL ATE estimators based on classical estimators of the ATE, like inverse propensity score weighting and outcome imputation. NATURAL estimators operate on entirely unstructured data under two novel data-access assumptions. • We implemented NATURAL estimators using an LLM-based pipeline.• We developed six observational datasets to systematically evaluate parts of this pipeline: two semi-synthetic datasets constructed using marketing data, and four clinical datasets curated from public (pre-December 2022) migraine and diabetes subreddits from the Pushshift collection [7] . • For each dataset, we treated the ATE from a corresponding real-world completely randomized experiment (CRE) as ground truth. Remarkably, our predicted ATEs all fell within 3 percentage points of the ground truth ATEs, a potential cost savings of many millions of dollars.Leveraging natural language data [42] to support causal claims is pervasive in applied research [43, 13] . Our work falls under the broad umbrella of accelerating the identification of real-world evidence (RWE) [40] . For instance, in the context of healthcare, RWE supports not only drug repurposing, but also post-market safety evaluations -its most common application. NATURAL expands the boundaries of how quickly one can obtain and validate such real-world evidence from observational data [41] .The use of natural language data in causal inference comes in different flavors: i) using text to measure confounders [23] , ii) using text to measure causal effect outcomes [15] , or iii) producing interpretable causal features from text [15, 6] , e.g., what words are more likely to explain the cause of an event. NATURAL distinguishes itself from these lines of research in two ways: i) NATURAL does not require any curated task-specific training data (it is zero-shot), and ii) NATURAL is not interested in how the text itself, i.e., its words, relate to the causal problem -that is, we are only leveraging the model's ability to predict the distribution of a specified variable conditional on the input text. We highlight that our work lies distinct from research at the intersection of text and causality that combines text and numerical or tabular data [14] , the latter of which may be unavailable or incomplete in settings involving neglected diseases, unrecorded abortions, or illicit drug use. Other work has also studied topic modelling approaches [2] or the ability of language models to infer latent variables (that are implied but not explicitly identified in text data) [36, 13] . Rather, we require the precise specification of covariates to condition on -we view this as being crucial to creating a more direct way for an end user to verify the validity of information extracted with our approach.Prior works have also leveraged LLMs in a black-box fashion for causal tasks by querying the model for causal statements. In the context of causal discovery, users directly ask for the existence of cause-and-effect relationships, e.g.,"Does changing the age of an abalone causes a change in its length?" [25, 33, 4, 5, 44, 21, 6] . Due to the large amount of training data, it is possible that the model learns to apply a causal model described in the training data and answer causal questions with it [35, 47] . The issue with this approach is i) the user is limited to the causal models observed in training, ii) the user is not aware of which causal model they are using, and iii) the queries tend to present high prompt sensitivity [28] . Finally, we note that a recent work created a benchmark and showed how LLMs struggle to distinguish pairwise correlation from causation [20] , while another shows that checking causal relationships in a pairwise manner can lead to invalid causal graphs [45] . 
preliminaries: 
 We are interested in estimating the causal effect of a treatment relative to either another treatment or no treatment in a population of interest. More precisely, we consider treatments t ∈ {0, 1} and the corresponding potential outcomes Y (1) and Y (0) under each treatment. We wish to compute the quantity τ := E[Y (1) -Y (0)], often referred to as Average Treatment Effect (ATE). Sometimes, Y (0) may correspond to no treatment (control). Throughout this work, we assume binary treatments and outcomes in the Neyman-Rubin causal model. We provide a full list of notation in appendix A.A Completely Randomized Experiment (CRE) with n participants requires no prior causal knowledge. In a CRE, the treatment assignment vector ( Ti ) n i=1 is a random permutation of n 1 ones and nn 1 zeros sampled independently of the outcomes. In this case, the difference-in-means1 n1 n i=1 Ti Y i (1) -1 n-n1 n i=1 (1 -Ti )Y i (0)provides us with an unbiased estimate of τ . Despite the indisputable necessity of CREs in high-stakes settings, it is often expensive and/or infeasible to have complete control over the treatment assignment. Instead, observational data is more readily available. Observational data often contains spurious correlations between the observed treatment T and the observed outcome Y = T Y (1) + (1 -T )Y (0) through a common cause (confounder). Typically, this confounding is formalized as a variable X, which we assume to be discrete throughout this work, representing covariates associated with each individual. Given i.i.d. samples {(X i , T i , Y i )} n i=1 from the target population, standard causal inference techniques can correct for confounding bias and provide consistent estimates of τ under Assumptions 1 and 2: Assumption 2 (Positivity.) For every treatment t and covariate set x, 0 < P (T = t | X = x) < 1.Following are two classical estimators of the ATE τ from observational data, each of which rely on X satisfying Assumptions 1 and 2. We refer the reader to Ding [11] for further details.Inverse Propensity Score Weighting (IPW). The propensity score is the conditional probability of receiving a treatment given the observed features, i.e., e(x) = P (T = 1|X = x). The IPW estimator is given byEQUATIONwhere ê(x) is an approximation of P (T = 1 | X = x). When ê(x) is the true propensity score, τIPW is an unbiased estimator of τ . When ê(x) is estimated as empirical probability, τIPW is consistent.Outcome Imputation (OI). Outcome Imputation learns a model to impute outcomes from features and treatment and then marginalizes away the features to estimate τ withEQUATIONwhere τ (x, t) approximates P (Y = 1 | X = x, T = t). Note that if τ (x, t) is an unbiased estimation of this quantity, τOI is an unbiased estimator of τ . 
natural estimators of the ate: 
 Both CRE and observational studies require direct access to tabulated data (X i , T i , Y i ) for every individual i. Our NATURAL estimators on the other hand estimate the ATE from observational, unstructured natural language data in the form of language reports R i . In addition to Assumptions 1 and 2, NATURAL estimators require the following assumptions to guarantee their consistency.Assumption 3 (Natural language report data.) The target population is described by an observational data-generating process P (X, T, Y, R) of data (X, T, Y ), which satisfies Assumptions 1 and 2 and is jointly distributed with a random natural language string R, called a report. We assume access to an i.i.d. sample of reports {R i } n i=1 from the marginal of this process.Assumption 4 (Access to the true observational conditional over (X, T, Y ).) We can either (i) compute the conditional P (X = x, T = t, Y = y|R = r) of the true data-generating process, or (ii) we can sample from P (X = x|R = r) and compute P (T = t, Y = y|R = r, X = x).Intuitively, these assumptions give NATURAL indirect access to (X, T, Y ) through R. They can be weak or strong, depending on the definition of the reports R. On the one hand, if reports are copies of the observational data, i.e., R = (X, T, Y ), then Assumption 4 is trivial to satisfy. On the other hand, if reports are all the constant, empty string, R = ϵ, then Assumption 4 guarantees that we have full access to the true observational joint density function over (X, T, Y ), which is a strong assumption.In other words, it requires a way to simulate trial outcomes unconditionally (without any data). We consider how we might satisfy these assumptions in practice in the next section. Here, we assume that they hold and develop a series of consistent estimators of the ATE.NATURAL Full. Given {R i } n i=1 and P (X = x, T = t, Y = y|R = r), we can construct an idealized version of NATURAL. Let us start by noting that the law of total expectation gives usEQUATIONA Monte Carlo estimate over reports is given byEQUATIONwhich further approximates êN-Full (x) from the given conditional. We used eq. ( 7) below. We note that τN-Full above is derived from IPW, but can also be derived from OI, as shown in appendix B.The estimator τN-Full above relies on enumerating all possible values of (X, T, Y ), making it computationally expensive for high-dimensional X. Below, we present two hybrid versions of our method which combine sampling of some variables and computation of conditional probabilities of others.NATURAL IPW. To construct our hybrid estimator, we augment the data {R i } n i=1 by sampling from P (X|R i ) independently for each report R i . This gives us a dataset {(R i , X i )} n i=1 drawn i.i.d. from P (X, R) by Assumption 4. Then, our hybrid estimator is derived from the form of IPW as follows:EQUATIONEQUATIONwhere êN-IPW (x) is consistently estimated in the following manner:EQUATIONNATURAL OI. Similarly inspired by the OI estimator in equation 2, we have for t ∈ {0, 1},EQUATIONThus, for our hybrid OI estimator, we augment the data {R i } n i=1 by sampling from P (X, T |R i ) independently for each report R i . This gives us a dataset {(R i , X i , T i )} n i=1 drawn i.i.d. from P (R, X, T ) by Assumption 4. Then, our consistent outcome predictor is given byEQUATIONand the final estimator is given by:EQUATIONNATURAL Monte Carlo. Further in the direction of sampling more variables, we can obtain samples (X i , T i , Y i ) from P (X, T, Y |R i ) and compute a Monte Carlo estimate, τN-MC . The set of samples {(X i , T i , Y i )} n i=1 constitute a tabular dataset which can be plugged into a standard ATE estimator like IPW or OI, as described in section 2. We refer to these sample-only estimators as N-MC IPW and N-MC OI, respectively.Inclusion Criteria conditioned ATE. We are sometimes interested in ATEs over populations defined by constraints on pre-treatment covariates X i , known as inclusion criteria and denoted by I. This conditional ATE is τ(I) = E[Y (1) -Y (0) | X ∈ I]and satisfies the following identity.EQUATIONThe conditional version of NATURAL IPW can be estimated by filtering out reports where P (X ∈ I|R i ) = 0, sampling X ij ∼ P (X|R i , X ∈ I) i.i.d., and finally weighting datapoints by the relative likelihood of matching the inclusion criteria given the report:EQUATIONwhere the inner expectation is estimated similar to eq. ( 6). A complete derivation for eqs. (11) and (12) and related discussion are included in appendix G. In practice, we took m = 1. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: While this is not a theoretical paper, we provide the full set of assumptions we operate under and method derivations in section 3, with strategies to satisfy these assumptions in section 4 and additional details and derivations about more nuanced steps of our method in appendix G. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
implementing natural estimators with large language models: 
 LLMs are trained on vast datasets of real-world data, e.g., [3] , which likely contain records of data generated by processes that are consistent with Assumption 3. Because LLMs can learn wellcalibrated conditionals [22] , our hypothesis is that LLMs can be prompted to approximate the conditionals required by Assumption 4 for real-world causal effect questions of interest. Our LLM implementation of NATURAL estimators is built on this hypothesis to try to satisfy Assumptions 3 and 4 (Assumptions 1 and 2 must be guaranteed by a domain expert). We defer exact prompts for LLM inference to appendix D, a full worked example to appendix C, and a discussion of the limitations of our approach to the next section. Figure 2 summarizes our pipeline.Filtering to match Assumption 3. For our real-data clinical settings, our first goal is to produce a dataset of i.i.d. reports R i that are very likely to be jointly distributed with the random variables (X i , T i , Y i ) of a specific observational study of interest. Given a study of interest and a dataset of real-world reports that are potentially relevant to the study, we pass it through a sequence of filters with increasing detail and strictness: (i) Initial filter. Inspired by other work with social media data [1, 38] , we first use deterministic rules to filter out uninformative reports: posts that were removed, are too short, have "bot" in the author's name, have no mention of any keyword related to the study, etc. (ii) Filter by relevance. We prompt an LLM to determine whether each report contains information that would make it relevant to the study. We remove reports that are deemed irrelevant. (iii) Filter by treatment-outcome. We ensure that each report pertains specifically to the treatments and outcomes of interest. We do so by prompting an LLM to extract only treatment and outcome information, and retaining only the posts that are deemed to both mention one of the treatments in question and also contain outcome information. (iv) Filter known covariates by inclusion criteria. For ATEs conditioned on inclusion criteria, as in our real-world datasets, we included a filter to enforce these criteria. Managing inclusion criteria is complicated by the fact that many reports R i contain no or partial information about covariates that are required to verify inclusion. So, in this filtering step, our goal was to ensure that the final set of reports have non-zero probability of matching the inclusion criteria. We begin by prompting an LLM to extract the full set of covariates X i , following constraints on the possible values each attribute can take, but we allow the LLM to extract Unknown if it is impossible for the LLM to determine the value of a covariate. We then remove reports, if any of the non-Unknown covariates are determined to fail their inclusion criteria. We found the JSON-mode made available for generation by certain LLM APIs, to suffice for this task; however more involved strategies for constrained generation are also possible [46, 50] .Sampling from and computing conditional probabilities to match Assumption 4. Given a set of reports {R i } n i=1 that pass the filtering stage above, our next steps use LLMs to extract the samples and conditionals P LLM (X, T, Y | R), required to compute NATURAL estimators. For each R i , we: (v) Extract covariates, both known and unknown. We run a final covariate extraction by prompting an LLM to determine the full set of covariates X i from the report R i , subject to the constraint that X i satisfies the inclusion criteria. In contrast to (iv), we ask the LLM to guess the values of Unknown covariates. We verified that this second extraction agreed exactly with the first extraction (iv) on the known covariates (i.e., the ones that were not extracted as Unknown in the first extraction). We contrast the empirical distributions of these known and unknown/guessed covariates for our experiments in appendix F.1. (vi) Infer conditionals. Given {R i , X i } n i=1 from the previous steps, we compute the probabilities P LLM (T = t, Y = y|R i , X i ) by prompting an LLM that makes log-probabilities accessible. Specifically, we ask an LLM to answer questions about T, Y given access to R i , X i , and we score every possible answer T = t, Y = y using the LLM log-probabilities. We exponentiate and renormalize these scores across the space of possible realizations to obtain a valid probability distribution. (vii) Weight reports according to inclusion criteria match. Similar to item (vi), we compute P LLM (X i ∈ I|R i ) to obtain the weights in eq. ( 12), by prompting an LLM with descriptions of the inclusion criteria that must be satisfied and each report R i . It may be possible to skip this weighting step under additional structural assumptions on the data. These assumptions as well as experimental results without the weighting are included in appendix G.Nevertheless, while our empirical results are remarkably consistent with the correctness of our pipeline, we cannot formally guarantee that it satisfies Assumptions 3 and 4. The final outcome of this pipeline is a dataset {R i , X i } n i=1 and a set of conditionals P (T = t, Y = y|R i , X i ) that can be plugged into the hybrid NATURAL estimators in section 3 to predict ATEs. Therefore, we see this as a first implementation of NATURAL estimators, which we anticipate can be improved.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Experimental details to reproduce our datasets and empirical results are provided in section 6 with further details like prompts used to query language models in appendix D. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Justification: We dedicate a section to the limitations and broader impacts of our work in section 5. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
limitations and broader impact: 
 In addition to the limitations that NATURAL shares with every observational study, i.e., the validity of the practitioner's causal assumptions, it comes with an extra dependence on how well one can approximate the desired conditional distributions. While more and more capable LLMs are being continually developed, the extent to which they satisfy NATURAL's assumptions is nearly impossible to formally test. Indeed, while pretraining tends to produce calibrated LLM predictions [22] , posttraining techniques can compromise calibration [34] . Therefore, we emphasize that NATURAL was not developed to recommend therapeutics directly to end-users or to directly inform high-stakes public policies. Instead, we envision NATURAL as a powerful tool to help us approximate ATEs at scale and prioritize confirmatory CREs. We strongly recommend that all predictions made by NATURAL estimators be validated experimentally before being used to inform high-stakes decisionmaking. Apart from its dependence on LLM capabilities, NATURAL is also limited by the nature of observational, unstructured natural language data:• Network Interference. In practice, acquiring i.i.d. reports can be challenging. For instance, social network users might talk to each other and influence their treatment choices. This is a well-known issue in causal inference and statistical sciences in general. Existing solutions rely on a known network structure to sample individuals or correct for their neighbors' treatments [10, 26, 17] . • Outcome Measurement. Since NATURAL deals with self-reports, subjects need to be able to report the outcomes of interest. For example, this cannot be applied if the outcome is measured with an expensive, inaccessible test. Therefore, the study design implemented with NATURAL must account for the accessibility of endpoints to users.• Selection Bias. Results might be biased towards individuals' choice of reporting an outcome given their experience with the treatment. Luckily, outcome missingness is a widely studied problem in causality research, see e.g., how to test [9] or how to mitigate [31] it. Note, however, that solutions will often accumulate assumptions on top of NATURAL and should always be critically evaluated by practitioners. Finally, apart from individuals' choice of reporting, selection bias might also arise from which individuals participate in online forums, i.e., our framework is only capable of estimating local ATEs -external validity is not guaranteed a priori. We demonstrate this challenge in section 6 by simulating systematic bias in semi-synthetic settings. 
empirical evaluation: 
 Evaluating an end-to-end pipeline for causal inference from unstructured real-world text data to ATEs presents challenges regarding access to data, ground truth ATEs and insightful intermediate metrics.We used two semi-synthetic datasets where we augmented randomized data to mimic real-world observations, while continuing to have access to ground truth evaluation. In addition, we study four real datasets, curated from publicly available Reddit posts from the PushShift dataset, as described in section 4. These six datasets allowed us to systematically evaluate NATURAL. Semi-synthetic Datasets. Causal effect estimation is typically evaluated using synthetic datasets with one or more relationships between the observed covariates, treatment and outcome being contrived. We instead synthesized unstructured observational text data from real randomized tabular datasets, using an LLM. Specifically, we (i) introduced confounding bias by sampling datapoints according to an artificial propensity score, (ii) randomly dropped covariates, (iii) described covariates, treatment and outcome in shuffled orderings, (iv) simulated realism by sampling a persona from the the Big Five personality traits [27] for each datapoint and finally, (v) prompted the LLM to generate a realistic report describing the provided information in the style of someone with the given traits (see appendix D for the full prompt). We used two standard, publicly available randomized datasets: Hillstrom [19] and Retail Hero [49] , and plan to open-source scripts to generate our data.Step (i) above is in a similar vein as Keith et al. [24] , in that our subsampling strategy does not modify the marginal distribution over covariates and the ATE remains identifiable from observational data.Real-world Datasets. To study how our framework may be deployed to test hypotheses using real data from online forums; we considered two medical conditions for which there exist abundant Reddit posts in the PushShift collection [7] , with individuals' personal experiences: the effect of diabetes medications (e.g. Semaglutide) on weight loss and the tolerability of migraine treatments. For each condition, we picked two clinical trials which performed head-to-head comparisons of two treatments that we expected to find references to in relevant subreddits. Moreover, to mitigate selection bias we selected pairs of similar treatments, e.g., comparable availability, where we believe the probability of a user reporting their experience is approximately equal in both. As we will discuss, our results suggest external validity as well, meaning that the probability of a user reporting their experience with the treatments seems to be (approximately) equal to the prior probability of a user reporting any experience. We limited our data collection to posts that were written before December 2022 and made publicly available in the PushShift archives. We curated four datasets for comparison between different treatments, each of which has a ground truth RCT: Semaglutide vs. Tirzepatide [18] and Semaglutide vs. Liraglutide [8] for their effect on weight loss and Erenumab vs. Topiramate [37] and OnabotulinumtoxinA vs. Topiramate [39] for their tolerability. We used the first of these to validate implementation choices NATURAL (like filtering, imputations, prompt specifications) and the other three as held-out test settings, see appendix C. We include further details for all our datasets in appendix E, including the definitions of covariates and outcomes.Next, we investigate several questions about the performance of NATURAL empirically. We used GPT-4 Turbo for sampling and LLAMA2-70B for computing conditional probabilities.How well does NATURAL estimate observational distributions from self-reported data? Our semi-synthetic datasets give us access to the true joint distributions P (X, T, Y ) and true propensity scores P (T = 1|X). The top row of fig. 3 shows the KL divergence between these distributions and those estimated by NATURAL Full, for Hillstrom (left) and Retail Hero (right). We find that these KL divergences decrease steadily as the number of reports used in the estimation increases.The bottom row shows corresponding root-mean-squared error (RMSE) between NATURAL and the true ATE. This corroborates the insight that as the joint distribution and propensity scores are estimated more accurately, the predicted ATE gets closer to its true value. In particular, we observe a clear correlation between the quality of estimated propensity scores and estimated ATEs.How do NATURAL methods compare to one another and to trained baselines? We present our estimated ATE and its RMSE on semi-synthetic datasets in table 1 . Further, we evaluate two trained baselines, which use a Bag-of-Words model and a sentence encoder respectively, to train representations of text data with their labels. Here, for each attribute in the set of covariates, treatments, and outcomes, we train a MLP model with 5-fold cross validation to predict that attribute. We then use these predicted attributes as a tabular dataset of samples that can be plugged into any causal inference estimator. We find that our methods are competitive with or outperform these baselines, despite not being trained with any labels. In particular, the sentence encoder baseline collapsed to an ATE of zero for Hillstrom, having learned the constant predictor for outcomes. IPW (Structured) is an oracle estimator, which assumes full access to structured data. Selection-biased N-IPW demonstrates the challenge of ATE estimation in the presence of bias which was systematically simulated as a function of covariates "channel" and "zip code" for Hillstrom and "age" for Retail Hero. This result is significant. Clinical trials can take on the order of years and costs in the tens to hundreds of millions of dollars. Going from the raw language observational data to ATE in our framework takes on the order of days and costs at most a few hundred dollars of compute. For problems in medicine, economics, sociology, and political science where randomization is infeasible or expensive, NATURAL provides a tractable way to leverage observational data to rank potential experiments prior to conducting them.How do different choices in the NATURAL pipeline effect ATE prediction? We assess the impact of key choices in our pipeline described in section 4, by ablating them one-by-one. We investigated and selected these choices on the Semaglutide vs. Tirzepatide experiment. Figure 4 (left) compares the RMSE of predicted ATEs when data is not filtered according to inclusion criteria and LLM imputations are replaced with samples from an uniform distribution. It shows that both inclusion-based filtering and imputations from a pretrained LLM are crucial for the performance of NATURAL. We also compared performance of our method when the conditional probabilities in eq. ( 6) are evaluated using models of different scales in fig. 4 (right), and found that performance improves at larger scales and with greater quantity of data. How well do different estimates of propensity score balance covariates? A property of accurate propensity score estimates is that they balance covariates across treatment cohorts (see Ding [11] for details and proofs), i.e. the average treatment effect on each covariate, corrected using propensity scores, is close to zero. Figure 5 visualizes this quantity for different covariates of the Semaglutide vs. Tirzepatide experiment and shows that propensity scores estimated using LLAMA2 conditional distributions balance the covariates far better than a uniform distribution does, with the 70B model consistently estimating the treatment effect on each covariate as close to zero. Similar visualizations for the test settings are shown in fig. 10 of appendix F.2. 
conclusion: 
 In this work, we introduced NATURAL, a family of text-conditioned estimators, to automate treatment effect estimation using free-form text data. We demonstrated NATURAL's efficacy with six semisynthetic and real datasets for systematic evaluation of its pipeline. We exposed the ability of LLMs to extract meaningful conditional distributions over structured variables and how their combination with classical causal estimators can predict real-world causal effects with remarkable accuracy. Given this promising performance, exciting directions for future work include (i) incorporating automatic prompt tuning methods into the pipeline, (ii) extending our methods to real-valued (X, T, Y ), (iii) exploring whether our assumptions can be weakened, (iv) exploring other domains in applied research, e.g., social sciences, (v) performing a more extensive evaluation of NATURAL on different study designs to better understand what type of treatments, outcomes, and reports show better or worse practical performance with NATURAL or (vi) deploying the pipeline to test hypotheses at even larger scales.NATURAL estimators have numerous use cases with potentially far-reaching impact. As long as patients have access to treatments and report their experiences, NATURAL can be used to compare two treatments in new indications or new populations. Therefore, our pipeline can in principle support efforts to prioritize trials for repurposed drugs or supplements in under-served diseases or populations. Further, a crucial step after drug approval is post-marketing surveillance for side effects (positive or negative) that may not have been measured or may have been too rare to identify in a smaller trial. NATURAL can leverage the diversity of available language data to detect these effects. While our motivations largely stem from the challenges of drug development, our NATURAL estimators are applicable to any effect estimation setting for which there exists relevant natural language data. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: We do not release any large models. We do however plan to release scripts to reproduce our dataset, ensuring they rely on publicly available data. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite all original owners of code, data, and model used in this work. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: We do not release new assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Random variable corresponding to unstructured natural language text from a social media post (or report).Random variable corresponding to features of an individual in a causal inference dataset.T Random variable corresponding to treatment or intervention assigned to an individual in a causal inference dataset.Random variable corresponding to outcome observed for an individual in a causal inference dataset.x Possible instance of X from its support X .tPossible instance of T from its support T = {0, 1} (binary treatments).y Possible instance of Y from its support Y = {0, 1} (binary outcomes).r Possible instance of R from its support R.Random variable corresponding to potential outcome observed for an individual after receiving treatment t.Propensity score function for binary treatments, equal to P (T = 1|X).X i Sampled value of X for individual i. n 0 Total number of individuals that are assigned treatment T = 0.In section 3, we derived the idealized version of our method, NATURAL Full using the form of the IPW estimator. In fact, any unbiased estimator would lead to same form as eq. ( 3), which can then be used to derive NATURAL Full using reports R and the law of iterated expectations. Here, we show this using the Outcome Imputation estimator. For discrete X and binary T and Y , we haveEQUATIONEQUATIONEQUATIONEQUATIONEQUATIONEQUATION= (x,t,y)EQUATIONwhich is equivalent to eq. ( 3).To make the NATURAL pipeline and its implementation more concrete, we now work through an end-to-end example using the Semaglutide vs. Tirzepatide dataset. We used this setting to develop our evaluation setup and constructed the pipeline as a function of the experiment design. While it is infeasible to exhaustively describe the entire decision space, major decisions that were found to impact ATE prediction are (i) filtering strategy, (ii) prompt tuning for extraction, and (iii) covariates' discretization. The strategies we tried are included in the pipeline description below, with the specific decisions made for each highlighted in this color.Pipeline for Semaglutide vs. Tirzepatide. Given the experimental design of the clinical trial in Frías et al. [18] (NCT03987919), we defined our experiment as follows:1 Decisions in the pipeline that are a direct function of an experimental design like the one above are highlighted in this color in the following description.Next, we implemented the entire pipeline as follows:1. Initial filter. We identified nine subreddits relevant to this problem setting: r/Mounjaro, r/Ozempic, r/fasting, r/intermittentfasting, r/keto, r/loseit, r/Semaglutide, r/SuperMorbidlyObese, r/-PlusSize. From each subreddit, we downloaded all submissions and comments posted upto December 2022 from the PushShift collection, so as to only use publicly available data. This resulted in a dataset of 577,733 submissions and comments. An initial deterministic, taskagnostic and rule-based filter removed any submission or comment if its content was not a string, if it had no score, if the content was "[deleted]" or "[removed]", if it was a comment with fewer than ten space-separated strings (presumably, words), if the author's name contained the string "bot", if there were no spaces in the first 2048 characters, and if less than 50% of all characters were alphabetic. This reduced the dataset size to 380,276.We then formatted this data into dictionary-like datapoints with fields: subreddit, title, date created, post/comment, author replies. We indcluded the last field because comments written by the author as replies to their own post may contain additional relevant information when combined with with original post and other replies. We then passed these through a task-dependent string-matching filters. For this dataset in particular, we listed strings used commonly to refer to the treatments, ["ozempic", "mounjaro", "semaglutide", "tirzepatide", "wegovy", "rybelsus", "zepbound"], included common misspellings generated with GPT-4 and Perplexity, and filtered out datapoints that did not contain any of these strings. Similarly, we listed keywords relevant to the outcome of interest, ["kg", "kilo", "lb", "pound", "weigh", "drop", "loss", "lost", "gain", "hb", "a1c", "hemoglobin", "haemoglobin", "glucose", "sugar"] and filtered out datapoints that did not contain any of these strings. This filtered dataset now contained 50,654 datapoints. 2. Filter by relevance. Next, we wrote a problem setting description and prompted GPT-3.5 Turbo to determine whether the posts, along with auxiliary information from the formatted dictionaries described above, were relevant to the described setting. The description and instructions for this particular dataset are shown in prompt 2. We manually labeled a handful of datapoints as Yes or No and included these as incontext examples to improve the LLM's generations. We removed datapoints that were deemed irrelevant, resulting in a "relevant" dataset of 21,229 datapoints. 3. Filter by treatment-outcome. To further filter the data to points that refer specifically to the treatments and outcome of interest, we prompted GPT-3.5-Turbo to extract only information required to ascertain the treatment and outcome, as shown in prompt 3. Since the outcome for this dataset, achievement of a target weight loss of 5% or more, may be reported in several ways, we attempted to cover all those possibilities. Specifically, we prompted the LLM to extract the user's starting weight, end weight, change in weight and percentage of change in weight. Several combinations of these attributes allow us to programmatically infer the final outcome. We also extracted the units in which weight was reported, converting all extractions to be in lbs. We filtered out any datapoint for which the extracted treatment was not one of the treatments considered for this task or for which it was not possible to infer the outcome using the above-mentioned extracted information. This finally gave us a natural language dataset of 4619 relevant reports, each of which contained treatment and outcome information pertaining to the defined problem setting. 4. Filter known covariates by inclusion criteria. To evaluate against the clinical trial, we further filtered the dataset according to its inclusion criteria. At a high level, different strategies for this filtering trade-off how strictly we match the criteria with how many datapoints remain in the filtered set. Strict filtering to match every criteria exactly resulted in very few reports. Less strict filtering to match a subset of the criteria resulted in predicted ATEs that varied depending on which criteria we chose to satisfy, with no apparent task-agnostic principle to determine crucial criteria. Finally, we used instructions shown in prompt 4 to extract covariates and aimed for a set of reports with non-zero probability of satisfying these criteria. As described in item (iv) of section 4 and motivated in appendix G, we extracted all covariates including ones related to the inclusion criteria and removed datapoints whose extractions were not Unknown and failed to satisfy the criteria above. This resulted in a dataset of 1265 reports. 5. Extract known and unknown covariates. Treating these 1265 reports as the final dataset from which to estimate an ATE, we extracted the set of covariates given in our experiment definition. We also included the duration of treatment as a covariate since this information is often reported and is likely to influence the outcome. This extraction step was conditioned on inclusion criteria being satisfied, a description of which was included in the extraction prompt, as in prompt 5.We tuned the prompts for extracting attributes, which include general instructions for the task and specific questions for each attribute. This was done by inspecting a handful of reports and corresponding extractions and then modifying prompts to correct any observed errors. 6. Infer conditionals. We inferred conditional distributions from LLAMA2-70B for different versions of NATURAL, with the strategy described in item (vi) of section 4 and LLM inputs of the form shown in prompt 6. Here, "conditioning on covariates" was implemented by adding questions about the covariates and their sampled answers to the input. For instance, for sex, the question "What is the reported sex of the user?" was followed by its previously extracted answer (Male or Female). The scoring strategy required enumerating possible options for treatments and outcomes for each input, which were ["Semaglutide like Ozempic or Wegovy or Rybelsus", "Tirzepatide like Mounjaro or Zepbound"] and ["No", "Yes"], respectively. 7. Weight reports according to inclusion criteria match. We also used LLAMA2-70B to compute the weighting terms described in item (vii) of section 4. Concretely, we constructed a prompt, like prompt 7, describing the inclusion criteria listed in item 4 of the experiment design above, followed by a report, R i and an instruction asking the LLM to determine whether all the described criteria are met. We then scored the possible answers, ["No", "Yes"], exponentiated and renormalized them to obtain P (X ∈ I | R i ). We marginalized over reports to compute the denominator, P (X ∈ I), in the weight. The contribution of each report to the ATE estimates was weighted by this relative likelihood of matching the in inclusion criteria of the experiment given the report. 8. Finally, given all the required extractions and conditional probabilities, we required discrete covariates to plug them into our NATURAL estimators. Hence, we converted any continuous covariates into discrete categories. These categories for each dataset are shown in table 4 for all our datasets. Different choices of discretization led to slightly different ATE predictions. We found it most helpful to discretize continuous numerical covariates into intervals such that the number of datapoints were roughly balanced across intervals. This avoided covariate strata with too many or too few datapoints and resulted in ATE predictions from all NATURAL estimators that were sufficiently close to the ground truth.Adapting the pipeline to test trials. The decisions above in this color directly or indirectly influenced the ATE and we made our choices with access to the ground truth for Semaglutide vs. Tirzepatide. Hence, we call this a "tuned" setting. We fixed these decisions for the three test settings. Note that no other aspect of the pipeline depends on the ATE. All the choices in this color are a function of the experimental design. Hence, the pipeline can be easily adapted to any new setting, given its experimental design and without knowledge of the true ATE.Prompt 1: Semi-synthetic report generation (Hillstrom) You are a user who used a website for online purchases in the past one year and want to share your background and experience with the purchases on social media.The following are attributes that you have, along with their descriptions.The following dictionary describes your personality with levels (High or Low) of the Big Five personality traits. > {traits} ## Your Instructions Write a social media post in first-person, accurately describing the information provided. Write this post in the tone and style of someone with the given personality traits, without simply listing them. Only return the post that you can broadcast on social media and nothing more.You are an expert researcher looking around reddit for posts/comments describing the effect of a treatment on weight loss or blood sugar level experienced by the author.## Problem Setting > You are interested in self-reported effects of a treatment on a user who took the treatment themselves. You want to be able to answer some or all of the following questions from the text of the post or comment: 1. Which treatment did the user take? 2. What change did they observe in their weight due to this treatment, and during what duration did they observe this change? You are a medical assistant, helping a doctor structure posts about weight loss treatments found on Reddit. Your task is to use the self-report to interpret accurate information about the following fields and store them in a JSON dictionary. ## Your Instructions I will provide a post along with its subreddit name, title and date of creation. You must return a valid JSON dictionary containing the following keys along with the corresponding accurate information: "start_weight": Numerical value for the user's starting weight, before starting the treatment described, sometimes referred to as SW. "end_weight": Numerical value for the user's current or final weight, at the end of the treatment regime, sometimes referred to as CW. "weight_unit": Units in which weight is reported: "kg" or "lb". "weight_change": Numerical value for net change in the user's weight. Use a postive sign to indicate weight gain and negative sign for weight loss.Leave blank if it is not possible to infer the change in weight. "percentage_weight_change": Numerical value for percentage reduction in user's weight relative to their start weight. Use a postive sign to indicate weight gain and negative sign for weight loss. Leave blank if it is not possible to infer the percentage. "drug_type": Treatment taken by the user: "Semaglutide", "Tirzepatide" or "Other". Semaglutide includes Ozempic, Wegovy or Rybelsus. Tirzepatide includes Mounjaro or Zepbound.Assign a valid value to each key above. If you can't find the required information in the post, assign the value "Unknown". Remember to ONLY return a valid JSON with ALL of the above keys and their accurate values.As a medical assistant aiding a physician, your role involves examining Reddit posts discussing weight loss treatments and interpreting self-reported information accurately. This data needs to be translated into a well-structured JSON dictionary, with the most suitable option chosen from the choices provided.Assume a user shares a post along with related data. Your job will be to create a dictionary comprising of the following keys as well as their matching accurate data: {covariate descriptions}Please ensure you fill all the fields and that you choose a valid value for each key from the provided options. Unfilled fields are not allowed. In instances where certainty is impossible, make your best educated guess, or provide the "Unknown" value. Note that your completed task should ONLY yield a JSON containing ALL the listed keys alongside their accurate values.Here Here is an entry that the patient wrote about themselves, which may be useful for your task. ## Input {report} ## Output > Prompt 6: Conditional distribution inference (Weight Loss) You are a medical assistant aiding a physician. I am going to ask you a few multiple choice questions about some posts I just found online. Please, answer accordingly.## Your Instructions I will give you a post about an individual's experience with a treatment and its effect on their weight, and a few questions with their correct answers, followed by additional multiple choice questions and options to choose from. Pick the right answer. We provide further details about the treatments, outcomes and covariates, along with inclusion criteria and discrete categories used in our experiments, for each dataset in tables 3 and 4. We refer the reader to Figure 10 for visualizations of the propensity score corrected average treatment effect on covariates for all test clinical settings. For each setting, our estimated propensity score balances each covariate, far better than a uniform propensity distribution would.Since covariates may take values at different scales, we computed the standard mean difference (SMD) across cohorts for each covariate X (i) [16] , given by:EQUATIONwhere X (i) (1) -X (i) (0) estimates the average treatment effect on X (i) , using propensity score weighting, and var(•) denotes sample variance. While we rely on domain expertise to define the confounder set for each setting such that necessary causal assumptions are satisfied, unobserved confoundedness remains a challenge. We followed strategies in Lu and Ding [29] to analyze the sensitivity of our ATE estimates to the degree of unobserved confoundedness. Specifically, the idea is to introduce sensitivity parameters, which quantify the degree of unobserved confoundedness. In our case, sensitivity parameters are the density ratio between likelihood of each potential outcome in the treated vs. untreated group. The ATE estimate is non-increasing in the sensitivity parameters. So, for positive ATEs, we are looking for the largest sensitivity parameters that maintain the positivity of the ATE, which tells us the degree of unobserved confoundedness that an estimator is robust to. Table 5 shows that the direction (sign) of NATURAL IPW ATE estimates for the Semaglutide vs. Tirzepatide dataset change from positive to negative at large values of sensitivity parameters, implying that they are robust to large degrees of unobserved confoundedness.ε 0 (X) = E[Y (0)|T = 1, X] E[Y (0)|T = 0, X] and ε 1 (X) = E[Y (1)|T = 1, X] E[Y (1)|T = 0, X] ,We further investigated the importance of each confounder as suggested by the leave-one-covariateout approach in Section 4 of Lu and Ding [29] , by dropping that covariate as if it is an unobserved confounder and measuring the corresponding worst-case (over all possible values of the remaining covariates) sensitivity parameters with the remaining covariates. Figure 11 shows contour lines depicting NATURAL IPW ATE estimates at different values of sensitivity parameters for all our real-world datasets. Orange regions denote sensitivity parameter values for which the sign of the estimate ATE does not change, while blue regions denote the values that flip the direction of the estimate. Stars show sensitivity parameters for each covariate in the different settings. Hence, this estimator is sensitive to the covariate set and all covariates are important in the case of Semaglutide vs. Tirzepatide, while most are important for Erenumab vs. Topiramate. The direction of ATE estimates is not sensitive to covariates in the case of Semaglutide vs. Liraglutide and OnabotulinumtoxinA vs. Topiramate, implying that the estimated causal effects could only be explained away by an unobserved confounder that is stronger than all observed confounders. We are interested in an ATE conditioned on inclusion criteria denoted I,EQUATIONLet τ (X, T, Y ) be a function such thatEQUATIONFor example, τ (I) can estimated by the IPW estimator,τ (X, T, Y ) = T Y e(X) - (1 -T )Y 1 -e(X) ,because the P (T = 1|X = x, X ∈ I) = P (T = 1|X = x) for all x ∈ I. Throughout this section, we operate under Assumptions 3 and 4 and assume that the LLM gives us access to the true data-generating conditionals.The law of total expectation gives us an estimator that can operate on samples of reports R:τ (I) = E X,T,Y [τ (X, T, Y ) | X ∈ I] = E R|X∈I [E X,T,Y [τ (X, T, Y ) | X ∈ I, R]] = r P (R = r|X ∈ I)E X,T,Y [τ (X, T, Y ) | X ∈ I, R] = r P (R = r) P (X ∈ I|R = r) P (X ∈ I) E X,T,Y [τ (X, T, Y ) | X ∈ I, R = r] = E R P (X ∈ I|R) P (X ∈ I) E X,T,Y [τ (X, T, Y ) | X ∈ I, R] .To summarize, we have the identities:EQUATION= E R P (X ∈ I|R) P (X ∈ I)EQUATIONWe prompted LLAMA2-70B with descriptions of the inclusion criteria and each report to estimate P (Xi∈I|Ri) P (Xi∈I) , similar to other conditional distributions described in section 4, and marginalized over reports for the denominator.It is also possible to avoid this weight computation at the expense of additional structural assumptions on the data. We discuss these conditions below and show corresponding results for NATURAL estimators in table 6 .Let X ∈ R D and let us make the following assumption on the inclusion criteria: d=1 . Recall from section 4 that inclusion-based filtering leaves us with reports whose covariates are either "known" and satisfy their criteria or Unknown. We also have the value of the known covariates. Let K ∈ {0, 1} D be the binary vector of variables K d that indicate whether the covariate X d is found to be "known" for a random report R. Let X K = (X d : K d = 1) be the vector of length d K d holding the values of the known covariates. For ease of notation, define the event that the known covariates satisfy their criteria and the event that the unknown covariates satisfy their criteria:EQUATIONNotice that {X K ∈ I K } ∩ {X 1-K ∈ I 1-K } = {X ∈ I}. Thus, after the filtering steps we haveEQUATIONwith the guarantee that the knowns satisfy their inclusion criteria, {X Ki i ∈ I Ki }. Formally, assuming that the LLM computes the true conditional distribution of the data-generating process (Assumption 4), this gives us data sampled i.i.d. from P (R = r, K = k, X k = x k |X K ∈ I K ). Note, that we are assuming the existence of an additional ground-truth random variable K in the data-generating process that describes whether a covariate is knowable from a report. Here, we show how to estimate τ (I) from this dataset of filtered reports using importance sampling, under the following assumption: Assumption 6 (Satisfaction of I by Unknown covariates) Satisfaction of inclusion criteria by unknown covariates is conditionally independent of the report and the known covariates given satisfaction of inclusion criteria by known covariates, i.e., for all r, k, x k :EQUATIONOne can derive the following identity in a similar fashion as eq. ( 25)EQUATIONFrom assumption 6, the fraction above simplifies to 1, leaving us with the following estimatorEQUATIONwhich can be computed from the information available at the end of filtering. In practice, we do not condition the LLM on K i in the final inference step (vi), which amounts to an additional conditional independence assumption:Assumption 7 (Conditional independence of knowable covariates) K ⊥ ⊥ (T, Y ) | (X, R).Equation (31) above can now be more efficiently estimated by prompting the LLM to extract covariates under the constraints of the inclusion criteria for each report in our filtered dataset, and then following the remaining steps in the pipeline to an ATE estimate. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: The paper develops an innovative, causal-aware way to estimate causal effects with the help of large language models. Overall, the reviewers including myself found the idea fresh and interesting. Here, I noted the term 'causal-aware' in the sense that the method does not have a theoretical grounding as many other papers in the causal inference track but offers a clever way together with strong empirics. 

The reviewers had some comments on the presentation (which can hopefully addressed in the camera-ready version). The question regarding more baseline is a difficult one, given that the task is fairly novel, so that simply baselines are not available (which is often the case in causal inference). 

Overall the reviewers find the contribution valuable and the authors have made a significant effort in addressing clarifying questions and concerns from reviewers. I do believe all additional experiments should be incorporated in the main paper. As a result, I think that the paper is ready to be accepted.

Reviews: review: paper_summary: paper_summary: The paper seeks to use LLMs for a workflow that extracts structured variables
from free text, filters the dataset, and then computes a causal estimate using
the imputed variables. The paper applies this methodology to two synthetic and
four real-world datasets and shows that it produces estimates that are
comparable to those from known ground truth or RCT estimates.
strength_weakness: strength_weakness: The paper at almost every turn relies on prompting an LLM and implicitly
assuming that the model returns data from a desired distribution. In general,
this will introduce measurement error which could systematically bias the
overall method's estimates. See for example [1] and [2] below.

The synthetic data evaluation is fairly limited. It would be nice to at least
understand how performance varies as sample size increases from a few hundred
examples to many thousands (especially as the IPW oracle baselines are not
particularly good). This would also allow you to stress-test the different
assumptions being made. If you have synthetic data in which reports for certain
patients are systematically truncated or censored, or where different groups
write reports in different languages or styles, how does that affect the
different steps (i-vi)?

[1] Ashwin, Julian, Aditya Chhabra, and Vijayendra Rao. "Using Large Language
Models for Qualitative Analysis can Introduce Serious Bias." Development
Research (2023).

[2] Egami, Naoki, et al. "Using imperfect surrogates for downstream inference:
Design-based supervised learning for social science applications of large
language models." Advances in Neural Information Processing Systems 36
(2024).
questions: questions: Line 267-268: What is the point of sampling a persona from Big Five personality
traits? How does this simulate realism?

Line 147-149 says "if all reports are the constant, empty string ... we have
full access to the true observational joint density." Am I correct that this is
a necessary assumption for the overall method to work in theory, but the actual
implementation (steps (v) and (vi) in lines 206-220) would not work if all
reports are the constant, empty string? This may be obvious, but it could be
worth clarifying that when you say "we cannot formally guarantee that [our
method] satisfies Assumptions 3 and 4" you mean the form of Assumption 4 that
does not involve all reports being empty strings.

In Tables 1 and 2, which LLMs are used? LLAMA2-70B? Overall, it would be
helpful to more clearly label which LLMs are used when. In Line 204, the paper
says that the GPT-4 API sufficed for filtering; was GPT-4 used anywhere else?
How was the decision made of which LLM to use where?

It might be helpful to have a figure or appendix that walks through every step
(the paper does this for many steps, just not all in one place) in which an
LLM is queried and discusses what assumptions are being made.
limitations: limitations: Overall, the paper does a good job of not overclaiming. However, see weakness 1
above. I would also suggest the authors discuss the challenges of using RCT
estimates to validate observational studies -- there are many reasons why those
estimates might not align (selection bias, dropout, etc).
paper_summary: paper_summary: Estimating causal effects is costly and time-consuming. The authors propose  to use large language models (LLMs) to mine unstructured text data for causal effect estimation. This paper introduces NATURAL, a family of causal effect estimators using LLMs to process unstructured text data. This seems to be a good application of LLMs for the task of causality.
strength_weakness: strength_weakness: - LLMs might inherit biases present in their training data or hallucinate. 

 - Social media users might not reflect the demographics or behaviors of the broader population.

- Finetuning the model to one domain may not lead to good generalisation in others, during fine-tuning does the LLM learn inductive biases regarding the "topic" or task?
questions: questions: -  How do the authors make sure that all of the confounders are accounted ? I can imagine there to be many confounders mentioned in the text.

- There could be a bias in social media text, I thought people would report negative cases more often than positive, did you observe any such biases?
limitations: limitations: Mentioned
paper_summary: paper_summary: The paper introduces a family of causal effect estimators named NATURAL, designed to use LLMs for mining causal effect estimates from observational text data. The authors address the challenge of automating data curation and using LLMs to impute missing information, presenting a method that conditions on structured variables to assist in computation of causal effect estimators. The authors develop six datasets (two synthetic and four real) to evaluate their method, and demonstrate that NATURAL estimators produce causal effect estimates within close range of ground truth from randomized trials.
strength_weakness: strength_weakness: 1. The original claims made in the abstract+intro seems to be too grandiose, but in fact it is about the use of LLMs to classify variables of interest from the text. I would change the framing of the paper a bit to better reflect that. It would make the claims of the paper less objectionable and would better demonstrate the usefulness of NATURAL.

2. There is a vast literature in the social sciences on recovering interpretable variables directly from text for estimating causal effects from observational data. Most of these approaches rely on probabilistic approaches like topic models, but recently there’s growing interest in doing so automatically with LLMs. The authors should better address this literature, and compare their approach to relevant methods if possible.

3. The approach is really cool, but like any ML-pipeline approach it surely introduces errors at each step of the process (see for example Egami et al @ NeurIPS 2023). Each error will then bias the causal effect estimation. From the results of the paper I’m convinced empirically your approach works well, but would I be able to understand whether it produces accurate estimates on a new dataset? A deeper analysis and perhaps a way to propagate error/uncertainty to the estimation would be very helpful here.
questions: questions: 1. Authors explain that as part of their process, they “filter for reports that are likely to conform to the experimental design”. Could this introduce bias?

2. All of the “classical strategies” used assume no-hidden confounding. Could you imagine how someone would use an LLM with a more complex identification strategy (say IV, or RDD)? You claim at the end of the intro that you can anticipate someone doing so, I wonder if you can elaborate.
limitations: limitations: The authors have addressed the limitations of their work, acknowledging the dependency on the quality and calibration of LLMs and the potential inaccuracies in approximating the required conditional distributions. They emphasize that NATURAL estimators should not be used for high-stakes decision-making without experimental validation and stress the need for domain expert involvement to ensure causal assumptions are met.
paper_summary: paper_summary: This work aims to the causal effect estimation from unstructured observational text data, proposing a pipeline based on LLM to extract the treatment, covariate and outcome from the unstructured data by LLM. It's a interesting exploration for causla inference and the pipeline process of the inference method is quite standard.
strength_weakness: strength_weakness: (1) some details are missing, leading to the poor readability.

(2) Some equations are not correct (based on my understanding).

(3) The presnetation of the whole methodology is not good enough, alought I am not an expert of LLM, I know how LLM works. However, i can not imagine how the proposed pipeline work and how it estimate ATE very clearly. 

(4) The evaluation part is kind of weak, given that baselines are too limited and simple.
questions: questions: 1.there are some claims that are not well expalined, leading to confuse people. For exampel, "value may be recouped from outcomes that would otherwise be lost" this claim sounds good but i don't what does it mean here.

2.the overall idea of this work to utilize LLM to explore the causal effect estimation from unstructured data is good, i like it. But the writing can not match the good idea. For example, line 54-59, i have no idea about what the authors try to express, The transition from sentence to sentence is too stiff 

3.The binary and discrete assumption about the outcome Y and covariates X is kind of too strict. Typically, one can assume the teatment assignment is binary but outcome and covariates are continous. It's not clear why this work adopt such assumption. if so, the applying scale of the proposed method would be limited. also, the assumption 4 is also too strong.

4.in the  outcome predictor eq. (9), why it is P(Y=1|R_i,X_i,T_i) instead of P(Y=t|R_i,X_i,T_i)? I don't understand. When t=0, the outcome predictor still ouput the expectation over P(Y=1|xxx)?

5.the description how to extrct the covariates and their corresponding values is not clear.  How to determine the feature values? For each report, the set of covarivates is the same, right? Can you present a specific example? what does each feature value mean in the real-world? The readability of the method section is poor.

6.In the evaluation table 1, the "IPW (Structured) " is a baseline or ground truth? there is no any discussion in the main body about this baseline and i am so confused about the discussion below because of it, given that the "IPW (Structured) " performs the best.

7.how does the ground truth of ate comes from? Can you clarify this point?

by the way, i like the idea of this work, but the presentation of this paper is really poor. thus if the above questions can be addressed well, i will consider to increase the score.
limitations: limitations: authors have adequately addressed the limitations.

Comments: title: Thank you for rebuttal
Rebuttal by Authors (continued)
Rebuttal by Authors (continued)
Rebuttal by Authors (continued)
Rebuttal by Author (continued)
comment: Dear Reviewer p9SY, 

Thanks for your review! The discussion is ending soon, and it'd be greatly appreciated if you could acknowledge the author's rebuttal and update your review if necessary.

Thank you!
AC
Thank you for your comments on our rebuttal! 

> I think it would be very useful to show that the IPW (structured) oracle converges to an arbitrarily low RMSE as sample size increases.

Thanks for clarifying your point further! We can demonstrate the convergence of the IPW (Structured) oracle to ground truth on a larger number of datapoints. Using all 2000 data points of the observational Hillstrom dataset, the oracle gives an ATE estimate of 6.08 (on a scale from -100 to 100), as compared to the ground truth of 6.09. To test and compare to NATURAL methods on even larger samples of data, we would need to generate more synthetic reports.

> What explains the mismatch between the low RMSE in Figure 3 (close to $10^{-2}$ on both synthetic datasets) and the N-FULL RMSEs in Table 1?

Thank you for catching this - the apparent mismatch is due to the scale on which the results (ATE and RMSE) are reported. The tables use a scale from -100 to 100, while the plots use -1 to 1. To maintain consistency, we will update the plots to use the same scale as the tables in the final version of our paper.

> Can you clarify the specifics of the ATE (%) versus RMSE metrics in Table 1? How are the standard errors being calculated? Are there any insights from the mismatches between the two metrics (e.g., N-MC OI on Hillstrom and N-OI on RetailHero have closer average ATEs to the ground truth but worse RMSE)?

The ATE (%), standard error and RMSE are all calculated over 10 runs, sampling 80% of the data points without replacement in each one. The difference in performance based on mean ATE vs RMSE can be explained by the standard deviation of the estimates. For instance, N-MC OI on Hillstrom estimates average ATE closer to the ground truth, but has higher variance across runs.

> Perhaps the easiest way to do this would be to have a “truncation model” that’s conditional on covariates (e.g., $\mathbf{P}(\text{Truncation}) = \text{logit}(w_0 + w_c\cdot \text{Channel} + w_z\cdot\text{ZipCode})$ or similar) where if you sample Truncation=1, you truncate the corresponding example’s report to contain only the first $k$ tokens. The real-world analogue I think is likely applicable is that there may be systematic biases in terms of the availability or verbosity of certain groups’ reports. 

Thanks for this suggestion! It makes complete sense to try and simulate varying verbosity as a function of individuals’ covariates. Since truncating to the first $k$ tokens for some $k$ will likely lead to incomplete words or sentences at the end of the report, these reports might not be very realistic data. Instead, we could use the “Openness” trait from the Big Five and set its probability to be a function of covariates (as in your example with channel and zip code). This could be a realistic way to vary verbosity because we found “Low Openness” to generate short and succinct reports, and vice versa. We are excited to explore these variations in future work!
thank you addressing my comments!
i remain supportive of this paper's acceptance.
I appreciate the authors’ careful response. The planned additions and clarifications to the paper will strengthen it.

> We would like to clarify that the oracle estimates in Table 1 are denoted by “IPW (Structured)” […] oracle estimates to be very close to the ground truth ATE

This was clear to me when writing my original review, but from my perspective Table 1 shows “closer” but not “very close” estimates. Given the many limitations of synthetic data (in general and specifically in the comparison between Hillstrom and Retail Hero compared to the domain to which you apply this method), I think it would be very useful to show that the IPW (structured) oracle converges to an arbitrarily low RMSE as sample size increases. That then provides a better comparison for how well the NATURAL methods are empirically performing compared to a method that theoretically should and practically does achieve unbiased estimation.

Two follow-up questions that I didn’t catch in my original reading:
- What explains the mismatch between the low RMSE in Figure 3 (close to $10^{-2}$ on both synthetic datasets) and the N-FULL RMSEs in Table 1?
- Can you clarify the specifics of the ATE (%) versus RMSE metrics in Table 1? How are the standard errors being calculated? Are there any insights from the mismatches between the two metrics (e.g., N-MC OI on Hillstrom and N-OI on RetailHero have closer average ATEs to the ground truth but worse RMSE)?

> We are also open to further discussion on how to investigate data that is biased in other ways, such as being “truncated”, if the reviewer has concrete suggestions.

Perhaps the easiest way to do this would be to have a “truncation model” that’s conditional on covariates (e.g., $\mathbf{P}(\text{Truncation}) = \text{logit}(w_0 + w_c\cdot \text{Channel} + w_z\cdot\text{ZipCode})$ or similar) where if you sample Truncation=1, you truncate the corresponding example’s report to contain only the first $k$ tokens. The real-world analogue I think is likely applicable is that there may be systematic biases in terms of the availability or verbosity of certain groups’ reports. This is a somewhat heavy-handed way to imitate that, but it should be easy to implement.

> helped simulate diversity in writing style, tone and report lengths, and hence made the synthetic data more realistic.

Thanks for the clarification. I was interpreting this as a claim that the Big Five traits were specifically relevant to these synthetic datasets, rather than this is a general way to introduce diversity in generation.
I would like to thank the reviewer for replying to my concerns regarding biases and confounders. I have increased my score.

Best regards,
> (Q6) In the evaluation table 1, the "IPW (Structured) " is a baseline or ground truth? there is no any discussion in the main body about this baseline and i am so confused about the discussion below because of it, given that the "IPW (Structured) " performs the best.

Thank you for pointing this out! We will clarify this result in the experiments section of the main paper. 
- IPW (Structured) corresponds to an oracle estimator that has access to the true structured treatments, outcomes and covariates, which can be plugged into the IPW estimator to estimate ATE. 
- Hence, it is not directly comparable to methods that do not have access to structured data, like NATURAL. We expect it to provide an upper bound on the performance, as verified in the results of Table 1.

> (Q7) how does the ground truth of ate comes from? Can you clarify this point?

As mentioned in lines 68-69 and again in lines 281-284, for every dataset we consider, there exists a randomized controlled trial which provides a gold standard estimate of the ATE. We treat this ATE value as “ground truth”.

We thank the reviewer for bringing up questions about the parts of our paper they found confusing. We believe the clarifications above improve the readability of our paper. We hope our responses address all their concerns and that they will consider increasing their score accordingly. We are more than happy to provide any further clarifications during the discussion period.
> (Q2) All of the “classical strategies” used assume no-hidden confounding. Could you imagine how someone would use an LLM with a more complex identification strategy (say IV, or RDD)? You claim at the end of the intro that you can anticipate someone doing so, I wonder if you can elaborate.

This is a great question! The general strategy behind deriving NATURAL estimators from classical strategies is as follows:
- Identify all (structured) variables required to compute the classical estimator. In the case of the Instrumental Variables (IV) approach, this includes variables representing the instrument $Z$, in addition to the $(T, Y)$ that our methods consider.
- Next, collect natural language reports that contain such information. For instance, electronic health records or clinical notes at hospitals may contain information relevant to the IV setting.
- Extract the required conditional distributions from an LLM using these natural language reports.
- Finally, marginalize out $(T, Y, Z)$ and average over reports to compute an ATE. For example, IV estimators are used to measure ATE conditional on compliance, which is a ratio of two treatment effects (see chapter 21 of Ding (2024)) and can be estimated consistently given the reports. 

[1] Ding, P. (2024). A first course in causal inference. CRC Press.

Again, thank you for your very thoughtful review! We hope we have addressed all your questions, but please let us know if we can clarify anything further.
> (Q2) There could be a bias in social media text, I thought people would report negative cases more often than positive, did you observe any such biases?
- Since this relates to the concern about selection bias above, please see our response to W2 above.  
- Since our experiments compare similar treatments, e.g., comparable availability, we believe the probability of a user reporting their experience is approximately equal in both. Our empirical results suggest low bias, relative to estimates from a real-world RCT.

Thank you for the great suggestions and questions! We believe our paper is significantly improved by the edits and clarifications above. We would be happy to clarify anything further in the discussion period.
> I would also suggest the authors discuss the challenges of using RCT estimates to validate observational studies -- there are many reasons why those estimates might not align (selection bias, dropout, etc).


Thank you for this suggestion! Since these are important considerations for future work, we will expand Section 5 of the paper to elaborate on the limitations of NATURAL.

Thank you for all your suggestions, which we believe will greatly improve our paper! Please let us know if there are any further questions we can address.