Proximal Causal Inference With Text Data

Authors: jacob m chen
rohit  bhattacharya
katherine a keith
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 23:50:21

Abstract: Recent text-based causal methods attempt to mitigate confounding bias by estimating proxies of confounding variables that are partially or imperfectly measured from unstructured text data. These approaches, however, assume analysts have supervised labels of the confounders given text for a subset of instances, a constraint that is sometimes infeasible due to data privacy or annotation costs. In this work, we address settings in which an important confounding variable is completely unobserved. We propose a new causal inference method that uses two instances of pre-treatment text data, infers two proxies using two zero-shot models on the separate instances, and applies these proxies in the proximal g-formula. We prove, under certain assumptions about the instances of text and accuracy of the zero-shot predictions, that our method of inferring text-based proxies satisfies identification conditions of the proximal g-formula while other seemingly reasonable proposals do not. To address untestable assumptions associated with our method and the proximal g-formula, we further propose an odds ratio falsification heuristic that flags when to proceed with downstream effect estimation using the inferred proxies. We evaluate our method in synthetic and semi-synthetic settings---the latter with real-world clinical notes from MIMIC-III and open large language models for zero-shot prediction---and find that our method produces estimates with low bias. We believe that this text-based design of proxies allows for the use of proximal causal inference in a wider range of scenarios, particularly those for which obtaining suitable proxies from structured data is difficult.

Main paper: 
 introduction: 
 Data-driven decision making relies on estimating the effect of interventions, i.e. causal effect estimation. For example, a doctor must decide which medicine she will give her patient, ideally the one with the greatest effect on positive outcomes. Many causal effects are estimated via randomized controlled trials-considered the gold standard in causal inference; however, if an experiment is unfeasible or unethical, one must use observational data. In observational settings, a primary obstacle to unbiased causal effect estimation is confounding variables, variables that affect both the treatment (e.g., which medicine) and the outcome. 1 and T pre 2 as inputs into LLM-1 and LLM-2, respectively, to infer zero-shot proxies Z and W . ( 7) If the proxies fail our odds ratio heuristic, analysis stops. (8, 9, and 10) Else, we use the proximal g-formula implied by the casual DAG to estimate the causal effect. causal estimates, including medicine (Zeng et al., 2022) , the behavioral social sciences (Kiciman et al., 2018) , and science-of-science (Zhang et al., 2023) . See Keith et al. (2020) ; Feder et al. (2022) ; Egami et al. (2022) for general overviews of text-based causal estimation. If all confounders are directly observed, then causal estimation is relatively1 straightforward with backdoor adjustment (Pearl, 2009) . However, known confounders are often unobserved. In such scenarios, researchers typically use supervised classifiers to predict the confounding variables from text data, but these text classifiers rarely achieve perfect accuracy and measurement error must be accounted for. To address this, another line of work has developed post-hoc corrections of causal estimates in the presence of noisy classifiers (Wood-Doughty et al., 2018; Fong and Tyler, 2021; Egami et al., 2023; Mozer et al., 2023) . These approaches, however, require ground-truth labels of the confounding variables for a subset of instances, a constraint that is not always feasible due to privacy restrictions, high annotation costs, or lack of expert labor for labeling.Our work fills this gap. We address the causal estimation setting for which a practitioner has specified a confounding variable that is truly unmeasured (we have no observations of the variable), but unstructured text data could be used to infer proxies. For this setting, our method combines proximal causal inference with zero-shot classifiers.Proximal causal inference (Miao et al., 2018; Tchetgen Tchetgen et al., 2020; Liu et al., 2024) can identify the true causal effect given two proxies for the unmeasured confounder that satisfy certain causal identification conditions. A major criticism of this method is that it can be difficult to find two suitable proxies among the structured variables; however, we conjecture that unstructured text data (if available) could be a rich source of potential proxies.In our proposed method, summarized in Figure 1 , we estimate two proxies from text data via zero-shot classifiers, i.e. classifiers that perform an unseen task with no supervised examples. In subsequent sections, we expand upon this method and its assumptions and empirically validate it on synthetic and semi-synthetic data with real-world clinical notes. Since large pre-trained language models (LLMs) have promising performance on zero-shot classification benchmarks (Yin et al., 2019; Brown et al., 2020; Wei et al., 2021; Sanh et al., 2021, inter alia) , we use LLMs to infer both of the proxies in our experimental pipeline. Our combination of proximal causal inference and zero-shot classifiers is not only novel, but also expands the set of text-specific causal designs available to practitioners. 2 In summary, our contributions are • We propose a new causal inference method that uses distinct instances of pre-treatment text data, infers two proxies from two different zero-shot models on the instances, and applies the proxies in the proximal g-formula (Tchetgen Tchetgen et al., 2020) . • We provide theoretical proofs that our method satisfies the identification conditions of proximal causal inference and prove that other seemingly reasonable alternative methods do not. • We propose a falsification heuristic that uses the odds ratio of the proxies conditional on observed covariates as an approximation of the (untestable) proximal causal inference conditions. • In synthetic and semi-synthetic experiments using MIMIC-III's real-world clinical notes (Johnson et al., 2016) , our odds ratio heuristic correctly flags when identification conditions are violated.When the heuristic passes, causal estimates from our method have low bias and confidence intervals that cover the true parameter; when the heuristic fails, causal estimates are often biased. Figure 2 : Causal DAGs (a) depicting unmeasured confounding and (b) compatible with the canonical assumptions used for proximal causal inference (Tchetgen Tchetgen et al., 2020) . 
problem setup and motivation: 
 To motivate our approach, imagine we are an applied practitioner tasked with determining the effectiveness of thrombolytic (clot busting) medications relative to blood thinning medications to treat clots arising from an ischemic stroke. Such medications are usually administered within three hours of the stroke to improve chances of patient recovery (Zaheer et al., 2011) . Given the urgency and the short treatment window, running a randomized experiment to compare these drugs is difficult. Left with only observational data, we examine electronic health records (EHRs) from a database like MIMIC-III (Johnson et al., 2016) .We formalize our causal estimand as follows: let A denote a binary treatment variable corresponding to clot busting (A = 1) or blood thinning (A = 0) medication, and let Y denote measurements of the D-dimer protein in the patient's blood which directly measures how much of the clotting has dissolved. In do-calculus notation (Pearl, 2009) , the target causal estimand is the average causal effect, ACE:= E[Y | do(A = 1)] -E[Y | do(A = 0)].Examining the EHRs, we find potential confounders (in structured tabular form), including biological factors, such as age, sex, and blood pressure, as well as socio-economic factors, such as income. We denote the observed confounders as the set C. However, we are worried about biased causal effects because atrial fibrillation (irregular heart rhythms) is an important confounder corresponding to a pre-existing heart condition that is not recorded in the structured data. We denote this unmeasured confounder as U , and assume for the rest of this paper that C and U form a sufficient backdoor adjustment set with respect to A and Y (Pearl, 1995) . Figure 2 (a) depicts this problem setup in the form of a causal directed acyclic graph (causal DAG) (Spirtes et al., 2000; Pearl, 2009) . With the presence of U , it is well known that adjusting for just the observed confounders via the backdoor formulac (E[Y |A = 1, c] -E[Y |A = 0, c]) × p(c)will give a biased estimate of the ACE (Pearl, 1995) . In response to this issue, we consider work that uses proxy variables of the unmeasured confounder. However, we are subject to the following restriction:(R1) We do not have access to the value of U for any individuals in the dataset.This kind of restriction is common in healthcare or social science settings when data privacy issues, high costs, or lack of expert labor make analysts unable to hand-label unstructured text data. We elaborate in Appendix A.In such cases, we turn to using proxies for U . Pearl (2010) proposed a method for obtaining unbiased effect estimates with a single proxy W under the assumption that p(W |U ) is known or estimable, which are impossible for us under (R1). However, a more recent line of work, building from Griliches (1977) and Kuroki and Pearl (2014) , called proximal causal inference (Miao et al., 2018; Tchetgen Tchetgen et al., 2020) is able to identify the true causal effect as long as the analyst proposes two proxies W and Z that satisfy the following independence conditions: A canonical example of proxies that satisfy these conditions is shown in Figure 2 (b) 3 . In addition to these independence relations that impose the absence of certain edges in the causal DAG, e.g., no edge can be present between Z and W to satisfy (P1), there is an additional completeness condition that imposes the existence of U → Z and U → W . This condition is akin to the relevance condition in the instrumental variables literature Angrist et al. (1996) and ensures that the proxies W and Z exhibit sufficient variability relative to the variability of U .EQUATION(P4) Completeness: for any square integrable function v(•) and for all values w, a, c, we haveE[v(U ) | w, a, c] = 0 ⇐⇒ v(U ) = 0, and E[v(Z) | w, a, c] = 0 ⇐⇒ v(Z) = 0.Intuitively, these completeness conditions do not hold unless Z and W truly hold some predictive value for the unmeasured confounder U . See Miao et al. (2018) for more details on completeness.EQUATIONwhere h(a, w, c) is the "outcome confounding bridge function" that is a solution to the equation et al., 2018) . Although the existence of a solution is guaranteed under (P1-P4), solving it can still be difficult. However, there exist simple two-stage regression estimators for the proximal g-formula (Tchetgen Tchetgen et al., 2020; Mastouri et al., 2021) that we make use of in Section 5 once we have identified a valid pair of proxies. This brings us to the primary criticism of proximal causal inference.E[Y |a, z, c] = w h(a, w, c) × p(w|a, z, c) (MiaoPrimary criticism When using proximal causal inference in practice, how do we find two proxies W and Z among the structured variables such that they happen to satisfy all of (P1-P4)? We often cannot, at least not without a high degree of domain knowledge. 4 Furthermore, empirically testing for any of (P1-P3) in general is, by definition of the problem, not possible because doing so requires complete access to the unobserved confounder U .Our approach Instead, in this work, we propose relying on raw unstructured text data (e.g., clinical notes) in an attempt to infer proxies that satisfy (P1-P4) by design.Returning to our motivating problem, our approach applies classifiers zero-shot-since we have no training data with U given (R1)-to two distinct instances of pre-treatment clinical notes of each patient and obtains two predictions, W and Z, for atrial fibrillation (our U ). In order to make it more feasible to estimate the ACE from data, we make the following two relatively weak assumptions:(S1) The unmeasured confounder U between A and Y can be specified as a binary variable.(S2) The text only causes W and Z (and no other variables).Assuming (S1) simplifies estimation since text classification typically performs better empirically than text regression (Wang et al., 2022) . Assumption (S2) asserts that the text data considered serves as a record of events rather than actionable data. 53 One could also add the edges W → Y and Z → A to Figure 2 (b), but these relations will not show up in our text-based setting. Shpitser et al. (2023) also propose a general proximal identification algorithm that is compatible with other causal DAGs, but we focus on the canonical proximal learning assumptions stated in Tchetgen Tchetgen et al. (2020) . 4 For instance, suppose we aim to find proxies (in the structured variables) of atrial fibrillation (U ) and have access to shortness of breath and heart palpitations. Although these proxies seem reasonable, we show how they may violate the proximal causal inference conditions. Patient complaints about shortness of breath may affect which medication a clinician prescribes (A); hence, using shortness of breath as the proxy W violates (P2). Furthermore, a lack of oxygen resulting from shortness of breath may affect the healing of blood clots, thus influencing measurements of the D-dimer protein Y . In this case, using shortness of breath as the proxy Z violates (P3). Shortness of breath can also sometimes be a symptom of heart palpitations, violating (P1). 5 We present an example of how to relax this assumption in Appendix B. We leave to future work more complicated scenarios, e.g., if the reader's perception of text differs from the writer's intent Pryzant et al. (2021) .A Y U C T Z W (a) A Y U C T pre Z W (b) A Y U C T pre 1 T pre 2 Z W (c) A Y U C T pre 1 T pre 2 Z W (d)Figure 3 : Causal DAGs depicting several different scenarios for inferring text-based proxies. Edges with different colors and patterns, e.g., T Z and T W, indicate that different zero-shot models were used. Our final recommended method is based on (d). 
designing text-based proxies: 
 In this section, we describe our method for designing text-based proxies. In doing so, we describe various "gotchas," pitfalls in attempting to use these text-based proxies in causal effect estimation. We describe each gotcha and explore how they lead us to our final recommended design, given by Figure 3(d) . Our empirical results in Section 5 demonstrate, as expected, that these pitfall approaches result in biased causal effect estimates.Gotcha #1: Using predictions directly in backdoor adjustment. Suppose we try to avoid the complications of proximal causal inference by using the predictions from one of our zero-shot models, say W , as the confounding variable itself. Proposition 1. Using a proxy W in the backdoor adjustment formula results in biased estimates of the ACE in general.Proof. If W ̸ = U for some subset of instances, there remains an open backdoor path through U , and the ACE remains biased as (Pearl, 1995) .U,C (E[Y | A = 1, U, C] -E[Y | A = 0, U, C]) × p(U, C) ̸ = W,C (E[Y | A = 1, W, C] -E[Y | A = 0, W, C]) × p(W, C) in generalThe most straightforward way to obtain unbiased results under Gotcha #1 is to have 100% accuracy between W and U , a scenario that is extremely unlikely in the real world. Further, under (R1)6 , we cannot measure accuracy at inference time since we do not have any observations of U . As expected, in our semi-synthetic experiments in Section 5, we find using predictions of W directly in a backdoor adjustment formula results in biased estimates; see Figure 4 .Gotcha #2: Using post-treatment text. While it is well-known that adjusting for post-treatment covariates in the backdoor formula often leads to bias (Pearl, 2009) , it is not obvious what might go wrong when using post-treatment text to infer proxies for the proximal g-formula. Proposition 2. If both W and Z are inferred from zero-shot models on text that contain post-treatment information, then the resulting proxies violate either (P2), (P3), or both.Proof. Consider Figure 3(a) , where the proxies are produced using text that is post-outcome and thus also post-treatment. We show that this violates both (P2) and (P3). Clearly the DAG violates (P3): by a simple d-separation argument we see thatZ ̸ ⊥ ⊥ Y | A, U, C due to the open path Y → T Z. Similarly, (P2) is violated from the open path A → Y → T W .Thus, before performing zero-shot inference, it is important that the text for each individual is filtered in such a way that it contains only the text preceding treatment7 . In our running clinical example, we can avoid this gotcha by using the time stamps of the clinical notes and information about when the patient was treated and discharged.Gotcha #3: Predicting both proxies from the same instance of text. After filtering to only pre-treatment text, T pre , for each individual, the intuitive next step is to use T pre to infer W and Z. Proposition 3. If W and Z are inferred via zero-shot models on the same instance of pre-treatment text, the resulting proxies violate (P1). In our synthetic experiments in Section 5, we find inferring proxies from the same instance of text results in biased estimates; see Table 1 . To avoid Gotcha #3, we select two distinct instances of pre-treatment text data for each unit of analysis, T pre 1 and T pre 2 , such that T pre 1 ⊥ ⊥ T pre 2 | U, C8 . For example, in the clinical setting we use metadata to select two separate categories of notes for each individual patient, e.g., nursing notes and echocardiogram notes. We hypothesize this satisfies T pre 1 ⊥ ⊥ T pre 2 | U, C since different providers will likely write reports that differ in content and style. Although the availability of distinct instances is domain-dependent, we hypothesize this type of data exists in other domains as well; for example, one could select distinct social media posts written by the same individual or multiple speeches given by the same politician.Gotcha #4: Using a single zero-shot model. After splitting text into T pre 1 and T pre 2 , should we apply the same zero-shot model to these two instances of text to infer the proxies W and Z, as in Figure 3 (c), or should we apply two separate zero-shot models as in Figure 3(d)? We find different answers in theory and practice. Proposition 4 shows that, in theory, both are valid as long asT pre 1 ⊥ ⊥ T pre 2 | U, C. However, we later describe how our semi-synthetic experiments demonstrate the need for two different models in practice in Section 5. We first establish the theoretical validity of both strategies. Proposition 4. If W and Z are inferred using zero-shot classification on two unique instances of pre-treatment text such that T pre 1 ⊥ ⊥ T pre 2 | U, C, then these proxies satisfy (P1-P3). Additionally, if the proxies are predictive of U , i.e., Z ̸ ⊥ ⊥ U | C and W ̸ ⊥ ⊥ U | C, then (P4) holds.Proof. Suppose we apply zero-shot classification models to two splits of pre-treatment text in a way that results in causal DAGs shown in Figure 3 (c) or (d) depending on whether we use one or two models, respectively. Applying d-separation confirms that the conditions (P1-P3) hold in both cases.Let |X V | denote the number of categories of a variable V . Kuroki and Pearl (2014); Tchetgen Tchetgen et al. (2020) state that when W and Z are predictive of U (as stated in the proposition), a sufficient condition for (P4) is min(|X Z |, |X W |) ≥ |X U |.Since U is binary under (S1) and since W and Z are discrete variables because they are inferred from classifiers, this condition is satisfied. Hence, (P4) is satisfied.Our Final Design Procedure Figure 1 and the causal DAG in Figure 3 (d) summarize our final design procedure-obtain two distinct instances of pre-treatment text for each individual and apply two distinct zero-shot models to obtain W and Z.In Proposition 4, we formalized how this procedure can be used to design proxies that satisfy the proximal conditions (P1-P4). However, this result relied on two important pre-conditions: (1) the conditional independence of the two instances of text, and (2) W and Z being (at least weakly) predictive of U . Yet, both of these conditions are untestable in general; this motivates the next section. 
falsification: odds ratio heuristic: 
 In practice, a major challenge for our procedure-and indeed all causal methods-are its assumptions. Sometimes, causal models imply testable restrictions on the observed data that can be used in falsification or confirmation tests of model assumptions, see Wang et al. (2017) ; Chen et al. ( 2023); Bhattacharya and Nabi (2022) for tests of some popular models. In our case, the proximal model Algorithm 1 for inferring two text-based proxies 1: Inputs: Observed confounders C; Text T; Zero-shot models M 1 , M 2 ; Specified γ high 2: Select two distinct instances of pre-treatment text T pre 1 and T pre 2 from T3: Infer Z ← M 1 (T pre 1 ) and W ← M 2 (T pre 2 ) 4: Compute the confidence interval (CI) for γ W Z.C , (γ CI low W Z.C , γ CI high W Z.C ) 5: if 1 < γ CI low W Z.C and γ CI high W Z.C < γ high then return W and Z 6: else return "stop" implies no testable restrictions (Tchetgen Tchetgen et al., 2020) , so the best we can do is provide a falsification heuristic that allows analysts to detect serious violations of (P1-P4) when using the inferred proxies and stop analysis. Our heuristic is based on the odds ratio function described below.Given arbitrary reference values w 0 and z 0 , the conditional odds ratio function for W and Z given covariates X is defined as Chen ( 2007), OR(w, z | x) = p (w|z,x) p(w0|z,x) × p(w0|z0,x) p(w|z0,x) . This function is important because W ⊥ ⊥ Z | X if and only if OR(w, z | x) = 1 for all values w, z, x. We summarize this odds ratio as a single free parameter, γ W Z.X , and, for the simplicity of our pipeline, we estimate it under a parametric model for p(W |Z, X)9 . Now, we describe our proximal conditions in terms of odds ratio parameters. If (P1-P3) are satisfied, then W ⊥ ⊥ Z | U, C and γ W Z.U C = 1. Further, if the zero-shot models are truly predictive of U , then (P4) is satisfied and W ̸ ⊥ ⊥ Z | C, which means that γ W Z.C ̸ = 1. Ideally, we would want to estimate both of these odds ratio parameters to confirm (P1-P4) empirically; however, γ W Z.U C cannot be computed from observed data alone due to (R1).Using a parameter we can estimate from observed data, γ W Z.C , we propose an odds ratio falsification heuristic in lines 3-6 of Algorithm 1. Now, we explain why, if this heuristic holds, an analyst can be reasonably confident in using their inferred text-based proxies for estimation.First, we examine a lower bound on γ W Z.C . Based on our previous discussion, if γ W Z.C is close to 1, then we should suspect that one or both of our zero-shot models failed to return informative predictions for U . Next, let us treat γ W Z.C as an imperfect approximation of γ W Z.U C . Let W, Z, U be binary with reference values w 0 = z 0 = u 0 = 0. VanderWeele (2008) proposed the following three conditions under which an odds ratio γ W Z.C that fails to adjust for an unmeasured confounder U is an overestimate of the true odds ratio γ W Z.U C : (i) {U } ∪ C satisfies the backdoor criterion with respect to W and Z; (ii) U is univariate or consists of independent components conditional on C; (iii) E[W |u, z, c] is non-decreasing in U for all z and c and E[Z|u, c] is non-decreasing in U for all c. Condition (i) is satisfied from Graph 3(d), and condition (ii) is satisfied by assumption (S1). Finally, condition (iii) is satisfied when our zero-shot models are reasonable predictors of the unmeasured confounder U by the following argument. Notice thatE[W |u, z, c] = E[W |u, c] = p(W = 1|u, c),where the first equality follows from d-separation in Graph 3(d) and the second equality follows from the definition of expectation for binary variable W . Then we should expect, if the zero-shot models are reasonably accurate, that p(W = 1|U = 1, c) > p(W = 1|U = 0, c). Therefore, the first part of condition (iii) is satisfied. Similar logic holds for E[Z|u, c].Hence, we have shown that under ideal conditions that satisfy (P1-P4), we should expect γ W Z.C > γ W Z.U C = 1, and we should reject proxies W and Z when we observe an odds ratio γ W Z.C ≤ 1. Next, we examine the upper bound on γ W Z.C . Consider the extreme case where γ W Z.C = ∞. This corresponds to a situation where W = Z, so (P1) is clearly not satisfied. In general, if γ W Z.C is higher than some threshold γ high , corresponding to the maximum association that one could reasonably explain by a single open path through U , we should suspect that perhaps the proxies W and Z are associated with each other due to additional paths through other unmeasured variables that make it so that the two instances of text are not independent of each other, i.e., T pre 1 ̸ ⊥ ⊥ T pre 2 | U, C. Following standard practice in sensitivity analysis, e.g., Liu et al. (2013) ; Leppälä (2023) , we leave it to the analyst to specify the upper bound γ high based on domain knowledge. In our experiments in Section 5, we found that, when the proximal conditions are not satisfied, γ W Z.C far exceeds ). We distinguish settings that passed the odds ratio heuristic (✓) from those that failed, with γ high = 2.Figure 4 : Semi-synthetic results for ACE point estimates (dots) and 95% CIs (bars). We distinguish settings that passed the odds ratio heuristic (✓) from those that failed, with γ high = 2.any reasonable setting of γ high . Hence, our heuristic works quite well in practice even with a generous suggestion for an upper bound. We describe our full design procedure with the diagnostic in Algorithm 1 and calculate 95% confidence intervals for γ W Z.C via the bootstrap percentile method (Wasserman, 2004) . We now evaluate its effectiveness for downstream causal inference. 
empirical experiments and results: 
 RQs In this section, we explore the following empirical research questions (RQs): How does Algorithm 1 compare to other alternatives in terms of bias and confidence interval coverage of the estimated causal effects? Does our odds ratio heuristic effectively flag when to stop or proceed?In causal inference, empirical evaluation is difficult because it requires ground-truth labels for counterfactual outcomes of an individual under multiple versions of the treatment, data that is generally impossible to obtain (Holland, 1986) . Thus, we turn to synthetic data and semi-synthetic data so we have access to the true ACE and U to evaluate methods. Semi-synthetic experimentswhich use real data for part of the DGP and then specify synthetic relationships for the remainder of the DGP-have been used extensively for other empirical evaluation of causal estimation methods; see Shimoni et al. (2018) ; Dorie et al. ( 2019); Veitch et al. (2020) . 10 We describe the experimental set-ups, the causal estimation procedure used by all experiments, and finally, the results to our RQs.Fully Synthetic Experiments We create our fully synthetic DGP based on the DAG in Figure 3(d) ; see Appendix C for full details. To summarize, A and U are binary, and Y and C are continuous. We generate (very simple) synthetic text data with four continuous variables, X 1 , X 2 , X 3 , X 4 , as functions of U and C. For training, we generate two realizations of these variables, which we call X train 1 and X train 2 , and likewise two realizations for inference time, X inf 1 and X inf 2 . At inference time, we explore using one or two zero-shot models, which we refer to as Proximal 1-Model (P1M) and Proximal 2-Model (P2M), respectively. For one zero-shot model, we train a logistic regression classifier to predict the true U from an aggregated variable X train = (X train 1 + X train 2 )/2 as P θ (U = 1| X train )11 . For the other zero-shot model, we use the following heuristic: predict 1 if X 1 > 1.1 else 0. P1M uses only the logistic regression model, and P2M uses both the logistic regression and heuristic models. Next, we vary whether Gotcha #3 holds at inference time for both P1M and P2M, i.e. whether the models infer Z and W from only X inf 1 or both X inf 1 and X inf 2 .Semi-Synthetic Experiments For our semi-synthetic experiments, we use MIMIC-III, a deidentified dataset of patients admitted to critical care units at a large tertiary care hospital (Johnson et al., 2016) . See Appendices D and G for detailed pre-processing steps, variables, and DGP. To summarize, we use the following real data from MIMIC-III in our DGP: ICD-9 code diagnoses, demographic information, and unstructured text notes. We choose the four diagnoses for oracle U that had the best F1 scores for a supervised bag-of-words logistic regression classifier (see Appendix E for full results): atrial fibrillation (Afib), congestive heart failure (Heart), coronary atherosclerosis of the native coronary artery (A-Sis), and hypertension (Hypertension). To avoid Gotcha #2, we explicitly exclude discharge summaries, which are post-treatment. In each experiment, we choose two out of the following four note categories as T pre 1 and T pre 2 : electrocardiogram (ECG), echocardiogram (Echo), Radiology, and Nursing notes. We hypothesize that notes written about different aspects of clinical care and by different providers will likely satisfy T pre 1 ⊥ ⊥ T pre 2 | U, C since each individual will write a conditionally independent realization of the patient's status. In Appendix I, we find distinct unigram vocabularies between the note sets that pass our falsification heuristic, lending preliminary evidence to this assumption of conditional independence. Consistent with the DAG in Figure 3 (d), we then synthetically generate binary A and continuous Y .For our zero-shot models, we use FLAN-T5 XXL (Flan) Chung et al. ( 2024) and OLMo-7B-Instruct (OLMo) Groeneveld et al. (2024) , both "open" instruction-tuned large language models. In Appendix F, we elaborate on our choice of these models. Following Ziems et al. (2024) , we use the prompt template 'Context: {T pre } \nIs it likely the patient has {U }?\nConstraint: Even if you are uncertain, you must pick either "Yes" or "No" without using any other words.' We assign 1 when the output from Flan or OLMo contains 'Yes' and 0 otherwise. P1M uses Flan for both proxies W and Z while P2M uses Flan for W and OLMo for Z. We compare P1M and P2M to a baseline that uses the inferred proxy W from Flan directly in a backdoor adjustment formula, corresponding to Gotcha #1.Estimation of Proximal g-formula For all experiments, we estimate the ACE by using the inferred W and Z in a two-stage linear regression estimator for the proximal g-formula provided by Tchetgen Tchetgen et al.. Although the linearity assumption is restrictive, it allows us to focus on evaluating the efficacy of our proposed method for inferring text-based proxies as opposed to complications with non-linear proximal estimation (Mastouri et al., 2021) . Briefly, we first fit a linear regression E[W |A, Z, C]. Next, we infer W , continuous probabilistic predictions for W , using the fitted model. For the second stage, we fit a linear model for E[Y |A, W , C]. The coefficient for A in this second linear model is the estimated ACE. We calculate 95% confidence intervals for the ACE via the bootstrap percentile method (Wasserman, 2004) . See Appendix F for additional implementation details (e.g., addressing class imbalance and sample splitting).Results Table 1 has synthetic results, and Tables 2 and Figure 4 have selected results for the semi-synthetic experiments. See Appendices E, H, and J for additional results. In short, our empirical results corroborate preference for Algorithm 1 over the baseline. First, we discuss the "gotcha" methods we showed to be theoretically incorrect in Section 3. Regarding Gotcha #1, Figure 4 shows that, across all settings of U , using the inferred W directly in the backdoor adjustment formula results in estimates with large bias. Regarding Gotcha #3 in the fully synthetic experiments, using the same realization X inf 1 results in high bias-0.130 and 0.107 for P1M and P2M, respectively, in Table 1 -and CIs that do not cover the true ACE. Regarding Gotcha #4, as in Proposition 4, using a single zero-shot model (P1M) results in low bias in the idealized setting of the synthetic DGP (Table 1 ; first row). However, using real clinical notes and Flan, the second columns of Figure 4 show that P1M produces estimates with higher bias compared to P2M across all three settings. We hypothesize this could be due to Flan over-relying on its pre-training and thus making similar predictions regardless of the clinical note.For all experiments we set γ high = 2. With this setting, we find low bias and good CI coverage for the two settings that pass our heuristic (P2M for U = A-Sis with Echo and Radiology notes and P2M for U = Heart with Echo and Nursing notes) and biased estimates in other cases. Although the confidence intervals sometimes cover the true ACE in settings that fail the heuristic, the interval is skewed and appears will asymptotically converge to a biased value. This gives us confidence that Algorithm 1 appropriately flags when to stop or proceed.Understanding why the odds ratio heuristic fails for a specific application requires oracle data. However, it may be helpful for analysts to reason about scenarios that influence this failure, such as when C and U are not a sufficient backdoor adjustment set; T pre 1 ̸ ⊥ ⊥ T pre 2 | U, C; or Z and W are poorly predictive of U . Using oracle data, we examined these scenarios for a few settings of our semi-synthetic experiments. For example, Figure 4 (right-most panel) shows that U =A-Sis with nursing and radiology notes fails the odds ratio heuristic. We qualitatively examined samples of the clinical notes and found some had unmeasured confounding, e.g., a patient had lung cancer-a variable that was not in our C or U -that influenced descriptions in both T pre 1 and T pre 2 . Figure 9 (bottom right-most panel) shows that for U =hypertension with echocardiogram (echo) and nursing notes also fails the odds ratio heuristic. However, we hypothesize this is due not to text dependence (see Table 4 ), but rather that the proxies W and Z have accuracies of 0.49 and 0.52 respectively (when evaluated against oracle U ), which is lower accuracy than predicting the majority class (Table 21 ). We leave to future work disaggregating the odds ratio heuristic into tests for these specific scenarios. 
conclusion, limitations, and future work: 
 In this work we proposed a novel causal inference method for estimating causal effects in observational studies when a confounding variable is completely unobserved but unstructured text data is available to infer potential proxies. Our method uses distinct instances of pre-treatment text data, infers two proxies using two zero-shot models on the instances, and applies these proxies in the proximal g-formula. We have shown why one should prefer our method to alternatives, both in theory and in the empirical results of synthetic and semi-synthetic experiments.If the conditions we present hold, the estimates will be unbiased. Further, as the underlying predictive accuracy of Z and W improve, the confidence interval width will narrow. Thus, we see promise in work improving the zero-shot performance of LLMs. Future work could possibly combine proxies from a greater number of zero shot models, i.e. inspired by work in Yang et al. (2024) .Although we are careful to infer valid proxies by design, it is generally impossible to empirically test whether conditions for proximal causal inference are fulfilled. To address this, we proposed an odds ratio falsification heuristic test, but acceptable values of γ high depend on the domain. If these are set incorrectly, our method could result in false positives (a setting for which the CI does not cover the true parameter and the estimated ACE is biased). In addition, our approach depends on the availability of pre-treatment text data and metadata to split text into independent pieces, which are not available across all applied settings.Although we use a clinical setting as our running example, our method is applicable to many other domains where it is infeasible to obtain ground-truth U labels due to privacy constraints or annotation costs, e.g., social media or education studies with private messaging data or sensitive student coursework. Other directions for future work include incorporating non-linear proximal estimators (including ones that use alternative assumptions to the completeness condition in (P4) (Kallus et al., 2021) ), expanding beyond text to proxies learned from other modalities (see Knox et al. (2022) ), providing more guidance for setting γ high in our odds ratio heuristic, extending our method to incorporate categorical U, W, Z, and using soft probabilistic outputs from the zero-shot classifiers. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes]Justification: We have reviewed the code of ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss broader impacts in Section 6 of the paper. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We credit the original creators of the datasets and models we use and mention the licenses. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Restriction (R1) is clearly present in settings for which we need to adjust for a confouding variable that is impossible or difficult to measure-for example, atrial fibrillation can go undiagnosed for years. A logical first attempt to mitigate this constraint is to train a supervised classifier on a subset of the data to create proxies for the rest of the dataset, as explored in Wood-Doughty et al. (2018) . This, however, requires humans to hand-label large amounts of text data. If labeling takes place on a crowd-sourcing platform, e.g., Amazon's Mechanical Turk, crowd-sourcing costs can quickly sky-rocket and often exceed tens of thousands of dollars, even for small datasets. Furthermore, many datasets-particularly those in clinical settings-require domain expertise (e.g., trained medical doctors) which will likely increase costs significantly and limit the availability of labelers.Cost and expertise of labelers aside, the possibility of supervised learning is further restricted by patient privacy legislation. We typically cannot transport sensitive data regarding patients' personal data to platforms such as Amazon's Mechanical Turk for labeling due to the Health Insurance Portability and Accountability Act (HIPAA)12 in the United States. In addition, legal acts such as the General Data Protection Regulation (GDPR)13 in the European Union and the California Consumer Privacy Act (CCPA)14 restrict the movement and repurposing of user data across platforms. Our proposed method overcomes restriction (R1) by using zero-shot learners that do not require previously labeled examples.Here, we first describe a scenario for which we may infer valid proxies using both post-treatment and pre-treatment text. In Figure 5 , T post is post-treatment text whereas T pre is pre-treatment text. Through simple d-separation arguments, we can see that each of (P1-P3) are fulfilled. Hence, it is still possible to use post-treatment text to generate valid proxies as long as we use pre-treatment text to generate one proxy and post-treatment text to generate the other. However, requiring both instances of text data to be pre-treatment is simpler to implement and validate, so we recommend this rule in our final method. Figure 6 further illustrates an example where we can use actionable text to generate valid proxies. Suppose T act is an instance of text data that influences a health practitioner's decision to assign different treatments to a patient, and T pre is an instance of pre-treatment text that is not actionable. Then, T act has a direct edge to the treatment A. Despite the addition of this edge, we can apply d-separation again to verify that (P1-P3) are still fulfilled. Therefore, our proposed method is still valid in a more relaxed setting where clinicians use one instance of the text data to make treatment decisions as long as the other instance of text data is both pre-treatment and not actionable. A Y U C T post T pre Z WThe DAG representing the fully synthetic data-generating process is shown in Figure 7 . We simulate U , a binary variable, as follows U ∼ Bernoulli(0.48) Next, we simulate baseline confounders and synthetic "text" X 1 , X 2 , X 3 , X 4 as follows:C ∼ N (0, 1)X 1 ∼ N (0, 1) + 1.95 * U + 3 * C X 2 ∼ N (0, 1) + exp(X 1 ) + U + 3 * C X 3 ∼ N (0, 1) + 1.25 * U + 3 * C X 4 ∼ N (0, 1) + X 3 2 + 0.5 * X 3 3 + U + 3 * CFinally, the treatment and outcome variables are generated via p(A = 1) = expit(0.8 In the fully synthetic data-generating process, increasing the coefficient for the variable C decreases the odds ratio conditional on observed confounders γ W Z.C while decreasing the coefficient for the variable C increases γ W Z.C .* U + C -0.3) A ∼ Bernoulli(p(A = 1)) Y ∼ N (0, 1) + 1.3 * A + 0.8 * U + C A Y U C X 1 X 2 X 3 X 4In the MIMIC-III dataset (Johnson et al., 2016) , data is organized into multiple tables. Patients are anonymized and each have a unique identifier. In addition, each unique hospital admission is assigned a hospital admission identification number (HAID). Each patient can have multiple HAIDs, and each HAID can have multiple notes (e.g., an ECG note and a nursing note). Our pre-processing steps are outlined in the following steps:1. We start with the table in MIMIC-III containing the clinicians' notes for all patients and drop rows of data where the HAID has a missing value. We refer to this main table we are working with as our reference table. 2. Next, for each unique patient in the reference table, we select one HAID by choosing the HAID with the earliest "chart date," the day on which the clinician's note was recorded. We limit each patient to only one hospital admission to better fit the assumption of i.i.d. data (which would be violated if there were multiple hospital admissions for a single patient since these multiple admissions would be dependent). 3. We drop all clinicians' notes that have the category label "Discharge summary" because such notes would be considered post-treatment text that violate (P2) as we have shown in Section 3. For each HAID, we combine separate notes of the same category into one string. 4. We now move on to the table in MIMIC-III that contains information on which patients were given which medical diagnoses for each HAID. For each HAID in the reference table, we select all the diagnoses (ICD-9 codes) assigned to the patient in that hospital visit. 5. The 10 most common diagnoses for the patients in our reference To ensure that the text contains predictive signal for the diagnoses (a precondition necessary to use our zero-shot classifiers later on in our analysis), we train supervised classifiers to predict the diagnoses, Y , separately using text data from each note category, X, with a bag-of-words representation.Our procedure is as follows. We first truncate all of the clinicians' notes data to 470 tokens (due to the context window of Flan-T5 which we use later in our pipeline). We ignore oracles and note categories combinations where positivity rates are 0 or 1 after subsetting on the note category in question. We then use the scikit-learn library's CountVectorizer to convert the text data to bag of words features with vocabulary size 5, 000 and train a linear logistic regression with the hyperparameter penalty set to "None". We further calculate the F1 score, accuracy, precision, and recall with the scikit-learn library Pedregosa et al. (2011) . The results are summarized in Table 3 .We choose oracles with at least two note categories that achieve F1 scores greater than 0.7 for downstream inference with zero-shot classifiers. Table 3 : Descending order of F1 scores (as well as accuracy, precision and recall) from bag of words linear logistic regressions predicting various medical diagnoses from the MIMIC-III dataset using different medical note categories as input. We set 0.7 as our F1 score cutoff for choosing the diagnoses to use in downstream analysis with zero-shot classifiers (dashed horizontal line). We bold the diagnoses that passed this threshold and also have at least two note categories. We used these (bolded) diagnoses in the final oracle U in downstream analyses.Choosing LLMs: FLAN and OLMo We used the large language models FLAN-T5 XXL and OLMo-7B-Instruct for inferring the oracle diagnoses from text data Chung et al. ( 2024); Groeneveld et al. (2024) . These are both fully "open" models in that we have full knowledge of their pre-training data, training objectives, and training details, and can be downloaded and updated on a local machine, all of which are important for reproducible science (Palmer et al., 2024) . We chose these two models because they have been fine-tuned for zero-shot classification and because they have been trained on different text corpora. Furthermore, we hypothesize that Flan will perform well on clinical notes because, according to Dodge et al. (2021) 's analysis, C4 (the pre-training data in Flan-T5) has among its top 25 domains patents.google.com, journals.plos.org, link.springer.com, www.ncbi.nlm.nih.gov, all of which likely contain medical text.Creating inferences from FLAN and OLMo We use the following string as the prompt for both models: 'Context: {T pre } \nIs it likely the patient has {U }?\nConstraint: Even if you are uncertain, you must pick either "Yes" or "No" without using any other words.' We consider only the first five tokens of output for both models and assign a value of 1 for the proxy if the output string converted to all lowercase characters contains the word 'yes' and 0 otherwise.Odds ratio confidence interval estimation We calculate a point estimate for the odds ratio γ W Z.C using the scikit-learn library (Pedregosa et al., 2011) by first training a linear logistic regression with W as the target outcome and {Z} ∪ C as the features. Whenever the positivity rate of W is less than 0.2 or greater than 0.8, i.e. there is a class imbalance, we set the hyperparameter class_weight to "balanced". This uses the values of W to automatically adjust weights inversely proportional to class frequencies in the dataset. Regardless of whether we adjust for class imbalance, we set the hyperparameter penalty to None to turn off regularization. Finally, we calculate the natural exponent of the coefficient of Z as the point estimate for the odds ratio γ W Z.C .We calculate the confidence interval for the odds ratio by drawing 200 bootstrap samples and repeating the steps above for each bootstrap sample to create a bootstrap distribution. We then take the 0.025th and 0.975th percentiles of the bootstrap distribution as γ CI low W Z.C and γ CI low W Z.C , respectively.Two-stage linear proximal causal inference estimator for the ACE Following sample splitting from the causal inference literature Hansen (2000), we start by splitting the semi-synthetic dataset into two splits-split 1 and split 2-where both splits are 50% of the original dataset. We then train a linear logistic regression using split 1 with W as the target outcome and {A, Z} ∪ C as the features. Like in the estimation procedure for the odds ratio, we set the hyperparameter class_weight to "balanced" whenever there is a class imbalance in the target outcome W and always set the hyperparameter penalty to None. Next, we make probabilistic predictions W for W with the previously trained linear logistic regression model using split 2. Finally, we train a linear regression using split 2 of the data with Y as the target outcome and {A, W } ∪ C as the features. The coefficient of A in the trained linear regression model is the estimate for the ACE.We calculate the confidence interval for the ACE by drawing 200 bootstrap samples and repeating the steps above for each bootstrap sample to create a bootstrap distribution. We then take the 0.025th and 0.975th percentiles of the bootstrap distribution as the lower and upper bound of the confidence interval for the ACE, respectively.The DAG representing the semi-synthetic data-generating process is shown in Figure 8 . The variable U represents atrial fibrillation, congestive heart failure, coronary atherosclerosis of the native coronary artery, or hypertension, depending on the setting. We use the reference table created in the pre-processing steps described in Section D of the Appendix and start by normalizing the variable Age with the formulaAge i = Age i -XAge σ Age .We then simulate draws of the binary variable A via p(A = 1) = expit(U + 0.9 * Gender + 0.9 * Age)A ∼ Bernoulli(p(A = 1))Next, we simulate draws of the continuous variable Y from: Y ∼ N (0, 1) + 1.3 * A + U + 0.9 * Gender + 0.9 * Age When estimating the ACE using the two-stage linear regression described in Section 5, we condition on the baseline covariates Age and Gender as well the other diagnoses not being used as U .In Figure 9 , we report additional experimental results from P1M using Flan to infer proxies from both note categories and P2M using Flan and OLMo to infer proxies from one note category each.As expected, using one inferred proxy from an LLM directly in backdoor adjustment produces biased estimates for the ACE. When using P1M and P2M, the odds ratio falsification heuristic successfully flags invalid estimates of the ACE. In the two cases where the inferred proxies pass the odds ratio falsification heuristic, we observe valid estimates for the ACE.A notable example of when the odds ratio heuristic correctly failed is for Echo and Nursing notes on the diagnosis hypertension. As shown in Figure 9 , the estimates for the ACE for both P1M and P2M are far from the true ACE of 1.3. This is explained by Table 21 where the oracle odds ratios between both proxies W and Z and the oracle U are nearly 1, suggesting that the zero-shot classifiers were not able to accurately predict the oracle. As expected in such a scenario, the estimates for the ACE are biased, and our odds ratio heuristic successfully detects the violation of Assumption (P4) for proximal causal inference.Recall, our procedure requires us to split text such that T pre 1 ̸ ⊥ ⊥ T pre 2 | U, C. This condition must hold at inference time when practioners are using our procedure and it must hold in the "real" part of our semi-synthetic experiments in Section 5. Thus, for our semi-synthetic experiments with real-world clinical notes, we aim to investigate when this condition does or does not hold. To the best of our knowledge, it is an open problem on how to perform (conditional) independence tests on high-dimensional, compositional language data. Instead, we turn to an approximation that can lend us some preliminary evidence and examine the probabilities of top unigram features for each pairing of text categories. We show the results in Tables 4 and 5 below.After tokenization into unigrams, we select the top five features via tf-idf scores. The tf-idf scores are created under two settings: (1) fitting tf-idf on both note categories combined, or (2) fitting tf-idf separately on each note category. Unique features (unigrams) must occur in a minimum of 10% of the documents and a maximum of 90% of the documents to be considered. Note, when we look at combinations of note pairings we subset to patients that have both notes, so subsets of documents are overlapping but not the exact same, resulting in different top features for the same category in different note pairs.We report key metrics, mostly requiring access to the oracle U , for the semi-synthetic simulations in the tables below. Although these metrics will typically not be available to practitioners at inference time, they are helpful in understanding the relationship between the inferred proxies W and Z under various conditions. The odds ratio values between the proxies and the oracle, γ W U.C and γ ZU.C , show how correlated the proxies are with the oracle, an important indicator for (P4). The difference between 1 and the positivity rate of the oracle describes the minimum accuracy threshold for which the inferred proxy must perform to be more accurate than simply predicting the majority class.Figure 9 : Semi-synthetic results for ACE point estimates (dots) and confidence intervals (bars) for all oracles and note categories that we detected sufficient signal in. Here, ✓ distinguishes settings that passed the odds ratio heuristic from those that failed it, with γ high = 2. To run the experiments in this paper, we used a local server with 64 cores of CPUs and 4 x NVIDIA RTX A6000 48GB GPUs. Our semi-synthetic pipeline ran for approximately 36 hours where Flan took roughly 12 hours and OLMo took roughly 24 hours to compute inferences from the text data.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We provide details of the compute resources in Section K of the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Setting aside challenges of high-dimensional covariate selection for causal estimation(Tamarchenko,2023).2 Supporting code is available at https://github.com/jacobmchen/proximal_w_text.In the absence of (R1), we direct readers to work that adjusts via measurement error estimates or assumptions that U is "missing at random"Wood-Doughty et al. (2018).In Appendix B we show how proxies could be inferred using a mix of pre-and post-treatment text while satisfying (P1-P3). However, we advocate for the the pre-treatment rule due to its simplicity.Note this independence condition does not imply the two pieces of text are completely uncorrelated. Since the text is written based on observations of the same individual, we certainly expect T pre 1 ̸ ⊥ ⊥ T pre 2 ; we simply require that the two pieces are correlated only due to C and U .More generally, the odds ratio can be treated as a finite p-dimensional parameter vector γ and estimated under semi-parametric restrictions on p(W |Z, X) and p(Z|W, X)(Chen, 2007;Tchetgen Tchetgen et al., 2010).Gentzel et al. (2019) andKeith et al. (2023) describe methods for evaluation that are more realistic than semi-synthetic evaluation by downsampling an RCT dataset to create an observational (confounded) dataset. However, for the proximal setting we investigated in this paper, we could not find a suitable existing RCT.Of course, having access to the true U may not qualify as "zero-shot" in the strict sense of the term, but we complement this idealized scenario with the difficult true zero-shot scenario in our semi-synthetic experiments.https://www.hhs.gov/hipaa/index.htmlhttps://www.consilium.europa.eu/en/policies/data-protection/ data-protection-regulation/https://oag.ca.gov/privacy/ccpaAtrial fibrillation (A-fib) with ECG, Echo, and Nursing notes.Congestive heart failure (Heart) with Echo and Nursing notes.Coronary atherosclerosis of the native coronary artery (A-sis) with Echo, Nursing, and Radiology notes. 


Decision: This paper treats causal inference in the setting where there is an "unobserved" confounder, but there is a proxy variable available in the form of text. The reviewers appreciated the problem setting, the principled approach, and the clarity of communication.

Reviews: review: paper_summary: paper_summary: The paper considers the causal effect estimation setting where a confounder is latent but with unstructured text data that could serve as proxies. Specifically, the paper proposes to incorporate zero-shot classifiers (to operate on text-based proxies), together with a falsification heuristic, into the proximal causal inference framework. The goal of the text-based proxy design is to craft W and Z that satisfy the identification condition in proximal causal inference framework. Empirical evaluations are presented in synthetic and semi-synthetic experiments.

---

**Post rebuttal**

I have increased my score, under the assumption that the revised manuscript could sufficiently address the original concerns on material organizations.
strength_weakness: strength_weakness: The weakness of the paper comes from the organization of the material (especially the constraints/conditions involved), and relatively simple settings considered in (fully-/semi-) synthetic experiments. In particular, further clarifications and/or discussions on following points would be very helpful (detailed in "Questions" section):

(1) regarding a series of different constraints, conditions, gotcha's, assumptions

(2) the fully and semi- synthetic experiments consider structured data, it is not exactly clear how these text proxies look like and if they correspond to all or part of the aforementioned constraints/conditions. Examples of text proxies would be much more intuitive than claiming "satisfy by design" (line 136)
questions: questions: (1) regarding a series of different constraints, conditions, gotcha's, assumptions

In Section 2, there are P1 -- P4, a set of conditions should be satisfied in order for the proximal g-formula to work. In order to respond to the criticism of potential unavailability of W and Z in structured data, further assumptions S1 -- S2 are proposed. Then in Section 3 and Section 4, a set of gotcha's, two additional pre-conditions (lines 216 -- 218), and another set of conditions related to odds-ratio based heuristics (lines 245 -- 247) are presented. How do these conditions fit together to respond to the criticism of proximal causal inference on structured data? Are they all identification conditions for the text-based proxy proximal causal inference (other than the fact that they are conditions specified by previous works for different components put together)?

(2) the fully and semi- synthetic experiments consider structured data, it is not exactly clear how these text proxies look like and if they correspond to all or part of the aforementioned constraints/conditions

For fully and semi- synthetic experiments, the setting is largely structured data instead of text-based proxies. Examples of text proxies that the proposed design procedure yields would be much more intuitive than just claiming "satisfy by design" (line 136).
limitations: limitations: The organization of material could be improved to better present how the proposed approach responds to criticism of proximal causal inference (with structured data). Examples beyond fully and semi- synthetic experiments will make it clearer w.r.t. how the designed text-based proxies help to address the aforementioned criticism on previous work.
paper_summary: paper_summary: Causal inference techniques often rely on the assumption that all confounders can be observed or inferred from available data. This paper proposes a method to address the setting where a confounder is entirely unobserved; instead there is available pre-treatment text that can be used to infer proxies, which in turn have predictive value for the unobserved confounder. The authors adapt proximal causal inference ideas to text-data settings, discuss "gotchas" in naive implementations of such a method, and explore various relevant choices before presenting best performing method. Specifically, they use multiple instances of pre-treatment text to infer two proxies for each datapoint using two different zero-shot language models. This is then used for average causal effect estimation via the proximal g-formula. The paper provides empirical evaluation on synthetic and semi-synthetic datasets as well as a falsifcation heuristic for untestable assumptions.
strength_weakness: strength_weakness: - The significance of the empirical evaluation is limited due to synthetic and semi-synthetic experiments. The impact of the proposed method in realistic settings is hard to assess. While the evaluation of real settings is hampered by the lack of ground-truth, it may still be possible to synthesize data that is as realistic as possible so that the more realistic challenges of deploying such a method may be uncovered.

- The experiment design of the paper carefully avoids post-treatment text to avoid biased effect estimation. However, the proxy inference from language models indirectly depends on the pretraining dataset used to train the language model. The data in that pretraining set may be post-treatment text. The impact of this dependence is unclear.

- It is unclear how sensitive the performance of the proposed method is to the choice of pretrained zero-shot model, or whether we can expect better performance as the zero-shot model used scales up.
questions: questions: As mentioned above:

- Do the authors expect the performance of their method to improve as they use larger language models to infer proxies? 

- Can the authors provide any intuition for what they expect to be the impact of using a language model that has been trained on post-treatment text data to infer proxies and plug them into the causal effect estimation?
limitations: limitations: The paper sufficiently addresses limitations of the setting and method proposed.
paper_summary: paper_summary: The situation with unobserved confounding variables is quite common in applied research and makes causal inference complicated. To deal with such settings, the authors propose a new causal inference method that uses multiple instances of pre-treatment text data, estimates two proxies from two zero-shot models on the separate instances, and applies these proxies in the proximal g-formula. It is shown that the identification conditions are satisfied, The method is evaluated in a simulation study on synthetic and semi-synthetic data.
strength_weakness: strength_weakness: - Literature review is incomplete. The idea of proximal inference goes back to Zvi Griliches (1977) who should get credit for it. As background reading the corresponding chapter in the textbook "Applied Causal Inference Powered by ML and AI" by Chernozhukov and coauthors (2024).
- To overcome the completeness condition the paper "Causal Inference Under Unmeasured Confounding With Negative Controls: A Minimax Learning Approach" (arXiv:2103.14029) might be helpful.
- The theoretical results (Prop 1-4) are all in some sense "negative" and straightforward / already known. What would be more interesting, would be some "positive" results about the proposed method, although I am aware that this might be quite challenging.
questions: questions: - How credible / realistic are (P1) and that $T^1_pre$ is conditionally independent from $T^2_pre$ in typical applications (e.g. using electronic health records)?
-  How could one use EHR to construct proxies? Are the assumptions fulfilled? (Might be questionable.. the example mentioned in the text is not convincing.)
limitations: limitations: Limitations are adressed adequately. As mentioned above, theoretical guarantees for the propsoed method might be very valuable.

Comments: title: Comment
Thank authors for the response
comment: Thank you for the clarifications! I will maintain my score.
First of all, thanks a lot for your efforts and the detailed answer. I still think that the negative results in Proposition 1-3 are not really deep and well known. I am aware that the reference can be considered as a complement, but I think a thorough literature review should contain all related sources. Finally, I still find the assumption of conditional independence hard to hold and I think here more convincing examples would be helpful. Overall, I will keep the score.
Thank authors for the responses, and my concerns are largely resolved.

While there is still some worry about the potential difficulty (upon readability, and upon the parsing of the key takeaway msgs) because of the conditions/assumptions/gothcha's introduced in various places, the proposed edits and material organization might help in the revised version of the paper. A brief paragraph (just like what authors presented in the response) or a table in the appendix would be helpful to provide a summary of how these requirements piece together.

As for the examples, if the space permits, having some of them in the main paper might be very helpful for readers to bridge the gap between the set of requirements, and what can be done empirically (beyond fully- and semi- synthetic experiments).

I have updated my evaluation score accordingly, under the assumption that the revised manuscript could sufficiently address the original concerns.