Causal Estimation for Text Data with (Apparent) Overlap Violations
Authors: Lin Gui
Victor Veitch
Venue: ICLR 2023 poster (2023)

Abstract: Consider the problem of estimating the causal effect of some attribute of a text document; for example: what effect does writing a polite vs. rude email have on response time? To estimate a causal effect from observational data, we need to adjust for confounding aspects of the text that affect both the treatment and outcome---e.g., the topic or writing level of the text. These confounding aspects are unknown a priori, so it seems natural to adjust for the entirety of the text (e.g., using a transformer). However, causal identification and estimation procedures rely on the assumption of overlap: for all levels of the adjustment variables, there is randomness leftover so that every unit could have (not) received treatment. Since the treatment here is itself an attribute of the text, it is perfectly determined, and overlap is apparently violated. The purpose of this paper is to show how to handle causal identification and obtain robust causal estimation in the presence of apparent overlap violations. In brief, the idea is to use supervised representation learning to produce a data representation that preserves confounding information while eliminating information that is only predictive of the treatment. This representation then suffices for adjustment and satisfies overlap. Adapting results on non-parametric estimation, we show that this procedure shows robustness with respect to conditional outcome misestimation and yields a low-bias estimator that admits valid uncertainty quantification under weak conditions. Empirical results show reductions in bias and strong improvements in uncertainty quantification relative to the natural (transformer-based) baseline.

Published as a conference paper at ICLR 2023
CAUSAL ESTIMATION FOR TEXT DATA WITH (APPAR- ENT) OVERLAP VIOLATIONS
Lin Gui and Victor Veitch 1.2
The University of Chicago Google Research
ABSTRACT
Consider the problem of estimating the causal effect of some attribute of text document; for example: what effect does writing polite rude email have on response time? To estimate a causal effect from ob- servational data, need adjust for confounding aspects of the text that affect both the treatment and outcome-e.g, the topic or writing level of the text. These confounding aspects are unknown priori, so seems natural to adjust for the entirety of the text (e.g,, using trans- former). However, causal identification and estimation procedures rely on the assumption of overlap: for all levels of the adjustment variables, there randomness leftover so that every unit could have (not) received treatment. Since the treatment here is itself an attribute of the text, it is perfectly determined, and overlap is apparently violated. The purpose of this paper is to show how to handle causal identification and obtain robust causal estimation in the presence of apparent overlap violations. In brief, the idea is use supervised representation learning to produce data representation that preserves confounding information while eliminating information that is only predictive of the treatment. This representation then suffices for adjustment and satisfies overlap. Adapting results on non-parametric estimation, we find that this procedure robust to condi- tional outcome misestimation, yielding low- www-lbbolute-oo biau estimator with valid uncertainty quantification under weak conditions. Empirical results show strong improvements in bias and uncertainty quantification relative to the natural baseline. Code, demo data and tutorial are available at TrrnbxxbTT-estimatrr. estimator.
https:/ wit.gg.chc...oooo com/
http:/ooo/gm/////////// g2-ybnbxb/TT
1 INTRODUCTION
We consider the problem of estimating the causal effect of an attribute of passage of text on some downstream outcome. For example, what is the effect of writing polite or rude email on the amount of time it takes to get a response? In principle, we might hope to answer such questions with randomized experiment. However, this can be difficult in practice-e.g, if poor outcomes are costly or take long to gather. Accordingly, in this paper, we will be interested in estimating such effects using observational data.
There are three steps to estimating causal effects using observational data (See Chapter 36 Murphy (2023)). First, we need to specify a concrete causal quantity as our estimand. That is, give formal quantity target of estimation corresponding to the high-level question of interest. The next step is causal identification: we need to prove that this causal estimator can, in principle, be estimated using only observational data. The standard approach for identification relies on adjusting for confounding variables that affect both the treatment and the outcome. For identification hold, our adjustment variables must satisfy two conditions: unconfoundedness and overlap. The former requires the adjustment variables contain sufficient information on all common causes. The latter requires that the adjust- ment variable does not contain enough information about treatment assignment to let us perfectly predict it. Intuitively, to disentangle the effect of treatment from the effect of confounding, we must observe each treatment state at all levels of confounding. The final
1

---

Published as a conference paper at ICLR 2023
step is estimation using finite data sample. Here, overlap also turns out to be critically important as a major determinant of the best possible accuracy (asymptotic variance) of the estimator Chernozhukov et al. (2016).
Since the treatment is linguistic property, it is often reasonable assume that text data has information about all common causes of the treatment and the outcome. Thus, we may aim to satisfy unconfoundedness in the text setting by adjusting for all the text the confounding part. However, doing so brings about overlap violation. Since the treatment is linguistic property determined by the text, the probability of treatment given any text is either 0 or 1. The polite/ /rude tone is determined by the text itself. Therefore, overlap does not hold if we naively adjust for all the text the confounding part. This problem is the main subject of this paper. Or, more precisely, our goal is to find causal estimand, causal identification conditions, and robust estimation procedure that will allow us to effectively estimate causal effects even in the presence of such (apparent) overlap violations.
In fact, there is an obvious first approach: simply use a standard plug-in estimation proce- dure that relies only on modeling the outcome from the text and treatment variables. In particular, do not make any explicit use of the propensity score, the probability each unit is treated. Pryzant et al. (2020) use an approach of this kind and show it is reasonable in some situations. Indeed, we will see in Sections and 4 that this procedure can be inter- preted as a point estimator of a controlled causal effect. Even once we understand what the implied causal estimand is, this approach has a major drawback: the estimator is only accurate when the text-outcome model converges at a very fast rate. This is particularly an issue in the text setting, where we would like to use large, flexible, deep learning models for this relationship. In practice, we find that this procedure works poorly: the estimator has significant absolute bias and (the natural approach to) uncertainty quantification almost never includes the estimand true value; see Section 5.
The contribution of this paper is method for robustly estimating causal effects in text. The main idea is to break estimation into a two-stage procedure, where in the first stage we learn representation of the text that preserves enough information to account for confounding, but throws away enough information to avoid overlap issues. Then, we use this repre- sentation as the adjustment variables in a standard double machine-learning estimation procedure Chernozhukov et al. (2016; 2017a). To establish this method, the contributions of this paper are:
1. We give a formal causal estimand corresponding to the text-attribute question. We show this estimand is causally identified under weak conditions, even in the presence of apparent overlap issues.
2. We show how to efficiently estimate this quantity using the adapted double-ML technique just described. We show that this estimator admits a central limit theo- rem at a fast (v/R) rate under weak conditions on the rate at which the ML model learns the text-outcome relationship (namely, convergence at n/ rate). This im- plies absolute bias decreases rapidly, and an (asymptotically) valid procedure for uncertainty quantification.
3.
We test the performance of this procedure empirically, finding significant improve- ments in bias and uncertainty quantification relative the catcome-modee--ome baseline.
Related work The most related literature is on causal inference with text variables. Pa- pers include treating text as treatment Pryzant et al. (2020); Wood-Doughty et al. (2018); Egami et al. (2018); Fong & Grimmer (2016); Wang & Culotta (2019); Tan et al. (2014)), as outcome Egami et al. (2018); Sridhar & Getoor (2019), as confounder Veitch et al. (2019); Roberts et al. (2020); Mozer et al. (2020); Keith et al. (2020), and discovering or predicting causality from text del Prado Martin & Brendel (2016); Tabari et al. (2018); Balashankar et al. (2019); Mani & Cooper (2000). There are also numerous applications using text to adjust for confounding (e.g., Olteanu et al., 2017; Hall, 2017; Kiciman et al., 2018; Sridhar et al., 2018; Sridhar & Getoor, 2019; Saha et al., 2019; Karell Freedman, 2019; Zhang et al., 2020). Of these, Pryzant et al. (2020) also address non non-paramttric es-
2

---

Published as a conference paper at ICLR 2023
timation of the causal effect of text attributes. Their focus is primarily on mismeasurement of the treatments, while our motivation is robust estimation.
This paper also relates to work on causal estimation with (near) overlap violations. D'Amour et al. (2021) points out sigh-dimensional adjustment (e.g., Rassen et al., 2011; Louizos al., 2017; Li et al., 2016; Athey et al., 2017) suffers from overlap issues. Extra assumptions such as sparsity are often needed to meet the overlap condition. These results do not directly apply here because we assume there exists low-dimensional summary that suffices to handle confounding.
D'Amour & Franks (2021) studies summary statistics that suffice for identification, which they call deconfounding scores. The supervised representation learning approach in this paper can be viewed as an extremal case of the deconfounding score. However, they con- sider the case where ordinary overlap holds with all observed features, with the aim of using both the outcome model and propensity score to find efficient statistical estimation procedures (in linear-gaussial setting). This does not make sense in the setting we con- sider. Additionally, our main statistical result (robustness to outcome model estimation) is new.
2 NOTATION AND PROBLEM SETUP
We follow the causal setup of Pryzant et al. (2020). We are in- terested in estimating the causal effect of treatment A on out- come Y. For examplc, how docs writing a negative sentiment (A) review (X) affect product sales (Y)? There are two imme- diate challenges to estimating such effects with observed text data. First, do not actually observe A, which is the intent of the writer. Instead, we only observe A, a version A that is inferred from the text itself. In this paper, will assume that A almost surely-e.g. a reader can always tell a review was meant to be negative or positive. This assumption is of- ten reasonable, and follows Pryzant et al. (2020). The next challenge that the treatment may be correlated with other aspects of the text (Z) that are also relevant to the outcome- e.g., the product category of the item being reviewed. Such Z. can act as confounding variables, and must somehow be adjusted for in a causal estimation problem.
Jonter propertin)
(Qutoom
[Parcelenn (isttnedd
Figure 1: The causal DAG of the problem. A writer writes text document linguistic properties A and Z, where A is the treatment in the causal problem. cannot be observed directly in data and can only be seen via text. The dotted line repre- sents possible correlation be- tween and Z. A reader per- ceives the treatment Á from the text. The perceived treat- ment À together with contents of x determine the outcome Y.
based on
and Z
Each unit Zi Y2) is drawn independently and identi- cally from an unknown distribution P. Figure shows the causal relationships among variables, where solid arrows rep- resent causal relations, and the dotted line represents possi- ble correlations between variables. We assume that text X contains all common causes of Á and the outcome Y.
3 IDENTIFICATION AND CAUSAL ESTIMAND
The first task is to translate the qualitative causal question of Interest-whan is the effect A on Y-into a causal estimand. This estimand must both be faithful to the qualitative question and be identifiable from observational data under reasonable assumptions. The key challenges here are that we only observe Å (not A itself), there are unknown confound- ing variables influencing the text, and À is deterministic function of the text, leading overlap violations if we naively adjust for all the text. Our high-level idea is to split the text both
into abstract (unknown) parts depending on whether they are iinfounding-affe Á and Y-or whether they affect Á alone. The part of the text that affects only À is not necessary for causal adjustment, and can be thrown away. If this part contains "enough" information about À, then throwing it away can eliminate our ability to perfectly predict Ā, thus fixing the overlap issue. We now turn to formalizing this idea, showing how can be used to define an estimand and to identify this estimand from observational data.
3

---

Published as a conference paper at ICLR 2023
Causal model The first idea is to decompose the text into three parts: one part affected by only A, one part affected interactively by and Z, and another part affected only by Z. We use XA, XAnz and Xz to denote them, respectively; see Figure 2 for the corresponding causal model. Note that there could be additional information in the text addition these three parts. However, since they are irrelevant to both A and Z, we do not need to consider them in the model.
Controlled direct effect (CDE) The treatment A affects the outcome through two paths. Both "directly" through X part of the text determined just by the treatment- and also through a path going through -he part of the text that relies on interaction effects with other factors. Our formal causal effect aims at capturing the effect A through only the first, direct, path.
X
CDE == Ex
E[Y| K,nz,X,, do(A 1)]
more sophisti- cated causal model with the decomposition of text X. X.. Xazi and X, are parts of the text affected by only A, both A and Z, and only Z, respectively. A and Z are linguistic properties writer based on and thus cannot observed directly from data. When investigating the causal relationship between À and Y, (xxz,X²) confound- ing part satisfying both uncon- foundedness and overlap.
Figure 2:
-E[Y X 0)] 0)]]
(3.1)
Here, do is Pearl's do notation, and the estimand is variant of the controlled direct effect (Pearl, 2009). Intuitively, it can be interpreted as the expected change in the outcome induced by changing the treatment from to 0 while keeping part of the text affected by Z the same would have been had we set A 1. This is a reasonable formalization of the qualita- tive "effect of A on Y". Of course, it is not the only possible formalization. Its advantage is that, as we will see, it can be identified and estimated under reasonable conditions.
Identification To identify CDE we must rewrite the expres- sion in terms of observable quantities. There are three chal- lenges: we need to get rid of the do operator, we don't 't observe A (only A), and the variables X 2 are unknown (they are latent parts of X).
Informally, the identification argument is as follows. First, XA block all backdoor paths (common causes) in Figure 2. Morcover, because we have thrown away XA, we now satisfy overlap. Accordingly, the do operator can be replaced by conditioning following the usual causal-adjusstmn argument. Next, A = Á almost surely, so we can just replace A with Å. Now, our estimand has been reduced to:
CδE := Ex Xx X,|i=
|[||
XAzz,X 1] E[Y X 0)]].
(3.2)
The final step to deal with the unknown Xa,z,X,. To fix this issue, we first define the conditional outcome Q according to:
Q(A))==(C Á,X).
(3.3)
A key insight here is that, subject the causal model in Figure 2, have Q(ī,x) E(Y (xXx)) But this is exactly the quantity in (3.2). Moreover, Q(Āx)) is an ob- servable data quantity (it depends only on the distribution of the observed quantities). In summary:
Theorem 1. Assume the following:
1. (Causal structure) The causal relationships among A, Á, Z, Y, and X satisfy the causal DAG
in Figure 2;
2. (Overlap) 0 P(A= Xx)z,X)) <;
3. (Intention equals perception) A almost surely with respect to all interventional distri-
butions. Then, the CDE is identified from observational data
CDE = - CDE == Exia [E[Y (x)x 1]- E[Y (x)))ā= 01],
(3.4)
where (x)) = (Q(0.)),((1x)).
4

---

Published as a conference paper at ICLR 2023
The proof is in Appendix B.
We give the result in terms of an abstract sufficient statistic 77X) to emphasize that the actual conditional expectation model is not required, only some statistic that is informa- tionally equivalent. We emphasize that, regardless of whether the overlap condition holds not, the propensity score of nxX) is accessible and meaningful. Therefore, we can casily identify when identification fails as long as (x) is well-estimated.
4 METHOD
Our ultimate goal is to draw a conclusion about whether the treatment has a causal effect on the outcome. Following the previous section, we have reduced this problem to estimating - defined in Theorem 1. The task now is to develop an estimation procedure, including uncertainty quantification.
4.1 OUTCOME ONLY ESTIMATOR
We start by introducing the naive outcome only estimator as first approach to CDE esti- mation. The estimator is adapted from Pryzant et al. (2020). The observation here is that, taking mxX) = (Q(O, X), Q(1,X) in (3.4), we have
CDB Ax E(Y A- 1,X) B(Y -A= 0. ].
(4.1)
Since (AA Namely, the solution the square error risk:
) is a function of the whole text data X, it is estimable from observational data.
Q arg B[(Y Q(A,)))).
(4.2)
With finite sample, estimate as Q by fitting a machine- model mini- mize the (possibly regularized) square error empirical risk. That is, fit a model using mean square error the objective function. Then, straightforward estimator is:
40 - Σ (i(x;)) -6(x)))
(4.3)
1A,11
where n, is the number of treated units.
It should be noted that the model for Q is not arbitrary. One significant issue for those models which directly regress Y on A and X is when overlap does not hold, the model could ignore A and only use X the covariate. As result, we need to choose class of models that force the use of the treatment A. To address this, we use a two-headed model that regress on for A 0/1 separately in the conditional outcome learning model (See Section 4.2 and Figure 3).
As discussed in the introduction Section 1, this estimator yields a consistent point estimate, but does not offer simple approach for uncertainty quantification. A natural guess for an estimate of its variance
vår( (t) 1 (x)) 11
Q).
(4.4)
That is, just compute the variance of the mean conditional on the fitted model. However, this procedure yields asymptotically valid confidence intervals only if the outcome model converges extremely quickly; i.e., if (
o(n +). We could instead bootstrap, refitting Q on each bootstrap sample. However, with modern language models, this can be prohibitively computationally expensive.
4.2 TREATMENT IGNORANT EFFECT ESTIMATION (TI-ESTIMATOR)
Following Theorem it suffices to adjust for (x) (Q(O,x), Q(1x)). Accordingly, we use the following pipeline. We first estimate (x) and (x(x) (using neural language model), as with the outcome-only estimator. Then, take nix)= = (Q) (,)x),((()) and estimate & P(A= 1 f)). That is, we estimate the propensity score corresponding to the estimated representation. Finally, we plug the estimated Q and &, into standard double machine learning estimator (Chernozhukov et al., 2016).
We describe the three steps in detail.
5

---

Published as a conference paper at ICLR 2023
Q-Net In the first stage, we estimate the conditional outcomes and hence obtain the esti- mated two-dimensional confounding vector (x)). For concreteness, we will use the drag- onnet architecture of Shi et al. (2019). Specifically, we train DistilBERT (Sanh et al., 2019) modified include three heads, as shown in Figure 3. Two of the heads correspond (x) and (x) respectively. As discussed in the Section 4.1, applying two heads can force the model to use the treatment A. The final head is single linear layer predicting the treatment. This propensity score prediction head can help prevent (implicit) regularization of the model from throwing away XaaZ information that is necessary for identification. The output of this head is not used for the estimation since its purpose is to force the DistilBERT representation to preserve all confounding information. This has been shown improve causal estimation (Shi et al., 2019; Veitch et al., 2019).
We train the model by minimizing the objective function 1 - ni = (x; (x;;)) y;)² + (crossEEttroyy(,,
P mm
(4.5)
where 9 are the model parameters, a, B are hyperparameters and (m is the masked language modcling objective of DistilBERT.
There a final nuance. In practice, split the data into K-folds. For each fold j, we train model Q_, the other K K-1 folds. Then, we make predictions for the data points in fold j using 2 Slightly abusing notation, we use (x) to denote the pre- dictions obtained in this manner.
DEAT
XIX)
Indt mp wewerrr
Figure 3: The architecture of Q-Net follows the dragonnet (Shi et al., 2019) for estimation of Q Specifically, given representations A(X) from input text data, the Q-Net predicts Y for samples with A and A using two separate heads. A third head predicting A is also included for train- ing, though the predictions are not used for esti- mation. Parameters in DistiBERTT and three pre- diction heads are trained together end-to-end manner.
Propensity score estimation Next, and esti-
define ff(x == (Q.(x), mate the propensity score & (x) P(A =
1 1 fxx)). To do this, we fit non- parametric cstimator to the binary classi- fication task of predicting A from ()) in a cross fitting or K-fold fashion. The im- portant insight here is that since (x( is 2-dimensional, non-marae estimation is possible at a fast rate. In Section 5, we try several methods and find that kernel regression usually works well.
We also define (X) == P(A = 1 n(X)) as the idcalized propensity score. The idca is that as 7, will also have & 8n so long we have valid non-parametric estimate.
CDE estimation The final stage combine the estimated outcome model and propen- sity score into CDE estimator. To that end, we define the influence curve of cDE follows:
as
(xx r CDE) (Y -((xx))))
(x)
(1-8,((x) (1 (1-A)
(4.6)
p
where PAA=1) -1A=1) 1). Then, the standard double machine learning estimator of CDE Chernozhukov et al. (2016), and the a-level confidence interval of this estimator, is given
by
R
TTI
$o CI -""==+" - a/sd - -.. TTI + s (T/p)//)))
/p)),
(4.7)
where
A (V,
8
$.
(1-A,) (V, i= (4.8)
D
(1 8x (x))
1 Ais 2/-0/2 is the a/2-upper quantile of the standard normal, and sd(.) the sample standard deviation.
i Σリ
6

---

Published as a conference paper at ICLR 2023
Validity We now have an estimation procedure. remains to give conditions under which this procedure valid. particular, we require that it should yield consistent estimate and asymptotically correct confidence intervals.
Theorem 2. Assume the following.
1. The mis-estimation of conditional outcomes can be bounded as follows
max (E(0,1)
=on
(4.9)
The propensity seore function 1(A=1l:;;) is continnuuuo continuous on IR,, and 0,
P (+ S 8,,()) < (1=8)=
3. The propensity score estimate converges at least quickly as k nearest neighbor; ie., (x( P(A=11 f(())²² f((x)} O(n +) Györfi et al. (2002);
4. There exist positive constants C₁ C2 c, and q such that
[[ S C2
sup pESup(((())
E[(Y - Q(A,X)² n(x) = =1]]= | s C2.
E[ Q(4x-((4)))) c ac(o,1)
max - s...
Then, the estimator ttl is consistent and
vn( 4 N(O,003
(4.10)
where o² = Bereeo====xx(_eeeeeeeei 8x,
The proof is provided in Appendix A.
The key point from this theorem is that we get asymptotic normality at the (fast) T-- rate while requiring only a slow (n!/4) convergence rate of Q. Intuitively, the reason is simply that, because (x) only 2-dimensional, it is always possible nonparametrically estimate the propensity score from o at fast rate-even naive KNN works! Effectively, this means the rate at which we estimate the true propensity score (x( = P(A = rxX)) is dominated by the rate at which we estimate r)(), which is in turn determined by the rate for Q. Now, the key property of the double ML estimator is that convergence only depends on the product of the convergence rates of Q and 8. Accordingly, this procedure is robust in the sense that we only need to estimate Q at the square root of the rate we needed for the naive Q-only procedure. This is much more plausible in practice. As will see in Section 5, the Tl-estimator dramatically improves the quality of the estimated confidence intervals and reduces the absolute bias of estimation.
Remark 3. In addition to robustness noisy estimation of Q, there are some other advan- tages this estimation procedure inherits from the double ML estimator. If Q is consistent, then the estimator is nonparametrically efficient the sense that no other non non-parametric estimator has smaller asymptotic variance. That is, the procedure using the data effi- ciently as possible.
5 EXPERIMENTS
We empirically study the method's capability to provide accurate causal estimates with good uncertainty quantification Testing using semi-synthetic data (where ground truth causal effects are known), we find that the estimation procedure yields accurate causal estimates and confidence intervals. particular, the Tl-estimator has significantly lower absolute bias and vastly better uncertainty quantification than the Q-only method.
Additionally, we study the effect of the choice of nonparametric propensity score estimator and the choice of double machine- machine-learning estimator, and the method's robustness in regard to Q's miscalibration. These results are reported in Appendices C and D. Although these
7

---

Published as a conference paper at ICLR 2023
The 'l-estimator significantly improves both bias and coverage relative to the baseline. Tables show average absolute bias and confidence interval coverage of CDE esti- mates, over 100 resimulations. The estimator & displays higher accuracy/smale solute bias of point estimate and much larger coverage proportions compared to outcome- only estimator 59. The treatment level equals true CDE, which takes 1.0 (with causal effect) and 0.0 (without causal effect). Low and high noise level corresponds 1.0 and 4.0. Low and high confounding level corresponds to &. to 50.0 and 100.0.
Table 1:
ab-
set
(a) Average absolute bias
Noise:
High
True CDE:
1.0
0.0
1.0
0.0
Confounding:
High
Low
High
High
High
0.100
0.376
0.076
0.326
0.563
0.548
0.502
0.498
0.059
0.074
0.049
0.089
Coverage proportions 95% confidence intervals
Noise:
Low
High
True CDE:
1.0
Confounding: Low
High
Low
High
Low
High
Low
High
49
0%
0%
2%
0%
0%
0%
0%
0%
t
57%
84%
57%
79%
87%
80%
77%
81%
choices do not matter asymptotically, we find they have significant impact in actual finite sample estimation. We find that, in general, kernel regression works well for propensity score estimation and the vanilla the Augmented Inverse Probability of Treatment weighted Estimator (AIPTW) corresponding to the CDE works well.
Finally, we reproduce the real-data analysis from Pryzant et al. (2020). We find that po- liteness has a positive effect on reducing email response time.
5.1 AMAZON REVIEWS
Dataset We closely follow the setup of Pryzant et al. (2020). We use publicly available Amazon reviews for music products as the basis for our semi-synthetic data. We include reviews for mp3, CD and vinyl, and among these exclude reviews for products costing more than $100 shorter than 5 words. The treatment A is whether the review is five stars (A= = 1) or one/two stars 0):
To have ground truth causal effect, must simulate the outcome. To produce a realistic dataset, wc choose a real variable as the confounder. Namcly, the confounder C is whether the product is a CD (C 1) or (C 0). Then, outcome Y generated according to Y - B, (o(CC)BB)+yNG 1). The true causal effect is controlled by Ba. We choose Pg 1.0, 0.0 generate data with and without causal effects. In this setting, B, is the oracle value of our causal estimand. The strength of confounding is controlled by B.- We choose B. = 50.0, 100.0. The ground-truth propensity score T(C) = P(A = 1[C). We set it to have the value :(0)= = 0.8 and T(1) = 0.6 (by subsampling the data). B, is an offset [[π[C)]] = (O))P(C = 0) + ({1)P(C = 1), where P(C = a), a = 0, are estimated from data. Finally, the noise level is controlled by y; choose 1.0 and 4.0 to simulate data with small and large noise. The final dataset has 10, 685 data entries.
Protocol For the language model, we use the pretrained distilbert basert--aae-unccsee model provided by the transformers package. The model is trained in the k-folding fashion with 5 folds. We apply the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 2e and batch size of 64. The maximum number of epochs set as 20, with early stopping based on validation loss with a patience of 6. Each experiment is replicated with five different seeds and the final Q(a, x,) predictions are obtained by averaging the predictions from the resulting models. The propensity model implemented by running the Gaussian process regression using WhiteKernel kernel. (We choose different random state for the GPR to guarantee the convergence of the GPR.) The coverage experiment uses 100 replicates.
in the sklearn package with PotPuotuct +
8

---

Published as a conference paper at ICLR 2023
Table 2: Politeness has positive causal effect on response time. The table displays differ- ent CDE estimates and their 95% confidence intervals. 'he unadjusted one the difference of sample means of treatment (polite) group and control group. The confidence interval of f" only covers positive values, which means politeness can increase the probability of timely response.
Estimator
CDE
Confidence Interval
unadjusted A naive
-0.038
[ [-0.0679, -0.0073
0.195
[ [011910, 0. 0.1993
A
0.200
[ [0.1708, 0. ]
Results The main question here is the efficacy of the estimation procedure. Table com- pares the outcome-only estimator 42 and the estimator t First, the absolute bias of the new method is significantly lower than the absolute bias of the outcome-only estimator: This particularly true where there is moderate to high levels of confounding, Next, we check actual coverage rates over 100 replicates of the experiment. First, we find that the naive approach for the outcome-only estimator fails completely. The nominal confidence interval almost never actually includes the true effect. It is wildly optimistic. By contrast, the confidence intervals from the new method often cover the truc value. This an enor- mous improvement over the basclinc. Neverthcless, they still do not actually achieve their nominal (95%) coverage. This may be because the Q estimate is still not good enough for the asymptotics kick in, and we are not yet justified in ignoring the uncertainty from model fitting.
5.2 APPLICATION: CONSUMER COMPLAINTS TO THE FINANCIAL. PROTECTION BUREAU
We follow the same pipeline of the real data experiment in (Pryzant et al., 2020, $6.2). The dataset consumers complaints made to the financial protection. Treatment A is politeness (measured using Yeomans et al. (2018)) and the outcome Y is a binary indicator of whether complaints receive a response within 15 days.
We the same training procedure for the simulation data. Table 2 shows point esti- mates and their 95% confidence intervals. Notice that the naive estimator show a signifi- cant negative effect of politeness accurate AIPTW method as well as the outcome-only estimator have confidence intervals that cover only positive values, so conclude that consumers' politeness has positive effect on response time. This matches our intuitions that being more polite should increase the probability of receiving a timely reply.
reducing response time. On the other hand, the more
6 DISCUSSTON
In this paper, we address the estimation of the causal effect of text document attribute using observational data. The key challenge is that we must adjust for the text to handle confounding- but adjusting for all of the text violates overlap. We saw that this issue could be effectively circumvented with a suitable choice of estimand and estimation procedure. In particular, have seen estimand that corresponds to the qualitative causal ques- tion, and an estimator that is valid even when the outcome model is learned slowly. The procedure also circumvents the need for bootstrapping, which is prohibitively expensive in our setting.
There are some limitations. The actual coverage proportion of our estimator below the nominal level. This is presumably due to the imperfect fit of the conditional outcome model. Diagnostics (see Appendix D) show that as conditional outcome estimations become more accurate, the TI estimator becomes less biased, and its coverage increases. It seems plau- sible that the issue could be resolved by using more powerful language models.
Although we have focused on text in this paper, the problem of causal estimation with apparent overlap violation exists in any problem where we must adjust for unstructured and high-dimensional covariates. Another interesting direction for future work understand how analogous procedures work outside the text setting.
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
Thanks to Alexander D'Amour for feedback on an earlier draft. We acknowledge the Uni- versity of Chicago's Research Computing Center for providing computing resources. This work was partially supported by Open Philanthropy.
REFERENCES
Susan Athey, Guido Imbens, Thai Pham, and Stefan Wager. Estimating average treatment effects: Supplementary analyses and remaining challenges. American Economic Review, 107(5):278-8)) 2017.
Ananth Balashankar, Sunandan Chakraborty, Samuel Fraiberger, and Lakshminarayanan Subramanian. Identifying predictive causal factors from news streams. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2338- 2348, 2019.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. and causal parameters. arXiv preprint arXiv: 00060, 2016.
machine learning for treatment
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, machine learning of treatment effects. American Economic Review, 107(5):261-65, May 2017a. doi: 10.1257 2257/2cc.200170338. URL http:/ wwwp://wwwwco www.wb.wwww.............. 1ttp:/arrrcceiid11111011111 her.pp217000..
and Whitney Newey,
Victor Chernozhukov, Denis Cherverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 2017b.
Alexander D'Amour and Alexander Franks. Deconfounding scores: Feature representations for causal effect estimation with weak overlap. arXiv preprint arXiv:2104.0576, 2021.
Alexander D'Amour, Peng Ding, Avi Feller, Lihua Lei, and Jasjeet Sekhon. Overlap in ob- servational studies with high-dimensional covariates. Journal of Econometrics, 221 (2): 644-654, 2021.
Fermin Moscoso del Prado Martin and Christian Brendel. Case and cause in icelandic: Reconstructing causal networks of cascaded language changes. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2421-221222 2421-2430, 2016.
Naoki Egami, Christian Fong, Justin Grimmer, Margaret E. Roberts, and Brandon M Stew- art. How to make causal inferences using texts. arXiv preprint arXiv:1802.02162 2018.
Christian Fong and Justin Grimmer. Discovery of treatments from text corpora. In Proceed- ings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1600-1609, 2016.
László Györfi, Michael Kohler, Adam Krzyzak, Harro Walk, et al. A distribution-free theory of nonparametric regression, volume 1. Springer, 2002.
Allie Hall. How hiring language reinforces pink collar jobs. Textio Word Nerd, 2017.
Daniel Karell and Michael Freedman. Rhetorics of radicalism. American Sociological Review, 84(4):726-753, 2019.
Katherine A Keith, David Jensen, and Brendan O'Connor Text and causal inference: A review of using text to remove confounding from causal estimates. arXiv preprint arXiv:2005.00649, 2020.
Emre Kiciman, Scott Counts, and Melissa Gasser. Using longitudinal social media analysis understand the effects of early college alcohol use. Twelfth International AAAI Conference on Web and Social Media, 2018.
10

---

Published as a conference paper at ICLR 2023
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6900, 2014.
Sheng Li, Nikos Vlassis, Jaya Kawale, and Yun Fu. Matching via dimensionality reduction for estimation of treatment effects in digital marketing campaigns. In IJCAI, pp. 3768- 3774, 2016.
Christos Louizos, Uri Shalit, Joris M Mooij, David Sontag, Richard Zemel, and Max Welling. Causal effect inference with deep latent-variable models. Advances in neural information processing systems, 30, 2017.
Subramani Mani and Gregory Cooper. Causal discovery from medical textual data. In Proceedings of the AMIA Symposium, pp. 542. American Medical Informatics Association, 2000.
Reagan Mozer, Luke Miratrix, Aaron Russell Kaufman, and L Jason Anastasopoulos. Match- ing with text data: An experimental evaluation of methods for matching documents and measuring match quality. Political Analysis, 28(4):445 16844446566--1 2020.
Kevin P Murphy, Probabilistic Machine Learning: Advanced Topics. MIT Press, 2023. URL http://probml.githee www.bbmml...tt...... htt.ggttu.u...obo...... 1o/book2.
Alexandra Olteanu, Onur Varol, and Emre Kiciman. Distilling the outcomes of personal experiences: A propensity-scone sccored analysis social media. In Proceedings of the 2017 ACM Conference Computer Supported Cooperative Work and Social Computing, pp. 370-386, 2017.
J. Pearl. Causality. Cambridge University Press, 2nd edition, 2009.
Reid Pryzant, Dallas Card, Dan Jurafsky, Victor Veitch, and Dhanya Sridhar. Causal effects of linguistic properties. arXiv preprint arXiv:2010.129: 12919, 2020.
Jeremy Rassen, Robert J Glynn, M Alan Brookhart, and Sebastian Schneeweiss. Covari- ate selection in nigh-dimensional propensity score analyses of treatment effects in small samples. American journal of epidemiology, 173(12):1404-1441 2011.
Margaret E Roberts, Brandon M Stewart, and Richard A Nielsen. Adjusting for confounding with text matching. American Journal of Political Science, 64(4):887-903, 2020.
Koustuv Saha, Benjamin Sugar, John Torous, Bruno Abrahao, Emre Kıcmman, and Munmun De Choudhury. A social media study on the effects of psychiatric medication use. In Proceedings of the International AAAÍ Conference on Web and Social Media, volume 13, pp. 440-451, 2019.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
Claudia Shi, David M. Blei, and Victor Veitch. Adapting neural networks for the estimation of treatment effects. In Advances in Newral Information Processing Systems, 2019.
Dhanya Sridhar and Lise Getoor. Estimating causal effects of tone in online debates. arXiv preprint 1xXiv:1906004177, 2019.
Dhanya Sridhar, Aaron Springer, Victoria Hollis, Steve Whittaker, and Lise Getoor. Esti- mating causal effects of exercise from mood logging data. In IJCAI/ICML Workshop on CausalM1,, 2018.
Narges Tabari, Piyusha Biswas, Bhanu Praneeth, Armin Seyeditabari, Mirsad Hadzikadic, and Wlodek Zadrozny. Causality analysis of twitter sentiments and stock market returns. Proceedings of the first workshop on economics and natural language processing, pp. 11- 19, 2018.
11

---

Published as a conference paper at ICLR 2023
Chenhao Tan, Lillian Lee, and Bo Pang. The effect of wording on message propa- gation: Topic-and author-controlled natural experiments on twitter. arXiv preprint arXiv:1405.1435 2014.
Victor Veitch, Dhanya Sridhar, and David M. Blei. Using text embeddings for causal infer- ence. arXiv e-prints, art. arXiv:1905.27441 May 2019.
Zhao Wang and Aron Culotta. When do words matter? understanding the impact of lexical choice on audience perception using individual treatment effect estimation. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 7233-7240, 2019.
Zach Wood-Doughty, Ilya Shpitser, and Mark Dredze. Challenges of using text classifiers for causal inference. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, volume 2018, pp. 4586. NIH Public Access, 2018
Michael Yeomans, Alejandro Kantor, and Dustin Tingley. The politeness package: Detecting politeness in natural language. R Journal, 10(2), 2018.
Justine Zhang, Sendhil Mullainathan, and Cristian Danescu-Niculescu-Mizz Quantify- ing the causal effects of conversational tendencies. Proceedings of the ACM on Human- Computer Interaction, 4(CSCW2))11244 2020.
12

---

Published as a conference paper at ICLR 2023
A PROOF OF ASYMPTOTIC NORMALITY
Theorem 2. Assume the following.
The mis-estimation of conditional outcomes can be bounded as follows
max eE(0,1)
=(n
(4.9)
The propensity score function P(A= is Lipschitz continuous on R², and 3 P P( 11-)==
3. The propensity score estimate converges at least quickly as k nearest neighbor; O(n Györfi et al. (2002);
i.e., - P(A= n(x))² [{))]
There exist positive constants C₁ Cz c, and q > such that
E[ | S Cz.
sup (ESupp(((0))
E[(Y - -(A,X) m(x) = n)] S C2.
E[(Y - Q((x))
max (x) s.. aE(0.1)
Then, the estimator T is consistent and
/n(*" - N(O,
(4.10)
where 0² = (,,( TDB)
Proof. We first prove that misestimation of propensity score has the rate n i, For simplic- ity, we use fg. f.: (u, R² denote conditional probability P(A 1|u, vv(u,v v) and the estimated propensity function by running the nonparametric regres- sion P(A v) f (u, v). Specifically, we have ((Q(0,)),Q(1xX))) (x)))))) P(A 1x),()),((()) 8x,x). Since B[(Qn(X)- - ( Q(,, J!, -((Q,(x)-Q(,xx))) o 1/44 and fg Lipschitz continuous, we have
8,(X) and
s [100000000000-
E
|| [00000...0000
27
SL E
(A.1)
2
=1. (((()))))0000))))=+((())) =o
Since the true propensity function fg is Lipschitz continuous on R.2, the mean squared eтror rate of the k nearest neighbor is 01/2 Györf et al. (2002). In addition, since the propensity score function and its estimation are bounded under we have the following equation
=/100.00000000--1,,00000... 2
1222
(A.2)
1

---

Published as a conference paper at ICLR 2023
due to the dominated convergence theorem. By (A.1) and (A.2), we can bound the mean squared error estimated propensity score in the following form:
[8, (,,)))
< (08())))-[(((0000)()))))))) 1 -
E (1000))),000000010000000000,0,
(A.3)
-2(
that is
=0
Before we apply the conclusion of Theorem 5.1 in (Chernozhukov et al., 2017b), we need to check all assumptions in Assumption 5.1 hold in Chernozhukov et al. (2017b), Let C: max (ex((22) (2c] + 29)8, J
(a) B[Y - Q(A,X) (X),A] = 0, B[A 8,(X) [x)]] 0 are casily checked by invoking definitions of Q and 8
(b) BIYNJ < C, E[(Y -(Ax))) c, and
sup nxsupp(((()) [ Q(A,x)) (x) = n) S C are guaranteed by the fourth condition in the theorem.
(c) (( S (x) < 1- $) is the second condition in the theorem.
(d) Since propensity score function and its estimation are bounded under 1, we have
-t()::)))))/((((00000000000000)0 (x) - 11)
$ (C + +
C
(e) Based on (A.3) and condition 1 in the theorem, we have
E() 1x))))
s <(onrb))o
so(n
B[ [[(26()) -1,
b)
(f) Based on condition 3 in the thcorem, we have
sup - P(A= x(x))) (xx = -1.
xxxupp(X)
2

---

Published as a conference paper at ICLR 2023
We consider smaller positive constant & instead of e. Note that for < we still - E) = 1. Then,
have P( s 8,((X) - 1
1 - =P
P
sup (resupp(x)
8,(x)
inf (resupp(X) 8,((2)
< -0)===+++ +P
sup
8,(()) > 1 1
2 2
(resupp(x)
inf xxssup(X))
P(A (x) f(xx)
inf xesupp(X)
8, 8-8)
sup (,(x) sup P(A ((x) f(xx)(()))))))) > 1-1-11-191
resupp(X)
xesupp(X)
= [inf (infxesupp()) (x - inf xesuppp)( P(A= (()= -(x)))
(E 8)2
E
(SuPrexppXX) 8,,(x) - sup xespps)))( P(A= f(x) = -(x))
(e (6-E) E)2
2 xsupxeppp(x) E - - 1(x)=
(8- 8)2
=0
Hence, respp)es (x) / - 1)
With (a)-(0, we can invoke the conclusion in Theorem 5.1 in (Chernozhukov et al., 2017b), and get the asymptotic normality of the estimator.
B PROOF OF CAUSAL IDENTIFICATION
Theorem 1. Assume the following:
1. (Causal structure) The causal relationships among A, Ā, Z, Y, and satisfy the causal DAG
in Figure 2;
2. (Overlap) 0 < P(A= | X2) <
3. (Intention equals perception) A = Å almost surely with respect all interventional distri- butions. Then, the CDE is identified from observational data as
CDE = T CDE : Bxja= [E[Y (x)x 1]-11100 E[Y =xx),A==]]]
(3.4)
where nxX) ::0))==((0000) (Q(0.)))((11))
Proof. We first prove that this two-dimensional confounding part r)() satisfies positivity.
Since (Q),,X), Q(1,X)) (E A = 1,X,, E[Y A= 0, is a function
of (Xazz, X2), the following equations hold:
P(A ((1,)))(( =E(A Q(0, Q(1,x))((1,X)
= (A X Xz) ((0,)),
(B.1)
PPA Xxnzx [ ((0,))
As 0 1| 1, we have 0 < P(4 2(1 )0)) <. Furthermore, we have P(A= ((1=1]((0,x).0(11,))) due to almost everywhere equivalence A and Å.
3

---

Published as a conference paper at ICLR 2023
Since A A= A, we can rewrite (3.1) by replacing A with Á in the following form:
CDE = Kanz X, 1 A==
E(Y do 1,
do(A xx) B(Y 1,= 0, 0,
Ex Exanx x AA=1
Bx Kune xx 1 A
E(Y (===1,x)-- - IĀ=
=Ex
[EY Ā= A=,,0(000),000.x))
= mxx | A=1
[[( | -
-B -=
=Bx A=1
IE(Y -= B A= o= (x))].
(B.2)
The equivalence of the first and the second line because X X₂ block all backdoor paths between À and Y (See Figure 2) and 0 < P(Ā = 1 Q( "do-operation" in the first line can be safely removed. Equivalence of the second line and the third line is due to Q(Ā,X) = E(Y A, xx,, which is subject to the causal model in Figure 2. The last equation is based on the fact that n(X is function of only XAAz and Xz. (It can be easily checked by using the definition of the expectation.)
< 1. Thus, the
(B.2) shows that (Q, Q(1,X) is two-dimensional confounding variable such that CDE is identifiable when we adjust for as the confounding part.
Note that f and h are two invertible functions on IR, (((Q(O X),, h(Q(1,)))) also suffices the identification for CDE. Since the sigma algebra should be the same for (((0,)),((1,3)) and ((Q(0,x)), iee. o ((00))))(.)))===((00))))))
Hence, we have
P(A= 1 1100000000..0003/311100000000000
(B.3)
C ADDITIONAL EXPERIMENTS
We conduct additional experiments to show how the estimation of causal effect changes over different nonparametric models for the propensity score estimation, and 2) when using different double machine learning estimators on causal estimation. Specifically, for the first study, we apply different nonparametric models and the logistic regression the estimated confounding part k{X) (X), Q (x)) to obtain propensity scores. We use ATT AIPTW in all above cases for causal effect estimation. For the second study, we fix the first two stages of the TI estimator, ie. we apply Q-Net for the conditional outcomes and compute propensity scores with the Gaussian process regression where the kemel function is the summation of dot product and white noise. Estimated conditional outcomes and propensity scores are plugged into different double machine learning estimators. We make the following conclusions with results of above experiments.
The choice of nonparametric models is significant. Table summarizes results with ap- plying different regression models for the propensity estimation. We can see that suitable nonparametric models will strongly increase the coverage proportion over true causal esti- mand. Therefore, we conclude that the accuracy in causal estimation is highly dependent on the choice of nonparametric models. In practice, when there is some prior informa- tion about the propensity score function, we should apply the most suitable nonparametric model to increase the reliability of our causal estimation.
The ATT AIPTW consistently the best double machine learning estimator. Table 4 shows results by applying different double machine learning estimators. We apply both estimators for the average treatment effect (ATE) and the controlled direct effect (CDE). The bias of "unadjusted" estimator Auaive is also included in Table (a). For absolute bias,
4

---

Published as a conference paper at ICLR 2023
ATT AIPTW f has comparable results with other double machine learning estimators in most cases. For coverage proportion of confidence intervals, though has lower rates in some cases, f has consistently the best performance. Especially in high confounding situations, the advantage of tLl is obvious.
Estimator For cach dataset, compute estimators as follows. n, and Hg stands for the number of individuals in the treated and controlled group. = + ng is the total number of individuals.
"Unadjusted" baseline estimator: &
1 Σ L -1 Y no - Yi
"Outcome-only" estimator: 40 1 "1 Lin, Q1; Qoii
ATT AIPTW: TT 1 - A
Qo,))
Table 3: The choice of nonparametric models for the Tl-estimator significant. 'Tables show average absolute bias and 95% confidence intervals' coverage t with applying different nonparametric models in the second stage. The Gaussian process regression with the dot product+ white noise kernel has the best performance (lowest absolute bias and highest coverage proportion). The treatment level is equal true CDE, which takes 1.0 (with causal effect) and 0.0 (without causal effect). Low and high noise level corresponds to y 1.0 and 4.0. Low and high confounding level corresponds to B. = 50.0 and 100.0.
(a) Average absolute bias
Noïse:
Low
High
Treatment (oracle causal effect):
1.0
0.0
1.0
0.0
Confounding:
Low
High
Low
High
Low
High
Low
High
GPR (Dot Producr-Whit Noise)
0.069
0.059
0.113
0.074
0.088
0.049
0.002
0.089
(RBF)
0.150
0.348
0.156
0.329
0.363
0.452
0.344
0.424
KNN
0.147
0.334
0.144
0.313
0.316
0.372
0.304
0,356
AdaBoost Logistic
0.074
0.349
0.061
0.323
0.526
0.497
0.479
0.464
0.070
0.057
0.114
0.073
0.086
0.047
0.001
0.087
(b) Coverage proportions of 95% confidencc intervals
Noise:
Low
High
Treatment (oracle causal effect):
1.0
1.0
Confounding:
Low
High
Low
High
Low
High
Low
High
GPR (Dot Product-Whit Noise)
57%
84%
57%
79%
87%
80%
77%
81%
GPR (RAF)
31%
0%
41%
0%
7%
7%6
17%
19%
KNN
18%
0%
39%
11%
8%
AdaBoost Logistic
25%
0%
35%
0%
0%
0%
0%
0%
84%
D DISCUSSION OF LOW COVERAGE
In this section, we discuss why the confidence intervals we get (See Table 1) have lower coverage than the nominated level 95%. We conduct diagnostics and find that the inaccu- racy of Q's estimations is responsible for the low coverage. We compute absolute biases, variances, and coverages of - T with different mean squared errors Te[(Q-Q) by using Figure 4-Figure 5, as the mean squared error
different numbers of datasets. According of Q increases, the bias of - grows and the coverage of - drops. Specifically, the highest coverage of each setting almost 95% (use 50 datasets with most accurate conditional outcome estimations). In practice, one direct way to improve the TI estimator's accuracy is apply better NLP models so that more accurate conditional outcome estimations can be obtained.
5

---

Published as a conference paper at ICLR 2023
Table 4: The ATT AIPTW is consistently the best double machine learning estimator for this causal problem. Tables show average absolute bias and 95% confidence intervals' coverage of different causal estimations. ATT AIPTW tt1 shows consistently the lowest absolute bias and highest coverage rate. For propensity score estimation, the Gaussian process regression with the dot product+ white noise kernel is applied for all estimators. The treatment level is equal to true CDR/true ATK, which takes 1.0 (with causal effect) and 0.0 (without causal effect). Low and high noise level corresponds y 1.0 and 4.0. Low and high confounding level corresponds B. 50.0 and 100.0.
(a) Average absolute bias
Noise:
High
Treatment (oracle CDE):
1.0
0.0
1.0
0.0
Confounding:
High
High
High
High
wnadjusted gaune
1.071
2.143
1.071
2.1453
1.068
2,140
1.069
2.140
ATE BMM
0.094
0.176
0.128
0,193
0.122
0.106
0.061
0.140
ATE IPTW
.0.574
-1.492
-1.839
-1.807
0.082
-0.592
-0.393
-0.649
AIPTW: 4TT
ATT BMM
0.075
0.147
0.031
0.062
0.621
0.454
0.464
0.337
ATT TMLE:
0.194
0.196
0.186
0.136
0.174
Coverage Proporroon 95% confidence intervals
Noise:
Law
Hlgh
Treatment (oracle CDE):
1.0
0.0
1.0
0.0
Confounding:
Low
Hlgh
Low
Hligh
Low
[lihh
Low
Hligh
69%
79%
79%
ATE BMM
39%
35%
70%
36%
75%
79%
79%
71%
ATE 1PTW
11%
1%6
0%
1%
90%
39%
44%
37%
ATT AIPTW: +71
57%
84%
57%
87%
80%
77%
81%
ATT BMM
26%
4%
49%
41%
1%
3%
156
14%
6

---

Published as a conference paper at ICLR 2023
0.110
0.0160 0.0165 0.0170 0.0175
0.100
Variance
Bls
0,050
5.35
5.45
5.55
5.65
5.35
5.45
5.55
5.65
Q loss
loss
0.91
Coverage
0.89
0.87
5.35
5.45
5.55
5.65
Q loss
Figure 4:
Absolute biases and variances increase while coverages decrease as the mean squared errors of Q (Q loss) becomes larger. This experiment uses 100 datasets with B. 1 (with causal effect), B. 50.0 (low confounding), and y 4.0 (high noise).
7

---

Published as a conference paper at ICLR 2023
0.115
Varfannc
B85
0.105
0.095
13.5
13.6
13.7
13.8
13.9
14.0
13.5
13.6
13.7
13.8
13.9
14.0
Q loss
loss
0.92
0.88
Coverage
0.84
0.80
13.5
13.6
13.7
13.8
13.9
14.0
Q loss
Figure 5: Absolute biases and variances increase while coverages decrease as the mean squared errors of becomes larger. This experiment uses 100 datasets with B. = (with causal effect), B. 100.0 (high confounding), and y 4.0 (high noise).
8


Reviews:
[{"review_id": "s4z_QYwde_", "reviewer": "Reviewer_yJuA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper concerns causal effect estimation when the confounders and treatment are both derived from texts. One issue herein is that when adjusting for the entire text, there is a violation of the overlap assumption which is required for drawing valid causal conclusions. The proposed solution is to use supervised representation learning to produce a data representation that preserves confounding information while eliminating information that is only predictive of the treatment. ", "summary_of_the_review": "This paper analyzes the identification conditions for causal effect estimation when there is an apparent violation of overlap. The paper then proposed to use a neural network to learn a representation of the text to predict the outcome, which is then fed into the common double machine learning method to estimate the average treatment effect for the treated. There is a seeming gap between the conceptual idea and the implementation (which should be addressed after some revision/clarification). The author should also address how their paper is novel compared to previous methods (Shi et al.) and theory (that of double machine learning).", "strength_and_weaknesses": "+1. the paper is well-written and technically correct.\n+2. there are theoretical justifications for the proposed methods.\n+3. the proposed method is backed up by simulations and two real data applications (Amazon reviews and consumer complaints.)\n+4. the topic of the paper is increasingly important and the paper is very timely.\n\n-1. there seems to be some small gap between the advocated main idea in Figure 2 (namely, one should eliminate the components in X that are predictive of A only) and the proposed method, which does not specifically make an effort to eliminate these components. Specifically, the authors proposed to use a three-head dragonnet where two heads predict Q0 and Q1 and one head predicts A. In the end, only the estimated Q0 and Q1 are used for making the causal effect estimation. Should I understand that the estimated A in the dragonnet represents the components in X predictive of A? If that's the case, how can we ensure that the two heads for Q0 and Q1 do not use information in X_A?  If not, how should we make the connection between the conceptual idea in Figure 2 and the implementation in Figure 3?\n-2. related to (-1) above, it seems that a critical point is that Equation (3.3) Q(\\tilde A, X) := E(Y | \\tilde A,X) is in fact E (Y | \\tilde A, X_A∧Z, X_Z). This seems to suggest that although conceptually CDE as in (3.2) should be based on X_A∧Z, X_Z which are unknown, in reality, one only needs to build models for Q(A,X) using the entire X. In this case, how does the proposed method differ from the naive method that \"naively adjust for all the text as the confounding part\"? Some clarification is needed.\n-3. It would be helpful to add the \"naively adjust for all the text\" method to the simulation, to see how this method is biased and problematic.\n-4. Clarification needed: following Theorem 1, the author said \"We emphasize that, regardless of whether the overlap condition holds or not, the propensity score of η(X) in condition 2 is accessible and meaningful.\" There is no η(X) in condition 2, and since η(X) is the outcome model, it does not have a propensity score.\n-5. abbreviation: On page 7, it says \"AIPTW is double robust\". AIPTW was not previously defined. AIPTW = augmented inverse probability T??? (treatment?) weighting? If you mean the estimator in (4.8), it should be said so.\n-6. bias: about Table 1, you said \"bias of the new method is significantly lower than the bias of the outcome-only estimator\". Bias can be both positive and negative, and hence lower bias is not necessarily better. Do you in fact mean average estimation error or root mean square estimation error? In fact, it would be good for you to define the bias that you are reporting in the paper. \n-7. simulation. It is counter-intuitive that low confounding actually leads to a low coverage rate of the confidence intervals. In randomized trials (zero confounding), shouldn't the confidence interval be valid? Some explanation/discussion is needed.\n-8. Please explain the specific differences between the current work and Claudia Shi, David M. Blei, and Victor Veitch. Adapting neural networks for the estimation of treatment effects. In Advances in Neural Information Processing Systems, 2019.\n-9. I am not sure how novel is the proof of Theorem 2. The stated results seem to inherit from double machine learning in Chernozhukov's work. The author should comment on how their proof differs from that of Chernozhukov. This is important as this part is claimed to be the second main contribution of the paper. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PbLlxef2sPs", "reviewer": "Reviewer_SeHw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the problem of estimating the causal effect of an attribute of a text on some outcome variable under a setting where overlap is violated, i.e., the treatment variable is fully determined by the text features. Under the assumption that the problem satisfies the constraints of a given causal model (Figure 1), the authors propose an identification formula for the target effect that adjusts for part of the text to block confounding while resolving the overlap violation. Then, they propose an estimation procedure for this formula using standard double machine learning. Empirical evaluation shows the advantage of the proposed technique over baseline work.", "summary_of_the_review": "The score reflects the concerns raised regarding the assumptions in the causal model and the derivation of the CDE expression which are discussed in the weaknesses section. I would be happy to revisit the score based on further clarification from the authors.", "strength_and_weaknesses": "Strengths: The evaluation of the proposed method shows a clear advantage over standard previous methods for estimation.\n\nWeaknesses and Comments:\n- CDE formulation: The authors state the following while explaining the formulation of the CDE expression on page 4: \"our formal causal effect aims at capturing the effect of A through only the first, direct, path\". Obviously, identifying the total effect of A on Y is different than identifying the direct effect, or at least the variant suggested in Equation 3.1. I'm missing the reasoning for why this is the purpose of the computation. Further clarification from the authors would be appreciated.\n\n- Generality of the causal DAG (Figure 1): The model assumes the absence of confounding between $X$ and $Y$ and between $A$ and $Y$. In general, it is not clear how valid this assumption is for estimating the causal effects of attributes of a text document even if it is justified for the suggested problem of sentiment effect on sales.\n\n- Citations: Most of the introduction states technical claims without any citation to back it. For example, the second paragraph discusses the identification of causal effects and the required conditions but there are no citations to support that.\nAnother example is in the following paragraph which states that \"it is often reasonable to assume that text data has information about all common causes...\"\n\n- typo: p.5, \"this estimator is yields\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ObMo0UxPnST", "reviewer": "Reviewer_HJbF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a way to overcome the violation of positivity in causal inference with text data. Following the problem setup of Pryzant et al. (2020), a piece of text X contains both the treatment (assumed binary) and confounders. The authors find a sufficient statistic $\\eta$ that prevents the violation of positivity. Another contribution of this paper is to combine the approach with the double machine learning idea and obtain easy valid confidence intervals.", "summary_of_the_review": "This paper proposes a way to overcome the violation of positivity in causal inference with text data. Following the problem setup of Pryzant et al. (2020), a piece of text X contains both the treatment (assumed binary) and confounders. The authors find a sufficient statistic $\\eta$ that prevents the violation of positivity. Another contribution of this paper is to combine the approach with the double machine learning idea and obtain easy valid confidence intervals.", "strength_and_weaknesses": "### Strengths\n* The problem is this paper is quite practical and interesting.\n* Theorem 1 seems to be clever. It finds a sufficient statistic that blocks the paths that lead to violation of positivity.\n* The paper is clear and easy to follow.\n\n### Weaknesses\n* Theorem 2 is a rather simple application of the DML approach. The theoretical results are also straightforward corollaries based on Chernokhukov et al (2016).\n\n### Request for clarifications:\nI am confused about several parts of the paper and I would like the authors to respond before I make the final decision.\n\n1. **The positivity issue**: Given positivity violation, the estimator of $Q(\\neg A, X)$ should be quite unreliable, where I assumed that the natural pair is $(X, A)$ and $\\neg A$ is the treatment that is not observed. There should be very few examples of $X$ and $\\neg A$. This might lead to violations of the assumptions of Theorem 2. Can you comment on this?\n2. **Beginning of Section 4.2**: Why is $\\eta$ a good sufficient statistic for $\\hat{g}$?\n3. **End of Section 3**: The phrase \"propensity score of $\\eta(X)$\" does not make sense.\n4. **Start of Section 4.1**: The phrase \"naive outcome _regression_\" is the right choice of words.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HkziTNmrjH7", "reviewer": "Reviewer_m2tm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the problem of estimating causal effect with overlap violations. The authors focus on a specific application---causal estimation for text data. The authors start with writing down the model in terms of a DAG and discuss identification results based on the DAG. The authors then proceed with discussing estimation strategies; they propose an outcome only estimator and a doubly robust TI-estimator. They further establish a theorem that allows inference on the proposed TI-estimator. Finally, the authors demonstrate the performance of the proposed estimator through simulations and real data analysis.", "summary_of_the_review": "Overall, I find this paper a good contribution to the literature. Here’re some comments/questions I have. \n\n-\tCorrect me if I’m wrong, but the identification result is not new: think about the line of research in prognostic score. It will be helpful if the authors can discuss the connection of the paper to the prognostic score literature. The general theme of the proposed method is the same as using prognostic score as deconfounding score in D’Amour & Franks (2021). The authors discuss briefly on page 3 the differences between this paper and D’Amour & Franks (2021). To me, the paper is still a good one with an interesting application---text data and a nice neural network model, but emphasizing more on the differences and new contributions can help readers understand the novelty of the paper better. \n\n-\tAssumptions in Theorem 2. Do the two rates of convergence have to be o(n^(1/4))? Is it possible for the theorem to hold true if one is smaller while the other is larger (in the sense that as long as the product is o(n^(1/2)))? Thanks!\n\n-\tIt will be helpful to state the full name of “CDE”. \n\n-\tThere are some typos. Here are a few examples:\n\no\tPage 3 line 3. “We assume there” -> “We do not assume”\n\no\tPage 7 line 1. “It remains to given” -> “It remains to give”\n", "strength_and_weaknesses": "Strengths\n\n-\tThe paper is very well-written. The proposed method is motivated and explained in a very nice way. \n\n-\tThe paper addresses an interesting and important question: causal estimation with overlap violations.\n\n-\tThe proposed method is very elegant conceptually. \n\n-\tThe proposed method appears to have good performance both theoretically and empirically. \n\nWeaknesses\n\n-\tThe identification result is not new: it is the same as using prognostic score as deconfounding score in D’Amour & Franks (2021). See the next section for more details. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster