SlowGuess/NAACL_2021_OCR huggingface dataset:
Reid Pryzant $^{1}$  Dallas Card $^{1}$  Dan Jurafsky $^{1}$  Victor Veitch $^{2}$  Dhanya Sridhar $^{3}$
1Stanford University
$^{2}$ University of Chicago
$^{3}$ Columbia University
$^{1}$  {rpryzant, dcard, jurafsky} @stanford.edu
victorveitch@gmail.com
<sup>3</sup>ds3778@columbia.edu
We consider the problem of using observational data to estimate the causal effects of linguistic properties. For example, does writing a complaint politely lead to a faster response time? How much will a positive product review increase sales? This paper addresses two technical challenges related to the problem before developing a practical method. First, we formalize the causal quantity of interest as the effect of a writer's intent, and establish the assumptions necessary to identify this from observational data. Second, in practice, we only have access to noisy proxies for the linguistic properties of interest—e.g., predictions from classifiers and lexicons. We propose an estimator for this setting and prove that its bias is bounded when we perform an adjustment for the text. Based on these results, we introduce TEXTCAUSE, an algorithm for estimating causal effects of linguistic properties. The method leverages (1) distant supervision to improve the quality of noisy proxies, and (2) a pre-trained language model (BERT) to adjust for the text. We show that the proposed method outperforms related approaches when estimating the effect of Amazon review sentiment on semi-simulated sales figures. Finally, we present an applied case study investigating the effects of complaint politeness on bureaucratic response times.
Social scientists have long been interested in the causal effects of language, studying questions like:
- How should political candidates describe their personal history to appeal to voters (Fong and Grimmer, 2016)?
- How can business owners write product descriptions to increase sales on e-commerce platforms (Pryzant et al., 2017, 2018a)?
How can consumers word their complaints to receive faster responses (Egami et al., 2018)?
- What conversational strategies can mental health counselors use to have more successful counseling sessions (Zhang et al., 2020)?
To study the causal effects of linguistic properties, we must reason about interventions: what would the response time for a complaint be if we could make that complaint polite while keeping all other properties (topic, pragmatics, etc.) fixed? Although it is sometimes feasible to run such experiments where text is manipulated and outcomes are recorded (Grimmer and Fong, 2020), analysts typically have observational data consisting of texts and outcomes obtained without intervention. This paper formalizes the estimation of causal effects of linguistic properties in observational settings.
Estimating causal effects from observational data requires addressing two challenges. First, we need to formalize the causal effect of interest by specifying the hypothetical intervention to which it corresponds. The first contribution of this paper is articulating the causal effects of linguistic properties; we imagine intervening on the writer of a text document and telling them to use different linguistic properties.
The second challenge of causal inference is identification: we need to express causal quantities in terms of variables we can observe. Often, instead of the true linguistic property of interest we have access to a noisy measurement called the proxy label. Analysts typically infer these values from text with classifiers, lexicons, or topic models (Grimmer and Stewart, 2013; Lucas et al., 2015; Prabhakaran et al., 2016; Voigt et al., 2017; Luo et al., 2019; Lucy et al., 2020). The second contribution of this paper is establishing the assumptions we need to recover the true effects of a latent linguistic property from these noisy proxy labels. In particular, we propose an adjustment for the confounding information in a text document and prove that this bounds the bias of the resulting estimates.
The third contribution of this paper is practical:
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4095-4109 June 6-11, 2021. ©2021 Association for Computational Linguistics
an algorithm for estimating the causal effects of linguistic properties. The algorithm uses distantly supervised label propagation to improve the proxy label (Zhur and Ghahramani, 2002; Mintz et al., 2009; Hamilton et al., 2016), then BERT to adjust for the bias due to text (Devlin et al., 2018; Veitch et al., 2020). We demonstrate the method's accuracy with partially-simulated Amazon reviews and sales data, perform a sensitivity analysis in situations where assumptions are violated, and show an application to consumer finance complaints. Data and a package for performing text-based causal inferences is available at https://github.com/rpryzant/causal-text.
2 Causal Inference Background
Causal inference from observational data is well-studied (Pearl, 2009; Rosenbaum and Rubin, 1983, 1984; Shalizi, 2013). In this setting, analysts are interested in the effect of a treatment  $T$  (e.g., a drug) on an outcome  $Y$  (e.g., disease progression). For ease, we consider binary treatments. The average treatment effect (ATE) on the outcome  $Y$  is,
$$
\psi = \mathbb {E} [ Y; \mathrm {d o} (T = 1) ] - \mathbb {E} [ Y; \mathrm {d o} (T = 0) ], \tag {1}
$$
where the operation  $\mathrm{do}(T = t)$  means that we hypothetically intervene and set the treatment  $T$  to some value (Pearl, 2009).
Typically, the ATE  $\psi$  is not the simple difference in average conditional outcomes,  $\mathbb{E}[Y|T = 1] - \mathbb{E}[Y|T = 0]$ . This is because confounding variables  $C$  are associated with both the treatment and outcome, inducing non-causal associations between them, referred to as open backdoor paths (Pearl, 2009). When all the confounding variables are observed, we can write the ATE in terms of observed variables using the backdoor-adjustment formula (Pearl, 2009),
$$
\psi = \mathbb {E} _ {C} \left[ \mathbb {E} [ Y \mid T = 1, C ] - \mathbb {E} [ Y \mid T = 0, C ] \right]. \tag {2}
$$
For example, if the confounding variable  $C$  is discrete, we group the data into values of  $C$ , calculate the average difference in outcomes between the treated and untreated samples of each group, and take the average over groups.
3 Causal Effects of Linguistic Properties
We are interested in the causal effects of linguistic properties. To formalize this as a treatment, we
Figure 1: The proposed causal model of text and outcomes. A writer uses linguistic property  $T$  and other properties  $Z$ , which may be correlated (denoted by bidirected arrow), to write the text  $W$ . From the text, the reader perceives the property of interest, captured by  $\tilde{T}$ , and together with other perceived information  $\tilde{Z}$ , produces the outcome  $Y$ . The proxy label of the property obtained via a classifier or lexicon is captured by  $\hat{T}$ .
imagine intervening on the writer of a text, e.g., telling people to write with a property (or not). We show that to estimate the effect of using a linguistic property, we must consider how a reader of the text perceives the property. These dual perspectives of the reader and writer are well studied in linguistics and NLP;<sup>1</sup> we adapt the idea for causal inference.
Figure 1 illustrates a causal model of the setting. Let  $W$  be a text document and let  $T$  (binary) be whether or not a writer uses a particular linguistic property of interest. For example, in consumer complaints, the variable  $T$  can indicate whether the writer intends to be polite or not. The outcome is a variable  $Y$ , e.g., how long it took for this complaint to be serviced. Let  $Z$  be other linguistic properties that the writer communicated (consciously or unconsciously) via the text  $W$ , e.g. topic, brevity or eloquence. The linguistic properties  $T$  and  $Z$  are typically correlated, and both variables affect the outcome  $Y$ .
<sup>1</sup>Literary theory argues that language is subject to two perspectives: the "artistic" pole – the text as intended by the author – and the "aesthetic" pole – the text as interpreted by the reader (Iser, 1974, 1979). The noisy channel model (Yuret and Yatbaz, 2010; Gibson et al., 2013) connects these poles by supposing that the reader perceives a noisy version of the author's intent. This duality has also been modeled in linguistic pragmatics as the difference between speaker meaning and literal or utterance meaning (Potts, 2009; Levinson, 1995, 2000). Gricean pragmatic models like RSA (Goodman and Frank, 2016) similarly formalize this as the reader using the literal meaning to help make inferences about the speaker's intent.
2We leave higher-dimensional extensions to future work.
We are interested in the average treatment effect,
$$
\psi^ {\text {w r i .}} = \mathbb {E} [ Y; \mathrm {d o} (T = 1) ] - \mathbb {E} [ Y; \mathrm {d o} (T = 0) ], \tag {3}
$$
where we imagine intervening on writers and telling them to use the linguistic property of interest (setting  $T = 1$ , "write politely") or not ( $T = 0$ ). This causal effect is appealing because the hypothetical intervention is well-defined – it corresponds to an intervention we could perform in theory. However, without further assumptions,  $\psi^{\mathrm{wri}}$  is not identified from the observational data. The reason is that we would need to adjust for the unobserved linguistic properties  $Z$ , which create open backdoor paths because they are correlated with both the treatment  $T$  and outcome  $Y$  (Figure 1).
To solve this problem, we observe that the reader is the one who produces outcomes. Readers use the text  $W$  to perceive a value for the property of interest (captured by the variable  $\tilde{T}$ ) as well as other properties (captured by  $\tilde{Z}$ ) then produce the outcome  $Y$  based on these perceived values. For example, a customer service representative reads a consumer complaint, judges whether (among other things) the complaint is polite or not, and chooses how quickly to respond based on this.
Consider the average treatment effect,
$$
\psi^ {\text {r e a .}} = \mathbb {E} [ Y; \mathrm {d o} (\tilde {T} = 1) ] - \mathbb {E} [ Y; \mathrm {d o} (\tilde {T} = 0) ], \tag {4}
$$
where we imagine intervening on the reader's perception of a linguistic property  $\tilde{T}$ . The following result shows that we can identify the causal effect of interest,  $\psi^{\mathrm{wri}}$ , by exploiting this ATE  $\psi^{\mathrm{rea}}$ .
Theorem 1. Let  $\tilde{Z} = f(W)$  be a function of the words  $W$  such that  $\mathbb{E}[Y|W] = \mathbb{E}\left[Y|\tilde{T},\tilde{Z}\right]$ . Suppose that the following assumptions hold:
1. (no unobserved confounding)  $W$  blocks backdoor paths between  $\tilde{T}$  and  $Y$ ,
2. (agreement of intent and perception)  $T = \tilde{T}$ .
3. (overlap) For some constant  $\epsilon > 0$ ,
$$
\epsilon <   P (\tilde {T} = 1 | \tilde {Z}) <   1 - \epsilon
$$
with probability 1.3
Then the ATE  $\psi^{rea}$  is identified as,
$$
\begin{array}{l} \psi^ {r e a.} = \mathbb {E} _ {W} \left[ \mathbb {E} \left[ Y \mid \tilde {T} = 1, \tilde {Z} = f (W) \right] - \right. (5) \\ \mathbb {E} \left[ Y \mid \tilde {T} = 0, \tilde {Z} = f (W) \right]. (6) \\ \end{array}
$$
Moreover, the ATE  $\psi^{rea}$  is equal to  $\psi^{wri}$ .
The proof is in Appendix A. Intuitively, the result says that the information in the text  $W$  that the reader uses to determine the outcome  $Y$  splits into two parts: the information the reader uses to perceive the linguistic property of interest  $(\tilde{T})$ , and the information used to perceive other properties  $(\tilde{Z} = f(W))$ . The information captured by the variable  $\tilde{Z}$  is confounding; it affects the outcome and is also correlated with the treatment  $\tilde{T}$ . Under certain assumptions, adjusting for the function of text  $\tilde{Z}$  that captures confounding suffices to identify the  $\psi^{\mathrm{rea}}$ ; in Figure 1, the backdoor path  $\tilde{T} \to W \to \tilde{Z} \to Y$  is blocked. Moreover, if we assume that readers correctly perceive the writer's intent, the effect  $\psi^{\mathrm{rea}}$ , which can be expressed in terms of observed variables, is equivalent to the effect that we want,  $\psi^{\mathrm{wri}}$ .
4 Substituting Proxy Labels
If we observed  $\tilde{T}$ , the reader's perception of the linguistic property of interest, then we could proceed by estimating the effect  $\psi^{\mathrm{rea.}}$  (equivalently,  $\psi^{\mathrm{wri.}}$ ). However, in most settings, one does not observe the linguistic properties that a writer intends to use ( $T$  and  $Z$ ) or that a reader perceives ( $\tilde{T}$  and the information in  $\tilde{Z}$ ). Instead, one uses a classifier or lexicon to predict values for this property from the text, producing a proxy label  $\hat{T}$  (e.g. predicted politeness).
For this setting, where we only have access to proxy labels, we introduce the estimand  $\psi^{\mathrm{proxy}}$  which substitutes the proxy  $\hat{T}$  for the unobserved treatment  $\tilde{T}$  in the effect  $\psi^{\mathrm{rea.}}$ :
$$
\begin{array}{l} \psi^ {\text {p r o x y}} = \mathbb {E} _ {W} \left[ \mathbb {E} \left[ Y \mid \hat {T} = 1, \tilde {Z} = f (W) \right] \right. (7) \\ - \mathbb {E} [ Y \mid \hat {T} = 0, \tilde {Z} = f (W) ] \Bigg ]. (8) \\ \end{array}
$$
3Informally, it must be possible to perceive a property  $(\tilde{T} = 1)$  for all settings of  $\tilde{Z}$ , and  $\tilde{Z}$  cannot perfectly predict  $\tilde{T}$ .
Grimmer and Fong (2020) studied a closely related setting where text documents are randomly assigned to readers who produce outcomes. From this experiment, they discover text properties that cause the outcome. Their causal identification result requires an exclusion restriction assumption, which is related to the no unobserved confounding assumption that we make.
This estimand only requires an adjustment for the confounding information  $\tilde{Z}$ . We show how to extract this information using pretrained language models in Section 5. Prior work on causal inference with proxy treatments (Wood-Doughty et al., 2018) requires an adjustment using the measurement model  $P(\tilde{T} \mid \hat{T})$ , i.e. the true relationship between the proxy label  $\hat{T}$  and its target  $\tilde{T}$ , which is typically unobserved. In contrast, the estimand  $\psi^{\mathrm{proxy}}$  does not require the measurement model.
The following result shows that the estimand  $\psi^{\mathrm{proxy}}$  only attenuates the ATE that we want,  $\psi^{\mathrm{rea}}$ . That is, the bias due to proxy treatments is benign; it can only decrease the magnitude of the effect but it does not change the sign.
Theorem 2. Let  $\epsilon_0 = \operatorname*{Pr}(\tilde{T} = 0\mid \hat{T} = 1,\tilde{Z})$  and let  $\epsilon_{1} = \operatorname *{Pr}(\tilde{T} = 1\mid \hat{T} = 0,\tilde{Z})$  . Then,
$$
\begin{array}{l} \psi^ {p r o x y} = \psi^ {r e a.} - \mathbb {E} _ {W} \left[ \left(\mathbb {E} [ Y \mid \tilde {T} = 1, \tilde {Z} ] \right. \right. \\ \left. \left. - \mathbb {E} [ Y \mid \tilde {T} = 0, \tilde {Z} ]\right) \left(\epsilon_ {0} + \epsilon_ {1}\right) \right] \\ \end{array}
$$
The proof is in Appendix E. This result shows that the proposed estimand  $\psi^{\mathrm{proxy}}$ , which we can estimate, is equal to the ATE  $\psi^{\mathrm{rea}}$ . that we want, minus a bias term related to measurement error. In particular, if the classifier is better than chance and the treatment effect sign is homogeneous across possible texts — i.e., it always helps or always hurts, an assumption the analyst must carefully assess — then the bias term is positive with the degree of attenuation dependent on the error rate of the proxy label  $\hat{T}$ . The result tells us to construct the most accurate proxy treatment  $\hat{T}$  possible, so long as we adjust for the confounding part of the text. This is a novel result for causal inference with proxy treatments and sidesteps the need for the measurement model.
5 TEXTCAUSE, A Causal Estimation Procedure
We introduce a practical algorithm for estimating the causal effects of linguistic properties. Motivated by Theorem 2, we first describe an approach for improving the accuracy of proxy labels. We then use the improved proxy labels, text and outcomes to fit a model that extracts and adjusts for the confounding information in the text  $(\tilde{Z})$ . In practice, one may observe additional covariates
$C$  that capture confounding properties, e.g., the product that a review is about or complaint type. We will include these covariates in the estimation algorithm.
5.1 Improved Proxy Labels
The first stage of TEXTCAUSE is motivated by Theorem 2, which said that a more accurate proxy can yield lower estimation bias. Accordingly, this stage uses distant supervision to improve the fidelity of lexicon-based proxy labels  $\hat{T}$ . In particular, we exploit an inductive bias of frequently used lexicon-based proxy treatments: the words in a lexicon correctly capture the linguistic property of interest (i.e., high precision, Tausczik and Pennebaker, 2010), but can omit words and discourse-level elements that also map to the desired property (i.e., low recall, Kim and Hovy, 2006; Rao and Ravichandran, 2009).
Motivated by work on lexicon induction and label propagation (Hamilton et al., 2016; An et al., 2018), we improve the recall of proxy labels, training a classifier  $P_{\theta}$  to predict the proxy label  $\hat{T}$  then using that classifier to relabel examples which were labeled  $\hat{T} = 0$  but look like  $\tilde{T} = 1$ . Formally, given a dataset of tuples  $\{(Y_i,W_i,C_i,\hat{T}_i)\}_{i=1}^n$  the algorithm is:
1. Train a classifier to predict  $P_{\theta}(\hat{T} \mid W)$ , e.g., logistic regression trained with bag-of-words features and  $\hat{T}$  labels.
2. Relabel some  $\hat{T} = 0$  examples (we experiment with ablating this in Appendix C):
$$
\hat {T} _ {i} ^ {*} = \left\{ \begin{array}{l l} 1 & \text {i f} \hat {T} _ {i} = 1 \\ \mathbb {1} [ P _ {\theta} (\hat {T} _ {i} = 1 | W _ {i}) > 0. 5 ] & \text {o t h e r w i s e} \end{array} \right.
$$
3. Use  $\hat{T}^*$  as the new proxy treatment variable.
5.2 Adjusting for Text
The second stage of TEXTCAUSE estimates the effect  $\psi^{\mathrm{proxy}}$  using the text  $W$ , improved proxy labels  $\hat{T}^{*}$ , and outcomes  $Y$ . This stage is motivated by Theorem 1, which described how to adjust for the confounding parts of the text. We approximate this confounding information in the text,  $\tilde{Z} = f(W)$  with a learned representation  $\mathbf{b}(W)$  that predicts the expected outcomes  $\mathbb{E}[Y|\hat{T}^{*} = t,\mathbf{b}(W),C]$  for  $t = 0,1$  (Eq.7).
We use DistilBERT (Sanh et al., 2019) to produce a representation of the text  $\mathbf{b}(W)$  by embedding the text then selecting the vector corresponding to a pretended [CLS] token. We proceed
<sup>5</sup>We prove in Appendix F that without the adjustment for confounding information  $\tilde{Z}$ , estimates of the ATE  $\psi^{\mathrm{rea}}$  will be arbitrarily biased.
Figure 2: The second stage of TEXTCAUSE adapts word embeddings to predict both of  $Y$ 's potential outcomes.
to optimize the model so that the representation  $\mathbf{b}(W)$  directly approximates the confounding information  $\tilde{Z} = f(w)$ . In particular, we train an estimator for the expected conditional outcome  $Q(t,\mathbf{b}(W),C) = \mathbb{E}[Y|\hat{T}^{*} = t,\mathbf{b}(W),C]$ :
$$
\hat {Q} (t, \mathbf {b} (W), C) = \sigma (\mathbf {M} _ {t} ^ {b} \mathbf {b} (W) + \mathbf {M} _ {t} ^ {c} \mathbf {c} + b)),
$$
where the vector  $\mathbf{c}$  is a one-hot encoding of the covariates  $C$ , the vectors  $\mathbf{M}_t^b \in \mathbb{R}^{768}$  and  $\mathbf{M}_t^c \in \mathbb{R}^{|C|}$  are learned, one for each value  $t$  of the treatment, and the scalar  $b$  is a bias term.
Letting  $\theta$  be all parameters of the model, our training objective is to minimize,
$$
\min  _ {\theta} \sum_ {i = 1} ^ {n} L \left(Y _ {i}, \hat {Q} _ {\theta} \left(\hat {T} _ {i} ^ {*}, \mathbf {b} \left(W _ {i}\right), C _ {i}\right) + \alpha \cdot R \left(W _ {i}\right), \right.
$$
where  $L(\cdot)$  is the cross-entropy loss and  $R(\cdot)$  is the original BERT masked language modeling objective, which we include following Veitch et al. (2020). The hyperparameter  $\alpha$  is a penalty for the masked language modeling objective. The parameters  $\mathbf{M}_t$  are updated on examples where  $\hat{T}_i^* = t$ .
Once  $\hat{Q} (\cdot)$  is fitted, an estimator  $\hat{\psi}^{\mathrm{proxy}}$  for the effect  $\psi^{\mathrm{proxy}}$  (Eq. 7) is,
$$
\begin{array}{l} \hat {\psi} ^ {\text {p r o x y}} = \frac {1}{n} \sum_ {i} \left[ \hat {Q} (1, \mathbf {b} (W _ {i}), C _ {i}) \right. \\ \left. - \hat {Q} (0, \mathbf {b} (W _ {i}), C _ {i}) \right], \tag {9} \\ \end{array}
$$
where we approximate the outer expectation over the text  $W$  with a sample average. Intuitively, this procedure works because the representation  $\mathbf{b}(W)$  extracts the confounding information  $\tilde{Z} = f(W)$ ; it explains the outcome  $Y$  as well as possible given the proxy label  $\hat{T}^*$ .
We evaluate the proposed algorithm's ability to recover causal effects of linguistic properties. Since ground-truth causal effects are unavailable without randomized controlled trials, we produce a semisynthetic dataset based on Amazon reviews where only the outcomes are simulated. We also conduct a real-world study using real-world complaints and bureaucratic response times. Our key findings are
- More accurate proxies combined with text adjustment leads to more accurate ATE estimates.
- Naive proxy-based procedures significantly underestimate true causal effects.
- ATE estimates can lose fidelity when the proxy is less than  $80\%$  accurate.
6.1 Amazon Reviews
6.1.1 Experimental Setup
Dataset. Here we use real world and publicly available Amazon review data to answer the question, "how much does a positive product review affect sales?" We create a scenario where positive reviews increase sales, but this effect is confounded by the type of product. Specifically:
- The text  $W$  is a publicly available corpus of Amazon reviews for digital music products (Ni et al., 2019). For simplicity, we only include reviews for mp3, CD, or Vinyl. We also exclude reviews for products worth more than $100 or fewer than 5 words.
- The observed covariate  $C$  is a binary indicator for whether the associated review is a CD or not, and we use this to simulate a confounded outcome.
- The treatment  $T = \tilde{T}$  is whether that review is positive (5 stars) or not (1 or 2 stars). Hence,
we omit reviews with 3 or 4 stars. Note that here it is reasonable to assume writer's intention  $(T)$  equals the reader's perception  $(\tilde{T})$  as the author is deliberately communicating their sentiment (or a very close proxy) with the stars. We use this variable to (1) simulate outcomes and (2) calculate ground truth causal effects for evaluation.
- The proxy treatment  $\hat{T}$  is computed via two strategies: (1) a randomly noised version of  $T$  fixed to  $93\%$  accuracy (to resemble a reasonable classifier's output, later called "proxy-noised"), and (2) a binary indicator for whether any words in  $W$  overlap with a positive sentiment lexicon (Liu et al., 2010).
- The outcome  $Y \sim \operatorname{Bernoulli}(\sigma(\beta_c(\pi(C) - \beta_o) + \beta_t \tilde{T} + \mathbb{N}(0, \gamma)))$  represents whether a product received a click or not. The parameter  $\beta_c$  controls confound strength,  $\beta_t$  controls treatment strength,  $\beta_o$  is an offset and the propensity  $\pi(C) = P(T = 1|C)$  is estimated from data.
The final data set consists of 17,000 examples.