SlowGuess/ACL_2020_OCR huggingface dataset:
Open problems: Estimates of causal effects are contingent on the "garden of forking paths" of data analysis, meaning any "paths" an analyst did not take could have resulted in different conclusions (Gelman and Loken, 2013). For settings with causal confounders from text, the first fork is the choice of representation (e.g., topic models or embeddings) and the second fork is the pre-processing
and hyperparameter decisions for the chosen representations.
We highlight that these decisions have been shown to alter results in predictive tasks. For instance, studies have shown that pre-processing decisions dramatically change topic models (Denny and Spirling, 2018; Schofield et al., 2017); embeddings are sensitive to hyperparameter tuning (Levy et al., 2015) and the construction of the training corpus (Antoniak and Mimno, 2018); and fine-tuned language model performance is sensitive to random restarts (Phang et al., 2018). Thus, reporting sensitivity analysis of the causal effects from these decisions seems crucial: how robust are the results to variations in modeling specifications?
5 Adjusting for confounding bias
Given a set of variables  $Z$  that satisfy the backdoor criterion (§3.2), one can use the backdoor adjustment to estimate the causal quantity of interest,
$$
\begin{array}{l} P (Y = y \mid d o (T = t)) = \\ \int P (Y = y \mid T = t, Z = z) P (Z = z) d z \tag {3} \\ \end{array}
$$
Conditioning on all confounders is often impractical in high-dimensional settings such as those found in natural language. We provide an overview of methods used by applications in this review that approximate such conditioning, leading to unbiased estimates of treatment effect; however, we acknowledge this is not an exhaustive list of methods and direct readers to more extensive guides (Morgan and Winship, 2015; Athey et al., 2017).
Open problems: Causal studies typically make an assumption of overlap, also known as common support or positivity, meaning that any individual has a non-zero probability of assignment to each treatment condition for all possible values of the covariates:  $\forall z, 0 < P(T = 1 \mid Z = z) < 1$ . D'Amour et al. (2017) show that as the dimensionality of covariates grows, strict overlap converges to zero. What are the implications of these results for high-dimensional text data?
5.1 Propensity scores
A propensity score estimates the conditional probability of treatment given a set of possible confounders (Rosenbaum and Rubin, 1984, 1983; Caliendo and Kopeinig, 2008). The true model of treatment assignment is typically unknown so one must estimate the propensity score from data (e.g., from a logistic regression model),
$$
\pi \equiv P (T = 1 \mid Z). \tag {4}
$$
Inverse Probability of Treatment Weighting (IPTW) assigns a weight to each unit based on the propensity score (Lunceford and Davidian, 2004),
$$
w _ {i} = t _ {i} / \hat {\pi} _ {i} + (1 - t _ {i}) / (1 - \hat {\pi} _ {i}), \tag {5}
$$
thus emphasizing, for example, treated units that were originally unlikely to be treated ( $t_i = 1$ , low  $\pi_i$ ). The ATE is calculated with weighted averages between the treatment and control groups, $^7$
$$
\hat {\tau} _ {\mathrm {I P T W}} = \frac {1}{n _ {1}} \sum_ {i: t _ {i} = 1} w _ {i} y _ {i} - \frac {1}{n _ {0}} \sum_ {j: t _ {j} = 0} w _ {j} y _ {j} \tag {6}
$$
5.2 Matching and stratification
Matching aims to create treatment and control groups with similar confounder assignments; for example, grouping units by observed variables (e.g., age, gender, occupation), then estimating effect size within each stratum (Stuart, 2010). Exact matching on confounders is ideal but nearly impossible to obtain with high-dimensional confounders, including those from text. A framework for matching with text data is described by Mozer et al. (2020) and requires choosing: (1) a text representation (§4); (2) a distance metric (cosine, Euclidean, absolute difference in propensity score etc.); and (3) a matching algorithm. As Stuart (2010) describes, the matching algorithm involves additional decisions about (a) greedy vs. optimal matching; (b) number of control items per treatment item; (c) using calipers (thresholds of maximum distance); and (d) matching with or without replacement. Coarsened exact matching (CEM) matches on discretized raw values of the observed confounders (Iacus et al., 2012).
Instead of directly matching on observed variables, stratified propensity-score matching partitions propensity scores into intervals (strata) and then all units are compared within a single strata (Caliendo and Kopeinig, 2008). Stratification is also known as interval matching, blocking, and subclassification.
Once the matching algorithm is implemented, counterfactuals (estimated potential outcomes) are obtained from the matches  $\mathcal{M}_i$  for each unit  $i$ :
$$
\hat {y} _ {i} (k) = \left\{ \begin{array}{l l} y _ {i} & \text {i f} t _ {i} = k \\ \frac {1}{| \mathcal {M} _ {i} |} \sum_ {j \in \mathcal {M} _ {i}} y _ {j} & \text {i f} t _ {i} \neq k \end{array} \right. \tag {7}
$$
which is plugged into the matching estimator,8
$$
\hat {\tau} _ {\text {m a t c h}} = \frac {1}{n} \sum_ {i} ^ {n} \left(\hat {y} _ {i} (1) - \hat {y} _ {i} (0)\right). \tag {8}
$$
Open problems: Ho et al. (2007) describe matching as a method to reduce model dependence because, unlike regression, it does not rely on a parametric form. Yet, estimated causal effects may still be sensitive to other matching method decisions such as the number of bins in coarsened exact matching, the number of controls to match with each treatment in the matching algorithm, or the choice of caliper. Are causal estimates made using textual covariates particularly sensitive or robust to such choices?
5.3 Regression adjustment
Regression adjustment fits a supervised model from observed data about the expected conditional outcomes
$$
q (t, z) \equiv \mathbb {E} (Y \mid T = t, Z = z) \tag {9}
$$
Then the learned conditional outcome,  $\hat{q}$ , is used to predict counterfactual outcomes for each observation under treatment and control regimes,
$$
\hat {\tau} _ {\text {r e g}} = \frac {1}{n} \sum_ {i} ^ {n} (\hat {q} (1, z _ {i}) - \hat {q} (0, z _ {i})) \tag {10}
$$
5.4 Doubly-robust methods
Unlike methods that model only treatment (IPTW) or only outcome (regression adjustment), doubly robust methods model both treatment and outcome, and have the desirable property that if either the treatment or outcome models are unbiased then the effect estimate will be unbiased as well. These methods often perform very well in practice (Dorie et al., 2019). Adjusted inverse probability of treatment weighting (A-IPTW) combines estimated propensity scores (Eqn. 4) and conditional outcomes (Eqn. 9), while the more general targeted maximum likelihood estimator (TMLE) updates the conditional outcome estimate with a regression on the propensity weights (Eqn. 5) and  $\hat{q}$  (Van der Laan and Rose, 2011).
5.5 Causal-driven representation learning
Several research efforts design representations of text specifically for causal inference goals. These
For alternative matching estimators see Abadie et al. (2004). This estimator is technically the sample average treatment effect (SATE), not the population-level ATE, since we have pruned treatment and control pairs that do not have matches (Morgan and Winship, 2015).
Lunceford and Davidian (2004) note there are two versions of IPTW, where both the weighted sum and the raw count have been used for the  $n_0$  and  $n_1$  denominators.
approaches still initialize their models with representations of text described in Section 4, but then the representations are updated with machine learning architectures that incorporate the observed treatment assignment and other causal information. Johansson et al. (2016) design a network with a multitask objective that aims for low prediction error for the conditional outcome estimates,  $q$ , and minimizes the discrepancy distance between  $q(1, z_i)$  and  $q(0, z_i)$  in order to achieve balance in the confounders. Roberts et al. (2020) combine structural topic models (STM; Roberts et al. (2014)), propensity scores, and matching. They use the observed treatment assignment as the content covariate in the STM, append an estimated propensity score to the topic-proportion vector for each document, and then perform coarsened exact matching on that vector. Veitch et al. (2019) fine-tune a pre-trained BERT network with a multi-task loss objective that estimates (a) the original masked language-modeling objective of BERT, (b) propensity scores, and (c) conditional outcomes for both treatment and control. They use the predicted conditional outcomes and propensity scores in regression adjustment and the TMLE formulas.
Open problems: These methods have yet to be compared to one another on the same benchmark evaluation datasets. Also, when are the causal effects sensitive to hyperparameter and network architecture choices and what should researchers do in these settings?
6 Human evaluation of intermediate steps
Text data has the advantage of being interpretable—matched pairs and some low-dimensional representations of text can be read by humans to evaluate their quality. When possible, we suggest practitioners use (1) interpretable balance metrics and/or (2) human judgements of treatment propensity to evaluate intermediate steps of the causal estimation pipeline.
6.1 Interpretable balance metrics
For matching and propensity score methods, the confounder balance should be assessed, since ideally  $P(Z \mid T = 1) = P(Z \mid T = 0)$  in a matched sample (Stuart, 2010). A standard numerical balance diagnostic is the standardized difference in means (SDM),
$$
S D M (j) = \frac {\frac {1}{n _ {1}} \sum_ {i : t _ {i} = 1} z _ {i j} - \frac {1}{n _ {0}} \sum_ {i : t _ {i} = 0} z _ {i j}}{\sigma_ {j} ^ {t = 1}}
$$
where  $z_{ij}$  is a single confounder  $j$  for a single unit  $i$  and  $\sigma_j^{t = 1}$  is the standard deviation of  $z_{ij}$  for all  $i$  such that  $t_i = 1$ . SDM can also be used to evaluate the propensity score, in which case there would only be a single  $j$  (Rubin, 2001).
For causal text applications, Roberts et al. (2020) and Sridhar and Getoor (2019) estimate the difference in means for each topic in a topic-model representation of confounders and Sridhar et al. (2018) estimate the difference in means across structured covariates but not the text itself. As an alternative to SDM, Roberts et al. (2020) use string kernels to perform similarity checks. Others use domain-specific, known structured confounders to evaluate the balance between treatment and control groups. For instance, De Choudhury and Kiciman (2017) sample treatment-control pairs across all propensity score strata and label the sampled text based on known confounders (in their case, from a previously-validated codebook of suicidal ideation risk markers).
Open problems: For embeddings and causally-driven representations, each dimension in the confounder vector  $z$  is not necessarily meaningful. How can balance metrics be used in this setting?
6.2 Judgements of treatment propensity
When possible, one can also improve validation by evaluating matched items (posts, sentences, documents etc.) to humans for evaluation. Humans can either (a) use a scale (e.g., a 1-5 Likert scale) to rate items individually on their propensity for treatment, or (b) assess similarity of paired items after matching. A simple first step is for analysts to do "in-house" evaluation on a small sample (e.g., Roberts et al. (2020)), but a larger-sample experiments on crowd-working platforms can also increase the validity of these methods (e.g., Mozer et al. (2020)).
Open problems: How can these human judgement experiments be improved and standardized? Future work could draw from a rich history in NLP of evaluating representations of topic models and embeddings (Wallach et al., 2009; Bojanowski et al., 2017; Schnabel et al., 2015) and evaluating semantic similarity (Cer et al., 2017; Bojanowski et al., 2017; Reimers and Gurevych, 2019).
7 Evaluation of causal methods
Because the true causal effects in real-world causal inference are typically unknown, causal evaluation is a difficult and open research question. As
algorithmic complexity grows, the expected performance of causal methods can be difficult to estimate theoretically (Jensen, 2019). Other causal evaluations involve synthetic data. However, as Gentzel et al. (2019) discuss, synthetic data has no "unknown unknowns" and many researcher degrees of freedom, which limits their effectiveness. Thus, we encourage researchers to evaluate with constructed observational studies or semi-synthetic datasets, although measuring latent confounders from text increases the difficulty of creating realistic datasets that can be used for empirical evaluation of causal methods.
7.1 Constructed observational studies
Constructed observational studies collect data from both randomized and non-randomized experiments with similar participants and settings. Evaluations of this kind include job training programs in economics (LaLonde, 1986; Glynn and Kashin, 2013), advertisement marketing campaigns (Gordon et al., 2019), and education (Shadish et al., 2008). For instance, Shadish et al. (2008) randomly assign participants to a randomized treatment (math or vocabulary training) and non-randomized treatment (participants choose their own training). They compare causal effect estimates from the randomized study with observational estimates that condition on confounders from participant surveys (e.g., sex, age, marital status, like of mathematics, extroversion, etc.).
Open problems: To extend constructed observational studies to text data, one could build upon Shadish et al. (2008) and additionally (a) ask participants to write free-form essays of their past educational and childhood experiences and/or (b) obtain participants' public social media posts. Then causal estimates that condition on these textual representation of confounders could be compared to both those with surveys and the randomized settings. Alternatively, one could find observational studies with both real covariates and text and (1) randomize treatment conditional on the propensity score model (constructed from the covariates but not the text) and (2) estimate causal effect given only text (not the covariates). Then any estimated non-zero treatment effect is only bias.
7.2 Semi-synthetic datasets
Semi-synthetic datasets use real covariates and synthetically generate treatment and outcome, as in the 2016 Atlantic Causal Inference Competition
(Dorie et al., 2019). Several applications in this review use real metadata or latent aspects of text to simulate treatment and outcome: Johansson et al. (2016) simulate treatment and outcome from two centroids in topic model space from newswire text; Veitch et al. (2019) use indicators of an article's "buzzy" keywords; Roberts et al. (2020) use "quantitative methodology" categories of articles that were hand-coded by other researchers.
Open problems: Semi-synthetic datasets that use real covariates of text seem to be a better evaluation strategy than purely synthetic datasets. However, with semi-synthetic datasets, researchers could be inadvertently biased to choose metadata that they know their method will recover. A promising future direction is a competition-style evaluation like Dorie et al. (2019) in which one group of researchers generates a causal dataset with text as a confounder and other groups of researchers evaluate their causal methods without access to the data-generating process.
Computational social science is an exciting, rapidly expanding discipline. With greater availability of text data, alongside improved natural language processing models, there is enormous opportunity to conduct new and more accurate causal observational studies by controlling for latent confounders in text. While text data ought to be as useful for measurement and inference as "traditional" low-dimensional social-scientific variables, combining NLP with causal inference methods requires tackling major open research questions. Unlike predictive applications, causal applications have no ground truth and so it is difficult distinguish modeling errors and forking paths from the true causal effects. In particular, we caution against using all available text in causal adjustment methods without any human validation or supervision, since one cannot diagnose any potential errors. Solving these open problems, along with the others presented in this paper, would be a major advance for NLP as a social science methodology.
The authors thank Sam Witty, Jacob Eisenstein, Brandon Stewart, Zach Wood-Doughty, Andrew Halterman, Laura Balzer, and members of the University of Massachusetts Amherst NLP reading group for helpful feedback, as well as the anonymous referees for detailed peer reviews.
Alberto Abadie, David Drukker, Jane Leber Herr, and Guido W Imbens. 2004. Implementing matching estimators for average treatment effects in stata. The Stata Journal, 4(3):290-311.
Maria Antoniak and David Mimno. 2018. Evaluating the stability of embedding-based word similarities. Transactions of the Association for Computational Linguistics, 6:107-119.
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In *ICLR*.
Susan Athey, Guido Imbens, Thai Pham, and Stefan Wager. 2017. Estimating average treatment effects: Supplementary analyses and remaining challenges. American Economic Review, 107(5):278-81.
Isabelle Augenstein, Kris Cao, He He, Felix Hill, Spandana Gella, Jamie Kiros, Hongyuan Mei, and Dipendra Misra. 2018. Proceedings of the Third Workshop on Representation Learning for NLP. In Proceedings of The Third Workshop on Representation Learning for NLP.
Isabelle Augenstein, Spandana Gella, Sebastian Ruder, Katharina Kann, Burcu Can, Johannes Welbl, Alexis Conneau, Xiang Ren, and Marek Rei. 2019. Proceedings of the 4th Workshop on Representation Learning for NLP. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019).
Ananth Balashankar, Sunandan Chakraborty, Samuel Fraiberger, and Lakshminarayanan Subramanian. 2019. Identifying predictive causal factors from news streams. In Empirical Methods in Natural Language Processing.
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent Dirichlet Allocation. Journal of Machine Learning Research, 3(Jan):993-1022.
Su Lin Blodgett and Brendan O'Connor. 2017. Racial disparity in natural language processing: A case study of social media african-american english. In *Fairness, Accountability, and Transparency in Machine Learning (FAT/ML) Workshop*, KDD.
Phil Blunsom, Antoine Bordes, Kyunghyun Cho, Shay Cohen, Chris Dyer, Edward Grefenstette, Karl Moritz Hermann, Laura Rimell, Jason Weston, and Scott Yih. 2017. Proceedings of the 2nd Workshop on Representation Learning for NLP. In Proceedings of the 2nd Workshop on Representation Learning for NLP.
Phil Blunsom, Kyunghyun Cho, Shay Cohen, Edward Grefenstette, Karl Moritz Hermann, Laura Rimell, Jason Weston, and Scott Wen-tau Yih. 2016. Proceedings of the 1st Workshop on Representation Learning for NLP. In Proceedings of the 1st Workshop on Representation Learning for NLP.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Jordan Boyd-Graber, David Mimno, and David Newman. 2014. Care and feeding of topic models: Problems, diagnostics, and improvements. Handbook of Mixed Membership Models and Their Applications, 225255.
John P Buonaccorsi. 2010. Measurement Error: Models, Methods, and Applications. CRC Press.
Marco Caliendo and Sabine Kopeinig. 2008. Some practical guidance for the implementation of propensity score matching. Journal of Economic Surveys, 22(1):31-72.
Dallas Card and Noah A Smith. 2018. The importance of calibration for estimating proportions from annotations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics.
Raymond J Carroll, David Ruppert, Leonard A Stefanski, and Ciprian M Crainiceanu. 2006. Measurement Error in Nonlinear Models: a Modern Perspective. CRC Press.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Vancouver, Canada. Association for Computational Linguistics.
Stevie Chancellor, Eric PS Baumer, and Munmun De Choudhury. 2019. Who is the human in human-centered machine learning: The case of predicting mental health from social media. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW):147.
Alexander D'Amour, Peng Ding, Avi Feller, Lihua Lei, and Jasjeet Sekhon. 2017. Overlap in observational studies with high-dimensional covariates. arXiv preprint arXiv:1711.02582.
Munmun De Choudhury and Emre Kiciman. 2017. The language of social support in social media and its effect on suicidal ideation risk. In International AAAI Conference on Web and Social Media (ICWSM).
Munmun De Choudhury, Emre Kiciman, Mark Dredze, Glen Coppersmith, and Mrinal Kumar. 2016. Discovering shifts to suicidal ideation from mental health content in social media. In Proceedings of the 2016 CHI conference on human factors in computing systems, pages 2098-2110. ACM.
Matthew J Denny and Arthur Spirling. 2018. Text preprocessing for unsupervised learning: Why it matters, when it misleads, and what to do about it. Political Analysis, 26(2):168-189.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Association of Computational Linguistics.
Vincent Dorie, Jennifer Hill, Uri Shalit, Marc Scott, and Daniel Cervone. 2019. Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition. Statistical Science, 34(1):43-68.
Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. 2015. Low resource dependency parsing: Crosslingual parameter sharing in a neural network parser. In Association for Computational Linguistics.
Naoki Egami, Christian J Fong, Justin Grimmer, Margaret E Roberts, and Brandon M Stewart. How to make causal inferences using texts. Working paper.
Felix Elwert and Christopher Winship. 2014. Endogenous selection bias: The problem of conditioning on a collider variable. Annual Review of Sociology, 40:31-53.
Seyed Amin Mirlohi Falavarjani, Hawre Hosseini, Zeinab Noorian, and Ebrahim Bagheri. 2017. Estimating the effect of exercising on users' online behavior. In Eleventh International AAAI Conference on Web and Social Media.
Christian Fong and Justin Grimmer. 2016. Discovery of treatments from text corpora. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1600-1609.
Wayne A Fuller. 1987. Measurement Error Models. John Wiley & Sons.
Andrew Gelman and Eric Loken. 2013. The garden of forking paths: Why multiple comparisons can be a problem, even when there is no "fishing expedition" or "p-hacking" and the research hypothesis was posited ahead of time. *Department of Statistics*, Columbia University.
Amanda Gentzel, Dan Garant, and David Jensen. 2019. The case for evaluating causal models using interventional measures and empirical data. In Advances in Neural Information Processing Systems.
Adam Glynn and Konstantin Kashin. 2013. Front-door versus back-door adjustment with unmeasured confounding: Bias formulas for front-door and hybrid adjustments. In 71st Annual Conference of the Midwest Political Science Association, volume 3.
Brett R Gordon, Florian Zettelmeyer, Neha Bhargava, and Dan Chapsky. 2019. A comparison of approaches to advertising measurement: Evidence from big field experiments at facebook. Marketing Science, 38(2):193-225.
MA Hernán and JM Robins. 2020. Causal Inference: What If. Boca Raton: Chapman Hall/CRC.