SlowGuess/NAACL_2022_OCR huggingface dataset:
Adel Daoud, Elias Nosrati, Bernhard Reinsberg, Alexander E. Kentikelenis, Thomas H. Stubbs, and Lawrence P. King. 2017. Impact of International Monetary Fund programs on child health. Proceedings of the National Academy of Sciences, 114(25):6492-6497.
Adel Daoud, Bernhard Reinsberg, Alexander E. Kentikelenis, Thomas H. Stubbs, and Lawrence P. King. 2019. The International Monetary Fund's interventions in food and agriculture: An analysis of loans and conditions. *Food Policy*, 83:204-218.
Amir Feder, Katherine A. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E. Roberts, Brandon M. Stewart, Victor Veitch, and Diyi Yang. 2021. Causal inference in natural language processing: Estimation, prediction, interpretation and beyond. arXiv preprint arXiv:2109.00725.
Michele Jonsson Funk, Daniel Westreich, Chris Wiesen, Til Stürmer, M Alan Brookhart, and Marie Davidian. 2011. Doubly robust estimation of causal effects. American journal of epidemiology, 173(7):761-767.
Matthew Gentzkow, Bryan Kelly, and Matt Taddy. 2019. Text as data. Journal of Economic Literature, 57(3):535-74.
Carolyn Heinrich, Alessandro Maffioli, and Gonzalo Vazquez. 2010. A primer for applying propensity-score matching. Inter-American Development Bank.
Katherine Keith, David Jensen, and Brendan O'Connor. 2020. Text and causal inference: A review of using text to remove confounding from causal estimates. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5332-5344, Online. Association for Computational Linguistics.
Shiho Kino, Yu-Tien Hsu, Koichiro Shiba, Yung-Shin Chien, Carol Mita, Ichiro Kawachi, and Adel Daoud. 2021. A scoping review on the use of machine learning in research on social determinants of health: Trends and research prospects. SSM - Population Health, 15:100836.
Aditya Krishna Menon and Robert C Williamson. 2018. The cost of fairness in binary classification. In Conference on Fairness, Accountability and Transparency, pages 107-118. PMLR.
Wang Miao, Zhi Geng, and Eric J. Tchetgen Tchetgen. 2018. Identifying causal effects with proxy variables of an unmeasured confounder. Biometrika, 105(4):987-993.
Reagan Mozer, Luke Miratrix, Aaron Russell Kaufman, and L. Jason Anastasopoulos. 2020. Matching with text data: An experimental evaluation of methods for matching documents and of measuring match quality. Political Analysis, 28(4):445-468.
Jessica A. Myers, Jeremy A. Rassen, Joshua J. Gagne, Krista F. Huybrechts, Sebastian Schneeweiss, Kenneth J. Rothman, Marshall M. Joffe, and Robert J. Glynn. 2011. Effects of adjusting for instrumental variables on bias and precision of effect estimates. American journal of epidemiology, 174(11):1213-1222.
Judea Pearl. 2015. Conditioning on post-treatment variables. Journal of Causal Inference, 3(1):131-137.
Jose M. Pena. 2020. On the Monotonicity of a Nondifferentially Mismeasured Binary Confounder. Journal of Causal Inference, 8(1):150-163. Publisher: De Gruyter Section: Journal of Causal Inference.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237-7256. Association for Computational Linguistics.
Severi Rissanen and Pekka Marttinen. 2021. A critical look at the consistency of causal estimation with deep latent variable models. In Advances in Neural Information Processing Systems, volume 34, pages 4207-4217. Curran Associates, Inc.
Margaret E. Roberts, Brandon M. Stewart, and Richard A. Nielsen. 2020. Adjusting for confounding with text matching. American Journal of Political Science, 64(4):887-903.
Paul R. Rosenbaum and Donald B. Rubin. 1983. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55.
Donald B. Rubin. 1974. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5):688-701.
Tyler J. VanderWeele. 2019. Principles of confounder selection. European Journal of Epidemiology, 34:211-219.
Zach Wood-Doughty, Ilya Shpitser, and Mark Dredze. 2021. Generating synthetic text data to evaluate causal inference methods. arXiv preprint arXiv:2102.05638.
A Synthetic Data Generation
We first summarize the general approach in this section and provide details for the simulation in §5 in the next section.
For each document  $i$ , we first draw observed and unobserved confounders  $\mathbf{X}_i$  and  $U_i$ , and then the
treatment  $T_{i}$ . For each paragraph  $j$  in the document, we draw a paragraph topic  $Z_{ij}$ , depending on the values of  $U_{i}$  and  $T_{i}$ , and then a prompt  $W_{ij}^{0}$  depending on the value of  $Z_{ij}$ . Finally, we sample from the GPT-2 language model to generate the paragraph text  $\mathbf{W}_{ij}$ , starting from the prompt  $W_{ij}^{0}$  and with a vocabulary distribution shift defined by  $Z_{ij}$ . Algorithm 1 shows the pseudocode.
In the pseudocode above, the functions  $f_{X}, f_{U}, f_{T}$ , and  $f_{Y}$  define the distributions of the observed confounders, unobserved confounder, treatment and outcome, respectively. On the paragraph level, the function  $f_{Z}$  defines a categorical distribution over paragraph topics, and  $f_{W^{0}}$  a categorical distribution over prompts.
Similarly to Wood-Doughty et al. (2021), we use two mechanisms to condition the generation of a paragraph on a topic  $Z$ : a prompt and a vocabulary distribution shift. The distribution shift is designed to promote a set of keywords related to the topic and we implement it by multiplying the language model probabilities by a topic-specific vector  $\theta_Z$  of scale factors:
$$
P ^ {\prime} (w | \text {c o n t e x t}, Z) \propto P _ {\mathrm {L M}} (w | \text {c o n t e x t}) \cdot \theta_ {Z} (w)
$$
B Parameterization Used in §5
In §5, we generated  $N = 10,000$  instances, each consisting of numerical values and a document. We used the following distributions to generate the document-level variables:  $f_{X}$  was a 3-dimensional isotropic Gaussian;  $f_{U}$  was an even coin toss;  $f_{T}$  was linear in  $X_{i}$  and  $U_{i}$ ;  $f_{Y}$  was Gaussian with a mean defined by a linear function of  $X_{i}$ ,  $U_{i}$ , and  $T_{i}$  and a fixed standard deviation.
Each document consisted of  $K = 20$  paragraphs. For the paragraph generation, we defined five dif
ferent topics: two corresponding to positive and negative treatment values; two corresponding to positive and negative values of the unobserved confounder; one general background topic that was unrelated to  $U$  or  $T$  (but conceptually thought of as controlled by other "residual" variables  $R$ ). For a document with given values of  $U$  and  $T$ , we set the topic distribution  $f_{Z}$  to select the  $U$  topic with a probability of 0.2, the  $T$  topic with a probability of 0.2, and the general topic with a probability of 0.6.
The generated texts were designed to simulate a hypothetical use case where the researchers want to investigate the effect of IMF programs on some country-level indicator (cf. Daoud et al., 2019). The treatment variable  $T$  represents the presence or absence of an IMF program; the unseen confounder  $U$  represents the political situation of the country with respect to the IMF. For each topic except the general topic, we define four different prompts: for instance, for a positive treatment value, one of the prompts was The International Monetary Fund mandates the deregulation of [COUNTRY]'s labor market. In the analysis, "[COUNTRY]" is substituted by randomly sampled country names.
All topics except the general topic defined a distribution shift used when generating from the language model. We used 8 topic keywords for each of these topics. For these keywords, the corresponding entries in the vocabulary distribution shift vector  $\log \theta_Z$  were set to a value that defines the strength of the effect of  $T$  on  $\mathbf{W}$ ; for all other words except these keywords,  $\log \theta_Z$  was 0. Since our focus in this paper is on a clear-cut use case where the effects are strong, we set the strength parameter to a value of 4, which gives a noticeable effect on the generated texts.
The text generation model was run on a single GPU (NVIDIA GeForce GTX TITAN X). Generating the 10,000 documents took around 10 hours. The generation of random text is within the intended use of the GPT-2 model.
The implementation of the algorithm to generate the synthetic data is available in our repository.2
C IPW Details
The ATE is defined as  $\tau = \mathbb{E}[Y_i(1) - Y_i(0)]$ , where  $Y_{i}(t)$  is the potential outcome for unit  $i$  under
$^{2}$ https://github.com/adeldaoud/ AIforTextandCausalInference
<sup>1</sup>We used the implementation from the HuggingFace repository, https://huggingface.co/gpt2.
treatment  $t$ . It can be identified in randomized experiments (Rubin, 1974). However, the situation is more complicated in the observational setting, where the treatment is not randomized to units but could be correlated with confounders,  $X_{i}$ , that are associated with the treatment and the outcome. In that setting, we can, with additional assumptions, still recover the ATE using Inverse Propensity Weighting (IPW) or related robust methods (Funk et al., 2011), where observations are weighted by the inverse of their estimated treatment probabilities  $\hat{\pi}(\mathbf{X}_i) = \widehat{\operatorname*{Pr}}(T_i = 1|\mathbf{X}_i)$  (Rosenbaum and Rubin, 1983):  $\widehat{\tau} = \frac{1}{n}\sum_{i=1}^{n}\left\{\frac{T_iY_i}{\hat{\pi}(\mathbf{X}_i)} - \frac{(1-T_i)Y_i}{1-\hat{\pi}(\mathbf{X}_i)}\right\}$ .
C.2 Estimation
ATE estimates based on Inverse Propensity Weighting (see §C.1) require the estimation of the propensity scores,  $\widehat{\operatorname*{Pr}}(T|\mathbf{X},\mathbf{W})$ . To estimate these scores, we applied a  $L_{1}$ -regularized logistic regression model using the glmnet package in R. The regularization strength  $(\lambda)$  was set automatically via 10-fold cross-validation. When estimating propensities, we represented the (non-distilled or distilled) document as an  $L_{2}$ -normalized TF-IDF vector using the 256 most frequent terms in the vocabulary, while the numerical covariates  $\mathbf{X}$  were standardized.