SlowGuess/NAACL_2021_OCR huggingface dataset:
Protocol. All nonlinear models were implemented using PyTorch (Paszke et al., 2019). We use the transformers<sup>6</sup> implementation of DistillBERT and the distilbert-base-uncased model, which has 66M parameters. To this we added 3,080 parameters for text adjustment (the  $\mathbf{M}_{\mathbf{t}}^{\mathrm{b}}$  and  $\mathbf{M}_{\mathbf{t}}^{\mathrm{c}}$  vectors). Models were trained in a cross-validated fashion, with the data being split into 12,000, 2,000, and 4,000-example train, validation, and test sets.<sup>7</sup> BERT was optimized for 3 epochs on each fold using Adam (Kingma and Ba, 2014), a learning rate of  $2e^{-5}$ , and a batch size of 32. The weighting on the potential outcome and masked language modeling heads was 0.1 and 1.0, respectively. Linear models were implemented with sklearn. For T-boosting, we used a vocab size of 2,000 and L2 regularization with a strength of  $c = 1e^{-4}$ . Each experiment was replicated using 100 different random seeds for robustness. Each trial took an average of 32 minutes with three 1.2 GHz CPU cores and one TITAN X GPU.
Baselines. The "unadjusted" baseline is  $\hat{\psi}^{\mathrm{naive}} = \hat{\mathbb{E}} [Y|\hat{T} = 1] - \hat{\mathbb{E}} [Y|\hat{T} = 0]$ , the expected difference in outcomes conditioned on  $\hat{T}$ .<sup>8</sup> The
proxy-\* baselines perform backdoor adjustment for the observed covariate  $C$  and are based on Sridhar and Getoor (2019):  $\hat{\psi}^{\mathrm{naive} + C} = \frac{1}{|C|}\sum_{c}(\hat{\mathbb{E}} [Y|\hat{T} = 1,C = c] - \hat{\mathbb{E}} [Y|\hat{T} = 0,C = c])$ , using randomly drawn and lexicon-based  $\hat{T}$  proxies. We also compare against "semi-oracle",  $\hat{\psi}^{matrix}$ , an estimator which assumes additional access to the ground truth measurement model  $P(\hat{T}\mid T)$  (Wood-Doughty et al., 2018); see Appendix G for derivation.
Note that for clarity, we henceforth refer to the treatment-boosting and text-adjusting stages of TEXTCAUSE as  $T$ -boost and  $W$ -Adjust.
6.1.2 Results
Our primary results are summarized in Table 1. Individually,  $T$ -boost and  $W$ -Adjust perform well, generating estimates which are closer to the oracle than the naive "unadjusted" and "proxy-lex" baselines. However, these components fail to outperform the highly accurate "proxy-noised" baseline unless they are combined (i.e., the TEXTCAUSE algorithm). Only the full TextCause algorithm consistently outperformed (i.e. produced higher quality ATE estimates) than the baselines. This result is robust to varying levels of noise and treatment/confound strength. Indeed TEXTCAUSE's estimates were on average within  $2\%$  of the semiconductor. Furthermore, these results support Theorem 2: methods which adjusted for the text always attenuated the true ATE.
Adjusting for the confounding parts of text is crucial: the results show that estimators that adjust for the covariates  $C$  but not the text perform poorly, sometimes even worse than the unadjusted estimator  $\hat{\psi}^{\mathrm{naive}}$ .
Does it always help to adjust for the text? We consider the case where confounding information in the text causes a naive estimator which does not adjust for this information  $(\psi^{\mathrm{naive}})$  to have the opposite sign of the true effect  $\psi$ . Does our proposed text adjustment help in this situation? Theorem 2 says it should, because  $\psi^{\mathrm{proxy}}$  estimates are bounded in  $[0,\psi ]$ . This ensures that the most important of bits, the bit of directional information, is preserved.
Table 2 shows results from such a scenario. We see that the true ATE of  $T$ ,  $\psi$ , has a strong negative effect, while the naive estimator  $\psi^{\mathrm{naive} + \mathrm{C}}$  produces a positive effect. Adding an adjustment for the confounding parts of the text with TEXTCAUSE
<sup>6</sup>https://huggingface.co/transformers
<sup>7</sup>See Egami et al. (2018) for an investigation into train/test splits for text-based causal inference.
See Appendix F for an investigation into this estimator.
Noise: Treatment: Confounding:	Low	High	Mean delta from oracle
Low	High	Low	High
Low	High	Low	High	Low	High	Low	High
oracle (ψ)	9.92	10.03	18.98	19.30	8.28	8.28	16.04	16.19	0.0
semi-oracle (ˆmatrix)	9.73	9.82	18.77	19.08	8.25	8.28	16.02	16.21	0.13
unadjusted (ˆnaive)	6.84	7.66	13.53	14.50	5.79	6.42	11.51	12.26	3.58
proxy-lex (ˆnaive+C)	6.67	6.73	12.88	13.09	5.65	5.67	10.98	11.12	4.43
proxy-noised (ˆnaive+C)	8.25	8.27	15.90	16.12	6.69	6.72	13.22	13.33	2.35
+T-boost (ˆnaive+C)	8.11	8.16	15.53	15.73	6.78	6.80	13.19	13.32	2.51
+W-Adjust (ˆproxy)	7.82	8.57	14.96	16.13	6.62	7.22	12.95	13.76	2.39
+T-boost +W-Adjust (TEXTCAUSE,ˆproxy)	9.42	10.27	18.20	19.32	7.85	8.53	15.45	16.30	0.11
Table 1: ATE estimates: expected change in click probabilities if one were to manipulate the sentiment of a review from negative to positive. TEXTCAUSE performs best in most settings. The true ATE is given in the top row ("oracle"). Estimates closer to the oracle are better. The last column gives the average difference between the estimated and true ATEs; lower is better. Rows 3-6 are baselines. Rows 7-9 are proposed. The second row and the bottom three rows use lexicon-based proxy treatments (we observed similar results using other proxy treatments). All columns have  $\beta_{o} = 0.9$ . Low and high noise corresponds to  $\gamma = 0$  and 1. Low and high treatment corresponds to  $\beta_{t} = 0.4, 0.8$ . Low and high confounding corresponds to  $\beta_{c} = -0.4, 4.0$ . All standard errors are less than 0.5.
Estimator	ATE	SE
oracle (ψ)	-14.99	± 0.1
proxy-lex (ψnaive+C)	6.29	± 0.3
+T-boost (ψnaive+C)	4.18	± 0.5
+T-boost +W-Adjust	0.50	± 1.3
(TEXTCAUSE, ψproxy)
Table 2: Estimator performance in a worst-case scenario where the estimated ATE of  $\hat{T}$  and  $\hat{T}^*$  indicates the opposite sign of the true ATE of  $T$  ( $\beta_c = 0.8$ ,  $\beta_t = -1$ ,  $\pi(C) = 0.8$ ,  $\beta_o = 0.6$ ).
successfully brings the proxy-based estimate to 0, which is indicative of the bounded behavior that Theorem 2 suggests.
Sensitivity analysis. In Figure 3 we synthetically vary the accuracy of a proxy  $\hat{T}$  by dropping random subsets of the data. This is to evaluate the robustness of various estimation procedures. We would expect (1) methods that do not adjust for the text to behave unpredictably, and (2) methods that do adjust for the text to be more robust.
These results support our first hypothesis: boosting treatment labels without text adjustment can behave unpredictably, as proxy-lex and  $T$ -boost both overestimate the true ATE. In other words, the predictions of both estimators grow further from the oracle as  $\hat{T}$ 's accuracy increases.
The results are mixed with respect to our second hypothesis. Both methods which adjust for
Figure 3: ATE estimates as the accuracy of  $\hat{T}$  is varied. Without text adjustment,  $T$ -boost's errors can increase with the error rate of  $\hat{T}$ . The dotted black lines correspond to the true ATE (left) and 0 error (right).
the text (W-Adjust and TEXTCAUSE) consistently attenuate the true ATE, which is in line with Theorem 2. However, we find that TEXTCAUSE, which makes use of  $T$ -boost and W-Adjust, may not always provide the highest quality ATE estimates in finite data regimes. Notably, when  $\hat{T}$  is less than  $90\%$  accurate, both proxy-lex and  $T$ -boost can produce higher-quality estimates than the proposed TEXTCAUSE algorithm.
Note that all estimates quickly lose fidelity as the proxy  $\hat{T}$  becomes noisier. It rapidly becomes difficult for any method to recover the true ATE
when the proxy  $\hat{T}$  is less than  $80\%$  accurate.
6.2 Application: Complaints to the Financial Protection Bureau
We proceed to offer an applied pilot study which seeks to answer, "how does the perceived politeness of a complaint affect the time it takes for that complaint to be addressed?" We consider complaints filed with the Consumer Financial Protection Bureau (CFPB). This is a government agency which solicits and handles complaints about financial products. When they receive a complaint it is forwarded to the relevant company. The time it takes for that company to process the complaint is recorded. Some submissions are handled quickly (< 15 days) while others languish. This 15-day threshold is our outcome  $Y$ . We additionally adjust for an observed covariate  $C$  that captures what product and company the complaint is about (mortgage or bank account). To reduce other potentially confounding effects, we pair each  $Y = 1$  complaint with the most similar  $Y = 0$  complaint according to cosine similarity of TF-IDF vectors (Mozer et al., 2020). From this we select the 4,000 most similar pairs for a total of 8,000 complaints.
For our treatment (politeness), we use a state-of-the-art politeness detection package geared towards social scientists (Yeomans et al., 2018). This package reports a score from a trained classifier using expert features of politeness and a hand-labeled dataset. We take examples in the top and bottom  $25\%$  of the scoring distribution to be our  $\hat{T} = 1$  and  $\hat{T} = 0$  examples and throw out all others. The final dataset consists of 4,000 complaints, topics, and outcomes.
We use the same training procedure and hyperparameters as Section 6.1, except now  $W$ -Adjust is trained for 9 epochs and each cross validation fold is of size 2,000.
Results are given in Figure 3 and suggest that perceived politeness may have an effect on reducing response time. We find that the effect size increases as we adjust for increasing amounts of information. The "unadjusted" approach which does not perform any adjustment produces the smallest ATE. "proxy-lex", which only adjusts for covariates, indicated the second-smallest ATE. The W-Adjust and TEXTCAUSE methods, which adjust for covariates and text, produced the largest ATE
Estimator	ATE	SE
unadjusted (ˆnaive)	3.01	± 0.3
proxy-lex (ˆnaive+C)	4.03	± 0.4
+T-boost (ˆnaive+C)	9.64	± 0.5
+W-Adjust (ˆproxy)	6.30	± 1.6
+T-boost +W-Adjust	10.30	± 2.1
TEXTCAUSE, (ˆproxy)
Table 3: Effect size can vary across estimation methods, with methods that adjust for more information producing larger ATEs. Each number represents the expected percent change in the likelihood of getting a timely response when the politeness of a complaint is hypothetically increased.
estimates. This suggests that there is a significant amount of confounding in real world studies, and the choice of estimator can yield highly varying conclusions.
Our focus fits into a body of work on text-based causal inference that includes text as treatments (Egami et al., 2018; Fong and Grimmer, 2016; Grimmer and Fong, 2020; Wood-Doughty et al., 2018), text as outcomes (Egami et al., 2018), and text as confounders (Roberts et al. (2020); Veitch et al. (2020); see Keith et al. (2020) for a review of that space). We build on Veitch et al. (2020), which proposed a BERT-based text adjustment method similar to our W-Adjust algorithm. This paper is related to work by Grimmer and Fong (2020), which discusses assumptions needed to estimate causal effects of text-based treatments in randomized controlled trials. There is also work on discovering causal structure in text, as topics with latent variable models (Fong and Grimmer, 2016) and as words and n-grams with adversarial learning (Pryzant et al., 2018b) and residualization (Pryzant et al., 2018a). There is also a growing body of applications in the social sciences (Hall, 2017; Olteanu et al., 2017; Saha et al., 2019; Mozer et al., 2020; Karell and Freedman, 2019; Sobolev, 2019; Zhang et al., 2020).
This paper also fits into a long-standing body of work on measurement error and causal inference (Pearl, 2012; Kuroki and Pearl, 2014; Buonaccorsi, 2010; Carroll et al., 2006; Shu and Yi, 2019; Oktay et al., 2019; Wood-Doughty et al., 2018). Most of this work deals with proxies for confounding variables. The present paper is most closely related to Wood-Doughty et al. (2018), which also
$^9$ https://wwwconsumer-action.org/downloads/english/cfgb_full_dbase_report.pdf/
deals with proxy treatments, but instead proposes an adjustment using the measurement model.
This paper addressed a setting of interest to NLP and social science researchers: estimating the causal effects of latent linguistic properties from observational data. We clarified critical ambiguities in the problem, showed how causal effects can be interpreted, presented a method, and demonstrated how it offers practical and theoretical advantages over the existing practice. We also release a package for performing text-based causal inferences.[10] This work opens new avenues for further conceptual, methodological, and theoretical refinement. This includes improving non-lexicon based treatments, heterogeneous effects, overlap violations, counterfactual inference, ethical considerations, extensions to higher-dimensional outcomes and covariates, and benchmark datasets based on paired randomized controlled trials and observational studies.
This project recieved partial funding from the Stanford Data Science Institute, NSF Award IIS-1514268 and a Google Faculty Research Award. We thank Justin Grimmer, Stefan Wager, Percy Liang, Tatsunori Hashimoto, Zach Wood-Doughty, Katherine Keith, the Stanford NLP Group, and our anonymous reviewers for their thoughtful comments and suggestions.
Jisun An, Haewoon Kwak, and Yong-Yeol Ahn. 2018. Semaxis: A lightweight framework to characterize domain-specific word semantics beyond sentiment. In Proceedings of ACL.
John P Buonaccorsi. 2010. Measurement error: models, methods, and applications. CRC press.
Raymond J Carroll, David Ruppert, Leonard A Stefanski, and Ciprian M Crainiceanu. 2006. Measurement error in nonlinear models: a modern perspective. CRC press.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Naoki Egami, Christian J Fong, Justin Grimmer, Margaret E Roberts, and Brandon M Stewart. 2018. How to make causal inferences using texts. arXiv preprint arXiv:1802.02163.
Christian Fong and Justin Grimmer. 2016. Discovery of treatments from text corpora. In Proceedings of ACL, pages 1600-1609.
Edward Gibson, Leon Bergen, and Steven T Piantadosi. 2013. Rational integration of noisy evidence and prior semantic expectations in sentence interpretation. Proceedings of the National Academy of Sciences, 110(20):8051-8056.
Gene Glass and Kenneth Hopkins. 1996. Statistical methods in education and psychology. Psyccritiques, 41(12).
Noah D Goodman and Michael C Frank. 2016. Pragmatic language interpretation as probabilistic inference. Trends in cognitive sciences, 20(11):818-829.
Justin Grimmer and Christian Fong. 2020. Causal inference with latent treatments. In Unpublished.
Justin Grimmer and Brandon M Stewart. 2013. Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political analysis, 21(3):267-297.
Allie Hall. 2017. How hiring language reinforces pink collar jobs.
William L Hamilton, Kevin Clark, Jure Leskovec, and Dan Jurafsky. 2016. Inducing domain-specific sentiment lexicons from unlabeled corpora. In Proceedings of EMNLP.
Wolfgang Iser. 1974. The implied reader: Patterns of communication in prose fiction from Bunyan to Beckett. Johns Hopkins University Press.
Wolfgang Iser. 1979. The act of reading: A theory of aesthetic response. Johns Hopkins University Press.
Daniel Karell and Michael Freedman. 2019. Rhetorics of radicalism. American Sociological Review, 84(4):726-753.
Katherine Keith, David Jenson, and Brendan O'Connor. 2020. Text and causal inference: A review of using text to remove confounding from causal estimates. In Proceedings of ACL.
Soo-Min Kim and Eduard Hovy. 2006. Identifying and analyzing judgment opinions. In Proceedings of NAACL.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In Proceedings of ICLR.
Manabu Kuroki and Judea Pearl. 2014. Measurement bias and effect restoration in causal inference. Biometrika, 101(2):423-437.
10 https://github.com/rpryzant/ causal-text
Stephen C. Levinson. 1995. Three levels of meaning. In Frank R. Palmer, editor, Grammar and Meaning: Essays in Honor of Sir John Lyons, pages 90-115. Cambridge University Press.
Stephen C. Levinson. 2000. *Presumptive Meanings: The Theory of Generalized Conversational Implicature*. MIT Press, Cambridge, MA.
Bing Liu et al. 2010. Sentiment analysis and subjectivity. Handbook of natural language processing, 2(2010):627-666.
Christopher Lucas, Richard A Nielsen, Margaret E Roberts, Brandon M Stewart, Alex Storer, and Dustin Tingley. 2015. Computer-assisted text analysis for comparative politics. Political Analysis, 23(2):254-277.
Li Lucy, Dorottya Demszky, Patricia Bromley, and Dan Jurafsky. 2020. Content analysis of textbooks via natural language processing: Findings on gender, race, and ethnicity in Texas U.S. history textbooks. AERA Open, 6(3).
Yiwei Luo, Dan Jurafsky, and Beth Levin. 2019. From insanely jealous to insanely delicious: Computational models for the semantic bleaching of english intensifiers. In Proceedings of the 1st International Workshop on Computational Approaches to Historical Language Change.
Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of ACL, pages 1003-1011.
Reagan Mozer, Luke Miratrix, Aaron Russell Kaufman, and L Jason Anastasopoulos. 2020. Matching with text data: An experimental evaluation of methods for matching documents and of measuring match quality. Political Analysis, 28(4):445-468.
Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Proceedings of EMNLP, pages 188-197.
Hüseyin Oktay, Akanksha Atrey, and David Jensen. 2019. Identifying when effect restoration will improve estimates of causal effect. In Proceedings of the 2019 SIAM International Conference on Data Mining, pages 190-198. SIAM.
Alexandra Olteanu, Onur Varol, and Emre Kiciman. 2017. Distilling the outcomes of personal experiences: A propensity-scored analysis of social media. In Proceedings of CSCW.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch:
An imperative style, high-performance deep learning library. In Proceedings of NeurIPS.
Judea Pearl. 2009. Causality. Cambridge university press.
Judea Pearl. 2012. On measurement bias in causal inference. Proceedings of UAI.
Christopher Potts. 2009. Formal pragmatics. The Routledge Encyclopedia of Pragmatics.
Vinodkumar Prabhakaran, William L Hamilton, Dan McFarland, and Dan Jurafsky. 2016. Predicting the rise and fall of scientific topics from trends in their rhetorical framing. In Proceedings of ACL, pages 1170-1180.
Reid Pryzant, Sugato Basu, and Kazoo Sone. 2018a. Interpretable neural architectures for attributing an ad's performance to its writing style. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.
Reid Pryzant, Youngjoo Chung, and Dan Jurafsky. 2017. Predicting sales from the language of product descriptions. In Proceedings of the SIGIR Workshop on eCommerce.
Reid Pryzant, Kelly Shen, Dan Jurafsky, and Stefan Wagner. 2018b. Deconfounded lexicon induction for interpretable social science. In Proceedings of NAACL, pages 1615-1625.
Delip Rao and Deepak Ravichandran. 2009. Semi-supervised polarity lexicon induction. In Proceedings of EACL.
Margaret E Roberts, Brandon M Stewart, and Richard A Nielsen. 2020. Adjusting for confounding with text matching. American Journal of Political Science, 64(4):887-903.
Paul R Rosenbaum and Donald B Rubin. 1983. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55.
Paul R Rosenbaum and Donald B Rubin. 1984. Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association, 79(387):516-524.
Koustuv Saha, Benjamin Sugar, John Torous, Bruno Abrahao, Emre Kiciman, and Munmun De Choudhury. 2019. A social media study on the effects of psychiatric medication use. In Proceedings of the International AAAI Conference on Web and Social Media.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
Cosma Shalizi. 2013. Advanced data analysis from an elementary point of view.
Di Shu and Grace Y Yi. 2019. Weighted causal inference methods with mismeasured covariates and misclassified outcomes. Statistics in medicine, 38(10):1835-1854.
Anton Sobolev. 2019. How pro-government "trolls" influence online conversations in russia. Unpublished.
Dhanya Sridhar and Lise Getoor. 2019. Estimating causal effects of tone in online debates. Proceedings of IJCAI.
Yla R Tausczik and James W Pennebaker. 2010. The psychological meaning of words: LIWC and computerized text analysis methods. Journal of language and social psychology, 29(1):24-54.
Victor Veitch, Dhanya Sridhar, and David Blei. 2020. Adapting text embeddings for causal inference. In Proceedings of UAI.
Rob Voigt, Nicholas P Camp, Vinodkumar Prabhakaran, William L Hamilton, Rebecca C Hetey, Camilla M Griffiths, David Jurgens, Dan Jurafsky, and Jennifer L Eberhardt. 2017. Language from police body camera footage shows racial disparities in officer respect. Proceedings of the National Academy of Sciences, 114(25):6521-6526.
Zach Wood-Doughty, Ilya Shpitser, and Mark Dredze. 2018. Challenges of using text classifiers for causal inference. In Proceedings of EMNLP.
Michael Yeomans, Alejandro Kantor, and Dustin Tingley. 2018. The politeness package: Detecting politeness in natural language. *R Journal*, 10(2).
Deniz Yuret and Mehmet Ali Yatbaz. 2010. The noisy channel model for unsupervised word sense disambiguation. Computational Linguistics, 36(1):111-127.
Justine Zhang, Sendhil Mullainathan, and Cristian Danescu-Niculescu-Mizil. 2020. Quantifying the causal effects of conversational tendencies. Proceedings of CSCW.
Xiaojin Zhur and Zoubin Ghahramani. 2002. Learning from labeled and unlabeled data with label propagation. CMU CALD tech report.
Consider the expected outcome  $\mu (t) = \mathbb{E}\left[Y;\mathrm{d}\mathrm{o}(\tilde{T} = t)\right]$ .
$$
\mu (t) = \mathbb {E} _ {W} \left[ \mathbb {E} \left[ Y \mid W; \operatorname {d o} (\tilde {T} = t) \right] \right] \tag {10}
$$
(by iterated expectation)
$$
= \mathbb {E} _ {W} \left[ \mathbb {E} \left[ Y \mid \tilde {T}, \tilde {Z}; \mathrm {d o} (\tilde {T} = t) \right] \right] \tag {11}
$$
(by definition)
$$
= \mathbb {E} _ {W} \left[ \mathbb {E} \left[ Y \mid \tilde {T} = t, \tilde {Z} \right] \right] \tag {12}
$$
(by overlap and no unobserved confounding)
The proof is complete because the estimand  $\psi^{\mathrm{rea.}}$  is simply  $\mu (1) - \mu (0)$ .
B  $C$ -restriction
Section 4 said that adjusting for confounding information  $\tilde{Z} = f(W)$  is sufficient for blocking the confounding backdoor path created by non-treatment properties of the text that readers might perceive. In Section 5.2 we proposed performing this adjustment with BERT. We could alternatively try to block this path by restricting a  $\hat{T}$ -boosting model to only use information related to the confounding covariates  $C$  (for which we can adjust). This could capture the same desired effect as conditioning on  $\tilde{Z} = f(W)$  by accounting for whatever extra information was leaked from  $W$  into  $\hat{T}$ . We accordingly experiment with restricting the model to features that are highly correlated with  $C$ : we compute point-biserial correlation coefficients (Glass and Hopkins, 1996) between each word and  $C$ , then select the top 2000 words as features for the bootstrapping model. Results are given in Table 4 and suggest that while it still gives an improvement over the raw  $\hat{T}$ 's,  $C$ -restriction yields more conservative estimates than adjusting for  $W$ .