SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Naoki Egami\*1, Musashi Hinck2, Brandon M. Stewart\* 2, Hanying Wei1 Columbia University, 2Princeton University
In computational social science (CSS), researchers analyze documents to explain social and political phenomena. In most scenarios, CSS researchers first obtain labels for documents and then explain labels using interpretable regression analyses in the second step. One increasingly common way to annotate documents cheaply at scale is through large language models (LLMs). However, like other scalable ways of producing annotations, such surrogate labels are often imperfect and biased. We present a new algorithm for using imperfect annotation surrogates for downstream statistical analyses while guaranteeing statistical properties—like asymptotic unbiasedness and proper uncertainty quantification—which are fundamental to CSS research. We show that direct use of surrogate labels in downstream statistical analyses leads to substantial bias and invalid confidence intervals, even with high surrogate accuracy of  $80 - 90\%$ . To address this, we build on debiased machine learning to propose the design-based supervised learning (DSL) estimator. DSL employs a doubly-robust procedure to combine surrogate labels with a smaller number of high-quality, gold-standard labels. Our approach guarantees valid inference for downstream statistical analyses, even when surrogates are arbitrarily biased and without requiring stringent assumptions, by controlling the probability of sampling documents for gold-standard labeling. Both our theoretical analysis and experimental results show that DSL provides valid statistical inference while achieving root mean squared errors comparable to existing alternatives that focus only on prediction without inferential guarantees.
Text as data—the application of natural language processing to study document collections in the social sciences and humanities—is increasingly popular. Supervised classifiers have long been used to amortize human effort by scaling a hand-annotated training set to a larger unannotated corpus. Now large language models (LLMs) are drastically lowering the amount of labeled data necessary to achieve reasonable performance which in turn sets the stage for an increase in supervised text as data work. Recent papers have shown that GPT models can (for some tasks) classify documents at levels comparable to non-expert human annotators with few or even no labeled examples [Brown et al., 2020, Gilardi et al., 2023, Ziems et al., 2023].
In social science, such text classification tasks are only the first step. Scholars often use labeled documents in downstream analyses for explanation of corpus level properties [Hopkins and King, 2010, Egami et al., 2022, Feder et al., 2022, Grimmer et al., 2022], for example, using a logistic regression to model a binary outcome  $Y \in \{0,1\}$  by regressing this outcome on some explanatory
*Correspondence to naoki.egami@columbia.edu and bms4@princeton.edu. We provide the supplementary materials in this link (https://naokiegami.com/paper/dsl_supplement.pdf).
(a) Simulated performance of Surrogate-Only Estimation (SO) and DSL. Even for highly accurate surrogates, ignoring measurement error leads to non-trivial bias and undercoverage of  $95\%$  confidence intervals in downstream regression. Correct coverage and asymptotic unbiasedness are essential properties for proper uncertainty quantification—a must in social science. These concerns are resolved using DSL. The data-generating process uses a logistic regression similar to the one in Vansteelandt and Dukes [2022] and is described in the supplement.
Step 1: Fit ML model to predict Y with (Q,X) using the gold-standard data
Step 2: Construct pseudo outcomes using gold-standard and predicted outcomes
$$
\widetilde {Y} = \widehat {g} (Q, X) + \frac {R}{\pi (Q , X)} (Y - \widehat {g} (Q, X))
$$
Step 3: Use pseudo outcomes in downstream analyses
$\widetilde{Y}\sim X$
(b) The DSL Estimator  $Y$  represent gold-standard outcomes available only for a subset of documents.  $Q$  represent surrogate labels (e.g. from an LLM).  $X$  represent explanatory variables that social scientists use in downstream statistical analyses.  $\widehat{g}(Q, X)$  is a supervised machine learning model to predict  $Y$  with  $(Q, X)$ . In the second step, we construct pseudo-outcomes by combining gold-standard outcomes  $Y$ , predicted outcomes  $\widehat{g}(Q, X)$ , an indicator variable for gold-standard labeling  $R$  (taking 1 if hand-coded and 0 otherwise), and the known probability of gold-standard labeling  $\pi(Q, X)$ . In the final step, researchers use pseudo-outcomes in downstream statistical analyses, e.g., regressing  $\widetilde{Y}$  on  $X$ . A full notation summary is available in Table 2.
Figure 1: Overview of the Problem and Design-based Supervised Learning (DSL)
variables  $X \in \mathbb{R}^{d_X}$ . In political science,  $Y$  could represent whether a social media post contains hate speech, and  $X$  could include posters' characteristics, such as gender, education, and partisanship. Regression estimates the share of hate speech posts within levels of explanatory variables. Importantly, this task of explanation in CSS is distinct from unit-level prediction—classifying whether each post contains hate speech. Because of this social science goal of explanation, simple regression models, like logistic regression, are often preferred as low dimensional summaries [Chernozhukov et al., 2018b, Vansteelandt and Dukes, 2022]. Ideally, all documents would be labeled by experts to achieve a gold-standard  $Y$  but this is costly and so researchers turn to more scalable approaches such as LLMs. We call these more scalable—but error-prone—labels, surrogate labels.
When using surrogates in downstream statistical analyses, researchers often ignore measurement error—the unknown and heterogeneous mismatch between the gold-standard label and the surrogate [Knox et al., 2022]. Measurement error is ubiquitous in CSS research due to the inherent difficulty of the task [Ziems et al., 2023], unknown social and racial biases in LLMs [Zhao et al., 2018, Bender et al., 2021], and performance sensitivity to classifiers or prompt engineering [Perez et al., 2021, Zhao et al., 2021]. Ignoring measurement error leads to estimator bias and invalid confidence intervals in downstream statistical analyses even when the surrogate labels are extremely accurate. In a simulated example (shown in Figure 1-(a)), even with surrogate accuracy of  $90\%$ , the bias is substantial and the coverage of a  $95\%$  confidence interval is only  $40\%$  in downstream regression. This is a fatal flaw in the
Method	Data Requirement	Statistical Properties
No Gold-Standard	Consistency	Coverage	Low RMSE
Surrogate Only (SO)	✓	✗	✗	?
Gold-Standard Only (GSO)	✗	✓	✓	✗
Supervised Learning (SL)	✗	✗	✗	?
Design-Based SL (DSL)	✗	✓	✓	?
Table 1: Design-based Supervised Learning (DSL) compared with existing approaches. The statistical properties are compared under settings where researchers control the probability of sampling documents for gold-standard labeling, while no additional assumptions about fitted supervised machine learning models are made. RMSE is marked as indeterminate (?) since the relative order between SO, SL, and DSL depends on the amount of bias specific to applications.
social sciences, where estimated effects are generally small—such that even low measurement error can overturn scientific conclusions—and valid uncertainty quantification is essential to distinguish signal from noise.
We propose a method to use surrogate labels as outcomes for common statistical analyses in the social sciences while guaranteeing consistency, asymptotic normality and valid confidence intervals. We assume a setting where the analyst has a large collection of documents and is interested in a regression of some text-based outcome on some known document-level characteristics. We observe imperfect surrogate labels for all documents and can choose to sample a small number of documents with known probability for an expert to annotate yielding gold-standard labels. Our proposed estimator, design-based supervised learning (DSL), combines the surrogate and gold-standard labels to create a bias-corrected pseudo-outcome, which we use in downstream statistical analyses (see Figure 1-(b)). The proposed DSL estimator is consistent and asymptotically normal, and its corresponding confidence interval is valid, without any further modeling assumptions even when the surrogates are arbitrarily biased. While we do not require accurate surrogates, as their accuracy improves, the efficiency of the DSL increases. These strong theoretical guarantees only require that the sampling probability of the documents selected for gold-standard be known and be bounded away from zero. Both conditions are straightforward to guarantee by design in many social science applications where the whole corpus is available in advance, which gives the name, design-based supervised learning.
After describing our contributions and related work, we formally characterize the problem setting (Section 2). In Section 3, we describe existing approaches including (i) using the surrogates ignoring the measurement error, (ii) using only gold-standard labels and ignoring the surrogates, and (iii) the conventional supervised approaches which use both. In Section 4, we describe our proposed method and prove its theoretical properties. Section 5 benchmarks our method against existing approaches in 18 diverse datasets, demonstrating that DSL is competitive in root mean squared errors while consistently delivering low bias and proper coverage. See Table 1 for a summary. Section 6 concludes with a discussion of limitations.
Contributions. We propose a unified framework for using imperfect surrogate labels in downstream statistical analyses which maintains the CSS priority for unbiasedness and proper coverage. We exploit three features of text-labeling tasks in social science: researchers often control the probability of sampling documents for gold-standard labeling, the accuracy of surrogate labels will likely keep increasing, and most quantities of interest can be written as a regression (i.e. not requiring individual labels). We (1) provide a new estimator, (2) prove strong theoretical guarantees, including consistency and proper asymptotic coverage, without requiring the accuracy of the surrogate or the correct specification of the underlying supervised machine learning estimator, (3) offer extensions to a broad range of moment-based estimators, and (4) demonstrate finite sample performance across 18 datasets which leverage LLMs for surrogate annotation. Our framework provides a theoretically-sound, design-based strategy for downstream analyses.
Related Work. In the text as data literature, there is limited work on addressing measurement error in regressions with predicted outcomes [although see, Bella et al., 2014, Wang et al., 2020, Zhang, 2021]. Existing approaches use a separate gold-standard sample to estimate the error rate and perform a post-hoc correction to the regression. This is related to approaches for calibrating a classifier [Platt, 1999, Zhang, 2021]. The related literature on quantification seeks to characterize
the share of documents in each class and thus corresponds to the intercept-only regression model with a categorical outcome [Forman, 2005, González et al., 2017]. The quantification literature has historically combined this task with domain shift since otherwise the mean of the training data is an extremely strong baseline [Hopkins and King, 2010, Keith and O'Connor, 2018, Card and Smith, 2018, Jerzak et al., 2023]. Our approach encompasses the quantification problem without domain shift (an intercept-only regression) and can handle cases where quantification is used to model different subgroups of the data that are available at training time (using regression).
Our proposed method also draws upon the large literature on double/debiased machine learning and doubly-robust estimation for missing data and causal inference [Robins et al., 1994, Laan and Robins, 2003, Chernozhukov et al., 2018a, Kennedy, 2022]. In particular, our use of bias-corrected pseudo-outcomes builds on foundational results on semiparametric inference with missing data [Robins and Rotnitzky, 1995, Tsiatis, 2006, Rotnitzky and Vansteelandt, 2014, Davidian, 2022] and the growing literature on doubly robust estimators for surrogate outcomes [Kallus and Mao, 2020] and semi-supervised learning [Chakraborty and Cai, 2018, Chakraborty et al., 2022]. A similar framework of using doubly robust estimation to debias measurement errors has also been recently used in other application areas [Angelopoulos et al., 2023, Mozer and Miratrix, 2023]. Like these papers, we exploit the efficient influence function to produce estimators with reduced bias.
Finally, we join an increasing number of papers that explore a variety of different uses of large language models for social science questions [Ornstein et al., 2022, Gilardi et al., 2023, Velez and Liu, 2023, Wu et al., 2023, Ziems et al., 2023]. Our work, in particular, focuses on how to correct biases and measurement errors in outputs from large language models in order to perform valid downstream statistical analyses. We also contribute to the growing literature on using predicted variables in downstream statistical analyses in the social sciences [Fong and Tyler, 2021, Knox et al., 2022, Katsumata and Yamauchi, 2023].
2 The Problem Setting and Design-based Sampling
Consider the case where a researcher wants to classify documents into a binary outcome  $Y \in \{0,1\}$  and then regress this outcome on some explanatory variables  $X \in \mathbb{R}^{d_X}$  using a logistic regression (we consider extensions to non-binary outcomes and more general moment-based estimators below).
Suppose we have  $n$  independent and identically distributed samples of documents. For all documents, we observe surrogate labels  $Q \in \mathbb{R}^{d_Q}$ , optional additional meta-data about the documents,  $W \in \mathbb{R}^{d_W}$ , which might be predictive of  $Y$ , and explanatory variables  $X$  to be included in our regression. We assume the outcome is costly to measure and we can choose to have an expert annotate a subset of the documents to provide the gold-standard  $Y$ . We use a missing indicator  $R_i \in \{0,1\}$  to denote whether document  $i$  is labeled by experts  $(R_i = 1)$  or not  $(R_i = 0)$ . Therefore, we observe the data  $\{R_i, R_iY_i, Q_i, W_i, X_i\}_{i=1}^n$ , and the total number of gold-standard documents is given by  $n_R = \sum_{i=1}^n R_i$ . We use  $\pi(Q_i, W_i, X_i) := \operatorname*{Pr}(R_i = 1 \mid Q_i, W_i, X_i)$  to denote the probability of sampling document  $i$  for gold-standard labeling. Formally, we assume that the sampling probability for gold-standard labeling is known and bounded away from zero.
Assumption 1 (Design-based Sampling for Gold-Standard Labeling)
For all  $i$ ,  $\pi(Q_i, W_i, X_i)$  is known to researchers, and  $\pi(Q_i, W_i, X_i) > 0$ .
The assumption holds when the researcher directly controls the sampling design. For example, if a researcher has 10000 documents and samples 100 of them to expert-announce at random,  $\pi = \frac{100}{10000} = .01$  for all documents. We also allow more complex stratified sampling schemes (shown later in our applications) and can cover any case where the sampling depends on the surrogates, optional covariates or, explanatory variables such that  $\pi(Q_i, W_i, X_i)$  is known. Restricting ourselves to this sampling mechanism allows us to guarantee,  $Y \perp R \mid Q, W, X$ . This assumption does rule out some applications—such as instances of domain shift where documents from the target population are not available at annotation time—but captures most social science research applications where researchers need to annotate a corpus of documents which is available in total from the outset.
Our estimand of interest is the coefficients of the oracle logistic regression  $\beta^{*}\in \mathbb{R}^{d_{X}}$ , which is a solution to the following moment equations.
$$
\mathbb {E} \left\{\left(Y - \operatorname {e x p i t} \left(X ^ {\top} \beta\right)\right) X \right\} = 0, \tag {1}
$$
Yi	Outcome, which is observed for documents labeled by experts.
Xi	Explanatory variables we include in downstream regression analysis.
β	Coefficients of the downstream regression analysis. Our main estimand of interest.
Qi	Surrogate outcome (e.g., LLM annotation).
Wi	Optional covariates that are predictive of Y.
Ri	Missing indicator, indicates whether we sample document i for gold-standard labeling.
π(Qi,Wi,Xi)	Prob. of sampling document i for gold-standard labeling depending on (Qi,Wi,Xi).
g(Qi,Wi,Xi)	Estimated supervised machine learning model predicting Y as a function of (Qi,Wi,Xi).
Table 2: Summary of Our Notation.
where  $\mathrm{expit}(\cdot)$  is the inverse of the logit function.  $\beta^{*}$  is defined as the solution to the moment equations above. Here,  $\beta^{*}$  is seen as a low-dimensional summary, and thus, this paper does not assume the underlying data-generating process follows the logistic regression.
3 Existing Approaches: Their Methodological Challenges
Because we do not observe  $Y$  for all documents, we cannot directly solve equation (1) to estimate  $\beta^{*}$ . This motivates the three existing strategies that researchers use in practice: using only the surrogate, using only the subset of documents that have gold-standard annotations, and using a conventional supervised learning approach. None of these approaches attain asymptotically unbiased estimation with proper coverage under minimal assumptions while also using surrogate labels to increase efficiency.
Surrogate Only Estimation (SO) The most common approach in practice is to replace  $Y$  with one of the surrogate labels ignoring any error. While we have motivated this with LLM-generated labels, this can be generated by any previously trained classifier, an average of labels produced from multiple LLMs/prompts, or any other strategy not using the gold-standard data. For example, researchers can construct LLM-based text label  $Q_{i}$  as a surrogate for the outcome  $Y_{i}$  in each document  $i$ , and then they can run a logistic regression regressing  $Q_{i}$  on  $X_{i}$ .
The appeal of this approach is that it uses all documents  $n$  for downstream analyses. This method is consistent with valid confidence intervals only when measurement errors are random and mean-zero conditional on  $X$ , which is rarely the case in practice. Recent papers have evaluated LLMs and concluded that the predictions are 'accurate enough' to use without correction in at least some settings [Ornstein et al., 2022, Ziems et al., 2023, Tornberg, 2023, Gilardi et al., 2023]. However as we saw in Figure 1-(a) even substantially more accurate LLM predictions can lead to non-trivial error.
Gold-Standard Only Estimation (GSO) Even if the researcher wants to use a surrogate, they presumably produce gold-standard annotations for a subset of documents to evaluate accuracy. The simplest approach of obtaining valid statistical inference is to use only this gold-standard data, ignoring documents that only have a surrogate label. Regressing  $Y_{i}$  on  $X_{i}$  only using gold-standard data with weights  $1 / \pi(Q_{i}, W_{i}, X_{i})$  is equivalent to solving the following moment equations,
$$
\sum_ {i: R _ {i} = 1} \frac {1}{\pi \left(Q _ {i} , W _ {i} , X _ {i}\right)} \left(Y _ {i} - \operatorname {e x p i t} \left(X _ {i} ^ {\top} \beta\right)\right) X _ {i} = 0, \tag {2}
$$
where the summation is taken only over documents with  $R_{i} = 1$ . Regular M-estimation theory shows that the estimated coefficients are consistent and their corresponding confidence intervals are valid [van der Vaart, 2000]. The key limitation of this approach is that it ignores the surrogate labels which, while biased, can help improve efficiency.
Supervised Learning (SL) Some researchers combine gold-standard data and LLMs-based surrogate outcomes in supervised learning [e.g., Wang et al., 2020, Zhang, 2021]. While the exact implementation varies, we consider a most common version of this strategy where researchers fit a black-box supervised machine learning model to predict  $Y$  with  $(Q,W,X)$  using the gold-standard data. Then, using the fitted supervised machine learning model,  $\widehat{g}(Q_i,W_i,X_i)$ , researchers predict labels for the entire documents and use the predicted label as the outcome in downstream logistic
regression. This is equivalent to solving the moment equation,
$$
\sum_ {i = 1} ^ {n} \left(\widehat {g} \left(Q _ {i}, W _ {i}, X _ {i}\right) - \operatorname {e x p i t} \left(X _ {i} ^ {\top} \beta\right)\right) X _ {i} = 0. \tag {3}
$$
Researchers can combine this with sample-splitting or cross-fitting to avoid overfitting bias. This estimation method is consistent only when  $g(Q_i, W_i, X_i)$  is correctly specified and consistent to the true conditional expectation  $\mathbb{E}(Y \mid Q, W, X)$  in  $L_2$  norm, i.e.,  $||\hat{g}(Q, W, X) - \mathbb{E}(Y \mid Q, W, X)||_2 = o_p(1)$  as sample size  $n$  goes to infinity. This assumption is trivial when the surrogate is binary and there are no covariates, but implausible in more general settings. More problematically, in general, this estimator cannot provide valid confidence intervals due to regularization bias, even when the underlying machine learning model is correctly specified [Chernozhukov et al., 2018a].
What we call SL here is a broad class of methods. It includes as special cases the surrogate only estimator  $(\widehat{g}(Q_i, W_i, X_i) = Q_i)$  and classical supervised learning with crossfitting (here there is no  $Q_i$  and we predict using other document features  $W_i$  and  $X_i$ ). With appropriate modification, it also includes post-hoc corrections using a gold-standard set as in Levy and Kass [1970], Hausman et al. [1998], Wang et al. [2020], Zhang [2021]. All of these strategies are expected to perform well in terms of RMSE. However, for these methods to be consistent, they need to assume the correct specification of the estimator for  $g$ , which is often implausible in social science applications. Even under such an assumption, they do not provide valid confidence intervals or p-values, unless other additional stringent assumptions are imposed.
4 Our Proposed Estimator: Design-based Supervised Learning
No existing strategy meets our requirements of asymptotically unbiased estimation with proper coverage while also efficiently using the surrogate labels. Design-based supervised learning (DSL) improves upon the conventional supervised learning procedure (which is not generally consistent and does not provide valid confidence intervals) by using a bias-corrected pseudo-outcome in downstream statistical analyses (summarized in Definition 1 and Algorithm 1).
For the proposed DSL estimator, we employ a  $K$ -fold cross-fitting procedure [Chernozhukov et al., 2018a]. We first partition the observation indices  $i = 1,\dots ,n$  into  $K$  groups  $\mathcal{D}_k$  where  $k = 1,\ldots ,K$ . We then learn the supervised machine learning model  $\widehat{g}_k$  by predicting  $Y$  using  $(Q,W,X)$  using all expert-coded documents not in  $\mathcal{D}_k$ . We then define a bias-corrected pseudo-outcome  $\widetilde{Y}_i^k$  for observations in  $\mathcal{D}_k$  as follows.
$$
\widetilde {Y} _ {i} ^ {k} := \widehat {g} _ {k} \left(Q _ {i}, W _ {i}, X _ {i}\right) + \frac {R _ {i}}{\pi \left(Q _ {i} , W _ {i} , X _ {i}\right)} \left(Y _ {i} - \widehat {g} _ {k} \left(Q _ {i}, W _ {i}, X _ {i}\right)\right). \tag {4}
$$
This pseudo-outcome can be seen as the sum of the predicted labels  $\widehat{g}_k(Q_i, W_i, X_i)$  (the same as in the conventional supervised learning) and the bias-correction term  $\frac{R_i}{\pi(Q_i, W_i, X_i)} (Y_i - \widehat{g}_k(Q_i, W_i, X_i))$ . Our use of the pseudo-outcome builds on a long history of the doubly robust methods [e.g., Robins et al., 1994, Rotnitzky and Vansteelandt, 2014, Chakraborty et al., 2022].