SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This bias-correction step guarantees that the conditional expectation  $\mathbb{E}_k(\widetilde{Y}^k\mid Q,W,X)$  is equal to the true conditional expectation  $\mathbb{E}_k(Y\mid Q,W,X)$  under Assumption 1, even when the supervised machine learning estimator  $\widehat{g}$  is misspecified. Here we use  $\mathbb{E}_k$  to denote the expectation over  $\mathcal{D}_k$ , which is independent of data used to learn  $\widehat{g}_k$  in cross-fitting.
$$
\begin{array}{l} \mathbb {E} _ {k} \left(\widetilde {Y} ^ {k} \mid Q, W, X\right) := \mathbb {E} _ {k} \left(\widehat {g} _ {k} (Q, W, X) + \frac {R}{\pi (Q , W , X)} \left(Y - \widehat {g} _ {k} (Q, W, X)\right) \Big | Q, W, X\right) \\ = \frac {\mathbb {E} _ {k} (R Y \mid Q , W , X)}{\pi (Q , W , X)} + \left(1 - \frac {\mathbb {E} _ {k} (R \mid Q , W , X)}{\pi (Q , W , X)}\right) \widehat {g} _ {k} (Q, W, X) \\ = \mathbb {E} _ {k} (Y \mid Q, W, X) \\ \end{array}
$$
where the first line follows from the definition of the pseudo-outcome and the second from the rearrangement of terms. The final line follows because  $\mathbb{E}_k(RY\mid Q,W,X) = \mathbb{E}_k(R\mid Q,W,X)\mathbb{E}_k(Y\mid Q,W,X)$  based on Assumption 1, and  $\mathbb{E}_k(R\mid Q,W,X) = \pi (Q,W,X)$  by definition. Importantly, this equality does not require any assumption about the supervised machine learning method  $\widehat{g}_k(Q,W,X)$  or about measurement errors. The proposed DSL estimator exploits this robustness of the bias-corrected pseudo-outcome to the misspecification of  $\widehat{g} (Q,W,X)$ .
Definition 1 (Design-based Supervised Learning Estimator) We first construct the bias-corrected pseudo-outcome  $\widetilde{Y}_i^k$  as in equation (4) using  $K$ -fold cross-fitting. Then, we define the design-based supervised (DSL) estimator for logistic regression coefficient  $\beta$  to be a solution to the following moment equations.
$$
\sum_ {k = 1} ^ {K} \sum_ {i \in \mathcal {D} _ {k}} \left(\widetilde {Y} _ {i} ^ {k} - \operatorname {e x p i t} \left(X _ {i} ^ {\top} \beta\right)\right) X _ {i} = 0. \tag {5}
$$
Algorithm 1 Design-based Supervised Learning
Inputs: Data  $\{R_i,R_iY_i,Q_i,W_i,X_i\}_{i = 1}^n$  , Known gold-standard probability  $\pi (Q_i,W_i,X_i)$  for all  $i$
Step 1: Randomly partition the observation indices into  $K$  groups  $\mathcal{D}_k$  where  $k = 1,\dots ,K$
Step 2: Learn  $\widehat{g}_k$  from gold-standard documents not in  $\mathcal{D}_k$  by predicting  $Y$  with  $(Q,W,X)$
Step 3: For documents  $i \in \mathcal{D}_k$ , construct the bias-corrected pseudo-outcome  $\widetilde{Y}_i^k$  (see equation (4))
Step 4: Solving the logistic regression moment equation by replacing  $Y_{i}$  with  $\widetilde{Y}_i^k$  (see equation (5))
Outputs: Estimated coefficients  $\widehat{\beta}$  and Estimated variance-covariance matrix  $\widehat{V}$
Proposition 1 (proof in supplement) shows that the DSL estimator is consistent and provides valid confidence intervals when the gold-standard probability is known to researchers, without requiring the correct specification of the supervised machine learning method.
Proposition 1 Under Assumption 1, when the DSL estimator is fitted with the cross-fitting approach (Algorithm 1), estimated coefficients  $\widehat{\beta}$  are consistent and asymptotically normal.
$$
\sqrt {n} \left(\widehat {\beta} - \beta^ {*}\right) \xrightarrow {d} \mathcal {N} (0, V) \tag {6}
$$
In addition, the following variance estimator  $\widehat{V}$  is consistent to  $V$ , that is,  $\widehat{V} \xrightarrow{p} V$ , where
$$
\widehat {V} := \widehat {\mathbf {M}} ^ {- 1} \widehat {\Omega} \widehat {\mathbf {M}} ^ {- 1}, \widehat {\mathbf {M}} := \frac {1}{n} \sum_ {i = 1} ^ {n} \operatorname {e x p i t} (X _ {i} ^ {\top} \widehat {\beta}) (1 - \operatorname {e x p i t} (X _ {i} ^ {\top} \widehat {\beta}))) X _ {i} X _ {i} ^ {\top},
$$
$$
\widehat {\Omega} := \frac {1}{n} \sum_ {k = 1} ^ {K} \sum_ {i \in \mathcal {D} _ {k}} \left(\widetilde {Y} _ {i} ^ {k} - \operatorname {e x p i t} \left(X _ {i} ^ {\top} \widehat {\beta}\right)\right) ^ {2} X _ {i} X _ {i} ^ {\top}.
$$
This proposition highlights the desirable theoretical properties of DSL relative to alternatives. First, the estimators of coefficients are consistent, and asymptotic confidence intervals are valid. These results hold even when  $g$  is arbitrarily misspecified as long as it does not diverge to infinity. This is unlike the SL approach which requires that  $g$  is correctly specified and is consistent to the true conditional expectation  $\mathbb{E}(Y|Q,W,X)$ . Second, even when LLMs-based labels are arbitrarily biased, such biased measures do not asymptotically bias final estimates of coefficients  $\beta$ . Therefore, unlike the SO approach, researchers can use LLMs-based labels and retain theoretical guarantees even when there is a concern of differential measurement errors. Finally, the asymptotic variance  $V$  decreases with the accuracy of the surrogate labels, allowing for use of non-gold-standard data unlike GSO.
These powerful theoretical guarantees mainly come from the research design where researchers know and control the expert-coding probability  $\pi(Q, W, X)$ . The well-known double/debiased machine learning framework mostly focuses on settings where the expert-coding probability  $\pi$  is unknown and needs to be estimated from data. In such settings, researchers typically need to assume (i) estimation of nuisance functions, such as  $g$  and  $\pi$ , is correctly specified and consistent to the true conditional expectations, and (ii) they satisfy particular convergence rates, e.g.,  $o_p(n^{-0.25})$ . We do not require either assumption because we exploit the research design where the expert-coding probability  $\pi(Q, W, X)$  is known.
Extension: Method of Moment Estimator Our proposed DSL estimator can accommodate any number of surrogate labels and a wide range of outcome models that can be written as a class of moment estimators with what we will call design-based moments. Importantly, many common estimators in the social sciences, including measurement, linear regression, and logistic regression, can be written as the method of moment estimator. In general, suppose researchers are interested in a method of moment estimator with a moment function  $m(Y,Q,W,X;\beta ,g)$  where  $(Y,Q,W,X)$
are the data,  $\beta$  are parameters of interest, and  $g$  is the supervised machine learning function. Then, the estimand of interest  $\beta_{M}^{*}$  can be written as the solution to the following moment equations.  $\mathbb{E}(m(Y,Q,W,X;\beta ,g^{*})) = 0$ , where  $g^{*}$  is the true conditional expectation  $\mathbb{E}(Y\mid Q,W,X)$ . We define the moment function to be design-based when the moment function is insensitive to the first step machine learning function.
Definition 2 (Design-based Moments) A moment is design-based when  $\mathbb{E}(m(Y,Q,W,X;\beta ,g)) =$ $\mathbb{E}(m(Y,Q,W,X;\beta ,g'))$  for any  $\beta$  and any machine learning functions  $g$  and  $g^{\prime}$  that do not diverge.
Note that the design-based moment is most often a doubly robust moment, and Chernozhukov et al. [2022] provide a comprehensive theory about doubly robust moments. In this general setup, the DSL estimator  $\widehat{\beta}_M$  is a solution to the following moment equation.
$$
\sum_ {k = 1} ^ {K} \sum_ {i \in \mathcal {D} _ {k}} m \left(Y _ {i}, Q _ {i}, W _ {i}, X _ {i}; \beta , \widehat {g} _ {k}\right) = 0. \tag {7}
$$
Proposition 2 Under Assumption 1, when the DSL estimator with a design-based moment is fitted with the cross-fitting approach,  $\widehat{\beta}_M$  is consistent and asymptotically normal.
We provide the proof and the corresponding variance estimator in the appendix.
In this section, we verify that our theoretical expectations hold in 18 real-world datasets. We compare DSL and all three existing approaches, demonstrating that only DSL and GSO meet the standard for bias and coverage, while DSL improves efficiency. We use generalized random forests via grf package in R to estimate  $g$  function [Tibshirani et al., 2022] and all cross-fitting procedures use five splits. A comparison to Wang et al. [2020] is included in the supplement.
Logistic Regression in Congressional Bills Data We use data from the Congressional Bills Project [CBP, Adler and Wilkerson, 2006] to construct a benchmark regression task. CBP is a database of 400K public and private bills introduced in the U.S. House and Senate since 1947. Each bill is hand-coded by trained human coders with one of 20 legislative topics. Our downstream analysis is a logistic regression to examine the association between whether a bill is labeled Macroeconomy  $(Y)$  and three traits of the legislator proposing the bill  $(X)$ —whether the person is a senator, whether they are a Democrat, and the DW-Nominate measure of ideology [Lewis et al., 2023]. For our simulations, we use 10K documents randomly drawn from documents labeled with the Macroeconomy (the positive class), or the Law and Crime, Defense and International Affairs topics (reflecting that the negative class is often diverse). We consider two scenarios: in the balanced condition, there are 5K documents in each class. In the imbalanced condition, there are 1K documents in the positive class and 9K documents in the negative class. For surrogate labels  $(Q)$ , we include zero-shot predictions using GPT-3 [Brown et al., 2020] which achieves accuracy of  $68\%$  (balanced) and  $90\%$  (imbalanced). As an additional document-covariate  $(W)$ , we include the cosine distance between embeddings of the document and the class description using MPnet [Reimers and Gurevych, 2019, Song et al., 2020]. Additional details and experiments using five-shot predictions are included in the appendix.
Figure 2 compares the performance of our four estimators on bias, coverage and RMSE. As expected, only GSO and DSL perform well on bias and achieve nominal coverage. While SL is able to achieve better RMSE at these sample sizes, DSL achieves a  $14\%$  gain in RMSE over GSO (balanced condition) as shown in Figure 3. This increases to  $31\%$  in the five-shot results. The empirical result accords with the theory: only DSL and SO provide a low-bias estimator achieving nominal coverage and DSL is notably more efficient.
Class Prevalence Estimation Ziems et al. [2023] evaluate zero-shot performance of a variety of LLMs on 24 diverse CSS benchmark tasks including detection tasks for emotion, hate-speech, ideology and misinformation. They, like others [e.g. Ornstein et al., 2022, Gilardi et al., 2023] make a qualified recommendation for zero-shot classification only (our SO estimator) in research-grade settings noting that "In some lower-stakes or aggregate population analyses,  $70\%$  [accuracy] may be a sufficient threshold for direct use in downstream analyses" [Ziems et al., 2023, p. 13]. We evaluate
Figure 2: Logistic regression estimation with Congressional Bills Project Data. Results for a three variable logistic regression model of a binary outcome indicating whether a bill is about Macroeconomy. Bias shows the standardized root mean squared bias (averaged over the three coefficients). Coverage shows proportion of  $95\%$  confidence intervals covering the truth. RMSE plots the average RMSE of the coefficients on a log scale. Each sampled dataset contains 10K datapoints with the X-axis providing gold-standard sample size. We average over 500 simulations at each point. Only DSL and GSO are able to achieve proper coverage, but DSL is more efficient.
Figure 3: Improvement of DSL over GSO. Both DSL and GSO attain asymptotic unbiasedness and proper coverage. Here we show the gain in efficiency for DSL over GSO in the balanced condition. As the quality of the surrogate rises (here as we move from the 0-shot to 5-shot setting) the efficiency gain from DSL grows.
on the 17 datasets in their Table 3 using flan-u12 [Chung et al., 2022] as the surrogate which is one of the highest performing LLMs in their study. Because there are no consistent covariates for these studies, we focus on class prevalence estimation—the simplest case of our regression setting.
Dataset (# cat.)	Obs.	LLM	Bias (×100)	Coverage (Percentage)	RMSE (×100)
Acc.	F1	SO	GSO	SL	DSL	SO	GSO	SL	DSL	SO	GSO	SL	DSL
Misinfo. (2)	500	77.6	77.4	9.0	0.0	0.2	0.0	0.6	94.8	95.2	96.2	9.2	5.0	4.3	4.3
Emotion (6)	498	70.3	70.1	2.0	0.1	0.6	0.1	67.2	94.2	91.3	94.0	2.9	3.7	3.2	3.2
Figur. (4)	500	64.0	61.7	5.5	0.2	0.3	0.2	41.2	94.1	91.8	94.3	6.0	4.4	3.8	3.9
Power (2)	500	60.8	57.8	26.6	0.0	0.2	0.1	0.0	94.8	93.2	94.8	26.6	5.0	5.0	4.9
Toxic. (2)	500	56.6	50.5	34.8	0.0	0.4	0.0	0.0	94.8	93.8	94.8	34.9	5.0	5.1	4.9
Disc. (7)	497	41.9	39.5	5.4	0.1	0.5	0.1	39.7	93.7	91.4	93.8	5.8	3.5	3.4	3.4
News (3)	498	40.3	38.8	10.6	0.0	0.3	0.0	4.3	95.1	94.0	95.1	10.8	4.7	4.6	4.5
Emp. (3)	498	39.8	34.7	24.7	0.0	0.2	0.0	0.0	95.1	93.7	94.9	24.8	4.7	4.8	4.7
Hate (6)	498	35.9	32.8	9.8	0.1	0.4	0.1	17.4	94.2	92.6	94.1	10.1	3.7	3.6	3.6
Table 3: Class prevalence estimation on a subset of the 17 datasets from Ziems et al. [2023] with  $n = 100$  gold-standard labels. Multi-class tasks are converted to 1-vs-all binary tasks with reported performance averaged over tasks. SO, SL and DSL use a surrogate label from f1an-u12. Numbers in green indicate any estimator within 0.1pp of the lowest bias; blue indicate any estimator achieving above  $94.5\%$  coverage; orange indicate any estimator achieving within 0.001 of the best RMSE. Remaining 8 tasks are shown in the supplement.
Table 3 shows the results for  $n = 100$  gold-standard labels (analyses at different  $n$  in the supplement) ordered by the accuracy of the LLM surrogate. As with the logistic regression our theoretical expectations hold. Even for surrogate labels above  $70\%$  accuracy, the SO estimator has high bias and consequentially poor coverage for this aggregate quantity. Using only 100 gold-standard annotations, DSL is able to achieve very low bias and nominal (or near nominal) coverage with consistent gains in RMSE compared to GSO (the only other estimator to attain acceptable bias and nominal coverage). Importantly, the RMSE of DSL is also comparable to SL.
We have introduced a design-based supervised learning estimator which provides a principled framework for using surrogate labels in downstream social science tasks. We showed competitive performance across 18 diverse CSS classification tasks with LLM surrogates, providing low bias and approximately nominal coverage, while achieving RMSE comparable to existing alternatives that focus only on prediction without inferential guarantees. Our approach works for any predicted outcome used in a downstream task [see e.g. Wang et al., 2020, for several examples across fields].
**Limitations.** We briefly describe three limitations of our work. First, DSL is focused on a very specific setting: outcome variables used in a regression where researchers need to annotate a corpus of documents available from the outset. This is a common setting in social science, but not a universal one: we might use text as a predictor [Fong and Tyler, 2021, Katsumata and Yamauchi, 2023], require individual document classifications, or have domain shift in the target population. Second, you need a way to construct gold-standard labels. This will often be naturally accessible using data that might otherwise be used to evaluate accuracy of the classifier. Like any method using gold-standard labels, DSL can be sensitive to the gold-standard being accurate—something which is not always true even in competition test sets and might not even be feasible in some settings. Finally, our method focuses on social science research settings where the researcher's priority is bias and coverage rather than RMSE. As SL explicitly targets RMSE, it may be the better option if that is the primary consideration. We often find that the RMSE tradeoff is small in order to have low bias and nominal coverage.
Future Work. In future work we plan to explore the entire analysis pipeline including improvements to prompt engineering and the implications of conditioning prompt engineering on preliminary assessments of performance. This paper assumed the probability of gold-standard labeling is given, but a natural future extension is to consider optimal sampling regimes for the gold-standard data.
Acknowledgments We are very grateful to Amir Feder for excellent feedback on an earlier draft. Research reported in this publication was supported by the Department of Political Science at Columbia University, the Data-Driven Social Science Initiative at Princeton University, and Princeton Research Computing.