Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models
Authors: Naoki Egami
Musashi Hinck
Brandon M. Stewart
Hanying Wei
Venue: NeurIPS 2023 poster (2023)
Keywords: Computational Social Science
Large Language Models
Statistical Inference
Causal Inference
TL;DR: We propose a new algorithm that uses large language model outputs for downstream statistical analyses in computational social science research, ensuring unbiasedness and proper uncertainty quantification.

Abstract: In computational social science (CSS), researchers analyze documents to explain social and political phenomena. In most scenarios, CSS researchers first obtain labels for documents and then explain labels using interpretable regression analyses in the second step. One increasingly common way to annotate documents cheaply at scale is through large language models (LLMs). However, like other scalable ways of producing annotations, such surrogate labels are often imperfect and biased. We present a new algorithm for using imperfect annotation surrogates for downstream statistical analyses while guaranteeing statistical properties—like asymptotic unbiasedness and proper uncertainty quantification—which are fundamental to CSS research. We show that direct use of surrogate labels in downstream statistical analyses leads to substantial bias and invalid confidence intervals, even with high surrogate accuracy of 80-90\%. To address this, we build on debiased machine learning to propose the design-based supervised learning (DSL) estimator. DSL employs a doubly-robust procedure to combine surrogate labels with a smaller number of high-quality, gold-standard labels. Our approach guarantees valid inference for downstream statistical analyses, even when surrogates are arbitrarily biased and without requiring stringent assumptions, by controlling the probability of sampling documents for gold-standard labeling. Both our theoretical analysis and experimental results show that DSL provides valid statistical inference while achieving root mean squared errors comparable to existing alternatives that focus only on prediction without inferential guarantees.

Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models
Naoki E Musashi Hinck?, Brandon M. Stewart' 2, Hanying Wei!
"Columbia University, Pprinceton University
Abstract
In computational social science (CSS), researchers analyze documents to explain social and political phenomena. In most scenarios, CSS researchers first obtain labels for documents and then explain labels using interpretable regression anal- yses in the second step. One increasingly common way to annotate documents cheaply at scale is through large language models (LLMs). However, like other scalable ways of producing annotations, such surrogate labels are often imperfect and biased. We present for downstream statistical analyses while guaranteeing statistical properties-like asymptotic unbiasedness and proper uncertainty quantification- mental to CSS research. We show that direct use of surrogate labels in downstream statistical analyses leads substantial bias and invalid confidence intervals, even with high surrogate accuracy of 80-90%. To address this, we build on debiased machine learning to propose the design-bass based supervised learning (DSL) estimator. DSL employs doubly-robust procedure to combine surrogate labels with a smaller number of high-quality, gold-standal labels. Our approach guarantees valid in- ference for downstream statistical analyses, even when surrogates are arbitrarily biased and without requiring stringent assumptions, by controlling the probability of sampling documents for gold-standard labeling. Both our theoretical analysis and experimental results show that DSL provides valid statistical inference while achieving root mean squared errors comparable to existing alternatives that focus only on prediction without inferential guarantees.
algorithm for using imperfect annotation surrogates
are funda-
Introduction
Text data the application of natural language processing to study document collections in the social sciences and humanities-is increasingly popular. Supervised classifiers have long been used amortize human effort by scaling hand-annotated training set larger unannotated corpus. Now large language models (LLMs) are drastically lowering the amount of labeled data necessary achieve reasonable performance which in turn sets the stage for an increase in supervised text as data work. Recent papers have shown that GPT models can (for some tasks) classify documents at levels comparable to non-expert human annotators with few or even no labeled examples [Brown et al., 2020, Gilardi et al., 2023, Ziems et al., 2023].
In social science, such text classification tasks are only the first step. Scholars often use labeled documents in downstream analyses for explanation of corpus level properties [Hopkins and King, 2010, Egami al., 2022, Feder al., 2022, Grimmer et al., 2022], for example, using a logistic regression to model a binary outcome Y E {0, 1} by regressing this outcome on some explanatory
"Correspondence to naoki, mentary materials in this link beetts::/nackiegeenn
. wgw.i@coolmmiaaaaa.m..maamaaaaaa edu and bms40princeton. edu. We provide supple-
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Bas
Coverage
RMSE
Standardizedd Blas
Coverage Percentage
50 051
0
RMSE
3-
00
0.
Accuracy of LLMs
Accuracy of LLMs
Accuracy of LLMs
(a) Simulated performance Surrogate-Onf Estimation (SO) and DSL. Even for highly accurate surrogates, ignoring measurement error leads to non-trivial bias and undercoverage of 95% confidence intervals down- stream regression. Correct coverage and asymptotic unbiasedness are essential properties for proper uncertainty quantification must in social science. These concerns resolved using DSL. The data generating process uses logistic regression similar to one in Vansteelandt and Dukes [2022] and described in the supplement.
Step 1:
Step 2:
Step 3:
Fit ML model to predict Y with (Q,X) using the gold-standard data
Construct pseudo outcomes using gold-standard and predicted outcomes
Use pseudo outcomes in downstream analyses
x
x(Q.))
x
X
y= 8(0.x) x) x(.x))
(Y 8(0,X))
y
(b) The DSL Estimator Y represent gold-standard outcomes available only for subset of documents. Q represent surrogate labels (e-g. from an L.MM). represent explanatory variables that social scientists use in downstream statistical analyses. x(Q is supervised machine learning model to predict Y with (Q,X). In the second step, construct pssudd-outcomes by combining gold cold-standard outcomes Y. predicted outcomes g(Q,X), X), indicator variable for gold-standard labeling R (laking hand-coded and otherwise), and the known probability of gold-standan labeling π (Q, X). In the final step, researchers use pseudo-outcomes downstream statistical analyses, e.g., regressing Yon X. full notation summary is available in Table 2.
Figure 1: Overview of the Problem and Design-based Supervised Learning (DSL)
variables X [dx In political science, could represent whether social media post contains hate speech, and X could include posters' characteristics, such as gender, education, and partisanship. Regression estimates the share of hate speech posts within levels of explanatory variables. Importantly, this task of explanation in CSS is distinct from unit-level prediction-clasiifiine classifying whether each post contains hate speech. Because of this social science goal of explanation, simple regression models, like logistic regression, are often preferred low dimensional summaries [Chernozhukov et al., 8b, Vansteelandt and Dukes, 2022]. Ideally, documents would labeled by experts to achieve gold-standard Y but this is costly and researchers turn to more scalable approaches such as LLMs. call these more scalable- arror-prone-labe surrogate labels.
When using surrogates in downstream statistical analyses, researchers often ignore measurement error-the unknown and heterogeneous mismatch between the gold-standard label and the surrogate [Knox et al., 2022]. Measurement error is ubiquitous in CSS research due to the inherent difficulty of the task [Ziems al., 2023], unknown social and racial biases in LLMs [Zhao al., 2018. Bender et al., 2021], and performance sensitivity to classifiers or prompt engineering [Perez et al., 2021, Zhao al., 2021]. Ignoring measurement error leads to estimator bias and invalid confidence intervals in downstream statistical analyses even when the surrogate labels are extremely accurate. In simulated example (shown in Figure 1 I-(a),, even with surrogate accuracy of 90%, the bias is substantial and the coverage of 95% confidence interval is only 40% in downstream regression. This is fatal flaw in the

---

Data Requirement
Statistical Properties
Method
No Gold-Standard
Consistency
Coverage
Low RMSE
Surrogate Only (SO)
X
X
Gold-Standard Only (GSO)
X
Supervised Learning (SL)
X
X
?
Design-Based SL (DSL)
Table 1: Design-based Supervised Learning (DSL) compared with existing approaches. The statistical properties are compared under settings where researchers control the probability of sampling documens for gold-standard labeling, while no additional assumptions about fitted supervised machine learning models made. RMSE is marked indeterminate (?) since the relative order between SO, SL, and DSL depends on the amount of bias specific to applications.
social sciences, where estimated effects are generally small-such that even low measurement error overturn seientific conclusions-ane and valid uncertainty quantification is essential to distinguish signal from noise.
We propose method to surrogate labels as outcomes for common statistical analyses in the social sciences while guaranteeing consistency, asymptotic normality and valid confidence intervals. We assume a setting where the analyst has a large collection of documents and is interested in a regression of some text- boxt-based outcome on some known document-level characteristics. We observe imperfect surrogate labels for all documents and can choose sample small number of documents with known probability for an expert to annotate yielding gold-standan labels. Our proposed estimator, design-based supervised learning (DSL), combines the surrogate and gold-standard labels create a bias-corrected pseudo-outcomer which we use in downstream statistical analyses (see Figure 1- (b)). The proposed DSL estimator is consistent and asymptotically normal, and its corresponding confidence interval is valid, without any further modeling assumptions even when the surrogates are arbitrarily biased. While we do not require accurate surrogates, as their accuracy improves, the efficiency of the DSL increases. These strong theoretical guarantees only require that the sampling probability of the documents selected for gold-standard be known and be bounded away from zero. Both conditions are straightforward to guarantee by design in many social science applications where the whole corpus is available in advance, which gives the name, design- based supervised learning.
After describing our contributions and related work, we formally characterize the problem setting (Section 2). In Section 3, we describe existing approaches including (i) using the surrogates ignoring the measurement error, (ii) using only gold-standard labels and ignoring the surrogates, and (iii) the conventional supervised approaches which use both. In Section 4, we describe our proposed method and prove its theoretical properties. Section 5 benchmarks our method against existing approaches in 18 diverse datasets, demonstrating that DSL is competitive in root mean squared errors while consistently delivering low bias and proper coverage. See Table I for summary. Section 6 concludes with discussion of limitations.
Contributions. We propose unified framework for using imperfect surrogate labels in downstream statistical analyses which maintains the CSS priority for unbiasedness and proper coverage. We exploit three features of lext-labelin tasks in social science: researchers often control the probability of sampling documents for wold-standard labeling, the accuracy of surrogate labels will likely keep increasing, and most quantities of interest can be written as a regression (i.c. not requiring individual labels). We (1) provide new estimator, (2) prove strong theoretical guarantees, including consistency and proper asymptotic coverage, without requiring the accuracy of the surrogate or the correct specification of the underlying supervised machine learning estimator, (3) offer extensions to broad range of moment-basee based estimators, and (4) demonstrate finite sample performance across 18 datasets which leverage LLMs for surrogate annotation. Our framework provides a theoretically-sound dbseg strategy for downstream analyses.
Related Work. In the text as data literature, there is limited work on addressing measurement error in regressions with predicted outcomes [although see, Bella et al., 2014, Wang et al., 2020, Zhang, 2021]. Existing approaches use a separate gold-standard sample to estimate the error rate and perform a post-hoc correction to the regression. This is related to approaches for calibrating a classifier [Platt, 1999, Zhang, 2021]. The related literature quantification seeks to characterize
3

---

the share of documents in each class and thus corresponds to the intercept-only regression model with a categorical outcome [Forman, 2005, González et al., 2017]. The quantification literature has historically combined this task with domain shift since otherwise the mean of the training data an extremely strong baseline [Hopkins and King, 2010, Keith and O' D'Conno,, 2018, Card and Smith, 2018, Jerzak et al., 2023]. Our approach encompasses the quantification problem without domain shift (an intercept-only regression) and can handle cases where quantification is used to model different subgroups of the data that are available at training time (using regression).
Our proposed method also draws upon the large literature on double/debiased machine learning and doubly-robust estimation for missing data and causal inference [Robins et al., 1994, Laan and Robins, 2003, Chernozhukov et al., 2018a, Kennedy, 2022]. In particular, our use of bias-corrected outuoo-su builds on foundational results on semiparametric inference with missing data [Robins and Rotnitzky, 1995, Tsiatis, 2006, Rotnitzky and Vansteelandt, 2014, Davidian, 2022] and the growing literature on doubly robust estimators for surrogate outcomes [ [Kallus and Mao, 2020] and semi-supervised learning [Chakrabortty and Cai, 2018, Chakrabortty et al., 2022]. A similar framework of using doubly robust estimation to debias measurement errors has also been recently used in other application areas [Angelopoulos et al., 2023, Mozer and Miratrix, 2023]. Like these papers, we exploit the efficient influence function to produce estimators with reduced bias.
Finally, we join an increasing number of papers that explore variety of different uses of large language models for social science questions [Ornstein et al., 2022, Gilardi et al., 2023, Velez and Liu, 2023, Wu al., 2023, Ziems et al., 2023]. Our work, in particular, focuses on how to correct biases and measurement errors in outputs from large language models in order to perform valid downstream statistical analyses. We also contribute to the growing literature on using predicted variables in downstream statistical analyses in the social sciences [Fong and Tyler, 2021, Knox et al., 2022, Katsumata and Yamauchi, 2023].
2
The Problem Setting and Design-based Sampling
Consider the case where researcher wants to classify documents into binary outcome Y {0, 1} and then regress this outcome on some explanatory variables X E Rdx using a logistic regression (we consider extensions to non-binary outcomes and more general moment-based estimators below).
Suppose we have n independent and identically distributed samples of documents. For all documents, we observe surrogate labels Q E Rd4, optional additional meta-data about the documents, W E Rdw, which might be predictive of Y, and explanatory variables X to be included in our regression. We assume the outcome costly measure and can choose to have an expert annotate subset of the documents to provide the gold-standard Y. We use a missing indicator Rg E {0, 1} to denote whether document/ is labeled by experts (R, 1) or not (R, 0). Therefore, we observe the data (R,, Y., Qú W X,) 1 and the total number of gold-standard documents is given by n = Σm RG, We use x (Q., X.) == Pr(R, 1 Q., W;, X;) to denote the probability of sampling document i for gold-standard labeling. Formally, we assume that the sampling probability for gold-s standard labeling is known and bounded away from zero
Assumption (Design- based Sampling for Gold-Standard Labeling) For all i, T W,, X:) is known to researchers, and T T(Q., W,) 0.
The assumption holds when the researcher directly controls the sampling design. For example, if a researcher has 10000 documents and samples 100 of them to expert-annotar at random, 10000 .01 for all documents. We also allow more complex stratified sampling schemes (shown later in our applications) and can cover any case where the sampling depends on the surrogates, optional covariates or, explanatory variables such that x (Q:, W,, X2) is known. Restricting ourselves to this sampling mechanism allows us to guarantee, Y I. R Q, W. This assumption does rule out as instances of domain shift where documents from the target population are not available at annotation time-but captures most social science research applications where researchers need to annotate a corpus of documents which is available total from the outset.
100 =
some applications-
Our estimand of interest is the coefficients of the oracle logistic regression 8 E Rdx, which is solution to the following moment equations.
E{ - expit(X (a)(xX = 0,
(I)
4

---

1
Oulcome, which is observed for documents labeled by experts. Explanatory variables include in downstream regression analysis. Coefficients of the downstream regression analysis. Our main estimand of interest. Surrogate outcome (e.g., LLM annotation).
X,
B
O1
W.
Optional covariates that predictive of
R
Missing indicator, indicates whether we sample document i for gold gold-standard labeling.
TQ Wi, X.) (Q,, W, X))
Prob. of sampling document for gold-standard labeling depending on (Qi. W., X.). Estimated supervised machine learning model predicting Y as a function (Qi, W.. Xz).
Table 2: Summary of Our Notation.
where expit(.) is the inverse of the logit function. 3 is defined the solution to the moment equations above. Here, 8r is seen as low-dimensional summary, and thus, this paper does not assume the underlying data-geneg process follows the logistic regression.
Existing Approaches: Their Methodological Challenges
Because we do not observe Y for all documents, we cannot directly solve equation (1) to estimate 3r. This motivates the three existing strategies that researchers use in practice: using only the surrogate, using only the subset of documents that have gold-standard annotations, and using a conventional supervised learning approach. None of these approaches attain asymptotically unbiased estimation with proper coverage under minimal assumptions while also using surrogate labels to increase efficiency.
Surrogate Only Estimation (8O) The most common approach in practice is to replace Y with one of the surrogate labels ignoring any error. While we have motivated this with LLM-generated labels, this can be generated by any previously trained classifier, an average of labels produced from multiple LLMs/prompts, or any other strategy not using the gold-standard data. For example, researchers can construct LLM-based text label Q; as a surrogate for the outcome Y; in each document i, and then they can run a logistic regression regressing Q, on X,
The appeal of this approach is that it uses all documents n for downstream analyses. This method is consistent with valid confidence intervals only when measurement errors are random and mean-zero conditional on X, which rarely the case in practice. Recent papers have evaluated LLMs and concluded that the predictions are 'accurate enough' to use without correction in at least some settings [Omstein al., 2022, Ziems al., 2023, Törnberg, 2023, Gilardi et al., 2023]. However as we saw in Figure 1-(a) even substantially more accurate LLM predictions can lead to non-trivial error.
Gold-Standard Only Estimation (GSO) Even if the researcher wants to use a surrogate, they presumably produce gold-standard annotations for a subset of documents to evaluate accuracy. The simplest approach of obtaining valid statistical inference is to use only this gold-standard data, ignoring documents that only have surrogate label. Regressing Y, on X, only using gold-stan standadd data with weights 1/T(Q,, W,, X.) is equivalent to solving the following moment equations,
Σ fR =(Q., W,, X,) (Y: expit (x, B))x 0,
where the summation is taken only over documents with R; = 1. Regular M-estimation theory shows that the estimated coefficients are consistent and their corresponding confidence intervals are valid [van der Vaart, 2000]. The key limitation of this approach is that it ignores the surrogate labels which, while biased, can help improve efficiency.
Supervised Learning (SL) Some researchers combine gold- standad data and LLMs- sur- rogate outcomes in supervised learning [e.g., Wang et al., 2020, Zhang, 2021]. While the exact implementation varies, we consider a most common version of this strategy where researchers fit a black-box supervised machine learning model to predict Y with (Q. W. X) using the gold-standard
data. Then, using the fitted supervised machine learning model, I(Q,, W, X2), researchers predict labels for the entire documents and use the predicted label as the outcome in downstream logistic
5

---

regression. This is equivalent to solving the moment equation,
(O,Q,, exp 8))X; = 0.
cross-fitting to avoid overfitting bias. This estimation method is consistent only when g(Qi, W,, X;) is correctly specified and consistent to the true conditional expectation E(Y Q, W. X) in L.2 norm, i.e., ( W, X) IE(Y QQ, W. X x)ll trivial when the surrogate binary and
Researchers can combine this with sample-splitingg
(1( as sample size n goes to infinity. This assumption there are no covariates, but implausible in more general settings. More problematically, in general, this estimator cannot provide valid confidence intervals due to regularization bias, even when the underlying machine learning model is correctly specified [Chermozhukov et al., 2018a].
What we call SL here is a broad class of methods. It includes as special cases the surrogate only estimator (Q( W. X²)) = Qi) and classical supervised learning with crossfitting (here there is no Qi and we predict using other document features W; and X.). With appropriate modification, also includes post-hoc corrections using a gold-standard set as in Levy and Kass [1970], Hausman al. [1998], Wang et [2020], Zhang [2021]. All of these strategies are expected to perform well in terms of RMSE. However, for these methods to be consistent, they need to assume the correct specification of the estimator for g. which is often implausible in social science applications. Even under such an assumption, they do not provide valid confidence intervals or p-values, unless other additional stringent assumptions are imposed.
4 Our Proposed Estimator: Design-based Supervised Learning
No existing strategy meets our requirements of asymptotically unbiased estimation with proper coverage while also efficiently using the surrogate labels. Design-based supervised learning (DSL) improves upon the conventional supervised learning procedure (which is not generally consistent and does not provide valid confidence intervals) by using a dias-corrected pseudo-outeome in downstream statistical analyses (summarized in Definition 1 and Algorithm 1).
For the proposed DSL estimator, employ K Kfolld cross- cooss-itting procedure [Chernozhukov et al., 20] We first partition the observation indices i = =,,... n into K groups Dr where k: = -1.... K. We then learn the supervised machine learning model ØK by predicting Y using (Q, W, X) using all expert-coded documents not in DR. We then define a bias-corrected pseudo-outcome Y for observations in D& as follows.
R Y (Q. W,, X,) + T(Q,, W, (Y, Ox,Qu, W,, X,)).
(4)
This pseudo-outcome can be seen as the sum of the predicted labels (Q. W,, X:) (the same as in Rg the conventional supervised learning) and the bias-correction term =[Q..W.. X.J (Y - (Qi, W., X.)). Our use of the pseudo- pseudo-outcome builds on long history of the doubly robust methods [e.g., Robins et al., 1994, Rotnitzky and Vansteelandt, 2014, Chakrabortty et al., 2022].
This bias correctono step guarantees that the conditional expectation E, (Y* Q. X) is equal to the true conditional expectation (y Q, W, X) under Assumption 1, even when the supervised machine learning estimator ÿ is misspecified. Here we use E, to denote the expectation over Dr. which is independent of data used to learn Is in fittsni
R
E, (Ý* Q, = E, (Q, W. X)
T(Q, W, X) (Y -O W. XT) Q. (.
Es (RY 1 Q, W,X)
(1- (R W,W,
(Q, W.X)
W,X)
E. (Y QQ, W.
where the first line follows from the definition of the pseudo- and the second from the rearrangement of terms. The final line follows because E, (RY Q, W. E, (R | Q, X)XXEB E, (Y Q, W. X) based on Assumption I, and Ex (R Q, W, X) x (Q, W, X) by definition. Importantly, this equality does not require any assumption about the supervised machine learning method 9& (Q, W. X) or about measurement errors. The proposed DSL estimator exploits this robustness of the bias-corrected pseudo-outeome to the misspecification of Φ(Q, X)

---

Definition 1 (Design-based Supervised Learning Estimator) We first construct the bias-corrected psendo-outcome Y as in equation (4) using K-fold cross-fitting. Then, define the design- supervised (DSL) estimator for logistic regression coefficient 3 to be a solution to the following
moment equations.
(Y, expit(X; 8))X, = 0.
(5)
k= DE
Algorithm 1 Design-based Supervised Learning
Inputs: Data {R,, Y. Qú, W,, X; ]= Known soandat probability x(Q,, W,, X;) for all i.
Step I: Randomly partition the observation indices into K groups Dr where k == ..... K.
Step 2: Learn 9K from gold-standard documents not in Dz by predicting Y with (Q, W X)
Step 3: For documents E Dx. construct the bias-corrected pseudo-outcome y (see equation (4))
Step 4: Solving the logistic regression moment equation by replacing Y, with Y (see equation (5))
Outputs:
Estimated coefficients 3 and Estimated variance-covarriance matrix V
Proposition 1 (proof in supplement) shows that the DSL estimator is consistent and provides valid confidence intervals when the gold-standard probability is known to researchers, without requiring the correct specification of the supervised machine learning method.
Proposition 1 Under Assumption 1, when the DSL estimator is fitted with the cross-fitting approach (Algorithm 1), estimated coefficients B are consistent and asymptotically normal.
Vn B*) 4 V()
(6)
In addition, the following variance estimator Vis consistent V, that is, V 2 V. where
V =
M
1ΩM
M
expit(X,) xxpxx xxpp(xxxxxxxx3))))xx/ X)),
1 K
Ω =
kED
This proposition highlights the desirable theoretical properties of DSL relative to alternatives. First, the estimators of coefficients are consistent, and asymptotic confidence intervals are valid. These results hold even when is arbitrarily misspecified as long as it does not diverge to infinity. This unlike the SL approach which requires that g is correctly specified and is consistent to the true conditional expectation E( W, W,X) X). Second, even when LLMs-based labels are arbitrarily biased, such biased measures do not asymptotically bias final estimates of coefficients 3.. Therefore, unlike the SO approach, researchers can use L.sMe- labels and retain theoretical guarantees even when there is concern of differential measurement errors. Finally, the asymptotic variance V decreases with the accuracy of the surrogate labels, allowing for use of non-gold-standan data unlike GSO.
These powerful theoretical guarantees mainly come from the research design where researchers know and control the expert-coding probability TQQ W,X) X). The well known double/debiased machine learning framework mostly focuses on settings where the expert-coding probability is unknown and needs to be estimated from data. In such settings, researchers typically need to assume (i) estimation of nuisance functions, such as g and æ. is correctly specified and consistent to the true conditional expectations, and (ii) they satisfy particular convergence rates, e.g., on (0-255).. We do not require either assumption because we exploit the research design where the expert-coding probability (Q, W, X) is known.
Extension: Method Moment Estimator Our proposed DSL estimator can accommodate any number of surrogate labels and a wide range of outcome models that can be written as class of moment estimators with what we will call design-based moments. Importantly, many common estimators in the social sciences, including measurement, linear regression, and logistic regression, can be written as the method of moment estimator. In general, suppose researchers are interested in method of moment estimator with moment function mYY, Q, W. X; B, where (Y, Q, W,
7

---

are the data, 9 are parameters of interest, and g is the supervised machine learning function. Then, the estimand of interest Shr can be written as the solution to the following moment equations. [),((y Q, W, X; g))) = O, where g" is the true conditional expectation B[ Q, W, X). We define the moment function to be design-based when the moment function is insensitive to the first step machine learning function.
Definition 2 (Design-base Moments) A moment is gesign-basee when E(m(Y, Q. X.W.X3.99) W.W.X: [(m(YY Q, W, X; g')") )) for any 3 and any machine learning functions 9 and g' that do not diverge.
Note that the design-based moment is most often a doubly robust moment, and Chernozhukov et al. [2022] provide a comprehensive theory about doubly robust moments. In this general setup, the DSL estimator Bas is solution to the following moment equation.
Σ
Σ mY,, Qi W., Xx O.
Proposition 2 Under Assumption I, when the DSL estimator with a design-based moment is fitted with the cross-fitting approach, By is consistent and asymptotically normal.
We provide the proof and the corresponding variance estimator in the appendix.
5 Experiments
In this section, we verify that our theoretical expectations hold in 18 real-world datasets. We compare DSL and all three existing approaches, demonstrating that only DSL and GSO meet the standard for bias and coverage, while DSL improves efficiency. We use generalized random forests via grf package in R to estimate g function [Tibshirani et al., 2022] and all cross-fitting procedures use five splits. A comparison to Wang et al. [2020] is included in the supplement.
Logistic Regression in Congressional Bills Data We use data from the Congressional Bills Project [CBP, Adler and Wilkerson, 2006] to construct a benchmark regression task. CBP a database of 400K public and private bills introduced in the U.S. House and Senate since 1947. Each bill is hand- coded by trained human coders with one of 20 legislative topics. Our downstream analysis is a logistic regression examine the association between whether a bill labeled Macroeconomy (Y)) and three traits of the legislator proposing the bill wXhhwhet the person is senator, whether they are Democrat, and the DW-Nominate measure of ideology [Lewis et al., 2023]. For our simulations, we use 10K documents randomly drawn from documents labeled with the Macroeconomy (the positive class), or the Law and Crime, Defense and International Affairs topics (reflecting that the negative class is often diverse). We consider two scenarios: in the balanced condition, there are 5K documents each class. In the imbalanced condition, there are 1K documents in the positive class and 9K documents in the negative class. For surrogate labels (Q), we include zero-shot predictions using GPT-3 [Brown et al., 2020] which achieves accuracy of 68% (balanced) and 90% (imbalanced). As an additional document-cone -ovariite (W), we include the cosine distance between embeddings of the document and the class description using MPnet [Reimers and Gurevych, 2019, Song et al., 2020]. Additional details and experiments using five-shot predictions are included in the appendix.
Figure 2 compares the performance of our four estimators on bias, coverage and RMSE. As expected, only GSO and DSL perform well on bias and achieve nominal coverage. While SL is able to achieve better RMSE these sample sizes, DSL achieves 14% gain in RMSE over GSO (balanced condition) as shown in Figure 3. This increases to 31 % in the five-shot results. The empirical result accords with the theory: only DSL. and SO provide a low-bias estimator achieving nominal coverage and DSL is notably more efficient.
Class Prevalence Estimation Ziems et al. [2023] evaluate zero-shot performance of a variety of LLMs on 24 diverse CSS benchmark tasks including detection tasks for emotion, hate-speech, ideology and misinformation. They, like others (e.g. Ornstein et al., 2022, Gilardi et al., 2023] make a qualified recommendation for zero-shot classification only (our SO estimator) in research-grade settings noting that "In some lower-stakes or aggregate population analyses, 70% [accuracy] may be sufficient threshold for direct use in downstream analyses" [Ziems et al., 2023, p. 13]. We evaluate
8

---

Bias
Coverage
RMSE
0.61
Standardized Blos
Percentage
0.37
RISSE
04
Balanced
0
0.22
100
250
500
1000
100
250
500
1000
100
250
500
1000
0.4
Standardized Bise
0.3
02
0.81
0.2
RMSE
Imbalanced
Coverage
0.37
0.1
0.22
00
100
250
500
1000
100
250
1000
100
250
500
1000
Size Gold-Standard Data
SO
GSO
SL
DSL (Ours)
Figure Logistic regression estimation with Congressional Bills Project Data. Results for a three variable logistic regression model of a binary outcome indicating whether a bill is about Macroeconomy. Bias shows the standardized root mean squared bias (averaged over the three coefficients). Coverage shows proportion of 95% confidence intervals covering the truth. RMSE plots the average RMSE of the coefficients on a log scale. Each sampled dataset contains 10K datapoints with the X-axis providing gold-stane gold-standard sample size. We average over 500 simulations at each point. Only DSL and GSO are able achieve proper coverage, but DSL is more efficient.
RMSE Improvement of DSL GSO
Percent Improvement (%)
0-shot
5-shot
80
30
20
10
100
250
500
1000
100
250
500
1000
Size Gold-Standard Data
Figure 3: Improvement of DSL GSO. Both DSL. and GSO attain asymptotic unbiasedness and proper coverage. Here we show the gain in efficiency for DSL over GSO in the balanced condition. As the quality of the surrogate rises (here we move from the 0-shot 5-shot setting) the efficiency gain from DSL grows.
on the 17 datasets in their Table 3 using flan-ul2 [Chung et al., 2022] as the surrogate which is one of the highest performing LLMs in their study. Because there are no consistent covariates for these studies, we focus on class prevalence cstimation- the simplest case our regression setting.

---

Dataset
LLM
Bias (x 100)
Coverage (Percentage)
RMSE (x 100)
(# cat.) obs.
Acc.
F1
SO
GSO
SL
DSL
so
GSO
SL
DSL
SO
GSO
SL
DSL
Misinfo. (2) 500
77.6 77.4
9.0
0.0
0.2
0.0
0.6
94.8
95.2
96.2
9.2
5.0
4.3
4.3
Emotion (6) 498
70.3
70.1
2.0
0.1
0.6
0.1
67.2
94.2
91.3
94.0
2.9
3.7
3.2
3.2
Figur. (4) 500
64.0
61.7
5.5
0.2
0.3
0.2
41.2
94.1
91.8
94.3
6.0
4.4
3.8
3.9
Power (2) 500
60.8
57.8
26.6
0.0
0.2
0.1
0.0
94.8
93.2
94.8
26.6
5.0
5.0
4.9
Toxic. (2) 500
56.6
50.5
34.8
0.0
0.4
0.0
0.0
94.8
93.8
94.8
34.9
5.0
5.1
4.9
Disc. (7) 497
41,9
39.5
5,4
0.1
0.5
0.1
39.7
93.7
91.4
93.8
5.8
3.5
3,4
3.4
News (3) 498
40.3
38.8
10.6
0.0
0.3
0.0
4.3
95.1
94.0
95.1
10.8
4.7
4.6
4.5
Emp. (3) 498
39.8
34.7
24.7
0.0
0.2
0.0
0.0
95.1
93.7
94.9
24.8
4.7
4.8
4.7
Hate (6) 498
35.9
32.8
9.8
0.1
0.4
0.1
17.4
94.2
92.6
94.1
10.1
3.7
3.6
3.6
Table 3: Class prevalence estimation on subset of the 17 datasets from Ziems et al. [2023]
with 100 gold-standard labels. Multi-class tasks are converted to 1-vs-all binary tasks with reported performance averaged over tasks. SO, SL and DSL use a surrogate label from flan-u12. Numbers in green indicate any estimator within O. I.Ipp of the lowest bias; blue indicate any estimator achieving above 94.5% coverage; orange indicate any estimator achieving within 0.001 of the best RMSE. Remaining tasks are shown in the supplement.
Table 3 shows the results for n 100 gold-standas labels (analyses at different in the supplement) ordered by the accuracy of the LLM surrogate. As with the logistic regression our theoretical expectations hold. Even for surrogate labels above 70% accuracy, the SO estimator has high bias and consequentially poor coverage for this aggregate quantity. Using only 100 gold-standard annotations, DSL is able to achieve very low bias and nominal (or near nominal) coverage with consistent gains in RMSE compared to GSO (the only other estimator to attain acceptable bias and nominal coverage). Importantly, the RMSE of DSL is also comparable to SL.
Discussion
We have introduced design-based supervised learning estimator which provides principled framework for using surrogate labels in downstream social science tasks. We showed competitive performance across 18 diverse CSS classification tasks with LLM surrogates, providing low bias and approximately nominal coverage, while achieving RMSE comparable to existing alternatives that focus only on prediction without inferential guarantees. Our approach works for any predicted outcome used in a downstream task [see e.g. Wang et al., 2020, for several examples across fields).
Limitations. We briefly deseribe three limitations of work. First, DSL is focused very specific setting: outcome variables used in regression where researchers need to annotate a corpus of documents available from the outset. This a common setting in social science, but not universal one: we might use text as a predictor [Fong and Tyler, 2021, Katsumata and Yamauchi, 2023], require individual document classifications, or have domain shift in the target population. Second, you need a way to construct gold-standard labels. This will often be naturally accessible using data that might otherwise be used evaluate accuracy of the classifier. Like any method using stad-standard labels, DSL can be sensitive to the gold-standard being securate-sometiee in competition test sets and might not even be feasible in some settings. Finally, our method focuses on social science research settings where the researcher's priority is bias and coverage rather than RMSE. As SL explicitly targets RMSE, it may be the better option if that is the primary consideration. We often find that the RMSE tradeoff is small in order to have low bias and nominal coverage.
which is not always true even
Future Work. In future work we plan to explore the entire analysis pipeline including improvements to prompt engineering and the implications of conditioning prompt engineering preliminary assessments of performance. This paper assumed the probability of stad-staddard labeling is given, but a natural future extension is to consider optimal sampling regimes for the gold-standard data.
10

---

Acknowledgments We are very grateful to Amir Feder for excellent feedback on an earlier draft. Research reported in this publication was supported by the Department of Political Science Columbia University, the Data-Driven Social Science Initiative at Princeton University, and Princeton Research Computing.
References
E Scott Adler and John Wilkerson. Congressional bills project. NSF, 880066:00880061, 2006.
Anastasios N. Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I. Jordan, and Tijana Zrnic. Prediction-poweree inference. Science, 382(6671) 2023. doi: 10. 126//ciencee URL https://www. science. wttp:///tt/t//////ttt/tt/t.otatt http:/....11////// 1126/sciencceeeee/e add00....0.......0...... adi6000.
Antonio Bella, Cesar Ferri, José Hernández-Oralle and María José Ramírez- Quintaaa. Aggregative quantification for regression. Data Mining and Knowledge Discovery, 28:475-518, 2014.
Emily M Bender, Timnit Gebru, Angelina McMillan-Maij Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610-623, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Dallas Card and Noah A Smith. The importance of calibration for estimating proportions from annotations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 1636-1646, 2018.
Abhishek Chakrabortty and Tianxi Cai. Efficient and adaptive linear regression in eeii-supervised settings. Annals of Statistics, 2018.
Abhishek Chakrabortty, Guorong Dai, and Eric Tchetgen Tchetgen. A general framework for treatment effect estimation in semi-supervised and high dimensional settings. arXiv preprint arXiv:2201.00466 2022.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/Debiased Machine Learning for Treatment and Structural Parameters. Econometrics Journal, 21:C1 C68, 2018a.
Victor Cheroozhukov, Mert Demirer, Esther Duflo, and Ivan Fernandez- Val. Generic machine learning inference on heterogeneous treatment effects in randomized experiments, with an application to immunization in india. Technical report, National Bureau of Economic Research, 2018b.
Victor Chernozhukov, Juan Carlos Escanciano, Hidehiko Ichimura, Whitney K Newey, and James M Robins. Locally robust semiparametric estimation. Econometrica, 10(4):150 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetus language models. arXiv preprint arXiv:2200. 1416, 2022.
Marie Davidian. Methods based on semiparametric theory for analysis in the presence of missing data. Annual Review of Statistics and Its Application, 9: 9:167-196, 2022.
Naoki Egami, Christian J. Fong, Justin Grimmer, Margaret Roberts, and Brandon M. Stewart. How to make causal inferences using texts. Science Advances, 8(42))cab22652, 2022. doi: 10.1 sci5dv.abg2652. URL https: http://wwwssccice www.eennccerrg//oooeeoo/eo wtgg/oo//abbaa 10. http://abb.....1///cc.eeoo 1ww///cii......... abg2652.
Amir Feder, Katherine Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Dos Noud-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E Roberts, et al. Causal inference in natural language processing: Estimation, prediction, interpretation and beyond. Transactions of the Association for Computational Linguistics, 10:1 18.1138-1110 1158, 2022.
11

---

Christian Fong and Matthew Tyler. Machine learning predictions regression covariates. Political Analysis, 2944)4677-484 2021.
George Forman. Counting positives accurately despite inaccurate classification. In Machine Learning: ECML 2005: 16th European Conference Machine Learning. Porto, Portugal, October 3-7, 2005. Proceedings 16, pages 564-575. Springer, 2005.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30), jul 2023. doi: 23.1077//naa.230006666
URL htwp:: //doi::////// org/ 10. 1073/ 1073/pnas, 2305016100.
Pablo González, Alberto Castaño, Nitesh V Chawla, and Juan José Del Coz. A review on quantifica- tion learning. ACM Computing Surveys (CSUR), 50(5): 2017.
Justin Grimmer, Margaret E Roberts, and Brandon M Stewart. Text as data: A new framework for machine learning and the social sciences. Princeton University Press, 2022.
Jerry A Hausman, Jason Abrevaya, and Fiona M Scott-Morton. Misclassification of the dependent variable in discrete-response setting. Journal of econometrics, 87(2):239-26, 1998.
Daniel Hopkins and Gary King. A method automated nonparametric content analysis for social science. American Journal of Political Science, 54(1):229-265 247, 2010.
Connor T Jerzak, Gary King, and Anton Strezhnev. An improved method of automated nonparametric content analysis for social science. Political Analysis, 31(1):42-58, 2023.
Nathan Kallus and Xiaojie Mao. On the role of surrogates in the efficient estimation of treatment effects with limited outcome data, arXiv preprint arXiv:2003. 2020.
Hiroto Katsumata and Soichiro Yamauchi. Statistical analysis with machine learning predicted variables. 2023.
Katherine Keith and Brendan O'Connor. Uncertainty-awanl generative models for inferring document class prevalence. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4575-4585, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 10.18653/v//Die 14877 URL https: https://aclanthology"
org/D18- 11877
Edward H Kennedy. Semiparametric doubly robust targeted double machine learning: a review. arXiv preprint arXiv:2203.0644 2022.
Dean Knox, Christopher Lucas, and Wendy Tam Cho. Testing causal theories with learned proxies. Annual Review of Political Science, 25:419 441, 2022.
Mark J Laan and James M Robins. Unified methods for censored longitudinal data and causality. Springer, 2003.
Paul S Levy and Edward H Kass. A dtree-population model for sequential screening for bacteriuria. American Journal of Epidemiology, 91(2):148-155 1970.
Jeffrey Lewis, Keith Poole, Howard Rosenthal, Adam Boche, Aaron Rudkin, and Luke Sonnet. Voteview: Congressional roll-call votes database. See https://ooteveew.com/ 2023.
Reagan Mozer and Luke Miratrix. Decreasing the human coding burden in randomized trials with text-based outcomes via model-assisted impact analysis. arXiv preprint arXiv:2309. 13666, 2023.
Joseph T Ornstein, Elise N Blasingame, and Jake S Truscott. How to train your stochastic parrot: Large language models for political texts. Working Paper, 2022.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. Advances in neural information processing systems, 34:1 10000 2021.
John Platt. Probabilistic outputs for support vector machines and comparisons to regularized likeli- hood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
12

---

Nils Reimers and Iryna Gurevych. Sentence-bert, Sentence embeddings using siamese bert-networks.
In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. As- sociation for Computational Linguistics, 11 2019. Btl.http://arat http:///aaa.....rr////////00..0
wwtp://.//bbbb//////// 10084.
James M Robins and Andrea Rotnitzky. Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Staristical Association, 90(429):122-129, 1995.
James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimation of Regression Coefficients When Some Regressors Are Not Always Observed. Journal of the American Statistical Association, 89
(427):846-866, 1994.
Andrea Rotnitzky and Stjjn Vansteelandt. Doublc-robust methods. In Handbook of missing data methodology, pages 185-212. CRC Press, 2014.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Yin Liu. Mpnet: Masked and permuted pre-training for language understanding. Advances in Neural Information Processing Systems, 33: 16857-16867, 2020.
Julie Tibshirani, Susan Athey, Erik Sverdrup, and Stefan Wager. grft Generalized Random Forests, 2022. URL https: /etpss//CRANN project. hagg/pakaage-grt R package version 2.2.1.
Petter Törnberg. Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning. arXiv preprint 2304.:2300.0658,,, 2023.
Anastasios A Tsiatis Semiparametric theory and missing data. Springer, 2006.
Aad W van der Vaart. Asymptotic Statistics, volume 3. Cambridge university press, 2000.
Stijn Vansteelandt and Oliver Dukes. Assumption-lean Inference for Generalised Linear Model Parameters. Journal of the Royal Statisticul Society Series B: Statistical Methodology, 84(3): 657-685, 07 2022. ISSN 1369-7412. doi: 10.1111/rssb.12 12504. URL hattts::/doi.org/// 1111/rssb.1255 12504.
Yamil Velez and Patrick Liu. Confronting core issues: A critical test of attitude polarization. APSA Preprints, 2023.
Siruo Wang, Tyler McCormick, and Jeffrey T Leek. Methods for correcting inference based on outcomes predicted by machine leamning Proceedings of the National Academy of Sciences, 117 (48):30266-3022) 30275, 2020.
Patrick Y. Wu, Jonathan Nagler, Joshua A. Tucker, and Solomon Messing. Large language models can be used to estimate the latent positions of politicians. arXiv preprint arXiv:2303. 12057, 2023.
Han Zhang. How using machine learning classification as variable in regression leads to attenuation bias and what to do about it. SocArXiv, 2021.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Short Papers). pages 15-20, New Orleans, Louisiana. June 2018. Association for Computational Linguistics. doi: 1111655/11/N18-20000 URL pettp::/aaclntthot/ http://aaaaatthoggo.ooooo Warg/N18-2000
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697-12706. PMLR, 2021.
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. Can large language models transform computational social science? arXiv preprint arX1v:2305.03514, 2023.
13


Reviews:
[{"review_id": "EoOgbcTlxU", "reviewer": "Reviewer_qD8b", "summary": "The authors consider using outputs of LLMs for labeling in downstream analysis in computational social science while guaranteeing statistical properties (asymptotic unbiasedness and uncertainty quantification). They show that directly using LLM-produced labels yields bias and invalid confidence intervals, then propose the design-based semi-supervised learning (DSL) estimator as an alternative. DSL combines surrogate labels from LLMs and gold-standard labels within a doubly robust procedure that guarantees valid inference even when the surrogates are arbitrarily biased and without requiring stringent assumptions. Experiments on 18 datasets show that DSL yields valid statistical inference while achieving error rates comparable to existing alternatives that focus on prediction with statistical guarantees. ", "questions": "In several places, the authors emphasize the need for knowing \\pi (the probability of obtaining a gold-standard label) and \\pi > 0 for all samples. However, this is not further discussed or described in the experiments. For instance, in Section 2.2 in the Appendix, the authors claim that \\pi is known because they can choose which samples to label, however how does that make \\pi for all samples that are not going to be labeled (with probability 1) larger than 0? ", "limitations": "The authors discuss some of the limitations of the proposed approach. However, the limitations are quite generally about the addressed setting and less so about the methodology. Also, they point to four limitations, however, only three are described (the particular setting, access to gold standard labels and the focus on bias and coverage rather than MSE).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is very well written, the setting and methodology are well described and motivated. The assumptions and potential use cases are clearly stated. Further, though the proposed approach is conceptually simple, it blends sound ideas from semi-supervised learning (not necessarily in the, transductive, machine learning sense), doubly-robust estimation, K-fold cross-fitting and automated labeling with LLMs. ", "weaknesses": "Perhaps the biggest weakness of the paper is that, though to the authors' admission, it prioritizes unbiasedness and coverage, however, DSL does not outperform GSO which is a very simple approach that does not require anything that is not available (a method for surrogate labels and the estimation of g). As a result, it is difficult to imagine a situation, more so in social science, where someone will prefer DSL over the far simpler GSO, unless the reviewer is missing something.\n\nAs minor points: i) the title of the paper is somewhat misleading because, as the authors point out, the proposed approach is agnostic to the methodology used to obtain the surrogate labels, so the fact they use LLMs is more of a detail of the experiments than a characteristic or contribution of the proposed approach; and ii) the paper (in its current form) does not seem a good fit for a conference proceeding considering the amount of important experimental details that need to be relegated to the supplementary material, without which, it is extremely difficult to properly understand the results. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the kind words about the paper! We are glad that you found the setting and methodology well-described and motivated.\n\n\n**(1)** *Perhaps the biggest weakness of the paper is that [...] it prioritizes unbiasedness and coverage, however, DSL does not outperform GSO which is a very simple approach that does not require anything that is not available[...]. As a result, it is difficult to imagine a situation [...] where someone will prefer DSL over the far simpler GSO, unless the reviewer is missing something.*\n\nAs the reviewer notes, gold standard labels only (GSO) and DSL are both asymptotically unbiased and have valid confidence intervals. In terms of the RMSE, we now see that Figure 2 did not convey precise information about how much gains DSL can provide compared to GSO. We clarify here that DSL indeed provides substantial RMSE gains over GSO. As written in the original draft, in our Congressional Bills Data experiment, we find that the RMSE gain by DSL over GSO is about 10% with zero-shot learning LLMs and about 30% with five-shot learning. To make these numbers more intuitive, we translated the RMSE improvement in terms of the number of hand-coded documents, i.e., how many more hand-coded documents does GSO need to achieve the same RMSE as DSL? This translation is possible because both estimators are asymptotically unbiased, and all the gains come from a smaller variance. We find that, for the zero-shot case, the RMSE gain by DSL is equivalent to having 50% more hand-coded documents, and for the five-shot case, the RMSE gain by DSL is equivalent to having 100% more hand-coded documents. Given the high cost of obtaining hand-coded documents, this shows that computational social scientists can obtain unbiased estimates and valid confidence intervals with much fewer hand-coded documents when they use DSL instead of GSO. We provide a new Figure R-1 (see PDF) based on our experiment. \n\nWe also emphasize that as the quality of LLMs improves and their deployment gets easier, the cost-benefit proposition for DSL over GSO will be even greater. In Figure R-2 of the attached PDF, we show the degree of gains of DSL for higher accuracies in simulated data.\n\n**(2)** *As minor points: i) the title of the paper is somewhat misleading because, as the authors point out, the proposed approach is agnostic to the methodology used to obtain the surrogate labels, so the fact they use LLMs is more of a detail of the experiments than a characteristic or contribution of the proposed approach;* \n\nFollowing your and other reviewers’ advice, we are changing the title to **“Using Imperfect Surrogates for Downstream Inference: Design-based Semi-supervised Learning for Social Science Applications of Large Language Models.”** We will also emphasize the generality of DSL and clarify that LLMs are one application in the revision. Please see more detailed responses in our Global Response (at the top of this page). \n\n**(3)** *[...] the paper (in its current form) does not seem a good fit for a conference proceeding considering the amount of important experimental details that need to be relegated to the supplementary material, without which, it is extremely difficult to properly understand the results.*\n\nWe will move more experimental details out of the supplement (in particular, we will be more clear about the specification of downstream analyses, the details of LLM annotations, and training procedures). We are open to suggestions about which would be most useful to move. While we know that many experimental details are in the appendix, we see the experiments as primarily the validation that the theory holds in practice and hence not the main focus. We think NeurIPS would be an excellent venue because this paper tries to address the intersection and frontier of computational social sciences: as some reviewers note, the common social science task of making downstream statistical analysis (and their focus on bias and coverages) is relatively new to computer science communities, while the valid use of LLMs is new to social scientists. Given this novelty and the interdisciplinary nature of our work, we believe NeurIPS is the best venue. \n\n**(4)** *[...] the authors emphasize the need for knowing \\pi (the probability of obtaining a gold-standard label) and \\pi > 0 for all samples. However, this is not further discussed or described in the experiments. [...] [T]he authors claim that \\pi is known because they can choose which samples to label, however how does that make \\pi for all samples that are not going to be labeled (with probability 1) larger than 0?*\n\nThank you for your question. $\\pi$ is the probability of being sampled for hand-coding. As an example, consider random sampling. If I have 10000 documents and I sample 100 of them to hand annotate completely at random, $\\pi$ = 1/100 for all documents regardless of whether any individual document is chosen or not. Therefore, in the social science applications where researchers can sample documents for hand-coding, this $\\pi$ is known, and it is greater than 0. In our experiment, to show the generality of our approach, we use the stratified sampling procedure where we stratified based on LLM labels such that we can oversample rare cases when we conduct hand-coding. We thank the reviewer for this clarification question, and we will add a simple example in the paper itself to hit this point home and will reinforce it when we describe applications. \n\n**(5)** *The authors discuss some of the limitations of the proposed approach. However, the limitations are quite generally about the addressed setting and less so about the methodology. Also, they point to four limitations, however, only three are described [...].*\n\nThank you also for flagging the concerns about the limitations, we will expand this (and get the count correct!) to focus more on limitations specific to the method with an emphasis on points raised by the reviewers.\n"}, {"review_id": "Ub8dhEeDKl", "reviewer": "Reviewer_qrXw", "summary": "This paper studies the use of surrogate labels in model training and it's bias, coverage and performance properties. Practically, the premise is to use a small number of gold data annotations for a classification task, then construct surrogate labels that have desirable bias, coverage and performance properties. The application of this would be doing additional analyses using covariates that would benefit from the additional support in data size.\n\nThe method proposed uses a small amount of gold labels to calibrate the model prediction and is agnostic to the choice of underlying model. The method is compared to using gold labels alone, surrogate labels alone from an external model not fitted to the data and a simple form of semi-supervised learning where a model is fitted to the gold labels.\n\nExperiments are conduced in depth on bias, coverage and RMSE in more depth on one data set on congressional bills. Further experiments are conducted over a battery of data sets related to computational social science applications on bias, coverage and RMSE with 100 gold labels. The results show that bias and coverage for the proposed method are closer to using gold standard data alone, while showing small improvements on some datasets in RMSE.", "questions": "See weaknesses.\n\nAn interesting analysis would be to use either synthetic data or to intentionally vary the accuracy of the underlying model on the same data set (rather than existing accuracy which differs across data sets). This would provide a more controlled analysis of these factors.\n\nWhat is the K (defined in line 192) in the experiments?", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The premise and application is interesting and has not been too well studied to the best of my knowledge.\n\nA large number of data sets were used in the bias/coverage/RMSE experiments, which provides a good starting point for such analyses.\n\nThe proofs and experiments supporting that bias and coverage properties of the proposed method.", "weaknesses": "The key weakness is that application presented as a motivation for this paper is not tested throughout any of the experiments. The covariates are not used in any of the experiments to enhance data analysis, which could have been done with several data sets in the collection, including the congressional bills. \n\nThe experiments show minor improvements in general on the RMSE, especially when compared to the vanilla self-supervised methods. The self-supervised methods represented here are very weak for this category of methods. For example, the gold labels are overwritten by the self-supervision and the self supervision is done in a single round over the entire data set, without any filtering by confidence as usually done, which makes this an artificially weak baseline, especially on bias and coverage. Yet, the SS methods achieve constantly higher RMSE, especially when considering the difference between Gold labels and the proposed method (DSL), which shows that the DSL methods does not capture much of the performance benefits that self-supervision alone could achieve. This leaves the main application of the approach the ability to use more data for covariate analysis, but this application is now shown through experimentation in the paper.\n\nThe paper should also present results in F1 metric (both surrogate label model and method comparison results) because RMSE is not natural for multi-class classification and having the surrogate label model and methods trained on top of predictions on the same method is useful for comparisons.\n\nAs a general comment: the method is agnostic to the source of surrogate labels, as any model can be used to produce the labels and the proposed method does not use any information about the model or its assumptions. Hence, while LLMs can be a source for labels, the paper should be rewritten to remove so many specific mentions of LLMs and discussions about LLMs, which serve no purpose for the paper (perhaps it would for paper publicity), but actually artificially narrow the generality of the method.\n\nThe paper also cites many papers that make unfounded claims or estimates about accuracy needed for 'annotations' (surrogate labels) for applications or model training, while none of the cited papers (e.g. Gilardi 2023) actually trained models to test the improvements those could bring (or not).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review! We are glad that you enjoyed the proofs, experiments, and premise. Below, we quote the weaknesses you identified and respond. \n \n**(1)** *The key weakness is that the application presented as a motivation for this paper is not tested throughout any of the experiments. The covariates are not used in any of the experiments to enhance data analysis. [...]*\n\nWe believe this might be a miscommunication due to our use of semisupervised learning (an overloaded term!). We first clarify our goal and then propose ways to incorporate your suggestions. \n\nOur goal is to *explain* a document label outcome using covariates with regression models (e.g., logistic regression). For example, political scientists are often interested in *explaining* what types of people are more likely to post hate speech. Here, outcome **Y** is whether a post contains hate speech, and covariates **X** could include posters’ characteristics (e.g. education and partisanship). This regression estimates the share of hate speech within strata of **X**. Social scientists run such statistical analyses to *test* social science hypotheses (e.g., whether highly educated people are less likely to post hate speech). These regressions are run explicitly or implicitly through the calculation of subgroup averages. This task of *explanation* and *hypothesis testing* in the social sciences is distinct from *unit-level prediction*—classifying whether each post contains hate speech. Document-level prediction is not less important; our goal is different. We developed DSL to perform downstream regression analyses that are asymptotically unbiased and have valid confidence intervals. Importantly the role of covariates is to use them as explanatory variables in the second-step downstream regression, not to improve model training for document-level predictions. \n\nWe think your understanding was: use an LLM to annotate a set of documents, use those as the labels to train a supervised classifier, and then return this fine-tuned classifier as the output. This is similar to the common computer science goal of training a better model to predict labels at each document level. This goal is different from ours. Your concern, in this case, appears to be that we could have a stronger baseline by only training the classifier using labels about which the LLM is confident and potentially repeating over multiple rounds, folding in new examples each time. Please correct us in the discussion if we have misunderstood your point.\n\nThese procedures are related but they tackle different goals. If the classifier trained on your procedure was used as the predicted labels in our model, it might yield a higher accuracy surrogate, but our main point is how to use such surrogates in downstream analyses. If we directly use such surrogates in the downstream regression, they will lead to substantial bias and invalid confidence intervals, even when the accuracy can be extremely high. \n\nYou were concerned that we didn’t use the covariates “to enhance data analysis,” but the CBP example is specifically about the downstream regression of the Macroeconomy category regressed on three covariates in our downstream logistic regression. Thus, we believe that we have demonstrated in our experiments the core use case, and it is consistent with the vast majority of social science use cases. We will make this point clear in our revision and include additional citations.\n\n**(2)** *The experiments show minor improvements in general on the RMSE, especially when compared to the vanilla self-supervised methods. The self-supervised methods represented here are very weak for this category of methods. [...] Yet, the SS methods achieve constantly higher RMSE.*\n\nWe want to clarify two points. First, we follow the social science priority for bias and coverage. SSL is biased and has invalid confidence intervals, in contrast to DSL. Second, focusing on RMSE, we expect that SSL, which is optimized for RMSE, can outperform DSL. However, in our experiment, we showed that DSL, while maintaining unbiasedness and valid confidence intervals, can achieve RMSE comparable to SSL (even though it is often higher than SSL in a finite sample). Importantly, we additionally show that, as sample size increases, DSL provably dominates SSL in terms of RMSE because bias of SSL does not vanish with sample size, while variance of DSL will vanish. See our new simulation evidence (Figure R-3 in the attached PDF). We have also added the state-of-the-art SSL baselines (Wang et al., *PNAS*), and found the same pattern (see the same Figure R-3). \n\n**(3)** *The paper should also present results in F1 metric [...].*\n\nWe added F1 metric for the surrogate in Table 2 (see Table 2 in PDF). For our main results, the target is the coefficients of the downstream regression and the F1 metric does not apply.\n\n**(4)** *As a general comment: the method is agnostic to the source of surrogate labels [...].*\n\nFollowing your and other reviewers’ advice, we are changing the title to reflect this point. Please see the global response at the top of this page. \n\n**(5)** *The paper also cites many papers [...], while none of the cited papers [...] actually trained models to test the improvements those could bring (or not).*\n\nWe think this is related to the miscommunication described above. Gilardi and others want to annotate documents so they can use LLM annotations as outcomes in downstream analyses without doing hand coding. We show this surrogate-only estimator is biased. \n\n**(6)** *An interesting analysis would be to use either synthetic data or to intentionally vary the accuracy of the underlying model on the same data set[...].*\n\nThis was the goal of Figure 1a, and we now extended it by including additional simulations based on synthetic data (see Figure R-2 in PDF). \n\n**(7)** *What is the K in the experiments?*\n\nThe number of splits which we set to five in both experiments (the default in Chernozhukov et al. 2018).\n"}, {"review_id": "E9T74EfYZ6", "reviewer": "Reviewer_m9JA", "summary": "In order to exploit the advantages of Large Language Models (LLMs) in computational social science (CSS) area, this paper proposed a novel DSL estimator, which employed a doubly-robust procedure to combine surrogate labels with a few gold-standard labels. By using the proposed method, not only LLMs can be used as surrogate label generator, but also the asymptotic unbiasedness and proper uncertainty quantification in CSS can be maintained. Finally, this paper conducted plenty of experiments to prove the effectiveness of the proposed method.", "questions": "N/A", "limitations": "Please see the weaknesses for more details.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper proposed a unified framework for using imperfect surrogate labels in downstream statistical analyses which maintains the CSS priority for unbiasedness and proper coverage. \n2.\tThis paper proposed a strong theoretical guarantee for the effectiveness of the proposed method.\n3.\tThis paper conducted extensive experiments across 18 datasets to compare the performance of the proposed method.\n", "weaknesses": "1.\tThis paper focused on the semi-supervised setting in some CSS scenarios. The difference between the proposed method and previous semi-supervised methods should be compared in a detailed way. \n2.\tThis paper had plenty of notations. An explanation table about these notations should be provided. \n3.\tIn experiments parts, this paper should provide a brief about the evaluation tasks and corresponding datasets, so that the results can be more convincing.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review! We are glad you enjoyed the theory and the extensive experiments. \n\n**Clarify the RMSE gain of DSL over GSO:** \n\nFirst, based on suggestions from reviewers, we further clarified the RMSE gains of our estimator compared to existing alternatives. In particular, we show that the RMSE gains of DSL compared to GSO (the only other method that is unbiased and has valid confidence intervals) are substantial: it is equivalent to having 50% more hand-coded documents (when using zero-shot LLM learning) and to having 100% more hand-coded documents (when using five-shot LLM learning). Please see the new Figure R-1 in the attached PDF. Given the high cost of hand-coding, this practically means that computational social scientists can obtain unbiased estimates and valid confidence intervals with much fewer hand-coded documents when they use DSL instead of existing alternatives. We wanted to emphasize the substantial efficiency gains from our method, which our previous Figure 2 did not convey well. \n\n**Below, we quote the weaknesses you identified below and respond.** \n\n**(1)** *This paper focused on the semi-supervised setting in some CSS scenarios. The difference between the proposed method and previous semi-supervised methods should be compared in a detailed way.*\n\nThank you for this point. We have added an additional comparison with the best state of the art we are aware of (Wang et al. 2020 in *PNAS*), which, as far as we know, is the only published article that proposes an estimator that covers exactly the same settings as ours (downstream regression analyses). As this existing estimator requires stronger parametric assumptions, it is, in general, biased and has invalid confidence intervals in our experiments, in contrast to our proposed DSL. Please see our new results in Figure R-3 in the attached PDF. Importantly, we additionally show that, as sample size increases, DSL provably dominates SSL in terms of RMSE because bias of SSL does not vanish with sample size, while variance of DSL will vanish. See the same Figure R-3. We will also use the additional page to describe the results of our experiments more clearly and what they tell us about how DSL compares to prior SSL methods — essentially that they have slightly better RMSE, while all of the SSL methods are biased and have invalid confidence intervals. We are also open to any other details you would like to see. Thank you for your suggestions. \n\n**(2)** *This paper had plenty of notations. An explanation table about these notations should be provided.*\n\nThank you for the suggestion. We will take the notation summary in Appendix 2.2 and append a condensed version to Figure 1, so readers will have an easily accessible notation guide.\n\n**(3)** *In experiments parts, this paper should provide a brief about the evaluation tasks and corresponding datasets, so that the results can be more convincing.*\n\nThank you! Per the suggestion from you and one other reviewer, we will substantially extend our discussion of the details of the evaluations in the main paper by pulling in material from the appendix. In particular, we will be more clear about the specification of downstream analyses, the details of LLM annotations, and training procedures. We have also extended Table 2 to include more information about each dataset (please see a snippet of Table 2 in the attached PDF; we show two rows as an example due to the space constraint). If there are any other details you would find particularly essential to include, please let us know, and we are happy to accommodate. We think of the theoretical results as the most convincing piece of evidence and the experiments as demonstrating that the results hold in practice. The review has been helpful in clarifying that we need to bring in more details.\n"}, {"review_id": "yKcNbTrVMU", "reviewer": "Reviewer_rbu9", "summary": "The author presents a complete and thorough study in their research regarding semi-supervised learning of social science application targeting bias and coverage control. The work include clear mathematical proof of the purposed approach, and experimental results demonstrating the well alignment of theory and application. The assumption is also clearly stated and limitations are considered. As a non-expert of computational social science, the paper is relatively easy to read and understand. Overall I think this is a solid work in the field. I have a few minor questions and comments below.", "questions": "Please see comments in weakness part", "limitations": "the limitations of the work is well discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. the mathematical proof of bias correction step, equation 4 and the conditionally unbiased estimation of Yi is simple and clear\n2. the surrogate labels can be arbitrarily biased, while the theoretical guarantee still holds is the most fascinating and important contribution of the work. I personally find this really interesting\n3. experimental results show great alignment of theory, comparable performance of RMSE compared with SSL although the method does not specifically target for it ", "weaknesses": "1. the biggest miss/weakness after reading from my perspective is from the title, using LLM annotations for valid statistical inference. In general the research is more like a mathematical proof of bound on semi-supervised learning and demonstration of usefulness with experiment. There is little regarding using LLM annotations. With such a title I would expect a substantial part of the work focusing on LLM generated annotations, e.g. how it is used and how can it be improved from LLM perspective, etc.\n2. the assumption of the purposed approach, though clearly stated, greatly limit the contribution of the approach. It may be a common setting in social science that test set is known, but not the case for majority of LLM annotation use cases.\n3. the asymptotic behavior given the gold-standard annotation, is not well discussed in the main paper. Aspects like the downstream coefficients expectation and variance given the size of golden labels compare with sample space size are not well covered. Some empirical numbers could be extremely useful, beyond the settings in experiment part\n4. some minor comments: page 8 figure 2 annotations, as a non expert of social science it would be good to explain in more detail what is coverage and why it is important to the study. In second experiment class prevalence estimation, better include more details like each dataset's distribution, total size of each dataset, etc. (see it in supplement however I think it is helpful to include some numbers in main paper, as it gives the important size ratio of golden-set and total set)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review! We are glad you found the theoretical and empirical work in the paper enjoyable! \n\n**Clarify the RMSE gain of DSL over GSO:** \n\nFirst, based on suggestions from reviewers, we further clarified the RMSE gains of our estimator compared to existing alternatives. In particular, we show that the RMSE gains of DSL compared to GSO (the only other method that is unbiased and has valid confidence intervals) are substantial: it is equivalent to having 50% more hand-coded documents (when using zero-shot LLM learning) and to having 100% more hand-coded documents (when using five-shot LLM learning). Please see the new Figure R-1 in the attached PDF. Given the high cost of hand-coding, this practically means that computational social scientists can obtain unbiased estimates and valid confidence intervals with much fewer hand-coded documents when they use DSL instead of existing alternatives. We wanted to emphasize the substantial efficiency gains from our method, which our previous Figure 2 did not convey well. \n\n**We quote the weaknesses below and respond.**\n\n**(1)** *the biggest miss/weakness after reading from my perspective is from the title, using LLM annotations for valid statistical inference. In general the research is more like a mathematical proof of bound on semi-supervised learning and demonstration of usefulness with experiment. There is little regarding using LLM annotations. With such a title I would expect a substantial part of the work focusing on LLM generated annotations, e.g. how it is used and how can it be improved from LLM perspective, etc.*\n\nFollowing your and other reviewers’ advice, we are changing the title to “**Using Imperfect Surrogates for Downstream Inference: Design-based Semi-supervised Learning for Social Science Applications of Large Language Models.**” We will also emphasize the generality of DSL and downplay the specific LLM angle in the paper. Please see more detailed responses in our Global Response (at the top of this page).\n\n**(2)** *the assumption of the purposed approach, though clearly stated, greatly limit the contribution of the approach. It may be a common setting in social science that test set is known, but not the case for majority of LLM annotation use cases.*\n\nWe completely agree and this is why we have tried to be so clear both about the social science connection and the assumption. We are adding more citations of applications to the paper to make clear that this covers a sizable portion of social science use cases. We also want to further clarify that our method only requires a random subset of the test data (i.e., we do not need the full test data when we hand-code documents). An interesting future direction is to incorporate the domain shift (i.e., researchers do not have access to a random subset of the test data). We hope the reviewer agrees that the use case is broad enough to support the work, but we agree on the limitation. \n\n**(3)** *the asymptotic behavior given the gold-standard annotation, is not well discussed in the main paper. Aspects like the downstream coefficients expectation and variance given the size of golden labels compare with sample space size are not well covered. Some empirical numbers could be extremely useful, beyond the settings in experiment part*\n\nThank you for raising this point! We want to clarify that this was indeed our goal in Figure 2 (which shows asymptotics for the CBP application). Bias is the difference between the expected values of estimated coefficients and the true coefficients, while RMSE is a summary measure of bias and variance. To make these points clearer, we created a new figure (Figure R-4 in the attached PDF) that directly visualizes the expected values of coefficients and variance. Please let us know whether the newly added figure is what you are thinking about. We are also happy to provide any additional detail you would like to see. In our new simulation results, we also consider a wide range of sample sizes (see Figure R-3 in the attached PDF). \n\n**(4)** *some minor comments: page 8 figure 2 annotations, as a non expert of social science it would be good to explain in more detail what is coverage and why it is important to the study.*\n\nThank you for your question. The coverage is a probability that the 95% confidence interval covers the true coefficients over the sampling distribution. If an estimator has valid uncertainty quantification and valid confidence interval, this coverage should be (asymptotically) at least 95%. This valid uncertainty quantification is essential because social scientists often conduct statistical analyses to *test social science hypotheses* rather than making predictions. For example, when studying online hate speech, social scientists would be interested in testing whether highly educated people are less likely to post hate speech. To conduct such hypothesis testing, valid uncertainty quantification (with confidence intervals and corresponding p-values) is fundamental. We will provide this discussion in the paper and extend the caption of Figure 2. \n\n**(5)** *In second experiment class prevalence estimation, better include more details like each dataset's distribution, total size of each dataset, etc. (see it in supplement however I think it is helpful to include some numbers in main paper, as it gives the important size ratio of golden-set and total set)*\n\nThank you for the suggestion! We amended Table 2 to include this information. Please see the updated Table 2 in the attached PDF. Please let us know whether the newly added table is what you are thinking about. \n"}]

Author rebuttal:
Thank you to all the reviewers for the careful engagement with our work. In this paper, we proposed how to use imperfect LLM annotations in downstream regression analyses, while guaranteeing asymptotic unbiasedness and proper uncertainty quantification, which are fundamental to social science research. 

All four reviewers agreed that the problem and setting of our paper are interesting, and our proposed methodology has strong theoretical guarantees. Most felt that our experiments using 18 different computational social science data sets were strong as well. In this reply, we address some of the common critiques and provide a guide to the new figures in the attached PDF.

**Why Bias and Coverage are Important in the Social Sciences**

We want to further clarify why we focused on bias and valid uncertainty quantification and why they are often prioritized over RMSE in the social sciences. This is primarily because social scientists often conduct statistical analyses to *test social science hypotheses* rather than making predictions. For example, when studying online hate speech, social scientists would be interested in testing whether highly educated people are less likely to post hate speech. To conduct such hypothesis testing, valid uncertainty quantification (with confidence intervals and corresponding p-values) is essential. If there exist several estimators that are asymptotically unbiased and provide valid confidence intervals, it is best to choose the one with a lower RMSE. In this paper, we showed that the gold-standard only (GSO) estimator and DSL estimator are the only two methods that are unbiased and have valid confidence intervals, and DSL outperforms GSO in terms of RMSE, which we elaborate more on below. 

**Concerns about RMSE Improvements over Existing Alternatives**

There was concern about the RMSE improvements of DSL over the gold standard only (GSO) and semi-supervised learning (SSL) benchmarks. 

In terms of comparisons against GSO, we admit that Figure 2 did not convey clear information about how much DSL can improve RMSE (this was masked because of a log scale). Please let us re-clarify this point here. As written in the original draft, in our Congressional Bills Data experiment, we find that the RMSE gain is about 10% with zero-shot learning LLMs and about 30% with five-shot learning. To make these numbers more intuitive, we translated the RMSE gains into the gains in the number of hand-coded documents, i.e., how many more hand-coded documents does GSO need to have the same RMSE as DSL? This translation is possible because both estimators are asymptotically unbiased, and all the gains come from a smaller variance. We find that, for the zero-shot case, the RMSE gain by DSL is equivalent to having 50% more hand-coded documents, and for the five-shot case, the RMSE gain by DSL is equivalent to having 100% more hand-coded documents. Given the significant cost of obtaining hand-coded documents, this shows that computational social scientists can obtain unbiased estimates and valid confidence intervals with much fewer hand-coded documents when they use DSL instead of GSO. We provide a new Figure R-1 (see PDF) based on our experiment. We also provide a new simulation result in Figure R-2 to show that this sample size gain increases as the accuracy of LLMs increases. We will incorporate these new results and clarification in the next revised version. 

In terms of comparison against SSL, we want to first re-emphasize that SSL does not meet the social science priority of bias and coverage. Following the reviewers’ comments, we added several additional variants of SSL methods, and all of them are still biased and have invalid confidence intervals, as our theory suggested. Focusing only on RMSE, we expect that SSL, which is optimized for RMSE, can outperform DSL. However, in our experiment, we showed that DSL, while maintaining unbiasedness and valid confidence intervals, can achieve RMSE comparable to SSL (even though it is often higher than SSL in a finite sample). Importantly, we also show that, as sample size increases, DSL provably dominates SSL in terms of RMSE because bias of SSL does not vanish with sample size, while variance of DSL will vanish with o_p(1/n). One reviewer raised concerns over the strength of the SSL baseline. We have added our SSL baselines, including an estimator by Wang et al. (2020), which, as far as we know, is the only published article that proposes an estimator that covers exactly the same settings as ours (downstream regression analyses). As it requires stronger parametric assumptions, it is, in general, biased and has invalid confidence intervals in our experiments (see Figure R-3 in the PDF). 

**Concerns about the Title**

Multiple reviewers raised concerns about the title because DSL can be applied to any surrogates and is not limited to LLMs. To address this, we are going to change the title to: “**Using Imperfect Surrogates for Downstream Inference: Design-based Semi-supervised Learning for Social Science Applications of Large Language Models.**” We will also emphasize the generality of DSL and downplay the specific LLM angle in the paper. But we hope to keep some of the framings for three reasons. First, LLMs are becoming a huge part of the computational social sciences (e.g., Ziems et al., 2023), and the application to LLMs will be more than 90% of the use cases of our approach. Second, we think the approach works very well with the applied zero/few-shot LLM workflow. You have some LLM annotations, but you have to sample some observations to check that it is working; in doing so, you already build up a gold standard dataset with which you can use DSL to improve performance. Third, we think LLMs will continue to get better, and that will just make the DSL estimator improve even more.

**Additional Changes**

Additional changes are specific to individual reviewers and discussed in those rebuttals. 


Meta-review: The paper proposes a strategy for unbiased estimation of regression coefficients using imperfect surrogate labels.

In the initial reviews there was some confusion, much of which the authors have adequately clarified with good answers. One concern raised, which I agree with, centers on the heavy emphasis the authors put on using large language models as a source of surrogate labels, even if the method works with many other sources of surrogate labels. The authors have committed to reducing this emphasis and updating the title, which I strongly urge them to follow through with.

Overall the method and results presented here are of interest to the community, and I recommend acceptance.
Decision: Accept (poster)