Valid Inference with Imperfect Synthetic Data
Authors: Yewon Byun
Shantanu Gupta
Zachary Chase Lipton
Rachel Leah Childers
Bryan Wilder
Venue: NeurIPS 2025 poster ()
Keywords: Human-AI Collaboration
Statistical Inference
LLMs for Computational Social Sciences
Primary area: evaluation
TL;DR: We provide a principled approach for incorporating fully synthetic samples in downstream statistical inference.

Abstract: Predictions and generations from large language models are increasingly being explored as an aid in limited data regimes, such as in computational social science and human subjects research. 
While prior technical work has mainly explored the potential to use model-predicted labels for unlabeled data in a principled manner, there is increasing interest in using large language models to generate entirely new synthetic samples (e.g., synthetic simulations), such as in responses to surveys. However, it remains unclear 
by what means practitioners can combine such data with real data
and yet produce statistically valid conclusions upon them.
In this paper, we introduce a new estimator based on generalized method of moments, providing a hyperparameter-free solution with strong theoretical guarantees to address this challenge. Intriguingly, we find that interactions between the moment residuals of synthetic data and those of real data (i.e., when they are predictive of each other) can greatly improve estimates of the target parameter.
We validate the finite-sample performance of our estimator across different tasks in computational social science applications, demonstrating large empirical gains.

Valid Inference with Imperfect Synthetic Data
Yewon Byun', Shantanu Gupta', Zachary C. Lipton', Rachel Leah Childers', Bryan Wilder" Machine Learning Department, Carnegie Mellon University' University of Zurich [yewonb, shantang, zlipton, wwilder]@cs cmu. edu, rachel, chileerseiddeesffff uzh ch
Abstract
Predictions and generations from large language models are increasingly being explored as an aid in limited data regimes, such in computational social science and human subjects research. While prior technical work has mainly explored the potential use model-predicted labels for unlabeled data in principled manner, there is increasing interest in using large language models to generate entirely new synthetic samples (e.g., synthetic simulations), such as in responses surveys. However, it remains unclear by what means practitioners can combine such data with real data and yet produce statistically valid conclusions upon them. In this paper, we introduce a new estimator based on generalized method of moments, providing hyperparameter-ffee solution with strong theoretical guarantees address this challenge. Intriguingly, we find that interactions between the moment residuals of synthetic data and those of real data (i.e., when they are predictive each other) can greatly improve estimates of the target parameter. We validate the finite-sample performance of our estimator across different tasks computational social science applications, demonstrating large empirical gains.
Introduction
Practitioners increasingly leverage large language models (LLMs) as cheap but noisy labelers for automating tasks traditionally reliant manual human annotations Ziems et al. [2024). Beyond annotation, recently, practitioners have started to explore the possibility of leveraging LLMs for more
diverse and open open-ended endee forms of model- generaeed data, such as outputting entirely new synthetic
samples, e.g., simulating human responses to surveys or human participants in early pilot studies
(Argyle al. 2023 Brand al. 2023 Dominguez-Omme Olmedo al.) 2024 Anthis al. 2025| Hwang
et al. 2025b Determining the extent to which researchers should integrate LLM simulations whether by simulating samples, combining simulated and real samples, or relying entirely on
human participants remiins an open and task-dependent question. While such pipelines leveraging fully synthetic simulations have yet to be fully realized, reliable mechanisms for aggregating these data sources are indeed what will inform both the feasibility of such design choices and how such pipelines should be implemented in practice.
A persistent challenge, however, is that naively aggregating synthetic samples with real data for downstream inference often leads to greatly biased estimates, compromising the statistical validity of downstream conclusions. Ideally, we would like to realize the benefits of incorporating information from these additional data sources while retaining favorable statistical properties-conse proper asymptotic coverage. We consider the setting where practitioners have access to corpus of unlabeled text and a small
and
comman-annotteed samples with labeled covariates and outcomes. leverage LLMs to (1) predict covariates and outcomes for the unlabeled text samples; and (2) generate new text samples conditioned on available samples and label the covariates and outcomes for them similarly to (1).
Here, practitioners
First of all, is not immediately obvious how even produce synthetic samples such that they can be used in a principled manner. Naively drawing samples from a generative model and treating them
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

as additional samples alongside real data makes it impossible to provide statistical guarantees for the resulting estimate if the generative model does not perfectly match the real distribution, which is expected in practice. We propose specific sampling strategy in which each synthetic sample is generated conditional on an individual real text Section 3] What makes this formulation statistically powerful is that it introduces correlation structure between each real text and synthetic sample. We will leverage this correlation structure, as enables us to more effectively share information across real and synthetic data.
In Section 4 we introduce a new estimation framework based on Generalized Method of Moments (GMM). The GMM framework allows us to incorporate multiple sources of information by adding moments. The optimal weighting in GMM produces combination of these moments that minimizes the variance of all estimators based on these moments [Chamberlain] 1987.. This optimal weighting measures the cross-correlations between the synthetic and real data, producing a weighting matrix that reduces the variance of the real data moments if there is information from the synthetic data moments. Prospectively, it is not intuitive that the incorporation of additional moments based exclusively on synthetic data (defined in terms of a separate parameter from the target parameter) should yield any benefits (or even affect) the estimation of the target parameter of the real data. Intriguingly, we find that the incorporation of synthetic data leads to better estimation and tighter confidence intervals when the synthetic data moment residuals are predictive of the real data moment residuals. When they are independent from each other, the variance reduces to the optimal variance based only on the fully observed data. That is, in the worst case where synthetic data is completely uninformative. including it does not hurt (at least asymptoticallyy. Finally, in Section 5 we analyze the finite-sampi performance of our estimator using real-world datasets that encompass varying computational social science tasks, demonstrating large empirical gains.
At a fundamental level, this work takes a step towards understanding how synthetic data from foundation models can systematically be leveraged to support valid inference. As the usage and future promise of foundation models continue to grow, so too will the complexity of pipelines that incorporate their outputs. Our framework provides a foundation for easily extensible estimation methods that can safely incorporate the growing variety and quality of synthetic data sources from such models. More broadly, this GMM-based estimation framework for incorporating auxiliary data may be of broader interest as an alternative to the predominant debiasing-base methods in the surrogacy literature [Angelopoulos et al.) [023a], it can more flexibly accommodate multiple proxy covariates and proxy outcomes compared to existing approaches.
2 Related Work
Statistical Inference and Debiasing Methods. Our work is broadly related to performing statistical inference with missing data, where past works have explored approaches to yielding valid and efficient parameter estimates [Robins et al., 1994]. Other work has notably explored the usage of ML models to estimate nuisance parameters [Chernozuukov et al. 2018). The most related line of research are debiasing methods that focus on combining ground truth data with surrogate predictions (often produced by a machine learning model) to perform statistical inference [Egami et al. 2023 Gligorić 2024 These frameworks are often referred to as prediction-pon pooeered inference [Angelopoulos et al.
2023a b] in the machine learning literature. Such methods have been well-studied in the context et al. of predicted outcomes and, more recently, predicted covariates [Ji et al. 2025). key difference between these works and our setting is that the primary focus of our work is how to incorporate fully synthetic samples, which remains unaddressed by previous work. For clarity, we refer to samples as fully synthetic when (1) the underlying text is synthetically generated and (2) both its covariates and outcomes are model predictions.
LLMs for Data Annotation and Synthetic Simulation Tasks, Our work is motivated by the increasingly growing use and future promise of LLMs for annotations and simulations (Ziems et al.) 2024 HWang et al., 2025a Anthis et al.) 2025). There has been growing interest in using LLMs in fully synthetic simulation studies, with primary applications in exploratory research or early pilot studies. For instance, recent work has studied simulating social interactions and behaviors [Chen et al. Park et al. 2023). Other works have explored LLMs for simulating survey responses (Dillion al. 2023 Rothschild et al.) 2024| Dominguez-O -Omedoo et al. [024]. J. analyzing how well simulations approximate human responses while cautioning about drawbacks such as limited diversity and lack of context-awareness. In summary, this line of work shows the potential of incorporating synthetic
2

---

data powered through strong generative models in such downstream pipelines but also exhibits clear failure modes and imperfect conclusions from such studies. While most of these works focus on qualitative takeaways and early signals for future experiments, we focus on a forward-lookinn setting of making statistically valid inference given such synthetic samples.
3 Preliminaries
Notation and Setup. We consider parameter estimation task where the goal is to estimate a target parameter 8* Rd. Let X, ~ D denote a random triple drawn from an unknown Gata-generating distribution D over text inputs T E TT, covariates about the text (e.g., structured metadata) X E X C RR, and labels Y E V. For example, T can be texts from online requests, where X are linguistic markers of hedging (i.e., notions of uncertainty) and Y is perceived politeness. Due to labeling budget constraints, we assume we only observe a small fraction of human-annotated data (i.e., ground-truth covariates and labels about the text). Specifically, we have access to labeled dataset Diabeled = XX,,, X=) that is sampled i.d.d from D and an unlabeled corpus of text Dunlabeled = {(T))) sampled i.i.d from Dr (i.e., the marginal distribution over T), where m > n. To supplement this limited supervision, we leverage machine learning models in the following two ways.
Proxy Covariates and Labels. We use machine learning model f to produce predictions (Tx(T)), fy (T for the available set of input texts T E TT Here, fx and fy denote the same ma-
chine learning model, using separate prompts for the target outcome (either a covariate X or outcome
Y) (see Appendix E for details). This yields the following Prroay { fx fy (T,))
fx fy (T, M For simplicity, we will refer to these as proxy samples and denote
them as (T, X, Y). We will refer to the distribution over proxy samples D. This is the main setting generally considered in the surrogacy literature (restricted to predicted outcomes).
Synthetic Covariates and Labels. We propose new data augmentation process which generates new samples using the same machine learning model f (employing it as generative model, instead of a classifier). Specifically, our method conditions the generation process on each individual text T3 as an example and asks the model to generate new synthetic sample given that context. Formally, for each i, we sample new text T;, conditioned on (Ti, if the sample is labeled and (T), X,) if the sample is unlabeled (since X, is not available, by definition). More concretely, Th ~ P(- T., X2) if labeled and T ~ TP( Tj, X,) if unlabeled. See Appendix for synthetic data generation. Based on the generated sample, which we denote as Ti, we then extract its corresponding covariates and outcomes using I similarly as in proxy samples. More concretely, X2 ~ P.(1 Tx) and Y2 P(- Ts). This results in Psymthetic- = [(T),, Xx. )x))
E for prompts used
We refer to the
distribution over synthetic samples (T, X, Y) as D.
This specific sampling process has two motivations. First, from a machine learning perspective it can be seen as form of in-context prompting, where the model is given an example from the dataset in order to align it more closely with the task. Iteratively prompting with different samples Ti is also likely to produce more diverse samples than asking for many samples with the same prompt. Second, from statistical perspective, it introduces correlation structure between each real text T; and synthetic sample T,.
Finally, we introduce some notation that combines all of these data sources into draws from a single joint distribution. Specifically, we introduce a new random variable E {0, which is an indicator for whether T is labeled (1) or unlabeled (0). Then, we view the complete generative process as draws (T, s, s X, s Y,X XXYY...... !..... Y-M) for M different kinds of auxiliary data. So far, we have discussed two kinds, proxy and synthetic, that we employ empirically (M = 2), but our methods are fully extensible to additional kinds of auxiliary data. For example, we could include samples from multiple different generative models. The real (X, Y) are observed only for labeled points with
= 1 while the auxiliary data available for all samples. The joint distribution over this full tuple induced by the composition of the generative processes for the components described above.
3

---

Combining Synthetic Information via Generalized Method of Moments
To estimate the target parameter 8", we adopt a generalized method of moments (GMM) approach [Hansen [982] that combines information from the different types of data in the following manner.
4.1
Moment Conditions
Our framework is applicable whenever the target parameter can be identified by set of moment conditions, functions whose expectation should be zero at the true value the parameter. Moment- based estimation is a broad and flexible framework that includes almost all commonly used statistical frameworks (e.g., maximum likelihood, generalized linear models, instrumental variables, etc). We begin by defining the moment conditions that identify 0* under the distribution of interest (i.e., the real-data distribution D). the following section, we introduce how this can be adapted to incorporate surrogate data (i.e., proxy and synthetic data).
Formally, we consider the problem of estimating a parameter 0 E Rd. The true value 8" identified as the solution to a set of p 2 d moment conditions
Elgi(0 (0*) = 00 f=
where the ()(() are continuously differentiable functions gd R. For example, in maximum likelihood model, we would have one vi for the derivative of the log-likelihood with respect to each parameter, and the moment conditions enforce that H" satisfies the first-order conditions for maximizing the likelihood. Let (b(0) = (0).)))))))) denote column vector stacking the moments.
4.2 Constructing Our G.M Estimator
To leverage the auxiliary data (i.e., proxy data and synthetic data) in making our GMM estimator more efficient, we can construct a set of auxiliary moments for each additional source of data. We estimate additional set of auxiliary parameters "1,,,,, set new auxiliary data. In the specific instantiation of the model that we use here, we always have M = 2 (proxy and synthetic data), but in principle method is extensible many sources of auxiliary data, for example synthetic samples generated from several different models. Roughly, each new parameter vector 17 be understood the parameter that would estimate using each auxiliary data source, and our augmented model will automatically determine how to use these auxiliary estimates inform the estimate of the parameter of interest 0.
RP,
parameter vector for each
For each new parameter vector Tk. we introduce a corresponding set new moments to estimate this parameter and allow its estimate inform the estimate of 0. Specifically, " two new blocks moments that are copies of the original moments for Intuitively, one block of moments will be evaluated only on the real (labeled) data, while the other will be taken on the pooled set of labeled data and auxiliary dataset i. The pooled-data moment will allow us to improve the estimation of π) using the larger sample. The version evaluated only the real data will allow GMM to evaluate how well the moments for the auxiliary parameter correlate with those of the true parameter on the same data, and share information across them if the auxiliary moments informative (as we would expect if the generated data is high quality).
introduce for each
Formally, let S, E IRP stack copies of the indicator variable for whether a data point is labeled. block matrix notation, the combined model takes the form of the augmented moments
S²
((n)
ge (<(0.))
S
((as))
E RR+2MM
(I)
w(m)
((MM)

---

We will then jointly estimate (f) as the solution to the moment condition (i 7)] = 0. For clarity, we refer our estimator that uses real and proxy data (M as GMM--Prox and our estimator that uses real, proxy, and synthetic data (M = 2) as GMM-Synth throughout the paper. We remark that since the parameter of interest 0 appears only in its original set of moments, which are evaluated only on the labeled data, this new moment condition still identifies the target parameter 0" However, as we discuss below, when we apply standard methods for efficiently estimating the augmented GMM, the new moment conditions will be leveraged to reduce the variance of the estimate without compromising consistency or asymptotic normality.
Before turning to estimation, we provide concrete example of our moment construction for the case of generalized linear models (GLMs) in wwo-dimensions.
4.3 Example 1. Generalized Linear Models
Recall that the standard GLM formulation optimizes the objective function,
(0 v) yyx T f(xT0),
where f is function that is convex and smooth. We remark that this recovers the setting of logistic regression when f(=) log(1 + exp(=)). Let assume two-dimensional setting for illustration.
This translates to the population moment conditions
0f(XT0") =
af(XTB*)
X,Y
0, E X
001
002
We have similar moments for proxy and synthetic data, where we use parameters which are also two-dimensional. Within GMM framework, construct the following set of moment conditions across the observed, proxy, and synthetic data.
(7)()),((2)))
X Y. 0f(X7 0)
80 X,,Y,
8t
0f(X 8) OB
81
8,,
0f(X; nj (1)))
81
X,
0f(8; (1)
8t
anh
8,,
0f(x; (2)
St
anp
9t (0, m)
81
8,24
0f(X; n (2)
8,1
85 m)
am
1
812 0f (1)
Omh
1
N, 81
a (2)
an (2)
1
X,2Yr Y
as(X; 1) (2)
amh (2)
4.4
GMM Estimation
Given augmented moment conditions g. estimate the parameters (0, m)) by minimizing the GMM objective:
θr. fr
arg
min BE@.TER?MP
@r(m.)
(2)
where
T gr
@r(0,)) =
Σ ge (0,
WT
(0, 1))
(3)
T
[=1
5

---

Two-step GMM estimator. We adopt the two-step GMM procedure as described in Newey and
McFadden 1994]. First, we compute the one-step estimator 0 1/r using an identity weight matrix WT = I. Then, we estimate the optimal weight matrix as:
T (
(0 m1 =
L
(6)) 12 Dge (0() 1/ (
(4)
T
and set
Per (08) ))=0)))
Wr
(5)
This optimal weighting has the interpretation as the inverse empirical covariance of the moment conditions on the one-step estimate. We then compute the final two-step estimator by minimizing (r()) with this updated weighting matrix. This choice of WT yields an asymptotically efficient estimator under standard GMM regularity conditions. Following Chamberlain (1877). this choice of weighting minimizes the semiparametric efficiency bound with respect the emmipparaettric model defined by these moments (see Appendix B for more details).
The adoption of two-step GMM is critical component of proposed estimation framework. Indeed, in the first-step estimates, the synthetic and proxy data will have no impact on the estimate of 0 because they never appear in the moment conditions concerning 0. In the second stage though, the weight matrix WT accounts for the covariance between moment conditions, where off off-diagonal terms in the matrix allow moments for the auxiliary data sources to influence the estimation of O.
4.5 Consistency and Asymptotic Inference
We now present results on the consistency and asymptotic behavior of our GMM estimators. Proposition! 1. Our estimate &r (as defined in Equation 2 is consistent and asymptotically normal. It converges in distribution
VT (0', (00))N0 w(0,
where the covariance V is given by
V=
(( n) ( (GG(.))
and where G(O, γi) is the Jacobian of the population moments at the ground truth parameter values 0, 11 and F is the asymptolic variance of the sample moments.
For optimal weight matrix in Equation 5 this simplifies to = V= n)) These are standard results on GMM estimators, which follow by straightforwardly applying the results in Hansen 1982]. We remark that these asymptotic results require a set of conditions on the sample moments, which are slightly nuanced in this setting with multiple sources of information. We discuss these conditions and prove that they are satisfied Appendix[A] for the setting of proxy and synthetic samples. Given this asymptotic behavior, we can derive valid confidence intervals for our parameter estimates.
4.6
Why does synthetic data improve performance?
To understand where the benefits arise from incorporating the proxy and synthetic data into our GMM estimator, we analyze the interaction between our moment conditions. Note that the functions e are often referred to as "residuals" in the GMM literature; since (0) should be zero in-expectation, deviations from zero are interpretable as a kind of residual. The key intuition is that synthetic data will improve performance when the synthetic-data residuals are predictive of the real-data residuals.
First, we note that the synthetic data were perfectly simulated, X and Y would be perfectly recovered from the unlabeled text T. With ground truth X, Y, we can perfectly recover the residual terms. In settings where we have good but imperfect simulations, X. and X, ý are highly correlated with the errors the true data, and we can approximately estimate the real-data residuals with the synthetic data. Within our GMM- approach, this is all handled implicitly in our two-step

---

estimation procedure. During the first estimation step, each set of parameters (e.g., defined on the observed, proxy, and synthetic data) is independently identified since the initial weighting is an identity matrix. The key insight is that, during the second estimation step, the weighting matrix W which is the inverse of the moment covariance matrix, captures the interactions between the observed residual terms and the residuals from the synthetic data in our GMM objective. This is captured in the off-diagonal terms of the moment covariance matrix. Partitioning the moments into observed data residuals me (0) and synthetic data residuals (1)),, we derive an explicit formula for the asymptotic variance of (Tr 0) in Theorem with the full proof in Appendix D
Theorem 1. The asymptotic variance of ( -0) 0) is given by
dR (0)) d0' A
d[(m((0]] cF[[((0]] dEAh(n)),
dR(h(n) dry'
di
[1(mm]]] dr
dB(h(n)] dn'
d[((m(0]
B
y-1
B
d0
d0'
dn
d0
with A, B, D, m(), h(n) defined in Appendix
We find two important conclusions. First, when these residuals are independent of the observed data, the formula reduces the optimal variance based only on the fully observed data. That is, in the worst case where synthetic data is completely uninformative, including it does not hurt (at least asymptotically). Second, when the real and synthetic residuals are correlated (as we would hope), we derive a lower bound on the variance which is proportional to the residual variance in regression of the observed data residuals on the span of the synthetic data residuals. This bound minimized by choosing moments that span the conditional expectation of the observed data residuals given T; sufficient condition for which is that the conditional distribution of X, Y or X, Y given T equals the conditional distribution of X, Y.
5
Experimental Results
5.1
Baselines
Existing methods in the literature are well-studied in the context of predicted outcomes and more recently, predicted covariates. However, remains unclear how to aggregate information from fully synthetic data. We consider how to adapt existing debiasing methods; PP1++ (Angelopoulos et al. 2023b] and recent variants [Ji ct al. 2025].
The most general approach is perhaps given by RePPI [Ji et al., 2025 which predicts the optimal loss through fitting an arbitrary model that maps the proxy and synthetic loss to the real loss. This results in the objective defined in Proposition 2 The resulting estimate retains asymptotic normality conditions (see Appendix E.2. for the proof and algorithm details).
RePPI
PPI++Proxy, PPI++Synth To adapt PPI++ [Angelopoulos et al.) 2023b) to this setting, we take an instantiation of RePPI, where the model is convex combination to limit the number of parameters. This results in the following objective defined in Proposition 3 We refer to the estimator with a = as PPI++Proxx, as the synthetic terms vanish, yielding data. We refer to the estimator with tunable a E [o, 1] as PPI++Synth, which combines real. proxy, and synthetic data. Note that the addition of this hyperparameter a adds increased complexity, and techniques such as cross-fitting must be used to select it in a statistically valid fashion. The resulting estimate retains asymptotic normality conditions (see Appendix E.2.2 for the proof and algorithm details).
estimator that combines real and proxy
PPI++Synth (Oracle) As an upper bound, we conduct a grid search over different possible a values without cross- cross-fitting. Note, this is not a valid solution in the setup, as it requires peeking in hyperparameter selection, but it provides an oracle version of the baseline for reference.
5.2 Experimental Setup
Datasets. We validate the finite-sample performance of our estimator for logistic regression and ordinary least squares (OI.S regression on the following computational social science tasks: First, use online requests posted on Stack Exchange and Wikipedia Daanssu-NNeuulesu--iizzzz et al.
7

---

1.0
0.225
1.8
Empirical MSE
0.200
Prt aitrayy
Human Only
Interval Width
0.175
PPt eynth la-- 0.8) RoFP)
0.150
2.4
0.125
0.100
1.2
0.075
0.06
0.07
0.08
0.09
0.10
0.06
0.07
0.08
0.09
0.10
0.06
0.07
0.08
0.09
0.10
Fraction Labeled
Fraction Labeled Data
Fraction Labeled
0.40
1.0
2.4
0.35
SyMth tours) Haman-Orly
2.2
Emeinical ISSE
PPta-Proay
Interval Witt
P/14 P754-Synth ta 0.50 RePP)
1.0
0.20
0.15
1.4
1.2
0.01
0.02
0.03
3.04
0.05
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0,04
0.05
ction of Labeled Data
Fraction of Labeled Data
Fraction Labeled Data
1.0
2.2
0.250
Synth
Empir MSE
0.225
PMt oitroxy
huamen-Srlly
Interval Width
2.0
0.200
Py--Syythh lam 0.8) IhePPi)
0.150
0.100
1.2
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07
0.08
0.09
0.10
Fraction Labeled
Fraction of Labeled Data
Fraction Labeled Data
0.8
1.0
0.7
Synth
Hummnn-Orly Only
Emeinical MSE
P
Interval Width
0.5
PP Synth (o 0.81
RePPI
0.4
0.3
0.1
2
0.006 0.088 0.010 0.012 0.014
0.006 0.008 0.010 0,012 0.014 Fraction Labeled
0.006 0.008 0.010 0.012 0.014 Fraction
Fraction Labeled
Figure 1: Main Results (Logistic regression). We observe large reductions in MSE, especially in very low-label regimes. Each row corresponds a task (i.e., 1pp. Hedging, Stance, Congressional Bills Data (from top to bottom)); each column corresponds to metric (i.e., MSE, coverage, confidence interval width (from left right)). Note that we report the PPI++Synth oracle number for PPI++Synth (see Figure
| for PPI++Synth with cross cross-fitting results). When the best performing PPI++Synth is equivalent to PPI++Proxx (i.e,, a 1), we report the second-best performing PPI++Synth method. See Figure in Appendix F for full grid-search results over different values. Results are averaged over 200 trials.
( in Appendix
2013b] to estimate how the presence of hedging markers (i.c., expressions of uncertainty) affect perceived politeness. Second, we use the same dataset estimate how the usage of first-person plural pronouns affect perceived politeness. Third, we use corpus of climate-related news headlines [Hmielowski et al. [014]] estimate the effect of affirming linguistic devices on media stance toward global warming (i.e., whether the news headline supports or rejects climate change). Lastly, we use congressional bills texts [Adler and Wilkerson| [01 to estimate the effect of a legislator's DW-Nominate measure [Lewis et al.) 2024] of ideology on the type of bill (whether the bill pertains to macroeconomy). In all tasks, the target parameter the regression coefficient corresponding the explanatory variable of interest.

---

1.0
0.014
Synth
Empiiiial MSE
Humbn Only
Interval Width
0.012
P/t oitrony
PPti Syrth (a- 0.8)
0.40
0.010
0.35
0.008
0.006
0.30
0.004
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07
0.08
0.09
0.10
Fraction Labeled
Fraction Labeled Data
Fraction Labeled Data
0.0200
GMM-Syytt lours)
Empirical ISE
0.0175
Human-nnny PPtayy
Interval Width
0.50
0.0150
PPy--Synth la- 0.8)
Coverage
0.45
0.0125
ReFFt
0.0100
0.35
0.0050
0.25
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
Fra coctioon of Labeled Data
Fraction of Labeled Data
Fraction of Labeled Data
1.0
Emplrcca calirccall MSE
0.12
Haman-Selly
PPi+-Prox
Interval Width
P7t4-Synth torm 0.8)
0.08
ReiP)
0.7
0.04
0.5
0.4
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
fraction of Labeled Data
Fraction of Labeled Data
Fraction Labeled Data
GHM-Syyth Synth Only
2.5
0.035
Empirical MSE
0.030
PRil Proxy
Interval Width
2.0
1911-4 Sym 0.20
0.025
RaPP1
3.5
0.020
0.015
0.010
0.005
0.006 0.000 0.010 0.012 0.014
0.006 0.008 0.010 0.012 0.014 Fraction of Labeled Data
0.006 0.008 0.010 0.012 0.014 Fraction Labeled
Fraction Labeled
Figure 2: Main Results (OLS). We again observe large reductions in MSE, especially in very low-label regimes. Each row corresponds a task (i.e., lpp. Hedging, Stance, Congressional Bills Data (from top to bottom)); each column corresponds to a metric (i.e., MSE, coverage, confidence interval width (from left right)). Note that report the PPI++Synth oracle number for PPI++Synth (see Figure
| for PPI++Synth with cross cooss-fitting results). When the best performing PPI++Synth is equivalent to PPI++Proxy (i.e.,
9 in Appendix
1), we report the second-best performing PPI++4ynth method. See Figure in Appendix F for full grid-search results over different a values. Results are averaged over 200 trials.
Models and Metrics. We use GPT-4o Hurst et al. 2024] without any task-specific fine-tuning generate both proxy and synthetic data. We also include additional results, using open-source, worse quality models (i.e., Llama-3-8b and Qwen-3-8b. in Appendix F (Figures 10
11
12 13).
We evaluate our method's s performance against the adapted baselines discussed in Section usinn four key metrics: empirical mean- -quured error (MSE), coverage, confidence interval width, and effective sample size. The effective sample size represents the number of human-labeled samples that a classical estimator would require to achieve the same MSE as our method's 's estimate. In other words, this metric quantifies how many human annotations the method effectively saves while maintaining equivalent mean squared error.

---

5.3 Results
The results for our method's performance shown in Figures I and 2 We will highlight some key observations. First, we observe that GMM-Synth achieves the lowest MSE, outperforming all baselines 8 out of 8 downstream tasks. Notably, performance gains (of more than 50% reductions in MSE) are most pronounced when the fraction of labeled data is small, precisely the setting where the need for synthetic data is best motivated. Crucially, this does not come at loss of validity of the parameter estimates; GMM-Synie retains valid coverage and results in tighter confidence intervals in 7 out of 8 downstream tasks. In Figures 4 and [5 (in Appendix (F), we further observe that our method substantially improves effective sample size in data-limited settings. In other words, our method effectively saves large amounts of human annotations (up to more than 50%) while maintaining equivalent mean squared error. Second, GMM-Synth consistently exhibits gains over GMM-Prox across all considered tasks. This demonstrates that synthetic data provides additional benefits beyond those of proxy-labeled data, and that our method effectively integrates these multiple sources, retaining their respective benefits. In other words, it shows how much additional benefit there is for the practitioner to not only use the model to label unlabeled samples but also use it to generate entirely new unlabeled samples to aid in statistical inference. Third, interestingly, unlike the additional gains demonstrated in GMM- compared to GMM-Proxy, we observe that incorporating synthetic data via PPI+++ynthh (compared to PPI++Proxy lead to benefits that are much less pronounced in 3 tasks, and, in fact, result in no gains in MSE in the remaining 5 tasks. This empirically demonstrates that our method is able to incorporate synthetic data much more effectively when compared to adaptations of existing debiasing methods in the literature. Importantly, we note that across all settings, using the proxy data and synthetic data alone yields greatly biased estimates (see Figure 3 in Appendix F.) Further, we note that the same conclusions hold with even worse quality synthetic data generations from weaker language models; see Appendix for results on open-source models such LLaMA [Grattafiori al., 2024 or Qwen YYng et al. 2025
[5
6 Discussion
How synthetic data pipelines should be designed and implemented in practice hinges on reliable mechanisms for integrating information from them. In this work, we introduce a principled framework for reliably incorporating fully synthetic samples into downstream statistical analyses. We provide practical guidance for constructing synthetic samples from text-based foundation models ways that support valid inference, and propose new estimator based on generalized method of moments (GMM) estimation, where the key intuition is that synthetic data will improve performance when the synthetic-data residuals are predictive of the real-data residuals. Across the studied inferential tasks, we indeed observe a large degree of improvement in estimation, especially in very low-label regimes. More broadly, this work takes a first step toward understanding how imperfect synthetic data from foundation models can systematically be leveraged to support valid inference and to make reliable downstream conclusions. With the increased adoption and growing capabilities of foundation models, pipelines that incorporate their outputs will only become more complex. Our method provides an easily extensible estimation framework that can safely integrate the increasing variety and quality of synthetic data sources.
Limitations and future directions. A potential limitation of our framework is its reliance on the quality of the generative model (e.g., an LLM), as expected. As with other debiasing approaches, very poor-quality synthetic data would yield little-to-no benefits in statistical efficiency. Moreover, our theoretical guarantees, like those of debiasing methods, hold asymptotically and thus may fail to hold in extremely low-data regimes, potentially leading to undercoverage of the target parameter.
Acknowledgments and Disclosure of Funding
We thank Michael Oberst and Gati Aher for helpful discussions in developing this work. We also thank Santiago Cortts--oomer for thoughtful comments on the manuscript. This work is supported by the AI Research Institutes Program funded by the National Science Foundation under Al Institute for Societal Decision Making (AI-SDM), Award No. 2229881.
10

---

References
Scott Adler and John Wilkerson. Congressional bills project. NSF 00880066 and, 880061, 2011.
Anastasios N Angelopoulos, Stephen Bates, Clara Fannjiang. Michael Jordan, and Tijana Zrnic. Prediction-powered inference. Science, 382(6671):669-666 2023a.
Anastasios N Angelopoulos, John C Duchi, and Tijana Zmic. Ppi++: Efficient prediction-powers inference. arXiv preprint arXiv:231 2023b.
Jacy Reese Anthis, Ryan Liu, Sean M Richardson, Austin Kozlowski, Bernard Koch, James Evans, Erik Brynjolfsson, and Michael Bernstein. Lim social simulations are promising research method. arXiv preprint arXiv:2504.02234 2025.
Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):37-331, 2023.
James Brand, Ayelet Israeli, and Donald Ngwe. Using Ilms for market research. Harvard Business School Marketing Unit Working Paper, (23-062), 2023.
Gary Chamberlain. Asymptotic efficiency in estimation with conditional moment restrictions. Journal of econometrics, 34(3):305-334, 1987.
Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan Yang, Tinghui Zhu, et al. From persona to personalization: A survey on role-playing language agents. Transactions on Machine Learning Research.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whit- ney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters: Double/debiased machine learning. The Econometrics Journal, 21(1). 2018.
Cristian Ranescu-Niiculesuu-Me Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. A computational approach to politeness with application to social factors. In Hinrich Schuetze, Pascale Fung, and Massimo Poesio, editors, Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 250-259, Sofia, Bulgaria, August 2013a. Association for Computational Linguisties. URL https:/7
aclanthology.org/PPP - 1025/
Cristian Dunescu-Nicuulssu-Miin Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. A computational approach to politeness with application to social factors. arXiv preprins arXiv: 1306.6078, 2013b.
Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. Can ai language models replace human participants? Trends in Cognitive Sciences, 27(7):977-600, 2023.
Ricardo Dominguez-Olmedo. Moritz Hardt, and Celestine Wendler-Dünnee Questioning the survey responses of large language models. Advances in Neural Information Processing Systems, 37:
45850-45878. 2024.
Naoki Egami, Musashi Hinck, Brandon Stewart, and Hanying Wei. Using imperfect surrogates for downstream inference: Design-based supervised learning for social science applications of large language models. Advances in Neural Information Processing Systems, 36:68589-6860) 2023.
Kristina Gligorić, Tijana Zance, Cinoo Lee, Emmanuel J Candès, and Dan Jurafsky. Can unconfident Ilm annotations be used for confident conclusions? arXiv preprint arXiv:2408, 15204, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinay Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama herd of models. arXiv preprint arXiv:2407.21788 2024.
Lars Peter Hansen. Large sample properties of generalized method of moments estimators. Econo- metrica: Journal of the econometric society, pages 1029-1054, 1982.

---

Jay D Hmielowski, Lauren Feldman, Teresa A Myers, Anthony Leiserowitz, and Edward Maibach. An attack on science? media use, trust in scientists, and perceptions of global warming. Public Understanding of Science, 23(7):866 883, 2014.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt- system card. arXiv preprint arXiv:2410.21277 2024.
Angel Hsing- Hwang, Michael S. Bernstein, S. Shyam Sundar, Renwen Zhang, Manoel Horta Ribeiro, Yingdan Lu, Serina Chang, Tongshuang Wu, Aimei Yang, Dmitri Williams, Joon Sung Park, Katherine Ognyanova, Ziang Xiao, Aaron Shaw, and David A. Shamma. Hu- man subjects research in the age of generative ai: Opportunities and challenges of applying lIm-simulated data hci studies. In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, CHI EA '25, New York, NY, USA, 2025a, Associa- tion for Computing Machinery. ISBN 97984007 13958. doi: 10.1 (0.1144/377659993776222 URL https://doi. org/10. 10. 01445//776699937166680
Angel Hsing-Chi Hwang, Michael S Bernstein, S Shyam Sundar, Renwen Zhang, Manoel Horta Ríbeiro, Yingdan Lu, Serina Chang, Tongshuang Wu, Aimei Yang. Dmitri Williams, et al. Human subjects research in the age of generative ai: Opportunities and challenges of applying llm-simulated data to hci studies. In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1-7, 2025b.
Wenlong Ji, Lihua Lei, and Tijana Zmic. Predictions surrogates: Revisiting surrogate outcomes in the age of ai. arXiv preprint arXiv:2501.0973 2025.
Jeffrey B Lewis, Keith Poole, Howard Rosenthal, Adam Boche, Aaron Rudkin, and Luke Sonnet. Congressional roll-call votes database. Published Online, 2024.
Yiwei Luo, Dallas Card, and Dan Jurafsky. Detecting stance in media on global warming. In Trevor Cohn, Yulan He, and Yang Liu, editors, Findings of the Association for Computa- tional Linguisties: EMNLP 2020, pages 3296-3315, Online, November 2020. Association for Computational Linguisties. doi: 10. 10.51//3vv1/2 1/202/0 222220.inninggsemmmp.99999 //aclanthology. - org/ 2020. w.w.2...iiniing... findings-nniiiiinn 2966/
URL https
Whitney K Newey and Daniel McFadden. Large sample estimation and hypothesis testing. Handbook of econometrics, 4:21 11 2245, 1994.
Joon Sung Park, Joseph O' Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1-22, 2023.
James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association, 89(477): 846-866,17 1994.
David M Rothschild, James Brand, Hope Schroeder, and Jenny Wang. Opportunities and risks of 1lms in survey research. Available SSRN, 2024.
Aad W Van Der Vaart and Jon A Wellner. Weak convergence and empirical processes: with applications to statistics. Springer.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388 2025.
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. Can large language models transform computational social science? Computational Linguistics, 50(1): 237-291, 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All claims supported by theoretical and/or empirical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We provide a discussion on the limitations of our work in separate Limitations paragraph.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

Justification:
We provide and discuss all assumptions in our theoretical results, and provide
proofs in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All details required for replicating our experiments are provided the Experi- mental Results section and in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, code and sufficient instructions to reproduce results are provided in the Supplementary Material, All data used is publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Yes, all details on the experimental setting are provided the code in the Supplementary Material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Yes, results are properly evaluated for statistical significance. We report runs averaged over 200 different random seeds and measure the statistical validity of our estimates.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources are detailed in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: :/tp::////euureee hww.npeu...../. hwtpripp..cpppbli///ehie... hccppbllic/ttthccccieeellllll
Answer: [Yes]
Justification:
The research in this paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the broader socictal impact of our work in both the Introduction and the Discussion section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
16

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper proposes no such risk - no data or models are released.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Licenses are properly referenced in the Appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Well-documented code is provided in the Supplementary Material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: The usage of LLMs (in generating synthetic data) are clearly described and outlined in the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
19

---

A Conditions for Consistency and Asymptotic Normality
We provide a discussion about the necessary conditions for a GMM estimator to be consistent and asymptotically normal, showing that these conditions are indeed met for our augmented GMM.
As mentioned in the construction of our estimator, define moment condition for each parameter on the observed data D. We also define two moments for each parameter on the proxy and synthetic data. This leads to overidentified system, with more moments than parameters, ensuring that the target parameter is identifiable.
Next, we establish few conditions for valid asymptotic properties of our GMM estimator, specifically about the convergence and distributions of the stacked vector of the sample moments ge .7 at its optimum, which we will refer as Jr for brevity. First, require that this vector of moments converges to its expectation, or that
1
9t E[gi].
where N = + m is our total amount of data. Next, all moments must jointly respect the central limit theorem, or that
VN
Σ 91 4 N(,
N
where F is some finite covariance matrix of all the moments
Under these standard regularity conditions on the moment vector Or Newey and Mcradden, 1994), these conditions are immediately satisfied for the moments defined on observed data, as each observation of the moments is independent. The same holds for the moments defined on proxy data, since X, y are functions of independent inputs T, and therefore also independent across observations. The case of synthetic data is slightly more nuanced, but we show that the required conditions still hold, through the following lemma.
Lemma 1. Let (6z 3 represent observations of the subset of moments corresponding to parameters of the synthetic data, and assume E| ,6
E|( 112 < cc. Then, they i.i.d., and consequently
N
N
N jj +
Σ
Bio,] and VN
Øj
4 E(()
N
1=1
j=1
where (d;; is the covariance matrix of Φ,-
Proof. We begin by noting that texts (Ti} are drawn i.i.d. from the marginal distribution Dr- For each T,. synthetic text Tý is produced by generative model (i.e., by LLM), which uses independent randomness for each call. The model conditioned only on an individual sample (T,, X2) j is labeled or (T), X2) otherwise. Since the generative process for each Tí is independent and the mapping T + (X,, YJ) is applied identically to each sample, the resulting pairs (X,, Yÿ) are also i.i.d. As these pairs are drawn i.i.d., then the conditions are met via the central limit theorem. O
This result shows that the required conditions on the sample moments hold in our setting of proxy and synthetic samples; under the regularity conditions of Newey and McFadden (1994) Theorem 3.2, one immediately obtains Proposition on the asymptotic behavior of our GMM estimator.
B Asymptotic Efficiency
From Chamberlain 1987]. we know that the lower bound on asymptotic variance among all regular estimators based on the moment restrictions is precisely the asymptotic variance that can be achieved the general case by using the GMM estimator. Specifically, achieves semiparametric efficiency bound-the smallest possible asymptotic variance attainable by any regular estimator using these moment conditions. This corresponds to the local asymptotic minimax risk over all statistical models such that these moment equalities hold, in the sense that for any E gd
20

---

lim inf N N000 ( (0) .D)
sup
EDI[ N 0)] 2 (G .1) ""
where r(0", D) is any local neighborhood of true parameter 0 and data-generating distribution D satisfying the moment conditions and regularity conditions C₁ in Chamberlain [1987] for parameter 0.
Proof. Follows directly from Chamberlain 1987 Theorem 2.
In words, this indicates that no estimator can achieve lower asymptotic variance than GMM uniformly over all local distributions satisfying the same moment conditions.
C Moment Conditions
We provide concrete example of our moment construction for the case of generalized linear models (GLMs) in two-dimensions.
C.1 Example 1. Generalized Linear Models
Recall that the standard GLM formulation optimizes the objective function,
Co y) yyx + f(x*0),
where f is function that is convex and smooth. We remark that this recovers the setting of logistic regression when f(z) log(1 + exp(2)). Let assume two-dimensional setting for illustration. This translates to the population moment conditions of
0f(x"0) O.
0f(x"0*) 0
E XIY
E X2Y
00,
002
We have similar moments for proxy and synthetic data, where use parameters 17 = (p)())))))))) which are also two-dimensional. Within our GMM framework, we construct the following set of moment conditions across the observed, proxy, and synthetic data.
X 0f(XT 0)
89
8t
X 0f(X. 0) Ob
81
8,,
0f(8; 1j (1))
81
on
X,2X 0f(8; (1)
81
Omh
8,,
Df(X!1)
9t
anp (2)
9e (0, m)
81
8128
0f(X? (2))
81 ())
an
1
as ())
Omh
1
N, V
as(X) (1)
X1 Y,
0s(X) (2))
Omh
D Partitioned GMM Asymptotic Variance
We now derive the asymptotic variance of our GMM estimator for specifically the target parameter
θr.
21

---

Theorem 1. The asymptotic variance of VT 8) is given by
[0)] d0'
dB(m((0]
dEim((0]
dB(h(m))
dE(M(n) dry' B
dB((((0)]
A
do' B
D
y-
))
d0
dn
dry'
dn
d0
with A, B, D, m(0), h(n) defined in Appendix
Proof. With the optimal choice of weight matrix for the full GMM estimation problem, the asymptotic variance of the vector (6, :) converges to (GT F-1G)-1 To obtain the variance for specifically, partition the moments into (e n) (m,(0) (n))),, where m,(0) = Se @ (0)), and
St (1)
Se
he(n)
Se
Given this partitioning, we can express
G(O,
dry
E (0),(0) (lhe(n)m,(())
(0),()) (7)4(1))
F
By the partitioned inverse formula, we can express
F-1 as
A B B D
where the upper left block A is
(m) - ([mm(()),(()]IEll
This term be interpreted the inverse of the asymptotic residual variance of regression of m, (0) on the span of the vector (T()).
The lower right block D is, symmetrically, the inverse of the asymptotic residual variance of regression of hx(r) on the span of the vector (0):
( (00m0 (0)))))))))))
Finally, the off- off-diagonal term multiplies A by the coefficient in a regression of m on h:
= AB
For the full variance,
dF(mm(0)
dR (m)
all aF(m(0)
B d (P(h(n)
GFF-G==
dbl
dg
dr
dB [(]]l
B
dB
(8))
de h(r)) JB h(())
D
dry
dry
dn
Applying the partitioned inverse formula again, the upper left block of (GT F-1 exactly the asymptotic variance of VT 0), equal
which gives
22

---

dl [m] d0'
dE
dB [0)
dE(k(())
dB dB(k(())
dB(h(n)) dE dE(A(n) dry'
di
dE( d0
)-
d0
do'
dr
dry'
dr
This can be interpreted similarly the inverse of the asymptotic variance of the residual prediction error from regression of A A-1/2 dm)()) onto the span of weighted linear combination of terms in dhin)
d0
dy
This formula can be used to derive several properties of the procedure.
Corollary 1. When the moments corresponding to real and simulated observations are uncorrelated, Cov(m)(() hx)n) = 0, the asymptotic variance of (T 8) equals the asymptotic variance of the GMM estimator only using the moments my corresponding to real data.
Proof. When (lm, 0. A of F-) equals corresponding to only using moments m by the claim of Appendix
and B 0, and so the upper left block (E(m, (0) (0)))) -1)))((1))) dE[m(0) -11 which the optimal variance
(0)
de'
L
Corollary The asymptotic variance of VT 0) is lower bounded in the positive semi-definite -1, which is minimized when A is maximized.
order by [B]m(0)]
20 d9
Proof. Let M, be positive semi-definite matrices such that M (B)()) diz m(
N
Then N
(M - N) = M-1 Apply this fact to M
d9
de
and
=
dB(m gddihlm) delkin) d3) dE[k(a)) BT do dr day' because they are symmetric quadratic forms.
dB(m(0) which are both positive semi-definite de
dy
dry'
by reverse monotonicity of the inverse in p.s.d. order, larger A corresponds smaller lower bound on the variance.
For any p.s.d. A1. A2, A1 2 Ag. ( [[ A d0
m( )-1 <
dE mm())
A2
de'
de
d9
Among choices moment functions hy (1) that depend solely on Tr, maximized the positive semi-definite order when the span of (T) contains E[m( A sufficient but not necessary condition for this is that for some j 1. 1...M, the conditional moments of the simulation are identical to those of the real data:
Elv(n)TTi = TT)
This calibration condition is satisfied when the conditional distribution of the simulated data given equals that of the real data, which is a natural simulation target, though not required for valid inference.
E Experimental Details
E.1 Resources and Licensing Details
Compute Details Each experiment run on a A6000 GPU. We evaluate and average performance over 200 random seeds for all experiments.
Asset Licenses The assets used our work are subject to the following licenses: Stack Exchange and Wikipedia Data [Danescu-Nicn) Niculescu-Mizii et al. 2013a]: CC BY-NC-SA 3.0: Climate news headlines data (Luo al. 2020]: CC BY-NC-SA 3.0; Congressional bills data [Adler and Wilkerson 2011: : MIT license; PPI++ Codebase (Angclopoulos et al. 2023b MIT license.
23

---

E.2 Baseline Details
E.2.1 RePPI Implementation
In adapting RePPI [Ji et al. 2025 to our setting, we can model the imputed loss function in PPI with a ML-based approach. While in their paper, they choose particular form of
go (
8 s" (X,Y), (x, 1) 1+
where s" is the conditional function. In our setting, we do not have access to X unlabeled instances, meaning that our model of the conditional score must take in inputs of (X, Y X, Y). In the case for GLMs and if we have access to unlabeled instances X, we know that the score is given by ( Y) ( would only need to model E M. However, in setting where only have access to proxy and synthetic data, need to directly model e (X, Y) or try to learn E X, Y)JX, Y, X, VJ. as we do not observe X to use in our predictions on unlabeled data.
B|y|X, X, IJ), where f is as defined in Section C.1 In that setting, we
We note that in our experiments, the sample splitting approach proposed in RePPI performs poorly due to pross-fitting; each split has a size of 3 of the number of labeled data to estimate (i) the ground truth parameter
fold, (ii) learn the ML model the second fold, and (iii) have accurate target parameter estimate on the final fold. As such, to learn the ML model for the imputed loss in RePPI, choose adopt linear regression model (defined over small number of covariates), which satisfies the required Donsker conditions to enable us to avoid any requirements on sample splitting as in standard DML approaches [Van Der Vaart and Wellner Chernozhukov al. 2018).
Therefore, our implementation of RePPI is as follows: we (1) fit 0 by optimizing the human-only loss, (2) optimize linear regression model that learns map h (X, Y, X,Y) Veg(X,Y, Y), and (3) we perform power tuning and produce our parameter estimate by minimizing the imputed loss that incorporates Ó and h, all on the full available data. We use the same linear regression in estimating the imputed loss; the exception to this on the Congressional Bills dataset, where use XGBoost as linear regression performs very poorly in estimating the score function.
Proposition 2. The RePPI objective with multiple predicted covariates and outcomes is given by
L (oPPP(()) = 1 n
1 Σ
N X.. (6)
Σ Lo(X,, Yi)
90 (o Y,, 8,, 8.)
Σ
9o
N
where
go (8, X, Y.)
1+r 8² (X,, Y, X,, Y1), S" -XV P, Y2),
and 6 is the target parameter estimate. The resulting estimate retains asymptotic normality condi- tions.
E.2.2 PPI++Proxy and PPI++Synth Implementation
We now present discussion adapted debiasing bebiasing-based based approach from Proposition
Proposition 3. The adapted +PI++ objective with multiple predicted covariates and outcomes is given by
PP
Σ [[1 Lo( 1.) a Lo(
(7)
1 Σ (Lo(X,, - (1- -( Co(x, Y1) + + lol lo 2.)).
(8)
where the estimate retains asymptotic normality conditions (see Appendix F.2.2 for the proof and algorithu details).
24

---

Algorithm 1 Cross-Fitting for PPI Syyth
Require:
Labeled data XY)
2: Proxy data D [[T), j, Y3) 1ntm
3: Synthetic data D ((T), x,, Iutm
4: K folds
Ensure: Debiased estimate OCF
5: Split D into folds {II..... Ix}
6:
7: for k ....... K do
8:
define train-fold Iirain = Urpp IT
9:
0T k arg ming (p;
b (1) initial fit on train-fold
10:
â- t arg (01 T.p ;0
(2) select mixture weight on train-fold)
12:
13:
@h + arg ming Lyp (0; (0; k
(3) final on held-out fold with chosen a)
14:
15:
end for
K
16:
return fC
gh
K
Asymptotic Normality of the true objective.
First, it relatively straightforward to show that this an unbiased estimate
B[ (1 ax) (X,, Y)] + -
+ B(eo(X,))) B[(1-0) -)) (x F)) - (X
E(Co(X,
Note that this holds for any choice of the hyperparameter a.
Under the same assumptions as in the PPI++ paper [Angelopoulos et al. for some constant and, in the case of generalized lınear models, the Hessran is non-singular, we perform their same approach to power tuning), we recover the asymptotic normality guarantees the parameter estimate (as in Corollary from Angelopoulos et al. (2023b]).
2023b (e.g., that n+m + c
Hyperparameter Selection via Cross-fitting The added complexity from these modified debiasing- based approaches arises from the hyperparameter a. We now discuss an approach for selecting a by performing cross-fitting. As previously mentioned,
simple version of RePPI [ii et al. 2025 where we fit a convex combination of proxy and synthetic losses.
treat
Namely, we partition our available data into two splits. We select a one fold by minimizing:
arg min T, (.1), [0
where 0 is defined as the solution to the naive minimzation of E[fo(X, Y)] on the same split. This essentially captures picking the that best combines the proxy and synthetic losses to best mimic the behavior of the standard loss function. We then take this optimal a and use to produce a parameter estimate the held-out fold. We aggregate these estimates as is standard in cross-fitting approaches. This process outlined in Algorithm T
E.3 Prompt Texts
We present the full text prompts that were used to generate proxy covariates and labels (for the proxy data) and synthetic data. Note that the prompts used to extract covariates and labels from the synthetic text are identical to those used for the proxy data.
25

---

Proxy Data Generation Prompts
Politeness (First Plural Pronouns) Covariates:
Does the following text contain first person plural pronouns (e.g., we, us, our, ourselves)?
Output either yes or no.
Text:
[content)
Answer:
Politeness (First Plural Pronouns) Labels:
Is the following text polite? Output either A or B. Output a letter only.
A) Polite
B) Impolite
Text: RA.
{content)
Answer:
Politeness (Hedging) Covariates:
Does the following text contain hedging devices exrices-expressioon that indicate uncertainty, caution, or a lack of full commitment to claim (e.g., may, might, could, would, possibly, probably, perhaps, apparently, suggest, indicate, seem, appear, it likely that, it seems that)? Respond with yes or no only. Text: =*"
{content}
nan
Answer:
Politeness (Hedging) Labels:
Is the following text polite? Output either A B. Output letter only.
A) Polite
B) Impolite
Text:
{content}
Answer:
Stance Dataset Covariates:
Does the following text contain any affirmative device words? Output either yes no. Text: иnn
[content} н*.
Answer:
Stance Dataset - Labels:
statement can agree, be neutral, or disagree with the statement: "Climate change/global warming is serious concern".. Classify the following statement into one of the three categories. Output either A. or C. Output letter only.
A) Agree
B) Neutral
C) Disagree
Statement: **.
{content)
Answer:
26

---

Proxy Data Generation Prompts (continued)
Congressional Bills Dataset Covariates:
You are political scientist familiar with the U.S. Congress and the DW-NOMINATE scoring system, which places legislators and legislation on a left-right ideological spectrum ranging approximately from -1 (most liberal) to +1 (most conservative). Below is the text of proposed bill. Based on the policy content, language, and framing the bill, estimate the DW-NOMINATE score that best represents its ideological position. Output single nonzero float between -1 and +1 representing the estimated OW-NOMINATE score of the bill. Bill:
{content}
Answer:
Congressional Bills Dataset Labels:
Does the following text relate to the economy? Output either true or false.
Text: и*в
{content>
Label:
Synthetic Data Generation Prompts
Politeness (First Plural Pronouns)
Consider texts taken from user requests on Stack Exchange Wikipedia. Each text is labeled as either polite or impolite, and either contains or does not contain first-person plural pronouns. Below an example that {x}: Example:
{example)
Now, generate new example of a request that also (x}.
Politeness (Hedging)
Consider texts taken from user requests on Stack Exchange or Wikipedia. Each text can be labeled either polite or impolite, and as either containing a hedging device not containing one. Hedging devices are expressions that indicate uncertainty, caution, or lack of full commitment to claim (e.g., may, might, could, would, possibly, probably, perhaps. apparently, suggest, indicate, etc.). Below is an example that {x}: Example: "*"
{example}
Now, generate new example of a request that also {x}.
Stance
Consider news headlines that take a stance - agree, disagree, or neutral on the statement:
"Climate change/global warming is serious concern."
Each headline also either contains or does not contain an affirmative device. Below is an example of headline. Example: an"
(example}
unn
Affirmative device: {x}
Now, generate new news headline about global warming that also {x}.
27

---

Synthetic Data Generation Prompts (continued)
Congressional Bills Data
You are political language model trained to generate realistic examples of U.S. congressional bills. Each bill is labeled either "related to the economy" or "not related the economy", and is associated with DW-NOMINATE score representing ideological position (ranging conservative).
from -1 liberal
Example:
Bill Text: n
{example)
DW-NOMINATE: Score: heo__omiinat__scrre
Now, generate a new example of bill that also has DW-NOMINATE score of <dv_nominatee_score). Output only the new bill text:
F
Additional Results
We present additional experimental results consisting of:
Prediction accuracy of GPT-4o, Llama-3-8b, Qwen-3-8b (Think) for the covariates and outcomes interest (Table
Performance of naive estimator that only uses synthetic data (Figure
Effective sample size results (Figures 5
Grid search results for PPI++Sytth (Oracle) across different values (Figures 6 and
Cross Cross-fitting results for PPI++Synth (Figures and 9
Llama-3-86 results for logistic regression (Figure 10 and OLS (Figure
Qwen-3-8b results for logistic regression (Figure 12] and OLS (Figure B
Results with zero-shot prompting for synthetic data generation (Figure 14)
Results with using random noise for synthetic data (Figure 15
Table 1: Accuracy of LLMs for prediction tasks. This represents the quality of the proxy covariates
and proxy labels (X, Y). Note, for the Congressional Bills dataset, we report MSE for the X Prediction task since DW-NOMINATE scores continuous values.
Hedging
1pp
Stance
Congressional Bills
Model
X Pred.
Y Pred.
X Pred.
Y Pred.
X Pred.
Y Pred.
X Pred.
Y Pred.
GPT-4o
0.764
0.785
0.994
0.785
0.864
0.743
0.184
0.827
Llama-3-8b
0.485
0.335
0.878
0.501
0.619
0.637
0.688
0.564
Qwen-3-8b (Think)
0.723
0.325
0.961
0.325
0.904
0.684
0.253
0.757
28

---

0.030
2.0
Human-arly
Haman-orly LiMagrthllioony
0.025
Human-oril
LLM synthetic only
1.5
Blas"
0.015
0.010
2
0.005
0.01
0.02
0.03
0.04
0.05
0.05
0.06
0.07
0.00
0.09
0.10
0.006
0.008 0,010 0.012 0.014
Fraction Labeled Data
Fraction of Labeled Data
Fraction Labeled Data
0.012
0.010
Human only
L.Mayttheli--nny
0.08
Hamen-onty
0.0025
Human-nny LM-yyntedeo-rtt
0.008
0.06
0.0020
Biax"
0.0015
0.04
0.004
0.0010
0.02
0.002
0.0005
0.000
0.00
0.0000
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
0.0050 0.0075 0.0100 0.0125 0.0150 Fraction Labeled Data
Fraction of Labeled Data
Fraction of Labeled Data
Figure Performance of a naive estimator for logistic regression (top) and OLS (bottom) using synthetic data only (Politeness (Hedging). Stance, Congressional Bills (from left to right)). We clearly observe that naively using only synthetic data the estimation task leads to largely biased estimates, as expected.
GMMC-Prxxy lours)
300
CMM--rrxy lours)
Effective Sample Size
Synth tours)
Effective Sample Size
Synth lours)
800
PPtn PPoxy
Human--nly
250
PPI-
Human-Dnily
700
PR1- syrth
PP1- Synth
Roip1
ReP91
600
500
400
300
0.06
0.07
0.08
0.09
0.10
0.01
0.02
0.03
0.04
0.05
Fraction Labeled Data
Fraction Labeled Data
300
Proxy lours)
CMM--ioay (ours)
Effective Sample Size
GMM-Syyth tours)
Effective Sample Size
GyMt-Sytth (ours)
Human--Dly
250
Human-Orily
250
PPt +Proxy
PPis-PProxy PPi-+Syrth 0.01
Syrth
200
ReFPi
RePPI
200
150
100
100
0.05
0,06
0.07
0.08
0.09
0.10
0.006 0.008 0.010 0.012 0.014 Fraction Labeled Data
Fraction of Labeled Data
Figure 4: Effective sample size for logistic regression (Politeness (Ipp), Politeness (Hedging). Stance, Congressional Bills (from left to right)). We observe large gains in effective sample size, up more than 50%. This represents how many human annotations the method effectively saves while maintaining the same performance (in terms of mean squared error).
29

---

1000
300
GMM-Prro tours)
Effective Sample Size
CMM-Synthh
Effective Sample Size
CMN--Synth lours)
800
PRle Proxy
Human- Only
250
Human Only
PRi- PProxy
PO1- +Syrth (ct 0.81
RePPI
600
150
400
100
0.05
0.06
0.07
0.08
0.09
0.10
0.01
0.02
0.03
0.04
0,05
Fraction of Labeled Data
Fraction of Labeled Data
Proxy
GMIM-Proxy CaIM-Synth (ours)
Effective Sample Size
120
GMM Synth lours)
Effective Sample Size
250
Human-Dnly
Human-Oriy
100
+Proxy
#
PRt +5ynth ia- 0.8
200
4Syrth (a- 0.21
Rappy
RePPI
60
40
100
20
50
0.02
0.006 0.008 0.012 0.014 Fraction of Labeled Data
Fraction of Labeled Data
Figure 5: Effective sample size for OLS (Politeness (1pp), Politeness (Hedging), Stance, Congres- sional Bills (from left to right)). The RePPI method is omitted from the Ipp plot because its effective sample size drops too low.
0.40
Human-Only
Human-Only
PPI++SYytth (a
0.35
PP+++Synth (a=
Empirical MSE
PBI++ (2)
Empirical MSE
PPI++Synth (a- 0.2)
Pylth (a 0.4)
0.30
PRi++4Synth (z- 0.4)
Pyl++ (a
PPI++Synth (a 0.6)
0.25
PRI++5yytth (a 0.8)
PPi++Synth
PPl++Synth (a 1]
0.20
PPi+- [a-
0.15
0.10
0.05
0.06
0.07
0.08
0.09
0.10
0.01
0.02
0.03
0.04
0.05
Fraction of Labeled Data
Fraction Labeled Data
0.250
Human-Only
Human-Only
0.6
PPI++Synth (a
Empirical MSE
PPi++5ynthh (a= 0.2)
Empirical MSE
PPI++Synth (a (.)00
Pyl+-+Synth (a 0.4)
0.5
PRI++5ynthh (.44
Pyl4-45ynth (a 0.6)
PPI++Synth (a=
0.175
PPi+++Synth (a 0.8)
PPI++Syntth (a
PPi++4Synth la 1
0.4
PPI++Synth (a=)
0.150
PPI++Synth (1)
0.3
0.125
0.100
0.2
0.05
0.06
0.07
0.08
0.09
0.10
0.006 0.008 0.010 0,012 0.014 Fraction of Labeled Data
Fraction of Labeled Data
Figure 6: Grid search results for logistic regression (Politeness (Ipp), Politeness (Hedging). Stance, Congressional Bills (from left to right)). This plot shows the grid search over different possible values without cross-fitting. Note, this not valid solution our setup, as requires peeking in hyperparameter selection, but it provides an oracle version of the baseline, which we as PPI++Synth (Oracle). The a value that leads to the smallest MSE is the one reported in Figure i the main text.
30

---

0.0225
Human-Only
Human-Only
0.016
PP14-45ynth la 0)
0,0200
PRI++Synth (a=0)
Empirical MSE
0.014
PPl+++5ynth la 0.4)
PPI++5ynnth (a= 0.2)
Empirical MSE
0.0175
PPlt+ (a
PRl++Synth (a=
0.012
5ynt+
0.0150
PRI++Synth
PPi++ la 0.8)
PPI++Synth (a= 0.8)
Synth la
0.0125
PRl++Synth (a 11
0.0100
0.0075
0.006
0.0050
0.05
0.06
0.07
0.08
0.09
0.10
0.01
0.02
0.03
0.04
0.05
Fraction Labeled Data
Fraction Labeled Data
0.035
Human Only
Human-Only
0.14
SPit++Sytth (== 0)
PP 4P1+4Synth (ar 0)
Empirical MSE
PPI++5yntth 0.20
Empirical MSE
0.030
PPI++Syntth (2
PPI++Synth (a= 0.4)
PP1++Syytth (ar
0.10
PPI++Synth (a 0.6)
0.025
PP 0.5)
0.08
PPI++5yntth (a 1)
PP1++5ynthh [a 0.8)
0.020
PPI++5ynth 0.6)
+ (ar - 0.8)
0.06
774+55ynth (ar 1)
0.015
0.04
0.02
0.01
0.02
0.03
0.04
0.05
0.006 0.008 0.010 0.012 0.014 Fraction Labeled Data
Fraction Labeled Data
Figure 7: Grid search results for OLS (Politeness (Ipp), Politeness (Hedging), Stance, Congressional Bills (from left to right)). The value that leads to the smallest MSE is the one reported in Figure the main text.
31

---

1.4
Haman-Drly
1.2
2.4
Empinical MMg
Syrth
Interval Widh
2.0
0.6
1.8
0.4
1.6
0.2
0.2
0.0
1.2
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07
0.08
0.09
0.10
Fraction of Labeled Data
Fraction of Labeled Data
Fraction of Labeled Data
Human-Only
2.0
Empiriall MSE
Syrth
Interval Width
2.5
1.0
0.0
0.0
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
Fraction of Labeled Data
Fraction of Labeled Data
Fraction of Labeled Data
Haman-Srly
1.0
3.0
MSE
Empinical MSE
Syrth
Interval Width
2.5
0.2
1.5
0.0
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07 0.08
0.09
0.10
Fraction of Labeled Data
Fraction of Labeled Data
Fraction of Labeled Data
Human Only
MSE
Empitical MSE
0.6
(91l Synth 0.80
Coverage
Interval Width
0.5
0.4
0.2
0.2
0.0
0.006 0.008 0.010 0.012 0.014 Fraction of Labeled Data
0.006 0.008 0.010 0.012 0.014 Fraction of Labeled Data
0.006 0.008 0.010 0.012 0.014 Fraction of Labeled Data
Figure 8: PPI++Synth results for logistic regression. This the valid implementation with cross- fitting to select hyperparameters in statistically valid fashion. We observe that it is upper-bounded by its oracle variant (PPI++Synth a = 0.8), as expected. In the main text (Fig. [.. we report the oracle variant results account for potential gains from improved cross-fitting techniques.
32

---

Only
1.0
0.60
Empirical MSE
0.012
0.014
P7t--Symth le 0.8)
PMt eSynth
Interval Wetth
0.55
0.50
0.6
0.45
0.40
0.008
0.35
0.006
0.30
0.05
0.07
0.09
0.10
0.05
0.06
0.07
0.08
0.05
0.07
0.08
0.09
0.10
Labeled
0.0
0.020
Praxy
Empirical MSE
PMN PMS--4 Synth
PPte 0.88)
0.0
Interval Width
0.7
0.015
0.6
0.5
0.4
0.2
0.3
0.0
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
Fraction Labeled Data
Fraction of Labeled Data
Fraction of Labeled Data
0.14
Human-Only Only PPt Prony
1.0
Empirical MSe
0.12
Interval Wiath
1.4
0.10
0.6
0.08
0.06
0.4
0.2
0.02
0.0
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
Fraction of Labeled Data
Fraction of Labeled Data
Fraction of Labeled Data
0.035
Human--Oil P
Empitical MSE
0.030
PPle- Synth
0.8
Interval Width
0.025
2.0
0.6
1.5
0.4
0.015
0.2
0.010
0.0
0.5
0.006 0.000 0,010 0.012 0.014 Fraction of Labeled Data
0.006 0.008 0.010 0.012 0.014 Fraction of Labeled Data
0.006 0.008 0.010 0.012 0.014 Fraction of Labeled Data
Figure SPI++Synth results for OLS. Similarly as above, we observe that it upper-bounded by its oracle variant (PPI++Synth = 0.8 and PPI++Synth = 0.2 for CBP), as expected.
33

---

0.30
GMM-Proxy lours)
2.0
Empinical MSE
Haman-Dnly
Interval Width
1.8
-Pt--rrary P Synth la 0.5)
1.6
RePP)
0.20
1.2
0.15
0.2
0.0
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07
0.08
0.09
0.10
Fraction of Labeled Data
Fraction of Labeled Data
Fraction of Labeled Data
GMM Prony lours)
Emplrical MSE
0.35
HamensDerly proay
Interval Width
PP Synth ta 0.5) ReiP))
2.00
0.25
1.75
0.15
1.50
1.25
0.05
0.0
1.00
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
Fraction of Labeled Data
Fraction of Labeled Data
Fraction of Labeled Data
2.0
0.250
GMMM-brocy Jour!
Emplnical MSE
PPt- -iroay 0.200
0.225
Human-Dely
Interval Width
1.8
PPi--Syyth ta- 0.5) ReFPi
1.6
0.175
0.150
0.2
1.2
0.125
0.100
0.0
1.0
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07
0.08
0.09
0.10
Fraction Labeled Data
Fraction of Labeled Data
Fraction of Labeled Data
0.8
GMyy-Proxy (ours)
4.5
Empircal MSE
0.7
Wil-4Prany
Interval Width
4.0
PPte Synth 0.5) ReFR)
3.5
3.0
0.4
0.3
2.0
0.2
0.0
0.006 0.008 0.010 0.012 0.014 Fraction of Labeled Data
0.006 0.008 0.010 0.012 0.014 Fraction of Labeled Data
0.006 0.008 0.010 0.012 0.014 Fraction of Labeled Data
Figure 10: Llama-3-8h results for logistic regression. Each row corresponds to a task (i.e., Ipp. Hedging, Stance, Congressional Bills Data (from top to bottom)); each column corresponds to metric (i.c., MSE, coverage, confidence interval width (from left to right)). Results are averaged over 200 trials.
34

---

GMM Prony touss
GMMM- Syrth Tours!
Empirical MSE
Interval With
0.40
<Synth farm
0.6
0.012
RaPP!
0.35
0.4
0.010
0.2
0.008
0.0
0.25
0.05
0.06
0.07
0.08
0.09
0.30
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07
0.08
0.09
0.20
Fraction of Labeled Data
Fraction of Labeled Data
Fraction of Labeled Data
0.0225
GMM--Proyy tourst
0.0200
GMM4-Symth (ours]
Empirical MSE
0.0
Interval Width
0.55
0.0150
PHi-SSyth
Coverage
0.6
0.50
RePR
0.45
0.0125
0.4
0.40
0.0075
0.2
0.30
0.0050
0.0
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
Fraction of Labeled Data
Fraction of Labeled Data
Fraction of Labeled Data
0.012
CMM6-Proay lours Syrth teurs) Human-Onty
Emglrical MSE
0.011
0.8
Interval Width
0.010
PPi--Syyth la RePPi
0.6
0.009
0.008
0.4
0.30
0.007
0.2
0.006
0.0
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07
0.08
0.09
0.10
Iraction Labeled Data
Fraction of Labeled Data
Fraction of Labeled Data
GoRt-Proxy lours)
GMM Synth (ours)
1.0
4.5
Empirical 19
Huml
0.0
Interval Width
4.0
0.6
Syrth
0.6
3.5
0.5
0.4
3.0
0.4
2.5
0.2
0.3
2.0
0.0
1.5
0.006 0.008 0.010 0.012 0.014 Fraction Labeled Data
0.006 0.008 0.010 0.012 0.014 Fraction of Labeled Data
0.006 0.008 0.010 0.012 0.014 Fraction Labeled Data
Figure 11: Llama-3-88 results for OLS. Each row corresponds to a task (i.e., lpp. Hedging, Stance, Congressional Bills Data (from top to bottom)); each column corresponds to metric (i.e.. MSE, coverage. confidence interval width (from left right)). Results are averaged over 200 trials.
35

---

0.30
GMM-Prooy lours)
2.0
Empinical MSE
0.25
-Pt4-Prory P Synth la 0.5) RePP)
Haman-Dnly
Interval Wlath
0.20
2.6
0.2
0.0
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07
0.08
0.09
0.10
Fraction of Labeled Data
Fraction of Labeled Data
Fraction of Labeled Data
GMM lours)
2.4
Emplrical MSE
0.35
P/14 Proay
Human Only
Coverage
Interval Width
2.2
PPt Synth la 0.5)
0.25
RoPP)
1.0
0.15
1.4
1.2
0.05
0.0
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
Fraction of Labeled Data
Fraction of Labeled Data
Fraction of Labeled Data
GMMl-hooyy lour)
GMM
2.2
Empirical MSE
Human-Drly PPt4-P
Interval Width
2.0
0.20
PPl Synth la 0.5) RePP)
0.15
2.6
0.2
0.10
0.0
1.2
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07 0.08
0.09
0.10
Fraction of Labeled Data
Fraction of Labeled Data
Fraction of Labeled Data
Grau-Proxy tours)
3.5
0.6
Empirical MSE
Interval Width
0.5
Wile-PProxy
Synth 0.55 BeP71
3.0
0.4
2.5
0.3
2.0
0.2
0.0
1.5
0.006 0.008 0.010 0.012 0.014 Fraction of Labeled Data
0.006 0.008 0.010 0.012 0.014 Fraction of Labeled Data
0.006 0.008 0.010 0.012 0.014 Fraction of Labeled Data
Figure 12: Qwen-3-8b results for logistic regression. Each row corresponds to a task (i.e., Ipp, Hedging, Stance, Congressional Bills Data (from top to bottom)); each column corresponds to metric (i.c., MSE, coverage, confidence interval width (from left to right)). Results are averaged over 200 trials.
36

---

GMM
0.45
Only
Interval Width
Emplrical MSE
0.014
PPi -Proxy
Prt -Synch la- 0.50
0.012
RaPPi
0.010
0.35
0.008
0.0
0.30
0.05
0.06
0.07
0.08
0.09
0.30
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07
0.08
0.09
0.10
Fraction of Labeled Data
Fraction of Labeled Data
Fraction of Labeled Data
GMM
EMM4-5ytth
0.55
Emplricall MSE
0.0175
hammnnODily
-Prr-
Interval Wltth
0.0150
P7t--yymhh 1o- 0.5)
0.45
0.0125
0.40
0.0100
0.35
0.0075
0.2
0.25
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
Fraction of Labeled Data
Fraction of Labeled Data
Fraction of Labeled Data
GMM lours! Only wihray Pitt -Synth la 0.5) RoPPi
0.012
Emplrical MSE
Interval Witth
0.010
0.40
0.006
0.30
0.004
0.0
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07
0.08
0.09
0.30
0.05
0.06
0.07
0.08
0.09
0.20
Fraction of Labeled Data
Fraction of Labeled Data
Fraction of Labeled Data
0.035
12/t-Prory
Synth Only PPle-
0.8
Empirical MSF
0.030
Interval Width
0.7
0.025
ReFPI
PPtn Synth 0.5)
0.020
0.5
0.2
0.010
0.0
0.4
0.006 0.008 0.010 0.012 0.014 Fraction Labeled Data
0.006 0.008 0.010 0.012 0.014 Fraction of Labeled Data
0.006 0.008 0.010 0.012 0.014 Fraction Labeled Data
Figure 13: Owen-3--b results for OLS. Each row corresponds to a task (i.c., 1pp. Hedging, Stance, Congressional Bills Data (from top to bottom)); each column corresponds metrie (i.c.. MSE, coverage, confidence interval width (from left to right)). Results are averaged over 200 trials.
37

---

0.25
GMH Praxy (ours)
1.0
GMM--5yrhh (ours)
1.8
Empirical MSE
0.20
Human-Driy
0.0
Interval Width
0.6
1.6
0.4
1.4
0.2
0.0
0.07
0.09
0.06
0.07
0.08
0.09
0.10
0.06
0.07
0.08
0.09
0.10
Fraction of Labeled Data
Fraction of Labeled Data
Fraction of Labeled Data
0.4
PMoM-Proxy
1.0
2.4
GMM Synth lours)
2.2
Empirical MSE
0.3
Human Only
0.8
Interval Witth
2.0
0.6
1.8
0.2
0.4
1.6
1.4
0.2
0.0
1.0
0.02
Fraction of Labeled Data
Fraction Labeled Data
Fraction Labeled Data
0.250
GMM Proxy lours)
1.0
Empirical MSE
0.225
Syrth (ours)
Human Only
0.8
Interval Interval Width
1.8
0.200
0.6
1.7
0.175
1.6
0.150
0.4
0.2
1.4
Fraction Labeled
Fraction Labeled Data
Fraction Labeled Data
GMM-Prrxc (oum))
1.0
GMM Synth lours)
Empirical MSE
0.6
Human Only
0.0
Interval width
2.6
0.5
0.6
2.4
0.4
0.4
2.2
0.3
2.0
0.2
0.0
0.010 0.012 0.014 Fraction Labeled Data
0.012 Fraction Labeled Data
Fraction of Labeled Data
Figure 14: Zero-shot prompting results. As ablation, description (instead giving an in-context real sample) for synthetic text generation (i.e., generating T). In all tasks, we observe that our method at least as good as Human-Only in terms of MSE and interval width. We that the benefits to using synthetic data is much less pronounced than when using the specific sampling strategy we propose, as expected. Each row corresponds to a logistic regression task (i.e., Ipp, Hedging, Stance, Congressional Bills Data (from top bottom)); each column corresponds to a metric (i.e., MSE, coverage, confidence interval width (from left to right)). Results averaged over 200 trials.
zero-shot prompts containing the task
38

---

1.0
Synth
1.0
Empirical MSE
Human Only
0.0
Interval Interval Width
2.6
2.5
1.4
0.07
0.08
0.09
0.10
0.07
0.0g
0.08
0.09
0.10
Fraction of Labeled Data
Fraction Labeled Data
Fraction Labeled
1.0
2.4
Empirical MSE
0.30
0.35
Htumen-Drkly
GyMh-Synth (ours)
Interval Width
2.2
2.0
0.25
1.8
0.15
2.4
0.10
1.0
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
Fra fraction of Labeled Data
Fraction of Labeled Data
Fraction Labeled
Synth mumen--iily
1.9
Empirical MSE
0.22
Interval Width
1.0
1.7
0.16
1.6
1.5
0.12
1.4
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07
0.08
0.09
0.10
Fraction of Labeled Data
Fraction of Labeled Data
Fraction Labeled
0.6
GMM-Syntr
2.0
MSE
Empincaal MSE
Aieevvl Wid
2.6
2.4
0.4
0.3
1.8
1.6
0.006 0.088 0.010 0.012 0.014 Fraction Labeled Data
0.006 0.008 0.010 0.012 0.014 Fraction of Labeled Data
0.006 0.088 0.010 0.012 0.014 Fraction Labeled
Figure 15: Random noise for labels results. As another ablation, we replace the proxy and synthetic labels (Y' and Y. respectively) with random noise. Importantly, we observe that our method still performs least as good as just having used the real data (Human-only) on all tasks. Each row corresponds to a logistic regression task (i.c., 1pp, Hedging, Stance, Congressional Bills Data (from top bottom)); each column corresponds metric (i.e., MSE, coverage, confidence interval width (from left to right)). Results are averaged over 200 trials.
39

---

0.225
0.09 GMMN-Proxy Jourst Syrth (ours)
1.0
1e6
Empircal MSE
0.200
PR-- PHa-yPraxy
himannsDll
0.8
Interval Width
2.0
0.175
Pyi--Synth
0.6
0.150
2.0
0.100
0.5
0.075
1e22
1.0
1e10
GitS-Pooyy tours)
1.5
GMM--Syni tours)
Empirical MSE
Interval Width
2.00
1.0
PPi++ Synth ia 0.8)
RCPPI
0.75
0.5
0.50
0.0
0.05
0.06
0.07
0.08
0.09
0.10
0.05
0.06
0.07
0.08
0.10
0.08 0.09 0.20
Fraction Labeled Data
Fraction of Labeled Data
Fraction of Labeled
Figure 16: RePPI results for 1pp task for logistic regression (top) and OLS (bottom). We report the RePPI method results for the Ipp task separately here, due to some very large values. In the main text (Figures 2) we exclude the RePPI numbers from the plots for better visibility.
40


Reviews:
[{"review_id": "4mnV0qJEc7", "reviewer": "Reviewer_7qA1", "summary": "The paper aims to find regression coefficients based on mostly-unlabeled data. It introduces `GMM+synth`, which relies on a black-box LLM to augment unlabeled data, and generate fully synthetic data. It uses a Generalized Methods of Moments approach for combining real, proxy, and synthetic data to get statistical estimates of the regression coefficients.", "questions": "See weaknesses. Additionally:\n\n1. In L.135, do you mean $\\tilde{X}_k=f(\\tilde{T}_k)$, idem ditto for $Y$?\n2. If we have enough texts $T$, what is the intuition for why modelling synthetic $T$ helps?", "limitations": "Yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. Robust low sample size inference is an underexplored problem in ML\n2. The approach---building statistically valid estimators for incorporating possibly erroneous LLMs for augmentation---is appealing\n3. Writing is generally good\n\n## Weaknesses\nI have two general concerns\n1. Method. The method is rather involved, with ML model $f$ giving proxies, the LLM giving synthetic data, and then a GMM doing the statistical inference. There are many hidden choices---which ML model $f$ for proxy data? Which LLM for synthetic data? Which LLM prompts?---which the writers do not explore. To me as a reader, I'm not convinced the method is the best way to leverage the internal knowledge of an LLM.\n2. Experiments. Related to the above, the experiments are rather limited.\n   1. There is just one baseline\n   2. Tasks are simple. \n   3. Just a single LLM, and unclear how accurate GPT-o is for these tasks. The authors pose in Section 4.5 that we'll never be worse, as the model learns to ignore the synthetic data if it's uncorrelated. It would have been interesting to test this in practice, either with more difficult tasks or worse LLMs.\n\n### Minor\n1. Key observations in Experiments should be structured better. Right now, it jumps from Figures 3 and 4, to Figure 6 in the Appendix, to finally get to Figure 2, but then going back to Figure 3.", "was_revised": true, "final_justification": "My initial concerns were about the somewhat limited exploration of some of the modelling choices (e.g. LLM choice) and empirical results. The authors have mostly resolved these issues, and I hope they will include these additional experiments in the revised manuscript. I retain my score and lean towards acceptance.", "rebuttal": "Dear Reviewer 7qA1,\n\nThank you for your feedback! We address your concerns below:\n\n### **Method details**\n\n> The method is rather involved, with ML model giving proxies, the LLM giving synthetic data, and then a GMM doing the statistical inference.  which ML model for proxy data? Which LLM for synthetic data? Which LLM prompts?\n\nWe apologize for the confusion! The **same model** (same LLM) is used to give proxy data (predictions) and synthetic data (generations). We have added the following clarification in the Methods section (Line 126) to clarify this point: “For simplicity, we use the same machine learning $f$ used above for proxy data also for synthetic data generation.”\nFor the LLM, we use GPT-4o (Line 303). For the LLM prompts, we include the full prompts used for all prediction and generation tasks in Appendix D.1 (Lines 853-860). \n\n### **Ablations and more baselines**\n\n> There is just one baseline\n\nWe would like to highlight that PPI++ is the current state-of-the-art method for these tasks. This line of work termed prediction-powered inference (also referred to as debiasing-based methods) has been the predominant strategy for incorporating model predictions in the literature [1][2] and represents the current most competitive method in incorporating imperfect LLM predictions [3][4]. Therefore, we focused on comparisons with this method rather than including multiple, weaker baselines from earlier iterations of this line of research, since they resulted in worse performance.\n\nWe would like to further highlight that we also implement **an “oracle” version of PPI++** that “cheats” in hyperparameter selection  (this is the baseline shown in the main plots). PPI++ requires selecting a hyperparameter alpha via cross-fitting, which cuts the sample size and leads to worse performance. To obtain an upper bound on the performance of this family of methods even if the hyperparameter could be chosen perfectly, the oracle method runs PPI++ on a large grid of hyperparameters and then reports the result closest to the ground truth (which isn’t knowable in practice). Our method outperforms even this strong baseline, while Appendix E (Fig 6) shows that the real, cross-fitting version of PPI++ performs worse than the oracle, as expected.\n\nTo better address your concern, we have also **added an additional baseline** Ji et. al [5]. We report the MSE (lower is better) averaged over 200 trials in the following Table. We clearly see that our method outperforms this method and the same conclusions hold. \n\n|                    | 1pp | Hedging | Stance | CBP |\n|--------------------|----------------|---------------------|--------|---------------------|\n| Ji et al. [2]      | 0.225          | 0.387               | 0.230  | 0.740               |\n| GMM-Synth (Ours)   | 0.125          | 0.162               | 0.208  | 0.311               |\n\n\n> Just a single LLM, and unclear how accurate GPT-o is for these tasks. It would have been interesting to test [worse synthetic data] in practice, either with more difficult tasks or worse LLMs.\n\nWe thank the reviewer for their valuable suggestion! We have added additional experiments using small, open-source models  (Llama-3-8b, Qwen-3-8b), where we observe that even with worse quality LLMs the same conclusions hold.  We have also added an experiment where the synthetic samples from the LLM are replaced by random noise with a set probability $p$ to further demonstrate how our method would empirically perform in even worse cases. More concretely, when p=0.5, we default to **complete random noise** for 50% of the samples and use the LLM predictions for the rest. We observe that even though the synthetic data is complete random noise for half of the samples, we don’t perform worse than not having used them (outperforms the Human-Only baseline). We would like to highlight that we theoretically provide an explanation and proof of this in Section 4.5 (Lines 257-258) and Appendix C (Lines 831-851). \n\nWe report performance (MSE; lower is better) with Llama-3-8b, Qwen-3-8b (respectively)\n\n|                    | 1pp   | Hedging | Stance | CBP |\n|--------------------|-------|---------|--------|---------------------|\n| Human-Only         | 0.301 | 0.384   | 0.250  | 0.566               |\n| GMM-Synth (Ours)   | 0.227 | 0.339   | 0.238  | 0.508               |\n\n|                    | 1pp   | Hedging | Stance | CBP |\n|--------------------|-------|---------|--------|---------------------|\n| Human-Only         | 0.301 | 0.384   | 0.250  | 0.566               |\n| GMM-Synth (Ours)   | 0.198 | 0.329   | 0.229  | 0.506               |\n\nFor 50% of the samples, we default to complete random noise. \n\n|                    | 1pp   | Hedging | Stance | CBP |\n|--------------------|-------|---------|--------|---------------------|\n| Human-Only         | 0.301 | 0.384   | 0.250  | 0.566               |\n| GMM-Synth (Ours)   | 0.300 | 0.374   | 0.248  | 0.567               |\n\n\nWe also report the accuracy of GPT-4o, Llama-3-8b, Qwen-3-8b for these tasks to more comprehensively understand performance. \n\n|                                           | GPT-4o | Llama-3-8b | Qwen-3-8b |\n|-------------------------------------------|--------|------------|-----------|\n| Hedging (Covariate Prediction) | 0.764  | 0.485      | 0.501     |\n| Hedging (Label)     | 0.785  | 0.150      | 0.230     |\n| 1pp (Covariate)     | 0.983  | 0.878      | 0.866     |\n| 1pp (Label)         | 0.785  | 0.150      | 0.230     |\n| Stance (Covariate)             | 0.864  | 0.620      | 0.645     |\n| Stance (Label)                 | 0.742  | 0.636      | 0.621     |\n| CBP (Covariate)| 0.125  | 0.087      | 0.098     |\n| CBP (Label)    | 0.827  | 0.558      | 0.557     |\n\n\n> Tasks are simple\n\nWe would like to remark that we run our experiments on standard benchmarks used in recent “LLM for statistical inference” works [3][4]. In fact, the inference tasks we use are actually more elaborate than previous work in this area, since we include a  nonlinear task (logistic regression) as opposed to more simple linear inference tasks (such as simple mean estimation, quantile estimation, etc.) used in past works [1][2]. To further address your concern, we have also included new results on ordinary least squares (OLS) – another common statistical inference task— in the following Table. We see that even for linear tasks and even when Y is not binary but continuous, the same conclusions hold. OLS and logistic regression are perhaps the most widely used statistical models in computational social science practice and strong performance on these tasks is widely applicable.\n\n|                    | 1pp   | Hedging | Stance | CBP |\n|--------------------|-------|---------|--------|---------------------|\n| Human-Only         | 0.022 | 0.021   | 0.036  | 0.034               |\n| PPI++Synth         | 0.020 | 0.022   | 0.035  | 0.026               |\n| GMM-Synth (Ours)   | 0.009 | 0.015   | 0.022  | 0.015               |\n\n\n### **Organization of figures**\n\n> Key observations in Experiments should be structured better. Right now, jumps from Figures 3 and 4, to Figure 6 in the Appendix, to finally get to Figure 2, but then going back to Figure 3.\n\nThank you for this suggestion! We have grouped Figure 3, 4 and Figure 6 together. Followed by Figure 2, for better flow. \n\n### **Questions**\n\n> In L.135, do you mean $\\tilde{X}_k = f( \\tilde{T}_k)$ idem ditto for $Y$?\n\nThanks for raising this clarification point! Yes, that is exactly what we mean. We have added this to the draft for better clarity. \n\n> If we have enough texts , what is the intuition for why modelling synthetic  helps?\n\nThank you for this great question! Even when the practitioner possesses a large collection of raw texts, the bottleneck for inference is the labels $(X,Y)$, because hand-labeling texts is expensive. Intuitively, prompting the LLM to generate a synthetic sample $(\\tilde{T}, \\tilde{X}, \\tilde{Y})$ for every real document $T$ can be viewed as obtaining a second noisy view of the same latent signal. \n\nTo further answer your question, we plot the relative improvement in variance (Variance_human_only - Variance_with_synthetic) / Variance_human_only, where the number of raw text samples (real T) grows. This metric represents the percentage reduction in variance achieved by adding synthetic data. We fix the labeled portion of the raw text samples (real T) to be fixed p=0.05. We run this on the Congressional Bills dataset, as it has the largest number of raw text samples. We observe that the relative improvement is almost constant, indicating that the improvement from adding synthetic data persists even when the number of real text grows large. \n\n|                    | n=1000 | n=2000 | n=3000 | n=4000 | n=5000 |\n|--------------------|--------|--------|--------|--------|--------|\n| Relative Variance  | 0.220  | 0.210  | 0.208  | 0.211  | 0.210  |\n\nWe thank the reviewer again for their time and consideration! We are more than happy to address any other concerns they might have.\n\n   1. Angelopoulos et. al. PPI++: Efficient Prediction Powered Inference.\n   2. Angelopoulos et. al. Prediction-Powered Inference.\n   3. Gligoric et. al. Can Unconfident LLM Annotations Be Used for Confident Conclusions.\n   4. Egami et. al. Using Imperfect Surrogates for Downstream Inference.\n   5. Ji et. al. Predictions as Surrogates: Revisiting Surrogate Outcomes in the Age of AI."}, {"review_id": "f8tLAgOiko", "reviewer": "Reviewer_psyf", "summary": "This paper proposes a principled framework for incorporating fully synthetic data, generated by LLMs, into downstream statistical inference tasks. Motivated by applications in computational social science where labeled data is often scarce, the authors introduce a novel estimator based on the Generalized Method of Moments that combines labeled data, model-predicted data (proxies), and LLM-generated synthetic samples. The method exploits correlation structures introduced by conditioning synthetic samples on real data and provides theoretical guarantees on consistency and asymptotic normality. Empirical results on four real-world regression tasks demonstrate significant performance gains in low-label regimes.", "questions": "1. Can your framework support generation strategies that are not conditioned on real examples? For instance, can it be extended to few-shot or zero-shot prompting?\n\n2. What's the perfomance gap of between open-source models and GPT-4o?\n\n3. What's the diversity of the generated samples, which should also be dicsussed and verified.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. This paper tackles an important and emerging challenge, i.e., how to use LLM-generated synthetic data for reliable statistical inference.\n\n2. It introduces GMM estimator for this task and leverages the correlation between real and synthetic data for synthetic data generation. Experiments on diverse real-world tasks from political science and psychology show strong gains, especially in low-data settings.\n\n3. The paper is well-written and clearly structured. And the theoretical and empirical sections are well balanced.\n\n**Weaknesses**\n\n1. While the paper mentions that low-quality synthetic data won’t help, it doesn’t empirically explore the effect of varying LLM quality or prompt design.\n\n2. The experiments are focused on regression tasks with structured covariates; it’s unclear how well the framework generalizes to classification, causal inference, or time series.\n\n3. The framework relies on the availability of strong LLMs like GPT-4o. This may not generalize well to low-resource settings, such as open-source models.\n\n4. It would be helpful to see a comparison between the proposed conditioned sampling vs. naive zero-shot generation to justify the design decision.", "was_revised": true, "final_justification": "After rebuttal, my concerns are properly addressed. Based on the paper quality and overall concerns, I will keep my score unchanged which lean to accept.", "rebuttal": "Dear Reviewer psyf,\n\nThank you for your feedback! We address your concerns below:\n\n### **Varying LLM quality/prompt design types**\n\n> While the paper mentions that low-quality synthetic data won’t help, it doesn’t empirically explore the effect of varying LLM quality…The framework relies on the availability of strong LLMs like GPT-4o. This may not generalize well to low-resource settings, such as open-source models.\n\nThanks for the suggestion! We have added additional experiments using small, open-source models (Llama-3-8b, Qwen-3-8b) to encompass varying types of LLM quality. We observe that even with worse quality LLMs, the same conclusions hold. We have also added an experiment where the synthetic samples from the LLM are replaced by random noise with a set probability $p$ to further demonstrate how our method would empirically perform in ***even worse cases***. More concretely, when p=0.5, we default to **complete random noise** for 50% of the samples and use the LLM predictions for the rest. We observe that even though the synthetic data is complete random noise for half of the samples, our method is able to safeguard against this (approximates performance you would get with human-only samples). This is important since practitioners can only know retrospectively if an LLM was not good for their task at hand. We would like to highlight that we theoretically provide an explanation and proof of this in Section 4.5 (Lines 257-258) and Appendix C (Lines 831-851). \n\nWe report performance (in terms of MSE; lower is better) with Llama-3-8b, Qwen-3-8b (respectively)\n\n|                    | 1pp   | Hedging | Stance | CBP |\n|--------------------|-------|---------|--------|---------------------|\n| Human-Only         | 0.301 | 0.384   | 0.250  | 0.566               |\n| GMM-Synth (Ours)   | 0.227 | 0.339   | 0.238  | 0.508               |\n\n|                    | 1pp   | Hedging | Stance | CBP |\n|--------------------|-------|---------|--------|---------------------|\n| Human-Only         | 0.301 | 0.384   | 0.250  | 0.566               |\n| GMM-Synth (Ours)   | 0.198 | 0.329   | 0.229  | 0.506               |\n\nFor 50% of the samples, we default to complete random noise. \n\n|                    | 1pp   | Hedging | Stance | CBP |\n|--------------------|-------|---------|--------|---------------------|\n| Human-Only         | 0.301 | 0.384   | 0.250  | 0.566               |\n| GMM-Synth (Ours)   | 0.300 | 0.374   | 0.248  | 0.567               |\n\nWe would also like to remark that our method offers a statistical framework for integrating information from LLMs, which provides value when LLMs offer useful signal (as you should only expect it to help if the LLM is “good”). In the case when the LLM is “bad”, what is important is that the method does not result in extremely biased estimates and can safeguard against this. The performance of models today and their rapidly improving accessibility (e.g., open models also keep improving) is what motivates practitioners to want to leverage predictions and generations from them, which in turn provides strong motivation for such integration frameworks. \n\n> Varying prompt design. It would be helpful to see a comparison between the proposed conditioned sampling vs. naive zero-shot generation to justify the design decision…Can your framework support generation strategies that are not conditioned on real examples? For instance, can it be extended to few-shot or zero-shot prompting?\n\nWe thank the reviewer for their valuable suggestion. We have added additional experiments encompassing varying types of prompt strategies: few-shot prompting and zero-shot prompting. We clearly see that conditional sampling leads to larger benefits downstream, justifying our design decision. However, even when the synthetic samples are of worse quality due to simpler prompt strategies, we still outperform baselines. \n\nResults for few-shot prompting, zero-shot prompting (respectively):\n\n|  | 1pp | Hedging | Stance | CBP |\n|----------------------|----------------|---------------------|--------|---------------------|\n| Human-Only           | 0.301          | 0.384               | 0.250  | 0.566               |\n| GMM-Synth (Ours)     | 0.246          | 0.321               | 0.209  | 0.497              |\n\n\n\n|  | 1pp | Hedging | Stance | CBP |\n|----------------------|----------------|---------------------|--------|---------------------|\n| Human-Only           | 0.301          | 0.384               | 0.250  | 0.566               |\n| GMM-Synth (Ours)     | 0.277          | 0.370              | 0.239  | 0.507               |\n\n\n### **More Tasks**\n> The experiments are focused on regression tasks with structured covariates; it’s unclear how well the framework generalizes to classification, causal inference, or time series.\n\nWe have added additional experiments to now cover regression, classification, causal inference, and time series tasks. We report performance in terms of MSE (lower is better) averaged over 200 trials. We observe that our method outperforms baselines across the additional tasks, demonstrating our framework's flexibility beyond regression: any estimand expressible as a moment condition can leverage our approach. \n\nFirst, in addition to the logistic regression results (classification task) in the paper, we run ordinary least squares (regression task). We clearly see that our method outperforms baselines and the same conclusions hold.\n\n|                    | 1pp   | Hedging | Stance | CBP |\n|--------------------|-------|---------|--------|---------------------|\n| Human-Only         | 0.022 | 0.021   | 0.036  | 0.034               |\n| PPI++Synth         | 0.020 | 0.022   | 0.035  | 0.026               |\n| GMM-Synth (Ours)   | 0.009 | 0.015   | 0.022  | 0.015               |\n\nNext, we construct a semi-synthetic causal inference task to evaluate treatment effect estimation under missing covariates. Starting with real data containing features $X_i$ and outcomes $Y_i$, we synthetically generate treatment assignments $D_i$ via covariate-dependent randomization and simulate causal outcomes with heterogeneous treatment effects across covariate strata. \nWe define moment conditions $m_i(\\tau) = S_i · (\\phi_i - \\tau)$, where $\\phi_i$ is the augmented inverse propensity weighted (AIPW) influence function combining outcome regression and propensity score components. Nuisance functions (propensity scores and outcome models) are estimated non-parametrically on labeled data, then we use these moments in our GMM estimator.\nThis demonstrates our framework's flexibility beyond regression: any estimand expressible as a moment condition can leverage our approach. \n\n| sel_prob | gmm-synth | gmm-proxy | ppi-synth | ppi-proxy | human-only |\n|----------|-----------|-----------|-----------|-----------|-------------|\n| 0.01     | 0.013897  | 0.014038  | 0.013891  | 0.014437  | 0.022648    |\n| 0.02     | 0.006132  | 0.006223  | 0.006490  | 0.006292  | 0.008589    |\n| 0.03     | 0.004239  | 0.004302  | 0.004402  | 0.004293  | 0.006024    |\n| 0.04     | 0.003213  | 0.003352  | 0.003300  | 0.003339  | 0.005192    |\n| 0.05     | 0.002316  | 0.002411  | 0.002484  | 0.002432  | 0.004100    |\n\nWe further demonstrate our applicability to time series by estimating autoregressive (AR) parameters under missing data. We generate $N$ independent AR(1) processes $Y_{it} - \\mu = \\phi(Y_{i,t-1} - \\mu) + \\varepsilon_{it}$ where $\\phi$ is the target parameter, supplemented by noisy proxy and auxiliary time series data. The GMM formulation uses AR(1) moment conditions: $E[\\varepsilon_{it}] = 0$ and $E[(Y_{i,t-1} - \\mu)·\\varepsilon_{it}] = 0$, where $\\varepsilon_{it}(\\phi,\\mu)$ represents model residuals. Again, we observe that our method outperforms baselines. \n\n| sel_prob | gmm-synth | gmm-proxy | ppi-synth | ppi-proxy | human-only |\n|----------|-----------|-----------|-----------|-----------|-------------|\n| 1        | 0.01053   | 0.01161   | 0.01387   | 0.01226   | 0.05426     |\n| 2        | 0.00421   | 0.00464   | 0.00548   | 0.00471   | 0.02714     |\n| 3        | 0.00257   | 0.00293   | 0.00313   | 0.00274   | 0.01645     |\n| 4        | 0.00194   | 0.00225   | 0.00246   | 0.00229   | 0.01062     |\n| 5        | 0.00146   | 0.00176   | 0.00198   | 0.00182   | 0.00935     |\n\n\n### **Questions**\n\n> What's the performance gap between open-source models and GPT-4o?\n\nWe report the performance gap between GPT-4o and open-source models (Llama-3-8b, Qwen-3-8b) in the table below.\n|                                           | GPT-4o | Llama-3-8b | Qwen-3-8b |\n|-------------------------------------------|--------|------------|-----------|\n| Hedging (Covariate Prediction) | 0.764  | 0.485      | 0.501     |\n| Hedging (Label)     | 0.785  | 0.150      | 0.230     |\n| 1pp (Covariate)     | 0.983  | 0.878      | 0.866     |\n| 1pp (Label)         | 0.785  | 0.150      | 0.230     |\n| Stance (Covariate)             | 0.864  | 0.620      | 0.645     |\n| Stance (Label)                 | 0.742  | 0.636      | 0.621     |\n| CBP (Covariate)| 0.125  | 0.087      | 0.098     |\n| CBP (Label)    | 0.827  | 0.558      | 0.557     |\n\n> What's the diversity of the generated samples, which should also be discussed and verified.\n\nThank you for this suggestion! We report Self-BLEU (lower is more diverse) and embedding-based metrics (average cosine distance) (higher is more diverse). We also report the metrics for samples generated by different prompt strategies (zero-shot, few-shot) for comparison. \n\n|                         | Self-BLEU | Embedding-based Diversity |\n|-------------------------|-----------|---------------------------|\n| Conditioned Sampling (Ours) | 0.324     | 0.687                     |\n| Few-Shot Sampling          | 0.8599    | 0.407                     |\n| Zero-Shot Sampling         | 0.6245    | 0.504                     |\n\nWe thank the reviewer again for their time and consideration! We are more than happy to address any other concerns they might have."}, {"review_id": "DnPLNQLlb4", "reviewer": "Reviewer_WjYv", "summary": "Using human annotators to label domain-specific text can be highly expensive and time-consuming. Therefore, recent works have used LLMs to replace humans in labeling and data generation. While this can remove the original costs, we are still not sure about the benefits and correctness of the synthetic data. This paper models this process through the Generalized Method of Moments. The authors model the contribution of real, auxiliary, and synthetic data. Their comprehensive model shows we can utilize synthetic data to improve performance under essential conditions. They should have similar residuals to real data.", "questions": "Conditions under line 135 are not clear. Are the equations showing how T, X, and Y are generated, or are T and X when the text is labeled, and T and Y when the text is unlabeled? The current formulation is misleading.", "limitations": "Yes", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Weakness**\n\nThe main weakness of the paper is that it is hard to follow; some basic concepts are overexplained, while the important part and the contribution section are left ambiguous. A very critical concept, the Generalized Method of Moments, is not explained in the related work.\n\n**Strength**\n\nThe contribution of the paper is significant. The authors have theoretically shown how to select synthetic datasets and when they can be beneficial. \n\nThe experimental results also support their theoretical results. \n\nThe paper has an interesting discussion section (4.5 and 5). \n\nMy suggestion for the authors is to summarize the content in the early pages of the paper and expand on how the Generalized Method of Moments works and how it can help with this problem.", "was_revised": true, "final_justification": "After reading the author's response to all the reviews, I decided to keep my score.", "rebuttal": "Dear Reviewer WjYv,\n\nThank you for your feedback! We’re grateful for your acknowledgement of our contributions. We address your concerns below:\n\n### **Clarity and Expansion of Related Work**\n\n> The contribution of the paper is significant. The main weakness of the paper is that it is hard to follow; some basic concepts are overexplained, while the important part and the contribution section are left ambiguous. A very critical concept, the Generalized Method of Moments, is not explained in the related work.\n\nWe appreciate that you find our contributions to be significant and thank you for this important feedback. We currently discuss our GMM approach in detail in the Methods section (Section 4). To address this further, we add the following subsection to the Related Work section (after Line 104) to provide further context earlier on in the draft:\n\n“””\n**Generalized Method of Moments (GMM).** The GMM framework [Hansen, 1982] is a flexible estimation method that generalizes many common statistical approaches (e.g., maximum likelihood, instrumental variables, least squares, etc.). GMM estimators are defined by moment conditions—functions of the data and parameters whose expectations equal zero at the true parameter values. The key insight is that when the number of moment conditions exceeds the number of parameters to estimate, GMM optimally combines these conditions using a weighting matrix that accounts for their variance-covariance structure. This framework has been widely adopted in statistics and econometrics due to its robustness to distributional assumptions and ability to handle overidentified systems (more conditions than parameters). In this work, we leverage GMM's flexibility to incorporate human-labeled, proxy, and synthetic data through separate moment conditions. The two-step GMM estimation procedure [Newey and McFadden, 1994] (see Section 4.3 for details) learns optimal weights for combining these data sources through the moment covariance matrix, eliminating manual hyperparameter tuning required by existing debiasing methods. The key intuition is that the weighting matrix captures the interactions between the real moment residuals and the synthetic moment residuals; the interactions between these moment residuals improves estimation when they are predictive of one another (see Section 4.5 for further discussion).\n“””\n\n\nCurrently, we provide a description of our GMM method in Section 4 in the following manner:\nIntroduce moment conditions (Section 4.1), estimator construction with proxy and synthetic data (Section 4.2), the two-step estimation procedure (Section 4.3), and theoretical results on consistency and asymptotic behavior of our GMM estimator (Section 4.4 with the full proof in Appendix A). \n\nWe further include a concrete example of our moment construction in Appendix B (Lines 821-831). However, we forgot to note a reference to Appendix B in the main text, which might have caused some ambiguity. To help improve clarity, we have updated the draft to move this concrete example from Appendix B into the main text before Section 4.3. This provides readers with a concrete illustration of our moment construction before turning to the estimation procedure (Section 4.3), making the method easier to follow. We hope these additions help step through the main contribution/method of the paper better. We would be happy to further improve the explanation if there are any additional remaining concerns.\n\n> My suggestion for the authors is to summarize the content in the early pages of the paper and expand on how the Generalized Method of Moments works and how it can help with this problem.\n\nThank you for this suggestion! We have updated the Introduction (Lines 61-70) with the following paragraph to summarize and further expand on how the GMM works and helps with the problem at hand more earlier in the draft:\n\n\"\"\"\nOur primary methodological contribution is to propose a new estimator based on the generalized method of moments (GMM) framework that naturally incorporates this synthetic data. The construction of our GMM estimator defines separate parameters and moments for each data source. It is not initially obvious whether the incorporation of moments based exclusively on synthetic data should yield any benefits (or even affect) the estimation of the target parameter. However, we find that in the GMM estimation procedure, the off-diagonal terms in the weight matrix captures interactions between the real moment residuals and synthetic moment residuals, allowing the information from the synthetic data moments to influence the estimation of the target parameter. The key intuition is that synthetic data will improve performance when the synthetic data residuals are predictive of the real data residuals. More concretely, the asymptotic variance is the remaining variance after you regress the real residuals on the span of the synthetic residuals. In other words, when the residuals are independent of each other, the asymptotic variance reduces to the optimal variance based only on real data. This means that in the worst case, when synthetic data is completely uninformative, it does not hurt performance asymptotically (see Section 4.5 and Appendix C for further details).\n\"\"\"\n\n\n### **Clarification Question (Line 135)**\n\n> Conditions under line 135 are not clear. Are the equations showing how T, X, and Y are generated, or are T and X when the text is labeled, and T and Y when the text is unlabeled? The current formulation is misleading.\n\nThank you for raising this clarification point! Yes, the equations are showing how T,X,Y are generated. We have updated the draft to list the equations in a list and not side-by-side to avoid this potential confusion:\n\n$\n\\tilde{T}_k \\sim \\mathbb{P} (\\cdot \\mid T_i, X_i) \n$\nif labeled\n\n$\n\\tilde{T}_k \\sim \\mathbb{P} (\\cdot \\mid T_j, \\hat{X}_j) \n$\nif unlabeled\n\n$\n\\tilde{X}_k \\sim \\mathbb{P} (\\cdot \\mid \\tilde{T}_k)\n$\n\n$\n\\tilde{Y}_k \\sim \\mathbb{P} (\\cdot \\mid \\tilde{T}_k)\n$\n\nWe thank the reviewer again for their time and consideration! We are more than happy to address any other concerns they might have."}, {"review_id": "baX4rnhaCq", "reviewer": "Reviewer_B76K", "summary": "This paper presents a novel methodology for generating synthetic data from Large Language Models (LLMs) that seeks to generate samples matching the distribution of human-labeled data. This is done by incorporating one human-labeled sample and specific request for a classification category into each LLM prompt. The paper also introduces a new estimator algorithm that seeks to correctly predict the true distribution from the (imperfect) artificial samples.", "questions": "Please explain why the experiments using PPI++ were conducted using a different set of artificially generated samples than the ones used by the authors' algorithm.", "limitations": "Limitations are addressed briefly, but not in enough detail to be satisfactory. In particular, the authors do not address the (very real) risks that researchers acting based on the claims of the paper might attribute more reliability to the algorithms result than is justified. In other words, because the authors make strong claims relating to the algorithm's ability to identify the true distribution from artificially generated samples, there is a risk that researchers may succumb to overreliance on the algorithm's outputs, even when research is conducted that is outside the scope of the algorithm's optimal performance bounds (for example, when a small or unreliable language model is used, or when the specific classification requested by the prompt (e.g. \"hedging\") lies outside of the language model's capabilities. For this reason, the authors should clearly identify the bounds of their algorithm's validity: For which category of language models and for which types of classification problems can the authors' novel prompting method be considered accurate? How can this reliability be externally verified by other researchers? What are the potential outcomes or dangers of naive reliance upon the authors' algorithm in contexts where the algorithm does nor provide good results?", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The paper seeks to address an important limitation in many social science contexts: Hand-labeled training data is expensive to produce and often limited in scope. Artificially-generated samples offer one way to overcome this limitation, but only if it is possible to infer the true distribution from an analysis of artificially generated samples (which are likely to contain unwanted biases)\n- The paper includes theoretical grounding for the proposed methodology, building upon established mathematical precedents\n- The authors ran multiple trials of all experiments, strengthening the reliability of the results\n- The introduction of a principled sampling method for artificial generated within the context of social science research is particularly valuable, and is worthy of deeper exploration\n\nWeaknesses\n- The prompting methods used by the researchers to generate artificial samples may not be as robust as the paper implies.\n- The assumption that prompting with T will result in text generations that exactly match the original distribution is incorrect, and in particular is highly dependent on the language model used. Highly distilled language models, in particular, tend to gravitate toward certain styles and patterns of generated text regardless of the prompt, and there is no guarantee that they will match T_i in all meaningful senses.\n- The authors do not address the question of how the selection of language model (e.g. GPT-4, GPT-4o, Qwen, Claude Sonnet, Llama, etc.) might impact the reliability of results, nor does the work explore the question of what will happen when the specific classification requested in the LLM prompt (e.g. hedging) represents a skill, behavior, or category that the language model is not equipped to- PPI++ is referenced within the paper and used as a key baseline, but this algorithm is not clearly explained within the text of the paper, making the comparisons difficult to understand. A more direct explanation of PPI++ within the paper body would reduce reviewer workload and make the paper more accessible to a broad audience. identify or generate.\n- The baseline comparisons used by the paper do not seem to be the correct way to evaluate this work. For example, PPI++ is used as a baseline, but it differs from the authors' methods along multiple dimensions (e.g. power tuning, the sampling method used to generate artificial data), making it difficult to draw meaningful conclusions from the results. In particular, it would have been nice to see a comparison of the authors' methods and PPI++ using the authors' novel sampling method for artificial data. It would also have been nice to see a comparison of PPI++ performance both with naively sampled artificial data and with data sampled using the authors' methodology.\n- The presentation of ideas within the paper do not flow naturally. Motivation for the research is not clearly established, many key points of information are not included in the paper proper and must be researched elsewhere in order for the paper to be properly understood. The title and abstract do not effectively convey the main contributions of the paper.", "was_revised": true, "final_justification": "This paper tackles an important challenge in modern social science by proposing a method to select synthetic datasets that are guaranteed not to decrease prediction accuracies.\n\nI am personally not convinced by the authors' assertions, but that may be due to my limited experience with the type of statistical analyses the authors are conducting. (My own expertise lies in the area of machine learning, regression models, and LLM architectures.)", "rebuttal": "Dear Reviewer B76K,\n\nThank you for your feedback! We address your concerns below:\n\n### **Assumption that text generations = original distribution**\n\n> The prompting methods used by the researchers to generate samples may not be as robust as the paper implies. The assumption that prompting with T will result in text generations that exactly match the original distribution is incorrect\n\nWe sincerely apologize for the potential confusion! To start, we must clarify: **our method does not require this assumption**. We have edited the Introduction (lines 45-48) to make this explicit (as we think the misunderstanding may be our fault due to some ambiguous wording). Changes are bolded:\n\n“””\nNaively…impossible to provide statistical guarantees for the resulting estimate if the generative model does not perfectly match the real distribution—which is expected in practice. **We propose** a specific sampling strategy…This correlation structure will prove critical for principled methods for integrating synthetic data, as it enables us to more effectively share information across them **without requiring the distributions to be equal to each other**.\n“””\n\nMore importantly, we would like to highlight that **in the Methods section we do not impose any assumptions on the quality of proxy or synthetic data to derive our key theoretical results on consistency** (Section 4.4) (see Appendix A for proof). In Section 4.5, we explicitly analyze why synthetic data improves performance even when they are imperfect and derive an explicit formula for the asymptotic variance in Appendix C, where we show, “in the worst case, where synthetic data is completely informative, including it does not hurt, asymptotically” (Lines 257-258). \n\nThe core idea behind how we construct our GMM estimator (see Section 4.2) is that we estimate **separate** parameters ($\\theta, \\eta$) for each data source, precisely how the method avoids assuming the samples (from different data sources) come from the same distribution. If we assumed all data came from the same distribution (as the original distribution) then we would constrain $\\theta = \\eta$. The fact that we keep these separate is precisely how we allow for distributional differences while still benefitting from cross correlations.\n\n>  “because the authors make strong claims relating to the algorithm's ability to identify the true distribution from artificially generated samples, there is a risk that researchers may succumb to overreliance on the algorithm's outputs. For this reason, the authors should clearly identify the bounds of their algorithm's validity…”\n\nWe apologize for the potential confusion! We would like to clarify again that we do not claim to identify the true distribution from artificially generated samples. Our theoretical results guarantee valid confidence intervals *regardless* of whether the synthetic samples match the real distribution; the confidence intervals will just be tighter if the synthetic samples match more closely. We state explicitly in the paper, even if the LLM is not good, the performance will still be at least as good as if the researcher had only used the human-labeled data: “When the [real and synthetic] residuals are independent, the formula reduces to the optimal variance based only on the real data” (Lines 256-257). \n\nTo further demonstrate this, we have added new experiments where the synthetic samples from the LLM are replaced by random noise with a set probability $p$ to further demonstrate how our method would empirically perform in even worse cases. More concretely, when p=0.5, we default to **complete random noise** for 50% of the samples and use the LLM predictions for the rest. We observe that even though the synthetic data is complete random noise for half of the samples, we don’t perform worse than not having used them (approximates the Human-Only baseline). We theoretically provide an explanation and proof in Section 4.5 (Lines 257-258) and Appendix C (Lines 831-851). \n\n|                    | 1pp   | Hedging | Stance | CBP |\n|--------------------|-------|---------|--------|---------------------|\n| Human-Only         | 0.301 | 0.384   | 0.250  | 0.566               |\n| GMM-Synth (Ours)   | 0.300 | 0.374   | 0.248  | 0.567               |\n\n\n> Ethical Concerns: Data quality and representativeness\n\nAgain, we would like to clarify that we do not make claims on data representativeness and one of the main strengths of our method is specifically that it does not make assumptions on the quality of the synthetic data. \n\n\n### **Different LLMs**\n\n> how the selection of language model might impact the reliability of results\n\nWe thank the reviewer for their valuable suggestion! We have added additional experiments using small, open-source models with worse performance (Llama-3-8b, Qwen-3-8b), where we observe that even with worse quality LLMs the same conclusions hold. \n\nWe report performance (MSE; lower is better) with Llama-3-8b, Qwen-3-8b:\n\n|                    | 1pp   | Hedging | Stance | CBP |\n|--------------------|-------|---------|--------|---------------------|\n| Human-Only         | 0.301 | 0.384   | 0.250  | 0.566               |\n| GMM-Synth (Ours)   | 0.227 | 0.339   | 0.238  | 0.508               |\n\n|                    | 1pp   | Hedging | Stance | CBP |\n|--------------------|-------|---------|--------|---------------------|\n| Human-Only         | 0.301 | 0.384   | 0.250  | 0.566               |\n| GMM-Synth (Ours)   | 0.198 | 0.329   | 0.229  | 0.506               |\n\n### **Details about Baseline Comparisons**\n\n> baseline comparisons do not seem to be the correct way to evaluate this work. PPI++ is used as a baseline, but it differs from the authors' methods along multiple dimensions (e.g. power tuning, the sampling method used to generate artificial data)\n\nWe would like to clarify that **PPI++ uses the same synthetic data samples**. The synthetic data samples used across all baselines are fixed to be the same across all methods. \n\nWe would also like to clarify that power tuning is a strategy the PPI++ authors use to improve their original weaker baseline PPI. Power tuning finds an optimal weighting parameter $\\lambda$ that takes a combination between the classical estimator (using only real data) and the PPI estimator (using both real and synthetic data). $\\lambda$ is chosen to minimize the width of the confidence interval. When the prediction model performance is poor, such that incorporating unlabeled data hurts performance, it sets $\\lambda$ = 0 to recover the classical estimator. Therefore, we are actually making a comparison with a stronger method by reporting numbers that involve power tuning for that baseline. We also report numbers without power tuning for more context:\n\n|                    | 1pp   | Hedging | Stance | CBP |\n|--------------------|-------|---------|--------|---------------------|\n| PPI++Synth         | 0.347 | 0.380   | 0.227  | 0.690               |\n| PPI++Synth (without power tuning)   | 0.358 | 0.402   | 0.258  | 0.710               |\n\n\n### **Title and Abstract**\n\n> The authors' actual contributions are far more interesting, but neither the title nor abstract clearly convey the ideas behind the authors estimator algorithm, nor the principled sampling method used by the authors to generate synthetic data.\n\nWe appreciate that you find our contributions interesting! We have updated the title to “Valid Inference with Imperfect Synthetic Data via Generalized Method of Moments” to better reflect the core methodology behind the estimator algorithm and have updated the abstract to include further details about the principled sampling method, the estimator algorithm, and intuition behind why/when synthetic data improves performance.\n\n\n### **Expand Motivation**\n\n> Motivation is not clearly established\n\nWe are motivated by limited-labeled data settings where practitioners face constraints in obtaining sufficient human-labeled data due to budget constraints. This has led researchers to increasingly turn to LLMs as cheap but noisy alternatives for data generation and annotation (e.g., simulating human responses to surveys) [1][2][3].\n\nThe persistent challenge is that naively combining these imperfect synthetic data samples with human-annotated samples for downstream inference tasks can lead to biased estimates, compromising statistical validity of its conclusions. Current methods are only able to incorporate noisy predictions and not fully synthetic samples [4][5], which further motivates the purpose of this work. Our work addresses this fundamental gap by providing the first theoretically-grounded framework that can safely incorporate fully synthetic samples while maintaining statistical validity, regardless of the quality of the synthetic data.\n\n### **Add Detailed Description of PPI++ in Main Text**\n\n> PPI++ is referenced within the paper and used as a key baseline, but this algorithm is not clearly explained within the text of the paper\n\nThanks for the suggestion! Currently, we state the PPI++ objective in Section 5, Proposition 2 (Lines 274-277) and the pseudocode for the algorithm implementation in Appendix F, Algorithm 1 (Lines 894-895). To address this concern, we have (1) updated the draft to add the pseudocode in the main text (Section 5) and (2) add a line-by-line explanation of the algorithm in Section 5 to help provide a more direct explanation of the method.\n\nWe thank the reviewer again for their time and consideration! We are more than happy to address any other concerns they might have.\n\n   1. Hwang et al. Opportunities and challenges of applying llm-simulated data, 2025.\n   2. Geng et al. Are large language models chameleons? an attempt to simulate social surveys, 2024.\n   3. Rothschild et al. Opportunities and risks of llms in survey research, 2024.\n   4. Gligoric et al. Can Unconfident LLM Annotations Be Used for Confident Conclusions, 2024.\n   5. Egami et al. Using Imperfect Surrogates for Downstream Inference, 2023."}]

Author rebuttal:

Meta-review: As I have weighed the reviewer's comments, the author's rebuttal, and my read of the paper itself, I have concluded that this paper has more potential impact that its ratings might suggest.  I believe that the problem setup and proposed solution could have application far beyond the CSS settings discussed here, and could provide important foundational ideas to increase our ability to harness synthetic data. To me, that is a critical capability that will become increasingly important in the future, and I believe now is a good time to publish papers to spark conversations and future work in this area.

I think the idea of using GMMs is mathematically elegant and well-justified (the method in this paper reminds me of control variates from control theory).  The core principle seems to be that synthetic data can be used as long as there is some sort of connection/signal/pattern between the real and synthetic data; in this paper, that signal is "covariance", but I could imagine building more sophisticated linkages.

I encourage the authors to respond to the reviewers comments and address their concerns, but especially after the additional work the authors have done during the rebuttal period, I find the paper worthy of publication.
Decision: Accept (poster)