Efficient Randomized Experiments Using Foundation Models
Authors: Piersilvio De Bartolomeis
Javier Abad
Guanbo Wang
Konstantin Donhauser
Raymond M Duch
Fanny Yang
Issa Dahabreh
Venue: NeurIPS 2025 poster ()
Keywords: statistical efficiency
causal inference
average treatment effect
data fusion
randomized trials
Primary area: probabilistic_methods

Abstract: Randomized experiments are the preferred approach for evaluating the effects of interventions, but they are costly and often yield estimates with substantial uncertainty. On the other hand, in silico experiments leveraging foundation models offer a cost-effective alternative that can potentially attain higher statistical precision. However, the benefits of in silico experiments come with a significant risk: statistical inferences are not valid if the models fail to accurately predict experimental responses to interventions.
In this paper, we propose a novel approach that integrates the predictions from multiple foundation models with experimental data while preserving valid statistical inference. Our estimator is consistent and asymptotically normal, with asymptotic variance no larger than the standard estimator based on experimental data alone. Importantly, these statistical properties hold even when model predictions are arbitrarily biased. Empirical results across several randomized experiments show that our estimator offers substantial precision gains, equivalent to a reduction of up to 20\% in the sample size needed to match the same precision as the standard estimator based on experimental data alone.

Efficient Randomized Experiments Using Foundation Models
Piersilvio De Bartolomeis
Javier Abad ETH Zurich jabadmarttine@ehh..cc
ETH Zurich & Harvard University pdebartol@ethz.co
Guanbo Wang
Konstantin Donhauser ETH Zurich
Raymond Duch
Harvard University gwangShsph.o harvard.
Oxford University
donhausk@ethz..ch ch
raymond. duch@ox.cccoooo ac uk
Fanny Yang
Issa J Dahabreh
ETH Zurich
Harvard University
fan. yang@in ettz..c..
idahabrehhhsph. harvard. edu
Abstract
Randomized experiments are the preferred approach for evaluating the effects of interventions, but they are costly and often yield estimates with substantial uncertainty. On the other hand, in silico experiments leveraging foundation models offer cost-effective alternative that can potentially attain higher statistical preci- sion. However, the benefits of in silico experiments come with a significant risk: statistical inferences are valid if the models fail accurately predict experi- mental responses to interventions. In this paper, we propose a novel approach that integrates the predictions from multiple foundation models with experimental data while preserving valid statistical inference. Our estimator is consistent and asymp- totically normal, with asymptotic variance no larger than the standard estimator based on experimental data alone. Importantly. these statistical properties hold even when model predictions are arbitrarily biased. Empirical results across several randomized experiments show that our estimator offers substantial precision gains, equivalent to reduction of up to 20% in the sample size needed to match the same precision as the standard estimator based on experimental data alone.
Introduction
Randomized experiments are widely considered the preferred approach evaluating the effects of interventions in scientific research. However, obtaining sufficiently large sample sizes can be costly and time-consuming, especially when studying rare outcomes. For example, Carlisle al. [11] reported that 481 out of 2579 recently completed clinical trials (19%) failed due to insufficient patient recruitment to meet the required sample size. In cancer trials, this failure rate can be as high as 40% due to strict eligibility and safety requirements [49]. As result, there is growing interest in exploring in silico experiments as a potential alternative randomized experiments. In silico experiments leverage the predictions from foundation models (6]-machine learning models trained on massive datasets and applicable to many downstream tasks to simulate the outcomes of hypothetical randomized experiments. This approach has already shown promising results in replicating the results of randomized experiments in several scientific disciplines, including clinical research [26, 19, 14] and the social sciences [3, 5, 4].
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Experimental data
(%
(X, A,A
AIPW estimators
(X))
T(xx
@(1:)
A
S
External data I
fi
f(Xx)) )
θ(
+
① H-AIPW
Re41
External data &
fk
Figure 1: Illustration of the Hybrid Augmented Inverse Probability Weighting (H-AIPW) estimator. For each unit observe covariates X,, treatment A; and outcome Yú; (X,, A., Yi);= forms the experimental data. An outcome regression model h fitted this sample yields the standard AIPW estimate 0(h). In addition, foundation models trained on external data provide the candidate outcome regression models fi...., fs. which result in & competing AIPw estimates Ø(fi), Ö(fe). By integrating the outcome regression models trained on a large external sample, rather than fitting a single model on the small experimental sample, H-AIPW can reduce the variance of the average treatment effect estimate.
However, for method be adopted in safety-critical fields like medicine, valid statistical inference is an absolute re- quirement. For instance, the Food and Drug Administration guidance strongly recommends that any method aimed at improving the efficiency of randomized experiments should provide valid inference under minimal statistical assump- tions [23]. Yet, statistical inference from in silico experi- ments is not valid if model predictions fail to reflect exper- imental responses to interventions. Since such an assump-
Reduction in sample size
20%
15%
10%
0%
tion is difficult to falsify, the growing consensus among researchers is that results from in silico experiments should be limited to exploratory stages of research, for example, pilot studies to predict effect sizes in larger experiments [27].
F8n22Z
20 HX
ELADD
F1.8
SyS
HR23
Scientific study
Figure 2: Our estimator achieves the same statistical precision as the standard estima- with up to 20% fewer samples. Each study is subsampled to n 75. We plot the percentage reduction the sample size needed to match the confidence interval width of the standard estimator using ours.
This limitation raises an important question: Can safely leverage the predictions from foundation models to improve efficiency while preserving valid statistical inference? In this paper, we introduce the Hybrid Augmented Inverse Probability Weighting (H-AIPW), novel estimator that can integrate predictions from multiple, potentially biased, foundation models while preserving valid statistical infer- ence under minimal assumptions. Specifically, we prove
that H-AIPW consistent and asymptotically normal, with asymptotic variance no larger than the standard estimator based on experimental data alone. Importantly, our results require no additional assumptions beyond those necessary for estimating treatment effects in classical randomized exper- iments. While our methodology applies broadly, we focus our empirical results on social science survey experiments, where large language models (LLMs) can provide rich predictive signals. Across several randomized experiments, we show that H-AIPW can offer substantial precision gains, equiv- alent to a reduction of up to 20% in the sample size required achieve the same precision as the standard estimator based on experimental data alone (see Figure 2).

---

2
Related work
Our work draws heavily from the literature on semiparametric inference and double machine learn- ing [42, 41, 47, 12]. In particular, our estimator is optimal combination of several Augmented Inverse Probability Weighting (AIPW) estimators, whose outcome regression models are replaced with foundation models. Importantly, the standard AIPW estimator, which relies regression model estimated using experimental data alone, also included in the combination.
outcome
Integrating foundation models Prediction-powerct inference (PPT) [1] is a statistical framework that constructs valid confidence intervals using small labeled dataset and large unlabeled dataset imputed by a foundation model. PPI has been applied in various domains, including generalization of causal inferences [18, 9], large language model evaluation [24, 20], and improving the efficiency of social science experiments [8, 21J. Recent work by Poulet et al. [40] introduces Prediction prediction--oweered powered inference for clinical trials (PPCT), adaptation of PPI estimate average treatment effects in randomized experiments without any additional unlabeled data. PPCT combines the difference in means estimator with an AIPW estimator that uses the predictions from one foundation model for both treatment and control groups. However, our work differs in a crucial aspect: PPCT does not include the standard AIPW estimator with the outcome regression model estimated from experimental data. Therefore, there is no mechanism to prevent PPCT from having higher variance compared to the standard AIPw estimator that uses experimental data alone (see e.g. Table 1). This risk of increased variance is critical limitation in many settings- for example, in clinical trials, pharmaceutical sponsors are highly risk-averse and methods that carry even a small chance of underperforming the established standard face significant barriers adoption. We refer the reader to Appendix A.2 for more complete discussion of the differences between our approach and PPI.
Integrating observational data There is growing interest in augmenting randomized experiments with data from observational studies to improve statistical precision [36]. One approach involves first testing whether the observational data is compatible with the experimental data [13, 37, 29, 17, 16), and then combining the datasets to improve precision, if the test does reject. These tests, however, have low statistical power, especially when the experimental sample size is small, which is precisely when leveraging observational data would be most beneficial. Another line of work combines a biased (but more precise) estimator from observational data with an unbiased estimator from experimental data obtain a debiased estimate [31, 15, 43, 48]. However, in small sample settings, the debiasing procedure often fails. Closest to ours, there are two lines of works that propose unbiased estimators: one integrates a prognostic score estimated from observational data as covariate when estimating the outcome regression model [44, 35), while the other incorporates an outcome regression model estimated from observational data directly into the AIPW estimator [25, 32]. However, both approaches rely access to well-structured observational data to improve statistical precision. In contrast, our approach not constrained by the availability of well-structured data, and instead leverages black-box foundation models trained on external data sources.
Background on randomized experiments
We observe dataset D of size from randomized experiment, containing tuples (X, Y, of covariates X Rdd bounded outcome Y R, and treatment variable A E (0, 1}. We assume that the data is drawn i.idd from a joint distribution P over (X, Y (0), Y ((), Y, A), where the potential outcomes (0), (1)) E RR2 are unobserved and Y (A). Our goal use D estimate the average treatment effect (ATE) in the randomized experiment population,
8 = (1) -
where the expectation is taken over P. In particular, we want to improve upon the statistical precision of classical ATE estimators by constructing an asymptotically valid confidence interval that is narrower. We further assume that the data is collected from proper randomized experiment that satisfies the following standard assumptions.
3

---

Assumption 3.1 (Identification assumptions). The data-generatinn process satisfies ((a) 1 A, for 0, 1. (ii) P(A=a - X) - no (0, 1), for =0,1.
We assume that the propensity score Ta is known by design, as is the case in the vast majority of experiments. Nevertheless, our framework can be extended to allow for covariate-adaptive randomization or settings where the probability of treatment needs to be estimated.
Under Assumption 3.1, we can identify the ATE as follows
= (1) [() = B|y A = 1 A =0]
Therefore, the standard approach is to estimate 0 using the difference in means (DM) estimator,
Σ
Y,
NO Σ
Y, where na = ((:: A; = a
#A,11
=A,=0
This estimator is consistent and asymptotically normal (see e.g. Wager [50, Theorem 1.2]):
VT(@nn - 8) N(0,, VnM),
where -- denotes convergence distribution and DM is the asymptotic variance. Therefore, provided that we can obtain a consistent estimator of the asymptotic variance, VoM = VoM + oF(1), we can construct asymptotically valid confidence interval
VDM
Com = Ouss 21-
such that lim,-+oo P(O E CRM) 1-a, where 2a is the a-quantile of the standard normal distribution. Arguably, 0pm all that is needed estimate average treatment effects in randomized experiments. However, the variance Vos is often very large, leading to wide confidence interval Cb In the next section, will show that it is possible to obtain narrower confidence intervals if we leverage the information contained in the covariates.
3.1 A class of valid estimators: Augmented Inverse Probability Weighting
Robins et al. [42] show that, when the propensity score is known, every regular and asymptotically linear estimator of 0 is asymptotically equivalent an AIPW estimator of the form below:
i A (h)
(h)),,
where h RRd {0,1) 1}
R. is square-integrable function, and
A (Y; - h(x,.1) (h) = T
(Y, (X,,,, 0)) + + M(X,,))
The most efficient estimator within this class uses an outcome regression model h that minimizes the asymptotic variance. Specifically, the semiparametric efficiency lower bound is attained by choosing h h,' a) = EE[|X = xr A a). which corresponds to the conditional mean of the outcome. In other words, the estimator Õxw (h*) attains the smallest asymptotic variance among all consistent and asymptotically normal estimators of 0, and, thus, the smallest possible confidence interval in large samples. In practice, however, we only have estimator of the conditional mean h², which achieves the efficiency lower bound only if lÍ h l= = (1).
Below, we adapt the standard result that establishes consistency and asymptotic normality of the AIPW estimator to our setting, where the treatment probability known. The key distinction from the standard setting is that asymptotic normality is achieved as long as the outcome regression model has an asymptotic limit. This implies that the confidence intervals are valid even when the outcome regression is estimated using complex machine learning models with unknown convergence rates.

---

Proposition (Asymptotic behavior of AIPW), Let D be an auxiliary sample, independent D. D. Let h be model trained on D, and let h be inntare-ntegrable limit such that E {0.1}.
[[((;a)) - (:aa))-h(::: (t,aaa))l
O,
where IP" denotes the joint law of (D, D). Then, it follows that (xww(()) is asymptotically normal: (T(OAup - 0) ~ NO, Vat),
where Va = -E [(v(ht) 00²] is the asymplolic variance.
We provide proof of this result in Appendix A.1.1, Proposition I shows that the choice of estimator for the outcome regression does not affect the validity of the inference, provided that it is trained on an independent sample-for example, by using cross-fitting on the experimental sample or training the model on a larger external dataset. Under these conditions, we can then construct an asymptotically valid confidence interval Cu outlined in Equation (1). Further, the asymptotic variance critically depends on the limiting model ht, and decreases as ht more closely approximates the true conditional mean h (see Appendix A.3 for formal result on the dependency of the excess variance on the difference between the outcome regression model and h*).
standard way to obtain an estimate h using the observed data would be to output the minimizers of the empirical risks of each a:
i(X, a) E arg min
Σ ((;))
(2)
NEH
iAA.m
where H is chosen model class (e.g. all linear functions) and L point-wise loss function (e.g. mean squared loss). We refer to the empirical risk minimizer AxP (h) in Equation (2) with H being the linear function class, as the standard AIPW estimator-as the name suggests, this is the most common estimator that is currently being deployed practice. Hence, a key desideratum for any new estimator is safety with respect to this standard baseline that is, it should never perform substantially worse in terms of variance, and ideally perform better than the standard AIPW estimator. However, key limitation of the standard AIPW estimator is that its outcome regression model is trained on a small sample size and is limited to a simple function class. In the next section, we introduce a novel estimator that instead leverages predictions from foundation models trained on vast amount of external data, significantly improving our chances of learning an accurate outcome regression model.
4 Methodology
We introduce Hybrid Augmented Inverse Probability Weighting (H-AIPW), estimator that, in contrast the standard AIPW, leverages the predictions from multiple foundation models to improve statistical precision. In what follows, we first provide formal definition of the H-AIPW estimator in Algorithm and then give theoretical results for its asymptotic distribution and variance.
4.1 Hybrid Augmented Inverse Probability Weighting
With the recent widespread availability of foundation models, we can potentially improve the accuracy of the outcome regression model beyond what is obtained from Equation (2) simply by replacing with a foundation model. This is the principle behind PPI-style estimators, yet such an approach offers no safety guarantee of doing no worse than the standard estimator-a critical desideratum for adoption in many domains. Further, as is often the case with language models, multiple competing models may be available, with clear way determine the best choice for a given task in advance. Therefore, we propose combining multiple AIPW estimators, each using different outcome regression model:
αxw ()), O xPP (f1), Dxw (fx).
Here, 7 is estimated exclusively from experimental data, as shown Equation (2), while f1,..., Sx foundation models trained on independent external data. The challenge of selecting optimal
5

---

Algorithm Hybrid Augmented Inverse Probability Weighting (H-AIPW)
Require: (i) Dataset D = {(X;, A, Y=) (ii) Collection of foundation models
(iii) Loss function & and function class H. (iv) na for 0, 1. (V) Significance level a.
1: Use cross-fitting to compute the estimate i Axrr (h) from the dataset D, where for each arm a:
1 Na i:d.=a
T((Xa) a) E arg min DEN
Σ ((Y,)((xi))
2: Compute Ā == 1--11/(11 1), where
n v
(h) ..... vv(ff))
Σ Σ (((h) (IB, -5).
3: Compute the estimate and its variance
02 = 21 (h) +
Σ
xxww (fi) 2z-1. and V2 : - ΣX.
(3)
4: Return: CLAIw
00
21
Px m
where 2a the a-quantile of the standard normal.
estimator from set of competing estimators for the same target quantity has been extensively studied in the statistical literature; see e.g. Lavancier and Rochet [34]. A common solution is to consider weighted average of the available estimators, which in our setting corresponds to
k+1 - j=1
0 == Amy (h)
Σ 1=1
O (f,))41, for some = (=
We illustrate the estimation pipeline in Figure 1. Further, we restrict the weights to the constraint set A so that the combined estimator 0x is still in the class of AIPw estimators. We can then choose the weights that minimize the asymptotic variance VA of the combined estimator 0x, that is:
X arg min VA arg min X =- XEA
ΛEA
(ff))) is the asymptotic covariance and ht the asymptotic limit where == = Cov((v(ht), ..... of h. However, in practice we only have an estimate Σ of the covariance matrix, and thus we use
x = arg min A Êx.
Asymptotic validity and efficiency We now establish that the H-AIPW estimator is consistent and asymptotically normal, with an asymptotic variance that no greater than that of the standard AIPW.
Theorem 2 (Asymptotic behavior of H- H-AIPw). Let be an outcome regression model that satisfies the conditions in Proposition 1, with asymptotic limit ht. Further, lei 0₂ be as in Equation (3), and assume that Σ is non-singular and -5 Ellop L, 0. Then, it holds that
v - 0) NO0 Vx-).
Moreover, the asymptotic variance of the combined estimator is greater than that of any individual estimator, Le. holds that
Vy Ejs, for j 1, k

---

Example System Prompt
Example User Prompt
Treatment: student organization de- nied Antifa's 's request for rally, citing safety concerns due to altercations at similar events.
You are 35-year-old female, politically Democrat, holding liberal views. Ad- ditionally, your religion is Christianity, and you once or twice month attend religious services. You reside a build- ing with two more apartments, and your household has a yearly income $85,000 to $99,999.
Outcome question: Do you agree or dis- agree with the statement:
" "Cancel culture a big problem in to- day's 's society." Choose integer be- tween (strongly agree) and 5 (strongly disagree).
You are responding to a scenario reflect- ing debate involving college campus events and broader social issues.
Instruction: Reflect on the scenario and your reasoning to assign value.
(a)
Figure 3: Examples of a system and user prompts used to generate synthetic responses for Fahey et al. [22].
We provide a proof of this result in Appendix A.1.2. Theorem offers a principled approach to combining multiple competing AIPw estimators, ensuring that the resulting estimator is at least as precise (asymptotically) as the best estimator in the ensemble. In particular, this approach allows us to leverage the strengths of foundation models without any risks: when these models give accurate outcome predictions, the combined estimator uses their extra information to improve precision. On the other hand, when the foundation models are biased, the final estimator falls back to AIPW.
4.2 Step-by-step recipe with Large Language Models
In this section, we provide a step-by-step guide for practitioners implement H-AIPW using Large Language Models (LLMs). Our guide focuses on LLMs as they are both widely accessible and have demonstrated good accuracy in predicting human behavior [27]. As concrete example, present a political science experiment that evaluates the effect of free speech framings on opposition cancel culture among Americans [22]. We provide simplified prompts here and refer readers to Appendix C.2 for the full LLM prompts.
Extract participant information. Extract the tuples Z; (X,, YY, A.) for each participant in the study. In Fahey et al. [22], covariates include age, gender, ideology, income, and religion. The treatment represents scenario where an Antifa protest is banned: for safety reasons only (A 0), or for safety reasons and cancel culture (A = 1). The outcome is measured on scale from 1 to 5, as the level of agreement with the statement: "Cancel culture is big problem in today's society."
2. Construct system prompts. For each participant i, create persona that matches X; and guides the LLM in simulating responses. In this study, personas summarize the participant's ' demographics. The persona then used as the system prompt for the LLM (see Figure 3a).
3. Construct user prompts. The user prompt includes the experimental treatment, the outcome question, and instructions to guide the LLM (see Figure 3b). We prompt the LLM to generate synthetic outcome for both treatment and control. The final instruction is sampled from predefined pool to introduce variability in the LLM's responses (see Appendix C.2.9).
4. Simulate outcome responses. Query the LLM using the user and system prompts. Validate that the responses numeric and conform to the specified outcome scale. For experiments where multiple instructions are sampled, compute the average response.
5. Estimate treatment effects. Compute the confidence interval CH-AIP via Algorithm 1. Using cross-fitting to fit the outcome models is key for coverage in small-sample settings.
7

---

Table I: Performance comparison of H-AIPW against baseline estimators (PPCT, DM, AIPW, PROCOVA) across several randomized experiments. We randomly subsample each study sample sizes n 100 and 200. We report the variance of each estimator averaged over A 10k subsampling repetitions. Cells shaded blue denote the standard AIPW baseline that should be improved upon using external data; green indicates better precision than standard AIPW; and red indicates worse precision than standard AIPW.
Melin et al. (2022)
Silverman et al. (2022)
Kennedy et al. (2020)
Fahey et al. (2023)
Estimator
100
200
100
200
100
200
100
200
H-AIPW
10.39
10.28
2.10
2.14
17.09
17.47
4.87
4.94
PPCT
11.00
11.06
2.25
2.26
17.87
17.97
4.88
4.91
PROCOVA
11.81
10.62
2.24
2.22
18.38
18.11
5.18
5.09
AIPW (boosting)
12.82
12.44
2.82
2.83
23.09
23.12
6.31
6.37
AIPW (standard)
11.72
10.57
2.22
2.20
18.09
17.95
5.09
5.04
DM
11.10
11.10
2.30
2.30
18.07
18.08
5.61
5.62
Caprariello et al. (2013)
Brandt (2013)
Haaland al. (2023)
Shuman et al. (2024)
Estimator
100
200
100
200
100
200
100
200
H-AIPW
5.88
5.96
11.86
11.90
4.49
4.44
8.46
8.91
PPCT
5.99
6,01
12.07
12.12
4.50
4.52
9.08
9.14
PROCOVA
6.41
6.13
12.77
12.25
4.73
4.44
9.12
9.55
AIPW (boosting)
7.79
7.60
15.20
14.70
5,39
5.22
10.53
10.67
AIpW (standard)
6.39
6.18
12.13
4.82
4.55
9.20
10.31
DM
6.15
6.15
12.81
12.80
5.72
5.71
13.83
13.83
5
Experiments
In this section, first show that H-AIPW improves statistical precision across eight randomized experiments without compromising empirical coverage. then evaluate performance several LLMs. highlighting the importance of both model scale and inference-tim compute: larger models (e.g., GPT-4o and LLaMA 3 70B) consistently outperform smaller ones in prediction accuracy, and averaging over multiple prompts inference time further improves performance.
5.1 H-AIPW offers improved precision
We evaluate H-AIPW across eight randomized experiments in Economics [28]. Psychology [7]. Political Science [22], Foreign Policy [46]. Sociology [33, 39, 10, 45]. These studies were selected from the multidisciplinary Time-Sharing Experiments in the Social Sciences (TESS) repository, along the lines of Ashokkumar et [4]. For each experimental study 8, we implement the following subsampling procedure: starting with a full dataset D of size N,, select target sample size For each subsampling repetition (1...., R), we sample n participants without replacement from D, ensuring the treatment and control groups are balanced, to create smaller dataset D,..
Estimators and metrics We implement H-AIPW by integrating predictions from three LLMs: GPT- 4o, Claude 3.5 Haiku, and LLaMA 3 70B. For each LLM,
10 different prompts for prediction and average over the responses (see Appendix C.2 for example prompts). We benchmark our estimator against two standard estimators: 0ps (DM) and 0xPw (h) (AIPW), where h is the solution to the optimization problem in Equation (2) with either a linear (standard) or complex (boosting) function class. We also implement the concurrent PPCT estimator [40] and the PROCOVA estimator [35, 44], both using GPT-4o as the external model. These two serve as more competitive baseline that also leverages predictions from foundation models (see Appendix C.I for implementation details). To benchmark statistical precision, for each estimator
compute the scaled variance R > n.Var(0,) where Var is the empirical variance obtained from the dataset D,-as the sample size grows, the scaled variance approaches the asymptotic variance of the estimator.

---

68
708
+ 278
X 2
DeepSook
LLaMA
Gomini Flash
Phi-4
Claude Haiku
2.4
Mean Squared Emror
2.2
Genna 2.
Fi-4
1.9
1.8
X
1.8
Gantri Flash
Grok
1.5
1.6
Cledde3
DoepSook
1.4
3
10
4.85
4.9
4.95
5
5.05
Number of prompts
Variance
(a)
(h)
Figure 4: Impact model scale and inference-time compute on the performance Н-AIPW the study by Fabey et al. [22]. (Left) Model scale: Figure 4a shows the relationship between the estimate of the H-A1PW variance (average varying sizes (10 prompts at inference time). (Right) Inference-time compute: Figure 4b shows the impact the MSE of increasing the number prompts inference time averaging the resulting predictions.
R = 10k repetitions, sample size 50) and mean squared error (MSE) for LLMs of
Results Table I reports the estimated variance of several competing estimators across eight different experimental studies and two sample sizes (n = 100 and n = 200). Across nearly all scenarios, H-AIPW consistently achieves the lowest variance among all estimators, and hence the tightest confidence interval. In particular, we observe variance reductions of roughly 5-11% compared to the standard AIPW estimator based on experimental data only. The gains especially pronounced in the small-sample setting (n = 100), where reducing variance is most critical. As expected, we observe that the PPCT estimator can be less precise than the standard AIPW estimator. This can be explained by noting that PPCT only guaranteed (asymptotically) to be at least as precise as the difference in means estimator (DM). Further, observe that the AIPW estimator using complex function class (boosting) suffers from very high variance, as the small sample sizes in the randomized experiments do not allow complex modeling choices. Lastly, while Theorem ensures that H-AIPW provides valid confidence intervals asymptotically, empirical results in Appendix B. confirm that all evaluated incluuunn
neww-maittain near-nominal coverage levels in finite samples.
Image treatments and contamination The study by Shuman et al. [45] is particularly relevant for two reasons. First, the data were published in December 2024, after the last known training cutoff for GPT-40, ensuring it was not included the model's training set. Second, the treatment is an image rather than text, allowing other foundation models do not support image inputs, we rely only on GPT-4o for outcome predictions in this study. Even so, H-AIPW achieves the lowest variance among all baselines, outperforming both PPCT and PROCOVA, suggesting that its gains over other approaches that integrate external models not only due the access multiple models.
to evaluate statistical framework beyond the text modality. Since the
5.2 Improving the accuracy of LLMs
We now study how two strategies for improving the accuracy of LLLMs-model scale and inference- time compute-affed the precision the H-AIPW estimator. In our setting, increasing inference-time compute boils down to presenting slight variations of the same prompt at inference time and averaging over the responses. We provide the complete list of the prompts used in Appendix C.2.9, For each LLM f. we evaluate prediction performance using the Mean Squared Error (MSE) on the full dataset: N E (f(X,, A,) Y2)2. Our findings indicate that larger models and increased inference-time compute improve prediction accuracy, which in turn can reduce the variance of H- AIP..

---

Model scale Figure 4a illustrates the precision gains achieved by H-AIPW when leveraging pre- dictions from LLMs of varying scales. We study the relationship between MSE and the estimate of the H- HTAW variance when integrating predictions from small models (LLaMA 3 8B, Gemma 2 9B, Phi-4, Gemma 2 27B) and large models (LLaMA 3 70B, GPT-4o, Gemini 1.5 Flash, DeepSeek-V3. Claude 3.5 Haiku, Grok 2). Large models consistently achieve lower MSE and thus lower variance than smaller models, with LLaMA 70B excelling despite having fewer parameters than GPT-4o and Claude 3,5 Haiku.
Inference-time compute Figure 4b shows that averaging over many prompts consistently reduces the MSE for the large models-4 similar trend expected for the smaller ones. As smaller MSE is associated with higher precision (see Figure 4a), using multiple prompts is expected to improve the precision of H- H-AIPW further. We confirm this observation in Appendix B.3, showing that H-AIPW precision improves with more prompts across several randomized studies.
6 Conclusion
We introduce H-AIPW, novel estimator that improve the efficiency of randomized experiments by integrating predictions from multiple foundation models. Our empirical results on social science data demonstrate that H-AIPW improves precision, especially in sample-constrained settings, without compromising validity of the inference. This approach holds significant promise in fields such as medicine, where leveraging well-curated foundation models could substantially lower the costs of clinical trials. However, a key limitation of H- H-AIPW is its reliance on the underlying foundation models: achieving meaningful gains in precision requires these models be accurate and well-aligned with the experimental domain of interest. Further, finite-sample covariance estimation can be unstable when the number of models is large relative to the sample size, which may lead undercoverage.
Acknowledgments and Disclosure of Funding
PDB was supported by the Hasler Foundation grant number 21050 and the Ermenegildo Zegna Founder's Scholarship. JA was supported by the ETH AI Center. KD supported by the ETH AI Center and the ETH Foundations of Data Science. This work was supported in part by National Library of Medicine (NLM) award R01LM013611 National Heart, Lung, and Blood INHL)))teNHHLBB award RO1HL136708; and Patient-Centere Outcomes Research Institute (PCORI) award ME-2021C3 22365. The content is solely the responsibility of the authors and does not represent the official views of NLM, NHLBI, PCORI, PCORI's Board of Governors or PCORI's Methodology Committee.
References
[1] Anastasios Angelopoulos, Stephen Bates, Clara Fannjiang, Michael Jordan, and Tijana Zrnic. Prediction-powered inference. Science, 382(6671):669-6777 6744, 2023.
[2] Anastasios Angelopoulos, John Duchi, and Tijana Zrni. PPI++: Efficient prediction-powered inference. arXiv preprini (rXiv:23/1.0115:: 2023.
[3] Lisa Argyle, Ethan Busby, Nancy Fulda, Joshua Gubler, Christopher Rytting, and David Wingate. Out of one, many: Using language models to simulate human samples. Political Analysis, 31 (3):337-351, 2023.
[4] Ashwini Ashokkumar, Luke Hewitt, Isaias Ghezae, and Robb Willer. Predicting results of social science experiments using large language models. Technical report, 2024.
[5] Christopher Bail. Can generative ai improve social science? Proceedings of the National Academy of Sciences, 121(21):e214402113 2024.
[6] Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.0722 2021.
10

---

[7] Mark Brandt. Onset and offset deservingness: The case of home foreclosures. Political Psychology, 34(2):2212233 2013.
[8] David Broska, Michael Howes, and Austin van Loon. large language models (potentially) informative observations. Technical report, 2024.
The mixed subjects design: Treating
[9] Riccardo Cadei, Ilker Demirel, Piersilvio De Bartolomeis, Lukas Lindorfer, Sylvia Cremer, Cordelia Schmid, and Francesco Locatello. Causal lifting of neural representations: Zero-shot generalization for causal inferences. arXiv preprint arXiv:2502.000 06343, 2025.
[10] Peter Caprariello and Harry Reis. To do, have, to share? valuing experiences over material possessions depends on the involvement of others. Journal of personality and social psychology, 104(2):199, 2013.
[11] Benjamin Carlisle, Jonathan Kimmelman, Tim Ramsay, and Nathalie MacKinnon. Unsuccessful trial accrual and human subjects protections: empirical analysis of recently closed trials. Clinical trials, 12(1))77-83, 2015.
[12] Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21 2018.
[13] Issa Dahabreh, Anthony Matthews, Jon Steingrimsson, Daniel Scharfstein, and Elizabeth Stuart. Using trial and observational data assess effectiveness: trial emulation, transportability. benchmarking, and joint analysis. Epidemiologic reviews, 46(1):1-16, 2024.
[14] Issa Dahabreh, Robert Yeh, and Piersilvio De Bartolomeis. Trial emulation, simulation, and augmentation using electronic health records and generative ai. NEJM AI, 2025.
[15] Lauren Eyler Dang, Jens Magelund Tarp. Trine Julie Abrahamsen, Kajsa Kvist, John B Buse, Maya Petersen, and Mark der Laan. A cross- validated targeted maximum likelihood estimator for aata-adeetive experiment selection applied to the augmentation of ret control arms with external data. arXiv preprint arXiv:22 2022.
[16] Piersilvio De Bartolomeis, Javier Abad, Konstantin Donhauser, and Fanny Yang. Hidden yet quantifiable: A lower bound for confounding strength using randomized trials. International Conference on Artificial Intelligence and Statistics, 2024.
[17] Piersilvio De Bartolomeis, Javier Abad, Konstantin Donhauser, and Fanny Yang. Detecting critical treatment effect bias in small subgroups. Uncertainty in Artificial Intelligence, 2024.
[18] Ilker Demirel. Ahmed Alaa, Anthony Philippakis. and David Sontag. Prediction- generalization of causal inferences. International Conference on Machine Learning, 2024.
[19] Nikita Dhawan, Leonardo Cotta, Karen Ullrich, Rahul Krishnan, and Chris Maddison. End- to-end causal effect estimation from unstructured natural language data. arXiv preprint arXiv:2407.07018, 2024.
[20] Florian Dorner, Vivian Nastl, and Moritz Hardt. Limits to scalable evaluation at the frontier: Llm as judge won't 't beat twice the data. arXiv preprint arXiv:2410.13:2 2024.
[21] Naoki Egami, Musashi Hinck, Brandon Stewart, and Hanying Wei. Using imperfect surrogates for downstream inference: Design-based supervised learning for social science applications of large language models. Advances in Neural Information Processing Systems, 2024.
[22] James Fahey, Damon Roberts, and Stephen Utych. Principled or partisan? the effect of cancel culture framings on support for free speech. American Politics Research, 51(1):69-75, 2023.
[23] FDA. Adjusting for covariates in randomized clinical trials for drugs and biological products. 2021.
[24] Adam Fisch, Joshua Maynez, Alex Hofer, Bhuwan Dhingra, Amir Globerson, and William Cohen. Stratified prediction-poue powered inference for hybrid language model evaluation. arXiv preprint 24K3:22006.0229, 2024.
11

---

[25] Johann Bognon-Bartsele Adam Sales, Edward Wu, Anthony Botelho, John Erickson, Luke Miratrix, and Neil Heffernan. Precise unbiased estimation in randomized experiments using auxiliary observational data. Journal of Causal Inference, 11(1):2220011, 2023.
[26] Javier González, Cliff Wong, Zelalem Gero, Jass Bagga, Risa Ueno, Isabel Chien, Eduard Oravkin, Emre Kiciman, Aditya Nori, Roshanthi Weerasinghe, et al. Trialscope: a unifying causal framework for scaling real-world evidence generation with biomedical language models. arXiv preprint arXiv:2311.01 2023.
[27] Igor Grossmann, Matthew Feinberg, Dawn Parker, Nicholas Christakis, Philip Tetlock, and William Cunningham. Ai and the transformation of social science research. Science, 380(6650); 1108-1109, 2023.
[28] Ingar Haaland and Christopher Roth. Beliefs about racial discrimination and support for pro-black policies. Review of Economics and Statistics, 105(1):40-55) 53, 2023.
[29] Zeshan Hussain, MingChich Shih, Michael Oberst, Ilker Demirel, and David Sontag. Falsifica- tion of internal and external validity in observational studies via conditional moment restrictions. International Conference on Artificial Intelligence and Statistics, 2023.
[30] Wenlong Ji, Lihua Lei, and Tijana Zmic. Predictions as surrogates: Revisiting surrogate outcomes in the age of ai. arXiv preprint arXiv:2501.09733 2025.
[31] Nathan Kallus, Aahlad Manas Puli, and Uri Shalit. Removing hidden confounding by experi- mental grounding, Advances in Neural Information Processing Systems, 31, 2018.
[32] Rickard Karlsson, Guanbo Wang. Piersilvio De Bartolomeis, Jesse Krijthe, and Issa Da- habreh. Robust integration of external control data in randomized trials. arXiv preprint arXiv:2406. 17971, 2024.
[33] Emily Kennedy and Christine Horne. Accidental environmentalist ethical elite? the moral dimensions of environmental impact. Poetics, 82:101448, 2020.
[34] Frédéric Lavancier and Paul Rochet. A general procedure to combine estimators. Computational Statistics & Data Analysis, 94:175-192, 2016.
[35] Lauren Liao, Emilie Hojbjerre-Frandsen, Frandsen, Alan Hubbard, and Alejandro Schuler. Prognostic adjustment with efficient estimators to unbiasedly leverage historical data in randomized trials. arXiv preprint arXiv:2305. /9/80, 2023.
[36] Xi Lin, Jens Magelund Tarp, and Robin Evans. Data fusion for efficiency gain in atc estimation: practical review with simulations. arXiv preprint 2rXIv:2407.01116. 2024.
[37] Alex Luedtke, Marco Carone, and Mark van der Laan. An omnibus non-parametric test of equality in distribution for unknown functions. Journal of the Royal Statistical Society Series B: Statistical Methodology, 81(1):75-99, 2019.
[38] Pranav Mani, Peng Xu, Zachary Lipton, and Michael Oberst. No free lunch: aoymasymioo analysis of prediction-powerte inference. arXiv preprint 250.::::505220178 2025.
[39] Julia Lee Melin and Jemnifer M Merluzzi. When women do "men's work": Hybrid femininity and within-gender inequality in job search. Academy of Management Proceedings, 2022.
[40] Bearneem Poulet, Maylis Tran, Sophie Tezenas du Montcel, Bruno Dubois, Stanley Durrleman, and Bruno Jedynak. Predietion-poweree inference for clinical trials. medRxiv, 2025.
[41] James Robins and Andrea Rotnitzky. Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association, 90(429): 1995.
[42] James Robins, Andrea Rotnitzky, and Lue Zhao. Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association, 89 8427):846-866, 1994.
12

---

[43] Evan Rosenman, Guillaume Basse, Art Owen, and Mike Baiocchi. Combining observational and experimental datasets using shrinkage estimators. Biometries, 79(4):2961-2977, 2023,
[44] Alejandro Schuler, David Walsh, Diana Hall, Jon Walsh, and Charles Fisher. Increasing the efficiency of randomized trial estimates via linear adjustment for prognostic score. The International Journal of Biostaristics, (8(2)329-3350 2022.
[45] Eric Shuman, Martijn van Zomeren, Tamar Saguy, Eric Knowles, and Eran Halperin. Defend, deny, distance, and dismantle: A new measure of advantaged identity management. Personality and Social Psychology Bulletin, 2024.
[46] Daniel Silverman, Daniel Kent, and Christopher Gelpi. Putting terror in its place: An experiment on mitigating fears of terrorism among the american public. Journal of Conflict Resolution, 66 (2):191-215) 216,, 2022.
[47] Anastasios Tsiatis. Semiparametric theory and missing data. Springer, 2006.
[48] Mark van der Laan, Sky Qiu, Jens Magelund Tarp, and Lars van der Laan. Adaptive-tmie for the average treatment effect based on randomized controlled trial augmented with real-world data. arXiv preprint 205.:240 2024.
[49] G Villacampa, S Dennett, E Mello, J Holton, X Lai, Lucy Kilbur,, J Bliss, Rekowski, and Yap. Acerual and statistical power failure in published adjuvant phase iii oncology trials: a comprehensive analysis from 2013 2023. ESMO open, 9(7):103603, 2024.
[50] Stefan Wager. Causal inference: statistical learning approach. Technical report, Stanford University, 2024.
[51] Zichun Xu, Daniela Witten, and Ali Shojaie. A unified framework for semiparamettrcally efficient sepi-super learning, arXiv preprint arXiv:250..17711, 2025.
13

---

Appendices
The following appendices provide deferred proofs, ablation studies, and experimental details.
Table of contents
A Methodology
15
A.1 Proofs
15
A.1.1 Proof of Proposition
15
A.1.2 Proof of Theorem
16
A.2 Connection with cnndictiio-powerre inference
17
A.3 Dependency of the variance term on the estimation error | h" (]))
18
Additional experiments
19
B.1 Empirical evaluation of coverage probability..
19
B.2 Impact of adding more foundation models on statistical precision
19
B.3 Impact of inference- -ime compute on statistical precision
20
Experimental details
22
C.1
Implementation details
22
C.2 Preprocessing of scientific studies and prompt design
22
C.2.1 Can Factual Misperceptions be Corrected? An Experiment on American Public Fears of Terrorism [46]
22
C.2.2
Cancel Culture for Friends, Consequence Culture for Enemies: The Effects of Ideological Congruence on Perceptions of Free Speech [22]
23
C.2.3
Beliefs about Racial Discrimination [28]
25
C.2.4
Accidental Environmentalists: Examining the Effect of Income Positive
Social Evaluations of Environnemaall--Friendly Lifestyles [33]
26
C.2.5 To Do, to Have, to Share? Valuing Experiences and Material Possessions
by Involving Others [10]
27
C.2.6 Onset and Offset Controllability in Perceptions and Reactions to Home Mortgage Foreclosures [7]
28
C.2.7 Testing Theory of Hybrid Femininity [39]
30
C.2.8 Understanding White Identity Management in a Changing America [45]
C.2.9 Introducing variability in multi-prompt experiments
32
14

---

A Methodology
A.1 Proofs
A.1.1 Proof of Proposition
We adapt here a classic result from the semiparametric inference literature to our specific setting where the probability of treatment is known by design. For clarity, refer to 0xP as
Let us define the summand of the AIPW estimator for fixed function h as:
4/s (h) = A (Y; ix,; 1)) 1(x..1)
- A (Y; h(x,, 0)) M(x;;)))
To
We can then decompose the estimation error the AIPW estimator as follows:
vn(((h) -0 = - - - ((h))
=
=
The first term, Ti. is an average of i.i.d. random variables with mean zero and finite variance. Therefore, by the Central Limit Theorem, we have:
vn(((h)) 0) vn (:!h! -0) ~ ~NN(0, Vi+),
where the asymptotic variance is given by Vn+ (l(ts(ht) - 7)
Bounding the remainder term We need to show that the second term T2 is asymptotically negligible, that TV = Op. (1).
We can rewrite this term as:
T3 -n(((h)-
(1)(h) - oi(tt))
vi
Further, with some simple algebra we can decompose the difference in the influence functions as:
A; T
Σ (w:(h) - bb((t))
(h 1),(x,,)),,,11))) 1))
Vn
T
1=1
A 9ι
1- T
Now, we will show that both terms in the sum above are asymptotically negligible. We focus our proof on the first term; the second follows from symmetric arguments.
Let Z, = (Xa, As, Yi) and Pr, denote the empirical measure over Zi...., Zn. and define the following functions:
f(Z.) A:
A π h(X,, 1).
(t,,,, 1) and f(Z,)
π
π
We can rewrite the first term as:
A πl
-x(x
π
where we use the fact that P(J î 0. since the treatment probability is known. Given that h is estimated from an independent sample, conditioning on the variables (f(2,) f(Z,) are i.idd with mean 0 and variance |l fl2 (m) Hence, by Chebyshev,
(P P) - /) =0= Oz- [15 flle,18) Vn
Op-
where it follows from assumptions that oiF-flleute op- (1). Therefore, have that Th Op- (1).
15

---

A.1.2 Proof of Theorem 2
and define the oracle weights as arg min A Σ'A. The corresponding oracle estimator is then
Recall that
AEA
02- 11o (h) Σ 1 (fz).
We now prove the theorem in the following three steps.
First, we observe that 0x can also be written
(2th+
I. 19h + Exitids
02. = @xIPW
j=1
since the constraint set E ΣS X = 1}. Further, it follows from assumptions that
24 Ajt1fs is also an outcome function estimator that satisfies the conditions in Proposition therefore Bx- is consistent and asymptotically normal, i.e. it holds that
vn - 0) ~ (O, Vx+), where Vi- = E
Second, we show that the asymptotic variance V. satisfies
VA Eggs for j = 1,............... b
By construction, the oracle weights X minimize X ΣA, ensuring 0x- attains the smallest asymptotic variance among all convex combinations of the initial estimators:
0x 11 (h) +
j=1
Moreover, since A is defined as the minimizer of AA subject to 1 T = 1, it holds that for any canonical vector ej g (which corresponds using the jth estimator alone) have
ΣA
V = x ΣX < Se, = Enj for j = 1, A +
Thus, the asymptotic variance of the hybrid estimator is larger than that of any individual estimator.
Third, we observe that 0z and 0x are asymptotically equivalent. Since || Ellop 2, 0 and Σ is nonsingular, the continuous mapping theorem implies
E''1 B
E-1
X
1T
1-w-1
Finally, using Slutsky's 's theorem, we get:
Vn(Øz - 0) vn(0,- 0) + op- (1) ~ N'0, Vx-)
which completes the proof.
16

---

A.2 Connection with prediction-powered inference
To further study the connection and differences with cnndiction-poweree inference (PPI) [1], it is instructive to consider the simpler problem of estimating the counterfactual mean, E|Y (1)* For this case, variant of PPI, referred to as PPI++ [2], can be shown to be equivalent to AIPW estimator.
The standard difference in mean estimator is the sample mean of outcomes for the treated group:
PpM
Σ Y,, where naa = Σ 1{A=aa
#A,=1
PPI++ improves the difference in mean estimator by using predictions from a black-box model f:
( Σ
(
0pm++
Σ Y
::A==1
:AA==
:A,=0
where the power-tuning parameter chosen to minimize the variance. Crucially, for assuming exact randomization, i.e. = m/nn, we have equivalence with the AIPW estimator for the counterfactual mean,
np
+n+
A,(Y, - f(X.)) + +f(x,)
8 p
=QXIPw (f)
i=1
few remarks are in order.
PPI++ replaces the estimated outcome regression with a black-box model f. However, when I is not equivalent to the outcome regression IE[Y X, A 1], the resulting estimator will not be efficient. In other words, 0pm++ will not achieve the smallest asymptotic variance among the regular estimators of the counterfactual mean. Concurrent work by Ji et al. [30] similarly identifies this limitation and proposes a recalibrated version of PPI overcome it. By contrast, the AIPW estimator will achieve the smallest possible asymptotic variance, assuming that the outcome regression estimator consistent Lo-norm. This condition is easier to satisfy in the setting of randomized experiments, since we can use flexible machine-learning models and still have valid inference as consequence of Proposition In particular, our H-AIPW estimator is guaranteed to have asymptotic variance no greater than the standard AIPW estimator (Theorem 2), and thus can be efficient even if the black-box model f is arbitrarily biased.
Extending PPI and PPI++ to average treatment effect estimation straightforward. To do so, Poulet et al. [40] proposes the following estimator:
Opper =
Σ ((--((()))) -Y A:=1
A.=0
However, a key limitation of the above estimator that it forces both outcome regressions, that is E[Y X = x, A = 1] and IE[Y X = A = OJ, to be replaced with the same black-box model f. This is particularly problematic when the treatment has a significant effect on the outcome, as single model f will fail to accurately capture both outcome regressions. In contrast, our approach allows for different black-box models and fo be plugged-in for the treated and control group, respectively.
PPI and its variants cannot integrate multiple competing foundation models. This is key limitation in the causal inference setting, as model selection is a non-trivial task due to the missingness of potential outcomes. Moreover, it is unclear whether they be extended to do so, as constructing a consistent estimate of the covariance matrix Σ poses a major hurdle. In contrast, our approach offers a simple way to estimate the covariance matrix Σ by exploiting the linear structure of the AIPW estimators.
We refer the reader Xu et al. [51] for discussion of the connections between AIPW and PPI.
17

---

A.3 Dependency of the variance term on the estimation error [[h h- (12)(PP)
As mentioned in Section 3.1, in small sample regimes the variance of the AIPw estimator crucially depends on the estimation error [] h* 1 (P)) For completeness, we formalize here this dependency by bounding the excess variance of the AIPW estimator that arises from using i instead of h*
Lemma 1. For any outcome regression h estimated from an independent sample, we have
[
Var( v OAr (hi) Var(y OxIOW (h*)) = E
T mp (MxX.0) - -xx,0))) + (x(X.1)-
And thus, it holds that
Var( / xxPP (h)) Vark /n xxw (h')) - |β[ 0) (). (.. (.
Proof of Lemma 1. Note that by the unbiasedness of the AIPW estimator, as well the independence of the samples used to compute the outcome regression ît the excess variance equals:
n. Var (DAIPw (M) - n. Var (DAIww (h*)) = E (w) 0)2 (vb(h*) - 0)²
E
2 (1b(h") 0) + Aub
with Ao (hh) *b(h)). We bound the terms Ti and T2 separately, Recall that by definition,
(h) =
(Y h(X, 1)) + M(X.1)
-0- (1-
(Y - hox,h + M(x.0))
and thus,
A Aho (1-1-A) with (x,(X) = h((x))))))))))))) - -
1
To
which does not depend on Y. Hence, taking the expectation first over Y for T1 yields:
Ti E ( (h*(X,1) ("(X,)) 0)], where we used the fact that E X that Ti = O.
X. A] h² (x, A). Finally, since |A bb X] = 0, also obtain
Next, bound the second term T2, we can write:
A
1
A
-2Ah Aho (1-
(1-1 1-1-A)
x0 ))
(1
A
T:
Ah?
1
An
T
To
straightforward computation (using Mh
To) yields:
27
= E Ab? (
1- :T -1)
E
A
27
and
E
Ahg
= E Ah
-
and
E
22 Aho
11
1-11 A
(1
}]
2E Ahh
As result, we obtain:
(V1-m
)
27
T2 = E
no
Noo +
To Abt
which completes the proof.
18

---

Table 2: Coverage probability comparison of H-AIPW against baseline estimators (PPCT, DM, AIPW, PROCOVA) across several randomized experiments. We report the empirical coverage of each estimator's 1 a confidence interval. The coverage probability is averaged over H 10000 subsampling repetitions
sample sizes 100 and n 200. The nominal level is set at a = 0.05. We implement H-AIPW by integrating predictions from three M.MMs: GPT-4o, Claude 3.5 Haiku, and I.L.MA 3 70B.
Melin et al. (2022)
Silverman et al. (2022)
Kennedy et al. (2020)
Fahey et al. (2023)
Estimator
100
200
100
200
100
200
100
200
H-AIPW
96.2
98.4
96.4
98.4
94.4
95.9
94.5
95.6
PPCT
96.4
98.4
96.7
98.7
95.0
96.5
94.9
95.7
PROcOVA
96.4
98.4
96.4
97.7
94.9
96.2
95.2
95.8
AIPW (standard)
96.5
98.5
96.3
97.7
94.8
96.1
95.3
96.0
AIPW (boosting)
95.6
97.2
95.6
96.6
94.4
95.0
94.1
94.6
DM
96.6
98.5
96.9
98.7
95.4
96.5
95.8
96.7
Caprariello et al. (2013)
Brandt (2013)
Haaland al. (2023)
Shuman et al. (2024)
Estimator
100
200
100
200
100
200
100
200
H-AIPW
96.8
99.3
95.7
97.2
95.0
96.2
95.3
96.2
PPCT
97.2
99.3
96.1
97.7
95.0
96.3
94.9
96.4
PROCOVA
97.5
99.4
96.3
97.6
95.2
96.2
95.0
95.8
AIPW (standard)
97.6
99.4
96.3
97.4
95.2
96.4
95.1
96.1
AIpW (boosting)
96.7
98.6
95.0
94.3
95.4
93.4
DM
97.4
99.4
96.6
98.1
95.3
96.3
95.5
96.5
B
Additional experiments
We present here additional ablations of method. The results reinforce the general trends observed in the main experiments: H-AIPW achieves better precision than the baselines while maintaining comparable coverage. Ablation studies provide insight into the number of models that be incorporated into our estimator without significantly compromising validity (due to finite sample effects), and they offer further evidence of the advantages of increasing inference-time compute.
B.1 Empirical evaluation of coverage probability
To benchmark validity, for each estimator, compute the fraction of confidence intervals containing the average treatment effect:
1 Coverage R
Σ 1
where C. is the confidence interval obtained from the dataset D, and 0 is the difference in means necessarily the true ATE, serves as the
ATE estimate from the full study dataset. While this best available proxy in the context of real randomized experiments. Table 2 shows that H- H-AIPW consistently achieves coverage probability close the nominal 95% level across all studies and both sample sizes. Importantly, this indicates that the variance reductions observed in Table do not come the expense of statistical validity.
B.2 Impact of adding more foundation models statistical precision
In this section, we study the impact increasing the number of models in H-AIPW. Specifically, Algorithm requires integrating predictions from multiple foundation models, which are combined with the standard AIPW to minimize the variance of the resulting estimator. In Figure 5, we show how increasing the number of language models from 1 to 7 affects the precision and validity of H-AIPW in the study by Fahey al. [22]. Models are incorporated in the estimator sequentially, starting from those with the lowest mean squared error (MSE) (i.c. LLaMA 3 70B) those with the
19

---

AIPW
H-AIPW (k=1)
H-AIPW (k=7)
Coverage probability
5.2
100%
Variance
4.8
95%
4.6
4.4
90%
4.2
85%
50
100
150
200
50
100 150
200
Sample Size
Sample Size
Figure 5: Impact of increasing the number models in H-AIPW on precision and validity in study by Fahey et al. [22]. Models are sequentially incorporated based on their mean squared error (MSE), starting with LLaMA 70B (lightest red, & = 1) and ending with Gemma 2 27B (darkest red, l = 7). following Figure 4a. The left panel shows the empirical variance, while the right panel shows empirical coverage. The standard AIpw estimator included reference. Each experiment is averaged over R 10k repetitions, with significance level set to a 0.05.
highest (stopping at Gemma 2 27B), following Figure 4a. We also include the standard AIPW (linear) estimator for reference.
Increasing the number of models improves precision compared to the standard AIPw estimator. In the setting with 50 samples, a single model improves variance by approximately 6%, while using 4 models increases this gain to nearly 12%, and 7 models yield an improvement of around 16%. However, the marginal benefits diminish with larger sample sizes: 200 observations, the variance difference between using and models shrinks to 4%. However, adding more models weakens empirical coverage. With 50 samples, combinations of 2%-4% relative to AIPW, failing to reach the nominal 95% coverage until the sample size reaches 200. contrast, combinations of to models maintain coverage levels comparable to AIPW.
models exhibit undercoverage of
Intuitively, the undercoverage observed in Figure 5 is driven by finite-sample error in estimating the covariance matrix. As the number of models increases, the covariance matrix is harder to estimate reliably from limited data. This, in turn, leads to a systematic underestimation of the combined estimator's 's variance and hence to undercoverage. Such effects have been formally proven in recent "no free lunch" results on prediction-powered inference [38]. A simple remedy to use sample- splitting or cross- cross-fitting when estimating the covariance matrix, ensuring that the same data are not reused for both covariance matrix estimation and inference.
Practitioners should therefore carefully determine both the number of models to include in the ensem- ble and whether to adopt sample-splitiing or cross-fitting. In our experiments, with moderately large samples (n 100 and 200) and only three outcome models, did not observe undercoverage, suggesting that the additional estimation error is negligible and standard plug-in inference is sufficient. By contrast, with smaller samples ensembles of several models, the covariance matrix's estimation error becomes non-negligible, and cross-fitting is recommended to ensure valid coverage.
B.3 Impact of inference-time compute on statistical precision
In Section 5.2, we showed that increasing inference- compute improves the precision H-AIPW: more prompts generally reduce mean squared error (MSE) of the foundation model predictions, which in turn lowers the estimator's variance. For completeness, Figure visualizes the relationship between the number of prompts, MSE, and variance.
We present results for three studies-Branne [7], Silverman et al. [46], Kennedy and Home [33] using H-A H-AIPW with predictions from GPT-4o. Figures 6a 66 show the empirical estimate of variance as function of the number of prompts, while Figures 6d to 6f illustrate the corresponding
20

---

Psychology (Brand: 2013)
Foreign (Silverrmn et al., 2022)
Socialogy (Kennedy et al. 2020)
12.75
2.11
17.5
Variacce
12.7
17.4
2.09
17.3
12.6
12:55
2.08
17.2
12.5
2.07
17
(a)
(b)
(c)
4.7
7.5
0.74
Meen Squared
4.65
0.68
0.6
4.55
4.5
Number of prompts
Number of prompts
Number of prompts
(d)
(e)
Figure 6: Impact of the number of prompts on the empirical variance and MSE. Results are reported for studies by Brandt [7]. Silverman et al. [46], Kennedy and Horne [33]. We randomly subsample each study to obtain sample size 50 and report the average over A 10% repetitions for each metric. (First row) Reduction variance the number of prompts increases. (Second row) Reduction in MSE the number of prompts increase. These results suggest that increasing inference-time compute improves the precision H-A1PW by reducing the MSE of the foundation model predictions.
changes MSE. The findings reinforce the conclusions from the main text: increasing inference-time compute through multiple prompts generally reduces the variance of H-AIPW,
21

---

C Experimental details
C.1 Implementation details
For all experiments, we first select the five features most correlated with the outcome variable. The AIPW estimator implements cross-fitting with 30 folds, using ridge regression with regularization X 1.0 in the standard case and XGBoost with default hyperparameters in the boosting case. For PPCT, we follow the implementation by Poulet et al. [40], using GPT-4o's predictions for the control scenario as the prognostic score. We implement PROCOVA using an AIPW estimator whose outcome regression estimator is augmented with a smart covariate, i.e. the prediction of GPT-4o for both arms. The coefficients for the optimal combination are computed using standard Python libraries. Finally, the DM estimator requires no hyperparameter tuning. We compute the ground-truth ATE for all studies using the DM estimator on the full study with sample size N.
Implementation of H-AIPW Our estimator integrates synthetic outcomes generated by multiple LLMs. Unless stated otherwise, we use predictions from LLaMA 3 70B, GPT-4o, and Claude 3.5 Haiku for all experiments in Section 5.1. Additional models, such as Gemma 2, Grok 2, and Gemini 1.5 Flash, are used in specific cases, e.g. Section 5.2. We leverage both proprietary and open-source LLMs. For open- source models, we apply nucleus sampling with temperature of 1.2, top-p of 0.9, and maximum of 100 new tokens. For proprietary models, use default decoding settings, except for Claude 3.5 Haiku, where we set the temperature to 1. In summary, H-AIPW extends the classic AIPw estimator by incorporating multiple AIpw estimators that integrate LLM predictions; see Algorithm for full details.
Reproducing Figure We randomly subsample each study with a sample size of n = 75 and compute the average confidence interval over IK repetitions using the standard AIPW estimator with linear regression. Then, we obtain AIPAIPw by progressively reducing n until the average confidence interval from H-AIPW (using GPT-40, LLaMa 3 70B, and Claude 3.5 Haiku) matches or exceeds the standard AIPw confidence interval. The percentage reduction in sample size then computed as:
100 (1
12H-APPW
C.2 Preprocessing of scientific studies and prompt design
In this section, we describe the preprocessing steps, selected outcomes, and control and treatment scenarios for the studies used in our experiments. We also provide an example prompt, including both system and user components, used to query the LLMs. The studies are sourced from the Time-sharing Experiments for the Social Sciences (TESS) repository, with findings published in peer-reviewed journals. These studies span various fields, demonstrating the versatility of our methodology.
C.2.1
Can Factual Misperceptions be Corrected? An Experiment on American Public Fears of Terrorism [46]
Abstract: An American's 's yearly chance of being killed by a terrorist attack sits at about I in 3.5 million. Yet over 40% of the American public consistently believes that they their family members are likely to be the victim of a terror attack. Can these inflated estimates the risks of terrorism be brought closer to reality? With trillions of dollars spent on the War Terror since 9/11, this question is not just theoretically but practically important. In order to investigate, we field a nationally representative survey experiment containing brief vignette with corrective information about the actual risks of terrorism vs. other dangers facing Americans. Additionally, we vary whether there political elite endorsement accompanying the information, with either Democratic politician, Republican politician, or senior military officer driving home the message.
Data availability: The study is publicly available at: https: tttps:://tessexpeiimeeete wtwperrinnnttttttttnnsrsoesoo org/study/ silverman1035
22

---

The primary outcome variable is Q5. The treatment condition is defined as P_TESS031 (corrective information). and the control condition is defined as P_TESS031 (no corrective information). The following variables are included as covariates: PARTYID7, IDEO, RELIG, ATTEND, GENDER, AGE, RACETHNICITY, EDUC4, INCOME. The final processed dataset contains n 503 observations.
Data pre-processing:
Prompting details: An example prompt provided below.
Example Prompt
System Prompt:
You are 33-year-old, ethnicity White, gender Male, strong Democrat, You hold very liberal views and college education. Additionally, your religion is Catholic, and you attend religious services nearly every week. Your household has a yearly income of $75,000 to $84,999. Your answer must be a single integer without additional text, in JSON format with key-value pair.
Treatment Condition:
The number of people who say that acts terrorism against Americans are imminent up 3% from last year, according new poll released this week. In the wake of attacks in San Bermardino, Orlando, Paris, and London, the Pew Research Center found that 63% of Americans think major terrorist attacks are likely to occur soon on American soil. Government officials have echoed these concerns. "We issuing new advisory that the terror threat is now elevated across the country," said Undersecretary for Homeland Security Stephen Krause. "We have to remain vigilant and we have stay alert. Terrorists can strike anytime, anywhere."
But does terrorism really pose critical threat to us? Below figure showing the average American's 's risk of death from different sources. As can be seen, around 90 Americans are killed each year by terrorism on U.S. soil. This means the risk of being victim of terrorism in given year is about in 3.5 million. In comparison, the risk of being killed by cancer 540, the risk of being killed in car accident is in 8,000, and the chance of being killed by your own home appliances is 1.5 million. These numbers provide some essential context when thinking about the different threats to public safety.
Control Condition:
The number of people who say that of terrorism against Americans are imminent is up 3% from last year, according to new poll released this week. In the wake of attacks in San Bernardino, Orlando, Paris, and London, the Pew Research Center found that 63% of Americans think major terrorist attacks are likely to occur soon on American soil. Government officials have echoed these concerns. "We are issuing new advisory that the terror threat is now elevated across the country," said Undersecretary for Homeland Security Stephen Krause. "We have to remain vigilant and we have to stay alert. Terrorists can strike anytime, anywhere."
Question:
How likely do you think it is that another terrorist attack causing large numbers of American lives to be lost will happen in the near future? Choose integer between 1 (very likely) and (not likely at all).
C.2.2 Cancel Culture for Friends, Consequence Culture for Enemies: The Effects of Ideological Congruence on Perceptions Free Speech [22]
Abstract: Political scientists have long been interested in the effects that media framings have on support tolerance for controversial speech. In recent years, the concept of cancel culture has
23

---

complicated our understanding of free speech. In particular, the modern Republican Party under Donald Trump has made "fighting cancel culture" cornerstone of its electoral strategy. We expect that when extremist groups invoke cancel culture as reason for their alleged censorship, support for their free speech rights among Republicans should increase. We use a nationally representative survey experiment to assess whether individuals' opposition to cancel culture is principled or contingent on the idcological identity of the speaker. We show that framing free speech restrictions as the consequence of cancel culture does not increase support for free speech among Republicans. Further, when left-wing groups utilize the cancel culture framing, Republicans become even less supportive of those groups' free speech rights.
Data availability: The study is publicly available at: study/fahey578
org/
Data pre-processing: The primary outcome variable is CC_1. The treatment condition is defined as P_ P_GROUP (safety reasons cancel culture), and the control condition as P_GROUP = 1 (safety reasons). The following variables are included covariates: PARTYID7, IDED, RELIG, ATTEND, GENDER, AGE, HOME_TYPE TYPE, INCOME. The final processed dataset contains n = 998 observations.
Prompting details: An example prompt provided below.
Example Prompt
System Prompt:
You are a 35-year-old male, politically Democrat, holding liberal views. Additionally, your religion is Christianity, and you attend religious services. You reside in a building with two or more apartments, and your household has a yearly income of $85,000 $99,999, You are responding to a scenario reflecting a debate involving college campus events and broader social issues.
twice month
Treatment Condition:
We are now going to ask you to imagine you have read about the following scenario, describing debate on a recent College Campus.
Local Group Denied Permit to Protest on Campus, Provoking Debate
bouu "Cancel Culture"
A debate on the merits of free speech erupted recently when the student chapter of the controversial far-left group Antifa attempted to obtain a permit conduct demonstration
the main quad of Rutgers University in New Jersey. Citing safety concerns, the president of the organization charge Registered Student Organizations (RSOs) initially denied the organization the right to conduct their rally, arguing that their presence would endanger college students. They cited a recent incident in Berkeley, CA where three Antifa members and two bystanders were injured by rocks thrown in an altercation between the group and counter protesters. A member of the local Antifa group. Luke Vargas, is appealing the decision, arguing that the permit denial represented "cancel culture amok," and the University was simply "afraid to hear the truth." When asked to comment, the University Ombudsman's ss Office promised that final decision on whether the rally would be permitted would be made by this Thursday, three days before the march is scheduled to take place on Sunday.
Control Condition:
We are now going to ask you to imagine you have read about the following scenario, describing debate on a recent College Campus.
Local Group Denied Permit to Protest Campus
A debate on the merits of free speech erupted recently when the student chapter of the controversial far-left group Antifa attempted to obtain a permit to conduct a demonstration on the main quad of Rutgers University in New Jersey. Citing safety concerns, the president of the organization in charge of
24

---

Registered Student Organizations (RSOs) initially denied the organization the right to conduct their rally, arguing that their presence would endanger college students. They cited a recent incident in Berkeley, CA where three Antifa members and two bystanders were injured by rocks thrown in an altercation between the group and counter protesters. A member of the local Antifa group. Luke Vargas, promised to bring an appeal to the desk of the University President. When asked to comment, the University Ombudsman's Office promised that a final decision on whether the rally would be permitted would be made by this Thursday, three days before the march is scheduled take place on Sunday.
Question:
Generally speaking, do you agree or disagree with the following statement: "Cancel culture is a big problem in today's society." Reply using numbers between (definitely agree) and 5 (definitely disagree).
C.2.3 Beliefss about Racial Discrimination [28]
This paper provides representative evidence on beliefs about racial discrimination and examines whether information causally affects support for pro-blact policies. Eliciting quantitative beliefs about the extent of hiring discrimination against blacks, we uncover large disagreement about the extent of racial discrimination with particularly pronounced partisan differences. An information treatment leads to convergence in beliefs about racial discrimination but does not lead to a similar convergence in support of pro-black policies. The results demonstrate that while providing information can substantially reduce disagreement about the extent of racial discrimination, is not sufficient to reduce disagreement about pro-black policies.
Abstract:
Data availability: The study is publicly available at: https: https://www.cr http://wwwwteeeepppeeetttttttss org/ study/Haaland874
The primary outcome variable Q2. The treatment condition is defined as GROUP 1 (statistics of white-sounding and black-sounding names), and the control condition is defined as GROUP (statistics white-sounding names). The following variables are included as covariates: PartyID7, INCOME, ATTEND, RELIG, GENDER, AGE, REGION9, RACETHNICITY. The final processed dataset contains n 1539 observations.
Data pre-processing:
Prompting details: An example prompt provided below.
Example Prompt
System Prompt:
You are 60-year-ol old, politically Independent, gender Female, ethnicity Hispanic. Additionally, your religion is just Christian and you never attend religious services. You live in a state of the West South Central region. Your household has yearly income of $30,000 to $34,999. You are responding to a survey experiment collecting data on people's 's beliefs about racial discrimi- nation and whether these beliefs affect people's 's views affirmative action policies.
Treatment condition:
Researchers from Harvard University conducted experiment study racial discrimination in the labor market. They did so by sending out fictitious resumes help-wanted ads in Boston newspapers. The resumes were exactly the same except for one thing: the name of the job applicant. Half of the resumes had typically white-sounding names like "Carrie" and "Todd". The other half of the resumes had typically black sounding names like "Tanisha" and "Kareem". The idea was make sure that the applicants were seen
25

---

as having identical qualifications. but that the employers would use the ap- plicants' names to infer whether they were white or black. Resumes with white-sounding names had to be sent out on average 10 times to get one callback for an interview.
Further, the researchers found that resumes with black-sounding names on average had to be sent out times to get one callback for an interview. Since resumes with white-sounding names on average only had to be sent out 10 times to get one callback for an interview, this means that employers were 50 percent more likely give callbacks to applicants with white-sounding names compared to applicants with black-sounding names.
Control condition:
Researchers from Harvard University conducted experiment to study racial discrimination in the labor market. They did so by sending out fictitious resumes help-wanted ads in Boston newspapers. The resumes were exactly the same except for one thing: the name of the job applicant. Half of the resumes had typically white-sounding names like "Carrie" and "Todd". The other half of the resumes had typically black-sounding names like "Tanisha" and "Kareem". The idea was make sure that the applicants were seen as having identical qualifications, but that the employers would use the ap- plicants' names to infer whether they were white or black. Resumes with white-sounding names had to be sent out on average 10 times to get one callback for an interview.
Question:
In the United States today, do you think that racial discrimination against blacks in the labor market a serious problem? Reply with JSON numerical answer using one of these numbers: (A very serious problem), 2 (A serious problem), 3 (A problem), 4 (A small problem), or (Not problem at all).
C.2.4 Accidental Environmentalists: Examining the Effect of Income on Positive Social Evaluations of Emvironmentally-Friidde Lifestyles [33]
Abstract: Many US households have adopted behaviors aimed at reducing their environmental impact. Existing scholarship examines antecedent variables predicting engagement in these pro-environmental behaviors. But little research examines the effect of making efforts to reduce environmental impact on positive evaluations. Based on our qualitative pilot data, we suspect that income may be an important factor in the extent to which green lifestyles earn social approval. We predict that household that reduces its environmental impact will be viewed more positively if that household has a high (rather than low) income. We manipulate houschold income (high vs low) and proenvironmental behavior (green vs typical). We then measure participants' approval of the houschold, how socially close they feel to the household, well as their evaluations of the houschold's competence, morality, and environmental commitment. This research allows us to identify the bases for social approval of green lifestyles and examine how social approval for household's 's green lifestyle varies with that household's 's income.
Data availability: The study is publicly available https: hta.s/sppprreeetttttorrstt/tta org/study/ kennedy1017
The treatment condition is defined as P_ TESS23 4 (green lifestyle), and the control condition is defined P TESS23 (typical lifestyle). The following variables are included as covariates: PartyID7, IDEO, ATTEND, GENDER, AGE. The final processed dataset contains n 1276 observations.
Data pre-processing:
The primary outcome variable is Q5.
Prompting details: An example prompt is provided below.
26

---

Example Prompt
System Prompt:
You are 45-ycar-old, lean Democrat, gender Female, and hold slightly conservative views. Additionally, you attend religious services several times year. We are going to give you some information about family. Please read the information very carefully, as we will be asking you questions about it. Your answer must be in JSON format with single key-value pair.
Treatment condition:
A family with two children lives neighborhood nearby to yours. You chat with them sometimes when you them in the neighborhood. As far as you can tell, they make a huge amount of money and seem to have plenty of extra money to spend. Their house is small and they often take public transit or walk to avoid driving. They also dry their clothes on a clothesline and don't have air conditioning in their home. This family has a much lower environmental impact than other people in their neighborhood.
Control condition:
A family with two children lives neighborhood nearby yours. You chat with them sometimes when you see them in the neighborhood. As far as you can tell, they make very little money and seem have no extra money to spend. Their house is small and they often take public transit or walk to avoid driving. They also dry their clothes on a clothesline and don't 't have air conditioning in their home. This family has much lower environmental impact than other people in their neighhorhood.
Question:
How much is the environment high priority for this family? Choose integer between (not at all) and 11 (very much).
C.2.5
To Do, to Have, to Share? Valuing Experiences and Material Possessions by Involving Others [10]
Abstract:
Recent evidence indicates that spending discretionary money with the intention of acquiring life experiences-events that one lives through-makes people happier than spending money with the intention of acquiring material pessessions-tangible objects that one obtains and possesses. We propose and show that experiences are more likely to be shared with others, whereas material possessions are more prone to solitary and that this distinction may account for their differential effects on happiness. In 4 studies, we present evidence demonstrating that the inclusion others is key dimension of how people derive happiness from discretionary spending. These studies showed dimensions were considered simultaneously,
that when the social-solitary and experiential-materie social discretionary spending was favored over solitary discretionary spending, whereas experiences showed no happiness-produuing advantage relative to possessions. Furthermore, whereas spending money socially shared experiences valued enacted alone or material possessions, solitary experiences were no more valued than material possessions. Together, these results extend and clarify the basic findings of prior research and add to growing evidence that the social context of experiences is critical for their effects on happiness.
than spending money either experiences
Data availability: The study publicly available at: https: tttp//rww.tttteeeeriiiieeeetttss org/ study/caprarielll113
The primary outcome variable is Q7A. The treatment condition is defined as XTESS086 (spend money with people), and the control condition defined as XTESS086 (spend money alone). The following variables are included as covariates: XPARTY7, XREL1, XREL2, XIDEO, PPAGE, PPGENDER. The final processed dataset contains 397 observations.
Data pre-processing:
Prompting details: An example prompt is provided below.
27

---

Example Prompt
System Prompt:
You are 53-year-old, not so strong Republican, gender Male, and hold moderate views. Additionally. regarding religion you are Buddhist and you more than once week attend religious services. You are responding to a survey on how you spend your discretionary money. Your answer must be a single integer without additional text, in JSON format with key-value pair.
Treatment condition:
We are interested in ways you spend your discretionary money. Discre- tionary money refers money that is spent anything that is NOT essential to basic activity (that is, essentials refer to things like tuition and textbooks, groceries, transportation, rent, gas for car, health care, etc.). We'd like you to answer the questions that follow for money that you spent on something discretionary. Please think of the last time you spent least $10 (but no than $10,000) of your discretionary money in order TO DO SOMETHING WITH AT LEAST ONE OTHER PERSON. The primary focus of this expense should have been on an activity doing something with at least one other person and not on buying something that could be kept. Maybe you bought tickets to see movic with some people, maybe you paid to visit an art museum with friends, maybe you and some other people went to a spa together ... any of these would be legitimate examples of spending money to do something with others.
Control condition:
We are interested in ways you spend your discretionary money. Discre- tionary money refers to money that is spent on anything that is NOT essential to basic activity (that is, essentials refer to things like tuition and textbooks, groceries, transportation, rent, gas for a car, health care, etc.). We'd like you answer the questions that follow for money that you spent something discretionary. Please think of the last time you spent at least $10 (but no more than $10,000) of your discretionary money in order TO DO SOMETHING BY YOURSELF. The primary focus of this expense should have been on an activity doing something by yourself and not buying something that could be kept. Maybe you bought a ticket to see a movie by yourself, maybe you paid to enter of these would be legitimate examples of spending money to do something by yourself.
museum, maybe you went a spa by yourself ... any
Question:
Think about the last time you used your possession. To what extent did it help you feel loved and cared about? Reply with JSON numerical answer using one of these numbers: I (not at all), 2 (slightly). 3 (moderately), 4 (very), or (extremely).
C.2.6 Onset and Offset Controllability in Perceptions and Reactions to Home Mortgage Foreclosures [7]
The circumstances and rhetoric surrounding home foreclosures provide an ideal and timely backdrop for an extension of research on attributional judgments. While people face foreclosure for many reasons, the current debate surrounding the mortgage crisis has highlighted reasons that are either onset or offset controllable; that is, the imitial cause, or the subsequent solution may be seen as controllable.nn the current study, examine how people use attributional evidence from multiple time points to determine affective reactions and helping intentions for people undergoing foreclosure, as well as ideological differences in these attributional processes. Participants read about people who were undergoing foreclosure for onset and offset controllable or uncontrollable reasons and then answer questions about their perceptions of these targets. The results suggested that both onset
Abstract:
28

---

longer and offset controllable information contributed to the emotional reactions and helping intentions of the participants with the participants experiencing more negative affect and less helping intentions when the target was in a controllable onset offset situation. Conservatives primarily relied on onset controllability information to decide who should receive government aid, while liberals updated their initial attributions with offset controllability information.
Data availability: The study publicly available at: https://www tessexper org/ study/brandt708
The primary outcome variable is Q7. The treatment condition is defined as XTESSO03 (family can afford mortgage), and the control condition is defined as XTESS003 2 (family might not afford the mortgage). The following variables are included as covariates: XPARTY7, XREL1, XREL2, PPAGE, PPGENDER. The final processed dataset contains n 624 observa- tions.
Data pre-processing:
Prompting details: An example prompt is provided below.
Example Prompt
System Prompt:
You are a 75- 75-year-old, not so strong Democrat, gender Female. Addition- ally, regarding religion you are Muslim and you once week attend religious services. You are responding to a survey on perceptions towards people who are facing foreclosure. Your answer must be single integer without additional text, in JSON format with a key-value pair.
Treatment condition:
Recently the growing number of home foreclosures has put strain on the financial system, which has weakened the United States economy. Foreclosure occurs when a person is behind on home mortgage payments to their bank and the bank decides repossess (i.e., take back) the home. People may go into foreclosure for a variety of reasons. We are interested in your perceptions towards people who facing foreclosure. In the following section you will be presented with a situation that describes some people facing foreclosure. Please carefully read the situation and answer the following questions about your reactions to the situation. Some people have a large monthly mortgage payment because they wanted to purchase larger house than they needed. Now they are facing foreclosure because they do not want to continue paying the mortgage, even though they are able afford the payments.
Control condition:
Recently the growing number of home foreclosures has put strain on the financial system, which has weakened the United States economy. Foreclosure occurs when person is behind on home mortgage payments to their bank and the bank decides repossess (i.c., take back) the home. People may go into foreclosure for a variety of reasons. We are interested in your perceptions towards people who are facing foreclosure. In the following section you will be presented with a situation that describes some people facing foreclosure. Please carefully read the situation and answer the following questions about your reactions to the situation. Some people have a large monthly mortgage payment because they wanted to purchase larger house than they needed. Now they are facing foreclosure because the primary income earner the household lost their job due to their company closing and they afford payments.
Question:
Do you strongly oppose or strongly support the following statement: The government should offer help (e.g., time, money, resources, etc.) in effort help people in this situation. Reply with an integer from (Strongly Oppose) to 7 (Strongly Support), where 4 Neutral stance.
29

---

C.2.7 Testing Theory of Hybrid Femininity [39]
Abstract: Although men experience advantages working in highly feminized occupations, they are commonly stigmatized as lesser men by outsiders- -he people they meet outside of their occupa- tions-for for doing "women's 's work." This experiment is designed to assess whether woman who has worked in hypermaseuline occupation would similarly be stigmatized as a lesser woman by workers outside of her hypermasculine occupation, or alternatively, whether she would viewed more favorably by such outsiders for doing "men's 's work." Specifically, this study aims to develop and empirically test a theory of hybrid femininity, which specifies the conditions under which hyper- masculinity signaled through occupation creates status and reward distinctions among women in external labor markets. The experiment asks respondents to provide recommended compensation and status ratings for a woman candidate while manipulating the gender-typing of her occupational history as well as her intended target job. By disentangling the underlying mechanisms driving these predicted status and reward differences, this study seeks to shed light on how gender inequality persists, even among women, through the privileging of masculinity over femininity, with important implications for the labor market and society at large.
Data availability: The study is publicly available at: http:://wwwttesserrrnnnnsss org/ study/melin1066
tessexperimentss
Data pre-processing: The primary outcome variable is Q7 1. The treatment condition is defined as P P_41 = 3 (applicant has experience in the Army), and the control condition is defined as P 41 6 (applicant has experience in the Cosmetics industry). The following variables included as covariates: P P_IDEO, P ATTEND, P RELIG, RELIG, GENDER, AGE, REGION9, RACETHNICITY, INCOME, P_PARTYID. The final processed dataset contains 545 observations.
Prompting details: An example prompt is provided below.
Example Prompt
System Prompt:
You are a 30-year-old, politically Independent, gender Male, ethnicity Híspanic. Your ideology is slightly liberal. Additionally, your religion is Protestant and you about once a month attend religious services. You live in a state of the Pacific region. Your houschold has a yearly income of $85,000 999, This task is part of a larger study on the design of Human Resources (HR) recruiting practices to pre-screen job applicants. Your answer must be a single integer without additional text, in JSON format with key-value pair.
Treatment condition:
Please imagine you work for a prominent management consulting company. You will provided with a job description and an applicant's 's résumé who is applying for a Senior Manager position. After thoroughly reviewing the job description and the applicant's résumé, you will be asked to provide your immediate and uncensored opinion. Job description for your review:
Senior Manager (Consulting) Responsible for: Leading high performance project teams across the organization Building professional relationships with key stakeholders Defining project objectives, roadmaps, and deliver- ables Aligning project tactics with project strategy for all new services The successful applicant will be hard-wor hard-working. results-oriented, and a team player. Required Qualifications: * Bachelor's 's degree in Business Administration related field 3-5 years of related experience Comfort with travel regionally globally (up to 30% of time) Self-motivated with potential for leadership Excellent communication skills Solid computer skills, including Microsoft software products
Applicant's 's résumé for your review:
Name: Amy Decker Motivated Project Manager with 5 years of experi- ence working in military and defense. Education: Rutgers University (New Brunswick, NJ), May 2017 (Graduated) B.A. in Business Administration,
30

---

GPA: 3.72/4.00 Work Experience: U.S. Anmy Project Manager (Active-duty Enlisted), 2014 Present Fort Dix Military Base (Fort Dix. NJ) Plan and track progress of entire life-cycle of military and defense projects. Build and maintain project plans, including actual and forecasted activities and timelines. Ensure project staffing and timely communications throughout project lifecycle. Identify and manage project risks. Skills and Interests: Computer: Proficient in Microsoft Office (including Word, Excel, Outlook, and PowerPoint). Interests: Running and traveling.
Control condition:
Please imagine you work for prominent management consulting company. You will be provided with a job description and an applicant's 's résumé who is applying for Senior Manager position. After thoroughly reviewing the job description and the applicant's résumé, you will be asked to provide your immediate and uncensored opinion. Job description for your review:
[Job description.
same as above]
Applicant's ' résumé for your review:
Name: Amy Decker Motivated Project Manager with 5 years of experi- ence working in military and defense. Education: Rutgers University (New Brunswick, NJ). May 2017 (Graduated) B.A. in Business Administration, GPA: 3.72/4.00 Work Experience Cosmetics Project Manager 2014 Present Precious Cosmeties (Lodi, NJ) Plan and track progress of entire life-cycle of cosmetics and beauty product projects. Build and maintain project plans, in- cluding actual and forecasted activities and timelines. Ensure project staffing and timely communications throughout project lifecycle. Identify and man- age project risks. Skills and Interests: Computer: Proficient in Microsoft Office (including Word, Excel, Outlook, and PowerPoint). Interests: Running and traveling.
Question:
On scale from "Not at all" to 7 "Fxtremely", to what extent do you perceive this applicant as MASCULINE.
C.2.8
Understanding White Identity Management in a Changing America [45]
This paper examines how White Americans manage their identity amidst societal shifts using a new measure of advantaged identity management, representative data (N = 2648), and latent profile analysis. The findings reveal five subgroups of White Americans, each managing their identity differently. Four profiles correspond to the main advantaged identity management strategies (defend, deny, distance, dismantle), with a fifth using strategies flexibly. Of predictions regarding how valuing hierarchy, meritocracy, and egalitarianism predict profile membership. 13 were supported. These profiles show contrasting attitudes toward social change, with defender-deniers opposing, denier-distaneers moderately opposing. distancers remaining neutral, and dismantlers supporting change. These findings provide some of the first empirical evidence for thcorized model of white identity management and suggest that how White Americans manage their identity has important implications for social change.
Abstract:
Data availability: The study publicly available at: https///www.tesseeeriinnnttssss org/ study/shuman1643
Data pre-processing:
The primary outcome variable is Q5D. The treatment condition defined as RND_00 01 = (disadvantage black people), and the control condition is defined RND __ (advantage white people). The following variables are included as covariates: AGE, GENDER, RACETHNICITY, EDUC5, REGION9, IDEO, PartyID7, RELIG, ATTEND, INCOME. The final processed dataset contains n = 1623 observations.
Prompting details: An example prompt provided below.
31

---

Example Prompt
System Prompt:
You are 41-year-old individual with gender Male, ethnicity Asian, and with Bachelor's degree education. You live in state of the New England region. You hold Moderate views and are not so strong Democrat, Additionally, your religion is Atheist and you attend religious services never. Your household has yearly income of $175,000 to $1 $199,999.
Treatment condition: The general purpose of this study is to examine the attitudes of people regarding social issues America today. You will now be presented with infographic:
DISADVANTAGES OF BEING BLACK IN AMERICA
LOWBR 40% 8.01871
EONER
WIIEE MEA TETAAAT
Gtu
Control condition: The general purpose of this study is to examine the attitudes of people regarding social issues America today. You will now be presented with an infographic:
ADVANTAGES OF BEING WHITE IN AMERICA
dtC
Question:
Rate the extent to which you agree with the following statement from I (STRONGLY DISAGREE) (STRONGLY AGREE): "There should be large scale criminal justice reform to address racial inequalities in the justice system." Your answer must in JSON format with single key-value pair.
C.2.9 Introducing variability in multi-prompt experiments
The user prompts described in the previous section include a final question or instruction sampled from a predefined pool to introduce variability in the of such instructions:
settings. Below some examples
"Consider all relevant factors and place this on the scale."
32

---

"Reflect on the scenario and use your reasoning to assign value."
Frro your understanding of the situation, quantify this feeling."
Given your insights and context described, provide your evaluation."
Wiith the provided details mind, rate your feeling the scale."
"Consider all the information and your perspective to choose suitable score."
EEvaluate the feeling here and align number with your reasoning."
Use the scale provided and your judgment to determine your feeling."
"Judge this scenario thoughtfully, considering the context and the details shared."
Reffecct the key aspects provided and numerically assess your feeling."
33

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract and introduction accurately describe our estimator that improves
precision of randomized experiments by integrating foundation model predictions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We clearly discuss limitations in the conclusion, particularly the reliance underlying foundation models' accuracy and alignment with the experimental domain.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
34

---

Answer: [Yes]
Justification: All theoretical results are stated with complete assumptions and detailed proofs are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The methodology section provides a detailed step-by-step procedure for implementing our approach, and experiment details are carefully documented.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
35

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code and data will be made available a public repository upon publication, with detailed instructions for reproducing all experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //tpps.ccppublleee wwtpccccppppcccccccccccccccccccc htalliicguideeee gewggeiiieeeeee /eeo//eeeeubiii ssi assionnolccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All experimental details including data splits, model specifications, and evaluation protocols are documented in the experiments section and appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
All experiments report confidence intervals.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
36

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Computational requirements are detailed in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpsss//arrrpp..ccpplll//////// www.tthisccclllllcccec ines?
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and our research fully complics with all guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The conclusion discusses both positive impacts (improving efficiency of clinical trials) and potential risks (reliance on potentially biased foundation models).
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
37

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Our research does not release high-risk models datasets that pose misuse concerns,
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets and foundation models used are properly cited with version information and license details the appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
38

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code implementation is thoroughly documented with usage instructions, and will be released with the paper.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsoureing or human subjects research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This research does not involve human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
39

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs are only used for minor writing edits.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //tprr//n.eu...eeeeeee cc//Conferennce//2255//11 for what should or should be described.
40


Reviews:
[{"review_id": "0VyUJ2VAl6", "reviewer": "Reviewer_X7Ms", "summary": "The authors introduce a method to combine randomized experimental design with predicted outcomes from foundation models to improve estimation efficiency. They leverage ideas from causal inference and proposes an Augmented Inverse Propensity Weighting (AIPW) estimator that integrates model-based predictions while retaining unbiasedness from randomization. In the paper, the authors demonstrate that this approach yields lower variance estimates of causal effects than the experimental data, while preserving consistency and asymptotic normality under standard assumptions. To support their metrics, the authors empirically validate the framework on societal experiment data and provide clear prompting processes.", "questions": "i) If possible, analyze robustness under misspecified predictive models: how much model error can the AIPW estimator tolerate before its benefits disappear? Consider adding experiments where the model is poor at generalizing to show limits of this method.\nii) Include cost analysis comparing the saved labeling effort vs. the compute cost of using foundation models.\niii) For more clarity, consider adding a schematic diagram illustrating the interaction between: experimental assignment, foundation model prediction, labeled outcomes and final estimator.", "limitations": "Clearly stated in the paper.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Overally a good paper and an interesting topic. \n\nStrengths:\n\nThe paper adapts classical AIPW estimators in a novel way to exploit predicted counterfactuals from foundation models, maintaining consistency while gaining efficiency, and therefore is a practical work. It shows both theoretically and empirically how the estimator reduces variance and performs well even with sparse outcome labels. It also tries to break down sources of variance and bias in the estimator, and the authors cleanly present the advantages of incorporating predictions.\n\nWeaknesses: \ni) Model Assumption Dependence: As the authors mentioned, the performance depends on the quality of the predictive model — in the domains where foundation models has weaker performance, discussion about workarounds is preferred. The paper does not formally analyze bias amplification under model errors, and doesn't provide adaptive design in this case.\n\nii) Although the paper discusses the impact of adding more foundation models, the cost of using large foundation models (e.g., running LLM inference on massive datasets) isn’t clearly addressed.", "was_revised": true, "final_justification": "I am happy about the authors' response during the rebuttal period. While I am not an expert about randomized experiments in the specific applications of this paper (clinical trials etc.), I lean to accept the paper as it shows clearly how the new estimator improve the experiment evaluation. After reading author's results about testing the robustness and model cost, I am satisfied about their arguments and want to maintain my score. I am not going to increase my rating as it is already not a borderline score and I don't feel confident to increase it.", "rebuttal": "We hope this response addresses your questions, but let us know if you have any additional concerns.\n\n**Model assumption dependence.** First, we want to clarify that our approach does not risk bias amplification under model errors. For example, even if a completely incorrect outcome model is used within the AIPW estimator, the estimator remains unbiased. However, variance reduction depends on model quality - when the predictive model is poor, our method offers reduced benefits. Indeed, we provide a simple result in Appendix A.3 (Lemma 1), showing that the variance depends on the l2-error of the model.\n\nWe provide here some additional experiments to understand when the benefits from our method (i.e. reduction in variance) disappear.  In particular, we ablate over the correlation between the external model predictions and the true outcome labels.\n\nWe simulated $n=100$ observations with $d=5$ Gaussian covariates $X\\sim N(0,I)$. Treatments were assigned at random with probability 0.5. The potential outcome functions are generated via\n$\ng_a(X_i)=  \\mathbb{I}  [a + \\beta^\\top X_i + 0.3 X_{i,0}X_{i,1} + 0.1 X_{i,2}^2 > 0 ].\n$\n\n\nWe compute the standard AIPW estimator (using a cross‑fitted logistic regression outcome model on the experimental data) and the H‑AIPW estimator, combining the standard AIPW with the external model obtained by flipping the ground truth outcomes to achieve the desired correlation.\n\n| ρ (correlation) | AIPW variance | H‑AIPW variance |\n| ---------------------- | ---------------------- | --------------- |\n| 0.2                    | 0.00295                | 0.00290         |\n| 0.5                    | 0.00302                | 0.00291         |\n| 0.7                    | 0.00302                | 0.00260         |\n| 0.9                    | 0.00302                | 0.00185         |\n\n\nAs expected, when the external model is highly correlated with the outcome, the variance of the H‑AIPW estimator is substantially lower than that of the baseline. For moderate correlation, the variance reduction is modest. When the external model is only weakly correlated, the H‑AIPW variance remains slightly lower than that of the baseline, demonstrating the estimator’s robustness. These results indicate that even with small to moderate correlation, our method continues to offer benefits and, importantly, does no harm.\n\n\n**Cost of using foundation models**. Thank you for raising this point. In our setting, inference costs are modest, as most datasets contain only 100 to 200 datapoints. Specifically, the cost of all the experiments reported in our paper was approximately $200 in API credits. We consider this cost negligible compared to the expenses associated with human experimentation.\n\n\n**Schematic diagram**. We’d be more than happy to adjust Figure 2 in the current manuscript, if the reviewer has suggestions for any particular edits that could clarify the illustration more."}, {"review_id": "4dP621oj8N", "reviewer": "Reviewer_oL2X", "summary": "This paper considers the idea of combining LLM output with real experimental data when computing estimates of the ATE. The set-up combines augmented inverse probability weighted estimators from experiment and various LLMs. They are combined using a linear combination, following the logic of [28].", "questions": "See Weaknesses", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n- Improving the quality of estimators, e.g. by reducing their variance, is particularly important for low-data experimental regimes.\n- Across a range of tasks, the use of LLMs appears to reduce the estimator variance compared to standard AIPW.\n- The proposed method allows one to use LLMs to improve the AIPW estimation on the experimental data, rather than adding artificial data directly into the estimator, meaning that no bias from the LLM is introduced to the actual estimate.\n- The authors conduct a study into using more inference time compute and larger LLMs\n\n# Weaknesses\n- Clarity. Part of the general approach is was not clear to me initially when reading the paper. The AIPW estimator introduced in Sec 3.1 depends on (a) the observed data tuples $(Y_i, A_i, X_i)$ and (b) the function $h$. When using an LLM, one could simulate entirely new data of the same form as the experimental data in (a), this is what the abstract and intro would imply is happening. On the other hand, one could simply use the LLM to create a better estimator for the $h$ function, and use that to create a slightly different estimate from the same pool of experimental data. I believe this is what is actually happening, otherwise the claim that the estimator can tolerate arbitrary bias in the foundation model would not be believable. \n- Furthermore, a key point of the approach as I understand it is that $h$ does not have to be estimated statistically when using LLMs because certain counterfactuals can be computed from the model that cannot be computed in real life. But again, this point is not really made expressly anywhere in the paper.", "was_revised": true, "final_justification": "Thank you for your assurances to improve the clarity and writing quality of the manuscript, with a focus on describing the high-level way in which LLMs are used as a top priority in the abstract/intro.", "rebuttal": "Thank you for your request for clarification. \n\nWe agree that the current framing in the introduction and abstract could be misinterpreted. In particular, we do not simulate entirely new data tuples. Instead, for given covariates $X_i$ from the experimental data, we use the LLM to predict the counterfactual outcome, i.e. $f(X_i,1-A_i) $. These predictions are then pluged into $\\psi(h)$, see equation between line 115 and 116.\n\n\nWe regret that Figure 2 did not clearly convey how the LLMs are used. To clarify, LLMs serve as the functions $f_1,\\ldots, f_k$ that appear in Algorithm 1.\n\n\nWe will revise the introduction and abstract in the camera-ready version to make this clearer."}, {"review_id": "T2K3LgBbkB", "reviewer": "Reviewer_jps4", "summary": "This paper proposes Hybrid Augmented Inverse Probability Weighting (H-AIPW), a new estimator that combines the classic AIPW estimator by an outcome regression fitted on randomized experimental data, with multiple AIPW estimators built from foundation model predictions. The authors prove that H-AIPW is consistent, asymptotically normal, and has no larger variance than the standard AIPW baseline under the minimal assumptions for randomized experiment if the weights are chosen by minimizing the joint asymptotic variance. The authors provide a recipe for applying H-AIPW with large language models including prompt design and cross-fitting, and showed the empirical results across social science studies that H-AIPW improves precision and maintains the validity of inference.", "questions": "Please refer to the Weakness section above.", "limitations": "The authors can explain how participant covariates and synthetic responses are safeguarded when applying LLM foundation models.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The proposed estimator is a natural and reasonable extension of classic AIPW estimators, and it has good theoretical qualities including consistency, asymptotically normality, and no larger variance relative to the baseline. Meanwhile, the valid inference requires no more than usual assumptions for randomized experiments and propensity score.\n2. The proposed framework is able to make use of different kinds of LLM foundation models flexibly. The provided prompt examples and cross-fitting procedures are friendly for replication.\n\nWeakness\nI want to share some concerns that may largely impact the generalization of the work\n1. All experiments are drawn from social science surveys with scale responses. The paper lacks demonstration in clinical trials or industrial AB tests where the responses are not from survey. Besides, the foundation model predictions may carry potential biases and privacy considerations for synthetic data generation are not addressed.\n2. Empirical undercoverage arises when many models are combined on very small sample size. The paper needs to discuss the practical guidance on ensemble size versus sample size.", "was_revised": true, "final_justification": "My major concern about this paper is merely the generalization of the proposed model since the response item theory (survey experiments) is a limited scenario. The authors provided numerical results for binary outcome and promised the results for continuous outcome, and I am convinced to correspondingly raise the score.", "rebuttal": "Thank you for the constructive feedback. We hope this response addresses your questions, but please let us know if you have any additional concerns.\n\n\n**All experiments are drawn from social science surveys.** Our evaluation focuses on survey \nexperiments due to the wide availability of high-quality, publicly accessible randomized datasets. This setting allows rigorous benchmarking across multiple studies. Further, we made the scope of our paper clear early on in the introduction (lines 54 and 55):\n > While our methodology applies broadly, we focus our empirical results on social science survey experiments, where large language\nmodels (LLMs) can provide rich predictive signals.\n\nIn contrast, clinical trials typically involve protected and proprietary data, making evaluation challenging. Further, the goal in our paper is to introduce a novel estimator with strong theoretical guarantees and demonstrate its promise within a well-controlled experimental domain. While we are actively working on applying our method in clinical trial settings, we consider that a separate, more application-driven effort (outside the scope of this paper and not well-suited to the NeurIPS venue).\n\n**Foundation models predictions may carry biases.** We want to clarify that our method is robust against any potential bias from the foundation models. Intuitively, this follows from the AIPW estimator being unbiased regardless of the choice of outcome regression $f$ (for any fixed function $f$, even if derived from a language model). \n\nFor clarity, we demonstrate this below for the treated arm (a symmetric argument holds for the control arm):\n\n$$\n\\mathbb{E} \\left[ \\frac{A}{\\pi} (Y - f(X)) + f(X) \\right] = \\frac{\\pi}{\\pi} \\mathbb{E}[Y \\mid A = 1] - \\mathbb{E}[f(X)] + \\mathbb{E}[f(X)] = \\mathbb{E}[Y \\mid A = 1],\n$$\nwhere we use the fact that in randomized experiments $ A \\sim \\text{Ber}(\\pi)$ and$f$ is an arbitrary fixed function.\n\n\n\n**Privacy considerations.** This is an important point, and we appreciate the reviewer raising it. ​​In a practical setting where privacy is important, the foundation models can be open source ones that are run locally (e.g., LLaMA, DeepSeek and Qwen). Nevertheless, we will discuss this issue carefully in the limitations section, emphasizing that rigorous privacy safeguards may be necessary when applying the method in practice, in particular when used in the medical setting where foundation models may not be as easily run locally. Future work could explore injecting calibrated noise into inputs or applying structured anonymization techniques (e.g., coarsening age or income). While these directions fall outside the scope of our current paper, they open up a promising research area at the intersection of privacy and causal inference.\n\n\n\n**Trade-off between sample size and ensemble size.** There is a clear trade-off in selecting the number of models to combine in small-sample regimes. As shown in Appendix B.2, empirical undercoverage can occur when too many models are ensembled on very limited data. This is a finite-sample effect: increasing the number of models in the ensemble increases the flexibility of the combined outcome model, akin to fitting a linear regression with too many covariates. As a result, asymptotic guarantees still hold, but require larger sample sizes to kick in.\n\nImportantly, these coverage issues vanish completely with more realistic sample sizes (e.g., n = 200), where our method consistently achieves nominal coverage. Indeed, most well-designed randomized experiments do not operate in extremely low-sample regimes (e.g., n = 50), which fall outside the regime where statistical inference remains reliable. \n\nWe will clarify these points and offer guidance for practitioners in the camera-ready version."}, {"review_id": "37aFVX5P0G", "reviewer": "Reviewer_j1Ah", "summary": "The paper introduces H-AIPW, a novel estimator for potentially improving the efficiency of randomized experiments by using the predictions of multiple foundation models (rather than just one model). The key motivation to use multiple foundation models seems to be i) that multiple models are available, ii) an individual model might be biased in its predictions. \n\nOverall, the problem of improving the efficiency of designs is an important problem in many disciplines.", "questions": "Why not use n=10, n=20, n=30 rather than n=100 and n=200 especially given the motivation in the introduction? You clearly cite the clinical trials in the introduction yet if you can get to n=200 you clearly have no recruitment problem. \n\nCan you empirically show the advantage of 3 models for table 1? Why do you restrict it to these three and why do not not compare to just having the best model and giving it the same compute as to all three? It might be that only using the best model actually works much better than having multiple models which are than worse by definition than the best one? \nThis is related to your analysis that \"suggesting that its gains over other approaches that integrate external models\nare not only due to the access to multiple models.\"  This seems to be true in general and raises the questions what the whole point of the motivation of using multiple models is? Can you provide more detailed results for the benefits of multiple models vs providing the same amount of compute to the best model? \n \nHow do you ensure that your assumptions for Proposition 1 are actually met in practice e.g. if n=10? How do you do sample splitting then? \n\nCan you show the effect of effect of sample splitting in n=10 or even n=50? This is in particular important since one of the raised critisms of PPI is that (from related work):  \"This risk of increased variance is a critical limitation in many settings—for example, in clinical trials, pharmaceutical sponsors are highly risk-averse and methods that carry even a small chance of underperforming the established standard face significant barriers to adoption\" Yet sample splitting and double ML approach likewise will increase the variance of your estimator. \n\n\"This approach has already shown promising results in replicating the results of randomized experiments in several scientific disciplines, including medicine [17] and social sciences (see e.g. [3, 5, 4])\" -> [5] is an opinion piece and 3 and 4 are technical reports without any clear evidence of promising results. Where do you see the promise of the approach in social sciences? \n\n\"Further, as is often the case with language models, multiple competing models may be available, with no clear way to determine the best choice for a given task in advance\" -> Is that not a contradiction to the principles of foundation models? \n\nRather than variance why not focus on the effective sample size for comparing approaches e.g. in Table 1?", "limitations": "A limitation section is completely missing. \"We clearly discuss limitations in the conclusion, particularly the reliance on\nunderlying foundation models’ accuracy and alignment with the experimental domain.\" This is one sentence in a 8 line conclusion where two sentence before laud the approach.", "rating": 3, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The paper has interesting applications, but the weaknesses and strengths of the approach are insufficiently explored and ablations are completely missing. \n\nI will provide more detailed feedback and questions in the questions section.", "was_revised": true, "final_justification": "I am unsure about the evaluation and rigorous benchmarking of the approach. I am however not a domain expert and the approach seems theoretical sound.", "rebuttal": "We hope this response addresses your questions, but let us know if you have any additional concerns.\n\n**Why not use very small sample sizes in experiments (e.g. n=10,n=20,n=30)?** We appreciate the interest in understanding the limits of our method under small-sample regimes. However, we believe it is important to emphasize that n = 100 represents a far more realistic and practically meaningful setting than n = 10 or n = 20. Well-designed clinical trials rarely operate with fewer than 100 total participants. Even Phase  2 trials typically enroll 130 to 300 participants on average, depending on the disease area (see, e.g., Table 2 in [1]). Our choice of sample sizes in simulation studies is consistent with this real-world setting.\n\nTherefore, we do not believe ablations with n = 10, 20, or 30 would provide useful insight. These sample sizes lie far outside the domain of established practice and produce conditions where even the simplest estimators (e.g., difference in means) fail to achieve nominal coverage due to severe data scarcity. In such regimes, inference becomes fundamentally unreliable - not just for our method, but for the broader class of semiparametric estimators.\n\nFinally, we note that Proposition 1 is built on standard and widely accepted assumptions in the semiparametric literature, adapted to our setting. As discussed above, small sample sizes like n=10 fall well outside the range of well-designed experiments and beyond the scope where any statistical estimator remains reliable.\n\n\n**Where do you see the promise of the approach in social sciences? The papers cited are technical reports without any clear evidence of promising results.** We respectfully disagree. The claim that [3] is a technical report is factually incorrect - the paper is published and has collected over 800 citations. As the authors state: \"We suggest that language models with sufficient algorithmic fidelity thus constitute a novel and powerful tool to advance understanding of humans and society across a variety of disciplines.\" This reflects growing recognition of foundation models as a transformative tool in the social sciences.\n\nFurther, we disagree with the claim that [4] offers no clear evidence of promising results. On the contrary, the paper presents one of the most comprehensive and rigorous evaluations to date of large language models' ability to predict the outcomes of social science experiments. They use GPT-4 to forecast 476 treatment effects across 70 nationally representative experiments. Overall, the authors report a correlation of r = 0.85 between GPT-4’s predicted and actual treatment effects. \n\n\n**It might be that using the best model actually works much better than having multiple models?** Our framework is flexible enough that it may ultimately rely on a single best-performing model. However, when different models capture complementary signals, a weighted combination can outperform any individual model, making the combination strictly better in such cases.\n\nNevertheless, we want to clarify that the best single model cannot outperform the combined estimator in terms of efficiency. This is guaranteed by Theorem 1, which establishes that the hybrid estimator’s variance is no worse than that of any individual component estimator. Thus, combining models is never harmful and often beneficial. \n\nThe use of multiple models is a flexible feature of our framework, not a requirement. It can be particularly valuable in high-stakes settings like clinical trials, where pre-specifying the analysis plan is often necessary and post hoc model selection is not allowed (as it can affect the validity of the inference).\n\n\n**Can you provide more detailed results for the benefits of multiple models vs providing the same amount of compute to the best model?** Figure 4(b) shows that performance plateaus after averaging the prediction over 5  different prompts. In Table 1, we use 10 prompts per model, meaning the total compute budget is already well beyond the point where additional sampling would lead to meaningful improvements. Therefore, we believe that reallocating the same compute to a single model - e.g. using 30 prompts - would offer no added benefit.\n\n\n**Effective sample size in Table 1.** We agree that reporting effective sample size could also  be informative. However, they are dual of each other, and hence the relative ranking between methods would remain unchanged.\n\n**Lack of limitations.** We agree that the limitations deserve more thorough discussion. Due to space constraints, we only briefly addressed them in the conclusion. In the camera-ready version, we will use the additional available space to include a dedicated limitations section covering model alignment.\n\n[1] Costs of Drug Development and Research and Development Intensity in the US, 2000-2018. Aylin Sertkaya; Trinidad Beleche; Amber Jessup; et al.\n\n[3] Out of One, Many: Using Language Models to Simulate Human Samples. Argyle et al.\n\n[4] Predicting Results of Social Science Experiments Using Large Language Models. Luke Hewitt, Ashwini Ashokkumar, Isaias Ghezae, Robb Willer"}, {"review_id": "HUXMPxxXmg", "reviewer": "Reviewer_cefv", "summary": "The paper introduces a novel statistical method H-AIPW estimator, to increase the statistical precision of randomized experiments by leveraging predictions from several foundation models.  The idea from my understand is to get the best convex combination of all the estimators. Theoretically, the paper proves that the H-AIPW estimator is consistent, asymptotically normal, and has an asymptotic variance that is guaranteed to be no larger than the standard estimator that relies solely on experimental data.", "questions": "Please see above.", "limitations": "Please see above.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "# Strength\n1. From my perspective, the problem itself is very important. How to utilize the foundation models in experimentation and causal inference has been of great interest recently.\n2. The technical part is quite solid.  The proof that H-AIPW's variance will be no worse than the standard AIPW estimator is interesting.\n3. The paper is well written. I enjoy the flow of the paper a lot.\n\n# Weakness\n1. My only major concern is that the core methodological idea—combining multiple AIPW estimators via optimal linear weighting to reduce variance—is conceptually close to classical shrinkage estimation, model averaging, and ensemble methods in semiparametric inference. Please do not get me wrong. I like simple ideas, which can usually be more useful in practice.\n2. Another minor point is what the unique feature of foundation models is used for the estimator? It seems that any prediction models trained on different dataset (not necessarily foundation model) can be included?", "was_revised": true, "final_justification": "I sincerely thank the authors for the clarification. However, I do not see significant difference. I choose to retain my score.", "rebuttal": "Thank you for highlighting the importance and clarity of the paper. We hope this response addresses your questions, but please let us know if you have any remaining concerns that would prevent an “accept score”. \n\n\n**Relation to shrinkage estimators.** We agree that the idea of combining multiple estimators to reduce variance is conceptually close to classical shrinkage estimators. However, we would like to emphasize a key distinction: unlike traditional shrinkage approaches, our method only includes unbiased estimators in the combination. We believe one of the contributions of our work is to show that it is possible to achieve variance reduction without introducing bias. This is a crucial property for practical adoption, especially in high-stakes applications. That said, we appreciate the connection and will expand our discussion of this relationship in the related work section of the camera-ready version.\n\n**Relation to ensemble methods and model averaging**. Ensemble methods such as bagging, random forests, boosting, and the Super Learner have long been used in semiparametric inference, typically combining models trained on the same (often small) dataset. In contrast, our approach integrates predictive models trained on vast amount of *external data, independent of the dataset at hand*. Further, our goal here is not to boost predictive accuracy per se, but to enable valid and more precise inference for treatment effects by leveraging these externally trained models. \n\n**Re foundation models.** Our framework is model-agnostic, and any externally trained prediction model can, in principle, be incorporated - it does not need to be a foundation model. We focus on foundation models in this paper because all our experiments are conducted using LLMs. Moreover, we expect foundation models to be the biggest use case of our general methodology, particularly within the social sciences. We will make this clearer in the camera-ready version to avoid any misunderstanding that our method is limited to foundation models."}]

Author rebuttal:

Meta-review: This paper introduces Hybrid Augmented Inverse Probability Weighting (H-AIPW), for improving the statistical efficiency of randomized experiments by integrating predictions from foundation models. It combines the standard, unbiased estimator from experimental data with model-based predictions in a way that reduces variance without introducing bias.

The paper received a largely positive reception from the reviewers. Most reviewers found the work to be technically solid, original, and significant. They highlighted the novelty of adapting classical AIPW estimators to leverage foundation models, the strong theoretical guarantees provided , and the practical importance of improving estimation efficiency, especially in low-data settings.

Initial reviews raised several concerns. However, the authors provided a thorough and convincing rebuttal. They clarified the technical details, presented new simulation results to address robustness and generalization concerns. This proactive engagement satisfied most reviewers, with one raising their score and others maintaining their positive ratings.

One reviewer remains less convinced, questioning the experimental design and the motivation for using multiple models. However, they also state a low confidence in their assessment and recommend weighing the opinions of other reviewers more heavily.

Given the overall enthusiastic response, the technical soundness of the work, and the authors' diligent and effective handling of the reviewers' feedback, I recommend accepting this paper.
Decision: Accept (poster)