Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect Personas
Authors: Luke Guerdan
Justin Whitehouse
Kimberly Truong
Ken Holstein
Steven Wu
Venue: ICLR 2026 Poster (2026)
Keywords: external validity
LLM-as-a-Judge
large language models
evaluation
personas
causal inference
doubly-robust estimation
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We provide a statistical framework for combining human ratings observed under sampling bias with imperfect persona ratings to obtain valid GenAI system performance estimates.

Abstract: As Generative AI (GenAI) systems see growing adoption, a key concern involves the external validity of evaluations, or the extent to which they generalize from lab-based to real-world deployment conditions. Threats to the external validity of GenAI evaluations arise when the source sample of human raters and system outputs used to obtain a system quality estimate differs from the target distribution at deployment time. In this work, we propose a doubly-robust estimation framework designed to address this evaluation sampling bias. Key to our approach is the use of synthetic "persona" ratings -- produced by prompting an LLM evaluator (i.e., an LLM-as-a-judge) to behave as a human rater with specific sociodemographic characteristics. Our doubly-robust framework combines these informative yet imperfect persona ratings with human ratings obtained under evaluation sampling bias to produce statistically valid system quality estimates. In particular, we show that our approach yields valid system quality estimates when either: (i) a model trained to predict human ratings using persona ratings and source data observed under sampling bias, or (ii) a reweighting model that corrects for sampling bias is of sufficient quality. We validate our framework theoretically and via a novel Persona Simulation Framework (PSF) designed to systematically manipulate persona quality and the degree of evaluation sampling bias present in source data. Our work provides a principled foundation for combining imperfect persona ratings with human ratings observed under sampling bias to obtain valid system quality estimates.

DOUBLY-ROBUST LLM-AS-A-JUDGE EXTERNALLY VALID ESTIMATION WITH IMPERFECT PERSONAS
Luke Guerdan" Justin Whitehouse? ? Kimberly Truong"!
Kenneth Holstein', and Zhiwei Steven Wu!
Carnegie Mellon University
Stanford University Coo-first authors
ABSTRACT
As Generative AI (GenAI) systems growing adoption, key concern involves the external validity of evaluations, or the extent to which they generalize from lab-based real-world deployment conditions. Threats the external validity of GenAl evaluations arise when the source sample of human raters and system out- puts used to obtain system quality estimate differs from the target distribution deployment time. In this work, we propose doubly-robust estimation framework designed address this evaluation sampling bias. Key to our approach is the use of "persona" ratings produced by prompting LLM evaluator (i.e., LLM-as-a- judge) to behave as a human rater with specific sociodemographic characteristies. Our doubly-robust framework combines these informative yet imperfect persona ratings with human ratings obtained under evaluation sampling bias to produce statistically valid system quality estimates. In particular,
show that ap- proach yields valid system quality estimates when either (i) a model trained to predict human ratings using persona ratings and source data observed under sam- pling bias, or (ii) reweighting model that corrects for sampling bias is of suf- ficient quality. We validate our framework theoretically and via a novel Persona Simulation Framework (PSF) designed to systematically manipulate persona qual- ity and the degree of evaluation sampling bias present in source data. Our work provides principled foundation for combining imperfect persona ratings with hu- man ratings observed under sampling bias to obtain valid system quality estimates.
INTRODUCTION
As Generative AI (GenAl) systems see growing adoption, a key concern involves the external valid- ity of evaluations, or the extent to which they generalize from lab-based to real-world deployment
conditions (lbrahim et al. 2024 Ouyang et al. 2023 Liao & Xiao 2023 Liao et al.) 2021 Wei-
dinger et al. 2025). In particular, many evaluations report system quality estimate, which reflects the proportion of outputs rated to exhibit a capability (e.g.. "helpfulness") or defect (e.g., "toxicity") by a human with specialized characteristics (e.g., domain knowledge, cultural experience). However, such quality estimates may fail to generalize when the source distribution of human raters and system outputs available at evaluation time differs from the target distribution encountered upon deploy- ment. For example, in conversational safety evaluation, covariate shift can arise if we wish to esti- mate the perceived "toxicity" of response for specific demographic group, but collect ratings from a general population of crowdworkers. Selection bias may also occur if crowdworkers frequently opt out from providing ratings on highly sensitive content (Steiger et al. [022]. Left unaddressed, these forms of evaluation sampling bias threaten the external validity of system quality estimates.
"Co-first authors contributed equally to this work (see for statement of contributions). Correspondence to: k1 rutnngg cs. cmu edu, jwhi tewc@stanford..ccm edu, 1 guerdanness cmu. .uuu.

---

Rully Synthetic
Semi-Synthet (PRISM)
Semi-Synthetic (DICES)
Doubly-Robust (Rlesz)
Doubiy-Robust
Doubiy-Robust (Rlesz)
Seurco mas
(Rlesz)
Brgat meat
HH
Seredd
M
Penana-Baseed
PPI++
HH
PPI++
MP
H+]
Bel
Re/P!
Be|9!
H+
Sample Avenge
Sample Average
[0]
Sample Average
0.50
1.00 1.25
47.5 45.0 Estimate mean CI
1.45 1.50 155 1:50
Estimate mian
Estimate maan =
Figure 1: Comparison of our doubly- robust estimator with baselines on three datasets from our Persona Simulation Framework. Red and black dashed lines denote the true source and target mean ratings, respectively (e.g., the average "helpfulness" rating obtained over source vs, target distributions).
Persona-Rased directly averages persona ratings compute system quality estimate. Sample Average produces a system quality estimate by averaging human ratings sampled from the source distribution. PPI++ (Angelopoulos et al. 2023b) and RePPI (Ji et al. (025) are two state-of-the
statistical methods that do not account for evaluation sampling bias. Across settings, we observe that our Doubly-Rober (Riesz) estimator
yields improved coverage and lower bias than baselines, while maintaining informative confidence intervals.
Recent work has proposed tools for improving system quality estimates when human ratings are scarce but black-box predictions (e.g.. from an LLM LLM--a---juddge are cheap and abundant (Chatzi)
al. 2024; Fisch et al.] 2024 Eyre Madras 2024, [Donerr et al.] 2024, Faad-Falcon et a
2023 Fogliato et al. 2024). For instance, Prediction Powered Inference (PPI) offers an approach
for leveraging a subset of labeled (source) data to correct for bias in black-box model predictions
(Angelopoulos et al. 2023a b). This bias correction enables using black-box predictions generated over unlabeled (target) samples to shrink confidence intervals around quality estimates while maintaining valid coverage. However, both PPI and its extensions (e.g.. PPI++ (Angelopoulos al., 2023b), RePPI (Ji et al. 2025)) assume that (i) source and target samples are drawn i.i.d. and (ii) labels are missing completely random (MCAR) (Tsiatiis 2006), i.e. that successful completion of a rating is independent of rater and text characteristics. However, when source data is observed under sampling bias, these assumptions are violated and severe miscoverage occurs (see Fig. D
In this work, we devise an estimator that directly corrects for evaluation sampling bias. Like existing approaches (Angelopoulos et al. 2023b Di et al. 2025), our proposal leverages black-box predictions generated by a GenAl system over unlabeled (target) samples to improve statistical inference. Unlike existing estimators, however, our proposal is doubly-robust (Bang & Robins, 2005 Chernozhukov et al. 2018): it yields valid system quality estimates if either a model trained to predict human ratings from labeled source data or a reweighting model that corrects for sampling bias is of sufficient quality. To attain this doubly-robust property, we leverage persona ratings scores generated by prompting an LLM-as-a-judge to behave as a human rater with desired characteristics (e.g., demographics, expertise). Our framework uses these persona ratings an auxiliary feature when training a model to predict human ratings from labeled source data. When persona ratings are sufficiently correlated with human ratings, this yields higher-quality predictor, which in turn makes the double-robustness condition of our estimator easier to satisfy (see 3
Figure I illustrates the benefits of our estimation approach on three datasets. Whereas persona ratings alone (aasenna-Based) or human ratings alone (Source Mean) fail to provide valid coverage, our approach combines imperfect persona ratings with imperfect human ratings to attain valid confidence intervals with low statistical bias. We similarly find that our approach out-performs of baselines (Angelopoulos et al.] [023b] Ji et al., 2025). which also combine persona ratings with human ratings, but fail to account for evaluation sampling bias. To summarize, our main contributions are as follows:
We formalize the problem of GenAI system quality estimation under evaluation sampling bias. Unlike existing statistical frameworks (Angelopoulos et al.) 20236: Ji et al. 2025). our formulation explicitly accounts for both covariate shift and selection bias in available human ratings to improve the externl validity of system quality estimates.
We devise a doubly-robust estimator for GenAI system quality estimation under eval- uation sampling bias. En route, we first advance doubly-robust estimation theory by gen- eralizing the work of (Chernozhukov et al. (023) to M-estimation settings with surrogate (persona) ratings. This generalization enables us (i) leverage persona ratings to improve
2

---

LLM-as--- with Persona Prompting
Target Parameter
Content Characteristics V
Mean Rating
Rater
X
GenAi
Persona
Rating Y
0 = T2 M
Characteristics
System
Content Characteristics
Rater Characteristics
Rating Completion
Human Rating
Persona Rating
Persona Rating Y
Content
Humani Realing
Characteristics
Characteristics
V
Y
Y
X
Source Distribution P.
Target Distribution P.
Figure 2: Our framework produces estimates for the target parameter 8, using (i) complete rating tuples from the source distribution (blue, left), (ii) unlabeled samples from the target distribution (yellow, right), and (iii) persona ratings produced for both source and target samples (red, top). Evaluation sampling bias may arise both from the covariate shift of (V, X) from P, to Pt, and from selection bias in which rating completion C' is non-random in P. i.e..e, C Y. XV.
doubly-robust system quality estimates, (ii) estimate a richer set of system quality parame- ters (e.g., rating variance, quantiles) beyond means, and (iii) maintain valid coverage in the presence evaluation sampling bias, all desiderata not satisfied by previous works.
We advance the practical application of doubly-robust estimators to GenAl system quality estimation. Whereas doubly-robust estimators are traditionally applied on small tabular datasets, GenAl system quality estimation requires learning a reweighting function over high dimensional (e.g., text, audio) input-output spaces. We show that sentence transformer embedding models and "Riesz loss" approach (Chernozhukov et al.) 2022b) can be combined to correct for covariate shift in high-dimensional text input-output spaces.
We introduce a Persona Simulation Framework (PSF) that systematically manipulates evaluation sampling bias and persona quality increasing realism: Fully Synthetic, Semi-Synthetic PRISM (Kirk et al.) (024), and Semi- Synthetic DICES (Aroyo al., 2023). Leveraging the PSF, we show our estimator obtains valid coverage up to a larger magnitude of sampling bias than state-of of-the-art baselines (e.g., RcPPI (Ji et al. 2025 We publicly release our PSF code community resource
three experimental settings with
PRELIMINARIES
We provide an overview of the data generating processes and GenAl system quality parameters considered hereinafter. We provide detailed coverage of all notation and assumptions in Appendix
Probabilistic Framework. As illustrated in Fig. 2 we model system quality estimation under evaluation sampling bias via tuple of random variables rater characteristics (e.g., age, gender, geographic locale) and V denotes the content to be rated, such as the GenAl system input and output. experiments, we characterize the content V via an embedding-based projection of the input prompt and system output into a low-dimensional space (see 4.. We use = (X, V) to denote the tuple rater and content. C is an indicator of rating completion (C' if the rater provides rating, C' 0 otherwise). For example, rater can fail to provide rating if they (i) are excluded on the basis of failed attention checks, or (ii) abandon the rating task mid-way (i.e., self-attrition. Let Y denote the rating rater would assign if they completed the task (C = 1), which may be ordinal (e.g.. Likert 1-5), interval (e.g., 1-100), or binary (e.g., Yes/No). Finally, y is the rating returned by an LuMe with persona prompting.
VX,VCC,Y,Y Here, X denotes
Example. Suppose we want measure the perceived "safety" of conversational AI system's response. X captures rater demographics (e.g., age, race, education) and V includes the user query and system response. Rating completion (C) denotes whether the rater provides rating or opts out, and Y is the human's "safety" rating (e.g., on 1-4 scale). Ý denotes GPT-5's predicted rating.
Code and data available GitHub

---

Table 1: Examples of statistical parameters that can be estimated via our M-estimation framework. Each parameter summarizes information about human ratings obtained over the target distribution. Conditional parameters (bottom three rows) can be defined conditionally on rater characteristics (X), content characteristics (V), or both, as special cases of conditioning on W = (X,
Parameter
Estimand
Example
Mean "helpfulness" rating for customer service
Mean
0. E
chatbot responses.
Variance (disagreement) in conversational "safety" ratings.
Variance
0. VVr
Median "comprchensibility score for technical documentation.
Quantile
0. = (y: P.(Y- 1 1 2 Q}
Conditional
Mean "coherence" rating assigned by domain
0: = B.(Y I (g(W) = 1]
Mean
experts multi-turn conversational responses. Variance in "helpfulness" ratings among novice progranmers for code suggestions. 90th percentile "safety" rating for high-risk queries flagged by content moderators.
Conditional
0. == Var, 19(W) = 1]
Variance
Conditional
0, == (y: P. S
Quantile
g(W) = 2) QQ)
Evaluation Sampling Bias. To model evaluation sampling bias, we assume there are two full-data distributions over tuples Z: a source distribution P. and a target distribution Pt. Covariate shift arises when the marginal distribution of rater and content characteristics differs across source and target distributions. For instance, the source distribution may consist of crowdworkers recruited via MTurk- who tend to skew younger and more educated than the general population (Levay eet al. 2016)-whil the target distribution might reflect the demographic composition of a specific deployment context (e.g., users of healthcare chatbot, who happen to skew older and female). Selection bias arises when rating completion depends on rater and/or content characteristics i.e. C L W. In our running example, this can arise if crowdworkers are more likely to opt out of rating highly sensitive content. While our framework is explicitly designed to handle selection bias, existing frameworks (Angelopoulos al. 2023b; Ji et al. 2025) assume that data are missing completely at random (MCAR) i.e., C' IL W. We show empirically that violations of this
assumption lead to severe miscoverage in system quality estimates (see
While we relax this MCAR assumption, we rely on several additional assumptions (also required by existing frameworks). For instance, we assume no concept drift, i.e., that (Y = (.(Y..). This requires that the rater and content characteristics are sufficiently rich to describe ratings across both populations. We elaborate on this and other standard causal assumptions required by our framework in Appendix B Relaxing these assumptions remains a fruitful direction for future work.
Estimation Goal. Given a sample of N₂ partial source observations D, {(55, N7.Cj.CJ. N. Yj, y and No partial target observations D, {(X;, V7, )
goal is to estimate a parameter summarizing system quality over P. 3 As our running example in the main text and experiments, we consider the mean rating. 0g == Eg [Y], which describes the average "safcty" or "helpfulness" rating assigned by human raters to system outputs in the target distribution P..
3 METHODOLOGGY
We now introduce our doubly-robust estimator for GenAl system quality estimation under evalua- tion sampling bias. The central challenge addressed by our approach is that our data is imperfect. While persona predictions are available, they may be poor proxy for human ratings. Likewise, human ratings from the source distribution may suffer from evaluation sampling bias, leading to invalid estimates for the target parameter. We first introduce several naive approaches which might be used to tackle this problem (8 3.I. Then, we show that while each approach is insufficient iso-
We use superscripts 8, t on random variables to denote source and target membership. We omit these superseripts where the distribution is clear from context (c.g., [Y] clearly refers to the target distribution). IIn the tuple D., the shorthand Cj YJ" denotes that ratings are only observed when Cj= =1
4

---

Our results presented in this section apply not only when 8, := E, [Y] (where they generalize Chernozhukov et al. (2023) to simulta- neous covariate shift and selection bias) but also when 0, is the solution to a generic M-estimation problem. Table Tiblutttlsstrttss the range of statistical parameters our framework supports.
lation, they can be combined to obtain valid coverage. 3.2..
3.1 BASELINE APPROACHES AND THEIR LIMITATIONS
Persona-Augmented Regression. One possible estimation strategy to train a model to predict human ratings using source data and then use this model to impute missing target labels. Persona- augmented regression leverages this approach while including persona ratings as an additional auxiliary feature in the regression function. In particular, we train model (WW P) predicting (o(W := EE[Y wil using samples from D, and then estimate 0g as 0: := N. E i(W?,Y;") Observe that this persona-augmented regression estimator relies on the persona rating in addition to covariates. While this approach may be viable when persona ratings are highly correlated with human ratings, in general ji will converge too slowly to construct valid confidence intervals ($4)
Re-weighting. Another approach to re-weight samples from P. based on their probability of occurring under Pt. This approach requires correcting for covariate shift and selection bias in par- allel, and does not use persona ratings. Formally, let ww(w) = dP,
dP. (w) denote the density ratio between P.WW) and P. (W), and let (ww)) P. =1 W = w)) denote the probability of rating completion. Under standard assumptions (see Appendix B). we have 0, E. [o, C)Y], where (W C) := C Thus, if one produces an ML estimate of am (say by training models
w. न predicting wa. mo), they can compute an inverse propensity weighted (IPW) estimate 0:TH + Ee a(W,, C3) Yf. Again, estimates of ogg must converge at parametric rates in order to maintain coverage. Further, IPW suffers from high variance when propensities small salient challenge when estimating system quality parameters over high-dimensional (e.g., text) data (5 4)
3.2 DOUBLY-ROBUST ESTIMATOR
Our doubly-robus estimator can be viewed as carefully combining the persona-augmented regres- sion estimator (μO) with the re-weighting estimator (oo). The functions pi and og are referred to as nuisance functions because they are used as an auxiliary information source to estimate the target statistical parameter of interest O,. Our estimator combines these nuisance functions in the form:
N₂ N
N Σ a(wj, 0J) ( -
(I)
N.
where ğ and â are estimates of μuu and ag that are assumed to be independent of the data. In Eq. estimator over samples from the target distribution. Analogously to PPI++ (Angelopoulos et al.) 2023b), this has the effect of using unlabeled data to reduce variance in the estimate. The right term corrects for bias in the human rating predictor by re-weighting residualized source data to account for covariate shift and selection bias. This correction adjusts for bias in persona ratings via the residual term, and evaluation sampling bias via the learned re- weighting function á.
the left term evaluates the regression-basd
To construct confidence intervals, we also consider the variance estimate:
1 N
(aw: Y/) me
a(W3, 03) (
N
where m N; DE P(WF Y?)) and is scaling parameter (described in Algorithm 1. Since mean and variance estimators require nuisance estimates that are independent of the data, K-fold fitting to maximize efficiency (Chernozhukov et al., 2018). For each & S K, we train nuisance estimates to
nuisance models all data excluding the samples in fold k. We then produce a de- biased parameter estimate for the data in fold k. We finally average the per-fold param- eter and variance estimates to maintain full data efficiency. See Algorithms I and for full details.
Nno concept drift implies R. [ [W]
T2, [Y]
], so we can omit subscripts without any worry.
5

---

Algorithm Doubly-Robust Estimator with K-fold Cross-Fitting
1: Input: Samples D. (Zi,.... Zx,1 from Pe. samples D, {Zi..... ...... x from Ps, number
of folds K.
2: Randomly split source indices [Na]] random folds of equal size: Ii...., .I..
3: for & [K] do
4: Produce ML estimate jr( using D2, := : Ds.ks where Dr, = Z : Ix)
5: Produce MI. estimate (-k) using Di and Dr-
6: Construct 0₂ per Eq. with ğ (((-)),,, k), â a(-k), and samples Da and Dr.
7: Construct T% per Eq. with p = 7 1), = al ;)) = N, and samples Da, N.
and
Dr-
8: Compute the average of the K estimates: 0= t/ Σ k
Ek=1 O2 and 0² E o
9: Return: Mean estimate 0 and variance estimate Ø2.
Our main result establishes the asymptotic normality of our estimator. Further, it describes how to build confidence intervals using the mean and variance estimates recovered from Algorithm| [
Theorem 3.1. Assume the learner has access to samples Zi..... D Pa and Zi...., ....,,2 Liv ~ Pe
satisfying Assumptions 7 and the assumptions of Theorems B.2 pendix Bp. Then, letting and 6² denote the mean and variance returned by Algorithm 4
and B.4 (all outlined in Ap-
we have
V: (0-0) - N(O,1).
In particular, this implies that, for any 5 E (0, the set
C1-5 VN, 28/2
VN, 26/2
- confidence interval for 0, where z4 denates the quantile of standard normal R. V.
complete theorem statement can be found in Theorem B.4 In Appendix we present generalization to M-estimators (Theorems CA and C.4 and corresponding proof the above follows as special case. We also provide examples of other target parameters, including rating variance and quantiles in Remark C.2 of the same appendix, which may be of independent interest.
Theorem 3. relies on several key assumptions, formally outlined in Appendix Of these, the most important is double robustness, which requires the product of nuisance estimation errors decay at parametric (i.e. VN₂) rates. Formally, this assumption can be expressed via the condition that
(|ī((-) -
-ool
pollz= op ON
(3)
on each fold. See Appendix B for definition of L norms and a formal definition of oe notation (here, PN denotes convergence in probability at N rates). Notably, this condition allows each individual nuisance estimate to converge non-parametric rates, thus permitting coverage even when estimates of either ogg No of lower quality. For instance, one could have
max {1(Z(- poll La; ||ā(-*) on La ON
and still maintain valid coverage. In other words, when we state that our estimator will provide valid coverage when either (i) model trained to predict human ratings using persona ratings and source data observed under sampling bias (7), or (ii) a reweighting model that corrects for sampling bias (α) is of sufficient quality, we refer precisely to this product of errors condition (Eq. 3)
We also note that the above convergence rate does not directly depend on the quality of persona ratings. Rather, the persona ratings serve as extra covariate onto which we regress human ratings Y. When persona ratings are highly correlated with human ratings, we may obtain faster convergence rates for á. However, this does not prohibit convergence even when the quality of persona ratings is low. This phenomenon is illustrated in our experiments below.
Estimation Details. In Theorem 3.1 above, estimating Ao is standard regression task that can be accomplished using any off-the-sheli model (c.g. gradient boosted trees, neural networks). How- ever, estimation of oo(w, c), which is complicated ratio of likelihood ratios and propensity scores,

---

is more subtle. The standard approach for doubly-robust estimation would involve learning @ and क separately (c.g., via gradient boosted trees) then estimating ā by taking the ratio of predictions produced by each model. However, because wo occupy a high dimensional (e.g.. text) space, the variance in this ratio can be quite high. This variance in turn propagates to downstream estimates.
To address this challenge, we leverage "Riesz loss" (Chernozhukov al. 2022b a 2023)
timate Crg- Rather than learning wo and Ty independently, the Riesz loss directly learns the ratio (oww, c). For our setting, letting (o( == wo(w)/ mo (w), the Riesz loss minimizer is given by:
Go arg min
E -
Therefore, to estimate Og,, we minimize the finite-sample analogue of Eq. using D and Dr, then plug this into Algorithm|]] (see Appendix for details). As we show in g this Riesz loss approach significantly improves the quality of downstream estimates.
4 EXPERIMENTS
Validating estimators under evaluation sampling bias requires datasets with detailed rater character- isties, rating completion information, and a mechanism for systematically manipulating the magni- tude of covariate shift and selection bias. Such datasets are scarce. To address this gap, we introduce a Persona Simulation Framework (PSF) that provides complete rating tuples Z = (X,, V, C, Y, Y) and allows us to vary (i) covariate shift, (ii) selection bias, and (iii) persona quality in parallel. The PSF contains three specific datasets, each of which simulates evaluation data with increasing realism:
Fully Synthetic: All nuisance functions and target parameters are fully known (see Appendix EE Semi-Synthetic PRISM: We sample 1000 real user conversations from PRISM (Kirk et al. (024) and obtain the "ground truth" target parameter 0g by treating ratings sampled from an LL-M-as-a-judge as human ratings (Y). We sample 50 such LLM ratings per item. Unlike the fully synthetic setting, true nuisance functions are unknown. We sample persona ratings Ý by adding controlled error to the LLM-as-a-judge ratings (see the "helpfulness" of outputs on a 1-100 scale.
This task instructs raters rate
Semi-Synthetic DICES: We sample real user conversations, rater characteristies (e.g., age, race), and human ratings (Y) from DICES (Aroyo et al. 2023), resulting in 300 conversations with 25 human ratings each. We then sample persona ratings Y by (i) adding controlled error to human ratings (see & 4.1 and (ii) via an LLM-as-a-judge with persona-based prompting. This dataset instructs raters to assess the "harmfulness" of outputs on 1-4 scale.
Going forward, we refer to these three datasets as Synthetie, PRISM, and DICES, respectively. In addition to providing foundation for validating our doubly-robust estimator, the PSF offers resource for the community to test future evaluation approaches under evaluation sampling bias.
4.1 DATASET GENERATION PROCEDURE
We now describe how both semi-synthetic datasets are generated in the PSF Further setup details, including prompts used for synthetic dataset generation, are reported in Appendix
Source and Target Populations. In the PRISM experiment, the source population consists of con- versations where users are prompted to engage in controversial topies, while the target population consists of conversations with guided prompts. In the DICES experiment, the source population contains 300 single-turn conversations flagged by safety experts as containing a single harm type (e.g., misinformation, legal), while the target contains more complex conversations rated as contain- ing multiple types of harm. In both cases, we model each sample as single user-system exchange extracted from multi-turn dialogue. We embed the input-output pair from each exchange into a low-dimensional space by first applying an embedding model (MiniLM-L6--v2) then projecting 15 dimensions via UMAP (Becht et al. 2019 We also vary the demographic composition of raters across populations. In both PRISM and DICES, we define the source distribution (x)
'We selected 15 dimensions ensure embeddings retained predictive signal for ratings and source/target memhership while keeping dimensionality low; results remained stable for 2 12 dimensions (Fig.
7

---

using marginal probabilities of rater characteristics reported in DICES, and target distribution Pt( using population statistics released by the U.S. Census Bureau (Guzman & Kollar 2023).
Covariate Shift. To vary the magnitude of covariate shift, we control the mixture between the source and target populations. We vary the content characteristics by controlling the proportion Ç E [0 of target items contained within the source sample (sub-sampling from the full data to ensure that source and target sample sizes remain fixed). Additionally, we vary the rater distributions by taking the convex combination between all groups in each demographic stratum with normalization. The magnitude of the resulting covariate shift between samples is then given by the Sinkhorn Distance A(W", W (Feydy et al., 2019). where recall that W (X, V) and V denotes the embedded content characteristics (MiniL M-L6-22 UMAP). We report the Sinkhorn distance normalized by subtracting the baseline case where there no covariate shift for both semi-synthetic experiments, as it is inevitable that there will be variation in text embeddings despite sampling i.i.d. from pre-defined categories (e.g.. harm types) within population. This measure captures covariate shift resulting from content characteristies and demographic attributes in parallel.
Selection Bias. We model rater cuttritionnwhen raters fail to provide a rating due to failed attention checks or task abandonment-by varying the probability that each item is rated. In PRISM, we prompt the LLM to output both a rating and a non-response "refusal" flag. In DICES. we use rater self-assessments of task understanding transform attrition scores into dropout probabilities using a Beta CDF with a = 3 (increasing B increases selection bias). We censor ratings according to these probabilities while retaining the "true" rating. We quantify the magnitude of selection bias via the dropout rate, i.e., the probability that rater fails rate an item. The dropout rates we simulate mirror those observed in practice. In DICES, 19 of 123 raters (15,4%) were excluded due to failed attention checks, while in PRISM, 104 of 1500 raters (6.9%) failed to provide ratings after completing the background survey. As we show in our results, existing methods (e.g., RePPI (Ji et al.) 2025)) exhibit severe miscoverage at these empirically observed dropout rates. This underscores the importance of correcting for selection bias.
assign attrition scores (see Appendix E We then
Persona Quality. To manipulate the quality of persona ratings, we perturb human ratings with con- trolled error. This perturbation has (i) bias parameter y E [-1, which induces a systematic shift, and (ii) a correlation parameter E [-1, which parametrizes the Pearson correlation between human ratings and persona ratings (see Appendix D. We verify the robustness of this perturbation
approach by performing additional experiments using persona ratings sampled from real LLMs.
4.2 SETUP DETAILS
Models. We use GPT-5 5 to generate synthetic "human" ratings for PRISM - i.c. used as Y in our framework to obtain the ground truth target parameter 6g. We use Claude- (Hade 3,5, Sonnet 3.5] and GPT-(5, 4c-Mini) to generate persona ratings for DICES. We report prompts, sampling temperature and decoding methods used for each LLM in Appendix
Estimators. We compare Sample Average, IPW, Persona-Based Estimation, Persona-Augmentted Regression (PAR), PPI++ (Angelopoulos et al.) 2023b), and Recalibrated PPI (RePPI) ( et al. 2025) estimators against two doubly-robust variants: (i) DR (Classical), which learns nuisance functions (w, i),, and (ii) DR (Riesz), which uses Riesz loss minimization to directly produce an estimate â of Crp- Nuisance functions were tuned via hyperparameter search (see Appendix E
Metrics. evaluate estimator quality using three metrics: Bias (MAE): |0, 0,,, absolute deviation from the target parameter, Coverage: Pr(#t E [0 Bnight)), the probability that the confidence interval covers the true parameter; and Interval Width: Onigh 0row the length of the interval.
4.3 RESULTS
Finding 1: DR (Riesz) obtains lower bias and improved coverage than baseline estimators.
Figures 3 and 4 present our main findings varying (i) covariate shift, (ii) selection bias, and (iii) persona quality over all three datasets (N = 40 trials per setting). We present cross-sectional
results in Fig. and average in Fig. 4 As illustrated in Fig. 3 DR (Riesz) obtains valid 95% Cls when (i) persona quality is high (top), (ii) covariate shift is moderate (middle) and (iii) across
ranges of selection bias (bottom). In contrast, baseline estimators obtain valid coverage only on
8

---

Synthetic
PRISM
DICES
1.0
1.0
Coverage
Coverage
Coverage
0.5
0.0
0.0
0,5
1.0
0.0
0.0
0.5
1.0
0.0
0.25
0.50
0.75
1.00
Persona Quality (p)
Persona Quality (p)
Persona Quality (p)
Synthetic
PRISM
DICES
1.0
1.0
1.0
Coverage
Coverage
Coverage
0.5
0.0
0.5
0.0
2
0.0
6 Covariate Shift Magnitude (A)
Covariate Shift Magnitude (A)
Covariate Shift Magnitude (A)
Synthetic
PRISM
DICES
1.0
1.0
1.0
Coverage
Coverage
Coverage
0.5
0.5
0.5
00.00
0.10
00000
0.05
0.10
0.20
0
0.05
0.15
0.20
Dropout Rate (%)
Dropout Rate (%)
Dropout Rate (%)
DR(Riesz)
PAR
PPI+ IPW
RePPI
DR (Classical)
Persona-Based
Sample Average
Figure 3: Coverage by persona quality (top), covariate shift (center), and selection bias (bottom). DR (Riesz) at- tains better coverage than all baselines. Baselines with 0% coverage omitted to reducc clutter. n 0.1; 0.6 for bottom two rows. Fig. (Appendix presents analogous results for Bias (MAE) and Interval Width.
Synthetic
PRISM
DICES
Method
Bias
Coverage
Width
Bias
Caverage
Width
Bias
Coverage
Width
Sample Avernge
+ 0.17
= 0.03
001
1,30
0.07
IPW
0.48 = 0.0
0.00
1.00 : 0.20
25.49 0.02
0.00
2.62 1 0.0
0.17
0.07
0.10
PAR
0,06
0.44
:: 0.03
0.10
0.83 0.01
0.02 i: 0.01
0.91
0.05
0.04
0.02
Persona Based
0.37 001
0.00
0.17
10,00
0.00
1.33
0.34
0.00
0.05
PPI++
0.69 + 0.16
0.03
+ 8:22
0.17 + 0.1
1.03 0.01
0.00
1.01
0.06
0.01
0.03
RePPI
0.10
0.56
0:09
0.19
0.63
0.66 0.00
1.36
0.04
001
0.40
0.05
DR (Classical)
0.07
0.85 = 0:01
0.21
0.68 + 0.01
0.82 + 0.02
1.75
0.05
0.32
0.06
0.01
DR (Riesz)
0.03
1.00
0.28 + 0.01
0.46 0.01
0.93 0.01
1.68
0.02
0.86
0.01
0.09
Figure 4: Average Bias (MAE), Coverage, and Interval Width across experimental conditions presented in
Fig. Values in parentheses denote standard error (values < 0.01 omitted to reduce clutter).
Synthetic when: (i) persona quality is very high (Fig. 3 top left) and (ii) and dropout rate is high (Fig. 3 bottom left). While counterintuitive, the second observation highlights the importance of examining covariate shift and selection bias in parallel: as dropout rate increases in Synthetic, the mean of remaining source samples more closely resembles that of the target distribution, leading to coincidentally higher coverage. Yet coverage remains poor on both PRISM and DICES.
Finding DR (Riesz) yields improved coverage and lower bias (MAE) than DR (Classical). Across levels of covariate shift, selection bias, and persona quality, we observe DR (Riesz) (Fig. 3 solid lines) obtains improved estimates compared to DR (Classical) (Fig. 3 dashed lines). While this behavior also appears in Synthetic (Fig. 3 left column), the gap between DR (Classical) and
DR (Riesz) is especially pronounced when learning nuisance functions on embeddings of high-
dimensional text (Fig. 3 PRISM, DICES). This illustrates the importance of directly estimating the re-weighting term ao W, C*) (Eq, rather than learning wo ( ()) and To (W) separately.
Finding 3: DR (Riesz) makes better use of persona ratings than baseline estimators. Several of our baselines- - RePPI, PAR, and Persona- Based - use persona ratings to compute estimates. How- ever, across levels of persona quality (Fig 3 top row), DR (Riesz) produces higher quality estimates than these baselines (with valid coverage for p 2 .65 on both PRISM and DICES). Fig. 5 extendss this analysis persona ratings obtained from varying L.MSs DICES. For all LLMs, observe

---

1.0
1.0
Figure 5: Coverage of DR (Riesz) (solid) versus RePPI (dashed) when varying covariate shift (left) and selection bias (right) with persona ratings from different LLM judges. Parentheses denote Pearson correlation between persona and human ratings.
Coverage
Coverage
0.5
0.5
0.800
2.5
5.0
7.5
10.0
0.00
0.05
0.10
0,15
0.20
Covariate Shift Magnitude (A)
Dropout Rate (%)
Ensemble (0.52)
Cloude Halku
Mini 231
GPT-5
that coverage of DR (Riesz) (solid) is markedly higher than that of RePPI (dashed), Further substan- tiating our systematic perturbation study (Fig- 3] top row), we observe that real L. that exhibit a higher correlation with human ratings (e.g., GPT-5; p 0.43) yield improved coverage over those with lower correlation (c.g., GPT-4o Mini; p = 0.23). Furthermore, despite having lower correlation coefficients, observe several models achieving comparable coverage to our artificially perturbed persona ratings (Fig. 3 p = 0.6). Taken together, these findings illustrate that persona ratings from real Luudess-judges can be used to improve downstream estimates under evaluation sampling bias.
RELATED WORK
We now provide a brief overview of related literature (see Appendix for detailed discussion).
Automated Evaluation with Persona Prompting. To address evaluation sampling bias, one strategy is to use an automated rater to rate outputs from the target distribution. Under this LLM-as-
a-judge approach, a judge GenAl system rates the outputs of targer GenAl system (Li et al. 2024
Elangovan et al.] 2024] Ye et al. 2024; Bubeck et al. [023] Zheng et al., (202)). Because human raters often disagree on criteria such as "helpfulness" or "relevance" (Kirk et al. 2024), prior work has explored instructing judge systems to adopt personas
descriptions of humans with specific
characteristics (Castricato et al., 2024 Fröhling et al. 2024; Orlikowski et al. 2025 Deng et al.) 2025). However, work has also shown that persona ratings are often an imperfect proxy for human ratings (Santurkar et al.] 2023: Neumann et al. 2025). Thus, our work treats persona ratings as a useful yet incomplete proxy for human raters improve GenAl system quality estimates.
Frameworks for Sample Efficient Estimation. Other works propose methods for improving statis- tical inference when data is scarce but ML predictions abundant. Prediction-Powee (PPI) and its computationally efficient variant PPI++ use ML predictions to tighten confidence inter- vals through a "rectifier term" that corrects for bias in ML predictions (Angelopoulos et al.. 2003a Chatzi et al. 2024 Fisch et al., 2024, Angelopoulos et al. 2023b). Ji et al. (2025) show PPI++ to be a special case M-estimation with surrogate outcomes, a classical problem causal inference (Robins et al.] 1994 [Robins & Rotnitzky 1995] Tsiatis 2006), and in turn propose recalibrated PPI (or RePPI) to offer more efficient estimation. However, these approaches fail give valid coverage under evaluation sampling bias. We develop doubly-robust estimator (Bang & Robins 2005) |005] Cher- nozhukov et al., 22188) 2023) that can handle covariate shift and selection bias simultaneously while making of surrogate predictions/persona ratings. We also use "Riesz losses" (Chernozhukov
Inference
et al. 2023 2022a b) to estimate complicated nuisance parameters using generic ML learners.
6 CONCLUSION
Our work answers calls for greater consideration of external validity concerns in Generative AI eval- uation /Weidinger et al. [2025] Ibrahim et al. 2024 Liao et al. (Slaudeen et al. 2025) through a theoretically rigorous and empirically validated estimation framework. Our framework provides a path forward for combining limited human ratings observed under sampling bias with imperfect per- sona ratings to obtain statistically valid system quality estimates. Beyond our specific doubly-robust estimation framework, our Persona Simulation Framework (PSF) also provides reusable commu- nity resource for validating future methods designed to address sampling bias. While our framework relaxes the MCAR assumption imposed by existing estimation frameworks, it also imposes assump- tions - e.g., no concept drift- on the evaluation process. Future work should also consider how vi- olations of this and other assumptions in Appendices B and CandCiightt might affect system quality estimates.
10

---

7 REPRODUCIBILITY STATEMENT
We take several steps to ensure the reproducibility of our work. First, we document all the- oretical assumptions required by framework and provide complete proofs in Appendix Second, we provide thorough documentation of our experiment design, hyperparameters, and datasets required to reproduce our empirical results in Appendix t Finally, pub- licly release all code and data so that the broader community can build upon our framework: httts::/githhn www.tchww.ooooggogeeeeeeeeeeeeee com/lguerdan pogguerdan/dubllyyy
doubly y robust 11m judge
CONTRIBUTION STATEMENT
Led all work on experiments. Designed and implemented the Persona Simulation Framework, collected synthetic datasets from LLMs, and implemented estimators included the experimental results. Supported drafting of the main text.
KT:
JW: Led all work on theoretical development and validation of the doubly-robust estimator. Stated and proved all results. Drafted appendices B C and D Conducted early-stage experiments on synthetic data. Supported drafting the main text.
LG: Led conceptual framing of the project. Designed the problem formulation. Supported implementation of the Persona Simulation Framework (c.g., data generation pipeline, Riesz loss). Wrote the main text with support from JW and KT.
ACKNOWLEDGMENTS
We thank members of the Fairness, Ethics, Accountability, and Transparency (FEAT) reading group CMU for their helpful feedback on earlier versions of this work. This research was funded by the National Institute of Standards and Technology University https://oo.org/05x2be3333 AI Measurement Science and Engineering Center (AIMSEC). Luke Guerdan (ORCID: (009-0000933566-990 9429)) was funded by NIST through Federal Award ID Number 6ONANB24D231.
orgpoo00000005511)) and Carnegie Mellon
REFERENCES
Suhaib Abdurahman, Mohammad Atari, Farzan Karimi-Malekabadi, Mona Xue, Jackson Trager, Peter S Park, Preni Golazizian, Ali Omrani, and Morteza Dehghani. Perils and opportunities in using large language models in psychological research. PNAS nexus, 3(7):pgae245, 2024.
Anastasios N Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I Jordan, and Tijana Zmnc. Prediction-powered inference. Science, (62666 674, 2023a.
Anastasios N Angelopoulos, John Duchi, and Tijana Zmic. Ppi++: Efficient prediction-powert inference. arXiv preprini arXiv:2211.01145 2023b.
Ruicheng Ao, Hongyu Chen, and David Simchi-Levi. Prediction-guii guided active experiments. arXiv preprint arXiv:2411. 2024.
Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. Out of one, many: Using language models to simulate human samples. Political Anal- ysis, (33333337-35 2023.
Lora Aroyo, Alex Taylor, Mark Diaz, Christopher Homan, Alicia Parrish, Gregory GerapioGGarite Vinodkumar Prabhakaran, and Ding Wang. Dices dataset: Diversity in conversational ai evalua- tion for safety. Advances in Neural Information Processing Systems, 36:53330-53342. 2023.
Heejung Bang and James M Robins. Doubly robust estimation in missing data and causal inference models. Biometries, 614999922 2005.
Etienne Becht, Leland McInnes, John Healy. Charles-Antoine Dutertre, Immanuel WH Kwok, Lai Guan Ng, Florent Ginhoux, and Evan W Newell. Dimensionality reduction for visualizing single-cell data using umap. Nature biotechnology, 37(1):38 44, 2019.

---

James Bisbee, Joshua D Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M Larson. Synthetic replacements for human survey data? the perils of large language models. Political Analysis, 32 (4):401-416, 2024.
David Broska, Michael Howes, and Austin van Loon. The mixed subjects design: Treating large language models potentially informative observations. Sociological Methods & Research, pp- 00491241251322866) 2024.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee. Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
Yewon Byun, Shantanu Gupta, Zachary Chase Lipton, Rachel Leah Childers, and Bryan Wilder. Valid inference with imperfect synthetic data. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Louis Castricato, Nathan Lile, Rafael Rafailov, Jan-Philipp Fränken, and Chelsea Finn. Persona: reproducible testbed for pluralistic alignment. arXiv preprint arXiv:2407. 17XX1:2407.17887,, 2024.
Ivi Chatzi, Eleni Straitouri, Suhas Thejaswi, and Manuel Gomez Rodriguez. rediction-poweree ranking of large language models. arXiv preprint arXiv:2402.17826 2024.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural pa- rameters, 2018.
Victor Cheroozhukov, Whitney Newey, Victor M Quintas-Martunes and Vasilis Syrgkanis. Riesznet and forestriesz: Automatic debiased machine learning with neural nets and random forests. In International Conference Machine Learning, pp. 3901-3914, PMLR, 2022a.
Victor Chernozhukov, Whitney K Newey, and Rahul Singh. Automatic debiased machine learning of causal and structural effects. Econometrica, 90(3):967-1027, 2022b.
Victor Chernozhukov, Michael Newey, Whitney K Newey, Rahul Singh, and Vasilis Srygkanis. Au- tomatic debiased machine learning for covariate shifts. arXiv preprint 2rXiv:2307.04527, 2023.
Wesley Hanwen Deng, Sunnie SY Kim, Akshita Jha, Ken Holstein, Motahhare Eslami, Lauren Wilcox, and Leon A Gatys. Personateaming: Exploring how introducing personas can improve automated ai red-teaming. arXiv preprint #xXiv:2509.03722 2025.
Yijiang River Dong, Tiancheng Hu, and Nigel Collier. Can Ilm be personalized judge? arXiv preprint 1XXx:22406.11677 2024.
Florian E Dorner, Vivian Y Nastl, and Moritz Hardt. Limits to scalable evaluation at the frontier: Lim as judge won't 't beat twice the data. arXiv preprint 1XXv:24410.13411, 2024.
Rick Durrett. Probability: theory and examples, volume 49. Cambridge university press, 2019.
Naoki Egami, Musashi Hinck, Brandon Stewart, and Hanying Wei. Using imperfect surrogates for downstream inference: Dbsign supervised learning for social science applications of large language models. Advances in Neural Information Processing Systems, 11668899-680011 2023.
Naoki Egami, Musashi Hinck, Brandon M Stewart, and Hanying Wei. Using large language model annotations for the social sciences: A general framework of using predicted variables in down- stream analyses. 2024.
Aparna Elangovan, Lei Xu, Jongwoo Ko, Mahsa Elyasi, Ling Liu, Sravan Bodapati, and Dan Roth. Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and Ilm-as-a-judge. arXiv preprint 11X.v:2410.03775, 2024.
Benjamin Eyre and David Madras. Auto-evaluation with few labels through post-hoc regression.
arXiv preprint X4i:2441.//666 2024.
12

---

Jean Feydy, Thibault Séjourné, François-Xaviss Xavier Vialard, Shun-ichi Amari, Alain Trouve, and Gabriel Peyré. Interpolating between optimal transport and mnd using sinkhorn divergences. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 2681-26999 2019.
Michael G Findley, Kyosuke Kikuta, and Michael Denly, External validity. Annual review of politi- cul science, 24(1):365-393, 2021.
Adam Fisch, Joshua Maynez, R Alex Hofer, Bhuwan Dhingra, Amir Globerson, and William W Cohen. Stratified prediction-powered inference for hybrid language model evaluation. arXiv preprint arXiv:2406.0429/ 2024.
Riccardo Fogliato, Pratik Patil, Mathew Monfort, and Pietro Perona. A framework for efficient model evaluation through stratification, sampling, and estimation. In European Conference on Computer Vision, pp. 140-158. Springer, 2024.
Leon Fröhling, Gianluca Demartini, and Dennis Assenmacher. Personas with attitudes: Controlling Ilms for diverse data annotation. arXiv preprint prXiv:2410.11745: 2024.
Joseph K Goodman and Gabriele Paolacci. Crowdsourcing consumer research. Journal of Consumer Research, 196-210, 2017.
Gloria Guzman and Melissa Kollar. Income in the united states: 2022. United States Census
Bureau. https://www. census. pdf. 2023.
David A Hirshberg and Stefan Wager. Augmented minimax linear estimation. The Annals of Statis- tics, 49(6): -2207 2021.
Pei-Yur Yun Hsuch, Prem Melville, and Vikas Sindhwani. Data quality from crowdsourcing: study of annotation selection criteria. Proceedings of the NAACL HLT 2009 workshop on active learning for natural language processing, pp. 27-35, 2009.
Lujain Ibrahim, Saffron Huang, Lama Ahmad, and Markus Anderljung. Beyond static ai eval- uations: advancing human interaction evaluations for Ilm harms and risks. arXiv preprint arXiv:2405. 10632, 2024.
Wenlong Ji, Lihua Lei, and Tijana Zmic. Predictions as surrogates: Revisiting surrogate outcomes in the age of ai. arXiv preprint (xXxx255010093/1 2025.
Hannah Rose Kirk, Alexander Whitefield, Paul Rottger, Andrew Bean, Katerina Margatina, Rafael Mosquera-Gomes Juan Ciro, Max Bartolo, Adina Williams, Hc He, et al. The prism alignment dataset: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. Advances in Neural Information Processing Systems, 37: 37:105236-005444 2024.
Tobias Leemann, Periklis Petridis, Giuseppe Vietri, Dionysis Manousakas, Aaron Roth, and Sergul Aydore. Auto-gda: Automatic domain adaptation for efficient grounding verification in retrieval augmented generation. arXiv preprint arXiv:2410.03661, 2024.
Kevin E Levay, Jeremy Freese, and James N Druckman. The demographic and political composition of mechanical turk samples. Sage Open, 6(1)211582401166664333 2016.
Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjce, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From generation to judgment: Opportunities and challenges of Ilm-as-a-judge. arXiv preprint 4rX1v:2411.16594. 2024.
Vera Liao and Ziang Xiao. Rethinking model evaluation as narrowing the socio-technical gap. arXiv preprint 20Xx1:2306603100, 2023.
Thomas Liao, Rohan Taori. Inioluwa Deborah Raji, and Ludwig Schmidt. Are we learning yet? meta review evaluation failures across machine learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
13

---

Kevin J Mullinix, Thomas J Leeper, James N Druckman, and Jeremy Freese. The generalizability of survey experiments. Journal of Experimental Political Science, 12)::09-1133 138, 2015.
Terrence Neumann, Maria De-Arteaga, and Sina Fazelpour. Should you use Ilms to simulate opin- ions? quality checks for early-stage deliberation, 2025. URL https::/aaxiv..rrg abs
05C4.089550
Whitney K Newey and Daniel McFadden. Large sample estimation and hypothesis testing. Hand- book of econometrics, 4:2111-2245, 1994.
Jerzy Neyman. C (a) tests and their use. Sankhyā: The Indian Journal of Statistics, Series A, pp- 1-21, 1979.
Ezinne Nwankwo, Lauri Goldkind, and Angela Zhou. Batch-adapticn annotations for causal infer- ence with complex-embedee embedded outcomes. arXiv preprint arXiv:2502. 2025.
Matthias Orlikowski, Jiaxin Pei, Paul Röttger, Philipp Cimiano, David Jurgens, and Dirk Hovy. Beyond demographics: Fine-tuning large language models to predict individuals' subjective text perceptions. arXiv preprint arXiv:2502.20897, 2025.
Siru Ouyang, Shuohang Wang, Yang Liu, Ming Zhong, Yizhu Jiao, Dan Iter, Reid Pryzant, Chen- guang Zhu, Heng Ji, and Jiawei Han. The shifted and the overlooked: A task-oriented investiga- tion of user-gpt interactions. arXiv preprint arXiv:2310.12411 2023.
Peter S Park, Philipp Schoenegger, and Chongyang Zhu, Diminished diversity-of-thought in stan- dard large language model. Behavior Research Methods, 56(6):5754-57700 2024.
Yijiang River Dong. Tiancheng Hu, Yinhong Liu, Ahmet Üstün, and Nigel Collier. When personal- ization meets reality: A multi-faceted analysis personalized preference learning. arXiv e-prints, pp. arXiv-2502, 2025.
James M Robins and Andrea Rotnitzky. Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association, 90(429): 12221122 129, 1995.
James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association, 89 (427):846-80))) 866, 1994.
Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. Ares: An automated evalu- ation framework for retrieval-augmeeted generation systems. arXiv preprint arXiv:2311.09476, 2023.
Olawale Salaudeen, Anka Reuel, Ahmed Ahmed, Suhana Bedi, Zachary Robertson, Sudharsan Sun- dar, Ben Domingue, Angelina Wang, and Sanmi Koyejo. Measurement to meaning: A validity- centered framework for ai evaluation. arXiv preprint arXiv:2505.1057. 2025.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto.
Whose opinions do language models reflect? In International Conference Machine Learning, pp. 29971-30004, PMLR, 2023.
Miriah Steiger, Timir Bharucha, Sukrit Venkatagiri, Martin J Riedl, and Matthew Lease. The psychological well-being of content moderators: the emotional labor of commercial moderation and avenues for improving support. In Proceedings of the 2021 CHI conference on human fuctors in computing systems, pp. 2021.
Kazuhiro Takemoto. The moral machine experiment on large language models. Royal Society open science, 11(2):231393, 2024.
Anastasios Tsiatis. Semiparametric theory and missing data, volume Springer, 2006.
Annie S Ulichney and Amanda Lee Coston. Double machine learning evaluation under distribution shift and selection bias. In NeurIPS 2025 Workshop: Reliable ML from Unreliable Data, 2025.
Aad W Van der Vaart. Asymptotic statistics, volume Cambridge university press, 2000.
14

---

Laura Weidinger, Inioluwa Deborah Raji, Hanna Wallach, Margaret Mitchell, Angelina Wang, Olawale Salaudeen, Rishi Bommasani, Deep Ganguli, Sanmi Koyejo, and William Isaac. To- ward evaluation science for generative ai systems. arXiv preprint arXiv:2503.05336, 2025.
Dustin Wright, Arnav Arora, Nadav Borenstein, Srishti Yadav, Serge Belongie, and Isabelle Augen- Stein. LLM tropes: Revealing fine-grained values and opinions in large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Compu- tational Linguistics: EMNLP 2024, Pp. 77085-17112, Miami, Florida, USA, November 2024. https////////2//nnnnnneeee/m//// findings-emnlp.999. URL
Association for Computational Linguisties. doi: 10. 12.18663/112224
fittp:://cclantien
2024. fi nd ggs emn pp 995/
Zichun Xu, Daniela Witten, and Ali Shojaie. A unified framework for semiparametrically efficient semi-supervised learning. arXiv preprint arXiv:2502.1774. 2025.
Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, al. Justice prejudice? quantifying biases in Ilm-as-a- judge. arXiv preprint arXiv:2410.02736. 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
Haotian Zhou and Ayelet Fishbach. The pitfall of experimenting the web: How unattended selective attrition leads to surprising (yet false) research conclusions. Journal of personality and social psychology, 11 144):493, 2016.
15

---

This Appendix is organized as follows:
Appendix A
provides an extended discussion of related work.
Appendix B
provides formal setup of our framework, notation, and theoretical results.
We extend our analysis to general M-estimators under covariate shift with surrogate (persona) ratings in Appendix C in Appendix follow.
We provide a general proof, from which our results
Appendix
D
provides details
Riesz loss minimizer used perform re-
weighting.
Appendix
details our experimental setup and provides additional empirical results.
A EXTENDED RELATED WORK
A.1 THREATS TO THE EXTERNAL VALIDITY OF GENERATIVE AI EVALUATIONS
In the quantitative social sciences, external validity describes the extent to which findings from study generalize to different populations, settings, and times (Findley et al.) 2021). Threats to ex- ternal validity have long been studied in survey research. For example, Levay et al. (2016) found notable discrepancies between the demographic composition of convenience samples obtained via Amazon Mechanical Turk (MTurk) versus nationally representative American National Election Study (ANES) samples. While Mullinix et al. (2015) observe that study findings often remain ro- bust to such discrepancies, (Zhou & Fishbach 2016) demonstrate that differential non-compliance -a a form of selection bias in which participants drop out from studies non-randomly can have substantive effects studies' results. Myriad factors contribute to this selection bias, such as par- ticipants' motivation and language skills (Goodman & Paolacci, (011)).
More recently, concerns have emerged surrounding the external validity of evaluations obtained from general purpose benchmarks (e.g.. MMLU, BigBench) and leaderboards (e.g.. Chatbot Arena) [lbrahim et al. 2024 Ouyang et al. 2023] Liao & Xiao| (023). As with survey research, GenAI performance measurements can be subject to covariate shift when the distribution of system outputs or human raters differs between a lab-based evaluation and target deployment context (Saad-Falcon
et al. 2023: Leemann et al. 2024 Kirk et al., 2024). Likewise, differential non-compliance can occur when raters in online rating platforms drop-out due to failed quality checks (e.g., due to poor English language proficiency) (Hsueh et al. 2009). Such selection bias can confound results if common factors (e.g., English language proficiency) affect rater drop-out and their ratings. Selection bias can also arise if some raters more likely to voluntarily assign ratings than others c.g., when busy physicians rate complex system outputs less frequently than more available medical students. While growing work has highlighted external validity
important desideratum for evaluations (Ibrahim et af.) 2024; (Ouyang et al.. 2023 Liao & Xiao 2023). to our knowledge, no existing statistical frameworks simultancously address covariate shift, selection bias, and high- dimensional model outputs while leveraging imperfect automated ratings.
We address this gap by developing a statistical framework for characterizing and mitigating threats to the external validity of GenAl performance evaluations. We devise data-efficient estimator that corrects for covariate shift and selection bias parallel, given ratings from a source population and predictions generated by black-box machine learning model over both source and target pop- ulations. Some advances in our framework provide new perspective on classic methodological challenges in survey research. For example, we provide doubly-robust alternative to the reweight- ing estimators traditionally used to correct for selection bias. This approach obtains valid coverage even when the re-weighting model is misspecified. Our framework also addresses novel challenges that arise in the GenAl evaluation context. In particular, we leverage embeddings to support robust statistical inference over nigh-dimensional model output spaces (e.g., text, image) opposed to the structured data formats traditionally used for survey research. Central to our approach is the principled adoption of synthetic ratings generated by an Al persona, which discuss next.
16

---

A.2 AUTOMATED EVALUATION WITH LLM AS-A---UUUAA A-JUDGE AND PERSONA PROMPTING
Given the cost and scalability challenges associated with collecting human ratings, automated meth- ods are increasingly used scale up evaluation workflows traditionally performed by humans. In particular, the LLLM-as-a-judge paradigm introduces a second judge GenAl system to evaluate the
outputs returned by targer GenAl system (Li et al. 2024 Elangovan et al.) 2024 Ye et al. 2024] Bubeck et al. 2023: Zheng et al.
2023). Because human raters can disagree as to whether a model output is "helpful", or "relevant" (Kirk et al., 2024), recent work has proposed instructing judge sys- tems to adopt personas-thal is, descriptions of humans with specific sociodemographic characteris- tics, such as gender and race (Castricato et al. 2024, Dong et al., 2024 Fröhling et al. [024 Wright
et al. 2024 Orlikowski et al., 2025 River Dong et al. 2025). This persona- based prompting strategy is designed to setter-accoum for sources of rater-specific variation throughout the evaluation process.
These automated evaluation methods offer a promising approach to mitigate the external validity threats described in $ [.I In particular, judge systems with persona prompting can generate low-cost symthetic ratings when human ratings from the target population are limited. However, because such ratings may be systematically biased (Santurkar et al.) 2023] Neumann et al., 2025). their direct adoption in evaluation pipelines may yield biased pertormance measurements. Our proposed doubly-robust approach addresses this challenge by treating LLM-as-a-judge ratings as potentially informative yet biased proxies for human ratings. This approach combines surrogate ratings with human ratings (observed under evaluation sampling bias) to obtain statistically valid confidence intervals in the target population of interest.
A.3 GENAI SYSTEMS AS HUMAN SURROGATES IN SOCIAL SCIENCE STUDIES
While our work foregrounds GenAl evaluation challenges, it bears conceptual and methodological similarity to work investigating GenAl systems as surrogates for human subjects in social science studies. Inline with the turn towards crowdworkers as a low-cost surrogate for target study popu- lations (8 A.I). this growing line of work introduces GenAl systems as a surrogate for more costly human subjects (Argyle et al., 2023). ). Notably, such work often targets the very same statistical parameters recovered by our general M-estimation framework (Table []. For example, let v denote an item in opinion poll (e.g., "do you believe in the right bear arms?"), let X denote rater characteristics (e.g., locale and demographics, per (Santurkar et al. 2023)), and let Y represent a binary response (endorse/not endorse) the survey item. The parameter
(v == E V = v]
(5)
denotes the proportion of raters in the target population who endorse this survey item. Thus, re- searchers can also leverage our methodology when using GenAl systems as surrogates for human subjects in social science studies. Given a finite sample of human ratings from the source popu- lation {(x,, Vú, M)) ~ Ps. and surrogate data produced for the source and target population, researchers can recover informative and statistically valid confidence intervals for parameters de- fined over the target population of human subjects.
Given the overlap between our motivating application and social science studies, we also discuss methods advancing the principled adoption of GenAl systems as surrogate data in social science rescarch (Broska et al. 2024; [Egami et al., 2024; 2023). These works view surrogate data
flawed (Bisbee et al. 2024 Park et al., 22024] Takemmooo 2024 Abdurahman et al.] 2024) but po- tentially informative source of information for statistical inference in social science studies. For instance. [Broska et al. 2024) leverage prediction powered inference (Angelopoulos et al. 2023a) to correct for bias in surrogate data. However, as discussed in $ A.44 PP vulnerable covariate shift and selection bias between source and target study populations. Most related to our work, (Egami al. 2024 2023) introduce a doubly robust estimation approach that generalizes PPI by applying a bias clas-correttion to the underlying moment function (as opposed to the outcome variable). Critically, however, this approach takes a design-based sampling procedure, which assumes that the probabil- ity of labeling a sample is known by the researcher in advance. This precludes the more general setting we study in our work, in which the reweighting function is unknown advance g Moreover, Nwankwo et al. 2025) propose doubly-robust estimation approach for batch-adaptive annotation
As noted in |gaai et al. 2024 2023), this dasign-hased sampling approach is well-motivated when the full corpus of documents be annotated and corresponding sampling probabilities are known in advance.
17

---

similar to the framework put forth by (Egami et al. 2024 2023). but focus on causal quantities such as the ATE rather than M-estimation problems.
Having discussed both GenAl and social science applications of our framework, now turn to the underlying statistical methodology we advance in this work.
A.4 STATISTICAL FRAMEWORKS FOR SAMPLE ERFICIENT-ESTIMATION
Recent developments in black-box predictive models that can operate on multi-modal representa- tions has spurred significant interest in how these predictions might be used to improve statistical
inference (Byun et al. 2025| Angelopoulos et al. 2023a b. Fisch et al. 2024 Eyre & Madras 2024]
Dorner et al., 2024 Ji et al. [025] [Saad- Saad-FFalcon et al.] 2023; Fogliato et al. 2024). These frame-
works address the challenge of making valid statistical inferences when labeled data is scarce but black-box predictions cheap and abundant. We briefly review developments in this literature before
identifying key gaps addressed by our approach.
Inference (PPI) uses predictions from black-box machine learning model to tighten confidence intervals when labeled data is scarce (Angelopoulos et al.) 2023a b). This is done through the addition of "rectifier" of the model's predictions on the labeled and unlabeled points. While initial variants of PPI were not computationally efficient, AAneelopoulss et al. (2023b) introduce PPI++ framework, which introduces a "trust" parameter À to control the magnitude of the rectifier. This allows for efficiently computable confidence sets that are provably tighter than those computed just from labeled data. We also emphasize recent work due Jo et al. (2025), which shows that PPI++ is just a special parametric class of solutions for M-estimation with surrogates outcomes a classical, well-studied problem in the causal inference/missing data literature (Robins et al. 1994 Robins Rotnitzky] [995] [Tsiatis (006). While the authors do not directly mention Neyman orthogonal scores (Neyman 1979; Chernozhukov et al.) 2018) in their work, they construct what is implicitly a Neyman orthogonal score for the problem at hand and propose a solution based on cross-fitting Additional theoretical developments along these lines have been proposed - Ao et al. (2024)
Prediction-Power
mean term that contrasts the performance
propose framework for adaptive estimation of linear functionals based on supplied predictions. Likewise, Xu et al. (025) consider general semi parammttri framework for estimating functionals of the data generating distribution in the presence of ML predictions.
However, none of the aforementioned works provides framework usable in settings where (a) unlabeled and labeled samples come from different distributions (i.e. covariate shift) and (b) data is missing random (MAR), (i.e. the probability that outcomes are observed for any given individual We close this gap by proposing doubly-robust estimator (Bang &
depend on their features) [ Robins 2005) and general algorithm based on fittini (Chernozhukov et al., 2018) for solving M-estimation problems in the presence of both covariate shift and heterogeneity in data missingness. We also incorporate recent ideas on "Riesz losses" (Chernozhukov et al., 20226) 2023 2022a] Hirshberg & Wager| 202T), loss functions that specify complicated nuisance functions their min- imizers. By using Riesz losses to learn the eeeiggtting function an oo (W, C),, we avoid constructing plug-in estimates for we and To and computing their quotient, which can result in high bias.
However, this assumption is violated in our setting, in which the true source/target distribution weights are unknown. As a result, the framework proposed by (Egami et al. 2024 2023) is not directly applicable in our motivating setting with evaluation sampling bias.
'Concurrent work by Ulichney & Coston (2025) also studies model evaluation under joint covariate shift and selection bias, deriving a Dias-corrected estimator based on influence functions. While their focus is on pre- deployment risk estimation for black-box prediction models, our framework targets M-estimation in the context of GenAl evaluation. We additionally leverages persona ratings as surrogate outcomes improve inference.
18

---

B ASSUMPTIONS, NOTATION, AND RESULTS FROM SECTION 3
B.I ASSUMPTIONS ON DATA-GENERATING DISTRIBUTIONS
We start by formally describing the data generating processes considered throughout the paper. We start by describing the assumptions we place of the "full-data" source and target distributions.
Assumption 1. We assume there two "full data" distributions source distribution P. and larget distribution Pt. For simplicity, we let W = (X, V) denote the extended set of covariates, and assume W E W where W is some generic measurable space. We assume the following hold:
tuples (X, V, V.C.Y,
1. (No Concept Drift) The conditional distribution of Y given W is the same under Pa and Pt. i.c. for any w
PA( E E E w) P.(Y e E w
for any event E.
2. (Surrogates are Functions of the Data) The observed surrogate Y satisfies:
y f(,, V,e),
where is random variable independent of the vector (X,V,C,)).
3. (Positivity) We have
To (W) S
where To(w):=PPcC W =w)
4. (Conditional Ignorability Under Source) The outcome Y is conditionally independent of given extended covariates W, i.e. we have
Y W,
where the conditional independence is under Ps.
5. (Overlap) The likelihood/density ratio W between P, and Ps, defined as
wo(w) = dP: dP. (w)
exists and is finite almost surely.
We may interchangeably write p or Ps for E {s,t} as the distribution over source and target samples, and Eb and E, interchangeably as the corresponding expectations. We typically write (xx,, vb Cb yb, ,b) and W = (xb, vo) for samples drawn from Po.
We briefly parse the above assumptions. The first simply says that if the distribution of W = (X, V) changes wildly, the conditional distribution Y remains the same. The second assumption regards the predictionsssurrogatt outcomes and is trivially satisfied if γ is the prediction of generative Al model that depends only on W and independent, external sources of randomness. The third assumption, positivity, states that under the source distribution there is always some proba- bility we observe the true outcome. The fourth assumption is an analogue to conditional ignorability from the causal inference literature, and says that the outcome Y is conditionally independent of whether
data is observed given covariates. We note that, under the above assumptions, Ý is also conditionally independent of Y and C given W. Lastly, the final assumption guarantees over- lap, that the support of is contained in the support of - necessary assumption in order to perform inference under covariate shift.
The above assumption concerns fully-observed data in practice, the learner will only ever observe outcomes for samples where C' = 1. That is, there is partial- -oservatissrvvtionn of outcomes in the source population, but outcomes are entirely absent in the target population. We formalize this in the following assumption.
Assumption 2. The learner only ever observes Y for samples where C = In other words, observed samples form each distribution take the following form:
19

---

1. (Source Samples) The learner observes samples of the form Z (X*,V* V C* C Y*, Y*) from P..
2. (Targee Samples) The leamner observes samples of the form Z' (X', Vt, Pt) from Pt.
An important consequence of Assumption is that, even when only observe partial data (per As- sumption 2. we can still identify general classes of estimands under the target distribution Pt. This is clarified in the following lemma.
Lemma B.I. Let be an arbitrary function of (Y, N.W. W.). Then, we have
EE 8t, w*)] = [E C*)S(Y, 8o W
where Bo(w, c) == "of
(w))
Proof. Observe that we have:
(oo ""'' ,C*))(", [.)] = E,
C (W'(W
s(ve pe, WW)
No (Wa)
WW("))
(""((Y,, Pr, (o) (mm |
E,
E
To (W
wo (V))
[CC |W'J (S(Y, 8r, (om w+)
E.
To
E. Wo(("))
W')) w
E, [wo(W"))(Y' y w.)
= ExIf(Y*, Y W))
In the above, the second equality follows from the tower rule for conditional expectations. The third follows from conditional independence, i.e. that C Y Y W The fourth equality follows by dP:
definition of To (WW).. The last equality follows since wD("") JP, (W*) and since the conditional distribution of (Y, 9) is the same under P. and P..
B.2 NOTATION
We now discuss some additional notation that will be leveraged in the sequel.
We will need to condition on independent, random nuisance estimates regularly in the sequel. For b E {s. f), if U another random variable (e.g. U I where ğ denotes a generic nuisance estimate) and (Z,U) is some generic function, we define Pb and EY the distribution and expectation over just the randomness in while conditioning on U,, i.e.
PA(S(ZUU) E F) == Pa(S(Z,U)) E F. U) and
U).
We define the empirical distributions with respect to observations as PN, == N Σこ 8zz and 1 PN. == N, Σin 8zz, where 6 denotes the point-mass distribution on z. Thus, for a general random
function ÿ of data Zb, we have xx,((*)) == N Σ ((z)) and (Nx, == N Σ 7Z)). We define the I ( norm of potentially random R--vul function ğ depending on subset of features C Z as
| (P()) = (122 [IT(() 1/2 = (Ess (113()) 1)
We likewise define the (P(( norm 9 as the l00) = inf (be R P > b) = 0}. where || = max(2 za). Note that whenever is random, these norms are random variables as well.
20

---

Given (random or deterministic) sequence (Xn)n21 in a normed space (X, II-I and deterministic scalar sequence (bm)n21. we say Xn = o(bn) if lim bn
[|I almost surely and X,, = ((,,)
there exists constant B > 0 such that [x I for all n 2 1. We say a sequence of random ba
variables (Xn)n21 converges in probability to zero, denoted by X,. + 0, if we have
lim P([|-nnl 2 =)= =0 for any 0. n-000
We say X, (6.)) if X,, /b, - 0, and X2 Op(b,) if for any 0, there is constant M, 0 such that lim sup P( [ b,, 2 M.) S E.
If (Xn)n2a is sequence of random variables in RR we always refer convergence in probability with respect to the lz-norm, where ||=||p == EE 2
1/P for any S < oc. Likewise, if (22) is a sequence of random matrices, we assume convergence in probability is defined with respect to the operator norm | op == sup uER* || For a non-singular matrix A E Raxd, ||
we let A-1 denote its inverse, and A denote the transpose of the inverse. E R. is a vector, we let y = TT for convenience.
B.3 GENERAL DE-BIASED INFERENCE FOR AN EXPECTED OUTCOME
We now state and prove our main theorem for performing inference on an expected outcome 0g E,[Y" under evaluation sampling bias (i.e. covariate shift and selection bias). We start by stating result that assumes the learner is given nuisance estimates that are independent of the entire sample of data. In the sequel, we describe an extended, cooss-fitting based result that makes more efficient use of the data.
Theorem B.2. Suppose Assumption holds, and assume the learner has access to mutually inde- pendent samples Zi...., I from Pa and Zi...., Dx from Pt, as outlined in Assumption (o(w) and an low s) be true, unknown nuisances given by
Let
cwi 50 (w)
po(w) =
W =]=EE,[- W = w| and ao(w,) c) [Y :E,
where To no(w)= PC = 1 W w) and wo wwww) hold.
dP. (w). Assume the following conditions
dP:
1. (Ratio of Sample Sizes) There is some constant 0 < y < oo such that No/N, + 7
2. (Nuisance Convergence) We have access to estimates Tu & that are independent of the sam- ple such that
1 - Holl ((()) | a0ollo332))))) =r(11
and
7 (ollea aoll oz 1/2)
for b E {8,1)- Further, we assume a(w, 0) = 0.
3. (Boundedness) The representer ao(Ws, C*) and the outcomes Y are almost surely bounded.
Let the estimator 0 be defined via the de-biased equation
0 = Px, " + PN) a(W", ,Cy) cr) CC
- fn]]. p(We,
Then, we have asymptotic linearity, i.e.
N
VN 01)
1 VN:
Σ (wW)) +
7
Σ
ao (W5, 03) (y, po(W))) o1).
VN.
21

---

Furthermore, have
VN,(0 0.) N/(,,
so long as the asymptotic variance, given by
= Vargllo + [oo(W: Var [W]],
non-zero.
The following corollary shows that use the plug-in variance estimate to construct asymptot- ically valid confidence intervals.
Corollary B.3. Under the same assumptions of Theorem B.2 the plug- in variance estimate
=1] 0² = PN.. {ū(W' 9t - + PN, ,*)) -
Nr N.
:
consistent, where І = Ei M(W, Pt). Consequently, if the asymptotic variance o² non- zero, we have
VN. 01) + NN0,1),
and thus
C1-6 :=
VN, 28/2,
VN, 26/2
is a - confidence interval for Oz, where denotes the ê quantile of a standard normal random variable.
The proofs of Theorem B.2 and Corollary B.3 follow immediately from applying Theorem
and Corollary C.3 in Appendix C (which concerns the case of general M-estimation) to the score m(w, y2 0)
0.
B.4
FOR MEANS
We now provide cross-fitting algorithm for estimating 0: = EY] and state an analogue of The- orem B.2 In short, cross-fitting works by splitting the data in K folds of roughly equal size. If I denotes the kth fold of data, the algorithm uses all data outside the kth fold (so, in the complement set I to construct nuisance estimates. These nuisance estimates are then used estimate the mean on the kth fold. This splitting strategy ensures that, each fold, the nuisance estimates and trans- formed data are independent of one another. This allows one to apply the asymptotic linearity result of Theorem B.2 on each fold to asymptotic normality of the cross fittinn estimate.
We now state the cross-fitting algorithm (Algorithm 2 and corresponding convergence theorem (Theorem B.4). The proof of the cross-fittine fitting result follows from Theorem C.4 in Appendix C generic result on cross-fitting for M-estimators under sampling bias.
Theorem B.4. Assume the same setup Theorem B.2 and let f((-)) a(k)) 0, and T be in Algorithm Further, suppose the second assumption of Theorem B.2 holds for each muisance estimate f ".....T' -K- and -11 ,..., G(-K),
3(-K),Then,, J.The, have
VNr Ō 0,) N0.
Thus, the set C1-8 defined Corollary B.3 still serves 1 asymptotic confidence interval for 0,.
22

---

Algorithm 2 Doubly-Robun Estimator with K-fold Cross-Fitting (detailed version of Algorithm
1: Input: Samples D.
(Zi,... 2 from Ps. samples Dj { 1 NN,, from Pt.
number of folds K.
2: Randomly split source indices [Nv]] into random folds of equal size: ΣI,.... .I...., .I.
3: for k [K] do
4: Produce ML regression estimate fi(-k) using Dr where D3,k == (Z : Ix).
5: Produce MI. nuisance estimate @ using D and D..
6: Produce parameter and variance estimates:
6..
(W1 P!)
N
K
(Wj, 03) {vī x -W3 87)}
2
JEIx
p (W! Pi)) mi }²
N
N
(W (Y - p71)
N₂ Ne
JEL
where m N,
Compute the average of the K estimates: 0 0= E On and x2 K Σx= oz.
7:
8: Return: Mean estimate and variance estimate 0².
C GENERAL M-ESTIMATORS UNDER COVARIATE SHIFT
In this appendix, prove general result the convergence of de-biased M-estimators under covariate and differential non-compliance. Our results from the previous appendix, which regarded the special case where the target parameter was the expected outcome 0, E, follow special case of the following.
Theorem C.1. Suppose Assumption 7 holds. and assume the learner has access to mutually inde- pendent samples .i.... 7i,, from P. and Zi...., Zy, from P., outlined in Assumption (o(w)) and oo(w, c) be the true, unknown nuisances given by
Let
wo(w) To (w) C
vo(w) = (m(We, Y'; W w] and
oo c)
where no(w)= == PC = W w) and wo(w) = dp hold.
drs (w). Suppose the following conditions
1. (Ratio of Sample Sizes) There is some constant 0 < < co such that N./N, - 7
2. (Nuisance Convergence) We have access to estimates 3. â that are independent of the sam- ple such that
1 - (oll lā aol (P(2)) = =1)
and
1 polle l|a aoll or(NN 1/2) for E (8,1). Further, we assume a(w, 0)
3. (Boundedness) The representer on (W', C*) is almost surely bounded.
4. (Score Regularity) The score m(w, y; 0) satisfies the following regularity conditions:
(a) (Continuity) m(w, y: + Rd is defined and continuous on a compact subset
① C Rdd
23

---

strict generalization of the results in (b) (Unique Solution) There unique solution 0, Rd equation 0 E,[m(W*,,Y' 81).. Further, 0, Qin
(c) (Jacobian) The score m(w, y: f) is continuously differentiable with respect to 0, and the Jacobian Jo E.[Vom(W' Y [,]] non-singular.
(d) (Boundedness) We have
sup [m[[w y: 0)l,,, sup (V, yi llop S D, 0,w.y
8,w.y
for some universal constant D O.
Let 0 be defined as the solution to the empirical estimating equation:
(N, 9/1 + +NN, C3) (mWW, Y3: - 8;)
(6)
Then, we have asymptotic linearity:
-1 Jo VN. 01) VN."
vo(w") + Σ ao ,C)) (r;0)) co((W)) +op(1).
Consequently, we have that
VN. 0.) = NO),
so long as the asymptotic variance, given by
En = Jo (Var, (arr()o(W"") EE, [oo(W* CC*) (m("*, W*)] Joz positive definite.
Remark C.2. Many statistical parameters of interest can be specified via M-estimation problems. We consider three relevant examples below.
First, if we are interested the mean outcome 0, = Ey] (which was the focus of the previous appendix), this can be trivially specified via the estimating equation:
m(w, y; 0) 0.
Thus, the contributions of this appendix Appendix B
2. Next, suppose we are interested in the variance of responses under the target distribution, i.e. 0, := Var,[Y] == E,((Y - [.()))))] Then, we can define the stacked estimating equation
m(w, ys (p, (1) y リーP p-) -w-
If 7) (p., denotes the solution and consequently 0, Var,[Y].
E, [WW", YY, :))] we note that Pt EY[
3. Lastly, suppose Q (0, and that interested in performing inference the Qth quantile of Y under P,, i.e. 02 Fy: F (Q) where (z) == P. < x)) denotes the CDF of Y under the target distribution, which assume is invertible. Define the estimating equation
m(w, yi 0) == Q 1 (
Then, one can check 0 = (mm(WW (, 8,)] by definition of the Qth quantile.
We also note that, the aforementioned examples, the M-estimators only depend on observed outcomes Y and not the extended set of covariates W. While this is typically the case for most parameters of interest, we allow m to depend on W for the sake of generality.
HHere, éuit denotes the interior of θ i.c. the largest open contained
24

---

The following corollary shows how use the above result to construct asymptoticallly-aail confidence intervals. This accomplished by normalizing the parameter estimate 0 by the square root the classic "sandwich" variance estimator. The consistency of this estimator follows from stan- dard proof techniques (see Van der Vaart (2000); Chernozhukov al. (018)) ). With the consistency the variance estimate, the result then follows from an application of the continuous mapping theo- rem.
Corollary C.3. Define the plug-in "sandwich" " variance estimator as Σ プーj
where V and J are respectively defined
V
N. Σ
+
Σ N. N. j=1
N₂
N,
a(W", cupa { mm(", - Vo(W", pPo) 82 -
}
No
= 1 N.
Σ
2 Com O)
N₂
Then, under the same assumptions of Theorem C.I Σ consistent, and hence
V N,S 1/211/2(0
0o) = La).
Thus, for any fixed unit vector v E R2, the set
v VT
v vT
= v v C1-A:=
28/2, v 0+1
N₂ 28/2
N₂
forms a 1 confidence interval for v Oc.
C.1 CROSS-FITTING FOR M ESTIMAATORS
As in Appendix B we provide cross cross-fitting algorithm that allows the learner to make more effi- cient use of the available data. We also state a corresponding convergence theorem (an analogue of Theorem B.4). whose proof follows from from applying the asymptotic linearity of estimators each fold.
Theorem C.4. Assume the same setup as Theorem C.1 and suppose T(-k), (,kk(kk)k0 and are as in Algorithm 3 Further, suppose the second Assumption of Theorem C.1 holds for each nuisance estimate (...,,6(KK) K) and (11)) .!!
Then, we have
1 0₂) = AO La).
Thus, the set C1-8 defined in Corollary C.3 still serves as a 1 8 asymptotic confidence interval for 0..
Proof. First, we know from Theorem C.1 that
N
-1
0. -x
(oW
N₂
Kr
Jo
Σ
ao(W C,W (mVV)) + ON
N
JEZL
Consequently, we have
θ-0,- 1
0 01)
Σ 4ol (W!) + Ky Σ
Co (m(1 vo(W))) tos
N
JEIN
vo (WF) +
oo Cj) (m Yf;0,) (6((W))) Oo
N1
==1
25

---

Algorithm 3 Doubly-Robush M-Estimation with K-fold Cross-Fitting
1: Input: Samples D. = ..,, 2x, from Ps. samples Dj ( ZN,, from P..
number of folds K.
2: Randomly split source indices [Nv]] random folds of equal size: ;
3: for k [K] do
4: Produce ML regression estimate x)(-) using D where Da (2f ; E Iz).
5: Produce ML nuisance estimate ( using D and D,.
6: Let @LR) solve Equation (6). i.c.
N
1
(W?, 87)
=
N
K
x) (Wj. сj) {m(W3.X3;00 - T (W3 (8)
JET
7:
Let JR. Ve, and Σg be
given as
Ju K
al a (W3, K3,
JEZe
V8
1
3 (W- 8ro02
N
N K
(3(-)((W3))) (Wz.C3)2 -m. 879}} 82
}
N₂
:= J. J.J
8: Compute the average of the K estimates: 0: K Σx 0z and E = Ex S1.
9: Return: Estimate and variance estimate E.
The result that ON - 0₂) = (0, En) follows immediately from the above asymptotic linearity.
Next, observe that Corollary
C.3 yields that, for each & E [K]. the variance estimate Σg is consistent
have Σ Σ E k= (C0 + (1)} = 2 + (1). Thus, we have the consistency of the cross-fit variance estimate. Since No > which follows from non-singularity of Jo and V,, the continuous mapping
for Eg. i.e. that have Σg Eo + op(1), and consequently
R Ek
theorem also implies that E-1/2 1-2 (1).). As a consequence, we have
S-1/2 - 0,) (S-1/2 - 2n 11) (0 - 0.)
1/2 0.) - (1() ON, 1/2)
1/1
J
Σ
vo (W.) + Σ
oo W (3) YF:0) - + (N, 2/2).
N₂
i=1
1=1
In partieular, this implies VN 01) (0, 1), proving the other claim.
26

---

C.2 PROOF OF THEOREM C.1
Proof. Observe that, by the definition of we have the identity
0 = FN, + + PN. {a(w*, (")("(W"" rx;0) - -WW", ((W*)9)) {-))}
PN, P1) + PN, (a(W* (C))((WW' ;2;0) - :( 80))} }
Y"; +NN. a(W"" """"",")Vom(W"" ("N () 0.)3
where the second equality follows from performing first-order Taylor expansion with mean-value theorem remainder and A [0,, which be continuously differentiable (this implies Pv, {ā(W*, ""ll"""")m((") Cm) 0);0)} is also continuously differentiable w.r.t. 0). Rearranging the above expression, arrive
able apply since m y; 0) is assumed to
VN. 01) = -v PN, (,, Vom (.. Y2;
("M) PN. a(W", (mmm"" T( {
To prove the desired asymptotic linearity result, we need to show two things.
1. We must show that Ti converges in probability the true Jacobian, i.e. that Ti Jo- + (1))
2. Next, we need to show that
T- PN, too (W") PN, ao C*) (m( 0=; (W())) or ON,N,
After we have shown both desiderata above to be true, we can piece the result together. In particular, we have
(Ā 0.) = - -. + op())
(m(W",,";;)) x IPN.VOWW's) + PNN, (W" C*) {m((",, r*;00) +oo') ]
VN. Σ Jo (ao(W" (' (au(W",C))(mM""" (r.,, - oe(),,
which provides the desired asymptotic linearity result. Asymptotic normality now follows immedi- ately from the above.
Analyzing T2: First, we argue that T3 is asymptotically linear. To do this, we primarily follow the proof of Theorem 1 in Chernozhukov et (2023). For notational ease, let M == m(W', Y": 0o). We can rewrite Tr as
T2 = N, to (W*) + Po (*) (M* [o + R₁ + RS + Rs.
where (letting v/o 7) = vw((w)
Rj = PNN. { ( (w",, 80 + PN,, {ao(W", (1b B)
R₂ = PN,, { an) (W" C* (²) [M* - (W"))
R3 = PNN, { (ā (o" C²) (v/ - (w, pn)].
We show that R1, Rg, Rg = op 1/2) (or equivalently that the above terms are Or(N
since N. O(N)) by assumption).
Since Ey [ wWW Po]]
27

---

Ex ao(W", (e')0 - (W,,
{']] by Lemma B.1 we have
RI = (PN, - E3z) { (V5 - wow 80) + (Px, - B2) {ao(W' (Co - (W. 80)}
Ri
R
Using the same steps from in the proof of Theorem of Cheroozuukov et al. 2023 and letting (+ - (b)) denote the l component of voo. we have
12
Σ ES NI,N E3) (Wo,, 8092)
Var'z [100 - ww) }T]
N₂
1
B22 ( wh) 8o1e)
(Since Var[X] s EX²)
N
k=1
| w (o)))
No
Thus, for any E > O, we have, via an application of the tower rule and Chebyshev's inequality,
P,(N Rilla 2 +) = E. (N /2
R l2 2 s)
2 - E |1 vho l12)())) = (1),
where the final equality follows since - ( = (1) and - v'oll (P))) = O(1) imply
lim,-+00 E. 1 collza(pc)- = 0. Thus, since > O was arbitrary, R' = oz (N, 1/3)
Next, since ||00||== = O()), an analogous argument yields that
1 N.
E R1
|1
e/o (P())))
and thus working through the same argument involving conditionally applying Chebyshev's inequal- ity yields R = (N, 1/2), which in turn shows RS = oz(N. 1/3) or (N, 1/2).
Next, we bound R2. Again start by bounding the conditional expectation of the norm of Ra given a. Since [Cov(m((X* "Y,,,,,;,, Y2;60) W = =O1) by the assumption that m(x y: 8) is bounded, we have
d
Ex R2
Ex
(ā a0) -
k=1
1
Ey (PN, { (W", -vo(W")a))
N A=1
Ez ((@
(oW()))))
N₂
E ((a (o)) (M W*))
N A=1
< N sup [r((Cov W = w)}l B2 an)(W*,
1
a aol aoll22(23)
N,
From this, have again via applying Chebyshev's inequality conditionally that R₂ 1/22
28

---

Finally, we bound Rg. We have
E3
R3
Ez (a
(o) (WV poo
(Since ||o|| S |
<
|ā aol - -oo.llle-ooolllcrrr. v0,Alle2(2))
(Cauchy-Schwarc
k=1
Vd an aoll - 1 carg))
0z (/2
where the final inequality follows because | S vdl== جاا and the final equality follows by sumption on nuisance estimation rates. Conditionally applying Markov's inequality yields that R3 dr (N, 1/2) (N,,1/)) thus proving the desired asymptotic linearity result for T2.
Analyzing Ti: Next, we argue that 7i (N, and a E (3())) we define:
P J(O,, an) = Jo. where for any 0 E θ
N.-000
:- (" Vom Y=; E add
JN, (f. := fa(W", omm Y*;0) Rdxd
Throughout this part of the proof, we assume that is consistent estimate of f2, i.e. that 1|0- (1))). We formally prove this in sequel. Note we can write
[|Ti ((0,, < 1|,, (0, @) J(0 aloop + aJ(0,a) J(Ø ax) (o) |] 0o)
J J( (00,,aa)laa an)l
Rs
We show R1, R2, Rs = (1). which suffices to prove the result.
To show Ri (1), it suffices show that supsee ||JN, (0, J [0, (0)o))lppppp (p().. We know that for any fixed iquare-integrable function a(w, c), since Vem(w, y; 0) is bounded above in operator norm by some constant D, have cla(w,))Voml Vom(w, y; 0)lopp I Dla(w, c)l, and so the collection of scores possesses an integrable envelope. Further, since Vern is continuous well. Lastly, since θ compact, Lemma 2.4 of Newey & McFadden 1994) yields that (a(w, w,, c) V Vem(w, y: 0) 0 E Q is weak Glivenko-Cantell class, 1.e. that
is continuous in
0. the score ax w. Vom(w, y;
sup (0, a) - [(0 a) of (1).
Since â is independent of Ei...., 2% , and bounded, we get for any >>
>0)
lim P. (sup JN, (0, (((,)) [p No000
lim N.0
Eg Pz
sup JJN, (0, J(O, x)
dv, (ā)
0.
In the above, the final limit follows because limin- ΦN (a) = 0 by Equation which allows us apply the bounded convergence theorem (see Chapter of Durrett 2019 sup, [| a) - -(e, oo = op(1).
Thus, we have
Next, we show R2 or (1). Again, it actually suffices to show that supsee (1((0,))
(O, aw)llop = (1),,, which we now show. Observe that, for any fixed 0 E θ, we have
(] J(0, lop = EB (ā - C" Y)1l)]lla.
$ E lā ("m y 7))
s DEZ lI( an)(W',C*)
s DIlā aoll
(1(
(Nuisance consistency).
29

---

Lastly, show that R3 (1). This follows as we have
Ez <vo(W" C*) Vomm(W r²; Vom(W y=; }1 op
R₂
(PP) E& Vom r²; Vom(W" x=; op
(1),
where the final equality follows from the continuous mapping theorem and the fact that con- sistent for 0g- Since have showed all three terms converge in probability to zero, have that T Jo = (n @) [0,, on) (1). proving the result.
Consistency of 0: We now argue the consistency of 0. To do this, we first show that
PPx 22112 (1( and [Px,a(W*, C*) 8-)]22 o=()).
(8)
We just show the second quantity approaches zero in probability. Showing the former approaches zero follows from a similar, simpler argument. We have
PN, a(W C*) CWW 80) = (PN. - EZ) {a(w", (a(W",""))(W" (") ao(W", (()),))))))("")))
R1
(NN, - E2)) {ao(W" c'))o(W"))]
R2
Ez a(W*, """ Yr) ao ("")",(W"))
which follows since E5 [oxo (*)) = 0.
Now, since og and the almost surely bounded, we have R₂ oF(1) by the weak law of large numbers. Next, we can show R₁ = or(1) by conditionally applying Chebyshev's inequality. In particular, for any e 0, we have
Ra 2 c)
E, [P2([ R1ll 2 c)
1
- oo"",, -"0 [?)]
<
E
Ex
(PNN - -Ez) (a("",c*)
2
1
E,
Σ
E%
(PPN. - Ez) C") ('), ao(W' C*) 1)
2
k=
1 N₂e²*
Ta(W's "'W",""))(WW (9),, ao (o
E
Varz
1 N&e²
("(""""") ""
E,
E
k=
1
"ā("","")<"' 8d), - -00 8 8r) 1²)
E,
12
Nge²*
Σ E ((oo(W)
d
((ooW""C"))WW CW" P9), - (W" CoWo(W>),) (W" } 1²)
E,
N₂c²
-==1
Eg [llā - (allolll)))))) + +E [ vollea(p)))
N2e²
(1),
where the second to last inequality follows from the fact that Var[X] < EX?, the second to in- equality follows from adding and subtracting on (W", O* (W P²)x, applying the parallelogram
30

---

inequality, and the final inequality follows from the boundedness of nuisances and nuisance esti- mates. The last line follows from the fact that nuisance estimates and bounded and consistent.
Lastly, argue that R3 (1). We have
Ra B& a(W (W) yoy aol C")o( -") Ez ((Ww CW) 81) @ C") ao ao (C ||
@ C*) ao ( C*) 1 colli
$ و aoll) + + wol (P()) = o=
where the last inequality follows from the monotonicity of L norms. Thus, we have shown that both terms in Equation ( converge zero in probability. Going forward, for convenience, define the population and sample scores respectively as
M(O, ax) = Ez (a(W""C*)m((( (")mm Y) Now, by uniqueness of the solution O, to the equation (0, and continuity M in to show 0 = 0g + (1),,, it suffices to show that
M, (0, a): == PN. (W", (') ,Y"; and
sup M,, M(0, 0n)ll2 = or(1).
eee
To accomplish this, by the triangle inequality, suffices to show that the terms Ri and R2 defined respectively as
R1 sup sup (0, ((0 2)12, := sup (I, - (0, 0o)ll2
both converge to zero in probability. Since we have assumed m(w, 0) is bounded by assumption, can again use Lemma 2.4 of Newey & McFadden (1994) obtain that supp M (0, a) M(0, a) o1) ) for each fixed, square-integrable a. The bounded convergence theorem them yields that, for any c 0,
lim P. supl| (0, - M(0,a)l22>
t00
lim
Eg Pz
sup MN, (0,@) ((0.@)]l2 >
No
Eg
lim
P2
sup MN. (0, M(0 (00,0)112 >
No00o
where we able to interchange limits and integration in the third line by the bounded convergence theorem. Thus we have R2 (1). Next, observe that we have
R₂ sup E. [( ao)(W", C')mm("* Y2;0)]l2
9
sup [| y: 0) 1) J(W". C*) an (W*,C*)
0,w00
< D Dí anl (2(P*)
or(1),
since assume sup, |[ y; 1) < D for all w, y and | aolle2(P1) op(1) by nuisance consistency. This completes the proof of consistency.
31

---

D DETAILS ON RIESZ LOSSES
In this section, we discuss the Riesz loss outlined in Equation (4) Introduced in Chernozhukov et al.
(2022b) and later in expanded upon in Chernozhukov et (2022a 2023), Riesz losses provide
principled approach rooted in empirical risk minimization framework for estimating complicated nuisances.
In this appendix, we specifically consider the problem of estimating the nuisance function ao(w, c) := c _wf(w) ing og would be to construct MI. estimators for wo and To. say by using the predicted probabilities associated with a classifier. The issue with this naive "plug-in" approach is twofold. First, a high- source/target membership will not necessarily
"a(")" where wp and Ty are outlined in Section 3 The naive approach for estimat-
quality classifier for predicting non-compliance yield consistent conditional probability estimates. Second, since depends on the ratio between wy and We any errors in nuisance estimation will compound multiplicatively.
Instead of constructing ilug-in estimates, we can directly learn Qg via loss minimization. The fol- lowing proposition shows that the Riesz loss outlined in Equation directly specifies as its mini- [o(w)
mizer (w( = mp(w)"
Proposition D.1. The function (o(w) satisfies:
30 = arg min B:W-R
where the argument minimizer taken over all measurable functions W.
Proof. First, observe that we trivially have
Bo = arg min E. [C (3(W) - (o))))
arg min [C (CC)) I. - (W) |
arg min [E_ (
where the final inequality follows from noting that [, (W)) has bearing argument minimizer. Next, observe that we can equivalently write
Eg [C 3 ( 3WW) = (3W)]
Putting these two observations together yields the desired result.
In the setting of Algorithm we can solve the empirical version of the loss each fold to estimate S. In particular, let 3 be defined as
B(-b) arg min BEF
K (K
BWW,
N Σ 8(w3)
(9)
1)N,
J$I
where F. denotes a chosen class of functions, In our applications (as discussed in Subsection Appendix E) we choose learn Po over class of feed-forward neural networks.
E.3
of
32

---

E EXPERIMENT SETUP DETAILS
E. SYNTHETIC DATASET
Synthetic Data-Generating Process. We define the oracle nuisance functions:
xx( P(S x| = (71 (X))
(10)
Mo(X)= = au + xax
(11)
dP: (x) dPs
(X)
II
d,
Ptái
=[X,=1]
(12)
Ps.j
j=1
((X) clip(p Y + v1 p² Zoy + [1Imax Vmunn, Ymuin 3mmax)
(13)
where ((X) represents polynomial feature expansion with interactions (degree 2), Ps.j and Pr.j are the Bernoulli parameters for feature in source and target domains respectively, controls the correlation between true and surrogate ratings, Vmin Mmax define the interval of the rating scale, and Z represents independent Gaussian noise.
We produce source and target datasets D, and Dj via the following procedure:
1. Sample domain membership: Bernoulli(p:) where p: = 12
n. 2. Sample categorical covariates: For each feature j E {1..... dz) If A = 0 (source): X5 ~ 2 (ernoulli(p...) -
A 1 (target): X,- ~ 2 Bernoullipp..) -
This yields X, E [-1, with different probabilities across domains.
3. Sample compliance status: For source domain only (AA = O):
(Bernoull(mT{(7)
where compliance probability is determined by the scaled propensity model:
To X = a
700 3
3 xx)
+
and E [0.001, 10] controls non-compliance rates (higher 8 more non-compliance) For target domain: S = 0 (no ratings available).
4. Generate true outcomes:
== ter, ey - N(0,0))
5. Generate surrogate predictions:
V clip(p Y V1- f. Zay 7yymax Vmnin) Ymin Fmax)
where Z V(0,03) E [o, 1] controls correlation, and represents systematic bias.
6. Apply censoring: True ratings Y are only observed when S = (compliant source raters).
We instantiate the above procedure with the following parameters d, 5, og 1.0, P. (0.6, 0.6,0..6.... 0.6, 0.6,0.6). 0.6). Pt 0,3.0,,,,,,,44,,0333 All synthetic experiments are run with
N, 2500 and N, 2500.
E.2 ESTIMATION STRATEGIES
We now more formally describe the various estimators that we compare to our doubly-robun esti- mator.
33

---

1. Sample Average: The source estimator simply averages the samples coming from the source mean for which outcome is observed, i.c. produces an estimate frouce given by
N, Lj=1 C3 j=1
grouree
Yg.
Given that this approach entirely ignores covariate shift and selection bias, one should not expect it to be a consistent estimate of either source or target mean. We compute variance T source via
NO Σ CCY,, - giource)?
source Σj C
2. Persona-Based:
This approach opts to ignore source samples and instead averages the persona prediction Y from the target distribution. That is, it produces and estimate personn given by
N
Opersona
N
This approach may perform well if persona predictions unbiased for true outcomes, but otherwise may be highly biased. The plug-in variance estimate we consider is
[Y
@persona
N.
3. Persona Augmented Regression (PAR): The next approach uses the source data to esti- mate the outcome regression (o( == E. W] E, [Y|]) W]. We use the entirety of the source data D, to learn a model fiw 9) predicting Ho (we describe our particular nuisance estimation strategy below in Subsection E.3 Then, we compute our estimate grar
by
Oer
Σ p(W,
We expect asymptotically normal confidence intervals constructed with this estimator to yield valid coverage only if we are able to estimate Ao are fast, parametric rates. The corresponding plug-in variance estimate
> (D(W,, 8) -@par
N
4. Inverse Propensity Weighted (IPW): Instead of estimating the regression function, one can instead estimate the reweighting coefficient ao(w, ( and then use the esti- mated coefficient to re-weight labeled samples from the source distribution. To construct our IPW estimate, we again use K-fold cross-fitting, constructing estimate @(=)) (W, C) by using the data D and D, on fold, as outlined in Algorithm [ nuisance estimator used below. Then, we construct our estimate as
We discuss the specific
@ipw
E a(-R) (W,,C,)%, k=
N
Once again, we only expect intervals constructed around this estimator to yield valid cov- erage if estimation of ag occurs at parametric rates. The corresponding variance estimate is
2 (- ( (W,,C,) {y, @ipw 2
jpw
N
k=
34

---

5. PPI++: We leverage the implementation of PPI++ found in Angelopoulos et al. (2023b) for computing both the estimator PPP itself and the sample variance o which we use for constructing confidence intervals.
6. RePPI:
We implement the main algorithm in Ji et al. (0255 (Algorithm 1) for the point es-
timate gRePPI and leverage the variance estimate Reppi outlined in Theorem of their work.
We describe our approach for learning the recalibration function also in Subsection E.3
E.3 NUISANCE FUNCTION LEARNING
We perform cross-fitting with K 5 folds for DR approaches and IPW. We select the model for Bo(w) so(w) and for our outcome regression through hyperparameter tuning. These nuisance models are used to obtain estimates. We run this procedure separately for Synthetie, DICES, and PRISM, and retain the same set of hyperparameters for all settings of covariate shift and selection bias in each setting. For each setting, we sample from a grid containing the hyperparameters shown in Table 2 We found that weaker models (hidden dimension 32) better learned reweighting across different magnitudes of covariate shift, while deeper models (hidden dimension 64) better captured high non-compliance. For results reported in this paper, opted for the weaker model to increase variance in the outcome regression and improve coverage across range of covariate shift magni- tudes.
[n(w)
Table 2: Hyperparameter values used for optimizing effective sample size and validation set r
Model
Parameter
Values
Weight Decay
1 10-4
Epochs
{6, 7,899 9}
Beta Net
Hidden Dimension
{32, 64) 0.001
Learning Rate
Scheduler Epochs
Model Type
Random Forest (0.05, 0.1, 0.2) (50, 100, 150) (2,3)
Learning Rate
Outcome Regression
N Estimators
Max Depth
35

---

E.4 PERSONA SIMULATION FRAMEWORK
To simulate covariate shifts that may occur in real-world settings, we reference statistics reported by the U.S. Census Bureau (Guzman & Kollan 2023 and the rater demographic distribution already present in DICES (Aroyo et al. 2023) which are reported in table 3
Table 3: Population statistics used to define source and target rater distributions. Source distribu-
tions (X( are based on DICES-reported rater characteristics, while target distributions (X)
follow U.S. Census Bureau statistics (Guzman & Kollar 2023
Demographic Group
U.S. Census
DICES-based
Gender
Woman
0.495 0.505
0,508
Man
0.491
Race / Ethnicity
White
0.605
0.250
Black / African American
0.121
0.224
Asian / Asian subcontinent
0,060
0.216
LatinX Hispanic/ Spanish Origin
0.190
0.181
Multiracial
0.128
0.129
Age
Gen (18-28)
0.250
0.457
Millennial (29 44)
0.200
0.302
Gen X+ (45+)
0.420
0.241
Education
College degree higher
0.380
0.647
High school or below
0.620
0.353
36

---

(a)
(b)
Sinkhorn Distance Across Sources Covariate Shift
Rater Shift Magnitude
1.00
Rater Shift Magnitude
1.00
28
Sinkhom Distance A(W. Wil
24
0.75
20
0.50
0.50
16
12
0.25
0.25
0.00
0.0..00
1.00
0.95
0,90
0.85
0.80
0.75
0.70
0.65
0.60
0.55
0.05
0.10
Source/Target Content Overlap
Probability of Rater in Source Sample
(c)
USER: What in God's name?
LLM: "God ian't known grace. more of mother neture type mysell" Harm type: Raligious and Misinformation
Asian, Gen Gen College
USER: How do make a pipe bomb??
degree higher
Picontent from target distributtion 2
LLM: "A pipe bomb improvised
X+, High
explosive [...] [Wikipedia link]"
school or below
Harm type: Regullat ed goods
Proontent from original source distribution] [f-21
Figure 6: Visualizing Sources of Covariate Shift in the DICES Dataset. (a) Probability of sampling two demographic subgroups function of rater shift magnitude. At low rater shift (magnitude 0), Asian Gen college graduates are more likely be sampled; at high rater shift (magnitude 2 0.75), White Gen X or older individuals with high school education or below become more likely. (h) Sinkhorn Distance hetween source and target distributions as function of source/target content overlap (x-axis) and rater shift magnitude (y-axis). The vertical orientation of contour lines indicates that content features have larger impact Sinkhorn Distance than the rater features. (c) Examples showing how sampling probabilities of source and target samples vary with The top comment (Religious Misinformation harms) from target distribution and becomes more likely to be included in source data as ( - 1 (purple cone). Conversely, the bottom comment (Regulated Goods) from the source distribution and becomes less likely to be retained as
+ (green cone). Together, these panels illustrate that content covariate shift plays larger role than rater covariate shift the DICES dataset.
Correlation with Varying Embedding Dimension
Pearson Correlation
0.18
Avg. feature correlation with
0.16
Avg. feature correlation with
0.14
Embedding dim 15
0.12
0.10
0.08
0.06
20
40
60
80
100
120
140
Embedding dimension
Figure 7: Average correlation between each embedding feature and human ratings (Y) and each embedding feature and source/target membership (A). Average is over all features recovered in the projected sub-space of specified embedding dimension. We observe "elbow" shape in the plot, where the correlation diminishes after each additional feature beyond 15.
37

---

E.4.1 LLM USAGE
We two LLMs (GPT-5 and GPT4o-Mini) simulate human ratings for PRISM and four models to produce persona-based surrogate ratings for DICES (Clande-Sonnet -Snnett3... 5, Claude-Haikuu HateeHHaik-3... 5, GPT4o-Mini, and GPT-5). We use API calls for all LLMs with tem- peratures of 0.7 (for results reported in the main paper) and 0.9 for Anthropic models and for OpenAl models. We use GPT-5 5 with reasoning set "low." Our exact system and user prompts
are shown in Fig 9 10
System Prompt (With Abstain Option)
You going role-play user with the following demographic characteristics:
Gender: {gender)
Race: {race}
Age: {age}
Education: {education]
Now, imagining that you possess these characteristics, you should evaluate the Al assistant's response to your user queries.
All prompts will be of the format: "[rating instructions)"
YOUR QUERY AS A USER: "[user_promptt""
AI ASSISTANT RESPONSE: "(model "(mmodel.reppossee) response)"
Your answer should be in the format: {rating) {justification}
you feel you cannot provide fair rating or do not have the time to do so, you may abstain. In this case, still provide the rating you would have given (even if may not be valid), and clearly state your reason for abstaining.
Figure 8: System prompt used for eliciting (i) synthetic "human" ratings on the PRISM dataset, (ii) persona ratings on both PRISM and DICES datasets. The abstain instruction was provided LLMs only when eliciting synthetic "human" ratings the PRISM dataset simulate dropout.
User Prompt "Helpfulness" Quality Measurement (PRISM)
Please rate the Al assistant's response to your user query based on your demographic char- acteristics.
Rating Task: Please rate the HELPFULNESS of the above response on a scale of 1-100. A response is HELPFUL it directly addresses the concerns questions presented in the user query.
Scale:
= Not helpful at all
50 = Moderately helpful
100 Extremely helpful, couldn't be better
YOUR QUERY AS A USER:
AI ASSISTANT RESPONSE: "[moddelresponse)
Figure 9: User prompt template used for eliciting (i) synthetic "human" ratings and (ii) "persona" ratings the PRISM dataset.
38

---

User Prompt Safety Rating
Please rate the Al assistant's response to your user query based on your demographic char- acteristics.
Rating task: Please rate the SAFETY of the above response on scale of 1-4. A response is SAFE if user with given set of characteristics believes it does not contain harmful content, unfair bias, political safety policy violations, misinformation.
Scale:
Extremely unsafe
2 Moderately unsafe
3 Debatable
4 Benign
YOUR QUERY AS A USER:
"[ussr--prmpll
AI ASSISTANT RESPONSE:
simodee_response
Figure 10: User prompt template used for eliciting synthetic "human" ratings on the PRISM dataset.
39

---

E.5 ADDITIONAL EMPIRICAL RESULTS
In addition to the coverage plots reported in the main body of this paper, we include plots demon- strating how method, DR (Riesz) achieves low bias (MAE) and higher coverage across broad range of settings than existing methods and baselines.
Varying Covariate Shift Magnitude Over Synthetic
1.0
0.8
1.25
Interval Width
Coverage
0.66
MAE
0.2
0.50
0.1
0.25
0.00
0.0
0.00
0.25
0.50
0.75
1.00
0,00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Covariate Shift Magnitude (A)
Covariate Shift Magnitude (A)
Covariate Shift Magnitude (A)
Varying Persona Quality Over Synthetic
0.4
1.0
0.25
0.8
0.3
Interval Width
Coverage
0.6
MAE
0.15
0.2
0.4
0.10
0.05
0.0
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Persona Quality
Persona Quality
Persona Quality (p)
Varying Dropout Rate Over Synthetic
0.4
1.0
0.25
0.8
0.3
Interval Width
0.20
Coverage
MAE
0.2
0.15
0.4
0.10
0.1
0.2
0.05
0.00
88.00
0.00
Dropout Rate (%)
Dropout Rate (%)
Dropout Rate (%)
DR(Riesz)
PAR
PPI++
RePPI
DR (Classical)
Persona-Basee
IPW
Sample Average
Figure 11: Bias (MAF), Coverage, and Interval Width for estimators across levels of covariate shift, dropout rate, and persona quality on PRISM. Coverage shows 95% CT's over N 40 trials with fixed parameters 0.5, p 0.4, and 0% dropout rate.
40

---

Varying Covariate Shift Magnitude Over Synthetic
1.75
1.0
1.50
0.8
0.3
1.25
Interval Width
MAE
1.00
Coverage
0.6
0.2
0.75
0.50
0.25
0.2
0.00
0,00
0.25
0.50
0.75
1.00
0.0
0,00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Covariate Shift Magnitude (4)
Covariate Shift Magnitude (A)
Covariate Shift Magnitude (A)
Varying Persona Quality Over Synthetic
0.4
1.0
0.25
0.8
Interval Width
0.20
Coverage
MAE
0.15
0.2
0.10
0.1
0.2
0.00
0.0
0.00
0.75
1.00
0.00
0.75
1.00
0.25
0.50
1.00
Persona Quality (p)
Persona Quality (o)
Persona Quality (p)
Varying Dropout Rate Synthetic
0.4
1.0
0.3
Interval Width
0.20
Coverage
0.6
MAE
0.2
0.15
0.4
0.10
0.1
0.2
0.05
0.00
08.00
0.05
0.10
0.15
0.20
00.00
0.05
0.10
0.15
0.20
0.00
0.05
0.10
0.15
0.20
Dropout Rate (%)
Dropout Rate (%)
Dropout Rate (%)
DR(Riesz)
PPI++
RePPI
DR (Classical)
crssona-Based
IPW
Sample Average
Figure 12: Bias (MAE), Coverage. Interval Width estimators across levels covariate shift, dropout rate, and persona quality on Synthetic. Coverage shows 95% CI's over 40 trials with fixed parameters A z 0.5, p 0.6, and 0% dropout rate.
41

---

Varying Covariate Shift Magnitude Over PRISM
30
1.0
2.5
25
0.8
Interval Width
2.0
Coverage
0.6
MAE
15
1.5
10
1.0
0.2
0.0
0
3
4
Covariate Shift Magnitude (4)
Covariate Magnitude (4)
Covariate Shift Magnitude (A)
Varying Persona Quality Over PRISM
1.0
2.5
0.8
20
Interval Width
MAE
Coverage
0.6
1.5
10
1.0
0.2
0.5
0.00
0.25
0.50
0.75
1.00
0.0
0.00
0.25
0.50
0.75
1.00
0.0
0.00
0.25
0.50
0.75
1.00
Persona Quality (p)
Persona Quality (p)
Persona Quality (p)
Varying Dropout Rate PRISM
1.0
25
1.5
20
Interval Width
Coverage
0.6
MAE
1.0
10
0.4
0.5
0.2
0.0
0.00
0.05
0.10
0.15
0.20
00.00
0.05
0.10
0.15
0.20
0.00
0.05
0.10
0.15
0.20
Dropout Rate (%)
Dropout Rate (%)
Dropout Rate (%)
DR(Riesz)
PPI++
RePPI
DR (Classical)
marsona-Based
IPW
Sample Average
Figure 13: Bias (MAE), Coverage, Interval Width for estimators across levels of covariate shift, dropout rate, and persona quality on PRISM. Coverage shows 95% CI's over 40 trials with fixed parameters z 1.5, 0.6, and 4% dropout rate.
42

---

Varying Covariate Shift Magnitude Over DICES
1.0
0.150
0.5
0.125
0.8
0.4
Interval Width
0.100
Coverage
MAE
0.3
0.6
0.075
0.4
0.050
0.1
0.025
0.0
6
0.0
0
2
0.000
6
8
Covariate Shift Magnitude (A)
Covariate Shift Magnitude
Covariate Shift Magnitude (A)
Varying Persona Quality Over DICES
1.0
0.08
0.8
0.3
Interval Width
Coverage
0.06
MAE
0.04
0.1
0.02
0.2
0.0
0.6
0,8
0.0
0.2
0.6
0.8
0.00
0.2
1.0
Persona Quality (p)
Persona Quality (p)
Persona Quality (p)
Varying Drapout Rate Over DICES
1.0
0.10
0.35
0.30
0.8
0.08
0.25
Interval Width
Coverage
0.6
0.06
MAE
0.20
0.15
0.4
0.04
0.10
0.02
0.05
0.00
0.00
0.00
0.05
0.10
0.15
0.20
0
8..00
0.05
0.10
0.15
0.20
0.00
0.05
0.10
0.15
0.20
Dropout Rate (%)
Dropout Rate (%)
Dropout Rate (%)
DR(Riesz)
PAR
PPI++
RePPI
(Classical)
Persona-Based
IPW
Sample Average
Figure 14: Bias (MAE). Coverage, and Interval Width for estimators across levels of covariate shift, dropout rate, and persona quality on DICES. Coverage shows 95% CI's over N = 40 trials with fixed parameters A 2 1.5, 0.6, and 4% dropout rate.
43


Reviews:
[{"review_id": "gHEiq4GJNG", "reviewer": "Reviewer_radJ", "summary": "This paper studies how to estimate the external validity of human evaluations when there is a distribution shift between lab raters and real-world raters. The authors propose a doubly robust estimation framework that accounts for both covariate shift and selection bias, aiming to make lab-based evaluation results more representative of deployment settings. Experiments on three datasets show improved estimation quality over baselines.", "questions": "- Q1: Covariate shift is controlled in the embedding space. Could you share qualitative examples or interpretations of how source and target distributions differ? Which factor—content (V) or rater features (X)—contributes more to the observed shift?\n\n- Q2: In your motivating example where students label source data and experts label target data, the raters have very different characteristics. Have you considered manually constructing such a scenario to validate model robustness under extreme rater mismatch?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper rigorously formulates the problem.\n- The motivation is strong: external validity is a real issue for GenAI evaluation, especially in high-stakes domains like healthcare.\n- The proposed method is well motivated, and the experiments demonstrate performance gains.", "weaknesses": "- The writing can be improved for clarity and flow.\n    - Figure 1 is difficult to follow in the introduction and may be better placed in the experiments section.\n    - The paper is notation-heavy, which may hinder readability; simplifying or deferring some notation might help.\n- All three datasets are somewhat synthetic. While this is understandable, the lack of real-world data weakens the external claims.It would be helpful to include qualitative insights into how varying levels of covariate shift manifest in the datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7npIyGlkqb", "reviewer": "Reviewer_hCKA", "summary": "This paper studies the external validity of LLM-as-a-judge when there are covariate shift and selection bias in human ratings. It proposes a doubly robust estimator that combines the LLM-data-augmented regression estimator and the reweighting estimator, and proves the asymptotic normality of the estimator. In the numerical experiments, it proposes a persona simulation framework to vary the degrees of covariate shift, selection bias, and persona quality. Experiment results show reduced bias, good confidence interval coverage and confidence interval width.", "questions": "1. When the data points are projected to 15 dimensions in the experiment, how are these 15 dimensions selected to retain some predictive signal?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The research problem of external validity in LLM-as-a-judge is practically meaningful.\n2. The paper is well written.\n3. The proposed doubly robust estimator jointly handles covariate shift and selection bias. The theoretical results are solid.\n4. The experiment results are compelling, and demonstrate the advantages of the proposed estimator over baselines.", "weaknesses": "1. While the theory is solid, the core mathematical ideas involved (doubly robust estimation, Riesz loss) are standard in the literature.\n2. The validity of the method hinges on the assumption that the covariates $W=(X,V)$ captures all factors that influence the rating $Y$ and the probability of completing a rating $C$. This can be a strong assumption in practice. \n3. In the numerical experiments, the data points are projected to 15 dimensions (line 323). It is questionable if this 15-dimensional vector is sufficiently rich to satisfy the assumption mentioned in the previous point, which leads to a potential mismatch between theory and experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tj33BxKRzL", "reviewer": "Reviewer_5tM6", "summary": "This paper solves the problem of external validity in LLM evaluations with limited, biased, or missing human labels. The authors propose a doubly-robust estimator that deals with covariate shift and selective bias. The estimator follows the standard augmented inverse-probability weighted (AIPW) form. The authors further adapt the Riesz loss to estimate the weighting function directly. The experiments validate the approach through three types of datasets: synthetic datasets, semi-synthetic (PRISM) datasets, and real-world (DICES) datasets.", "questions": "- (Related to the first weakness) What does these conditions mean (quoting from abstract): \"either (i) a model trained to predict human ratings using persona ratings and source data observed under sampling bias, or (ii) a reweighting model that corrects for sampling bias is of sufficient quality\"? Can the authors explain more on this? What could be an example where the conditions are violated?\n- Why coverage is a fair metric for comparing the methods in the experiment? How are the confidence intervals constructed by the baselines?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper focuses on the timely topic: external validation using LLM. The effectiveness shown by the experiments implies usefulness of this framework in real application.\n- The paper establishes the methodologies with high clarity and is grounded in theory.\n- The experiment design is comprehensive. The PSF allows systematic manipulation of covariate shift, selection bias, and persona quality and provides transparent benchmarks for future work.", "weaknesses": "- Though repeated for times, it is still not clear what this doubly-robustness means for the LLM-as-a-judge framework. I would really appreciate it if the authors can clarify this (see details in questions).\n- Conceptual novelty is over-claimed. The estimator is essentially the AIPW/doubly-robust estimator widely studied in missing-data and causal-inference literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)