QuEst: Enhancing Estimates of Quantile-Based Distributional Measures Using Model Predictions
Authors: Zhun Deng
Thomas P Zollo
Benjamin Eyre
Amogh Inamdar
David Madras
Richard Zemel
Venue: ICML 2025 poster (2025)
Keywords: prediction-powered inference
L-statistics
auto-evaluation
distortion risk measures
synthetic data
Primary area: theory->everything_else
TL;DR: Rigorous method for combining real, observed data with machine learning model predictions to produce better estimates of important distributional risk measures.

Abstract: As machine learning models grow increasingly competent, their predictions can supplement scarce or expensive data in various important domains. In support of this paradigm, algorithms have emerged to combine a small amount of high-fidelity observed data with a much larger set of imputed model outputs to estimate some quantity of interest. Yet current hybrid-inference tools target only means or single quantiles, limiting their applicability for many critical domains and use cases. We present QuEst, a principled framework to merge observed and imputed data to deliver point estimates and rigorous confidence intervals for a wide family of quantile-based distributional measures. QuEst covers a range of measures, from tail risk (CVaR) to population segments such as quartiles, that are central to fields such as economics, sociology, education, medicine, and more. We extend QuEst to multidimensional metrics, and introduce an additional optimization technique to further reduce variance in this and other hybrid estimators. We demonstrate the utility of our framework through experiments in economic modeling, opinion polling, and language model auto-evaluation.

QuEst: Enhancing Estimates of Quantile-Based Distributional Measures Using Model Predictions
Zhun Deng" 1 Thomas Zollo 2
Benjamin Eyre
Amogh Inamdar
David Madras
Richard Zemel
Abstract
Observed
tf fif
Estimate
As machine learning models grow increasingly competent, their predictions can supplement scarce or expensive data in various important do- mains. In support of this paradigm, algorithms have emerged to combine a small amount of high- fidelity observed data with a much larger set of imputed model outputs to estimate some quan- tity of interest. Yet current sybrid-infereent target only means or single quantiles, limiting their applicability for many critical domains and use cases. We present QuEst, a principled frame- work to merge observed and imputed data to de- liver point estimates and rigorous confidence in- tervals for wide family of quantile-based dis- tributional measures. QuEst covers range of measures, from tail risk (CVaR) to population segments such as quartiles, that are central to fields such as economics, sociology, education, medicine, and more. We extend QuEst to multi- dimensional metrics, and introduce an additional optimization technique to further reduce variance in this and other hybrid estimators. We demon- strate the utility of our framework through exper- iments in economic modeling, opinion polling, and language model auto-evaluation.
Top
Machine Learning
Model
QuEst
Imputed Data
Estimate
Bottom
25%
tools
Figure Small gold-standard (observed) datasets may noisy, while model-predicted (imputed) data are often biased. QuEst combines both rigorously characterize various distributional measures, such the wealth of the top and bottom 25% of house- holds in developing nation.
since they often exhibit biases or misalignment with user ob- jectives. Instead, promising approach leverages both small, expensive observed datasets (i.e., true measurements) and large imputed datasets (i.e., model-predicted values), in or- der to improve the estimation of important quantities across various critical domains (Angelopoulos et al., 2023). This paradigm can be applied, for example, to improve the mea- surements generated by experiments in biology (Angelopou- los et al., 2023) or environmental science (Angelopoulos et al., 2024), or to enhance evaluation reliability the large language model (LLM) development cycle (Boyeau et al., 2024; Eyre & Madras, 2024). range of existing meth- ods (Angelopoulos al., 2023; 2024; Fisch et al., 2024; Hofer al., 2024) can produce valid confidence intervals for quantities such as mean, quantile, or linear regression coefficient in this hybrid manner, without any knowledge of the model that produced the predictions.
1. Introduction
As machine learning (ML.) models grow increasingly com- petent, their predictions are being used to simulate or other- wise represent diverse phenomena across economics (Hor- ton, 2023), politics (Argyle et al., 2023a), genetics (Jumper et al., 2021), and other fields, especially when such data is scarce or expensive gather. Despite their convenience, these predictions cannot be trusted perfect surrogates,
Despite their utility, existing frameworks ultimately focus on a narrow set of quantities, leaving other important as- pects of distribution unaddressed. In applications with societal or high-stakes implications- rrangnng from wealth disparity assessments to identifying the most harmful out- puts of LLMs- unddersttttnnnnngggggggg the tails or other segments of distribution can be crucial (Snell et al., 2022; Deng et al., 2024; Zollo al., 2023). For instance, policy-maken,
UNC at Chapel Hill CColumbia University 'Google Deepmind. Correspondence to: Zhun Deng <ffrstl.lass xxx.edu>.
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

QuEst: Enhancing Estimates of Quantile-Basee Based Distributional Mcasures Using Model Predictions
may be interested in the upper or lower 10% of an income distribution when studying wealth inequality (Espey al., 2015), while LLM safety experts must care about the rare, highly problematic outputs of an ML system (Ganguli et al., 2022). By applying to a narrow set of measures, existing methods cannot offer these key distributional insights.
2. Background and Related Work
A growing paradigm in data science and applied statistics seeks to combine a small sample of observed data (i.e., high- fidelity but expensive ground truth) with a larger sample of imputed data (i.e., model predictions) for improved estima- tion of key statistics. The recently introduced Prediction- Powered Inference (PPI) framework (Angelopoulos et al., 2023) addresses this challenge for a broad class of inference problems, provided the target parameter is an M-estimator. Concretely, M-estimators are statistical parameters express- ible as the minimizer of an empirical loss function, such as the sample mean of loss. PPI leverages small, gold- standard dataset (with reliable labels) in tandem with large, model- tenelliimputed dataset to construct unbiased estimators and corresponding confidence intervals. In particular, PPI de- biases the potential systematic errors in the imputed data via carefully designed correction terms, yielding inference procedures that enjoy theoretical guarantees without requir- ing knowledge of the underlying data distribution or the model that generates the imputations (Angelopoulos et al., 2024; Fisch et al., 2024). PPI has been successfully applied to settings like large language model evaluation (Boyeau et al., 2024; Eyre & Madras, 2024) and survey analysis (Angelopoulos et al., 2024), showcasing its flexibility in leveraging large-scale imputed data while maintaining valid statistical guarantees.
To address this gap, we introduce QuEst (Quantile- -uase Estimation), a framework designed to offer point estimates and valid confidence intervals on key distributional fea- tures by combining both observed and imputed data. Specifi- cally, QuEst estimates range of quantile-related metrics, in- cluding tail measures like Conditional Value at Risk (CVaR) (Rockafellar& & Uryasev, 2002) and population-leve seg- ments (e.g., deciles or quartiles). These measures are use- ful for understanding extreme values, variability, and other trends in a distribution, and are of particular salience in human-centric domains such as economics, sociology, edu- cation, and medicine. Beyond the unidimensional setting, QuEst also provides an approach for multidimensional mea- surements, enabling a more nuanced understanding of dis- tributional properties in real-world scenarios. Finally, we propose an extension of our method (also applicable to other similar methods) based optimizing weighting function applied to the imputed data, leading better estimates and tighter confidence intervals.
Here, we summarize our contributions:
While PPI applies to wide range of M-estimators (e.g.. means, regression coefficients, and single-quantile esti- mates), many important metrics in, e.g., economics, finance, social science, and risk management, cannot be expressed as such. In these fields, one often seeks to understand the tails or shape of distribution through quantile- -based distri- butional measures (QBDM). A canonical example is the Conditional Value at Risk (CVaR), used to quantify tail risk (Rockafellar & Uryasev, 2002) in financial engineering.
We introduce QuEst, a rigorous, eecory-grounded framework to leverage both observed and imputed data to estimate and provide rigorous confidence intervals quantile-based distributional measures.
We derive complementary method for estimating these measures for multi-dimensional quantities, for example when multiple loss functions are considered in model evaluation.
Definition 2.1 (Quantile-Based Distributional Measures). Given a CDF F, a quantile-based distributional measure for F is given by
Qv(F)= ( (bp)
We propose an advanced method for optimizing QuEst estimates, as well as those from existing methods for hybrid estimation.
(I)
where vo is a weighting function satisfying ab 2 0 and
I (dp)dp= = Here, F-)(p) = inf{æ : F(x) > p} is
the general inverse of CDF F, also known as the quantile function.
We perform experiments highlighting a wide set of use cases for our algorithms, showcasing applications in opinion polling, gene analysis, LLM auto-evaluation, and wealth modeling with satellite imagery. Through this work, we aim to enable more informed decision-making in domains with societal, ethical, or operational importance. By extending the applicability of prediction- -ooeiiee powered frameworks, QuEst paves the way for richer, more robust, and context-sensitive evaluations in machine learning and beyond.
By plugging in different weighting functions v. we can recover many classic measures; see Table (where VaR ab- breviates
These measures are instrumental for capturing tail behavior, inequality, or beesshold-bsse phenomena. In practice, they reveal insights that a mere mean or single quantile cannot fully characterize. For ex- ample, in order for an economist to study trends in wealth
2

---

QuEst: Enhancing Estimates of Chantilc-Basel Distributional Mcasures Using Model Predictions
described in Section 2, this might include CVaR, VaR, or other tail/segment--bssed measures.
Table 1. Several quantile-based distributional measures and their corresponding weight functions (see Definition 1). The Dirac delta function centered at & is denoted by óp.
The challenge in many scenarios is that collecting the ground-truth observations Y (which we call observed data) can be costly and/or difficult to obtain. However, we of- ten have access to a large amount of unlabeled data points (X"). which can be paired with model-imputed values Y = g(}") generated by some predictive model We assume that each unlabeled input X; drawn from the same distribution as the labeled inputs X,.
Measure
Weighting Function (p)
Expected Mean
1
3-VaR
(p)
p 3
8-CVaR
(p))
1-
(0,
otherwise
E [81, 82] otherwise
Interval VaR
vips B1, 32)
0,
We collect datasets
{(x,, Y,, KIYE and ((X), Xy)),
inequality, they might compare the income growth of the top 20% of earners to that of the bottom 20% (Pew Research Center, 2020). To understand variety in human development, genomics study may consider the 10% of the population that most strongly expresses some gene (Taylor et al., 2024). Further, VaR a central risk measure in finance and port- folio management, widely used to gauge possible losses in investment positions. However, current tools are unable to leverage large pools of model-imputed data to more effi- ciently estimate these measures.
where < N, and each X,, Xÿ is drawn i.idd from the same marginal distribution of inputs. We further define:
CDF of (X,, Y3) as F, and the corresponding empir- ical CDF (built on {M(X, YD) as Fn.
CDF of M(X;, Yj) as F and the corresponding empir- ical CDF as F..
CDF and empirical CDF of M(X,, Y1) as Fo and Fx-
3. QuEst for Quantile-Based Distributional Measures
Our assumption that X; and X," are drawn from the same distribution implies that F = F. Thus, the CDF and em- pirical CDF of M(X,, Yru) can be denoted as F and Fx for short and we use them exchangeably with jru and Fr.
We now introduce our QuEst framework, which produces hanced point estimates and confidence intervals for quantile- based distributional measures (QBDMs) using a combina- tion of observed and imputed data. We first describe the setup and notation, and then we present the main QuEst esti- mators and describe how they correct for model-imputation bias. Finally, we extend the idea multiple dimensions, covering scenarios where practitioners want quantify mul- tiple QBDMs (or QBDMs of multiple metrics) simultane- ously.
3.2. Methods
We focus on estimating QBDM of F:
Qu(F)= = f (p) F (p)
where (-) is nonnegative weighting function that inte- grates to (Table lists examples). A naive "classical" approach would be to simply compute
3.1. Setup
Consider general predictive setting adapted from Boyeau et al. (2024) in which each instance has an input X E X and an associated observation Y e V. We are given a user- specified metric interest, M(,.)) : X' x + R, that maps an input-observation pair (X, Y) to a real value [X,, Y). M is flexible: for instance, if we want to measure the per- formance of predictive model h, then we might define
/ (b)) F.))
Qs (F))
(p) dp,
where (p) is the empirical quantile function of the observed values (M, Y.) Unfortunately, with relatively small, this estimate can be noisy and unreliable.
M(X, Y)- e(h(X), Y) for some loss f. Alternatively, if we simply want to examine the distribution of Y itself (e.g., monthly carnings in an economics setting), we can choose M(X,Y) = Y (treating X as a "dummy" argument). Then, letting F' denote the true CDF of M(X, Y), our ultimate goal is to estimate various QBDMs derived from F. As
Our QuEst framework integrates model-imputed data from the large unlabeled set to reduce variance in the estimate, while carefully correcting for potential model bias. Specifi- cally, we define the following estimator for Qp (F):
2x) = xQx(F)) + (Qoo(F)
(F.)).
3

---

QuEst: Enhancing Estimates of Quantile-Based Distributional Mcasures Using Model Predictions
For any given A, Qu (A) is asymptotically unbiased es- timator of Q when n, N + oo. Intuitively, strong annotator model that produces near-perfect imputations will result in the first term (F(x)) almost exactly recovering (F). In the second term, we use the observed sample to measure and remove potential statistical bias: the imputed data F2 systematically deviates from Fn, this correction term will account for this and rectify the estimate.
The derivation builds on standard asymptotic expansions for L-statistics (Van der Vaart, 2000), then extends them to incorporate the imputed CDF F. The result also shows that in the limit, ( is unbiased, but different choices of À lead to different tradeoffs variance.
3.2.1. SELECTING THE OPTIMAL
To minimize variance, we can pick the value of that solves
Varying À allows to adapt reliance the imputed data, where À = 0 ignores the predictions and recovers the classical estimator on observed data. By selecting À based on the quality of the imputed predictions, we can ensure that our estimator is no worse than Qu (Fn) (see Section 3.2.1). We note that our method is a strict generalization of the method in (Angelopoulos al., 2023), since the mean and the quantile are special cases of QBDMs.
min (2,
This approach is referred to as power-tuning by Angelopou- los et al. (2024). In practice, of course, F and F are unknown; we replace them with their empirical versions Fn; Fu, Fÿ to obtain
i argminy (A, Fn, F,, FN)
To analyze our estimator, we must understand its asymp- totic normality and the corresponding variance. The main challenge that Q (F.) is not in the typical form of sum of ii.d. random variables. The method presented by Angelopoulos al. (2023) only considers estimators of this form, known M-Estimators. Our estimator involves order statistics of the observations, since F is quantile. However, we can harness the classic theory of L-statistics (Aaronson et al., 1996) to rewrite:
where CA, Fn, F%) is a consistent estimator of P F, F) of the form:
(1 + +r+ + (Fa)) 22m (F:: Fr))
Because (.) is quadratic in X, there is a closed-form solu- tion. In particular:
(p(Fn)) = E (p) dp)
M(3)
nos no Fn, Fw)
=
(1+ (1+ of
where May is the i-th order statistic of the sorted sam- ple (M,,,, YI,,..,,M((.. Y.)). Under some regularity conditions, we can show that
Under standard regularity conditions, an argument using Slutsky's 's rule shows that we still apply the central limit theorem (Theorem 3.1) with À = X
DD (())
Summarizing these ideas, we obtain the following corollary and a blueprint for constructing confidence intervals:
where +D means convergence in distribution, the func- tional (.) is defined as
Corollary 3.1. Under certain regularity conditions, if converges to constant, then
( /
Fn Fn, Fn,FN F%) ( 24 N(O,
and u/v minfu, vJ. Using similar ideas, analyze (X) and obtain:
Theorem 3.1. For any fixed X, under certain regularity conditions, if f//N + for some r 2 0, we have
In other words, we can plug the estimated Å, compute an estimated standard error
V (T((((() - -Q((F)) AD (1..
Pos Fn Fn,
SE
Here, we define the functional pv((s.;:) (A, F) A and no (F, F) is the covariance Cov Cor(Qu (F), 2z) (F)).
and then form finite-sample (1-x) confidence interval as
+ (F) - 22 F)
2x + 21-a/2 SE,
We also note that work is the first to introduce optimal À selection for quantile, also known as VaR.
where 21 2/-a/2 is the usual two-sided standard normal quan- tile.

---

QuEst: Enhancing Estimates of Quantile-Based Distributional Measures Using Model Predictions
Crucially, our final asymptotic variance can never exceed that of the classical estimator Q (F... From Corollary 3.1, we have
We provide specific closed form expressions for VyJ in Ap- pendix C.4. These expressions involve calculating new quantities such as Cov(Q (Fw), Qu,(F))
(Fo. F.)) (1+ o
In some applications. it may be beneficial to choose each X by separate univariate ceriance-minimizenn as in the single-QBDM setting. In others, we may prefer a single joint objective that balances all is coordinates. One example the sum of the asymptotic variances:
Fn. oZ((F))
approach,
so the second (non-negative) term is subtracted off from the classical variance oz (F)). This leads to more precise confidence intervals whenever the model imputations Y have nontrivial correlation with Y.
h
min Σ AL AR
Pe (A. Fu. Fm: FN),
We remark that the user may elect to clip À to J0, 1J to stabi- lize the final estimator in small, finite samples. The above analysis still holds since X will still converge to a constant. We also note that since À enjoys a closed-form solution, QuEst does not require any hyperparameter selection from the user.
but other utility or risk functions are possible, depending on the user's goals.
Thus, by permitting multiple weighting functions ......, 1/x (and/or multiple metrics can deliver a cohesive picture of multi-dimensional distributional statistics while retaining valid coverage. Our experiments in Section 4.2.2 illustrate these multi- dimensional analyses in the context of LLM evaluation.
QuEst
3.3. Calti-Dimennsonal QuEst
In many real-world scenarios, it is not enough to estimate a single QBDM in isolation. Rather, we might simultaneously want to know the (5%, 95%) 95%) pair, or we might want to compute distributional statistics for multiple metrics, such as different loss functions in model evaluation.
4. Experiments
We now evaluate the empirical performance of Qulist across two categories of tasks: (1) research settings where expen- sive experimental data is combined with predictions from an ML model, and (2) LLM auto-evaluation settings where large, more expensive LLM supplies small number of high-quality labels (treated as "observedd"), while cheaper model provides predictions (treated as "imputed"") for the majority of data. In each experiment, we have both observed and imputed labels for all instances, allowing us to gauge the estimation error and interval coverage of different methods with respect the true quantity.
To address this challenge, we extend our method to a multi- variate version for simultaneously evaluating (1) multiple QBDMs of the same metric or (2) QBDMs of multiple met- rics. Here, we will mainly discuss the case of evaluating multiple QBDMs in tandem, but a similar argument could be easily extended to estimating QBDMs for several metrics.
In particular, we are interested in estimating k QBDMs simultaneously, i.e.,
QUVE: () (z). (F).
In each experiment trial, we randomly sample some amount of observed data, and use a fixed amount of imputed data. We compare QuFist's point estimates to those derived using either of these data sources alone. For the confidence in- tervals, establishing some baseline still requires our highly non-trivial CLT derivation. In order to enable some compar- ison for contextualizing QuEst's performance, we compare the confidence intervals when À 0 to those when is se- lected using our algorithm. Once again, we note that QuEst involves no additional hyperparameters to be tuned or set by the user. Some details are deferred to Appendix A.
If we were to simply estimate each quantity separately, then correct via naive Bonferroni approach, our confidence intervals/regions will become overly conservative. Thus, we need to derive a multi-dimensional central limit theorem.
Our estimator will be
Qu) Ox)))))))
for any wt's and X.s. We can then offer following theorem: Theorem 3.2. Suppose S satisfy that X -> X for con- stant X3 's, under certain regularity conditions, if n/N +r for some 2 O, we have
4.1. Improving Measurement in Research
In scientific and industrial research, is often useful to char- acterize not only average measurements, but also those that refer to some segment(s) of the full distribution. Economists, for example, may want to compare the top 20% vs. bottom 20% of incomes (Pew Research Center, 2020), while ge-
1-1 (T(Q(VI.. )1x) Q(VI.. AD 10,
where V is the le k covariance matrix, where Vi- Cor(Qu, Qu, (2z))-
5

---

QuEst: Enhancing Estimates of Quantile-Based Distributional Mcasures Using Model Predictions
PovertyMap, & E [0.33, 0.66]
GeneExpression, & E [0.8, 1.0]
OpinionQA, & E [0.25, 0.75]
Estimation Error
0.020
Observed Imputed QuEst
0.15
0.015
0.4
0.10
0.010
0.2
0.005
0.05
Conf. Int. Width
0.10
À 0
2.5
0.8
QuEst
0.08
2.0
0.6
0.06
1.5
0.04
1.0
200
400
600
800
1000
200
400
600
800
1000
200
400
600
800
1000
Num. Observed
Num. Observed
Num. Observed
Figure 2. Experimental results for estimating Interval- VaR and CVaR using three datasets (PovertyMap, GeneExpression, OpinionQA). Results are averaged over 2000 random data splits, and error bars are included (although too small to observe in most cases). Top: Estimation error vs. number of observed labels. Bottom: Confidence interval width vs. number observed labels.
nomics studies may focus on the 10% of the population that most strongly expresses certain gene (Taylor et al., 2024). Such subpopulation measures further exacerbate the scarcity of expensive gold-standard experimental data, as splitting dataset into segments decreases sample size. QuEst ad- dresses this challenge by mixing scarce gold-standar mea- surements with abundant but potentially biased model pre- dictions to provide rigorous estimates and valid confidence intervals. Here, we showcase relevant example applications in economics, genomics, and social science.
exhibit systematic biases and inconsistencies (Dominguez- Olmedo al., 2024). Using the OpinionQA dataset, we show how QuEst can produce more reliable estimates of public opinion-groundder in limited, high-quality human labels by rigorously leveraging abundant synthetic LLM- generated data from -Jaama-.1-700--nnstit
Opinions are measured on a 0-1 scale, and we target the middle 50% of the population for estimation, once again a case of the Interval-VaR.
We run 2000 trials each with varying numbers of randomly sampled observed data (100, 200, 500, 1000) combined with 2000 randomly sampled imputed data, and calculate point estimates and 95% confidence intervals for the target measure. To evaluate QuEst and baselines, we plot 3 metrics averaged across all trials: absolute error of point estimates, width of (valid) confidence intervals, and confidence inter- val coverage (i.e., the percentage of trials where intervals contain the true value).
Experiment Details We perform our initial experiment using data from 3 publicly available datasets.
PovertyMap is dataset containing satellite imagery and socioeconomic data, used for estimating poverty metrics and wealth distribution across regions, typically in developing nations (Yeh et al., 2020; Koh et al., 2021). Predictions are obtained from a pre-trained deep regression model, where for each satellite image the model predicts scalar "wealth index". Our goal is to estimate and provide confidence intervals on the average wealth index of the middle third of houscholds in the distribution, a measure of Interval- VaR..
Results and Discussion Figure 2 shows estimation er- ror and interval width results for estimating Interval-Va VaR for PovertyMap and OpinionQA, and CVaR for GeneEx- pression. QuEst consistently produces better estimates and tighter confidence intervals than baseline methods across all three datasets. The benefits over using only observed data are most pronounced when examples are scarce: on Gene- Expression, for instance, we roughly 50% reduction in both estimation ervor and interval width with only 100 gold- standard examples. Similar gains appear in PovertyMap and OpinionQA when fewer examples are available, and using only observed data becomes competitive only once about 1, 1,000 examples are collected. While the imputed es-
In GeneExpression, the goal is to predict the level of gene expression caused by some regulatory DNA (Vaishnav et al., 2022; Angelopoulos et al., 2023). Predictions are taken from transformer sequence model that outputs gene expression level from 0 to 20. For our target measure, we consider the CVaR, or tail behavior of the 20% of genes with the highest expression level.
LLMs are increasingly used to simulate and study human behavior in social science (Argyle et al., 2023b), yet often
6

---

QuEst: Enhancing Estimates of Quantile-Base Based Distributional Mcasures Using Model Predictions
timate performs reasonably in PovertyMap, it is markedly worse for GeneExxression and OpinionQA. Taken together, these results illustrate the dynamics for which our method is designed: improving estimation when observed data are limited while avoiding blind reliance on potentially biased model predictions.
set of labels from an expensive model (which in this case becomes the observed data) with abundant labels from cheaper model (imputed data), thus retaining high-fidelity anchor for the evaluation while significantly reducing over- all cost. In the following, we illustrate how QuEst can effectively manage these trade-off's and yield dependable distributional assessments of LLM behavior.
In Appendix Figure 6, we further evaluate the empirical coverage of our confidence intervals, finding them be valid across all datasets. Overall, the results in this section highlight QuEsst's effectiveness in combining limited human labels and abundant model predictions to achieve more ac- curate and reliable quantile- -ased estimates in important research work.
4.2.1. RED-TEAMING CHATROTS FOR TOXICITY
Public-facing LLM applications like chatbots bring partic- ular concerns above those used in internal private appli- cations. Besides factually inaccurate or otherwise incorrect generations, they must be evaluated for their potential to produce toxic, abusive, violent, or otherwise offensive or dangerous material. A collection of methods for LLM red- teaming has emerged, wherein a collection of adversarial inputs are created (either by human or L.LM()) in order to probe the vulnerability of a model to producing such undesirable content. Research has shown that within such a context, the developer must be concerned not only with reducing the average rates of, e.g., toxic content, but also must be concerned with the tail of the distribution of worst responses (Ganguli et al., 2022).
QuEst for Individual Quantiles While the previous ex- periment emphasized quantile-based distributional measures that cannot be addressed by standard PPI methods, we can also explore how our proposed QuEst framework behaves when applied to individual quantiles. Focusing on the sim- pler task of estimating a single quantile allows us to make direct comparison between QuEst and the original PPI framework (Angelopoulos et al., 2024), which offered method for quantile inference under a fixed parameter X. By contrast, QuEst enables an adaptive selection of X, thereby potentially reducing variance and leading to more efficient estimates.
Experiment Details We perform red-teaming on a set of 8 candidate LLMs, where one will be chosen for deploy- ment as chatbot (see Appendix A for list). Prompts are obtained from the red-teaming split of the Helpful/Harmless dataset (Ganguli et al., 2022), giving to elicit potentially toxic behavior. We score the outputs from each model for toxicity on a scale from 0 (least toxic) to (most toxic) with two models: L.ama-3.1-70B (ob- served data), and Llama-3.1-8P (imputed data). The goal is to estimate the toxicity in the worst 25% of responses. We run 1000 trials with all models, varying numbers of observed inputs, and 2000 imputed inputs.
To study the effects of this increased flexibility, we conduct an experiment examining performance on the 8 0.75 quantile on two datasets, GeneExpression and PovertyMap. We compare estimates and corresponding confidence inter- vals produced by QuEst against those obtained from PPI. The results, depicted in Figure 7, show that QuEst's ability to adjust À yields lower estimation error overall. The confi- dence intervals from both methods maintain a similar level of tightness, and thus in this case the primary improvement from selecting À stems from QuEst's enhanced accuracy rather than smaller confidence intervals.
set of inputs meant
Results and Discussion See Figure 3 for results. On the left, we examine the performance for one particular model Diistral-7--Instruet--oo
4.2. QuEst for LLM Auto-Evaluation
and find that QuEst once again offers consistently lower estimation error than using only small set of observed data (imputed estimation error is very large and thus not included in the plot for clarity). On the right, we evaluate the effects of performing model se- lection based on QuEst, as opposed to baseline methods. Our evaluation metric rank correlation, which is the av- erage correlation across trials between the model rankings produced by each evaluation method and model rankings based on the true measure. We find that QuEst produces rankings that are more correlated with the true rankings, and that these improve with more observed data.
A growing trend in large language model (LLM) devel- opment involves using an LLM to evaluate the outputs of another (or the same) LLM, a process commonly called "auto-evaluation" (Zheng et al., 2023; Boyeau et al., 2024). In practice, developers must choose between an expensive, higher-quality model for labeling- which yields more re- liable judgments at a higher cost-or cheaper, weaker model that saves money but can offer worse judgments (or labels). Although many developers default to using the expensive model, they then face a trade-off between con- trolling costs (by limiting labeled examples) and tolerating noisy estimates (by sampling fewer high-quality labels). With QuEst, developers can rigorously combine a small
Finally, we aim to understand when QuEst gives low estima- tion error, and when it most improves performance over the
7

---

QuEst: Enhancing Estimates of Quantile- -aseed Distributional Mcasures Using Model Predictions
0.90
Observed QuEst
Table 2. Results of estimating the [0,0.2] interval VaR three separate scores averaged over 50 trials.
Estimation Error
Rank Correlation
0.07
0.85
Estimation Type
Classical Volume
QuEst Volume
% Change
0.06
Univariate
1.57 e-2
1.04e-2
-34%
Multivariate
5,55e-3
5.17e-3
-7%
0.05
0.80
Estimation Type
Classical MSE
QuEst MSE
Change
Imputed
0.04
Multivariate
1.17e-2
1.08c-2
-7%
100
150
200
250
100
150
200
250
Num. Observed
Num. Observed
Summarization Evaluation Confidence Regions
Figure 3. LLM auto-evaluation on a red-teaming toxicity task.
Consistency Interval VaR. g 0.2]
Multivariate QuEst Mu Multivariate À True Value
Left: Estimation error for representative candidate model as function of observed labels (the weak model's predictions have much higher error, omitted for clarity). Right: Rank corrolation with true toxicity ordering across 8 candidate models,
0.06
0.80
averaged over trials.
+
QuEst Estimation Error %
Labelled/QuES: Error Ratio
1.2
120
100
0.61
80
1.1
0.55
60
0.78
0.80
0.82
0.84
0.86
0.88
Coherence Interval VaR, [ 0.2]
40
20
1.0
Figure 5. Visualization of two-dimensional 90% confidence gion defined using different estimation methods. When simulta- neously estimating different QBDMs, QuEst yields smaller confidence region than using the observed data alone.
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
Correlation
Correlation
Figure 4. I.IM auto-evaluation on red-teaming toxicity task. Each point corresponds one model; the æ-axis the corre- lation between the "big" and "small" labelers for that model, while the y-axis captures QuEst's absolute error (left) or the error ratio relative to Observed-only (right).
sistent with the summarized article (Zhang al., 2024), and this can erode user's trust in the platform. In this circum- stance, the developer must ensure that the worst summaries achieve a reasonable score on average across three different criteria. Accounting for stochasticity, the developer will also want to ensure the LLM exceeds this threshold with high probability, requiring a confidence region.
baselines. In the plots in Figure 4, each point corresponds to one of the candidate LLMs. For both plots, the æ-axis the correlation between observed and imputed labels for that particular candidate LLM (i.e., how well the small la- beller's scores match the large labeller). In the left plot, the y-axis is the average eиror of the QuEst estimate percentage of the true error value. On the right, the y-axis shows the average етror ratio between the QuEst estimate and the observed estimate. We see in general that QuEst esti- mates improve, including relative to the bascline, as labeller correlation increases.
Experiment Details We derive news articles from the XSum dataset (Narayan al., 2018). Summaries are gener- ated using Qwen2-7B-Inntruet., We score the outputs from each model for coherence, relevance, and consistency scale from 0 (worst) to 1 (best) with two models: Llama- 3.1-70B (observed data). and Llama-3. 3-BB (imputed data).
We estimate the average coherence, relevance, and consis- tency amongst the bottom 20% of reviews for each criterion. In addition point estimate, we construct a 90% confi- dence region defined over the three criteria. We perform this estimation with both classical estimation and QuEst, and construct the region using the cube defined by the three uni- variate confidence intervals (univariate) or by the covariance matrix (Multivariatee. Volume and MSE calculations are av- eraged over 50 trials. We examine low label setting where only 100 observed examples and 10,000 imputed examples
4.2.2. EVALUATING NEWS SUMMARIZATION ACROSS MULTIPLE METRICS
News aggregation services may provide article summaries produced by a local LLM as a feature for their users to quickly scan for relevant content. It paramount that these summaries meet some minimum level of quality; LLMs may produce summaries that are confusing, irrelevant, or incon-
8

---

QuEst: Enhancing Estimates of Quantile-Based Distributional Mcasures Using Model Predictions
available.
Following the arguments in Section for any fixed &, if n/N r for 0, then
Results and Discussion QuEst yields not only superior point estimates in comparison to classical estimation in terms MSE, but it also produces 90% confidence re- gions with lower volume (Table 2). The smallest three- dimensional confidence region is achieved by multivariate QuEst, underpinning QuEst's improved utility when esti- mating multiple quantities of interest at once. Figure 5 visualizes this effect: we can observe multivariate QuEst yielding a confidence region with lower area than multivari- ate classical estimation.
(Q( Vb) - Q+(F)) 2s N(O.P)(.,F.F)).
where (5. F.F) is suitable functional (see the Ap- pendix C for details). We show that a consistent empirical version of variance P5, Fn, Fn, Fx) satisfying
PLI Fn, Fn, (N) 7n (E.
is always a convex function of ç. Hence, we can mini- mize p (E. Fn. Fr, F%) with respect to achieve better variance reduction. Since n is typically small (limited gold- standard observed data), we aim to avoid separate data split. The following theorem shows that, with a mild regu- larization term, the resulting solution still satisfies a central limit theorem without any additional sample splitting.
5. Extension: Generalizing
Our QuEst framework, as described in Section 3, already provides robust quantile-based estimates and confidence intervals that were not attainable with existing tools for integrating observed and imputed data. However, further variance reduction, and therefore a reduction in error and confidence interval width, is possible by applying a different weighting function the imputed data (1) than the QBDM being measured (ę)). By generalizing the weighting function J to a parameterized function, we can flexibly adapt the imputed data term for the estimator to our finite sample to maximize variance reduction.
Theorem 5.1. Under certain regularity conditions, we have
P P5 (S, Fn, Fn, Fx) VTl - 2, N(00 1),
where
= argming PLIE, Fn, Fx)
+
and a any fixed positive constant.
We demonstrate that applying the adaptive weighting func- tion v this way still yields an asymptotically unbiased esti- mator. This extension not only strengthens QuEst's perfor- mance in limited-data settings, but also offers theoretical foundation for extending hybrid inference techniques to more complex, high-dimensional problems.
We remark that > can be made arbitrarily small and is included only ensure unique solution é. facilitating the central limit theorem result.
5.2. Experiment
For proof-of-concept, we implement this extended version of QuEst (referred to as QuEst-Opt) and evaluate its perfor- mance in estimating QBDMs in Appendix B. Our empirical results indicate that QuEst-Opt performs favorably relative to the simpler QuEst variant (with power-tuned A) when the observed sample size is small.
5.1. Method
We modify our earlier estimator (A) to the more general form Q ) defined by
(P%)
+
(Fn)
6. Conclusion
Adaptive function Ø
Target QBDM weighting &
We propose QuEst, a framework for producing enhanced estimates and valid confidence intervals for guantile-based distributional measures by leveraging both observed and imputed data. QuEst shows consistent improvements over baselines in our experiments, producing better point esti- mates and tighter confidence intervals across applications from genomics to LLM evaluation. Future work might con- sider more rigorous characterization of the conditions der which our QuEst framework and other related methods are likely to aid in estimation, including how large the pool of unlabeled data for imputation must be, or what level of correlation between observed and imputed data is necessary in order to see performance gains.
(2)
Our adaptive weighting function e applies only to the im- puted data, while the weighting function t associated with our target QBDM is still applied to the observed data. Note that this is strict generalization of our previous estima- tor: if choose v Xuh then Q v reduces to our original (A). Crucially, the new estimator also remains asymptotically unbiased.
For simplicity, we let (() = cT) where x(-) a dimeniiammenional vector of basis functions and & is a tuning parameter vector that we will optimize.

---

QuEst: Enhancing Estimates of Quantile-Based Distributional Mcasures Using Model Predictions
Acknowledgement
Boyeau, P., Angelopoulos, A. N., Yosef, N., Malik, J., and Jordan, M. I. Autoeval done right: Using synthetie data for model evaluation, 2024. URL https://aatt /arxiv. org/ abs / 24003.403.0 07008.
This project was supported by an award by the Columbia Center of AI and Responsible Financial Innovation. We for its generous
also thank ONR Grant N00014-23-1 support. This work is supported by the funds provided by the National Science Foundation and by DoD OUSD (R&E) under Cooperative Agreement PHY-2229929 (The NSF AI Institute for Artificial and Natural Intelligence).
Deng, Z., Zollo, P., Snell, C., Pitassi, T., and Zemel, Distribution- free statistical dispersion control for soci- etal applications, 2024. URL bttps::/aaxiv..tt abs/2309.13785
All experiments using Llama or other open-source models were run Columbia University.
Dominguez-Olmedo, R., Hardt, M., and Mendler-Dümenn C. Questioning the survey responses of large language mod- els, 2024. URL https: /ttp:///aavvv//eeeeeeeeeeeeeevve org/ abs/ 6 07951.
Impact Statement
QuEst has the potential to improve statistical reliability in applications in diverse and important domains. Though our framework can enable more precise assessments of critical distributional features, its performance depends on the qual- ity of the underlying model predictions, raising concerns about bias, fairness, and robustness in high-stakes settings. Future research should address the responsible deployment of QuEst, ensuring that imputation methods are transparent, well-calibrated, and aligned with domain-specific ethical considerations. There are not any negative impacts which we feel must be specifically addressed here.
Espey, J., Swanson, E., Badiee, S., Christensen, Z., Fischer, A., Levy, M., Yetman, G., de Sherbinin, A., Chen, R., Qiu, Y., Greenwell, G., Klein, T., Jütting. J., Jerven, M., Cameron, G., Aguilar Rivera, A. M., Arias, C., Mills, S. L.,, and Motivans, A. Data for development: A needs assessment for sdg monitoring and statistical capacity development, April 2015,
Eyre, B. and Madras, D. Auto-evaluation with few labels through post-hoc regression, 2024. URL https:// arxiv.org/asae /2411 12665.
Eyre, B., Creager, E., Madras, D., Papyan, V., and Zemel, R. Out of the ordinary: Spectrally adapting regression for covariate shift, 2023. URL https: //ap:://ariiiee www.xivvorrg abs/2312 1463.
References
Aaronson, J., Burton, R., Dehling, H., Gilat, D., Hill, T., and Weiss, B. Strong laws for l-and u-statistics. Trans- actions of the American Mathematical Society, 348(7): 2845-2866, 1996.
Fisch, A., Maynez, J.. Hofer, R. A., Dhingra, B., Globerson, A., and Cohen, W. W. Stratified prediction-poweret infer- ence for hybrid language model evaluation, 2024. URL wbwpr///a//aa/eeeeeeeeeeeeeeeeee https: / // araia.iiivv.... org/ abs /406 04291.
Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., and Zmic, T. Prediction-powered inference, 2023. URL htttps://rxxii arxiv. tttp//t..........aaa///////ooo.t wtt///gg//////e////.///eeee/e//e abs/2301111 09633.
Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Ka- davath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., Jones, A., Bowman, S., Chen, A., Conerly, T., Das- Sarma, N., Drain, D., Elhage, N., El-Showk, S., Fort, S., Hatfield-Dodds, Z., Henighan, T., Hernandez, D., Hume, T., Jacobson, J.. Johnston, S., Kravec, S., Olsson, C., Ringer, S., Toan-Jo,,,on, E., Amodei, D., Brown, T., Joseph. N., McCandlish, S., Olah, C., Kaplan, J., and Clark, J. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned, 2022. URL https::t//// arxiv. org/ abs / 220/ 07858.
Angelopoulos, A. N., Duchi, J. C., and Zmic, T. Ppi++: Efficient prediction inference. 2024. URL https: httss///rxvv.ooa abs///b///// bbbs/23111 211..1131.........
Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., and Wingate, D. Out of one, many: Using language models to simulate human samples. Political Analysis, (3):337-351, February 2023a. ISSN 1476-4989. doi: 10. 1017/pan.2023.2. URL http://dx.doi..ore//// 1017/pan.2023.
Hofer, R. A., Maynez, J., Dhingra, B., Fisch, A., Glober- son, A., and Cohen, W. W. Bayesian prediction-powere inference, 2024. URL https: http/vivvvvv.. .rg. abs/ 2405. 06034.
Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., and Wingate, D. Out of one, many: Using language models to simulate human samples. Political Analysis, 31 (3):337335,, February 2023b. ISSN 1476-4989. doi: 10. 1017/pan.2023.2. URL http:////// dx. doi doi.ooo/oo.g..... 1017/pan. 2023.
Horton, J. Large language models as simulated economic agents: What can we learn from homo silicus?, 2023. URL hal.https//arre htt.ps::/arrvv..rrg/a///////
.35223333311.0..3.33 07543.
10

---

QuEst: Enhancing Estimates of Quantile-Based Distributional Mcasures Using Model Predictions
Jumper, J. M., Evans, R., Pritzel, A., Green, T., Figurnov,
Vaishnav, E. D., de Boer, C. G., Molinet, J., Yassour, M., Fan, L., Adiconis, X., Thompson, D. A., Levin, J. Z., Cubillos, F. A., and Regev, A. The evolution, evolv- ability and engineering of gene regulatory dna. Na- ture, 603(7901))453 463, March 2022. ISSN 0028- 0836. doi: 10. 10.10388/41136--22-045500 URL https: / //europepmc: http:///wworrrrrrreee/////e/ee//
M., Ronneherger, O., Tunyasuvunakool, K., Bates, R., Zidek, A., Potapenko, A., Bridgland, A., Meyer, C., Kohl,
S. A. A., Ballard, A., Cowie, A., Nomera-Paredee. B., Nikolov, S., Jain, R., Adler, J., Back, T., Petersen, S., Reiman, D., Clancy, E., Ziclinski, M., Steinegger, M., Pacholska, M., Berghammer, T., Bodenstein, S., Sil- ver, D., Vinyals, O., Senior, A. W. Kavukcuoglu, K., Kohli, P., and Hassabis, D. Highly accurate protein struc- ture prediction with alphafold. Nature, 596:583 589, 2021. URL https: http://pp...aaaa.. semant cscholar.
Van der Vaart, A. W. Asymptotic statistics, volume 3. Cam- bridge university press, 2000.
Yeh, C., Perez, A., Driscoll, A., Azzari, G., Tang, Z., Lobell, D., Ermon, S., and Burke, M. Using publicly available satellite imagery and deep learning to understand eco- nomic well-being in africa. Nature Communications, http:/ppp..saaaacccsc..ooooooooa org/CorpusID:2133 :11
Koh, P. W., Sagawa, S., Marklund, H., Xie, M., Zhang, M., Balsubramani, A., Hu, W. Yasunaga, M., Phillips, R. L., Gao, I., Lee, T., David, E.. Stavness, I., Guo, W., Earsshw, B. A., Haque, I. S., Beery, S., Leskovec, J., Kundaje, A., Pierson, E., Levine, S., Finn, C., and Liang, P. Wilds: A benchmark of wnlthe-wild distribu- tion shifts, 2021. URL https::///// /ttp.vaaxvvv...//...//../...at.. org/ abs/ 2012.07421.
11, 2020. URL
Zhang. T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., and Hashimoto, T. B. Benchmarking large language models for news summarization. Transactions of the As- sociation for Computational Linguistics, 12:39-57, 57, 2024.
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., and Zhu, C. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303. /6634, 2023.
Zheng, L.,, Chiang, W.-L.,, Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z. Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Ad- vances Neural Information Processing Systems, 36: (6595-46623, 2023.
Narayan, S., Cohen, S. B., and Lapata, M. Don't give me the details, just the summary! topie-aware convolutional neural networks for extreme summarization. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), Pro- ceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, Brussels, Belgium, Octoher-Novemther 2018. Association for Com- putational Linguistics. doi: 1/111/663/11111--1222 URL https: /ttt:::/acllanthoooyy..oooo 1ttp:thhhhhoooooooo1-00000000000
Zollo, T. P., Morrill, T., Deng, Z., Snell, J. C., Pitassi, T., and Zemel, R. Prompt risk control: A rigorous framework for responsible deployment of large language models. arXiv preprint arXiv:2311. 2023.
Pew
Research Center. and wealth inequality, January 2020.
Trends in income
URL org/social-trends: /ttp://eedd///////////3333// frends-in-incone aedd-walltae www.wwetttthhhaaaaalaaa-a----a-a
http://wwwwwwweeearhhtte
Accessed: 2024-12-11.
Rockafellar, R. T. and Uryasev, S. Conditional value-at-risk for general loss distributions. pp. 29, 2002.
Snell, C., Zollo, T. P., Deng, Z., Pitassi, T., and Zemel, R. Quantile risk control: flexible framework for bound- ing the probability of high-loss predictions, 2022. URL https: aariiiiivv. org/ abs/ 2212 13629.
Taylor, D., Chhetri, S., Tassia, M., Biddanda, A., Yan, S., Wojcik, G., Little, A., and McCoy, R. Sources of gene expression variation a globally diverse hu- man cohort. Nature, 632:122-1333 07 2024. doi: N0.1038//41188 1038/44158--024--770---

---

QuEst: Enhancing Estimates of Quantile-Based Distributional Mcasures Using Model Predictions
A. Additional Experiment Details
Below are additional details for our experiments. Our code will be made public upon release of this paper.
A.1. Mav
PovertyMap is dataset containing satellite imagery and socioeconomic data, used for estimating poverty metrics and wealth distribution across regions (Yeh et al., 2020; Koh et al., 2021). Further, there is shift from training to test distribution, where test data comes from separate set of countries. Each satellite image in the dataset is labelled with some scalar wealth index; we use 5000 random datapoints from the full test set for our experiments. Our predictions are obtained from pre-trained deep regression model, with the domain-robustnes: technique from Eyre al. (2023) applied after training. We scale all observed and imputed data to be in the range [0,1], based on the maximum and minimum values observed.
A.2. GeneExpression
In GeneExpression, the goal is to prediet the level of gene expression caused by some regulatory DNA (Vaishnav et al., 2022; Angelopoulos al., 2023). Predictions are taken from transformer sequence model that outputs gene expression level from 0 to 20. We derive this data from the code repository released with Angelopoulos et al. (2023). 2, and use 5000 random samples for our experiments.
A.3. OpinionQA
LLMs are increasingly used to simulate and study human behavior in social science (Argyle et al., 2023b), yet often exhibit systematic biases and inconsistencies (Doninguez-Olmedo et al., 2024). Using the OpinionQA dataset, we show how QuEst can produce more reliable estimates of public opinion grounded in limited, high-quality human labels-by by rigorously leveraging abundant synthetic L.LM -.M--genratted data from Llama-3.1 1...--..--0--
the OpinionQA dataset, each person has answered different subset of Pew polling questions. Opinions are measured on 0-1 scale, and we target the middle 50% of the population for estimation.
For the target opinion question on which we wish to estimate the quantile-based measure, we use the following: "Do you think same-sex marriages being legal in the U.S. is generally good or bad for our society?"
The answer options are l'Very good for society', 'Somewhat good for society', 'Neither good nor bad for society', 'Somewhat bad for society', 'Very bad for society', 'Refused"), J, which we assign to float values from 0 to I, evenly spaced.
Generating Response Predictions Responses are generated with persons in the dataset that have real answers to this question, which is 10151 total. In order to make prediction for each person, we include 20 in-context examples using other question/answer pairs from the same person. The in-context examples are chosen based on the embedding similarity of the target question to other questions that person has answered. We also include any demographic information that is available for the given person, although some persons include more of this than others.
We make predictions for all
A.4. Toxicity
For our adversarial prompts, we use 20000 prompts randomly sampled from the red-teaming split of the Anthropic RLHF dataset (Ganguli et al., 2022).3 Our 8 candidate LLMs used to produce responses are taken from the Huggingface model repository:
Phi.3 mini 4k instruct
Mistral_7B_Instt Jns
Llama.2 2.7b_char hf
Pnttps: ://gtthb.// http:///aaggeoooooos///p.....pp
hhttps://huggiggg: co/moelss
12

---

QuEst: Enhancing Estimates of Quantile-Basch Based Distributional Mcasures Using Model Predictions
Meta_ Instrruct
Llama.3.1.3 Instruct
gemma 2 9b it
Qwen2_7P 7B Instruct
Responses are generated with a temperature 0.75, using the system prompt: "You are helpful assistant. Answer the question fully as possible."
Scoring Responses for Toxicity Auto-evaluation relies the of scoring models evaluate model of interest. To evaluate QuEst as method to balance the cost and efficacy of auto-evaluation, we score all model responses adversarial prompts with a powerful LLM that is expensive to operate (Llama_3.1 1.._a0............. Instruustr as well as a less powerful and relatively inexpensive LLM (Llama_3.) 1_8.BB...BB... 1_8B___Bmnei Instruct).. The scoring models are prompted with template that contains prompt-response pair from our generations, as well as instructions to evaluate the toxicity of a response with single token out of two suggestions (e.g., (safe/unsafe), (yes/no)). We then compute the probabilities the positive and negative responses from the model's 's output logits for the first output token and compute the toxicity
unsafe token prob
t Lunsafe.token.pee + Safe_token_ppe
The template for toxicity scoring prompt is presented below. The unsafe categories input identical to those used in the prompt template of the Alama-Guat family of LLMs, which are specifically tuned for toxicity. Note that do not use these models as scoring models as their logits tend to saturate, resulting in highly polar (poorly calibrated) scores.
You will be given one conversation between User and an Agent that contains potentially unsafe content.
Your task is to check if there is unsafe content in the Agent messages in the conversation according to our safety policy with the below categories.
<BEGIN UNSAFE CONTENT CATRGORIES>
(atafe__attgoriess)
<END UNSAFE CONTENT CATEGORIES>
Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.
<BEGIN CONVERSATION>
[[conversation)] <END CONVERSATION>
Provide your safety assesament for the Agent in the above conversation in the form below. Your response should be a SINGLE WORD, either if the response is safe or ([unsafe_indicatoro) if it is unsafe.
}
SAFETY ASSESSMENT:
A.5. News Summarization
We derive news articles from the XSum dataset (Narayan et al., 2018).5 We use 20000 randomly sampled articles. Summaries are generated using Qwet2..7B.Intrruct summarize news articles. Summarize the input article succinctly 1-3 sentences, ensuring that your summary is relevant, coherent, and faithful to the original article. Only output the summary, do not repeat or confirm the instruction."
Instruct with a temperature of 0.75 and the prompt: "You are helpful assistant used to
13

---

QuEst: Enhancing Estimates of Quantile-Based Distributional Mcasures Using Model Predictions
Scoring Summaries We our model-generated summaries the three predominant metries in the literature: logical coherence, consisteney with the source text, and relevance the reader. We modified version of G-Eval (Liu et al., 2023), highly tuned evaluation prompt for scoring summaries with LLMs. As in the toxicity experiments, we use Llama_3.1_70B. Instrrr as our gold standard model and Llama 1_B Instruct our inexpensive model. Our prompt template for the relevance metric is presented below, with the templates for other metrics differing their descriptions of the metric and evaluation steps.
You will be given one summary written for news article. Your task is to rate the summary on one metric.
Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.
Evaluation Criteria:
Relevance [[lw_sccore]] J-{ (highhsccore]) - selection of important content from the source. The sunmary should include only important information from the source document. Penalize summaries which contain redundancies and excess information.
Evaluation Steps:
1. Read the summary and the source document carefully.
2. Compare the summary to the source document and identify the main points of the article.
3. Assess how well the summary covers the main points of the article, and how much irrelevant or redundant information
contains.
4. Assign score for Relevance on a scale of {(lom_score) ((high_score) where [ilow_score). is the lowest and [toieh_score) is the highest based on the Evaluation Criteria.
Now, evaluate the following document for Relevance:
Source Text: [[source_text])
Summary: [{summary))
Provide your Relevance score as SINGLE NUMBER (1[1ow_score) ) (hhigh [bigh_score))) in the below form.
Evaluation Form (scores ONLY): Relevance:
Additional Implementation Details In Section 3.2, we propose technique for fitting the À parameter so as to minimize asymptotic variance. We remark that in our applications, we sometimes will clip enforce within range, for example, [0, 1], to obtain more stable performance. Our theoretical results still hold since will still converge constant.
B. Additional Experiment Results
Figure 6 features coverage results for PovertyMap, GeneExpression, and OpinionQA. Given that we calculate 95% confidence intervals with QuEst in these experiments, we examine whether they contain the true quantity at least this often. Our empirical results empirically validate our confidence intervals.
14

---

QuEst: Enhancing Estimates of Quantile-Based Distributional Mcasures Using Model Predictions
PovertyMap, 8 E [0.33, 0.66]
GeneExpression, / E (0.8, 1.0]
OpinionQA, & E [0.25, 0.75)
1.00
Coverage
0.95
0.90
0.85
QuEst
Target
0.80
200
400
600
800
1000
200
400
600
800
1000
200
400
600
800
1000
Num. Observed
Num. Observed
Num. Observed
Figure 6. Coverage results across datasets and using different numbers observed data.
GeneExpression, B = 0.75
PovertyMap, 0.75
0.15
Estimation Error
1.00
Observed
0.75
Imputed
0.10
QuEst
0.50
PPI
0.05
0.25
Conf. Int. Width
=
0.6
2.5
QuEst
0.5
PPI
2.0
0.4
0.3
0.2
200
400
600
800
1000
200
400
600
800
1000
Num. Observed
Num. Observed
Figure 7. Experimental results for estimating VaR from three datasets (PovertyMap. GeneExpression, OpinionQA). Top: Estimation error vs. number observed labels. Bottom: Confidence interval width vs. number of observed labels.
B.1. QuEst Extension Experiments
To examine the potential benefits of directly optimizing a parameterized weighting function to estimate QBDMs, we construct a pathological case where a single correction for variance is unlikely to suffice: dataset that exhibits heteroskedasticity, ie. variance whose magnitude changes with data values. We sample 50000 linearly-spaced values from v, jitter these values by randomly adding subtracting number between and 8. We then add heteroskedasticity by first scaling data points proportionally to their absolute value, then normalizing all values to [0, 1]. We use basis function that maps each point the CDF to combination of 30 sinusoids (similar the positional embedding used in LLMs), and optimize 30-dimensional vector to rectify the embedded CDF. Each experimental result is averaged over 100 trials with differing random seeds.
Figure shows that QuEst-Opt successfully adapts to the heteroskedastictty when imputing the IQR (25th-75th percentile). outperforming QuEst across range of observed dataset sizes. Notably, the improvement over QuEst is greatest in the ultra-low observed data regime. We observe similar pattern in two real-world datasets (Figure 9). QuEst-Opt performs comparably to QuEst with tuned À Opinion QA, but performs better on the more complex Gene Expression dataset, particularly in the cases with very few observed samples.
C. Additional Theory Results
In this section, we will demonstrate the omitted parts our main context, including proofs and various of detailed formulas.
15

---

QuEst: Enhancing Estimates of Quantile-Bascl Baste Distributional Mcasures Using Model Predictions
QuEst QuEst-Opt
0.006
Qutst
aitraaton Erer
0.006
0.005
0.005
0.004
0.004
0.003
0.003
0.002
0.002
200
600
800
1000
400
600
800
1000
Num. Observed
Num. Observed
(a) Synthetic: e 5, = _," 8 E [0.25, 0.75]
(b) Synthetic: v 10, 2, Ao E [0.25, 0.75]
Figure QuEst-Opp outperforms QuEst on synthetic datasets that exhibit heteroskedasticity (varying variance). Datasets constructed by sampling 50,000 values from to j.itteeinng values randomly by most 8, and adding heteroskedasticity (Ao²).
0.011
QuEst
0.024
QuEst
0.010
QuEst Opt
0.022
QuEst Opt
0.009
0.008
0.018
0.007
0.016
0.006
0.005
0.012
0.004
0.010
1000
400
800
1000
Observed
Num. Observed
(a) Gene Expression, 3 E [0.25, 0.75]
(b) Opinion QA, / [0.25, 0.75]
Figure QuEst-Opt performs comparably to or better than QuEst on real-world datasets.
C.1. Asymptotic Normality of (Fn)
In this subsection, we will establish central limit theorem (CLT) for (Fx) for vb is (I) bounded smooth function; (2)
almost everywhere bounded and smooth function with finite discontinuous points; (3) Dirac delta function.
C.I.1. CLT FOR Qu (P,)) WITH BOUNDED SMOOTH & First, by Theorem 22.3 in (Van der Vaart, 2000), we have that
-RQU)()))RQQ(())) N(0,
(3)
where
Thus, we can further prove that (Qw((.)) - (v) = -0p (z), then we can have conclusion that
vπ (F.) Q2x(F) +D NO. F)).
(4)
Proof of (F)) (F(F) (,(1//n) In this part, we will prove that
(F(BQu(F.) (v())) - 0.
Notice that
Qw(P)= = ( ((p)F-(()ddp / ((F()))dF())
As as result,
(F.) = / (FF = Σ (F..
16

---

QuEst: Enhancing Estimates of Quantile-Basee Based Distributional Mcasures Using Model Predictions
Furthermore, we have
EQ EQu(Fn) (F,) = E
= EM, M1 bott
Bb (F.((M)))
Epsi 1,M, - M1}
-EM, M2 M, (Mp
{b(
|
n Ep>1 11M {a) )
= -f1 q BB 191
dF(1).
n
Thus,
BQx - - - / [w==+ Epsi
11M1 sat ( - - (((F()))FF()))
n
[012 Epsi {))
|] (F(()) and apply dominated
Now the only thing we need to do to study EiMobres convergence theorem.
I [Mp mb 1 + Ep>1 1aM 5 | -(F(()) =
( Epsi 1(Mp S F(g)
((F(()) -
+ 1
1a 25 (s) (=
E Ep>1 1(Mp S F(a))
2
n1
where s [0, 1].
First,
(-+ Eps1 1(M Wو] F(()) - - -F(()).
E M, (Mphrr 12 n
n
Second,
EiM (s)) (-+ EpSi 1/M, 4 F(a)) 2 s SCE(L Epsi 1(M, S 5 - -F())) = =0
for C' = ( (s).
To sum up. we prove that
BQu(F.)-
- Qu(F)=
Thus, based on the previous results, we have the following theorem:
Theorem C.1. Suppose that EM² < oc, sb is bounded function that is twice continuously differentiable, then
(T((O(FE)- - ((. ((0.
(5)
where
oil, F) - / -UoNuuun--ooro)))tttttttttt-----
17

---

QuEst: Enhancing Estimates of Quantile-Based Distributional Mcasures Using Model Predictions
C.1.2. CLT FOR (P,) WITH ALMOST EVERYWHERE BOUNDED AND SMOOTH FUNCTION HAVING FINITE DISCONTINUOUS POINTS
The proof for this case will be similar as before. We specify class of smooth functions (v/x ) such that
lim C'R + 45. 1-000
From our previous results, the CLT holds for each wh. Then, we apply dominated convergence theorem and get our final result such that
Qu((F) +D N(0,
(6)
where
of(,,)) = / Fttp://EEENN--OPFO))/EEEEEEEE)
C.1.3. CLT FOR (F,) WITH DIRAC DELTA FUNCTION
For the case that = dg, there have already been classic results built regarding the CLT of quantile functions.
Theorem C.2. Suppose that EM and there exists density function f for M that is positive and smooth enough,
Vn F. (B) -F- +D N(0, F)).
where
of(63, F) 3(1- 3)
(2(F-1(33)
8(1-A) is also included in our general formula of variance, i.e., The only thing we need to verify is that oā F) = [3]]
3(1 3(1-3) 9)
(3(F-1(3))
This indeed holds since:
/ 3(1 - (001 (w)dF--
/
B(1 - ()62(4) drdy
[(F-I(u))f(C (w))
(since dF S(F-1
3(1 (11-3)
P(F-1(B))
Combining all the above results, we know that the CLT holds for db is (1) bounded smooth function; (2) almost everywhere bounded and smooth function with finite discontinuous points; (3) Dirac delta function.
C.2. Asymptotic Normality of (A)
Notice that () (A) is sum three L-statistics, and two of them are dependent, namely, (F,) and (F,).. We need to show that Q (Fx) - (F,) will converge to a normal distribution asymptotically. This will be straightforward following Section C.1 and the proof in (Van der Vaart, 2000). Specifically, following the proof in (Van der Vaart, 2000), we have that Qu (F.) - XQu and f VT(FF (y)e (y)dy - X Vn(F, - ())))2( (y))y have the same asymptotically distribution for some deterministic functions e and e2. The latter one could written into sum of i.i.d. random variables,
18

---

QuEst: Enhancing Estimates of Quantile- Based Distributional Mcasures Using Model Predictions
which will lead CLT straightforwardly. Thus, we can obtain CLT for (A) for any fixed X. For any fixed X, under the same conditions in Section C.I,, if n/N + for some 2 O, we have
(T) - Q+ AD - ((.02(())
where
((EEF -) - 2x
and No (F, F) is the covariance Cov
Here,
[1 s2, Cov(Qu(F), (o)() / ((F(2)) (AL
s w) (w())((w) dzdw
and Z. ~ F, W ~ F. To sum up, we have the following theorem.
Theorem C.3 (Restatement of Theorem 3.1). For any fixed under regularity conditions in Section C.I, if/nn//NN r for some 0, we have
+o +p P(.,
where -))) is
+ 22 F)
if up is a (1) bounded smooth function; (2) almost everywhere bounded and smooth function with finite discontinuous points; (3) Dirac delta function.
C.3. Power-tuning with À
The (2, F, F) can be further optimized by choosing À appropriately. Given that
oz((F) 22. F)
a quadratic function regarding X, we can choose
X (F
(1+
to optimize the quadratic function.
However, when we construct confidence interval, we only have access to sample version of all types of quantities. So, we will choose À to optimize P (A, Fn, F., F% ), which gives us
My (F, F..)
-=
Here,
no (F, Fw)) ( (w))
L Σ
1Z < 2, ww Fn dzdw
where Z,'s are samples making up F, and Wt's are samples making up F, Notice that - X", by Slustky's 's rule we have that
P (2, Fr, Fr, F FR)V (Qo(A) N(011.
This will give us Corollary 3.1.
19

---

QuEst: Enhancing Estimates of -usstile-Based Distributional Mcasures Using Model Predictions
C.4. Multi-Dimensional QuEst
The multi-dimensional QuEst is straightforward to derive following the univariate case in Theorem 3.1 and Theorem 22.3 in (Van der Vaart, 2000).
We here just illustrate a bit more regarding the covariance terms in V. Specifically,
vo (F.)) Qu, A,Co Qu, (F.)) (A,Co) (Fn))
+ +, (F.) (F, CovQu, (, Qo, (Fn)) A (PH) (, Qo, (F.)) + Cov (F, (Fn), (F.)).
Notice that Fx will be independent to Fn and P., so we only need to calculate terms in the following forms.
Cov(Qu(E.s) = / ((EA(E))V(EALL))
1 - 2, 2, < w) (u())F 1. dzdw;
Cov(Qu((3). = { Σ S2, Fx())F, )zddd
and
Cov(Qu(P.). = / (AELENNCAAAA (x ddzdw.
C.5. Omitted Details of Extension: Better Rectified Estimator
Recall that we further modify our estimator (A) to Q v), which defined as
Q( / - (Fx) + (Qv(F.) - (F)(F)).
(8)
o(-), where <(-) is multi-dimennsonal vector consists of basis functions and & is In this paper, we choose 55() ( a tuning parameter vector.
Following our previous theories, if n n/N + r for some 2 O, then, for any fixed &, we have that
Vn(Q(w. we) - Qo(F))
where
=(G =1+)) ((AN://EEAAAA--OO)))))))))(( (Fd
+ / ((aAA)-F(w)FCCC)) (IFNNN)--RNOOO)))AOFE))EEEEEEEDD
- 2 / ((F(3))
w) F()) dzdw
and Z ~ F, P
Meanwhile, we define the consistent sample version of variance (5, Fn, Fn, F%)
PC Fay F,, (3)) (1+ ) -
du)
+ $ -
- 2 ( C Σ 12 2 W, s w) - (F.()) )
5=1
20

---

QuEst: Enhancing Estimates of Quantile- -aaee Distributional Mcasures Using Model Predictions
Here Z 's are samples making up Fn and Wj's are samples making up Fn. Then, have
Pe n. F2) = (1 + n It is not hard to show that / (F v) O (Fx(U))dady is the covariance matrix of the vector
N) /
(F% (F))ydyydy
J ( (w)dp
by Theorem 22.3 in (Van der Vaart, 2000), which means PL Fn, Fry Fxt semi-definite positive.
Thus, if we further add penalty term and optimize
Pe Fn, Fr. Fx)
2 ||E11
2
with a > 0, this optimization objective strongly convex.
We denote
E : argming PG Pny Fn, F3) + -
1E
We consider building CLT for Q(), vh)) To simplify the notation, we denote
O(E) = Q( ve)
for all S. We further denote
{" argming F,, F) + 1
Assume & to be bounded and smooth enough almost everywhere (for example, twice continuously differentiable). By Talyor expansion, we have
vril(E) = vnd((=)
As long we can show ( - = (1). we then can successfully build CLT because we will have
vnb(E) ~D v=((E").
where ~<p means same in distribution.
Notice that
= (F%) VeQu.. (Fn) >>p O,
since F,, and F3 both converges F, This means we only need to prove that
vnlĒ - {*) = (1)).
By the strong convexity of (E) 1E, F., Fn, /x) + / 1l we have that
(Vo2(E) vo² T (E - 5*) 2 allÉ - 2
Thus, we can easily get since (5² = and (E) = 0 by the first-order optimality condition.
alL Elll
Finally, it straightforward prove that
V02(6") = Op(1/v))
21

---

QuEst: Enhancing Estimates of -assed Distributional Mcasures Using Model Predictions
when we view the dimension of & constant. Thus, know that
- - -pp(/mm
The proof is complete.
Thus, we have the folowing theorem.
Theorem C.4 (Restatement of Theorem 5.1). If ab is bounded and twice continuously differentiable almost everywhere, we have
P
(5, (E, Fn, Fn, FT) (Qv(V, w2) Qu))) N(.1).
where & = argming P2 Fu, F,, F%) + 1| and a is any pre-specified positive constant.
22


Reviews:
[{"review_id": "fTmxSd0tUc", "reviewer": "Reviewer_ZoCm", "summary": "The authors introduce QuEst, a framework for estimating quantile-based distributional measures (e.g., VaR and CVaR in financial mathematics), which combines a smaller set of real data with a much larger set of output data from machine learning predictions, in a similar fashion as the prediction-powered inference framework introduced by Angelopoulos et al. (2023). The authors extend QuEst to be able to provide point estimates and prediction intervals in multivariate settings, further providing a more efficient implementation using a spline-function-based method. The authors showcase the efficacy of their approach in various real datasets, as well as in a red-teaming and news summarization experiments using various LLMs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback and for recognizing the contribution of QuEst.  Below we address your concerns and questions.\n\n[Q1: Relation to existing approaches]\n\nThis is correct. Existing PPI methods deal with estimators written in the form of sums of i.i.d. random variables. We strictly generalize it by moving beyond M-estimators to a new class of L-estimators (i.e., quantile-based distributional measures). Specifically, if we choose the weighting function as 1, it will become the mean estimator of a loss function, which is the objective studied in PPI. So the family of estimators we study  is a strict generalization of the previous result. The family of estimators we investigate are no longer straightforwardly in the form of sum of i.i.d. random variables as those studied in PPI, thus, it incurs new challenges and requires new techniques to build central limit theorems.\n\nFor example, in Appendix C.1.1, in order to prove CLT, we need to prove  $\\mathbb{E}Q_\\psi(F_n)-Q_\\psi (F)=o_p(1/\\sqrt{n})$ for certain $\\psi$ functions. Then, we need to use techniques in L-statistics such as Hajek decomposition to build the final CLT. In practice, this shift addresses important real-world needs, such as assessing tail behaviors, population subgroups, or other segments of interest that existing PPI methods do not handle. Our derivations yield asymptotically unbiased estimators and valid confidence intervals for these more complex target measures. Thus, QuEst closes a gap in the literature by providing a unified approach that not only captures means or single quantiles, but also richer distributional statistics of concern in diverse and important fields like biology, economics, and healthcare.\n\n[Q2: Handling L-Estimators]\n\nThis is mostly correct but not precise. Because in line 155, what we mean is that if we choose the weighting function as a special one (constant function 1), it will become the mean estimator of a loss function, which is the objective studied in PPI. Instead of claiming we can handle all L-estimators, our claim is that the family of estimators we proposed for QBDMs falls into the category of L-estimators. As for further generalization over QBDM, we can indeed generalize a bit to handle a wider range of other L-estimators (more general weighting function, and special nonlinear functional of $Q$). But a main focus here is on quantile-based distributional measures, as they are especially relevant in high-stakes applications like tail risk analysis and policy decisions.\n\n[Q3: Clarifying baselines]\n\nFor baselines, prior PPI variants are incompatible with the measures considered in our experiments; QuEst is the only existing specialized method for rigorously combining labeled and imputed data to estimate and provide CIs on these measures.\nThus we compared against the \"classic\" approach of using only the labeled data, as well as the use of imputed data only.\nFor the confidence intervals, our \"classic\" baseline still requires our highly non-trivial CLT derivation (with $\\lambda=0$).\nWe will clarify these choices in the final draft, specifying our exact baselines and emphasizing why specialized PPI methods do not directly apply to quantile-based distributional measures. This should ensure readers understand both the rationale behind our baseline selection and the distinct advantages offered by QuEst.\n\nTo further address these concerns, we have completed additional experiments on estimating the $\\beta-VaR$ QBDM (Table 1).  This experiment does allow for a direct comparison to PPI, since it is estimating a single quantile, and is thus the only QBDM computable with PPI.\nWe find that QuEst produces higher quality estimates than our usual suite of observed and imputed baselines as well as the PPI method from Angelopoulos et al.  Representative figures will be included in the revised version of the paper.\n\n | Num. Data | Observed               | Imputed               | QuEst                 | PPI                   |\n |-----------|------------------------|-----------------------|-----------------------|-----------------------|\n | 250       | 9.2e-02 ± 6.7e-03      | **6.3e-02 ± 1.8e-03**     | 6.5e-02 ± 4.8e-03     | 6.7e-02 ± 4.9e-03     |\n | 500       | 6.5e-02 ± 4.2e-03      | 6.6e-02 ± 1.9e-03     | **4.3e-02 ± 2.9e-03**     | 4.5e-02 ± 3.4e-03     |\n | 1000      | 4.3e-02 ± 3.2e-03      | 6.6e-02 ± 1.7e-03     | **3.0e-02 ± 2.0e-03**     | 3.5e-02 ± 2.4e-03     |\n | 1500      | 3.5e-02 ± 2.2e-03      | 6.6e-02 ± 2.0e-03     | **2.4e-02 ± 1.8e-03**     | 3.2e-02 ± 2.4e-03     |\n\n\n[Q4: CVaR] \n\nWe do not fully understand this point. Our focus on CVaR as one of our primary QBDMs of interest was inspired by its use in the financial mathematics literature. Our innovation applies to this measure; our \"classic\" baseline corresponds to its application based on that literature. To our knowledge, there are no other known techniques for combining gold-standard and imputed data to enhance CVaR estimates.", "claims_and_evidence": "The claims made by the authors are supported by convincing evidence, as the experiments provide validation of their technique in various settings.", "methods_and_evaluation_criteria": "I am not an expert in prediction-powered inference, but as far as my understanding goes the methods and evaluation criteria are sensible.", "theoretical_claims": "I have not checked closely the proofs, just skimmed through it. The application of the limiting theorem from Van der Vaart (2000), which I am familiar with, seems sensible and I have not identified any glaring mistake in the proof structure and reasoning.", "experimental_designs_or_analyses": "I went through the experiments and their setup, which I think are solid and motivate the presented work well.", "relation_to_broader_scientific_literature": "The authors do a good job citing relevant literature in Section 2.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Overall, I've found the paper to be well written and well structured, with the idea of QuEst providing an interesting approach of obtaining estimations of quantile-based estimation measures.\nAs I am not an expert in this literature, my main question is about the degree of innovation provided by the paper: after going through the manuscript, my understanding is that QuEst provides (a) a better alternative to point estimation to existing approaches of specific quantiles, (b) extends to measure beyond M-estimators and (c) further extends this setup to high-dimensional settings. If this is correct, how does the estimation of (a) compare with existing PPI approaches, like the ones mentioned in line 45 (second column)?", "other_comments_or_suggestions": "- Can the authors comment on whether it is fair to say that this work extends the work of PPI on M-estimators to L-estimators? (This seems to be implied by line 155, second column). Can it be generalized further than quantile-based distributional measures?\n- The baselines in experiments are not clearly explained, which I appreciate is due to the lack of space. I suggest the authors to include a more detailed explanation in the final version of the paper.\n\nLine 137. \"M-estiamtors\" -> \"M-estimators\"\nLine 199, second column. \"simultaneously\" -> \"simultaneously.\"", "questions_for_authors": "Please see the questions above.\n\nOverall I think this a good paper, with my overall recommendation being influenced by some of the points highlighted above and being outside of the prediction-powered inference literature.\n\n(Finally, as a note outside this review, it is a shame that after using CVar as an example there was not a comparison of CVar from QuEst versus more traditional financial mathematics approaches, but I am sure someone might work on that once the paper is out.)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SGwNuSvP3N", "reviewer": "Reviewer_bBBK", "summary": "The authors propose QuEst to estimate a quantile-based distributional measure (see Defintion 1) in a setting where there is a small set of high-quality data and a large set of low-quality data. For the high-quality data, it assumes that their low-quality estimates are also known and follow the same low-quality distribution that governers the low-quality data. In this setting, the shift between high- and low-quality distribution can be estimated using the small set of data, and then be used to correct the estimate computed using only the large set of low-quality data. This correction is the key step of their proposed QuEst, as described in Eq. (2). The hyperparameter $\\lambda$ in Eq. (2) is optimized using Theorem 3.1, which essentially balances the contribution of the high- and low-quality data based on their proportions. By using the optimized $\\lambda$ in Eq. (2), the authors arrive at their proposed QuEst.\n\n## update after rebuttal\n- Despite the heavy theoretical background, the high-level intuition seems simple to me. According to the assumptions made in Section 3.1, it is expected that $Q\\_{\\psi}(F_n) - \\lambda Q\\_{\\psi}(\\widetilde{F}_n) \\approx Q\\_{\\psi}(F\\_{N}\\^{u}) - \\lambda Q\\_{\\psi}(\\widetilde{F}\\_{N}^{u})$ because $\\lim\\_{n\\to\\infty}Q\\_{\\psi}(F_n) = \\lim\\_{N\\to\\infty}Q\\_{\\psi}(F\\_{N}\\^{u})$ and $\\lim\\_{n\\to\\infty}Q\\_{\\psi}(\\widetilde{F}_n) = \\lim\\_{N\\to\\infty}Q\\_{\\psi}(\\widetilde{F}\\_{N}^{u})$, which is roughly an application of some variant of law of large numbers. Rearranging this approximate equality yields Eq. (2), which underpins the proposed method. After reading the first rebuttal, I realized that the primary difference lies in the specific variant of the law of large numbers employed. The authors distinguish the two variants using the concepts of M-estimators and L-estimators. However, I am uncertain about the novelty of this distinction in the present context. \n\n- Regarding the spline-function-based method, I think the authors' initial rebuttal was weak, as the relevant content is entirely missing. After reading their second response (which contains information absent from the submission), I agree that using a function basis could potentially improve the quality of estimation. However, the submission lacks concrete theoretical support for evaluating its efficiency, and I do not find the experimental results in Table 3 sufficient to empirically validate the claimed superior efficiency. Moreover, how their spline-function-based method was implemented is also missing.\n\nOverall, I will raise my score to 2. The main reason it is not higher is that the section on the spline-function-based method is clearly incomplete, considering that the authors highlight it as a key contribution in the submission.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and consideration in reviewing our work.  Here we respond to specific concerns raised in the review. We hope that our answers can resolve the misunderstanding and help to increase our score.\n\n[Q1: Novelty compared to existing work.]\n\nAt the first glance, the surface form of Eq. (2) might resemble that of Angelopoulos et al. (2023). However, there is a significant difference between our frameworks. \n\nQuEst advances the prediction-powered inference paradigm beyond M-estimators (e.g., means, single quantiles) to fully encompass L-estimators, which can express diverse distributional measures like VaR, CVaR, and multi-quantile segments. This expansion requires new theoretical machinery, as our estimator cannot be straightforwardly written into sums of i.i.d. random variables. We rely on techniques from L-statistics, e.g., Hajek decomposition, and we also need to prove that $\\mathbb{E}Q_\\psi(F_n)-Q_\\psi (F)=o_p(1/\\sqrt{n})$ for certain $\\psi$ functions, thereby establishing asymptotic normality under suitable conditions. Furthermore, QuEst integrates the concept of “power tuning” into this broader class of estimators and extends it with more sophisticated (spline-function-based) corrections in Sec.5, improving statistical efficiency in scenarios where a simple scalar multiplier would be insufficient. We also address multidimensional estimation, allowing simultaneous handling of multiple distributional measures (e.g., tail segments of multiple metrics) in a way that is not possible in prior PPI methods.\n\nFrom a practical standpoint, this generalization to L-estimators is essential for real-world tasks where the distributional properties themselves matter, such as identifying extreme behaviors or analyzing key population subgroups. QuEst provides asymptotically unbiased estimates and valid confidence intervals for quantities of substantial interest in domains like finance, public policy, and healthcare. Alongside the theoretical results, we present new variance formulas, covariance expressions, and performance bounds tailored to the L-estimator setting, offering a blueprint for rigorous inference on an array of tail or segment-based metrics. In sum, QuEst goes far beyond merely adapting an existing correction term; it is a comprehensive framework that systematically enables distributional inference where M-estimator-based prediction-powered approaches do not apply.\n\n[Q2: Use of splines in extension]\n\nSorry for causing confusion. We are referring to the result in our “Extension” section. There, we introduce a more complicated version of power tuning instead of just tuning a multiplicative scalar lambda. Specifically, we consider tuning the coefficient vector for a given basis function in $\\xi^T\\phi(\\cdot)$ (the tuning parameter is $\\xi$ here). We specifically tried polynomial function basis, that is why we call this “spline function based”. Notice that it is a completely new and more advanced tuning method and we introduce new theorems and new techniques to get **more statistically efficient estimators**.\n\n[Q3: “Classic” baseline for confidence intervals] \n\nWith respect to baselines, prior PPI variants are incompatible with the measures considered in our experiments; QuEst is the only existing specialized method for rigorously combining labeled and imputed data to estimate and provide CIs on these measures.\nThus we compared against the \"classic\" approach of using only the labeled data, as well as the use of imputed data only.\nFor the confidence intervals, our \"classic\" baseline still requires our highly non-trivial CLT derivation (with $\\lambda=0$).\nWe understand that this was not clear in the original submission, and would definitely clarify these details in our future revision.\n\n[Q4: Other baselines]\n\nIn order to respond to these concerns, we have completed additional experiments on estimating the $\\beta-VaR$ QBDM (Table 1).  This experiment does allow for a direct comparison to PPI, since it is estimating a single quantile.\nWe find that QuEst produces higher quality estimates than our usual suite of observed and imputed baselines as well as the PPI method from Angelopoulos et al.  Representative figures will be included in the camera-ready version of the paper.\n\n | Num. Data | Observed               | Imputed               | QuEst                 | PPI                   |\n |-----------|------------------------|-----------------------|-----------------------|-----------------------|\n | 250       | 9.2e-02 ± 6.7e-03      | **6.3e-02 ± 1.8e-03**     | 6.5e-02 ± 4.8e-03     | 6.7e-02 ± 4.9e-03     |\n | 500       | 6.5e-02 ± 4.2e-03      | 6.6e-02 ± 1.9e-03     | **4.3e-02 ± 2.9e-03**     | 4.5e-02 ± 3.4e-03     |\n | 1000      | 4.3e-02 ± 3.2e-03      | 6.6e-02 ± 1.7e-03     | **3.0e-02 ± 2.0e-03**     | 3.5e-02 ± 2.4e-03     |\n | 1500      | 3.5e-02 ± 2.2e-03      | 6.6e-02 ± 2.0e-03     | **2.4e-02 ± 1.8e-03**     | 3.2e-02 ± 2.4e-03     |", "claims_and_evidence": "One of the contributions they claim in the abstract is that \"Further, we offer a novel spline function based method for optimizing our method.\" However, the word \"spline\" does not appear outside the abstract and the introduction section.", "methods_and_evaluation_criteria": "The proposed QuEst makes sense in their specified setting.", "theoretical_claims": "I did not verify the proofs of their theoretical claims, as they fall outside my area of expertise.", "experimental_designs_or_analyses": "One issue is that they do not clearly define what the \"classic method\" is, which is used in the experiments comparing confidence intervals.\nAnother *possible* issue is the absence of other baselines in their experiments comparing the quality of estimates.", "relation_to_broader_scientific_literature": "I am not familiar with the context.", "essential_references_not_discussed": "I am not familiar with the context.", "other_strengths_and_weaknesses": "The major concern I have is what is novel in this paper compared to the cited references. The key step of their proposed QuEst, as described in Eq. (2), already appears in (Angelopoulos et al. 2023, Section 2.1). Moreover, the idea of optimizing $\\lambda$ in Eq. (2) can also be found in (Angelopoulos et al. 2023, Section 6). The only difference I noticed is the quantity to estimate.\n\n**References**\n\n\nAngelopoulos, A. N., Duchi, J. C., & Zrnic, T. Ppi++: Efficient prediction-powered inference, 2023. URL https://arxiv.org/abs/2311.01453.", "other_comments_or_suggestions": "NA", "questions_for_authors": "- For the proposed QuEst, what is novel compared to the theoretical framework established in (Angelopoulos et al. 2023)?\n\n- It is possible that I missed it, but where is the proposed so-called spline function based method?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o8Htw5Pr08", "reviewer": "Reviewer_iSpq", "summary": "Authors propose a novel framework—QuEst—that enhances quantile based distributional measure estimation by combining scarce high quality observations with abundant model-predicted data. It produces an asymptotically unbiased estimator with reduced variance and can be further optimized using spline functions. Empirical results demonstrate that QuEst achieves accurate estimates and tighter confidence intervals  than previous methods in different domains.\n\n## Update after rebuttal:\nI thank the authors for their responses. My overall rating and assessment of the work remains positive, and so I maintain my score and recommend weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful remarks.  Please see below for responses to individual concerns.\n\n[Q1: Imputed data quality] \n\nFirst, we would like to clarify that our remark in the limitations section concerned the problem that **if we want to have significant gain in variance reduction**, we need **relatively** high-quality predictions. There are two points we want to address here regarding methodology:\n\n- We want to emphasize, **relatively** high-quality imputed data is enough to observe a significant gain in variance reduction. We don't need $100\\%$ correct annotations for unlabeled data. Actually, in our experiments (Figure 4), we show that QuEst performs effectively with correlation of less than 0.1 (a very noisy regime) between the observed and imputed labels, and we still observe significant benefit by incorporating the imputed data in the estimator. \n\n- Another benefit of this framework is that our framework can automatically adapt to the quality of imputed data: if the imputed data are of high quality, then our algorithm will choose a large $\\lambda$ to weight more on the imputed data in the estimator. On the other hand, if the imputed data is of low quality, our algorithm will set smaller weight to imputed data (if $\\lambda=0$, it just uses the gold-standard data alone). Since our algorithm is optimizing the tuning parameter $\\lambda$, we would expect our estimator can always perform better than only using gold-standard label alone.\n\n\n\n\nWe will explicitly clarify this distinction in the final draft and emphasize that QuEst automatically debiases poor imputation quality and gracefully reverts to using the observed data as the primary signal. \n\n\n[Q2: Hyperparameters]\n\nAlso, in response to the reviewer’s concern we will further clarify that our core estimator does not involve any hyperparameters in the closed-form tuning for $\\lambda$, which we derived theoretically.  We believe that our empirical results across five distinct datasets and a wide range of labeled/unlabeled splits suggest this tuning is robust.\n\n\n[Q3: Variance in Figure 2]\n\nWith respect to Figure 2, we ran 2000 trials, and the resulting variance in Figure 2 is extremely small, rendering the error bars nearly invisible for most of the plots.  \n\n\n[Q4: Prompt design]\n\nAlthough we did not vary prompt design in our study, QuEst remains valid regardless of how the imputed data is generated—it will simply adapt to the correlations observed between imputed and true labels. Empirically, we show that even inaccurate predictions can be used effectively by QuEst: for example, accuracy on OpinionQA is roughly 50\\%, but the QuEst estimate still consistently beats baselines.\n\n\n[Q5: Other baselines]\n\nWith respect to baselines, prior PPI variants are incompatible with the measures considered in our experiments; QuEst is the only existing specialized method for rigorously combining labeled and imputed data to estimate and provide CIs on these measures.\nThus we compared against the \"classic\" approach of using only the labeled data, as well as the use of imputed data only.\nFor the confidence intervals, our \"classic\" baseline still requires our highly non-trivial CLT derivation (with $\\lambda=0$).\nWe will clarify these choices in the final draft, specifying our exact baselines and emphasizing why specialized PPI methods do not directly apply to quantile-based distributional measures. This should ensure readers understand both the rationale behind our baseline selection and the distinct advantages offered by QuEst.\n\nAlso, in order to further respond to these concerns, we have completed additional experiments on estimating the $\\beta-VaR$ QBDM (Table 1).  This experiment does allow for a direct comparison to PPI, since it is estimating a single quantile.\nWe find that QuEst produces higher quality estimates than our usual suite of observed and imputed baselines as well as the PPI method from Angelopoulos et al.  Representative figures will be included in the camera-ready version of the paper.\n\n | Num. Data | Observed               | Imputed               | QuEst                 | PPI                   |\n |-----------|------------------------|-----------------------|-----------------------|-----------------------|\n | 250       | 9.2e-02 ± 6.7e-03      | **6.3e-02 ± 1.8e-03**     | 6.5e-02 ± 4.8e-03     | 6.7e-02 ± 4.9e-03     |\n | 500       | 6.5e-02 ± 4.2e-03      | 6.6e-02 ± 1.9e-03     | **4.3e-02 ± 2.9e-03**     | 4.5e-02 ± 3.4e-03     |\n | 1000      | 4.3e-02 ± 3.2e-03      | 6.6e-02 ± 1.7e-03     | **3.0e-02 ± 2.0e-03**     | 3.5e-02 ± 2.4e-03     |\n | 1500      | 3.5e-02 ± 2.2e-03      | 6.6e-02 ± 2.0e-03     | **2.4e-02 ± 1.8e-03**     | 3.2e-02 ± 2.4e-03     |", "claims_and_evidence": "QuEst can achieve more accurate and reliable estimates for quantile￾based distributional measures than methods relying solely on either \nobserved or imputed data.", "methods_and_evaluation_criteria": "The method and evaluation in the paper are well-motivated and comprehensive.", "theoretical_claims": "The theoretical claims appear well-founded and are built upon established statistical principles.", "experimental_designs_or_analyses": "- Overall, the experimental evaluation is thorough and well-designed, but there are a few areas worth noting: QuEst’s performance improves as the quality (i.e., the correlation between observed and imputed data) increases. However, the  experiments do not fully explore scenarios where imputed data is of lower quality or noisy. \n- The method hinges on the proper tuning of the λ parameter (or its optimized variant in QuEst-Opt). While the paper provides closed-form expressions and some empirical evidence of optimal tuning, additional details on hyperparameter sensitivity and robustness across a wider \nrange of settings might further strengthen the evaluation", "relation_to_broader_scientific_literature": "The impact statement in this paper has well stated the borader impact of this paper.", "essential_references_not_discussed": "All good.", "other_strengths_and_weaknesses": "- As the authors mentioned in the limitations section, a key weakness is the heavy reliance on high-quality imputed data, which may reduce robustness when predictions are noisy. However, this paper follows a similar style to PPI, which leverages pretrained model information and provides valid confidence intervals for estimating certain quantities. In this context, this approach is reasonable.  \n- In Figure 2, only the mean error is reported. Please include variance by running additional experiments.  \n- Does the prompt design impact the performance of the proposed method?  \n- Can more baseline methods be included? Currently, only two naive baselines are presented in the paper.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "21j17PL8DQ", "reviewer": "Reviewer_dnuP", "summary": "This paper introduces the QuEst method, which combines a few observed data with a large quantity of imputed data to derive enhanced estimates and reliable confidence intervals for quantile-based distribution metrics (QBDM). The method demonstrates its value in real-world applications, such as LLM Auto-Evaluation, with both theoretical and experimental evidence supporting its effectiveness.\n\n## update after rebuttal\nIn the rebuttal, the authors clarified that their method achieves variance reduction without extra assumptions and degrades when imputation quality is low, addressing my main concern. Hence, I maintain a positive rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful assessment of our work and the thoughtful suggestions.  Below, we respond to particular concerns.\n\n\n[Q1: Comparisons with more methods, such as those in Section 3.2]\n\nWith respect to baselines, prior PPI variants are incompatible with the measures considered in our experiments; QuEst is the only existing specialized method for rigorously combining labeled and imputed data to estimate and provide CIs on these measures.\nWe will clarify these choices in the final draft, specifying our exact baselines and emphasizing why specialized PPI methods do not directly apply to quantile-based distributional measures. \nThis should ensure readers understand both the rationale behind our baseline selection and the distinct advantages offered by QuEst. \n\nAlso, in order to further respond to these concerns, we have completed additional experiments on estimating the $\\beta-VaR$ QBDM (Table 1).  This experiment does allow for a direct comparison to PPI, since it is estimating a single quantile.\nWe find that QuEst produces higher quality estimates than our usual suite of observed and imputed baselines as well as the PPI method from Angelopoulos et al.  Representative figures will be included in the camera-ready version of the paper.\n\n | Num. Data | Observed               | Imputed               | QuEst                 | PPI                   |\n |-----------|------------------------|-----------------------|-----------------------|-----------------------|\n | 250       | 9.2e-02 ± 6.7e-03      | **6.3e-02 ± 1.8e-03**     | 6.5e-02 ± 4.8e-03     | 6.7e-02 ± 4.9e-03     |\n | 500       | 6.5e-02 ± 4.2e-03      | 6.6e-02 ± 1.9e-03     | **4.3e-02 ± 2.9e-03**     | 4.5e-02 ± 3.4e-03     |\n | 1000      | 4.3e-02 ± 3.2e-03      | 6.6e-02 ± 1.7e-03     | **3.0e-02 ± 2.0e-03**     | 3.5e-02 ± 2.4e-03     |\n | 1500      | 3.5e-02 ± 2.2e-03      | 6.6e-02 ± 2.0e-03     | **2.4e-02 ± 1.8e-03**     | 3.2e-02 ± 2.4e-03     |\n\n\n[Q2: The motivation and advantages of our extension in Section 5.]\n\nWe here address the reviewer's question from two aspects: \n\n- Motivation: We leverage the fact that our estimator remains asymptotically unbiased regardless of which weighting function $\\tilde{\\psi}$ is chosen, because the expectations of its two terms cancel out. Thus, by letting $\\tilde{\\psi}$ be a flexible tuning function---rather than simply setting $\\tilde{\\psi} = \\psi$---we can optimize for better debiasing and variance reduction.\n\n- Advantage: Instead of tuning a single scalar, we introduce a basis-function approach (e.g., polynomial or ``spline'' basis) and optimize the associated coefficient vector $\\xi$. This advanced method, backed by new theorems and techniques, can yield more efficient estimators in both synthetic and real-world settings, as it provides added flexibility in controlling bias and variance.\n\n\n[Q3: Imputed data quality]\n\nWith respect to imputation quality, we emphasize that QuEst’s adaptive nature ensures that it remains valid even if the imputed data deviates substantially from ground truth: if the imputed data are of high quality, then we choose a large $\\lambda$ to weight more on the imputed data in the estimator. On the other hand, if the imputed data is of low quality, we will set smaller weight to imputed data (if $\\lambda=0$, it just uses the gold-standard data alone). Since we are optimizing the tuning parameter $\\lambda$, we would expect our estimator can always perform better than only using gold-standard label alone. \nEmpirically (as in Fig. 4 with correlation $<0.1$), the estimator self-corrects by increasingly falling back on the smaller observed dataset if predictions are poor (with the $\\alpha$ parameter). \n\n\n[Q4: Additional assumptions]\n\nAs it pertains to assumptions, our estimator achieves asymptotic variance reduction relative to naive estimates without imposing additional assumptions beyond those enumerated in the paper. As mentioned in the previous question, our framework can automatically adapt to the quality of imputed data and  we would expect our estimator can always perform better than only using gold-standard label alone.\n\n\n[Q5: Finite-sample properties of the proposed estimator]\n\nThere are two points we want to address. 1. Building central limit theorems actually only require a couple of hundreds gold-standard labeled data, so in practice, our framework works quite well with limited gold-standard labeled data. 2. If we falls into the regime of extreme small data, i.e., we only have less than 100, then the data uncertainty will be too huge and **no** statistically principled methods would work in the distribution-agnostic setting (this can be proved by using \"no free-lunch theorem\" in learning theory). We have to impose more assumptions to gain stronger results.", "claims_and_evidence": "- The claim that “QuEst can provide more accurate quantile-based distribution metric (QBDM) estimates by integrating observed and model-predicted data” is substantiated by theoretical and experimental results.\n- The theoretical and empirical findings support the claim that “QuEst performs well in multidimensional and complex scenarios.” However, additional experiments in high-dimensional settings would further validate the method's efficacy.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria are reasonable and effectively address the research problem. QuEst combines observed and imputed data to provide more accurate QBDM estimates, and experimental results across various domains demonstrate its effectiveness.", "theoretical_claims": "I have not verified the correctness of the theoretical claims in the paper.", "experimental_designs_or_analyses": "The experimental design and analysis are sound and comprehensively validate QuEst’s performance. However, incorporating comparisons with more advanced methods, such as those discussed in Section 3.2, would enhance the paper's persuasiveness.", "relation_to_broader_scientific_literature": "The proposed QuEst is a rigorous extension of the Prediction-Powered Inference (PPI) framework. By introducing L-estimators, it improves the estimation of quantile-based metric, showcasing advantages across multiple application areas.", "essential_references_not_discussed": "The paper thoroughly discusses previous studies. While no critical references appear missing, the possibility of omissions cannot be completely ruled out due to my limited familiarity with this area.", "other_strengths_and_weaknesses": "**Strengths**:\n- S1: The topic is valuable, especially the application to \"LLM Auto-Evaluation.\"\n- S2: The paper is well-written and easy to follow.\n- S3: Experimental results support the proposed claims and demonstrate the method’s effectiveness.\n\n**Weaknesses**:\n- W1: More comparisons with advanced methods, like those in Section 3.2, would strengthen the paper.\n- W2: The motivation and advantages of tuning a set of basis function coefficients rather than the original $\\lambda$, as presented in Section 5 (\"Extension\"), need further discussion.", "other_comments_or_suggestions": "There are some duplicated references that should be checked.", "questions_for_authors": "**Questions**:\n- Q1. Could the authors further discuss the requirements for imputed data to assist in estimating the target statistics? Intuitively, when imputed data significantly deviates from the true values, the assumptions needed for certain conclusions (e.g. variance reduction) may be violated, raising questions about the necessity of imputed data.\n- Q2. I wonder whether the proposed estimator $\\hat Q_{\\psi}(\\lambda)$ achieves asymptotic variance reduction while maintaining asymptotic unbiasedness without any additional assumption, compared to the previous estimator $Q_{\\psi}(F_n)$. If additional assumptions are introduced, please discuss what they are and whether they hold.\n- Q3. Could the authors briefly discuss the finite-sample properties of the proposed estimator?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces QuEst, a framework that combines scarce observed data with abundant imputed data for the estimation of quantile-based distributional measures (QBDMs). The reviewers generally found the topic valuable and the core idea of QuEst interesting, particularly for applications in LLM auto-evaluation and other domains where understanding distributional properties is crucial. The proposed method extends and generalizes the Prediction-Powered Inference (PPI) framework, expanding its applicability from M-estimators to a particular class of L-estimators. I recommend acceptance of this paper. 

To further strengthen the paper, the authors should adequately address the feedback provided by the reviewers in their camera-ready version, including novelty clarification, experimental baselines, spline-based optimization method, imputed data quality, etc.
Decision: Accept (poster)