$\texttt{BetaConform}$: Efficient MAP Estimation of LLM Ensemble Judgment Performance with Prior Transfer
Authors: Huaizhi Qu
Inyoung Choi
Zhen Tan
Song Wang
Sukwon Yun
Qi Long
Faizan Siddiqui
Kwonjoon Lee
Tianlong Chen
Venue: NeurIPS 2025 poster ()
Keywords: LLM-as-a-Judge
Distribution Estimation
LLM Ensemble
Primary area: applications
TL;DR: We present an efficient judgmetn distribution estimation method for LLM ensembles.

Abstract: LLM ensembles are widely used for LLM judges. However, how to estimate their accuracy, especially in an efficient way, is unknown. In this paper, we present a principled $\textit{maximum a posteriori}$ (MAP) framework for an economical and precise estimation of the performance of LLM ensemble judgment. We first propose a mixture of Beta-Binomial distributions to model the judgment distribution, revising from the vanilla Binomial distribution. Next, we introduce a conformal prediction-driven approach that enables adaptive stopping during iterative sampling to balance accuracy with efficiency. Furthermore, we design a prior transfer mechanism that utilizes learned distributions on open-source datasets to improve estimation on a target dataset when only scarce annotations are available. Finally, we present $\texttt{BetaConform}$, a framework that integrates our distribution assumption, adaptive stopping, and the prior transfer mechanism to deliver a theoretically guaranteed distribution estimation of LLM ensemble judgment with minimum labeled samples. $\texttt{BetaConform}$ is also validated empirically. For instance, with only $10$ samples from the TruthfulQA dataset, for a Llama ensembled judge, $\texttt{BetaConform}$ gauges its performance with an error margin as small as $3.37\\%$.

BetaConform: Efficient MAP Estimation of LLM Ensemble Judgment Performance with Prior Transfer
Huaizhi Qu', Inyoung Choi², Zhen Tan', Song Wang', Sukwon Yun', Qi Long', Faizan Siddiqui", Kwonjoon Lee', Tianlong Chen' 'University of North Carolina at Chapel Hill, Univversity of Pennsylvania Arizona State University UUniversity of Virginia Honda Research Institute USA [huaizhiq, tianlong]@cs. unc
Abstract
LLM ensembles are widely used for LLM judges. However, how to estimate their accuracy, especially in an efficient way, is unknown. this paper, we present a principled maximum posteriori (MAP) framework for an economical and precise estimation of the performance of LLM ensemble judgment. We first propose a mixture of Beta- eeta-Binomial distributions to model the judgment distribution, revising from the vanilla Binomial distribution. Next, we introduce a conformal prediction- driven approach that enables adaptive stopping during iterative sampling to balance accuracy with efficiency. Furthermore, we design a prior transfer mechanism that utilizes learned distributions on open-source datasets to improve estimation on a target dataset when only scarce annotations are available. Finally, we present BetaConfora. a framework that integrates our distribution assumption, adaptive stopping, and the prior transfer mechanism to deliver theoretically guaranteed distribution estimation of LLM ensemble judgment with minimum labeled samples. BetaConfors fomm orm is also validated empirically. For instance, with only 10 samples from the TruthfulQA dataset, for Llama ensembled judge, BetaConfors orm gauges its performance with an error margin as small as 3.37%.
Introduction
With the improving performance of large lan- guage models (LLMs), there a proliferation of adopting LLMs as judges for various tasks
Ensemble
Hallucination: "Elephant the biggest animal"
Wrong Judgment: By majarity is correct
[Liang al. 2023 Yuan et al. 2024b ZZhng et al. 2025 In applications of LLM judge en- sembles, the judgment distribution is critical to
Judge
How jucigas correct?
?8
How many samples needed?
the service quality [Chen et al.) 2024, Schoeneg-
Ensemble
Num. Questions
ger et al. 2024
Quu et al. 2025 Many datasets
samples
[Zheng al.) 2023 Zeng et al. 2023 Yuan
Judge
Judge 2 Judge 3
0123
Correct
Judges
et al.] 2024a have been employed to evaluate the performance of LLM judges. However, these datasets rely on human annotations, which are impractical large scale due the substan- tial time and financial costs of annotating. This challenge highlights the need of how estimate
Figure I: In this paper, we aim to answer (1) how estimate the judgment distribution of LLM en- semble on dataset, and (2) how to achieve effi- cient estimation to reduce annotation effort.
the L.M ensemble judging performance efficiently.
In this work, we consider the following judgment distribution estimation problem:
P(# correct judgments
L.LMs judge sample .:).
We propose an efficient method for MAP estimation of the distribution of L.LM ensemble judgment answer two research questions shown in Figure
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

RQ1: How to efficiently and accurately estimate the judgment distribution?
RQ2: How many samples are needed for estimation under given enror margin threshold?
Given a small number of samples, one intuitive estimation is to directly adopt the distribution of the samples as the judgment distribution the entire dataset. However, this is susceptible to the sampling bias. To avoid this, one common practice is to first calculate the single LLM accuracy on the samples and then model the distribution on the full dataset as Binomial. We first posit that the judgment distribution is not Binomial. Theoretically. a Binomial distribution implies increasing accuracy in majority voting as the ensemble size grows (De Condorcet et al.. 2014 Austen-Smith and Banks) 1996]. However, this is unrealistic since the accuracy of LLM ensembles remains bounded even with large number of judges. To testify to this, we start by observing the distribution of LLM ensemble judges on various benchmarks. We find marked deviations from the Binomial distribution and show a stratification between questions that can be classified as "easy" and "hard". To this end, we propose to model the judgment distribution with mixture of Beta-Binomia distributions to reflect the stratification. We show that under this assumption, by utilizing an expectation maximization (EM) estimation method, it can achieve accurate judgment distribution estimation with high data efficiency.
To rigorously guide the sampling process and determine how many samples to use for the es- timation, we draw inspiration from the confor-
BetaConform
Num of Quertions
Semple
Transfor (4)
mal prediction (CP) [Shafer and Vovk 2008
Fontana et al.] 2023 that can efficiently esti- mate the sampling deviation. Based on this, we propose a novel adaptive stopping strategy for it- erative sampling, designed to meet pre-defined deviation threshold. Our experiments demon- strate the effectiveness of this method for limit- ing the sample amount while maintaining high estimation precision.
(b) Adaptiiv Stopping
(eliideibution Prior Transfer Pu
Souree Datret
Nunber
Dravel))
Moreover, we hypothesize that the prior knowl- edge of judgment distribution open-source datasets can benefit the estimation of a new dataset when only few samples are available. To achieve this, we propose a text similarity- based distribution prior transfer mechanism. This method embeds text inputs from both source and target datasets and calculates embed- ding similarities to determine the transfer weight, Our design greatly improves the estimation ac- curacy when transferring from similar datasets and avoids performance degradation when the datasets are distinct. Notably, this method relies solely the text inputs, making it practical for Our contribution can be summarized as follows:
Figure 2: target dataset, adaptive stopping is adopted to de- termine the sample amount (b, Section [5). Dur- ing iterative sampling, the sampling deviation monitored by using conformal prediction. The sampling process stops when the deviation is suf- ficiently low. Next, the estimation of the small number of samples from the previous step is fur- ther enhanced by transferring distribution priors from source datasets (c, Section 6). The transfer will assign larger weight the dataset that is textually closer to the target dataset.
Overview of BetaConform Given
application vast amounts of unlabeled data.
We present pioneering work in judgment distribution estimation. We point out that the Binomial assumption of judgment distribution is inaccurate. By replacing it with a mixture of Beta-Binonnnn Binomial distributions, we could achieve efficient and accurate estimation.
We design a rigorous conformal prediction-based adaptive stopping strategy during iterative sampling when the sampling deviation is sufficiently low.
We introduce distribution prior transfer mechanism that leverages judgment distributions open-source datasets to improve few-sample estimations.
Extensive experiments show BetaConform 's high estimation efficiency. For example, using only 10 samples could result in an average of 10.84% error margin.
2 Related Works
L.MSs for Judgment. Reliable model evaluation critical problem. Traditional human eval- uations remain the gold standard, but their scalability is significant bottleneck in large-scale
2

---

applications. Thus, recent works have proposed leveraging LLMs to evaluate the text quality, ranking
outputs, and ensuring alignment with human preferences [Zheng et al. 2023 Liu et al. et al., [024]. While initially focused on text generation evaluation, the use of LLMs judges has expanded to diverse applications including model alignment and safety assessment [Lee et al. 2024
2023 Dubois
code quality evaluation (Zhao et al. 2024b), and knowledge verification [Min et al. 2023 etc.
Challenges and Limitations. The reliability of such frameworks is not without concerns. Studies have found that even advanced models like GPT-4 often exhibit systematic biases such as position bias and egocentric bias (Zeng al.] 2023 Wang al. 2023], overconfidence in their judgments [Koo et al. 2024]. and self-preference effects [Panickssery et al. 2024). Moreover, many studies
employing LLM annotations do not explicitly measure the alignment between LLMs and humans,
thus further raising questions about their dependability (Calderon et al. 2025]. While researchers have proposed various solutions, including dynamic evaluation pipelines Yu et al. 2024 Zhao et al.
2024a Moniri et al. [024]. self-reflection mechanisms [Wu et al. 2024 Li ct al. [2033b] Wang et al.
2024 and specialized benchmarks for assessing judge performance Zheng et al. [023] [Ta et al.
2024. Park et al., 2024 Li et al. [024] /Zhao et al. [2024b], these methods often fall short in offering
rigorous guarantees their outcomes. A related line of research is Item Response Theory (IRT) (Cai et al. 2016 Baker] 2001 Harvey and Hammer] 1999], which assesses respondents" latent abilities using responses to calibrated questions. However, the requirement for calibrated questions limits the direct applicability of IRT in the context of judgment distribution estimation, as datasets in this domain are frequently unlabeled.
Statistical Approaches. Another direction of research focuses on providing statistical guarantees for LLM performance. Researchers have explored conformal methods (Angelopoulos et al. 2023) to ensure correctness and factuality [Mohri and Hashimoto 2024] and to determine when LLMs should abstain from responding (Yadkori et al. [024]]. While these methods provide some statistical rigor, there is still need for unified framework that establishes reliable, theoretically grounded approaches for assessing LLM performance across diverse applications.
3 Problem Setup
We consider the task of using LLM ensemble to evaluate and judge samples by discerning. choosing, or scoring. Let:
Total number of samples in the dataset be judged.
k: Number of LLMS in an ensemble.
S: The random variable of correct judgments.
7: Number of samples to estimate S.
D: A dataset to estimate the judgment distribution.
Definition 1 (LLM Ensemble Judgment). Let J = (J1, J&} be an ensemble of LLM judges. For given input x, each LLM Jj generates an output o, = (z)), yielding the set of all judgments = 01, ................................ }. In this paper, we focus on binary and scoring judgments. We consider the LLM ensemble to be composed of multiple instances of the same underlying model (e.g.. 11 Llama models). Variations in their judgments for a given input are due to Top P token sampling Zhou et al.) 2024 and the difference in random seeds.
Definition (L.LM Ensemble Correct Judgment). For an ensemble of LLMs,, the random variable S = - =i Match y) represents the number of correct judgments. y denotes the ground truth, and Match(-) is the criterion for correct judgment. For instance, for binary classification judgments, Match (-) could be an exact match; for scoring judgments, it could be whether the score falls within predefined range of the human average score. The ensemble's decision is deemed correct if S S [k/2]. To prevent ties, which can occur if l is an even integer and S = [k/2],, we stipulate that l must be odd integer.
4 Mixture of Beta-Binomial Distribution
4.1 Examination of Binomial Distribution
We start by examining the common assumption of follows Binomial distribution, i.e. the probability of having correct judgments when single judge accuracy p is,
3

---

GPT-4 on HaluBval
GPT- en JudgeBench
Llama on HaluBval
Llama on JudgeBench
63
Fraction
Nomber Correct Judgments
Number of Correct Jodgments
Number Correct Judgments
Coreee/udggeett
Number of Correct Judgments
Bisonual Distriration
Our Dismibution
Actul Distiberion
Figure 3: Comparison of judgment distributions among actual, Binomial, and ours. Llama-3.3-70B
and GPT-4 ensembles of 11 models are tested on HaluEval and JudgeBench, respectively. The Binomial distribution is estimated by using single judge accuracy p. Our mixture distribution is estimated with 100 samples and scaled to the full dataset. Our distribution is consistently better.
HaluEvaal
GPT4 JudgeBench Bench
Llama Halulvval
Llama JudgeBench
Errar Rate
0.2
Number of Judges
Number of Judges
Namber of Judges
Number of Judges
Distrirutton
Figure 4: Majority voting error rate of actual, Binomial, and mixture distribution. Binomial uses single judge accuracy p. Our distribution estimated with 100 random samples and tested for 3 times. The line denotes the average error rate and the shadow represents the standard variance. Binomial shows decreasing error rate, while our distribution captures the actual trend.
kk
PBi (S s) Bin(s p)
10(1 1)
8
The error rate Pain of ensemble judgment is:
[k/2]-1
k (5)
Bin Pini (S < [k/27] Σ
(1(1)))
(2)
==0
We first examine the common assumption that S follows a Binomial distribution in Equation ( Specifically, we 1 evaluate individual LLMs datasets across domains and @ use the single LLM accuracy p in Equation ( and 2 to estimate both the distribution datasets and the majority voting егтоr rate for different numbers A: of LLMs. Specifically, we evaluate GPT-4 [OpenAl et al. 2024] and 1.3ma-3.3-700 [Dubey et al., 2024 on hallucination detection
LLM ensembles on these
(HaluEval, Li ct al. 2023a) and Human alignment (JudgeBench, Tan al. 2024) datasets. Results are shown in Figure and Figure
The results in Figure and Figure 4 demonstrate the large deviation of Binomial distribution to the real distribution. On both datasets, the real distributions LLM ensemble judgments consistently show two peaks centering at the two ends, while Binomial distribution results in single peak with large shift to either of the two peaks. Notably, in Figure 4 the assumption of Binomial distribution leads to an always decreasing majority voting error rate, which is in sharp contrast with the actual error rate that remains at the same level when the ensemble becomes larger.
4.2 Mixture of Beta-Binomial Distributions
Assumption (Mixture of Beta-Binee Binomial Distribuitons)..
S wBB(k, (1) + w:) BB.k 022, B2),
where BB(.,..) is the Reta-Binomiia distribution, is the number of judges in the ensemble, an, B1, 022, f2 are parameters of the two distributions, and w is the mixture weight.
Corollary 1 (Mixture Distribution Error Rate). The error rate of the mixture of Beta-Binomial distributions is
[k/2]
[k/2]-1
k B(s + an, k- 8- B1) B(a1, B1)
k
B(s + 02 k + B2)
PBB = w
Σ
+ (1 - ww)
Σ
B ((az, 32)
(4)

---

where B(.,-) is the Beta function.
After examining the common Binomial distribution assumption in Figure and Figure we notice that the real distribution keeps showing two peaks centering near all wrong and all correct. Motivated by this observation, in Assumption distributions, where one distribution models the LLM ensemble judgments on simple questions and the other one for hard problems. derive all the parameters, we utilize labeled samples from the dataset and design distribution-tailored expectation maximization (EM) algorithm.
I we model the distribution
mixture of two Beta-Binona
4.3 Expectation Maximization
Samples as Distribution Evidence. Given samples, each containing judgments from k: LLMs, S; is the number of correct judgments in the i-th sample and = S./k as the estimated probability of success for the i-th sample.
For the i-th sample, considering the first Beta- Binomial distribution, a responsibility 7f is assigned as (Peta (1)
(5)
w wBetaap: | 31) + Betaea 022 .2)
where Beta(p; la, (8) is the probability density of beta distribution at for the i-th sample under the corresponding Beta component. of represents the probability that the i-th sample belongs the first Beta component, and 7/ ii is the probability for the second component.
Parameters Update. The parameters are updated based on the weighted contributions of samples. The parameters of two distributions j = =11 are updated as
a/
vi St. B3 Σ ri (I: S.), w
(6)
We verify our distribution assumption by first sampling r = 100 judgments made by two models on two datasets and apply our distribution-tailore EM algorithm estimate the parameters. Our method is evaluated in two scenarios: O In Figure we fix the ensemble size k = 11 and compare the estimated distribution against the real distribution and Binomial distribution, and @ in Figure 4 we estimate the eтror rate of majority voting with different ensemble sizes.
In Figure 3 the mixture of Beta-Binomial distributions is significantly closer the real distribution compared to the Binomial, with clear two-peak patterns that are analogous to the observation. In Figure 4 it shows that our distribution is consistently close the real majority voting error rate across all ensemble sizes. Contrary to the Binomial distribution that produced decreasing error rate, our distribution successfully modeled the stable error rate when the ensemble becomes larger. Additionally, the narrow confidence interval demonstrates the high stability of our method.
5 Guide Sampling via Conformal Prediction
In the experiments above, we used fixed number of samples. However, in practical settings where datasets are unannotated and being labeled, it is essential to determine when the number of annotated samples sufficient for accurate estimation. Inspired by conformal prediction (CP), which does not rely on prior knowledge of the dataset distribution and can rigorously estimate the sampling deviation, we propose leveraging its principles to address this challenge.
5.1 Conformal Prediction for Adaptive Stopping
CP provides a principled approach to dynamically evaluate the sampling deviation in the distribution of the number of correct judgments S, which can be used guidance.
Nonconformity Scores. major part of CP is the nonconformity score, which measures how test sample differs from the rest of the data. In our implementation, set the nonconformity score as
score(S.) =|S -E[[)],
5

---

Splits
Embeddings TA Splins
Benbokking Din.
Errer Merpin
0.5-
1..0
Eunbokking Dim.
Lye
Micetaption
0.35
Erree
0.55
Embudding Din.
Embodding Dim.
Sumples
HaluEval Embodding
Margin
TruthfuQQA Embbodding
TruthuuQA Estimation Margin
Figure 5: Examples of distribution prior transfer. Splits from HaluEval form distinct clusters in the embedding space, and transfer does not degrade performance compared to only using target dataset samples. contrast, topics TruthfulQA exhibit closer proximity, where transfer leads significant performance improvements compared to solely using the limited samples of the target dataset.
which quantifies the deviation of each observed value of S from the expected value.
Calibration Data and Quantile Computation. Suppose samples have been used test the LLM ensemble with S1, ...... S, correct judgments, the CP sampling computes the nonconformity scores for all calibration data as s; score(S,) and these scores are sorted in ascending order as For desired estimation confidence 1 the (1 é -quantile with r samples T- is
hhe = (1-() (r1c))))+)))
(8)
Adaptive Stopping Criteria. Adaptive stopping is achieved by monitoring the variation of the conformal prediction quantile. After r samples, the (1 ququnntee recomputed and compared with the one from samples. The sampling process stops when the quantile satisfies
1af - E
where & is a predefined threshold.
Proposition 1 (Sample Amount with Adaptive Stopping). For a given sampling deviation threshold & and a scale т, the sample amount r should satisfy
) Sc.
V
(10)
This proposition offers an estimation of the sample amount under the threshold ç.
Proposition 2 (Error Rate with Adaptive Stopping). Under the sampling threshold &, the majority voting error rate of the mixture distribution becomes
- 1-min
T )) < Padape < (1 +1++m)
min(s, (1nmmnn5
T ))
(11)
This proposition provides a theoretical error bound for estimation under adaptive stopping, suggesting the mild degradation of estimation performance.
We leave the proofs of Proposition and in Appendix B.I and B.2 respectively. our experiments,
we set ç = 0.03, and = 25, which leads to r 2 56.
Text Similarity for Distribution Prior Transfer
To further improve the data efficiency when only a few samples are available and enhance estimation accuracy, we propose to incorporate prior knowledge about the LLM ensemble on other open- source datasets and transfer the estimated judgment distributions to the target dataset. However, one challenge is that the prior transfer could bring performance degradation if the distributions of the source datasets and the target dataset are very different. To resolve this challenge, we design text similarity- based distribution prior transfer, which leverages the strong text embedding capability of the recent models to understand and measure the textual difference among datasets.

---

Text Embedding.
To embed the text inputs of the LLM ensemble, we use NV-Embed-Vi V2 LLee al. 2025]. Given sets of samples {D,, ....... ] from m source datasets, the embedding model (() is utilized to transform the sets of samples to sets of embeddings for the source datasets
(Ei, Eg..... Fm (E(D)). {)m)
(12)
The average embedding F. = Σラ j= Fy of the i-th dataset is used to represent it.
Distribution Prior Transfer. To transfer the distribution from source datasets to the target dataset Do, the process starts by embedding the target dataset Eg = E(D)) and acquiring average embedding Eg. For the dataset D; its transfer weight is
A = log(r)) a (p1 (CosSim (Eo,E)) - p2))
(13)
where (-) is the sigmoid function, 7 is the number of samples and #1 and p2 are hyperparameters. We adopt this design to avoid the degradation of estimation caused by transferring datasets with dissimilar text inputs. This is achieved by setting a threshold and applying the sigmoid function to suppress the weight when the similarity is low. log(r:) is included as datasets with more samples could produce a more accurate estimation and thus should have a higher impact on the transfer. The transfer from the source datasets to the target dataset is performed
E Ay Wi W0 Di=0 A a0s
E A Cze
m
N. Pag i== X
fr
Ar
gtr Po.j
j (1,2).
(14)
Σ d
Oss and Bi, are the j-th parameter the mixture distribution of i-th dataset. The parameters in the weighted sum with index O denote direct estimation the target dataset.
In Equation [44.
Examples. To verify our distribution design, we evaluate the distribution within splits of HaluEval [Li al., 2023a] and TruthfulQA [Lin al., [021] datasets. For HaluEval, we use Dialogue and Summarization splits as source datasets and transfer to QA split; for TruthfulQA, we transfer from topics of Health and Law to Misconceptions. As shown in Figure S] the embeddings form distant clusters in HaluEval, as the text inputs of the three splits have different hallucination detection requirements, and embeddings from TruthfulQA overlap due the similarity of judgment format. When clusters are separated, our method will not bring performance degradation compared to solely using samples from the target dataset, while when clusters are overlapping, our method brings significantly lower estimation error rate margin compared to only using target dataset samples. This supports the effectiveness of our distribution transfer design.
We present the algorithm and Python implementation of BetaConform in Section A and Section
7 Experiments
7.1 Estimation Accuracy
We begin by evaluating BetaConf orm with adaptive stopping on datasets to verify its accuracy. We choose Binomial distribution and a single Beta-Binomias distribution as baselines and compare the error margin, which is the absolute difference between the estimation error rate and the actual value. The results are reported in Table Please see Section for implementation details.
From the results, the following observations can be drawn: Compared to the Binomial distribution, BetaConf orm achieves consistently lower error margin, with 32.4% ~ 54.1% improvements of average error margin of all models. This demonstrates an effective answer to RQ1 by modeling judgment distribution as mixture of Reta-Binomiial distributions. @ The number of samples is close to the theoretical estimation. The average sample amount of models on all datasets exhibit slight deviation of the estimated value 56 by 3.14 12.86 samples. This validates our design of using the distribution-fii -free CP for adaptive stopping, which effectively solved RQ2
7.2 Distribution Prior Transfer
We then verify our text similarity-bate -baed distribution prior transfer when only limited samples are available. We constrain to 10 samples from the target dataset and assume the full source datasets are accessible. Transfer compared with estimating only on the target dataset samples (w/o Transfer). Error margins are shown in Table 2 We also conduct ablation studies of the transfer design in Table
7

---

Table 1: The comparison of error margins between our mixture of Reta--iioomial distributions and Binomial distribution. The Err. Margin and # Samples nasser[RQ] and RQ2, respectively. The error margin is calculated the absolute difference between the actual error rate and the estimation. Estimations using both distributions are done on samples obtained through iterative sampling with adaptive stopping. For each run, the error margin is computed from k- to 11, and the average margin of ensemble sizes is used as the result for that run. We conduct 30 runs and report the average and standard deviation. The average number of samples across runs is also reported.
Mara- 13:30
beardd M-30
GPT4
Medwol
Magis (1)
Saaplo-
Broe Magan 111
Saxepko L1: Errer Magin (1)
Semples (1)
Eise Maggt
Saxplix [2) Erice Magis Saxplo
IaBacieatio Ditection
Beverii Segg 58
17:63 R-F7
15:05 -
16 11:4
5:7 1158
9:1e +
HakEral
14.66-2 f1.6
20.47
5.14 0621
61/12
30.57
5.2 (07)
14.90
5.72 24.54
40.55
Beaxeed
14.00 Du65
19:06:0::0 20:40 40
19.55 2166
14,8 +0
25 0.55
Our
1,89 $9.55
7.18 6:44
6:24 9.59
6.55 0.58
6.79 9936
Nalabaa
Siell 58
11 13 DU6
06.91
16.75 -080
15.78
7.05 +0 14
22 0.99
41.51
8:43 +0 15
42:31
73M 8868
6647
8:50
5,22 8.399
Rearveing
FRMBOCK
Segl 58
16 6 1:35
43,33
00:00
43.90
9:21 60
45.13
4.55 0.75
51.38
21.11 0.331
$4.67
24.64
7.80
9.48
Rinerii
13:28.11 0.78
16.17 -0.00 10
14:33
15 0.7
Steak
à
$1.31
1.
18.15 0:00
4.99 0..56
12.99 8846
1/1 9.59
Bexxsed 10
15.15.00
THAM
0.70
55.99
17.66
9.44
57.78
70077
4 4.30
8-99 866
6.20
6.10-
FLP
Algenee:
JadadBeads
Stagk 38
7,00 20
00.35
7.04 0.56
$8.40
14224
$7.36
20.79
41007
1/62 20.25
46.35
600 0.56
5.30 6..9
7403- 1681
4.55
Bexesed
20 15
3.59 0,22
16.5 1,41
11.2250.11 0.33
0.20
Recosfflencs
Stagle
16.22 38 1.34
-1.22
11,48 - 1.30
45:30
$2.04
02:27
6.99 a15 4D
Bixcria
13.08 à D.96
14,0
15.08
LLMB
90
Oues
68.39 6.71
7.52 645
0 4:353
15:71 00.44
1,56 9.50
4.25
360
3,66 8887
Kii Scoe
16.55
41.23
9.24 13.9
43.75
1.27
19:57
36.93
Okz
899 20:45
6.91 0.99
18 1033
3.58 0.32
5/76 20.06
Beworid
0:7
15:08 0013
0175
15
COMP-Areyysk
Sagl 58
5.56-066
53
4.93
53.38
4A1 +.37
57.18
7:3--000
46.00
13 6.43
$1.82
Bloezid
12.78 1.56
1G75
91.9 11.9
Averag
Sagl 58
12
0000
49.33
0.30 01.6
51.81
5.0 +033
32.96
2 0.92
43.14
10 11 045
46.47
Theoretical Estimation
# Sumples
10
X
Liama Judged Samples
Qwen Judged Samples
# Samples
5-0.5
t-0.0
5-0.01
10'
10
Halafval
TedhfalAA
HallaDial
PEMROOK
Bin-bocc
TRAM Jodgdtoch RowanBienhh
LIMBa ICE Sum
COMP Andyys
0.5
0.4
0.3
0.2
0.1
0.0
Threshold
Figure 6: The actual number of samples un- der various thresholds & versus the theoretical value from Equation (TO). The actual sample numbers match with the theoretical bound.
Figure 7: The actual number of samples from different datasets under three & values Our sampling with adaptive stopping shows consistent results on all the datasets.
From the results, we observe that by transferring from other datasets in the same category (e.g., from TruthfulQA and HalluDial to HaluEval), the average error margin across all datasets is reduced by 5.0% ~ 25.0% and is consistently lower compared to no transfer, suggesting the effectiveness of using prior knowledge of the judgment distributions on open-source datasets can benefit estimation.
7.3 More Research Questions
RQ3: Is sampling with adaptive stopping consistent to the theory? We examine our adaptive stopping to see if Equation (1O) matches the real sampling amount. We set series of & while keeping = 25 and sample with adaptive stopping from judgment samples produced by Llama, Qwen, and GPT-4, and compare with the thcoretical value of Equation (10). The actual sample amounts under different thresholds in F closely with the theoretical estimation, which proves the effectiveness of quantifying sampling deviation through CP and the Proposition
RQ4: Is adaptive stopping really distribution-free" sampling deviation is distribution irrelevance. To testify to this, we consider sampling with various thresholds on all datasets to see if the sample amount remains consistent. The results in Figure| 7
One benefit of adopting CP to quantify
8

---

Table 2: The comparison of error margins with and without distribution prior transfer. Estimations are performed using the mixture of veta-Binomiia distributions, with 10 samples randomly drawn for estimation. In experiments, each dataset is chosen the target dataset, and the left datasets in the same domain are used as source datasets. Bold denotes lower margin. Scores are in percent (%).
Dataset
Method
Qwen-225-72B
IntmmLM-2.5-208
GPT-3.5
GPT-4
Hallocination Detection Datasets
HaluFval
we Transfer
12.43 0.87
12.50 0.92
10.09 0.64
14,07 0
12.85 1 0.83
w/ Transfer
9.19 + 0.75
8.60 0.64
0.71
TruchfulQA
w/o Transfer
15.30 0.81
13.88 0.85
13.17 1.11
02,54
13.21 + 1,03
Transfer
8.55
10.18 0.10
HalluDial
w/o Transfer
17.53 :: 0.81
16.15 1 0.60
11.35 0.83
16.62 a 0.70
14.64 0.85
w/ Transfer
12.89 0.77
13.42 0.53
8.72 0.54
23.79 a: 0.84
18.77 0.92
Reasoning Datasets
PRMS00K
wio Trunsfer
15.02 0.78
12.85 0.88
8.22 a: 0.58
9.27 0.84
9.97 0.53
0.62
10.96
0.60
0.84
BIG-bench
w/o Trunsfer
15.22 =: 0.74
13.81 0.82
9.44 01.3
14.39 + 0.74
13.31 1.15
Transfer
0.69
TRAM
w/o Transfer
w/ Transfer
14.77 a: 0.84
12.52 0.92
12.27 1 0.69
11.03 + 1.04
10.85 0.97
11.67 0.76
13.52 a: 0.81
11.81 1.00
12.69 +: 1.26
11:25 0.57
Alignment Datasets
JudgeBench
Trunsfer
14.05 0.88
12.41 0.66
11.37 0.79
8.23 0.75
12.32 0.69
w/ Transfer
8.19 + 0.66
8.03 0.54
14.36 0.68
15,30 1.19
RewardBench
w/o Trinsfer
12.73 + 0.68
9.47 + 1.07
10.34 0.67
15.17 + 0.92
13,30 0.77
Transfer
16.35
0.38
L.M.Barr
w/o Transfer
16.97 1.10
15.91 + 0.70
10.03 =: 0.88
17.00 a: 0.64
12,90 # 0.97
Transfer
8.03 0.39
9.95 0.30
8.61 0.41
21 11.44+11 0.42
17:20 0.40
Scoeing Darasets
ICE-Score
wfo Trinsfer
14.08 0.53
11.90 1.05
19.59 0.78
12.11 + 0.82
13.98 + 0.88
w/ Transfer
11.32 = 0.66
11.99 0.76
19.25 1.05
10.63 + 0.66
12.30 0.67
COMP-Analysis
w/o Transfer
14.85 1.45
10.83 0.60
10.29 0.60
10.22 + 0,53
16.18 1.00
12.28
10.23
0.82
Average
Average
w/ Transfer
Transfer
11.11 = 0.58
0.86
11.15 0.70
12.91
11.41
10.84 0.60
0.74
13,01
13.62 a 0.68
13.12 0.74
show only slight variance of sampling amounts across datasets, demonstrating superior stability. This verifies that our adaptive stopping is truly distribution-free, and stable on diverse datasets.
RQ5: Is CP-based Adaptive Stopping efficient? To validate the effectiveness of our CP-based adaptive stopping, compare it against variance-based stopping. Specifically, we calculate the variance sampling as
Or Var (sampling) (a, + 3,) + +1)
(15)
where and 3, = rr- Cty are the number of correct and wrong judgments in r samples, respectively, As shown in Table 3 is consistently more effective for adaptive stopping under the same deviation threshold S. which results a reduced number of samples and achieves a reduction of up to 46.3%.
8 Conclusion
We present BetaConf orm, a framework for effi- cient estimation of LLM ensemble judge distri- bution. As part of our framework, we propose mixture of Beta-Binomin the judgment distribution after examining the inaccuracy of the Binomial assumption. We de- sign conformal prediction-basee adaptive stop- ping for sampling, which monitors the sampling deviation and effectively determines the sample amount for estimation. When only limited sam-
Table 3: Comparison of variance- adaptive stopping and ours. We compare the sample amount of both methods under the same threshold. Bold denotes less samples.
distributions model
HaluEval
JudgsBesch
PRMBOOK
Thesshold
Methods
Samples
Samples (1)
Sumples
Sumples
C-0.05
Ours
Variunce
36.57
85.37
36.37
M6.37
36,00
30.47
24.77
31.33
Vanaacce
K3.09
74,43
79.76
$1.47
Ours
$4.72
$3.90
43.33
45.37
S-000
194.72
101.28
ples are available, we incorporate text similarity baased distribution prior transfer mechanism to improve the estimation accuracy. As shown by experiments, the conformal prediction- based adaptive stopping effectively guided the sampling. Our mixture of Beta- Binomial distributions significantly outperforms the common Binomial assumption. With the transfer mechanism, BetaConf orm can achieve high estimation precision with as few as 10 samples from the target dataset.

---

References
Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control, 2023. URL httpttt///aaxxii..rrrro wwtp://///abbb//////e 02814
David Austen-Smite and Jeffrey S Banks. Information aggregation, rationality, and the condorcet jury theorem. American political science review, 90(1):34 1996.
Frank B Baker. The basics of item response theory. ERIC, 2001.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Li Cai, Kilchan Choi, Mark Hansen, and Lauren Harrell. Item response theory. Annual Review of Statistics and Its Application, 3(1):297-321, 2016.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internim2 technical report. arXiv preprint arXiv:2403.17297, 2024.
Nitay Calderon, Roi Reichart, and Rotem Dror. The alternative annotator test for llm-as-a-judge:
How to statistically justify replacing human annotators with Ilms, 2025. URL https://arxiv org/ 2/50/ 2501. 10970
Beiduo Chen, Xinpeng Wang, Siyao Peng, Robert Litschko, Anna Korhonen, and Barbara Plank. seeing the big through the small": Can iims approximate human judgment distributions nli from a few explanations? arXiv preprint arXiv:2406. / 76000, 2024.
Nicolas De Condorcet et al. Essai sur l'application de l'analyse à la probabiliéé des décisions rendues la pluralité des voix. Cambridge University Press, 2014.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprini arXiv:2407.21783 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-comtrolled alpacaeval: A simple way to debias automatic evaluators, 2024. pttttps://arxiv.rrppp.
2404. 04475)
Matteo Fontana, Gianluca Zeni, and Simone Vantini. Conformal prediction: a unified review of theory and new challenges. Bernoulli, 29(1):1-23, 2023.
Robert Harvey and Allen L. Hammer. Item response theory. The Counseling Psychologist, 27(3): 353-383, 1999.
Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking cognitive biases in large language models as evaluators, 2024. URL https
//arxiv. org/ 2abb/2399 17012
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13788 2024.
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Ny-embed: Improved techniques for training lims as generalist embedding models, 2025. httpssttss///arr
org/ abs/ 17428
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback, 2024. URL. https
abs/ 00267
Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian- Yun Nie, and Ji-Rong Wen. Halueval: A large-scale hallucination evaluation benchmark for large language models. arXiv preprint arXiv:2305. | 1147, 2023a.
10

---

Lei Li, Yuancheng Wei, Zhihui Xie, Xuqing Yang, Yifan Song, Peiyi Wang, Chenxin An, Tianyu Liu, Sujian Li, Bill Yuchen Lin, Lingpeng Kong, and Qi Liu. Vlrewardbench: A challenging benchmark for vision-language generative reward models, 2024. URL https:///axii: org/ 0bs/2411.17455
Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. Rain: Your language models can align themselves without finetuning, 2023b. URL hattts://aaiittt/ owwpxii.orrgoooo www./a/.//////o/.eo...//...e..e/ 2b209 07124
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang. Yan Wang, Rui Wang, Yujiu Yang. Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305. 19118, 2023.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprins arXiv:2305.20050. 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human 2021.
falsehoods. arXiv preprint
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 251 1-2222 Singapore, December 2023, Association for Computational Lin- guistics. doi: 10.1866. 122653511/22223.mmmmmmaaall emmlp-mae www..ainn..... 153/
153. URL httph:tt htata:://aallanhhooogyy orw.oogy.orrgggg. org/ 2023
Wen Luo, Tianshu Shen, Wei Li, Guangyue Peng, Richeng Xuan, Houfeng Wang, and Xi Yang. Halludial: A large-scale benchmark for automatic dialogue-level hallucination evaluation. arXiv preprint arXiv:2406.0700 2024.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1 10076- 12100, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18655/v1/2023..mmll main.741. URL http:://scclmmaaooogg.ooo/////// org/ emnlp main 741/
Christopher Mohri and Tatsunori Hashimoto. Language models with conformal factuality guarantees, 2024. URL htRpsttps:// http:s://aaxivvooo www.xii..oog/aba. httgg/abs/24022 10978
Behrad Moniri, Hamed Hassani, and Edgar Dobriban. Evaluating the performance of large language models via debates, 2024. /tt///ttp////rrrv..rrr///// htsp:/bb//240000 11044
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Srrmadett-Shapiro. Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai,, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu. Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lppes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo
11

---

Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning. Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko. Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk. David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh. Sarah Shoker, Pranav Shyam. Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky. Yang Song. Natalie Staudacher, Felipe Petroski Such. Natalic Summers, lly Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss. Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng. Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL https://arxiv. org/ 2303 08774
Arjun Panickssery, Samuel R. Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations, 2024. URL htttss:////: org aba/ab/a////// 2404. 13076
Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. Offsetbias: Leveraging debiased data for tuning evaluators, 2024. URL https: /tass:////xvvvaa org/ htsp//bs/2249// 2407
06551
Jiaxing Qiu, Dongliang Guo, Papini Natalie, Peace Noelle, Levinson Cheri, and Teague R. Henry, Ensemble of large language models for curated labeling and rating of free-text data, 2025. URL https://arxiv. .rgg abs 25a..///500.055 08413
Philipp Schoenegger, Indre Tuminauskaite, Peter S. Park, and Philip E. Tetlock. Wisdom of the
silicon crowd: Llm ensemble prediction capabilities rival human crowd accuracy, 2024. URL
https://arxiv. org/ abs/2402. 19379
Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro. Aditya Gupta, Adrià Garriga-Alonse et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206. 15. 2022.
Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and lon Stoica. Judgebench: A benchmark for evaluating judges. arXiv preprint arXiv:2410.12784. 2024.
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao. Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators, 2023. URL https:tt///// aattps://arxin brg/abs//a arg/abs/23055... 2205. 17926
12

---

Tianlu Wang, Iia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi- Yu, Richard Yuanzhe Pang, Maryam 2024. URI.httper /ttp://axxiiv org/ wtbbs//448.. 02666
Jason Weston, and Xian Li. Self-taught evaluators,
Yuqing Wang and Yun Zhao. Tram: Benchmarking temporal reasoning for large language models. arXiv preprint arXiv:2310.00833 2023.
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with hwwg /407 1
llm-as-a-metta-uudee 2024. htthtttss//axiivv
Yasin Abbasi Yadkori. Ilja Kuzborskij. David Stutz, András György, Adam Fisch, Arnaud Doucet, Iuliya Beloshapka, Wei-Hung Weng. Yao-Yuan Yang, Csaba Szepesvári, Ali Taylan Cemgil, and Nenad Tomasev. Mitigating Ilm hallucinations via conformal abstention, 2024. URL https
//arxiv. . org/ aba 2asss/205.0000 01563
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arX1v:2412.15115, 2024.
Zhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang, Wei Ye, Jindong Wang. Xing Xie, Yue Zhang, and Shikun Zhang. Kieval: A knowledge-groundee interactive evaluation framework for large language models, 2024. URL http::t:p/ss....iprrrrrrrre wbwpg////////aggoe 15043
Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang. et al. R-judge: Benchmarking safety risk awareness for Ilm agents. arXiv preprint arXiv:2401.10019, 2024a.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint (XXi:24401110222 2024b.
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models evaluating instruction following. arXiv preprint arXiv:2310.07641, 2023.
Chen Zhang, Luis Fernando D' D'Haro, Yiming Chen, Malu Zhang, and Haizhou Li. A comprehensive analysis of the effectiveness of large language models as automatic dialogue evaluators. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19515-19524, 2024.
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning, 2025. URL http::ttts:aaattt
org abs 2ttps/2320000001111
Ruochen Zhao, Wenxuan Zhang, Yew Ken Chia, Weiwen Xu, Deli Zhao, and Lidong Bing. Auto- arena: Automating I1m evaluations with agent peer battles and committee discussions, 2024a. URL
https://arxiv.. org/ www.oogggg//////ooooooooo...e/// abs/2405 20267
Yuwei Zhao, Ziyang Luo, Yuchen Tian, Hongzhan Lin, Weixiang Yan, Annan Li, and Jing Ma. Codejudge-eval: Can large language models be good judges in code understanding?, 2024b. URL
https://arxiv.orge org abs/ 2408. 10718
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric Xing, et al. Judging llm-as jsjgge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
Yuxuan Zhou, Margret Keuper, and Mario Fritz. Balancing diversity and risk in 11m sampling: How to select your method and parameter for open-ended text generation. arXiv preprint arXiv:2408. 13586, 2024.
Terry Yue Zhuo Ice-score: Instructing large language models to evaluate code. arXiv preprint arXiv:2304. 14317, 2023.
13

---

A BetaConform
In this section, we introduce BetaConform, a framework de- signed for the efficient estimation of judgment distributions, as illus- trated in Figure 2 and Algorithm[] [ The framework operates in two sce- narios: when only limited samples
Algorithm BetaConform
I: Input: target dataset Do. source datasets Di..... Dmn
judges J = [JI...... ) EM algorithm EM(-)
2: Output: distribution parameters S2 on the target dataset
3: if limited samples in Du then
4: Compute distribution parameters on Do
are available on the target dataset, and when a larger number of sam- ples can be collected. In the former case, BetaConf orm leverages prior distributions from source datasets to enhance estimation. In the latter, it employs adaptive stopping during it- erative sampling to balance sample efficiency and estimation accuracy.
Compute parameters of distributions on Di...., Dm
6: Compute transfer weights by Equation (33
7: Ω 4 Compute transferred parameters by lq. 14)
8: else
9: Initial D < 0. 9 < -00
10: while Equation (g)
is not satisfied do
11:
Add a sample from Do D and update qi-e
12:
end while
13: 52 <- Compute distribution parameters on samples D
(1) When only a small number of samples are available from the tar- get dataset, BetaConform follows these steps: First, it estimates the mixture of Beta-Binomial distributions using the available samples. Next, it incorporates prior knowledge by transferring distributions from source datasets. Specifically, it estimates the distributions calculates transfer weights based on Equation 13
14: end if
15: return ?
the source datasets using all available samples and Finally, the distributions from the source datasets aggregated using Equation 14 to produce enhanced estimation for the target dataset.
(2) When the target dataset contains a large number of unlabeled samples, BetaConf orm employs the following process: ① It uses conformal prediction (CP)-based adaptive stopping strategy to guide the labeling process. @ During iterative sampling, batches of samples are drawn and labeled, while the variation in the nonconformity score is monitored. The sampling process stops when the variation falls below a predefined threshold. @ Once sufficient labeled samples are collected, the mixture of -inomiin distributions is directly estimated using these samples.
orm
B Proofs
B.1
Determination of Sample Amount.
To derive theoretical estimation of the sample amount for the adaptive stopping criteria above, we utilize the fundamental statistical properties of variance reduction with increasing sample size. Specifically, for i.i.d samples, the variance of the quantile decreases as:
Var(gi-)) T. f()
(16)
where f(gi-e) is the density function the quantile. The standard deviation of the estimator, which determines the variability of the quantile estimate, thus decays as:
StdDev
(17)
Vr
By the asymptotic theory of quantile estimation, for a large enough number of samples r, the empirical quantile o- converges to the quantile on the whole dataset q - with a known distribution based on Bahadur's 's representation:
VF - -q1-)) N (0, c(1 c)
(18)
This implies:
1
h- = =1-= +Op
(19)
14

---

where Op(-) denotes the order in probability. Thus, we can determine that the quantile itself decays as:
1 (
---
(20)
This decay behavior shows that as increases, the estimated quantile approaches the theoretical quantile Qi-c, reflecting decreasing sampling deviation by using more samples. We will use this property to derive the relationship between the stopping criteria and the sample size From the stopping criteria in Equation
o. S
(21)
According to the calculations in Equation
20
we can rewrite the bound for g- as
o - -
(--TT)
(22)
Thus we have
1
g9 - q=c - =0p
(23)
VT
Vr
This suggests to meet Equation (). requires
< <:
(24)
which proves Equation (T)).
B.2 Error Rate with Adaptive Sampling
In this section we develop a theoretical estimation the error bound for adaptive sampling. We first consider the base case and as shown in Equation we know that the mixture distribution error rate is:
[k/2]-1 -1 Σ
[k/2] -1 Σ
B(s + 01, h: 8 31)
k B(s + 022 k - 8 + 32) (25)
PBB =
+(1 w)
B(a1, B1)
B(a 32)
$=0
==0
The adaptive stopping criterion is given by Equation 9
g // 5.
(26)
The sample size requirement is given by Equation 100:
S.
(27)
VF
Based on the two equations and large number theory, we know that the difference between the quantile on samples The and the quantile on the whole dataset qı-, decays proportionally vr In addition, the non-conformity score s; is defined in Equation 77]:
8; sC,re(S.) = SS, E[S]],
(28)
where S; is the number of correct judgments in the i-th sample. As the - eu-unnie of the sorted scores <... 8y at stopping time with r samples is:
T-
(((1-c)
(99
When the stopping criterion is met, this implies the confidence region for E[S] has stabilized and the following holds:
P(S E[S] < - = 1
(30)
For the
mixture model, E[S] relates to the error rate via:
PBB P(S < [k22).
(31)
We will use the quantile stability argument follows. For sequence of independent samples (S1,... Sr), let 8; be the non-conformity score defined as:
s score S.S [[S]],
(32)
15

---

where S; is the number of correct judgments in the i-th sample. By the theory of quantile estimation, for a large enough number of samples r, the empirical quantile ai converges to the population quantile qi-c with known distribution:
(oc-e----)--- (1,
<(1 <(1-)) 6)
(33)
[(112))
where /()) is the density function. This implies:
1
T- - Qi-e Op
(34)
where (-(() denotes the order in probability. As the (l-e)-quantile of the sorted scores < < 8r stopping time with samples is:
h- = (1-)))))))
(35)
When the stopping criterion is met, this implies the confidence region for E[S] has stabilized and the following holds:
P(S: E[S] S -
(36)
For the Beta- Seta-Binomial mixture model, E[S] relates to the error rate via:
Pan P(S [k/27).
(37)
By the quantile stability argument above, we have the bound:
T (1 (1- min(,, T )マ ))[STBBB < E[S]adapt < (1 +1+ min(S, -ー ))[STBB
(38)
The error probability PBB is defined using the Reta-Binomial cumulative distribution function:
Pnn P(S [k/2]) Fnn - 1),
(39)
where FBB the Beta-Binomin cumulative distribution function. Since FBB is monotonically increasing, the error probability Padapt follows the same proportional bound.
(1 - Liminn fa Padaps (1 + min(E, 3
(40)
Therefore, we have:
Padapt (1 + min(E, - 1)
(41)
C Implementation Details
In this section, we elaborate on the implementation details of BetaConform.
We evaluate LLM ensembles with A: 1, 3, 5, 7,9, 9, models, including GPT-3.5 (Brown al. 2020), GPT-4 [OpenAl al. 2024]. (luma-3.3-705 (Dubey al. 2024], Qwen-2.5-72B Yang et al. 2024]. and IntemL --5- CCai et al., 2024). The experiments cover four domains: hallucination
detection (HaluEval Li al. 2023a, TruthfulQA Lin et al., 2021 HalluDial Luo et al. 2024). reasoning (PRM800K Lightman et al.
2023
BIG-bench Srivastava et al. 2022 TRAM Wang and Comp-Analysis Zhang al. 20024), and alignment
Zhao 2023 scoring (ICE-Scome Zhuo
2023
(JudgeBench Tan et al. 2024 RewardBench Lambert et al.) 2024 LLMBar Zeng et al. 2023).
For all experiments, the sampling temperature of LLMs is set to 1, and the random seeds are not fixed. The randomness comes from the Top P sampling of token generation. Each experiment repeated 30 times to compute the mean and standard deviation of the error margin. The adaptive stopping threshold is set to e = 0.03 and т = 25, requiring at least r 2 56 samples to meet the stopping criteria.
D Additional Experiments
In Table we conduct ablation studies on our distribution transfer. Compared to ablated variants, our full design achieves the smallest error margin, indicating the effectiveness of our transfer design.
16

---

Table 4: The ablation study of BetaConform distribution prior transfer. log(r:) rs means the first term log(r;) in Eq. 14 is replaced with ri to still asign a larger dataset higher weight while not considering source datasets could be magnitudes larger. @ CosSim( Eo: E.) - Eo-E, Edl refers to replacing the cosine similarity to measure the source datasets and the target dataset with the reciprocal of the Euclidean distance between the embeddings of the two datasets. This still assigns more similar datasets higher weights. No a--) means the transfer weight is computed as A log(r:) CosSim Eu, E.). without using the sigmoid function o(-) to reduce the weight of low similarity datasets
Llama-3.3-708
Qwen-2.5-72B
IntemLM-208
Dataset
Ablation
Emor Margin
Eroor Margin
Eroo Margin
log(n;) + 73
10.94 0.57
9.53 0.70
0.75
CosSimi Ra, Pa)
10.29
HaluEval
No
10.04 = 0.23
23.03 = 0.12
8.45 0.10
Ours
8.82 =: 0.42
9.19 0.75
8.60 0.64
lug(r;) 73
0.66
10.65
CusSimn Eg,, E.) Bia
15.13 0.71
13.14 0.96
11.03 0.80
TruthfulQA
No.
6.87 +: 0.01
16.52 = 0.03
12.47 = 0.06
Ours
3.37
8.55 = 0.07
log(n,) 71
13.55 0.58
15.43 0.86
10.42
CosSim E.,. E.) No
15.54 0.59
15.89 0.65
10.47 0.67]
HalloDial
12.39 0.00
16.61 = 0.09
13.00 = 0.07
log(n,)
25.97 0.03
21.23 0.04
15.46 0.06
CosSim/ Ea, B.) 1.
14.43 1.12
11.26 0.99
11.47 0.74
JudgeBench
(o))
24.57 0.44
19.26 = 0.13
10.42 = 0.08
Ours
9.45
8.19 0.66
8.03 0.54
log(rs)
15.00 0.01
17.33 = 0.02
20.32 = 0.01
CosSim/ Ea, Ba) M86
13.29 0.87
14.48 0.45
16.75 0.34
RewardBench
(.)
12.88 0.59
16.45 0.26
Ours
12.72 0..30
12.84 s: 00.48
16.35 0.36
log(r;)
13.88 0.01
15.88 = 0.01
15.45 = 0.01
CosSim! En, E.) (Ap
16.27 0.81
15.55 = 0.83
11.90 1.07
LLMBar
No))))
12.58
Ours
8.03 0.39
9.95 0.30
8.61 0.41
E
Python Implementation
Below provide the Python-style code for the implementation of methods
Listing 1: Adaptive Conformal Sampling
import math
import random
import numpy as np
Helper Function for Conformal Sampling .ww.conformity.....momm.oom..... acore_abs_dee difffdeeeeeeeeeeeeeee mean (value, mean_ value) """Caleulates LI distance between value the mean as nonconformity measure.
def
return abs (value - mean_value)
Core Function 2: Adaptive Conformal Sampling
def run
full dull_ddaaset_iteen: vaalue_num_modelss num_samples_pen batch,
17

---

max_batches,
peeilon_conformalle
convergence ott. check-5 check-
min bin_baacches_beiie stetore_stopping.coee
Performs adaptive sampling for fixed k-value (number of models) using conformal prediction.
Samples are drawn in batches until the width of the conformal interval (related to g-value) stabilizes.
Args:
full, dataset, items (list of lists): Each inner list contains binary outcomes for a data point across all available models.
(int): Number of models/outcomes to consider from the start of each ätem.
samplesses.p.r bet.bch batch (int): Number of items to sample per batch.
man batches (int): Nazimum number of batches to draw. epstlon, conformal (float): Significance level for conformal prediction (e.g., 0.05 for 95% interval).
convergence, aareshold.g.diti (float): Threshold for q-value change to determine stopping. Snn_batches.befor beforeestoppingeeeeeeeeeeeeeeepp before checking q q-value convergence.
(int); Hinimum batches
Returns:
tuple: (collected, (collected_success.. success_counts. final_q.valas
num num_ bat.het.chepp... proccessdd sndpled,indices. indices, overalll) succedddssccssccccccusss
List of success counts for all sampled items.
fenal_ q. value: q-value from conformal prediction at stopping maz batches.
num_ www.batches_proccssed Actual number of batches processed.
campled, indices overall: List of original indices of the samples led items.
not Hall_dataset_items: return () None, 0, 0
if not (0 < k_value_nuu_modell <= [em [0])): raise ValueError Invalid _.value_nu__oddll1
k_value_ "aalue_num_models).
all Sol._oolll____eeeee vvalues [ # Stores $.4 sum (item
v_value_ num models]) for calibration set
q q-previous None
final.q.value None
fndexed_full_datare = list (enumerate (full_das (aall_datast__temms)
evailable_indice_ffor. fom_soomm_iinggggeeaaappppp_gete list (range (len
indexed_full,aataset smppedd.iniices.vvera []
for batch_idz in range (aaxxeaaxaaaaaaaaaa
if len wotearaiaaaleeiiieeee.ooooeaaaaa
nun_samples_per._mm
if not how.laale_iddice_foooo...o
break No more
samp les available
If remaining samples less than batch, sample all remaining actual actual_samples caalles_tttt_bbee this_batch batch = len
iiaallaleiindices. inticcesfoo_ssppliing.
18

---

else:
actual_sanples.thieee
nuw_samples_per_ooe batch
Sample indices for the current batch without replacement from available indices
chosen_pool_ indices random sample
available_indices_ for.amalin....m.igg...... aampling, actual satual_
purrent_batch_ittan [] ourrent.batch..riggnna.. indicess
temp_available_inde the nent round
To update available indices for
Build set for quick removal chosen indices loceen_poo__indiesseeoe for pool_idx
if pool_ids idx in original_data_1dx: item fudxed__flll pool_idx]
ourrent_batch_items. append (item)
original_datajirr)
else:
temp comp_availableenndcce... append append
samplede oag.aaaceeercllltttttttaaoaaaarr indicess)))
Gaailable_indics_ffoocoommm
temp_available
for item in current.batch.itee..
8 v_value sum wtw.::::aaaaaeee__moeeea ell_collected_S.allee app.ees.edeeeee.ppdddddee ap
if not all col continue
s s_mean [p..emalll..ollleeeeee nonconformity
[_nomnooformmttty.ccrre..bbb diff (o
mean (s,
s_mean) for a in ail_collected__ anconformity__ccores hctp:racrrrsssoeo_
sorted (ontednnonnnfformiity...o scoress
r_calib_size len tatooccntooiityyccrrroorrdlllll
quantile_idx = int (math cat.atccccc((_ccaib..iiiiia
1) (1
-
quantile_idx min (max (quantile_ixx, 0), r_calib_size 1)
Ensure indeæ valid
current_q_value sanconormitty.coree.ccom [quantile_ixx] final.q.value current_q.value
batch_ids min bih.bacchee..bofooessstpppp check -1 batch_ide is 0-Sndezed if q-previous is not None:
if abs (currentt__vvaauu q.previous) boverreene_.hrreeolld.... return all.colleeted...vaaeeee
final_q.valus value,
batch_idz 1, ianpleed_idicees_oveee
g-previous current_q_valie
elif batch_idx q-previous current.q.value
Set q- previoss for the first iteration
return all Seoollected___valuee sampled
max_batches,
Listing 2: Mixture of Beta Distributions Fitting via EM
19

---

import numpy as up
Helper Function for Distribution Transfer def vornalliee (v):
"""L2 normalizes a vector.
norm np. up.linalg norm(v)
return
if
0 else
Core Function 3: Distribution Transfer for Beta Nizture
Parameters
def transfer.borrauuuueeeereaaaaae
target_direct_params
source.parres__list,
target.mean.omecdingg
sorrce_nea.eebbeddnggg......
target_data_siize,
source.data.sizee_list,,
ambodinggssallaartttyrbeebbbbb.e
Amalaartyy_ccanng_ffcco-0000000
ala.souurce.veegtt.fccccrr......
): ~.~
Transfers/adjuse Beta miature parameters from source domains to target domain
based embedding similarity and data size.
á Args:
target weddddie_paareee params (tuple): (al.t, bl_t, a d2.t, wl.t) Directly estimated parameters for the target domain. source.params. list (list of tuples): Each tuple contains parameters for a source domain. marget_mean, embedding array): Hean embedding vector for the target domain.
source_mean, embeddings_ list (list of up array): List of embedding vectors for source domains.
target data_ (ent); Number of samples in the target domain
source, data sizes, list (lest of int): List data sizes for source domains.
threshold (float): Threshold for cosine
simi similarity.
(float): Sealing factor for the man son_ weighrrre.w.itt..ccco (float): Hinimum source weight factor ensuring non con-negativity.
similarity score.
Returns:
tuple: Transferred parameters (al.f, b1.f, a2_s, b2.f, w1_f).
if not source_params__istt list:: # No source, return target 's own parameters
darget_direct_pare
len wawuurce_mea_emmeddieeee not (len (sourcee.premm__iiat)
) == len raise ValueError ("Lengthss of source parameters, embeddings, and size lists must match.")
norm norm_targen targetrrge_eeeee emb pormalize_vectonn (np. asarray target_mean. embedding, dtype-float))
veight_target float (target_data, source_final_
for i in range (len (source_params.
20

---

sorn_source_emb__ = www.mmalizee.ectooooe (np asarray( source_ _eae___ea__eeeeeeeeeeeeeeeeeeeee www.eeeeeness.iiidddeeee.oe.eeee dtype=float)) similarity np dot (norm_targget_ent norn_source_emb_bt
# Calculate similarity based weight factor, ensuring non Regativity
factor simllarity.ccalln.feee.. factor similarity bopeddmn_iiiilariityttheeeee a ammlaaritybased_aaccee max wowcenoorreeeeeghheeeeeeee
current_ sount__soureeeeigille source_data_sizos_lia
source coce_final__o0 furre__iinal_weihheee www.wts.ggt...... httpee/iii..eeettt.pppapaarrrrre
total_
weight_targe sum fiaa
if total_combingg cotall <= 1e-9: # If total weight is too small, return target 's own parometers
return targe_drret_paras
num cum_params_to_ transfer len beaget_ddreet.paraan final_ final_ hwwaalarraaeeerdppaaeeeeeo
[0.0] num_params, to transfer
Contribution from target parameters
i range (tw.__paramm_tttttrattttt
timal 1ist [i] += veight_target
farget_direct_parame
Contrabution from source parameters for i, src.params_ in (saurceea.o
if len(src.params_tupll
I= wtt.parrmstt_trrasttet
raise ValueError Source parameter tuple {i} length
mismatch with target parameters.")
for j in range ton__pppaaarramsttotttr
final_ ten.ittreeffrrr..rramm.... -istraammmmsssssip [j] +- fource_.iinal_weghhee
gec.params_tuplelle
final, ainaalparams_.. values [p total, total_ cot.:/combiedddeeeeeeee for p in final wwwttttaeeeeeeeerrrrssss..
# Post Posteprocess parameters: ensure alpha, beta positive, and wi is in [0,1]
Assuming the order is (al, b1, a2, 02, w1)
al_f, b1_f, a2_f, b2_f, w1_f final params. values
al_f max (al_f, 1e-6)
b1 f max cai_biff... 1e-6)
a2_f max (a2_f, 1e-6)
b2_f max (b2_f, 1e-6)
wi_f mp. clip (wl_f, 1e-6, 1.0 1e-6)
return (al_f, bi_f, a2_f, b2_f, wl_f)
Listing 3: Distribution Transfer for Beta Mixture Parameters
import math import random
import numpy as from scipy.state stats import beta from scipy. special import betaln, gammaln 1gamma g gammaln scipy is log gamma
from math import comb math. comb for combinations
Helper Functions for Beta Minture and Beta Binomial
21

---

def
bot_pet__ppddfffeeeeeeeeee (probabilities):
Replaces Os and is a list of probabilaties with close values to avosd issues with beta. pdf calculations.
return 999999 if 1.0 else 0.000001 if <- 0.0 else for in probabilities]
def Denamiiiommal. log (k trials, nun_successes, alpha, beta.param)
aan
Calculates the log of the Beta Binomial PMF: log dewum_successe ))
where I (B trials, alpha, beta.param). P(I==) a) Beta(alphatz, betatk-a) Balpha deta)
not num_successes k_trials): return -np. inf Log probability of zero
# Ensure alpha and beta_param alpha_stable max (alpha, 19-9) beta_stable max (beta_ 1e-9)
posstive
1og_c_k_x = 1gamma tkatai 1) (1gamma (num_successes 1)
1gamma (k-trials suceeuse 1))
Log_beta_nt num betaln (alahaaaalaasttabla num_successes, beta_stable
k_ k_trials num_ sun_sucessss)
log_beta_dde den betaln(alpha_statt beta_stable)
return 1og_C_k_x log_beta_num log_ bog_beta_den
def histure_bbtaa heteeebbbtaabinniallaaaaam pmf (num_successes, alphal, betal, alpha2,
beta2, w1, k_trials): ann
PMF of the mizture Beta Binomial model:
P. mim (I=m) = wf (B(k, alphal, betal) + (1-w1) BB alpha2, betal)
log-p1 bet.eeatbbnnn b (k_triast, nun_successes, alphal, betal)
1og-p2 beta2)
(k trials, num_successes, alpha2,
p1 np. exp (log-p1)
p2 mp. exp (log_p2)
return p1 (1 w1) p2
Core Function 1: Miature of Beta Distributions Fitting via EN
def feiiiturr__o_betaae_
raw raw_samples_outen
outcomes,
num_trials_per__ee
max_iters=100,
tol=1e-6.
alphal.init--oone, betal_init=Mone,
alphaz_init=Mone beta2_init=None vl_init-None
):
Fits mizture of Beta diatributions using the EN algorithm. This model is used for modeling observed success rates P. successes for sample i) / num, perallsper.ssaneeeee sample.
22

---

Args:
raw_samples les. cestcoplessuuccomss (list of lists): Bach inner list contains binary outcomes or for data point. tum_trials.per
(int): Number of trials/outcomes to consider from the start of each inner list (K or m). maz_iters (int): Hazimum number iterations for the BM algorithm
fol (float): Tolerance for convergence. alphal_ betal, alpha2, init, beta2, init, wI wiinni: Optional initial parameters.
Returns:
tuple: (alphal, betal, alpha2, betal, w1) The estimated parameters.
data.points len (rwwwaaaalees_uutoooo num_ cun_data_pointt
raise ValueError (" I'Iput raw rav_ sav_samlls__oue outcomes cannot be empty."
)
if num www.:rraass.persanee <" 0: raise ValueError
must be positive.")
# Initialize parameters (heuristic based on original code)
alphal alphal_init if alphal_init is not None else 10 *
mun_trrials_eer_sapll
betal betal_init if betai_init is not None else
num_trialss_eer_ampll
al alpha2 alpha2_init if alpha2_init is not None else mum_triala_per_aampe
beta2 beta2_init if beta2_init is not None
10
num_triale_per_sapp
wi wi_init if wi_init is not None else 0.5
alphal, betal max 1e-6), max (betal, 1e-6) alpha2, beta2 max max (alpha2, 1e-6), (beta2, 1e-6) w1 clip 1e-6, 1.0 - 1e-6)
up array [[um (sample num_ www.tialss.perreeeeeeeeem _ampll for sample in www.saaples_oouccomess proportions observed_successes / num cooportios__for_pee
lbserved_successt
np. array eet formmmeetssooooreeoooeeoooooeee bort proportions tolist
for iteration in range(max,iters):
# Steppp Calculate responsibilitiie pdf pdf_valsi beta. pdf (proportions_for for_pdff alphal 1e-9, betal (pta..dd(propottioonss
1e-9) $ Add small epsilon for stability
pdf_vals2 beta, pdf (proportione_for.pppr for_pdf, alpha2 1e-9, beta2 1e-9)
numeratori = w1 pdf oalsv numerator2 (1 s1) * pdf_vals2
denominator numerator1 numerator2
denominator [denominator < 1e-9] 1e-9 Avoid division by zero
respi numeratori denominator reap2 numerator2 denominator
# # Stepp: Update parameters (using weighted method of moments for Beta parameters)
wi_new np (resp1) w1_new mp. clip (el_new, 1e-6, 1.0 1e-6)
# Update alpha, beta for component 1
23

---

sum_respi mp. sp.sum (resp1) if sum resp1 1e-6:
alphai_nen new, betal_new alphal, betal Keep old if weight is too small
else:
mean_pl_v np. (resp * proportions) / sum_respl var-pi_w up sumsum (reap1 ((proportions mean_pl_w) *22)
/ sum_resp
mean_pi_v np (m., 1e-6, 1.0 1e-6)
if var-pi_s <= 1e-9 or var var-pi_w >= mean_pi_w (1.0
mean_pl_w) (1-1e-6): # Check if veriance is valid Invalid or too small varsance, use heuristic (e.g., high confidence)
al alphal_nee mean_pl_v (num_trials_ppr sample 10)
# Larger concentration betal_new (1.0 mean_pi_x) (nun_trials_perr sample
10)
else:
common_ (mean_pl_w * (1.0 mean_pl_w) var var_pi_w) 1.0
alphal_nes mean_pl_v common_
betal_new (1.0 mean_pl_w) common_factor
Update alpha, beta for component 2 sum_resp2 nu. (resp2) if sum_resp2 1e-6: alpha2_new, beta2_nes new alpha2, beta2
else:
mean_p2_v sum (resp2 * proportions) / sum_resp2
var var_p2_w np.sum (resp2 ((proportions mean.p2_w) .*2))
/ sum_ sum_resp2
mean_p2_v np. (2ea,_p,._22_ 1e-6, 1.0 1e-6) if var-p2_w <= 1e-9 var-p2_w >= mean_p2_w (1.0
mean_p2_w) (1-1e-6);
alpha2_nex mean_p2_v
beta2_new new (1.0 mean_p2_w)
(num ttun_trials.per_eee pampl sample
10)
else:
common common_facton factor2 (mean_p2__ (1.0 mean_p2_w) / var_p2_w) 1.0
alpha2_new mean_p2_r common factor2
beta2_new = (1.0 mean_p2_w) common_factor?
alphal_new, betal_new (alphhai__eee 1e 1e-6), (betal_new, 1e-6)
a alpha2_nex. new, beta2_new max 1e 6), max (beta2_nev, 1e-6)
Check for convergence baffm_diff (abs alphai_new) abs (betai betal_nes )
(alpha2 alpha2_new) abs (beta2 beta2_new )
abs (w1 wi_new))
if param param_diff diff < tol:
alphal, , betal, alpha2, beta2, w1 alphai_new, betai_new, alpha2_nex, beta2_new, vi_new break
alphal, betal, alpha2, beta2, w1 alphai_new, betal_new, new, beta2_new, vi_new
return alphal, betal, alpha2, beta2, w1
--- Utility Function: Calculate Majority Vote Success Probability from Hizture

---

def oalculee..abootyyy......eee...p. s_trials_for_voll
alphal, betal, alpha2, beta2, seniitture_wiihht
Calculates the probability of achieving majority success given Pata-Binomian miature parameters.
#ajority success is defined as number of successes > ceel( k k.trials_for_vo / 2).
Args:
k. torraals_for.. vote (int): Total number of trials g.. number LLNs).
alphal betal: Parometers for the first Beta BetomBinomial component.
alpha2, beta2: Parameters for the second Binomiil component.
wi_mizture_ weight (float); Hizture weight for the first component.
Returns:
float: Probability of majority vote success.
k_trials_for_vote 0: return 0.0
majority.threshold math. ceil (ktrraassssrsssrrrooooo / 2.0)
prob_sum_for.majrrle 0.0
num_successes range (int (majority..hhreehood) vot 1):
+= .wn.ture__btta.bnnomaaa..
num num_successes, alphal, betal, alpha2, beta2,
wi_mixture_weight f_trials_for_w
return grob_sum_for fot.orrrmaaattyy....
Limitations and Future Work
The two-component Beta-Binomial mixture improves simpler models but may still underfit complex judgment distributions. Prior transfer depends on text embedding quality and assumes textual similarity implies similar judgments- design also focuses on binary/scoring tasks and requires an odd number of annotators.
assumption that may not always hold. The current
Future work could explore more flexible mixture models, robust prior transfer methods beyond textual similarity, task specific features, and extensions to diverse judgment formats and ensemble sizes.
G Broader Impacts
BetaConf orm reduce the cost of LLM ensemble evaluations, supporting broader in QA, benchmarking, annotation, and MLOps. It enables scalable, reliable assessment but requires careful attention to estimation error and modeling assumptions, especially in high-stakes applications.
25

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Our method sections and the experiment section matches the decription of our method in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
26

---

In the Section F we discuss about the limitations and future work of BetaConform.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-ten system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: For Proposition I aad Proposition] 2 we provide the assumption and proof in
Section B. I3..andd and Section B.2
Guidelines:
The answer NA means that the paper does not include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Section we provide the implementation detail of our method and experi- ments. In Section A provide the detailed description of our method.
27

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We do not release the code. All the datasets used in this paper are open-source datasets and can be found online with their names.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctiips.cc/puulic/guui
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
28

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our work do not involve training. and directly the designed valida- tion/test splits of each dataset. The hyperparameters of experiments are described in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide the mean and standar deviation of our experimental results. The setting described in Section E
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the eтror bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [NA]
Justification: The judging process only require inference of L.MMs. The distribution estima- tion solely uses CPU.
29

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The anonymity is preserved and follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Section discuss the broader impact of our method.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
30

---

Answer: [NA]
Justification:
Our paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We make proper citation of each dataset used our paper, and we follow the license of each dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should citep the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the bataasess.ppprsstttt paperswi theode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
31

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
hatta///eeeuuppssccc//ooeeeerree for what should or should not be described.
Please refer to our LLM
32


Reviews:
[{"review_id": "YPtZNgMUbV", "reviewer": "Reviewer_NToF", "summary": "The paper proposes a method to efficiently and accurately estimate the judgment performance of LLM ensembles. The method proposed determines: (RQ1): how to estimate the judgment distribution of LLM ensembles and (RQ2): how many samples are needed to achieve a desired estimation under a given error margin threshold. To achieve more accurate estimation, the paper proposes to use a mixture of Beta-Binomial distributions to model the different judgment for \"easy\" and \"hard\" questions and use expectation maximization (EM) to estimate the parameters of the distribution from the samples. To achieve the accurate estimation more efficiently, the paper determines the number of samples to use for a good estimation using conformal prediction to compute when to stop sampling when a pre-defined threshold is reached. This limits the number of samples to take while maintaining estimation precision. Finally, to efficiently estimate when only a few samples are available, the paper proposes a text similarity-based method to transfer from the distributions of similar datasets to the target dataset. The paper then conducts experiments across various datasets and shows that the proposed method achieve high estimation precision and achieve low error margin with as few as 10 samples.", "questions": "- Why the choice of 2 distributions in the mixture model? Why not 3, 4, or other values? Were other values explored?\n- How were the parameter estimates for prior dataset distributions combined with the parameter estimates of the target data? Were the prior parameters used as initial parameters to be further refined through EM? Or were other methods of combining used?\n- How will the proposed model change when ensembles with different underlying models are used? Was this explored?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- the paper proposes an original method to efficiently and accurately estimate judgment distribution of LLM ensembles, which combines the choice of distribution (Beta-Binomial) and EM, adaptive stopping (using conformal prediction), and text similarity-based transfer from distributions of existing datasets. \n- the problem the paper is tackling is well-motivated, since there is a need to estimate performance of LLM ensembles with few samples. The proposed method can estimate the performance of LLM ensembles with lower human annotation costs (both in terms of time and financial costs). \n- while the paper starts from theories in statistics, the proposed method is shown to perform well empirically on multiple datasets. \n- the paper is written really well and is very easy to read. The code is given to ensure reproducibility. \n\nWeaknesses:\n- while the choice of 2 beta-binomial mixture model is shown to be effective empirically, it is unclear why the authors choose 2 distributions instead of 3, 4, 5, and so on. The reasoning that it is because questions given to LLMs can be categorized into \"easy\" vs. \"hard\" seems rather coarse-grained and heuristic. A deeper exploration of this choice will make the paper stronger. \n- it will be good to add more intuitive explanations on statistical theories such as the conformal prediction, for readers who are not familiar with the mathematical reasoning behind these theories. \n- it is unclear how the distribution transfer was done. How were the parameter estimates for prior dataset distributions combined with the parameter estimates of the target data? Were the prior parameters used as initial parameters to be further refined through EM? Or were other methods of combining used?\n- finally, the proposed method seems to be limited in that the LLMs in the ensemble have to all be the same model: same architecture, same size, same version with variations only in terms of the top-P token sampling. This seems to limit the generalizability of the method to real world LLM ensembles that may be composed of different LLM models (e.g., Llama with Qwen).", "was_revised": false, "final_justification": "", "rebuttal": "Thank you, **Reviewer NToF**, for the detailed review and acknowledging our paper as original, well-motivated, theory-rooted, empirically strong, and clearly written. Below, we address your questions point by point:\n\n**Q1: The choice of two distributions**\n\n**A1:** We observed that the judge distribution consistently shows **two peaks near all-wrong and all-correct** responses, as illustrated in **Figure 3**. Motivated by this, we adopted a mixture of two Beta-Binomial distributions to **capture this pattern**. Furthermore, our experimental results (**Table 1**) demonstrate that our chosen mixture distribution **outperforms using a single Beta-Binomial distribution**.\n\nTo further validate this design choice, we compared our two-distribution mixture with mixtures of three, four, and five distributions, following the setup in **Table 1** on judgments from Llama-3.3-70B:\n\n|            | 2 Distributions (Ours) | 3 Distributions | 4 Distributions | 5 Distributions |\n| ---------- | :--------------------: | :-------------: | :-------------: | :-------------: |\n| HaluEval   |      6.68 ± 0.53       |   6.43 ± 0.64   |   7.35 ± 0.51   |   6.73 ± 0.45   |\n| JudgeBench |      6.98 ± 0.56       |   6.31 ± 0.41   |   6.82 ± 0.43   |   6.98 ± 0.47   |\n| BIG-bench  |      11.15 ± 0.60      |  12.51 ± 0.53   |  12.35 ± 0.68   |  12.70 ± 0.54   |\n\nThe results show that adding more distributions provides **minimal or no performance gain** compared to our two-distribution choice, justifying our decision.\n\n**Q2: Clarification of mathematical concepts**\n\n**A2:** Thank you for this valuable suggestion. We will add further explanations in the revised manuscript. Brief, intuitive explanations for key concepts are provided here:\n\n- **Beta-Binomial Distribution and Mixture**: Imagine flipping a coin whose fairness you're unsure about. The Binomial distribution describes how many heads you expect if you know the coin’s bias (probability of heads). The Beta distribution captures your uncertainty about this bias. Combining these, the Beta-Binomial distribution models both the number of heads and your uncertainty about the coin’s fairness. We observed that some questions are \"easy\" and others are \"hard\" for LLM judges; thus, we use a **mixture** of two Beta-Binomial distributions to model both question types simultaneously.\n- **Conformal Prediction**: Conformal prediction generates prediction intervals with guaranteed confidence without making strong assumptions about data distributions. It helps determine \"how many samples are enough?\" by continuously sampling until the prediction interval is acceptably narrow, ensuring efficiency.\n- **Expectation-Maximization (EM)**: EM is an iterative algorithm used to estimate model parameters. It alternates between two steps: the Expectation (E-step), where it estimates missing information based on current parameters, and the Maximization (M-step), where it updates parameters to better fit the data. This process repeats until convergence.\n\n**Q3: Procedure for distribution transfer**\n\n**A3:** We apologize for any confusion. The distribution transfer involves the following steps:\n\n1. **Estimate distributions** on both target and source datasets using EM, initially **without transfer**.\n2. **Compute text embeddings** for the target and source datasets (**Lines 230-233**, **Equation 12**).\n3. **Transfer distributions** from source datasets to the target dataset by refining initial estimates using information from the source datasets (**Lines 234-244**, **Equations 13 and 14**). Specifically, we perform a weighted sum of source and target distributions as described in **Equation 14**.\n\n**Q4: Ensemble of different models**\n\n**A4:** Currently, our method only considers ensembles composed of the same model, aligning with common practice in LLM judging. We plan to explore ensembles of different models as future work."}, {"review_id": "2TXUAiq6HA", "reviewer": "Reviewer_vhVT", "summary": "The paper presents a principled maximum a posteriori (MAP) framework for an economical and precise estimation of the performance of LLM ensemble judgment. It replaces the Binomial assumption with a mixture of Beta-Binomial distributions to capture bimodal judgment patterns, introduces a conformal prediction-driven approach to enable adaptive stopping during iterative sampling, and uses text-similarity-based transfer of learned distributions from source datasets to improve target dataset estimation with few labels. \n\nThe BetaConform framework delivers a theoretically guaranteed estimation of LLM ensemble judgment with minimum labeled samples, and also validated empirically.", "questions": "Apart from the weaknesses above, I would like to see a discussion on the overall computational overhead, as the cost of text embedding, EM algorithm and conformal prediction may bring extra computational overhead while it effectively reduces annotations.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The approach is able to dramatically cut down annotation costs.\n- The approach is theoretically rigorous in that it is formally grounded in MAP estimation, conformal prediction, and Bayesian transfer.\n- The framework is broadly validated on 11 diverse datasets and five SoTA LLMs.\n\nWeaknesses:\n- The mixture model only assumes two components (i.e. easy/hard tasks), which may not well generalize to more components that reflect more complicated situations.\n\n- The distribution prior transfer is highly reliant on the quality of text embeddings, which may be a problem when the target dataset has only a few samples.", "was_revised": true, "final_justification": "The paper presents a principled maximum a posteriori (MAP) framework for an economical and precise estimation of the performance of LLM ensemble judgment.  I think the paper may have high practical impacts to the community and therefore recommend accepting the paper.", "rebuttal": "Thank you, **Reviewer vhVT**, for your detailed review and for recognizing our paper as precise, highly efficient, theoretically rigorous, and broadly validated. Below, we address your questions point by point:\n\n**Q1: Two-component mixture distribution**\n\n**A1:** While our initial design of the two-component mixture distribution was based on observations presented in **Figure 3**, we **extensively validated our method across 5 models and 11 datasets**. The results consistently demonstrate that our approach outperforms baseline methods, highlighting its **generalization capability**.\n\n**Q2: Reliance on the quality of text embedding**\n\n**A2:** We argue that, in practice, **ample unlabeled data for computing embeddings is typically available**. Our method significantly reduces the annotation effort required from humans to achieve accurate estimations, making it capable of leveraging the unlabeled data. Moreover, in scenarios where only a few dataset samples are accessible, **our method can directly perform estimations without transfer**, similar to the setting of **Table 1**.\n\n**Q3: Computational overhead**\n\n**A3:** Our method incurs **minimal computational overhead**. The conformal prediction and EM algorithm are executed **solely on a CPU**, typically **completing within three minutes per dataset**. Additionally, adaptive sampling using conformal prediction allows for efficient estimation **using only a small fraction** (approximately 200 samples) of the entire dataset. Furthermore, generating text embeddings relies on efficient model inference, further minimizing computational costs."}, {"review_id": "fQnktZENrg", "reviewer": "Reviewer_Qt2U", "summary": "The authors propose a problem of estimating the error rate distribution of LLM ensembles. They provide several key points: (1) the distribution of the correct votes can be better modeled by a two-level mixture of Beta-Binomial distribution, compared to the Binomial distribution model; (2) the number of samples can be chosen via the conformal prediction's square-root bounds; and (3) the distribution parameters can be transferred from other source datasets based on text similarities. The algorithm combines the above three points and is verified by empirical evidence.", "questions": "1. Line 33's goal should be carefully revised. All the guarantees are made for the population-level rather than arbitrary $x$.\n\n2. About the mixture of the Beta-Binomial model: (1) How to rule out the possibility of the Binomial distribution. Lines 42 and 43 are confusing. Condorcet's Jury Theorem is only correct if (a) each judge has an equal chance of $p>0.5$ making the correct choice; and (b) each judge is making the decision independently. Both conditions are hardly met in the LLM ensemble scenario, especially the second one (I think the LLMs' decisions are highly correlated). So, the Binomial assumption alone doesn't imply that improving the judge number leads to a correct decision with an arbitrarily high chance. The statement is incorrect. (2) How to justify the choice of the mixture of the Beta-Binomial model. I don't quite see the intuition behind this model. The \"mixture\" step stands for two levels of difficulty. Then what does the \"Beta\" step stand for? Have you tried other models (e.g., the mixture of Binomials with more mixtures)? I understand that the mixture of Beta-Binomials works the best in your experiments, but the modeling step should be justified more sufficiently.\n\n3. The experiments are using the error margin as the performance metric. As I mentioned in the Weaknesses part, the metric is not sufficient for the entire distribution. Could you provide any further performance metrics to further validate your findings?\n\n4. Typos and minor issues: (1) Line 175 \"eta\" should be \"beta\"? (2) Line 237 $\\alpha$ and $\\beta$ have already been used for Beta distribution's parameters, so please consider replacing the hyperparameters with other characters. (3) Line 255 missing reference. (4) Some texts in the math mode are not in the text mode.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The authors provide a novel problem of estimating the distribution of correct judges when applying LLM ensembles. This may help develop a better understanding of how to model the uncertainty when LLM-as-a-judge is widely adopted today.\n\n2. The paper gives some empirically/theoretically validated key observations (see the Summary part) when modeling the uncertainty: the first is supported by empirical evidence, and the second is supported by conformal prediction theory.\n\n3. The numerical experiments cover a wide range of datasets and models.\n\nWeaknesses:\n\n1. The motivation behind studying this problem is not fully motivated (or at least I do not quite understand). Estimating the total correct votes from an LLM ensemble for an *entire* dataset may be less meaningful compared to estimating the uncertainty on some *certain* prompt (e.g., please see the individual calibration v.s. marginal/populational calibration).\n\n2. The target proposed at the beginning (estimating the whole distribution) is not fully aligned with the numerical experiment metric (estimating the population-level error rate v.s. the true error rate, if I understand correctly). The latter target is *much* easier. To validate the findings, the authors need other metrics (e.g., the estimated $\\tau$-quantiles v.s. the actual $\\tau$-quantiles for different $\\tau$'s). The current status of the experiment doesn't support the goal proposed in line 33.\n\n3. The presentation is a little bit confusing. The paper proposes the target of estimating $\\mathbb{P}(S=n|x)$ (line 33), which seems to be an individual/contextual estimation based on observing $x$, yet all the remaining discussions are without the dependence on $x$ (which is population-level). I think the authors need to clarify this point when revising the draft. Besides, the authors need to define what the error rate and the error margin are *formally* in the main text. The authors also need to present the main algorithm framework in the main text rather than in the appendix.", "was_revised": true, "final_justification": "I think the authors have proposed a novel way of modeling the LLM-as-a-judge distribution, and it is worth noticing for the community. But I'm not sure if this distribution is fully justified outside empirical evidence; I'm also not fully convinced of the value of calibrating the marginal error in a dataset rather than the individual one for each prompt. Therefore, I'm giving my score of borderline accept and leaving the decision to the AC.", "rebuttal": "Thank you, **Reviewer Qt2U**, for your detailed review and recognizing our work as economic, well-written, and extensively validated. Below, we address your questions point by point:\n\n**W1: Motivation for population-level estimation**\n\nAn intuitive example of population-level estimation is a **classification task**, where the model’s accuracy is assessed across the entire dataset rather than a single sample. When given new samples, **the estimated accuracy is the accuracy on the entire dataset**. Following this intuition and standard practice in LLM-as-a-Judge, we adopt population-level performance estimation.\n\n**W2 (1), W3 (1) & Q1: Equation on Line 33**\n\nThank you for highlighting this. Our intention was to indicate that our task involves estimating the number of correct judgments for any input question, but the original equation missed this assumption. We will revise the equation on **Line 33** as follows:\n$$\n\\mathbb{P}(\\\\#\\ \\text{correct judgments}=n\\ \\vert\\ \\forall x\\in D, k\\ \\text{LLMs judge sample}\\ x),\n$$\nwhere $x$ is a sample and $D$ is the dataset. To achieve this goal, we adopt **population-level estimation to represent the distribution for any given sample**.\n\n**W3 (2): Definition of error rate and error margin**\n\nWe apologize for the confusion. Definitions are clarified as follows:\n\n- **Error rate:** The fraction of erroneous judgments made by the ensemble majority vote over the total number of samples in the dataset. It intuitively represents how frequently samples are judged incorrectly.\n- **Error margin:** The absolute difference between the estimated and actual fraction of incorrectly judged samples. This metric assesses the quality of our estimated judgment performance relative to the true performance.\n\n**Q2 (1): Application of Condorcet's Jury**\n\nWe reference Condorcet’s Jury Theorem to highlight the inadequacy of assuming a binomial distribution. Given the following reasons, we believe **the Condorcet’s Jury Theorem is applicable**:\n\n1. $p>0.5$ implies the model performs better than random guessing. This condition **typically holds true** in our experimental settings.\n2. We respectfully disagree with the notion of high correlation in LLM decisions. While **LLMs judge independently without intercommunication**, decisions are conditionally dependent since judgments are made based on the same question $x$. Thus, the joint judgment distribution $\\mathbb{P}(y|x)$ is conditional, not independent.\n\n**Q2 (2): Meaning of “Beta”**\n\nThank you for this critical inquiry. The motivation behind our choice stems from addressing empirical shortcomings in simpler Binomial models. A Binomial distribution assumes a single, fixed probability of correctness ($p$) across all questions, which contradicts our empirical observations (Figure 3). Introducing the \"Beta\" component enables modeling the probability itself as a random variable, effectively capturing variations in question difficulty.\n\n**Q2 (3): Further validation of the Beta-Binomial mixture**\n\nTo further validate our method, we include additional baseline comparisons involving mixtures of three, four, and five Beta-Binomial distributions, as well as a mixture of two Binomial distributions:\n\n|            | 2 BB Dist. (Ours) |  3 BB Dist.  |  4 BB Dist.  |  5 BB Dist.  | 2 Bin Dist.  |\n| ---------- | :---------------: | :----------: | :----------: | :----------: | :----------: |\n| HaluEval   |    6.68 ± 0.53    | 6.43 ± 0.64  | 7.35 ± 0.51  | 6.73 ± 0.45  | 9.12 ± 0.34  |\n| JudgeBench |    6.98 ± 0.56    | 6.31 ± 0.41  | 6.82 ± 0.43  | 6.98 ± 0.47  | 8.63 ± 0.70  |\n| BIG-bench  |   11.15 ± 0.60    | 12.51 ± 0.53 | 12.35 ± 0.68 | 12.70 ± 0.54 | 12.14 ± 0.24 |\n\nThe results show that adding more Beta-Binomial distributions provides **minimal or no performance gain** compared to our two-distribution choice, justifying our decision. Compared to the mixture of Binomial distributions, **our method achieves lower error margins**, suggesting our practice is effective.\n\n**W2 (2) & Q3: Additional metrics for evaluation**\n\nTo further assess our method's distribution estimation quality, we provide additional comparisons using **Wasserstein Distance** and **KL Divergence**:\n\n**Table a: Comparison with baselines using Wasserstein Distance**\n\n|           | HaluEval | JudgeBench | BIG-bench |\n| --------- | :------: | :--------: | :-------: |\n| Binomial  |  3.8810  |   3.8404   |  3.1527   |\n| Single BB |  2.0787  |   2.1123   |  1.6547   |\n| Ours      |  1.9614  |   1.7246   |  0.5205   |\n\n**Table b: Comparison with baselines using KL Divergence**\n\n|           | HaluEval | JudgeBench | BIG-bench |\n| --------- | :------: | :--------: | :-------: |\n| Binomial  |  6.2672  |   6.1436   |  4.3358   |\n| Single BB |  3.3084  |   3.1690   |  2.4090   |\n| Ours      |  2.9744  |   2.3681   |  0.3692   |\n\nIn the results above, we select HaluEval, JudgeBench and BIG-bench and report the **average numbers across the three datasets**. As shown by the results, our method still outperforms baselines under the **Wasserstein Distance** and **KL Divergence**. This shows that our model is consistently better in different metrics。\n\n**Q4: Typo and minor issues**\n\nThank you for identifying these issues. We will thoroughly address and correct them in our manuscript revision."}, {"review_id": "TUMk3s71Te", "reviewer": "Reviewer_gw97", "summary": "This work introduces BetaConform, an economic framework for measuring the accuracy of an ensemble of LLM-based judges. The framework includes (1) a bi-modal beta-binomial mixture model to model the judgment distribution, (2) an adaptive stopping for collecting annotations, and (3) a text-similarity-based transfer mechanism to incorporate mixture parameters learned on other datasets. Extensive experiments show that BetaConform produces higher error margins under the same annotation budget compared to other methods.", "questions": "1. On some configurations in Table 2, the prior transfer hurts. Have you found any patterns in the failure cases? For example, it seems like the prior transfer helps Llama-2.2-70B considerably, but slightly hurts GPT-3.5. Why?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written. The motivation, methodology, and experimental results are clear and accessible to readers.\n2. The paper tackles an important challenge: measuring the reliability of LLM-based evaluations, particularly in low-data regimes.\n3. Extensive experiments and ablations support the design of the framework. \n\nWeaknesses:\n1. In the definition of \"LLM Ensemble Correct Judgment\", Match(·) is binary i.e., a judge is either correct or incorrect, which constrains the types of tasks where this framework is applicable (e.g., ranking candidate responses or multi-label rubrics).\n2. The mixture model assumes two peaks (i.e., easy/hard problems), which may not hold on all datasets, in the presence of adversarial prompts, or in sampling regimes with high variability.", "was_revised": true, "final_justification": "The authors propose a principled approach for measuring the performance of an ensemble of LLM-based judges. I'm convinced that the proposed distribution holds in most cases. The early stopping mechanism and transfer mechanism make the method economical to apply. Because of this, I recommend acceptance. This work is especially valuable for the community given the popularity of LLM-based judges and the lack of research into their reliability.", "rebuttal": "Thank you, **Reviewer gw97**, for your detailed review and for recognizing our work as economic, well-written, and extensively validated. Below, we address your questions point by point:\n\n**Q1: Binary $\\mathrm{Match}(\\cdot)$ function**\n\n**A1:** Our rationale for adopting the binary $\\mathrm{Match}(\\cdot)$ function is that many tasks evaluated by LLM-as-a-Judge can be **naturally framed as binary classification**, resulting in conclusions of \"yes\" or \"no.\" Additionally, accuracy in scoring-based judgments can be considered binary, where a judgment is **deemed incorrect if the deviation exceeds a certain threshold**, otherwise correct.\n\nWe acknowledge that the current approach does not support ranking and appreciate your helpful suggestion. We plan to explore incorporating ranking as part of our future work.\n\n**Q2: Mixture distribution assumption**\n\n**A2:** Our work assumes a typical judging process without adversarial conditions. Adversarial prompts represent special cases of challenging questions designed specifically to **induce incorrect judgments from LLMs**. Our sampling assumption is based on **random availability of samples**.\n\nAlthough our two-peak assumption originated from observations presented in **Figure 3**, extensive experiments (see **Table 1**) demonstrate that our assumption **consistently yields better performance** compared to baseline methods. This supports the **validity of our two-peak assumption across various datasets**.\n\nWe acknowledge the challenge posed by adversarial prompts, which can intentionally create challenging distributions that degrade estimation accuracy. Addressing this will be part of our future research.\n\n**Q3: Performance degradation with GPT-3.5**\n\n**A3:** While distribution transfer aims to enhance estimation accuracy, it may also introduce noise. To mitigate potential negative impacts from the transfer, we designed the transfer mechanism (see **Equation 13**) to carefully balance reliance on source distributions, thus preventing over-dependence and reducing possible degradation.\n\n**Q4: Typos**\n\n**A4:** Thank you for pointing these out. We will revise them in the final manuscript."}]

Author rebuttal:

Meta-review: The paper proposes BetaConform, a framework to efficiently and accurately estimate the performance of LLM ensembles. The authors first propose to use binomial judgment to better fit the observed judgment patterns. They then propose to use conformal prediction to enable early stopping during iterative sampling. Finally, they show that the distributional parameters can be transferred to new dataset based on text similarities.

The paper proposes a novel principled approach for efficient judgment estimation from LLM ensembles (NToF, Qt2U), is easy to read (NToF), efficient and effective (vhVT, gw97, Qt2U) and well-motivated (NToF). The theoretical work is supported by comprehensive experiments (vhVT, Qt2U, gw97).

The concerns raised by the reviewers regarding the choice of hyper-parameters (number of components; gw97, vhVT),  generalizability (NToF) and choice of evaluation metrics (Qt2U) were well addressed during the rebuttal. Reviewer Qt2U had concerns about calibrating the marginal error vs errors in individual prompts which was addressed by the author in that they will clarify that they conduct population-level performance estimation and add relevant motivation.

Given the overall clarity, novelty, and strong empirical evidence, as well as the effective rebuttal, I recommend acceptance.
Decision: Accept (poster)