Title: $\texttt{BetaConform}$: Efficient MAP Estimation of LLM Ensemble Judgment Performance with Prior Transfer
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM-as-a-Judge

Distribution Estimation

LLM Ensemble
Abstract: LLM ensembles are widely used for LLM judges. However, how to estimate their accuracy, especially in an efficient way, is unknown. In this paper, we present a principled $\textit{maximum a posteriori}$ (MAP) framework for an economical and precise estimation of the performance of LLM ensemble judgment. We first propose a mixture of Beta-Binomial distributions to model the judgment distribution, revising from the vanilla Binomial distribution. Next, we introduce a conformal prediction-driven approach that enables adaptive stopping during iterative sampling to balance accuracy with efficiency. Furthermore, we design a prior transfer mechanism that utilizes learned distributions on open-source datasets to improve estimation on a target dataset when only scarce annotations are available. Finally, we present $\texttt{BetaConform}$, a framework that integrates our distribution assumption, adaptive stopping, and the prior transfer mechanism to deliver a theoretically guaranteed distribution estimation of LLM ensemble judgment with minimum labeled samples. $\texttt{BetaConform}$ is also validated empirically. For instance, with only $10$ samples from the TruthfulQA dataset, for a Llama ensembled judge, $\texttt{BetaConform}$ gauges its performance with an error margin as small as $3.37\\%$.


Peer reviews:
comments: SUMMARY:
This work introduces BetaConform, an economic framework for measuring the accuracy of an ensemble of LLM-based judges. The framework includes (1) a bi-modal beta-binomial mixture model to model the judgment distribution, (2) an adaptive stopping for collecting annotations, and (3) a text-similarity-based transfer mechanism to incorporate mixture parameters learned on other datasets. Extensive experiments show that BetaConform produces higher error margins under the same annotation budget compared to other methods.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper is well-written. The motivation, methodology, and experimental results are clear and accessible to readers.
2. The paper tackles an important challenge: measuring the reliability of LLM-based evaluations, particularly in low-data regimes.
3. Extensive experiments and ablations support the design of the framework. 

Weaknesses:
1. In the definition of "LLM Ensemble Correct Judgment", Match(·) is binary i.e., a judge is either correct or incorrect, which constrains the types of tasks where this framework is applicable (e.g., ranking candidate responses or multi-label rubrics).
2. The mixture model assumes two peaks (i.e., easy/hard problems), which may not hold on all datasets, in the presence of adversarial prompts, or in sampling regimes with high variability.

QUESTIONS:
1. On some configurations in Table 2, the prior transfer hurts. Have you found any patterns in the failure cases? For example, it seems like the prior transfer helps Llama-2.2-70B considerably, but slightly hurts GPT-3.5. Why?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a problem of estimating the error rate distribution of LLM ensembles. They provide several key points: (1) the distribution of the correct votes can be better modeled by a two-level mixture of Beta-Binomial distribution, compared to the Binomial distribution model; (2) the number of samples can be chosen via the conformal prediction's square-root bounds; and (3) the distribution parameters can be transferred from other source datasets based on text similarities. The algorithm combines the above three points and is verified by empirical evidence.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The authors provide a novel problem of estimating the distribution of correct judges when applying LLM ensembles. This may help develop a better understanding of how to model the uncertainty when LLM-as-a-judge is widely adopted today.\n\n2. The paper gives some empirically/theoretically validated key observations (see the Summary part) when modeling the uncertainty: the first is supported by empirical evidence, and the second is supported by conformal prediction theory.\n\n3. The numerical experiments cover a wide range of datasets and models.\n\nWeaknesses:\n\n1. The motivation behind studying this problem is not fully motivated (or at least I do not quite understand). Estimating the total correct votes from an LLM ensemble for an *entire* dataset may be less meaningful compared to estimating the uncertainty on some *certain* prompt (e.g., please see the individual calibration v.s. marginal/populational calibration).\n\n2. The target proposed at the beginning (estimating the whole distribution) is not fully aligned with the numerical experiment metric (estimating the population-level error rate v.s. the true error rate, if I understand correctly). The latter target is *much* easier. To validate the findings, the authors need other metrics (e.g., the estimated $\\tau$-quantiles v.s. the actual $\\tau$-quantiles for different $\\tau$'s). The current status of the experiment doesn't support the goal proposed in line 33.\n\n3. The presentation is a little bit confusing. The paper proposes the target of estimating $\\mathbb{P}(S=n|x)$ (line 33), which seems to be an individual/contextual estimation based on observing $x$, yet all the remaining discussions are without the dependence on $x$ (which is population-level). I think the authors need to clarify this point when revising the draft. Besides, the authors need to define what the error rate and the error margin are *formally* in the main text. The authors also need to present the main algorithm framework in the main text rather than in the appendix.\n\nQUESTIONS:\n1. Line 33's goal should be carefully revised. All the guarantees are made for the population-level rather than arbitrary $x$.\n\n2. About the mixture of the Beta-Binomial model: (1) How to rule out the possibility of the Binomial distribution. Lines 42 and 43 are confusing. Condorcet's Jury Theorem is only correct if (a) each judge has an equal chance of $p>0.5$ making the correct choice; and (b) each judge is making the decision independently. Both conditions are hardly met in the LLM ensemble scenario, especially the second one (I think the LLMs' decisions are highly correlated). So, the Binomial assumption alone doesn't imply that improving the judge number leads to a correct decision with an arbitrarily high chance. The statement is incorrect. (2) How to justify the choice of the mixture of the Beta-Binomial model. I don't quite see the intuition behind this model. The \"mixture\" step stands for two levels of difficulty. Then what does the \"Beta\" step stand for? Have you tried other models (e.g., the mixture of Binomials with more mixtures)? I understand that the mixture of Beta-Binomials works the best in your experiments, but the modeling step should be justified more sufficiently.\n\n3. The experiments are using the error margin as the performance metric. As I mentioned in the Weaknesses part, the metric is not sufficient for the entire distribution. Could you provide any further performance metrics to further validate your findings?\n\n4. Typos and minor issues: (1) Line 175 \"eta\" should be \"beta\"? (2) Line 237 $\\alpha$ and $\\beta$ have already been used for Beta distribution's parameters, so please consider replacing the hyperparameters with other characters. (3) Line 255 missing reference. (4) Some texts in the math mode are not in the text mode.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper presents a principled maximum a posteriori (MAP) framework for an economical and precise estimation of the performance of LLM ensemble judgment. It replaces the Binomial assumption with a mixture of Beta-Binomial distributions to capture bimodal judgment patterns, introduces a conformal prediction-driven approach to enable adaptive stopping during iterative sampling, and uses text-similarity-based transfer of learned distributions from source datasets to improve target dataset estimation with few labels. 

The BetaConform framework delivers a theoretically guaranteed estimation of LLM ensemble judgment with minimum labeled samples, and also validated empirically.

STRENGTHS AND WEAKNESSES:
Strengths:
- The approach is able to dramatically cut down annotation costs.
- The approach is theoretically rigorous in that it is formally grounded in MAP estimation, conformal prediction, and Bayesian transfer.
- The framework is broadly validated on 11 diverse datasets and five SoTA LLMs.

Weaknesses:
- The mixture model only assumes two components (i.e. easy/hard tasks), which may not well generalize to more components that reflect more complicated situations.

- The distribution prior transfer is highly reliant on the quality of text embeddings, which may be a problem when the target dataset has only a few samples.

QUESTIONS:
Apart from the weaknesses above, I would like to see a discussion on the overall computational overhead, as the cost of text embedding, EM algorithm and conformal prediction may bring extra computational overhead while it effectively reduces annotations.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method to efficiently and accurately estimate the judgment performance of LLM ensembles. The method proposed determines: (RQ1): how to estimate the judgment distribution of LLM ensembles and (RQ2): how many samples are needed to achieve a desired estimation under a given error margin threshold. To achieve more accurate estimation, the paper proposes to use a mixture of Beta-Binomial distributions to model the different judgment for \"easy\" and \"hard\" questions and use expectation maximization (EM) to estimate the parameters of the distribution from the samples. To achieve the accurate estimation more efficiently, the paper determines the number of samples to use for a good estimation using conformal prediction to compute when to stop sampling when a pre-defined threshold is reached. This limits the number of samples to take while maintaining estimation precision. Finally, to efficiently estimate when only a few samples are available, the paper proposes a text similarity-based method to transfer from the distributions of similar datasets to the target dataset. The paper then conducts experiments across various datasets and shows that the proposed method achieve high estimation precision and achieve low error margin with as few as 10 samples.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- the paper proposes an original method to efficiently and accurately estimate judgment distribution of LLM ensembles, which combines the choice of distribution (Beta-Binomial) and EM, adaptive stopping (using conformal prediction), and text similarity-based transfer from distributions of existing datasets. \n- the problem the paper is tackling is well-motivated, since there is a need to estimate performance of LLM ensembles with few samples. The proposed method can estimate the performance of LLM ensembles with lower human annotation costs (both in terms of time and financial costs). \n- while the paper starts from theories in statistics, the proposed method is shown to perform well empirically on multiple datasets. \n- the paper is written really well and is very easy to read. The code is given to ensure reproducibility. \n\nWeaknesses:\n- while the choice of 2 beta-binomial mixture model is shown to be effective empirically, it is unclear why the authors choose 2 distributions instead of 3, 4, 5, and so on. The reasoning that it is because questions given to LLMs can be categorized into \"easy\" vs. \"hard\" seems rather coarse-grained and heuristic. A deeper exploration of this choice will make the paper stronger. \n- it will be good to add more intuitive explanations on statistical theories such as the conformal prediction, for readers who are not familiar with the mathematical reasoning behind these theories. \n- it is unclear how the distribution transfer was done. How were the parameter estimates for prior dataset distributions combined with the parameter estimates of the target data? Were the prior parameters used as initial parameters to be further refined through EM? Or were other methods of combining used?\n- finally, the proposed method seems to be limited in that the LLMs in the ensemble have to all be the same model: same architecture, same size, same version with variations only in terms of the top-P token sampling. This seems to limit the generalizability of the method to real world LLM ensembles that may be composed of different LLM models (e.g., Llama with Qwen).\n\nQUESTIONS:\n- Why the choice of 2 distributions in the mixture model? Why not 3, 4, or other values? Were other values explored?\n- How were the parameter estimates for prior dataset distributions combined with the parameter estimates of the target data? Were the prior parameters used as initial parameters to be further refined through EM? Or were other methods of combining used?\n- How will the proposed model change when ensembles with different underlying models are used? Was this explored?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}