Cost-aware LLM-based Online Dataset Annotation
Authors: Eray Can Elumar
Cem Tekin
Osman Yagan
Venue: NeurIPS 2025 spotlight ()
Keywords: large language models (LLMs)
dataset annotation
optimization
Primary area: deep_learning

Abstract: Recent advances in large language models (LLMs) have enabled automated dataset labeling with minimal human supervision. While majority voting across multiple LLMs can improve label reliability by mitigating individual model biases, it incurs high computational costs due to repeated querying. In this work, we propose a novel online framework, Cost-aware Majority Voting (CaMVo), for efficient and accurate LLM-based dataset annotation. CaMVo adaptively selects a subset of LLMs for each data instance based on contextual embeddings, balancing confidence and cost without requiring pre-training or ground-truth labels. Leveraging a LinUCB-based selection mechanism and a Bayesian estimator over confidence scores, CaMVo estimates a lower bound on labeling accuracy for each LLM and aggregates responses through weighted majority voting. Our empirical evaluation on the MMLU and IMDB Movie Review datasets demonstrates that CaMVo achieves comparable or superior accuracy to full majority voting while significantly reducing labeling costs. This establishes CaMVo as a practical and robust solution for cost-efficient annotation in dynamic labeling environments.

Cost-aware LLM-based Online Dataset Annotation
Eray Can Elumar
Cem Tekin
Dept. of Electrical and Computer Eng. Carnegie Mellon University Pittsburgh, PA 15213
Dept. of Electrical and Electronics Eng.
Bilkent University
Ankara, Türkiye
eelumar@andrew. cmu. edu
cemtekin@ee bilkent. edu. tr
Osman Yagan
Dept. of Electrical and Computer Eng. Carnegie Mellon University
Pittsburgh, PA 15213 oyagan&ece cmu edu
Abstract
Recent advances in large language models (LLMs) have enabled automated dataset labeling with minimal human supervision. While majority voting across multiple LLMs can improve label reliability by mitigating individual model biases, it incurs high computational costs due to repeated querying. this work, we propose a novel online framework, Cost-aware Majority Voting (CaM for efficient and accurate L.LM-based dataset annotation. CaMVo adaptively selects a subset of LLMs for each data instance based on contextual embeddings, balancing confi- dence and cost without requiring pre-training or ground- labels. Leveraging LinUCB-based selection mechanism and a Bayesian estimator over confidence scores, CaMVo estimates lower bound on labeling accuracy for each LLM and aggregates responses through weighted majority voting. Our empirical evalua- tion on the MMLU and IMDB Movie Review datasets demonstrates that CaMVo achieves comparable superior accuracy full majority voting while significantly reducing labeling costs. This establishes CaMVo as a practical and robust solution for cost-efficient annotation in dynamic labeling environments.
Introduction
The rapid proliferation of data across domains has created urgent need for accurate, large-scale annotation pipelines. While human experts and crowd workers have been the gold standard for dataset labeling, manual annotation is notoriously slow, expensive, and prone to inter-annotator inconsistency Petrović et al. 2220)_ As machine learning models become increasingly sophisticated, their demand for high-quality, richly labeled datasets only intensifies, exacerbating this bottleneck.
Recent advances large language models (LLMs) offer promising remedy: by leveraging transformer-based architectures such as GPT, it is now possible to automate much of the label- ing workload. LLMs excel at natural-language understanding, reasoning, and contextual inference, enabling rapid generation of annotations with minimal human effort Naveed et al. 2023].
However, relying on a single LLM introduces issues biases inherited from its training data and stochastic variability across repeated queries, undermining reliability and reproducibility |mricaa et al. 2024 Li et [2244]. common strategy to bolster label quality ensembling: querying multiple LLMMs, multiple samples from the same model, and aggregating their outputs via majority voting.
'Wor done while at Carnegie Mellon University. Now at Intel Corporation.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

This reduces hallucinations and offsets the bias of individual models but substantially increases cost, as each additional model increases latency and compute expenditure Yang et al. (2023]. In practice, querying every available L.M for each instance is often wasteful and unnecessary.
In this paper, we address this trade-off by adaptively selecting subset of L.MSs for majority voting on each input, achieving comparable accuracy to full-ensemble voting while dramatically cutting cost. Unlike prior work on LLM weight optimization or query routing-whiee presumes access to
ground-truth labels or a pre-trained routing model Chen et al. [2024] Nguyen el al. (2024). Ding
al.
2024
our method operates online, without any held-out training set or ground truth.
Our contributions are as follows:
1. Online formulation. To the best of our knowledge, this is the first work on LLM-based dataset labeling in which both vote weights and the queried subset of LLMs are adapted in real time, i.e. without relying on pre-trained model or dedicated training set.
2. Cost-aware Majority Voting (CaMVo). We propose CaMVo, algorithm that combines LinUCB-style contextual bandit with a Bayesian Beta-mixture confidence estimator. For each candidate L.M,, CoMVo computes lower confidence bound on its correctness proba- bility given the input's embedding, then selects the smallest-cost subset whose aggregated confidence exceeds user-specified threshold 6.
3. Empirical validation. Through experiments the MMLU benchmark and the IMDB Movie Review dataset, we show that CaMVo matches or exceeds the accuracy of full majority voting (majority voting with all available LLMs) while significantly reducing the cost: on MMLU, CaMVo achieves higher accuracy with around 40% lower cost, on IMDB, it attains only 0.1 0.17% drop in accuracy while halving query expenditure.
1.1 Related Work
Ensembling and Majority Voting with LLMs. Aggregating outputs from multiple LLMs (or repeated queries to single LLM) via majority voting has become a popular strategy to boost annotation reliability. [Chnn et al. 2024] analyze the effect of repeated queries to a single model and observe con-monotoiic accuracy curve: performance improves initially but degrades beyond an optimal number of calls due to task heterogeneity. While additional LM calls enhance accuracy on easier queries, they may introduce noise inconsistency that degrades performance on more challenging ones. To address this, the authors propose a scaling model that predicts the optimal number of LM calls required maximize aggregate performance. TTad and Chehab (024 compare re- re-querying and multi-mode ensembles, showing that ensemble strategies are most effective when individual models or prompts exhibit comparable performance levels, and ensemble gains may diminish when individual model accuracies diverge.
Yang et al. 2023 propose a weighted majority-votinn ensemble for medical QA, combining dynamic weight adj adjustment with clustering- based model selection. However, their approach relies on offline training data and focuses solely improving accuracy, without accounting for the cost of querying. In contrast, our approach selects a cost-effective subset of heterogeneous LLMs online, without any pre-training.
LLM Query Routing. Query routing addresses the problem of selecting a single LLM per query to optimize cost latency under performance constraints. Nguyen al. [2024] cast LLM selection as contextual bandit problem, training offline on labeled data to learn a routing policy that maps query embeddings to a single optimal LLM under total budget constraint of b. Ding et al. [2024] train router to distinguish "casy" versus "hard" queries, sending easy tasks to local LLMs and hard ones to cloud APIs Unlike these methods, our method operates in fully online setting without access to labeled training data, updating the model dynamically during the labeling process. Moreover, rather than routing to a single I.IM, we select a cost-efficient subset for majority voting.
Confidence Estimation in LLM Outputs. Estimating model confidence can guide automatic label selection. Kadavath et al. 2022 explore how a language model's own uncertainty estimates can serve as predictors of answer correctness, and find that the cumulative bog-robbability the model assigns to its generated token sequence correlates strongly with factual accuracy across diverse benchmarks. Li ct al. 2024a] generate code multiple times and use output similarity as a proxy for confidence,

---

choosing the most consistent result. While these works focus on self-consistency of a single model, we estimate probabilistic lower bound on each LLM's correctness via Bayesian Beta-mixture model, incorporating both past performance and context.
Weighted Majority Voting. Beyond simple voting, weighted schemes assign each annotator or model a reliability score such the accuracy of the annotator, or the label confidence reported by the annotator. One notable approach is the GLAD model by Whitehill et al. (2009), which formulates weighted majority voting as probabillittic inference problem over annotator expertise and task difficulty. GLAD jointly estimates per- annotator reliability and per-item ambiguity via generative model, using an EM algorithm infer the latent variables. Li and Yu (2014) introduce Iterative Weighted Majority Voting (IWMV) to aggregate noisy crowd labels by iteratively estimating worker reliability, and show it approaches the oracle Maximum A Posteriori (MAP) solution.
Crowdsoureing. Rangi and Franceschetti [2018) utilize the bandits-with withktts-wii--kaasscckkkkkk knapsacks framework to dynamically estimate worker accuracy and allocate tasks in real time to maximize overall labeling quality within budget. Another influential model is by Raykar et al. (010), which jointly infers true labels and annotator reliabilities by modeling each worker's s confusion matrix. Through an expectation pepetation-maximizzatinn procedure, the method -weihtt inconsistent annotators and yields more accurate aggregated labels without prior knowledge of worker quality. Our method parallels this online estimation but differs in that leverages contextual embeddings and targets LLM ensembles rather than human annotators.
The comparison of work with prior work is summarized in Table
Method
Ensemble Type
Pretrained
Online
Contextual
Ours (C)MVo)
Subset voting
No
Yes
Yes
Yang et al. 2023
Weighted voting
Yes
No
Yes
Nguyen et al. 2024
Single-model routing
Yes
No
Yes
Ding et al. 2024
Single-model routing
Yes
No
Yes
Chen et al. 2024
Re-querying Re-querying
No
No
No
Li et al. 2024a
No
No
Yes
L1 and Yu 2014
Crowd aggregation
No
Yes
No
Rangi and Franceschetti 2018
Crowd assignment
No
Yes
No
Raykar al. (0T0))
Crowd aggregation
No
Yes
No
Table I: Comparison of our work with prior ensemble and routing approaches.
2 Problem Statement
In this section, we formally define the problem setting, introduce the baseline algorithm, and provide some results that will be used by our proposed algorithm, which will be introduced in $3
22...., Tr). text sample). Let there be a set [K] of K distinct large language models (LLMs). where each LLM 1; is associated with known cost per token Pt and can be represented mapping query g E Q to a response
Consider an unlabeled dataset D = {æ1, {z1.22......
where each 2, denotes a data instance (e.g., a
function 1; Q - R. R. We denote the total number of possible labels for the dataset D as M. The objective in this setting is to assign predicted label ÿ. E [M] to each data instance x; by querying a subset of the available LLMs and aggregating their outputs. Labeling is performed sequentially, where each data instance x, is processed in round t. In each round, LLMs are queried independently of other LLMS and without memory of prior interactions (i.e., no context is preserved between rounds). Furthermore, all queries are made in zero-shot setting, meaning that no speciffc fine-tuning or additional training data used.
To serve as a baseline, we introduce the following weighted majority voting scheme, In this scheme, the predicted label ÿn. for instance x, determined by aggregating the votes of all K LLMS using:
K
Je = arg max ME[M]]
Warr 1 (VuL m
(1)
3

---

where 1 {.} is the indicator function that returns 1 if the condition is true and 0 otherwise, and e denotes the label outputted by LLM I, for instance zy. Since the true label is not available in our setting, we use the empirical accuracy of model 1, relative to the predicted labels as the voting weight. Hence, a (t) = (2 1 ll(y,, 9.))/Na.. where Ni, denotes the number of times LLM 4, has been queried up to round i, and Ds is the predicted label for round 8. The pseudocode for this scheme, which we refer to as the baseline method, is provided in Algorithm 2 in Appendix C
The goal is label the dataset D in a cost-efficien manner by dynamically selecting a subset of LLMs for each data instance zt. To facilitate this selection, we assume access to model Emb(-) that generates a d-dimensional embedding et = Emb(xr,) for 2e in round This embedding serves as representation of the instance and plays an analogous role to the context in contextual bandit framework. Using this embedding, our approach, which will be introduced S estimates a lower bound on the probability that given LLM 4 will produce the correct label for 24,, and utilizes this bound to determine the subset of LLMs. The details for estimating this lower bound given in 3
Naturally, selecting only a subset of LLMs rather than querying all available models may result in reduced labeling accuracy. To manage this trade-off, we introduce a user-defined parameter 5 E [0, 1] that specifies the desired minimum relative confidence of the selected subset compared to the full majority vote using all K LLMs. Let Life denote the lower confidence bound on the estimated probability that I.I. 1, correctly labels instance at round f. Our algorithm identifies the cost- most-minimizing subset of LLMs whose aggregated confidence, relative to that of the baseline method, satisfies the accuracy constraint imposed by 8.
We will leverage the following result estimate the confidence of the majority vote label based on the Li,t and Wi, values when majority voting is performed over a subset A of LLMs.
Lemma 2.1. Let wi denote the weight and L; the lower confidence bound on the correctness of the ouiput from LLM li Suppose the oulputs of LLMs are conditionally independent given the data instance. Then, for a subset of LLMs A S [K], the lower bound on the probability that majority voting over the subset yields the correct label is given by
W)(
Σ SEA
II iES
II (1-1))))
JEAYS
Exeg
where WA- EieA Wi Proof of this result is provided in Appendix A
For practical purposes and computational efficiency, this expression may be approximated as (L) R 1 Fetta (0.5; WL,A, WA- WL,A), where FBeaa (I; a. (3 is the CDF of Beta(a,, distribution evaluated at point x, WI,A = EiEA Li, and W = ΣiEA wi.
Note that Lemma 2.1 can be extended to remove the munditional-nndeeenden assumption using Bahadur's ' model, which characterizes the joint distribution of binary variables using their marginal probabilities, second-order dependence terms known Bahadur parameters, and higher-order inter- action terms [Bahadur] 1961]. This framework allows the computation of majority vote confidence by summing the probabilities of all outcomes where the majority label is correct. However, the absence of a closed-form expression and the exponential growth in computational complexity with the number of variables limit its practical use. In practice, Monte basee simulation methods can approximate the majority-vote confidence by generating correlated LLM outputs; we describe and analyze CaMVo with one such approach in SG
Finally, we introduce a user-specified parameter kmin which enforces a floor on the number of LLMs queried per instance. By requiring at least kmin votes in every round, this constraint further safeguards label quality-ensuring that no annotation is based on fewer than kmin model predictions.
The proposed Cost-aware Majority Voting (CaMVo) algorithm, which incorporates these results and user-defined parameters, is presented in $
3
The CaMVo Algorithm
We propose novel algorithm, Cost-aware Mujority Voting (CaMVo), for efficient dataset labeling with large language models (L.MMs). CaMVo aims to select a cost-effective subset of L.LMs for each input instance by leveraging contextual embeddings to estimate lower confidence bound on

---

the probability that each model will produce a correct label. These bounds are computed using MaUCCB-bssed framework [Li et al. 2010]. Based on this information, CaMVo identifies subset of LLMSs such that the confidence of their weighted majority vote exceeds user-specified threshold 6. no such subset exists, the algorithm defaults to querying available models. The pseudo-code of and consists of six main steps described below.
Vo is provided in Algorithm 1
Algorithm 1 Cost-aware Majority Voting (CaMVo) Algorithm
1: Input: Set of LLMs [K], cost per token
11:
end for
for each LLM embedding model Emb(-),
12:
A: <- Oracle Wr, knin)
confidence threshold 8. LinUCB regulariza-
13:
Query LLMs: Vi = (2te) i E As
tion parameter ÅL, exploration parameter a,
14:
Út < arg maxm LiEAt. W (t)
regularization parameter XR
15:
if A > then
2: A,,0 <- Ar, 0 f- 0a, Vi E [K]
16:
for each LLM A do
3: for each round / = 2, ...., T do
17:
Fi.e 4 1 {yr. = 9r
4: Get context vector: 6₁ f- Emb(
18:
Update: Ni, t No,t11 +
5: for each LLM i 1, 2. ..... K do
19:
Update: A f- A + e,e
6:
g.,((e;) - e, A bi,-11
20:
Update: b,, f bi,t- ri
7:
(,,((e,) (e.( - av c, A 1
21:
Update: piút f 2 (
N..
8:
Li,1 - (s;,
22:
Update the parameters of Est,
9:
Li & Ns 1
23:
end for
24:
end if
Wh,t t pi,t-1 (e()
25:
end for
ManUBB-Based Confidence Estimation. For each LLM 4, CaMVo maintains matrix A, e Rdxd and vector b, E Rd, initialized as A,, = Ayla, 6,,0 = O, where λ > is a user-defined parameter. Given e, Emb(x;), the estimated confidence, and its confidence bound is computed as:
(e,((e)) = e, b, Ciales) = a A
From these, the lower confidence bound (LCB) of LLM confidence can be found as: (e,)(ex) = di,t (ex) Ce.)(e:).
Since this is an LCB of a probability, we clip it between 0 and 1 when it is out of range [0.1] 1].
Bayesian Estimation of Label Correctness. Given the inherent probabilistic nature of LLM outputs, the estimated confidence score may not reliably indicate the correctness of label. To address this, we introduce a Bayesian estimator E-t,(-) that models the posterior probability that prediction correct, conditioned on this confidence. First, we define a latent variable ha, (yii: = ÿt where Ú, is the assigned label. Note that ideally, the true label should be used instead of ÿt. but since we do not have access to the true label, we instead use ÿt. We model the conditional likelihood of (e,)))), given the latent variable hi,, as Beta-distributed random variable:
hi. = ~ 1eta(on,,,,1)) (.)(er)
(e+( /4, = ~ Beta(ox,o, 33,0)-
Further, to model Ph = 1). we use the empirical historical relative accuracy of 4, Mi,t1 which captures the accuracy of LLM 4; relative to the predicted labels up to round t - 1. Similarly, 11 Applying Bayes' rule with these models, the posterior probability is
P(h,, = 0)
modeled as:
(g); = P(h 11
Beta, (g) 04,1,, Mi Beta, ( B,1))
Pi.
(a,-1) Beta, (g: as 3,,o)
We apply this estimator to 0 (er), the LCB of the estimated LLM confidence as L,,- = Est, (0.,, ( (e))) to encourage exploration under the UCB principle. Note that unlike traditional UCB- based methods that promote exploration upper bounds, we use the LCBB as expands the size of the set of L.LSs likely to satisfy the confidence threshold 8.
5

---

Regularization. In the absence of ground-truti labels, empirical relative accuracy estimates in majority voting can overfit, resulting in overconfident weights and biased aggregation. Further, since subset selection is based on these estimated confidences, early estimation errors can bias the selection process and lead to compounding errors over time. To address this issue, we regularize the I.BB of the estimated confidence of LLM 1, using Laplace smoothing:
L L N₁., + AR log(t + 1)/2
(3)
N1. + XR log(t + 1)
where AR 0 is user-defined regularization parameter that controls the strength of smoothing. Laplace smoothing is chosen since it corresponds to using a uniform prior on the Beta estimator, and hence combines well with the Beta estimator. We use log(t) term with the Laplace smoothing as compared to a constant term, as the log(t) term does not decay as quickly as the constant term and prevents overfitting in the long run.
Subset Selection with Oracle. We define the weight of LLM I for majority voting as wi. == Mi,,-1 (e.) This way the weights of LLMS reflect both their past performance, and also their expected performance for the current data instance. An Oracle is used to find the lowest cost subset whose label confidence is above the threshold ō using the computed Life and Wi, values of LLMs. Using Lemma 2.1 this can be expressed as
A Oracle(L,, Wi, kmin) == arg min (t():: dA wi) 8, |A] Kmin
where (t) -EicA Pi Hx4 , Pi the cost per token for LLM and (xxx)) is the token count of the query to the LLM 4 for data instance 24 under the tokenization method of LLM 4. Since we expect only one token as output (which will be the label for the data instance 2t), we ignore the output tokens. Note that as in Lemma 2.1 we assume that the outputs of LLMs are conditionally independent given the data instance. We analyze the correlated version of CaMVo that does not have this assumption in Appendix G Also note that if no such A exists, CaMVo Vo defaults to querying all LLMs.
Label Assignment. We query the LLMs in A, and receive their responses. The label for æ can be assigned via weighted majority vote using ÿt arg maX EiEA. wi(t) 1 {yr, m).
Parameter Updates. If [Al > 1, for each 1 E Ac, CaMVo updates Ai,, b and jia,, which is the empirical relative mean accuracy, as:
As, t A + e b,, + b + Ti, Mi,e + El 1 [3 = d=}
No,
where ri, 1 [3y. = Je Note that parameters are not updated when |AA,]
the reward will
always be 1 in that case. The Beta distribution parameters (ax,), Br,s) can be updated via maximum likelihood estimation or method-of-non mofo-otnntts approximation based on the mean and variance of past confidence scores. These approaches are discussed in Appendix B
Experiments
4.1 Experiments on the MMLU Dataset
We first evaluate CaMVo on the MMLU dataset Hendrycks et al. [2021a]b]. a challenging multiple- choice benchmark spanning 57 diverse subjects including mathematies, U.S. history, law, and computer science. MMLU is well-suited to our setting, as it demands broad world knowledge and strong reasoning capabilities- crabblitees-condtiiionn under which majority voting is particularly effective. To reduce computational cost, we restrict our evaluation to the test split, which contains 14,042 instances.
Claude 3 Sonnet and Haiku from Anthoopic Anthropic [024). GPT-4o, o3-mini, and ol-mini from OOenAA] (2024), and LLaMA-3.3 and LLaMA-3.1 from Meta Meta [2024]. All models are queried using temperature 0.35 and top-p = 1, where applicable. To extract contextual embeddings for CaMVo, we use the 384-dimensional sentence transformer 111-MiniLM-LL--
Models and setup. We use the following LLMs:
Wang et al. (2020. For computational efficiency. we approximate the confidence (L, w) using the cumulative distribution function of Beta distribution. Further implementation details and experimental setup provided in Appendix D

---

2 (Left) reports the accuracy and cost of individual LLMs, as well as two baselines: Majority Vote, which aggregates all LLMs using weights proportional to their true accuracy. and the Baseline Method, which corresponds to Algorithm? 2 Cost corresponds to the average input token cost when labeling the dataset and is reported dollars per million input tokens. Accuracy reflects the percentage of data instances correctly labeled. CaMVo's performance under varying confidence thresholds 6 and minimum vote counts kmin E {1, 3] is shown in Table 3 Note that we include kmin = 3 in our experiments model parameters do not get updated when only a single LLM is queried, a scenario that can occur under kmin = 1. As a result, users may prefer to avoid setting kmin 1. Moreover, for Amiin = 2, the selected pair of LLMs will always yield a majority vote in favor of the LLM with the higher weight, limiting the informativeness of the voting outcome. The algorithm is configured with 0.25, Ag = and λ 1. The Target Accuracy column reflects the targeted relative accuracy, which is the minimum relative accuracy CaM must exceed to satisfy the threshold 6, and is computed ō x (Majority Vote Accuracy) 5 x 88.18%.
Results. Table
From Table 2 we observe that among individual models, 03-mini achieves the highest accuracy at 85.92% while Majority Vote attains 88.18% at substantially higher cost of $9.14 per million tokens. The Baseline Method also matches this accuracy and cost. Table 3 shows that CaMVo consistently meets or exceeds the desired accuracy levels specified by 8, across all settings of kmin- From Table B it can be observed that CaMVo consistently satisfies all target accuracy levels defined by the confidence parameter 8. Moreover, the accuracy and cost of CaMVo exhibit a predictable trade-off: as 5 decreases, the cost of labeling decreases accordingly, while accuracy also declines in controlled manner. This behavior highlights the flexibility of CaMVo in adapting to wide range of practical scenarios with varying accuracy and budget constraints. Further, at ā = 0.97 and knin 1, CaMVo achieves 88.33% accuracy cost of only $7.18, outperforming both baselines in cost-efficiency. This improvement stems from CaM ability to dynamically select LLM subsets based on both global accuracy estimates and instafce-specific contextual confidence.
We also observe that when kimin = 3, lowering ō below 0.85 has negligible effect on cost or accuracy, This occurs because Vo settles on the lowest-cost trio: LLaMA-3.3, LLaMA-3.1, 1, and Claude-3.5; whose combined cost ($1.44) represents a lower bound given the constraint on kminn
LLM/ / Method
Accuracy (%)
Cost
LLM / Method
Accuracy (%)
Cost
03-mini
85.92
1.10
gpt-4o
95.68
2.50
claude-3-7-somnet
85.65
3.00
03-mini
95.40
1.10
ol-mini
84.82
1.10
claude-3-5-haiku
95.05
0.80
gpt-4o
83.58
2.50
gpt-4o-mini
94.60
0.15
Ilama-3.3-700
81.70
0.59
ol-mini
94.52
1.10
llama-3.1-88
68.01
0.05
1lama-3.1-8h
94.06
0.05
claudc-3-5-han h-5-haiku
64.09
0.80
llama-3.3-706
92.23
0.59
Majority Vote
88.18
9.14
Majority Vote
95.62
6.29
Baseline Method
88.18
9.14
Baseline Method
95.61
6.29
Table 2: Accuracy and cost of individual L.LMs and baseline ensemble methods the MMLU
dataset (Left), and the IMDB Movie Reviews Dataset (Right).
Figure (Left) illustrates the accuracy trade-off of All Subsets versus CaMVo. Each gray point represents one of the 2 11 possible LLM subsets voted via ground-truth accuracies, while the yellow curve depicts those that are Pareto-optimal. CaMVo's 's results appear as blue markers for Amin = and green markers for kmin 3. The red marker denotes the Baseline Method. Remarkably, even without any a priori knowledge of LLM performance, or any pre-training, CaMVo consistently tracks, and sometimes surpasses the Pareto frontier, demonstrating its ability to approximate optimal cost-accuracy trade-off's in an online manner.
Figure I (Right) shows CaMVo's 's cumulative average accuracy (blue) and cost (red) for 8 = 0.96, kmin = I; the green horizontal line indicates the target accuracy. In early rounds, CaMVo explores larger, more expensive subsets, yielding both high cost and high accuracy. As the LCB estimates converge, the algorithm rapidly shifts to smaller, cheaper subsets that still satisfy the accuracy threshold. Cost declines steeply while accuracy stabilizes just above the target, illustrating CaMVo's ability to quickly identify and exploit the most cost-effective ensembles without sacrificing labeling quality. Additional plots with different parameters are provided in Appendix D
7

---

Target Acc. (%)
Acc. (%)
Cost
Acc. (%)
Cost
CaMVo 8
kmin
kmin
kmin
Rmin = 3
0.99
87.30
88 $8.47
9.14
88.47
9.14
0.98
86.42
88.59
8.57
88.59
8.57
0,975
85.98
88.49
7.80
88.49
7.80
0.97
85.53
88.35
6.67
88.33
6.67
0.965
85.09
88.27
5.66
88.27
5.66
0,96
84.65
87.98
4.74
88,03
4.74
0.955
84.21
87.40
3.38
87.01
3,36
0.95
83.77
86.82
2.76
87.01
2.96
0.90
79.36
84.88
1.19
84.80
1.81
0.85
74.95
84.41
1.03
82.14
1.58
0.80
70.54
82.12
0.70
81.32
0.75
66.14
68.80
0.16
81.24
1.50
0.70
61.73
68.38
0.14
81.22
1.50
Accuracy and cost of CaM the MMLU dataset under varying confidence thresholds 6 Table 3:
and Kmin E (1.3). For reference, the cost the baseline method is $9.14 per million tokens.
Accuracy Cast
Accuracy and Cost Over Rounds
100
Accuracy
Accuracy Target for 6 0.96
Accuracy
Majority Voce
CaMVV Kow
86-
Single LLM
84-
2000
4000
5000
8000
10000
12000
14000
Cost
Round
Figure 1: (Left) Cost-accuracy trade-off for MMLU dataset: gray dots show every LLM subset via weighted majority voting, yellow dots trace their Pareto-optimal frontier, blue markers are CaMVo at kaiin = 1, green markers at kmin 3, cyan markers denote the individual single LLMs, and the red marker denotes the Baseline Method. (Right) Cumulative average accuracy and cost of CaMVo with 8 0.96, Amin 1 over rounds.
4.2 Experiments on the IMDB Movie Reviews Dataset
We next test CaMVo on the IMDB Movie Reviews dataset Maas et al. [2011] balanced binary- sentiment benchmark of 50,000 movie reviews. As before, we compare CaMVo against each individual LLM, full-ensemble Majority Vote, and the Baseline Method (Algorithm 2)
Models and setup. We employ Anthropic's Claude 3-5 Haiku Anthropic [024]; OpenAl's GPT-4o, 03-mini, GPT-4o-mini, and ol-mini OpenAl [2024]: and Meta's LLaMA-3.3 and LLaMA-3.1 Meta [2024). All queries use temperature = 0.25 and top-p 1, where applicable. We extract 384- dimensional contextual embeddings with all all-MiniLM-L6-- the confidence bound (L, w) via the Beta-CDE, in $4.1
Wang et al. (0200) and approximate
Results. Table 2 (Right) reports the accuracy and cost (in dollars per million input tokens) of each LLM and the two baselines. The baseline underperforms the best individual model (95.68% vs. 95.61% 1)) despite incurring significantly higher cost. This is partly due to the relative ease of the IMDB Movie Reviews dataset, where individual LLMs already achieve high accuracy, limiting the marginal benefit of ensembling. As noted by Li al. (2024b), ensemble gains most pronounced on harder tasks. Additionally, Trad and Chehab 2024 highlight that large performance gaps among models reduce ensemble effectiveness, making smaller, selective subsets preferable in such cases.
Table presents CaMVo's accuracy-coe cost trade-off across various thresholds and kimin E (1,3). CaMVo's hyperparameters 0.7, AR 5, and d 1; and the Target Accuracy is computed

---

similarly as x 95.62%. Across all configurations, CaMVo meets or exceeds its target accuracy. Further, CaMVo achieves less than half the cost (when 8 0.997 and kmin 1) a slightly lower accuracy of 95.45% compared to the baseline, confirming its practicality for large-scale sentiment annotation without any pre-training or ground-truth labels.
Target Acc. (%)
Acc. (%)
Cost
Acc. (%
Cost
CaMVo 8
Amin
Kmin
mmn
kmin
3
0.999
95.52
95.59
6.15
95.59
6.15
0.998
95.43
95.43
4.03
95.43
4.03
0.997
95.33
95.45
2,83
95.45
2.83
0,995
95.14
95.25
2,06
95.25
2,06
0.99
94.66
95.10
1.09
95.12
0.99
0.985
94.20
94.69
0.34
95.06
0.84
0.98
93.71
94.69
0.31
95.07
0.83
0.97
92.75
94.56
0.22
95.07
0.82
0.96
91.80
94.21
0.13
95.06
0.81
0.95
90.84
94.28
0.14
95.07
0.81
0.9
86.06
94.24
0.10
95.06
0.81
Table 4: Accuracy and cost of CaMVo on the IMDB dataset under varying confidence thresholds and Kmin E (1,3). For reference, the cost of the baseline method is $6.29 per million tokens.
Accuracy and Cost Over Rounds
06.0
100
Accuracy forget 5.9955
99
95.0
ACCUPSE 100
04.3
Accurscc (86)
03.5
10000
20000
30000
40000
50000
Round
Figure 2: (Left) Cost-accuracy trade-off for IMDB dataset: gray dots show every LLM subset via weighted majority voting, yellow dots trace their Pareto-optimal frontier, blue markers are CaMVo at kmin 1, green markers kinin 3, cyan markers denote the individual single LLMs, and the red marker denotes the Baseline Method. (Right) Empirical average accuracy and cost of CaMVo with over rounds.
8 0.995, Kmin
Figure 2 (Left) presents the analogous comparison of Figure (Left) on the IMDB sentiment task. As before, gray points and the yellow Pareto-frontier points show all possible subset combinations, while blue and green markers plot CaMVo at kmin and 3, respectively. The red marker denotes the Baseline Method. CaMVo closely matches the Pareto front in the low-cost regime (cost < 1), but lags behind in higher-cost regions. This exposes a key limitation: when majority voting with additional LLMs is ineffective, CaMVo's reliance the independence assumption, which suggests that aggregating more LLMs improves accuracy; can lead to suboptimal performance.
Figure (Right) plots CaMVo's 's cumulative average accuracy (blue) and cost (red) IMDB with = 0.995 and kmin = 1; the green line marks the target accuracy. As before, early rounds involve querying larger, costlier ensembles robustly explore each model's performance. Once the lower confideeneee bounds stabilize, CaMVo swiftly transitions to minimal-cost subsets that still meet the accuracy requirement. This demonstrates CaMVo's rapid convergence cost-effective model combinations without compromising annotation quality. Additional results for other parameter settings appear in Appendix E

---

4.3 Additional Experiments
We present additional experimental results on the AG News Classification Dataset in Appendix F Appendix introduces VC (Correlated VoM, a practical extension CaMVo that estimates subset confidence by modeling correlations among LLMs, thereby relaxing the independence assump- tion. We provide the corresponding algorithm, experiments, and a comparison with CaMVo, showing that accounting for correlations yields only marginal improvements in accuracy. Finally, Appendix HH reports a sensitivity analysis evaluating how CaMVo's performance degrades under varying levels of correlation among LLM predictions.
5 Limitations
Our work relies on the assumption that the outputs of LLMs are independent of each other. Under this assumption, aggregating any subset of models with individual accuracy above 50% strictly improves majority-vote performance.
practice; i.e., on the IMDB sentiment task ($4.2). LLM outputs can be highly correlated, and majority voting may underperform the best single model. Consequently, CaMVo inherits these failures and can yield lower ensemble accuracy when independence is violated. Nevertheless, even in such regimes CaMVo still achieves the user-specified accuracy threshold while reducing cost relative to the full-ensemble baseline. This mostly due to the fact that our results are relative to the full-ensemble baseline which also suffers from the same issue.
Appendix inttouuucss CCaMVo (Correlated CaMVo), a practical extension that estimates subset con- fidence thorugh estimating the correlation matrix among LLMs; this approach yields only marginal gains in accuracy. More broadly, extending CaMVo account for inter-model correlations a prin- cipled manner-ee e.g., via joint confidence estimation or diversity- subset selection selereonn a promising and challenging direction for future work.
Conclusions
We have introduced Cost-aware Majority Voting (CaMVo), the first fully online framework for LLM-based dataset labeling that jointly adapts both vote weights and the subset of models queried on a per- per-instance basis. By combining a LinUCB-styll contextual bandit with Bayesian Beta-mixture confidence estimator, CaMVo estimates a lower bound each LLM's correctness probability for the given input and selects the minimal-cost ensemble that meets a user-specified accuracy threshold.
Empirical results the MMLU and IMDB benchmarks demonstrate that CaMVo matches or exceeds full-ensemble majority-vote accuracy while reducing labeling cost. On MMLU, CaMVo surpasses the true Pareto frontier of all possible weighted mubsets-despin having prior knowledge individual model performance. These findings establish CaMVo as a practical solution for cost-efficient, automated annotation in dynamic labeling environments without any ground-truth labels or offline training.
Our analysis assumes independence among LLM outputs, which can be violated in practice and may degrade ensemble gains. Nonetheless, Co still enforces the user's accuracy target and delivers significant cost savings even under these conditions.
Our method can be naturally extended beyond classification tasks several ways. For regression, LLM outputs can be aggregated via weighted average or median, with weights updated based on how closely each LLM's output aligns with the aggregate. For ranking tasks, each LLM can assign scores to items, and weighted combination of these scores would produce a final ranking: weights can then be adjusted according to agreement of the LLM's ranking generated from its scores with the aggregated ranking. These extensions involve only minor modifications to the aggregation and weight update steps, making them straightforward to integrate into our existing framework. Future work will explore diversity-aware selection and joint confidence models to mitigate correlated errors. We will also extend Co to support iterative relabeling, allowing previously annotated instances to be revisited and refined as additional contextual information becomes available.
10

---

Acknowledgments and Disclosure of Funding
We thank Dr. Samarth Gupta for his thoughtful insights and stimulating discussions that helped shape the initial direction of this paper. This work was supported in part by the Office of Naval Research Grant N00014-23, 24-23-1-227 and by CyLab Enterprise Security Initiative. The work of Cem Tekin was supported in part by the Scientific and Technological Research Council of Türkiye (TÜBITAK) under Grant 124E065; by the Turkish Academy of Sciences Distinguished Young Scientist Award Program (TÜBA-GEBiP-2023);; and by TÜBÍTAK 2024 Incentive Award.
References
Anthropic. Claude 3 technical overview. httpps//www..atttrrpppcc.cooo/e/ Accessed: 2025-04-24.
2024.
R. Bahadur. On the classification based on responses dichotomous items. In Herbert Solomon, editor, Studies in Item Analysis and Prediction, pages 158-176. Stanford University Press, Stanford, CA. 1961.
Lingjiao Chen, Jared Quiney Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei A Zaharia, and James Y Zou. Are more Ilm calls all you need? towards the scaling properties of compound ai systems. Advances in Neural Information Processing Systems, 0444576--47799 2024.
Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim. Subhabrata Mukherjee, Victor Ruhle, Laks VS Lakshmanan, and Ahmed Hassan Awadallah. Hybrid Ilm: Cost-efficient and quality-aware query routing. arXiv preprint arXiv:2404. 14618, 2024.
Federico Errica, Giuseppe Siracusano, Davide Sanvito, and Roberto Bifulco. What did do wrong? quantifying llms sensitivity and consistency to prompt engineering. arXiv preprint arXiv:2406. 12334, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Pruceedings of the International Conference on Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021b.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac HaffieldDoodss, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint ar wiv::2207.05222 2022.
Hongwei Li and Bin Yu. Error rate bounds and iterative weighted majority voting for crowdsourcing. arXiv preprint arXiv: /4/ 1.4086, 2014.
Jia Li, Yuqi Zhu, Yongmin Li, Ge Li, and Zhi Jin. Showing Ilm-generated code selectively based on confidence of lms. arXiv preprint arXiv:2410.03234 2024a.
Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. More agents is all you need. arXiv preprint arXiv:2402.05120. 2024b.
Lihong Li. Wei Chu, John Langford, and Robert E. Schapire. contextual-bandee approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web (WWW 100,, pages 661-670. ACM, 2010.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150. Portland, Oregon, USA, June 201 1. Association for Computational Linguistics. URL http!
7/ www..aclwbb.... org/ htt///atoooog/11--0000
Meta. Llama-3 models. //ww:/ www. 1lama. com/modeess/llaaa http::///sssaaaaaaacooooooooooa/ 2024. Accessed: 2025-044 24.

---

Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models. arXiv preprint 23Xiv:3377.06335 2023.
Quang Nguyen, Duy C Hoang, Juliette Decugis, Saurav Manchanda, Nitesh V Chawla, and Khoa D Doan. Metallm: A hegf-performant and cost-efficient dynamic framework for wrapping llms. arXiv preprint arXhv:2477108844, 2024.
OpenAL. Openai models. https://platforts www.effro...o.aai....o.. com//ocs//mdeels 2024. Accessed:
2025-04-24.
Nataša Petrović, Gabriel Moyà-Alcover, Javier Varona, and Antoni Jaume-i Capó. Crowdsourcing human-based computation for medical image analysis: systematic literature review. Health Informatics Journal, 26(4):2446-248) 2020.
Anshuka Rangi and Massimo Franceschetti. Multi-armed bandit algorithms for crowdsourcing systems with online estimation of workers' ability. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS 2018), pages 1345-1352, Stockholm, Sweden, 2018. International Foundation for Autonomous Agents and Multiagent Systems.
Vikas C. Raykar, Shipeng Yu, Liangliang Zhao, Gilbert H. Valadez, Christopher Florin, Luca Bogoni, and Lawrence Moy. Learning from crowds. Journal of Machine Learning Research, 11(pr)): 1297-1322, 2010.
Fouad Trad and Ali Chehab. To ensemble or not: Assessing majority voting strategies for phishing detection with large language models. In International Conference on Intelligent Systems and Pattern Recognition, pages 158-173. Springer, 2024.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. In Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), pages 5776-5788. Curran Associates, Inc., 2020. URL https: http://aaxvvvvvaa org/ htngg/ab/20002.. 10957
Jacob Whitehill, Ting-fan fan Wu, Jacob Bergsma, Javier Movellan, and Paul Ruvolo. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems, volume 22, pages 2035-2043, 2009.
Han Yang. Mingehen Li, Huixue Zhou, Yongkang Xiao, Qian Fang, and Rui Zhang. One Ilm is not enough: Harnessing the power of ensemble learning for medical question answering. medRxiv, 2023.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances Neural Information Processing Systems (NeurIPS) 28, pages 649- 657, 2015.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main scope of the paper is reducing costs while maintaining user defined threshold on accuracy compared to majority voting with all LLMs, which is described in the Abstract and Introduction. In the abstract, to the full majority voting baseline on two datasets, which are MMLU and IMDB. These claims are backed by experimental results in $ While assume the responses of LLMs to be independent of each other in the design of our algorithm, we do not have this assumption in the experiments, and hence claims do not depend on this assumption.
only claim performance gains compared
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in the Limitations Section ($5).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate 'Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specificattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
13

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We only have minor theoretical result, which is Lemma 2.1 provided the independence assumption on the outputs of LLMs the statement of the lemma, and also provide the proof of the lemma in Appendix A Note that this is minor result to be used in the design the algorithm.
We have
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided the pseudo-code of our algorithm, and the parameters used are listed in the Experiments section (4.) Additional details on the experimental setup, such
as the prompt template for querying the LLMs, are available in Appendices D and The datasets used in the experiments are publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
14

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
While do not directly provide data code, experimental results
easily be reproduced. We have provided the pseudo-code of our algorithm in Algorithm [ and have provided details on the experimental setting in Experiments Section ($ 4). Dnd
Appendices D and E The datasets we used and the LLMs we queried are publicly available. Based these, we believe results can easily be reproduced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr co/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
//npsscccppullcc/eeeecc http://ggu..eeeoooouuummmiibiii
/CodeSubbeeii ssionPolccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We use online learning in our paper, and do not train any models beforehand. We have provided the hyperparameters used in each experiment in its relevant section. Datasets used in experiments are publicly available. We provide most of the important details regarding the experiments in our Experiments Section ($ and we also provide additional details in Appendices and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
15

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our experiments involve labeling dataset with a given method, and for each method we record the average accuracy relative to ground truth labels, and the cost for entire dataset. Since we have only one result per method, we do not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer YYe' if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: LLMs were queried using online LLM providers. The experiments (apart from querying LLMs) do not consume much computer resources, and were performed with personal computer.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://pw...aoooo..ccoo/ppo/aaaa
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics, and we confirm that our paper conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
16

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
This paper considers optimizing the cost of LLM--ased dataset labeling. Since we only introduce the optimization methods, and use only publicly available LLMs and datasets, we do not expect any societal impact of this paper.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No data or model is released with this paper, and the datasets used in our experiments are publicly available.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
17

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We provide the appropriate citations for the LLMs and the public datasets that used in the paper in the Experiments Section (
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
18

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: Usage LLMs is core component our paper as our paper studies cost optimization of LLM-based dataset labeling. In our Experiments Section ($ 4) and in Appendices
we provide in detail how LLMs are queried to generate labels for the D and
datasets.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
19

---

A Proof of Lemma 2.1
In order to find lower bound the probability of correct labeling, consider the worst-case where each LLM's probability of correct labeling is exactly equal its lower bound. Hence, assume that each LLM Define the random variable Z; ~ Bernoulli L,) to represent whether model I correctly labels the data instance, where E[Z,] 2 L;- The total weight of LLMs that output the correct label is given by:
AA produces correct label with probability L; independently of other models.
WA.A = Σ
Wi Zz.
(5)
iE
and the total weight of all LLMs in A is:
WA Σ
Wi-
(6)
iEA
Majority voting yields the correct label if the cumulative weight of correctly labeling LLMs exceeds half of the total weight, i.e. when
WA Wo.A
2
Hence, )A(L, can be expressed
(A( <PWWA>
WA 2
(8)
To compute this probability, we consider all possible label correctness outcomes for the subset A. Let S A denote the subset of LLMs that produce correct labels, while A A S corresponds to those that produce incorrect labels. The probability of this joint outcome under the independence assumption is
Ps (w(Lww)
II T II (1-4)) iES
JEAYS
Summing over all subsets S A for which the total weight of correctly labeling models exceeds half the total weight gives the desired result:
w)
Σ
II
II (1-L,).
(10)
SCA
iES
JEAiS
EreS
B Estimating the Shape Parameters of the Beta Distribution
In this section, we present two methods for estimating the shape parameters of the Beta distributions used in CaMVo. The first is maximum--lkellhooe estimation (MLE) approach that yields a closed- form system of equations, while the second is an efficient approximation based on the method of moments. Due its computational practicality, the second method used in experiments.
Maximum-likelihodd Estimation. ar, and Bi,1 for the Beta distribution Rota,(as,1,,..1) corre- sponding to LLM 1, can be estimated by maximizing the log- ligelikkh:hoo::
((.,,1,8111) (at,1 1) (e.. (Br, 1) Σ In(1 qi(e.. t In B(a,,, 81,1)
(11)
Taking derivatives with respect to a,1 and 8r,1 and setting them to zero yields the MLE system:
ae:
In (e,, 8) - (bb(a,,)) vbaas,1 +B.1))
(12)
00g,1
0
In(1 (n(1-g(ee,,)) 8)) (v.B..) - ((s,,1 + (r,1))
(13)
03,1
where w-)) the digamma function (xx) = 4 In [(x). Solving this system yields the MLE
estimates for the parameters of each LLM. However, these equations are nonlinear and hence solving them can computationally expensive. To address this, we employ an alternative estimation procedure based on the method of moments.
20

---

This approach provides computationally efficient and sufficiently accurate alternative for parameter estimation and is used in our experimental pipeline in $ 4 For each LLM 4, we compute sample statistics separately for rounds in which 4 output matched the predicted label, and the rounds in which did not match. Let S = {s: hiss = 1, S i} be the set of rounds s until t where hi. = 1. The empirical mean and variance for each case can be computed as:
mfooenssofmmomnnss.
Q1,1
Σ
Ri.o (e,))
u
Σ (gi,ole.) 1t,11
(14)
[1 sESI,
sESr,
9:,0
(t,)
1/.0 t |
(g),) - -q.u)²
(15)
t |5
NE NE(A)
Using the empirical means and variances, define:
1/,1 0s,1 (1 k..1) 1,
14.0
(16)
$2,0
We estimate the Beta distribution parameters using the following proposition.
Proposition B.1. Let Beta(a, 3) be a Beta-distributed random variable with unknown parame- ters a and B, and let ( qn) be observed samples with sample mean m and variance s? Then, the method-of ofmmomens estimates are:
B = (1 mm) v, where v/ m(1 - m)
â = m
(17)
g2
Proof. The Beta distribution has mean and variance:
Elgl
a
+ B Var[g]
aB
(a + (
Substituting q to Ejg], and s to Var(g); and solving for and 3 yields the expressions for å and 3 as stated.
Using Proposition B.1 the parameters can be updated as
0,,1 = fi, 1/1
(18)
Br1 (t,1)
(19)
0,,0 = f0,0 Mi
(20)
8., k.o) v0
(21)
To ensure numerical stability, we clip small variance values below a threshold > 0 to prevent division by near-zero values.
C The Baseline Algorithm
The pseudocode of the Baseline Algorithum is provided below in Algorithm
Algorithm 2 Baseline Algorithm (Online Weighted Majority)
1: Input: The set of L.MMs [K] dataset to label D
2: for each round = 1, 2, ... do
3: Query all LLMs: Vs.t (xxx)
4: De < maxmmaX ΣK (aet: (Vi, = m)
5: Generate rewards for LLMs: Ti,t 1 (yi, = 5r)
6: Update LLM weights: (t).:(t) Eimi 1 N..
7: end for
21

---

D Supplementary Details for Experiments on the MMLU Dataset
This section provides additional details regarding our experimental setup for the MMLU dataset.
First, to improve computational efficiency, we approximate the confidence score dا (L, w) using the cumulative distribution function (CDF) of the Beta distribution rather than the closed-form expression in Lemma 2.1
(A(L,w) = 1 Fuea (0.5; WL,A, WA WL,A), where o 3) is the CDF of a Beta(o, (3) distribution, WL.A = ΣiEA Wi L; and WA =
EiEA Wi.
The Beta distribution parameters are updated online using the method-of-mooents estimator defined Eq. (21). with a regularization term 10
We query LLMs using a consistent format tailored to the multiple-choice structure MMLU, The standard prompt template is shown below:
Query Format for MMLU Dataset
System: Select the correct answer. Answer with A, B, C, or D only.
User: Question: <question>
A. <choice-A>
B. <choice-B>
C. <choice-C>
D. <choice-D>
Answer:
the LLM API does not support a system instruction prompt, the instruction is prepended directly to the user message. An example query, using actual MMLU question, is shown below:
Example Query for MMLU Dataset
System: Select the correct answer. Answer with A, B, C, or D only.
User: Question: Find the degree for the given field extension Q( 2, v3, /18) over Q
A. 0
B. 4
C. 2
D. 6
Answer:
We apply single random permutation the dataset and maintain this identical ordering across all methods to ensure fair and consistent comparison (except in experiments in Appendix [D.] where analyze the sensitivity of CaMVo to dataset ordering).
D.1 Additional Experimental Results
To facilitate a more detailed analysis of the results in Table 3. for each label category in Table we report precision, recall, and Fl score as additional performance metrics. For comparison, the corresponding results for majority voting are presented in Table These metrics allow to evaluate category-win wise performance. The results indicate that category is slightly more challenging to label, as reflected by its consistently lower scores across all metrics. Furthermore, as the confidence threshold decreases, all three metrics degrade across label categories, with more pronounced decline observed for category 0, likely due to its possibly higher intrinsic difficulty.
Figure illustrates CaM cumulative average accuracy (blue) and cost (red) over rounds for kmin under different confidence thresholds to explore CaMVo's learning dynamics various 8 values. The green line marks each d-specific target accuracy. In all cases, the algorithm begins by querying larger, more expensive ensembles gather reliable performance estimates, then swiftly transitions to cheaper subsets once the lower-confidence bounds stabilize. This yields steep decline
22

---

Accuracy and Cas Ove Bounds
1034
Accurnsy and Oet Over founch
Acoracy Con Over
Accese No
Asursy (X)
2000 400 6000 EXC 1860 L00 1963
2000
4000
50:00
9000
1840
1250
16600
200
450
630
5X0
Li0 14030
102
Ncurncy Tergel
Accres
Jccevy
34600
18090
Aeru and Cant Ouer Sourdh
Fer and East Over Sounds
Acnunny and Coe ConrBoance
300
305
160
Accutiyy auget 500-
aay Larel
55
Accuragy and Ove fourdh
Accrrsy Over Tounds
Acorady Dest Boyrd
Acevca 36
kcnss N0
Onl
0ol
50
2000
400
620
3000
10000
1250
24000
2000
4000
6000
3000
2000
13300
34000
0
2000
4006
5000 000
1000
10000
1455
Figure 3: Cumulative average accuracy (blue) and cost (red) of CaMVo (kiin 1) on the MMLU dataset across rounds for various confidence thresholds d. The green line marks each ô-specific target accuracy.
in cost concurrent with accuracy settling at a value above the target line. A temporary dip in accuracy around round 1,000 appears consistently, reflecting cluster of harder instances in our fixed data shuffle.
For high thresholds (6 0.99), CaMVo predominantly queries the full ensemble, producing an almost linear cost profile. At intermediate levels (5 = 0.98, 0.975), cost initially falls but momentarily rises when accuracy dips below the target, prompting the algorithm select slightly costlier subsets to regain the required confidence as the accuracy estimations of individual LLMs decrease. When 5 < 0.965, the cost curve decreases monotonically and converges to stable minimum, indicating rapid identification of the context-specific optimal subsets.
Finally, for low thresholds (ō 0.85, 0.80), observed accuracy significantly exceeds the target owing to the performance gaps among individual LLMs: no model has true accuracy between 70% and 80%, hence CaMVo's conservative lower-bound estimates result in consistently higher realized accuracy. Overall, these results underscore CaMVo's ' capacity to balance exploration and exploitation, quickly pinpoint cost-effective ensembles, and reliably meet -see-speedfeed accuracy requirements.
To evaluate Vo'so's robustness to input ordering, Figure 4 shows the mean cumulative average accuracy and cost trajectories (solid lines) for 8 = 0.96, kinin 1, averaged over 20 random shuffles of the MMLJU dataset. Shaded bands denote one standard deviation. Although the accuracy band initially wide due to the exploration of CaMVo, and also different mixes of casy and hard examples
23

---

across the shuffles; it contracts rapidly, underscoring CaMVo's 's consistent attainment of the target accuracy across permutations. The cost band also narrows over time, illustrating stable convergence to low-cost ensembles. Notably, the accuracy band remains much tighter than the cost band, since CoMVo targets above the accuracy threshold but does not optimize for fixed cost.
Accuracy and Cost Over Rounds
100.0
Acturacy
97.5
Accuracy "Trget for 5 0.96
95.0
Accuracy (8))
92.5
90.0
87.5
85.0
82.5
60.0
2000
4000
6000
8000
10000
12000
14000
Round
Figure 4: Mean (solid lines) and one-staanarrdddeviatinn bands (shading) of CaMVo's cumulative average accuracy (blue) and cost (red) over 20 random shuffles of MMLU (6 0.96, Kinin 1). The green line indicates the accuracy target of 84.65% for 5 = 0.96.
Aecurnry Ouet Sourds
Aecurnyy Sounds
Accumey and Cast
300
10C
Aiaady beget bi0
Aira ons Sagel Mih-DAs
Araeiiyy Seget brc
Acouracy and Det Ovrr Sourds
Accurecy and Det Over Soundh
Acouracy and Cest Cost Bounds
100
300
Accmnor
100
01.
Accanoy Tanget
015
Accasco pareeterdee 1:0
Accano 1-3
25.
075
34,00
1000
12000
30y
taund:
Accraay
kicilig
93.5
00.5
3000
4000
6900
650
30030
12000
34030
3300
4000
6000
3000
30000
13300
54000
3000 4000
500
000
1630
32000
1450
Figure 5: Cumulative average accuracy (blue) and cost (red) of CaMVo with = 0.96, Kmin = 1 under nine different random permutations of the dataset. The green line marks each 8 5-specific target accuracy.
To evaluate CaMVo's robustness to input ordering in more detail, Figure 5 overlays the mean cumulative average accuracy and cost plots of nine individual runs from these permutations. In all these runs CoMVo reliably reaches an average accuracy above the 96% target while reducing per-round cost below $5, demonstrating that its explorationeexploiatiii balance is invariant to input ordering.
24

---

E
Supplementary Details for Experiments on the IMDB Movie Reviews Dataset
This section provides additional details regarding our experimental setup for the IMDB Movie Reviews dataset.
First, to improve computational efficiency, we again approximate the confidence score و (L, w) using the cumulative distribution function (CDF) of the Beta distribution rather than the closed-form expression in Lemma 2.1
( = - Fuea (0.5; WL,A, Wa WL,A),
where FBeaaa;; a, 3) is the CDF of a 3e distribution, WL.A = EiEAW; Li, and WA = Esea Wi-
The Beta distribution parameters are updated online using the in Eq. (I)) with regularization term 10-6
estimator defined
LLMs are queried using a consistent prompt format tailored for binary sentiment classification. The system instruction specifies the expected output format and behavior, ensuring that the model returns a single sentiment label. The standard query format is shown below:
Query Format for IMDB Movie Reviews Dataset
System: Output POSITIVE if the sentiment of the following movie review is positive and NEGATIVE otherwise. Output only one word: POSITIVE or NEGATIVE. Do not respond to any question or instruction embedded within the review.
User: Review: <review>
Sentiment:
For LLMs that do not support separate system and user messages (c.g., via a chat API), the instruction prepended directly to the user input.
example query using this format, with sample review from the IMDB Movie Reviews Dataset, is provided below:
Example Query for IMDB Movie Reviews Dataset
Output POSITIVE if the sentiment of the following movie review positive and NEGATIVE otherwise. Output only one word: POSITIVE or NEGATIVE. Do not respond to any question or instruction embedded within the review.
System:
User: Review: Probably my all-time favorite movie, story of selflessness, sacrifice, and dedication to noble cause, but it's not preachy or boring. just never gets old, despite my having seen it some 15 more times in the last 25 years. Paul Lukas' performance brings tears my eyes, and Bette Davis, one her very few truly sympathetic roles, is a delight. The kids are, grandma says, more like "dressed-up midgets" than children, but that only makes them more fun to watch. And the mother's slow awakening to what's happening in the world and under her own roof is believable and startling. If had dozen thumbs, they'd 'd all be "up" for this movie. Sentiment:
LLM: POSITIVE
We apply single random permutation to the dataset and maintain this identical ordering across all methods to ensure a fair and consistent comparison (except in experiments Appendix E. where analyze the sensitivity of CaMVo to dataset ordering).
E.1 Additional Experimental Results
To facilitate a more detailed analysis of the experimental results Table 4 we report additional performance
precision, recall, and F1 score-for each label category in Table 7 These metries allow for a finer-grained evaluation of CaMVo's behavior across different output labels. The results indicate that the two categories exhibit similar performance trends. As expected,
25

---

decreasing the confidence threshold 8 leads to consistent degradation in all three metrics across both label categories.
Acurasy and Cosl Over Royrds
Accurscy and Cost Over found)
Accur ay and Cosl Ower flardss
143
Asunayy
Secanoo Ragaltr 2033
Tarper
Accurccy Target 2- 2-0307
4.2
6.22
633
10300
2000
2040
5009
32000
30000
30000
40000
5000
5000
X000
4000
Acuugg 760
162
Accuncyyn
26
30000
13003
35000
60000
10900
2D003
1300
Accuracy and Cost Over Sourds
uecy and Over Sounds
Acouracy Conl Coer Rounds
10C
300
160
Acauscy laged fo 5-000
Aco my lagel 5007
Accatiy 1ag
Accursoy and Doot Over tourds
Acrrray and Dast Over tourds
Acauery Terent 83635
20000
20000
2000
40000
50000
33000
20000
30003
40000
50000
Figure 6: Cumulative average accuracy (bluc) and cost (red) of CaMVo (kmin 1) the IMDB Movie Reviews Dataset across rounds for various confidence thresholds ó. The green line marks each 8-specific target accuracy.
Figure 6 visualizes CaMVo's learning trajectories on IMDB for kmin = 1 across all the confidence thresholds values reported in Table In all cases, CaMVo begins by querying larger, higher-cost ensembles to obtain reliable performance estimates, then rapidly shifts to cost-optimal subsets once the lowfr-cone bounds converge. This transition yields a sharp decline in cost while maintaining accuracy above the target line.
At the extreme threshold 8 0.999, CaMVV predominantly queries the full ensemble, resulting in a near-linear cost profile until about round 25,000. For 5 S 0.98, cost quickly converges to a stable minimum, reflecting identification of the least-expensive subset that meets the target accuracy, Across all plots, CaMVo achieves or exceeds the respective accuracy target. For very high thresholds (8 2 0.995), final accuracy hovers just above the threshold, as expected; as ò decreases, the accuracy surplus grows. Below d = 0.96, accuracy plateaus at approximately 94.06%, corresponding to the performance of the single cheapest model ('1lama-3.1-8h)
Overall, these results demonstrate CaMVo's ability to balance exploration and exploitation, swiftly discover cost-effective subsets, and reliably satisfy the target accuracy requirements.
To assess CaMVo's sensitivity to dataset ordering, Figure the mean cumulative accuracy and cost curves (solid lines) for d 0.995, Amin 1, averaged over 20 random shuffles. Shaded regions
26

---

Accuracy Over Rounds
100
Accuracy (%)
Cost
Target 6 0.995
96
Accuracy (%)
96
Coot
93-
10000
20000
30000
40000
50000
Round
Figure 7: Mean (solid lines) and one-standardddeviation bands (shading) of CaMVo's cumulative average accuracy (blue) and cost (red) over 20 random shuffles MMLU (6 0.995, kimin = 1). The green line indicates the accuracy target of 95.14% for 8 0.995.
Accurooy Loot founds
ACCUTZOY tounds
Acdracy
Actunti
30000
20000
3000
40000
3000
33000
20000
3000
40090
5000
65
10000
13003
30000
40900
5300
Aecurary and Cart Over Sourds
Aeeurnyy Over Sounds
Acnunay Coe Geer Rounds
300
30C
102
Kiacin lagel SISTER
55
Got
Accuragy and Dost Over fourds
Accuraty and Dost Over Tourdh
Acouracy and Cost Cosr Bounds
100
Acaurcy Tagget 5w0
Asule Tagel 0230
Accanc 1
20003
30000
30000
2003
40000
10000
2003
2000
Figure 8: Cumulative average accuracy (blue) and cost (red) of CaMVo with 0.96, kimin 1 under nine different random permutations of the dataset. The green line marks each 8 5-specific target accuracy.
indicate one standard deviation. Although the accuracy band starts wide, which reflects both the initial exploration and also the varying mixes of 'casier' and 'harder' instances; this rapidly contracts over time, confirming CaMVo's reliable attainment of the target accuracy across permutations. The cost band likewise narrows, demonstrating stable convergence to low-cost ensembles. Notably, the accuracy variability remains much smaller than the cost variability, as CaMVo does not optimize toward fixed cost.
Figure g further investigates ordering effects by overlaying nine individual runs from these permu- tations. In every run, CaMVo exceeds the 95.14% accuracy target while driving per-round cost strategy, and the resulting cost-accuracy
below $2.10 corroborating that its exploration-exploiaa performance is effectively invariant to the input sequence.
27

---

Experiments on the AG News Classification Dataset
We further evaluate Co on the AG News Classification Dataset Zhang et al. 2015). which contains news articles gathered from over 2,000 online sources. Each article is categorized into one of four topies: World, Sports, Business, or Science/Techin Sieene//ecchology. From the original training set of 120, 000 samples, uniformly sample 50, 000 instances for our experiments. As before, we compare CaMVo against each individual LLM, a full-ensemble Majority Vote, and the Baseline Method (Algorithm 2)
Models and setup. We employ Anthropic's Claude 3-7 Sonnet, and Claude 3-5 Haiku Anthropic 2024];
OpenAl's 03-mini, ol-mini, and GPT--4-mini OpenAl [0244] and Meta's LLaMA-3.3 and
LLaMA-3.1 Meta (0024). All queries use temperature = 0.3 and top-p 1, where applicable. We
extract 384- 384-dimensional contextual embeddings with 41l-MiniLM--LG
Wang et al. (2200 and
approximate the confidence bound A L, w) via the Beta-CDF, as in $4. To improve computational
efficiency, again approximate the confidence score (L, w) using the cumulative distribution
function (CDF) of the Beta distribution rather than the closed-form expression in Lemma
(A w) = 1 FBe (0.5; WL.A, WA - WL,A);
where (a; a, is the CDF of a Beta(a, 3) distribution, WI..A EiEA wi Ta, and W EiEA W4.
The Beta distribution parameters are updated online using the fathod-of-momente estimator defined in Eq. (1), with regularization term 10-6
LLMs are queried using consistent prompt format tailored for categorical output. The system instruction specifies the possible categories that the news article can be classified into, and the expected output format and behavior, ensure that the model returns single category. To form each instance for the User prompt of the L.M,, we concatenate the article's title and description provided in the dataset. The standard query format is shown below:
Query Format for AG News Classification Dataset
System: Classify the following news article WORLD, SPORT, BUSINESS, or SCIENCE/TECHIE BENNIC///TCHNLLYY... Respond with only the chosen category. the article is ambiguous, select the closest matching category.
User: <Title> <Description>
For LLMs that do not support separate system and user messages (e.g., via a chat API), the instruction prepended directly the user input.
An example query using this format, with a sample review from the AG News Classification Dataset, provided below:
Example Query for AG News Classification Dataset
System: Classify the following news article as WORLD, SPORT, BUSINESS, or OCIENCE/TECHNOLOOG, Respond with only the chosen category. If the article is ambiguous, select the closest matching category.
User:
Celtie beat Dunfermline 2-0: Celtic regained the top spot in the Scottish Premier League after 2-0 victory away to Dunfermline. The result leaves Celtic top-of-the-table with 44 points from 18 games, single point clear of arch-rivals Glasgow Rangers. Sentiment:
LLM: SPORT
We apply single random permutation to the 50, 000 sampled data instances from the dataset and maintain this identical ordering across all methods ensure fair and consistent comparison.
Results. Table 8 (Right) reports the accuracy and cost (in dollars per million input tokens) of each LLM and the two baselines. Once again, the baseline underperforms the best individual model 185.68% vs. 87.16%) despite incurring a substantially higher cost. This performance gap may stem
28

---

from the inherent ambiguity of the dataset, certain news articles may belong to multiple categories, making consensus through majority voting more difficult to achieve.
Table presents CaMVo's accuracy-cost trade-off across various thresholds and kimin E {1,3), 3). 0.7, An 5, and Λz. 1; and the Target Accuracy is computed similarly as x 85.68% Across all configurations, CaMVo meets or exceeds its target accuracy. Further, CaMVo achieves less than half the cost (when ò 0.99 and kiiin 1) at a slightly lower accuracy of 56%18% compared to the baseline, confirming its practicality for large-scale sentiment annotation without any pre-training or ground-truth labels.
Vo' hyperparameters
Accurasy Comparison
Figure 9: Cost-accuracy trade-off for AG News Classification Dataset: gray dots show every LLM subset via weighted majority voting, yellow dots trace their Pareto-optimal frontier, blue markers are CaMVo at kminn 1, green markers at kmin = 3, cyan markers denote the individual single LLMs, and the red marker denotes the Baseline Method.
Figure 9 presents the analogous comparison of Figure (Left) on the AG News Classification Dataset. As before, gray points and the yellow cyan markers show individual single LLMs, while blue and green markers plot CaMVo at kmin = and 3, respectively. The red marker denotes the Baseline Method. Again, CaMVo closely matches the Pareto front the low-cost regime (cost < 0.8), but lags behind in higher-cost regions.
[igft points show all possible subset combinations,
Figure 10 visualizes CaMVo's learning trajectories on the AG News Classification Dataset for kmin across all the confidence thresholds values reported in Table Similar to experiments on the other datasets, CaMVo begins by querying larger, higher-cost ensembles to obtain reliable performance estimates, then rapidly shifts to cost-optimal subsets once the lower-confidence bounds converge.
Further, to assess CaMVo's sensitivity to dataset ordering. Figure plots the mean cumulative accuracy and cost curves (solid lines) for = 0.99, kmin = 1, averaged over 20 random shuffles. Shaded regions indicate one standard deviation. Similar to the experiments in other datasets, although the accuracy band starts wide, which reflects both the initial exploration and also the varying mixes of "easier' and 'harder' instances; this contracts over time, confirming CaMVo's reliable attainment of the target accuracy across permutations. The cost band likewise narrows, demonstrating stable convergence low-cost ensembles.
Figure 12 further investigates ordering effects by overlaying nine individual runs from these per- mutations. every run, CaMVo exceeds the 84.82% accuracy target while driving per-round cost below $3.3, corroborating that its exploration-expie exploitation strategy, and the resulting cost cost-accuracy performance effectively invariant to the input sequence.
G Extension of CaMVo to Correlated LLM Outputs
In the original CaMVo algorithm, we assume that LLM outputs are independent. To relax this assumption, we introduce the Correlated CaMVo (CCaMVo), an extension that models dependencies between L.M outputs while maintaining the practicality of the original method. Unlike CoMVo, which relies on Lemma 2.T for confidence estimation, CCaMVo estimates subset confidences via Monte Carlo simulation over correlated LLM correctness samples.
29

---

Accurecy
Accurecy Col
Accurccy and Ove Boadss
Acuuss
41930000044
475000000304
175
20030 3000
2000
4000
3000
22.00
1X00
3000
4000
X000
32000
20000
30000
+0.00
3006
2.55
Accrec
Accuso 5-0
24
435
135
165
0.75
15000
Accuracy and Cae Bearde
Accuracy and Coet Ouer Bourds
Aerurnyy and Cod Doee Rrarsts
300
205
160
130
0.0
Acoracy and
Over Noards
Acrer Ail
X
1.0
10000
20000
2000
40000
12000
Figure 10: Cumulative average accuracy (blue) and cost (red) of Co (kimin = 1) on the AG News Classification Dataset across rounds for various confidence thresholds 6. The green line marks each 8-specific target accuracy.
CCaMVo differs from CaMVo by incorporating three additional components. The first is an online correlation estimator (Algorithm(3, (3), which incrementally estimates pairwise correlations between LLM accuracies as new rounds of predictions arrive. Let μ4 and o; denote the empirical mean and standard deviation of LLM 1,'s historical accuracy, respectively, and let C' denote the estimated correlation matrix across LLMs. For each LLM, the algorithm maintains running counts n,, the number of times LLM has been selected up that round. Further, for each pair (i. of LLMs. Nij. the co-occurrence counts that designate the number of times LLMs 4 and have been selected at the same round; and Mij. the accumulated deviation statistics, are maintained. At each round, updates the univariate statistics (14; o.)) for selected L.MMs and adjusts the terms M,j using Welford's numerically stable online formulas. Pairwise correlations are computed as P.j = cov (o,0,, where cov. /Ngg 1) and clipped to [-1, 1] for numerical stability.
The second component (Algorithm uses these estimated correlations to generate correlated LLM correctness samples via a Gaussian copula. Specifically, we first project the estimated correlation matrix Ceu to positive semi-definite matrix Cpsd and draw n samples Zi..... . Z, N(OO.. K. Cpsa). Applying the standard normal CDF elementwise yields uniform samples U, = & (Z.;), which pre- serve the dependence structure Cpa-- Binary correctness samples are then obtained by thresholding each variable as X;; Il (Uiz < {tj), ensuring marginal correctness probabilities of μyj While the
binary correlations differ slightly from Gpsa due to thresholding, this approximation offers a practical trade-off between accuracy and computational efficiency.
30

---

Accuracy and Cost Over Rounds
Accuracy (%)
Cost
Target 6
110
4.0
100
Accurary (%)
70
25
10000 20000
30000
40000
50000
Round
Figure 11: Mean (solid lines) and one-standard-deviatio bands (shading) of CaMVo's cumulative average accuracy (blue) and cost (red) over 20 random shuffles using the AG News Classification Dataset (Ó 0.99, Kaii 1). The green line indicates the accuracy target of 95. 95.14% for 8 = 0.995.
Recusacy
Accoracy
Aasunry Cosl
Sclanip
30000
20030
3000
4000
50300
33000
33000
30.30
4000
5030
1000
2032
5000
4003
5800
Aecurny Cant Aounds
Over Rourds
and Coel Duar Rourde
300
Accuragy and Dest Over Toundh
Accorery and Dost Over Tourdh
Acoraay and Ceal Over Rognds
Aecnnor Ai
Kcsson
Jecunsy Targe 60-
Acunoo 5-000
57.
15.3
52.5
3000
30000
30000
50300
30000
20000
30000
3000 40003
3000
Figure 12: Cumulative average accuracy (blue) and cost (red) of CaMVo with 5 = 0.99, Amin 1 under nine different random permutations of the dataset. The green line marks each 8-specific target accuracy.
Finally, CCaMVo estimates the confidence cach LLM subset by aggregating majority-vote correct- ness outcomes across the generated samples, averaging over simulated draws to approximate subset confidence. The remaining components of CCaMVo follow the same structure as CaMVo, preserving its efficiency while enabling robust confidence estimation under correlated LLM outputs.
We evaluated CCaM using the same experimental setup as CaMVo. On the MMLU dataset, Table 10 compares CaMVo and CCaMVo for Kmin 1 across varying confidence thresholds 8, while
Table 11 shows the results for kimin 3. Similarly, for the IMDB dataset, Table 12) reports results
for Kmin = 1, and Table 13 presents the corresponding results for Kmin = 3. For the AG News
Classification Dataset, Table reports results for kmin 1, and Table 15 presents the corresponding
results for Kmin 3.
31

---

Algorithm 3 OntineCCorelationSstimaooo
Require: Number of variables K
1,..., K 2: Initialize Mij < D, Nij +- O, and correlation matrix Cij 4 Ili = j]
1: Initialize TAg 0, jAk k oß 0 for all k
3: function UPDATB(T, A)
Step 1: Update univariate statistics
4:
for all E A do
5:
old_1t t- pii
6:
n4 s n; +1
7:
8' f r; old M4
8:
14 - old old_1 44+8/m 8/n;
9:
Ma + Mii + (ri (i)(ri old_
n. > then
11:
- (aa/nni- -I)
12:
else
13:
σ: +
14:
end
15:
end for
Step 2: Update pairwise covariance and correlation
16:
for all unordered pairs (i, in do
17:
5 f r; Hig
8j f rj - Pi
18:
Mij Mis de dj
19:
Mí < Mij
20:
No. +- Nes +1, N8 t No
21:
if Na, > then
cov;j + Mij/(N, - 1)
23:
Pij + clip(covis) (a Oj),, -1,
C. Cj + Pij
25: end
end for
27:
end function
28: function GETTCORRELLTONNNAARRR
29: return C
end function
Algorithm 4 Sampling Correlated Binary Outcomes via Gaussian Copula
Require: Number of samples n, marginal probabilities E R estimated correlation matrix
Cest E R
1: Cnd <- MAKEPSD(Cea)
Project to nearest PSD matrix
2: 7 <- Sample vectors from (0K, Cpsa)
3: U + 4(Z)
> Apply standard normal CDF elementwise
4: for i 1.....,. do
5: for
j = -, L.... K do
6:
X,j < (Un < 1b)
7: end for
8: end for
9: return X
These results indicate that incorporating the copula-bases based simulation to model the correlation does not significantly improve performance on MMLU and yields a modest cost improvement on IMDB, where L.LM outputs exhibit higher correlation. We attribute this to two factors: (1) LinUCB's use of input embedding vectors to predict L.M confidences already implicitly captures correlations among L.M outputs conditioned on the input context, and (2) estimation errors in the correlation matrix reduce the potential gains from explicitly modeling correlations.
32

---

H Experiments on Sensitivity to Correlated LLM Outputs
We conduct a sensitivity study to evaluate the impact of correlation among LLM outputs. Given a target mean accuracy vector l4 = (fi...., and target covariance matrix C E R we generate synthetic LLM labels using a Gaussian copula-based procedure (Algorithm 5 This approach ensures that the outputs of different LLMs are both context-dependent and correlated according to the specified C. By varying C, we analyze how increasing inter-model correlation affects the performance of CaMVo.
In this experiment setting, we denote the number of LLMs by K, the context dimension by d, and the number of rounds by T. Each LLM 4 is associated with target mean accuracy μ4 E (0. 1). The context correlation parameter a [0, controls the degree of shared dependence among LLMs, and O denotes the scale of latent Gaussian noise.
We first transform the target reward correlation matrix C. into valid latent Gaussian correlation matrix R = [pig] that can reproduce the desired binary LLM outputs after thresholding. Specifically, for each pair of LLMs (i,j), we find a latent correlation coefficient Pij such that
Corr [[Z > + (1 ()), -- (1--11,)))) = Cij.
where (Z,, -2) ~ (0, (pi)) and 4 is the inverse standard normal CDF. The resulting matrix R = [Piz] is then projected onto the nearest positive
matrix to ensure numerical stability. This step guarantees that the generated binary outputs exhibit empirical correlations close to C.
Each LLM 4 is associated with context-dependent accuracy weight vector 8, R defined as
0 shared (1 a) uniiuu,
where Oshared (0, Ia) captures shared structure among LLMs, and uniqque NO Ia) introduces model-specific variation.
At each round / = ...... T,
context vector
N ((,0 Ia)
is independently sampled, representing the input context of the data instance. A correlated Gaussian noise vector E (0, R) is drawn jointly across all LLMs. The latent correctness score for each LLM 4 is then computed as
21,4 0, 2t + a Ct,s.
The observed binary correctness outcome is given by
Fr, = 1(2,, > T 10 14)}-
When Ti, = 1, it means the LLM produces the correct label. . This data generation process produces context-output pairs whose expected accuracies close to the target μ4, while the inter-model dependencies exhibit empirical correlations close to C. Note that because of the additive term 0,, which encodes correlations with the context, this method cannot exactly reproduce the target accuracy and covariance matrix values. Although more sophisticated approaches could more precisely match the specified targets, we adopt this formulation for its computational simplicity and because our experiments focus primarily on analyzing the effects of varying correlation strength rather than achieving exact target statistics.
H.1 Experiment Results
In the first experimental setup the diagonal entries of C' representing self-correlation are set to 1, i.c. C₁ = 1, Vi E [K] and off-diagonal entries are to a parameter C, = Y, Vi + j that represents the pairwise correlation. We sweep 7 across a range of values to evaluate how varying correlation strengths impact performance. The results are summarized in Table [6] which reports the accuracy and cost of CaMVo under kmin = 1 and Kmin 3, alongside the accuracy of majority voting and the target accuracy, across different confidence thresholds 8 and correlation parameters C.. The synthetic data are generated with parameters d = 5, 0.5. and a 0.1, while the costs and mean accuracies of the L.MS are taken from the MMLJ experiments, given by [0.05, 1.1, 0.59, 2.5, 3, 1.1, 0.8) and µ [0.6801, 0.8482, 0.8170, 0.8358 0.8565, 0. 0.8592, 0. 6449). Note that the cost of majority
33

---

Algorithm 5 Gaussian Copula-based Correlated Data Generation
Require: Number of rounds T, number of LLMs K, context dimension d, target mean rewards {ui-=1 target correlation matrix C RK*X context correlation parameter a, noise scale Ensure: Contexts (at binary rewards {rt,i)t,i
1: function GENERATEDATACT, K, K,d, d,, C,
2: Compute latent Gaussian correlation matrix R + CALIBRATECOPULNGU C)
3: Draw Oshored NO0 Ia)
4:
for = I..... K do
5:
Draw uunique, (O, Id) Ga f- O Iaharod + (1 a) unique.
6:
7:
end for
8:
for t= 1, ......
T do Sample context Dt NO Ia)
9:
10:
Sample c NN0, R)
for a = l....... do
12:
2ta t 02 xt E1,0
13:
Nt. f I[2r, > 1 (1 ma))
end for 15: end for
return ((ax,
17:
end function
18: function CALIBRATECOPULAAG C)
19:
for each pair (i.j) with < do
20:
Find Pij such that
Com 112 > 41 1i),,(z x- pb)}} Cij
where (Z, Z,) (0,
21:
end for
22: Construct R= [pt]] and project to nearest PSD matrix
23: return I
24: end function
voting is fixed at $9. 14 per million tokens across all settings. Tables 17 and present the results for 0.2 and 0.4, respectively.
The results indicate that the accuracy of CaMVo generally decreases as C, increases; similar trend is observed for majority voting. Importantly, CaM is able to maintain the target accuracy across all tested correlation levels, demonstrating its effectiveness even settings with correlated LLM outputs.
In the second experimental setup, we use the correlation matrix
11.00
0.445
0.147
0.423
0.211
0.306
0.320
0.445
1.000
0.373
0.448
0.242
0.420
0.317
0.147
0.373
1.000
0.265
0.172
0.118
0.142
C
0.423
0.448
0.265
1.000
0.301
0.436
0.325
(22)
0.211
0.242
0.172
0.301
1.000
0.158
0.180
0.306
0.420
0.118
0.436
0.158
1.000
0.388
L0.320
0.317
0.142
0.325
0.180
0.388
1.000_
so that, unlike the first experiment, the correlations between different LLMs are heterogeneous. We sweep the confidence threshold and the parameter to evaluate how the strength of correlation between LLM outputs and the context affects accuracy. The results are summarized in Table [9 which reports the accuracy and cost Co for kinin = and kmin = 3, alongside the accuracy of majority voting and the target accuracy, across varying 5 and Across all experiments, d = 5 and 0.5, while the costs and mean accuracies the L.MSs are taken from the MMLUU dataset as in the previous experiment. The cost of majority voting remains fixed $9.14 per million tokens.
34

---

As expected, the accuracy of VaMVo increases with larger a, higher correlation with the context allows LLM outputs better reflect the arm rewards. contrast, the aceuracy of majority voting de- creases with increasing o, since it increases correlations among LLMs through the context. Crucially, CaMVo consistently maintains the target accuracy across all instances, demonstrating robustness even under correlated LLM outputs.
35

---

Precision,
Recall,
F1 Score,
Precision,
Recall,
F1 Score,
8
label
Kmin
1
Kmin =
Kmin
Kmin 3
Kmin 3
Kmin = 3
0.99
0
0,87
0,86
0.87
0.87
0.86
0.87
0.99
0.87
0.89
0.88
0.87
0.89
0.88
0.99
0.88
0.9
0.89
0.88
0.9
0.89
0.99
0.91
0.89
0.9
0.91
0.89
0.9
0.98
0.88
0.86
0.87
0.88
0.86
0.87
0.98
0.88
0.89
0.88
0.88
0.89
0.88
0.98
0.88
0.9
0.89
0.88
0.9
0.89
0.98
0.91
0.9
0.9
0.91
0.9
0.9
0.975
0.88
0.85
0.87
0.88
0.85
0.87
0.975
0.87
0.89
0.88
0.87
0.89
0.88
0.975
0.88
0.9
0.89
0.88
0.9
0.89
0.975
0.91
0.9
0.9
0.91
0.9
0.9
0.97
0.88
0.85
0.86
0.88
0.85
0.86
0.97
0.87
0.89
0.88
0.87
0.89
0.88
0.97
0.88
0.9
0.89
0.88
0.9
0.89
0.97
0.91
0.9
0.9
0.91
0.9
0.9
0.965
0.89
0.85
0.87
0.89
0.85
0.87
0.965
0.86
0.88
0.87
0.86
0.88
0.87
0.965
0.88
0.9
0.89
0.88
0.9
0.89
0.965
0.91
0.9
0.9
0.91
0.9
0.9
0.96
0.88
0.85
0.86
0.88
0.85
0.86
0.96
0.86
0,88
0.87
0.86
0.88
0.87
0.96
0.87
0.89
0.88
0.87
0.89
0.88
0.96
0.9
0.89
0.9
0.9
0.89
0.9
0,955
0.87
0.84
0.86
0.87
0.84
0.86
0.955
0.86
0.88
0.87
0.86
0.88
0.87
0,955
0.87
0.88
0.88
0.87
0.88
0.88
0.955
0.89
0.89
0.89
0.89
0.89
0.89
0.95
0.87
0.84
0.85
0.87
0.84
0.85
0.95
0.85
0.87
0.86
0.85
0.87
0.86
0.95
0.87
0.87
0.87
0.87
0.87
0.87
0.95
0.89
0.89
0.89
0.89
0.89
0.89
0.9
0.85
0.82
0.84
0.85
0.82
0.84
0.9
0.84
0.85
0.84
0.84
0.85
0.84
0.9
2
0.85
0.85
0.85
0.85
0.85
0.85
0.9
0.86
0.87
0.87
0.86
0.87
0.87
0.85
0.84
0.82
0.83
0.84
0.82
0.83
0.85
0.81
0.86
0.84
0.81
0.86
0.84
0.85
0.85
0.84
0.84
0.85
0.84
0.84
0.85
0.87
0.85
0.86
0.87
0.85
0.86
0.8
0.8
0.82
0.81
0.8
0.82
0.81
0.8
0.76
0.87
0.81
0.76
0.87
0.81
0.8
0.85
0.8
0.82
0.85
0.8
0.82
0.8
0.88
0.8
0.84
0.88
0.8
0.84
0.75
0.59
0.73
0.65
0.59
0.73
0.65
0.75
0.71
0.65
0.68
0.71
0.65
0.68
0.75
0.76
0.63
0.69
0.76
0.63
0.69
0.75
0.72
0.74
0.73
0.72
0.74
0.73
0.7
0.58
0.73
0.65
0.58
0.73
0.65
0.7
0.71
0.64
0.67
0.71
0.64
0.67
0.7
0.75
0,63
0.68
0.75
0.63
0.68
0.7
0.71
0.74
0.73
0.71
0.74
0.73
Table 5: Precision, recall, and F1 scores of Vo with kmin and Kmin across different confidence thresholds 5 for each label category on the MMLU dataset.
36

---

label Precision
Recall
F1 Score
Maj. voting
0.87
0.86
0.86
Maj. voting
0.87
0.88
0.88
Maj- voting
0.88
0.89
0.89
Maj- voting
0.91
0.89
0.9
Table 6: Precision, recall and Fl score of majority voting over each output category on the MMLU dataset.
Precision
Recall
F1 Score
Precision (kmin 3)
Recall
Fl Score
5
label
(kmin
(kmin 1)
(kmin 1)
(kmin 3)
(kmin 3)
0.999
0
0.96
0.96
0.96
0.96
0.96
0.96
0.999
0.96
0.96
0.96
0.96
0.96
0.96
0.998
0
0.95
0.95
0.95
0.95
0.95
0.95
0,998
0,95
0,95
0.95
0.95
0.95
0.95
0.997
0
0.96
0.95
0.95
0.96
0.95
0.95
0.997
0.95
0.96
0.95
0.95
0.96
0.95
0.995
0.96
0.95
0.95
0.96
0.95
0.95
0.995
0.95
0.96
0.95
0.95
0.96
0.95
0.99
0.95
0.95
0.95
0.95
0.95
0.95
0.99
0.95
0.95
0.95
0.95
0.95
0.95
0.985
0.94
0.95
0.95
0.94
0.95
0.95
0.985
0.95
0.94
0.95
0.95
0.94
0.95
0.98
0.94
0.95
0.95
0.94
0.95
0.95
0.98
0.95
0.94
0.95
0.95
0.94
0.95
0.97
0.94
0.95
0.95
0.94
0.95
0.95
0.97
0.95
0.94
0.95
0.95
0.94
0.95
0.96
0.94
0.94
0.94
0.94
0.94
0.94
0.96
0.94
0.94
0.94
0.94
0.94
0.94
0.95
0.94
0.95
0.94
0.94
0.95
0.94
0.95
0.95
0.94
0.94
0.95
0.94
0.94
0.9
0.94
0.94
0.94
0.94
0.94
0.94
0.9
0.94
0.94
0.94
0.94
0.94
0.94
MJV
0.96
0.96
0.96
MJV
0.96
0.96
0.96
Table 7: Precision, recall, and scores of CaMVo with kiiin
Kmin across different
confidence thresholds for each output label the IMDB dataset. Note that the last two rows report the corresponding precision, recall, and scores obtained with majority voting.
LLM / Method
Accuracy (%)
Cost
03-mini
87.16
1.10
1lama-3.3-70b
86.35
0.59
ol-mini
85.77
1.10
Hlaue-3-7-ssonne
85.27
3.00
84.71
0.80
gpt-4o-mini
82.63
0.15
1--m
77.85
0.05
Majority Vote
85.55
6.79
Baseline Method
85.68
6.79
Table 8: Accuracy and cost of individual LLMs and baseline ensemble methods on the AG News Classification Dataset.
37

---

Target
Acc. (%)
Cost
Acc. ( (%)
Cost
CaMVo
Acc.
kmin
kmin
kmin
kmin
3
0.999
85.59
85.98
6.82
85.98
6.82
0.998
85.51
85.98
6,82
85.98
6.82
0.995
85.25
86.36
5.24
86.36
5.24
0.99
84.82
86.18
2.98
86.18
2.98
0.98
83.97
84.72
1.39
84.72
1.39
0.96
82.25
83.38
0.52
83.85
0.99
0.95
81.40
83.18
0.40
83.77
0.96
0.9
77.11
80.66
0.29
83.75
0.88
0.85
72.83
79.62
0.26
83.74
0.87
0.80
68.54
79.34
0.23
83.73
0.87
Table 9: Accuracy and cost of CaMVo on the AG News Classification Dataset under varying confidence thresholds and Kmin {1, For reference, the cost of the baseline method is $6.79 per million tokens.
Target Acc. (%)
CaMVo Acc. (%)
CaMVo Cost
CCaMVo Acc. (%)
CCaMVo Cost
0.99
87.30
88.47
9.14
88.55
9.14
0.98
86.42
88.59
8.57
88.53
8.72
0.975
85.98
88.49
7.80
88.41
8.14
0.97
85.53
88.35
6.67
88.25
7.10
0.965
85.09
88.27
5.66
88.07
4.88
0.96
84.65
87.98
4.74
86.96
3.07
0.955
84.21
87.40
3.38
86.84
2.58
0.95
83.77
86.82
2.76
86.33
2.32
0.90
79.36
84.88
1.19
82.29
0.76
0.85
74.95
84.41
1.03
81.72
0.67
0.80
70.54
82.12
0.70
80.09
0.60
0.75
66.14
68.80
0.16
68.30
0.14
0.70
61.73
68.38
0.14
68.30
0.14
Table 10: Accuracy and cost of CaMVo and Correlated CaM (CCaM on the MMLU dataset
under varying confidence thresholds ö and kmin For reference, the cost of the baseline method is $9.14 per million tokens.
Target Acc. (%)
CaMVo Acc. (%)
CaMVo Cost
CCaMVo Acc. (%)
CCaMVo Cost
0.99
87.30
88.47
9.14
88.55
9.14
0.98
86.42
88.59
8.57
88.50
8.73
0.975
85.98
88,49
7.80
88.43
8.16
0.97
85.53
88.33
6.67
88.33
6.90
0.965
85.09
88,27
5,66
88.29
5.21
0.96
84.65
88.03
4.74
86.40
3.73
0.955
84.21
87.01
3.36
87.08
2.69
0.95
83.77
87.01
2.96
86.42
2.45
0.90
79.36
84.80
1.81
84.17
1.77
0.85
74.95
82.14
1.58
81.45
1.52
0.80
70.54
81.32
1.51
81.23
1.50
0.75
66.14
81.24
1.50
81.18
1.50
0.70
61.73
81.22
1.50
81.17
1.50
Table 1: Accuracy and cost of CaMVo and Correlated CaM Vo (CCaM on the MMLU dataset under varying confidence thresholds 8 and kmin = 3. For reference, the cost of the baseline method is $9.14 14 per million tokens.
38

---

Target Acc. (%)
CaMVo Acc. (%)
CaMVo Cost
CCaMVo Acc. (%)
CCaMVo
Cost
0.999
95.52
95,59
6.15
95.43
3.61
0.998
95,43
95,43
4.03
95.28
2.29
0.997
95.33
95.45
2.83
95.10
1.32
0.995
95.14
95.25
2.06
95.09
0.99
0.99
94.66
95.10
1.09
95.09
0.89
0.985
94.20
94.69
0.34
95.05
0.82
0.98
93.71
94.69
0.31
94.78
0.38
0..7
92.75
94.56
0.22
94.68
0.28
0.96
91.80
94.21
0.13
94.18
0.20
0.95
90.84
94.28
0.14
94.61
0.26
0.9
86.06
94.24
0.10
94.11
0.15
Table 12: Accuracy and cost of CaMVo and Correlated CaM (CCaM the IMDB dataset
under varying confidence thresholds and kmin = For reference, the cost the baseline method $6.29 per million tokens.
Target Acc. (%)
CaMVo Acc. (%)
CaMVo Cost
CCaMVo Acc. (%)
CCaMVo
Cost
0.999
95.52
95.59
6.15
95,46
3.61
0.998
95.43
95.43
4.03
95.29
2.21
0.997
95.33
95.45
2.83
95.10
1.21
0.995
95.14
95.25
2.06
95.10
0.99
0.99
94.66
95.12
0.99
95.07
0.88
0.985
94.20
95.06
0.84
95.07
0.85
0.98
93.71
95.07
0.83
95.08
0.83
0.97
92.75
95.07
0.82
95.07
0.83
0.96
91.80
95.06
0.81
95.07
0.82
0.95
90.84
95.07
0.81
95.06
0.82
0.9
86.06
95.06
0.81
95.06
0.81
Table 13: Accuracy and cost of CaMVo and Correlated CaM (CCaMVo) on the IMDB dataset under varying confidence thresholds d and kmin = 3. For reference, the cost of the baseline method is $6.29 million tokens.
Target Acc. (%)
CaMVo Acc. (%)
CaMVo Cost
CCaMVo Acc. (%)
CCaMVo Cost
0.999
85.59
85,98
6.82
85.98
6.82
0.998
85.51
85.98
6.82
85.98
6.81
0.995
85.25
86.36
5.24
86.10
5.12
0,99
84.82
86.18
2.98
85.82
2.75
0.98
83.97
84.72
1.39
83.94
1.13
0.96
82.25
83.38
0.52
83.3
0.51
0.95
81,40
83.18
0.40
83.18
0.44
0.9
77.11
80.66
0.29
81.5
0.33
0.85
72.83
79.62
0.26
79.66
0.26
0.80
68.54
79.34
0.23
79.26
0.24
Table 14: Accuracy and cost of CaMVo and Correlated CaMVo (CCaMVo) on the AG News Classification Dataset under varying confidence thresholds ó and Kmin 1. For reference, the cost of the baseline method is $6.79 per million tokens.
39

---

Target Acc. (%)
CaMVo Acc. (%)
CaMVo Cost
CCaMVo Acc. (%)
CCaMVo
Cost
0.999
85.59
85,98
6.82
85.98
6.82
0.998
85.51
85.98
6,82
85.98
6.81
0.995
85.25
86.36
5.24
86.14
5.08
0.99
84.82
86.18
2.98
85.67
2.29
0.98
83.97
84.72
1,39
83.94
1.10
0.96
82.25
83.85
0.99
83.89
1.01
0.95
81.40
83.77
0.96
83.75
0.93
0.9
77.11
83.75
0.88
83.75
0.89
0.85
72.83
83.74
0.87
83.72
0.87
0.80
68.54
83.73
0.87
83.72
0.87
Table 15: Accuracy and cost of CaMVo and Correlated CaMVo (CCaMVo) on the AG News Classification Dataset under varying confidence thresholds å and kmin 3. For reference, the cost of the baseline method is $6. 56.79 per million tokens.
40

---

CaMVo
CaMVo
CaMVo
CaMVo
Maj. vote Acc. (%)
Target Acc. (%)
6
(Kmin 1)
(kmin 1)
(Knin 3)
(kmin Cost
Acc. (%
Cost
Acc. (%)
0.96
0.1
93.36
89.63
93.35
9.14
93,40
9.14
0.96
0.2
92.84
89.13
92.83
9.14
92.80
9.14
0.96
0.3
92.30
88.61
92.30
9.14
92.30
9.14
0.96
0.4
91.32
87.67
91.32
9.14
91.30
9.14
0.96
0.5
90.99
87.35
90.99
9.14
91.00
9.14
0.96
0.6
90.63
87.00
90.63
9.14
90.60
9.14
0.96
0.7
89.84
86.25
89.84
9.14
89.80
9.14
0.96
0.8
89.16
85.59
89.16
9.14
89.20
9.14
0.95
0.0
93.85
89.16
93.85
9.14
93.90
9.14
0.95
0.1
93.36
88.69
93.35
9.14
93.40
9.14
0.95
0.2
92.84
88.20
92.83
9.14
92.80
9.14
0.95
0.3
92.30
87.69
92.30
9.14
92.30
9.14
0.95
0.4
91.32
86.76
91.32
9.14
91.30
9.14
0.95
90.99
86.44
90.99
9.14
91.00
9.14
0.95
0.6
90.63
86.09
90.63
9.14
90.60
9.14
0.95
89.84
85.35
89.84
9.14
89.80
9.14
0.95
0.8
89.16
84.70
89.16
9.14
89.20
9.14
0.93
0.0
93.85
87.28
93.87
7.59
93.90
7.59
0.93
0.1
93.36
86.83
92.74
7.20
92.70
7.20
0.93
0.2
92.84
86.34
90.76
5.68
90.80
5.68
0.93
0.3
92.30
85.84
91.52
6.54
91.50
6.54
0.93
0.4
91.32
84.93
90.31
6.03
90.30
6.03
0.93
0.5
90.99
84.62
89.56
5.43
89.60
5.43
0.93
0.6
90.63
84.28
89.35
5.11
89.40
5.11
0.93
0.7
89.84
83.55
88.68
4.57
88.70
4.57
0.93
0.8
89.16
82.92
88.23
4.35
88.20
4.35
0.90
0.0
93.85
84.47
86.98
2.23
86.70
2.66
0.90
0.1
93.36
84.03
86.79
2.23
86.20
2.57
0.90
0.2
92.84
83.56
86.84
2.03
87.50
2.63
0.90
0.3
92.30
83.07
85.61
3.53
87.80
3.78
0.90
0.4
91.32
82.19
85.68
1.85
84.50
2.29
0.90
0.5
90.99
81.89
85.50
1.69
84.10
2.15
0.90
0.6
90.63
81.56
85.23
1.66
83.70
2.16
0.90
0.7
89.84
80.86
84.69
1.49
83.10
2.02
0.90
0.8
89.16
80.24
84.64
1.47
82.80
2.01
0.80
0.0
93.85
75.08
84.56
1.13
80.10
1.55
0.80
0.1
93.36
74.69
84.35
1.16
80.80
1.56
0.80
0.2
92.84
74.27
84.58
1.12
80.00
1.52
0.80
0.3
92.30
73.84
84.42
1.14
79.10
1.51
0.80
0.4
91.32
73.06
84.54
1.14
79.10
1.52
0.80
0.5
90.99
72.79
84.68
1.13
78.70
1.49
0.80
0.6
90.63
72.50
84.74
1.15
78.60
1.50
0.80
0.7
89.84
71.87
83.05
0.92
78.20
1.47
0.80
0.8
89.16
71.33
80.90
0.64
78.30
1.48
Table 16: Accuracy and cost of CaMVo with Aiin and Kmin 3. alongside the accuracy
of majority voting and the target accuracy, under varying confidence thresholds 8 and correlation parameters C, in the synthetic correlated data. The synthetic data are generated with parameters
d = 5, 0.5, and 0.1. Note that the cost of majority voting fixed at 19.14 per million tokens for all settings.
41

---

CaMVo
CaMVo
CaMVo
CaMVo
Maj. vote Acc. (%)
Target Acc. (%)
6
(kmin 1)
(Kmin
(Kmin 3)
(kmin 3)
Acc. (%
Cost
Acc. (%)
Cost
0.96
0.0
93.95
90.19
93.95
9.14
93.95
9.14
0.96
0.1
93.43
89.69
93.43
9.14
93.43
9.14
0.96
0.2
92.86
89.15
92.86
9.14
92.86
9.14
0.96
0.3
92.28
88.59
92.28
9.14
92.28
9.14
0.96
0.4
91.57
87.91
91.57
9.14
91.57
9.14
0.96
0.5
91.07
87.43
91.07
9.14
91.07
9.14
0.96
0.6
90.65
87.02
90.65
9.14
90.65
9.14
0.96
0.7
90.06
86.46
90.06
9.14
90.06
9.14
0.96
0.8
89.41
85.83
89.41
9.14
89.41
9.14
0.95
0.0
93.95
89.25
96.00
8.54
96.00
8.54
0.95
0.1
93.43
88.75
95.84
8.51
95.84
8.51
0.95
0.2
92.86
88.22
95.48
8.50
95.48
8.50
0.95
0.3
92.28
87.66
94.95
8.54
94.95
8.54
0.95
0.4
91.57
86.99
94.54
8.52
94.54
8.52
0.95
0.5
91.07
86.51
94.25
8.50
94.25
8.50
0.95
0.6
90.65
86.12
91.65
7.82
91.65
7.82
0.95
0.7
90.06
85.56
89.90
6.97
89.90
6.97
0.95
0.8
89.41
84.94
89.76
6.60
89.76
6.60
0.90
0.0
93.95
84.55
86.55
1.71
85.40
2.24
0.90
0.1
93.43
84.08
86.44
3.09
90.51
3.27
0.90
0.2
92.86
83.58
86.32
1.68
84.94
2.14
0.90
0.3
92.28
83.05
87.63
2.20
88.51
2.91
0.90
0.4
91.57
82.42
86.06
1.64
84.53
2.12
0.90
0.5
91.07
81.96
85.72
1.52
84.98
2.02
0.90
0.6
90.65
81.58
84.09
1.98
84.09
1.98
0.90
0.7
90.06
81.06
85.31
1.44
83.69
1.99
0.90
0.8
89.41
80.47
85.55
1.36
82.90
1.95
0.80
0.0
93.95
75.16
85.32
1.13
80.11
1.51
0.80
93.43
74.74
85.33
1.15
80.10
1.53
0.80
0.2
92.86
74.29
82.66
0.73
79.91
1.50
0.80
0.3
92.28
73.82
85.24
1.14
78.78
1.49
0.80
0.4
91.57
73.26
85.27
1.13
78.67
1.48
0.80
0.5
91.07
72.85
85.30
1.13
78.76
1.47
0.80
0.6
90.65
72.52
84.12
0.98
78.58
1.48
0.80
0.7
90.06
72.05
81.77
0.63
78.40
1.48
0.80
0.8
89.41
71.53
85.45
1.13
77.93
1.47
Table 17: Accuracy and cost of CaM Vo with Aiiin and kmin = alongside the accuracy of majority voting and the target accuracy, under varying confidence thresholds 5 and correlation parameters C, the synthetic correlated data. The synthetic data are generated with parameters d = 5, = 0.5, and 0.2. Note that the cost of majority voting is fixed at $9.14 per million tokens for all settings.
42

---

CaMVo
CaMVo
CaMVo
CaMVo
Maj. vote Acc. (%)
Target Acc. (%)
6
(Kmin 1)
(kmin 1)
(Knin 3)
(kmin 3)
Acc. (%
Cost
Acc. (%)
Cost
0.96
0.1
91.83
88.16
96.69
6.62
96.69
6.62
0.96
0.2
91.54
87.88
96.77
6.54
96.77
6.54
0.96
0.3
90.75
87.12
96.18
6.64
96.18
6.64
0.96
0.4
90.28
86.67
95.58
6.46
95.58
6.46
0.96
0.5
89.61
86.02
95.68
6.31
95.68
6.31
0.96
0.6
89.27
85.70
95.28
6.24
95.28
6.24
0.96
0.7
88.79
85.24
89.15
4.99
89.15
4.99
0.96
0.8
88.32
84.79
88.07
4.45
88.07
4.45
0.95
O
92.45
87.83
97.50
5.69
97.50
5.69
0.95
0.1
91.83
87.24
96.55
5.52
96.55
5.52
0.95
0.2
91.54
86.96
97.07
5.51
97.07
5.51
0.95
0.3
90.75
86.21
96.47
5.56
96.47
5.56
0.95
0.4
90.28
85.77
95.85
5,41
95.85
5.41
0.95
89.61
85.13
95.33
5.14
95.33
5.14
0.95
0.6
89.27
84.81
91.03
4.74
91.03
4.74
0.95
88.79
84.35
88.68
3.46
88.68
3.46
0.95
0.8
88.32
83.90
94.52
5.00
94.52
5,00
0.93
92.45
85,98
92.72
3.56
92.72
3,56
0.93
0.1
91.83
85.40
91.07
3,48
91.07
3,48
0.93
0.2
91.54
85.13
90.44
3.23
90.44
3.23
0.93
0.3
90.75
84.39
90.23
3.34
90.23
3.34
0.93
0.4
90.28
83.96
90.12
3.21
90.12
3,21
0.93
0.5
89.61
83.33
89.63
3.26
89.63
3.26
0.93
0.6
89.27
83.02
86.25
1.95
86.25
1.95
0.93
0.7
88.79
82.57
86.35
1.88
86.35
1.88
0.93 0.8
88.32
82.14
86.45
1.66
86.45
1.66
0.9
92.45
83.20
86.80
1.42
86.80
1.42
0.9 0.I
91.83
82.65
85.45
2.03
85.45
2.03
0.9 0.2
91.54
82.38
86.63
1.52
86.63
1.52
0.9 0.3
90.75
81.67
86.78
1.57
86.78
1.57
0.9 0.4
90.28
81.26
83.97
1.90
83.97
1.90
0.9 0.5
89.61
80.65
83.75
1.44
83.75
1.44
0.9 0.6
89.27
80.35
82.89
1.08
82.89
1.08
0.9 0.7
88.79
79.91
86.36
1.28
86.36
1.28
0.9 0.8
88.32
79.49
82.69
0.92
82.69
0.92
0.8 Q
92.45
73.96
82.53
0.61
82.53
0.61
0.8 0.1
91.83
73.46
86.15
1.13
86.15
1.13
0.8 0.2
91.54
73,23
85.71
1.10
85.71
1.10
0.8 0.3
90.75
72.60
85.88
1.14
85.88
1.14
0.8
0.4
90.28
72.23
82.57
0.64
82.57
0.64
0.8
0.5
89.61
71.69
86.01
1.13
86.01
1.13
0.8
0.6
89.27
71.42
82.70
0.63
82.70
0.63
0.8
0.7
88.79
71.03
82.85
0.64
82.85
0.64
0.8
0.8
88.32
70.65
82.71
0.62
82.71
0.62
Table 18: Accuracy and cost of CaMVo with Amin and Kmin = 3. alongside the accuracy
of majority voting and the target accuracy, under varying confidence thresholds 8 and correlation parameters C, in the synthetic correlated data. The synthetic data are generated with parameters d = 5, 0.5, and 0.4. Note that the cost of majority voting fixed at 19.14 14 per million tokens for all settings.
43

---

CaMVo
CaMVo
CaMVo
CaMVo
Maj. vote Acc. (%)
Target Acc.
Q
8
(kmin 1)
(kmin 1)
(kmnn 3)
(kmin 3)
Acc. (%)
Cost
Acc. (%)
Cost
0.1
0.96
92.21
88.53
92.21
9.14
92.21
9.14
0.1 0.95
92.21
87.60
92.21
9.14
92.21
9.14
0.1 0.93
92.21
85.76
92.41
6.97
92.41
6.97
0.1 0.9
92.21
82.99
85,82
1.81
85.82
1.81
0.1 0.85
92.21
78.38
84.61
1.21
84.61
1.21
0.1 0.8
92.21
73.77
80.73
0.66
80.73
0.66
0.2 0.96
92.36
88.67
92.36
9.14
92.36
9.14
0.2 0.95
92.36
87.75
94.94
8.53
94.94
8.53
0.2 0.93
92.36
85.90
91.12
5.10
91.12
5.10
0.2 0.9
92.36
83.13
85.69
1.45
85.69
1.45
0.2 0.85
92.36
78.51
85.35
1.18
85.35
1.18
0.2 0.8
92.36
73.89
84.93
1.10
84.93
1.10
0.4 0.96
91.23
87.58
93.59
6.70
93.59
6.70
0.4 0.95
91.23
86.67
96.61
5.49
96.61
5.49
0.4 0.93
91.23
84.85
90.10
3.03
90.10
3.03
0.4
0.9
91.23
82.11
84.71
1.91
84.71
1.91
0.4
0.85
91.23
77.55
82.96
0.69
82.96
0.69
0.4
0.8
91.23
72.99
82.90
0.63
82.90
0.63
Table 19: Accuracy and cost of CaMVo with Amin 1 and Kmin 3, alongside the accuracy of majority voting and the target accuracy, 8 and values for synthetic data generated using the correlation matrix Eq. (22). The synthetic data are generated with parameters d = 5, and a = 0.5. Note that the cost of majority voting is fixed at $9. 14 million tokens for all settings.
44


Reviews:
[{"review_id": "SUKZ6Js9mr", "reviewer": "Reviewer_fBz7", "summary": "The paper proposes a method to approximate the majority voting of an ensemble of LLMs in an online setting, using a subset of the ensemble. They aim to achieve comparable accuracy to the full-ensemble at much lower costs. The method is supported by experiments showing that their method is close to the Pareto-optimal frontier of cost and accuracy.", "questions": "- As mentioned in the weaknesses, the choice of priors for the label correctness and regularization is not clear. Could you give a justification? How would it compare to different choices?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "#### Strengths\n- The paper is well-written and easy to follow for the most part.\n- I appreciated the in-depth experiments, especially the cost accuracy trade off plots in Figure 1 and 2.\n\n#### Weaknesses\n- The language in section 3 is imprecise and might be misleading. For instance, $y_t$ is defined the predicted label by aggregating the votes of the LLM ensemble in a weighted way, see eq. (1). As the authors say that the true label is not available, they replace it with the *ensemble prediction* and define *empirical accuracy* as agreeing with the *ensemble prediction*. This has nothing to do with *accuracy* as we know it, i.e., agreeing with the *true label*. Note that the majority vote (based on weights proportional to accuracy) and their so called baseline method (based on weights proportional to agreeing with the previous ensemble prediction) perform very similar in experiments. However, it is unclear whether (1) the weights are actually similar, (2) both weights are unstable but the aggregation is not sensitive or (3) weights are both close to uniform. Moreover, if the weights were close, this might be a results of the ensemble accuracy being rather high, i.e. above 85%, and would not hold in a more general case.\n- The authors model the the label correctness as a beta distributed random variable, but give no justification for this choice. I only found that the parameters of the shape beta distribution are estimated by MM oder MLE in the supplements.\n- The authors propose a regularization based on Laplace smoothing to avoid overconfident weights and bias aggregation, again without any reasoning.\n- Table 3 is a bit misleading. As the best single model accuracy is 85.92% by o3-mini in Table 2, there is no reason to apply their method to ensemble a subset of LLMs to attain a much lower target accuracy at higher costs. This is the case in all but the first few rows. Similarly, the individual models are missing in the cost accuracy trade off plots in Figure 1 and 2, or are at least not clearly indicated.\n- Although the authors mention that their method is based on the independence of LLM predictions, they do not conduct a proper sensitivity study on how vulnerable their method is.", "was_revised": true, "final_justification": "The authors addressed the issues I mentioned before, that is, revising the terminology to make it more clear, explaining the modeling choices and adding a sensitivity study to evaluate the impact of correlated LLM outputs (preliminary in the rebuttal and promise of extension in the final version).", "rebuttal": "Dear Reviewer, \n\nThank you for the comments. Our responses to the reviewer's comments are below.\n\n## Responses to Weaknesses: \n\n**W1:** In our paper, we consider a problem setting where a user wished to label a dataset. Without our method, the default approach would be majority voting. Our primary motivation is to reduce the labeling cost by allowing a trade-off with accuracy. Accordingly, the accuracy we define is relative to the accuracy of majority voting, which serves as the baseline for comparison. We acknowledge that this distinction could be more clearly articulated. In the final version, we will revise the terminology to avoid ambiguity. For instance, we will \"accuracy\" to \"relative accuracy\" and \"target accuracy\" to \"target relative accuracy\". Note that the accuracy values reported in our simulation results correspond to comparisons between the generated labels and the true labels, so the use of \"Accuracy\" in those contexts remains appropriate.\n\nRegarding the LLM accuracies, the individual model accuracies (used as weights in majority voting) on the MMLU dataset are\n[0.680,0.848,0.817,0.835,0.856,0.859,0.640], and the final weights learned by the baseline method are [0.728,0.905,0.875,0.893,0.901,0.911,0.689]. As expected, these weights are higher than the original accuracies, since under the baseline method, any LLM that agrees with the majority vote is treated as having produced a correct label—even in cases where all LLMs are jointly incorrect. This inflates the estimated accuracy. Nonetheless, the baseline performs comparably to majority voting because for the majority voting output to be correct, the weights of the correct labels need to exceed the weights of incorrect labels. Even though the weights are not exactly correct, the baseline method learns the correct ordering of the weights (if LLMs are sorted by weight both set of weights produce the same result). With this, the weights of the correct labels can still exceed the weights of incorrect labels, and this helps the baseline method achieve similar results as majority voting. This explains why the baseline method achieves results close to majority voting.\n\n\n**W2:** Using a Beta distribution is a standard Bayesian approach for modeling binary outcomes, as it is the conjugate prior to the Bernoulli likelihood. This allows for efficient incremental posterior updates as new binary observations (correct/incorrect labels) are collected. While maximum likelihood estimation (MLE) is the formal method for updating the shape parameters of the Beta distribution based on observed data, it has no closed-form solution. In practice, the method of moments is commonly used as a computationally efficient alternative to approximate the parameter updates.\n\n\n**W3:** Since no labeled data is available at the start, early accuracy estimates of the LLMs can be noisy and prone to overfitting. Because subset selection is based on these estimated confidences, early estimation errors can bias the selection process and lead to compounding errors over time. To mitigate this, we use regularization to stabilize the accuracy estimates. We choose Laplace smoothing as the regularizer as Laplace smoothing corresponds to using a uniform prior on the Beta estimator, and hence combines well with the Beta estimator. We use a log(t) term with the Laplace smoothing as compared to a constant term, as the log(t) term does not decay as quickly as the constant term and prevents overfitting in the long run.\n\nWe will include the reasoning behind our design choices in the final version of the paper. \n\n\n\n\n**W4:** While we use the true labels to compute accuracy in our simulations, these labels are not available to the user in the actual problem setting. As a result, the user does not know a priori which LLM performs best, or whether any individual model outperforms majority voting. Crucially, if only a single LLM is queried in each round, its output will be treated as the true label. Since it will always agree with itself, this prevents any meaningful estimation of the true accuracy of the LLM. Hence, to reliably estimate individual LLM accuracies over time, it is necessary to query a subset of LLMs in each round, which incurs a cost. This is a fundamental aspect of the exploration-exploitation trade-off in our framework.\n\nRegarding the cost-accuracy trade-off plots, although individual models are already included in the plots, we agree that their presentation can be improved. In the final version, we will use a distinct color to clearly differentiate individual LLMs for better visual clarity.\n\n\n**W5:** We have conducted the following sensitivity study to evaluate the impact of correlated LLM outputs. Using the true accuracies of the LLMs, we generate synthetic labels via a Gaussian copula approach. This method preserves the original accuracies while introducing correlations among the LLMs as specified by a correlation matrix. In this matrix, the diagonal entries representing self-correlation are set to 1, and all off-diagonal entries are set to a parameter $C_i$ that represents the pairwise correlation. We run experiments using these synthetic labels, sweeping $C_i$ across a range of values to analyze how varying correlation strengths affect performance.\n\nThe results of this experiment for the MMLU dataset are given below .\n|        | $C_i=0$   | $C_i=0.1$   | $C_i=0.2$   | $C_i=0.3$   | $C_i=0.4$   | $C_i=0.5$  | $C_i=0.6$   | $C_i=0.7$   | $C_i=0.8$   |\n|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|\n| CaMVo ($\\delta=0.96, k_{min}=1$) | 96.67  | 94.84  | 93.24  | 91.72  | 89.53  | 88.19  | 87.01  | 84.02  | 78.96  |\n| Maj vote  | 96.66  | 94.15  | 91.46  | 89.88  | 87.67  | 86.35  | 84.53  | 83.59  | 82.49  |\n| Target acc.  | 92.79  | 90.38  | 87.80  | 86.29  | 84.16  | 82.90  | 81.15  | 80.25  | 79.19  |\n\nAs can be seen from the results, although the performance of CaMVo declines as the correlation among LLMs increases, the performance of majority voting also deteriorates similarly. Importantly, CaMVo is able to maintain the target accuracy across all tested correlation levels except at the very high correlation coefficient of $C_i=0.8$, verifying the utility of CaMVo even in correlated regimes. \n\nCurrently, we have used a single uniform value for all off-diagonal entries in the cross-correlation matrix. In the final version of the paper, we plan to extend this sensitivity study by exploring varying cross-correlation values across different matrix elements. Additionally, we will extend the experiments to the IMDB dataset and conduct experiments with a broader range of $\\delta$ values to provide a more comprehensive analysis.\n\n\n\n## Responses to Questions: \n\nWe have addressed the reviewer’s questions as part of our responses to the weaknesses."}, {"review_id": "qmUTHhtNVe", "reviewer": "Reviewer_ZiN8", "summary": "The core of this paper lies in exploring the methods that if it is possible to dramatically reduce the cost of dataset annotation by adaptively selecting a small, optimized subset of LLMs for each specific data instance, while achieving an accuracy comparable or even superior to using a full, expensive ensemble of all available LLMs.", "questions": "1. Given CaMVo’s focus on _online_ cost efficiency (§1.1, Table 1), how might it integrate task-specific fine-tuning (e.g., LoRA adapters) without violating the zero-shot constraint? Could dynamically tuned LLMs alter the cost-accuracy trade-off?\n2. In correlated-error regimes (e.g., IMDB), does the Beta-mixture estimator’s performance degrade? Would modeling inter-model dependencies (e.g., via copula-based confidence aggregation) enhance robustness?", "limitations": "1. **Independence Assumption**: As noted in §5, correlated LLM errors reduce ensemble gains, particularly in low-diversity tasks (e.g., IMDB). This remains an open challenge.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The authors combine two well-established concepts -- contextual bandits and Bayesian estimation, the design is clean and makes intuitive sense: use the bandit to learn which models are good at which types of questions, and use the Bayesian estimator to quantify the underlying reliability of each model.\n2. The online approach make the method more practical for many \n3. The use of Pareto frontier plots provides a clear, visual benchmark of how the system performs against a theoretical \"perfect\" selector.\n\n**Weakness**\n\n1. The NP-hard nature of subset selection (Eq. 4) warrants deeper discussion. While the paper mentions approximation via Beta-CDF, the absence of complexity analysis or runtime metrics obscures practical deployability.\n2. The independence assumption (§2, Lemma 2.1) undermines performance when LLM outputs correlate strongly (e.g., IMDB in §4.2). While acknowledged in limitations, preliminary mitigation strategies (e.g., covariance-aware confidence bounds) could be explored.", "was_revised": true, "final_justification": "I recommend Acceptance. The authors' rebuttal successfully resolved my primary concerns regarding the NP-hard complexity and the independence assumption. Their proposed greedy algorithm and copula-based simulation are practical solutions that make the work significantly more robust. With these key issues addressed, the paper is technically sound, and my initial reservations are gone.", "rebuttal": "Dear Reviewer, \n\nThank you for the comments. Our responses to the reviewer's comments are below.\n\n## Responses to Questions:\n\n**Q1:** We propose the following implementation strategy to integrate task-specific fine-tuning. Initially, we will start with zero-shot LLMs to label data. As more data is labeled, we can fine-tune the LLMs using the accumulated labels. However, blindly fine-tuning on all generated labels risks incorporating incorrect annotations, which can degrade model performance and lead to error propagation. To mitigate this, we introduce a threshold $\\delta_u$ and update the LLMs only with labels whose confidence exceeds $\\delta_u$.\n\nSince CaMVo already selects subsets with confidence above a threshold $\\delta$, it will not necessarily choose subsets with confidence above $\\delta_u$. To enable this, we can use  $\\delta_u$ as the update threshold for a user-defined fraction of the data. Additionally, as more data is collected, we can recalculate label confidences and include them for fine-tuning once they surpass $\\delta_u$. To control cost, fine-tuning can be performed in batches once a sufficient number of high-confidence labels is accumulated.\n\nTo account for changes in model behavior due to fine-tuning when estimating model parameters such as empirical accuracy, we can use discounted updates on model parameters, assigning higher weight to recent observations.\n\nFinally, since fine-tuning may increase correlation among LLMs, it may be beneficial to keep the original zero-shot LLMs available for selection in the subset. In such cases, we can introduce a constraint to avoid simultaneously selecting both the fine-tuned and zero-shot versions of the same model, as they are likely to be highly correlated.\n\n\n**Q2:** Yes, since the Beta estimator does not model correlations between LLMs, its performance will degrade in regimes with correlated LLM outputs. To overcome this degradation in performance, and also to remove the assumption of independence, we have implemented a practical Monte Carlo simulation approach to evaluate majority vote confidence given LLM accuracies and their correlations. Using a Gaussian copula, we generate $m$ binary outcomes (correct/incorrect) for each LLM that capture both the marginal accuracies and correlations. From these simulated outcomes, we compute $m$ majority vote results weighted by the LLM weights. Average accuracy over these instances provides an empirical confidence estimate for the majority voting scheme.\n\nApplying this method on the MMLU dataset with CaMVo ($\\delta=0.96$, $k_{\\min}=1$) yields an accuracy of 86.96\\% at a cost of 3.07, exceeding the target accuracy of 84.65\\%. For comparison, the original CaMVo results are 87.98\\% accuracy at 4.74 cost ($\\delta=0.96$), and 86.82\\% accuracy at 2.76 cost ($\\delta=0.95$). On the IMDB dataset, the simulation with $\\delta=0.99$ and $k_{\\min}=1$ produces 95.09\\% accuracy at a cost of 0.83, surpassing the target accuracy of 94.66\\%. The original CaMVo results are 95.10\\% accuracy at 1.09 cost ($\\delta=0.99$), and 94.69\\% accuracy at 0.34 cost ($\\delta=0.985$).\n\nThese results indicate that incorporating the copula-based simulation does not significantly improve performance on MMLU and yields a modest cost improvement on IMDB, where LLM outputs exhibit higher correlation. We attribute this to two factors: (1) LinUCB’s use of input embedding vectors to predict LLM confidences already implicitly captures correlations among LLM outputs conditioned on the input context, and (2) estimation errors in the correlation matrix reduce the potential gains from explicitly modeling correlations.\n\nWe plan to extend this study across all tested $\\delta$ and $k_{\\min}$ values in our simulations and include these results in the final version of the paper.\n\nAdditionally, Lemma 2.1 can be extended to remove the conditional-independence assumption using Bahadur’s model (R.R. Bahadur, “On classification based on responses to n dichotomous items,” 1961) characterizes the joint distribution of $n$ binary variables using their marginal probabilities $p_i$, second-order dependence terms known as Bahadur parameters, and higher-order interaction terms. This framework allows the computation of majority vote confidence by summing the probabilities of all outcomes where the majority label is correct. However, the lack of a closed-form solution and the rapid growth in computational complexity with increasing $n$ limit its practical applicability. We will include a note on this in the discussion of Lemma 2.1 in the final version of the paper.\n\n\n## Responses to Weaknesses:\n\n**W1:** The NP-hard nature of subset selection is well known in the combinatorial bandits literature (e.g., CUCB, Combinatorial Thompson Sampling), and many algorithms assume access to an oracle (either exact or approximate) that solves the offline combinatorial optimization step.\n\nAn exact oracle, which evaluates all possible subsets, has a time complexity of $O(2^K)$ per round. However, for small $K$ (i.e., $K \\leq 10$), the number of subsets remains tractable. In our simulations, $K=7$, and we did not observe any meaningful impact on runtime.\n\nFor larger $K$, an approximate oracle can be employed to reduce computation at the cost of approximation. In our setting, one such approach is to use Monte Carlo simulations to generate $m$ realizations for each LLM. Given these samples, the confidence of a subset of size $K$ can be estimated in $O(mK)$ time. As we discussed in our response to Q2, this approach can also incorporate LLM correlations using a given correlation matrix, however time complexity will increase to $O(mK^2 + K^3)$. To efficiently search a low-cost subset that satisfies the confidence threshold, a greedy approximation can be used. Starting with all LLMs in the subset, at each step we can remove the highest cost LLM that does not reduce the confidence of the subset below the threshold. This will continue until no more LLMs can be removed. In this method, each iteration will have $O(K)$ comparisons, and in total there will be $O(K)$ iterations. As confidence estimation for one subset is $O(mK)$, in total the runtime will be $O(mK^3)$ for this method. We will add this discussion to the final version of the paper. \n\n**W2:** Please refer to our response Q2.\n\n## Responses to Limitations: \n\n**L1:** Please refer to our response Q2."}, {"review_id": "OPtOMYqf4X", "reviewer": "Reviewer_Ld4S", "summary": "This paper introduces a new algorithm called Cost-aware Majority Voting (CaMVo) for efficient and accurate dataset labeling using LLMs. Traditional approaches to LLM-based annotation often rely on majority voting across multiple models to improve label reliability, but this incurs high computational costs. CaMVo addresses this by adaptively selecting a subset of LLMs for each data instance, balancing cost and confidence without requiring pre-training or ground-truth labels. It uses a LinUCB-based contextual bandit algorithm combined with a Bayesian confidence estimator to determine which models to query, aiming to meet a user-defined accuracy threshold while minimizing cost.\n\nThe core innovation of CaMVo lies in its online, cost-aware selection mechanism. For each data instance, it computes contextual embeddings and estimates a lower bound on each LLM’s probability of producing a correct label. These estimates are used to select the smallest-cost subset of models whose aggregated confidence exceeds a specified threshold. The algorithm also incorporates regularization using Laplace smoothing to prevent overfitting and uses a weighted majority voting scheme where weights reflect both historical and expected instance-specific performance. This dynamic adaptation allows CaMVo to operate effectively without offline training or labeled data.\n\nEmpirical evaluations on two datasets, MMLU and IMDB, demonstrate CaMVo’s effectiveness. On MMLU, CaMVo achieves higher or comparable accuracy to full majority voting while reducing costs by around 40%. On the IMDB dataset, where individual LLMs already perform well, CaMVo still maintains high accuracy with comparable or lower cost. These results highlight CaMVo’s ability to adapt to different task complexities and cost constraints, suggesting it may be a practical tool for scalable annotation.\n\nCaMVo is the first fully online method that jointly adapts both the subset of queried models and their voting weights on a per-instance basis. By leveraging contextual embeddings and probabilistic confidence estimation, CaMVo achieves strong performance without relying on labeled data or pre-training. Its flexibility and efficiency make it well-suited for dynamic labeling environments, and its design opens avenues for further research into more robust and adaptive ensemble strategies.", "questions": "How did you decide on the individual models, especially since they didn’t necessarily exhibit comparable performance levels?\n\nHow often did each LLM give an incorrectly formatted response? Did this complicate the evaluations?\n\nIn practice, how do you choose a confidence threshold?\n\nHow might results compare with human annotation? Is there a good indication that LLMs approximate human annotation for these types of tasks?\n\nHow did you choose the embedding model? What informed your decision?\n\nHow could CaMVo be extended to incorporate human in the loop annotations?", "limitations": "The paper discusses limitations, particularly the assumption of independence among LLM outputs. In practice, LLMs may produce correlated outputs, which can reduce the benefits of majority voting and lead to suboptimal performance. This issue is especially evident in simpler tasks like IMDB sentiment analysis, where ensemble methods may not outperform the best single model. The authors suggest future work could involve modeling inter-model correlations and incorporating diversity-aware selection strategies.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n-\tThe proposed algorithm seems very practical for real-world annotation scenarios.\n-\tThe authors take cost into account and design the selection of the subset of LLMs to reduce cost, again making this very practical for real world applications.\n-\tThe work is based on a strong theoretical foundation.\n-\tThe two empirical examples give an intuitive understanding of both the strengths and weaknesses of the approach.\n\nWeaknesses:\n-\tAs noted by the authors, the assumption of independence in LLM outputs is likely problematic.\n-\tThe algorithm was only tested on two datasets; additional empirical evidence for the efficacy of the approach is needed.\n-\tAccuracy was the only evaluation metric considered. \n-\tNo guidance or insight around how to select a confidence threshold is provided.", "was_revised": true, "final_justification": "I thank the authors for their additional comments and answers to my questions. My final score remains the same.", "rebuttal": "Dear Reviewer, \n\nThank you for the comments. Our responses to the reviewer's comments are below.\n\n## Responses to Questions:\n\n**Q1:** In selecting the individual models, we prioritized those commonly used in related work while also aiming to ensure diversity in both model architecture and per-token cost. Since our primary objective is cost optimization, it is important to span a wide range of cost levels to enable meaningful trade-offs.\n\nAdditionally, we included different versions of similar LLMs with varying parameter sizes (e.g., llama-3.1 and llama-3.3) to allow CaMVo to learn cost-aware selection strategies. The intent is to allow CaMVo to favor cheaper, smaller models on simpler tasks, and to choose the larger, more expensive models when the input task is more complex. \n\n\n**Q2:**  On the MMLU dataset, more than 20\\% of the LLM responses were incorrectly formatted. For example, when the expected output was the answer choice ‘A’, common variations included responses like “A) 5”, “5”, “The answer is A”, or full explanations preceding the answer. Similarly, on the IMDB dataset, LLMs frequently generated summaries of user comments before stating their classification decision.\n\nTo handle these inconsistencies, we post-processed the outputs using keyword matching and other simple heuristics to extract the intended answer. In fewer than 2\\% of cases, the LLMs failed to produce a usable response (i.e. by answering a question in the movie review in the IMDB dataset or providing an answer that is not in the answer choices). In such cases, we were able to obtain valid responses by requerying the LLM.\n\n\n**Q3:**  In practice, the confidence threshold can be set to reflect a desired target accuracy. If no specific accuracy requirement is given, the threshold can instead be adapted dynamically to meet a target cost budget or tuned over time as feedback on the output labels becomes available. This flexibility allows the method to operate under varying application constraints, whether accuracy-driven, cost-driven, or feedback-guided.\n\n\n**Q4:**  Yes, there is a growing body of work showing that LLMs can approximate, and in some cases even surpass human annotators across a wide range of tasks, including the classification tasks considered in our work. For example, Gilardi et al. (“ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks”) demonstrate that LLMs can not only match but often exceed the performance of crowd-sourced human annotations.\n\n\n\n**Q5:**  Since CaMVo is designed for online use, low-latency embedding generation is crucial. We selected SBERT over BERT because SBERT is significantly faster for producing sentence embeddings. To inform our choice in choosing an SBERT model, we referred to the accuracy and speed trade-off comparisons provided on the official Sentence Transformers (SBERT) library website. We chose the all-MiniLM-L6-v2 model as it offers a strong balance between efficiency and performance, making it well-suited for our setting.\n\n\n\n**Q6:** In the current implementation of CaMVo, we query all LLMs when no subset achieves confidence above the threshold $\\delta$. For example, with $\\delta=0.96$ and $k_{\\min}=1$ on the MMLU dataset, this occurs an average of 597 times across 14,042 data instances. To extend this mechanism, an additional threshold $\\delta_{\\text{human}} < \\delta$ can be introduced: if the highest subset confidence falls between $\\delta_{\\text{human}}$ and $\\delta$, we query all LLMs; if it falls below $\\delta_{\\text{human}}$, the instance can be deferred to a human annotator. This extension provides a principled way to incorporate human in the loop annotations for low-confidence labels.\n\n\n## Responses to Weaknesses:\n\n**W1:** Please see our response to the limitations section.\n\n**W2:** We will further extend our simulation results to an additional dataset and include the results in the final version of the paper, and we have conducted the following synthetic experiment to analyze the sensitivity of our method to correlated LLM outputs, and will also include its results in the final version of the paper. Using the true accuracies of the LLMs, we apply the Gaussian copula approach to generate synthetic labels that preserve the individual accuracies while introducing controlled correlations via a specified correlation matrix. The diagonal entries are set to 1, and off-diagonal entries are set to a shared correlation coefficient $C_i$, which we vary across experiments. The results from these experiments show that while CaMVo’s performance declines with increasing correlation, the performance of majority voting also degrades. Importantly, CaMVo maintains its target accuracy except under very high correlation levels. See our response W5 to Reviewer fBz7 for additional details.\n\n\n**W3:** We will include additional evaluation metrics such as precision, recall, and F1 score in the final version of the paper to provide a more comprehensive assessment of performance.\n\n**W4:** Please refer to our response Q3. \n\n\n## Responses to Limitations\n\n**L1:** To overcome the assumption of independence, we have implemented a practical Monte Carlo simulation approach to evaluate majority vote confidence given LLM accuracies and their correlations. Using a Gaussian copula, we generate $m$ binary outcomes (correct/incorrect) for each LLM that capture both the marginal accuracies and correlations. From these simulated outcomes, we compute $m$ majority vote results weighted by the LLM weights. Average accuracy over these instances provides an empirical confidence estimate for the majority voting scheme.\n\nApplying this method on the MMLU dataset with CaMVo ($\\delta=0.96$, $k_{\\min}=1$) yields an accuracy of 86.96\\% at a cost of 3.07, exceeding the target accuracy of 84.65\\%. For comparison, the original CaMVo results are 87.98\\% accuracy at 4.74 cost ($\\delta=0.96$), and 86.82\\% accuracy at 2.76 cost ($\\delta=0.95$). On the IMDB dataset, the simulation with $\\delta=0.99$ and $k_{\\min}=1$ produces 95.09\\% accuracy at a cost of 0.83, surpassing the target accuracy of 94.66\\%. The original CaMVo results are 95.10\\% accuracy at 1.09 cost ($\\delta=0.99$), and 94.69\\% accuracy at 0.34 cost ($\\delta=0.985$).\n\nThese results indicate that incorporating the copula-based simulation does not significantly improve performance on MMLU and yields a modest cost improvement on IMDB, where LLM outputs exhibit higher correlation. We attribute this to two factors: (1) LinUCB’s use of input embedding vectors to predict LLM confidences already implicitly captures correlations among LLM outputs conditioned on the input context, and (2) estimation errors in the correlation matrix reduce the potential gains from explicitly modeling correlations.\n\nWe plan to extend this study across all tested $\\delta$ and $k_{\\min}$ values in our simulations and include these results in the final version of the paper.\n\nAdditionally, Lemma 2.1 can be extended to remove the conditional-independence assumption using Bahadur’s model (R.R. Bahadur, “On classification based on responses to n dichotomous items,” 1961) characterizes the joint distribution of $n$ binary variables using their marginal probabilities $p_i$, second-order dependence terms known as Bahadur parameters, and higher-order interaction terms. This framework allows the computation of majority vote confidence by summing the probabilities of all outcomes where the majority label is correct. However, the lack of a closed-form solution and the rapid growth in computational complexity with increasing $n$ limit its practical applicability. We will include a note on this in the discussion of Lemma 2.1 in the final version of the paper."}, {"review_id": "22AWKopeOy", "reviewer": "Reviewer_U74S", "summary": "The paper introduces Cost-aware Majority Voting (CaMVo), an online framework that labels data with large language-model (LLM) ensembles while minimizing inference cost. CaMVo first uses a LinUCB-style contextual bandit to compute, for every candidate LLM, a lower-confidence bound (LCB) on its probability of producing the correct label for the current instance. A Bayesian Beta-mixture then converts that LCB into a posterior correctness estimate, and a knapsack-style “Oracle’’ picks the cheapest subset of models whose weighted-vote confidence exceeds a user-specified threshold and a minimum vote count. If no subset passes the test, all models are queried. Experiments on MMLU and IMDB show that CaMVo matches or surpasses full-ensemble majority-vote accuracy while cutting cost by about 40 % on MMLU and about 50 % on IMDB under some settings.", "questions": "1. How do you plan to test and, if necessary, correct for the violation of the conditional-independence assumption that underlies Lemma 2.1 and the Oracle’s confidence bound? (e.g., measuring pair-wise correlations)\n2. How will you extend the method beyond classification tasks?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Quality**: the paper thoroughly specifies CaMVo, including pseudocode and all hyper-parameters. The experiments were conducted on two benchmarks, seven off-the-shelf LLMs, and ablations over $\\omega$ and $k_\\min$ demonstrate consistent cost savings at equal (or better) accuracy. However, the theory and subset-confidence bound assume conditional independence of LLM outputs, a condition the authors admit is violated on the IMDB sentiment task and can degrade performance.\n\n**Clarity**: writing is generally clear, with a step-by-step derivation of the bound and well-annotated figures showing cost–accuracy trade-offs.\n\n**Significance**: this paper addresses a practical bottleneck: escalating API cost of ensemble labelling at a time when LLM annotation pipelines are widely adopted. It demonstrates sizeable savings without ground-truth labels or offline training. The paper can benefit more from analysis on harder structured-output problems or real-time latency. Practical impact thus remains to be validated beyond the chosen domains.\n\n**Originality**: this paper proposes the first fully online cost-aware subset-of-LLMs selection scheme that jointly adapts both voting weights and the subset size, combining contextual bandits with a Bayesian confidence model. However, it is conceptually related to prior bandit-based router and query-budget work mentioned in the related work section.", "was_revised": true, "final_justification": "The authors provided clear and detailed responses that resolved the main concerns. They clarified the definition of relative accuracy, justified the use of the Beta prior and Laplace regularization, and explained the subset selection complexity with both exact and approximate strategies. The correlation sensitivity study using a Gaussian copula showed that CaMVo remains effective even when LLM outputs are correlated. The empirical results are strong, with CaMVo often matching or exceeding the Pareto frontier while significantly reducing cost. Responses to reviewer questions were thoughtful and demonstrated practical awareness. While evaluation is limited to two datasets and correlation modeling is not fully integrated, the core method is technically sound, well-motivated, and impactful. I recommend acceptance.", "rebuttal": "Dear Reviewer, \n\nThank you for the comments. Our responses to the reviewer's questions are below.\n\n**Q1:** Lemma 2.1 can be extended to remove the conditional-independence assumption using Bahadur’s model (R.R. Bahadur, “On classification based on responses to n dichotomous items,” 1961) characterizes the joint distribution of $n$ binary variables using their marginal probabilities $p_i$, second-order dependence terms known as Bahadur parameters, and higher-order interaction terms. This framework allows the computation of majority vote confidence by summing the probabilities of all outcomes where the majority label is correct. However, the lack of a closed-form solution and the rapid growth in computational complexity with increasing $n$ limit its practical applicability. We will include a note on this in the discussion of Lemma 2.1 in the final version of the paper. \n\nDue to this computational intractability, we implemented a practical Monte Carlo simulation approach to evaluate majority vote confidence given LLM accuracies and their correlations. Using a Gaussian copula, we generate $m$ binary outcomes (correct/incorrect) for each LLM that capture both the marginal accuracies and correlations. From these simulated outcomes, we compute $m$ majority vote results weighted by the LLM weights. Average accuracy over these instances provides an empirical confidence estimate for the majority voting scheme.\n\nApplying this method on the MMLU dataset with CaMVo ($\\delta=0.96$, $k_{\\min}=1$) yields an accuracy of 86.96\\% at a cost of 3.07, exceeding the target accuracy of 84.65\\%. For comparison, the original CaMVo results are 87.98\\% accuracy at 4.74 cost ($\\delta=0.96$), and 86.82\\% accuracy at 2.76 cost ($\\delta=0.95$). On the IMDB dataset, the simulation with $\\delta=0.99$ and $k_{\\min}=1$ produces 95.09\\% accuracy at a cost of 0.83, surpassing the target accuracy of 94.66\\%. The original CaMVo results are 95.10\\% accuracy at 1.09 cost ($\\delta=0.99$), and 94.69\\% accuracy at 0.34 cost ($\\delta=0.985$).\n\nThese results indicate that incorporating the copula-based simulation does not significantly improve performance on MMLU and yields a modest cost improvement on IMDB, where LLM outputs exhibit higher correlation. We attribute this to two factors: (1) LinUCB’s use of input embedding vectors to predict LLM confidences already implicitly captures correlations among LLM outputs conditioned on the input context, and (2) estimation errors in the correlation matrix reduce the potential gains from explicitly modeling correlations.\n\nWe plan to extend this study across all tested $\\delta$ and $k_{\\min}$ values in our simulations and include these results in the final version of the paper.\n\n\n**Q2:** Our method can be naturally extended beyond classification tasks in several ways. For regression, LLM outputs can be aggregated via a weighted average or median, with weights updated based on how closely each LLM’s output aligns with the aggregate. For ranking tasks, each LLM can assign scores to items, and a weighted combination of these scores would produce a final ranking; weights can then be adjusted according to agreement of the LLM's ranking generated from its scores with the aggregated ranking. These extensions involve only minor modifications to the aggregation and weight update steps, making them straightforward to integrate into our existing framework. We will highlight these as directions for future work in the final version of the paper."}]

Author rebuttal:

Meta-review: The paper propose CaMVo, a cost-aware framework for dataset annotation with LLMs. The idea is to select adaptively a subset of models using contextual bandit with Bayesian estimation, in this way saving cost but keeping accuracy. This is a relevant and timely problem because LLM annotation is becoming very expensive, and the approach is simple but novel.

Experiments are convincing: results are close or better than majority voting but with much lower cost. The method is well motivated and ablations are strong. Some concerns on assumptions were raised but authors reply was reasonable. Overall I think the paper is solid and useful for the community, so I recommend accept.
Decision: Accept (spotlight)