Poor-Supervised Evaluation for SuperLLM via Mutual Consistency (2024)

Abstract: The guidance from capability evaluations has greatly propelled the progress of human society and the development of Artificial Intelligence. However, as LLMs evolve, it becomes challenging to construct evaluation benchmark with accurate labels for LLMs whose capabilities approach or even surpass those of humans (denoted as SuperLLMs). To credibly conduct evaluation without accurate labels (denoted as poor-supervised evaluation), we first prove that the consistency between the model under evaluation and the reference model, when their prediction distributions are independent and the sample size is infinite, can equivalently assess the true capabilities of the model to be evaluated. However, using either humans or LLMs as the reference model cannot sufficiently meet the conditions, for which we propose the PEEM algorithm. By treating all models under evaluation as reference models, PEEM alternately optimizes model weights and filters reference models based on EM algorithm to maximally alleviate the insufficiency of the conditions. Comprehensive experiments across 3 types of tasks with 16 mainstream LLMs validate the efficiency, universality, and effectiveness of PEEM. More generally, PEEM has advanced the evaluation paradigm evolution from human-centric to human&model-centric, alleviating the limitations of human capabilities for evaluating SuperLLMs1.

heading: 1 Introduction
text: Evaluations, such as IQ tests and Olympic competitions, effectively identify areas of strength and weakness for individuals or groups, providing valuable guidance for the enhancement of various human abilities. In the field of artificial intelligence, benefiting from high-quality annotated data (Deng
*Corresponding author. 1Our code and data have been released on https://
github.com/ypw0102/PEEM.
et al., 2009; Wang et al., 2019), neural network models have been well evaluated and optimized specifically, leading to significant advancements across various tasks.
Nowadays, advanced large language models (LLMs) (OpenAI, 2023), called SuperLLMs (Burns et al., 2023), have reached the level of human experts in certain tasks (Tedeschi et al., 2023; Lei et al., 2023; Pu et al., 2023; Webb et al., 2023). To further guide SuperLLMs towards AGI, evaluating them on a wider range of challenging tasks is of crucial importance.
However, building benchmarks for challenging tasks is especially intelligence-intensive and sometimes even difficult to ensure the accuracy of labels. As shown in Figure 1, annotating hard tasks requires scarce human experts and some tasks are even beyond the capability boundary of experts, such as “Is the Riemann hypothesis true?”. So here comes the question: How to evaluate LLMs accurately on benchmarks with poor supervision (with inaccurate or even no labels)?
Some works have explored unsupervised (labelfree) model evaluation, including using LLM-based
11614
pseudo labels (Bai et al., 2023; Li et al., 2023), evaluating with logits (Yu et al., 2022; Peng et al., 2024), and detecting conflicts with designed inputs (Fluri et al., 2023). Unfortunately, they are not proposed for evaluating SuperLLMs, presenting issues such as limited applicability across tasks, pseudo labels may still be inaccurate, logits of some LLMs are unavailable (e.g., GPT4), and dependence on prior knowledge. Overall, SuperLLMs are in eager need of an accurate, universal, and efficient poor-supervised evaluation paradigm.
Consequently, we propose the following theorem and prove it in Appendix §A.
Theorem 1. For benchmark X lacking true labels Y , if reference model Ṁ is independent of predictions {Ŷ i}Li=1 of models {Mi}Li=1 to be evaluated (Condition 1), and performs better than random guess (Condition 2), when the size of X is infinite (Condition 3), the following equation holds:
Cons(Ṁ(X), Ŷ i) < Cons(Ṁ(X), Ŷ j) ⇔ Cons(Y, Ŷ i) < Cons(Y, Ŷ j)
(1)
where Cons(·) denotes mutual consistency. Theorem 1 allows us to credibly evaluate and compare the capabilities of models (consistency between their predictions and true labels) without true labels Y .
LLMs themselves or humans can naturally serve as reference model Ṁ, but the conditions are not guaranteed to be sufficiently satisfied in real scenarios. Some LLMs may exhibit mutual affinity and tend to make the same predictions (e.g., Llama series), and the size of X is finite. Thus, we propose PEEM algorithm (Poor-supervised Evaluation with EM algorithm) to alleviate these insufficiency. PEEM takes models predictions (with optional human annotations) as inputs, and calculates the weighted average consistency of certain model with reference models as the measurement of its capability. All the models to be evaluated are treated as reference models and assigned equal weights initially. Based on the EM algorithm, PEEM iteratively adjusts the weights of reference models during E-step, and filters reference models based on current weights during M-step to optimize the proxy optimization objective inferred from preliminary experiments. PEEM mitigates the limitation of sample size (Condition 3) by comprehensively utilizing the predictions of all models under evaluation. And it alleviates the dependency
among model predictions (Condition 1) by optimizing the weights of reference models and filtering out the ones with high affinity tendency.
Our poor-supervised evaluation paradigm based on PEEM algorithm holds several advantages. Efficient: No additional model inference overhead is required except for model predictions. Universal: It can be applied across various tasks by proper definition of Cons(·), as we demonstrate in §5. Accurate: Experimental results show that PEEM aligns perfectly with true label evaluation results, achieving an average of 0.977 Pearson and 0.972 Spearman correlation coefficient.
Furthermore, we validate that PEEM with human annotations and model predictions, as opposed to the traditional consistency check between models and humans, allows for a more accurate evaluation results. More generally, PEEM has advanced the evaluation paradigm evolution from humancentric to human&model-centric, addressing the limitations of human evaluation capabilities.
Our contributions are summarized as follows:
1. We analyze and define the task of evaluating SuperLLMs as poor-supervised evaluation.
2. We theoretically provide a method and its conditions for poor-supervised evaluation.
3. We propose the PEEM algorithm to alleviate the insufficiency of the conditions for the introduced theorem under real-world scenarios by alternating between weight correction and reference model filtering.
4. We experimentally validate the efficiency, universality, and effectiveness of PEEM across regression, classification, and reasoning tasks.

heading: 2 Related Work
text: Surrounding our study, we discuss the researches on poor-supervised evaluation and model consistency, and briefly introduce the EM algorithm.
Poor-supervised Evaluation We refer to the process of evaluating models on benchmarks that contain inaccurately labeled or unlabeled data as poorsupervised evaluation, an area where many related efforts have been invested. Various directions have been explored to assessing model’s capabilities on datasets without labels: examining distribution discrepancy (Yu et al., 2022; Deng et al., 2021) , relying on model confidence (Garg et al., 2021; Wang
et al., 2023a; Lu et al., 2023; Peng et al., 2024), calculating models’ disagreements (Baek et al., 2022; Jiang et al., 2022; Chen et al., 2021; Jiang et al., 2021) and bucketing based on decision boundaries (Miao et al., 2023; Tu et al., 2023; Xie et al., 2023). However, these studies mainly focus on classification tasks and often require model logits, thus lacking universality and not being applicable to LLMs whose logits are unavailable. In addition, Fluri et al. (2023) and Jain et al. (2023) consider detecting conflicts with designed inputs to assess model’s capabilities. However, the design of input format highly depends on prior knowledge and cannot be generalized across tasks. Conducting self-evaluation (Zheng et al., 2023) or peer evaluation (Li et al., 2023; Bai et al., 2023) with LLMs is a promising direction, but they have not addressed the fundamental issue of how to ensure the effectiveness of evaluations on benchmarks that exceed their own capabilities. Additionally, such methods require extra inference overhead. Overall, an accurate, universal, and efficient poor-supervised evaluation paradigm for SuperLLMs remains to be researched, for which we propose PEEM.
Model Consistency Consistency has long been studied for training models (Miyato et al., 2019; Chen et al., 2020), enhancing performance during inference (Wang et al., 2023b; Yao et al., 2023), and examining specific attributes such as reliability (Jang and Lukasiewicz, 2023) and hallucination (Ji et al., 2023). We investigate using mutual consistency between inaccurate predictions to achieve poor-supervised evaluation for SuperLLMs.
EM Algorithm The EM algorithm (Dempster et al., 1977) alternates between the Expectation step (E-step) and the Maximization step (M-step) to estimate parameters of models with latent variables. The E-step estimates the distribution of latent variables. During the M-step, parameters are updated to maximize the expected likelihood function. We use the EM algorithm to optimize the parameters of mapping from inter-prediction consistency to prediction-label consistency.

heading: 3 Preliminary

heading: 3.1 Task Definition
text: Given LLMs {Mi}Li=1, datasets D : (X = {xi}Ni=1, Y = {yi}Ni=1) and predictions of Mi on X as Ŷ i = {ŷij}Nj=1, the consistency between {Ŷ i}Li=1 and true labels Y , denoted as
{Cons(Ŷ i, Y )}Li=1, shortened to B ∈ RL, is the accurate measurement of the capabilities of these LLMs. Our research aims to assess LLMs when Y is unavailable. Therefore, we consider designing a mapping algorithm F that can map LLM predictions {Ŷ i}Li=1 (with inaccurate human annotations if available) to capability vector B̂ ∈ RL , where B̂i can substitute for Bi in evaluating the capabilities of modelMi. We calculate the correlation coefficient between B̂ and B to validate the effectiveness of F . It should be noted that the calculation method of Cons(·) depends on the label domain, which we will introduce in §5.

heading: 3.2 Intuition of Theorem 1
text: We propose Theorem 1 to credibly implement poor-supervised evaluation, which is grounded in an intuition that is straightforward to understand: For the samples where the reference model predicts correctly, strong model tends to have higher consistency with the reference model compared to weak model, Cons(Mstrong,Ṁ)> Cons(Mweak,Ṁ). While for the samples where the reference model predicts incorrectly, if the reference model does not tend to make the same predictions as either strong or weak models, the consistency of both strong and weak model with the reference model should be the same, Cons(Mstrong,Ṁ)= Cons(Mweak,Ṁ). Therefore, across all the samples, Cons(Mstrong,Ṁ)> Cons(Mweak,Ṁ).

heading: 3.3 Visualization of Consistency and Affinity
text: Although Condition 2 can be assumed to be satisfied naturally, Condition 1 and 3 cannot be completely fulfilled in real scenarios as discussed before. To explore the impact of the insufficiency of Condition 1 and 3 in real scenarios, we visualize the consistency matrix C and the affinity matrix A of models to be evaluated. Ci,j ∈ [0,1] denotes Cons(Ŷ i, Ŷ j), the average consistency between the predictions ofMi andMj on X . And Ai,j ∈ [-1,1] reflects the affinity ofMi toMj , which can be calculated as follows:
Ai,j = Ci,j Sum(Ci) − Bj Sum(B) (2)
A positive Ai,j implies usingMi as the reference model to estimate B̂j will overestimate the performance ofMj and vice versa.
We conduct this preliminary experiment on subset Precalculus of MATH (Hendrycks et al., 2021)
benchmark with 16 LLMs (See detailed setup in §5.1.1). As shown in Figure 2(a), the top 16 rows demonstrate the C, the third-to-last row calculates {Avg(Ci)}Li=1, the penultimate row shows B, while the ith column of the last row represents the Pearson coefficient between Ci and B (the closer to 1, the stronger the linear correlation). From Figure 2(a), we can observe the following tendencies and corresponding insights. (1) The third-to-last and penultimate row show a positive correlation → Insight 1: stronger models tend to have higher consistency with other models; (2) The last and penultimate row also show a positive correlation→ Insight 2: stronger models’ consistency with other models can better reflect the true performance of these models. Considering that model predictions tend to converge towards the true labels and predictions of stronger models are closer to the true labels, the above two insights are natural and reasonable.
From Figure 2(b), we confirm that some LLMs belonging to the same series (Llama-2 series, MetaMath series) indeed exhibit mutual affinity, which makes their consistency biased to reflect their true performance. Furthermore, we observe Insight 3: weaker models tend to exhibit higher affinity. We attribute this to the consistency between weak models and strong models underestimating the performance of the strong models, indirectly leading to a higher affinity among weak models. More generally, the proportion of samples correctly predicted by a weak model is too low, using it as a reference model may not suffice to differentiate between strong and weak models.

heading: 4 Methods
text: In this section, we propose multiple mapping algorithms F to obtain B̂ based on the insights above, as shown in Figure 3. For each algorithm, we first calculate the consistency matrix C according to Ŷ and task types, thus F : C → B̂.

heading: 4.1 Naive Ensemble
text: Due to the lack of prior knowledge about the capabilities of models, a straightforward approach is to consider all the models as reference models and calculate the ensemble consistencies of certain model with other models to measure its capability:
Fensemble(C) = {Avg(Ci)}Li=1 (3)
Compared with randomly choosing certain model as reference model, Fensemble can enlarge sample size by L times to alleviate the insufficiency of Condition 3 through ensemble, thus offering a more stable measurement result.

heading: 4.2 Weight Calibration
text: Building upon Fensemble, considering that Ci of stronger modelMi aligns better with B (Insight 2), we contemplate calibrating the ensemble weights based on the capabilities of the models through iteration.
Let αkj be the weight assigned to Ci,j after iteration k, which is initialized as 1/L. The capability estimation of Mi for iteration k, B̂ki , is the calibrated average of its raw consistencies with all the
models:
B̂ki =
L∑
j=1
Ci,j × αk−1j (4)
For each iteration k, we calibrate αkj according to B̂k following Insight 2:
αkj = B̂kj∑L i=1 B̂ k i
(5)
Given the set of equations above, we look for the converging point after k̂ iterations where the following equation holds:
L∑
j=1
|αk̂j − αk̂−1j | < 1e− 2 (6)
Through the above process, we have managed to calibrate the ensemble weights using the capability estimation of the models, thereby obtaining a better algorithm Fcalibrate:
Fcalibrate(C) = { L∑
j=1
Ci,j × αk̂j }Li=1 (7)

heading: 4.3 Reference Model Filtering
text: To alleviate the impact of non-independent distributions between model predictions (insufficiency of Condition 1) shown in Figure 2(b), we consider filtering out models with strong affinity tendencies. Insight 3 indicates that weak models generally have high affinity tendencies, thus we contemplate filtering them from reference models based on the estimation of their capabilities according to Fensemble as follows:
B̂ens = Fensemble(C) (8) Sref = arg{i|B̂ensi > Max(B̂ens)× p} (9) Ffilter(C) = {Avg({Ci,j}j∈Sref )}Li=1 (10)
The hyperparameter p serves as a threshold for filtering out weak models whose performance is not up to par with the strongest model to a certain extent according to B̂ens. Through the above process, the weak model with high affinity tendencies can be filtered for calculating a debiased B̂.

heading: 4.4 PEEM
text: Based on the discussions above, we consider designing an algorithm that can simultaneously leverage all three insights to mitigate the insufficiency of Condition 1 and 3. To this end, we propose an EM-based algorithm FPEEM as follows:
FPEEM (C) = { L∑
j=1
Ci,j × αk̂j × βk̂j }Li=1 (11)
We set βj ∈ {0, 1} to control whetherMj serves as a reference model and αj ∈ [0, 1] to control how much doesMj contributes to B̂.
Initialization. We initialize α0i as 1 L and β 0 i as 1.
Optimizing Objective. From Insight 1 and 2, we deduce the following chain-of-thought: greater Avg(Ci) Insight 1←−−−→ stronger Mi
Insight 2←−−−→ better alignment between Ci and B. On this basis, Avg(B̂) can serve as a proxy optimizing objective for B̂ to attain a better approximation of B.
Expectation Step. In this step, we conduct Fcalibrate among the reference model set constructed by βk−1 to obtain the calibrated expectation of αk as follows:
Sk−1ref = arg{i|βk−1i = 1} (12) {αki }i∈S = Fcalibrate(Ci∈Sref ,j∈Sref ) (13)
Maximization Step. Further, we optimize βk to maximize our objective Avg(B̂k) as follows:
Indexkmin = argmin i αki (14)
βki = { βk−1, i ̸= Indexkmin 0, i = Indexkmin
(15)
Avg(B̂k) = 1
∑L i=1 β k i
L∑
i=1
L∑
j=1
Ci,j × αkj × βkj
(16)
Termination condition. We iteratively conduct the E-step and M-step for k̂ + 1 rounds until the following formula holds true for the first time:
Avg(B̂k̂) > Avg(B̂k̂+1) (17)
In summary, we alternately calibrate the model weights and filter out weak models with high affinity tendency according to the estimation under current parameter, iterating this process until the optimal proxy objective Avg(B̂) is achieved.

heading: 5 Experiments
text: Since it is impossible to obtain accurate true labels for samples that human capabilities can not handle well, theoretically, we cannot directly measure the effectiveness of the proposed algorithms on such benchmarks. Thus we have designed a comparable experimental setup: we select benchmarks (MATH (Hendrycks et al., 2021) §5.1, USR (Mehri and Eskénazi, 2020) §5.2) that are currently considered fully manageable by human experts, and treat human predictions as the true labels. Under this setup, we validate whether FPEEM (Cons(LLMs)) can align well with Cons(LLMs, true label). Furthermore, MultiRC (Khashabi et al., 2018) §5.3 uses the predictions of single human annotator as human annotations and approximates the ensemble of predictions from multiple human annotators as true labels. We adopt this setting to verify if FPEEM (Cons(LLMs and human)) aligns better with Cons(LLMs, true label) than traditional evaluation paradigm Cons(LLMs, human).

heading: 5.1 Reasoning Task
text: We first validate the effectiveness of our proposed algorithms on MATH (Hendrycks et al., 2021) benchmark, which contains 7 subsets (Table 7), to assess the reasoning capabilities of LLMs.

heading: 5.1.1 Experimental Setup
text: We have selected 16 mainstream LLMs with a broad range of size and capabilities for the experiments involved in this paper, as shown in Table 6. To enhance the stability of evaluation, for each modelMi and sample xj , we sample T (Ŷ i) (= 5 across our experiments) times at temperature 0.5 to attain predictions {ŷij,k} T (Ŷ i) k=1 (Results of greedy search are also provided in §C.1). As the domain of true label y is discrete in MATH, we calculate Cons(Y, Ŷ i) as follows:
Cons(Y, Ŷ i) = 1
NT (Ŷ i)T (Y )
N∑
j=1
T (Y )∑
u=1
T (Ŷ i)∑
v=1
1yj,u=ŷij,v
(18)
The inter-model consistency Cons(Ŷ i, Ŷ j) can be similarly calculated. T (Y ) = 1 as the label is unique for each xi. To enhance the significance of the results, we randomly sample models at a ratio of qmodel = 0.7 from 16 candidates to conduct experiments and take the mean of 500 outcomes to report. As forFfilter, we set filtering threshold p as 0.9 where it can achieve the best performance (See Appendix §C.2 for results with varying p). The Pearson coefficient rp can measure the degree of linear correlation, while the Spearman coefficient rs can assess the monotonic relationship between two variables. We use them to measure the degree of alignment between B̂ and B, which are separately obtained from our algorithm and the true labels.

heading: 5.1.2 Experimental Results
text: As shown in Table 1, our algorithms generally achieve correlation coefficient greater than 0.8 (0.9 for FPEEM ) across most subsets, indicating that they can assess model capabilities well through mutual consistency in the absence of true labels.
In a horizontal comparison, Fensemble ranks last as it only naively treat all the models as reference models. Fcalibrate significantly improves the ordering of model capability (higher rs) by adjusting the weights of models during ensemble. However, altering the original weights also leads to a slight decrease in the linear relationship (lower rp). By filtering out weak models with high affinity tendency on the basis of Fensemble, Ffilter achieves significant improvements on both rs and rp. Finally, by integrating the ideas of both weight adjustment and reference model filtering into EM algorithm and directly optimizing the proxy target Avg(B̂),
FPEEM achieves evaluation results that align perfectly with the true labels (0.971 rp and 0.989 rs).
Meanwhile, we notice that the performance of the proposed algorithms vary across different subsets. To this end, we compare the average accuracy of all the models Avg(B), the mean of the absolute values of affinity matrix Avg(Abs(A)), and rs between B̂ from FPEEM and B of each subset. We plot the 20th power of rs and 0.5−Avg(Abs(A)) for better observation. As shown in Figure 4, they exhibit a clear correlation in the graph. We speculate the reason is as follows: on subsets where the model accuracy (Avg(B)) is higher, their predictions tend to be consistent with the true label and thus show a weak affinity tendency (Avg(Abs(A))). Therefore, Condition 2 can be well satisfied for FPEEM to attain good performance (rs), and vice versa. Inspired by this, we suggest that, where feasible, samples of moderate difficulty rather than those obviously beyond the model’s capability should be chosen to evaluate the model, thereby making the evaluation results more accurate. Meanwhile, FPEEM can be used to estimate whether samples are too difficult for the model during benchmark construction.

heading: 5.2 Regression Task
text: We select the USR benchmark (Mehri and Eskénazi, 2020) to validate our proposed algorithms within the context of regression task.

heading: 5.2.1 Experimental Setup
text: USR is a dialog evaluation testbed requiring models to predict appropriate scores based on specific criteria for the provided dialogues. We follow Mehri and Eskénazi (2020); Liu et al. (2023) to calculate Cons(Y, Ŷ i) as shown below:
Cons(Y, Ŷ i) = rp(Y, Ŷ i) (19)
We choose the Overall criteria for experiments, which contains 360 samples. To validate the effectiveness of the proposed methods varying sample sizes, we conduct experiments with sample sampling ratios qsample ∈ [0.1, 0.2, 0.4, 0.6, 0.8, 1].

heading: 5.2.2 Experimental Results
text: As shown in Table 2, FPEEM shows the best performance as expected, achieving 0.966 rp and 0.942 rs when qsample = 1. As qsample
0.2 0.4 0.6 0.8 1.0 qsample
gradually increases, the performance of all algorithms improves accordingly. This is natural since Condition 1 is satisfied to a greater extent with the increase of sample size. We also notice that rs and rp exhibit logarithmic curves as qsample increases (see Figure 5). This inspires us that the benefits of increasing sample size are greatest in the early stages. Therefore, when sample size is small (our experiments suggest 100 being a suitable threshold), we should try to collect samples as much as possible. However, when the number of samples is sufficiently large, we can afford to filter out some samples to save on inference resources, without significantly affecting the evaluation results.

heading: 5.3 Classification Task
text: To verify the effectiveness of FPEEM within the context of classification task, we select MultiRC (Khashabi et al., 2018) benchmark for experiments.

heading: 5.3.1 Experimental Setup
text: MultiRC evaluates the reading comprehension ability of models by having them answer multiplechoice questions. Based on our previous observation that a large sample size can lead to performance gain saturation, we randomly select 200 samples to conduct experiments for saving inference costs with 16 LLMs. We use the macroaverage F1 score to calculate Cons(Y, Ŷ i) follow-
ing Khashabi et al. (2018), as shown below:
Cons(Y, Ŷ i) = 1
NT (Y )T (Ŷ i)
N∑
j=1
T (Y )∑
u=1
T (Ŷ i)∑
v=1
F1(yj,u, ŷ i j,v)
(20)
Each sample in MultiRC is annotated by multiple individuals, and the ensemble result of these multiple annotations is considered the true label. The prediction made by a certain person is regarded as the human-level label ŷhuman. Traditionally, Cons(ŷhuman, ŷi) is used to measure the capability of Mi. Considering human as a model, we input the consistency matrix among {ŷi}i∈{human}∪{1,...,L} into the proposed algorithm to verify whether our method can assist humans in better evaluating model capabilities compared to Cons(ŷhuman, ŷi).

heading: 5.3.2 Experimental Results
text: As shown in Table 3, we find that FPEEM surpasses traditional human-centric evaluation approach by an obvious margin, especially when qsample is small. FPEEM has advanced the evolution of evaluation paradigm from human-centric to human & model-centric paradigm, utilizing the capabilities of LLMs to compensate for the insufficiencies in human ability during evaluation.

heading: 6 Conclusions
text: In this paper, we propose PEEM, an algorithm for conducting poor-supervised evaluation without true labels for SuperLLMs. Comprehensive experiments validate the efficiency, effectiveness, and university of PEEM across 3 types of tasks with 16 mainstream LLMs. PEEM has advanced the evaluation paradigm evolution from human-centric to human&model-centric, alleviating the limitations of human capabilities for evaluating SuperLLMs. Our work is a preliminary exploration of evaluating SuperLLMs by analogizing existing LLMs and benchmarks, longing for further improvement in the future.
Limitations
From an objective perspective , we think there are two main limitations of this paper:
1. Since humans currently cannot construct benchmarks with accurate labels that exceed the boundaries of human capabilities, we can only conduct analogy experiments on datasets within the scope of human capabilities. Despite this, such an analogy is reasonable because changes in task difficulty do not affect the form of the experimental setup.
2. The experiments above have verified the effectiveness of FPEEM in various tasks within closed label domain. Meanwhile, opendomain text generation is also an important aspect of LLM capability assessment. Unlike closed domain tasks holding a one-to-one relationship between input and output, opendomain tasks can have multiple appropriate answers for a given input. Therefore, evaluation based on consistency may not be suitable for open-domain evaluation and current mainstream methods depend on human-evaluator and LLM-evaluator (Liu et al., 2023). However, we still need a method to judge the capability of the LLM-evaluators when their capability surpass that of human-evaluator. The output of the LLM-evaluator is closed-domain, where we can use FPEEM to carry out such evaluations, as demonstrated in §5.2. To summarize, FPEEM can enhance the accuracy of model capability assessment in open-domain tasks by aiding in the selection of SuperEvaluators.
Ethics Statement
All of the datasets used in this study were publicly available, and no annotators were employed for our data collection. We confirm that the datasets we used did not contain any harmful content and was consistent with their intended use (research). We have cited the datasets and relevant works used in this study.

heading: Acknowledgments
text: This work is supported by the Beijing Natural Science Foundation, China (Nos. 4222037, L181010).

heading: A Proof of Theorem 1
text: Definition 1. Given models M i,M j under evaluation, reference model Ṁ , benchmark D : (X = {xi}Ni=1, Y = {yi}Ni=1) and predictions of M i on X as Ŷ i = {ŷij}Nj=1, the consistency between Model M i and Ground Truth Y , denoted as Cons(Ŷ i, Y ), is typically used to measure the performance of Model M i. We prove that the following equation holds:
Cons(Ẏ , Ŷ u) < Cons(Ẏ , Ŷ v)
⇔ Cons(Y, Ŷ u) < Cons(Y, Ŷ v) (21)
when condition 1: Both P (Ẏ |X), P (Ŷ i|X) and P (Ẏ |X), P (Ŷ j |X) are independently distributed; condition 2: Ṁ performs better than random predictor; condition 3: N →∞.
Proof. We prove the theorem separately in the continuous and discrete label domains as follows:
Discrete Domain In discrete label domain, we define Cons(A,B) = Avg(1ai=bi |ai ∈ A, bi ∈ B). Without loss of generality, we assume that the label domain has T possible values and the distribution of P (Ŷ i|X) are as follows:
P (ŷij = t|xj) = { σij , t = yj (1− σij)× λi,tj , t ̸= yj ( ∑ t ̸=yj λ i,t j = 1)
(22)
As Ṁ performs better than random predictor, E(σ̇) > 1T holds. Based on the assumptions above, we can derive as follows:
Cons(Ŷ u, Ẏ ) < Cons(Ŷ v, Ẏ )
⇔ N∑
i=1
Cons(ŷui , ẏi) < N∑
i=1
Cons(ŷvi , ẏi)
⇔ N∑
i=1
1ŷui =ẏi <
N∑
i=1
1ŷvi =ẏi
⇔ N∑
i=1
σui < N∑
i=1
σvi
⇔ NE(σu) < NE(σv) , according to condition 3 ⇔ E(σu) < E(σv)
(23)
From another direction:
Cons(Ŷ u, Ẏ ) < Cons(Ŷ v, Ẏ )
⇔ N∑
i=1
Cons(ŷui , ẏi) < N∑
i=1
Cons(ŷvi , ẏi)
⇔ N∑
i=1
1ŷui =ẏi <
N∑
i=1
1ŷvi =ẏi
⇔ N∑
i=1
(σui σ̇i + ∑
t̸=yi
λu,ti λ̇ t i) <
N∑
i=1
(σvi σ̇i + ∑
t̸=yi
λv,ti λ̇ t i)
⇔ NE(σuσ̇ + ∑
t̸=yi
λu,ti λ̇ t i) < NE(σvσ̇ +
∑
t̸=yi
λv,ti λ̇ t i)
, according to condition 3
⇔ E(σu)E(σ̇) + (T − 1)E(λu)E(λ̇) < E(σv)E(σ̇) + (T − 1)E(λv)E(λ̇) , according to condition 1
⇔ E(σu)E(σ̇) + (1− E(σu))(1− E(σ̇))/(T − 1) < E(σv)E(σ̇) + (1− E(σv))(1− E(σ̇))/(T − 1)
⇔ (E(σu)− E(σv))(E(σ̇)− 1− E(σ̇) T − 1 ) < 0 ⇔ (E(σu)− E(σv))(TE(σ̇)− 1) < 0 ⇔ E(σu) < E(σv) , according to condition 2
(24)
According to Eq 23 and Eq 24, Eq 21 holds in discrete label domain.
Continuous Domain In continuous label domain, we define Cons(A,B) = −Avg(Abs(ai − bi)|ai ∈ A, bi ∈ B). Without loss of generality, we assume ŷui = yi + g u i , where g u i is zero-mean Gaussian noise with variance as σui . Based on the assumptions above, we can derive as follows:
Cons(Ŷ u, Ẏ ) < Cons(Ŷ v, Ẏ )
⇔ N∑
i=1
Cons(ŷui , ẏi) < N∑
i=1
Cons(ŷvi , ẏi)
⇔ N∑
i=1
|ŷui − ẏi| > N∑
i=1
|ŷvi − ẏi|
⇔ N∑
i=1
|gui | > N∑
i=1
|gvi |
⇔ NE(|gu|) > NE(|gv|) , according to condition 3 ⇔ E(σu) > E(σv)
(25)
From another direction: Cons(Ŷ u, Ẏ ) < Cons(Ŷ v, Ẏ )
⇔ N∑
i=1
Cons(ŷui , ẏi) < N∑
i=1
Cons(ŷvi , ẏi)
⇔ N∑
i=1
|ŷui − ẏi| > N∑
i=1
|ŷvi − ẏi|
⇔ NE(|ŷu − ẏ|) > NE(|ŷv − ẏ|) , according to condition 1
⇔ N ∗ (E(σu) + E(σ̇)) > N ∗ (E(σv) + E(σ̇)) , according to condition 2
⇔ E(σu) > E(σv) (26)
According to Eq 25 and Eq 26, Eq 21 holds in discrete label domain.

heading: B Detailed Statistics
text: We conduct experiments with 16 mainstream LLMs as shown in Table 6. The detailed statistics of MATH benchmark are shown in Table 7.

heading: C Further Experimental Results
text: C.1 Results with Greedy Search
Apart from the main results of the proposed algorithms with random sampling at temperature as 0.5 and sampling times as 5, we also show the results with greedy search as shown in Table 4. PEEM performs best as expected. Compared with random sampling at sampling times as 5, the correlations between B̂ and B decrease a little bit across all the algorithms. We believe this is because multiple sampling can eliminate the noise impact brought by single sampling.
C.2 Hyperparameter Analysis We have explored the effectiveness of Ffilter under different values of p. As shown in Table 5, Ffilter performs relatively stably across different p-values, reaching its best when p = 0.9.

heading: Future Work
text: Our work is a preliminary exploration of evaluating SuperLLMs by analogizing existing LLMs and benchmarks, longing for further improvement in the future. Limitations
From an objective perspective , we think there are two main limitations of this paper:
1. Since humans currently cannot construct benchmarks with accurate labels that exceed the boundaries of human capabilities, we can only conduct analogy experiments on datasets within the scope of human capabilities. Despite this, such an analogy is reasonable because changes in task difficulty do not affect the form of the experimental setup. 2. The experiments above have verified the effectiveness of FPEEM in various tasks within closed label domain. Meanwhile, opendomain text generation is also an important aspect of LLM capability assessment. Unlike closed domain tasks holding a one-to-one relationship between input and output, opendomain tasks can have multiple appropriate answers for a given input. Therefore, evaluation based on consistency may not be suitable for open-domain evaluation and current mainstream methods depend on human-evaluator and LLM-evaluator (Liu et al. , 2023). However, we still need a method to judge the capability of the LLM-evaluators when their capability surpass that of human-evaluator. The output of the LLM-evaluator is closed-domain, where we can use FPEEM to carry out such evaluations, as demonstrated in §5. 2. To summarize, FPEEM can enhance the accuracy of model capability assessment in open-domain tasks by aiding in the selection of SuperEvaluators