Silencer: From Discovery to Mitigation of Self-Bias in LLM-as-Benchmark-Generator
Authors: Peiwen Yuan
Yiwei Li
Shaoxiong Feng
Xinglin Wang
Yueqi Zhang
Jiayi Shi
Chuyi Tan
Boyuan Pan
Yao Hu
Kan Li
Venue: NeurIPS 2025 poster ()
Keywords: large language model
benchmark generation
self-bias
Primary area: evaluation

Abstract: LLM-as-Benchmark-Generator methods have been widely studied as a supplement to human annotators for scalable evaluation, while the potential biases within this paradigm remain underexplored. 
In this work, we systematically define and validate the phenomenon of inflated performance in models evaluated on their self-generated benchmarks, referred to as self-bias, and attribute it to sub-biases arising from question domain, language style, and wrong labels.
On this basis, we propose Silencer, a general framework that leverages the heterogeneity between multiple generators at both the sample and benchmark levels to neutralize bias and generate high-quality, self-bias-silenced benchmark. Experimental results across various settings demonstrate that Silencer can suppress self-bias to near zero, significantly improve evaluation effectiveness of the generated benchmark (with an average improvement from 0.655 to 0.833 in Pearson correlation with high-quality human-annotated benchmark), while also exhibiting strong generalizability.

SILENCER:
From Discovery to Mitigation of Self-Bias in LLM-ss-Banchmark--enrrrboo
Peiwen Yuan', Yiwei Li', Shaoxiong Feng" Xinglin Wang', Yueqi Zhang', Jiayi Shi', Chuyi Tan'. Boyuan Pan', Yao Hu², Kan Li
School of Computer Science, Beijing Institute of Technology
Xiaohongshu Inc (peixenyuan,liyieee wangxingl wiw.gng.aanggaaaa (shaoxiongfeng2023) @gmall.mam..ooo..m com (panboyuan, wanchuyl,1ikannnaee (bitan]bbit edu. com
Abstract
-Geerrrtt methods have been widely studied as a supplement to human annotators for scalable evaluation, while the potential biases within this paradigm remain underexplored. In this work, we systematically define and validate the phenomenon of inflated performance in models evaluated on their self-generated benchmarks, referred to as self-bias, and attribute it to sub-biases arising from question domain, language style, and wrong labels. On this basis, we propose SILENCER, a general framework that leverages the heterogeneity between multiple generators both the sample and benchmark levels to neutralize bias and generate high-quality, self- bias-silenced benchmark. Experimental results across various settings demonstrate that SILENCER suppress self-bias to near zero, significantly improve evaluation effectiveness of the generated benchmark (with an average improvement from 0.655 0.833 in Pearson correlation with high-quality human-annotated benchmark), while also exhibiting strong generalizability.
Introduction
The rapid evolution of large language models (LLMs) (Guo et al. 2025 OpenAl 2025) has led to three key trends in evaluation: (1) benchmarks saturate
quickly, increasing the demand for continual benchmark construction (Wang et al. 2024); (2) LLMs are applied to a wider range of downstream tasks, amplifying the need for more eustomized benchmarks (Chang et al. 2024): (3) as LLMs achieve and even surpass human capability, ever greater intelligence is demanded of benchmark generators (Patel et al., 2025). Traditional human-centered benchmark construction paradigm struggles to meet these demands due to high annotation costs and the inherent ceiling of human intelligence. To address this, LLM-as a-aaanc-BBark----- Geeerrtto has been proposed as a potential paradigm (Huang et al. [0244] Farchi et al.) 2024: Maheshwari et al.) 2024 Yuan et al. 2025) for automatically constructing customizable benchmarks that are scalable diffculty based on L.MMs.
However, the potential biases -.ne-rreerteed benchmarks remain underexplored, which may lead unreliable evaluation results. In this work, we investigate the self-bias of Generator, a phenomenon where LLMs tend to achieve overestimated performance on benchmarks they generate themselves. Specifically, we first introduce formal and quantifiable definition of self-bias B, and then empirically demonstrate its widespread presence across various LLMs and evaluation tasks. To gain a deeper understanding of the origin of bias-bias, we heuristically disentangle B into three contributors: question domain bias B", language style bias B", and wrong label bias B¹. Investigative experiments show that the three sub-biases collectively contribute to self-bias, with their relative contributions ordered as B > B B
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To eliminate self-bias for achieving more accurate evaluation results, we propose SILENCER, general Self-bias neutralization framework for benchmark generation that exploits the heterogeneity of multiple generators to construct self-bias-silencee benchmark. Specifically, to mitigate the sub- biases at sample level, we introduce (1) Attribute Integration to encourage the L.LM generators to create more domain-unbiased question sets (5²); (2) Cross Paraphrase to avoid stylistic monotony in language (B*); (3) Label Calibration to reduce the inherent alignment between wrong labels and predictions offered by generators (B'). To suppress self-bias at benchmark level, we propose a Bias-Neutra Ensemble algorithm, which iterates through two steps: (1) estimating each generator's 's self-bias by measuring the consistency of its evaluation results with those of the ensembled benchmark, and (2) ensembling the generated benchmarks with weights that are inversely proportional to the estimated oeff-bas, thereby reducing the self- introduced to the final ensembled benchmark.
We validate the effectiveness and generalizability of SILENCER across three tasks, seven L.LM generators, and two demonstrate that SILENCER significantly suppresses self-bias, improving evaluation consistency of LLM-generated benchmark with high-quality human-annotater benchmark by 27.2% (from 0.655 to 0.833) on average. Detailed ablation studies and visualizations confirm the effectiveness and working mechanism of each component in SILENCER. Building this, we further explore the impact of factors such as the number of generators and benchmark size on self- bias and evaluation effectiveness, offering practical insights for applying SILENCER in real scenarios.
methods. Comprehensive experimental results
2 Related Works
2.1 LLM-as- Beraa---Becccmatt--eeertarr
Benchmark saturation (Glazer et al., 2024), data contamination (Balloccu et al. 2024), and accelerated LLM evolution have collectively motivated the research community to leverage the LLMs for automatic benchmark construction, moving toward the vision of self-evaluation (Maheshwari et al.. 2024). In particular, a specific LLM adopts a designated generation strategy to construct a benchmark based on benchmark definition inputs such as the task description and seed samples (Lei et al. 2023 Huang et al. 2024). The evaluation effectiveness of the generated benchmark is typically validated Via benchmark agreement testing (Perlitz et al., 2024), where the performance of some L.MMs on L.M--genrrted and human-annotated benchmarks are calculated using certain agreement metric (e.g., rank correlation). Metries such as sample correctness and diversity are computed to support a more fine-grained evaluation of benchmark quality (Yuan et al. 2025). However, existing studies lack consideration of the generator's self-bias, raising the risk of performance overestimation.
2.2 Self-Bias of LLMs
Variations in training corpora, model architectures, and training strategies cause LLMs to exhibit different domain-specific biases (Gallegos al. 2023). Among them, self-bias generally refers to the tendency of the LLM to favour its own generations when acting as a judge (Panickssery et al. 2024 Xu al. 2024 Bai et al. 2023). In this work, we extend the notion of self-bias to the setting self-bias
LaLM------Bnhhhatte Geeekratonn....... While judge-side self- bias primarily stems from content- based self-preference (Gu et al.) 2024). our experiments show that self-bias of benchmark generators arises from multiple, more intricate factors and thus requires mitigation efforts from multiple aspects 0
3 Preliminary
3.1 Formal Definition and Verification of Self-Bias
Humans tend perform better on tests they create themselves (Corrigan and Craciun] 2013 Saucier ct al. (2024). This motivates the necessity to investigate whether LLMs show similar self- when evaluated on benchmarks they generate, which previous studies have largely neglected (Huang et al.. 2024 Yuan et al. 2025). We define R(M)Mque .Mlab) as the performance (e.g. accuracy) of model M evaluated benchmarks where questions provided by M9ue and labels are provided by
"Unless otherwise noted, all subsequent mentions self-bias refers to that in LLM-as LLM-as-Benchmark, Bennraaak-Genneratorr..

---

Testers
0.4
M2: 6PT-4o mini
My
0.082
0.12
Mgi Qwen-Plus
Ma
0.068
0.11
Mg: Cloude 3.5 Haiku
Ma
0.39
0.13
M4: DeepSeak-Distill
M4
-0.021
0.006
Qwen-328
14
My: GPT-4o
14
0.008
Mg: QwQ-328
Mr
0.027
Gemini 2.0 Flosh
M2
Mg
Mg
Me
Mg
M
My
M1
M2
M₁
M4
Mg
Mg
My
-0.2
Benchmark Generators
Reasoning
Language Understanding
Figure 1: Evaluation bias of LLMs across tasks. The value at position (M;, Mi, represents M(M))MM) (Eq. (2). the evaluation bias of M, on the benchmark generated by M,
Mlab, As different benchmarks vary in difficulty, to compare the evaluation results across them, calculate the relative performance of M with respect to K reference models M as follows:
K- (M 1M9e (Maaab) Σ R (M AMeu,,Mb
R (M M M M MiK)
On this basis, the evaluation bias of M on benchmark generated by M can be calculated as:
B(M R(MM). M .M.k)
Ma Mhuuman .M
(2)
where M(M)MMuuma Mumhum M 1:K) denotes the relative performance on human-annotated bench- mark. Accordingly, the self-bias of M can be denoted as ((M)MM, abbr. B(M). A larger value of (JM) indicates greater degree of self-bias in the benchmark generated by the M.
Based on the definition above, we validate the existence of self-bias on math reasoning and language understanding tasks. We select human-annotated annotated MATH (Hendrycks et al., (221)) and MMLUP AMLU-PPr
(Wang et al. 2024) benchmarks for comparison and employ BenchMaker method (Yuan et al. 2025) for benchmark generation, which has been validated construct high-quality benchmarks (See detailed introduction of BenchMaker in Appendix and exact experimental settings in $5.1 The seven LLMs illustrated in Figure I are employed both generators, testers and reference models, with the results of evaluation bias presented accordingly. Based on Figure []] we identify two primary findings: (1) Models generally perform better benchmarks they generate themselves (highlighted by red boxes on the diagonal), exhibiting average self- self-biases of O. 103 and 0.058 across the two tasks, respectively. (2) Some models demonstrate prevalent negative evaluation bias on model-generater benchmarks, as reflected by the blue rows (c.g., MM in both tasks). We verify (see Appendix [ for details) that the reason lies in data contamination issue (Balloccu et al. 2024). which lead to the inflated relative performance R(MMMtuuma human M Mummnn 'M of these models on human-annotanl annotated benchmarks, subsequently resulting in negative evaluation bias according to Eq. 22
3.2 Disentangling of Self-Bias
After confirming the widespread presence of self-bias, we proceed investigate its underlying causes. Since benchmark typically consists of question set and a label set, and its presentation can be influenced by language style, we heuristically decouple self-bias into three sub-biases:
Language style bias B", The language style of questions generated by a model may align with what it is familiar with, enabling it to betler understand the questions and answer them correctly. We formally measure B as follows:
B RRM (M (uman)) human
M
R[MM)Muunne human human
M 1:k
where we measure how much M is overestimated on human-constructed benchmark whose question set has been paraphrased by M.
Models without data contamination consequently exhibit underestimated relative performance, reflected by the red rows (e.g., Mi in both tasks).

---

Question domain bias B", The domain of questions generated by model may largely align with the domain it excels in, making likely to answer them correctly. We formally measure 1 as follows:
(MM) MRMM (M); MM .MEK) R(M MM .Mbumun
,M
(4)
where we measure how much M is overestimated on a benchmark built from its gefereene questions (while paraphrased by humans to disentangle the impact of (*) and labels annotated by humans. Given the impracticality of manual annotating across multiple experimental settings, we adopt the powerful OpenAl o3-mini (OpenAl 2025) proxy for human annotators.
Wrong label bias B'. The model may produce incorrect labels, and the model's 's tendency to repeat the same mistakes during labeling and answering can lead to many false positives, inflating the accuracy. We formally measure B' as follows:
B' (M) RRMMMMhumm .M, .M. R[M human
.Mhuman
.M 1:K)
(5)
where we measure how much M is overestimated on a benchmark constructed from -urttewrittenn questions and M-generated labels. In Appendix C.I we further theoretically prove that models tend to perform better on samples labeled by themselves.
Table 1: Quantitative results of the decoupled sub-biases of B across tasks.
MATH REASONING
LANGUAGE UNDERSTANDING
B
B
B
B
B
B'
VALUE RELATIVE CONTRIBUTION (%)
0.1032
0.0205
0.0247
0.0921
0.0575
0.0061
0.0077
0.0545
14.96
17.99
67.05
8.967
11.29
79.74
Following the experimental setup in $3.1, Table rreports the magnitudes and relative contributions of each sub-bias. As we hypothesized, all three sub-biases are indeed present and collectively constitute the self-bias illustrated in Figure T We further observe that their relative contributions follow the order: B > B > B This indicates that, although both language style and question domain biases have an impact, the high alignment between the model acting labeler and tester contributes more significantly to the self-bias.
4 SILENCER Framework
4.1 Motivations
Since different LLMs generally exhibit distinct biases (Gallegos et al. 2023). the most straightforward strategy to mitigate self-bias to ensemble benchmarks generated by multiple LLMs M to achieve bias neutralization. However, this method suffers from two limitations. First, each sample still carries all three types of sub-biases from its generator, and the accumulation of these sub-biases leads to a distribution shift away from the unbiased sample domain. Meanwhile, since different LLMs exhibit varying degrees of self-bias, uniformly ensembling their generated benchmarks can be suboptimal. To this end, propose SILENCER (as shown in Figure 2). general framework for LLM-as- M--ns-Bencchark- Generator methods that (1) individually mitigates the three sub-biases at the sample level ($4.2). and (2) performs weighted ensemble of benchmarks from different generators the benchmark level The prompts used can be found in Appendix
4.2 Sample-wis wise Sub-biases Mitigation
methods (Huang et al.) 2024 Yuan et al. 2025) generally adopt an attribute- guided approach to ensure the diversity of the question domain Yu et al.) 2023). Specifically, certain generator I.LM M, is first prompted to generate set of attributes A, (e.g., topic) based on the task definition TD, with each attribute corresponding to multiple candidate values (e.g., sports). Afterwards, a list of (attribute, value) pairs is sampled for guiding the LLM to generate a question q: meeting the attributes, and corresponding label 4;
Attribute Integration. Current
A, AttributeGGeneraiion... (TD)
(6)
(ge, ()) = SampleGennration M. (AttributeSampling (At))

---

Attribute Generation
Attribute Integration
Benchmark Generation
Cross Paraphrase
Prediction Generation
Label Calibration
Bias Neutrrllzzng Benchmark Ensemble
performance matnx
A
A,
(gc, 4))
(7,4)
P
tosk
Ihercte convergance
descr- iption
ensemble weights
observed performance
debiased benchmark
D
Somple- Wise Debias
Question Domain
Language Style Debias
Wrong Label Debias
Benchmark Wise Debias
Debias
Figure 2: Overall illustration our SILENCER framework. Each generator LLM represented by unique color. Colored arrows indicate the process by which a certain LLM takes the content on the left as input and produces the content on right as output (excluding the gray arrows).
Since the attributes largely determine the content of the generated question, we look to mitigate the question domain bias of M, by suppressing the bias inherently contained in Ag:
Å, Attributenteegration,, M. (Ai.T TD)
(7)
where M, takes as input the attribute sets produced by all generators Mi.T and integrates the high-quality attributes aligned with the task definition TD, thereby deriving unbiased attribute A; for subsequent sample generation.
Cross Paraphrase. After each M₁ completes its own benchmark generation D, based on A, enhance the linguistic diversity of these benchmarks by having the generators paraphrase each other's question without altering the original meaning:
S QuestionParaphrase M. (q:), r r () chosen uniformly at random
(8)
Building on this, the question set of D, no longer exhibits single language style, thereby significantly suppressing 8
Label Calibration. Due to the limitations and invariance in knowledge and capabilities, the label l provided by model Mr may be incorrect, and its predictions tend to exhibit consistent errors, leading to obvious as shown in Table To this end, we consider leveraging the predictions from other generators p!:f assist M, in calibrating the old label 4e:
p Prediction, (g:)
T LabelCalibration 4..
Through jointly analyzing and comparing multiple predictions (with their rationales), Me is able to produce more reliable label 4, which, even when incorrect, is less likely to align with its erroneous predictions.
4.3
Benchmark-wise Self-bias Suppression
After obtaining the benchmarks D in which we consider how to ensemble them to further eliminate self-bias. Ideally, benchmarks with less self-bias should be assigned greater weight during ensembling, so that the resulting benchmark exhibits minimal overall self-bias. However, in real-world scenarios, we often lack high-quality human-annotated benchmark Dumman to evaluate the self-bias Bi.7 of DT.T. Therefore, we design the Bias-Neutralizing Ensemble Algorithm [ which uses the ensembled benchmark as proxy for D and iteratively optimizes the weights 01.7 used for benchmark ensembling.
have been mitigated at the sample level,
5

---

Algorithm/ Biias-euttralizing Ensemble Algorithm.
Require: Generated benchmarks DI.T. Relative performance of M; on Dj a (for [1, E (1,77).
Desired benchmark size N, Minimum value (default: 1c-6)
Ensure: Ensembled benchmark with low self-bias: D
1: 0i f- 0,
ai < T
for E [1,
2: while Ei=1 0;
new
ac > do
3:
anrr
4: 37 - Σラ=1 ==1 ag o for E [1, 7]
5:
now.
UpdateAlpha(z] 1:7 F1.r)
6:
Ct;
new
for i E [1
7: end while
8:
D
-
RandomSampling((D, D,,, IN oal)
Specifically, we first initialize the weight as T of each M, on all the generated benchmarks using 01:7 (model performance x 1IT can be calculated from the predictions in Eq. (9). Here, ã; essentially represents the performance ensembled benchmark ò of current iteration. Treating Ó as a proxy for Dhuman we use the algorithm UpdateAlpha(-) to update This process is repeated until convergence.
1 Then, we compute the weighted performance T; 1.
M, on the
Ideally, we can design an effective UpdateAlpha(.) such that Q1:r exhibits a clear negative correlation with Bi.r, then each iteration will yield a ensembled benchmark Í with lower self-bias. This, in turn, means that D can serve as better proxy for Dhuman, thereby facilitating a more accurate estimation of 01.7 in the next iteration. To identify the most effective UpdateAlphal we explore the following three strategies in our early-stage experiments:
Self-Bias, Naively, we use F. as the basis and compute the reciprocal of each model's estimated self-bias its ensemble weight:
UpdateAlpha SelfBias
(1TT T1rr);
(10)
x x4
Accuracy. The Koaning-KKugge Effect (Kruger and Dunning 1999) suggests that individuals with lower competence tend to overestimate their abilities. Thus, we directly use the accuracy as weight:
Accuuer UpdateAlpha Accuracy 1:T Z1:T): = 3t
(11)
Evaluation Consistency, As stronger self-bias tends to result in less reliable evaluation results, quantify the self-bias of each generated benchmark by assessing its evaluation consistency with the ensembled benchmark, which in turn determines its ensemble weight:
HipdteeAlhaaComee Consistency (I I:T BI.T; PearsonCorrelatiionss... Zi.T)
(12)
To verify the effectiveness of the three strategies, we compute the Pearson correlation between the reciprocal of the ensemble weights derived from each strategy and the true self-bias under the settings described in 13.1 The results show: Consistency(O..r (i.isteecc(0.86)))... indicating that Evaluation Consistency better reflects the degree of self-bias. We hypothesize that the reason lies in the fact that, when there is discrepancy between ă and the true model performance, the Evaluation Consistency strategy can fully leverage the original model perfor- mance data 21.7 compared to the Self-Bias and Accuracy strategies, thus partially mitigating error accumulation during the iterative process. We also evaluate the correlation between the ensemble weights and the effectiivnees of each generated benchmark (effectivenees of D, is de- fined as the Pearson correlation between reference models' performances on D, and on Dhuman): Consistency(O..g) (0.965) > Self- (olff-Bas(O..88)) weights derived from the Evaluation Consistency strategy also better reflect the evaluation effective- ness each D, thereby enabling the ensembled benchmark to more accurately assess model performance. Based these findings, we adopt UpdateAIpha Consisteeyy (.) as our default setting. To ensure the non-negativity of the weights and the convergence of the iteration (the proof of convergence provided in Appendix C.2 we adopt the following form in practice, where is set as le-6:
(aff-Bias00.500) > Accuracy(0.33)
Accuracy(O.2411 This indicates that the ensemble
UpdateAlpha Consistemy
Consistency (1):T, ZIT)) ReLU(PeassonCorccationi (rr TIT) +e (13)

---

Experiments
In this section, we conduct a comprehensive evaluation of the proposed SILENCER framework. Specifically, we introduce the detailed experimental settings in $5.1 In $5.2 we validate the effectiveness of SILENCER in mitigating self-bias and enhancing evaluation effectiveness, examine its generalizability across generators, tasks, and ablations on its individual components. In $5.3, we further investigate other impact SILENCER on the generated benchmark with fine-grained metrics. In $5.4 we explore how varied settings influence the self-bias and evaluation effectiveness of the generated benchmarks in practice.
methods, and perform
5.1 Experimental Settings
Tasks. We select three tasks to evaluate the cross-task effectiveness of SILENCER, with each task paired with a
benchmark for comparison: math reasoning (MATH (Hendrycks et al., (02T)) language understanding (MMLU-Pro (Wang et al. 2024)), and common- sense reasoning (HellaSwag (Zellers al. 2019)).
Models. As for generators, we select seven representative open- and closed-source LLMs: GPT-
4o mini OpenAl 2024b). Qwen-Plus (Qwen-Team) 2024), Claude 3.5 Haiku (Anthropic 2025). Dirtoc--DDstll--weeeee
(a GPT-4o ( QwQ-32B Qwen Team 2025). and Gemini 2.0 Flash (DeepMind 2025). As for reference models, ensure the robustness of experimental results, we additionally include nine more LLMs (see Appendix D except for generators, resulting in a 16-model reference set. For each setting, we report the averaged result of 1,000 seeds, each time randomly selecting T (T > 2) generators and K reference models (K > 7) from the candidate pool.
2025) as the baseline method for bench- mark generation and evaluate its performance under the ST..ENCER framework. We take the task descriptions used in Yuan et al. (2025) as input and have the generators construct benchmarks of a specified size. We additionally include simple and broadly applicable baseline method AttrPrompt (Yu et al. 2023) to further assess the mross-method generalizability of SILENCER.
Baselines. We primarily adopt BenchMaker Yuan et al.
Details. Since multiple settings are involved, we default the benchmark size N for each sub-task within a task (e.g.. MATH- Algebra) to 50. We explore the impact of larger sizes in Section $5.4 To ensure fairness in the comparison, we keep the benchmark size consistent before and after ensembling. The sampling temperature for the LLMs is set to 1. In addition to self-bias, we also compute the Pearson correlation rp of reference models' performance between model-generated and high-quality human-annotated benchmarks to assess the evaluation effectiveness the generated benchmarks.
5.2 Effectiveness and Generalizability of SILENCER
Overall Effectiveness and Ablation Studies. The main experimental results shown in Table 2 The benchmarks generated using the baseline methods exhibit significant self- bias, consistent with the findings in $3.1 The results in rows 2-5 demonstrate that our proposed sample-level strategies effectively mitigate different sub-biases, thereby reducing self-bias B and improving evaluation effectiveness Tp- Among them, the Label Calibration strategy yields the largest gain by alleviating the wrong label bias B¹, which is consistent with the conclusion in $3.2 that 13 is the major contributor to self-bias. Building on the mitigation of sub- biases, directly ensembling the -gneeraee benchmarks can substantially reduce B and significantly improve ry (row 6), which is consistent with our hypothesis in Section 4.1 Meanwhile, after applying our proposed reweighted ensemble strategy, B is reduced nearly zero and rp reaches the highest value across all tasks, verifying the effectiveness of Biutaleutr Ensemble Algorithm.
Generalizability. Across all settings, observe that SILENCER consistently yields significant reductions in (from 0. 13 to 0.009 on average) and improvements in тр (from 0.655 to 0.833 on average), demonstrating its generalizability across tasks, generators and methods.
5.3 A Closer Look at What SILENCER Brings
To gain deeper insights into how SILENCER affects benchmark generation, we analyze from four perspectives with fine- grained metrics (on Math Reasoning task by default).
7

---

Table 2: Main results of the baseline methods with different modules of SILENCER. and rp denote the average self-bias and evaluation effectiveness (Pearson correlation) across multiple settings. All results of rp are statistically significant (p-value < 0.05).
MATH REASONING
LANGUAGE UNDERSTANDING
COMMONSENSE REASONING
METHODS
B4.
rp
B4
Tpl
B
Tp.
BASELINE METHOD: BENCHMAKER
YUAN
2025
BASELINE
0.1205
0.7863
0.0802
0.6816
0.1102
0.6479
W. ATTRIBUTE INTEGRATION: B
0.1179
0.7892
0.0748
0.6907
0.1059
0.6557
CROSS PARAPHRASE: B
0.1074
0.7952
0.0759
0.6840
0.1107
0.6511
LABEL CALIBRATION: B
0.1109
0.8302
0.0583
0.7058
0.1021
0.6582
BY 1 B'
0.1002
0.8371
0.0517
0.7145
0.0934
0.6627
W: BY 1" W. ENSEMBLE
0.0218
0.9216
0.0076
0.7953
0.0178
0.7826
W. SILENCER
0.0140
0.9386
0.0011
0.8340
0.0053
0.8149
BASELINE METHÓD: ATTRPROMPT
YU ET AL..
2023)
BASELINE
0.1427
0.6179
0.1148
0.5843
0.1093
0.6106
W: SILENCER
0.0187
0.8641
0.0125
0.7725
0.0074
0.7762
Sample Semantic Distribution. The presence B and B results significant discrepancies the semantic distributions of samples generated by different generators, as shown in Figure 3(a) After applying Cross Paraphrase alone (Figure 3(b) and both Cross Paraphrase and Attribute Integration simultaneously (Figure 3())., the separation among benchmarks from different LLMs is visibly reduced, indicating that 5 and 8 have been mitigated. We further compute the average Maximum Mean Discrepancy (Gretton et al. 2012) and Wasserstein Distance (Villani et al., 2008) between benchmarks from different generator pairs, both of which measure the divergence between two distributions. After applying ST.ENCCE, both metrics decrease (from 0. 174 to O. 0.2555 and from 12.267 to 10.903, respectively), providing additional evidence for this mitigation.
DispSeik-Diit Quan----e2-
Qwer Plus
(a) Raw Generated Benchmark
(b) Mitigate B*
(c) Mitigate 8 and B
Figure 3: T-SNE visualization results of the raw generated benchmarks (a), those with mitigated language style bias (b), and those with both language style and question domain biases suppressed (c) the Language Understanding task. The presence of 13 clusters is probably due to the task
comprising 13 sub-tasks. We use text-embeeding-ada--00 (OpenAl 2024c) as the embedding model.
Benchmark Diversity. By expanding the question domain and improving the monotonicity of language style, we observe a clear improvement in the diversity of benchmarks. As shown in the Table after applying SILENCER, the average word frequency entropy (Montahaei al. 2019) increases from 9.749 to 10.078, and the 2-gram count (Li et al.) 2016) rise from 33,560 to 36,932.
Label Accuracy. Besides suppressing B3, we further examine the impact of the Label Calibration strategy on label accuracy. We use OpenAl o3-mini as the judge model to assess label correctness. Experimental results in Table 3 show that Label Calibration can also improve the label accuracy from 0.877 to 0.892, possibly because the model corrects some of its erroneous labels based on right predictions from other generators. This effect further promotes improved evaluation effectiveness.
Ensemble Weight. Finally, we calculate the average Pearson correlation between the reciprocal of the ensemble weights estimated by SILENCER and the self-biases across all settings. The results show high correlation 0.8252, validating that SILENCER, as expected, assigns greater weights to benchmarks with less self-bia bias, thereby leading higher-quality ensembled benchmark.

---

Table 3: The impact of the three proposed sample-level strategies for mitigating sub-biasee benchmark diversity (entropy and 2-gram) and label accuracy.
ENTROPYT
2-GRAM
LABEL ACCURACY RAW
GENERATORS
Raw
SILENCER
RAW
W: SILENCER
SILENCER
DEEPSEEK- CEETSSEK-DDSTILL--CE DISTILL-QWEN-33 -WBN
9.691
9.828
25493
24747
0.855
0.862
QWEN-PLU
9.307
10.149
25488
36707
0,819
0.846
GPT-4o MINI
9.610
9.931
23231
25742
0,912
0.927
CLAUDE 3.5 HAIKU
9.274
9.989
14172
25286
0.833
0.884
GPT-40
9.942
10.023
55440
56010
0.893
0.899
GEMINI 2.0 FLASH
10.300
10.363
56985
53378
0,900
0.891
QwQ-32B
10.120
10.269
34111
36657
0,927
0.936
AVERAGE
9.749
10.078
33560
36932
0.877
0.892
Table 4: The impact of generator number T.
METRICS
T= 3 T
T =
5
T
6
T
NAIVE ENSEMBLE: SELF BIAS B
0.031 0.020
0.015
0.011
0.008
NAIVE ENSEMBLE: EVALUATION EFFECTIVENESS Tpp
0.909 0.920
0.925
0.932
0.937
REWEIGHTED ENSEMBLE: SELF BIAS B
0.030 0.016 0.010
0.008
0.005
REWEIGHTED ENSEMBLE: EVALUATION EFFECTIVENESS Pp
0,912
0.936
0.949
0.951
0.955
REWEIGHTED ENSEMBLE: WEIGHT ESTIMATION ACCURACY Pp
0.453
0.734
0.884
0.922
0.948
5.4 Impact of Setting Variants
Generator Number T. SILENCER primarily suppresses self-bias by leveraging mutual bias neutral- ization among different generators. Therefore, examine the effect of varying the generator number T. As shown in the Table 4 with a fixed ensembled benchmark size N, both Naive Ensemble and Reweighted Ensemble strategies lead reduction B as increases. This suggests that using more generators can introduce greater bias diversity, thereby better diluting belf-bias and improving Evaluation Effectiveness Tp- We also observe that larger results in higher consistency between the reciprocal of the estimated ensemble weight and self-bias (the last row), indicating that the proposed Reweighted Ensemble benefits more from scaling T than Naive Ensemble.
Benchmark Size N. We also investigate the effect of scaling the benchmark size N. As N increases (50 100 + 200), the self-bias B under the Naive Ensemble strategy remains nearly unchanged (0.021), while under the Reweighted Ensemble strategy, B shows a consistent decline (0.014 0.012 + 0.011). We hypothesize that the Reweighted Ensemble benefits from larger N because more samples enable better estimation of model performance, thereby improving the precision ensemble weight estimation we have observed: 0.788 0.820 + 0.839.
Conclusions
In this work, systematically analyze and mitigate the Self-Bias of LLM-as LLM--as-enchmarrk Benchmark-Generattor Specifically, we validate the existence of Self-Bias and attribute to three sub-biases. Based on this, we propose SILENCER, framework that helps generate high-quality, seff-bias-silenced benchmarks from multiple levels. We validate the effectiveness and generalizability of SILENCER through experiments across various settings. We further explore the finer- anar-ee impacts of SILENCER on the generated benchmarks and provide insights into the setting variants selection.
Limitations. To mitigate self-bias and attain higher-quality benchmarks, our SILENCER framework incurs approximately 30% additional token costs. However, we believe this overhead is worthwhile because (1) users generally prioritize the accuracy of evaluation results
cost, and (2) benchmark sizes used in practice are typically not very huge. For example, we often use MATH-500 (Lightman al. 2024 instead of the full MATH benchmark. In such cases, the extra cost is acceptable.
Broader Impacts. We believe that the SILENCER framework help mitigate potential biases in synthetic data, thereby reducing the risk of distributional shifts associated with internet-based corpora.

---

Acknowledgments
This work supported by Beijing Natural Science Foundation (No.4222037, L181010).
References
Anthropic. 2025. Claude 3.5 haiku.
Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He. Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et 2023. Benchmarking foundation models with language-mode -aaeaaa--aa-----aan examiner. Advances in Neural Information Processing Systems, 36:781 12-7716677
Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondrej Dusek. 2024. Leak, cheat, repeat,
Data contamination and evaluation malpractices in closed-source Ilms) In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2024 Volume 1: Long Papers, St. Julian's Malta, March 17-22, 2024, pages 67-93, Association for Computational Linguistics.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1-45.
Hope Corrigan and Georgiana Craciun. 2013, Asking the right questions: Using student-written exams as innovative approach to learning and evaluation. Marketing Education Review, 23(1):31-36.
DeepMind. 2025. flash
Gemini 2.0 flash.
DeepSeek-AL. 2025. Deepseck-distill-qqen-- DeepSeek-R1-DDe Ri-issiii---iiiill-----l- Qwen-32B
htttp::/hugginggfcccccoo///e cotpe//aeecccccceeeeeeeeeeeeeee.
Jasper Dekoninck, Mark Niklas Müller, and Martin T. Vechev. 2024. Constat: Performance- based contamination detection in large language models) In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024.
Eitan Farchi, Shmulik Froimovich, Rami Katan, and Orna Raz. 2024. Automatic generation of benchmarks and reliable LLM judgment for code tasks CoRR, abs/2410.21071.
Isabel O. Gallegos, Ryan Rossi, Joe Barrow, Md. Mehrab Tanjim, Sungchul Kim, Franck Dernon- court, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2023. Bias and fairness in large language models: A survey CORR, hbs/22009.0070..
Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falk- man Olsson, Stan-Stansslas Denain, Anson Ho. Emily de Oliveira Santos, Olli Järviniemi, Matthew Barnett, Robert Sandler, Matej Vrzala, Jaime Sevilla, Qiuyu Ren, Elizabeth Pratt, Lionel Levine, Grant Barkley, Natalie Stewart, Bogdan Grechuk, Tetiana Grechuk, Shreepranav Varma Enu Enuganlla. and Mark Wildon. 2024. Frontiermath: A benchmark for evaluating advanced mathematical reasoning in Al CORR, ab482411.04872.
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. 2012. A kernel two-sa test. The Journal of Machine Learning Research, 13(1):723-773
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. 2024. A survey on llm-as-s llm-as-a-judee CoRR, 0bs/24111.1599.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948
10

---

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt, 2021. Measuring mathematical problem solving with the MATH dataset) In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual.
Yue Huang, Siyuan Wu, Chujie Gao, Dongping Chen, Qihui Zhang, Yao Wan, Tianyi Zhou, Chaowei Xiao, Jianfeng Gao, Lichao Sun, et al. 2024. Datagen: Unified synthetic dataset generation via large language models. In The Thirteenth International Conference on Learning Representations.
Justin Kruger and David Dunning. 1999. Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead inflated self-assessments. Journal of personality and social psychology, 77(6): 1221.
Fangyu Lei, Qian Liu, Yiming Huang, Shizhu He, Jun Zhao, and Kang Liu. 2023. S3eval: A synthetic scalable, systematic evaluation suite for large language models) CoRR, abs/2310.15147.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-rromoting
objective function for neural conversation models] In NAACL IILT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California. USA, June 12-17. 2016, pages 110-119, The Association for Computational Linguistics.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let's verify step by step) In The Twelfth International Conference Learning Representations, TCLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
Gaurav Maheshwari, Dmitry Ivanov, and Kevin El Haddad. 2024. benchmark CoRR, abs/2409. 11968.
Efficacy of synthetic data as
Ehsan Montahaei, Danial Alihosseini, and Mahdieh Soleymani Baghshah. 2019. Pointly measuring diversity and quality in text generation models, CoRR, abs11944.39997
OpenAl. 2024a. http://////aatttt.o....../.ooo/r
OpenAl. 2024b. Gpt-4o mini. http:////paaaaar.....ea.cc/oo//a
OpenAl. 2024c. ext-embeedding-aaa--000 https://paattorr.pppaall.cooo/ embeddings
OpenAI. 2025. Openai o3 and 04-mini system card
Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024, LLM evaluators recognize and favor their own generations In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024.
Arkil Patel, Siva Reddy, and Dzmitry Bahdanau. 2025. How to get your Ilm to generate challenging problems for evaluation. arXiv preprint arXiv:2502.14462
Yotam Perlitz, Ariel Gera, Ofir Arviv, Asaf Yehudai, Elron Bandel, Eyal Shnarch, Michal Shmueli- Scheuer, and Leshem Choshen. 2024. [Bnchmarr agreement testing done right: A guide for LLM benchmark evaluation CoRR, abs/2407.13666.
Qwen-Team. 2024.
Qwen-plus.
what -s-t-iss-oe isntein---en---- 11m
Qwen-Team. 2025. Qwq-32b.
Donald A Saucier, Ashley A Schiffer, and Tucker L Jones. 2024. "exams by you": Having students write and complete their own exams during the covid-19 pandemic. Teaching of Psychology. 51(2):158-165.
Cédric Villani al. 2008. Optimal transport: old and new, volume 338. Springer.
11

---

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhramil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang. Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark CoRR, abs/2406.01574.
Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Wang. 2024. Pride and prejudice: LLM amplifies self-bias in self-refinement) In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 15474-15492. Association for Computational Linguisties.
Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander J. Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. 2023. Large language model attributed training data generator: A talc of diversity and bias, In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA. USA, December 10 16, 2023.
Peiwen Yuan, Shaoxiong Feng. Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, and Kan Li. 2025. Llm-powered benchmark factory: Reliable, generic, and efficient. arXiv preprint arXiv:2502.01688
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791 4790- Association for Computational Linguistics.
12

---

A LLMassBenccnaark-CGnnrroo Baselines
We introduce the Be-ch-aa-enccaak----e baselines used in the experiments as follows:
A.1 AttrPrompt
AttePrompt (Yu al. 2023) a widely used data synthesis method. Its core idea is first prompt generator model to produce a set of task-related attributes, each associated with several values; then, based on this, random of (attribute, value) pairs is selected each time constraints guide the model in generating samples that satisfy both task and attribute constraints.
A.2 BenchMaker
The BenchMaker (Yuan et al. 2025) method aims to generate high-quality benchmarks by optimizing multiple attributes such as diversity, accuracy. and difficulty. For a given task description, sample diversity is enhanced through attribute-driven generation and in-batch diversity boosting strategy; benchmark difficulty and controllability are improved via a difficulty diffusion mechanism and strategy guidance; sample accuracy is elevated through stepwise self-correction and conflict-guice -enfliccttgieed contrastive discrimination. Experiments demonstrate that this method produces high-quality benchmarks that are well-aligned with human annotations. In experiments, adopted the task descriptions following Yuan et al. (2025). Due to the use of a large number of generators, we used a benchmark size of 50 per sub-task instead of the 500 used in BenchMaker.
B Data Contamination Issue Identified in Our Preliminary Experiments
In our preliminary experiments, we observed that some models exhibited relatively better performance on the human-annotated benchmark compared most (or even all) model-generated benchmarks, resulting in general negative evaluation bias, as shown in Figure 3 We hypothesize that this may be due to data leakage specific the suman-annotater benchmark. To verify this, we employed the Benchmark- Specific Contamination detection method proposed by ConStat (Dekoninck et al. (024), benchmark by fitting the performance
which infers the expected performance on the distributions of various models on synthetic benchmarks. The deviation between the expected and the Math Reasoning task, ConStat indicated statistically significant contamination (p-value < 0.05) for DeepSeek Daasseek--Dsstll-eeee 32B (M4) and QwQ-32B (Mg). On the Language Understanding task, significant contamination was detected for DesSeek-Disill--eeeeeee
actual performance used to estimate the degree of contamination.
(M4), GPT-4o (Ms), QwQ-32B (Ma), and Gemini 2.0 Flash (M7). These results align with the models exhibiting consistent negative evaluation bias in Figure supporting hypothesis.
C Proof Section
C.1
Self-Labeling Inflates Expected Accuracy
Setup. Suppose we have T models, denoted by (M1, M2,..., Mr each mapping an input x to a probability distribution over possible outputs in a label space y. For each model M,, let
Pi x) Pr(model M, outputs y x), y V.
We examine two expected accuracy measures:
(1) Self-labeling expectation Ej: the case where each model both generates the label and predicts
T
E =
Σ yEy
Pa(u
1)) poly 1
=
Σ
paly x))
T
T
tabel
predict
i=
NEY
13

---

(2) bross-labeling expectation Ey: the case where a model My generates the label and another model
M, predicts it:
E2 T
Pyly p
yEy
label
predict
Claim. We aim prove that E1 2 E2.
Proof. Subtract Eg from E
T
E E2 T
1
Σ
Σ
pily x)
Psly x)) z).
T
yEy
vE
Rewriting this via a common factorization yields
T
F1 - -
(pe(v x;)) + (py(v x))² 2p x)) m)
2T
i=1
j=1 vEY
pily p5(U x) 2 0.
27
vEy
Hence
E This proves that using each model as its own label generator cannot be worse (in terms of expected accuracy) than having labels generated by a different model.
E
E2-
C.2 Convergence of the Proposed Bias-Neutralizing Ensemble Algorithm
Setup. Let X E [0, NNN be a non-negative matrix (the performance matrix in our scenario) whose columns are æ1, 22, .... zN RRV. We use N to denote the number of generators to avoid confusion with the transpose symbol. Consider an weight vector a = (ai,...,ay) where
N-1
N-1 {ZERN 24 1
2 11.
For u. V RN, let
Σ (uk -)( - 0)
N
N
corr(u,v)
N
Uks
0
N
Σ
uk-
E (uk= u)2 v Eit1( (2kk 8)
k=1
Iterative process.
Fix 5 > 0. Define Ft N - N-1 by
f ==oo,
(a) ReL ReLU corr (7, (z,
+ 8,
i I....,N.
(14)
(b((a)
a
new
i= 1 N.
N E))
where Rt)U(t)- = max(0, (0,t).
1. Existence of a fixed point
Lemma (Existence). There exists at least one a" E N-1 with =
Proof. The simplex A N-1 is compact and convex. The map F continuous because (i) 7 Xa depends linearly on a, (ii) corr(ā, (x)) is continuous in ã (outside degenerate constant vectors), and (iii) ReLU continuous and 1-Lipschitz on R. Hence Fa continuous self-map on compact convex set, and Brouwer's fixed-point theorem guarantees at least one fixed point.
14

---

2. Uniqueness and contraction
Theorem (Uniqueness & Contraction). With d > O, the map F in 14 strict contraction N-1 under the e norm. Consequently, the fixed point in Lemmal iis unique.
Sketch. Take a, A AN1 and set ca(a) corr(oo, ca(2) corr(X2,, 2,). Because ReLU 1-Lipschitz,
(0)(0) - 0=(=)] = (eLUU(((a)) - (=))( (c((a) - cx(2))-
The mapping a + Xa is linear, while the correlation Lipschitz on the bounded set [0, 1]M Thus there is L < co with llø(a) - Φ(2)1 & L| z||1.
Now decompose the update as
1 (a)) = da) Aa
Φ(a) 51
(Φ(a) 81 | Xa llo(a) - 51/11 N8
[0, 1).
[δ((a) ó1|[1"
Because > 0, we have sup, Aa =: q < 1. A direct calculation using the above representation yields (a) (z)ll s a/|a z||1. Hence J is strict contraction, so it admits exactly one fixed point.
3. Global convergence to the unique fixed point
Theorem (Global convergence). Let (()) A and define a (k+1) (a There exists unique a² E A with F (a") ) = a", and
[1a a"1l1 S 4* ||a - a" 0<q<l,
so the convergence is geometric.
Proof. By Theorem I
F* is contraction complete metric space, hence Banach's fixed-point theorem applies directly.
Remark. If for some iterate all correlations are non-positive, ReLU sets them 0, d(a) 81 and F maps every point to the uniform distribution. In that (rare) event the process converges in single step, which is even faster than the geometric rate guaranteed above.
D Reference Model List
Apart from the seven generator LLMs, we also include the following LLMs in our reference model set:
2.
3. 3atpap/ss.iiii.3../r.rr/r//////e
4. DeepSeees-Rl-DDiiillAAaaaa---2
http:///hugginnfacccco//eeeeepe
DeeiSekkRR--Dstti1--11mm--
5. htwp://hhhaeielaarer////aaaaaa/a
6. Llama-3.1-70B-Instruccc
httes///hngiigghbcccc///aaaaaaaa
1-70B-Instruc
7. Deepseets-R--DDsiil-LLAAm-BBB-B
http:://uuginngfacc.cc//eessss
JeepSeek-Ri-Distll--aaaa.
8. Athene-70B. httpae/eo...rrrr//ggggggeeeo///r -7B
9. gemmaa2-27b-it.
https://huginngaae.ccogggll/ 2-27b-i1
15

---

E
Prompt List
Prompt for Attribute Integration
Your task is to help create benchmark with multiple choice questions about "[(ability))". Specifically:
#*## Task**** ([task content]}
*** Questionn Content*** ((task content analysis)} #*## *** #### ***0ptions*** ((option analysis)}
Below is a set of attributes and their corresponding values.
([ttriiuuees}
step 1. Please evaluate them sequentially on two dimensions:
Dimension 1: Assess whether the attribute is closely related to the appropriate attribute for diversity of generated sample.
given task and whether it serves controlling Dimension 2: Evaluate whether the values corresponding to the attribute are comprehensive. If not, suggest any additional revisions that should be included.
step 2. Based on this evaluation, please consolidate the high Aigh-qualit, attributes and their values. The consolidation process should include the folloving steps:
1. For attributes with high redundancy, merge them (particularly focusing on merging their values).
2. Provide the optimized attributes and their corresponding values
step 3. Finally, please output all the attributes that you consider relevant and meaningful for evaluating the given task along with their values. The output should include no fewer than attributes and the more, the better
Please present final output step in following format (Use *###hhe Final Attributes###' separate this part from the previous analyses and end with ###End of Final Attributes#4#)
###The Final #ttributes:::
(Name of Attribute 1): (Value 1)=u(ealue 2Vauue
(e.g., Reasoning Steps: 1-2##3 -4##5 -##5-64#>6) 6##>6)
(Nlmm of Attribute 2}: (Value 1)#a(ealue 2)=u(Value 3)
(Name of Attribute N}: (Value 1)#*(value 2} 3) ###End of Final Attributes###
Prompt for Cross Paraphrase
You are an assistant skilled in rewriting samples. Given sample: #*# Sample Start ###
Question: finput question content) Options: (input options content) Label: {input correct option}
#*# Sample End
Your task is to execute the following process for the given sample: Step 1: Analyze the question and understand the correaponding correct ansver.
Step 2: Think about how to rewrite the question while keeping the original meaning of the question, options, and label unchanged. Focus language style transformation rephrasing (e.g.,
16

---

synonym substitution), but ensure the correctness of the rewritten sample remains intact.
Step 3: Output the revritten question the following format (you can also refer to the template below):
Template:
### Sample Start ###
Question:
{question content)
Options:
{options content]
Label:
[correct option}
###
Sample End #*#
Prompt for Label Calibration
Question: ([question)}
Candidate Reaponses: {[candidates))
Your task is conduct the steps below strictly:
1. Read the given Question
2. Analyze all
given Candidate Responses One by One
3. Based on the given Candidate Responses, determine the correct
answer through comprehensive analysis
4. Output your final answer in the format: ** answer is ##{[Cpptio *
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We substantiate our claims with both experimental evidence and theoretical
analysis.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
We have included detailed assumptions and proof in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided detailed experimental settings in the experiments section and detailed module designs of our method.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification:
We will release our code soon in the github.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//guiii wwwiii/gguie///eeeeeeoeic CoodeSubeiiieeee ssi wswinnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided detailed experimental settings in the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For each setting, we run 1,000 times and report the average results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
20

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: We primarily use API calls to access the model generator, and the invocation depends the service provider.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpp::/neuurpp.
Answer: [Yes]
Justification:
We have read the NeurlPS Code of Ethics and followed it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed this in the conclusion section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have correctly cited all the assets use.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets released.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "JkhzRzwNCv", "reviewer": "Reviewer_8ACx", "summary": "This paper investigates self-bias in benchmarks generated by large language models (LLMs), where an LLM tends to overestimate its own performance on benchmarks it creates due to alignments in question domains, language styles, and labeling errors. It decouples self-bias into sub-components (question domain bias, wrong label bias, and language style bias) and proposes the SILENCER framework to mitigate it. At the sample level, SILENCER introduces three strategies: Attribute Integration (to diversify question domains), Cross Paragraphs (to vary language styles), and Label Calibration (to correct wrong labels). At the benchmark level, it employs a Bias-Neutralizing Ensemble algorithm that weights benchmarks inversely to their estimated self-bias. The framework is evaluated on three tasks (math reasoning, language understanding, commonsense reasoning), seven LLM generators, and two benchmark generation methods, demonstrating a 27.2% average improvement in evaluation consistency with human-annotated benchmarks (Pearson correlation from 0.655 to 0.833).", "questions": "1. Have you compare the annotations of o3-mini with human annotators? \n2. How would results change if genuine human raters supplied labels for a subset of samples? A small‑scale human audit would bolster the claim that Silencer improves true benchmark quality.\n3. Have you tested Silencer on free‑form generation benchmarks (e.g. writing or coding tasks) where label calibration is non‑trivial?\n4. How does Silencer perform on difficult datasets?\n4. The ensemble assumes bias correlates with consistency. What happens when one generator is much weaker (or stronger) than all others?", "limitations": "The authors acknowledge extra compute cost.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper explores sources of self-bias in LLM-as-a-judge, which is important but underexplored.\n2. The silencer framework combines sample-level prompting strategies with a benchmark-level ensemble algorithm that dynamically estimates and neutralizes bias.\n3. The experiment result shows lower bias and better consistency with human-annotated benchmarks.\n\n\nWeaknesses:\n1. My main concern is that the experiments adopt o3-mini as a proxy of human annotators. o3-mini still has huge bias.\n2. The experiments are conducted only on three datasets that are not difficult enough. Results may vary on datasets that are super difficult. Besides, these three datasets may not fully capture broader domains (e.g., code generation or multimodal tasks)", "was_revised": true, "final_justification": "The rebuttal has addressed most of my concerns.", "rebuttal": "**Dear Reviewer 8ACx, thank you for recognizing the motivation, method and experimental results of our work. Also, we sincerely appreciate you taking the time to read our manuscript and provide valuable feedback. Below are our responses to all the concerns you raised, which will be incorporated into the updated version to further enhance the quality of our submission.**\n\n---\n> **`My main concern is that the experiments adopt o3-mini as a proxy of human annotators. o3-mini still has huge bias. Have you compared the annotations of o3-mini with human annotators? How would results change if genuine human raters supplied labels for a subset of samples? A small‑scale human audit would bolster the claim that Silencer improves true benchmark quality.`**\n\nWe fully understand your concern. In Section 3.2, when quantifying the Question Domain Sub-bias, due to the large scale of the test data (over 1,000 samples), we employed o3-mini as a human proxy to (1) paraphrase the generated questions and (2) provide labels for these questions. We understand your concern regarding whether using o3-mini might introduce additional bias in terms of language style  and label that could affect the results. Below, we analyze the impact in details:\n\n(1) The questions generated by the LLM generator inherently carry both Question Domain Sub-bias and Language Style Sub-bias. To eliminate the Language Style sub-bias, we used o3-mini to paraphrase these questions. To analyze whether the paraphrased questions may introduce an o3-mini-specific Language Style Bias, we calculate as follows:\n\n$\\mathcal{B}^{s}_{\\mathcal{M}^{\\text{o3-mini}}}(\\mathcal{M}) = R(\\mathcal{M}|\\text{para}\\_{\\mathcal{M}^{\\text{o3-mini}}}(\\mathcal{M}^{\\text{human}}), \\mathcal{M}^{\\text{human}},\\mathcal{M}^{\\text{ref}}\\_{1:K}) - R(\\mathcal{M}|\\mathcal{M}^{\\text{human}}, \\mathcal{M}^{\\text{human}},\\mathcal{M}^{\\text{ref}}\\_{1:K})$\n\nThis metric evaluates the performance difference of the seven evaluated LLMs on the o3-mini-paraphrased human benchmark versus the original human benchmark. The results show that for Math Reasoning and Language Understanding tasks, $ \\mathcal{B}^{s}\\_{\\mathcal{M}^{\\text{o3-mini}}}(\\mathcal{M}) $ is measured as **-0.0009** and **0.0004**, respectively. In comparison, $ \\mathcal{B}^{q} $ is **0.0247** and **0.0077**, respectively. **Thus, the bias introduced by o3-mini paraphrasing is negligible and does not affect the experimental conclusions.**\n\n(2) We understand your concern that o3-mini’s labels might still contain errors, potentially with bias, and thereby affect the measured Question Domain sub-bias. Following your suggestion, we conducted a human audit. For each sub-task, we randomly sampled 20 generated questions and manually (at least with a bachelor’s degree) verified o3-mini’s labels. The audit revealed that the error rates of o3-mini’s labels were **7.8%** for Math Reasoning and **6.9%** for Language Understanding. We further tested whether these labeling errors introduced bias using:\n\n$$\\mathcal{B}^{l}\\_{\\mathcal{M}^{\\text{o3-mini}}}(\\mathcal{M}) = R(\\mathcal{M}|\\mathcal{M}, \\mathcal{M}^{\\text{o3-mini}},\\mathcal{M}^{\\text{ref}}\\_{1:K}) - R(\\mathcal{M}|\\mathcal{M}, \\mathcal{M}^{\\text{human}},\\mathcal{M}^{\\text{ref}}\\_{1:K})$$\n\nThe results show that for Math Reasoning and Language Understanding tasks,\n$ \\mathcal{B}^{l}\\_{\\mathcal{M}^{\\text{o3-mini}}}(\\mathcal{M}) $ is **0.0014** and **0.0007**, respectively.\nAgain, these values are negligible compared to $ \\mathcal{B}^{q} $ (**0.0247** and **0.0077**), **indicating that the labeling bias introduced by o3-mini is minimal.**\n\n**In summary,  the experimental evidence demonstrates that using o3-mini as a human proxy for measuring sub-bias does not affect the conclusions presented in Section 3**. This is intuitively reasonable: models with training trajectories more similar to o3-mini may benefit from its proxy setting, while models more dissimilar to o3-mini may be slightly disadvantaged, thereby leading to the two effects offsetting each other.\n\nWe sincerely thank you for this insightful suggestion. \n\n---\n> **`The experiments are conducted only on three datasets that are not difficult enough. Results may vary on datasets that are super difficult. Besides, these three datasets may not fully capture broader domains (e.g., code generation or multimodal tasks).`**\n\nWe understand and agree with your suggestion that conducting experiments on more challenging and diverse tasks (e.g., open-domain tasks) would better validate the generalizability of Silencer. We would also like to clarify that prior works on LLM-as-benchmark-generator generally focused on a limited set of closed-domain tasks. For instance, [1] conducted evaluations on the same three tasks as ours, [2] tested on Math Reasoning, Language Understanding, Commonsense Reasoning, and QA tasks, while [3] covered QA and Gender Bias classification tasks.\n\nThis is mainly because assessing LLM-as-benchmark-generator approach is costly (requiring a large number of tester inference results), and closed-source tasks are more conducive to unifying the benchmark generation framework.\n\nNevertheless, we agree that verifying the generalizability of Silencer on a more challenging, free-form generation task such as **competitive coding** would be valuable. To this end, we chose OJBench [4] as high-quality human-annotated benchmark. We modified BenchMaker’s label generation setting to prompt it to generate 10 test cases that thoroughly evaluate the function’s correctness, thereby enabling support for free-form generation tasks. During Label Calibration, Silencer accepts test cases provided by other LLMs as input, and jointly analyzes and compares all candidate test cases to regenerate calibrated 10 test cases. We followed [4] in evaluating the Pass@1 metric, and chose Python as the programming language with PyPy3 interpreter. All other experimental settings are aligned with Section 5.1.\n\n**Results on Competitive Coding Task**\n\n|Methods|$\\mathcal{B}$|$r_p$|\n|-|:-:|:-:|\n|Baseline|0.1336|0.7172|\n|w. Attribute Integration|0.1204|0.7346|\n|w. Cross Paraphrase |0.1098|0.7461|\n|w. Label Calibration|0.1130|0.7487|\n|w. Silencer|0.0169|0.8229|\n\nAs shown above, our findings on this task are consistent with other tasks:\n- **All modules of Silencer effectively suppress self-bias, ultimately reducing self-bias to near zero. This significantly improves the alignment between the de-biased benchmark and the human-annotated benchmark (from 0.7172 to 0.8229 in $r_p$).**\n- Among the sub-bias mitigation strategies, Label Calibration brings the greatest improvement in $r_p$.\n\nMeanwhile, unlike other tasks, Cross Paraphrase results in a larger reduction of self-bias. Our case study indicates that the longer questions in competitive coding amplify language style bias, making its mitigation through Cross Paraphrase more effective.\n\nAdditionally, we found that the average pass@1 of the tested models was only 27%, which is significantly lower than their accuracies on other tasks (all above 35%). This demonstrates that **Silencer maintains strong generalizability even on the more challenging, free-form generation task of competitive coding**. In the future, given sufficient resources, we plan to evaluate Silencer on a wider range of tasks (such as multi-modal and writing tasks) to further explore its generalizability.\n\n---\n> **`The ensemble assumes bias correlates with consistency. What happens when one generator is much weaker (or stronger) than all others?`**\n\nWe consider this suggestion highly insightful, as it helps explore the stability of Silencer under extreme conditions. To this end, we extend the experimental setup of the Math Reasoning task by separately adding an LLM to the original set of 7 generator LLMs:\n- A weaker LLM, Qwen2.5-7B-Instruct (models weaker than this lack sufficient instruction-following capability to support the task).\n- A stronger LLM, OpenAI o4-mini (the strongest API model available to us, which significantly outperforms other generators on mathematical reasoning tasks).\n\nFor both extended experiments, we measure the **Pearson correlation coefficient $r_p$ between the reciprocal of evaluation consistency and self-bias**. \n\n**Extended Results on Math Reasoning Task**\n\n|Settings |  Original 7 Generators |  w. a Much Weaker Generator |  w. a Much Stronger Generator |\n|-|:-:|:-:|:-:|\n|$r\\_p$|0.861|0.882|0.873|\n\nWe observe that the added weaker generator exhibits the largest self-bias and the lowest evaluation consistency with the ensembled benchmark, which aligns with our original hypothesis and leads to an increase in $r_p$.\nSimilarly, the added stronger generator exhibits the smallest self-bias and the highest evaluation consistency with the ensembled benchmark, also resulting in an increase in $r_p$.\n\nMoreover, we find that when adding a much weaker generator, directly ensembling benchmarks from different generators with equal weights significantly reduces the consistency with the human benchmark (from 0.9216 to 0.9008). In contrast, by leveraging the assumption that bias is consistent with evaluation consistency, Silencer substantially suppresses the ensemble weight of the weaker generator, mitigating the impact on the consistency with the human benchmark (only a slight drop from 0.9386 to 0.9369).\nWhen adding a much stronger generator, Silencer also yields additional benefits (from 0.9386 to 0.9451) compared to equal-weight ensembling (from 0.9216 to 0.9242).\n\n**These results empirically confirm that under extreme conditions, Silencer’s ensemble strategy guided by evaluation consistency leads to higher-quality benchmarks**.\n\n**References**\n\n[1] LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient.\n\n[2] Unigen: A unified framework for textual dataset generation using large language models.\n\n[3] Discovering language model behaviors with model-written evaluations.\n\n[4] OJBench: A Competition Level Code Benchmark For Large Language Models."}, {"review_id": "cP6jainBub", "reviewer": "Reviewer_sV2H", "summary": "The paper studies the self-bias of LLM generated benchmarks with respect to the same or similar LLMs to be evaluated or trained on the generated data. Typically, an LLM will be biased to overperform on a benchmark generated by itself. The paper classifies this self-bias into three categories: question domain, language style and labeling and proposes techniques using reference models and human annotated datasets to measure these self-biases. The authors show experimentally the significant self-bias of each LLM in a collection of seven popular LLMs from the industry, on Math Reasoning and Language Understanding datasets. Based on these measures, the paper proposes attribute integration, cross paraphrase and label calibration methods respectively to alleviate the three kinds of self-bias, and further proposes a weighted ensemble approach to combine the benchmarks generated by different LLMs in the collection so that the resultant dataset has low self bias. Experiments on 7 industry LLMs (with 9 others to complete the reference set) are conducted using ablations over baseline generator methods, which show that the three types of self-bias are improved using the proposed method.", "questions": "1. It will be helpful to provide examples of generator outputs: question, answer and label for the benchmarks generated/used in the experiments.\n2. Line 241: define $r_p$ formally.\n3. In Table 2, what is the difference between “$\\mathcal{B}^q, \\mathcal{B}^s, \\mathcal{B}^l$ w. Ensemble” and SILENCER ?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is a fairly comprehensive study of self-bias in LLM generated Benchmarks.\n2. The notions and metrics of self-bias introduced by the paper are natural, well reasoned, and also shown to be prevalent in popular models.\n3. The proposed self-bias mitigation method combines individual approaches targeting each of the categories of self-bias – along with weighted ensembling – and is shown experimentally to significantly decrease the self bias for several LLMs across two tasks.\n\nWeaknesses:\n1. On line 115, the authors state that they use o3-mini as a proxy for human annotators. Could this not make similar models (e.g. those trained on similar datasets) erroneously perform better in terms of self-bias in the experiments.\n2. The presented experimental results seem to have a few gaps: (i) only aggregated bias scores over all generator models are included in Tables 1 and 2, instead of those for each of the seven models and, (ii) the intermediate ablations are not included for AttrPrompt baseline method in Table 2, effectively, the SILENCER method is fully evaluated only for BenchMaker.", "was_revised": true, "final_justification": "The main concern of using o3-mini as a proxy for human annotators has been addressed by the authors, through additional experiments showing that the model has low error rate compared to human annotators, and can therefore serve as a proxy for the latter. Apart from this the authors have included detailed experimental results and examples of the generator outputs. My concerns have largely been addressed.", "rebuttal": "**Dear Reviewer sV2H, thank you for acknowledging the motivation of our work, the comprehensiveness of our research, and the effectiveness of our method. Also, we sincerely appreciate you taking the time to read our manuscript and provide valuable feedback. Below are our responses to the concerns you raised, which will be incorporated into the updated version to further enhance the quality of our submission.**\n\n---\n> **`On line 115, the authors state that they use o3-mini as a proxy for human annotators. Could this not make similar models (e.g. those trained on similar datasets) erroneously perform better in terms of self-bias in the experiments?`**\n\nWe fully understand your concerns, and we greatly appreciate that your question is based on a thorough reading of our paper. Our response is as follows:\n\nIn Section 3.2, when quantifying the Question Domain Sub-bias, due to the large scale of the test data (over 1,000 samples), we employed o3-mini as a human proxy to provide labels for the generated questions. We recognize your concern that labels provided by o3-mini may still contain errors, and that such errors might be biased, thereby affecting the measured Question Domain Sub-bias.\n\nTo address this, for each sub-task, we randomly sampled 20 generated questions and manually verified the labels given by o3-mini. All human annotators held at least a bachelor's degree and were instructed to mark the answer as \"uncertain\" if they were unsure about its correctness. In the end, all annotators provided confident validation results. The verification showed that the error rates of o3-mini’s labeling were **7.8%** for Math Reasoning and **6.9%** for Language Understanding tasks.\n\nFurthermore, we examined whether such labeling errors introduce additional bias, as measured by the following formula:\n\n$$\\mathcal{B}^{l}\\_{\\mathcal{M}^{\\text{o3-mini}}}(\\mathcal{M}) = R(\\mathcal{M}|\\mathcal{M}, \\mathcal{M}^{\\text{o3-mini}},\\mathcal{M}^{\\text{ref}}\\_{1:K}) - R(\\mathcal{M}|\\mathcal{M}, \\mathcal{M}^{\\text{human}},\\mathcal{M}^{\\text{ref}}\\_{1:K})$$\n\nExperimental results show that, on Math Reasoning and Language Understanding tasks, the average values of $ \\mathcal{B}^{l}\\_{\\mathcal{M}^{\\text{o3-mini}}}(\\mathcal{M}) $ across 7 generators were **0.0014** and **0.0007**, respectively. Compared to the measured values of $ \\mathcal{B}^{q}$ (**0.0247** and **0.0077**), the magnitude of this error is negligible and does not affect our experimental conclusions.\n\n**In summary, the results demonstrate that using o3-mini as a human proxy to evaluate sub-biases in LLM-as-benchmark-generator does not impact the conclusions in Section 3**. In fact, this is understandable: models with training trajectories similar to o3-mini may perform better under this setting due to higher homogeneity (indeed, we observed that $\\mathcal{B}^{l}\\_{\\mathcal{M}^{\\text{o3-mini}}}(\\mathcal{M}^{\\text{GPT-4o}})$ was 0.0062 for Math Reasoning), while more distinct models (e.g., $\\mathcal{B}^{l}\\_{\\mathcal{M}^{\\text{o3-mini}}}(\\mathcal{M}^{\\text{Gemini 2.0 Flash}})$ was -0.0051 for Math Reasoning) may perform worse. As a result, these effects nearly offset each other on average, as shown by our experimental results above.\n\nWe sincerely thank you for your suggestion. By incorporating this additional analysis, we believe our results are now more comprehensive and convincing.\n\n---\n> **`The presented experimental results seem to have a few gaps:(i) only aggregated bias scores over all generator models are included in Tables 1 and 2, instead of those for each of the seven models.(ii) the intermediate ablations are not included for AttrPrompt baseline method in Table 2. Effectively, the SILENCER method is fully evaluated only for BenchMaker.`**\n\nDue to space limitations, we did not include these fine-grained experimental results in the main tables, but instead reported the results that support the core conclusions of our study. We fully acknowledge your suggestion; presenting such details would help researchers gain a more comprehensive understanding of the distribution of self-bias and the effectiveness of Silencer from different perspectives. Below are the experimental results you mentioned. We will add these information to the revised version.\n\n**Detailed Results of Table1-Math Reasoning** \n \n| Models | $\\mathcal{B}$|$\\mathcal{B}^s$|$\\mathcal{B}^q$|$\\mathcal{B}^l$|\n|-|-|-|-|-|\n|GPT-4o mini|0.0923|-0.0067|0.0064|0.0675|\n|Qwen-Plus|0.0576|0.0162|0.0088|0.0455|\n|Claude 3.5 Haiku|0.3895|0.0357|0.0524|0.2464|\n|DeepSeek-Distill-Qwen-32B|-0.0210|-0.0146|-0.0021|0.0135|\n|GPT-4o|0.0567|0.0028|0.0004|0.0741|\n|QwQ-32B|0.0076|0.0750|0.0731|0.0844|\n|Gemini 2.0 Flash|0.1398|0.0355|0.0341|0.1140|\n|Avg.|0.1032|0.0205|0.0247|0.0921|\n\n**Detailed Results of Table1-Language Understanding** \n\n| Models | $\\mathcal{B}$|$\\mathcal{B}^s$|$\\mathcal{B}^q$|$\\mathcal{B}^l$|\n|-|-|-|-|-|\n|GPT-4o mini|0.1230|0.0152|0.0213|0.0765|\n|Qwen-Plus|0.1108|0.0186|0.0127|0.0772|\n|Claude 3.5 Haiku|0.1297|0.0136|0.0167|0.1234|\n|DeepSeek-Distill-Qwen-32B|0.0057|-0.0128|-0.0017|0.0083|\n|GPT-4o|0.0129|0.0092|0.0031|0.0288|\n|QwQ-32B|-0.0060|-0.0090|0.0086|0.0229|\n|Gemini 2.0 Flash|0.0268|0.0082|-0.0066|0.0444|\n|Avg.|0.0575|0.0061|0.0077|0.0545|\n\n\n**Detailed Results of Table2-AttrPrompt Baseline** \n\n |_|Math|Reasoning|Language|Understanding|Commonsense|Reasoning|\n |-|:-:|:-:|:-:|:-:|:-:|:-:|\n|Methods| $\\mathcal{B}$|$r_p$|$\\mathcal{B}$|$r_p$|$\\mathcal{B}$|$r_p$|\n|Baseline|0.1427|0.6179|0.1148|0.5843|0.1093|0.6106|\n|w. Attribute Integration:$\\mathcal{B}^q$|0.1211|0.6158|0.1058|0.6109|0.1087|0.6149|\n|w. Cross Paraphrase:$\\mathcal{B}^s$|0.1169|0.6349|0.1102|0.5953|0.1016|0.6095|\n|w. Label Calibration:$\\mathcal{B}^l$|0.1048|0.6508|0.1025|0.6228|0.0988|0.6327|\n|w. $\\mathcal{B}^q$ $\\mathcal{B}^s$ $\\mathcal{B}^l$|0.0957|0.6677|0.1034|0.6341|0.0927|0.6384|\n|w. $\\mathcal{B}^q$ $\\mathcal{B}^s$ $\\mathcal{B}^l$ w. Ensemble|0.0340|0.8255|0.0203|0.7437|0.0257|0.7505|\n|w. Silencer|0.0187|0.8641|0.0125|0.7725|0.0074|0.7762|\n\nThe table presents the full evaluation of the SILENCER method based on the AttrPrompt Baseline, providing further validation for the effectiveness of the proposed sub-modules.\n\nIt should be noted that for Table 2 of the original manuscript, if we were to present detailed results for each task (3 in total), each baseline method (2 in total), and each experimental setting (7 in total), and report individual bias scores for each model, this would result in a total of 3 × 2 × 7 = 42 tables. Each table would contain 7 rows (for the seven generator models) and 4 columns ($\\mathcal{B}$, $\\mathcal{B}^q$, $\\mathcal{B}^s$, $\\mathcal{B}^l$), which would be extremely difficult to present. In practice, we have already visualized these detailed results as figures, which makes them easier to read and analyze. We will include these visualizations in the appendix of the revised version.\n\n---\n>  **`It will be helpful to provide examples of generator outputs: question, answer and label for the benchmarks generated/used in the experiments.`**\n\nWe fully agree with your suggestion. Below are examples of samples generated by the Silencer framework on the two tasks. We will include more cases in the appendix of the revised version.\n\n**Math Reasoning - Algebra**\n\nQuestion:\n\nAn investor divides \\\\$100 into two parts, $x$ and $y$, invested in two different accounts. The first account offers an annual return modeled by the equation \\( R_A = x^2 - 10x + 25 \\), and the second account offers a return of \\( R_B = y \\). The total return after one year is \\\\$125, and the total investment is \\\\$100. Determine the amount invested in the first account.\n\nCandidates:\n\nA. \\\\$5  B. \\\\$10  C. \\\\$11  D. \\\\$15\n\nLabel: C\n\n**Commonsense Reasoning**\n\nQuestion:\n\nIn a South Asian city, a lawyer is providing legal advice to a client regarding a delicate family issue that involves a family dispute with relatives. During the consultation, the client becomes clearly distressed, expressing feelings of betrayal and anxiety. Despite the lawyer's attempts to reassure them, the client shows hesitation to proceed with legal action, stating that they now believe the issue can be resolved through family mediation. Considering the lawyer's responsibilities, the cultural value placed on family unity, and the client's emotional state, what is the most likely next step the lawyer will take?\n\nCandidates:\n\nA. Advise the client to seek professional counseling to address their anxiety.\n\nB. Strongly push the client to file a lawsuit immediately, ignoring their wish for mediation.\n\nC. Explain the legal process for family mediation and help the client contact a qualified mediator.\n\nD. Suggest the client abandon legal proceedings entirely to preserve family harmony.\n\nLabel:  C\n\n---\n> **`Line 241: define rp formally.`**\n\nWe have formally defined $r_p$ in line 233-235: “In addition to self-bias, we also compute the Pearson correlation $r_p$ of reference models' performance between model-generated and high-quality human-annotated benchmarks to assess the evaluation effectiveness of the generated benchmarks.” It can be calculated as：\n\n$$\nr\\_p = \\text{PearsonCorrelation}([\\text{Performance}(\\mathcal{M}^{\\text{ref}}\\_i\\ |\\ \\text{human-annotated benchmark})]\\_{i=1}^K,[\\text{Performance}(\\mathcal{M}^{\\text{ref}}\\_i\\ |\\ \\text{model-generated benchmark})]\\_{i=1}^K)\n$$\n\n---\n> **`In Table 2, what is the difference between “$B_q,B_s,B_l$ w. Ensemble” and SILENCER?`**\n\nThe distinction is as follows: “$B_q$, $B_s$, $B_l$ w. Ensemble” refers to equally weighted ensembling of benchmarks generated by different generators. In contrast, “Silencer” denotes weighted ensembling of benchmarks generated by different generators, where the weights are set as the inverse of the self-bias magnitude estimated by Algorithm 1. This approach enables further suppression of self-bias at the benchmark level, as reflected in the improvements shown in Table 2 compared to “$B_q$, $B_s$, $B_l$ w. Ensemble”."}, {"review_id": "XYpkaZcKKs", "reviewer": "Reviewer_iMmg", "summary": "The paper discusses the phenomenon of self-bias in large language models (LLMs) used as benchmark generators, proposing a framework called SILENCER to mitigate this bias and improve evaluation accuracy. The framework includes strategies to mitigate sub-biases at both the sample and benchmark levels. ​ SILENCER employs three main strategies at the sample level:\n\n- Attribute Integration to create domain-unbiased question sets. ​\n- Cross Paraphrase to enhance linguistic diversity and reduce language style bias. ​\n- Label Calibration to improve label accuracy and reduce wrong label bias. ​\n\nAt the benchmark level, a Bias-Neutralizing Ensemble algorithm is used to weigh benchmarks based on their self-bias, improving overall evaluation effectiveness. \n\nSILENCER reduces self-bias to near zero, achieving an average improvement in Pearson correlation from 0.655 to 0.833 with high-quality human-annotated benchmarks. ​The framework is tested across three tasks and seven LLM generators, confirming its generalizability and  effectiveness. Detailed ablation studies validate the contribution of each component in SILENCER, providing insights into the impact of factors like the number of generators and benchmark size on evaluation effectiveness.", "questions": "Is there any evaluation on the paraphrases generated wrt quality and correctness? Overall, qualitative assesments are lacking in the paper and could be interesting to look at.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "- The paper introduces a novel framework, SILENCER, to mitigate self-bias in LLM-as-Benchmark-Generator methods, addressing an under-explored issue in the field. ​\n- The paper provides detailed explanations of the methodology, including formal definitions, experimental settings, and theoretical proofs, ensuring transparency and understanding. ​\n- The work has broad implications for improving the reliability of LLM-generated benchmarks, which are increasingly used in AI research and applications.", "was_revised": true, "final_justification": "No changes based on author discussion. All limitations I highlighted were addressed.", "rebuttal": "**Dear Reviewer iMmg, thank you for your comprehensive recognition of the motivation, novelty, and significance of our work.\nAlso, we sincerely appreciate you taking the time to read our manuscript and provide valuable feedback. Below are our response to the concern you raised, which will be incorporated into the updated version to further enhance the quality of our submission.**\n\n---\n> **`Is there any evaluation on the paraphrases generated wrt quality and correctness?Overall, qualitative assessments are lacking in the paper and could be interesting to look at.`**\n\n\nWe sincerely appreciate your suggestion to conduct a qualitative analysis of the paraphrased questions, as this provides us with a more intuitive understanding of how the items change through paraphrasing. To this end, we randomly selected ten samples generated by DeepSeek-Distill-Qwen-32B from the Language Understanding Task and analyzed the characteristics of the paraphrased items produced by different generator models. Below is one example:\n\n**\\<DeepSeek-Distill-Qwen-32B generated one\\>**\n\nQuestion:\nHow did the introduction of cash crops in 19th-century African, Caribbean, and Southeast Asian colonies primarily affect their economies and societies?\n\nOptions:\n\nA. Enhanced social equality through equitable distribution of cash crop profits.  \nB. Increased economic diversification and self-sufficiency in local markets.  \nC. Decreased labor availability due to migration to urban areas.  \nD. Improved infrastructure development to support cash crop exports.  \nE. Enhanced cultural preservation through traditional agricultural practices.\nF. Economic dependency on colonial powers due to export-focused agriculture.  \nG. Reduced environmental impact due to sustainable farming practices.  \nH. Improved local economies through diversified crop production.  \nI. Reduced reliance on colonial powers through export profits.  \nJ. Increased political autonomy through economic independence. \n\n\n**\\<paraphrased by QWQ-32B\\>**\n\nQuestion:\nWhat were the main impacts on the economies and societies of 19th-century African, Caribbean, and Southeast Asian colonies as a result of the introduction of cash crops?\n\nOptions:\n\nA. Improved social equality due to fair sharing of cash crop profits.  \nB. Boosted economic diversity and self-reliance in local markets.  \nC. Reduced workforce availability as people moved to cities.  \nD. Enhanced infrastructure to facilitate cash crop exports.  \nE. Strengthened preservation of traditional farming methods.  \nF. Economic reliance on colonial powers due to export-oriented agriculture.  \nG. Lower environmental effects thanks to sustainable farming.  \nH. Economic growth through varied crop production.  \nI. Less dependence on colonial powers through export earnings.  \nJ. Greater political independence through economic self-sufficiency.\n\n- **QWQ-32B’s paraphrase adheres to the principles of minimal alteration and maximum fidelity, without changing the answer direction.**\n\n**\\<paraphrased by Qwen-Plus\\>**\n\nQuestion:\nHow did the economies and societies of African, Caribbean, and Southeast Asian colonies mainly change as a result of the introduction of cash crops during the 19th century?\n\nOptions:\n\nA. Promoted equal social status by fairly sharing earnings from cash crops.  \nB. Encouraged a wider variety of economic activities and greater independence in regional markets.  \nC. Caused a shortage of workers as people moved toward cities.  \nD. Strengthened transportation and communication systems to facilitate the export of cash crops.  \nE. Maintained cultural traditions by continuing customary farming techniques.  \nF. Created economic reliance on colonial rulers due to a focus on export agriculture.  \nG. Lowered environmental damage through eco-friendly farming methods.  \nH. Boosted domestic economies by producing a range of crops.  \nI. Lessened dependence on colonial authorities by generating income from exports.  \nJ. Expanded political freedom as a result of economic self-reliance.\n\n- **Qwen-Plus’s paraphrase enhances clarity with a more formal and definitive style; its explanations are more explicit but do not go beyond the intended scope, resulting in high quality.**\n\n\n**\\<paraphrased by GPT-4o mini\\>**\n\nQuestion:\nWhat was the primary impact of cash crop cultivation in the 19th century on the economies and societies of African, Caribbean, and Southeast Asian colonies?\n\nOptions:\n\nA. Promoted social equity through fair distribution of cash crop revenues.  \nB. Led to greater economic diversification and self-reliance in local markets.  \nC. Reduced labor availability as people migrated to urban centers.  \nD. Fostered infrastructure improvements to facilitate cash crop exports.  \nE. Strengthened cultural preservation via traditional farming methods.  \nF. Created economic dependency on colonial authorities due to a focus on export-oriented agriculture.  \nG. Minimized environmental damage through sustainable farming techniques.  \nH. Boosted local economies through a variety of crop production.  \nI. Decreased reliance on colonial powers through profits from exports.  \nJ. Enhanced political independence through economic self-sufficiency.  \nK. Improved literacy rates through agricultural education programs.\n\n- **GPT-4o mini’s paraphrase introduces a new option K, which does not align with our prompt instruction that paraphrasing should only modify the linguistic style without altering the substantive content. However, after examining the other nine cases, we found this to be an isolated occurrence.**\n\n**\\<paraphrased by Claude3.5 Haiku\\>**\n\nQuestion:\nExamine the primary socioeconomic transformations induced by the introduction of cash crop economies in colonized territories across Africa, the Caribbean, and Southeast Asia during the 19th century.\n\nOptions:\n\nA. Stimulated localized agricultural innovation and technological adaptation.\nB. Promoted horizontal economic integration among indigenous communities.\nC. Generated significant population mobility and demographic shifts.\nD. Catalyzed infrastructural development through colonial investment.\nE. Preserved traditional agricultural methodologies and cultural practices.\nF. Established systemic economic subordination to metropolitan colonial powers.\nG. Minimized ecological disruption through sustainable agricultural strategies.\nH. Expanded agricultural diversity and market complexity.\nI. Facilitated economic emancipation through international trade mechanisms.\nJ. Enhanced indigenous political agency through economic transformations.\n\n- **Claude 3.5 Haiku’s paraphrase systematically academicizes both the correct and distractor options, increasing the difficulty and confusion while maintaining correctness.**\n\n**\\<paraphrased by GPT-4o\\>**\n\nQuestion:\nWhat was the primary impact of cash crop cultivation introduced in 19th-century colonies across Africa, the Caribbean, and Southeast Asia on their economies and societies?\n\nOptions:\n\nA. Promoted social equity by fairly distributing cash crop revenues.  \nB. Fostered economic self-reliance and diversification in domestic markets.  \nC. Reduced the availability of labor due to urban migration.  \nD. Stimulated infrastructure development to facilitate cash crop exports.  \nE. Preserved cultural traditions through indigenous farming methods.  \nF. Created economic reliance on colonial powers through export-oriented agriculture.  \nG. Minimized ecological damage through sustainable agricultural practices.  \nH. Strengthened local economies by introducing a variety of crops.  \nI. Lessened dependence on colonial authorities by generating export income.  \nJ. Advanced political independence through economic self-sufficiency.\n\n- **GPT-4o’s paraphrase demonstrates a natural, balanced, and stable style, without expanding the set of options or altering the difficulty structure.**\n\n**\\<paraphrased by Gemini 2.0 Flash\\>**\n\nQuestion:\nWhat was the principal consequence of introducing cash crop cultivation in 19th-century African, Caribbean, and Southeast Asian colonial territories on their economic systems and societal structures?\n\nOptions:\n\nA. It led to greater social equity through the fair distribution of agricultural profits.  \nB. It fostered increased economic variety and self-sufficiency within local markets.  \nC. It resulted in a decline in available labor due to significant urban migration.  \nD. It spurred significant infrastructure enhancements primarily to support export activities.  \nE. It reinforced cultural preservation by maintaining traditional farming methods.  \nF. It established economic reliance on imperial powers due to an agriculture system centered on exports.  \nG. It minimized environmental degradation through the adoption of sustainable farming practices.  \nH. It strengthened domestic economies via the diversification of agricultural output.  \nI. It decreased dependence on colonial authorities through substantial export earnings.  \nJ. It advanced political self-governance through the achievement of economic independence.\n\n- **Gemini 2.0 Flash’s paraphrase is characterized by more comprehensive and explanatory narratives.**\n\n**In summary**, each paraphrasing model exhibits a distinct linguistic style, which qualitatively reveals the language style sub-bias. At the same time, through detailed case studies, we found that apart from the single instance with GPT-4o mini introducing a new option (which does not affect the correctness of the question), all paraphrased questions (a total of 6 × 10 = 60 cases) maintained alignment with the original items in both content and assessed capability, demonstrating high quality. This indicates that paraphrasing for language style is well within the capability of current LLMs.\n\n**We will include this discussion in our revised version. We believe this addition will further improve the quality of our work.**"}, {"review_id": "mobpHKELuU", "reviewer": "Reviewer_wEJW", "summary": "This paper focuses on investigating a timely problem with LLM-generated benchmarks, specifically the self-bias of LLM-as-Benchmark-Generator. In this phenomenon, LLMs tend to achieve overestimated performance on benchmarks they generate themselves. To tackle this problem, the paper first formally defines and quantifies self-bias and empirically demonstrates its prevalence across multiple LLMs and evaluation tasks. Sequentially, the paper heuristically decouples self-bias into three contributors with different relative contributions: question domain bias, language style bias, and wrong label bias. Finally, the paper proposes a new framework, SILENCER, for benchmark generation to address these issues, which mitigates the self-bias at both the sample and benchmark levels. Experimental results on three tasks have validated the effectiveness and generalizability of SILENCER.", "questions": "1. Missing reference: \n   - AutoBencher: Towards Declarative Benchmark Construction. ICLR 2025.", "limitations": "See above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Pros:**\n1. The problem addresses a highly practical and timely issue of NLG evaluation: despite the recent emergence of using LLMs as benchmark generators, potential biases in this paradigm remain underexplored. This paper provides valuable insights into possible biases of LLM-as-Benchmark-Generator and corresponding mitigation strategies.\n2. Overall, the paper is well-written, with clear and logical progression from problem identification and analysis to solution development. Each stage is supported by relevant experiments, effectively substantiating its claims.\n3. The paper conducts extensive experiments, including different models, tasks, and ablation studies. The experimental results sufficiently demonstrate the effectiveness of the proposed SILENCER framework.\n\n**Cons:**\n1. As shown in Figure 1, the self-bias of LLM-as-Benchmark-Generator varies substantially across different base models and tasks. A detailed analysis of these differences would greatly enhance the paper. For instance, the two reasoning models, DeepSeek-Distill-Qwen-32B and QwQ-32B, appear to consistently exhibit the lowest levels of self-bias.  In addition to self-bias, there appears to be evaluation bias between different models as well  (for example, between M1 and M7 in the Math Reasoning task). It would be valuable for the authors to further investigate whether the SILENCER framework is effective in mitigating cross-model evaluation bias.\n2. The decomposition of self-bias in Section 3.2 seems a little ad hoc. The authors could further clarify the completeness and rationality of the sub-bias decomposition empirically or theoretically.\n3. For Section 4.3, considering that the Bias-Neutralizing Ensemble Algorithm is an iterative algorithm, it would be beneficial for the authors to analyze the number of convergence steps and the evolution of self-bias during the iterative process in their experiments.", "was_revised": true, "final_justification": "Thanks for the author's response. I think the additional discussion improves the clarity of Silencer. I decide to maintain my score and still be positive about this paper.", "rebuttal": "**Dear Reviewer wEJW, thank you for recognizing the significance of our work, the clarity of our writing, and the effectiveness of our method. Also, we sincerely appreciate you taking the time to read our manuscript and provide valuable feedback. Below are our responses to the concerns you raised, which will be incorporated into the updated version to further enhance the quality of our submission.**\n\n---\n> **`As shown in Figure 1, the self-bias of LLM-as-Benchmark-Generator varies substantially across different base models and tasks. A detailed analysis of these differences would greatly enhance the paper. For instance, the two reasoning models, DeepSeek-Distill-Qwen-32B and QwQ-32B, appear to consistently exhibit the lowest levels of self-bias. In addition to self-bias, there appears to be evaluation bias between different models as well (for example, between M1 and M7 in the Math Reasoning task). It would be valuable for the authors to further investigate whether the SILENCER framework is effective in mitigating cross-model evaluation bias.`**\n\nWe appreciate your insightful suggestions, and indeed, the two phenomena you mentioned also puzzled us during our experiments: (1) $\\mathcal{M}\\_4$ and $\\mathcal{M}\\_6$ consistently exhibited the lowest levels of self-bias across multiple tasks; (2) in the Math Reasoning task, $\\mathcal{M}\\_1$ and $\\mathcal{M}\\_7$ consistently showed higher evaluation bias.\n\nThrough our subsequent systematic analysis, we identified the underlying cause of these phenomena as the **data contamination issue , as described in lines 97–100 of our paper**. Using the ConStat method proposed in [1], we verified in Appendix B (lines 747–752) that, on the MATH dataset, both DeepSeek-Distill-Qwen-32B ($\\mathcal{M}\\_4$) and QwQ-32B ($\\mathcal{M}\\_6$) face statistically significant risks of data contamination (p-value < 0.05). This indicates that the performance and relative performance of $\\mathcal{M}\\_4$ and $\\mathcal{M}\\_6$ measured on the human-annotated benchmark are overestimated (e.g., $R^{\\text{observed}}(\\mathcal{M}\\_4|\\mathcal{M}^{\\text{human}},\\mathcal{M}^{\\text{human}},\\mathcal{M}^{\\text{ref}}\\_{1:K})$). According to the formula for relative performance (Equation 1):\n\n$$\nR(\\mathcal{M}|\\mathcal{M}^{\\text{que}},\\mathcal{M}^{\\text{lab}},\\mathcal{M}^{\\text{ref}}\\_{1:K})=\\frac{K \\cdot R(\\mathcal{M}|\\mathcal{M}^{\\text{que}},\\mathcal{M}^{\\text{lab}})}{\\sum\\_{k=1}^K R(\\mathcal{M}\\_k^{\\text{ref}}|\\mathcal{M}^{\\text{que}},\\mathcal{M}^{\\text{lab}})}\n$$\n\nwe can see that these inflated performances indirectly lead to lower relative performance for other models (such as the $\\mathcal{M}\\_1$ and $\\mathcal{M}\\_7$ you mentioned) on the human-annotated benchmark (e.g., $R^{\\text{observed}}(\\mathcal{M}\\_1|\\mathcal{M}^{\\text{human}},\\mathcal{M}^{\\text{human}},\\mathcal{M}^{\\text{ref}}\\_{1:K})$). Furthermore, according to Equation 2:\n\n$$\n\\mathcal{B}(\\mathcal{M}|\\mathcal{M}^{\\text{gen}}) = R(\\mathcal{M}|\\mathcal{M}^{\\text{gen}}, \\mathcal{M}^{\\text{gen}},\\mathcal{M}^{\\text{ref}}\\_{1:K}) - R(\\mathcal{M}|\\mathcal{M}^{\\text{human}}, \\mathcal{M}^{\\text{human}},\\mathcal{M}^{\\text{ref}}\\_{1:K})\n$$\n\nwe can see that $\\mathcal{B}(\\mathcal{M}\\_4|\\mathcal{M}^{\\text{gen}})$ will be underestimated since $R^{\\text{observed}}(\\mathcal{M}\\_4|\\mathcal{M}^{\\text{human}},\\mathcal{M}^{\\text{human}},\\mathcal{M}^{\\text{ref}}\\_{1:K})$ is overestimated, while $\\mathcal{B}(\\mathcal{M}\\_1|\\mathcal{M}^{\\text{gen}})$ will be overestimated since $R^{\\text{observed}}(\\mathcal{M}\\_1|\\mathcal{M}^{\\text{human}},\\mathcal{M}^{\\text{human}},\\mathcal{M}^{\\text{ref}}\\_{1:K})$ is underestimated, which is consistent with the phenomena observed by both you and us in Figure 1. The same explanation applies to the observations in the Language Understanding task, as detailed in Appendix B.\n\nThank you again for your careful reading. We will highlight and clarify this part in the revised version of the paper.\n\n---\n> **`The decomposition of self-bias in Section 3.2 seems a little ad hoc. The authors could further clarify the completeness and rationality of the sub-bias decomposition empirically or theoretically.`**\n\nThank you very much for your suggestion. We will clarify the rationale and completeness of decomposing the self-bias of LLM-as-Benchmark-Generator into question domain sub-bias, wrong label sub-bias, and language style sub-bias from two perspectives.\n\n**First, regarding the rationale**, two empirical studies in the education field investigating students making their own test papers [2,3] provide supporting evidence:\n\n- Students tend to focus on “topics they consider important or are good at”; open-ended responses in [2] frequently mention “being able to focus on what I think are the most important points,” which leads to higher grades. At the same time, [3] also notes that student-generated questions may fail to cover all knowledge points, confirming that students making their own questions introduces a question domain bias. **These findings support the rationality of our defined question domain sub-bias**.\n- Students prefer to organize the questions and answers using their own language style, believing that it is “fairer and better showcases learning,” and helps to “relieve anxiety and unease,” allowing them to perform at a higher level [2,3]. **This evidence supports the rationality of our defined language style sub-bias**.\n- [2] points out that in human scenarios, it is necessary for teachers to grade accuracy to mitigate the bias from “self-determined answers,” but there can still be ceiling effects if grading is too lenient. They specifically use “accuracy” and “breadth” metrics to counter the risks of “only setting easy questions or giving favorable answers.” [3] also notes that teachers must specifically revise the answers to questions written by students to ensure fairness in grading. **This is fully consistent with our extrapolation of the wrong label sub-bias**: without dedicated mitigation, correlated errors from the same source will translate into higher apparent scores.\n\n**Regarding completeness**, we note that the question set determines what content is assessed, the language style determines the presentation, and together these determine what the tester actually sees. The label set determines the tester’s final score. Therefore, the question set, language style, and label set together constitute a complete benchmark. Based on this, we consider potential sub-biases from these three aspects, making our decomposition of sub-biases comprehensive and complete.\n\n---\n> **`For Section 4.3, considering that the Bias-Neutralizing Ensemble Algorithm is an iterative algorithm, it would be beneficial for the authors to analyze the number of convergence steps and the evolution of self-bias during the iterative process in their experiments.`**\n\n\nThank you for your comprehensive suggestion. In response, for the Language Understanding task, we have supplemented our analysis by tracking four metrics throughout the iterative process: self-bias $\\mathcal{B}$, the evaluation effectiveness $r\\_p$ of the integrated benchmark, the convergence criterion $\\sum\\_{i=1}^T |\\alpha\\_i^{\\text{new}}-\\alpha\\_i|$, and the proportion of cases that achieved convergence. The changes in these metrics across iteration steps are shown in the table below:\n\n| Iteration Step | $\\mathcal{B}$ |Pearson Correlation $r\\_p$| $\\sum\\_{i=1}^T \\|\\alpha\\_i^{\\text{new}}-\\alpha\\_i\\|$|Convergence Rate %|\n|-|:-:|:-:|:-:|:-:|\n|0|0.0076|0.7953|3.36|0|\n|1|0.0042|0.8237|2.43|0|\n|2|0.0029|0.8285|0.22|0|\n|3|0.0021|0.8320|0.11|0|\n|4|0.0017|0.8320|0.07|0|\n|5|0.0015|0.8335|0.05|0.3|\n|18.57|0.0011|0.8340|<1e-6|100|\n\nHere, step=0 corresponds to the scenario where benchmarks generated by all generators are integrated with equal weights. As shown, in the early stages of iteration (the first five steps), both the self-bias $\\mathcal{B}$ and the evaluation effectiveness $r_p$ are significantly optimized as the convergence criterion $\\sum\\_{i=1}^T |\\alpha_i^{\\text{new}}-\\alpha\\_i|$ rapidly decreases. Meanwhile, since our convergence threshold is set at $\\sum\\_{i=1}^T |\\alpha_i^{\\text{new}}-\\alpha\\_i|<1\\text{e}{-6}$, only 0.3% of cases had converged by the end of the fifth step. On average, it takes 18.57 iterations to reach convergence. Although the marginal benefit from further iterations gradually diminishes, given that the time consumed by each iteration is almost negligible, we adopted the current settings.\n\n---\n> **`Missing reference: AutoBencher: Towards Declarative Benchmark Construction. ICLR 2025.`**\n\nThank you for pointing out the additional related work. AutoBencher [4] is also an LLM-as-Benchmark-Generator approach (powered by GPT-4), and we will include this method in the Related Work section of the revised version.\n\n\n**References**\n\n[1] Constat: Performance-based323contamination detection in large language models\n\n[2] “Exams by You”: Having Students Write and Complete Their Own Exams During the COVID-19 Pandemic\n\n[3] Asking the right questions: Using student-written exams as an innovative approach to learning and evaluation\n\n[4] AutoBencher: Towards Declarative Benchmark Construction"}]

Author rebuttal:

Meta-review: The paper studies LLM self-bias in benchmark generation and evaluation. It then proposes Silencer to alleviate this phenomenon. Reviewers agree that the method is effective and the analysis is well-conducted. We would also like to remind the author to incorporate reviewers' comments to the final version.
Decision: Accept (poster)