IDGen: Item Discrimination Induced Prompt Generation for LLM Evaluation
Authors: Fan Lin
Shuyi Xie
Yong Dai
Wenlin Yao
TianJiao Lang
Yu Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM
Data Generalization
Discrimination Indexes
Primary area: natural_language_processing

Abstract: As Large Language Models (LLMs) become more capable of handling increasingly complex tasks, the evaluation set must keep pace with these advancements to ensure it remains sufficiently discriminative. Item Discrimination (ID) theory, which is widely used in educational assessment, measures the ability of individual test items to differentiate between high and low performers. Inspired by this theory, we propose an ID-induced prompt synthesis framework for evaluating LLMs so that the evaluation set continually updates and refines according to model abilities. 
Our data synthesis framework prioritizes both breadth and specificity. It can generate prompts that comprehensively evaluate the capabilities of LLMs while revealing meaningful performance differences between models, allowing for effective discrimination of their relative strengths and weaknesses across various tasks and domains.
To produce high-quality data, we incorporate a self-correct mechanism into our generalization framework and develop two models to predict prompt discrimination and difficulty score to facilitate our data synthesis framework, contributing valuable tools to evaluation data synthesis research. We apply our generated data to evaluate five SOTA models. Our data achieves an average score of 51.92, accompanied by a variance of 10.06. By contrast, previous works (i.e., SELF-INSTRUCT and WizardLM) obtain an average score exceeding 67, with a variance below 3.2.
The results demonstrate that the data generated by our framework is more challenging and discriminative compared to previous works.
We will release a dataset of over 3,000 carefully crafted prompts to facilitate evaluation research of LLMs.

IDGen: Item Diserimination Induced Prompt Generation for LLM Evaluation
Fan Lin 1,2 Shuyi Xie²*, Yong Dai
Wenlin Yao?. Tianjiao Lang², Yu Zhang'
SouthEast University, Nanjing, China, Tencent, Shenzhen, China
Abstract
As Large Language Models (LLMs) grow increasingly adept at managing complex tasks, the evaluation set must keep pace with these advancements to ensure it remains sufficiently discriminative. Item Discrimination (ID) theory, which is widely used in educational assessment, measures the ability of individual test items differentiate between high and low performers. Inspired by this theory, we propose an ID-induced prompt synthesis framework for evaluating LLMs to ensure the evaluation set can continually update and refine according model abilities. Our data synthesis framework prioritizes both breadth and specificity. It can generate prompts that comprehensively evaluate the capabilities of LLMs while revealing meaningful performance differences between models, allowing for effective discrimination of their relative strengths and weaknesses across various tasks and domains. To produce high-quality data, we incorporate self-correct mechanism into our generalization framework, and develop two models to predict prompt discrimination and difficulty score to facilitate our data synthesis framework, contributing valuable tools to evaluation data synthesis research. We apply our generated data to evaluate five SOTA models. Our data achieves average 51.92, accompanied by variance 10.06. By contrast, previous works (i.e., SELF-INSTRUCT and WizardLM) obtain average a variance below 3.2. The results demonstrate that the data generated by our framework is more challenging and discriminative compared to previous works. We will release a dataset of over 3,000 carefully crafted prompts to facilitate evaluation research of L.MMs.
exceeding 67, with
1 Introduction
The rapid advancement of LLMs, such as OpenAl's ChatGPT, Anthropic's Claude [1], and Facebook's 's LLaMA series [2. 3]. has revolutionized the field Natural Language Processing (NLP) in recent years. Model evaluation plays crucial role in the development of LLMss, as it guides the iterative improvements during training, enables the selection of the best model variations, and facilitates their deployment in real-world applications [4, 5]. Recognizing the importance of model evaluation, researchers have made great efforts to create comprehensive benchmarks. Many of these benchmarks consist of multiple-choice questions in English [6. 7], as the results are easily obtainable through string matching. Some researchers [8] have extended these datasets to non-English languages, adapting the content to new linguistic and cultural contexts through translation. These datasets often result from either extensive public data collection or through manual or model-assisted data synthesis processes.
"qqual contribution. Work done during the internship Lin at Tencent.
i "Corresonndigg author.
Codde and data are available at
git
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Despite these advances, existing evaluation frameworks exhibit crucial limitations, particularly in their ability to discriminate between LLMs of varying capabilities. The predominant use of multiple-choice questions restricts the evaluation to specific competencies, potentially overlooking the full generative potential of LLMMs, including their crruuction-ollowwing ability. Merely translating prompts from one language to another language may not adequately demonstrate model's proficiency within specific cultural context. Furthermore, current generation methods lack comprehensive mechanism to ensure the correctness of the generated questions, which is especially important for producing mathematic questions.
More importantly, the evaluation set should evolve adaptively as LLMs' abilities improve to ensure it remain sufficiently discriminative. As LLMs become more capable of handling increasingly complex tasks, the evaluation set must keep pace with these advancements. Static evaluation sets may be ineffective in differentiating between the performance of various L.Mss. To maintain the discriminative power of the evaluation set, it is essential to continually update and refine the questions and tasks according to model abilities. This involves incorporating new challenges that push the boundaries of LLMs' abilities, such as more difficult reasoning, deeper understanding of context, and generating coherent responses to complex instructions. By adaptively updating the evaluation set in the development of LLMs, we can ensure that the benchmarks keep providing valuable insights into the strengths and weaknesses of different models.
To address these challenges, we propose robust framework to produce high-quality. discriminative test data that evolves in alignment with advancements in LLM capabilities. Our framework inspired by Item Discrimination (ID) Theory [9] that is introduced to assess how well individual questions (items) on test distinguish between students who perform well on the overall test and those who do not. We adopt ID Theory to ensure each test question's 's effectiveness in differentiating between higher and lower-ability LLMs. Our framework can generate open-ended questions automatically in both English and Chinese, aimed at capturing wide spectrum of tasks. Central to approach is the application of discriminative techniques that enhance the test sets' ability to distinguish between different levels of language understanding, thereby allowing for more precise evaluation of LLM performance. To achieve this goal, we also introduce two key metrics: question discriminative power and question difficulty, and train corresponding models to measure them. Additionally, establish an iterative verification process to guarantee the logical soundness and precision of our questions. This multi-round iterative process can better enhance the usability of questions with logical coherence.
Our contributions can be summarized as follows:
We propose a framework for data production and generalization that enables the rapid and high-quality creation of test datasets capable of effectively testing and differentiating LLMss
We innovatively adopt discrimination as the guiding principle for data production and generalization, employing rigorous data correction methods throughout the entire data production process to ensure the generated data has high usability and quality.
We release a comprehensive set of over 3,000 questions, created and retined through our rigorous iterative verification process, to support and enrich the community's 's resources for LLM evaluation.
We develop and train two models to measure question discriminative power and difficulty, which we have made available to the open-source community.
2 Method
In this section, demontstrate generalization framework in Figure 1. Assuming We have meticulously handcrafted a batch of high-quality seed data, the first thing is to exploit "instruction gradient" "4 i.e., specially designed rules from the instruction perspective, generalize the questions (in Section 2.1). Subsequently, we employ "response gradient" to generalize questions, where the "gradient" refers to the rules for generalizing questions based on LLMs' responses (in Section 2.2). Next, we discuss a self-correct method to rectify the generalization questions, enhancing the usability
'Inspired by [10]. the generalization of instruction data is similar to forward propagation in gradient descent while generalizing instructions by asking questions about the model's response is similar to backpropagation. Therefore, we name methods as "instruction gradient" and "response gradient" respectively.

---

Response Gradient
Instruction Gradient
L.M Response
General Text Question
Question
Instruction Gradient
CoT Check
Mathematicnl Question Pool
Feedback
Mathematical Question
Instruction Correction
Figure 1: Self-Correct Instruction Generalization Framework with "Instruction Gradient". Firstly, handcraft a batch of seed data, dividing into math category and general text category. Next, we generate batch of dataset through "instruction gradient". For instructions in the general text category, we generate responses using a LLM, then generate new instructions through "response gradient", i.e., propose new questions based on the response. For problems in the math category, we check them through CoT check, and apply self-correct according to the CoT check's 's feedhack.
of these data (in Section 2.3). Finally, we illustrate how we get high-quality answers from LLMs (in Section 2.4). To ensure the discrimitive power of the generation evaluation set, we propose to train an discrimination estimation model and an difficulty estimation model formulate two metries (in Section 2.5 and 2.6).
2.1 Data Generalization Based on "Instruction Gradient"
From the perspective of instruction, aim design constraints guide the generated content, ensuring the generated questions adhere to specified content and also possess diversity and distinc- tiveness. Inspired by previous work, refer this feedback from the instruction perspective as "instruction gradient". We apply Hunyuan for data generalization? A.2. Since different types of data require distinct generalization techniques, we create various methods tailored different categories of data [11]. We systematically develop several strategies that enhance both the difficulty and the discriminative power of the generated questions. In our study, delineate 12 strategies tailored for addressing general text questions, such as "restricting the language used in responses", and formulate 8 distinct strategies for tackling mathematical questions. including "introduce additional variables", A comprehensive enumeration of these methodologies is presented in Appendix Table 5. In the data production process, for general text questions, we select 1-3 suitable generalization strategies. This approach aims to increase the complexity and differentiation of the generated questions, making them richer and more diverse. In contrast, for mathematical questions, randomly select single strategy. This choice helps to minimize the risk of generating unusable questions and ensures consistency in the problem generation process.
2.2 Instruction Generalization Reliant on "Response Gradient"
Generalizing questions from seed data based on the "instruction gradient" restricts the diversity and confines Especiallynt specific topics. To enhance the diversity of general evaluation questions, adopt a two-pronged approach. Firstly, we ensure overall diversity by expanding the variety of seed data. Secondly, we amplify question diversity by leveraging the "response gradient."
UUnles otherwise specified, all data in this document are generated by Hunyuan (Hunyuan-standard) which is a Large Language Model developed by Tencent. Additionally, data generated using other LLMs are also employed, and the relevant experiments and analyses provided in the Appendix

---

For general text questions, we rephrase the question based on the response from the LLM. Specifically. we append brief instruction to the question, which serves to guide the LLM in generating responses with more comprehensive information. After acquiring additional information, we generate new questions based on them. However, to ensure the difficulty and discrimination of the data, we embed reference question in the prompt. We present the instruction that guides more information for the LLM and the prompt rephrasing questions based on response information in Appendix Table 6 and Table 7.
For example, for the question "How can NLP technology be used to detect and prevent the spread of fake news?", using the instruction gradient for generalization, we can obtain new question "List three specific methods detect and prevent the spread of fake news using NLP technology and explain their principles," which still revolves around the original question for expansion or transformation. To address this, we consider discarding the original question and using the --generattedd response as information or knowledge. At this point, we only generate questions based on piece of text, and the questions may become more interesting based on the content of the response. In the above example, we could generate a new question "What NLP tasks are typically addressed by fact-checking and source analysis techniques?"
2.3 Evaluating Question Usability
Assessing the Usability of General Text Questions Inspired by the methodologies outlined in [10], we craft a comprehensive set of evaluation criteria encompassing safety, neutrality, integrity and feasibility. These criteria are important in assessing the suitability of general text questions for our purposes. The detailed descriptions of these evaluative measures are presented in Appendix Table 8. We consider a question to be unusable it fails to meet any of these criteria.
CoT Check for Mathematical Questions For mathematical questions, it is insufficient to estimate whether the generalization question is reasonable or not using a simple instruction for an LLM. As depicted in Figure2, consider the question "There are ten red, yellow, and blue balls in a box. You wish to draw ball at random from the box. What are the chances of drawing red ball?". The generated question includes two conditions, "totaling 30 balls" and "the probability of drawing a yellow ball is 1/4", which leads to result of 7.5 yellow balls. This result contradicts common sense because there should not be "half" yellow ball. Such scenarios are frequently undetectable by simply asking LLMs to determine whether the problem is reasonable.
Inspired by CoT (Chain of Thought) [12], we come up with a CoT-based approach to check whether generated questions are reasonable or not. Specially, we start with the concepts and move on to analyze each element of the problem, ensuring the rationality and precision of mathematical questions by assessing logical connections, solvability, and meticulously examining assumptions and calculation outcomes the present context. The details are depicted in Appendix Table 10. Through our proposed inspection mechanism, we can dramatically eliminate the problems of conceptual errors, logical contradictions, violations of common sense, missing conditions, and unsolvable questions. In the example shown in Figure 2, we use Hunyuan to assess the reasonableness of the question, which successfully identifies the unreasonableness of the problem and corrects based on the assessment process. During data production, to further improve the usability of the questions, we invoke both Hunyuan and Hunyuan-pro to assess the reasonableness of the questions separately. We consider a question to be reasonable only when both models judge it to be reasonable.
2.4 Acquiring reference answers
To ensure the highest quality of responses for general inquiries, we adopt a sophisticated multi-modes strategy. We use five SOTA LLM models: Hunyuan, GPT-4, GProo Wenxin 4, and Qwen to generate preliminary answers independently. This diverse approach leverages the unique strengths of each model and cover a broad spectrum of perspectives. For general text questions, inspired by [13), we design each response from the following perspectives: Safety (0-30 points), Correctness (0-10 points), Relevance (0-10 10 points), Comprehensiveness (0-10 points), Readability (0-20 points), Richness (0-10 points), and Humanization (0- points). Hunyuan scores each response according to
'In this document, GPT-4 refers to 4pt-4-32k-066 GPT-4 Turbo refers to turboo204-- Wenxin 4 refers to ERNIE- Bott--B-444--- 4.0, Qwen refers Qwen-Max, Claude refers claude-3-oput 20242--99ppu--2224222....

---

Ressonableness Judgement
Reasonableness Judgement
50 times 1/4 results 7,5 yellow balls, unreasorable.
There are 7.5 yellow bails, but the number of balls should be integer, which is unressonable here.
The question is reasouable as has well- defined prohability, logical mathemanical relationships sufficient information.
The question unrcasomable.
The question is reasonable.
CoT Check
CoT Check
Seed
Generalization Question:
Cerrected Question:
In box with 10 red,
Logical Steps
a box 30 bails red, yellow, and blue- of drrwing
In box of 30 balls rod, yellow, and blue- with 1/3 chance drowing red and
10 yallow, and 10 blue balls, the prohability of drawing red ball?
chance drawing a yellow, what's the probability of drswing loest vod ball two draws?
chance of drawing yellow, what's probability of drawing at least one red ball in draws!
Figure 2: Chain of Thought Check Illustrated with Mathematical Question Example
these criteria to maintain a high standard of consistency and fairness. The response with the highest score from Hunyuan is then selected the reference answer and is used establish benchmark.
For mathematical questions, our approach is equally robust but tailored to the specificity of the subject. The most accurate response is determined through collective voting mechanism? involving three models: Hunyuan, GPT-4 Turbo, and Qwen. The answer that obtains the majority of votes from these models is then selected as the reference answer. In cases where there is a tie, one of the tied responses is randomly chosen to serve as the reference. To further ensure the precision of our answers, we enlist mathematics experts to review and refine the responses where necessary. This step crucial to validate the accuracy and dependability of the answers we provide.
2.5 Discrimination Estimation Model
To facilitate data synthesis and ensure new data are discriminative enough, we train a model to measure discrimination of each data instance. Each training instance includes prompts and its label diserimination indexes. The prompt includes four features: question, its corresponding category, mean length of this category, and length ratio. These features are significant and provide meaningful reference for understanding the discrimination of the questions. We apply five-point rating system to score each response from different models and obtain the discrimination indexes. The specific scoring criteria can seen in Table 1.
Table 1: Score Evaluation Criteria.
Evaluation Criteria
Evaluation Score
The answer is irrelevant or harmful.
0
The answer is wrong or contains factual errors.
The answer is correct but the process has flaws.
2
The answer is right. The answer exceeds expectations.
3
4
Refer to the discrimination indexes proposed by KeLlKy [15] in education studies, we design calculation formula for discrimination indexes by utilizing the evaluation data derived from several models including GPT-4, ChatGPT, Wenxin 4, and Qwen. Regarding the same question, arrange each model's is average score a descending order. The average score for the top 50% is denoted as PH, while the average score for the bottom 50% is indicated PL.. The computation of the discrimination indexes is articulated by the following formula:
"We hope to select high-quality responses as reference answers as much as possible. Related work| 14] studies the theoretical basis of the collective voting mechanism and discusses the impact of different voting methods on social welfare. Inspired by this, we introduce a "collective voting mechanism" to select reference answers by comparing and voting among multiple responses.
5

---

used as the backbone to N/2 2 PH
Σk=1 k=
score,k
(1)
N
2 * M
Ei= ++1 Ek=1 score,k
PL
(2)
N 2
* *M
PH PL max_scort
discrimination_indees
where N is the number of models, M is the total number of evaluators, score;; is the k-th evaluator's s score for the i-th evaluation model's answer, and max score is the highest score of the evaluation (in our scoring system, the max score is 4). We map the discrimination indexes to four levels: "Low" for values less than or equal 0.1, "Relatively Low" for values greater than 0.1 but less than or equal to 0.15, "Relatively High" for values greater than 0.15 but less than or equal to 0.25, and "High" for values greater than 0.25. The threshold here is estimated based on the distribution of 100,000-Ievel evaluation data.
We construct the training data by sampling from 12 widely adopted models (GPT-4, ChatGPT, Wenxin 4, Claude3, LLaMa2, Baichuan3, GLM-4, etc.). A training sample includes information such as the question, category, reference answer, and the ratio of the question length the average length of its category, etc. The expected label is discrimination level label ranging from 0-3, which implies superior discrimination when the number is high. Then, Baichuan2-133 be supervised and finetuned as discrimination model.
To more accurately obtain the discriminative power of the dataset, calculate the discrimination indexes through manual annotation. Specifically, we first invoke multiple models to respond to the questions. Then we engage relevant experts score the responses of various models according to Table 1. Subsequently, we calculate the discrimination indexes for each sample using Formula 3 and then determine the average value across all samples obtain the discrimination indexes for this batch of data.
2.6 Difficulty Estimation Model
In our research, we utilize the "difficulty level" metric to assess a dataset's ability differentiate various model by categorizing data into varying levels of difficulty. However, assessing difficulty using a general-purpose LLM such as GPT-4 can yield inaccurate estimation. Moreover, manually annotating the difficulty level of each instance is time-consuming and labor-intensive, and there's 's often a discrepancy between the difficulty perceived by humans and the difficulty perceived by models. To address these challenges, we have developed specialized model designed specifically to evaluate the difficulty of each question. We train this model using dataset compiled from the evaluation results of various LLMs, similar to those used in training our discrimination estimation model. The difficulty of each sample is determined based on these models' evaluation scores. This method provides a more standardized and efficient means of measuring difficulty, avoiding the biases and limitations of manual annotation and annotation by general-purpose models.
EE=
Σド scoret,
difficulty_scoe max score
M
Where N is the number of evaluation models, M is the total number of evaluators, and scoreij is the j-th evaluator's 's score for the i-th evaluation model's answer. We map the difficulty scores to three difficulty levels: "easy" for scores less than or equal to 1.5, "medium" for scores greater than 1.5 but less than or equal to 2.5, and "hard" for scores greater than 2.5. The difficulty level is applied to evaluate the quality of generated instructions.
We believe that the difficulty score can serve as a reference for discriminability. In addition, a high difficulty score for a question does not necessarily mean that it is more discrininative. For example, for a question with max score of 3, if the evaluation scores are both 0 and O, according to the formula, its difficulty score is 3, and the discrimination score is 0, meaning that the question is very

---

difficult, and the LLMs cannot answer it correctly, so the question is not discriminative. However, if the evaluation scores are and 3, we can calculate that its difficulty score is 1.5, and the discrimination score is 1, indicating that the question can effectively distinguish the level of LLMss.
We propose difficulty estimation model by fine-tuning BaiChuan2-13B pretrained model. The training sample input is the same with the discrimination estimation model. The output is 1-3, representing the difficulty level, and the training instruction changed estimate the difficulty of the problem. The complexity of generalization questions can be predicted via utilizing our difficulty estimation model. With the predicted complexity, we can sift out evaluating data exhibiting a specified degree of difficulty.
In order to obtain more accurate measure of the difficulty of the dataset, calculate the difficulty scores through manual annotation. After obtaining the annotators' scores for the responses of various models to the questions, calculating the average value of the difficulty scores for all samples in the dataset, we obtain the difficulty score for these samples.
calculate the difficulty score for each sample using Formula 4. By
Experiment
In this section, we first introduce the experimental setup, including the baselines and the seed data. Then we compare our generalization data with some publicly usable datasets and analyze the results. Subsequently, we assess the usability of our data, as well the discrinination indexes and difficulty score, and provide relevant analysis. Finally, we describe the performance of our proposed discrimination and difficulty estimation models.
3.1 Experiment setting
Baselines (1) SELF-INSTRUCT [16]: it generates approximately 82k instances from 175 human- created handwritten instructions.
(2) Instruction Tuning with GPT-4 Dataset [8]: in this task, GPT-4 is used to generate responses the 52k English data from Alpaca dataset. The questions are then translated into Chinese using chatgpt, and responses are generated again using GPT-4.
(3) WizardLM [17]: leverages the ChatGPT API to generate 250k instructions based on the training data from Alpaca Dataset.
Seed Data We establish a dataset comprising 6,000 instances by employing human annotators,
which consists of Chinese and English subsets. The Chinese subset[1 composed of approximately 5,000 instances, while the English subset contains 1,000 instances. The English instances include 175 sourced from the SELF-INSTRUCT dataset [16] and the remainder from the Alpaca dataset [18]. These questions are categorized into general text questions and mathematical questions, which are generalized separately. Furthermore, the seed data typically exhibit a high degree of diversity, while the categories of generalized data generally remain unchanged.
3.2 Comparison to Public Datasets
Discrimination indexes and difficulty score analysis Including the first three baselines that have already been introduced, have also incorporated other datasets:
(1) SELF-INTTRUCT BeceCCC_Seeee_dttee data: 175 seed data used to generate the SELF-INSTRUCT dataset.
(JSELF-INSTRUCT--O) the dataset created by generalizing the 175 seed data points from the SELF-INSTRUCT dataset using our proposed method.
(3) Ours (hard seed data): the data obtained by applying our method to questions that human experts consider to be more challenging.
We sample responses from GLM-4, GPT-4 Turbo, GPT-4, Claude 3, and Qwen. ask 104 domain experts to score the responses from each model according to the criteria outlined in Table I and calculate the discrimination and difficulty. By averaging these values, we obtain the overall discrimination indexes and difficulty scores for each dataset. The results are presented in Table 2.
7

---

Table 2: Comparison of Discrimination Indexes and Difficulty Score on Public Datasets.
Dataset
Discrimination Indexes
Difficulty Score
WizardLM
0.140
1.235
Instruction Tuning with GPT-4
0.098
1.215
SELF-INSTRUCT LeCCC__see__a__e SELF-INSTRUCK SEL.F-INSTRUCT-Or
0.061
1.146
0.109
1.319
0.137
1.541
Ours (hard seed data)
0.204
1.941
From Table among the public datasets for generalization tasks, the WizardLM dataset stands out with a discrimination indexes of 0. 140. It is slightly outpaced by the SELF-INSTRUCT dataset, which has discrimination indexes of 0.109. SELF-INSTRUCT dataset also leads in difficulty score of 1.319. Generalization data with the same 175 seed data, using our method, achieving higher distinctiveness of 0. 137,, close the WizardI.M dataset, and the highest difficulty score of 1.541 among its variants.
Applying our method to more complex seed data yields even better results, with top scores of 0.204 in discrimination indexes and 1.941 in difficulty scores. These findings highlight that our method not only improves discrimination indexes and difficulty scores but also benefits significantly from the use of challenging seed data, emphasizing the seed data's quality as crucial factor for generating superior generalized datasets.
Performance across LLMs We convert the expert assigned to each model into percentage- based scale. We then compute the average scores for each dataset and determine the mean and variance of the scores for each model across the various datasets. The detailed evaluation results presented in Table 3.
Table 3: Evaluation Scores for Various Models on Different Datasets.
Model
GLM-4
GPT-4 Turbo
GPT-4
Claude3
Qwen
Mean
Var.
WizardLM
69.85
72.06
66.91
68.01
68.75
69.12
3.08
Instruction Tunning with GPT-4
69.89
69.25
67.58
71.29
70.14
69.63
1.49
SELF-INSTRUCT Seed data
71.86
72.01
70.06
71.71
71.11
71.35
0.51
SELF-INSTRUCT
67.73
69.48
66.86
63.95
67.15
67.03
3.20
SELF-INSTRUCT-ON
70.51
74.29
68.70
66.87
67.48
69.57
7.12
Ours (hard seed data)
51.75
56.73
47.51
53.75
49.85
51.92
10.06
In Table 3, "Var." refers "Variance". We draw the following conclusions from table mentioned above. Firstly, The datasets of WizardLM, Instruction Tuning with GPT-4, and SELF-INSTRUC" exhibit improvements in both initial seed data. Notably, the SELF-INSTRUC dataset has the lowest mean score and the highest variance, suggesting that can effectively differentiate the performance of various models certain extent. Secondly, the generalization data based on CELFUUT seed_data using our method (SELF-INSTRUCT-
scores and variances across the five models compared their
has lower average score than the seed data, implying that method may increase the difficulty of the questions. In addition, its variance of 7.12 is higher than that of other datasets generalized from the same seed data, reinforcing the notion that our method can enhance the distinctiveness of the data. Lastly, the dataset generated by our method using more challenging seed questions has the lowest average score of 51.92 and the highest variance of 10.06 among all datasets. This highlights the difficulty and distinctive nature of the questions, underscoring the importance of the seed data. Our analysis also reveals that the choice of seed data plays a crucial role in differentiating the performance of various models.

---

3.3 Analysis on the generalization questions
To evaluate the effectiveness of our framework's generalization, we collect 192 general text questions and 385 mathematical questions as seed data, and conduct generalization within our framework. For both the seed data and the generalization data, we generate responses from GPT-4, Wenxin 4, and Qwen. Subsequently, we hire 43 experts to assess the usability of the questions and score the responses according to Table I. Based these scores, we calculate the discrimination indexes and difficulty scores for both seed seed and generalization questions. The results are shown Table
Table 4: Evaluation Scores for Seed Data and Generalization Questions.
General Text Question
Mathematical Question
Data
Usa.
Dis.
Dif.
Usa.
Dis.
Dif.
Sced Data
0.08
0.52
0.09
1.21
Generalization Question
94.0%
0.17
1.08
96.4%
0.20
1.58
The data in Table 4 are all obtained from manual annotation, where "Usa." " stands for "Usability". "Dis." represents "Discrimination Indexes", and "Dif." denotes "Difficulty Score".
From the table, we draw the following conclusions: Firstly, the generalization questions have high usability rate, which proves the effectiveness our method for identifying or correcting the reasonableness of questions. Secondly, by comparing the values of the generalization questions with seed data, our method can enhance the discrimination indexes and difficulty score of the questions to some extent.
3.4 Discrimination and Difficulty Estimation Models Performance Evaluation
Accuracy of Discrimination Estimation Model We utilize 1500 evaluation data to validate the agreement between the discrimination estimation model predictions and human evaluations. The agreement is 0.72.
Comparison of Difficulty Estimation Model with Human Evaluation We select 1,500 human- evaluated questions and let both humans and models predict their difficulty levels respectively. Then, based on the scores from the evaluations, we calculate the difficulty of each question as the gold label according difficulty formula. Surprisingly, the model's predictions get consistency rate of 0.70 with the gold label, while the human predictions have a consistency rate of only 0.52. This result indicates that the model may find problems that humans consider difficult hard-to-understand be simple.
4
Related Work
4.1 Instruction Data Generation
Instruction data generation from LLM aims to minimize the expenses of human-written instruction and enhance the quality of the data. With the growing capabilities of LLMs, they are now also capable of generating and evaluating datasets. Pioneer works include [16]. [8], [18], which generate instruction data with LLM achieve remarkable success. WizardLM[17] introduces Evol-Instruct, which begins with basic set of data and expands it into more comprehensive and complex instructions. The specific approach incorporates both in-depth evolving (applicable to complex instructions) and in breadth evolving (aiming to increase topic coverage and diversity). Ultimately, unqualified data is filtered out using the Evolutionary Elimination rules. Subsequently, the Wizard series of works [19] [20] that utilize Evol-Instruct have emerged, further refining the system to form more comprehensive and robust framework. self-Alignmen((11] proposes an iterative self-training algorithm that utilizes large amount of unlabeled data create high-quality instruction datasets.

---

4.2
Data Quality
LIMA (Less is more for aligmment)(22] is primarily debunking the myth of RLHF by demonstrating that, given a really good dataset, it is possible to train a small supervised model that can perform almost as same GPT-3 or in fact better than Google's ' BARD and in some cases like GPT-4 equivalent. Finding high-quality data without resorting to human curation remains a significant challenge. Utilizing the super L.M to assess the validity of data and evaluate quality is also one of the prevalent methods. The design of =aif-Alligment2111 involves a scoring standard on 5-point scale with the help of LLM to assess the quality of generated instructions and responses, focusing on aspects such relevance, completeness, usefulness, and the accuracy of the responses to the questions. Furthermore, some studies have attempted to directly extract metrics from existing data to reflect the quality of the data, such as Information Fidelity (IFD) [23]. This approach aims to quantify the richness and accuracy of information the dataset, thereby providing an intuitive measure of data quality. However, the calculation of metries like IFD often relies on additional large language models, which to some extent increases the complexity and computational cost of the method. Despite this, these metries offer automated means of data quality assessment that does not depend manual annotation, which is of significant value for rapid evaluation of large-scale datasets.
4.3
LLM Evaluation
Due to the high convenience in both data collection and automatic evaluation, many evaluation benchmarks have emerged. AGIEval [24] collects official, public, and high-standard admission and qualification exam questions to the human-level capabilities of LLMs. C-Eval [25] is a comprehensive Chinese evaluation suite and contains 13,948 multi-choice questions, including middle school, high school, college, and professional. However, they have overlooked the discrimination indexes of the evaluation questions.
Conclusion
In our research, we emphasize the importance of data discrimination and difficulty and introduce a new framework for instruction generalization. Experimental results prove that this framework effectively enhances the discrimination and difficulty of instructions, generating data that more effectively distinguish the capabilities of different models. We release batch of generalization data help the community evaluate models more effectively, thus promoting the enhancement of model capabilities. Additionally. we provide models for identifying discrimination and difficulty to help quickly judge the quality of data.
Limitations The effectiveness of our framework relies on the performance of large models, and we hope see the advent of even more powerful large models in the future. Our method does not directly yield accurate reference answers for mathematical problems that require strong logical reasoning, and the accuracy of these answers requires improvement.
Broader Impact The data generalized by our framework effectively differentiates the performance of current mainstream models, offering research direction for the effective improvement of model capabilities. We also note that the quality of seed data affects the discriminability and difficulty the data after generalization. We look forward to the arrival of high-performance models and high-quality data in the future, creating a complementary trend.
6 Acknowledgement
Our work was supported by Tencent. Shenzhen, China, and Southeast University, Nanjing, China. We thank Zishan Xu, Zhichao Hu, Xiao Xiao, and Yuhong Liu of Tencent for their assistance with our work.
10

---

References
[1] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2Xiv:2204.05822 2022.
[2] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Amin Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
[3] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[4] Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models, ACM Transactions on Intelligent Systems and Technology, 15(3):1-45, 2024.
[5] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang. Zican Dong, al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
[6] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.091100 2022.
[7] Ahmad Ghazal, Tilmann Rabl, Minqing Hu, Francois Raab, Meikel Poess, Alain Crolotte, and Hans-Arno Jacobsen. Bigbench: Towards an industry standard benchmark for big data analytics. In Proceedings of the 2013 ACM SIGMOD international conference on Management of data, pages 120--120, 2013.
[8] Baolin Peng. Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, 2023.
[9] C Boopathiraj and K Chellamani. Analysis of test items difficulty level and discrimina- tion index in the test for research in education. International journal of social science & interdisciplinary research, 2(2): 189-193, 2013.
[10] Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with "gradient descent" and beam search, 2023.
[11] Shuyi Xie, Wenlin Yao, Yong Dai, Shaobo Wang, Donlin Zhou, Lifeng Jin, Xinhua Feng, Pengzhi Wei, Yujie Lin, Zhichao Hu, Dong Yu, Zhengyou Zhang, Jing Nie, and Yuhong Liu. Tencentlmeval: hierarchical evaluation of real-world capabilities for human-aligned Ilms, 2023.
[12] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thouuhh prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 248337 2022.
[13] Yilun Liu, Shimin Tao, Xiaofeng Zhao, Ming Zhu, Wenbing Ma, Junhao Zhu, Chang Su, Yutai Hou, Miao Zhang, Min Zhang, et al. Automatic instruction optimization for open-source Ilm instruction tuning. arXiv preprint arXiv:2311.132 132466, 2023.
[14] Amartya Sen. Collective choice and social welfare: Expanded edition. Penguin UK, 2017.
[15] Truman Lee Kelley. The selection of upper and lower groups for the validation of test items. Journal of Educational Psychology, 30:17-24, 1939.
[16) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self self-generated instruc- tions, 2023.
11

---

[17] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models follow complex instructions, 2023.
[18] Rohan Taori Ishaan Guirajani Tianyi Zhang*, Yann Dubois* Xuechen Li Carlos Guestrin, Percy Liang, and Tatsumori B. Hashimoto. Alpaca: strong, replicable cestructionfollowiiie model. Website, 2023. https:/ crrm..rfmm// www.f.ttw.....oo.....t..t..tf.ft oodf 2atd/2223/033 00///03/133 ///appaca html.
[19] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma. Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprini arXiv:2306.0055 08568, 2023.
[20] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical rea- soning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308. 095583, 2023.
[21] Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer. Omer Levy, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation, 2023.
[22] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment, 2023.
[23] Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting Ilm performance with self- guided data selection for instruction tuning, arXiv preprint arXiv:2308, 220..23200.2222 2023.
[24] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: human-centric benchmark for evaluating foundation models. arXiv preprint urXiv:2304.06366 2023.
[25] Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al. C-eval: A multi-level multi multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322 2023.
12

---

A Appendix Supplemental Material
A.1 Additional Details on the Method
Generalization Methods for Different Categories We believe that for evaluation data, discrimi- nation and difficulty are important measures of data quality. Inspired by traditional gradient ideas, we hope to find suitable "gradient" generalization method in existing instruction generation to improve the discrimination and difficulty of data. Considering that for different types of data, there should be different suitable generalization methods. Therefore, we have designed different generalization methods for different categories. We have carefully designed some generalization schemes that improve the difficulty and discrimination of the problem. The list of schemes is presented in Table 5:
Table Generalization Methods for Different Categories
Category
Generalization Method
1. Increase the requirements for creativity and novelty
2. Replace general concepts with specific ones
3. Raise the level abstraction, abstracting problems from concrete instances
4. Integrate knowledge across domains
5. Restrict the language used in responses
6. Design forbidden specific vocabulary, constrain vocabulary usage fre- quency, require the use of specific vocabulary
General Text Question
7. Limit the number of sentences, word count, special formatting, or the number of paragraphs
8. Impose constraints on punctuation marks, such using or using specific punctuation symbols
9. Limit the number of placeholders, and choose whether to add postscript not
10. Restrict the starting or ending words
11. Require highlighting, JSON formatting, or partial quantities
12. Employ multiple constraint methods from the above list
1. Change variables
2. Provide programming code
3. Introduce dynamic processes
4. Introduce additional variables
Mathematics
5. Limit methods
6. Combine with non- oon-mathemaical domain knowledge
7. Introduce advanced mathematical concepts
8. Combine different mathematical domains
Information Inducer To generate more enriched responses from the LLM for subsequent questions, we incorporate a simple instruction into the questions, which name the "Information Inducer".
Prompt of Generating Questions Based on Response For general text questions, the prompt for generating questions based on responses shown in Table 7. We provide responses from large models and request the design of new questions, thereby generating more diverse set of questions.
13

---

Table 6: Information Inducer for General Text Question
Category
Instruction
Please describe the background and relevant details of this problem in detail. Think deeply about the problem from multiple dimensions. Based on this information, provide a comprehensive and in-depth answer or suggestion, and explain the thought process.
General Text Question
The method_ Jist in the prompt refers to the generalization strategies listed in Table 5, which can serve reference during the question design process.
Table 7: Prompt of Generating Questions Based on Response
You are an experienced educational master with rich expertise. Please combine your expertise to play the role of an "examiner". The candidates are existing Al systems (such as ChatGPT, Qwen, GPT4, etc.). Your task is to design question based on the given information.
Information: [response]
Question requirements:
Please first consider the important criteria within the field of education and use them
reference for designing the question.
The designed question should be logically clear and complete in content.
The designed question should challenging for the candidates.
The designed question should have discrimination as much as possible, so that some candidates can answer correctly, while others cannot.
The question should be novel and not limited to the given information.
Reference question ideas: To increase the discrimination and difficulty of the question, you can refer to the following question ideas: (method_list
Output format requirements:
Line 1: Require point-by-point discussion, give the thinking points that can be mined from the given information, and the "thinking process" for improving the difficulty and discrimi- nation.
Line 2: Based on the above thinking, output the designed question.
Please output your design:
General Text Question Usability Evaluation Criteria For general text questions, we consider the usability of the problems from the perspectives of safety, neutrality, integrity, and feasibility, as shown in Table and further detailed in the instruction scoring prompt presented in Table 9.
Table 8: General Text Question Usability Evaluation Criteria
Dimension
Description
Safety
No explicit, politically sensitive, violent content
Neutrality
No bias racial discrimination in instructions Sufficient information provided to clarify the task
Integrity Feasibility
Instructions within the AI system's 's capability range
14

---

Table 9; Instruction Scoring Prompt
You are an instruction scorer, responsible for scoring the following instruction: (instruction)
Evaluation Criteria:
Safety (1 point):
The instruction does not contain any inappropriate, politically sensitive,
violent, illegal content.
Neutrality (1 point); The instruction is unbiased and does not involve racial discrimina-
tion.
Information completeness (1 point): The instruction provides enough information to clearly execute the task, including all necessary context and specific details, without omitting key information or requiring additional assumptions or inferences.
Feasibility (1 point): The instruction does not exceed the capabilities the Al system.
Additional Notes:
Based on the above criteria, score the instruction. Ensure that the instruction contains all necessary information. For example, if the instruction requires rewriting sentence, analyzing text, or providing an explanation, must provide the original sentence be rewritten, the text to analyzed, or the specific content to be explained. Instructions lacking this information can be penalized in "information completeness" and "feasibility."
Output Format:
Safety:
Neutrality:
Information completeness:
Feasibility:
Total score:
Each point should be 0/1, with total score output of 0-4, without any additional content.
CoT Check Usability for Mathematical Questions For mathematical questions, we design Chain of Thought (CoT) approach to check the usability of the problems. Starting from the concepts, we delve into each component the problem, evaluate the logical relationships and solvability, and carefully examine the assumptions and calculation results in the problem to ensure the reasonableness and accuracy of the mathematical questions. As shown in Table 10.
Case Study for General Text Questions For general text questions, provide additional example to further illustrate the generalization process, as shown in Figure 3.
Analysis of Effectiveness Usability: Human-annotated datasets not necessarily all usable, and they often contain errors. They also need to be repeatedly checked and reviewed to ensure a high level of usability (e.g., above 95%). The usability of the questions in generated data reach 94% (based on human-am -anattte results), and the usability of the evaluation data is satisfactory. In contrast, the usability of Self celf-Instruct(161 is 79%.
Production Efficiency: In this paper, takes 2-5 calls check -ennnrree question, with an average time of about 20 seconds per question. In contrast, manual writing takes about 5 minutes per question, and it is subject to fatigue effects.
Cost: In this paper, generating and checking question with the machine involves the input and output of about 9k tokens, costing approximately S0.03. In contrast, the market price for manually writing a usable question is about $2, making the cost of human buman--nnotalle annotated datasets relatively high.
15

---

Table 10: CoT Check of Usability for Mathematical Questions
Step 1: Analyze each component of the problem in detail, identify and understand the relevant concepts involved the problem, and check whether they are defined in mathematics and used appropriately.
Step
Think deeply about the logical relationships between each component. Evaluate whether the relationships in the problem are mathematically reasonable. If possible, provide supporting mathematical proofs or identify potential contradictions.
Step 3: Fully assess the solvability of the problem. Determine whether the problem can be solved and whether there is sufficient information conditions solve it. If the problem cannot be solved, point out the missing information or conditions and explain why these are necessary.
Step 4:
Carefully check to determine whether there are any counter-intuitie or unreasonable assumptions in the problem steps. Check whether the numbers in the problem and the results of the calculations are consistent with the actual situation, such as whether the relevant results of people/objects integers, whether there any violations of odd and even cognition the problem or process, etc.
1. Integrate other philosophies
Add keyword constraints
Generalization Question 1:
3. Restrict answer length
Investigate Nietzsche's Superman philosophy, linking it to his other concepts. Utilize keywords "Nietzsche", "Superman", and "philosuphy" in 50-100 word explanation.
Instruction Gradient
Select appropriale
strategies
Seed Data:
What is Nietzsche's philosophy of Superman?
Generalization Question
Link Nictzsche's "will to power" to his Superman philosophy, highlighting its importance to humanity. Cite theory and examples.
LLM Response:
Response Gradient
In Nietzsche's philosophy of the Superman
Figure 3: Example of generalization for general text questions. First, commence with the seed data comprising general text questions and choose to techniques from the method library to furnish specific generalization recommendations for the seed data. In this example, the seed data mandated to be generalized by "incorporating other philosophical viewpoints". "Add keyword constraints" and "restricting the answer length". Through these methodologies, the generalization question becomes more challenging. Subsequently, we assess the generalization question for safety, neutrality, integrity and feasibility to ascertain their usablity. We retain the qualified instructions and discard unqualified questions. If the generalization questions are qualified, we can employ L.M to generate responses for them and restructure questions based these responses. In example, the generalization question that emerges from the LLM's response incorporates philosophical concepts like the "will to power" and the "aspiration become the Übermensch". The rephrased question introduces novel perspective, largely contingent on the language model's reply, thereby enriching the diversity of viewpoints in the question set through the applied generalization technique.
16

---

A.2 Supplementary Experiment
Ablation Study of Multiple LLMs for generation data We apply our proposed method some other LLMs, such as GPT-4-turbo 200-4-ttubo-2024-004 batch of small amount of seed data, and manually scoring the models' responses calculate discrimination indexes and map them to the four levels of discrimination indexes. The experimental results are shown in the table below. The results show that there are differences the effects of these models, and using more powerful models may generate higher quality data. This also confirms the limitation mentioned in the conclusion section of our paper: our framework relies on the performance of large models.
and Qwen (Qwen-max) using the same
Model
Amount
Low
Relatively Low
Relatively High
High
Seed data
50
45
0
Hunyuan
50
29
8
8
5
Qwen
50
28
13
6
Gpt4-turbo
50
21
5
10
14
Table 11: Comparison of different models based on performance metries.
Ablation Study of Multi Models for CoT check In the proposed framework, the idea of 'one problem, multiple evaluations' is operationalized by aggregating outcomes from several models. Specifically, we utilize both Hunyuan-standard and Hunyuan-pro to adjudicate the reasonableness of generalization questions. These models apply our Chain of Thought (CoT) method to systematically assess the validity of each question. If either model identifies question lacking in reasonableness, that model will initiate a corrective iteration based on its CoT reasoning process. In the event that both models concur on the unreasonableness of a question, the correction process will be guided by the CoT reasoning mechanism employed by Hunyuan-standard. The question will then undergo subsequent evaluation of its reasonableness. This iterative process is capped at two cycles. Questions that continue to be classified as unreasonable after two iterations are subsequently removed from the question pool.
To further investigate the mathematical question usability recognition using single and multiple models, conduct ablation study the generalization data that has an expert-judged usability rate of 64.8%. In this study. we separately count the usability of data after filtering by Hunyuan- standard and Hunyuan-pro. as well as the usability of data under their combined filtering. The results are presented in Table 12.
Table 12: Usability of Data after Filtering by Different Models
Judgment Model
Generalization Data Usability
Correction Data Usbility
munyuan-ssanddan
87.0%
93.3%
Hunyuan-pro Hunyuan-standard Hunyuan-pro
84.6%
90.3%
90.0%
96.4%
In Table 12, "Generalization Data Usability" refers to the usability rate of the data generated by applying our generalization method set of seed data, accompanied by the exclusion of any questions considered unreasonable by our proposed Chain of Thought (CoT)) method. The "Correction Data Usability" section details the process where the model attempts to correct questions identified as unreasonable by the proposed CoT in generalization data, while leaving the reasonable questions unchanged. The resulting usability rate of the data is then gained.
As indicated in Table 12, employing a single model, either Hunyuan-standann or Hunyuan- -aro utiliz- ing the proposed Chain of Thought (CoT) approach to assess question usability yields commendable results. After the rectification of questions and subsequent removal of data still considered unsuitable, the usability rate reaches threshold of 90%. Furthermore, when both models are deployed in tandem to evaluate question usability and filter out inadmissible questions, there is an observable enhancement in the usability rate in both scenarios.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims presented in the abstract and introduction accurately reflect the paper's contributions and scope, providing clear and concise overview of the novel findings, innovations, and the research topic covered the paper.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In the Conclusion section, we discuss the limitations of our work, addressing considerations of robustness with respect to potential assumption violations and providing insights into the computational efficiency of our approach scales with dataset size.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
18

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not include theoretical results
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in the appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We propose new framework, provide detailed and specific descriptions of its components, and make the content open-source, enabling other researchers to easily reproduce our experimental results. Furthermore, we open-source of data to promote related research and development. These materials are currently under review, and we will make them publicly available afterward.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
19

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The data and code currently under review. We will make them publicly available at a later time.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc
Seriides/CodeSubmiin
lyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips.ccc..cccc wtt.cc/ccccccccdudcccooooo gwtp///dde///ee// wwwrdddSubmsssiooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experimental section provides a detailed description of the experimental
setup and specifics.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
Due to our limited computational and human resources, we don't report enror
bars.
Guidelines:
20

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of computing workers, memory, time of cxecution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the experiment section, provide detailed description of the human resources required for the experiments and the situation of models' API interface calls.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttpe:///eeupseeeeeeeeeee wwwpeeee.../puuuuieeeeeeeeeeeeee httpppicc/thicauiieeeeee.
Answer: [Yes]
Justification:
The paper ensures the preservation of anonymity.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
21

---

Justification: We discuss the potential positive and negative societal impacts of our work in the conclusion section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: the methods section, we consider data safety by implementing filtering processes and manually inspecting the publicly released data to ensure its security and prevent potential misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make the best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the relevant works and provide the appropriate licenses for the released assets properly, ensuring compliance with their terms of use.
Guidelines:
The answer NA means that the paper does not existing assets.
22

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license ge.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pace re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide documentation for the assets released alongside the assets them- selves. The content of documentation is still under review and will be made public shortly.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots. if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We employ annotators for our research and provide them with lawful compen- sation.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
23

---

The paper involves crowdsourcing experiments and provides a clear description potential risks incurred by study participants, as well the disclosure of these risks to the subjects.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
24


Reviews:
[{"review_id": "UZwVXcmL62", "reviewer": "Reviewer_UD21", "summary": "The paper introduces a novel framework for evaluating Large Language Models LLMs) based on Item Discrimination ID theory, which generates adaptive, high- quality prompts to effectively differentiate model performance. Key contributions include a dynamic evaluation set that evolves with LLM advancements, a self- correct mechanism for prompt precision, and models to estimate prompt discrimination and difficulty. The authors validate their framework by testing it on five state-of-the-art models and release a dataset of over 3,000 prompts to aid further research, demonstrating enhanced challenge and discrimination over previous methods.", "questions": "The concerns are included in the weaknesses.", "limitations": "The authors have identified some limitations; however, there are additional ones that I have raised in the weaknesses.", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes a novel prompt generation method to produce more challenging evaluation data.\nThe paper is well-structured and clearly written. The methodology and evaluation criteria are explained clearly, making the paper accessible to a broad audience.", "weaknesses": "The paper only used one LLM Hunyuan) to generalize data and did not verify whether the proposed method can generalize to other LLMs.\nIt is debatable whether using test data generated by an LLM to evaluate the performance of LLMs has practical value. The paper lacks validation of the effectiveness of the machine-generated test set, such as comparing its metrics with those of other human-annotated datasets.\nThe paper lacks an analysis of the diversity of the data used to produce the test set.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q:** *The paper only used one LLM Hunyuan) to generalize data and did not verify whether the proposed method can generalize to other LLMs.*\n\n**A:** Thank you for your question about our paper. Our proposed method is designed for existing LLMs and is **not limited to a particular model**. The work of using LLMs to automatically generate data often involves selecting only one LLM for data generation, such as the Wizardlm[1] work using gpt-3.5-turbo to generate instruction data, and the Self-instruct[2] work using gpt-3 to generate instruction data. \n\nWe apply our proposed method to some other LLMs, such as GPT-4-turbo (gpt-4-turbo-2024-04-09) and Qwen (Qwen-max), using the same batch of a small amount of seed data, and **manually scoring the models' responses** to calculate discrimination indexes and **map** them to the four levels of discrimination indexes. The experimental results are shown in the table below. The results show that **there are differences in the effects of these models**, and using more powerful models may generate higher quality data. This also **confirms the limitation mentioned in the conclusion section** of our paper: our framework relies on the performance of large models.\n\n| Model       | Amount | Low | Relatively Low | Relatively High | High |\n|:-----------:|:------:|:---:|:-------------:|:--------------:|:----:|\n| Seed_data   |   50   |  45 |       0       |       4        |  1   |\n| Hunyuan     |   50   |  29 |       8       |       8        |  5   |\n| Qwen        |   50   |  28 |      13       |       6        |  3   |\n| Gpt4-turbo  |   50   |  21 |       5       |      10        | 14   |\n\n**Q:** *It is debatable whether using test data generated by an LLM to evaluate the performance of LLMs has practical value.*\n\n**A:** The issue you metioned is a very in-depth one. On the one hand, if the **data produced by the model** is not **controlled and filtered**, it can have a significant negative impact on the model [1], so a good filtering mechanism is crucial for the **effectiveness** of the model's data production. On the other hand, if **manually annotated data** does not have a **good filtering mechanism**, it can also have a significant impact on model training, as shown in research work like LIMA[2]. However, a single evaluation often requires **tens of thousands of evaluation data** to fully measure the capabilities of large models. The cost of manually writing questions is too high and the speed is relatively slow. It is necessary to use LLM-produced test data in conjunction with manually written questions to evaluate the capabilities of large models.\n\nTherefore, this paper proposes an automated approach for constructing high-quality evaluation data, with contributions including the following two aspects: (1) Globally, it explores how to **ensure the diversity of data**, such as the classification of seed data and the diversity of data generation methods; (2) For each data, a very effective **data usability check mechanism** is designed. This process is reusable and will also be fully open-sourced.  The method we proposed has also been validated in an **actual production environment**, helping to stably and comprehensively improve the performance of the production model, thereby validating its effectiveness in both experimental testing and production environment testing.\n\n[1] Shumailov, Ilia, et al. \"AI models collapse when trained on recursively generated data.\" Nature 631.8022 (2024): 755-759.\n\n[2] Zhou, Chunting, et al. \"Lima: Less is more for alignment.\" Advances in Neural Information Processing Systems 36 (2024).\n\n**Q:** *The paper lacks validation of the effectiveness of the machine-generated test set, such as comparing its metrics with those of other human-annotated datasets.*\n\n**A: ** Thank you for your suggestion on our paper. We will explain your suggestion from the perspectives of usability, production efficiency, and cost to supplement the missing part.\n\n**Usability:** Human-annotated datasets are not necessarily all usable, and they often contain errors. They also need to be repeatedly checking and reviewing to ensure a high level of usability (e.g., above 95%). The usability of the questions in our generated data can reach **94%** (the 94% usability is based on hunman-annotated results), and the usability of the evaluation data is satisfactory.\n\n**Production efficiency:** In this paper, it takes 2-5 calls to check a machine-generated question, with an average time of about **20 seconds per question**. In contrast, manual writing takes about 5 minutes per question, and it is subject to fatigue effects.\n\n**Cost:** In this paper, generating a question and checking it with the machine involves input and output of about 9k tokens, costing approximately **0.03$**. In contract, the market price for manually writing a usable question is about 2$, making the cost of human-annotated datasets relatively high.\n\n**Q:** *The paper lacks an analysis of the diversity of the data used to produce the test set.*\n\n\"A:\" Thanks for your suggestion on our paper. We appreciate your suggestion and provide a response to this issue, supplementing the explanation of the diversity of the data.\n\nWe ensure **the diversity of the seed data** through a rich variety of categories. Our seed data consists of two parts: Chinese and English. The Chinese data refers to the work of TencentLLMEval, which includes 6 primary categories and 61 secondary categories. The English data uses the seed data from Self-instruct, which contains 175 different task types.\n\nIn terms of methods, we design **diversified generalization strategies** including \"Instruction Gradient\" and \"Response Gradient\" to promote the generation of diversified questions. \n\nIn the actual production process, we **filter out similar questions**. In this process, we cite the work of Self-instruct and remove data samples with ROUGE-l greater than 0.7 to filter out similar sample data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kLpxn5sGzh", "reviewer": "Reviewer_K8YF", "summary": "The paper proposes a prompt synthesis framework for evaluating LLMs to accurately reflect different Large Language Model abilities. The authors develop two models to measure LLMs’ question discriminative power and difficulty. This study presents “instruction gradient” and “response gradient” methods to exploit rule sets to generalize questions.", "questions": "1. The proposed methods - “Instruction gradient” and “response gradient” are not properly described in the manuscript. Authors should write the working procedure of these methods in detail in the main manuscript, as these are the centerpiece of the whole question generation process.\n\n    2. “Generalizing questions from seed data based on the \"instruction gradient\" restricts the diversity and confines the content to specific topics” - is unclear. Consider explaining.\n\n    3. In section 2.3 - Assessing the Usability of General Text Questions: How is the assessment done? Is it done manually with human input? Or by an autonomic process/model?\n    4. In section 2.3 - CoT Check for Mathematical Questions: “we use Hunyuan to assess the reasonableness of the question, which successfully identifies the unreasonableness of the problem and corrects it based on the assessment process.” - How can it be ensured that the model successfully identifies the unreasonableness? Provide a theoretical/experimental study.\n    5. In section 2.4 - Acquiring reference answers: lines 133-136, are the answers scored by human participants?\n    6. In section 2.4 - Acquiring reference answers: line 140, What is meant by a “collective voting mechanism”? Please explain clearly.\n    7. In section 2.5 - lines 148-149, what are “label discrimination indexes”?\n        a. In line 149, “the prompt includes four features” - How did you select these features? Provide some analysis.\n        b. In lines 162-164, How did you select the threshold values? (e.g., “Low” means less than or equal to 0.1, “High” means values greater than 0.25, etc.).  \n        c. In line 168, “discrimination level label ranging from 0-3” - Is this range acquired by observations? Or have you performed some analyses on the score expressions?\n    8. In equation 4, what does the “score” mean? Is it the evaluation score that is depicted in Table 1?\n        a. If you are using the same “score” to calculate the difficulty score and the discrimination indexes, does that mean a question is more difficult if a question is more discriminative?", "limitations": "While this proposed method is understood to work on general text questions fairly well, mathematical questions are the weakest part of this study.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The paper focuses on the generation of a large number of queries and corresponding answers on general language and mathematical topics. They have released a set of over 3000 questions for LLM evaluation. Their proposed metrics (discrimination index and difficulty score) show significant improvement in the quality of the benchmark datasets.", "weaknesses": "Although the paper tries to solve a crucial research area in the scope of LLM evaluation, the study lacks in many different ways. The textual flow is difficult to follow. Many of the concepts introduced were not properly described or not cited with previous work’s references. These issues restricted the reviewability of this study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q:** *The proposed methods - “Instruction gradient” and “response gradient” are not properly described in the manuscript. Authors should write the working procedure of these methods in detail in the main manuscript, as these are the centerpiece of the whole question generation process.*\n\n**A:** Thanks for your question on our paper. We explain \"Instruction gradient\" and \"response gradient\" **in footnote 1 on the second page** of the manuscript. \"Instruction gradient\" and \"response gradient\" are the names we give to our methods. The process of **generating generalized questions from seed data is analogous to forward propagation**, where the LLM generates responses to the questions, and this process should be further pushed forward. Based on these responses (considered as information or knowledge), new questions are generated again, and **this process is pushed backward**, which can be compared to **backpropagation**. Therefore, we thought of **using the term \"gradient\" for naming**, and we named the process of generating questions based on seed data as \"instruction gradient\" and the process of generating generalized questions based on LLM responses as \"response gradient\".\n\nWe **further detail the description of the working procedure** in the annotation of Figure 1 in the manuscript. Our working procedure is as follows: First, we collect a batch of seed data and divide it into mathematical and general text categories. Next, we apply the \"instruction gradient\" to both types of questions. For the **\"instruction gradient,\"** the specific generalization strategies for the two types of questions are different due to the various question types. We **provide the core generalization strategies** in  Table 5 of the appendix. We have the LLM(Specially referring to Hunyuan-standard in our paper) rewrite the seed data according to the generalization strategies, thus obtaining new questions. For **general text questions**, we can implement the \"response gradient,\" i.e., first obtain the LLM's response to the question, and then **ask questions based on the content of the response**. We show the prompt for this process in Table 7 of the appendix. For **mathematical questions**, after generating questions based on the \"instruction gradient,\" we focus more on **the usability of the questions**. Therefore, we design **CoT** to check, using multiple models (referring to Hunyuan and Hunyuan-pro in our paper) to judge the usability of the questions, and modify or discard the questions based on the inspection results. We show the specific CoT content in Table 9 of the appendix.*\n\n**Q:** *“Generalizing questions from seed data based on the \"instruction gradient\" restricts the diversity and confines the content to specific topics”- is unclear. Consider explaining.*\n\n**A:** Thank you for your suggestion on our paper. We provide further explanation for this part. The content generated by generalizing seed data through the \"instruction gradient\" is relatively **close to the topic of the seed data.** \n\nTo make the generalized evaluation data more diverse, on the one hand, we can ensure the overall diversity of the evaluation data through **the diversity of seed data**. On the other hand, we enhance the diversity of questions through the **\"response gradient.\"** For example, for the question \"How can NLP technology be used to detect and prevent the spread of fake news?\", using the **instruction gradient** for generalization, we can obtain a new question \"List three specific methods to detect and prevent the spread of fake news using NLP technology and explain their principles,\" which still revolves around the original question for expansion or transformation. To address this, we consider **discarding the original question** and **using the LLM-generated response as information or knowledge.** At this point, we only generate questions based on a piece of text, and the questions **may become more interesting** based on the content of the response. In the above example, we may generate a new question \"What NLP tasks are typically addressed by fact-checking and source analysis techniques?\"\n\n**Q:** *In section 2.3 - Assessing the Usability of General Text Questions: How is the assessment done? Is it done manually with human input? Or by an autonomic process/model?*\n\n**A:** This part explains **the usability check for general text data**, which is **implemented automatically by the LLM**. We propose **four criteria** that we believe are important for general text questions: safety, neutrality, integrity, and feasibility. Here, **safety** refers to the absence of explicit, politically sensitive, or violent content in the question; **neutrality** refers to no bias or racial discrimination in the instructions; **integrity** refers to sufficient information provided to clarify the task; and **feasibility** refers to instructions within the AI system's capability range.\n\nWe use the LLM (specifically, Hunyuan) to score general text questions based on these four criteria. Questions that do not receive a perfect score are considered **unusable**. For general text questions, the incidence of unusability **occurs less frequently**, so we **discard questions deemed unusable** without modifying them.\n\nIn the experiment, we **manually annotate** the generated general text questions and found that the usability reached 94.0%.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ecG0hpo8bm", "reviewer": "Reviewer_VhvU", "summary": "This paper proposes a method of generating prompts for evaluating large language models such that the prompts are dynamic and allow for showing meaningful performance gaps between different language models.The authors show that the generated data is more-challenging and discriminative than prior datasets.", "questions": "- If a particular language model is used to generate data using the proposed method, is there any bias where that model will perform better at solving those problems? For example, if Claude generates the prompt set, will the prompt set be easier for Claude than GPT?\n- Is the data generation done for a set of language models or for each individual language model? In other words, are the prompts being dynamically changed with respect to a single language model's response or all language model responses? Specifically, Section 2.2 says that the method \"rephrases the question based on the response from the LLM\" - which LLM is this statement referring to?\n- Are there any experiments to verify the robustness of each individual step in the pipeline? It seems like the current experiments are meant to verify the final output of the pipeline, not the in-between steps.", "limitations": "The authors mention the most-important limitations of this work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Work is very timely and addresses a major issue in how we can better evaluate LLMs which are continuously improving and saturating existing benchmarks.\n- Good to see that the generated prompts are indeed harder than baseline datasets - this should indicate that the prompts are challenging enough to provide decent signal on a language model's capabilities.\n- Experimented with many SOTA models and compared with several baseline datasets.", "weaknesses": "The main weakness of this work is that much of the pipeline relies prompting language models to modify seed data. This means that the performance of the language model plays a huge role in the quality of the resulting data. Given that the pipeline seems to have many different steps, each of these steps can introduce errors since LLMs are not fully reliable. It then becomes crucial to have a way of verifying that the generated questions are of high quality. There's also a concern that the ground truth answers might not be entirely accurate. The authors mention both of these issues as limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q:** *If a particular language model is used to generate data using the proposed method, is there any bias where that model will perform better at solving those problems? For example, if Claude generates the prompt set, will the prompt set be easier for Claude than GPT?*\n\n**A:** Thank you for your question, and your question is very professional. In our experiments, we find that for the data generated by Hunyuan, its performance may not be as good as potentially better-performing models, such as GPT4 and Claude3. However, in the data generated by GPT4, there are cases where Hunyuan performs better than GPT4. We cannot entirely confirm that the bias you mentioned does not exist, but even if it does, its impact would **not be decisive**. To avoid potential biases, we have also done related work and processing.\n\nIn our paper, we try to **minimize this potential bias as much as possible**: on the one hand, we **pay great attention to the usability of the generated data**. During the data production process, we use models to participate in the usability check of the generated data. For the final generated data, we also hire **expert personnel to conduct checks**. The expert personnel's judgment of the usability of the questions is based on the human preference perspective to determine whether the questions are reasonable. The inspection process includes **result sampling or re-inspection schemes** to ensure the accuracy of the judgment. The inspection results show that the evaluation data obtained by the method in this paper have ideal usability. Therefore, we can believe that these data can be used to evaluate the performance of LLMs. On the other hand, to avoid suspicion, we **don't  allow the model that generate the data to participate in the evaluation**. In the experiments, the calculation of evaluation results-related data **don't include the evaluation results of Hunyuan**, to avoid potential biases that may exist in Hunyuan-generated data as much as possible. The evaluation results also confirm that **our data can effectively distinguish the performance of existing LLMs**.\n\n\n**Q:** *Is the data generation done for a set of language models or for each individual language model? \nIn other words, are the prompts being dynamically changed with respect to a single language model's response or all language model responses? \nSpecifically, Section 2.2 says that the method \"rephrases the question based on the response from the LLM\" - which LLM is this statement referring to?*\n\n**A:** For data production, we only use a **individual** language model, i.e., Hunyuan-standard. For the **automated usability check of mathematical questions**, to improve the usability of the data, we use **Hunyuan-standard** and **Hunyuan-pro** to perform checks separately and cross-validate the usability of the data. For other processes, we only use Hunyuan-standard.\n\nWe **input the large model's response into the prompt** for generating questions, and the large model responsible for generating questions will produce new questions based on the prompt. Therefore, **the prompt does not dynamically change** with the model's response; it simply incorporates the model's response. While we use Hunyuan, **other large models can also be utilized** for this purpose.\n\nIn Section 2.2, \"rephrases the question based on the response from the LLM,\" **the LLM refer to in this sentence is Hunyuan (Hunyuan-standard)**. In the footnote 2 on page 3 of the article, we mention, \"Unless otherwise specified, all data in this document are generated by Hunyuan (Hunyuan-standard), which is a Large Language Model developed by Tencent.\" Therefore, we use LLM as a reference here.\n\n**Q:** *Are there any experiments to verify the robustness of each individual step in the pipeline? It seems like the current experiments are meant to verify the final output of the pipeline, not the in-between steps.*\n\n**A:** Your question is indeed very insightful. We have not verified the robustness of each independent step; if necessary, we can **incorporate relevant experiments in future versions** of our work. Theoretically, each individual step in our process is robust.\n\nFrom a global perspective, the data we utilize is both reliable and stable. Our dataset comprises Chinese and English components. For the Chinese data, we reference the work of TencentLLMEval[1], which has been **stably applied in various business scenarios**. For the English data, we employ the seed data from Self-instruct[2], which is **extensively used in academia**. Additionally, during the data production process, we set the **temperature of the LLMs** to 0 and use **fixed prompts** to guide the LLM in data production.\n\nFor each data point, we implement several measures when calling the LLM API. These measures include **multiple requests** using a counter in case of call failures, capturing and handling exceptions, and **pausing** for three seconds before resubmitting a request if a call fails. These steps aim to increase the probability of successful API calls. Furthermore, we conduct **validity checks** on the input data for each step to enhance the robustness of the data production process.\n\n[1] Xie, Shuyi, et al. \"TencentLLMEval: a hierarchical evaluation of Real-World capabilities for human-aligned LLMs.\" arXiv preprint arXiv:2311.05374 (2023)\n\n[2] Wang, Yizhong, et al. \"Self-instruct: Aligning language models with self-generated instructions.\" arXiv preprint arXiv:2212.10560 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The main concerns raised by the reviewers are
- a lack of practical applicability
- textual flow of the paper and lack of proper explanation of terms and missing references
- lack of comparison to human annotations
The rebuttal from the authors was quite extensive and they assure that the method has actually been tested in a production environment. They also promise to incorporate all the suggestions regarding writing and including references. I find that also their response as to why they did not compare with human annotations is convincing as this would be much more expensive than the experiments performed for the paper. Therefore, while I believe that the paper can be improved, I do not think there are major issues that stand in the way of acceptance.
Decision: Accept (poster)