StructEval: Deepen and Broaden Large Language Model Assessment via Structured Evaluation (2024)

Abstract: Evaluation is the baton for the development of large language models (LLMs). Current evaluations typically employ a single-item assessment paradigm for each atomic test objective, which struggles to discern whether a model genuinely possesses the required capabilities or merely memorizes/guesses the answers to specific questions. To this end, this paper proposes a novel evaluation framework referred to as StructEval. Starting from an atomic test objective, StructEval deepens and broadens the evaluation by conducting a structured assessment across multiple cognitive levels and critical concepts, and therefore offers a comprehensive, robust and consistent evaluation for LLMs. Experiments on three widely-used benchmarks demonstrate that StructEval serves as a reliable tool for resisting the risk of data contamination and reducing the interference of potential biases, thereby providing more reliable and consistent conclusions regarding model capabilities. Our framework also sheds light on the design of future principled and trustworthy LLM evaluation protocols1.

heading: 1 Introduction
text: Evaluation is fundamental for the development of large language models (LLMs) (Ouyang et al., 2022; Touvron et al., 2023b; OpenAI, 2023), providing essential measurements, feedback, and insights that facilitate enhancements in helpfulness, reliability and security (Chang et al., 2023). Consequently, a variety of large-scale benchmarks are proposed to assess LLMs’ capabilities, such as language understanding (Hendrycks et al., 2021; Huang et al., 2023a), instruction following (Li et al., 2023; Zheng et al., 2023a), reasoning capabilities (Cobbe et al., 2021; Srivastava et al., 2022a).
Unfortunately, current evaluations for LLMs typically employ a single-item assessment
1We openly release our source code and benchmark at https://github.com/c-box/StructEval
Which one is not a variety of apples? A. Granny Smith B. Fuji C. Clementine D. Lobo
Single instance
Isolated evaluate Apple Cultivars
Atomic test objective
(a) Previous evaluation uses single-item assessment for each atomic test objective.
Knowledge graph for critical conceptsBloom-based multi-level instances
paradigm(Milton et al., 2011), which still suffer from their weakness on validity, robustness and comprehensiveness. As demonstrated in Figure 1a, to evaluate the factual knowledge in LLMs, they segment the factual knowledge into a set of atomic test objectives (e.g., apple cultivars, function of insulin), and evaluate each with a single instance (e.g., which one is not a variety of apple). However, such a single-item assessment paradigm struggles to discern whether a model genuinely possesses the required capability or merely memorizes/guesses the answers to specific questions. On the one hand, the single-item assessment relies on the correctness of isolated instances, which is sensitive to confounders correlated to specific instances (Poerner et al., 2020; Zhu et al., 2023b), and susceptible to biases or shortcuts (Cao et al., 2022; Xie et al., 2023; Wang et al., 2023a), making it difficult to discern whether a model’s correct response is due to genuine understanding or mere memorization (Cao et al., 2021, 2024). On the other hand, the rapid expansion of LLMs’
5300
training data and memorization capacity have heightened the risk of data contamination in static benchmarks(Carlini et al., 2022; Jiang et al., 2024), potentially leading to inflated evaluations of model capabilities (Magar and Schwartz, 2022; Oren et al., 2023; Shi et al., 2023). That is, the true capabilities of the models might be overestimated owing to the potential contamination of the training dataset by test instances. Moreover, due to the huge resources required for benchmark construction, currently most benchmarks assess models in a static manner. Consequently, they may quickly reach saturation due to the inability to update in timeliness, complexity and diversity.
To address the aforementioned challenges, previous research has primarily attempted to manually construct newer, harder, and more diverse benchmarks. For instance, Kasai et al. (2022); Yu et al. (2023) devised evaluation benchmarks drawing from recent news or articles; Wang et al. (2021, 2023b) added perturbations into the original datasets to assess model robustness; Hendrycks et al. (2021); Huang et al. (2023a) collected test instances from human professional examination to increase difficulty and diversity. Despite the substantial resource invested, the single-item assessment paradigm of previous benchmarks still struggles with determining whether the evaluated performance can faithfully and fairly reflect the capabilities of models.
In this paper, we propose a novel structured evaluation framework named StructEval, which can comprehensively, robustly and validly evaluate LLMs. This is achieved by employing a structured assessment guided by pedagogy theories to evaluate model ability for each test objective across multiple cognitive levels and critical concepts, rather than relying on the correctness of a single test instance. Specifically, as illustrated in Figure 1b, StructEval consists of two modules which deepen and broaden current evaluation respectively. Given a seed instance, the first module identifies its underlying test objective, and then generates multiple test instances around this test objective which are aligned with the six cognitive levels outlined in Bloom’s Taxonomy (Krathwohl, 2002). Meanwhile, the second module extracts the key concepts that must be understood to answer the seed question (Trochim, 1989), and then develop a series of instances revolving around these concepts based on knowledge graph. Unlike single-item assessment, for each test objective, StructEval requires
LLMs to demonstrate knowledge across multiple cognitive levels and a thorough comprehension of critical concepts for good performance. In this way, for each test objective, the assessment conclusion is no longer determined by the correctness of a single instance. As a result, it does not depend on confounders introduced by specific instances, such as prompt selection, surface form shortcut, data distribution, etc. Therefore, StructEval can reduce the impact of biases brought by these confounders, providing more consistent and accurate assessment conclusions for various LLMs. Meanwhile, a model with data contamination can merely memorize specific answers but still lacks corresponding structured knowledge, therefore, StructEval can robustly provide stable assessment results even when the training data is contaminated. Moreover, due to StructEval’s capability to automatically generate large-scale and high-quality instances, thereby realizing dynamic evaluation through updating of knowledge sources, it can also prevent benchmarks from rapidly reaching saturation.
To demonstrate the effectiveness of our framework, we implement StructEval based on 3 widely used benchmarks. The experiments on a variety of LLMs demonstrate that StructEval: 1) enables the automating generation of large-scale benchmarks and completion of structured evaluations, while ensures instance correctness, relevance, and helpfulness. 2) effectively resists the risk of data contamination, providing robust evaluation results even under data contamination settings. 3) significantly enhances the consistency of model rankings across different experiments, offering more precise and stable conclusions from evaluations. 4) substantially outperforms previous augmentation-based strategies such as word perturbation, paraphrasing, back translation, option shuffle, etc.
The main contributions of this paper include:
• We propose a novel evaluation framework named StructEval, which can comprehensively evaluate LLMs’ capability by assessing each test objective across multiple cognitive levels and critical concepts in principle, rather than previous single-item assessment.
• We implement StructEval on widely-used benchmarks, and human evaluation results demonstrate that StructEval can automatically construct large-scale benchmark with high quality.
• We conduct comprehensive experiments regarding data contamination and rank consistency, demonstrating the effectiveness, robustness and consistency of StructEval for LLM evaluation.

heading: 2 Preliminaries
text: Evaluation is the cornerstone for the progress of LLMs (Chang et al., 2023). Unfortunately, there still exist several grand challenges for achieving comprehensive and trustworthy evaluation for LLMs. For instance, the inability to scale in complexity and diversity at the same pace as the rapid advancements in model capabilities(Srivastava et al., 2022b; Huang et al., 2023b); the biases or shortcuts that lead to unfaithful assessments (Liang et al., 2022; Xie et al., 2023); and the lack of reliable metrics for providing trustworthy results (Zheng et al., 2023b; Wang et al., 2023c). To this end, previous studies have mainly devoted to improving the diversity, scale, difficulty and timeliness of test instances (Kasai et al., 2022; Zhu et al., 2023a), exploring the robustness and trustworthiness vulnerabilities in current evaluations (Zhu et al., 2023b; Wang et al., 2023a), and proposing metrics or protocols more suitable for generative LMs (Lin and Chen, 2023; Zhang et al., 2023). In comparison, this paper aims to propose a structured evaluation framework for LLM evaluation.
StructEval framework is guided by two pedagogy theories which are widely used for educational assessment. Bloom’s Taxonomy Theory is a hierarchical model2 used for classification of educa-
2We adopt the cognitive domain list since it is frequently
tional learning objectives into six levels, including remember, understand, apply, analysis, evaluate and create (Krathwohl, 2002). Therefore, to comprehensively evaluate the model’s ability across various cognitive levels on test objective, StructEval would generate multiple test instances covering six cognitive levels in Bloom’s Taxonomy. Concept Mapping theory is another well-known tool for student assessment. Educators use concept maps to assess the breadth of a student’s understanding of a subject, which reveals how well students grasp connections among concepts (Trochim, 1989). Therefore, to assess whether the model genuinely possess the knowledge required for test instance, StructEval would develop a series of instances revolving the critical concepts based on knowledge graph.

heading: 3 StructEval Framework
text: The overall framework of StructEval is illustrated in Figure 2, which consists of two modules. Given a seed instance, the first module would evaluate the model’s ability on test objective across multi cognitive level. It first identifies the underlying test objective of this instance and then generates multiple relevant instances covering six cognitive levels of Bloom’s Taxonomy. The second module evaluates the model’s comprehensive understanding of all critical concepts related to the seed instance. It extracts the essential concepts that must be understood and develops a series of extended questions around these concepts using a knowledge graph. In the following, we would describe the StructEval framework in detail. used for educational assessment.

heading: 3.1 Bloom’s Taxonomy-based Instance Generation
text: As shown in Figure 2, given a seed instance, the first module of StructEval automatically generates test instances corresponding to the six cognitive levels in Bloom’s Taxonomy through the following steps: 1) extract the test objective examined by the seed instance; 2) retrieve relevant documents and re-rank the document chunks based on their relevance to seed instance; 3) generate candidate evaluation instances for each cognitive level in Bloom’s Taxonomy using in-context learning; 4) select instances that best meet the requirements and refine them to be more challenging. Subsequently, each component will be introduced in detail.
Test Objective Extraction aims to identify the underlying test objective for each seed instance. For example, the test objective for question “Which one is not a variety of apple?” is “apple cultivars”. However, such a single question is insufficient to thoroughly evaluate the LLM’s related knowledge. Therefore, to comprehensively assess how much knowledge the LLM possesses about the test objective and its level of understanding across different cognitive tiers, we conduct a structured evaluation around this test objective. In our framework, we prompt LLM with few-shot demonstration to extract the test objective examined by each instance in the benchmarks.
Relevant Document Retrieval Given the test objective corresponding to the seed instance, an intuitive approach is directly prompting LLM to generate instances for each cognitive level. However, this approach is severely compromised by the LLM’s hallucinations, resulting in a significant proportion of incorrect instances. Therefore, StructEval would first retrieve relevant passages, and then re-rank document chunks based on the correlation with the seed instance. This procedure ensures that the generation of subsequent instances is firmly based on the retrieved context, guaranteeing the precision and pertinence of the generated instances.
Candidate Instances Generation aims to generate multiple candidate instances for each cognitive level in Bloom’s Taxonomy, based on the test objective with relevant document chunks. As demonstrated in Table 1, we meticulously design the prompt for LLM to generate relevant, correct and helpful instances corresponding to each cog-
nitive level. The prompt begins with introducing Bloom’s Taxonomy and current cognitive level, following by the task instruction which includes three principles to ensure the answerability, accuracy and relevance of the generated instances. Subsequently, we provide manually created few-shot demonstrations, and ask LLM to generate candidate instances using these demonstrations as references.
Instance Selection and Refinement Since the quality and difficulty of these instances may vary greatly, as shown in Figure 2, we introduce a postprocessing module aimed at selecting the highest quality instances for each cognitive level. 1) To ensure the answerability and correctness of instances, we prompt LLM to eliminate questions necessitating specific contextual information for resolution, and employ Retrieval-Augmented Generation (RAG) module to exclude questions that cannot be correctly answered based on the provided context, thereby ensuring the accuracy of the generated answers; 2) To enhance the quality and difficulty of instances, inspired by Clark et al. (2018); Lin et al. (2022), we establish a comprehensive pool of diverse LMs. Questions that all models could answer correctly were eliminated, thus ensuring the discriminative efficacy.
Ultimately, for each instance within the original benchmark, we develop a hierarchical evaluation
system capable of extensively assessing the tested model’s knowledge across all six cognitive levels in Bloom’s Taxonomy.

heading: 3.2 Concept Mapping-based Instance Expansion
text: The second module evaluates LLMs’ knowledge for each test objective with a concept map. The hypothesis behind is also intuitive: if a model genuinely possesses the necessary knowledge to answer a given question, it should demonstrate a comprehensive understanding of the critical relevant concepts. Specifically, as illustrated in Figure 2, StructEval utilizes LLM and knowledge graph to expand the breadth of existing benchmarks with following steps: 1) Identify the key concepts that must be understood to correctly answer the seed question; 2) Retrieve relevant knowledge sub-graphs for each concept and select the necessary knowledge triplets from all the candidates for understanding the original question; 3) Transform the selected triplets into test instances and optimize their difficulty.
Critical Concept Identification aims to identify the critical concepts that must be understood to correctly answer the seed question. These concepts are then linked to the entries in knowledge graph to facilitate subsequent knowledge retrieval. Previous approaches such as BLINK (Wu et al., 2020) are constrained on the entity label set and fail to discern between critical and non-critical concepts. Therefore, we prompt LLM with few-shot demonstration to identify the critical concepts in instances.
Knowledge Graph Retrieval and Selection involves retrieving the identified critical concepts across the entire knowledge graph and extracting relevant knowledge triples from the sub-graph as candidates. Given the potential enormity of the candidate set, which may contain extraneous triplets not aiding in determining the model’s ability to answer the seed question, similar to Guan et al. (2023), we prompt the LLM to select the helpful knowledge triplets with few-shot demonstrations.
Instances Generation and Optimization transforms the selected factual triples into evaluation instances. Similar to Petroni et al. (2019), we utilize the subject entity and its relation to formulate the question, with the object entity as the answer. For multiple-choice questions, in order to ensure the difficulty of the questions, we first use the taxon-
omy of the knowledge graph to identify the finestgrained entity category corresponding to the correct answer. Then, we select the incorrect options from other entities within the same category.
Ultimately, we construct a multi-nodal evaluation framework for each test instance, offering a comprehensive assessment of the language model’s grasp of pertinent critical concepts.

heading: 4 Implementations and Experiments of StructEval
text: In this section, we first implement StructEval across three widely-used benchmarks. Through human evaluation, we demonstrate capability of StructEval to automatically construct large-scale benchmarks while ensuring the helpfulness, answerability and correctness of generated instances. Then, we demonstrate how StructEval could improve the robustness and consistency of LLM evaluation from the following perspectives. Firstly, StructEval requires LLMs to understand the test objective across multiple cognitive levels and critical concepts. In this case, a contaminated model which merely memorize specific answers may achieve high performance in original benchmark, but cannot gain performance improvements on the structured evaluation since it lacks of corresponding knowledge. Therefore, StructEval can effectively resist data contamination issues, providing robust evaluation results even when the test data is leaked. Secondly, since the evaluation results do not rely on the correctness on single instance, it does not depend on confounders introduced by specific instances, such as prompt selection, surface form shortcut and data distribution. Therefore, compared with single-item assessment, StructEval can provide a much more robust and consistent evaluation conclusion.

heading: 4.1 StructEval -based Benchmarks
text: Finding 1. By leveraging the advanced generative capabilities of LLMs, and meticulously orchestrating the construction process guided by pedagogy theories and grounded in credible knowledge sources, StructEval is capable of automatically construct large-scale benchmarks while ensuring the helpfulness, answerability and accuracy of generated instances.
To demonstrate the reliability and quality of the automatically generated instances by StructEval, we adopt StructEval on three widely used bench-
marks for LLMs, and conduct human evaluations from three aspects.
Seed Benchmarks include the following three wildly used LLM evaluation benchmarks, and the corresponding statistics are demonstrated in Table 2: 1) MMLU (Hendrycks et al., 2021) is a large-scale benchmark designed to measure the knowledge in large language models. We filtered out subjects unsuitable for StructEval, such as logical reasoning and numerical computation, and used the remaining 48 subjects for experiments. 2) ARC (Clark et al., 2018) consists of a set of science exam questions drawn from a variety of sources, which is widely used to assess the knowledge in LLMs. The benchmark is partitioned into a challenge set and easy set, and we include both of them in our experiments. 3) OpenBook QA (Mihaylov et al., 2018) is a question-answering datasets that consists of multiple-choice elementary-level science questions. For each benchmark, we randomly sample 200 test instances generated by StructEval to conduct human evaluations.

heading: LLM and Knowledge Source Implementation
text: Considering the balance between cost, efficiency, and the quality of generation, we select ChatGPT (gpt-3.5-turbo-1106) for LLM generation tasks in this paper, and our framework can also be easily adopted to other large models and knowledge sources. We adopt BM25 (Maron and Kuhns, 1960) for document retrieval and BGE (Xiao et al., 2023) for chunk re-ranking. We select Wikipedia (Wikipedia contributors, 2004) as knowledge source since it encompasses the vast majority information about test objectives and possesses an high density of knowledge, and use Wikidata (Vrandečić and Krötzsch, 2014) for fact retrieval, since it is one of the most comprehensive knowledge bases covering structured knowledge.
Metrics We propose the following metrics to comprehensively assess the quality of test instances generated through StructEval: 1) Instance Helpfulness, which is calculated by the proportion of generated evaluation instances that conform to the target test objective or critical related concepts; 2) Question Answerability, which is calculated by the proportion of generated questions that can be answered without relying on external context. 3) Answer Correctness, which is calculated by the proportion of generated evaluation instances that contain the correct answer. The detailed annotation

heading: Benchmark Original Bloom Concept Structured

heading: Benchmark Helpfulness Answerability Correctness
text: guideline is presented in the appendix.
Benchmark Assessment From table 2, we find that StructEval is able to automatically construct a large-scale multi-level and multi-nodal evaluation system based on original benchmark, provide novel test instances and structured evaluation protocol for existing benchmarks. Moreover, the human evaluation results are shown in table 3. We can find that StructEval could construct the structured evaluations while ensure the high quality of generated instances in all the aspects of instance helpfulness, question answerability, and answer correctness. The few errors mainly due to that GPT-3.5 generate questions relying on context to answer or with multiple correct choices, we also provide a detailed error analysis and the annotation guideline in the Appendix E due to page limitations.

heading: 4.2 Robustness of StructEval
text: Finding 2. By expanding the benchmark across both depth and breadth dimensions, StructEval is capable of robustly evaluating the capabilities of LLMs, resisting the risks of data contamination, and providing stable results even under data contamination settings.
Data contamination refers to the inclusion of test data in the training dataset of evaluated models, which can significantly skew the apparent performance and capabilities of models, leading to misleading conclusions about their true effectiveness. Addressing data contamination becomes increasingly crucial for large language models as the training data grow exponentially with the data sources
and processing recipes being obscure.
To demonstrate the effectiveness of StructEval in resisting the risk of data contamination, we compare the performance divergences of LLMs with and without data contamination, on the original benchmark, the data augmented benchmark and StructEval-constructed benchmark respectively. Specifically, for a seed benchmark and a base model, we use instruction fine-tuning (IFT) to train the model on both a clean dataset and a dataset contaminated with test data. To make a fair comparison, we ensure that both datasets maintain identical scale and similar data composition. Simultaneously, we integrate Alpaca-GPT-4 (Taori et al., 2023) dataset into both the training data to ensure data diversity and prevent training collapse. In this case, the contaminated set consists of AlpacaGPT-4 and the test data, while the clean set consists of Alpaca-GPT-4 and an equal number of multichoice questions which are randomly sampled from an out-of-distribution benchmark Xiezhi (Gu et al., 2023). To ensure the robustness of our conclusions, we consider 5 wildly used base LLMs of various scales including LLaMa-7B&30B(Touvron et al., 2023a), LLaMa-2-7B&13B(Touvron et al., 2023b) and Mistral-7B(Jiang et al., 2023). Each model is trained through 3 epochs with batch size of 256
sequences, using Adam with learning rate 2e− 5. We also compare our method with the following augmentation-based approaches including character-level, word-level and instance-level, which are able to generate adversarial samples while ensuring the answerability and correctness of test instances: 1) CharDisturb (Morris et al., 2020): which randomly substitutes, deletes, inserts and swaps characters in original question. 2) WordNet (Miller, 1992), which randomly replaces words with WordNet synonyms. 3) Paraphrasing (Zhu et al., 2023c), which prompts ChatGPT to generate paraphrasing for each test question. 4) BackTranslation (Sennrich et al., 2016), which translates the test question into another language and translates it back. 5) OptionShuffle (Yang et al., 2023), which re-ordered the options for each question to prevent LLMs memorizing specific option for question.
The results in Table 4 clealy demonstrate the significant role of StructEval in resisting data contamination: 1) The performance of original benchmark can severely suffer from data contamination due to the superior memorizing capabilities of LLMs, resulting in a serious overestimation of the model’s capabilities. For instance, the performance of all models on MMLU increase by at least 29% when the training data is contaminated 2) Previous
augmentation-based approaches struggle to resist data contamination. Despite adjustments to the surface form of the original instances, due to the LLMs’ advanced memorizing and language comprehension capabilities, they still achieve significant benefits from data contamination. 3) StructEval is able to provide stable evaluation results, regardless of whether the training data is contaminated. For example, due to data contamination, the performance of LLaMa-2-13B improves by 31.71% on the original MMLU, but changes by only 0.79% on the structured-MMLU generated by StructEval, which remains almost unchanged. The finding remains consistent across all base LLMs and benchmarks. Such results effectively demonstrate that StructEval can play a role in anti-attack and contamination monitoring for evaluation.

heading: 4.3 Consistency of StructEval
text: Finding 3. By conducting structured assessments across various cognitive levels and essential concepts, instead of basing assessments solely on the accuracy of a single instance, StructEval achieves valid assessment of models, providing consistent conclusions regarding various model capabilities.
As we discussed above, StructEval can also serve as a more stable reference for assessing the knowledge capabilities of language models, which can give more stable evaluation results to various LLMs, and reach a consistent conclusion. Demonstrating this requires to collect numerous benchmarks with similar evaluation objective and distribution, and observe whether the evaluation conclusions are consistent on original data, aug-
mented data and StructEval-constructed data respectively. To facilitate our experiments, we refer to Cao et al. (2022) and use rank consistency across multiple runtimes as the evaluation metric. Specifically, we randomly sample 10000 sub-set with K subjects from MMLU, and evaluate rank consistency by measuring the percentage of the most popular rank of each model in 10000 runtimes. For instance, if ChatGPT ranks at 3rd place in 6500 of the 10000 runtimes, then the rank consistency of ChatGPT would be 65%. To make a comprehensive evaluation, we conduct experiments on 13 different open-source large language models across various parameter scales, including LLaMA-7B&30B, LLaMA-2-7B&13B, Mistral-7B&8*7B, Baichuan2-7B&13B(Baichuan, 2023), Qwen-7B&14B, Qwen1.5-7B&14B(Bai et al., 2023) and Yi-6B. We report the rank consistency of each model, as well as the rank consistency across all models.
The results in Table 5 and Figure 3 demonstrate that StructEval can significantly improve the evaluation consistency: 1) The consistency of current LLM evaluations are relatively poor: when using original isolated instances to compare the ability of different models, the overall rank consistency is only 1.24%. 2) Previous strategies can hardly improve the rank consistency. Although they modify the original data, they still adhere to the paradigm of single-item assessment. As a result, they remain susceptible to interference from confounders and struggle to provide more consistent evaluation conclusions across all models. 3) StructEval provides much more consistent evaluation conclusions regarding the ability of different LLMs: the overall rank consistency improved from 1.24% to 33.17% when K = 15, and the rank consistency of most LLM is substantially improved, reaching a
more reliable conclusion.

heading: 5 Conclusion
text: This paper proposes a novel evaluation framework for large language models named StructEval. Through structurally evaluating model’s capability for each test objective across multiple cognitive levels and critical concepts, StructEval achieves more comprehensive, robust and consistent evaluation for LLMs. Experimental results demonstrate that StructEval could effectively resist the risk of data contamination and significantly improve the rank consistency across models. The corresponding benchmarks and leaderboard will be released, which will benefit our understanding of LLMs’ capabilities. StructEval is also broadly applicable to various applications. For instance, StructEval can function as a customizable benchmark construction framework, capable of automating evaluations for any granularity of assessment objectives, please refer to Appendix A for details and experiments. Furthermore, our study also sheds light on the design of future principled and trustworthy instance collection and LLM evaluation protocols.

heading: Limitations
text: Considering the balance between cost, efficiency and quality for benchmark construction, we currently use GPT-3.5 for generation in this paper, which may limit the difficulty and quality of generated instances. In the future, we will introduce more powerful LLMs (e.g., GPT-4) or incorporate human to our framework, to further improve the qualify of test instances, and release the corresponding updated benchmarks. Moreover, to facilitate the assessment of our framework, we currently select to implement StructEval based on multi-choice benchmarks. Please also kindly note that our framework can be easily adapted to other formats of benchmark such as open-end QA and multi-turn conversation, which will be included in our future work.

heading: Acknowledgements
text: We sincerely thank all anonymous reviewers for their insightful comments and valuable suggestions. This research work is supported by the National Natural Science Foundation of China under Grants no. 62122077 and no. 62106251.

heading: A Customized Benchmark Construction based on StructEval
text: The majority of current benchmarks assess models in a static and coarse-grained manner. They typically start by defining a broad assessment domain, such as general knowledge, medical knowledge, or legal knowledge, and then extensively collect questions and answers within that domain. These instances are then fixedly used to evaluate models.
However, with the rapid development of large-scale models, this assessment paradigm faces two issues: 1) The obsolescence rate of static assessments is accelerating, and they are prone to rapid invalidation after reaching benchmark saturation. 2) As the application scenarios of large models become more diverse and refined, such coarse-grained assessment methods struggle to meet the rapidly growing needs for customization in real-world scenarios. For example, evaluating an AI assistant designed to aid in railway museum explanations should target "railway knowledge" rather than "general knowledge". Manually collecting data for each of these customized scenarios is not feasible.
Benefiting from the automatic and dynamic features of StructEval , we can restructure it into a multi-agent-based customized benchmark construction framework. As illustrated in Figure 4a, given a customized assessment objective (e.g., 2022 FIFA World Cup), two agents including topic proposer and concept proposer would list the essential test objectives and important concepts for comprehensive evaluate LLMs within target objective. Then, StructEval would follow the same procedures in Figure 2, and automatically construct a multi-level and multi-nodal benchmarks for evaluation. In order to validate the effectiveness of our approach, we followed the aforementioned steps with GPT-4 to construct a smallscale dataset named FWC_2022, with "2022 FIFA World Cup" as the evaluation objetive. Subsequently, we compare the performance of various models on both a large-scale general benchmark MMLU and FWC_2022. FWC_2022 comprises a total of 240 multiple-choice questions pertaining to various aspects of “2022 FIFA World Cup.” Please refer to the appendix for details and instances of the dataset due to page limitations.
The results in Table 4b demonstrate the necessity for customized fine-grained evaluations: 1) In the large-scale general benchmark MMLU, GPT-3.5-turbo perform significantly better than other two LLMs, which indicate that the GPT-3.5 has a stronger general knowledge ability. 2) However, the knowledge cuttoff of GPT-3.5-turbo is September, 2021. Therefore, in FMC_2002, the evaluate datasets about “2022 FIFA World Cup”, GPT-3.5-turbo perform worse than other two LLMs which are newly released. The inconsistent conclusion between these two benchmarks indicate that previous static and fine-grained evaluation could not adapt to many scenarios, and StructEval could serve as an valuable tool for a customized, dynamic and fine-grained evaluation automatically.

heading: B Examples of Test Instances
text: Here is an example of generated instances by StructEval.
Level Instance Explanation Remembering What is the cranial nerve number associated with
the facial nerve? A.CN VII B.CN VI C.CN V D.CN VIII Correct Answer: A.CN VII
As mentioned in the provided document, the facial nerve is also known as the seventh cranial nerve, cranial nerve VII, or simply CN VII.
Understanding Where does the facial nerve exit the skull? A.Temporal bone B.Parietal bone C.Occipital bone D.Frontal bone Correct Answer: A.Temporal bone
As mentioned in the document, the facial nerve typically exits the skull at the stylomastoid foramen, which is located in the temporal bone.

heading: C Framework Design Details
text: This section will introduce the more details about our framework design.
C.1 Instance Generation based on Bloom’s Taxonomy

heading: C.1.1 Test Objective Extraction Instruction
text: We use the following instruction to identify the underlying test objective for a seed instance.

heading: C.1.2 Details of Instance Selection and Refinement
text: The post-processing modules are crucial for ensuring the quality of generated instances since there exist several issue for test instances directly generated by LLMs. Table 9 present a case study to demonstrate how this post-processing module to filter out candidate instances with low quality. And Table 8 shows the instruction for retrieval-augmented generation.
C.2 Instance Expansion based on Concept Mapping

heading: C.2.1 Critical Concepts Extraction
text: We use the following instruction to extract the critical concepts that must be understood to correctly answer the seed question.

heading: C.2.2 Helpful Knowledge Triplets Selections
text: We use the following instruction to select helpful knowledge triplets from all candidates.

heading: D Human Annotation Guidelines
text: Here is the annotation guidelines for our human evaluation is shown in Figure 5. We recruit 5 annotators to participate in the human evaluation, each of whom possesses a bachelor degree. To ensure the clarity and consistency in the evaluation, we provided detailed instructions and examples in the annotation guidelines. Each instance is annotated by 3 participants, and the final results are determined by a majority vote.

heading: E Error Analysis of Constructed Benchmark
text: As we discussed in Section 4.1, according to the human evaluation results, there still exist a fewer instances which not meet the standard. In order to find the underlying causes of these errors, we conduct a detailed error analysis which is demonstrated in Table 12.
Test Objective Example Cause Answerability Issue Economic growth Question: What constraints to economic growth are highlighted? A. Government intervention B. Depleted resources and energy consumption C. Technological advancements D. Increased labor force Context-dependent instance not be filtered Moon What will happen in 50 billion years according to the text? A. The Moon’s rotation will stop B. The Sun will become a red giant C. The Moon will collide with Earth D. The Earth’s rotation will match the Moon’s orbital period Context-dependent instance not be filtered Dementia What do physicians need to include in any dementia evaluation? A.A memory assessment B.A somatic disturbance evaluation C. A depression screening D.A sensory function test Instance with multianswers not be filtered by RAG Fish farming What is the most important fish species produced in fish farming worldwide? A.Tuna B.Carp C.Mussels D.Salmon Instance with multianswers not be filtered by RAG
Human subject research Question: What are the three guidelines that serve as the baseline for the report? A. Beneficence, justice, and respect for persons B. Beneficence, integrity, and respect C. Prudence, integrity, and honesty D. Beneficence, justice, and honesty Unclear reference

heading: Future Work
text: In the future, we will introduce more powerful LLMs (e. g. , GPT-4) or incorporate human to our framework, to further improve the qualify of test instances, and release the corresponding updated benchmarks. Moreover, to facilitate the assessment of our framework, we currently select to implement StructEval based on multi-choice benchmarks. Please also kindly note that our framework can be easily adapted to other formats of benchmark such as open-end QA and multi-turn conversation, which will be included in our future work