xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation

## Abstract

Abstract The continuous advancement of large language models (LLMs) has brought increasing attention to the critical issue of developing fair and reliable methods for evaluating their performance. Particularly, the emergence of subjective or non-subjective cheating phenomena, such as test set leakage and prompt format overfitting, poses significant challenges to the reliable evaluation of LLMs. Since evaluation frameworks often utilize Regular Expression (RegEx) for answer extraction, some models may adjust their responses to comply with specific formats that are easily extractable by RegEx. Nevertheless, the key answer extraction module based on RegEx frequently suffers from extraction errors. This paper conducts a comprehensive analysis of the entire LLM evaluation chain, demonstrating that optimizing the key answer extraction module can improve extraction accuracy, reduce LLMs’ reliance on specific answer formats, and enhance the reliability of LLM evaluation. To address these issues, we propose xFinder, a model specifically designed for key answer extraction. As part of this process, we create a specialized dataset, the K ey A nswer F inder (KAF) dataset, to ensure effective model training and evaluation. Through generalization testing and evaluation in real-world scenarios, the results demonstrate that the smallest xFinder model with only 500 million parameters achieves an average answer extraction accuracy of 93.42%. In contrast, RegEx accuracy in the best evaluation framework is 74.38%. xFinder exhibits stronger robustness and higher accuracy compared to existing evaluation frameworks. All resources for xFinder are available at https://github.com/IAAR-Shanghai/xFinder .

## 1 Introduction

Large language models (LLMs) have seen rapid advancements in recent years [, ]. To explore the strengths, weaknesses, and differences of various LLMs, numerous researchers have introduced a variety of evaluation benchmarks aimed at evaluating the performance of LLMs across various dimensions [, , , ]. Furthermore, to facilitate comparisons of different LLMs on these benchmarks, several institutions have developed unified LLM evaluation frameworks, such as LM Eval Harness(^1^11LM Eval Harness is the backend for Hugging Face’s popular Open LLM Leaderboard [].) [] and OpenCompass []. These frameworks are capable of uniformly evaluating LLM in reasoning, text generation, and domain-specific knowledge.

Despite the widespread application of the aforementioned evaluation benchmarks and frameworks, some researchers have raised concerns about the fairness and reliability of current LLM evaluation efforts [, ]. For instance, the benchmarks may suffer from data contamination and leakage in their test sets [, ]. Furthermore, a systematic review has revealed multiple unreliabilities at various stages of the existing evaluation pipeline, as illustrated in Figure 1.

Figure: Figure 1: Typical LLM Evaluation Pipeline.

##### Data Item Transformation

Given the uncertainty of LLM output content, evaluation frameworks often transform datasets to achieve uniform evaluation across different tasks. For example, they may convert datasets into the Multiple-Choice Question-Answer (MCQA) with alphabet options. However, it has been shown that for some tasks, the evaluation results under this MCQA form with alphabet options may differ from those under the open-ended setting  [].

##### Question Prompting and LLM Answering

Various institutions may employ different configurations (such as temperature settings, decoding strategies, etc.) and prompts when evaluating LLMs. LLMs are highly sensitive to these settings []. Even minor modifications (such as changing the case of letters in options or adding extra spaces) can significantly alter the model’s responses [, , , , ], thus affecting the fairness of the evaluation outcomes. Moreover, while employing consistent prompts may ensure a degree of fairness, uniform prompts may not suit all LLMs, potentially contradicting the initial intent of evaluating the models’ inherent capabilities.

##### Key Answer Extraction and Matching

Current evaluation frameworks predominantly rely on Regular Expression (RegEx) to extract answers from model outputs. For example, they may select content following phrases like "The answer is" as the key response. However, many LLMs fail to generate standardized answers [], which complicates the extraction of key answers from their outputs. Figure 2 illustrates instances where current evaluation frameworks failed to extract responses correctly. There are two primary scenarios where regular rules fail to extract answers: first, when the model does not produce a relevant response to the evaluation question, and second, when the model’s response does not conform to standards. In the first scenario, it is unreasonable to classify a model’s failure to provide a relevant response as an error, as the evaluation aims to measure the model’s knowledge comprehension and reasoning capabilities, not merely its instruction-following ability. In the second scenario, although the LLM’s response effectively addresses the question, the response’s non-standard format impedes correct extraction using RegEx, resulting in ’false judgments’.

Figure: Figure 2: Cases where LM Eval Harness and OpenCompass fail in extracting key answers. A/T/C/M stands for tasks with alphabet / short text / categorical label / math options, respectively.

Consequently, existing LLM evaluation practices still face numerous deficiencies in terms of fairness and reliability. How to conduct reliable evaluations has become an urgent issue in the field of LLMs. For extracting and matching key answers, the most effective and reliable methods involve using advanced LLMs (such as GPT-4) alongside manual evaluation []. However, due to time and financial constraints, these methods are seldom applied in practical evaluations. As a result, some initiatives have developed models, known as Judge models, through fine-tuning strategies to automatically determine the correctness of LLM responses [, ]. Unfortunately, these methods significantly lag behind advanced LLMs or manual evaluation in terms of generalizability.

To address the conflict between fairness and cost, we propose xFinder, a novel evaluation component designed to replace the answer extraction phase of existing frameworks. We have optimized the logic for matching extraction results, thereby enhancing the reliability of evaluations. The core of xFinder involves constructing a large-scale LLM response evaluation dataset and proposing a high-precision, low-parameter model based on this dataset to serve as the evaluator. By replacing the existing RegEx-based answer evaluation methods, xFinder can evaluate LLM outputs more accurately and efficiently, further enhancing the reliability of LLM evaluations.

To construct xFinder, we initially created the Key Answer Finder (KAF) Dataset. The training set is used to fine-tune xFinder, and we successfully fine-tuned 19 LLMs of varying sizes, all of which performed well on both the test and generalization sets. For instance, on the testing set, xFinder-qwen1505 achieved an average extraction accuracy of 96.88%. To ensure the generalizability of xFinder, we constructed a generalization set using outputs from various models on different tasks beyond the original KAF dataset. xFinder-qwen1505 achieved an average extraction accuracy of 93.42% on this dataset, demonstrating the model’s substantial generalizability. Finally, to ensure consistent experimental settings, we utilized xFinder and the traditional evaluation framework’s RegEx answer extraction module to assess 10 LLMs on a common set of evaluation tasks under real-world conditions, subsequently ranking the 10 LLMs’ performance. We found that the RegEx-based key answer extraction components in existing frameworks often fail to extract answers correctly or at all. Moreover, rankings produced by different evaluation frameworks show significant inconsistencies. In contrast, our fine-tuned xFinder models of various sizes achieved high consistency in LLM rankings, demonstrating greater stability in the evaluation results.

We summarize our primary contributions as follows:

- •
We provide a comprehensive review of LLM evaluation processes in the industry, identifying critical factors that can lead to unreliable evaluation results.
- •
We introduce xFinder, a model specifically designed for key answer extraction. The KAF dataset supports its effective training and evaluation.
- •
In our extensive experiments, we demonstrate that RegEx-based evaluation methods are unreliable, while our xFinder model significantly improves reliability.

## 2 Related Work

From the perspective of LLM capabilities, current evaluations can be categorized into knowledge and capability evaluations [], vertical domain evaluations [], and alignment and safety evaluations [, ]. When considering the types of questions in benchmark datasets, evaluations can be divided into objective questions (such as multiple-choice, classification, mathematical questions, fill-in-the-blank, etc.) and subjective questions (such as summarization, text continuation, open-ended questions, and code generation). For objective questions, metrics like accuracy and F1-score are widely used. These metrics rely on the ability to exactly match the LLM’s output with the correct answer.

Currently, many popular unified LLM evaluation frameworks exist, including LM Eval Harness [], OpenCompass [], OpenAI Evals [], and UltraEval []. When using objective question datasets to evaluate LLMs, these frameworks first conduct data item transformation, converting classification and fill-in-the-blank questions into multiple-choice questions. They then extract and match key answers using RegEx methods. This process either involves literally extracting the option letters for text response evaluation or extracting the probabilities of different option letters during generation for first-token evaluation. However, both methods have notable drawbacks when evaluating objective questions. In text response evaluation, inaccurately extracting the key answer from the LLM’s response can lead to incorrect evaluation results. On the other hand, first-token evaluation may not accurately reflect the LLM’s intended answer, as it overlooks the LLM’s reasoning process [].

In addition to RegEx-based answer extraction, recent research has increasingly focused on fine-tuning LLMs for evaluation, known as LM-based methods. Specifically, these methods include training evaluation models to select the better response to the same question (Pairwise Selection) [, , ] and training evaluation models to score a given question directly, response, and reference (Pointwise Grading) [, , ]. However, these methods have significant drawbacks. The current prompt and fine-tuning dataset designs often result in evaluation LLMs with poor generalization, making them difficult to compare with strong LLMs like GPT-4 [, ].

Recently, the concept of reliable evaluation has been proposed (^2^22[https://sites.google.com/view/real-info2024/overview](https://sites.google.com/view/real-info2024/overview)). This concept aims to more scientifically assess the authenticity of content generated by LLMs and the potential risks to information systems []. However, systematic research and efforts to improve reliable evaluation are still lacking. To the best of our knowledge, our work is the first to conduct a systematic study in this area.

## 3 Problem Definition

The existing methods described in Section 2 aim to align the scores assigned by evaluation models with human-expected scores. In other words, these methods strive to ensure that the evaluation model’s judgments of LLM outputs match human judgments. Since scoring is subjective, unlike the objective determination of correctness, these methods are more suitable for open-ended tasks like text translation and summarization. In contrast, our method aims to align xFinder’s extraction of key answers from LLM outputs with the key answers expected by humans. Fundamentally, this method is more suitable for tasks with deterministic answers such as multiple-choice questions and mathematical problems. We define the key answer extraction task as follows:

Let $\mathcal{D}$ denote an evaluation dataset consisting of triplets: a question, a set of options, and the correct answer, as shown in Equation 1. Here, $q$ denotes the question, $\mathcal{C}$ denotes the set of options, and $a$ denotes the correct answer. Let $\Sigma$ be the set of all tokens, and $\mathcal{P}$ the power set.

$$ $\mathcal{D}\subset\{(\boldsymbol{q},\mathcal{C},\boldsymbol{a})\mid\boldsymbol {q}\in\Sigma^{*},\boldsymbol{a}\in\Sigma^{*},\mathcal{C}\in\mathcal{P}(\Sigma^ {*})\}$ (1) $$

Each data point $(\boldsymbol{q},\mathcal{C},\boldsymbol{a})$ in $\mathcal{D}$ is fed into the LLM, producing an output $\boldsymbol{y}$, as shown in Equation 2:

$$ $\boldsymbol{y}=\text{LLM}(\boldsymbol{q},\mathcal{C},\boldsymbol{a})$ (2) $$

If a unique substring in $\boldsymbol{y}$ can be identified as one of the three types—Direct, Prompt wrapped, or Converted question wrapped—then this substring is considered the Key Answer Sentence $\boldsymbol{s}$. Intuitively, $\boldsymbol{s}$ is a sentence that contains the final answer to the question $\boldsymbol{q}$. The definitions of these three types and their corresponding sets are as follows:

##### Direct

This refers to $\boldsymbol{s}$ directly providing the final answer. Its corresponding set is specifically defined as shown in Equation 3. Here, $\tau:\Sigma^{*}\rightarrow\mathcal{P}(\Sigma^{*})$ denotes the function for synonym transformation, such as $\tau\left(\text{Option A}\right)=\{\text{A},\text{The First Option},\left<
\text{corresponding content of the option}\right>,\cdots\}$.

##### Prompt wrapped

Here, $\boldsymbol{s}$ consists of relevant prompts combined with the final answer, as detailed in Equation 4. The subset $\mathcal{F}\subset\mathcal{P}(\Sigma^{*})$ represents a collection of prompts guiding to the answer, where the placeholder <final answer> indicates the position of the final answer. The expression $\boldsymbol{s}\circ\boldsymbol{f}$ denotes the new sentence formed by replacing the <final answer> token in $\boldsymbol{f}$ with $\boldsymbol{s}$.

##### Converted question wrapped

In this approach, $\boldsymbol{s}$ consists of the original question $\boldsymbol{q}$ converted into a declarative statement, combined with the final answer. The corresponding set is specified in Equation 5. The function $\zeta:\Sigma^{*}\rightarrow\Sigma^{*}$ represents the statement conversion function, which transforms $\boldsymbol{q}$ into its declarative form and identifies the position of the placeholder. For example, $\zeta(\text{What is the age of Mark 10 years later?})=\text{Mark will be <
final answer> years old in 10 years.}$.

Particularly, when multiple substrings meet the criteria, it is necessary to determine if a Chain of Thought (CoT) process(^3^33CoT (Chain of Thought []) refers to the process of progressively reasoning through $\boldsymbol{q}$ to arrive at the final result.) exists within $\boldsymbol{y}$. If a CoT process is present in $\boldsymbol{y}$, $\boldsymbol{s}$ is defined as the unique substring that meets the criteria and appears after the CoT process in $\boldsymbol{y}$. If no CoT process is present, $\boldsymbol{s}$ does not exist in $\boldsymbol{y}$.

$$ $\displaystyle\mathcal{S}_{1}$ $\displaystyle=\{\boldsymbol{c},\tau(\boldsymbol{c})\mid\boldsymbol{c}\in \mathcal{C}\}$ (3) $\displaystyle\mathcal{S}_{2}$ $\displaystyle=\{\boldsymbol{s}\circ\boldsymbol{f}\mid\boldsymbol{s}\in\mathcal {S}_{1},\boldsymbol{f}\in\mathcal{F}\}$ (4) $\displaystyle\mathcal{S}_{3}$ $\displaystyle=\{\boldsymbol{s}\circ\boldsymbol{x}\mid\boldsymbol{s}\in\mathcal {S}_{1},\boldsymbol{x}\in\zeta(\boldsymbol{q})\}$ (5) $$

The definition of the key answer $\boldsymbol{k}$ is divided into two scenarios:

- 1.
If $\boldsymbol{s}$ does not exist, $\boldsymbol{k}$ is set to [No valid answer];
- 2.
If $\boldsymbol{s}$ exists, $\boldsymbol{k}$ is the option within the set $\mathcal{C}$ that best corresponds to what is described by $\boldsymbol{s}$, as defined in Equation 6:

$$ $\exists\boldsymbol{k}\subset\mathcal{C},\text{s.t.}\\ \exists\boldsymbol{x}\in\{\boldsymbol{k},\tau(\boldsymbol{k})\},\boldsymbol{x} \subset\boldsymbol{s}$ (6) $$

## 4 Methodology

To effectively fine-tune xFinder, a high-quality dataset is essential. To our knowledge, there is currently no dedicated dataset for Key Answer Extraction. In this section, we will detail the implementation process of xFinder, including the construction of the KAF dataset and the training of xFinder, as illustrated in Figure 3. The KAF dataset encompasses a variety of evaluation tasks, including questions, optional answer ranges, LLM responses to the questions, and the extracted key answers. This dataset is divided into three segments: a training set, a test set, and a generalization set, which are used for fine-tuning, testing, and assessing xFinder’s performance, respectively. The training set comprises 26,900 samples, the test set includes 4,961 samples, and the generalization set contains 4,482 samples. For more detailed information, please refer to the Appendix B.1. Section 4.1 details the process of LLM response collection, Section 4.2 describes our method and workflow for annotating key answers in the LLM-generated content, and Section 4.3 elaborates on our detailed approach to training xFinder.

Figure: Figure 3: Schematic of the research framework. The first three stages correspond to Sections 4.1, 4.2, and 4.3 respectively; the final stage illustrates the replacement of RegEx with xFinder in the current evaluation pipeline. The experiments in Section 5.2 demonstrate the efficacy of our approach within this pipeline. Note: The percentages 80.3%, 77.0%, and 2.2% in the center of the figure around ’unreliable evaluation’ indicate results from the Llama3-8B-Instruct on the GSM8K benchmark using RegEx evaluation via the LM Eval Harness, OpenCompass, and UltraEval frameworks, respectively, while our method achieves a reliable result of 80.2%.

### 4.1 LLM Response Generation

As shown in the first part of Figure 3, we initially generate question-response pairs using different LLMs across multiple datasets. Currently, the tasks for evaluating LLM performance are quite diverse. We investigated the leading LLM benchmarks and evaluation reports to cover a wide range of popular evaluation tasks. Ultimately, we selected 18 typical evaluation task datasets, with detailed information provided in Appendix A.1. We categorized these tasks into four types, each with similar option formats, which serve as prompts to induce the LLMs to generate varied responses (as shown in the left boxes in Figure 2): (1) Alphabet option tasks. By referring to the design of mainstream evaluation rankings, these tasks are formatted as alphabet options, where each task aims to select the single correct option from multiple possibilities (i.e., A, B, C, D, etc.). (2) Short text option tasks. The requirement for the model to correctly map letters to option texts can affect the assessment of the model’s true capabilities []. Therefore, we designed tasks with key answer types as short texts in the form of alphabet options, where the goal is to select the single correct short text from multiple short texts. (3) Categorical label tasks. For traditional classification tasks (e.g., sentiment classification, topic classification, entity classification), we retained their original format, using the labels themselves as options. (4) Math option tasks. Given the output types of mathematical tasks, which could be integers, decimals, or even LaTeX code, we categorized them separately.

To further enhance the heterogeneity and richness of the dataset, we have primarily designed eight types of prompt configurations to elicit diverse responses from the LLMs. These configurations include 0-shot(-restrict(^4^44-restrict indicates that the prompt ends with the sentence ”End your final answer with ’The answer is ¡answer¿.’” to standardize the LLM’s response format.)), 0-shot-cot(-restrict), 5-shot(-restrict), and 5-shot-cot(-restrict). Specific examples of these prompts are provided in Appendix F.1.

For generating responses, it is crucial to capture a wide range of output content, given that LLMs of different series and sizes exhibit varied reasoning processes and may provide different answers to the same question. Consequently, we utilized 10 distinct LLMs for this purpose. Specific details on these models can be found in Appendix A.2.

It is important to note that the KAF dataset’s training and test sets consist of Question-Response Pairs generated by 10 LLMs across 8 evaluation tasks. To ensure the validity and fairness of the generalization evaluation, the Question-Response Pairs used in the generalization set are derived from responses by 8 LLMs on 11 entirely new evaluation tasks.

### 4.2 Auto Labelling and Human Recheck

The foundation of constructing the KAF dataset is the extraction of the key answer $\boldsymbol{k}$ from the LLM response $\boldsymbol{y}$, which is then used as the label. We employed distinct annotation strategies for the training, test, and generalization sets. For the training set, to improve the efficiency of the annotation process, a semi-automated procedure was adopted. This involved generating two sets of annotations from GPT-4 using varied prompts F.2. Utilizing the Self-Consistency strategy [], items with discrepancies between the two annotation rounds, along with all mathematics questions, were consolidated for manual re-annotation using Label Studio []. To ensure the validity and reliability of the test and generalization sets, all labels needed to be accurate, necessitating two rounds of manual annotation for these datasets. Detailed information about the annotation methods can be found in Appendix B.2.

### 4.3 Training xFinder

According to the definition of key answer extraction outlined in Section 3, our goal is to fine-tune xFinder to precisely locate $\boldsymbol{s}$ within $\boldsymbol{y}$ and accurately extract $\boldsymbol{k}$ from $\boldsymbol{s}$. The training input for xFinder includes the question $\boldsymbol{q}$, the LLM response $\boldsymbol{y}$, the key answer type, and the answer range, while the output is the key answer $\boldsymbol{k}$.

During the training process, we primarily utilized the XTuner [] tool developed by the InternLM team for fine-tuning. To fully exploit the instruction-following potential of xFinder, we meticulously crafted prompts as demonstrated in Appendix F.3. Additionally, to enhance the generalization capability of xFinder, we implemented data augmentation on the KAF training set, which included:

- •
Simulating LLM responses to alphabet option questions. We randomly selected 50% of the alphabet option questions from the KAF training set and altered their answer choices by either adding one or two options or removing one option with equal probability.
- •
Enriching the variety of prompt forms in Prompt wrapped key answer sentences. From the LLM responses in the KAF training set that contained wrapped key answer sentences, we extracted 10% and substituted the $\boldsymbol{f}$ with other elements from $\mathcal{F}$, such as replacing "The final answer is A" with "Based on the context of the question, A is the most likely answer."

## 5 Experiments

In this section, we initially fine-tuned 19 LLMs with parameter sizes ranging from 0.5B to 8B using the KAF training set. We compared their accuracy in extracting key answers on both the KAF test and generalization sets against the mainstream evaluation frameworks using RegEx methods. Subsequently, we evaluated the performance of 10 LLMs on 14 evaluation task datasets using xFinder-qwen1505, xFinder-llama38it, and the current mainstream evaluation frameworks, analyzing differences in evaluation results.

### 5.1 Extraction Accuracy: xFinder vs. RegEx

##### Results on KAF test set

We employed the QLoRA method from the XTuner framework [] [] for fine-tuning the foundation models, with specific tuning parameters detailed in Appendix C. To assess the efficacy of xFinder, we conducted tests using the KAF test set. We set the current mainstream frameworks—OpenCompass, LM Eval Harness, and UltraEval—with their RegEX extraction modules as baselines. The comparison metric is extraction accuracy, defined as the proportion of correctly extracted key answers relative to the dataset size. Partial results are presented in Table 1; for complete experimental results, refer to Table 18 in Appendix D.1.

**Table 1: Comparison of Extraction Accuracy on KAF Test Set: Mainstream Frameworks vs. xFinder Models. Specifically, xFinder-qwen1505, xFinder-qwen1518, xFinder-gemma7, xFinder-chatglm36base, and xFinder-llama38 are models fine-tuned using the KAF training set on Qwen1.5-0.5B, Qwen1.5-1.8B, Gemma-7B, ChatGLM3-6B-base, and Llama-8B, respectively.**
|  | alphabet option | short text | categorical label | math | Overall |
| --- | --- | --- | --- | --- | --- |
| OpenCompass | 0.8098 | / | / | 0.7481 | 0.7937 |
| LM Eval Harness | 0.8500 | 0.7540 | 0.8208 | 0.7496 | 0.8113 |
| UltraEval | 0.6057 | / | / | 0.4766 | 0.5722 |
| xFinder-qwen1505 | 0.9735 | 0.9683 | 0.9805 | 0.9276 | 0.9688 |
| xFinder-qwen1518 | 0.9735 | 0.9781 | 0.9755 | 0.9457 | 0.9712 |
| xFinder-gemma7 | 0.9751 | 0.9805 | 0.9736 | 0.9367 | 0.9704 |
| xFinder-chatglm36base | 0.9735 | 0.9793 | 0.9730 | 0.9291 | 0.9684 |
| xFinder-llama38 | 0.9698 | 0.9829 | 0.9692 | 0.9276 | 0.9661 |

It is evident that the xFinder models, fine-tuned using the KAF training set, are highly effective. Furthermore, despite a 16-fold difference in the parameter sizes of the models, the variation in extraction accuracy does not exceed 0.5%. This implies the high quality of the KAF dataset we constructed.

##### Results on KAF generalization set

To further explore xFinder’s extraction capabilities on not-seen tasks, we conducted tests using the KAF generalization set. Specific results are shown in Table 2. Only two xFinder models are displayed here for brevity, but the complete experimental results can be found in Table 19 of Appendix D.1. The results indicate that xFinder maintains a high level of extraction accuracy on the generalization set. In contrast, mainstream regular expression extractors perform poorly(^5^55Here, the extraction accuracy of OpenCompass, LM Eval Harness, and UltraEval has significantly decreased compared to Table 1. This is because the LLM responses in the KAF test set are generated from restricted prompts, making it easier for regular expressions to extract the key answer; however, the responses in the KAF generalization set are generated from non-restricted prompts.), especially in extracting key answers for mathematical problems. Even GPT-4 struggles to extract key answers from complex and varied LLM responses accurately.

**Table 2: Comparison of Extraction Accuracy on KAF Generalization Set: Mainstream Frameworks, GPT-4, and xFinder Models. Specifically, xFinder-qwen1505 and xFinder-llama38it represent models fine-tuned with the KAF dataset on Qwen1.5-0.5B and Llama3-8b-instruct, respectively.**
|  | alphabet option | short text | categorical label | math | Overall | $\Delta_{acc}$ | $\Delta_{acc}/N$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| OpenCompass | 0.7750 | / | / | 0.6813 | 0.7438 | / | / |
| LM Eval Harness | 0.6594 | 0.7484 | 0.8381 | 0.2094 | 0.6780 | / | / |
| UltraEval | 0.5945 | / | / | 0.1781 | 0.3978 | / | / |
| GPT-4 | 0.6578 | 0.8046 | 0.6706 | 0.6703 | 0.6957 | / | / |
| xFinder-qwen1505 | 0.9477 | 0.9335 | 0.9281 | 0.9234 | 0.9342 | 0.0277 | 0.0554 |
| xFinder-llama38it | 0.9547 | 0.9428 | 0.9537 | 0.9547 | 0.9518 | 0.0453 | 0.0057 |

### 5.2 Evaluation in Real-World Scenarios

The experiments conclusively demonstrate that xFinder plays a critical role within evaluation frameworks. Next, we will use xFinder for automated evaluations in real-world evaluation tasks instead of only inspecting the extraction accuracy. To balance evaluation efficiency and quality, we calculated two metrics for all xFinder models on the KAF Generalization Set: $\Delta_{acc}$ and $\Delta_{acc}/N$ (where $N$ is the parameter size of the LLM, measured in billions). Based on these metrics, xFinder-qwen1505 (with the highest $\Delta_{acc}/N$) and xFinder-llama38it (with the highest $\Delta_{acc}$) were selected. Here, $\Delta_{acc}$ represents the difference in extraction accuracy on the KAF Generalization Set between the current xFinder and the lowest extraction accuracy among the 19 xFinder models. For specific results, refer to Table 19 in Appendix D.1.

We evaluated the performance of 10 LLMs across 14 mainstream evaluation tasks using xFinder-qwen1505 and xFinder-llama38it, along with regular extraction methods based on OpenCompass, LM Eval Harness, and UltraEval. The comprehensive scoring results are provided in Appendix D.2.

##### Setup

Currently, the parameter settings for each evaluation framework differ significantly, including preprocessing methods for task datasets, prompt template configurations, and hyperparameter settings for the models being tested. Therefore, to compare the performance of our xFinder extraction method with RegEx methods on these frameworks, we replicated only the RegEx methods on each framework and ensured consistency in other settings. This primarily included: (1) applying uniform preprocessing to the task datasets; (2) employing a consistent zero-shot approach for all evaluation tasks, with specific prompts detailed in Appendix F.1; and (3) selecting the following LLMs for comprehensive evaluation: Phi-2, Baichuan series, Gemma series, Llama series, InternLM series, and Qwen series. Additionally, to ensure the stability of experimental results and maintain consistency in parameter settings across different LLMs, we set the temperature to 0, top_p to 0.9, and top_k to 5.

Figure: Figure 4: Bump charts: Changes in LLM rank over different evaluation frameworks.

##### Results and Analysis

Figure 4 illustrates the ranking changes of LLMs across four tasks based on evaluation results extracted from different frameworks. Additional chart results can be found in Appendix E. From the analysis, we can derive the following insights:

RQ1: Are existing evaluation frameworks reliable? The graph reveals significant discrepancies in the rankings of LLMs obtained using different evaluation frameworks. For instance, in the CommonsenseQA (alphabet) task, InternLM2-1.8B-Chat ranks first in UltraEval but only fifth in OpenCompass. Such disparities in rankings indicate the unreliability of RegEx methods within current mainstream evaluation frameworks, potentially affecting model evaluation and ranking.

RQ2: Is xFinder reliable? Firstly, by comparing our presented checkpoints, xFinder-qwen1505 and xFinder-llama38it, we observe relatively consistent rankings of LLMs across different base models on the same task. This demonstrates the high stability of xFinder, even on CommonsenseQA datasets that have not been used for training. Furthermore, considering the extraction accuracy of key answers from our previous experiments, xFinder exhibits higher accuracy compared to the RegEx methods in other frameworks. This implies that our evaluation results are both stable and accurate. Therefore, it can be concluded that our model demonstrates higher reliability compared to other frameworks.

(additional) RQ3: Is setting evaluation tasks as alphabet options reliable? Existing evaluation frameworks often aim to convert all questions into multiple-choice format with alphabet options. However, as shown in Figure 4, there are significant differences in the rankings of the CommonsenseQA task when using the alphabet option format compared to the short text format (more experimental details can be found in Appendix E.1). This suggests that alphabet options may be unreliable, and we advocate for reducing the reliance on the alphabet option format in existing evaluation frameworks.

## 6 Limitations

Although xFinder has achieved encouraging results, we must acknowledge some limitations. Our smallest xFinder model, with only 500 million parameters, performs well as a key answer extraction component in evaluation tasks. Nevertheless, the deployment cost may still be high for some researchers. In the future, we promise to continuously update xFinder by exploring model compression techniques to support smaller versions, potentially enabling CPU deployment and thereby further enhancing xFinder’s flexibility.

## 7 Conclusion

In this paper, we analyzed the key factors in existing LLM evaluation frameworks that may lead to unreliable assessment results, highlighting the critical answer extraction module in LLM responses as a weak point. To address this issue, we introduce a key answer extraction model, xFinder, specifically for extracting key answers in LLM responses. Extensive experimental results demonstrate that xFinder, compared to existing RegEx-based key answer extraction methods, can extract key answers from LLM outputs more accurately and efficiently, thereby improving evaluation reliability. This study marks the first step in constructing a reliable evaluation framework. In the future, we will continue addressing other key issues in the reliable evaluation of LLMs, providing a foundation for accurately evaluating LLM performance.

## Appendix A Datasets and Models

### A.1 Dataset Details

Table 3 provides a summary of all the datasets utilized in our experiments. We undertook extensive preprocessing activities on the raw data from these datasets, ensuring that all samples were transformed into a consistent and unified format. From our original dataset, we sampled 2000 data points from the training set and 1000 data points from the test set (or dev set if a test set was unavailable). Subsequently, we categorized the questions into four types based on the key answer format in the dataset: alphabet, short text, categorical label, and math type.

**Table 3: Datasets Description. The datasets are sorted by their names in alphabetical order. Key Answer Type refers to the format of the key answer: A/T/C/M stands for alphabet / short text / categorical label / math type options respectively. Asterisk (*) denotes that the data within these datasets are utilized for fine-tuning our xFinder model.**
| Dataset | Key Answer Type | #Train | #Test | License |
| --- | --- | --- | --- | --- |
| AgNews^∗ | C | 2000 | 1000 | Unspecified |
| Amazon^∗ | C | 2000 | 1000 | Apache-2.0 |
| ARC-challenge^∗ | A/T | 2000 | 1000 | CC BY-SA 4.0 |
| ARC-easy^∗ | A/T | 1391 | 1000 | CC BY-SA 4.0 |
| BOOlQ | C | 2000 | 1000 | CC BY-SA 3.0 |
| ComonsenseQA | A/T | 2000 | 1000 | MIT License |
| DBPedia^∗ | C | 2000 | 1000 | CC-BY-SA-3.0 |
| GSM8K^∗ | M | 2000 | 1000 | MIT License |
| hellaswag | A | 2000 | 1000 | MIT License |
| MATH^∗ | M | 1974 | 986 | MIT License |
| MetaMathQA | M | 2000 | 1000 | Apache-2.0 |
| MMLU^∗ | A/T | 2000 | 638 | MIT License |
| MultiArith | M | 420 | 180 | Unspecified |
| OpenbookQA | A/T | 2000 | 500 | Apache-2.0 |
| QNLI | C | 2000 | 1000 | Unspecified |
| SIQA | A/T | 2000 | 1000 | Unspecified |
| Subj | C | 2000 | 1000 | Unspecified |
| TREC | C | 1000 | 500 | Unspecified |
| WiC | C | 2000 | 638 | CC BY-NC 4.0 |

### A.2 Large Language Models

We selected the models listed in Table 4 from a vast array of open-source LLMs for our experiments.

**Table 4: LLMs sorted by release date. All LLMs are chat or instruct models. Asterisk (*) denotes estimated value, NaN denotes no public data available, and 175B denotes 175 billion.**
| Model | #Para. | Type | Publisher | Date |
| --- | --- | --- | --- | --- |
| LLaMA2 | 7B | Chat | Meta | 2023.07 |
| Baichuan2 | 7B | Chat | Baichuan Inc. | 2023.09 |
| ChatGLM3 | 6B | Chat | Tsinghua | 2023.10 |
| GPT4-1106-preview | NaN | Chat | OpenAI | 2023.11 |
| Phi-2 | 2.7B | Base | Microsoft | 2023.12 |
| InternLM2 | 7B | Chat | ShLab | 2024.01 |
| InternLM2 | 1.8B | Chat | ShLab | 2024.01 |
| Qwen1.5 | 14B | Chat | Alibaba | 2024.02 |
| Qwen1.5 | 4B | Chat | Alibaba | 2024.02 |
| Qwen1.5 | 1.8B | Chat | Alibaba | 2024.02 |
| Qwen1.5 | 0.5B | Chat | Alibaba | 2024.02 |
| Gemma | 7B | Instruct | Google | 2024.02 |
| Gemma | 2B | Instruct | Google | 2024.02 |
| Qwen1.5-MoE | 14.3B | Chat | Alibaba | 2024.03 |
| LLaMA3 | 8B | Instruct | Meta | 2024.04 |

## Appendix B KAF Dataset

### B.1 Dataset Structure

The KAF dataset is designed to enhance the accuracy and reliability of key answer extraction in LLMs. It is divided into three segments: training set, test set, and generalization set, each serving distinct purposes in the development and evaluation of the xFinder model.

#### B.1.1 Training Set

The training set contains 26,900 samples. Detailed information about the models, key answer types, and prompt configurations used in the training set is presented in Tables  5,  6,  7, and  8, respectively.

**Table 5: Number of samples from each LLM in the training set. Asterisk (*) denotes our manually expanded enhanced data.**
| LLMs | Count |
| --- | --- |
| Baichuan2-7B-Chat | 3726 |
| Qwen1.5-4B-Chat | 3051 |
| Anonymity^∗ | 2972 |
| Qwen1.5-1.8B-Chat | 2724 |
| Gemma-7B-it | 2409 |
| InternLM2-7B-Chat | 2383 |
| Gemma-2B-it | 2214 |
| LLaMA2-7B-Chat | 2102 |
| Qwen1.5-0.5B-Chat | 1927 |
| ChatGLM3-6B | 1755 |
| PHI2 | 1637 |

**Table 6: Number of samples for each dataset in the training set. "enh" indicates the enhanced alphabet option questions.**
| Dataset | Count |
| --- | --- |
| AgNews | 1813 |
| Amazon | 2029 |
| ARC-c (alpha) | 2452 |
| ARC-c (enh) | 577 |
| ARC-c (text) | 2549 |
| ARC-e (alpha) | 2403 |
| ARC-e (enh) | 581 |
| ARC-e (text) | 2670 |
| DBPedia | 2303 |
| GSM8K | 3394 |
| MATH | 1540 |
| MMLU (alpha) | 2439 |
| MMLU (enh) | 574 |
| MMLU (text) | 1576 |

**Table 7: Number of samples for each key answer type in the training set.**
| Key Answer Type | Count |
| --- | --- |
| alphabet option | 9026 |
| short text | 6795 |
| categorical label | 6145 |
| math | 4934 |

**Table 8: Number of samples for each prompt configuration in the training set.**
| Prompt Configuration | Count |
| --- | --- |
| random-0-shot | 3341 |
| random-0-shot-cot | 5201 |
| random-0-shot-cot-restrict | 2976 |
| random-0-shot-restrict | 2350 |
| random-5-shot | 3267 |
| random-5-shot-cot | 5321 |
| random-5-shot-cot-restrict | 2390 |
| random-5-shot-restrict | 2054 |

#### B.1.2 Test Set

The test set comprises 4,961 samples. Detailed information about the models, key answer types, and prompt configurations used in the test set is presented in Tables  9,  10,  11, and  12, respectively.

**Table 9: Number of samples from each LLM in the test set.**
| LLMs | Count |
| --- | --- |
| Baichuan2-7B-Chat | 786 |
| Qwen1.5-4B-Chat | 669 |
| Qwen1.5-1.8B-Chat | 545 |
| InternLM2-7B-Chat | 541 |
| Gemma-7B-it | 495 |
| LLaMA2-7B-Chat | 436 |
| Phi-2 | 412 |
| Gemma-2B-it | 398 |
| ChatGLM3-6B | 344 |
| Qwen1.5-0.5B-Chat | 335 |

**Table 10: Number of samples from each dataset in the test set.**
| Dataset | Count |
| --- | --- |
| AgNews | 478 |
| Amazon | 600 |
| ARC-c (alpha) | 651 |
| ARC-c (text) | 345 |
| ARC-e (alpha) | 584 |
| ARC-e (text) | 387 |
| DBPedia | 512 |
| GSM8K | 663 |
| MMLU (alpha) | 652 |
| MMLU (text) | 89 |

**Table 11: Number of samples for each key answer type in the test set.**
| Key Answer Type | Count |
| --- | --- |
| alphabet option | 1887 |
| short text | 821 |
| categorical label | 1590 |
| math | 663 |

**Table 12: Number of samples for each prompt configuration in the test set.**
| Prompt Configuration | Count |
| --- | --- |
| random-0-shot | 1157 |
| random-0-shot-cot | 1248 |
| random-5-shot | 1270 |
| random-5-shot-cot | 1286 |

#### B.1.3 Generalization Set

The generalization set comprises 4,482 samples. Detailed information about the models, key answer types, and prompt configurations used in the generalization set is presented in Tables  13,  14,  15, and  16, respectively.

**Table 13: Number of samples from each LLM in the generalization set.**
| LLMs | Count |
| --- | --- |
| LLaMA2-7B-Chat | 562 |
| InternLM2-1.8B-Chat | 562 |
| Gemma-2B-it | 561 |
| Qwen1.5-14B-Chat | 560 |
| Llama3-8B-Instruct | 560 |
| Qwen1.5-MoE-A2.7B-Chat | 560 |
| ChatGLM3-6B | 559 |
| Phi-2 | 558 |

**Table 14: Number of samples from each dataset in the generalization set.**
| Dataset | Count |
| --- | --- |
| BoolQ | 320 |
| CommonsenseQA (alpha) | 320 |
| CommonsenseQA (text) | 322 |
| hellaswag (alpha) | 320 |
| MetaMathQA | 319 |
| MultiArith | 321 |
| OpenbookQA (alpha) | 320 |
| OpenbookQA (text) | 320 |
| QNLI | 320 |
| SIQA (alpha) | 320 |
| SIQA (text) | 320 |
| Subj | 320 |
| TREC | 320 |
| WiC | 320 |

**Table 15: Number of samples for each key answer type in the generalization set.**
| Key Answer Type | Count |
| --- | --- |
| alphabet option | 1280 |
| short text | 962 |
| categorical label | 1600 |
| math | 640 |

**Table 16: Number of samples for each prompt configuration in the generalization set.**
| Prompt Configuration | Count |
| --- | --- |
| random-0-shot | 1122 |
| random-0-shot-cot | 1120 |
| random-5-shot | 1119 |
| random-5-shot-cot | 1121 |

### B.2 Human Annotation Details

In our manual annotation project, all annotators are volunteers with a background in natural language processing, and each annotator holds at least a master’s degree. The project team consists of five annotators, with a gender ratio of three men to two women. Regarding their remuneration, they first receive the standard employee salary. We also pay them an additional 2 RMB per annotated data item, with the average time for annotating each task being about 20 seconds. Additionally, our engineering team participates in data review, primarily responsible for supervising and evaluating annotation quality. The entire annotation process lasted for 15 days.

## Appendix C Fine-Tuning Hyperparameters

When fine-tuning xFinder using the XTuner framework and the QLoRA method, all base models were trained with identical hyperparameters. The training was conducted on 8x A100 GPUs. We focused on the hyperparameters specified in Table 17, while all other hyperparameters were set to their default values.

**Table 17: xFinder fine-tuning hyperparameter setting.**
| Hyperparameter | Setting |
| --- | --- |
| Batch Size | 1 |
| Maximum Length | 2048 |
| Learning Rate | 2e-4 |
| Pack to Max Length | True |
| Optimizer Type | AdamW |
| Betas | (0.9, 0.999) |
| Weight Decay | 0 |
| Warmup Ratio | 0.03 |

## Appendix D Detailed Experimental Results

### D.1 Extraction Accuracy Results

Table 18 presents the extraction accuracy of all 19 LLMs fine-tuned on the KAF generalization set. It can be observed that LLMs with different parameter magnitudes achieve stable and high accuracy levels after training on the KAF training set.

**Table 18: Extraction accuracy of xFinder fine-tuned on all 19 LLMs on the KAF test set. The first column lists the models fine-tuned using the KAF training set, including Qwen1.5-0.5B, Qwen1.5-0.5B-Chat, Qwen1.5-1.8B, Qwen1.5-1.8B-Chat, InternLM2-1.8B, Gemma-2B, Gemma-2B-it, Qwen1.5-4B, Qwen1.5-4B-Chat, ChatGLM3-6B-base, ChatGLM3-6B, Qwen1.5-7B, Qwen1.5-7B-Chat, InternLM2-7B, Baichuan2-7B-Chat, Gemma-7B, Gemma-7B-it, Llama-3-8B, and Llama-3-8B-Instruct models.**
|  | alphabet option | short text | categorical label | math | Overall |
| --- | --- | --- | --- | --- | --- |
| xFinder-qwen1505 | 0.9735 | 0.9683 | 0.9805 | 0.9276 | 0.9688 |
| xFinder-qwen1505chat | 0.9634 | 0.9635 | 0.9811 | 0.9351 | 0.9653 |
| xFinder-qwen1518 | 0.9735 | 0.9781 | 0.9755 | 0.9457 | 0.9712 |
| xFinder-qwen1518chat | 0.9709 | 0.9708 | 0.9736 | 0.9382 | 0.9673 |
| xFinder-internlm218 | 0.9608 | 0.9695 | 0.9667 | 0.9201 | 0.9587 |
| xFinder-gemma2 | 0.9698 | 0.9744 | 0.9774 | 0.9276 | 0.9673 |
| xFinder-gemma2it | 0.9661 | 0.9756 | 0.9755 | 0.9246 | 0.9651 |
| xFinder-qwen154 | 0.9730 | 0.9708 | 0.9698 | 0.9291 | 0.9657 |
| xFinder-qwen154chat | 0.9671 | 0.9793 | 0.9686 | 0.9306 | 0.9647 |
| xFinder-chatglm36base | 0.9735 | 0.9793 | 0.9730 | 0.9291 | 0.9684 |
| xFinder-chatglm36 | 0.9698 | 0.9769 | 0.9742 | 0.9261 | 0.9665 |
| xFinder-qwen157 | 0.9703 | 0.9744 | 0.9774 | 0.9306 | 0.9680 |
| xFinder-qwen157chat | 0.9735 | 0.9842 | 0.9748 | 0.9321 | 0.9702 |
| xFinder-internlm27 | 0.9740 | 0.9756 | 0.9730 | 0.9336 | 0.9686 |
| xFinder-baichuan27chat | 0.9724 | 0.9756 | 0.9717 | 0.9276 | 0.9667 |
| xFinder-gemma7 | 0.9751 | 0.9805 | 0.9736 | 0.9367 | 0.9704 |
| xFinder-gemma7it | 0.9682 | 0.9708 | 0.9226 | 0.9367 | 0.9498 |
| xFinder-llama38 | 0.9698 | 0.9829 | 0.9692 | 0.9276 | 0.9661 |
| xFinder-llama38it | 0.9709 | 0.9683 | 0.9711 | 0.9382 | 0.9661 |

Table 19 displays the extraction accuracy, $\Delta_{acc}$, and $\Delta_{acc}/N$ of all 19 LLMs fine-tuned on the KAF generalization set.

**Table 19: Extraction accuracy of xFinder fine-tuned on all 19 LLMs on the KAF generalization set, along with $\Delta_{acc}$ and $\Delta_{acc}/N$ values. The first column lists the models fine-tuned using the KAF training set: Qwen1.5-0.5B, Qwen1.5-0.5B-Chat, Qwen1.5-1.8B, Qwen1.5-1.8B-Chat, InternLM2-1.8B, Gemma-2B, Gemma-2B-it, Qwen1.5-4B, Qwen1.5-4B-Chat, ChatGLM3-6B-base, ChatGLM3-6B, Qwen1.5-7B, Qwen1.5-7B-Chat, InternLM2-7B, Baichuan2-7B-Chat, Gemma-7B, Gemma-7B-it, Llama-3-8B, and Llama-3-8B-Instruct.**
|  | alphabet option | short text | categorical label | math | Overall | $\Delta_{acc}$ | $\Delta_{acc}/N$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| xFinder-qwen1505 | 0.9477 | 0.9335 | 0.9281 | 0.9234 | 0.9342 | 0.0277 | 0.0554 |
| xFinder-qwen1505chat | 0.9289 | 0.9283 | 0.9163 | 0.9375 | 0.9255 | 0.0190 | 0.0380 |
| xFinder-qwen1518 | 0.9508 | 0.9439 | 0.9406 | 0.9500 | 0.9456 | 0.0391 | 0.0217 |
| xFinder-qwen1518chat | 0.9508 | 0.9356 | 0.9181 | 0.9531 | 0.9362 | 0.0297 | 0.0165 |
| xFinder-internlm218 | 0.9500 | 0.9407 | 0.8400 | 0.9344 | 0.9065 | 0 | 0 |
| xFinder-gemma2 | 0.9484 | 0.9335 | 0.9337 | 0.9437 | 0.9393 | 0.0328 | 0.0164 |
| xFinder-gemma2it | 0.9344 | 0.9262 | 0.9263 | 0.9391 | 0.9304 | 0.0239 | 0.0120 |
| xFinder-qwen154 | 0.9484 | 0.9345 | 0.9387 | 0.9484 | 0.9420 | 0.0355 | 0.0088 |
| xFinder-qwen154chat | 0.9492 | 0.9407 | 0.9237 | 0.9531 | 0.9389 | 0.0324 | 0.0081 |
| xFinder-chatglm36base | 0.9484 | 0.9324 | 0.9481 | 0.9484 | 0.9449 | 0.0384 | 0.0064 |
| xFinder-chatglm36 | 0.9563 | 0.9376 | 0.9163 | 0.9484 | 0.9369 | 0.0304 | 0.0051 |
| xFinder-qwen157 | 0.9570 | 0.9345 | 0.9519 | 0.9391 | 0.9478 | 0.0413 | 0.0059 |
| xFinder-qwen157chat | 0.9563 | 0.9345 | 0.9319 | 0.9422 | 0.9409 | 0.0344 | 0.0049 |
| xFinder-internlm27 | 0.9563 | 0.9563 | 0.9281 | 0.9375 | 0.9436 | 0.0371 | 0.0053 |
| xFinder-baichuan27chat | 0.9477 | 0.9345 | 0.9431 | 0.9313 | 0.9409 | 0.0344 | 0.0049 |
| xFinder-gemma7 | 0.9477 | 0.9345 | 0.9456 | 0.9500 | 0.9444 | 0.0379 | 0.0054 |
| xFinder-gemma7it | 0.9398 | 0.9200 | 0.8938 | 0.9469 | 0.9201 | 0.0136 | 0.0019 |
| xFinder-llama38 | 0.9563 | 0.9376 | 0.9469 | 0.9516 | 0.9482 | 0.0417 | 0.0052 |
| xFinder-llama38it | 0.9547 | 0.9428 | 0.9537 | 0.9547 | 0.9518 | 0.0453 | 0.0057 |

### D.2 Evaluation in Real-World Scenarios Results

#### D.2.1 Alphabet & Short Text Option Tasks

**Table 20: Evaluation results of LLMs on the ARC-c (alphabet) task, presenting accuracy scores using different key answer extraction methods.**
| LLMs | OpenCompass | UltraEval | LM Eval Harness | xFinder-llama38it | xFinder-qwen1505 |
| --- | --- | --- | --- | --- | --- |
| Baichuan2-7B-Chat | 39.17% | 38.15% | 52.80% | 54.93% | 55.04% |
| ChatGLM3-6B | 56.56% | 44.46% | 56.26% | 57.27% | 57.17% |
| Gemm-2B-it | 43.23% | 42.22% | 43.23% | 44.56% | 43.44% |
| Gemma-7B-it | 50.76% | 45.88% | 35.10% | 71.01% | 70.80% |
| InternLM2-1.8B-Chat | 59.31% | 59.21% | 58.70% | 59.51% | 59.61% |
| InternLM2-7B-Chat | 76.70% | 77.72% | 77.72% | 77.92% | 77.72% |
| Llama3-8B-Instruct | 78.03% | 77.92% | 77.92% | 78.03% | 78.03% |
| Phi-2 | 66.12% | 65.82% | 70.91% | 70.91% | 71.01% |
| Qwen1.5-14B-Chat | 83.32% | 64.29% | 76.60% | 85.96% | 86.06% |
| Qwen1.5-MoE-A2.7B-Chat | 70.30% | 67.96% | 70.19% | 73.35% | 73.45% |

**Table 21: Evaluation results of LLMs on the ARC-c (text) task, presenting accuracy scores using different key answer extraction methods.**
| LLMs | OpenCompass | UltraEval | LM Eval Harness | xFinder-llama38it | xFinder-qwen1505 |
| --- | --- | --- | --- | --- | --- |
| Baichuan2-7B-Chat | 40.80% | 40.80% | 47.50% | 48.40% | 48.40% |
| ChatGLM3-6B | 26.70% | 26.70% | 37.30% | 40.70% | 39.70% |
| Gemm-2B-it | 8.00% | 8.00% | 25.10% | 31.70% | 29.60% |
| Gemma-7B-it | 31.80% | 31.80% | 58.10% | 65.20% | 63.90% |
| InternLM2-1.8B-Chat | 34.10% | 34.10% | 43.80% | 44.30% | 42.20% |
| InternLM2-7B-Chat | 57.20% | 57.20% | 70.40% | 71.00% | 70.00% |
| Llama3-8B-Instruct | 60.50% | 60.50% | 75.10% | 74.80% | 74.70% |
| Phi-2 | 41.30% | 41.30% | 48.40% | 49.80% | 49.50% |
| Qwen1.5-14B-Chat | 60.10% | 60.10% | 77.80% | 80.60% | 80.60% |
| Qwen1.5-MoE-A2.7B-Chat | 41.30% | 41.30% | 61.10% | 63.20% | 62.50% |

**Table 22: Evaluation results of LLMs on the ARC-e (alphabet) task, presenting accuracy scores using different key answer extraction methods.**
| LLMs | OpenCompass | UltraEval | LM Eval Harness | xFinder-llama38it | xFinder-qwen1505 |
| --- | --- | --- | --- | --- | --- |
| Baichuan2-7B-Chat | 44.30% | 42.70% | 71.20% | 74.10% | 73.60% |
| ChatGLM3-6B | 71.40% | 47.50% | 70.60% | 71.90% | 72.10% |
| Gemm-2B-it | 61.70% | 59.50% | 62.00% | 62.80% | 61.90% |
| Gemma-7B-it | 58.80% | 54.80% | 39.20% | 84.40% | 84.10% |
| InternLM2-1.8B-Chat | 74.90% | 75.60% | 76.00% | 76.40% | 76.10% |
| InternLM2-7B-Chat | 89.40% | 90.50% | 90.20% | 90.40% | 90.50% |
| Llama3-8B-Instruct | 92.70% | 92.70% | 92.70% | 92.70% | 92.70% |
| Phi-2 | 78.00% | 77.70% | 84.20% | 84.60% | 84.60% |
| Qwen1.5-14B-Chat | 93.40% | 76.70% | 88.20% | 95.60% | 95.60% |
| Qwen1.5-MoE-A2.7B-Chat | 83.40% | 81.20% | 84.20% | 86.10% | 85.80% |

**Table 23: Evaluation results of LLMs on the ARC-e (text) task, presenting accuracy scores using different key answer extraction methods.**
| LLMs | OpenCompass | UltraEval | LM Eval Harness | xFinder-llama38it | xFinder-qwen1505 |
| --- | --- | --- | --- | --- | --- |
| Baichuan2-7B-Chat | 61.40% | 61.40% | 65.20% | 67.00% | 67.20% |
| ChatGLM3-6B | 39.60% | 39.60% | 49.20% | 52.70% | 51.60% |
| Gemm-2B-it | 8.20% | 8.20% | 32.80% | 45.80% | 41.20% |
| Gemma-7B-it | 51.80% | 51.80% | 76.80% | 82.70% | 81.60% |
| InternLM2-1.8B-Chat | 53.30% | 53.30% | 60.70% | 60.50% | 58.70% |
| InternLM2-7B-Chat | 79.40% | 79.40% | 86.50% | 86.90% | 86.20% |
| Llama3-8B-Instruct | 80.40% | 80.40% | 89.90% | 89.70% | 89.40% |
| Phi-2 | 62.80% | 62.80% | 67.30% | 69.50% | 69.30% |
| Qwen1.5-14B-Chat | 74.10% | 74.10% | 88.50% | 91.90% | 91.80% |
| Qwen1.5-MoE-A2.7B-Chat | 56.70% | 56.70% | 75.80% | 78.60% | 77.60% |

**Table 24: Evaluation results of LLMs on the CommonsenseQA (alphabet) task, presenting accuracy scores using different key answer extraction methods.**
| LLMs | OpenCompass | UltraEval | LM Eval Harness | xFinder-llama38it | xFinder-qwen1505 |
| --- | --- | --- | --- | --- | --- |
| Baichuan2-7B-Chat | 36.00% | 27.50% | 58.60% | 60.00% | 59.90% |
| ChatGLM3-6B | 61.20% | 31.80% | 57.00% | 62.30% | 61.70% |
| Gemm-2B-it | 44.70% | 39.90% | 44.90% | 45.90% | 45.00% |
| Gemma-7B-it | 55.10% | 34.90% | 41.00% | 69.90% | 69.90% |
| InternLM2-1.8B-Chat | 71.50% | 71.40% | 73.70% | 73.10% | 72.90% |
| InternLM2-7B-Chat | 82.60% | 66.90% | 83.70% | 83.70% | 83.80% |
| Llama3-8B-Instruct | 71.60% | 58.30% | 71.70% | 71.30% | 71.30% |
| Phi-2 | 59.50% | 51.90% | 61.30% | 61.40% | 61.30% |
| Qwen1.5-14B-Chat | 83.40% | 60.20% | 74.80% | 83.60% | 83.40% |
| Qwen1.5-MoE-A2.7B-Chat | 73.00% | 57.80% | 75.70% | 77.70% | 77.50% |

**Table 25: Evaluation results of LLMs on the CommonsenseQA (text) task, presenting accuracy scores using different key answer extraction methods.**
| LLMs | OpenCompass | UltraEval | LM Eval Harness | xFinder-llama38it | xFinder-qwen1505 |
| --- | --- | --- | --- | --- | --- |
| Baichuan2-7B-Chat | 51.20% | 51.20% | 51.50% | 52.40% | 52.20% |
| ChatGLM3-6B | 31.80% | 31.80% | 41.90% | 46.60% | 46.60% |
| Gemm-2B-it | 5.70% | 5.70% | 31.40% | 39.10% | 38.10% |
| Gemma-7B-it | 53.20% | 53.20% | 63.80% | 68.00% | 67.60% |
| InternLM2-1.8B-Chat | 58.00% | 58.00% | 58.60% | 59.00% | 58.60% |
| InternLM2-7B-Chat | 74.80% | 74.80% | 75.80% | 78.50% | 78.00% |
| Llama3-8B-Instruct | 70.20% | 70.20% | 71.40% | 71.20% | 71.50% |
| Phi-2 | 41.80% | 41.80% | 42.40% | 42.60% | 42.60% |
| Qwen1.5-14B-Chat | 70.50% | 70.50% | 75.60% | 78.60% | 78.90% |
| Qwen1.5-MoE-A2.7B-Chat | 55.50% | 55.50% | 67.80% | 70.40% | 70.20% |

**Table 26: Evaluation results of LLMs on the MMLU (alphabet) task, presenting accuracy scores using different key answer extraction methods.**
| LLMs | OpenCompass | UltraEval | LM Eval Harness | xFinder-llama38it | xFinder-qwen1505 |
| --- | --- | --- | --- | --- | --- |
| Baichuan2-7B-Chat | 36.60% | 34.80% | 41.30% | 44.40% | 44.30% |
| ChatGLM3-6B | 39.10% | 31.20% | 39.20% | 39.40% | 39.60% |
| Gemm-2B-it | 37.40% | 36.50% | 37.50% | 38.10% | 37.30% |
| Gemma-7B-it | 40.80% | 35.40% | 24.90% | 50.00% | 49.70% |
| InternLM2-1.8B-Chat | 42.10% | 44.30% | 45.00% | 45.80% | 45.10% |
| InternLM2-7B-Chat | 54.10% | 54.80% | 55.00% | 55.70% | 55.00% |
| Llama3-8B-Instruct | 62.50% | 62.00% | 62.70% | 62.60% | 62.50% |
| Phi-2 | 46.10% | 45.20% | 48.30% | 48.80% | 48.60% |
| Qwen1.5-14B-Chat | 65.40% | 49.70% | 59.80% | 66.00% | 66.00% |
| Qwen1.5-MoE-A2.7B-Chat | 50.80% | 48.60% | 51.60% | 54.50% | 54.20% |

**Table 27: Evaluation results of LLMs on the MMLU (text) task, presenting accuracy scores using different key answer extraction methods.**
| LLMs | OpenCompass | UltraEval | LM Eval Harness | xFinder-llama38it | xFinder-qwen1505 |
| --- | --- | --- | --- | --- | --- |
| Baichuan2-7B-Chat | 26.00% | 26.00% | 37.20% | 35.70% | 35.90% |
| ChatGLM3-6B | 13.40% | 13.40% | 24.00% | 22.70% | 22.40% |
| Gemm-2B-it | 5.20% | 5.20% | 23.20% | 24.50% | 21.50% |
| Gemma-7B-it | 19.60% | 19.60% | 44.00% | 42.30% | 41.70% |
| InternLM2-1.8B-Chat | 22.30% | 22.30% | 33.30% | 29.50% | 29.80% |
| InternLM2-7B-Chat | 30.10% | 30.10% | 48.90% | 44.60% | 44.80% |
| Llama3-8B-Instruct | 36.80% | 36.80% | 55.20% | 49.60% | 49.80% |
| Phi-2 | 22.30% | 22.30% | 32.90% | 30.60% | 30.70% |
| Qwen1.5-14B-Chat | 31.60% | 31.60% | 53.10% | 52.60% | 51.90% |
| Qwen1.5-MoE-A2.7B-Chat | 25.60% | 25.60% | 43.60% | 41.80% | 41.30% |

**Table 28: Evaluation results of LLMs on the OpenbookQA (alphabet) task, presenting accuracy scores using different key answer extraction methods.**
| LLMs | OpenCompass | UltraEval | LM Eval Harness | xFinder-llama38it | xFinder-qwen1505 |
| --- | --- | --- | --- | --- | --- |
| Baichuan2-7B-Chat | 39.00% | 37.00% | 47.80% | 50.40% | 50.40% |
| ChatGLM3-6B | 52.80% | 41.80% | 52.00% | 53.20% | 53.20% |
| Gemm-2B-it | 51.80% | 51.00% | 51.80% | 52.60% | 51.80% |
| Gemma-7B-it | 57.40% | 28.60% | 49.40% | 65.20% | 65.00% |
| InternLM2-1.8B-Chat | 55.40% | 56.80% | 57.20% | 56.20% | 57.20% |
| InternLM2-7B-Chat | 77.60% | 78.20% | 78.00% | 78.40% | 78.60% |
| Llama3-8B-Instruct | 74.60% | 74.60% | 74.40% | 74.60% | 74.60% |
| Phi-2 | 65.40% | 64.60% | 68.00% | 68.00% | 68.00% |
| Qwen1.5-14B-Chat | 83.00% | 68.20% | 75.40% | 84.20% | 84.20% |
| Qwen1.5-MoE-A2.7B-Chat | 69.80% | 66.60% | 69.20% | 72.00% | 71.60% |

**Table 29: Evaluation results of LLMs on the OpenbookQA (text) task, presenting accuracy scores using different key answer extraction methods.**
| LLMs | OpenCompass | UltraEval | LM Eval Harness | xFinder-llama38it | xFinder-qwen1505 |
| --- | --- | --- | --- | --- | --- |
| Baichuan2-7B-Chat | 41.60% | 41.60% | 44.20% | 45.40% | 45.60% |
| ChatGLM3-6B | 22.60% | 22.60% | 34.20% | 36.20% | 36.20% |
| Gemm-2B-it | 5.20% | 5.20% | 24.00% | 30.80% | 30.60% |
| Gemma-7B-it | 42.60% | 42.60% | 58.60% | 63.40% | 63.00% |
| InternLM2-1.8B-Chat | 38.40% | 38.40% | 44.00% | 43.80% | 42.20% |
| InternLM2-7B-Chat | 61.40% | 61.40% | 70.20% | 70.00% | 70.00% |
| Llama3-8B-Instruct | 67.80% | 67.80% | 72.80% | 72.80% | 72.60% |
| Phi-2 | 34.60% | 34.60% | 36.60% | 37.40% | 37.40% |
| Qwen1.5-14B-Chat | 68.00% | 68.00% | 75.40% | 81.00% | 80.80% |
| Qwen1.5-MoE-A2.7B-Chat | 39.60% | 39.60% | 54.20% | 55.40% | 55.20% |

**Table 30: Evaluation results of LLMs on the SIQA (alphabet) task, presenting accuracy scores using different key answer extraction methods.**
| LLMs | OpenCompass | UltraEval | LM Eval Harness | xFinder-llama38it | xFinder-qwen1505 |
| --- | --- | --- | --- | --- | --- |
| Baichuan2-7B-Chat | 44.50% | 43.50% | 58.90% | 60.40% | 60.20% |
| ChatGLM3-6B | 66.10% | 60.30% | 66.00% | 66.50% | 66.50% |
| Gemm-2B-it | 53.00% | 52.70% | 53.00% | 53.40% | 53.00% |
| Gemma-7B-it | 63.00% | 15.10% | 55.40% | 66.00% | 66.30% |
| InternLM2-1.8B-Chat | 69.20% | 69.80% | 70.80% | 69.80% | 70.00% |
| InternLM2-7B-Chat | 75.90% | 77.30% | 77.70% | 77.10% | 77.20% |
| Llama3-8B-Instruct | 71.00% | 70.50% | 70.90% | 70.60% | 70.70% |
| Phi-2 | 63.60% | 63.60% | 64.00% | 64.00% | 64.00% |
| Qwen1.5-14B-Chat | 79.00% | 58.30% | 62.40% | 78.90% | 79.10% |
| Qwen1.5-MoE-A2.7B-Chat | 68.90% | 67.30% | 71.60% | 72.50% | 72.50% |

**Table 31: Evaluation results of LLMs on the SIQA (text) task, presenting accuracy scores using different key answer extraction methods.**
| LLMs | OpenCompass | UltraEval | LM Eval Harness | xFinder-llama38it | xFinder-qwen1505 |
| --- | --- | --- | --- | --- | --- |
| Baichuan2-7B-Chat | 50.00% | 50.00% | 52.90% | 53.70% | 53.50% |
| ChatGLM3-6B | 32.40% | 32.40% | 49.70% | 52.50% | 52.90% |
| Gemm-2B-it | 8.40% | 8.40% | 35.20% | 45.30% | 43.90% |
| Gemma-7B-it | 32.50% | 32.50% | 62.60% | 64.00% | 64.30% |
| InternLM2-1.8B-Chat | 53.10% | 53.10% | 60.20% | 59.60% | 59.80% |
| InternLM2-7B-Chat | 58.40% | 58.40% | 68.60% | 67.70% | 68.00% |
| Llama3-8B-Instruct | 61.30% | 61.30% | 64.90% | 64.60% | 64.70% |
| Phi-2 | 43.50% | 43.50% | 44.20% | 45.80% | 45.70% |
| Qwen1.5-14B-Chat | 62.50% | 62.50% | 68.80% | 72.90% | 72.70% |
| Qwen1.5-MoE-A2.7B-Chat | 48.40% | 48.40% | 62.70% | 66.10% | 65.40% |

**Table 32: Evaluation results of LLMs on the hellaswag (alphabet) task, presenting accuracy scores using different key answer extraction methods.**
| LLMs | OpenCompass | UltraEval | LM Eval Harness | xFinder-llama38it | xFinder-qwen1505 |
| --- | --- | --- | --- | --- | --- |
| Baichuan2-7B-Chat | 46.40% | 45.80% | 46.80% | 47.20% | 46.70% |
| ChatGLM3-6B | 44.50% | 43.60% | 43.90% | 44.20% | 43.80% |
| Gemm-2B-it | 29.80% | 27.80% | 29.80% | 29.80% | 29.70% |
| Gemma-7B-it | 50.20% | 21.30% | 42.30% | 51.20% | 51.10% |
| InternLM2-1.8B-Chat | 61.20% | 62.20% | 60.30% | 61.80% | 61.30% |
| InternLM2-7B-Chat | 73.40% | 72.70% | 73.40% | 73.40% | 73.20% |
| Llama3-8B-Instruct | 69.30% | 68.80% | 69.30% | 68.80% | 68.80% |
| Phi-2 | 34.40% | 34.20% | 34.40% | 34.30% | 34.20% |
| Qwen1.5-14B-Chat | 78.50% | 75.60% | 75.50% | 78.60% | 78.60% |
| Qwen1.5-MoE-A2.7B-Chat | 62.60% | 59.70% | 63.60% | 66.10% | 65.50% |

#### D.2.2 Categorical Label Option Tasks

**Table 33: Evaluation results of LLMs on the Amazon task, presenting accuracy scores using different key answer extraction methods.**
| LLMs | OpenCompass | UltraEval | LM Eval Harness | xFinder-llama38it | xFinder-qwen1505 |
| --- | --- | --- | --- | --- | --- |
| Baichuan2-7B-Chat | 83.40% | 83.40% | 83.60% | 83.70% | 83.60% |
| ChatGLM3-6B | 86.50% | 86.50% | 95.20% | 95.20% | 95.20% |
| Gemm-2B-it | 81.50% | 81.50% | 85.30% | 85.10% | 85.30% |
| Gemma-7B-it | 73.60% | 73.60% | 94.10% | 94.30% | 94.20% |
| InternLM2-1.8B-Chat | 96.40% | 96.40% | 96.60% | 96.50% | 95.50% |
| InternLM2-7B-Chat | 96.10% | 96.10% | 96.20% | 96.10% | 96.10% |
| Llama3-8B-Instruct | 93.40% | 93.40% | 93.40% | 93.40% | 93.40% |
| Phi-2 | 91.60% | 91.60% | 91.80% | 91.80% | 91.80% |
| Qwen1.5-14B-Chat | 95.50% | 95.50% | 95.80% | 95.50% | 95.70% |
| Qwen1.5-MoE-A2.7B-Chat | 90.50% | 90.50% | 93.70% | 92.50% | 92.60% |

**Table 34: Evaluation results of LLMs on the BoolQ task, presenting accuracy scores using different key answer extraction methods.**
| LLMs | OpenCompass | UltraEval | LM Eval Harness | xFinder-llama38it | xFinder-qwen1505 |
| --- | --- | --- | --- | --- | --- |
| Baichuan2-7B-Chat | 63.60% | 63.60% | 63.60% | 63.60% | 63.60% |
| ChatGLM3-6B | 20.60% | 20.60% | 60.50% | 60.20% | 61.00% |
| Gemm-2B-it | 24.80% | 24.80% | 63.50% | 63.20% | 63.80% |
| Gemma-7B-it | 68.60% | 68.60% | 69.40% | 69.70% | 69.70% |
| InternLM2-1.8B-Chat | 55.00% | 55.00% | 64.60% | 61.50% | 58.20% |
| InternLM2-7B-Chat | 76.70% | 76.70% | 76.70% | 76.50% | 76.50% |
| Llama3-8B-Instruct | 68.60% | 68.60% | 68.50% | 68.60% | 68.60% |
| Phi-2 | 64.10% | 64.10% | 64.10% | 64.10% | 64.10% |
| Qwen1.5-14B-Chat | 41.90% | 41.90% | 74.60% | 74.90% | 74.90% |
| Qwen1.5-MoE-A2.7B-Chat | 24.30% | 24.30% | 63.60% | 63.50% | 63.60% |

**Table 35: Evaluation results of LLMs on the QNLI task, presenting accuracy scores using different key answer extraction methods.**
| LLMs | OpenCompass | UltraEval | LM Eval Harness | xFinder-llama38it | xFinder-qwen1505 |
| --- | --- | --- | --- | --- | --- |
| Baichuan2-7B-Chat | 58.40% | 58.40% | 58.40% | 58.50% | 58.50% |
| ChatGLM3-6B | 42.60% | 42.60% | 63.50% | 64.90% | 65.50% |
| Gemm-2B-it | 54.30% | 54.30% | 55.40% | 55.40% | 55.20% |
| Gemma-7B-it | 53.30% | 53.30% | 56.80% | 56.80% | 57.70% |
| InternLM2-1.8B-Chat | 51.50% | 51.50% | 69.50% | 67.60% | 68.40% |
| InternLM2-7B-Chat | 60.80% | 60.80% | 61.60% | 60.80% | 67.80% |
| Llama3-8B-Instruct | 72.10% | 72.10% | 72.10% | 72.10% | 72.20% |
| Phi-2 | 48.40% | 48.40% | 48.40% | 48.40% | 48.40% |
| Qwen1.5-14B-Chat | 37.20% | 37.20% | 79.70% | 78.00% | 79.60% |
| Qwen1.5-MoE-A2.7B-Chat | 35.80% | 35.80% | 77.00% | 76.80% | 77.10% |

**Table 36: Evaluation results of LLMs on the WiC task, presenting accuracy scores using different key answer extraction methods.**
| LLMs | OpenCompass | UltraEval | LM Eval Harness | xFinder-llama38it | xFinder-qwen1505 |
| --- | --- | --- | --- | --- | --- |
| Baichuan2-7B-Chat | 53.92% | 53.92% | 53.92% | 53.92% | 53.92% |
| ChatGLM3-6B | 32.29% | 32.29% | 51.88% | 52.35% | 52.35% |
| Gemm-2B-it | 54.39% | 54.39% | 54.39% | 54.39% | 54.39% |
| Gemma-7B-it | 50.31% | 50.31% | 50.31% | 50.47% | 50.47% |
| InternLM2-1.8B-Chat | 54.08% | 54.08% | 54.08% | 53.76% | 54.08% |
| InternLM2-7B-Chat | 51.57% | 51.57% | 51.57% | 51.57% | 51.57% |
| Llama3-8B-Instruct | 56.90% | 56.90% | 56.90% | 56.90% | 56.90% |
| Phi-2 | 50.31% | 50.31% | 50.31% | 50.16% | 50.31% |
| Qwen1.5-14B-Chat | 47.65% | 47.65% | 64.42% | 64.42% | 64.42% |
| Qwen1.5-MoE-A2.7B-Chat | 45.45% | 45.45% | 57.99% | 57.84% | 57.84% |

#### D.2.3 Math Option Tasks

**Table 37: Evaluation results of LLMs on the GSM8K task, presenting accuracy scores using different key answer extraction methods.**
| LLMs | OpenCompass | UltraEval | LM Eval Harness | xFinder-llama38it | xFinder-qwen1505 |
| --- | --- | --- | --- | --- | --- |
| Baichuan2-7B-Chat | 6.60% | 5.50% | 5.50% | 6.70% | 6.80% |
| ChatGLM3-6B | 43.60% | 0.00% | 37.10% | 43.60% | 43.60% |
| Gemm-2B-it | 7.90% | 3.10% | 3.20% | 8.70% | 7.90% |
| Gemma-7B-it | 35.70% | 1.20% | 33.30% | 36.80% | 36.60% |
| InternLM2-1.8B-Chat | 14.00% | 2.40% | 23.90% | 31.80% | 26.20% |
| InternLM2-7B-Chat | 49.50% | 5.10% | 50.80% | 63.10% | 58.60% |
| Llama3-8B-Instruct | 77.00% | 2.20% | 80.30% | 80.30% | 80.20% |
| Phi-2 | 15.60% | 12.40% | 13.40% | 16.80% | 16.50% |
| Qwen1.5-14B-Chat | 74.40% | 0.40% | 76.00% | 77.00% | 76.70% |
| Qwen1.5-MoE-A2.7B-Chat | 46.80% | 3.80% | 42.10% | 48.00% | 48.20% |

**Table 38: Evaluation results of LLMs on the MATH task, presenting accuracy scores using different key answer extraction methods.**
| LLMs | OpenCompass | UltraEval | LM Eval Harness | xFinder-llama38it | xFinder-qwen1505 |
| --- | --- | --- | --- | --- | --- |
| Baichuan2-7B-Chat | 6.39% | 6.39% | 5.68% | 7.20% | 7.10% |
| ChatGLM3-6B | 23.02% | 8.22% | 4.56% | 23.02% | 22.82% |
| Gemm-2B-it | 6.39% | 4.56% | 5.07% | 8.82% | 8.82% |
| Gemma-7B-it | 13.18% | 1.72% | 11.46% | 14.40% | 14.10% |
| InternLM2-1.8B-Chat | 11.16% | 3.55% | 8.32% | 11.66% | 10.55% |
| InternLM2-7B-Chat | 19.57% | 4.67% | 16.23% | 21.20% | 20.39% |
| Llama3-8B-Instruct | 30.73% | 7.10% | 24.44% | 30.63% | 30.73% |
| Phi-2 | 9.03% | 8.82% | 6.90% | 10.24% | 9.33% |
| Qwen1.5-14B-Chat | 31.34% | 4.26% | 26.06% | 36.71% | 36.31% |
| Qwen1.5-MoE-A2.7B-Chat | 17.65% | 7.91% | 6.69% | 22.21% | 22.52% |

**Table 39: Evaluation results of LLMs on the MultiArith task, presenting accuracy scores using different key answer extraction methods.**
| LLMs | OpenCompass | UltraEval | LM Eval Harness | xFinder-llama38it | xFinder-qwen1505 |
| --- | --- | --- | --- | --- | --- |
| Baichuan2-7B-Chat | 5.00% | 3.33% | 3.33% | 5.00% | 5.00% |
| ChatGLM3-6B | 72.78% | 1.11% | 63.33% | 72.78% | 72.78% |
| Gemm-2B-it | 23.89% | 4.44% | 4.44% | 26.11% | 23.89% |
| Gemma-7B-it | 84.44% | 1.67% | 77.78% | 84.44% | 83.33% |
| InternLM2-1.8B-Chat | 30.56% | 16.67% | 36.67% | 59.44% | 45.56% |
| InternLM2-7B-Chat | 56.67% | 36.67% | 67.78% | 81.67% | 70.00% |
| Llama3-8B-Instruct | 97.78% | 1.67% | 97.78% | 97.78% | 97.78% |
| Phi-2 | 33.33% | 29.44% | 28.33% | 41.67% | 38.89% |
| Qwen1.5-14B-Chat | 98.33% | 0.56% | 98.33% | 98.33% | 98.33% |
| Qwen1.5-MoE-A2.7B-Chat | 62.78% | 7.78% | 52.78% | 63.33% | 63.33% |

## Appendix E Bump Charts of All Tasks

### E.1 Alphabet & Short Text Option Tasks

Figure: Figure 5: Bump Chart of ARC-c

Figure: Figure 6: Bump Chart of ARC-e

Figure: Figure 7: Bump Chart of CommonsenseQA

Figure: Figure 8: Bump Chart of MMLU

Figure: Figure 9: Bump Chart of OpenbookQA

Figure: Figure 10: Bump Chart of SIQA

Figure: Figure 11: Bump Chart of hellaswag (alphabet)

### E.2 Categorical Label Option Tasks

Figure: Figure 12: Bump Chart of Amazon

Figure: Figure 13: Bump Chart of BoolQ

Figure: Figure 14: Bump Chart of QNLI

Figure: Figure 15: Bump Chart of WiC

### E.3 Math Option Tasks

Figure: Figure 16: Bump Chart of GSM8K

Figure: Figure 17: Bump Chart of MATH

Figure: Figure 18: Bump Chart of MultiArith

## Appendix F Prompts

### F.1 Prompts for LLM Responses Generation & Real-World Scenarios Evaluation

In these figures, the red part represents the System prompt, the blue part represents the Demonstration, and the black part represents the Question.

Figure: Figure 19: 0-shot prompt for LLM response generation (using alpha option type question as an example).

Figure: Figure 20: 0-shot-restrict prompt for LLM response generation & Real-World scenarios evaluation (using alpha option type question as an example).

Figure: Figure 21: 0-shot-cot prompt for LLM response generation (using alpha option type question as an example).

Figure: Figure 22: 0-shot-cot-restrict prompt for LLM response generation (using alpha option type question as an example).

Figure: Figure 23: 5-shot prompt for LLM response generation (using alpha option type question as an example).

Figure: Figure 24: 5-shot-restrict prompt for LLM response generation (using alpha option type question as an example).

Figure: Figure 25: 5-shot-cot prompt for LLM response generation (using alpha option type question as an example).

Figure: Figure 26: 5-shot-cot-restrict prompt for LLM response generation (using alpha option type question as an example).

### F.2 Prompts for Auto Labeling Using GPT-4

As shown in Step 3 of Figure 3, we employed a self-consistency strategy for the automatic annotation using GPT-4. Specifically, we used two forms of prompts: a normal form and an XML form, as shown in Figure 27 and Figure 28, respectively. In these figures, the red part represents the System prompt, the orange part represents the Instruction prompt, and the black part represents the Question.

Figure: Figure 27: The prompt for automatic annotation by GPT-4: normal form.

Figure: Figure 28: The prompt for automatic annotation by GPT-4: XML form.

### F.3 Prompts for Extracting Key Answers Using xFinder

As shown in Figure 29, this is the carefully designed prompt template we used for extracting key answers with xFinder.

Figure: Figure 29: The prompt for extracting key answers by xFinder.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see above.
soundness: 3
strengths: 1 A large dataset that can be used for further LLM-based evaluation.
2 A new model that can be used for more reliable evaluation.
summary: This paper proposes a novel evaluator for answer extraction and matching in LLM evaluation. The main idea is to first construct a large-scale LLM response evaluation dataset, and then train (small) LLMs on it. This paper conducts an extensive evaluation of multiple tasks with comparison with multiple LLM-based evaluators.
weaknesses: To be perfectly honest, I am not an expert in LLM-based evaluation. But to me, the main contribution is the construction of a dataset that can help train LLM evaluators, with the help of other LLMs (e.g GPT-4). Thus the novelty of the proposed model is less convincing as it does not provide any new architecture. Training LLMs on evaluation data as evaluators have also been explored in previous research, such as [1] and its subsequent work. Could the authors explain more on its novelty? For example, in terms of training process, and model architectures, how does the xFinder differ from previous work that trains LLMs on evaluation data? Also, would it possible to prompt GPT-4 or other very big LLM using ICL with the constructed data, and how would it perform?

[1] Towards a Unified Multi-Dimensional Evaluator for Text Generation

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: see weaknesses
soundness: 3
strengths: The identified challenges in LLM response extraction and matching are realistic and merit attention.

The paper proposes a novel method to improve answer extraction modules, addressing limitations in existing approaches.

The paper is well-structured, with a clear progression from problem definition to methodology and experimental analysis.
summary: This paper introduces xFinder, a novel evaluator designed for answer extraction and matching in the context of LLM evaluation. The study identifies the limitations of current answer extraction modules, particularly those based on RegEx, in handling inconsistencies in model response formats. To address these issues, the authors propose xFinder to enhance extraction accuracy and evaluation reliability. The authors developed a dataset called the Key Answer Finder (KAF) dataset to train xFinder. Experimental results demonstrate that xFinder significantly outperforms existing frameworks and model-based evaluators in terms of extraction accuracy and evaluation efficiency.
weaknesses: Although the KAF dataset is used to validate xFinder’s performance, the paper lacks a comprehensive exploration of the model’s generalizability to entirely different datasets and error types.

The paper does not include sufficient experimental analysis on the impact of xFinder on final evaluation outcomes, such as a comparison between using xFinder and other extraction methods in terms of evaluation results.

There is a lack of detailed analysis of the KAF dataset’s quality, such as inter-annotator agreement metrics.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: What is the prompt you used for GPT-4 as Judge (CoT)?
soundness: 3
strengths: - The significance of accurate answer extraction in evaluations is often underestimated, yet it critically impacts results. This study rightly emphasizes this aspect.
- xFinder demonstrates strong performance in accuracy over conventional RegEx frameworks.
- Both the model and its dataset are immediately usable for enhancing the reliability of LLM assessments.
- The paper effectively outlines the problems in current evaluation methods and introduces a well-structured solution.
summary: The paper introduces xFinder, a tool designed to enhance the accuracy of evaluating large language models by improving key answer extraction and matching. It identifies flaws in current methods like test set leakage and RegEx limitations, proposes a new dataset (KAF) for training, and shows xFinder outperforms traditional and other automated judge models in extraction and judgment accuracy, thus contributing to more reliable LLM evaluations.
weaknesses: - The techniques may not be applicable to responses where the answer is not a short, extractable phrase.
- Although the results are promising, I suspect the technique might be replaced by stronger LLMs used as judges with improved prompting techniques in the near future, which could also generalize better for longer responses. The results in Table 3 are good, showing that even GPT-4 as a judge does not perform as well as xFinder. Therefore, I believe xFinder remains useful at this moment for tasks that have a similar distribution to its training data. It will also be interesting to discuss the combination of xFinder and other techniques.

---

confidence: 5
contribution: 4
initial_rating: 5
presentation: 3
questions: - **Why does the key extraction task need to have ``short text`` and ``alphabet option`` categories, as the former could be transformed into the latter?**

- **Does the order of the xticks in the bump charts affect the comparison between ``alphabet option`` and ``short text``?**

- **What is the trade-off between xFinder and simply adding one or more regular expression patterns in RegX?**: It is known to us that LLMs are good at learning patterns and following instructions, which could have further enhancements as the models' reasoning capabilities are further enhanced. xFinder indeed makes good improvements, but where do they come from? We must be sure that xFinder (or future work) helps answer extraction better than RegX, such as finding answers from those nasty answer patterns. After all, it is easy to write RegX patterns nowadays by simply prompting GPT-4 and LLM-based methods are still inefficient compared to lexical methods.
soundness: 4
strengths: The paper is solid enough on all claims with its comprehensive experiments. The proposed KAF dataset is suitable for future research on developing more reliable evaluation systems. The xFinder models are more efficient and reliable than current LLM-based methods. In all, the paper did a good engineering research on using LLMs to better find the answers from their own responses.
summary: The paper proposes a training dataset KAF for key answer extraction and the correspondingly trained models, xFinder. The motivation lies in improving the extraction of answers in LLM responses for more reliable evaluation. The authors create KAF based on various LLMs, different evaluation tasks, and widely used prompting techniques, i.e., CoT and few-shot prompting. Based on the comprehensive experiments in the paper, xFinder outperforms RegX and other LLM-based methods in answer extraction.
weaknesses: - **Missing Related Work**: The work [1] is also highly related to this work, especially for the Judgement Accuracy part. It holds a similar idea by comparing different evaluation methods, including LLM-based ones, in directly evaluating open-question answering.

- **Annotation Agreement**: Human rechecking is one of the significant parts of the data generation pipeline. What are the annotation agreements between annotators? 

- **Human/Case Study**: Except for those numbers in the experiment tables, it is important to do the case study on the output of xFinders. For example, **How and Why is xFinder better**, **Is it worth using xFinder other than RegX or other LLM-based methods?**, **Summarize the failure modes of those inferior methods and how xFinder could perform better in these cases.**, etc.

- **Writing**: I recommend adding more content about experimental settings, such as evaluation metrics, baseline models, etc, to the main text. There are too many staff in the Appendix. Things could be clearly explained in the main text for better reading.

(see the below ``Questions`` section for more)

[1] Wang C, Cheng S, Guo Q, et al. Evaluating open-qa evaluation[J]. Advances in Neural Information Processing Systems, 2024, 36.

Meta-review: The paper proposed to improve the robustness of answer extraction when evaluating large language models, focusing on the limitations of existing RegEx-based evaluation frameworks. It argues that these methods often lead to extraction errors and unreliable evaluations. The paper proposes xFinder, a novel evaluator designed to enhance the accuracy and reliability of answer extraction and matching. The authors also introduce the KAF dataset to support training and evaluation of the extraction task. Experiments show that xFinder has a significantly higher extraction accuracy compared to RegEx-based methods. It improves final judgment accuracy to 97.61%, outperforming GPT-4.

Strengths

The work studied an important issue in the evaluation of LLMs.
The experimental results demonstrate significant improvements in accuracy and efficiency over existing methods, particularly RegEx and LLMs like GPT-4.
The KAF dataset is a strong contribution, providing a benchmark for future research in automated evaluation.

Weaknesses
While the KAF dataset is valuable, more detailed exploration of its quality (e.g., inter-annotator agreement, error types) could strengthen the paper.
The method may struggle with long or complex responses, as noted by reviewers. This limitation is not explored in depth.
Some details, particularly experimental setups and case studies should be moved out of appendix.

Given its strengths in addressing an important problem, with its robust empirical results, I recommend acceptant of this paper.

Decision: Accept (Poster)