xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation

Authors: Qingchen Yu, Zifan Zheng, Shichao Song, Zhiyu li, Feiyu Xiong, Bo Tang, Ding Chen
Primary area: datasets and benchmarks
Keywords: Large Language Models; Reliable Evaluation

Abstract:
The continuous advancement of large language models (LLMs) has brought increasing attention to the critical issue of developing fair and reliable methods for evaluating their performance. Particularly, the emergence of cheating phenomena, such as test set leakage and prompt format overfitting, poses significant challenges to the reliable evaluation of LLMs. As evaluation frameworks commonly use Regular Expression (RegEx) for answer extraction, models may adjust their responses to fit formats easily handled by RegEx. Nevertheless, the key answer extraction module based on RegEx frequently suffers from extraction errors. Furthermore, recent studies proposing fine-tuned LLMs as judge models for automated evaluation face challenges in terms of generalization ability and fairness. This paper comprehensively analyzes the entire LLM evaluation chain and demonstrates that optimizing the key answer extraction module improves extraction accuracy and enhances evaluation reliability. Our findings suggest that improving the key answer extraction module can lead to higher judgment accuracy and improved evaluation efficiency compared to the judge models. To address these issues, we propose xFinder, a novel evaluator for answer extraction and matching in LLM evaluation. As part of this process, we create a specialized dataset, the Key Answer Finder (KAF) dataset, to ensure effective model training and evaluation. Generalization tests and real-world evaluations show that the smallest xFinder model, with only 500 million parameters, achieves an average extraction accuracy of 93.42\%. In contrast, RegEx accuracy in the best evaluation framework is 74.38\%. The final judgment accuracy of xFinder reaches 97.61\%, outperforming existing evaluation frameworks and judge models.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730164615783

thread_type:
official_review

<thread_object>
creator: reviewer_xqzp

<content>
summary:
The paper proposes a training dataset KAF for key answer extraction and the correspondingly trained models, xFinder. The motivation lies in improving the extraction of answers in LLM responses for more reliable evaluation. The authors create KAF based on various LLMs, different evaluation tasks, and widely used prompting techniques, i.e., CoT and few-shot prompting. Based on the comprehensive experiments in the paper, xFinder outperforms RegX and other LLM-based methods in answer extraction.

strengths:
The paper is solid enough on all claims with its comprehensive experiments. The proposed KAF dataset is suitable for future research on developing more reliable evaluation systems. The xFinder models are more efficient and reliable than current LLM-based methods. In all, the paper did a good engineering research on using LLMs to better find the answers from their own responses.

weaknesses:
- **Missing Related Work**: The work [1] is also highly related to this work, especially for the Judgement Accuracy part. It holds a similar idea by comparing different evaluation methods, including LLM-based ones, in directly evaluating open-question answering.

- **Annotation Agreement**: Human rechecking is one of the significant parts of the data generation pipeline. What are the annotation agreements between annotators? 

- **Human/Case Study**: Except for those numbers in the experiment tables, it is important to do the case study on the output of xFinders. For example, **How and Why is xFinder better**, **Is it worth using xFinder other than RegX or other LLM-based methods?**, **Summarize the failure modes of those inferior methods and how xFinder could perform better in these cases.**, etc.

- **Writing**: I recommend adding more content about experimental settings, such as evaluation metrics, baseline models, etc, to the main text. There are too many staff in the Appendix. Things could be clearly explained in the main text for better reading.

(see the below ``Questions`` section for more)

[1] Wang C, Cheng S, Guo Q, et al. Evaluating open-qa evaluation[J]. Advances in Neural Information Processing Systems, 2024, 36.

questions:
- **Why does the key extraction task need to have ``short text`` and ``alphabet option`` categories, as the former could be transformed into the latter?**

- **Does the order of the xticks in the bump charts affect the comparison between ``alphabet option`` and ``short text``?**

- **What is the trade-off between xFinder and simply adding one or more regular expression patterns in RegX?**: It is known to us that LLMs are good at learning patterns and following instructions, which could have further enhancements as the models' reasoning capabilities are further enhanced. xFinder indeed makes good improvements, but where do they come from? We must be sure that xFinder (or future work) helps answer extraction better than RegX, such as finding answers from those nasty answer patterns. After all, it is easy to write RegX patterns nowadays by simply prompting GPT-4 and LLM-based methods are still inefficient compared to lexical methods.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713147833

thread_type:
official_review

<thread_object>
creator: reviewer_vwgu

<content>
summary:
The paper introduces xFinder, a tool designed to enhance the accuracy of evaluating large language models by improving key answer extraction and matching. It identifies flaws in current methods like test set leakage and RegEx limitations, proposes a new dataset (KAF) for training, and shows xFinder outperforms traditional and other automated judge models in extraction and judgment accuracy, thus contributing to more reliable LLM evaluations.

strengths:
- The significance of accurate answer extraction in evaluations is often underestimated, yet it critically impacts results. This study rightly emphasizes this aspect.
- xFinder demonstrates strong performance in accuracy over conventional RegEx frameworks.
- Both the model and its dataset are immediately usable for enhancing the reliability of LLM assessments.
- The paper effectively outlines the problems in current evaluation methods and introduces a well-structured solution.

weaknesses:
- The techniques may not be applicable to responses where the answer is not a short, extractable phrase.
- Although the results are promising, I suspect the technique might be replaced by stronger LLMs used as judges with improved prompting techniques in the near future, which could also generalize better for longer responses. The results in Table 3 are good, showing that even GPT-4 as a judge does not perform as well as xFinder. Therefore, I believe xFinder remains useful at this moment for tasks that have a similar distribution to its training data. It will also be interesting to discuss the combination of xFinder and other techniques.

questions:
What is the prompt you used for GPT-4 as Judge (CoT)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730728679250

thread_type:
official_review

<thread_object>
creator: reviewer_rbe3

<content>
summary:
This paper introduces xFinder, a novel evaluator designed for answer extraction and matching in the context of LLM evaluation. The study identifies the limitations of current answer extraction modules, particularly those based on RegEx, in handling inconsistencies in model response formats. To address these issues, the authors propose xFinder to enhance extraction accuracy and evaluation reliability. The authors developed a dataset called the Key Answer Finder (KAF) dataset to train xFinder. Experimental results demonstrate that xFinder significantly outperforms existing frameworks and model-based evaluators in terms of extraction accuracy and evaluation efficiency.

strengths:
The identified challenges in LLM response extraction and matching are realistic and merit attention.

The paper proposes a novel method to improve answer extraction modules, addressing limitations in existing approaches.

The paper is well-structured, with a clear progression from problem definition to methodology and experimental analysis.

weaknesses:
Although the KAF dataset is used to validate xFinder’s performance, the paper lacks a comprehensive exploration of the model’s generalizability to entirely different datasets and error types.

The paper does not include sufficient experimental analysis on the impact of xFinder on final evaluation outcomes, such as a comparison between using xFinder and other extraction methods in terms of evaluation results.

There is a lack of detailed analysis of the KAF dataset’s quality, such as inter-annotator agreement metrics.

questions:
see weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730752393749

thread_type:
official_review

<thread_object>
creator: reviewer_b8rq

<content>
summary:
This paper proposes a novel evaluator for answer extraction and matching in LLM evaluation. The main idea is to first construct a large-scale LLM response evaluation dataset, and then train (small) LLMs on it. This paper conducts an extensive evaluation of multiple tasks with comparison with multiple LLM-based evaluators.

strengths:
1 A large dataset that can be used for further LLM-based evaluation.
2 A new model that can be used for more reliable evaluation.

weaknesses:
To be perfectly honest, I am not an expert in LLM-based evaluation. But to me, the main contribution is the construction of a dataset that can help train LLM evaluators, with the help of other LLMs (e.g GPT-4). Thus the novelty of the proposed model is less convincing as it does not provide any new architecture. Training LLMs on evaluation data as evaluators have also been explored in previous research, such as [1] and its subsequent work. Could the authors explain more on its novelty? For example, in terms of training process, and model architectures, how does the xFinder differ from previous work that trains LLMs on evaluation data? Also, would it possible to prompt GPT-4 or other very big LLM using ICL with the constructed data, and how would it perform?

[1] Towards a Unified Multi-Dimensional Evaluator for Text Generation

questions:
Please see above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732087876490

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A Friendly Reminder to the Reviewers

comment:
Dear Reviewers,  

Thank you once again for your thoughtful comments and valuable feedback. To address your comments and suggestions, we have submitted our responses and revised manuscript, with the revised sections highlighted in blue font. Since the discussion phase is halfway through, we kindly request the reviewers to confirm receipt of our responses. We also welcome any further concerns or suggestions regarding our responses.  

Thank you for your time and consideration.  

Sincerely,  

The Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732893612113

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response to Area Chairs and Reviewers

comment:
Dear Area Chairs and Reviewers,

We deeply appreciate your thoughtful feedback and the time you’ve invested in reviewing our work. Your insights during both the review and discussion phases have been invaluable in enhancing the quality of our paper.

In response to the reviewers' concerns and suggestions, we have provided detailed explanations or clarifications in the discussion and submitted a revised manuscript. All updates are highlighted in blue font for easy reference, and the main revisions include the following four improvements:

1. **Additional analysis of related work** [Reviewer XqZp] (see Lines 163-165 of the manuscript).  
2. **More detailed manual annotation protocol** [Reviewer XqZp, Reviewer rbE3] (see Appendix B.2).  
3. **Improved descriptions of the experimental setup** [Reviewer XqZp] (see Lines 326-347 of the manuscript).  
4. **Prompt for GPT-4 as Judge (CoT)** [Reviewer vwGU] (see Appendix F.3).  

Additionally, we noticed that **the ICLR discussion period has been extended to December 3, 2024 (AoE).** Given this change in the review process and its potential impact on the evaluation standards, we would like to take this opportunity to further summarize the core contributions of our paper and reiterate its importance and potential value in the field of reliable LLM evaluation. Our research not only makes significant improvements to existing methods but also provides a unique perspective for the LLM evaluation domain:

- **Significance of the Work:** We analyzed the existing LLM evaluation pipeline and introduced the concept of **reliable evaluation**. By examining critical factors that compromise evaluation reliability, particularly in answer extraction and matching stages, we identified challenges that hinder accurate model evaluation and model improvement. Enhancing evaluation reliability is essential for advancing LLM research.

- **Contributions and Impact:** We constructed the high-quality KAF dataset for training and evaluating LLMs as automated evaluators, and developed xFinder to automate answer extraction and matching. Experiments show that xFinder outperforms existing methods like RegEx and LLM as a judge in judgement accuracy (e.g., xFinder-qwen1505 achieves 97.48% accuracy) and demonstrates stronger generalization across datasets. Additionally, xFinder offers superior efficiency and cost benefits (e.g., the smallest model with only 0.5B parameters), providing an effective, low-cost solution for automated evaluation.

- **Challenges and Our Solution:** Automated evaluation, exemplified by LLM as a judge, has become crucial in LLM evaluation [1, 2, 3]. However, current models combine multiple steps (e.g., answer extraction and matching) in a single process, leading to intermediate skipping issues and lower accuracy [4]. Moreover, these models struggle with generalization, fairness, and efficiency [5]. By separating the evaluation into two stages—answer extraction and matching—xFinder improves judgement accuracy, generalization, and efficiency, thus enhancing the overall reliability and practicality of automated evaluation.

- **Future Applications and Potential:** Beyond automated evaluation, the KAF dataset and xFinder have broader applications in other tasks and domains. For instance, the strategy of separating answer extraction and matching can be applied to automate evaluation in other open-ended question tasks. Additionally, the high-quality annotated KAF dataset can be used in structured text generation tasks. Our analysis of the unreliabilities in LLM evaluation also provides valuable insights for future research and the development of more reliable evaluation methods, contributing to more robust and scalable LLM evaluation frameworks.

Once again, we sincerely thank all reviewers for their detailed and thoughtful feedback on this paper. Most of the suggestions aim to clarify issues or explore further improvements, which we have carefully addressed in individual responses. Additionally, we hope that, with the extension of the discussion period, we can further engage with the reviewers to explore potential issues or optimization directions, and contribute to the ICLR community with a higher-quality paper. If you have any additional suggestions or require further clarification, please do not hesitate to let us know.

Sincerely,

The Authors

---
**References**

[1] Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. *NeurIPS 2023.*

[2] Wang, Y., et al. (2024). PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization. *ICLR 2024.*

[3] Gu, J., et al. (2024). A Survey on LLM-as-a-Judge. *arXiv preprint arXiv:2411.15594.*

[4] Yang, S., et al. (2024). Do Large Language Models Latently Perform Multi-Hop Reasoning? *ACL 2024.*

[5] Wang, Y., et al. (2024). Large Language Models are not Fair Evaluators. *ACL 2024.*
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposed to improve the robustness of answer extraction when evaluating large language models, focusing on the limitations of existing RegEx-based evaluation frameworks. It argues that these methods often lead to extraction errors and unreliable evaluations. The paper proposes xFinder, a novel evaluator designed to enhance the accuracy and reliability of answer extraction and matching. The authors also introduce the KAF dataset to support training and evaluation of the extraction task. Experiments show that xFinder has a significantly higher extraction accuracy compared to RegEx-based methods. It improves final judgment accuracy to 97.61%, outperforming GPT-4.

Strengths

The work studied an important issue in the evaluation of LLMs.
The experimental results demonstrate significant improvements in accuracy and efficiency over existing methods, particularly RegEx and LLMs like GPT-4.
The KAF dataset is a strong contribution, providing a benchmark for future research in automated evaluation.

Weaknesses
While the KAF dataset is valuable, more detailed exploration of its quality (e.g., inter-annotator agreement, error types) could strengthen the paper.
The method may struggle with long or complex responses, as noted by reviewers. This limitation is not explored in depth.
Some details, particularly experimental setups and case studies should be moved out of appendix.

Given its strengths in addressing an important problem, with its robust empirical results, I recommend acceptant of this paper.