UltraEval: A Lightweight Platform for Flexible and Comprehensive Evaluation for LLMs (2024)

Abstract: Evaluation is pivotal for refining Large Language Models (LLMs), pinpointing their capabilities, and guiding enhancements. The rapid development of LLMs calls for a lightweight and easy-to-use framework for swift evaluation deployment. However, considering various implementation details, developing a comprehensive evaluation platform is never easy. Existing platforms are often complex and poorly modularized, hindering seamless incorporation into research workflows. This paper introduces UltraEval, a user-friendly evaluation framework characterized by its lightweight nature, comprehensiveness, modularity, and efficiency. We identify and reimplement three core components of model evaluation (models, data, and metrics). The resulting composability allows for the free combination of different models, tasks, prompts, benchmarks, and metrics within a unified evaluation workflow. Additionally, UltraEval supports diverse models owing to a unified HTTP service and provides sufficient inference acceleration. UltraEval is now available for researchers publicly 1.

heading: 1 Introduction
text: LLMs have been deployed in diverse domains, such as finance(Zhang and Yang, 2023), education(Kasneci et al., 2023), and law(Blair-Stanek et al., 2023), demonstrating their versatility and efficacy(Zhao et al., 2023). This advancement is significantly bridging the gap between the realization of the current state and Artificial General Intelligence (AGI)(Bubeck et al., 2023). Nevertheless, the expansion of model parameters and training datasets engenders increasing uncertainties and emergent capabilities, posing potential risks to humanity and
*Corresponding author:Xu Han (thu.hanxu13@gmail.com) and Zhiyuan Liu (liuzy@tsinghua.edu.cn)
†Work done during internship at ModelBest Inc. 1The website of UltraEval is at https://github.com/
OpenBMB/UltraEval and a demo video is at https://youtu. be/C0O6BVzNAS8.
challenges to stable training models (Chang et al., 2023; Bommasani et al., 2021; Wei et al., 2022a). Consequently, it is imperative to continuously and meticulously evaluate the evolving capabilities of LLMs throughout their development to ensure their responsible and beneficial applications.
Traditional benchmarks (Zellers et al., 2019; Suzgun et al., 2022; Austin et al., 2021; Clark et al., 2018) typically focus on evaluating model performance in a specific capability, making it challenging to assess the comprehensive abilities of a model. Additionally, these benchmarks generally do not include model deployment. Building pipelines from scratch for each combination of tasks and models for evaluation is highly inefficient and repetitive. Therefore, an integrated evaluation framework is crucial. Currently, some evaluation frameworks covering the entire pipeline from model deployment to model evaluation are proposed, and predominantly divided into two types: conversational websites, exemplified by platforms like Chatbot Arena 2, and open-source evaluation tools, such as lm-evaluation-harness 3. The former effectively assesses the conversational abilities of a model
2https://chat.lmsys.org/ 3https://github.com/EleutherAI/
lm-evaluation-harness
247
but lacks comprehensive task coverage and transparency in the evaluation process. The latter opensource frameworks face challenges such as incomplete task coverage, complex code structure, heavy implementations, high difficulty to use, and tightly coupled functionalities. These issues hinder both convenient and comprehensive assessments.
In this paper, we identify three core components that form the evaluation process: models (or systems), task data, and metrics (i.e., evaluation methods), as illustrated in Figure 1. Rethinking the implementations in these three aspects would benefit the construction of a lightweight and easy-to-use evaluation framework, which covers mainstream tasks and complete evaluation pipeline, and can be easily expanded according to user customization. To this end, we introduce UltraEval, a lightweight and user-friendly open-source framework for LLMs evaluation. It stands out for its modular and scalable design, enabling thorough assessment of model capabilities. As illustrated in Figure 2, we segment the evaluation pipeline into three main modules: Data, Model, and Metrics, each operating independently and interacting through data exchange.
Specifically, UltraEval is characterized by the following features:
1. Lightweight Usage Modes. UltraEval is designed with minimal dependency requirements and features straightforward design and installation, complemented by detailed documentation. Users can initiate automated evaluations with just a few simple commands.
2. Comprehensive Evaluation Tools. UltraEval offers an extensive benchmarks suite, com-
prising over 50 commonly used benchmarks, and provides a customized prompt for each task. During the evaluation process, we replicated commonly used metrics and incorporated post-processing methods for more accurate metric calculation. We replicate some benchmarks from the LLaMA2(Touvron et al., 2023), achieving consistent results, which demonstrates UltraEval’s reliability.
3. Modular Architecture and Interfaces. The three main modules are independent and have clear functions, enhancing the system stability of UltraEval. Moreover, its excellent scalability allows users to flexibly customize the evaluation workflow, such as by adding new models, tasks, metrics, and more.
4. Efficient Inference Engines. UltraEval deploys models as HTTP services, supporting the evaluation of LLMs from different sources, including the models deployed locally and the web-based API. When deployed locally, we also provide the interface to utilize vLLM 4(Kwon et al., 2023) and Gunicorn to enable multi-GPU acceleration.
Evaluation is currently in a phase of rapid and exploratory growth. UltraEval will be continuously updated and provide detailed tutorials to help researchers to efficiently deploy evaluation pipeline.

heading: 2 Related Work
text: The advancement of LLMs has led to the emergence of various evaluation frameworks, each with
4https://github.com/vllm-project/vllm
its unique features. This section will provide a detailed overview of the current state of evaluation frameworks (also see Table 1).
Chatbot Arena (Zheng et al., 2024) offers a LLM evaluation platform where users vote on model responses, using a crowdsourced, anonymous Elo-rating system. Although innovative, its reliance on human judgment limits its suitability for fast, routine assessments. AlpacaEval5(Li et al., 2023) and FastChat6(Zheng et al., 2023a) conduct evaluation by employing GPT-4(Achiam et al., 2023) for automated judging. Yet, in evaluating complex reasoning tasks, they tend to favor verbose responses and face issues with robustness. Additionally, the scope of their evaluation capabilities is limited.
HELM7(Liang et al., 2022) streamlines language model evaluation but is constrained by its support solely for AutoModelForCausalLM8, excluding models without namespaces or stored locally. It lacks support for user models, demonstrates potential module coupling issues and absence of acceleration options. FlagEval’s9 “capability-task-indicator” framework is original but criticized for its closed-source approach and overly simplistic benchmark choices, raising data security and assessment depth concerns. Despite their innovations, both platforms fall short of the adaptability and comprehensiveness seen in more versatile frameworks like UltraEval.
5https://github.com/tatsu-lab/alpaca_eval 6https://github.com/lm-sys/FastChat 7https://github.com/stanford-crfm/helm 8https://huggingface.co/docs/transformers/main 9https://flageval.baai.ac.cn
LLM harness (Gao et al., 2023), used by HuggingFace’s Open LLM Leaderboard, and OpenCompass10(Contributors, 2023) have emerged as comprehensive solutions, offering extensive dataset support and rapid updates. These feature-rich environments, however, entail a trade-off: their complexity and dependency on specific software can complicate usage and customization. This underscores the importance of detailed documentation for those looking to adapt or extend these frameworks. Similarly, InstructEval11 (Chia et al., 2023), leveraging the LLM harness infrastructure, caters specifically to models fine-tuned with instructions such as Alpaca and Flan-T5. Despite its targeted approach, InstructEval’s limitations in model and task coverage hint at its niche application rather than widespread utility. The adoption of such frameworks reflects the evolving landscape of model evaluation, where finding a balance between comprehensiveness and usability poses an ongoing challenge.
OpenAI Evals12 and GPT-Fathom13(Zheng et al., 2023b). OpenAI Evals offers a straightforward, open-source framework for appraising OpenAI models, while GPT-Fathom expands upon this by analyzing the progression from GPT-3 to GPT-4 using a wider dataset array. Although it provides valuable insights into LLM development, GPT-Fathom shares OpenAI Evals’ limitations in supporting a diverse range of models.
10https://github.com/open-compass/opencompass 11https://github.com/declare-lab/instruct-eval 12https://github.com/openai/evals 13https://github.com/GPT-Fathom/GPT-Fathom

heading: 3 UltraEval
text: As illustrated in Figure 1, evaluation is a comprehensive process that integrates models, data, and metrics. With this in mind, the design philosophy considers both the independence and interconnectivity of each module. As shown in Figure 2, UltraEval encompasses the entire evaluation lifecycle(Chang et al., 2023) and segments the evaluation workflow into three main modules. In this section, we delve into the design and implementation of each component within UltraEval in detail.

heading: 3.1 Data Preparation
text: Data preparation involves transforming raw data into the final input format for the model, encompassing data preprocessing and prompt templates.
Data Preprocessing. We collect commonly
used benchmarks for evaluating LLMs, such as MMLU(Hendrycks et al., 2020), GSM8K(Cobbe et al., 2021), and Hellaswag(Zellers et al., 2019), covering multiple dimensions of capabilities. Currently, we have 59 benchmarks, listed in Table 2, and we plan to continually expand our collection of benchmarks.
To ensure the accuracy of the data, we source it from reputable platforms like Hugging Face14 and GitHub, rather than relying on data modified by third parties. Given the varying data formats across benchmarks, we design a set of templates to standardize these diverse formats into JSON, serving as the starting point for evaluations. As shown in Figure 3, different data items are categorized under unified attributes.
Prompt Templates. Prompts are used to guide models to generate specific outputs, and UltraEval supports prompt engineering (White et al., 2023), including few-shot and Chain of Thought (CoT) (Wei et al., 2022b), to enhance the model’s accuracy. The sensitivity of LLMs to prompts (Zhu et al., 2023) and the variability of prompts across different tasks often make it challenging for researchers to replicate results from papers, hindering research progress. UltraEval addresses this issue by providing customized, stable prompt templates for each task to facilitate result alignment. Figure 4 showcases an example of a prompt template for MMLU, demonstrating the rigorous process for forming the final prompt input.
14https://huggingface.co/datasets

heading: 3.2 Model Delpoyment
text: UltraEval employs a unique design approach, deploying models as HTTP online services and leveraging vLLM and gunicorn technologies to enable multi-GPU acceleration.
Http Service. In traditional evaluations, model deployment is closely integrated with task assessment, requiring models to be redeployed for each new task, which can lead to unnecessary consumption of time and hardware resources. To address this, we deploy models as HTTP services with Flask, facilitating modularization and efficient resource use. This approach has several advantages:
1. Independence. We provide a unified interface through which models receive task data and hyperparameters, returning results upon completing inference. This setup, which allows for adjustments via hyperparameters, ensures model independence.
2. Comprehensiveness. In UltraEval, we enable direct model loading from the Huggingface Transformers library. Given the independent deployment, UltraEval theoretically supports all models, including those from personal experimentation under different frameworks, greatly enhancing research and development flexibility. We utilize vLLM15 as the foundational acceleration framework, granting loading priority to models it supports.
3. Scalability. Thanks to its excellent scalability, users can easily extend models from languagebased applications to multimodal models.
Multi-GPU Acceleration. We use the Gunicorn web server with Flask to deploy models via web endpoints, achieving a flexible and decoupled architecture for model deployment and evaluation. This setup allows for dynamic GPU acceleration, where
15https://github.com/vllm-project/vllm
the Gunicorn server, configured with environmentspecific parameters, manages multiple worker processes. Each process, handling a slice of the available GPUs, executes inference tasks in parallel, significantly improving computational efficiency and throughput. A highlight of UltraEval’s performance is its ability to utilize 4 A800 GPUs to evaluate a test set of 41k data points in under 1.5 hours, showcasing remarkable efficiency16.

heading: 3.3 Evaluation Methods
text: Refined data and models are instantiated through the Task and Model classes, respectively, initiating the model inference process. The model performs inference based on the input data and its hyperparameters, generating prediction outcomes. Typically, between the model’s output and the final score calculation, there are intermediate steps including post-processing and metric calculation.
Post-Processing. Model outputs, influenced by task characteristics, prompt templates, and the model’s performance, often contain extraneous information beyond the answers needed(Park et al., 2024). As shown in Figure 7, when ChatGPT responds to HumanEval questions, the response may include both code and additional textual descriptions, complicating automatic evaluation. To more accurately assess the model, it is necessary to postprocess the model’s outputs to extract the most crucial answers.
Post-processing is bifurcated into two dimensions: model and task. Variations in model training approaches result in different versions, such as chat and base, necessitating distinct processing methods. Additionally, certain tasks employ specific evaluation methodologies. Taking Figure 7 as an example, the initial step involves extracting the code segment from ChatGPT’s response. Subsequently, due to HumanEval(Chen et al., 2021)’s unique evaluation
16The model used in this experiment is Llama2-7B, and the benchmarks includes BBH, MMLU, C-Eval, CMMLU, HumanEval, MBPP, GSM8K and MATH. In total, they consist of 40,938 data points.
criteria, it is necessary to extract the function body from the code while omitting the function name, yielding a cleaner and more precise answer. UltraEval develops several post-processing methods tailored to the tasks and models currently available.
Metric Calculation. Evaluation methods are categorized based on their ability to be computed automatically into automatic evaluation and human evaluation(Chang et al., 2023). For automatic evaluation, we implement common metrics such as exact match for text generation tasks, F1 score for binary classification tasks, ROUGE(Lin, 2004) for translation tasks, and pass@k(Chen et al., 2021) for coding tasks. Specifically, for exact match, we develop extensions like in match and prefix match to more effectively capture a wide range of scenarios.
Given that UltraEval is an automated evaluation framework, for human evaluation, we integrate GPT-4(Achiam et al., 2023) as a discriminator to substitute for human evaluation. Moreover, all data results can be saved according to user preferences, allowing users to decide on human evaluation if desired, thus offering significant flexibility for a more objective assessment.

heading: 4 Evaluation
text: UltraEval aims to provide a lightweight, comprehensive, and user-friendly evaluation framework to support research. As illustrated in Figure 5, UltraEval’s modular design effectively combines various models, tasks and metrics for evaluation. Using UltraEval, we evaluate models from the LLaMA2 series(Touvron et al., 2023) and Mistral(Jiang et al., 2023) on these widely-used benchmarks. As indicated in the Table 3, some reproduced results are higher, while others are lower, but within a certain margin of error, our reproduced results are consistent with the results reported in the papers, underscoring our framework’s reliability. The sources of error include hyperparameters (e.g., temperature, top-p) and hardware configurations. Since the evaluation details are not provided in the papers, verification is not possible. This highlights the importance of having an open and reproducible evaluation framework.
Furthermore, UltraEval supports innovative research efforts, research on predictable scaling(Hu et al., 2023), OlympiadBench(He et al., 2024) and model training, such as with MiniCPM(Hu et al., 2024).

heading: 5 Discussion and Future Work
text: In this section, we discuss future work following this study. Specifically, we focus on addressing data contamination and supporting a broader range of evaluation scenarios.
Data contamination refers to the phenomenon that examples from the evaluation set are also found in the training data (Li et al., 2024), causing inaccuracies in model evaluation. Common methods for contamination detection include n-gram over-
lap and embedding similarity search. However, none of these methods are perfect, making research on contamination detection still crucial. We will integrate these methods.
Supporting more evaluation scenarios, such as multimodal, long-text, and Retrieval-Augmented Generation (RAG), is crucial for meeting a broader range of evaluation needs. This will be an important direction for our future work.

heading: 6 Conclusion
text: We introduce UltraEval, a lightweight, userfriendly, and comprehensive framework for model evaluation. UltraEval establishes a unified structure with well-defined modules and flexible interactions, aiding researchers and developers in efficiently deploying evaluation workflows. Moving forward, we plan to continuously integrate new technologies and features into UltraEval, extending beyond large language models to support the evaluation of multimodal models, Retrieval-Augmented Generation (RAG), agents, and more, to advance the research on AGI. Additionally, we aim to expand our collection of representative benchmarks and also develop our own, exploring the capabilities and limits of large models.

heading: Acknowledgements
text: This work is supported by the National Key R&D Program of China (No. 2022ZD0116312), Quan Cheng Laboratory (Grant No. QCLZD202301), the Postdoctoral Fellowship Program of CPSE (Grant No. GZB20230343), China Postdoctoral Science Foundation (Grant No. 2023M741945), and Institute Guo Qiang at Tsinghua University.

heading: Limitations
text: Currently, our approach primarily utilizes text domain evaluation. However, we are looking to expand the scope of UltraEval by integrating multimodal and long-context evaluation datasets. This enhancement aims to facilitate more thorough and diverse assessments. Additionally, there is room for improvement in the visualization of our results. Future improvement will focus on enabling multidimensional visualization, thereby enriching the interpretability and depth of our evaluation results.

heading: Ethical Considerations
text: In this paper, we present UltraEval, a lightweight, user-friendly, flexible, and comprehensive frame-
work for model evaluation. Adhering to the principles of modularity, UltraEval segments the evaluation process into three distinct modules: Data, Model, and Metrics. This approach enhances the framework’s extensibility and flexibility, allowing for the easy integration of new models and tasks. We offer an extensive benchmark suite and replicate commonly used models and benchmarks. Our results align with those reported in the corresponding papers, underscoring the stability and reliability of our framework. Committed to sustainable development, we publicly release all our code to minimize unnecessary carbon footprint. Throughout our experiments, we adhere to all licenses related to models and data.

heading: A More details

heading: A.1 Post-process
text: We present the post-processing code in Figure 6 and explain the reasons necessitating postprocessing in Figure 7.

heading: A.2 UltraEval Usage
text: In the tutorial 17, we provide a detailed guide on UltraEval, including an introduction to its modules and instructions on how users can customize evaluations, such as adding their own new tasks and new models. This ensures that diverse evaluation needs are met.

heading: A.3 Model as Judge
text: Currently, UltraEval supports ChatGPT as a substitute for human evaluation. However, since ChatGPT is a commercial model, users need to provide the relevant API key. Due to the modularity and flexibility of our framework, users can also use other closed-source models or highly performant open-source models as evaluators.

heading: A.4 Multi-GPU Acceleration
text: Multi-GPU acceleration requires sufficient GPU resources. For larger models, such as Llama270B, deploying a single instance requires two A100 GPUs. Therefore, when using Multi-GPU acceleration, it is essential to consider both the model and the available GPU resources to ensure optimal configuration.
17https://github.com/OpenBMB/UltraEval/blob/ main/docs/tutorials/en/ultraeval.md

heading: Future Work
text: Future improvement will focus on enabling multidimensional visualization, thereby enriching the interpretability and depth of our evaluation results