JudgeLM: Fine-tuned Large Language Models are Scalable Judges

## Abstract

Abstract Evaluating Large Language Models (LLMs) in open-ended scenarios is challenging because existing benchmarks and metrics can not measure them comprehensively.
To address this problem, we propose to fine-tune LLMs as scalable judges (JudgeLM) to evaluate LLMs efficiently and effectively in open-ended benchmarks.
We first propose a comprehensive, large-scale, high-quality dataset containing task seeds, LLMs-generated answers, and GPT-4-generated judgments for fine-tuning high-performance judges, as well as a new benchmark for evaluating the judges.
We train JudgeLM at different scales from 7B, 13B, to 33B parameters, and conduct a systematic analysis of its capabilities and behaviors.
We then analyze the key biases in fine-tuning LLM as a judge and consider them as position bias, knowledge bias, and format bias.
To address these issues, JudgeLM introduces a bag of techniques including swap augmentation, reference support, and reference drop, which clearly enhance the judge’s performance.
JudgeLM obtains the state-of-the-art judge performance on both the existing PandaLM benchmark and our proposed new benchmark.
Our JudgeLM is efficient and the JudgeLM-7B only needs 3 minutes to judge 5K samples with 8 A100 GPUs.
JudgeLM obtains high agreement with the teacher judge, achieving an agreement exceeding 90% that even surpasses human-to-human agreement 1 1 1 As a reference, the max agreement among humans in MT-bench (Zheng et al., 2023 ) is 82%. .
JudgeLM also demonstrates extended capabilities in being judges of the single answer, multimodal models, multiple answers, and multi-turn chat.

## 1 Introduction

Recent advancements in large language models (LLMs) have fostered significant interest due to their remarkable performance in following instructions and their broad capabilities in dealing with open-ended scenarios.
Based on the open-source LLMs, including OPT (Zhang et al., ), Flan-T5 (Chung et al., ), LLaMA (Touvron et al., ), and Pythia (Biderman et al., ), researchers propose numerous methods to align these models with human preferences through instruction fine-tuning. These aligned LLMs demonstrate enhanced abilities in comprehending human instructions and generating more coherent responses. Nonetheless, existing benchmarks (Hendrycks et al., ; Liang et al., ) and traditional metrics (Lin, ; Papineni et al., ; Zhang et al., ; Sellam et al., ; Yuan et al., ) do not adequately estimate the capabilities of LLMs in open-ended scenarios. Therefore, a new benchmark method that could evaluate LLMs comprehensively in open-ended tasks is needed.

Concurrent works are making efforts to explore various methods for evaluating the performance of LLM. The arena-format (Zheng et al., ) methods leverage crowdsourced platforms to extract anonymous LLM competition results. While evaluations by humans are trustworthy, they are also time-consuming and financially demanding. Some approaches (Chiang et al., ) utilize GPT-4 as a judge. Nevertheless, these methods grapple with challenges of potential data exposure and volatile API model transitions, potentially compromising the judge’s reproducibility. PandaLM (Wang et al., ) attempts to fine-tune open-source LLMs for evaluating answers. However, limitations stemming from the model’s size, training data quality, and inherent LLM biases, undermine the effectiveness of such fine-tuned models in the role of a judge.

In this paper, we propose to evaluate LLMs through fine-tuned open-source LLMs, which serve as scalable judges (JudgeLM) achieving satisfactory agreement with the teacher judge. Our methodology incorporates scalable judges as evaluators in open-ended tasks, coupled with a high-quality dataset conducive to both training and evaluating the judge models. Within our framework, we adapt open-source LLMs to serve as judges and analyze their scaling ability in relation to model size (ranging from 7B to 33B) and volume of training data (extending from 3.5K to 100K). Our curated dataset comprises 105K seed questions, LLM answer pairs, and judgments from the teacher judge, GPT-4, as shown in Fig. [1(a)](#S1.F1.sf1). Note that we generated two judgments for each seed task with and without reference answers. This dataset is partitioned, with 100K seed questions allocated for training ($\times$2 larger than PandaLM) and the remainder for validation ($\times$29 larger than PandaLM).

Figure: (a) Data generation pipeline of our JudgeLM. We first collect 105K seed tasks as questions. Then, we extract answers from 11 LLMs and randomly sample a pair of answers from the answer set. Last, we input the tasks, the sampled answer pairs, and optionally reference answers to GPT-4, which generates scores and detailed reasons as a judge teacher.
Refer to caption: /html/2310.17631/assets/x1.png

Utilizing LLMs as judges inevitably introduces biases such as position bias (favoring answers in specific positions), knowledge bias (over-reliance on pre-trained knowledge), and format bias (optimal performance only under specific prompt formats) as shown in Fig. [8](#A1.F8),  [10](#A1.F10),  [12](#A1.F12),  [13](#A1.F13).
We propose methods to address them. Moreover, our JudgeLM system presents extended capabilities as shown in Fig. [1(b)](#S1.F1.sf2), including grading single answers, judging multiple answers, judging multimodal models, and multi-turn chat.

In contrast to arena-format methodologies, our approach is rapid and has a low cost. For instance, a model like JudgeLM-7B requires only 8 A100 GPUs and can evaluate 5000 response pairs in just 3 minutes. In comparison to closed-source LLM judges, JudgeLM ensures reproducibility and protects user privacy. When compared to concurrent open-source LLM judges, our system explores both the scaling ability and biases in LLM fine-tuning. Furthermore, the dataset we introduce stands as the most diverse and high-quality one, significantly benefitting subsequent research in judge model investigations.

Our main contributions can be summarized as follows:

- •
We introduce a high-quality, large-scale dataset for judge models, enriched with diverse seed tasks, LLMs-generated answers, and detailed judgments from GPT-4, laying the foundation for future LLMs evaluating research.
- •
We propose JudgeLM, a scalable language model judge, designed for evaluating LLMs in open-ended scenarios. It achieves an agreement exceeding 90% that surpasses the human-to-human agreement. Our JudgeLM also has broad capacities to deal with extended tasks.
- •
We analyze the biases inherent to LLM judge fine-tuning and introduce a series of methods to address them. Our methods significantly improve the consistency of the model in different cases, making the JudgeLM more reliable and flexible.

## 2 Related Works

### 2.1 Instruction Fine-tuning of Large Language Models

With the development of large language models (LLMs), researchers find that fine-tuning pre-trained LLMs such as GPT-3 (Brown et al., ), T5 (Raffel et al., ), OPT (Zhang et al., ), and PaLM (Chowdhery et al., ) enables LLMs to follow human instructions and help with open-ended tasks. The instruction fine-tuned LLMs such as InstructGPT (Ouyang et al., ), ChatGPT (OpenAI, ), FLAN-T5 (Chung et al., ), FLAN-PaLM (Chung et al., ), OPT-IML (Iyer et al., ), and GPT-4 (OpenAI, ) exhibit stronger ability in zero-shot or few-shot tasks than their base models. After Meta released the powerful open-source LLM LLaMA (Touvron et al., ) and LLaMA2 (Touvron et al., ), lots of instruction fine-tuning works based on LLaMA or LLaMA2 were proposed in the natural language generation or multimodal generation domain, such as Alpaca, Vicuna (Chiang et al., ), OpenFlamingo (Awadalla et al., ), LLaMA-Adapter (Zhang et al., ), and Emu (Sun et al., ). Our JudgeLM also belongs to the LLaMA family and takes the Vicuna series as base models.
Our JudgeLM follows the instruction fine-tuning manner to create LLM judges and proposes to model the judgment-generation task as “grading, judging, and reasoning”. We further collect a high-quality, large-scale dataset for research in judging the performance of LLMs.

### 2.2 Evaluation of Large Language Models

As many open-source large language models (LLMs) and their fine-tuned variants are proposed and present remarkable performance on various tasks, evaluating the capabilities of LLMs becomes a popular and challenging task. To address this problem, Chatbot Arena (Zheng et al., ) aims to build a crowdsourced platform that ranks the LLMs through pairwise comparison and Elo rating. The crowdsourced way to evaluate LLMs has more reliable results but faces high costs and low efficiency.
Vicuna (Chiang et al., ) uses GPT-4 as a judge to select the better answer. Although the GPT-4-based method can judge LLMs like a human expert, the API-based methods have potential risks of data leakage and unstable performance. Zeno Build (Alex & Graham, ) proposes to evaluate LLMs at a customer service dataset, but using traditional metrics such as ChrF (Popović, ) and BERTScore (Zhang et al., ) can not fully evaluate the answers of LLMs in open-ended tasks. Besides, PandaLM (Wang et al., ) developed a judge model based on LLaMA (Touvron et al., ) to compare answers produced by LLMs. When serving as judges, PandaLM achieves an accuracy close to ChatGPT (OpenAI, ) but it only has a 7B model size that limits its performance further.
Our JudgeLM contains scalable judges from 7B-parameter to 33B-parameter and achieves state-of-the-art performance in both PandaLM and our benchmarks. Furthermore, researchers can use the proposed JudgeLM locally which ensures reproducibility and data security.

## 3 Dataset

High-quality, large-scale datasets are crucial for effectively fine-tuning large language models (LLMs) to act as evaluative judges. However, the concurrent datasets, such as the one by Wang et al. (), present limitations in terms of diversity and the granularity of judgment criteria. To address this, we introduce a novel dataset replete with a rich variety of seed tasks, comprehensive answers from modern LLMs, answers’ grades from the teacher judge, and detailed reasons for judgments. Section [3.1](#S3.SS1) elucidates the data generation process, while Section [3.2](#S3.SS2) delineates the methods adopted for training and evaluation using our dataset.

### 3.1 Data Generation

Figure: Figure 2: The input and output of the proposed JudgeLM data sample. By comparing the answers with ground truth, traditional metrics can not judge answers accurately. However, the LLM judges can understand the questions and answers and give accurate scores and reasons.
Refer to caption: /html/2310.17631/assets/x4.png

The primary objective of our data generation is to create a large-scale and diversified dataset that maximizes the evaluative capabilities of judge models.
We sample 105K instruction seed tasks from a large-scale set that contains Alpaca-GPT4 (Peng et al., ), Dolly-15K (Conover et al., ), GPT4All-LAION (Anand et al., ), and ShareGPT.
To enhance the heterogeneity of the dataset, answers are collated from 11 leading open-source LLMs including, but not limited to, LLaMA (Touvron et al., ), Alpaca, and Vicuna (Chiang et al., ). Following this, we amalgamate LLM-generated answers with the reference answer to create answer sets. Pairs are randomly selected from the sets, upon which, fine-grained scores and detailed reasons are assigned by the advanced teacher model, GPT-4. To ensure robust and comprehensive judgments, we utilize detailed templates as demonstrated in Fig. [3](#A1.F3). Additionally, to allow the model to judge with reference answers, the reference-inclusive template is employed as Fig. [4](#A1.F4). This encourages the model to integrate external knowledge during the evaluative process.

### 3.2 Training and Evaluating

To better utilize our dataset to train and evaluate the judge models, we partition it into a training split and a validation split. The training set contains 100K judge samples, while the validation set has 5K. We then introduce the way we use this dataset to train and evaluate, respectively.

Training.
The training process of JudgeLM adheres to the instruction fine-tuning paradigm. As illustrated in Fig. [2](#S3.F2), the model is fed a question alongside a pair of answers, and an optional reference answer, yielding outputs comprising scores and detailed reasons. It is imperative to note the significance of a detailed crafted prompt template to harness the full potential of JudgeLM’s instruction-following ability. Distinct input templates cater to scenarios with and without references, as depicted in Fig. [3](#A1.F3) and Fig. [4](#A1.F4) respectively.

To further analyze the scaling ability of our JudgeLM, we fine-tune JudgeLM with sizes of 7B, 13B, and 33B parameters. The specific hyperparameters are enumerated in Table [7](#A1.T7). As for the scaling analysis for dataset size, we also fine-tune JudgeLM on varying data scales including 3.5K, 10K, 30K, and 100K samples. JudgeLM demonstrates scaling ability both in terms of parameter size and data volume.

Evaluating.
For the judge’s result, we model it as “grading, judging, and reasoning”. The judge model first generates scores for answer pairs. Subsequently, we can get the judge result from three situations: “Answer 1 wins” if the answer 1’s score is higher than the answer 2’s, “Answer 2 wins” if the answer 2’s score is higher, or “Tie” if the scores of two answers are the same. Last, the model generates detailed reasons if needed. The advantage of this modeling is that the judge model just needs little time to grade and judge, and generates time-consuming reasoning optionally.

For the metrics, we employ the objective metrics and reliability metrics to evaluate the judge models comprehensively.
For the objective metrics, we compute the agreement, precision, recall, and F1-score between the model’s judge results and those of the teacher. This provides insights into the alignment of judge models with established benchmarks, such as GPT-4 or human experts.
As for reliability metrics, we first compare the results before and after swapping LLM answers. Then we calculate the consistency to measure the judge model’s reliability. Last, we further calculate the metrics like “bias toward 1st”, “bias toward 2nd”, and “delta bias” to get insights from specific position biases and their variance.

## 4 Inherent Biases

In this paper, we also study the inherent biases that influence the reliability of fine-tuned LLM judges through reliability metrics and visualizations.

Position Bias.
Position bias means that the LLM judges prefer answers in a certain position and it widely exists in natural language processing tasks (Ko et al., ; Wang et al., ) and decision-making of humans (Blunch, ; Raghubir & Valenzuela, ). The powerful LLMs, ChatGPT and GPT-4, also face this challenge when working as judges (Wang et al., ; Zheng et al., ). As the qualitative and quantitative results shown in Fig. [8](#A1.F8) and Table [5](#S6.T5), JudgeLM also faces the position bias and prefers the first answer when swapping the positions of answers.

Knowledge Bias.
Knowledge bias arises when the pre-trained data lacks the knowledge of some seed tasks or induces possibly undesirable knowledge (Ko et al., ) that could degenerate the generative capabilities of LLMs.
Fig. [10](#A1.F10) provides an example that LLM judges can not give correct judgments to open-ended tasks if they lack related truth.

Format Bias.
Researchers expect that the judge model can make judgments based on pre-trained knowledge when the reference is not available and can make judgments following the reference when it is available.
However, our experiments revealed that judge models have a specific preference for the fine-tuning format whether with references or not. We name the situation that a judge fine-tuned without reference but validated with reference as a mismatched format, and vice versa.
As shown in Fig. [12](#A1.F12), Fig. [13](#A1.F13), and Table [6](#S6.T6) the judge models perform badly in mismatched formats. We hypothesize that the reason for format bias is that judge models are overfitting with the fixed fine-tuned template format.

## 5 Mehtods

In evaluating LLM-generated answers for a seed question, the LLM judge aims to determine the superior answer from a pair of candidates. Motivated by recent methods (Touvron et al., ; Chiang et al., ; Ouyang et al., ), we present JudgeLM, a scalable judge model, and address inherent biases in such models. Our methodology is depicted in Fig. [1(b)](#S1.F1.sf2). The subsequent sections provide a detailed breakdown of our approach.

### 5.1 Swap Augmentation

MT-bench (Zheng et al., ) and PandaLM (Wang et al., ) alleviate the position bias by judging twice with original and reverse order. These methods regard the result as a tie if the judgments are not the same. This kind of method casting double time to evaluate, can be regarded as a compromise and does not fix the inherent position bias of LLMs.

Intuitively, swapping the positions at the fine-tuning stage could push the judge model to pay more attention to the contents of answers rather than positions. Leveraging our structured judge data, we can easily swap the positions of answers to generate a new input sample. Correspondingly, we also swap the scores and question indexes of the judgment from the teacher (i.e., GPT4) to get the new ground truth. As shown in Fig. [15](#A1.F15), the augmented judge sample keeps the same results but exchanges the positions of answers. Overall, it is simple but effective to augment the training data and address position bias. The JudgeLM-with-swap-augmentation can give good judgment to the same judge sample as shown in Fig. [9](#A1.F9).

### 5.2 Reference Support

Introducing external knowledge is an intuitive way to make up for the lack of related pre-trained knowledge. To do so, we propose the reference support method to teach the model to judge with the help of reference answers. We collect reference answers for all judge samples and re-generate reference-guided judgments by GPT-4. Please note that GPT-4 also gives different scores and judgments for most judge samples with or without references. This proves that the differences between pre-trained knowledge and reference answers greatly impact judgments. As shown in Fig. [11](#A1.F11), the JudgeLM with reference support can avoid factual errors and give reliable judgments. Furthermore, introducing reference support to LLM judges can simply insert judge preferences. JudgeLM with reference support training can flexibly set reference answers with different preferences for different scenarios and needs. As shown in Fig. [16](#A1.F16), changing reference answers does not need extra training and makes JudgeLM more flexible to different preferences.

### 5.3 Reference Drop

To address the format bias, we introduce a method, named reference drop, in which we randomly drop the training sample with reference and use the corresponding sample without reference. As shown in Fig. [14](#A1.F14), judge models with reference drop can alleviate the overfitting for fine-tuning formats and give fair judgments with or without reference. Furthermore, the reference drop method also makes the judge model easy to use and decreases the cost of fitting into different formats.

## 6 Experiments

We study the performance of JudgeLM as follows: Section [6.1](#S6.SS1) presents the main results of JudgeLM comparing with concurrent methods, Section [6.2](#S6.SS2) analyzes the scaling ability of JudgeLM from both model sizes and data scales, and Section [6.3](#S6.SS3) shows ablation studies of proposed methods in detail.

### 6.1 Main Results

Comparison on JudgeLM Benchmark.
We first evaluate the proposed JudgeLM on our $val$ set. As shown in Table. [1](#S6.T1), we give the quantitative results of GPT-3.5, PandaLM-7B, and our JudgeLM with three model sizes. Among them, GPT-3.5 is used in the form of APIs with the help of templates in Fig. [3](#A1.F3) and Fig. [4](#A1.F4). PandaLM-7B is deployed with the released checkpoint and template. These two methods could be regarded as zero-shot methods because they are not fine-tuned by the JudgeLM dataset.
Our JudgeLMs are fine-tuned with proposed methods, i.e., swap augmentation, reference support, and reference drop. So, they can handle the situations with or without references simultaneously.
It can be observed that our JudgeLM-7B outperforms PandaLM-7B in all metrics, and even surpasses GPT-3.5. Furthermore, the proposed JudgeLM-33B exhibits the most powerful judge ability.

**Table 1: Main results for our JudgeLM and concurrent methods on our $val$ set, which uses GPT-4 annotation results as ground truth.**
| Methods | Agreement ↑<br>(w/ GPT-4) | Precision ↑<br>(w/ GPT-4) | Recall ↑<br>(w/ GPT-4) | F1 ↑<br>(w/ GPT-4) | Consistency ↑<br>(w/ swap.) |
| --- | --- | --- | --- | --- | --- |
| Judge w/o reference. |  |  |  |  |  |
| GPT-3.5 | 73.83 | 70.70 | 52.80 | 52.85 | 68.89 |
| PandaLM-7B | 68.61 | 40.75 | 38.82 | 39.41 | 74.78 |
| JudgeLM-7B | 81.11 | 69.67 | 78.39 | 72.21 | 83.57 |
| JudgeLM-13B | 84.33 | 73.69 | 80.51 | 76.17 | 85.01 |
| JudgeLM-33B | 89.03 | 80.97 | 84.76 | 82.64 | 91.36 |
| Judge w/ reference. |  |  |  |  |  |
| GPT-3.5 | 71.46 | 56.86 | 51.12 | 51.14 | 62.94 |
| PandaLM-7B | 63.77 | 39.79 | 34.82 | 35.18 | 55.39 |
| JudgeLM-7B | 84.08 | 75.92 | 82.55 | 78.28 | 84.46 |
| JudgeLM-13B | 85.47 | 77.71 | 82.90 | 79.77 | 87.23 |
| JudgeLM-33B | 89.32 | 84.00 | 86.21 | 84.98 | 92.37 |

Comparison on PandaLM Benchmark.
We also zero-shot evaluate our JudgeLM on the PandaLM $test$ set. PandaLM uses the result of three human annotators’ votes as ground truth. It requires swapping the positions of two answers to perform inference twice and the conflicting evaluation results are modified to ‘Tie’. Following the manner of this dataset, we present the zero-shot results of JudgeLM in Table [2](#S6.T2). It can be observed that the JudgeLM-7B outperforms GPT-3.5 and PandaLM-7B. When compared with GPT-4, JudgeLM-7B has lower accuracy and higher Precision, Recall, and F1-score than GPT-4. Furthermore, JudgeLM-33B achieves higher results than GPT-4, which demonstrates that fine-tuned JudgeLM can outperform its teacher in this specific task.

**Table 2: JudgeLM zero-shot evaluation results on PandaLM $test$ set, which uses human annotation results as ground truth. “^∗” means the results are reported in PandaLM (Wang et al., )**
| Methods | Accuracy ↑ | Precision ↑ | Recall ↑ | F1 ↑ |
| --- | --- | --- | --- | --- |
| zero-shot methods. |  |  |  |  |
| GPT-3.5^∗ | 62.96 | 61.95 | 63.59 | 58.20 |
| GPT-4^∗ | 66.47 | 66.20 | 68.15 | 61.80 |
| Fine-tuned on PandaLM $train$ set. |  |  |  |  |
| PandaLM-7B^∗ | 59.26 | 57.28 | 59.23 | 54.56 |
| Ours (zero-shot). |  |  |  |  |
| JudgeLM-7B | 65.07 | 66.89 | 71.95 | 61.92 |
| JudgeLM-13B | 68.97 | 68.21 | 74.15 | 65.12 |
| JudgeLM-33B | 75.18 | 69.30 | 74.93 | 69.73 |

Efficiency comparison.
To further compare the efficiency between our JudgeLM and PandaLM, we conduct experiments on our $val$ set to display the time cost using the same machine with 8 NVIDIA-A100 (40G) GPUs. As shown in Table [3](#S6.T3), we display the methods and model sizes in the first and second columns. The third column shows the needed GPUs for each judge model. The models with 7B or 13B parameters run on 1 A100 GPU with 40G memory while the 33B-parameter needs 2 GPUs. The fourth column shows whether the methods can judge answers in parallel. PandaLM only runs on 1 GPU because it does not support parallel running but all our JudgeLM can run in parallel to fully use all the 8 GPUs. The fifth column indicates whether judge reasons are generated at runtime. Our JudgeLM can choose whether to generate reasons but PandaLM must generate reasons that make it too time-consuming. The sixth column presents the total time cost including the time of loading checkpoints, judging samples, and calculating metrics. It can be observed that our JudgeLM is much more flexible and efficient than PandaLM.

Compared to the API-based judge methods,
the proposed JudgeLM proves to be more cost-effective.
For instance, using GPT-4 to evaluate on JudgeLM $val$ set (without generating reasons) incurs a cost of approximately $60. In contrast, JudgeLM-7B accomplishes the same task at a cost of merely $0.5, representing just 0.8% of the expense associated with using GPT-4.

**Table 3: Efficiency comparison for our JudgeLM and PandaLM on our $val$ set. We use a machine with 8 Nvidia-A100 GPUs with 40G memory to evaluate their efficiency.**
| Methods | model size | GPUs per model | parallel judge? | generate reason? | total time |
| --- | --- | --- | --- | --- | --- |
| PandaLM | 7B | 1 | ✗ | ✓ | 6 hrs 40 mins |
| Ours. |  |  |  |  |  |
| JudgeLM | 7B | 1 | ✓ | ✓ | 50 mins |
| JudgeLM | 7B | 1 | ✓ | ✗ | 3 mins |
| JudgeLM | 13B | 1 | ✓ | ✗ | 5 mins |
| JudgeLM | 33B | 2 | ✓ | ✗ | 15 mins |

### 6.2 Scaling Analysis of JudgeLM

In this section, we analyze the scaling ability of the plain JudgeLM on our $val$ set without reference as illustrated in Table [4](#S6.T4). As we increase the model size and data scale, we can observe the metrics increase. It demonstrates that the proposed JudgeLM is scalable and can reach up to 90.06% agreement and 87.93% consistency with 33B-parameter and 100K fine-tuning data.

**Table 4: Performance analysis for the scaling JudgeLM on our $val$ set.**
| Judge<br>Size | Data<br>Scale | Agreement ↑<br>(w/ GPT-4) | Consistency ↑<br>(w/ swap.) | Bias ↓<br>toward 1st | Bias ↓<br>toward 2nd | Delta bias ↓ |
| --- | --- | --- | --- | --- | --- | --- |
| 7B | 3.5k | 75.87 | 73.45 | 19.83 | 6.72 | 13.11 |
| 7B | 10k | 78.89 | 78.25 | 17.3 | 4.45 | 12.85 |
| 7B | 30k | 81.43 | 80.89 | 14.54 | 4.57 | 9.97 |
| 7B | 100k | 83.71 | 82.62 | 12.31 | 5.07 | 7.24 |
| 13B | 3.5k | 80.61 | 78.91 | 14.68 | 6.41 | 8.27 |
| 13B | 10k | 83.19 | 81.9 | 13.42 | 4.68 | 8.74 |
| 13B | 30k | 84.39 | 82.99 | 11.96 | 5.05 | 6.91 |
| 13B | 100k | 85.87 | 83.01 | 11.53 | 5.46 | 6.07 |
| 33B | 3.5k | 85.38 | 85.16 | 9.34 | 5.5 | 3.84 |
| 33B | 10k | 87.49 | 86.4 | 8.32 | 5.28 | 3.04 |
| 33B | 30k | 88.84 | 87.34 | 7.57 | 5.09 | 2.48 |
| 33B | 100k | 90.06 | 87.93 | 6.85 | 5.22 | 1.63 |

### 6.3 Ablation Study

In this section, we present the ablation studies of the proposed methods. For all ablation studies, we use JudgeLM-7B as the base model and 3.5K data for fine-tuning. Based on this baseline, we analyze the improvements brought by swap augmentation, reference support, and reference drop.

Improvements of Swap augmentation.
As shown in Table [5](#S6.T5), swap augmentation can bring comprehensive improvements to the baseline model. It improves consistency by 5.44%, which demonstrates that swap augmentation can reduce the influence of position bias and push the judge to pay more attention to the contents of answers.

**Table 5: Ablation study for the swap augmentation on our $val$ set.**
| Methods | Agreement ↑<br>(w/ GPT-4) | Consistency ↑<br>(w/ swap.) | Bias ↓<br>toward 1st | Bias ↓<br>toward 2nd | Delta bias ↓ |
| --- | --- | --- | --- | --- | --- |
| baseline | 75.87 | 73.45 | 19.83 | 6.72 | 13.11 |
| + swap aug. | 76.51 | 78.89 | 15.34 | 5.77 | 9.57 |

Improvements of Reference support.
As shown in the rows with the matching format of Table [6](#S6.T6), JudgeLM fine-tuned with reference support exhibits superior performance on every metric. It demonstrates that the introduction of reference answers induces the judge to rely on external knowledge and addresses the limitation of pre-trained knowledge.

Improvements of Reference drop.
As shown in Table [6](#S6.T6), baselines can not reach satisfactory performance when facing mismatched formats. With the help of the reference drop, the JudgeLM can handle both the format with or without reference and achieve higher agreement and consistency. It demonstrates that reference drop can address the format bias and avoid the JudgeLM overfitting to a single format.

**Table 6: Ablation study for the reference support and reference drop on our $val$ set.**
| Methods | $ft$<br>w/ ref? | $val$<br>w/ ref? | Agreement ↑<br>(w/ GPT-4) | Consistency ↑<br>(w/ swap.) | Bias ↓<br>toward 1st | Bias ↓<br>toward 2nd | Delta<br>bias ↓ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| matching format. |  |  |  |  |  |  |  |
| baseline | ✗ | ✗ | 75.87 | 73.45 | 19.83 | 6.72 | 13.11 |
| baseline | ✓ | ✓ | 80.15 | 81.23 | 11.55 | 7.22 | 4.33 |
| mismatched format. |  |  |  |  |  |  |  |
| baseline | ✗ | ✓ | 73.09 | 67.75 | 29.44 | 2.81 | 26.63 |
| baseline | ✓ | ✗ | 75.69 | 73.40 | 20.89 | 5.71 | 15.18 |
| w/ ref. drop. |  |  |  |  |  |  |  |
| baseline | ref. drop | ✗ | 76.86 | 77.13 | 17.30 | 5.57 | 11.73 |
| baseline | ref. drop | ✓ | 80.35 | 81.24 | 11.48 | 7.28 | 4.20 |

### 6.4 Extensions of JudgeLM

Not only judging answer pairs, but our JudgeLM can also be applied to many other extended tasks, such as grading a single answer, judging and ranking multiple answers, evaluating multimodal models, and multi-turn chat about judgments.

Grading a single answer.
The Concurrent judge method (Wang et al., ) usually judges a pair of answers to decide which one is better or tie but they lack the ability to evaluate a single answer.
Thanks to our judging mode of scoring first and then calculating the judging results, our JudgeLM provides an alternative practice to grade a single answer by slightly modifying the template as shown in Fig. [5](#A1.F5).
By putting the reference answer in the first position and giving it a full grade as a prior, JudgeLM can give quantitative fine-grained evaluations as shown in Fig. [17](#A1.F17).

Judging multiple answers.
To get the optimal ranking for N answers from different LLMs, other judge models need to call the model $O(n^{2})$ times to get the full matrix, which is a much less efficient solution. We attempt to resolve this limitation by extending our JudgeLM to process multiple answers at the same time. We first need to modify the template as shown in Fig. [5](#A1.F5). As shown in Fig. [18](#A1.F18), JudgeLM can judge and rank the multiple answers within the context limit of LLM.

Judging multimodal models.
Traditional multimodal evaluation needs prediction match the ground truth exactly. For some open-ended questions, a human-like evaluator is needed to determine whether the prediction is close to the ground truth range. Our JudgeLM provides good practice for multimodal evaluation by a slightly modified template as shown in Fig. [7](#A1.F7). Thanks to its capacity to judge open-ended answers, our JudgeLM can also perform well in judging multimodal models, as shown in Fig. [20](#A1.F20). Furthermore, the proposed JudgeLM can be easily applied to the modern open-ended multimodal benchmarks replacing the closed-source LLM judges like GPT-4. As shown in Fig. [21](#A1.F21), Fig. [22](#A1.F22), and Fig. [23](#A1.F23), we utilize JudgeLM to evaluate Emu (Sun et al., ) on MM-Vet (Yu et al., ) to get specific scores and detailed reasons.

Multi-turn chat about judgments.
It is worth noting that fine-tuning with judge samples does not compromise the multi-turn chat ability extended from base models. As illustrated in Fig. [19](#A1.F19), our JudgeLM retains the capability to engage in meaningful dialogues with users, providing them with a richer context, detailed information, additional examples, and specific details.

## 7 Conclusion

In this paper, we propose JudgeLM as scalable judges for evaluating LLMs in open-ended tasks efficiently, achieving state-of-the-art judge performance on two benchmarks.
The three key biases in fine-tuning LLMs as judges are analyzed and addressed by the proposed methods. We hope our work can motivate more studies to explore the judge models for LLMs in open-ended tasks and build more powerful LLMs with guidance from judge models.

Limitations.
Although the proposed JudgeLM achieves encouraging performance and efficiency, the cost of the judge dataset limits further scaling up in the judge dataset. Currently, we spend about 4000 dollars to provide 100K high-quality GPT-4-generated judge data to the public.
We expect to further improve the performance of judge models with the help of synthetic judge data.

## Appendix A Appendix

### A.1 Fine-tuning Settings

We list the hyper-parameters we used.

**Table 7: JudgeLM fine-tuning setting.**
| config | JudgeLM / -7B / -13B / -33B |
| --- | --- |
| base model | Vicuna / -7B / -13B / -33B |
| model max length | 2048 |
| fine-tuning data source | JudgeLM-100K |
| learning rate | 2e-5 |
| learning rate schedule | cosine decay |
| optimizer | AdamW Kingma & Ba (); Loshchilov & Hutter () |
| optimizer hyper-parameters | $\beta_{1}$, $\beta_{2}$, $\epsilon$ = 0.9, 0.999, 1e-8 |
| weight decay | 0.0 |
| GPU nums | 8 / 8 / 16 |
| batch size | 128 |
| training epochs | 3 |
| warmup ratio | 0.003 |
| numerical precision | bf16, tf32 |
| ZeRO optimizer Ramesh et al. () | stage 3 |
| gradient checkpointing | True |

### A.2 Prompt templates

We list all the prompt templates we used.

Figure: Figure 3: The template for judging answers without the reference.
Refer to caption: /html/2310.17631/assets/x5.png

Figure: Figure 4: The template for judging answers with the reference.
Refer to caption: /html/2310.17631/assets/x6.png

Figure: Figure 5: The template for grading a single answer. We set the reference answer in the position of Answer 1. Then, we set the score of the reference answer to 10. Last, the JudgeLM outputs the score of the single answer with such a prior.
Refer to caption: /html/2310.17631/assets/x7.png

Figure: Figure 6: The template for judging multiple answers.
Refer to caption: /html/2310.17631/assets/x8.png

Figure: Figure 7: The template for multimodal judging.
Refer to caption: /html/2310.17631/assets/x9.png

### A.3 Case Studies

We list several case studies.

Figure: Figure 8: Bad judgment caused by position bias. The answer placed in the first position always gets a higher score. The judge models generate reasons as possible from the perspective of making the scores reasonable.
Refer to caption: /html/2310.17631/assets/x10.png

Figure: Figure 9: Good judgment generated by the judge fine-tuned with swap augmentation. The judge can give judgments based on the content of answers rather than a certain position. The reason is convincing and reasonable.
Refer to caption: /html/2310.17631/assets/x11.png

Figure: Figure 10: Bad judgment caused by knowledge bias. This seed task is out of the judge model’s pre-trained knowledge, so it can not be correctly judged by the judge model. The judge model gives contradictory reasons in the judgment.
Refer to caption: /html/2310.17631/assets/x12.png

Figure: Figure 11: Good judgment generated by the judge model fine-tuned with reference support. Even though the judge model itself lacks related information, it can also give a reasonable judgment with the reference answer.
Refer to caption: /html/2310.17631/assets/x13.png

Figure: Figure 12: Bad judgment caused by format bias. For judging without reference, the judge model trained without reference is matched, so it performs well. However, the judge model trained with reference is mismatched, so it performs badly.
Refer to caption: /html/2310.17631/assets/x14.png

Figure: Figure 13: Bad judgment caused by format bias. For judging with reference, the judge model trained with reference is matched, so it performs well. However, the judge model trained without reference is mismatched, so it performs badly.
Refer to caption: /html/2310.17631/assets/x15.png

Figure: Figure 14: Good judgment generated by the judge model with reference drop, which addresses the preference for specific fine-tuning formats and gives fair judgments with or without reference.
Refer to caption: /html/2310.17631/assets/x16.png

Figure: Figure 15: An illustration of swap augmentation. We use swap augmentation to exchange the positions of answers, and our GPT-4-generated judgments can be modified correspondingly easily due to their structure.
Refer to caption: /html/2310.17631/assets/x17.png

Figure: Figure 16: An illustration of changing the reference answer to control model preference. When we change to a different reference answer, the model turns to prefer another answer.
Refer to caption: /html/2310.17631/assets/x18.png

Figure: Figure 17: An illustration of grading a single answer.
Refer to caption: /html/2310.17631/assets/x19.png

Figure: Figure 18: An illustration of judging multiple answers.
Refer to caption: /html/2310.17631/assets/x20.png

Figure: Figure 19: An illustration of multi-turn chat. Users can get more details, advice, examples, etc. by chatting with JudgeLM.
Refer to caption: /html/2310.17631/assets/x21.png

Figure: Figure 20: An illustration of multimodal judging. Our JudgeLM has the capacity to judge the VQA task without images.
Refer to caption: /html/2310.17631/assets/x22.png

Figure: Figure 21: An illustration of multimodal high-score grading on MM-Vet benchmark. The proposed JudgeLM can replace GPT-4 to grade multimodal answers.
Refer to caption: /html/2310.17631/assets/x23.png

Figure: Figure 22: An illustration of multimodal mid-score grading on MM-Vet benchmark. The proposed JudgeLM can replace GPT-4 to grade multimodal answers.
Refer to caption: /html/2310.17631/assets/x24.png

Figure: Figure 23: An illustration of multimodal low-score grading on MM-Vet benchmark. The proposed JudgeLM can replace GPT-4 to grade multimodal answers.
Refer to caption: /html/2310.17631/assets/x25.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: - did you notice any favoring of GPT-4 answers by the JudgeLM? 
- What tasks are seen in PandaLM that arent seen in the JudgeLM dataset?
soundness: 4
strengths: - The problem setting of building cheaper, scalable LLM judges is an important problem now that LLM-driven evaluation is becoming standard, and the provided benchmark will be incredibly valuable to the community
- This paper is easy to follow and provides a comprehensive analysis comparing JudgeLM to existing LLM judges on both accuracy and efficiency
- Improvements over previous LLM judges is impressive and provides a promising alternative to expensive closed source model judges
summary: This paper targets evaluating and building LLM's specifically for judging answer correctness on open-ended tasks. To do this, they construct a dataset which consists of llm answers across a variety of tasks along with GPT-4 generated judgements used as a ground truth (ground truth references responses are sometimes supplied). This dataset is used to train smaller LLMs to provide judgements with comparable accuracy to SOTA LLMs. They also discuss the biases that result from the LLM judge finetuning (position bias, knowledge bias, format bias) and propose methods to address them.
weaknesses: - Knowledge bias is not a bias but rather a limitation of the LLM, and the proposed solution to this seems to be providing that knowledge via a reference (AKA making the out of distribution task in distribution). While I am not opposed to this solution, I would argue this is not a bias addressed but rather a universally known failure case of the model that the authors try to mitigate by training on more knowledge via reference examples - a universally known solution to address model knowledge gaps.
- While the validation set was manually checked and corrected by the authors, it does still rely on GPT generated outputs. This provides somewhat of an unfair evaluation as JudgeLM is trained on GPT generated judgements as well. Even with the human validation, there is a reasonable chance that if this dataset where annotated by a different LLM and produced different judgements, humans checking responses would also consider them reasonable. An unbiased way of annotating is for humans to provide judgements *without* knowing what the GPT judgement is. If the agreement between humans and the GPT judgements are similar, than I would consider this evaluation relatively fair across all judge models. 
- The authors did not provide clear evidence that this model is able to maintain good performance across tasks not in the training set. I suspect that the comparison to the PandaLM test set is showing this to some extent, but I did not see any prose on *how* these two datasets differ. What tasks are seen in PandaLM that arent seen in the JudgeLM dataset? If the authors can show that the task distribution is significantly different from the training set I would be satisfied

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Given that JudgeLM reportedly surpasses human-to-human agreement, how does it handle cases where human judgments may rely on subjective or context-dependent insights? Could the authors discuss potential scenarios where JudgeLM might still fall short of human evaluators in nuanced judgments, and possibly include or consider adding complex tasks to explore this?
soundness: 3
strengths: The paper demonstrates a novel approach to scaling LLM evaluation by fine-tuning models as judges, creating a comprehensive system that addresses biases inherent in model evaluations. This creative combination of fine-tuning techniques with practical augmentation methods (swap, reference support, reference drop) removes limitations from prior works that struggled with consistent, scalable evaluation in open-ended tasks. 

The quality of the work is solid, backed by a large-scale, carefully curated dataset, including GPT-4 judgments and human validation, which strengthens the empirical basis of the results. In terms of clarity, the paper effectively communicates its methodology and contributions. 

Given the increasing role of LLMs across various fields, there is pressing need for scalable, unbiased evaluation frameworks. JudgeLM’s seems a valuable tool in AI evaluation, with potential impact on future benchmarks and research in LLM development.
summary: The paper proposes a scalable method for evaluating LLMs in open-ended tasks. It leverages a large, diverse dataset with tasks, LLM-generated answers, and GPT-4 judgments to fine-tune models that assess other models effectively. To mitigate biases such as position, knowledge, and format biases, the authors introduce techniques like swap augmentation, reference support, and reference drop. JudgeLM achieves high agreement rates with GPT-4, surpassing human-level consistency, and can judge multiple formats efficiently.
weaknesses: The improvements seem The dataset, although extensive, primarily relies on GPT-4 for initial judgments, which may inadvertently transfer GPT-4’s specific limitations to JudgeLM. A more diverse range of teacher models such as Claude could minimize over-reliance on any single model’s limitations, making JudgeLM’s judgments more adaptable.

---

confidence: 4
contribution: 2
initial_rating: 8
presentation: 3
questions: - Can JudgeLM’s performance be reliably extended to evaluate answers in different languages or domain-specific contexts (e.g., math, coding, legal or medical)?
- How does the model handle cases where the reference answer may introduce bias rather than mitigate it?
soundness: 3
strengths: - JudgeLM’s ability to process up to 5K samples in 3 minutes on an 8-GPU system is impressive, and it supports multiple use cases including single-answer and multimodal evaluations.
- The swap augmentation and reference-based adjustments offer a nice way to mitigate biases that impact LLM judgments, contributing to more reliable scoring across scenarios.
- The JudgeLM dataset, with its human-verified judgments and optional reference answers, is a notable contribution.
summary: The paper introduces JudgeLM, a scalable framework for evaluating LLMs by fine-tuning models as judges, using a dataset of LLM-generated answers and GPT-4 judgments. This framework addresses evaluation biases such as position and format biases through techniques like swap augmentation and reference drop. The authors show that JudgeLM achieves high alignment with the GPT-4 judge and is more efficient than comparable methods.
weaknesses: - The model's robustness under varied task complexities or unseen domains is not extensively tested (e.g. math/coding/reasoning task). Additional benchmarks or diverse human annotations would reinforce its generalizability.
- The study acknowledges that scaling up the judge dataset is costly and currently relies on GPT-4 outputs. Exploring alternative sources or synthetic judgment data could be beneficial.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In Table 3, why is JudgeLM able to judge answers in parallel, while PandaLM cannot? The authors mention certain “engineering optimizations for parallel judging”; Please provide specific details about these engineering optimizations, such as the parallel processing techniques used or any architectural changes that enable parallel judging. This would help readers better understand the efficiency advantages of JudgeLM over PandaLM.
2. In both the Abstract and the Introduction, the authors emphasize that JudgeLM achieves an agreement exceeding 90%, surpassing *human-to-human agreement*. Could you clarify this comparison? My understanding is that this primarily reflects strong internal consistency within JudgeLM’s evaluations, yet the emphasis seems to be on JudgeLM’s superior performance beyond consistency alone. If so, I suggest to provide a direct comparison between JudgeLM-to-human agreement and GPT-4-to-human agreement. This would help clarify whether JudgeLM's performance truly surpasses human-level agreement or if it's primarily a measure of internal consistency. Or if not so, feel free to correct me :)
3. In the Limitations section, the authors only mention the need to further scale up the judge dataset, which I find somewhat trivial. I recommend considering discuss more improvements to the dataset, such as refining the data distribution to real user quires. This could enhance JudgeLM’s performance in areas where the current data may lack diversity or balance. Please refer to my comments in the weaknesses section for further insights.

I will increase the score if all these concerns are addressed.
soundness: 3
strengths: 1. **Strong Motivation**: Training a judge LLM for efficient, effective evaluation of LLMs in open-ended benchmarks is highly valuable, as it enhances privacy, consistency, and cost-effectiveness.
2. **Thorough Exploration of Key Research Questions**: The paper addresses significant questions around agreement, consistency, efficiency, scalability, and generalization for LLM-as-judge models.
3. **Solid Experiments**: A convincing ablation study supports the effectiveness of data augmentation strategies, and the “grading, judging, and reasoning” pattern retains significant agreement and consistency benefits while improving efficiency.
4. **Open-Sourced Code and Dataset**: The code and dataset are readily accessible and user-friendly, enabling further research and reproducibility.
summary: A high-quality judge LLM should ideally demonstrate the following features: strong agreement with ground truth (often human annotations, though GPT-4 judgments are acceptable in some cases), consistency to diverse input formats (resilience to various biases), inference efficiency, scalability with data and parameter size, and generalization across scenarios.   
JudgeLM, as presented by the authors, offers an effective approach to optimizing these aspects:

1. **Achieving High Agreement**: The authors collect a diverse, high-quality dataset at scale through a structured pipeline (though the pipeline itself is not entirely novel).
2. **Achieving High Consistency**: To mitigate three critical biases—position, knowledge, and format biases—the authors employ three straightforward yet effective data augmentation methods: swap augmentations, reference support, and reference drop. This augmentation not only enhances consistency but also boosts agreement.
3. **Enhancing Efficiency**: The authors adopt a “grading, judging, and reasoning” pattern, as opposed to an explanation-first (CoT) approach. This method achieves a balance, trading slight reductions in agreement and consistency for increased efficiency and flexibility.
4. **Scalability**: Experimental results demonstrate JudgeLM’s scalability, as tested across varying model and data sizes. The 33B JudgeLM model even exceeds its generalist teacher, GPT-4’s agreement on a human-labeled validation set.
5. **Generalization**: JudgeLM exhibits promising generalization across various judging tasks (e.g., math problems, code generation) and diverse benchmarks (e.g., human-annotated, multimodal, and retrieval-based benchmarks).
weaknesses: 1. **Assumption of GPT-4 Judgments as Ground Truth**: Although GPT-4 is a common choice for cost-effective labeling, it would be beneficial if the authors could further substantiate its reliability before using it as a standard.
2. **Suboptimal Training Query Distribution**: While the authors highlight the diversity and quality of the training data by statistics, further optimization of the ideal data distribution for training judge LLMs would add depth. What constitutes a more ideal queries distribution? For example, the distribution of user queries collected in Chatbot Arena reflects a large volume of real user queries, which is one of the reasons Chatbot Arena has become such a well-recognized benchmark. If we align the queries distribution in the training data more closely with the distribution of actual user queries, the resulting Judge LLM would evaluate samples more fairly,  drawing on the philosophy behind MixEval/MixEval-X paper.
3. **Potential Overclaiming**:
    - Bias Analysis: In Section 4, while the paper discusses “position bias” and “knowledge bias,” these concepts are well-established in prior literature. The novel contribution here lies in addressing “format bias,” so the phrase “shed light on three key biases” could overstate the novelty. It might be more precise to note that this work builds on previous discussions of position and knowledge biases while introducing and addressing format bias as a new focus.
    - Scalability Comparison: In Appendix 4, the authors mention JudgeLM’s scalability as a differentiator from PandaLM. While PandaLM also explores different model sizes (up to 70B), JudgeLM offers a more granular analysis of scalability. It would clarify the novelty here to acknowledge PandaLM’s scalability exploration but emphasize JudgeLM’s finer scalability insights.
4. **Paper Structure**: Despite the solid research content, the paper’s organization could benefit from a clearer structure, such as arranging sections by key research questions.

Meta-review: This paper introduces resources to enable training of JudgeLM, a language model trained specifically to behave as an automatic evaluator. A large-scale dataset to train judge models is proposed, covering diverse seed tasks. The data also includes generated answers and detailed responses generated by GPT-4. A training recipe is also contributed. Since multiple answers are obtained for each given query during data generation, training follows the instruction tuning framework. Authors evaluate trained models and observe a high agreement rate.

The paper is clear and easy to follow. The contributions are of strong relevance to the community. I recommend acceptance.

Decision: Accept (Spotlight)