Tuning LLM Judge Design Decisions for 1/1000 of the Cost
Authors: David Salinas
Omar Swelam
Frank Hutter
Venue: ICML 2025 poster (2025)
Keywords: LLM judge
LLM evaluation
hyperparameter optimization
multi-fidelity
multi-objective optimization
Primary area: deep_learning->large_language_models
TL;DR: We propose a cost-effective approach to tuning LLM judges hyperperameters and analyze their performance across various dimensions, including scaling and hyperparameters.

Abstract: Evaluating Large Language Models (LLMs) often requires costly human annotations. To address this, LLM-based judges have been proposed, which compare the outputs of two LLMs enabling the ranking of models without human intervention.  While several approaches have been proposed, many confounding factors are present between different papers. For instance the model, the prompt and other hyperparameters are typically changed at the same time making apple-to-apple comparisons challenging.
In this paper, we propose to systematically analyze and tune the hyperparameters of LLM judges. To alleviate the high cost of evaluating a judge, we propose to leverage multi-objective multi-fidelity which allows to find judges that trades accuracy for cost and also reduce significantly the cost of the search. Our method identifies judges that not only outperform existing benchmarks in accuracy and cost-efficiency but also utilize open-weight models, ensuring greater accessibility and reproducibility.

Tuning LLM Judge Design Decisions for 1/1000 of the Cost
David Salinas' Omar Swelam
Frank Hutter' !2
Abstract
et al., 2023), but also to select instruction-tuning recipes (Grattafiori et al., 2024; Lambert et al., 2024). While they can reduce cost significantly compared
Evaluating Large Language Models (LLMs) of- ten requires costly human annotations. To ad- dress this, LLM-based judges have been proposed, which compare the outputs of two LLMs enabling the ranking of models without human interven- tion. While several approaches have been pro- posed, many confounding factors are present be- tween different papers. For instance the model, the prompt and other hyperparameters are typ- ically changed at the same time making apple- to-apple comparisons challenging. In this pa- per, we propose to systematically analyze and tune the hyperparameters of LLM judges. To al- leviate the high cost of evaluating a judge, we propose to leverage multi-objective multi- fideliityy which allows find judges that trade accuracy for cost and also significantly reduce the cost of the search. Our method identifies judges that not only outperform existing benchmarks in ac- curacy and cost-efficiency but also utilize open- weight models, ensuring greater accessibility and reproducibility. The code to reproduce our ex- periments is available this repository https: / //gittttt tig., com/ geoal go/ judee/udgetunnnnn ng.
human evals, LLM judges have limitations they may rely super- ficial style aspects, such the length of a response (Dubois et al., 2024) the order of their input (Zheng et al., 2023).
While some of these issues can be easily fixed, several aspects limit the improvement of LLM judges. The first is that evaluating L.L.M judge configuration expensive. For instance, evaluating one model in Alpaca-Eval (Li et al., 2023) costs ~24 (Ni et al., 2024), and evaluating a judge multiplies this by the large number of model evaluations it has to do in order to compute the correlation with human ratings. Given this, many confounding factors are typically present between different judges approaches. For instance, between Alpaca-Eval (Li al., 2023) and Arena-Hard (Li et al., 2024), the judge model, the prompt, the score-type and the set of instructions were changed. This makes it hard to learn which contributions important.
In this work, we propose to analyze and tune systematically the hyperparameters of an LLM judge, including the LLM model used, the prompt, the inference parameters (such as the temperature) as well as the parsing mechanism used to extract the judge preference. We first analyze the impact of scaling the LLM judge model or the number of instructions the judge performance. This highlights that scaling is insufficient to reach good performance and also helps us to identify a cheaper way to evaluate judges than evaluating Spearman correlation with Chatbot Arena with grid of model annotations. We then show how to systematically tune the hyperparameters of a judge using a multi-objective (to account for accuracy and cost) and multi-fidelity ap- proach which saves tuning cost by stopping early poor con- figurations. Finally, we show that the configurations found outperform previous state-of-the-ar judges on range of real-world test datasets.
1. Introduction
Instruction tuned models are difficult evaluate as they provide free-form text given arbitrary instructions that may include summarization (Zhang et al., 2024), code writing (Ni et al., 2023) translation (Elshin et al., 2024). While humans can annotate the quality of the outputs of an LLM given instructions, this quickly becomes expensive and also delays the evaluation and development of instruction-tumm models (Li et al., 2023).
As an alternative, LLM judges have been proposed to pro- vide an indicative ranking for instruction-tuned models (Li
Our key contributions are the following:
We study scaling laws of judges showing how much model size and number of instructions alone impact key metrics used to evaluate judges
University of Freiburg FELLIS Institute Tübingen. Correspon- dence to: David Salinas <david.salinas, pro@ comm
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
We propose a way to tune judges hyperparameters at reasonable tuning cost

---

Tuning LLM Judges' Hyperparameters
We show the approach is able identify configura- tions that outperform previous approaches while rely- ing solely on open-weight models
them is their dependence on superficial stylistic aspects of answer, for instance, favoring longer answers (Dubois et al., 2024), the first answer when using judges that make pairwise comparison (Li et al., 2024), or their own outputs (Panickssery et al., 2024).
We analyze which prompt strategy and hyperparame- ters work best for judges highlighting important pat- terns that may be used by the community to build better judges
While some of those issues have been fixed by averaging the order of the answers (Li et al., 2024) or using a causal model to isolate the length effect from the judge preference (Dubois al., 2024), the challenge of favoring its own an- swer is more problematic. Such biases pose some challenge in aligning with human evaluations which requires more intricate design for the scoring methods (Liu et al., 2024). Indeed, many previous works used judges from close mod- els such as GPT-4 which may bias the leaderboard or render evaluations infeasible if the model becomes unavailable or its cost increases.
2. Related work
LLM judges. LLM judge has been emerging as way to alleviate large human annotation costs that are re- quired to evaluate instruction-tuned models. For instance, Llama3 used an earlier version of itself as a reject sampler to select best completions (Grattafiori et al., 2024) or more re- cently (Lambert et al., 2024) used an LLM judge to annotate preference data to perform instruction tuning. LLM judges have also been used for leaderboards such as Alpaca-Eval (Li al., 2023) Arena- (Li et al., 2024) which of- fer cheaper alternatives than human-annotated leaderboards such as ChatBot Arena (Chiang et al., 2024).
Prompt optimization. LLM judges tend to be very reliant the design on optimization of the prompts used. Such strong sensitivity makes the prompt design an important aspect addressed in previous works. Zhou al. (Zhou et al., 2024) introduce prompt optimization framework for ad- dressing such sensitivity even for semantically equivalent instructions. Other works (Shi et al., 2024) have explored the importance of prompt optimization at constrained bud- get which is essential for scaling. Another important factor is how prompt tuning, together with fine-tuning, can lead to boosted performance (Soylu et al., 2024).
Zero-shot and fine-tuned judges. Two main approaches have been proposed for LLM judges. The first one, referred to as zero-shot judges, prompts LLM to rate a pair of model completions (Li et al., 2023; 2024) or a single com- pletion possibly against a baseline (Zheng et al., 2023). The second one fine-tunes an existing model on a set of human- annotated preferences (Zhu et al., 2023; Wang et al., 2024). In contrast with the first approach, it requires fine-tuning model which may occur at extra cost, along with lower robustness under distribution shift (Huang et al., 2024).
Judge tuning. To the best of our knowledge, hyperparam- eter tuning for LLM judges has not been comprehensively explored in a way that accounts for the various factors con- tributing to a high-performing judge. One reason is the as- sociated compute cost with naive approaches. For instance, Alpaca-Eve and Arena-Hard Hard evaluate each judge config- uration across a grid of models and instructions, leading to substantial expenses when comparing multiple judges! Cost is therefore strong limitation for the tuning of judges limiting the comparison and tuning of judges to simple as- pects such as the choice of the underlying LLM model but excluding other key factors such as the prompt strategy, the output format or the temperature.
Zero-shot judges. Many strategies for zero-shot judges have been proposed. Li et al. (2024) proposed ask the judge to answer the instruction to perform some form of Chain of Thought (Wei et al., 2022). To avoid the order of outputs to matter in pairwise comparison, previous work proposed to randomize or average the two possible positions (Dubois et al., 2024; Li et al., 2024). To parse the model output, Li et al. (2023) asks the judge letter to denote the best model, Li al. (2024) uses instead a Likert scale (such as AwB, A>B, A=B to indicate respectively when model A is much better, better or comparable wrt B), and another alternative (Cui et al., 2024; Lambert et al., 2024) outputs a score for various criteria such as instruction-followe following, honesty, or helpfulness. The underlying LLM models often vary between papers, along with multiple other dimensions, making it difficult to determine which strategies inherently perform better.
In this work, we propose a method to search for optimal judge configurations, including the best prompt parameteri- zation. While prompt tuning approaches such as (Fernando et al., 2023) are related, they do not specifically address tuning judges. A key distinction is that we focus not only on optimizing prompts but also on other judge hyperparam- eters, such as model selection and temperature. Similarly,
'We estimate that our approach costs approximately 2K$ to search through 4,480 judge configurations and that evaluating the same number of judges using Alpaca-Eval or Arena-Hard methodology would cost around 2MS, see B.3 for details.
Judge limitations. In parallel with their adoption, several limitations of L.LM judges have been highlighted. Among
2

---

Tuning LLM Judges' Hyperparameters
Doddapaneni al. (2024) analyzed the effectiveness of five different prompt strategies for LLM judges. However, their work primarily introduces a benchmark, whereas our ap- proach is centered on systematically tuning and analyzing judges within a search space that includes 80 prompting strategies and a total of 4,480 judge configurations.
it accounts for differences in scale between the golden scores and the judge annotations, focusing on the ranking consis- tency rather than absolute values. Other metrics which can be used to compare the order of models include Brier score and calibration described (Li et al., 2024).
Human agreement. Another approach to evaluating judges is to measure their agreement with aunan-annotated preferences in list of pairwise model battles. Each bat- tle consists of prompt, a pair of model outputs, and binary label indicating which output the human annotator preferred.
We begin by providing background on LLM judges and ex- amining the extent to which improvements can be achieved through scaling alone. This analysis underscores the im- portance of human agreement differentiating between judges. We then demonstrate how judges can be effectively tuned by optimizing their prompts and other hyperparameters, such as model selection and temperature.
more efficient metric for
Let us denote a set of annotated battles:
(pi, os, o', ( 0i, o())
(I)
3. Background
where o,, of denotes the output of two models from prompt p; and (pi, o of) E {0, 0.5, 1} denotes a human choice for respectively oG, a tie or of.
3.1. Judge
We denote LLM model as a function P that produces an output string o given prompt A judge compares two models To and иı and outputs a score (f, To(p), m (p)) E [] which is close to zero if Tp(p) better than mı (p) or close to one otherwise.
To evaluate the judge quality, one can measure the human agreement which is the percentage of times where the human and judge agrees, c.g.
E (ps, of) = o O, df)
A common choice is to use a fixed baseline n² for one of the models, typically frontier model which allows to obtain a score for model to be evaluated: (fp, x()) ((p, nt (p), π (p)). Given that the order can sometimes in- fluence the judge decision (Li et al., 2023), previous works have proposed to either randomize the baseline position (Li et al., 2023) or compute the judgement given both orders and average out the result (Li et al., 2024).
Having defined two metrics to evaluate judges, we next evaluate how those impacted by scaling the base models or number of instructions.
4. Scaling judges
Rather than tuning judges a natural question is: can we just scale them up? LLM judges can be scaled in multiple ways: by increasing the number of parameters of the LLM judge by using more instructions. Here we investigate how well the Spearman correlation and human agreement metrics scale with both dimensions.
3.2. Judge metrics.
Spearman correlation. One approach to see how well judge performs is check how close are its rankings on grid of models and instructions compared to rankings obtained through human annotations.
Spearman correlation. In Fig. 1, we show the scaling behavior when increasing the LLM judge size and the num- ber of instructions. We use Qwen2.5 with a default prompt and compute Spearman correlation common set of 26 models available on both Alpaca-Eval and Arena-Hard.
Assume we are considering a finite set of models M and in- structions P and that we are given a list of golden scores for the models denoted x E RRMMM for instance, the elo-ratings from the Chatbot Arena obtained from human ratings.
As expected, the judge performance improves when scal- ing the LLM model and when using more instructions as it allows to cover more areas of the models to evaluate. Alpaca-Eval contains easier prompts and therefore gives bet- ter performance to smaller judges compared to Arena-Hard In contrast, Arena-Hart contains harder and technical ques- tions. This allows to achieve better Spearman correlation as the instructions allow better separability among ad adannedd models, provided that the judge base model is strong enough to measure the performance on this more complex set of instructions.
To obtain scores from the judge, we average for each model the preference against baseline x*(p)) (p), e.g.
Ep~p [p x" (p), (p))].
One can then use the Spearman correlation between the scores estimated with the judge and the golden score, e.g.
p(s", s cov cov(R(C"),R(E_ (o)
[-1,1] where R(z) and (z) denotes respectively the rank operator and the standard deviation. Using Spearman correlation is beneficial because
3

---

Tuning LLM Judges' Hyperparameters
Instruction Alaac-Evval
Instruction Arena-Hard
Instruction both
1.0
Speamann correiation
0.8
0.58
0.6
1.58
38
0.4
0.2
326
728
0.0
length
-0.2
Instructions
Instructions
Instructions
Figure Effeec of scaling the LLM judge and increasing the number of instructions on Spearman correlation. In contrast human agreement, neither Aipaca-Eval, Arena-Hart nor their union distinguishes the quality difference between 32B and 72B models.
# params
Sp. corr. (t)
CV(J)
Hum. agr. (f)
CV(J)
6 10
0.5B
0.09 + 0.189
207.60
0.36 + 0.006
1.70
Human agreement
1.5B
0.33 + 0.137
41.26
0.37 = 0.006
1.61
3B
0.82 + 0.066
8.11
0.42 + 0.006
1.45
7B
0.83 +: 0,082
9.84
0,45 + 0,006
1.33
32B
0.90 + 0.052
5.75
0.48 + 0.006
1.29
3 10-1
72B
0.88 + 0.074
8.43
0.50 + 0.006
1.27
0.5B
Random Length
1.58
Table I. Comparison of the variability Spearman-correlation correlation and aggeemtt metrics when using 6500 random annotations and the same default prompt for all model sizes.
38
72B
10-1
10a
10a
10
#instructions
rankings. Both metrics are correlated, we can see for in- stance that the ranking w.r.t. the base model size used for the judge is mostly consistent across both Spearman correlation and human agreement in Figs. and For both metrics, observe that, for this prompt and model family, LLM judges bellow 7B are not able to outperform a simple length baseline which favors the output with the longest answer.
Figure 2. Effect on scaling the LLM judge and the number of instructions on human-agrecment.
Human agreement. In Fig. 2, we study the effect of scal- ing the LLM judge and the number of battles this time on human agreement. Here, we use the same prompt and base LLM models in the previous paragraph compute hu- man agreement on the LMSys dataset (lin Chiang et al., 2024).
In Table 1, we compare the variability of both metrics when using a random set 6500 instructions We report the stan- dard deviation and Coefficient of Variation (CV) which is computed a a/μ= * 100 and indicates the percentage of vari- ation of metric. It can be seen that while both metrics are correlated, human-agreement allows to differentiate larger models much more easily as the signal to noise ratio of the metric is better. In particular, allows to statistically distin- guish 32B and 72B models whereas those models are tied for Spearman correlation given the number of instructions considered.
We also observe that scaling the LLM judge base model improves performance. However, compared to the previous case, the performance is stationary which is expected since auraa-agreementt the average an instruction based property. In contrast, Spearman correlation gets better with more instructions as shown in Fig. I as the judge gets better sense of model performance (more scenarios are seen and with larger frequency).
In what follows, we therefore use human agreement the metric to optimize it allows to rank judge configurations much more cheaply than Spearman correlation and allows to separate judge configurations with much fewer instructions.
Which metric optimize. We have metrics eval- uate the judge quality: the Spearman correlation and the human agreement. As discussed in the previous paragraph, they behave differently as human agreement is an average of an atomic property (does the judge and human agrees on instruction on average) whereas Spearman correlation a global metric requiring evaluating many models to compare
'In the case of Spearman correlation, we sample 250 instruc- tions from both Alpaca-Eval and Arena-Hard datasets (which yields to 6500 annotations given that have 26 models). For LMSys, we just sample 6500 random instructions.

---

Tuning LLM Judges' Hyperparameters
5. Tuning judges
Providing its confidence is meant to help the judge pro- vide more calibrated preference scores (e.g. to not give strong score for one option when it is uncertain) while the latter two are meant to elicit chain of thought reasoning.
We first describe the search space used also summarized in Table which includes searching for the inference and prompt hyperparameters.
To use one of the three options, we add in the prompt a description of the option and asks the LLM to generate it, see Fig. 3 for an illustration where the prompt asks the judge to provide its answer and an explanation on its judgement.
5.1. Inference hyperparameters
For the LLM model, we search among apen-weights op- tions: Llama3.1 (8B and 70B), Qwen2.5 (7B, 27B and 70B) and Gemma2 (9B, 27B). All models with more than 9B pa- rameters are quantized with half-precision We also search for the LLM temperature in [0.0, 0.01, 0.1, 1.0] and whether to average predictions when considering two possible orders or using just a single order.
JSON formating. Formats used to query the output of an LLM have different trade-off's (He et al., 2024). Some are more controllable such as JSON but may loose performance against simpler format (such as raw text) in particular given less capable models.
5.2. Prompt hyperparameters
We search for two formats, using either JSON or raw text. When using ISON, we generate the prompt so that the tem- plate asks the LLM to provide a JSON with all the fields needed (the preference in the right format, how provide its hwwnaaatiii/anwwer///onnneaan when needed). In addition, we enforce the LLM to provide a valid JSON by only select- ing completions that follow the expected JSON schema. In the case of raw text, we ask the model to provide its output of each field by writing the field first, then its value.
We now descrihe how we parametrize different prompt op- tions and we illustrate one such option in Fig. 3.
Output format. When prompting a LLM judge, we must be able to parse its output into a preference. We consider the following options where the judge outputs:
best-model-identtffe:: a letter corresponding to the best assistant as in Li et al. (2023)
Prompt further details. In total, we get 5 x 2 = 80 different prompts. We test each of them by making sure that a judge based on this prompt and using (lama3.1-8B is able recognize the correct output between obviously bad and good completion when judging the pair with both orders.
likert: a likert scale (such as B, A>B, A-B to indicate respectively when model A is much better, better or comparable wrt B) as proposed in Li et al. (2024)
pair: score for both assistants in [0-10] similar to (Zhu et al., 2023)
Overall, our search space contains 4480 possible judge con- figurations, which corresponds to 80 different prompts and 56 choices for the 7 LLM models, temperatures and the choice whether to average or not output permutations. We give more details in the appendix the different prompting mechanisms as well as output preference formats.
preference: score in [0, 1] where 0 (resp. 1) indicates preference for model A (resp. B)
multi: the average score for 5 criteria conciseness, clarity, adherence, comprehensiveness and style similar to Cui et al. (2024); Lambert et al. (2024).
5.3. Multi-fidelity and multi-objective optimization
Provide answer other information. Asking LLM to reflect before providing its answer is known to be beneficial in cases requiring reasoning (Wei et al., 2022). In addition, providing example (few-shot learning) can also be beneficial as shown in (Zheng et al., 2023). We therefore search for the following options and ask the judge to provide before its preference:
Multi-fidelity optimization. Next, perform multi- fidelity multi-objective optimization in order to find judge configurations that are good for both accuracy and cost while keeping the cost of the search feasible with
Evaluating all judges on all battles is expensive and would cost N x P annotations we have / judges and P battles. One way to reduce the cost of the search is to apply a multifidelity approach such as crtesssullhalvinn (Karnin et al., 2013).
confidence: its confidence on its preference
answer: its own answer to the instruction as proposed in (Li et al., 2024)
We perform the tuning by running configurations in three steps. In the first step, we run all A configurations with only P/9 battles. We then run the top N/ /3 judges on P/3 battles second step and finally run N/9 on the full P
explanation: its explanation on the given preference
5

---

Tuning LLM Judges' Hyperparameters
Prompt Template
5.4. Hyperparameter analysis
You highly efficient assistant, please evaluate and select best large language model based on quality of their responses to given instruction.
On Fig. 5, we show the validation performance of all judge configurations at the lowest fidelity. We see that while the number of parameters influence the performance, the prompt and other judge hyperparameters have significant impor- tance given the wide variation of performance obtained for a fixed model.
User Prompt:
Who is Geoffrey Hinton?
Assistant A: Geoffrey Hinton is research scientist.
Assistant B: do not know who Geoffrey Hinton is.
Next, we examine which hyperparameters and prompt char- acteristics contribute to the best judge performance. In Fig. 6, we analyze all 4480 judge configurations the low- est fidelity (i.e., evaluated on 400 instructions) where we conduct a survival analysis, measuring how often each hy- perparameter appears in the top 100 configurations with the highest human agreement. We perform this analysis separately for large models (blue bars) and smaller models (orange bars) identify which hyperparameters are most effective in each case.
# Your Output
## Format Description
Your output should follow this format:
"answer": <your answer to the user
prompt>,
"explanation"; <your explanation on why you think or is better>,
"score_A"" <between 0 and 10 to rate the quality of AP, "score_B"; <between and 10 to rate the quality of B>
Without surprise, the model used for the LLM judge is the most impacting hyperparameters and larger is generally better. Llama3 performs better than Qwen2.5 and Gemma when used as a judge.
## Your output, do not repeat the input above.
Figure 3. Illustration of the prompt templating approach. We parametrize the prompt with the following hyperparameters: Pro- vide answer, Provide explanation, Provide example, use ISON, output preference format, Given each of the 2' x = 80 prompt hyperparameter, generate prompt like this onc.
This analysis also reveals less obvious hindsights:
The output format used to obtain judge preferences plays a big role, almost as important as the choice of the model used for the judge. Small models are more sensitive which is expected as they struggle more with more complex output mechanism. For both small and large models, the pair format performs better.
battles.
This reduces the cost of the full search from N x P to N x P/3. One could also use a more aggressive cutoff and save further in the computation but this would naturally increase the risk of missing good configurations.
Increasing temperature negatively impacts perfor- mance
Averaging the judgement after evaluating a pair of outputs in both orders gives performance boost
Providing an example helps the judge provided that large model is used smaller models gets confused by this extra information
Multi-objective optimization. When sorting the top judge configurations, have objectives take into account since we would like to find judges that both accu- rate and cheap. For instance, prompting a judge to perform Chain-of-thoughn of to provide its answer may improve per- formance, but the extra-cost may be better spent on a better and more expensive base model.
Asking the judge to provide an explanation, or its an- swer hurts performance
Using JSON does not impact much the performance
5.5. Prompt stability
To sort configurations while accounting for the two objec- tives, we use non-dominated sort (Emmerich & Deutz, 2018) as it was shown efficient in multi- fidelitty settings (Salinas et al., 2021; Schmucker et al., 2021; I; Izquierdo et al., 2021). We illustrate the priority given to the judge configurations in Fig. 4 in the first and second selection step where one can see that the priority model the geometry of the Pareto front
Next, we investigate how much the performance of a prompt varies between different model judges in Fig. 7, where show the correlation between different judge models on how well they perform under different prompts. For each of the m = 7 models, we measure the average human agreement for all the different n = 2 x 5 = 80 prompts and compute the covariance matrix X X where X E RR is the matrix
well.
6

---

Tuning LLM Judges' Hyperparameters
Hummnn agreement (val)
Initial configurations 4480 configs
Step 1 1200 configs
Step 2 400 configs
0.4
+
0.0002
0.0004
0.0006
0.0008
0,0010
0.0002
0,0004
0.0006
0.0008
0.0010
0.0002
0.0004
0.0006
0.0008
0.0010
Cost per annotation (5)
Cost per annotation (5)
Cost per annotation
Figure Illustration of the selection process. 4480 configurations are evaluated on 400 instructions (left), the top 1200 configurations then evaluated on 1200 instructions (center) and finally the top 400 configurations evaluated 3548 instructions (right). The color denotes the ranking assigned by the non-dominated sort procedure.
Validation scores
(validation)
Judge
Human agr. (t)
Cost per 1K ann. (1)
Random
0.33 +/- 0.01
Length
0.42 +/- 0.01
Model
#Params
PandaLM-7B
0.38 +/- 0.01
6.0
0.30
Owen2.55
JudgeLLM-7B
0.42 +/- 0.01
8.6
Gemma2
32
Arena-Hard
0.50 +/- 0.01
1.2
0.25
llama3
72
Ours-small
0.45 +/- 0.01
0.21
0.0002
0.0004
0.0006
0.0008
0.0010
Ours-medium
0.47 +/- 0.01
0.48
per annotation (S)
Ours-large
0.49 +/- 0.01
0.48
Table 2. Comparison of judges on LMSys test instructions. For each judge, report the bootstrap mean and std for human- agreement on 3K test instructions with 100 seeds.
Figure We plot the cost per annotation and human agreement of all 4480 judges when using 400 instructions. The model family and the number of parameters are represented with color and size respectively.
of scores for all models and prompts. Interestingly, smaller models and larger models are highly correlated which shows that two group of prompts works well for large and small models. This is expected as lower capacity models may struggle obey more complex instructions (e.g. rate models for style and accuracy using JSON) that are beneficial to evaluate better models.
Judge
Human agr. (t)
Random
0.33
Length
0.60
GPT-3.5
0.63
5.6. Results on test datasets
GPT-4
0.67
PandaL
0.62
In this section, we report the performance of three judges found by our search on several test sets. While the multi- objective search returns a list judges with continuous cost tradeoffs as seen in Fig. 4, we report results for only judges: small, medium and large with numbers of parame- ters respectively lower than 10B, lower than 32B and lower than 72B. We do this as it may offer an easier choice for practitioner when picking judge, for instance accounting for the memory constraint of a given GPU. For each cate- gory, we select the judge with the best validation score on the 3548 instructions of the validation set of LMSys.
PandaLM-70B
0.67
Ours-small
0.67
Ours-medium
0.78
Ours-large 0.76
Table 3. Comparison with PandaLM on PandaLM test set. Note that method is fine-tuned opposed to PandaLM.

---

Tuning LLM Judges' Hyperparameters
model
temperature
e-saapleeittt-shhit
score-type
provide explanaixpi
provide-cxample
provide-answer
jso-ccupput
100
0.6
>108
0.2
Proportion
0.0
qwenz 5.76
Ilame 3mna-31100 3-480
gemma-2-220
15.320
awen2.5 2.00225-7200
llama 3.1 3.1
0.1
10
best-modall
preference
mulu
likert
pair
True
False
True
Fals
True
Fasc
True
Figure Fraction of time each hyperparameter appears in the top 100 configurations for small (<10B) large models (>10B).
Judge
Sp. corr. (f)
Cost per IK amn. (J)
Length
0.50 +/- 0.21
Arena-bard + Claude
0.82 +/- 0.12
75.0
Correlation between
Arena-hard GPT4
0.90 +/- 0.06
50.0
0:92
0:79
2.00
Ours-small
0.81 +/- 0.10
0.21
Ours-medium
0.93 +/- 0.05
0.48
Ours-large
0.86 +/- 0.09
0.48
0:82
0.76
5.72
quer12 5-72b
0:72
0.78
0.75
0.63
0.5
0.75
-gmmms-2-777
Table 4. For each judge, we compute the Spearman correlation between win-rates using the protocol of Arema-Hard and EL.O- ratings computed from human annotations from Chatbot Arena. We report mean and std over 100 boostraps of the set of models.
0.78
0.63
0.69
2.63
1.770
Ban 1-48
agreement but strongly outperforms it in term of cost.
PandaLM. In Table 3, we show the performance on Pan- daLM test (Wang et al., 2024). The judges that found all outperforms strongly PandaLM, even our small judge with less than 10B parameters outperforms its PandaL. counter with 70B parameters. Importantly, we recall that our approach only consider zeroshot judges, c.g. it does not fine on the training dataset which would improve further the performance on this dataset although it may also hurt gen- eralization as seen for the LMSys dataset. We that the 70B model underperforms slighly the 32B model however, both scores are high and close to inter-agreement rates seen in real-world data.
Figure 7. Prompt performance stability across different models. We show correlation matrix between models when looking at their performance on all the 80 different prompts.
LMSys. In Table 2, we compare tuned judges with sim- ple baseline that picks the longest answer, PandaLM (Wang ct al., 2024), JudgeLM (Zhu et al., 2023) and Arena- Arena-Hard (Li et al., 2024). For the latter, we use GPT-4o mini to ob- tain a similar cost per annotation than other methods. We compute scores by measuring huran-agreement 3 000 test instructions that are not used for the model selection and report mean and standard errors over 100 bootstraps.
Arena-Hard. In Table 4, we report the Spearman corre- lation of the judges found by our method, compared to the judge proposed in Arena-Hard using the 20 models available for both Claude-Opus and authors repository (Li et al., 2024). For each judge, we anno- tate all the 20 models on the 500 instructions of Arena-Hart against baseline and compute winrates against this base- line. We then compute the Spearman correlation between the winrates and ELO-ratings from Chatbot Arena. We esti- mate the cost ofArena EEAren--Har judges using the cost estimate from Ni et al. (2024) 25$ to annotate one model on all 500 instructions and multiply this estimation by 50% for Claude
While all methods outperform a random baseline, Panda- LM-7B underperforms and JudgeLM-7B only matches a simple baseline Length that picks the longest answer. This because the instructions on LMSys are more complex and significantly longer from the distribution used for fine- tuning and confirms previous findings that fine-tuned judges performance can be affected by change of distributions (Huang et al., 2024).
judges the
The judges we found outperforms baselines and slightly underperforms or matches Arena-Hard for human-
8

---

Tuning LLM Judges' Hyperparameters
Opus given the difference as it corresponds roughly to the additional token cost compared to GPT-4-1 106- p06-preee
Impact Statement
This paper demonstrated how to tune judge hyperparameters while balancing cost considerations and ensuring that the search remains feasible. Our approach optimizes judge hyperparameters to maximize human agreement while also considering open-weight alternatives.
The judge we found match or outperforms the judge con- sidered at much lower cost. Importantly the judge config- urations are open-weight models which provide additional benefits, in particular for applications such as building a community leaderboard or building an open model.
The benefits of this approach include enabling fairer and more cost-effective leaderboards and helping the community adopt judges that do not rely on closed systems. However, L.MM judges may also reinforce undesirable superficial bi- ases, such as favoring stylistic elements over substantive quality or perpetuating human biases present in the training data, including biases against certain groups. We conducted preliminary review of the selected LMSys data and did not observe obvious issues, though our analysis was limited to a small sample. As a result, such systems should not be de- ployed without safeguards and additional bias evaluations.
6. Limitations
We currently select judges solely based on accuracy and cost, not on potential biases such as position (where LLMs favor responses based on order (Li et al., 2024)). To investigate whether our selection criteria worsen this bias, we measured the flip rate, how often a judge changes its decision when the response order is swapped, and found a strong negative cor- relation with the human agreement scores (r" 78.789) of the judge configurations of the top rung. While the position bias is not worsen, other biases, such as stylistic preferences or verbosity, may be worsen by our method. Future work could consider those biases by including them as additional objectives.
Our approach analysed the prompting strategy and hyperpa- rameters of the current generation of LLMs while expect our conclusion to hold given the relative stability of prompt strategy across this family (see Fig. 7), the conclusion could change over time with the introduction of distinctive new capabilities such as reasoning.
7. Conclusion
In this paper, we examined how judge performance is in- fluenced by scaling and hyperparameter choices. We in- troduced multi-fidelity, multi-objective approach to tune judge hyperparameters - including prompt design and base models- at a feasible cost. Our results demonstrate that this method can produce judges that outperform previous approaches across different budget constraints.
A Acknowledgments
This research was partially supported by the following sources: TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215; the Deutsche Forschungsemmiincchatt (DFG, German Re- search Foundation) under grant number 417962828 and 539134284 through EFRE (FEIH_ 2698644)))44)) and the state of Waadln--Wrtttebbergg the European Research Council (ERC) Consolidator Grant "Deep Learning 2.0" (grant no. 101045765); EC under the grant No. 101 101195233 (OpenEu- roLLM), Frank Hutter acknowledges financial support by the Hector Foundation. The authors acknowledge support from ELLIS and ELIZA. Views and opinions expressed are however those of the author(s) only and do not nec- essarily reflect those of the European Union the ERC. Neither the European Union nor the ERC can be held re- sponsible for them. The authors gratefully acknowledge the computing time made available to them on the high- performance computer NHR @ KIT Compute Cluster at the NHR Center NHR @ KIT. These Centers are jointly supported by the Federal Ministry of Education and Re- search and the state governments participating in the NHR
While some limitations of LLM judges persist, we hope that enabling cost-effective tuning will help the community refine their use and address remaining deficiencies. For ex- ample, our multi-objective procedure could be extended to optimize for additional criteria such as stability or explain- ability.
We release the code to reproduce our results, along with dataset containing all annotations at https: / /t///ttttt thub. com/ gann/geolgo//// judgetuning. We hope this resource will support further analysis and improvement of L.LM judges.
OPEN EURO
Baden-Württemberg
Co-funded by the European Union
M
LLM

---

Tuning LLM Judges' Hyperparameters
References
Ilm evaluation, 2024. URL https:://aaiiii abs/ 0.l//2403.220
Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., and Stoica, I. Chatbot arena: An open platform for evaluating Ilms by human preference, 2024. URL https: // //aaxaixvv////////////////aaaav/ www.i.orggaa..........t...oooaar htrg/abs///0 2403. 04132.
Izquierdo. S., Guerrero- Viu, J., Hauns, S., Miotto, G., Schrodi, S., Biedenkapp. A., Elsken, T., Deng, D., Lin- dauer, M., and Hutter. Bag of baselines for multi- objective joint neural architecture search and hyperparam- eter optimization. In 8th ICML Workshop on Automated Machine Learning (AutoML), 2021.
Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., Liu, Z. and Sun, M. Ultrafeedback: Boosting language models with scaled ai feedback, 2024. URL https:// http///xxvvv//o////ee abs/ 2310. 01377.
Karnin, Z., Koren, T., and Somekh, O. Almost optimal exploration in multi-armed bandits. In Dasgupta, S. and McAllester, D. (eds.), Proceedings of the 30th Interna- tional Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 1238- 1246, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. URL https: http//oooceennngeeee/eeeeeeeeeee . .mlr. press/ /v28/ karnin13 hhmml.
Doddapaneni, S., Khan, M. R., Verma, S., and Khapra, M. M. Finding blind spots in evaluator Ilms with inter- pretable checklists. arXiv preprint arXiv:2406. 13439, 2024.
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint wrXiv:2404.04475, 2024.
Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V., Liu, A.. Dziri, N., Lyu, S., et al. T\ ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.1512 2024.
Elshin, D., Karpachev, N., Gruzdev, B., Golovanov, I., Ivanov, G., Antonov, A., Skachkov, N., Latypova, E., Layner, V., Enikeeva, E., Popov, D., Chekashev, A., Ne- godin, V., Frantsuzova, V., Chernyshev, A., and Denisov, K. From general LLM to translation: How dramat- ically improve translation quality using human evalua- tion data for LLM finetuning. In Haddow, B., Koemi, T., Koehn, P., and Monz, C. (eds.), Proceedings of the Ninth Conference on Machine Translation, pp. 247- 252, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18855/v1/2224, 1.1- URL https://aclanthont 2024. wmt 1. 17/.
Li, T., Chiang, W.-L., Frick, E., Dunlap, L., Wu, T., Zhu, B., Gonzalez, J. E., and Stoica, I. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. 1939, 2024.
Li, X., Zhang, T., Dubois, Y. Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacaeval: An automatic evaluator of mnstruction-following models, 2023.
org/
lin Chiang, W., Zheng, L., Dunlap. L., Gonzalez, J. E., Stoica, I., Mooney, P., Dane, S., Howard, A., and Keat- ing. N. Lmsys chatbot arena human preference predic- tions. https: / /aggle. e com/ compet t tinss/ imsys chatbot arena, 2024. Kaggle.
Emmerich, M. T. and Deutz, A. H. A tutorial on multi- objective optimization: fundamentals and evolutionary methods. Natural computing. 17:585-609, 2018
Liu, Y., Zhou, H., Guo, Z., Shareghi, E., Vulić, I., Korhonen, A., and Collier, N. Aligning with human judgement: The role of pairwise preference large language model evaluators. In First Conference on Language Modeling, 2024. URL https: /owpnrvveeeeeeeeeeeeeeeeeerreeee net/ forum? i id=9gdZI7c6yr.
Fernando, C., Banarse, D., Michalewski, H., Osindero, S., and Rocktäschel, T. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309.16797 2023.
Grattafiori, A., Dubey, A., Jauhri, A., and Abhinav Pandey, e. a. The laaa herd of models, 2024. URL https: / //arxiv. org/ abs/ 24007-40 21783.
Ni, A., Iyer, S., Radev, D., Stoyanov, V., Yih, W.-T., Wang, S., and Lin, X. V. LEVER: Learning to verify language-to- code generation with execution. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Ma- chine Learning Research, pp. 26106 26128. PMLR, 23- 29 Jul 2023. URL ht https: //ppoceedinggs mmlr. vrees/v/02// press v202//22//2//2 / n 23b. html.
He, J., Rungta, M., Koleezek, D., Sekhon, A., Wang, X., and Hasan, S. Does prompt formatting have any impact on Ilm performance?, 2024. URL https: // arxiv. org/ abs/ / 2411.1111.. 10541.
Huang, H., Qu, Y., Zhou, H., Liu, J., Yang, M., Xu, B., and Zhao, T. On the limitations of fine-tuned judge models for
10

---

Tuning LLM Judges' Hyperparameters
J., Xue, F., Yue, X., Deng, Y., Shah, M., Jain, K., Neubig, G., and You, Y. Mixeval: Deriving wisdom of the crowd from Ilm benchmark mixtures, 2024. URL https:// orgiv.org/ abs 240.. 06565.
Zhou, H., Wan, X., Liu, Y., Collier, N., Vulic, I., and Korho-
nen, A. Fairer preferences elicit improved human-aligned large language model judgments. CoRR, abs/2406.11377 2024.
Panickssery, A., Bowman, S. R., and Feng, S. Llm eval- uators recognize and favor their own generations, 2024. URL https: //rxiii://axxv....// arxii.
Zhu, L., Wang, X., and Wang, X. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310.76311, 2023.
org/ abs/2404 13076. 13076.
Salinas, D., Perrone, V., Cruchant, O., and Archambeau, C.
A multi-objecttive perspective on jointly tuning hardware and hyperparameters, 2021. URL https: org/abs/ 2106. 0568C.
Schmucker, R., Donini, M., Zafar, M. B., Salinas, D., and Archambeau, Multi-objective asynchronous successive halving, 2021. URL https://arxiv.oor abs/ 2106. 12639.
Shi, C., Yang, K., Chen, 7., Li, J., Yang. J., and Shen, C. Efficient prompt optimization through the lens best identification. In The ehirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Soylu, D., Potts, C., and Khattab, O. Fine-tuning and prompt optimization: Two great steps that work better together. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, Miami, Florida, USA, November 2024. Association for Computational Linguisties.
Wang, Y., Yu, Z., Yao, W., Zeng, Z., Yang. L., Wang, C., Chen, H., Jiang, C., Xie, R., Wang, J., Xie, X., Ye, W. Zhang, S., and Zhang, Y. PandaLM: An tomatic evaluation benchmark for LLM instruction tun- ing optimization. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: / //opereeiiew www.riiwweee..eeee fiee/fooree forum wttprmmmmd=5nnnnni..mmmm dd=5N22LN7755
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E. H., Le, Q.. and Zhou, D. Chain of thought prompt- ing elicits reasoning in large language models. CoRR, abs/2201. 11903, 2022. URL abbs/2001.11 11903.
Zhang, T., Ladhak, F,, Durmus, E., Liang, P., McKeown, K., and Hashimoto, B. Benchmarking large lan- guage models for news summarization. Transactions of the Association for Computational Linguistics, 12:39- 57, 2024. doi: 10. 10.1162/tac__ 206322 URL https: //aclanthology org/ 22tp://hw..oog....222224... wttp:////2.....11.......eeeee. tta.llllll......
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46623, 2023.

---

Tuning LLM Judges' Hyperparameters
Your task evaluate how well the following imput prompts can assess the capabilities of advanced Al assistants. For the input prompt, please analyze it based on the following criteria. For each criteria, make sure explain before determine whether the input satisfy it.
1. Specificity: Does the prompt ask for specific, well-defined output without leaving any ambiguity? This allows the Al to demonstrate its ability to follow instructions and generate a precise, targeted response.
2. Domain Knowledge: Does the prompt test the Al's knowledge and understanding in a specific domain or set domains?
The prompt must demand the Al to have a strong prior knowledge or mastery of domainspecific concepts, theories, or principles.
3. Complexity: Does the prompt have multiple components, variables, or levels of depth and nuance? This assesses the Al's capability to handle complex, multi-faceted problems beyond simple queries.
4. Problem-Solving: Does the prompt require active problem-sol volv::::::: analyzing and clearly defining the problem and
systematically devising and implementing a solution? Note active problem-solving is not simply reciting facts or following a fixed set of instructions.
5. Creativity: Does the prompt require a creative approach or solution? This tests the Al's ability to generate novel ideas tailored the specific needs of the request or problem hand.
6. Technical Accuracy: Does the prompt require an answer with a high degree of technical accuracy, correctness and
precision? This assesses the reliability and truthfulness of the Al's outputs.
7. Real-World Application: Does the prompt relate to real-world applications? This tests the Al's ability to provide practical and actionable information that could be implemented in real-life scenarios.
After analyzing the input prompt based on these criteria, you must list the criteria numbers that the prompt satisfies in the format of a Python array. For example, "Criteria Satisfied: [1, 2, 4, 6, 7]"
Figure Prompt to evaluate instruction quality.
A. Datasets
A.1. Alpaca-Eval and Arena-Hard datasets
We consider two datasets that contains prompts, model completions and judge annotations for a grid of prompts and model pairs. The first one is Alpaca-Eval which contains 47 models completions on 805 prompts (Li et al., 2023). The second one is Arena-Hard which contains the completions on 500 instructions for 57 models (Li et al., 2024). both cases, we select the 26 models that also appear in Chatbot Arena in order to be able to compute how well judge configurations approximate human judgement.
A.2. LMSys
We use the LMSys dataset (lin Chiang et al., 2024) which contains 51734 battles and allows to measure the human-agreement of given judge configuration.
LMSys validation and test split. When using LMSys, we rate instructions with the prompt from (Li al., 2024) given in Fig. 8 and we use Alama3--B-instruus which assigns
to each instruction in [0, 7].
We select instructions which have score greater than or equal to 5 and also have the criterion 1. Specificity detected since it is important to avoid large ambiguity prompt to evaluate judge (e.g. we want to discard the instruction like "say hello" since they provide no value to distinguish models).
This gives 6548 instruction which we split randomly into 3548 validation instructions and 3000 test instructions. All model selection (c.g. non-dominated sort) is done only using validation instructions and only the best models on the validation set evaluated on the test set.
B. Experiment details
To generate inference with open models, we host the models locally using VLLM on L40 GPUs for models up to 32B parameters and on H100 GPUs for models with more than 70B parameters. Given that we use two clusters with two different job queues, we favored using synchronous successful halving rather than an asynchronous approach such as (Schmucker
12

---

Tuning LLM Judges' Hyperparameters
Hyperparameter
Values
model
Llama3 (8/70B), Qwen2.5 (7/27/70B).
Gemma2 (9/27B)
temperature
0.0, 0,01, 0.1, 1.0
average-orders
False, True False, Truc
provide-example
provide-answer
False, Truc
provide-cxplanation
False, True
use-json
False, True
output-type
likert, best-model-letter, pair, preference, multi
Table 5. Judge Hyperparameters considered. In total, the search-space contains 4480 different judge configurations.
Model
Cost / IK token (S)
qwen2.5-72b
0.58
qwen2.5-32b
0.36
llama-3.1-2
0.35
gemma-2-27b
0.30
gemma-2-44
0.14
qwen2.5-7b
0.12
llama-3, 3.1-8b
0.11
Table 6. Cost per token estimated from runtime evaluations
al., 2021). submitted all the 4480 configurations of the first fidelity with 400 instructions, then applied non-dominated sort and submitted the top 1200 configurations with 1200 instructions before finally submitting the top 400 configurations with 3548 instructions.
B.1. Cost
To compute the cost judge annotation, we first estimate the token price and then multiply the number of tokens by the token price?. To obtain the average token price for model, we measure the total runtime and number of tokens on large collection of judge annotations for the given model. We then derive the cost per token using the H100 hourly price of runpod (2. 2.79%//hur) for models requiring more than 48GB of VRAM (Llama3 70B, Qwen2.5 70B and Gemma2 27B) and using L40 hourly price for other models (0. (0.995/hour).
We arrive at the cost per token given in Table 6. The estimate are lower than a public provider such as Together which is expected as such service needs to operate a margin and over-provision machines to meet demand. The cost we estimate highly conservative given that no optimization was done to optimize VLLM hyperparameters.
For close models, compute the cost identically by multiplying tokens seen in prompt and completion with the corre- sponding token price.
B.2. Prompt templating
Prompt examples.
In Fig. 9, we show a full prompt corresponding to the prompt hyperparameter:
"Provide answer": True, "Provide explanation": True, "Provide example":
True, "use JSON": True, "output preference format": "Pair"
and in Fig. 10, we show the prompt corresponding to:
"Provide answer": True, "Provide explanation": False, "Provide example":
False, "use JSON": False, "output preference format": "Likert").
AAn alternative approach would However, this approach yields to noisy estimation and large costs some judges just because of hardware noise.
by multiplying the runtime of judge with the hourly price of the machine used.
13

---

Make sure to not over-cenfidently prefer one assistant or the other and also make sure to Tuning LLM Judges' Hyperparameters
You are highly efficient assistant, who evaluates and selects the best large language model based on the quality of their responses to given instruction.
You will be shown
instruction and the cutput of Assistant A and Assistant B and will have to decide which one was best.
- not bias your preference based on the ordering or on the length of the answers.
Example
Let us first look example.
Input
<|User Prompt
What is the square rect of 81? Just provide the answer.
<| Start of Assistant A Answer|>
The answer is 81, this <he End of Assistant A's Answer|>
9*9 81.
The Start of Assistant B's Answer 81
< The End of Assistant B's Answer |>
## Your expected output (must be valid JSON)
"answer": "81",
"explanation"; "Both model are correct however, the output from model A is verbose and does not provide just the answer whereas the instruction asked for conciseness. "score_A"; 2, "score_B":
For the explanation, do not exceed three sentences.
Now is the judgement would like to make, please follow the format I just described.
## Input
<|user Prompt Who is Barack Obama?
The Start of Assistant A's Answer Barack Obama is former president. The End of Assistant A Answer
<h Start of Assistant B's Answer do not know who Barack Obama is.
The End of Assistant B's Answer
# Your output, do not repeat the input above (must be valid JSON)
Figure 9, Examplc of prompt for the user prompt "Who is Barack Obama?". In this the judge is asked to provide its answer, an explanation, provided an example. asked to use Pair format and provide answer ISON..
14

---

Tuning LLM Judges' Hyperparameters
You are highly efficient assistant, who evaluates and selects the best large Language model based the quality of their responses to given instruction.
You will be shown
instruction and the cutput Assistant A and Assistant B and will a have to decide which one was best.
Make sure to not over-confidently prefer one assistant or the other and also make aure to not bias your preference based on the ordering or on the length of the answers.
<see Prompt >> Who is Barack Opama?
The Start of Assistant A's Answeri> Barack Obama is a former US president. <I The End of Assistant A's Answer|>
The Start of Assistant B's Answer|> do not know who Barack Obama is. The End of Assistans B's Answer
Your output
#d Format description
Your output should fellew this format:
answer: syour answer the prompt>
score: <one of A>>B, A>B, A=B, A<B, A<<3, see instruction bellow>
The "score" value should indicate your preference for the assistant. You must output only one of the follewing choices as final verdict with a label:
A>>B: Assistant is significantly better
A>B: Assistant is slightly better
A-B: Tie, relatively the same
B>A: Assistant is significantly better
н>>A: Assistant B is signifi better
#l Your output, do not repeat the input above
Figure 10. Example of prompt for the user prompt "Who is Barack Obama?". In this case, the judge is asked to provide its answer. It is asked to use the Likert format and provide answer raw text.
15

---

Tuning LLM Judges' Hyperparameters
B.3. Tuning cost estimation
Alpaca-Eval and Arena-Hard. In both cases, evaluate Spearman correlation one must annotate a judge on a grid of models and instructions.
Let us assume we evaluate Njudges = 4480 as done in this work for Temodels = 20 models as done in (Li et al., 2024) on the 805 instructions of Alpaca Eval. We get the cost to annotated one model as 24$ by using the estimation of Ni et al. (2024), This gives a cost of 4480 24 * 20 = 186 240$ for Alpaca-Eval and cost of 240 000S for Arena-Hard whose cost to annotate the instructions for one model was estimated to 25 in (Ni et al., 2024).
Cost estimation of our approach. We evaluate 4480 judge configurations on 400 instructions, then the top 1200 judge configurations on 1200 instructions, then the top 400 judge configurations on the full set of P 3548 validation instructions. This requires a total of 651 200 annotations. On average, a single annotation takes about 0.6s on H100. If using runpod with cost of 2. 2.799/hour per H100 hour, get total cost of 4651200/ 2451100036600 0.6 2.79 x 2.1KS.
The savings are obtained by identifying a more efficient metrics to distinguish judges than Spearman correlation (which requires evaluating a grid of models and instructions in (Li et al., 2023) and (Li et al., 2024)) and applying multi-fidelity which allows us to save roughly a factor of 3 since it avoids to annotate the full set of N x P.
C. Multi-objective background
In hyperppraamterroppmiiizaiiioo one seeks to find the best hyperparameter 0* of a blackbox function f RR + RR e.g. to find:
f* arg min f(0).
BE
The blackbox may be for instance a neural network that we want to train and the hyperparameter may include the number of layers or the learning rate.
Multi-objective optimization. When considering several objectives, we now want to minimize a function f(0) R... Since we have more than one objective, there is not single best hyperparameter 6* general but set of non-dominated solutions.
We say that hyperparameter 0 dominates 0' if and only if:
vi, f(0), < f(0", and Bi, f(0) < f(0"),
and denotes it with 0 < 0' i.e. when all components of f(0) are lower equal than the ones of f(0)) and one component strictly better.
We aim to find the Pareto front P which consists of non-dominated solution:
p = (0 E IRdd| 700' 0' x <0}
Non-dominated sort. When applying successful-halee halving, we need to sort the top configurations, for instance to keep the top 50% and let those configurations run with larger budget.
Since we have multiple objectives, something must be done to adapt the algorithm. While averaging out the objective (or doing more advanced scalarization) allows to go back to the scalar case, it only works on some cases (where the Pareto front convex the case of averaging the objectives for instance).
Another approach is to non-dominated sort (Emmerich & Deutz, 2018) which illustrate in Fig. 11 and now describe. The approach first computes the Pareto front of the current set of observations and assign top-ranks with a heuristic to break ties. Then, the approach is applied recursively until no points are left. To break ties, multiple heuristic can be used, in our case we use an epsilon-net as was shown to perform well in (Schmucker et al., 2021). ) As can be seen in Fig. 4, the approach models the geometry of the Pareto front compared to scalarization approaches.
16

---

Tuning LLM Judges' Hyperparameters
Layer
Layer 2
Layer
Layer 4
26
Objeerive
10
15
24
Objective
Objective #2
Objective #2
Objective #2
Figure 11. Illustration of the non-dominated sorting approach. The process first computes the Pareto front, assigning top ranks to the points in this layer. Next, the Pareto front is determined for the remaining points, which are then assigned the next set of rankings. This process continues iteratively until all points are ranked. resolve ties within each layer, heuristic is applied to balance both sparsity and coverage.
0.55
Human (val sett
0.45
0.40
0.35
0.30
False
True
Thue
0.50
Human (val
0.45
qwen22 5-32b
0.40
gemma-2-9b 2-0m----99bb-
likert
gemma 2:27b
0.35
best-nnodelidentiilfe preference
qwen2 5-7b
0.0001
llama-3 1ite-3,1-770
False
0,1
multi
lama-3.1-88
0.25
pair
qwen2.5-722
0.0002
0,0004 0. 0.0006 0.0008 0.0010 cost
0.0002 0. 0.0004 0.0006 0.0008 0.0010
0,0002 0.0004 cost
0.0006 0.0008 0.0010
0.0002 0.0004 0.0006 0.0008 0.0010
cost
cost
Figure 12. Scatter plot of cost and human agreement on the 400 validation instructions for all judges. We color-code each hyperparameter differently illustrate the performance of all judges. Even using the same LLM model, there is large spread of performance when varying other hyperparameters without an obvious pattern to distinguish the best configuration which motivates the need to search for optimal hyprrparameters.
400 Instr p-0.88
1200 instr p- 0.93
3548 instr p-0.95
3.60
0.55
0.50
0.45
0.40
0:35
Haman
0.55
0.20
agr. on
agr:
agr.
Figure 13. Correlation for different fidelity sizes. For cach fidelity, we randomly split the instructions into two buckets and plot the human-agreemenn on the first bucket versus the same metric computed second bucket of instructions for all the available judges, we also report the Spearman correlation between the two groups This allows to see the correlation one can obtain between the two sets. the final fidelity, we measure validation performance 3548 instructions and use 3000 test instructions so we expect higher correlation between validation and test scores.
17


Reviews:
[{"review_id": "83jKAPpmng", "reviewer": "Reviewer_ETKe", "summary": "This paper proposes a cost-effective approach to systematically tune hyperparameters of Large Language Model (LLM)-based judges for evaluating other LLMs, significantly reducing the required resources. The authors leverage a multi-objective, multi-fidelity optimization framework to efficiently search through 4,480 configurations, considering factors like model choice, prompting strategy, inference parameters, and output parsing method. Their method identifies judges outperforming existing benchmarks in accuracy and cost while exclusively utilizing open-weight models to ensure accessibility and reproducibility. The authors find that optimal judge performance strongly depends on the selected LLM, prompt style, temperature settings, and response parsing mechanism, rather than simply scaling model size or instruction count.\n\n## update after rebuttal\nI thank the authors for the rebuttal, but I keep my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and thorough review. We are delighted to hear that you found the approach sound and well-executed.\n\nYour point on potential bias is indeed very relevant. One concern one could have is that the selection could on one hand improve human agreement but, on the other hand worsen bias. We analyzed the position bias and found that it was negatively correlated with human-agreement performance, e.g. better models for human-agreement tend to have lower positional bias. For length, we could not conduct an analysis in time. \n\nWe will add a discussion of this point in our paper. In particular, we will point that in case improving human-agreement worsens bias (which could be the case for length for instance), one could then just add this measured bias as an objective with our proposed approach.", "claims_and_evidence": "The claims made in the paper are supported by clear and convincing evidence, primarily through systematic empirical experiments comparing the tuned judges against existing benchmarks across multiple datasets. The extensive hyperparameter search, multi-objective optimization analysis, and direct comparisons against baselines strongly substantiate their conclusions.", "methods_and_evaluation_criteria": "The methods and evaluation criteria proposed by the authors make sense for the problem they’re tackling. They use human agreement and correlations with established datasets (LMSys, PandaLM, Arena-Hard) as evaluation metrics, which fits nicely with their goal. Their choice to optimize hyperparameters using a cost-saving, step-wise tuning method is logical given the high expense typically involved in evaluating these models.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The multi-fidelity procedure (successive halving across three fidelity steps: 400, 1200, and 3548 instructions) is logically sound and efficiently manages computational costs. The analyses on hyperparameter sensitivity (e.g., prompt formatting, model size, and temperature) are well-executed, clearly demonstrating which factors contribute significantly to judge performance. There were no issues identified; the approach is robust, transparent, and methodologically sound.", "relation_to_broader_scientific_literature": "The paper positions itself within existing research on automatic evaluation of LLMs using LLM judges, emphasizing the cost and complexity of human annotation as motivation. The authors specifically build upon previous work like Alpaca-Eval and Arena-Hard, systematically addressing confounding factors (such as simultaneous changes in judge model, prompts, and scoring methods) that hindered clear comparisons in prior research. By employing multi-objective multi-fidelity optimization, they extend beyond existing prompt-tuning approaches (e.g., Promptbreeder) and earlier explorations into prompt stability across models. Unlike related efforts such as PandaLM and JudgeLM—which primarily use fine-tuning or closed models—their approach notably relies exclusively on open-weight, zero-shot judges, significantly enhancing accessibility and reproducibility.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n- The use of open-weight, zero-shot models increases accessibility and encourages reproducibility, addressing critical limitations of prior work relying on expensive closed models.\n- The analysis of hyperparameter importance (prompt style, inference parameters) is thorough and insightful, providing valuable practical guidelines.\n\nWeaknesses:\n- Potential biases or superficial stylistic preferences of judges, despite being briefly mentioned, are not systematically analyzed in depth.", "other_comments_or_suggestions": "None", "questions_for_authors": "You briefly mention potential superficial biases (e.g., preference for longer answers). Could you provide further insights or data on how your tuned judges mitigate or exacerbate such biases compared to existing approaches?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0SPZNVQD6p", "reviewer": "Reviewer_rLGG", "summary": "This paper is more like an extensive experimental report. The authors systematically analyze the hyperparameters of LLM judges, including the choice of model, inference parameters, and prompt hyperparameters (e.g., output format, provide answer or other information, JSON formatting). Overall, our search space contains 4480 possible judge configurations, which corresponds to 80 different prompts and 56 choices for the 7 LLM models, 4 temperatures, and the choice of whether to average or not output permutations. They leverage multi-objective and multi-fidelity optimization techniques to balance accuracy and cost, significantly reducing the expense of tuning compared to prior methods. By evaluating on datasets including Alpaca-Eval, Arena-Hard, and LMSys, the optimized judges outperform existing methods in terms of accuracy and cost-efficiency while relying solely on open-weight models. The study highlights the importance of prompt design and model selection in improving judge performance and provides insights into the trade-offs between cost and accuracy. The results show that their approach can identify superior judge configurations at a fraction of the cost, making the evaluation of LLMs more accessible and efficient.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and valuable feedback. We answer here the main three points raised by your review.\n\n**Comparison with other search approaches. The methodology relies on brute-force search.**\n\nWe believe the characterization of brute-force may be a bit strong since we are using an efficient multi-fidelity approach but we agree with you that we evaluate a full grid of all options. Of-course, an alternative could be to use a model-based approach which can sample a better part of the space, for instance a Gaussian Process or an Evolution Algorithm. However, we prefer to explore the whole search space for two important reasons. The first one is that the analysis of hyperparameter performance is easier and does not require fitting for instance a random-forest to understand hyperparameter importance. The second is that having the full set of options allows one to simulate any (model-based) method and compare their performance. We will make the data available in a way where one can easily compare different optimizers, we did not do it in this paper as we believe this is a point orthogonal to the main point of the paper (showing how LLM judges can be tuned by reducing the tuning cost). Note that the cost of tuning on judge configuration with our approach is still reasonable costing around \\$285 per judge.\n\n**This paper ignored the related works on automatic prompt optimization.**\n\nThank you for providing these relevant references. We cited two papers related to prompt optimization (Fernando et al. 2023 and Doddapaneni et al. 2024), but we agree that both references are relevant, and will discuss them in the paper. The first one in particular is a great reference as it is another one discussing prompt tuning for judges. Regarding the second one, it is relevant for LLM judges in general but does not mention prompt tuning. Is it the correct one that you wanted to discuss?\n\n**The paper does not adequately address whether the identified optimal configurations generalize or are merely artifacts of the specific LLMs being evaluated.**\n\nYou are right that different LLMs may behave differently regarding the prompting strategy. We did one analysis regarding this with Figure 7, which shows how much prompt ranking changes across different models. The figure showed that similar model capacities have high correlation (the Spearman correlation of qwen2.5-32b, llama-3.1-70b, and qwen2.5-72 is larger than 0.88 for instance).\n\nWe agree that this analysis does not fully resolve this question, but we believe it quantifies how much we can expect our results to generalize through current LLMs (also it is clear that no absolute statement can be made for LLMs as future generations may always differ from the current one in any aspect). We also want to point out that our method costs about 2k dollars to evaluate all the 4480 configurations for the 7 families which gives a cost of 285 dollars per LLM, which we believe is reasonable for a new model.\n\nFollowing your remark, we propose to add the following statement in the Impact Statement:  “Our approach analysed the prompting strategy and hyperparameters of the current generation of LLMs while we expect our conclusion to hold given the relative stability of prompt strategy across this family (see Fig 7), the conclusion could change over time with the introduction of distinctive new capabilities such as reasoning.”\n\nLet us know if you have any feedback on the wording or generally regarding this point.", "claims_and_evidence": "The claims made in the paper are generally well-supported by clear and convincing evidence. The authors provide an approach to optimizing LLM judges through extensive experimentation and analysis.", "methods_and_evaluation_criteria": "In general, the proposed method and evaluation criteria make sense for the problem.", "theoretical_claims": "This paper does not include any theoretical claims. It is more like an experimental report.", "experimental_designs_or_analyses": "I think most of the experimental designs of this paper are sound. I only have one question about the effectiveness of multi-objective optimization. The authors claim that their multi-objective and multi-fidelity optimization approach significantly reduces the cost of tuning LLM judges while maintaining or improving performance. However, the paper could benefit from a more detailed comparison with other optimization techniques to further validate the superiority of their approach.", "relation_to_broader_scientific_literature": "The key contributions of the paper are closely related to the broader scientific literature on LLM-as-Judge and prompt engineering.\nTheir analysis also reveals several insights of using LLM judges:\n1. Without surprise, the model used for the LLM judge is the most impacting hyperparameters and larger is generally better. \n2. The output format used to obtain judge preferences plays a big role.\n3. Increasing temperature negatively impacts performance.\n4. Averaging the judgement after evaluating a pair of outputs in both orders gives a performance boost.\n5. Providing an example helps the judge provided that a large model is used as smaller models gets confused by this extra information.\n6. Asking the judge to provide an explanation, or its answer hurts performance.\n7. Using JSON does not impact much the performance.", "essential_references_not_discussed": "I think there are several related works that need further discussion, especially the works on automatic prompt optimization. It seems to me that the authors completely ignore the works related to this relevant topic.\n1. Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments (EMNLP 2024): a prompt optimizer for bridging the gap between LLM evaluators and human judgments.\n2. Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators (COLM 2024): an uncertainty-guided search-based rank aggregation method for LLM judges.", "other_strengths_and_weaknesses": "My primary concern is that the authors claim that their optimized judges are cost-effective and accessible due to the use of open-weight models. However, they do not address the potential challenges of maintaining performance over time as new LLMs or datasets emerge. The adaptability of their optimized judges to future changes in LLM technology or evaluation standards could be a potential area of concern. Do the insights summarized above (prompt design, decoding temperatures, etc.) still hold for later LLMs, such as reasoning LLMs such as OpenAI o1 or QwQ (DeepSeek R1 is released on Jan 2025, it is OK that this paper does not test it)?\n\nIt seems to me that the methodology appears to rely heavily on brute-force search across configurations without presenting a more principled approach to parameter optimization. The paper does not adequately address whether the identified optimal configurations generalize or are merely artifacts of the specific LLMs being evaluated, also this paper ignored the related works on automatic prompt optimization (see the first paper in the section of \"Essential References Not Discussed\").", "other_comments_or_suggestions": "No", "questions_for_authors": "See the Other Strengths And Weaknesses above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rxoIahovgx", "reviewer": "Reviewer_8kjw", "summary": "This paper proposes an efficient way to finetune LLM judges via a multi-objective multi-fidelity approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We are delighted to hear that you found the results convincing. \n\nPlease find our answers below to the three points raised in your review.\n\n**The number 1/1000 in the title does not have clear support in the main text. Therefore, it should be replaced with non-quantitative words like \"lower\".**\n\nIn the second page of the main text, line 106 we made this statement: \n“We estimate that our approach costs approximately \\\\$2k to search through 4 480 judge configurations and that evaluating the same number of judges using Alpaca-Eval or Arena-Hard methodology would cost around \\$2M, see B.3 for details.”\nThis is where the 1/1000 numbers came. We will avoid the footnote to make it easier to find.\n\n**How does the prompt template look like when the prompt hyperparameters include \"provide confidence\" (an option included in Page 5) ?**\n\nThe prompt to ask for confidence looks like this, thank you for the callout, we added this point in the appendix as we realized that this prompt for this hyperparameter was missing.\nYour output should follow this format:\n\n```\nanswer: <your answer to the user prompt>\nconfidence: <a number between 0 and 1 to indicate your confidence>\nbest_assistant: <either \"A\" or \"B\" to indicate which Assistant gave the best answer>\n```\n\n**Can you provide the resulting prompt template after judge tuning in one of your experiment setting?**\n\nOf-course, here is the prompt found for the middle size LLM. It is a great suggestion we will add it to the appendix as it is revealing for a reader.\n\n```\nYou are a highly efficient assistant, who evaluates and selects the best large language model based on the quality of their responses to a given instruction.\nYou will be shown one instruction and the output of Assistant A and Assistant B and will have to decide which one was best.\nMake sure to not over-confidently prefer one assistant or the other and also make sure to not bias your preference based on the ordering or on the length of the answers.\n\n<|User Prompt|>\n{USER_PROMPT}\n\n<|The Start of Assistant A's Answer|>\n{ANSWER_A}\n<|The End of Assistant A's Answer|>\n\n<|The Start of Assistant B's Answer|>\n{ANSWER_B}\n<|The End of Assistant B's Answer|>\n\n# Your output\n\n## Format description\nYour output should follow this format:\n\\```\nanswer: <your answer to the user prompt>\nscore_A: <a number between 0 and 10 to indicate the quality of Assistant A's answer>\nscore_B: <a number between 0 and 10 to indicate the quality of Assistant B's answer>\n\\```\n\n## Your output, do not repeat the input above\n```", "claims_and_evidence": "The claim is supported by experiments on three models and three datasets. Therefore the result is convincing.", "methods_and_evaluation_criteria": "No significant flaws in method and evaluation.", "theoretical_claims": "Not applicable for this paper.", "experimental_designs_or_analyses": "No significant problems in experiments or analyses.", "relation_to_broader_scientific_literature": "This paper pushes forward the research in LLM judge tuning.", "essential_references_not_discussed": "No missing reference found.", "other_strengths_and_weaknesses": "No additional strength or weakness.", "other_comments_or_suggestions": "The number 1/1000 in the title does not have clear support in the main text. Therefore, it should be replaced with non-quantitative words like \"lower\".", "questions_for_authors": "1. How does the prompt template look like when the prompt hyperparameters include \"provide confidence\" (an option included in Page 5) ?\n2. Can you provide the resulting prompt template after judge tuning in one of your experiment setting?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jTqRAaXMz7", "reviewer": "Reviewer_tVtX", "summary": "This paper aims to broadly profile the various factors impacting judge LLM performance and results, systematically analyzing the impact of factors related to prompt, hyperparameter selection, answer extraction method, and model design. It adopts a cost-effective approach to minimize search overhead while preserving performance. The paper identifies how to improve over previous SOTA baselines, presenting configurations which notably boost judge LLM performance in relevant downstream settings.\n\n---\n**Update after rebuttal**\n\nThanks for the authors’ response, I have reviewed the rebuttal and discussions and will be keeping my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for reviewing all appendices. We are delighted that you found the analyses relevant and insightful across all settings.\n\nPlease find our answer for the two points raised in your review.\n\n**Experiments could consider finer gradations of inference temperature.**\n\nIndeed you are right that the optimal temperature may not be exactly covered with the grid we selected. Our intent was to analyse the range of temperatures that works to guide practitioners when considering judges. We think that just getting the range may be enough for the temperature to get good performance (e.g. a low temperature is enough to get most of the performance; at least it is not the dominant hyperparameter to select as opposed to the base model or the prompting strategy).\n\n**Confidence elicitation in LLMs is generally dependent on prompt wording and the range of confidence scores the model is asked to produce. How did the paper elicit confidence and were variations in elicitation strategy considered?**\n\nWe totally agree, the range is indeed important, and LLM judges performance indeed differ when their performance is graded between [1-5] or [0-10] for instance. For confidence, we elicited it by asking for a score in [0, 1]. We agree that setting a score in [1-5] for instance may change the performance with respect to this hyperparameter. We tried other ranges in smaller scale experiments but we did not observe a significant effect of this hyperparameter on performance so we decided on a simple option.\n\nHere is the prompting corresponding to confidence, we will add it to the appendix.\nYour output should follow this format:\n```\nanswer: <your answer to the user prompt>\nconfidence: <a number between 0 and 1 to indicate your confidence>\nbest_assistant: <either \"A\" or \"B\" to indicate which Assistant gave the best answer>\n```", "claims_and_evidence": "Claims are generally supported in the paper.", "methods_and_evaluation_criteria": "The paper develops a comprehensive evaluation framework, considering a diverse array of prompting strategies in addition to having good coverage of variations related to model selection, to profile the impact of various decisions involved in using LLMs as judges.", "theoretical_claims": "The theory presented in the paper is sound.", "experimental_designs_or_analyses": "Experiments utilize a range of model families and sizes, with sensible selections of inference hyperparameters. Analyses are relevant and insightful across all settings.", "relation_to_broader_scientific_literature": "The paper provides a complementary set of findings to existing work concerning judge LLMs, systematically characterizing the impact of model selection and other experimental factors on judge LLM performance. Recommendations provided in the paper serve to guide design choices in future research involving judge LLMs.", "essential_references_not_discussed": "N/A — To my understanding the paper sufficiently covers relevant key works for the LLM-as-a-Judge paradigm.", "other_strengths_and_weaknesses": "Strengths\n* The paper exhibits good organization and writing quality. Figures are well-presented and easy to read and understand. \n* The paper answers an important need in the community toward understanding and improving use of LLMs as judges in various settings.\n\nWeaknesses\n* Experiments could consider finer gradations of inference temperature.\n* Confidence elicitation in LLMs is generally dependent on prompt wording and the range of confidence scores the model is asked to produce. How did the paper elicit confidence and were variations in elicitation strategy considered?", "other_comments_or_suggestions": "Minor formatting suggestions\n* Bullet points of key contributions in Section 1 utilize no period for the first three bullets versus use of a period in the fourth bullet.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a cost-effective approach to systematically tune hyperparameters of LLM-as-a-judge for evaluating other LLMs, significantly reducing the required resources. The authors leverage a multi-objective, multi-fidelity optimization framework to efficiently search through 4,480 configurations, considering factors like model choice, prompting strategy, inference parameters, and output parsing method. By evaluating on datasets including Alpaca-Eval, Arena-Hard, and LMSys, the optimized judges outperform existing methods in terms of accuracy and cost-efficiency while relying solely on open-weight models. 

The paper answers an important need in the community toward understanding and improving use of LLMs as judges in various settings. The use of open-weight, zero-shot models increases accessibility and encourages reproducibility, addressing critical limitations of prior work relying on expensive closed models. The analysis of hyperparameter importance (prompt style, inference parameters) is thorough and insightful, providing valuable practical guidelines.

Please add the missing references about prompt optimization in your related work section.
Decision: Accept (poster)