Self-Taught Evaluators

## Abstract

Abstract Model-based evaluation is at the heart of successful model
development –
as a reward model for training, and as a replacement for human evaluation.
To train such evaluators, the standard approach is to collect a large amount of human preference judgments over model responses, which is costly and the data becomes stale as models improve.
In this work, we present an approach that aims to improve evaluators without human annotations , using synthetic training data only. Starting from unlabeled instructions, our iterative
self-improvement scheme generates contrasting model outputs and
trains an LLM-as-a-Judge to produce reasoning traces and final judgments, repeating this training at each new iteration using the improved predictions. Without any labeled preference data, our Self-Taught Evaluator can improve
a strong LLM (Llama3-70B-Instruct) from 75.4 to 88.3 (88.7 with majority vote) on RewardBench.
This outperforms commonly used LLM judges such as GPT-4 and matches the performance of the top-performing reward models trained with labeled examples.

## 1 Introduction

Large language models (LLMs) rely on strong evaluators
at every stage of the
development lifecycle.
They are used at training time as reward models to align with human preferences Bai et al. (); Ouyang et al. () or for iterative self-improvement Yuan et al. (), and at inference time as an alternative to human evaluation
Li et al. (); Chiang and Lee (); Wang et al. (); Liu et al. ().
Improvements in evaluation capabilities will thus clearly benefit this entire
workflow –
including empowering the scientific research process itself as we aim to develop better overall techniques.

Building such strong evaluator models usually relies on large amounts of high-quality preference data from human annotation over model responses, which can be costly and time-consuming to collect, as it requires expert annotation for challenging tasks (e.g., coding and mathematics). This dependency on human-generated data poses significant challenges for scaling to new tasks or evaluation criteria. Furthermore, as new models inevitably improve over older ones, these existing annotations will typically become outdated, as the judgments are based on annotations of older,
less performant, model responses.

In this work, we instead explore an iterative self-training approach which uses no human annotated preferences in the training loop, relying purely on synthetically generated data.
Given a seed model, our method first uses prompting to generate contrasting synthetic preference pairs for a given input, such that one response is designed to be inferior to the other.
Next, using the model as an LLM-as-a-Judge, we generate reasoning traces and judgments for these pairs, which we can label as correct or not given our synthetic preference pair design.
After training on this labeled data we obtain a superior LLM-as-a-Judge, from which we can then iterate the whole process in order for it to self-improve.

In our experiments, starting from Llama-3-70B-Instruct, the proposed method improves the accuracy on RewardBench (Lambert et al., ) from 75.4 to 88.7 (with majority vote, or 88.3 without).
This matches or outperforms the performance of reward models derived from the same Llama-3-70B-Instruct model that uses human annotations, for example using the HelpSteer2 dataset (Wang et al., ) of 10k annotations
achieves a performance of 85.6 using the same LLM-as-a-Judge setup.

Figure: Figure 1: Self-Taught Evaluator iterative training scheme.
Refer to caption: x1.png

## 2 Related Work

#### LLM-based Evaluators

While traditional evaluation benchmarks employ automated metrics that require a reference answer Wang et al. (); Rajpurkar et al. (), these types of benchmarks can pose severe limitations when evaluating open-ended or complex instructions where multiple valid answers are possible (e.g., creative writing and coding). Because human evaluation per response can be costly, many recent works have proposed LLMs as effective evaluators. These come in several flavors: as classifiers that output scores directly (Zhu et al., ; Wang et al., ) or via LLM-as-a-Judge prompting that can first generate a chain-of-thought in natural language, which helps provide explanations for judgments (Zheng et al., ). Responses
can also be scored alone (Kim et al., ) or pairwise relative to each other (Dubois et al., ; Li et al., ; Bai et al., ; Saha et al., ).
LLM evaluation shows great promise as a scalable proxy
for human raters, and in the case of LLM-as-a-Judge as an explainable proxy as well Ye et al. (); Zheng et al. (). However, many of these “off-the-shelf” evaluators demonstrate high variance across many tasks Bavaresco et al. (), indicating the need for improved methods.

#### Synthetic Data

Synthetic data has emerged as a promising solution for efficiently acquiring training examples and can be particularly valuable in settings where real-world data can be hard to access (e.g., weather data covering all conditions Lam et al. ()) or where correct annotations can be challenging to acquire (e.g., coding tasks Liu et al. ()). Additionally, synthetic data has the benefit of being easily customizable to specific requirements, such as different evaluation criteria or safety constraints Kim et al. (); El Emam et al. (); Howe et al. (). The use of synthetic data has been beneficial in model alignment Lee et al. (), improving the original model’s capabilities Yuan et al. (); Li et al. (); Yu et al. (); Li et al. (), and teaching the model new skills Schick et al. (); Lanchantin et al. (). In the context of evaluation, synthetic data has been used to measure tasks such as factuality Wei et al. (); Feng et al. (), safety Perez et al. (); Hubinger et al. (), coding Gu et al. (), and general instruction following Zeng et al. (), showing strong correlation with real human judgments. The West-of-n approach (Pace et al., ) has been used to improve reward models by
constructing preference pairs using the best and worst scoring pairs from an initial model. For LLM-as-a-Judge models specifically, synthetic responses have been generated prompting the LLM to produce a given quality response (Kim et al., ).

## 3 Method

We consider the setting of pairwise evaluation using the LLM-as-a-Judge approach (Zheng et al., ) that takes:

- •
an input (user instruction) $x$; and
- •
two possible assistant responses $y^{(A)}$ and $y^{(B)}$ to the user instruction $x$; and
- •
the evaluation prompt containing the rubric and asking to evaluate and choose the winning answer, see e.g., Figure 8.

The goal of the LLM-as-a-Judge model is to output a preference of which response $y$ is better: A or B.
In order to do this it is common to output, prior to the final judgment, a chain-of-thought (or “reasoning chain”), which is a set of steps generated in natural language that helps the model decide its final judgment.

Such models can be used as pairwise reward models to build training data for preference optimization, e.g., for training methods like DPO (Rafailov et al., ), Iterative DPO (Xu et al., ) and Self-Rewarding methods (Yuan et al., ).
They can also be used for evaluation; e.g., many popular benchmark leaderboards are built
by using a fixed LLM-as-a-Judge evaluation model Li et al. () such as GPT4 Achiam et al. ().

We propose a novel recipe for training such an evaluator.
Our overall method is an iterative training scheme that bootstraps improvements by
annotating the current model’s judgments using constructed synthetic data – so that the Self-Taught Evaluator is more performant on the next iteration.

Figure: Figure 2: Generating Synthetic Response Pairs. We use the following prompt template which is used to generate a “worse response” $y^{l}$. Given an instruction $x$ and baseline response $y^{w}$ generated by an instruction-following LLM as usual, this prompt is used to first generate a “noisy” version $x^{\prime}$ of the original instruction $x$, and then a best-attempt $y^{l}$ at responding to $x^{\prime}$. $y^{l}$ is then treated as a poor response to $x$, giving a preference pair $y_{i}^{w}\succ y_{i}^{l}$.

Our overall pipeline is thus as follows:

- •
Initialization: We assume access to a large set of human-written user instructions, e.g., of the type that is commonly collected in production systems, and an initial seed LLM.
- •
Instruction Selection: We next select a challenging, balanced distribution of user instructions from the uncurated set by categorizing them via LLM.
- •
Response Pair Construction: For each user instruction (example) we create a preference pair of two model responses (chosen & rejected), generating them via prompting such that the rejected response is likely of lower quality than the chosen response.
- •
Iterative Training: We then iterate the following two steps:
(i)
Judgment Annotation: For each example, we sample from the current model up to $N$ times LLM-as-a-Judge generated reasoning traces and judgments. If we find a correct judgment we add that example to our training set, otherwise we discard it.
(ii)
Model Fine-tuning: We fine-tune the model on the newly constructed training set which yields an updated model for the next iteration.

Note that in each iteration of training the size of the training set depends on the quality of the current model. We expect that as the model improves, the size of the training set will increase as well, as the model will be able to find more correct judgments, giving the model a kind of automatic curriculum.

We next describe each of these steps in detail.

### 3.1 Initialization

We assume we have access to a pool of user instructions $\{x_{i}\}$. Each sample $x_{i}$ can either be one single text instruction or a multi-turn dialog history of turns between the user and the assistant, with the last turn being an instruction or question from the user.
Instructions typically involve different skills such as general knowledge and reasoning, coding, safety, and mathematical reasoning.

### 3.2 Instruction Selection

Given a pool of human-written user instructions, there may be a large degree of noise, as well as an imbalance in terms of topic, variety, difficulty, and ability of the model to answer.
We therefore aim to select a subset of instructions to generate high-quality synthetic responses and judgments that can be further used for training.

We classify each input using an LLM into a given category, for example coding, reasoning, brainstorming, etc. The precise prompt we use is given in Figure 7.
We are then free to select data from within those categories, and to discard certain categories not deemed to be useful for training.

### 3.3 Response Pair Construction

For each input $x_{i}$ in our curated training pool, we next generate preference data involving two responses $y_{i}^{(w)}$ and $y_{i}^{(l)}$ where $w$ is expected to be preferable (winning) over $l$ (losing).
We achieve this by generating the data in a synthetic manner without using human annotation.

Given the instruction $x_{i}$, we first prompt an instruction-following LLM to generate a baseline response $y_{i}^{w}$ as usual.
We then prompt the LLM to generate a “noisy” version of the original instruction $x^{\prime}_{i}=\phi(x_{i})$. We do this using the prompt template given in Figure 2, where we ask
to “generate a modified instruction that is highly relevant but not semantically identical to
the instruction above from the user.” We then prompt the LLM for a high-quality response $y_{i}^{l}$ to $x^{\prime}_{i}$, which would not be a good response for $x_{i}$.
This yields a synthetic preference $y_{i}^{w}\succ y_{i}^{l}$ for the original input $x_{i}$.

This paired data is then used to construct training examples:

$$ $(x_{i},y_{i}^{(A)},y_{i}^{(B)})$ $$

where
we randomize the order of whether the winner is $w=A$ or $w=B$, which is important to deal with position bias for LLM-as-a-Judge inference.

### 3.4 Judgment Annotation

Our LLM-as-a-Judge model is used to generate evaluation judgments (reasoning chains and verdicts)
$\{j_{i}\}$ for each training example
$e_{i}\coloneq(x_{i},y_{i}^{(A)},y_{i}^{(B)})$
in the following manner: for a given input $e_{i}$, we collect $N$ diverse evaluations $\mathcal{J}\coloneq\{j_{i}^{1},\ldots,j_{i}^{N}\}$ by sampling from the model. We then apply rejection sampling to filter $\mathcal{J}$ by removing $j_{i}^{n}$ when the final verdict disagrees with the ground truth labeling, derived from Subsection 3.3. We then select a single correct reasoning chain and verdict at random from the pool of correct solutions.
If no such judgment exists ($\mathcal{J}$ is empty) then we discard the example.

This now allows us to construct our final training examples of synthetic preferences for fine-tuning:

$$ $((x_{i},y_{i}^{(A)},y_{i}^{(B)}),j_{i}).$ $$

### 3.5 Model Fine-tuning (Iterative Training)

Our Self-Taught Evaluator (LLM-as-a-Judge model) is first initialized with the seed LLM. The model is then trained in an iterative manner. At each iteration,
we annotate the training examples with judgments as described in Subsection 3.4 using the current model,
giving training examples $\{(x_{i},y_{i}^{(A)},y_{i}^{(B)},j_{i})\}$. These are used to train the next iteration’s model by fine-tuning. Note that we initialize from the seed model at each iteration.

## 4 Experiments

### 4.1 Experimental Setup

#### Training.

Our initial model $M_{0}$ is initialized from Llama3-70B-Instruct. In each iteration $i=1,\dots T$, we use model $M_{i-1}$ from the previous iteration to generate synthetic preferences followed by judgments on the training data, and then fine-tune Llama3-70B-Instruct again. We use fairseq2 library (Balioglu, ) for instruction finetuning and vLLM (Kwon et al., ) for inference. During training the negative log-likelihood loss is only applied to the evaluation part, i.e., $j_{i}$ of the training example. Training hyperparameters are provided in Table 7. Model selection is done using a combination of pairwise judgment accuracy and position bias computed over the held out set. Sampling parameters used for generations are provided in Table 8.

#### Instructions and Responses.

We start with a large pool of human-written instructions $\{x_{i}\}$ from the WildChat dataset Zhao et al. (). To perform prompt selection, we annotate the category of each instruction with the Mixtral 22Bx8 Instruct model, using the template in Figure 7 and select 20,582 examples in the reasoning category, as we expect these to be challenging inputs. For the selected inputs we generate synthetic responses $y_{i}^{w}$ and
$y_{i}^{l}$ using Mixtral 22Bx8 Instruct following Subsection 3.3 and
Figure 2.

#### Judge Annotation.

For each training example, we sample $N=15$ judgments from the model $M_{i-1}$
and retain one positive sample $j_{i}$ per example.
Then over the entire dataset we sample the same amount of examples from different labels (“A is better”, “B is better”) to ensure balanced training.
Judgements for training $M_{0}$ were sampled from Mixtral 22Bx8 Instruct, and from the Llama model being trained in all subsequent iterations.

The training data is constructed as (<system prompt>, $\{(x_{i},y_{i}^{(A)},y_{i}^{(B)},j_{i})\}$). We use the standard system prompt from MT-Bench and RewardBench as shown in Figure 8.

#### Majority Vote Inference.

As LLM-as-a-Judge uses chain-of-though reasoning chains generated by the LLM followed by a verdict, it is known that majority vote inference can yield improvements in these cases (Wang et al., ). At inference time when evaluating final performance we sample generations $N$ times, and take the final judgment to be the most common verdict.

### 4.2 Other Data Sources

To understand the effectiveness of the
proposed method, we generate synthetic judgments
using the same approach but based on the following data sources:

- •
HelpSteer2 Wang et al. (). We generate evaluations conditioned on the scores of helpfulness, correctness, coherence, complexity and verbosity provided the dataset. We use the aggregated score to derive the ground truth preference for each example using the recommended weighting $[0.65,0.8,0.45,0.55,-0.4]$(^1^11Recommended weighting was taken from https://huggingface.co/nvidia/Llama3-70B-SteerLM-RM.).
- •
GSM8K Cobbe et al. (). We sample from an instruction-following model multiple times to get $y^{w}$ when the final solution agrees with the ground truth and $y^{l}$ vise versa.
- •
Coding instructions from WildChat. Similar to the “reasoning” prompts we selected from WildChat used in the main experiment, we also experimented with prompts annotated with the “Coding” category.
- •
hh_rlhf Bai et al. (). We generate evaluations on the prompts and responses provided in the “harmless_base" training split. Then we take human preferences provided by the dataset as ground truth to perform rejection sampling to construct
judgments.

### 4.3 Evaluation

We evaluate the accuracy of our
Self-Taught Evaluator model
on the following benchmarks:

- •
RewardBench Lambert et al. (). We use the standard evaluation protocol provided by the leaderboard.
- •
MT-Bench Zheng et al. (). We report agreement rate with human judgments when examples with ties are excluded.
- •
HelpSteer2 Wang et al. (). We evaluate on the validation split.

## 5 Results

### 5.1 RewardBench

Results on RewardBench are given in Table 4. We find that our
Self-Taught Evaluator which is trained iteratively on synthetic data without any annotated preference labels significantly improves over the seed Llama3-70B-Instruct model, matching top-performing reward models trained with labeled data.
Our approach improves its results across training iterations, and achieves an overall score of 88.3 on iteration 5, while the seed model it starts from obtains 75.4. Training an LLM-as-a-Judge in a similar manner starting from the same seed using the labeled HelpSteer2 data we only obtain 85.6, hence we obtain superior performance without using human labeled data.
Compared to the seed model,
we observe improvements using our approach in evaluating instructions in the Chat Hard, Safety and Reasoning categories, while being worse on the easier Chat category – perhaps because our unlabeled training data focused the model on harder examples.

**Table 1: RewardBench Results. Our Self-Taught Evaluator trained on synthetic data without any human annotated preference labels matches top-performing reward models trained with labeled data. Models marked with (*) are taken from the RewardBench leaderboard.**
| Model | Overall | Chat | Chat Hard | Safety | Reasoning |
| --- | --- | --- | --- | --- | --- |
| Llama-3-70B-Instruct (seed) | 75.4 | 97.6 | 58.9 | 69.2 | 78.5 |
| Self-Taught Evaluator, trained on synthetic data only |  |  |  |  |  |
| Iteration 1 | 83.9 | 98.3 | 69.0 | 85.7 | 82.6 |
| Iteration 2 | 86.0 | 97.5 | 75.4 | 89.5 | 81.7 |
| Iteration 3 | 87.5 | 97.2 | 79.1 | 89.7 | 83.9 |
| Iteration 4 | 87.7 | 98.0 | 80.3 | 90.5 | 82.2 |
| Iteration 5 | 88.3 | 96.6 | 84.2 | 91.5 | 81.0 |
| w/ majority voting using 32 samples | 88.7 | 96.9 | 84.0 | 91.5 | 82.5 |
| Baselines with Labeled Data |  |  |  |  |  |
| Llama-3-70B-Instruct w/ HelpSteer2, LLM-as-a-Judge | 85.6 | 96.9 | 70.0 | 88.8 | 86.7 |
| nvidia/Llama3 70B RM with HelpSteer2, classifier * | 88.8 | 91.3 | 80.3 | 92.8 | 90.7 |
| Other SoTA LLM-as-a-Judge baseline models |  |  |  |  |  |
| GPT4 0125 * | 84.3 | 95.3 | 74.3 | 87.2 | 86.9 |
| Gemini 1.5 Pro 0514 * | 88.1 | 92.3 | 80.6 | 87.5 | 92.0 |
| Llama3.1-405B-Instruct | 83.7 | 98.0 | 75.1 | 74.7 | 86.8 |
| Llama3.1-70B-Instruct | 82.2 | 97.8 | 69.7 | 76.3 | 85.2 |

#### Improving results further with majority voting

As also shown in Table 4, with 32-sample majority voting, our third iteration of Self-Taught Evaluator model reaches an overall performance of 88.7 on RewardBench, outperforming many other existing reward models.

### 5.2 MT-Bench

We report results on MT-Bench in Table 4.
Unlike RewardBench, the MT-Bench dataset contains tie votes (A and B are considered equally good).
Since our models are trained to give binary decisions, we only report the agreement on non-tie examples. For each pair of responses A and B, we test two orders: where response A appears first and response B appears first, and average the results.
We find that our Self-Taught Evaluator again outperforms the Llama3-70B-Instruct seed model, and is on par or slightly outperforms
GPT4-0125.

### 5.3 HelpSteer2

Results on the HelpSteer2 validation set are given in Table 4.
We report the average accuracy of two orders and three seeds by swapping the response order in a similar manner, as well as reporting both orders separately (right answer first or second) to test for position bias. We further compute the position-consistent accuracy, treating a judgment as incorrect when a model has different predictions on the two orderings. We use the human labels from the Helpsteer2 dataset and treat the response with higher summed scores as the better response. We find that our
Self-Taught Evaluator method
improves both average accuracy and position-consistent accuracy compared to the seed Llama-3-70B-Instruct model.

## 6 Ablations and Analysis

### 6.1 Synthetic Data from Other Sources

In Table 4, we compare Self-Taught Evaluator models trained on synthetic preferences constructed from different sources. We found data sources focusing on different skills, such as coding, mathematical reasoning, etc. are all effective in turning a strong instruction-following LLM into a strong LLM-as-a-Judge. Intuitively, we find that data sources generally improve the categories in RewardBench that are related to their distribution.

Figure: Figure 3: Distribution of curated training set of selected instructions compared to the full WildChat dataset.
Refer to caption: extracted/5781816/figures/wildchat_input_length.png

**Table 5: Iterative training with labeled data also shows improvement on RewardBench. However, it does not outperform iterative training with synthetic preferences .**
| Model | Overall | Chat | Chat Hard | Safety | Reasoning |
| --- | --- | --- | --- | --- | --- |
| Llama-3-70B-Instruct (seed) | 75.4 | 97.6 | 58.9 | 69.2 | 78.5 |
| Self-Taught Evaluator, trained on labeled HelpSteer2 preferences |  |  |  |  |  |
| Iteration 1 | 85.6 | 96.9 | 70.0 | 88.8 | 86.7 |
| Iteration 2 | 86.3 | 96.1 | 72.4 | 91.1 | 85.7 |
| Iteration 3 | 87.0 | 95.0 | 74.2 | 91.2 | 87.8 |
| Iteration 4 | 87.0 | 94.1 | 77.2 | 91.6 | 85.1 |

### 6.2 Synthetic Bad Response Generation

In our experiments we generate synthetic data by first generating a similar instruction, and then a good response for the similar instruction – with the aim that this will be a bad response for the original instruction. An alternative is to just prompt an LLM to generate a bad response to the original instruction directly.
We use the prompt template given in Figure 10 and otherwise conduct training as before on the same set of reasoning-based instructions. This approach obtains a RewardBench overall score of
80.7, which still works – but is
worse than using our proposed approach, which achieves 83.8.

### 6.3 Comparison of Synthetic Data with Human Annotated Data

We conducted the same iterative training using labeled preference data from HelpSteer2 Wang et al. (), rather than synthetic data. On RewardBench, as is shown in Table 6, the improvement from each iteration is smaller and the final model did not outperform iterative training on synthetic preferences. We note that these experiments use data to train an LLM-as-a-Judge. Other results in the literature have used the HelpSteer2 to train classifier-based reward models with slightly better results on RewardBench, e.g., obtaining 88.8 using Llama-3-70B, see Table 4.

### 6.4 Iterative Training by Initializing from Labeled Data

We further explore how to utilize labeled data in our pipeline. We first finetune a model on Helpsteer2 Wang et al. () and use this model to generate judgements. In this way, we obtain synthetic data by utilizing a model finetuned on labeled data. We conducted iterative training and present results in Table 12. We observed good performance compared to the seed model (Llama-3-70B-Instruct), however it does not clearly outperform conducting iterative training with unlabeled data alone.

### 6.5 Combining Synthetic and Human Labeled Preference Data

We compare how combining synthetic preference data with human labelled preference data affects model performance.
In particular, we combine synthetic preferences generated from reasoning WildChat prompts with the human labeled HelpSteer2 dataset (train split) and report performance in Table 6. We compare to first-iteration models trained on single data source, and select the best checkpoint for joint training using the validation split of HelpSteer2 and holdout set of synthetic preferences (in-distribution), as well as safety and code synthetic preferences (out-of-distribution). We then report evaluation results on RewardBench. The results show that overall the models retain strong performance across different data mixing weights, with slight improvements on overall accuracy.

### 6.6 Instruction complexity

We analyze the length distribution of the curated training set of selected instructions
in Figure 6. The dataset has a long-tail distribution of input length, with most of the examples less than 500 tokens. In contrast, the full dataset (i.e., the full data before the instruction selection step of Subsection 3.2) has a cluster of very long instructions, containing content such as long-form coding instructions or transcripts.

We further instruct Llama-3-70B-Instruct to infer the complexity (using a score of 1–5) and category of each input instruction, as well as the length of the expected output, following the procedure in Yuan et al. ().
From Figure 6 and Figure 6, we see that the curated dataset has more complex instructions involving logical reasoning/science whereas the full dataset has a greater proportion focused on relationships and entertainment. Finally, in Figure 6 we see that the anticipated length of the response is higher for the full dataset than the curated one, perhaps because of the greater frequency of lengthy, and sometimes repetitive instructions.

## 7 Conclusion

We present a scalable approach to build a strong generalist evaluator to perform model-based evaluation of LLM outputs. Our method constructs synthetic preferences over pairs of responses without using any human annotation. Our Self-Taught evaluator with iterative training over these synthetic preferences greatly boosts the accuracy of a strong seed LLM (Llama3-70B-Instruct) as an evaluator, from 75.4 to 88.7 on RewardBench, a new state-of-the-art for generative LLM-as-a-Judge methods.

## 8 Limitations

Generative LLM-as-a-Judge models usually have longer outputs and thus higher inference cost than reward models that simply output a score, as LLM-as-a-Judge typically first generates a reasoning chain. Further, we have used relatively large LLMs in this work (70B parameters) and made no study of whether our approach works on smaller models.
Since we use a seed model to generate first synthetic preferences during our iterative training scheme, one of the assumptions is that the model is capable of generating reasonable evaluations. Thus, our approach is limited by having a capable instruction fine-tuned model which is already reasonably aligned to human (or legal/policy) preferences.
Furthermore, we only investigated and reported metrics involving evaluation accuracy improvements, rather than computational requirement concerns.
We also only investigated pairwise evaluation, i.e., comparing two responses, whereas it is also possible to use LLM-as-a-Judge models (or any other model) to evaluate the quality of single responses, e.g., giving them a score out of 5 or 10, rather than a pairwise A vs B judgment. We leave evaluating single responses to future work.

## 9 Acknowledgements

We thank Jing Xu and Janice Lan for discussions and support in the project overall, and Can Balioglu for his feedback and support in LLM training using the fairseq2 (Balioglu, ) library. We thank Nathan Lambert for the help with RewardBench, and Yuntian Deng and Yejin Choi for the help with WildChat.

## Appendix A Appendix

### A.1 Prompt Templates

We provide the prompt templates used for annotating and selecting instructions (Figure 7), annotating judgments with synthetic preferences (Figure 8), and generating ablation synthetic preference data with bad responses (Figure 10). Figure 9 illustrates an training example based on synthetic preference data.

Figure: Figure 7: Prompt template for Selecting Instructions. We prompt an instruction following model to annotate the category of each instruction in order to curate our training data instructions.

Figure: Figure 8: Prompt template for Judgment Annotation. This is the same prompt as used in MT-Bench and RewardBench.

Figure: Figure 9: An illustrative example of judgment generation given an instruction and two responses.

Figure: Figure 10: Generating a Bad Response for an Instruction. This approach is an ablation compared to our proposed approach described in the main paper.

### A.2 More Training and Evaluation Details

We include training hyper-parameters in Table 7 and sampling parameters in Table 8.

**Table 7: Training hyper-parameters used during fine-tuning.**
| Name | Value |
| --- | --- |
| max_seq_len | 4096 |
| max_num_tokens | 8192 |
| model | llama3_70b_instruct |
| dtype | bfloat16 |
| data_parallelism | fsdp |
| tensor_parallel_size | 8 |
| activation_checkpointing | true |
| lr | 1.0e-06 |
| betas | 0.9, 0.95 |
| final_lr_ratio | 0.2 |
| weight_decay | 0.1 |
| num_lr_warmup_steps | 100 |
| gradient_accumulation | 1 |
| max_num_data_epochs | 2 |
| checkpoint_every_n_steps | 100 |
| seed | 2 |

**Table 8: Sampling parameters (temperature and top p) used during generations at each stage of training and evaluation.**
| Stage | Generation for | Temperature | Top p |
| --- | --- | --- | --- |
| Train | Judgment | 0.7 | 0.9 |
| Eval | MT-Bench | 0.0 | 1.0 |
| Eval | Reward Bench (RB) | 0.0 | 1.0 |
| Eval | RB w/ maj voting | 0.7 | 0.9 |
| Eval | Helpsteer 2 valid | 0.7 | 0.9 |

### A.3 Position Order Evaluation on RewardBench

We notice that when we evaluate generative models on RewardBench, the order of two responses in each example is not fixed. More specifically, for each example, the winning response ($y^{w}$) can be randomly placed before or after the losing response ($y^{l}$). Generative models may output different judgements when the order of responses changes. Thus, we analyze how the performance varies when different seeds are used to decide response order. In Table 9, we test our model from the 5th iteration of training on RewardBench with the response order randomly shuffled, as well as two extreme cases where the winning answer always appear first or last. We recommend to report the average performance (88.3 for our 5th iteration) of “$y^{w}$ always first” and “$y^{l}$ always first” as it fairly considers both orders.

**Table 9: Average accuracy on RewardBench when order of responses changes.**
| Seed | Average Accuracy |
| --- | --- |
| 1 | 88.9 |
| 11 | 88.4 |
| 111 | 88.6 |
| 1111 | 88.7 |
| 11111 | 88.3 |
| $y^{w}$ always first | 85.5 |
| $y^{l}$ always first | 91.1 |

### A.4 Using Different Models for Training Data Generation

In Table 10 we present evaluation on RewardBench of models finetuned on different training data. Note in our Self-Taught Evaluator approach we can use different LLMs to generate responses and judgements. Specifically, we try using Mixtral 22Bx8 Instruct or Llama-3-70B-Instruct in various combinations. We then finetune the Llama-3-70B-Instruct model and test on RewardBench. As shown in Table 10, the model finetuned on data generated by using the Mixtral 22Bx8 Instruct model to judge Mixtral 22Bx8 Instruct model generated responses achieves the best performance.

**Table 10: Performance on RewardBench of models finetuned on different training data.**
| Model | Overall | Chat | Chat Hard | Safety | Reasoning |
| --- | --- | --- | --- | --- | --- |
| Llama-3-70B-Instruct (seed) | 75.4 | 97.6 | 58.9 | 69.2 | 78.5 |
| Self-Taught Evaluator, trained on synthetic data only |  |  |  |  |  |
| Mixtral judge Mixtral | 83.9 | 98.3 | 69.0 | 85.7 | 82.6 |
| Llama3.0 judge Llama3.0 | 81.4 | 97.2 | 66.0 | 85.0 | 77.5 |
| Llama3.0 judge Mixtral | 80.0 | 97.5 | 70.0 | 72.8 | 79.4 |

**Table 11: MT-Bench Per-category Results. Our Self-Taught Evaluator trained on synthetic data without any human annotated preference labels performs on par with GPT-4 judgments.**
| Model | Overall | writing | stem | coding | math | humanities | reasoning | roleplay | extraction |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Llama-3-70B-Instruct (seed) | 77.8 | 70 | 76.9 | 73.8 | 80 | 79.85 | 78.8 | 78.8 | 85.1 |
| Self-Taught Evaluator, trained on synthetic data only |  |  |  |  |  |  |  |  |  |
| Iteration 1 | 78.95 | 71.15 | 78.95 | 76.6 | 81.65 | 80.25 | 82.3 | 80.7 | 80.25 |
| Iteration 2 | 78.65 | 69.6 | 77.9 | 82.55 | 79.15 | 82.2 | 80.8 | 77.6 | 79.8 |
| Iteration 3 | 78.9 | 70.4 | 78.6 | 79.35 | 79.55 | 82.9 | 82.3 | 77.9 | 80.7 |
| Iteration 4 | 77.45 | 71.15 | 77.25 | 75.8 | 73.3 | 82.6 | 81.3 | 78.85 | 79.35 |
| Iteration 5 | 78.9 | 68.45 | 78.2 | 81.75 | 82.5 | 82.25 | 81.35 | 75.15 | 83.75 |
| w/ majority voting @ 32 | 79.45 | 68.45 | 78.55 | 82.95 | 83.75 | 82.9 | 82.8 | 76.35 | 81.6 |
| Other SoTA LLM-as-a-Judge baseline models |  |  |  |  |  |  |  |  |  |
| GPT4-0125 | 79.15 | 70.4 | 79.9 | 82.9 | 82.1 | 80.55 | 80.8 | 77 | 80.7 |

**Table 12: Iterative training on synthetic data generated by a model that is first fine-tuned on labeled data (Helpsteer2).**
| Model | Overall | Chat | Chat Hard | Safety | Reasoning |
| --- | --- | --- | --- | --- | --- |
| Llama-3-70B-Instruct (seed) | 75.4 | 97.6 | 58.9 | 69.2 | 78.5 |
| Self-Taught Evaluator, trained on synthetic data generated by a finetuned model (Helpsteer2) |  |  |  |  |  |
| Iteration 1 | 87.0 | 95.8 | 75.8 | 90.7 | 85.8 |
| Iteration 2 | 86.6 | 92.2 | 77.4 | 91.2 | 85.8 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: -Please use \citep in place of \cite where appropriate to ensure citation consistency.

- I would consider revising the score if the paper includes a more comprehensive discussion of potential drawbacks, along with a comparison to other LLM-as-a-judge baselines.
soundness: 3
strengths: - The self-taught framework presented in this paper is a quite novel. It uses an iterative self-improvement process that enables the model to independently refine its judgment skills, offering valuable insights for future research in self-supervised evaluation methods for LLMs.

- The paper is well-structured and clearly written.

- The proposed self-taught method could effectively reduces the dependency on human labeling by enabling the model to generate its own synthetic data and preference labels. It offers a scalable solution for training evaluation models for LLMs.

- The empirical results are highly promising. On competitive benchmarks like RewardBench and MT-Bench, the self-taught models match or even exceed the performance of traditional reward models that rely on human annotations, highlighting the method’s strong potential for practical applications.
summary: This paper presents a novel approach to training LLMs to act as **evaluators**, or "judges," without relying on human-labeled data. The key method involves a self-supervised, iterative training process where the model generates its own **synthetic data** and **preference labels**, using these to refine its evaluative skills over multiple rounds, resulting in curriculum learning. This process allows the model to progressively improve its judgment abilities without external human supervision. In tests on benchmarks like RewardBench and MT-Bench, the self-taught evaluator achieves competitive, sometimes superior, results compared to reward models trained with human annotations.
weaknesses: - The method has only been tested on one specific LLM variant (LLAMA3-70B-Instruct), making it unclear whether the approach generalizes well to other types of LLMs or models of different sizes and architectures.

- The process for generating contrasting synthetic preference pairs is relatively simple and lacks refinement. The prompt used to generate suboptimal responses often results in fairly static patterns, and it remains unverified whether the generated response $y^l$ is indeed worse than the original. There is limited discussion on corner cases or situations where synthetic data quality might be compromised. A more carefully crafted design for preference pairs could improve the model’s ability to distinguish subtle judgment differences.

- The paper provides limited comparison with other LLM-as-a-judge methods, relying primarily on comparisons with GPT-4-0125 while omitting other competitor models from existing literature in LLM-as-a-judge frameworks.

- The computational complexity of the iterative self-taught evaluation process is not fully discussed. Generally, this approach involves a curriculum learning process requiring multiple iterations, and further discussion on the computational demands and any associated performance gains would strengthen the analysis.

- In practice, determining the correctness of a judgment can be challenging. The proposed method relies on a fairly straightforward approach for judgment annotation, and it’s uncertain whether this consistently leads to high-quality judgments. The quality of judgments has not been independently verified.

- While the performance of the self-taught evaluator generally improves with each iteration, there are scenarios (e.g., reasoning ability for RewardBench doesn't increase as more iterations are gone through) where performance declines. More in-depth analysis and discussion on these cases would provide valuable insights, and investigating a potential upper bound for the model’s performance could also be beneficial.

- The paper lacks clarity on the amount of synthetic data needed per iteration and how performance might vary with different volumes of synthetic data. Providing guidelines on optimal data requirements for each iteration would improve the method’s practical applicability.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: - I would like to understand the motivation behind using two language models for different steps in the experiments. Why is it necessary? What would happen if the same model is used everywhere? How does the fact of using two models reflect on positioning the method as self-taught? How the use of an additional model should be reflected in the baselines in order to make a fair comparison?
- I would like to understand the importance of various components of the proposed method and how it compass to other existing synthetic data methods like the Best-of-N mentioned in the related work section.
soundness: 3
strengths: - This paper presents an efficient method for an important problem of LLM evaluation. The method relies on synthetic data without the use of human data and thus it is easier to scale in practice.
- The presentation of the paper is very clear and it is easy to follow and find relevant information
- The proposed method is quite original, especially the component on generation of artificial pairs of responses where one element is constructed to be better than the other element.
- The experimental section contains comprehensive experiments showcasing the benefits of synthetic data over human data. The results are quite encouraging and have a potential to have an impact in the community.
summary: This paper proposes a new way to train the evaluators for the capabilities of LLMs which relies on only synthetically generated data. The authors start with a set of user instructions which are classified and sampled to represent different categories. Then, they generate pairs of responses where one is constructed to be better than the other. First, they generate a response from the model which is going to represent a better response. Second, they prompt a LLM to modify the initial user task to be semantically similar but not identical and they generate the response for that task and this sample is going to represent a worse response in the pair. For each pair, they sample N reasoning traces and judgements with LLM-as-a-Judge and select a trace which agrees with the synthetically constructed order of the samples. This trace is then used for finetuning the model. They repeat this process iteratively by using better and better LLM-as-a-Judge models but finetuning the same original model. The experiments show that on several benchmarks their method results in a better model than the same initial model trained on human data.
weaknesses: - My main concern is about using *two* language models in the experiments. While the method is presented as a "self-taught" evaluator, the actual experiments rely on the use of two models: Mistral 22Bx8 Istruct for generating the initial synthetic responses, initial judgments and categorizing the queries, and Llama3-70B-Instruct for everything else. How is the need for the second model motivated? In this case, wouldn't it be the situation of distilling the knowledge of two LLMs into one rather than being self-taught?
- Regarding the experimental section and the baselines, the authors mentioned some related work that sounds to be highly applicable in the studied setting, such as the Best-of-N method. Were there any attempts to compare against it?
- The proposed method contains many steps / components that contribute to the performance of the policy. While there are several ablations and comparisons presented in the experimental section, it would be nice to understand in which degree each component affects the solution.
- In table 2 it seems that the performance on iteration 5 is worse than on iteration 4. Why is this the case? What is happening here?

Some minor questions and clarifications:
- When comparing against the human data, were the sizes of the datasets identical? Was the user instructions with human labels filtered in the same way (which seems to preserve more complex instructions) as the user instructions used to generate the synthetic pairs?
- In terms of formatting, I think putting brackets around citations would make it easier to read the text.
- Line 090: West->Best?
- For the understanding of the algorithm, I would like to see some examples of the "related but different" prompts and to see in what kind of answers they result. Intuitively, it is hard to make difficult comparison examples synthetically, and I would be interested to read examples from this approach
- What proportion of the traces agree with the synthetic labels over the training iterations?
- Details about the model selection (line 225-227 are not very clear)
- As far as I understand, for the final inference, there are N samples generated and the majority vote is performed for the proposed method (line 244-247). Is the same done for the baselines?
- What is the conclusion from Table 6? What combination to use, if any?

---

confidence: 4
contribution: 4
initial_rating: 5
presentation: 2
questions: 1. **Validation of Prompt Selection and Evaluation**  
   Could you provide further clarity on the categories chosen for “reasoning” and explain why they were deemed the most challenging, specifically over alternatives like “Coding” or “Social Studies”? 

2. **Judgment Selection Consistency**  
   In the data collection process, was there a standardized method to ensure that “worse responses” generated were indeed inferior? Examples of these responses, especially in the appendix, would improve understanding of this process. 

3. **Majority Vote Implementation**  
   How was the majority vote for final judgments implemented? What impact did this have on the model’s improvement scores?

4. **Synthetic Data Generation Process**  
   Could the authors provide more concrete steps for generating synthetic judgments based on additional data sources? This would aid in replicability.

5. **MT-Bench Experiment**
  Since MT-Bench is multi-turn, did the author observe any difference in the performance of the evaluator on first turn versus the second turn?
soundness: 2
strengths: 1. **Novel Contribution**: The paper addresses a significant challenge in model evaluation by eliminating the need for costly human annotations, which can be both expensive and quickly outdated.

2. **Technical Implementation**: The proposed solution is an end-to-end pipeline that includes data curation, iterative synthetic data generation, and model training. The training produceure is also properly outlined.  

3. **Performance**: The method shows decent performance, competing with human-annotated evaluation models and LLM judges such as GPT-4-Turbo-125.

4. **Scalability and Cost-effectiveness**: This approach offers a scalable and more affordable alternative for model evaluation, which is particularly valuable for model developers.
summary: The paper proposes a novel approach, "Self-Taught Evaluator," which enhances model-based evaluation without human annotations by using synthetic data only. The iterative self-improvement process generates contrasting outputs, curating training data from WildChat for a large language model (LLM) to improve as a judge. The authors demonstrate that this method can improve Llama-3-70B-Instruct from 75.4% to 88.3% accuracy on RewardBench, matching performance levels typically achieved with human-annotated data.
weaknesses: ## Methodological Concerns

**Reliability of LLM-as-a-Judge**: The paper does not thoroughly validate the reliability of various components in the LLM-as-a-Judge system used throughout the pipeline, raising concerns about the accuracy and consistency of judgments.
   
> Line 230: To perform prompt selection, we annotate the category of each instruction with the Mixtral 22Bx8 Instruct model, using the template in Figure 7 and select 20,582 examples in the reasoning category, as we expect these to be challenging inputs.

1. *Selection and Justification of Categories*: It is unclear whether the reasoning category chosen is specific to “Knowledge and Reasoning” or includes others like "Coding" or "Social Studies." Clarifying which categories were deemed “challenging” and why could strengthen the rationale behind the selection process.

2. *Assumption of Prompt Difficulty*: The choice to prioritize the “reasoning” category as the most challenging is not fully justified. Other categories like “Coding” or “Social Studies” could also be complex; a justification for the choice of the reasoning category would enhance the paper’s clarity.

3. *Lacks Validation of Categorical Classification*: The study's reliance on Mixtral 22Bx8 as a judge for classification raises concerns about potential biases and reliability. Without proper validation and ablation studies, the accuracy of the classification process and the quality of selected prompts remain questionable. Including accuracy metrics or a confusion matrix would significantly strengthen the claims made about classification quality. Ablation on using different LLMs as judge is also desirable. 

> Line 126: We use the following prompt template which is used to generate a 'worse response' y^l. Given an instruction x and baseline response y^w generated by an instruction-following LLM as usual, this prompt is used to first generate a 'noisy' version x′ of the original instruction x, and then a best-attempt y^l at responding to x′. y^l is then treated as a poor response to x, giving a preference pair y^w_i ≻ y^l_i.

4. *Assumption of Poor Response Quality*: While the methodology treats responses from certain prompts as “worse,” the inherent biases of LLMs and potential quality variances raise questions about consistency. 
5. Correct me if I am mistaken, I cannot find examples for the generation of “worse responses” anywhere in the paper, including the appendix. Examples would be helpful for further illustration. 

> Line 244: At inference time when evaluating final performance, we sample generations N times and take the final judgment to be the most common verdict.

6. *Majority Vote Methodology*: The approach of using a majority vote to finalize judgments could benefit from additional clarification, specifically on how judgments are consolidated and what impact this has on final model performance.

> Line 250: To understand the effectiveness of the proposed method, we generate synthetic judgments using the same approach but based on the following data sources.

7. *Synthetic Data Generation Steps*: The paper proceed to explain high-level steps for synthetic curation, however, lacks sufficient detail on the exact generation steps, limiting reproducibility.

8. *Data curation details*: It is unclear how exact the data is being curated. Starting with WildChat dataset, how many conversations were applied using the prompt selection procedure? Did the author deduplicate the dataset? Did author ran PII detection? Were multi-turn conversation being included? If multi-turn conversation are being include, how are they handled during prompt selection and judgment annotation (eg. concat all the turns)? Does the curated dataset include non-English conversation? Addressing these question and provide a detailed explanation will greatly improve reproducibility of the work. 

## Analysis Concerns

> Line 497: We further instruct Llama-3-70B-Instruct to infer the complexity (using a score of 1–5) and category of each input instruction.

9. *Validation of Complexity Inference*: Relying on Llama-3-70B-Instruct to infer prompt complexity raises questions regarding its ability to accurately gauge task difficulty. Validation experiments could bolster the validity of the complexity categorization.

10. *Presence of Simple Prompts in Curated Data*: Despite aiming to filter for challenging prompts, many simpler prompts remain, suggesting possible gaps in the selection methodology. Additional validation could enhance prompt filtering accuracy.

## Suggestions for Improvement

1. **Validation of Pipeline Components**  
   Additional validation experiments are needed for components like prompt selection and judgment annotation, particularly since both stages heavily rely on LLM-based evaluators.

2. **Ablation Studies**  
   The paper would benefit from ablation studies to justify choices, such as using Mixtral-22Bx8 for prompt selection, as well as alternative configurations that might affect the pipeline’s effectiveness.

3. **Generalizability**  
   Demonstrating the approach’s adaptability to other models would help establish broader applicability and enhance its value to the field.

4. **Bias Analysis**  
   A thorough analysis of potential biases, including length bias or stylistic preference, in the trained evaluator would clarify whether the approach mitigates or exacerbates common biases in LLM evaluation.

5. **More baselines**
  The paper compare trained evaluator against gpt-4-turbo-0125 on MT-Bench. The result can be strengthen by using more than one baseline and include other SOTA LLM judges. Otherwise, it is also unclear whether gpt-4-turbo-0125 is actually a good LLM judge.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: The proposed method iterates over instruction and response pairs that are generated at the start, and then fixed.
Could this not cause the model to over-fit to these specific instructions and responses, neglecting performance on others?

Since the model is being fine-tuned to re-produce its own previous outputs (with some filtering applied), how can it learn to generate better critiques?
Where is the exploration coming from, and why would it favor generating good critiques over poor ones which still agree with the ground truth?

What happens if you train an LLM to optimise the resulting reward model, especially on prompts not in the initial instruction distribution of the LLM-as-a-Judge?
Does this policy LLM then outperform LLMs fine-tuned on different sources of reward?
This seems like a very important test to run to properly verify the method.

Do you know or have any idea what happens if you include response pair construction or instruction selection as part of the iterative process?

How does the proposed method avoid model collapse, as is implied to eventually happen with these iterative recursive schemes by Shumailov et al. (2024, https://www.nature.com/articles/s41586-024-07566-y)?

Multiple seeds and computing standard error/deviation would help discern the signal from the noise in regards to whether the iterative nature of the method provides significant, consistent benefits.
The reviewer notes that this might not be possible due to computational requirements involved.

In table 2, it's not clear whether the GPT4 baseline model was using 32 sample majority vote.
Please can you clarify this, as if it is not, the results of this table seems to be misleading and making an unfair comparison.
soundness: 3
strengths: The problem the paper is trying to address is somewhat well motivated.

The paper is well presented, with minimal errors.

The method is reasonably well communicated, and to this reviewers knowledge, reasonably novel, extending existing work nicely.
The experimental setup is also clearly communicated, with details on hyper-parameters that would greatly aid reproducibility.

The method is tested on a variety of datasets, and using a variety of data sources.
Additionally, scores per iteration and for different subsets of the RewardBench dataset are given.
Thus, the empirical data generated to evaluate this method is extensive.

Some of the empirical evidence supports the claim that the proposed method outperforms existing SotA reward models.

The authors perform several ablations and additional analysis of their method, empirically justifying some of their design choices.
summary: The authors propose a new approach to training an LLM-as-a-Judge that, unlike prior work, does not need human supervision.
It can improve the judgement quality of strong instruction-tuned LLMs to be comparable to SotA judge models trained in a supervised manner.

The method utilises clever prompting to generate pairs of completions for an instruction where one is known to be preferred to the other.
Judgements with chain-of-thought reasoning are then generated from the target LLM, with correct ones being SFT trained on in an iterative procedure.

It is tested on a variety of benchmarks, along with some ablations.
The original claim is somewhat supported by the empirical evidence.
weaknesses: It's not entirely clear why the method should work, especially if given many iterations.
See "Questions" section for more details.

Experimentally, it appears that the iterated training does not always help, with the score often decreasing or noisily bouncing around after the first iteration.
See Table 1 "Chat" and "Reasoning" columns, table 2, and table 5 "Chat" column for clear examples of this.
In many other columns, score does not monotonically increase with iteration.
Thus, it's not clear to what extent the iterative nature of this method provides *consistent* improvements.

From table 1, the method is beaten both overall and in the "Safety" and "Reasoning" categories by other methods.
This slightly undermines the claim that the method outperforms or matches existing top-performing reward models.

The main motivation behind training a reward model is to provide a reward signal to optimise a downstream LLM's outputs to better reflect human values.
The proposed method has not been evaluated on its ability to provide such a reward signal.
 
## Errata
* Figure 1 is not referenced in the main paper text
* Table 5 "Chat" column has the Iteration 3 element in bold, but the Iteration 1 element is highest scoring.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 2
questions: Please see the weakness part.
soundness: 3
strengths: (1) Writing: This paper is well-written and easy-to-understand.

(2) Method: Instead of adhering to the conventional LLM-as-a-Judge paradigm for providing judgment explanations, this paper introduces an innovative approach leveraging synthetic training data to enhance evaluators, thus avoiding the need for expensive and time-consuming human annotations during model development.

(3) Experiments: Through iterative experiments on LLama3-70B-Instruct, this paper convincingly showcases the effectiveness of the proposed Self-Taught Evaluators across RewardBench, MT-Bench, and HelpSteer2 datasets.
summary: This paper introduces a novel approach focused on leveraging synthetic training data to enhance evaluators without the need for resource-intensive human annotations during model development. Moreover, it delves into an iterative self-training methodology and introduces an iterative self-improvement framework capable of producing contrasting outputs and training an LLM-as-a-Judge. The proposed Self-Taught Evaluator demonstrates notable performance enhancements on RewardBench.
weaknesses: (1) Concerning the explanation provided in line 044 and Figure 2, could you elaborate on the methodology used to ascertain the contrastiveness of the generated synthetic preference pairs beyond the prompt instructions?

(2) The definition of "similar" in Figure 2 appears ambiguous. Specifically, while x' is deemed similar to x, the term "similar" implies a high level of relevance without strict semantic identity. Clarification on this aspect would be beneficial.

(3) Several typographical errors, such as the presence of "?" in Figure 2 and line 241, have been noted.

(4) The citation format lacks consistency, as evidenced by variations between line 137 and line 139. It would be beneficial to standardize this aspect.

(5) In terms of experiments, aside from Llama, have results been obtained using different models of varying sizes?

(6) Could you provide insight into the interpretation of performance fluctuations observed in Chat and Reasoning across multiple iterations?

Meta-review: **summary**

The paper introduces Self-Taught Evaluator for training LLMs as evaluators without relying on human annotations, using a fully synthetic, iterative self-training process. The method generates contrasting response pairs for given instructions, curating synthetic preference labels to train the model in an iterative curriculum learning approach. By leveraging CoT reasoning and progressively improving the model’s evaluative capabilities through self-supervised fine-tuning, the approach demonstrates substantial performance improvements on benchmarks like RewardBench and MT-Bench. 

**strengths**

* By enabling models to generate their own synthetic data and preference labels, the method significantly reduces the need for human labeling, offering a scalable solution for training evaluators.
* The method demonstrates strong performance on benchmarks like RewardBench and MT-Bench, matching or surpassing traditional models reliant on human annotations, which underscores its potential for practical application.
* The paper is well-structured and clearly written, making it accessible and easy to follow.

**weaknesses**

* Judgment quality assurance: As pointed by several reviewers, the method's reliance on generated judgments without independent validation raises concerns about the accuracy and reliability of these judgments, which are crucial for training the evaluative models. 

**decision**

This paper introduces an innovative method that employs an iterative self-improvement process to enhance the judgment skills of LLMs. Considering the impacts/importance of LLM evaluators, I think this paper provides valuable insights to the community. However, there are several areas that require further exploration. Notably, addressing the concerns regarding judgment selection consistency is crucial. While the use of synthetic preferences is a significant advantage of the proposed method, the lack of independent validation for judgment evaluation raises questions about the trustworthiness and actual quality of the models. Although the authors have demonstrated the quality of their method on standard benchmarks, this does not necessarily guarantee its correctness. This issue is particularly critical given that one of the primary uses of LLM evaluators is to ensure the alignment of LLMs with human values.

Decision: Reject