Generative Judge for Evaluating Alignment
Authors: Junlong Li
Shichao Sun
Weizhe Yuan
Run-Ze Fan
hai zhao
Pengfei Liu
Venue: ICLR 2024 poster (2024)
Keywords: Generative
Evaluation
Alignment
Primary area: generative models
TL;DR: We release Auto-J, a cutting-edge, flexible and interpretable judge with 13B parameters, to evaluate alignment in various real-world scenarios.

Abstract: The rapid development of Large Language Models (LLMs) has substantially expanded the range of tasks they can address. In the field of Natural Language Processing (NLP), researchers have shifted their focus from conventional NLP tasks (e.g., sequence tagging and parsing) towards tasks that revolve around aligning with human needs (e.g., brainstorming and email writing). This shift in task distribution imposes new requirements on evaluating these aligned models regarding *generality* (i.e., assessing performance across diverse scenarios), *flexibility* (i.e., examining under different protocols), and *interpretability* (i.e., scrutinizing models with explanations). In this paper, we propose a generative judge with 13B parameters, **Auto-J**, designed to address these challenges. Our model is trained on user queries and LLM-generated responses under massive real-world scenarios and accommodates diverse evaluation protocols (e.g., pairwise response comparison and single-response evaluation) with well-structured natural language critiques. To demonstrate the efficacy of our approach, we construct a new testbed covering 58 different scenarios. Experimentally, **Auto-J** outperforms a series of strong competitors, including both open-source and closed-source models, by a large margin. We also provide detailed analysis and case studies to further reveal the potential of our method and make a variety of resources public at https://github.com/GAIR-NLP/auto-j.

Published as a conference paper at ICLR 2024
GENERATIVE JUDGE FOR EVALUATING ALIGNMENT
Junlong Li
Shichao Sun 36
Weizhe Yuan*
Run-Ze Fan
Hai Zhao'
Pengfei L2.
'Shanghai Jiao Tong University SShangaai Artificial Intelligence Laboratory
'Hong Kong Polytechnic University "New York University 'Chinese Academy of Sciences Gennratiie AI Research Lab (GAIR)
ABSTRACT
The rapid development of Large Language Models (LLMs) has substantially ex- panded the range of tasks they can address. In the field of Natural Language Processing (NLP), researchers have shifted their focus from conventional NLP tasks (e.g., sequence tagging and parsing) towards tasks that revolve around align- ing with human needs (e.g., brainstorming and email writing). This shift in task distribution imposes new requirements on evaluating these aligned models regard- ing generality (i.e., assessing performance across diverse scenarios), flexibility (i.e., examining under different protocols), and interpretability (i.e., scrutinizing models with explanations). In this paper, we propose a generative judge with 13B parameters, AUTO-J, designed address these challenges. Our model is trained on user queries and LLM-generated responses under massive real-world scemarios and accommodates diverse evaluation protocols (c.g., pairwise response comparison and single-resper response evaluation) with well-structured natural language critiques. To demonstrate the efficacy of our approach, we construct a new testbed covering 58 different scenarios. Experimentally, AUTO-J outperforms series of strong competitors, including both open-source and closed-source models, by large margin. We also provide detailed analysis and case studies further reveal the potential of our method and make a variety of resources public at https://github.. Cot.///....oooc/AAAAL////AA/
INTRODUCTION
In natural language processing, the evaluation methodology for generation tasks is continually updating with the advancement of modeling techniques, ranging from ROUGE (Lin) 2004] to
ROUGE-WE (Ng & Abrecht 2013) (a metric enhanced with word embedding (Mikolov ct al. 2013)
and then to ScRT (Zhang et al. 2019). BARTScore (Yuan al.) 22221) and GPTScore (Fu et al.
2023) (metrics enhanced by pre- tra-trained language models (Peters et al.) 2018 Devlin et al. 2019 Lewis al. (0200), aiming for more reliable evaluation for over-growing modeling techniques. Recently, the advent of large language models (Brown et al., 2020 Touvron et al.) 2023a b) has not only reshaped the implementation for modeling techniques (i.e., paradigm shifi from "pre-train, fine-tuning" to "pre-train, supervised fine-tune, and reward model-based tune" (Ziegler et al. 2019) Stiennon et al.) 22020; Ouyang et al. 2022)) but also broadened the spectrum of tasks that modeling techniques seek to address (i.c., task distribution shift from traditional NLP tasks towards those more
aligned with human needs (Bai al., 2022a OpenAl 2023 Zhou al. 2023 Taori et al. 2023)).
Given the evolving modeling techniques, the evaluation methods are in urgent need of upgrading and improvement adapt to new challenges and requirements, particularly in the following aspects: (i) generality: the evaluation method should support massive real-world scenarios where gold references are usually unavailable. Traditional approaches frequently require human references and apply a single evaluation metric to constrained tasks (e.g.. ROUGE (Lin) 2004) for text summarization, BLEU (Papineni et al. 2002) for machine translation) are struggling to keep pace with the current demands for evaluation. (i) fexxibiiiy: the evaluation method should accommodate different protocols with desirable performance. The current LLLM-base modeling paradigm requires methodological support of the evaluation in various aspects, and the evaluation protocols they demand also exhibit variations. For instance, when learning a reward model, it is necessary to compare two responses,
Corresponding author

---

Published as a conference paper at ICLR 2024
while evaluating the final system output often involves assessing single response (Stiennon et al. 2020) I (iii) interpretability; evaluation results are encouraged to provide more than solely numerical scores. Additional explanations crucial to enhance the reliability of evaluation outcomes and facilitate humans' involvement in the evaluation loop (Saunders et al. 2022).
In this context, researchers have engaged in some explorations, with the central idea being to conceptualize evaluation as an instruction-follwiiin problem (Fu et al.] 2023 Liu et al. 2023 based
on a high-capacity LLM. For example, Zheng al. (2023); Zhou et al. (2023 Dubois et (023) employ proprictary LLMs (e.g.. ChatGPT, Claude or GPT-4) through API calls to perform various evaluation protocols. Such methods have shown decent agreement with human judgment, but they also face challenges in terms of consistency and reproducibility due to the opacity of API models as well as the high API cost. An alternative
train a specialized evaluator based on open-source LLMs. PandaL.M (Wang et al.] 2023c) is able to compare a pair of responses for a given query with a brief explanation of the evaluation process, and Shepherd (Wang et al., 2023b) can provide critiques LLM's ' response to pinpoint its shortcomings. These models have achieved remarkable performance in certain settings; however, they are relatively limited in the following aspects: (a) Some not optimized to evaluate various deployed LLMs under massive real-world scenarios but are only trained on synthetic data (e.g., the Alpaca dataset Taori et al. (2023) by GPT-3.5), online forums, or traditional NLP datasets, without the consideration of scenario-specific evaluation criteria. (b) Each of these models only supports one evaluation protocol, like pairwise comparison or single-response evaluation, making them less flexible for various evaluation requirements. (c) They only provide brief or no natural language explanation for their evaluation, reducing the reliability of the result.
To address the above challenges, we develop AUTO-J, a generative judge with 13B parameters trained on user queries and model-generated responses from massive real-world scenarios. Methodologically, to train more generalized judge, we created new dataset from a large collection of data, encom- passing 58 different scenarios, with most samples coming from real-world user queries and LLMs' responses. Based on the dataset, we guide GPT-4 (OpenAl, 2023) with carefully hand-written criteria for each scenario to collect desired evaluation judgments our supervised training signals and apply heuristic filtering strategies and prot-procss to unify output formats and mitigate noise. We also design new testbeds from the above dataset for pairwise comparison and single-response evaluation, with diverse and balanced scenario distribution ($5.1). Through comprehensive meta-evaluation on its evaluation functionalities, we show that AUTO-J outperforms various strong baselines, including both open-source and closed-source models I $6.,
6.3).. We also conduct detailed analysis and case studies ($6.4) show series of advantages offered by AUTO-J, from lessened positional bias in pairwise comparison, more specific critiques in single-response evaluation to the potential as a generative reward model to help improve base LLMs. To summarize, our contributions are:
(i) We develop AUTO-J, new open-source model that can effectively and flexibly evaluate LLMs for both pairwise comparison and single-response assessment, with well-structured natural language critiques to support its evaluation. It establishes new state-of-the-ar performance among open- source models across all 58 scenarios (e.g.. 8.9% improvement pairwise evaluation in $6.1) and surpasses strong proprietary models such as ChatGPT and Claude-2 (e.g., with 12. 12.1% % and 12.4% gains pairwise evaluation in 1 (ii) We construct a judgment dataset ($3) that covers 58 real- world scenarios. Each judgment consists of both numerical rating (or a pairwise comparison result) and a critique generated in accordance with our curated 332 criteria. These data resources serve as a valuable foundation for both training and benchmarking evaluation methodologies under emerging technologies. (iii) We have released a wealth of resources to meet the diverse needs for future research: out-of-the-bou models with superior performance; scenario typology and classifier; curated scenario-aware evaluation criterion and prompts; judgments with well-formatted critiques.
2 RELATED WORK
2.1
EVALUATION OF LLMS
It is universally known that the best way to evaluate LLMs is human judgment, but collecting human annotations can be costly, time-consuning, and laborious (Ouyang et al., 2022 Zheng et al.) 2023).
Traditional metrics such as BLEU and ROUGE are capahle of but not adept at conducting pairwise evaluation due the worse performance in sample-level evaluation. (Bhandari et al. 2020)
2

---

Published as a conference paper at ICLR 2024
Email Writing
Code Writing
Criteria Accuracy
Data Sources
Scenario Classifier
Sample
Sample
Criteria
Evaluation
Subject Pupose
Subjeet
Judgments
Efficiency
Purpose
Pairwise
Sumple
Reasecing
Sumple
Sample Structure
Preferanee
Query GPT-4
Math
Criteria
Single
Reasoning
Logic
Quary
Notation
Sumple
Response
Sample
Generate & Reformat
Ressoning
Response
Raiing
User
Preference
Step1:
Define Scenario and Criteria
Step2: Collect Data for Each Scenario
Step3: Generate Judgements
Figure I: An overview for our data construction pipeline three steps.
automated proxy for assessing LLMs has become a natural choice (Zhou et al. 2023). With appropriate prompt design, the quality of evaluation and agreement human judgment can be promising (Dubois et al.) 2023; Zheng et al.] 2023; Zhang et al. 2023: Wang et al. 2023a). However, the cost concern still exists when calling the APIs of these proprictary models, especially when there frequent need for model validation on large-scale data. Moreover, closed-source evaluation leads to low reproducibility due to potential changes in models behind the API. Some recent works have started to make attempts for open-source alternatives. SelFee (Ye et al. 2023) collects generations, feedback, and revised generations from ChatGPT and fine-tunes LLaMA models to build a critique model. Shepherd (Wang al. 2023b) trains model that can output critiques for single-response with the data of feedback from online communities and human annotation. PandaLM (Wang et al., 2023c) trains a model to conduct pairwise comparison for LLM Instruction Tuning Optimization, and [heng et al. (2023) also fine-tune Vicuna (Chiang et al., 2023) on a 20K pairwise comparison dataset to explore the potential of open-source models as cost-friendly proxy.
Using strong LLMs (usually closed- ones, e.g., GPT-4, Claude, ChatGPT)
2.2 META-EVALUATION TESTBED FOR LLM EVALUATORS
Besides the evaluators themselves, there is also practical need to construct comprehensive testbed to meta-evaluate them (i.e., assessing the quality of their evaluation). In In/Zheng et al. (2023), MTBench and Chatbot Arena Conversations are proposed. The former has only 80 human-crafted queries, each with several LLMs' responses and expert-level human annotation on pairwise comparison; the latter large collection of crowdsourced data, with more than 30K queries from real-world users and their vote on pairs of responses from different LLMs. FairEval (Wang et al. 2023a) is based on the 80 queries from VicunaBench (Chiang et 2023) with human annotated labels between ChatGPT and Vicuna responses. PandaLM (Wang et al. (023c) constructs a test set comprising 999 pairwise samples, with queries from 252 user-oriented instructions in Wang et al. (2022). LLMEval (Zhang et al. 2023) is much larger than the previous two, with 2,553 samples compiled from multiple data sources with numan-annotated preferences. Shepherd (Wang et al., 2023b) collects 352 samples from multiple sources for its critique model test set to evaluate the quality of the critiques.
3 DATA CONSTRUCTION
We construct data from massive real-world scenarios with high-quality evaluation judgments for both training and testing. The data construction pipeline involves three main steps: (1) defining evaluation scenario and criteria, (2) collecting real-world queries and responses from different models for these scenarios and (3) generating desired evaluation judgments for different evaluation protocols. An overview our data construction pipeline is shown in Fig. T
3.1 SCENARIO AND CRITERIA DEFINITION
Scenario We define 58 scenarios (including one "others" scenario), categorized into eight major groups: Summarization, Exam Questions, Code, Creative Writing, Functional Writing, Rewriting,
3

---

Published as a conference paper at ICLR 2024
Aspect
clarity; written should cleadyy outline the
objectives, tasks, and timeline-
2. feasibility; the writtmn plan should propose reali-
3. creativity: the wattenn plan should demoestrate
stic and schievable steps and actions..
seppiingg perend
shacke
creative thinking and innovative
prisna
4, thoroughnes:: the written plam should cover all essential ispects details the
Bowerting
Eum
Cammaa
Format Aspect
Coom
stracture: the wrilles plan should be well strac- tured, logical flow ideas...
Fanshoonal
lasout: the written plan encouraged use headings, bullet puints, lists, tables,
Crearing
completeness Basic Bollowing: for instractions g, answer multiple...
2. socuracy; all contemss provided or mentioned in response should information richnase the response encorrgeed to provide rich, detriled...
(a) Criteria for "planning" scenario.
(b) Scenario distribation.
Figure 2: An example of the criteria for the "planning" scenario and a demonstration of the defined scenarios. In (b), Summa. - Summarization, Commu. - General Communication.
General Communication, and NLP Tasks, as shown in Fig. (b). The detailed description for each scenario is shown in Tab. A
Criteria Besides the definition and description, we also design a set of criteria for each scenario that serves reference to guide models how to do the evaluation. Each criterion has name and a description. We show condensed version of the set of criteria for the "plamning" scenario in Fig. 2 (a) (the complete version is in Fig. 10) Generally, criteria for each scenario consists of specific ones and basic ones (more general, shared by multiple scenarios). In total, we craft 332 different criteria. When we use a of criteria, we put them in the system message for LLMs, as shown in Tab, g
3.2 QUERIES AND RESPONSES COLLECTION
To start with, we first collect a large collection of data from the following sources: Chatbot Arena Conversations and MTBench (Zheng et al. 2023). OpenAl Summary (Stiennon et al. 2020). OpenAl WebGPT (Nakano et al. 2021). ), Stanford SHP (Ethayarajh et al. 2022). Synthetic GPT-J (Havrilla 2023), and PKU-SafeRLHI (Ji et 2023). All these datasets are publicly available preference datasets with human preference comparisons containing two or tie) sharing the same query (and previous dialogue). We remove the non-English samples and only keep the first turn for multi-turn dialogues. In short, all samples share a common structure: A query, Response & and preference label (1/2/Tie).
responses (win, lose,
The next step is to classify the collected data based on the scenarios. Although this is trivial for datasets with relatively homogeneous components (OpenAl Summary, OpenAl WebGPT) or small query size (MTBench), this is quite challenging larger and more complex ones. Therefore, we train a classifier to help us with this. The complete training details are in $B Based on the classifier, are able to classify all the data have collected.
3.3 JUDGMENT GENERATION
Pairwise: This part of the data comes from all datasets the data source except MTBench. We guide GPT-4 to make pairwise response comparisons, with scenario-specific criteria as the system message in Tab, and the user message prompt as in Tab. TT After that, we reformat the raw GPT-4 output with heuristic rules to achieve a unified format in Tab. 19 We discard samples where the predictions of GPT-4 are inconsistent with existing human annotations or the predictions cannot be reformatted. For each scenario, the collection process continues until either all samples of this scenario have been
4

---

Published as a conference paper at ICLR 2024
Model
Summ
Exam
Code
Rewriting
Crea W
Ponc W
Comm
NLP
Overall
Closed sounee Models
ChaiGPT
33.3
40.4
45.8
42.7
Claude-2
30.6
36.1
41.7
34.2
48.1
42.5
40.6
48.5
42.4
GPT-4
59.7
51.4
69.2
58.3
66.7
60.4
58.3
65.2
61.2
Open-source Models
SteamSHP
33,3
29.2
26.7
33.3
40.7
31.3
51.4
51.9
40.6
PandaLM
29.2
33.3
31.7
23.3
43.5
32.9
44.8
48.9
38.9
LLMMA-2-Chat1133
20.8
27.8
19.2
20.0
31.5
27.5
35.8
31.8
29.0
Vicuna: Dicuma-13B-vv 138-v1.5
30.6
23.6
35.0
28.3
36.1
37.5
45.5
39.8
37.3
I33
22.2
32.5
19.2
28.7
25.4
29.2
33.0
27.8
LLaMA-2-cha--70B
34.7
33.3
36.7
33.8
31.4
54.2
47.2
47.7
45.9
AUTO-J
45.8
38.9
59.2
47.5
54.6
57.1
58.0
57.6
54.8
Table I: Agreement rates for pairwise comparison different scenario groups and overall results. Results with underline are the best among all models and results in bold are the second-best The mapping from abbreviations to names of scenario groups Summ - Summarization, Crea W Creative Writing, Fune + Functional Writing, and Comm General Communication.
annotated with reformatted judgment (or discarded), we have collected 100 samples for this scenario. The final size of pairwise training data is 3,436, and the detailed statistics are in Tab. 21
Single-response: For single-responss we pick 960 query-response pairs from Chatbot Arena Con- versations with a balanced sampling on different scenarios. In preliminary experiments, directly incorporating the scenario criteria the system message (as in pairwise evaluation) impairs GPT-4's performance single-response assessment, overly constraining its generated output the scenario- specific criteria. Therefore, we adopt a "divide-and-conquee strategy: We collect two pieces of critiques from GPT-4 for single response with and without scenario criteria system message, and then in the third inference, we get the final evaluation judgment by asking GPT-4 to combine these two critiques into
comprehensive critique and give final rating. The message prompt and the prompt for combining critiques are in Tab. 12 [3) 13 and the detailed statistics are shown in Tab. 22 Tab. 20 shows example from the "planning" scenario. We find that critiques generated with and without scenario criteria exhibit distinct stylistic differences: The former is longer and closely adheres the given criteria, whereas the latter is more concise yet capable of incorporating details not covered by the criteria. Finally, combining the above two critiques, a comprehensive critique simultaneously contains general criteria for this scenario and specific details for this sample.
Besides the collected evaluation judgments, we also need to determine the input format for AUTO-J. In early-stage experiments, we attempted to include the scenario criteria as the system message in the input. However, models trained in this manner performed poorly, often simply paraphrasing the scenario criteria. Therefore, we adopt a technique akin to Context Distillation (Bai al. 2022b) and Ghost Attention (Touvron et al.) 2023b), where omit the inclusion of scenario criteria in the input for the training data, allowing the model to learn them from the output end implicitly. This design significantly enhances the generality of AUTO-J. The final input formats for pairwise comparison and single-respe response evaluation are Tab. T7 and Tab. [8] respectively.
Input format:
4 TRAINING AUTO-J
By integrating data from both pairwise and single-response evaluations, we train our model to seamlessly toggle between diverse evaluation protocols simply by applying the corresponding prompts. To lessen the positional bias (Wang et al.] (023a) in pairwise comparison, we apply simple data augmentation trick. For each pairwise training sample, swap the order of two responses in the input and alternate the "Response I" and "Response 2" the evaluation judgment. Since this doubles the pairwise data, balanced the dataset by duplicating each single-response samples well.
We train AUTO-J from LLaMA-2-11 -hahchhtt (Touvron et al. 2023b) with the DeepSpeed (Rasley et al. 2020) library, Zero Redundancy Optimizer (ZeRO) (Rajbhandari et al. 2020 Ren al. 2021 Stage
3, cha.eiinttccccckiiiiiing.e (Chen et al. 2016) and FlashAttention (Dao et al. 2022 Dao 2023) on
NVIDIA A100 GPUs. We the 6 (BF16) and tfloat32 (TF32) mix computation precision
5

---

Published as a conference paper at ICLR 2024
Auto.J Wins
Tie
Auto-J Losns
Auto-J Wins
Tie
Auto-J Lases
SelFee
94, 0%
0.4% 5.6%
SelFee
78.
L2Chat
84.5%
0.9%
14.7%
L2Chat
49.1%
34.5%
16.4%
WizardLM
80.6%
16.4%
WizardLM
56.0%
31.9%
12.1%
Vicuna
77.6%
0.4%
22.0%
Vicuna
69.0%
20.7%
10.3%
ChatGPT
73.7%
2.2%
24.1%
ChatGPT
50.4%
31.0%
18.5%
GPT-4
54.3%
0.3%
44,8%
46.1%
17.7%
36.2%
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
GPT-4 judgments.
(h) Human judgments.
Figure 3: Win-rate of AUTO-J against baselines single-response critique generation task, judged by GPT-4 and Human, L2Chat refers LLaMA-2-Chat-133
options to further optimize training and efficiency. The model is trained for 5 epochs (675 parameter update steps in total) and we save checkpoints for every 50 steps. We use AdamW (Loshchilov & Hutter 2017) as our optimizer with B1 0.9, 82 = 0.95 and weight decay of 0. We use peak learning rate le-5 with 3% warmup steps and cosine learning rate decay to O, and set the batch size to 64 and maximum sequence length to 4,096. The loss is only calculated on the output end.
EVALUATION SETTING
5.1 TASK AND TEST SET
Task I: Pairwise Response Comparison (Eval-P) In this task, the evaluators will see a pair of generated responses for given query and decide which better or is tied. From each scenario defined in $3.1 we randomly sample 24 pairwise comparison samples from the data we collected in $3.2 and skip those that have been used as training data. For some scenarios, the number of paired samples with pre-existed human annotation is smaller than 24, so extract queries from either ShareGPT or the brainstormed seed data for training scenario classifier in S Samples from these two sources have annotated pairwise labels, a new pair of responses from two random selected LLM and manually annotate them. In total, have 58x24=1.33) testing samples, each with responses generated by different LLMs and a human-annotated preference label. We refer to this test set as Eval-P, with the distribution on Win/Tie/Los being 520/373/499.
only the query for each sample, generate
Task II: Critique Generation for Single Response (Eval-C) this task, we evaluate the quality of the generated critiques for single-response evaluation. The evaluators are required to write critiques for a response to pinpoint its shortcomings addressing the query. We apply both GPT-4 and human evaluation to compare critiques generated by different models. In GPT-4 evaluation, we randomly shuffle the order of two critiques to mitigate the positional bias, and use the instruction Tab. [6] In human evaluation, we recruit four expert-level annotators (graduate students) and guide them with the same instruction for GPT-4. We build the test set for this task on the basis of Eval-P by sampling 4 out of 24 queries for each scenario and pick the less preferred response for each query (if tie, randomly pick one). We refer to this test set as Eval-C, with 58 x 232 guery-response pairs.
Task III: Overall Rating for Single Response (Eval-R) In this task, we evaluate the usefulness of the final rating for single-response evaluation in ways: (I) The first is use the ratings as verbal "rewards" to help improve the base policy models through the Best-of-N selection (Lightman)
2023 Gao al. 2023), i.e., selecting the best response among the first N candidates with the assigned rewards, and use GPT-4 to grade the selected response. Generally, more reliable model will select better response with higher GPT-4 rating more often. (2) The second is calculate the
al.
From 1aMMA-2-cha family, Vicuna family, WizardLM family, Claude-2, ChatGPT and GPT-4

---

Published as a conference paper at ICLR 2024
response-level correlations between model-generated ratings and GPT-4 ratings. To save cost, only collect the GPT-4 ratings on the previous "best-of-N"" responses. The test set for this task is built on the basis of Eval-C by sampling 2 out of 4 queries for each scenario. We ask two different base LLMSs (LLaMA-2-chat-77 and Vicuna-7B-vl (0vv- to generate 32 responses for each query through uniform sampling (temperature as 1.0). We refer this test Eval-R, with 58x2=1 16 queries and 1 116,322,,712 query-response pairs for each base LLM.
5.2 BASELINES
General-purpose models: We use LLaMA-2- 2-22-h-tt133 Touvron et al. 2023b), Vicuna-13B-vI. (Chiang et al. 2023), M-3-r-MM--3BBvll (Xu et al. 2023). and ChatGP (GPT-3. 3. 5-turbo-0613 3).
We also use GPT-4 (GPT-4-6113) in the pairwise comparison and critique generation, and Claude-2 and LLaMA-2-Chatt 70B in pairwise comparison. These models are used with corresponding prompt for each task: pairwise comparison prompt in Tab. [4 critique generation prompt in Tab. 18 (the same input format for AuTo-J's single-response evaluation). and rating prompt in Tab. 15 Evaluation- specific models: We SelFee (Ye et al.
2023) in critique generation, SteamSHP (Ethayarajh et al.)
2022) in pairwise comparison and overall rating, Open-Assistants 's reward model (Köpf et al. 2023
overall rating, and PandaLM (Wang et 2023c) pairwise comparison.
EXPERIMENTS
6.1 PAIRWISE RESPONSE COMPARISON
common problem in pairwise response comparison is positional bias (Wang et al. 2023a), where an LLM may tend to favor specific positions, causing inconsistency in comparison results when response orders swapped. To pursue stable and reliable results, we conduct two compar- isons for each sample by swapping the order of the two responses in the prompt. We consider a model's judgment agree with human only when the two comparison results are consistent and align with the human judgment.
WizarLLM
46.4
L3Chat13B
47.8
Vicuna
ChatGP
62.5
Claude-2
63.1
SteamSHP
63.6
PandaLM
66.8
L2Char70B
69.9
Auto-J
83.4 $
GPT-4
85.9
The agreement rates for AUTO-J and the baselines Eval-P are in Tab. [ AUTO-J achieves a significantly higher agreement rate than all baselines except GPT-4 on every scenario group. We also plot the prediction consis- tency for each model in Fig. 4 AUTO-J has a similar
50
60
70
80
Consistency Rate(())
Figure 4: Consistency of prediction
when swapping the response order.
consistency rate to GPT-4 and is far more consistent than all other baselines, which makes it more reliable and robust judge for pairwise comparison.
6.2 CRITIQUE GENERATION FOR SINGLE RESPONSE
The comparison results on Eval-C given by GPT-4 and human are in Fig. and the complete comparison results for different scenario groups in Tab. 23 In both evaluation settings, AUTO-J performs significantly than all baselines, including GPT-4, reflecting the strong ability to criticize other LLMs' outputs. We also observe that GPT-4 tends provide judgments with very few ties, whereas humans often give tie judgments in comparisons, sometimes even exceeding 30%. One possible explanation is that the critique from AUTO-J exhibit a clearer structure and readability, which leads GPT-4 to pay less attention to the content when making comparisons, while humans are able to read more attentively and discern subtle differences between two critiques.
6.3 OVERALL RATING FOR SINGLE-RESPONSS
We conduct experiments on Eval-R with the N in Best-of-N selection set as 8, 16, and 32. In practice, if two responses share common model rating, choose the one with a higher output probability. Results in Tab. [ show that responses selected by AUTO-J generally get higher GPT-4 ratings than those selected by baselines different N.
7

---

Published as a conference paper at ICLR 2024
Base LLM
BoN
Open-Assistant
SteamSHP
CharGPT
L2Chat
Vicuna
WizardLM
AUTO-J
8.17
8.02
8.20
8.13
8.09
7.93
8.21
HaMMA-2Chht-77
16
8.28
8.01
8.14
8.19
8.03
7.89
8,33
Selection
8.25
8.05
8.34
7.51
7.47
7.28
7.07
7.19
7.49
Vicuna-7B-vi.5
7.69
7.74
7.29
7.02
7.53
6.46
7.74
7.66
7.66
7.32
7.07
7.63
6.88
7.97
Pearson
0.36
0.13
0.06
0.16
-0.05
0.41
0.57
Correlation
Spcarman
0.42
0.13
0.06
0.24
-0.01
0.35
0.55
Table 2: Top half: Average GPT-4 Rating the Best-of-N (BoN) responses selected by different rating models. Bottom half: Correlations between different models and GPT-4 on all selected Best- of-N responses by different rating models, i means p-value >0.05. L2Chat: LLaMA-2- 2-Mhatt--t----- 13B.
Based on the 1,993 query-response pairs with GPT-4 rating in the above best-of-N experiment, we calculate the response-level Spearman and Pearson correlations between model's rating and GPT-4 ratings. Results in Tab. show better correlation between AUTO-J and GPT-4 than all baselines.
6.4 ANALYSIS AND CASE STUDIES
System-level Ranking Besides response-level evaluation, and we also investigate the potential of AuTO-J on the system level, which is useful when we benchmark existing LLMs with leaderboard. We use the AlpacaEval leaderboard each submitted model. We use AUTO-J in single-response evaluation protocol and calculate average ratings on the dataset for all open- source LLMs on the leaderboard. The Spearman and Pearson correlations with GPT-4's ranking on the leaderboard are 0.97 and 0.96 respectively (Fig. 5). and we show detailed ranking in Tab. [4 This extremely strong correlation indicates that AUTO-J can also serve good system-level judge for ranking open-source LLMs.
has archived complete outputs for
80
60
Auto-J
Figure 5: System-level corre-
lation AlpacaEval.
Ablation Studies (1) We train model that outputs only the final
decision using the same pairwise training data for AUTO-J, Its agreement rate with human on Eval-P is 55.0 ( (AUTO-J gets 54.8, in Tab. [. We conclude that our model does not sacrifice the pairwise comparison performance for supporting multiple evaluation protocols and generating supporting explanations. (2) Using the same pairwise training data, we train a standard reward model to output scalar rating for each query-response pair (its agreement rate on Eval-P is 54.5). We conduct best-of-32 response selection experiments. As shown in Tab. 3 despite not being directly optimized for scalar output, AUTO-J achieves comparable performance reward model. It also demonstrates higher correlation with GPT-4 ratings than the reward model trained solely for that purpose.
Case Studies We show a pairwise comparison case from the test set (Eval-P) in Tab. 44 (complete version in Tab. 25 and 2nd ). This example shows only AUTO-J (and GPT-4) emphasize the advantages of the second response in terms of tone and interactivity for family email, and make the correct choice.
Base LLM
AUTO-J
ScalarRM
1.2haa78 Vicuna7B
8.34
8.42
7.94
Correlation with GPT-4
We show a single-response evaluation case from the test set (Eval-C) in Tab. (complete version in Tab. (7) shows that the critique given by AUTO-J is more aware of the user's status novice in cooking, and pinpoint more essential concerns this.
Peurson Spearman 0.57
0.39
Table 3: Best-of-32 response selection for AUTO-J and standard RM, and their corre- lation with GPT-4 ratings.
The Best-of-N selection case from the test set (Eval-R) in Tab. 28 shows the usefulness of its rating in single-response evaluation. With more candidate responses given by the base LLM VVicuna-7B-vl.5
AUTO-J is able to select a better response measured by both GPT-4 rating and human observation.
$53 models their variants on httts:://tats/t
wttihhub.ii/appaaae http:iiiialppaaeaaaalaaal eval/ when the paper is written.
8

---

Published as a conference paper at ICLR 2024
Query:
Draft an email to my family telling them booked flights for Thanksgiving. I'll arrive on November 22 and leave on the 30th
Response (from G. (3B-soorryy))))
Response 2 (Preferred, from Claude-vl)
Subject: Exciting News: Flights Booked for Thanksgiving!
Here's a draft:
Dear Family.
Subject: Thanksgiving Travel Plans Hi everyone,
I hope this email linds you all doing well! I am thrilled to share exciting
with you all. have hooked my flights for Thanksgiving! I will be arriving on November 22 and leaving on November 30.
wanted to let you know that have booked my flights for Thanks- giving. I will be arriving on Wednesday, November 22nd and departing Thursday, November 30th. I'm looking forward to seeing you all and spending the boliday together.
I cannol wait to spend quality time with all of you and enjoy the de- licious Thanksgiving feast logether. There might few sumprises store for you all, so make sure keep open mind!
Please
anything I should bring there are any plans slready in the works for meals activities. I'm happy to contribute however can.
know
I have attached the flight itinerary lor your reference. Please feel free reach out me if you have any questions
Safe rravels anyone else coming into town. See you all very sapn! Love,
Wishing you all a very happy Thanksgiving!
Love.
YYour name]
(Your Name]
Feel free to modilyy the email us you see lit. Just let me know if you' like me make any changes to email before you send
Judgment by Auto-J:
1. The key factors to distinguish these two responses:
purpose: Both responses meet this criterion they both inform the family about the booked flights for Thanksgiving. However, Response 2 does better jub by asking lur any plans for meals or activities, which aligns more closely with the parpose of the email, which is inform and also plan for the Thanksgiving celebration.
tone: Both responses maintain casual and friendly tone, which is approprinte for the situation. However, Response I'ss tone is more formal and scrious, while Response 2's tone more casual and friendly, which is more suitable for a family email.
body: Both responses provide details about the tri in the body of the email. However, Response provides more detailed information about the trip, such as the arrival and departure dates, and also asks for any plans for meals or activities, which makes more informative and engaging.
personalization Response 2 is more personalized as it asks for any plans for meals or activities, which shows that the sender is family's 's plans and wants to involved information richness: Response 2 provides more detailed information about the trip and also asks for any plans for meals activitics, which makes rieber in information compared to Response 1.
interested
celebration.
2. The final decision:
Although both responses meet the criteria and are well wriiten, Response 2 is hetter hecause is more informative, engaging, and personalized, which makes more suitable for a family email.
So, the final decision is Response 2.
Table 4: An example of pairwise response comparison and the judgment given by AuTo-J, Response 2 is preferred than Response 1, we highlight the content that distinguishes these two responses most in red, and the key parts of the judgment that align with the user's preference in green.
Query: What's 's the hest way to cook salmon on the grill.
The best way to cook salmon on the grill is to preheat the grill to medium high heat, brush the salmon with olive oil, season with salt and pepper, and place skin down
Response:
grill. Allow the salmon cook 5-7 minutes, untill the skin is crispy and fish flakes easily with fork. Flip the salmon, and cook for additional 3-4 minutes. Serve with your favorite sides and enjoy.
Judgment from Auto-J: The response provided generally informative and offers a basic method for grilling salmon. However, lacks depth and detail in several key areas. A main sbottcoming is the absence of specilic temperatures or cooking times, which are crucial for achieving the perfect grilled salmon. The response vaguely states "medium high heat" and "5.7 minutes" for the first side cooking, and
"3-4 minutes' for the second side. More precise guidance would be beneficial for users who may not be experienced grill masters,
The response also fails to address the quality of the salmon that roquired for the recipe. It is essential to meation the type of salmon that would be best suited for this method of cooking. F Furthermore, the response does provide detailed information on bow to determine when the salmon is cooked. This could potemially lead to overcooked or undercooked salmon, which is comman problem for navice cooks. more interactive and personalized by asking the user about their specific grill or their experience with cooking salmon. This would have made the response more engaging and helpful. Additionally the response could have been more professional by citing credible sources providing references hack up the cooking instructions. This would have added credibility to the response,
The response could have
Lastly. the response missed an opportunity to provide a comprehensive answer by not addressing potential questions that the user might
have, such what to do the salmon sticks to the grill or how to ensure the salmon cooked evenly. In conclusion, while the respanse is a good starting point, it lacks the necessary detail and personal touch that would make it a truly excellent response. Rating: ||5||
Table 5:
An example of single-response evaluation and the judgment given by AUTO-J. We highlight the parts of the judgment that capture the crux of the matter in green.
7 CONCLUSION
In this work, we develop AUTO-J, a generative judge with 13B parameters for evaluating alignment, which is devised to address the challenges in generality, flexibility, and interpretability. We create new judgment dataset for diverse evaluation protocols, containing user queries and responses from different LLMs under massive real-world scenarios, and sell-structured natural language critiques. Experiments demonstrate that AUTO-J significantly outperforms both open- and closed-source source baselines models. Last but not least, we release wealth of resources to facilitate future research.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We thank Chunpu Xu and Yuqing Yang for supporting the human annotation process, and Yuan Guo for his support the post-submission maintenance and development of Auto-J. We also thank the anonymous reviewers for their valuable feedback and helpful suggestions. This project is supported by Qingyuan Research Project and Shanghai Artificial Intelligence Laboratory.
REFERENCES
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 05862, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitutional ai: Harmlessness from ai feedback. arXiv preprini arXiv:22/ 2022b.
Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. Re- evaluating evaluation text summarization. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9347-9359, Online, November 2020. URL
Association for Computational Linguistics. doi: 00.186653/112022.mmmmpp/// T/aclanthology.org/22200
emnlp-main. 751
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint 1oX4v:66000077777 2016.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open- chatbot impressing gpt-4 with 90% * chatgpt quality, March 2023. |RL|https/
//Lmsys.org/bl.gg 00--0---03--viiii
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.0866 2023.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16359 2022.
Jacob Devlin, Ming-Wee Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.1 10.18633v1/N19.1 1423. URL https
//aclanthology.co wrw.o/g...rg///99 ONg/N19-/111 1423.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305. 14387, 2023.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In International Conference on Machine Learning, pp. 5988-6008. PMLR, 2022.
Jinlan Fu, See-Kiong Ng. Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04466 2023.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In
International Conference Machine Learning, pp. 10835-10866 PMLR, 2023.
10

---

Published as a conference paper at ICLR 2024
Alex Havrilla.
2023. URL httoss//nggginnffeeccoottttssss
Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Il via a human-preference dataset. arXiv preprint arXiv:2307.044: 37.00407777 2023.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richard Nagyfi, et Openassistant conversations demmorainns large language model alignment. arXiv preprint arXiv:2304.03327, 2023.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequeene pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 787 1-7880, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/w1/2022.ccc main.703. URL 2020. acl -ainn. 703.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20055, 2023.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https:://aclathhhhtt/// 000.ooo0yyoo///-111333
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint drXiv.230.16633, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17Xiv:1711.051 2017.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv: 2at::2222.0932.2 2021.
Jun Ping Ng and Viktoria Abrecht. Better summarization evaluation with word embeddings for rouge. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1925-1930, 2015.
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feed- back. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Ad- vances Neural Information Processing Systems, volume 35, pp. 27744 Curran Asso- ciates, Inc., 2022. UtL/htttp://prroceeddngg neurips wwtppppppcpppppprreaaeepp.p httpeeerriieeeaapee//////////// htt.:/aee5..a331111168022221-333 Confeeerooooeoeeeeneeee pdf
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting the the Association for
Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 05111//007303.17333333 URL https::t///att https::/aacaanthologg org/P02-
11

---

Published as a conference paper at ICLR 2024
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computationnll Linguistics: Human Language Technologies, Volume / (Long Papers), pp. 2227-2231 New Orleans, Louisiana, June 2018. Association for Computational Linguisties. doi: 10. 00.18633/1//118 1202. htt/htpss
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference High Performance Computing, Networking. Storage and Analysis, pp. 1-16. IEEE, 2020.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505-3506, 2020.
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. Zero-offload: Democratizing billion-scale model training, In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pp. 551-564, 2021.
William Saunders, Catherine Ych, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.085 05802, 2022.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feed- back. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 3008 3021. Curran Asso- ciates, Inc., 2020. URL htpp://ees// hewpeeeiddinnnnnnnii..ss.... wwt.uripssccppppprrr..eppp cccpappe__aaiieeeeeeeeeeeeeeeeea wwwes/lllaaaaeee pwtp:/pppr////// www.ee/ppapr///2///1// 11888555566228888
pdf
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction followwig Ilama model. 2023.
www.:tttw...ottttaeeeeeeeteettee http:///ttorlaattttfrrrr_aaaaee https://github.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307..09288 2023b.
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305. 230.:2305..7926 2023a.
Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O' O'Brien, Ramakanth Pasunuru, Jane Dwivedi- Yu, Olga Golovneva, Luke Zettlemoyer, Maryam Faran-Zarandi, and Asli Celikyilmaz. Shepherd: A critic for language model generation. arXiv preprint arXiv:2308. 04592, 2023b.
Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang. Xing Xie, et al. Pandalm: An automatic evaluation benchmark for Ilm instruction tuning optimization. arXiv preprint arXiv:2306. 05087, 2023c.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212. 10.6600 2022.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint ar3iv:2304. 122234444.2244...000 2023.
12

---

Published as a conference paper at ICLR 2024
Seonghyeon Ye, Yongrae Jo, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, and Minjoon Seo. Selfee: Iterative sevisigg Ilm empowered by self-feedback generation. Blog post, May 2023. iot.ub.e////eeee////
URL https::/kaistaai....
Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34:27263-27277, 2021.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. International Conference Learning Representations, 2019.
Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yong- bin Li. Wider and deeper IIm networks are fairer Ilm evaluators. arXiv preprint arXiv:2308. 01862.
2023.
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306. (00,22360006855, 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et Lima: Less more for alignment. arXiv preprint arXiv:2305. 11206, 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffiey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 1e.Xiv:1009008555 2019.
13

---

Published as a conference paper at ICLR 2024
A SCENARIO DESCRIPTION
Summarization
uss_aummarization
Write a summary for reddit post.
sommarization Dole,_ summarization
Write summary for piece of text. Write note to summarize piece of text.
Exam Questions
math reasoning
Write an answer with step-by-step reasoning process for math question. Solve an exam question (like fill in the blank, multiple choice, peoblem solving, etc) with math involved. Solve an exam question fil-innthe-blank, multiple choice, problem solving, etc) with no math involved.
exam, questiqnninnnoooo with math
exam question without math
Rewriling
WAL simplification
Reduce the complexity of the vocabulary and sentence structure of text while retaining its original meaning. Polish a piece text to make more fluent, natural, and reulable. Rewrite given text with a specific instraction.
language polishing
instructional rewriting
text_ correction paraphrasing
Correct the potential crrors in a piece text.
Paraphrasing given text.
Code
code_ sidpliiiittoiiaiionnn
Rewrite piece of code to make more concise and easy to understand. Write piece of cadee based on the given description.
sode_ generation
explaining code
Write an explanation for piece of code.
code_correction rewriting
Correct the potential
piece code rewrite the code user's requirements.
code_to, to toc_ttee__oooeeeeeeoooo translation
Convert the given code into another programming language.
Creative Writing
writing_somg__ lyniclyries
Write song lyrics. Write a post that will be posted on social media such as Twitter, Instagram, Facebook or LinkedIn. Conduct creative writing task, like writing stories, pocms, dramas, novels, screenplays, etc. Answer questions or write texts under counterfactual premises.
writing_socii social _ost post
general_ treativewwitinn counterfactull
writing personal_esssee cassy
Write an essay that explores topics through personal experiences, insights or undorstanding. Write blog post the website.
writing_bog__tt
writing. advenisemnnt
Write un advertisenent for product or service.
writing markating materralss
Write marketing materials that help you communicate your brand's products services to your target market.
protintggrecttationnn. script
Write a spocch presentationn script for public spocch.
Functional Writing
writing_product prsducc__esccpptionn
Write product description that describes and explains your product service.
writing, news, article
Write news article for the newspaper.
writing. biography
wiiting_biugraphy
Write biography for a person.
writing pegtin document
Write legul document involving one or multiple parties that can be relied upon in court. describes the function and structure technical product.
writing, technical document
Write technical document
writing_job cob_applicttion
Write job application for your job search.
writing_scientific papee
Write scientific paper that shares your own original research work with other scientists.
general funetiooal_ writing
Conduct functional writigg task, like proposals, reports, memos, resumnes polls, questionnaires, scheduless, etc.
writing cacil_coooko_g_eeeeeee
Write a cooking recipe that teaches people how to prepare a meal.
General Communication
asking how question
Give relevant and complete instractions when ask 'how to do' something. Respond well to users when they scek advice.
socking_adve advice
verilying fact
Verily the given fact true false.
open question
The user's 's query is an open domain question with no attached passage or article. a topic, issue, material, text etc.) given by the user.
analyzing general
Lemerrl
Analyze certain thing
explaining_genenal
Explain something the user wants to know.
brainstorming
Brainstorm ideas or items for a given topic.
roleplay
Pretend to be specific person, churacter, profession identity, and complete the required task on this busis.
planning
Write plan for an event or activity.
chitchat
Chitchat with user. Give recommendations to users. Provide value judgment on given topic or statement.
recommendation
value judgment
NLP Tasks (including "others")
ranking
Sort some things, according to some criteria.
text_io textttt__e____eeeeeeeeeette translation
Translate the given text into another language.
data annlysis
Analyze certain data given by the user.
classification iddentifccatonn
Clussiyy or identify one or multiple objects given by the user into specific categories. Generate title for the given text based on description of the work.
title generation
question_ generation
Generate one or multiple questions based on the given topic or attached text. Answer questions that can be directly answered by the attacbed passage. Extract the key words from piece of text.
reading compechension
keywords, extraction
information, extraction
Extract one multiple user- specifid categories information
piece of text attached the user's query. Extract the high-level topics or themes from given text, i.e., what kind of topics are discussed in the text.
topie_modee modeling
Table 6: Detailed description for each scenario.
14

---

Published as a conference paper at ICLR 2024
socnario
train
test
socnario
train
test
sccuario
train
test
others
317
79
writing, www.keoooooccieeeeee recipe
40
classification identification
24
6
functional_ writing
128
32
explaining,coe code
40
10
language_ ppolishing
22
4
brainstorming
writing, legal, document
22
soeking_ advice
88
25
asking Boking_how_ir to question
40
10
writing_ productt description
20
5
open_ question
writing_ prwsertttionnn script
data analysis
18
5
esplaining sglainiinggeeerral
17
writing mocial_meiia, post
wriling marketing materials
17
instructional, rewriting
15
question_ generation planning
38
mote summarization
17
verifying_ fact
13
paraphrasing
17
5
49
13
writing Belinn_blog_Pee blog_post
36
9
writing_tochnical tochnical document
17
5
ritle generation
writing job application
text simplification
16
5
code _el__eecration
48
12
writing personal_ _ssay
36
10
information extraction
16
2
roleplay
value__judee jalue_judeementt
35
writing biography
16
4
rejecting
12
code Jo code translation
32
9
lext correction
12
6
creative writing
12
writing advertisement
31
8
reading, comprehemsion
12
3
exam, question_ without_ math
12
writing_emml
8
keywords, extraction
12
3
writing song__ognnn... Jyties
44
11
recomendationn
29
8
topie_ modeling
10
3
text to text translation
ranking counterfactual
writing scientiffe_pape
10
text summarization
43
12
26
7
peer_ review
2
exam question with_mmtth math
24
code simessippiiiiiaionn
math mettores
12
newiserticle writing news, article
24
6
overil
2383 623
Table 7: The scenario distribution in the training and test for scenario classifier, note that "rejecting" and "peer_review" are two early-defined scenarios that have been removed by us.
B TRAINING DETAILS OF SCENARIO CLASSIFIER
In this section we describe in detail the training process of the scenario classifier mentioned in $3.2
We model the scenario classification task as a generation task. The classifier are re- quired to generate only the scenario name when given the query, with the prompt as "Identify the scenario for the user's query, output 'default' if you are uncertain. An An'nnQuery: (wnuuury:n/nnnnnuut)nnnn.... hww.iiiinnntttttnnnnnnaooooooo " (the "default" scenario in the prompt the early naming for "others" scenario).
In general, the training involves three steps:
1. We first brainstorm about 10 seed queries for each scenario with the help of ChatGPT, and train model that can directly output the scenario name when given a query as conditional generation task on this small synthetic dataset.
2. Using the trained model, we conducted an initial classification for queries in Chatbot Arena Conversations and ShareGPT as they cover much more scenarios than other datasets. Based on this preliminary classification, we randomly select up 50 queries from each scenario for secondary manual validation, involving data cleaning and correcting misclassified labels.
3.
We combine the newly-collected dataset and the small synthetic dataset in step 1, and retrain our final classifier, We divide queries in each scenario in an 8:2 train/test split (Tab. 77) The accuracy and Fl of the final classifier on test set are 72.55 and 74. respectively.
Our scenario classifier trained from 1LLMMA2-133 (Touvron et al. 2023b). and we set the max sequence length as 2,048, and the max length for query 2,048-50=1,99 both in training and inference. If a query Q with length L exceeds that limit, we truncate it from the middle and replace the dropped part with "..." since the front and end of the sequence usually contain more important information for identifying scenario of the (such as the user's 's instruction): QILL > (Q1.999: .... QL-1000:L]]
We train the scenario classifier for epochs on the training set, and set the batch size as 64. Without warmup steps, set the initial learning rate le-5 and cosine decaying to 0 by the end of training. The optimizer is AdamW W with 31 = 0.9, B2 = 0.95 as in training AUTO-J, and we also use the speedup and GPU memory saving techniques like DeepSpeed Zero 3, BF16, TF32, and gradient- checkpointing. The loss is only calculated on the output end as well.
"This dataset is collected from https: http://hhaaegpt.oom com/ containing shared conversations with ChatGPT GPT-4. We use public available subset of it.
15

---

Published as a conference paper at ICLR 2024
Eval-P
Eval-C
Model
Seen
Unseen
Seen
Unseen
Unseen scenarios:
Randomly select one scenario from each group.
Auto-J (Complete Version)
54.5
56.8 55.7
146/200 143/200
25/32
Auto-J (Training w/o unseen scenarios)
53.5
25/32
Unseen scenarios: Scenarios of NLP Tasks group.
Auto-J (Complete Version)
54.2
57.6
136/188
35/44
Auto-J (Training w/o unseen scenarios)
54.2
54.9
130/188
38/44
Table 8: Auto-J's generality on unseen scenarios. We train two new variants by removing a set of scenarios from the training data, and compare their performance with the complete version of Auto-J that has been trained on all data. We report the agreement rate with human label on the pairwise response comparison task (Eval-P), and the winrate against ChatGPT judged by GPT-4 on the critique generation task (Eval-C).
You given the criteria crafl good responses for this type query from [scenario description]
The criterin follows:
[Criteria start]
(criteria scenario]
(Criteria end]
Table 9: Scenario criteria as system message in prompt.
C AUTO-J's GENERALITY ON UNSEEN SCENARIOS
is quite important to see how Auto-J performs on the scenarios that are not included its training data. To investigate this research problem, retrain variants of Auto-J by holding out sets of unseen scenarios.
1. We randomly select one scenario from each scenario group as the unseen scenarios, and retrain Auto-J with the remaining scenarios. This in total leads to 8 unseen scenarios in testing.
2. We take the complete "NLP Tasks" group as the scenarios, and retrain Auto-J with the remaining scenarios. This in total leads to 11 unseen scenarios in testing.
Under both setting, select the complete version of Auto-J as the baseline to compare with. We report the agreement with human annotattion labels on the pairwise response selection task (Eval-P) and the win rate against ChatGPT in the critique generation task (Eval-C) judged by GPT-4. The results are shown in Tab. g
Compared with the complete version of Auto-J, the two re-trained variants only show slightly degraded performance on the two evaluated tasks both on the seen and unseen scenarios, which indicates that Auto-J can generalize well to scenarios unseen during training.
D PROMPTS
Tab. 96 shows different prompts. Tab. 9 guide GPT-4 to generate training data ($3.2) Tab.
and 10 provide GPT-4 system messages, where the scenario and the criteria are defined. Tab.
show GPT-4 user messages, providing the instance-related information. Tab. [4 15 elaborate the
prompts (55.)), which all baseline models use to generate the testing results. Tab. 16 is used for GPT-4 evaluation that conducts pairwise comparison between AUTO-J with baseline.
16

---

Published as a conference paper at ICLR 2024
The following are the specific criteria for this type of query, focusing the content aspect:
1. clarity: The writlen plan should clearly outline the objectives. tasks, and timeline of the event or activity, easuring that the reader can easily understand the proposed plan.
2. fcasibillty:: The written plan should propose realistic and achievable steps and actions, considering available resources, coustraints, and logistical factors.
3. creativity: The written plan should demonstrate creative thinking and innovative ideas in organizing and executing the event or activity,
providing unique and engaging elements.
4. thoroughness: The written plan should cover all osscatial aspects and details of the event or activity, like logistics. budget, promotion, and
participant engagement.
The following the specific criteria for this type of query, focusing the format aspect:
1. structure: The written plan should be well-structured, with a logical flow of ideas and clearly defined sections or headings for different
compoments of the plan.
2. layout: The written plan is encourgeed to use headings, hullet points, lists, tables, or other devices to enhance readahility and coherence.
The following are the basic general criteria:
completeness of instruction following: For all key instructions (e.g.. answer multiple questions or perform multiple tasks) and explicit comstrainis (e.g. word count, response length limil, word usage, ouiput format, etc.) provided by the user, the response should be complete in following all them without any omission.
2. accuracy: All contents provided or mentioned in the response should be accurate and correct. This criterion is not upplicable if the user ask for an opinion subjective response.
3. information richness: The response is encouraged to provide rich, detailed and professional information, by providing examples, explanations, citations, and additional information. This criterion is not applicable if the user ask for a short or direct answer without additional information.
The response should be devoid of affensive, insulting, or inappeopriate content and should strictly avoid any form discrimination, including but limited racial, gender, ago, secual orientation, religious, disability, socioconomic status, cultural or ethnic. and language baved discrimination.
4. barmlessmess:
quality: The response should he grammatically correc, of spelling emors typos, use punctuation marks properly and consistently. The overall text should be flucnt and coherent, and consistent in its style, tone and provided information.
user intention inference: If the user's 's intention is not elearly expressed by the query. the response should provide some relevant information. reasonable inference and ask more information for clarification This criterion is applicable if the user's intention is clearly expressed by the query.
Table 10: The complete criteria for "planning" scenario.
You are assessing twn submittee responses on given user's 's query hased on the criteria you have known and judging which response is hetter they are tied (ineluding both good and both bad). Here is data:
[BEGIN DATA]
New
[Queryl: [query]
[Response I]: |response I]
wew
|Response 2): fresponse 2)
[END DATA]
Here are the instructions to assess and compare the two responses:
1. Review the two response and the given criteria to identify **only*** the criterion(s) that can significantly distinguish the two responses. Ignore the criteria that camnot signilicantly distinguish the two responses (like both neither responses meet criterion) and the criteria that not suitable for this query.
important factors that can significanlly distinguish two responses, especially the factors suecialized for the user's query and the two responses.
Besides the given criteria, brainstorm provide
decision which response is better they tied (including both good and had). Begin your final decision statement with 'So, the final decision Response I/Response 2/Tic", Ensure that your decision aligns coherently with the compreheasive evaluation and comparison you'te 've provided.
Conclude your comparison hy providing
Table 11: Prompt used when collecting raw output for pairwise evaluation protocol from GPT-4.
E INPUT AND OUTPUT FORMATS
This section shows the input and output (judgment) formats (Tab. 18 20 where some examples also provided. These formats are supplemental details of $3.3
F TRAINING DATA STATISTICS
This section shows the train data statistics (Tab.
22
These supplemental details of $3.
17

---

Published as a conference paper at ICLR 2024
You are writing critiques for submirted response on given user's query. Here the data:
[BEGIN DATA]
[Query': (query]
[Response]: (response]
[END DAIA]
Here are the instructions you should Tollow:
commentt strengths. critiques specific possible by quoting details from the respoose and query. and don't include the criterion name.
You only
criliques shoricomings;
Table 12: Prompt used when collecting raw output for single-response evaluation from GPT-4.
Write meta-cribque by combining the following critiques for submitted response query, and grade the response:
[BEGIN DATA]
[Query]] [query]
vee
[Response]: (response)
<46
[Critique 1]: (critique 1]
<ax
[Criique 2]: [critique 2]
...
[END DATA]
You should give meta crilique merging the two critiques into comgrehensive critique for the response iluent langunge. Afler that. you should give final rating for the response on scale of to 10 by strictly following this format: "[[tating]]. for example: 'Rating: [[5]I".
Table 13: Prompt for asking GPT-4 to combine two critiques to comprehensive critique and give final rating.
G COMPLETE RESULTS AND CASES
Tab.
23
contains the complete comparison results of Fig.
6.2)
Tab. 24 28
provide the comprehensive details of 6.4
Tab. 24 shows the detailed ranking of Fig.
The complete cases of $6.4 are shown in Tab. 25
28
18

---

Published as a conference paper at ICLR 2024
SYSTEM MESSAGE
Please act an impartial judge and evaluate the quality of the responses provided two Al assistants the user question displayed below. You should choose the assistant that follows the user's instructions and unswers the user's s question betler. Your evaluation should consider lactors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Begin your evaluation by comparing the responses and provide short explanation. Avoid any position biases and ensure that the order which the responses were presented does not influence your decision. Do allow the longth the resooses influence your evaluation. Do as objective as possible, After providing your explanation, oulput your tinal verdict by striedl following this format: "[[A]|" if assistant A is better, '[[B]]' ussistant better, "[[C]]" for tie.
the
Be
USER MESSAGE
[User Question]
[question]
TThe Start of Assistant A's Answer]
(answer_ad
[The End of Assistant A's Answer]
[The Start of Assistant B's Answer]
[answer
[The End of Assistant B's Answor]
Table 14: Pairwise comparison prompt for baseline models.
SYSTEM MESSAGE.
quality of response provided by assistant the question displayed below. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of the respenee. Begin your ovaluation by providing short explanation. as objective as possible. After providing your explanation, please rate the response on scale to by strictly jollowing this format: "[[rating for example: 'Rating: [[5||".
act impertial judge and
USER MESSAGE
Question]
[question]
TThe Start of Assistant's s Answer]
(unswer]
(The End Assistant's x Answer]
Table 15: Single-response rating prompt for baseline models.
You helpful and precise assistant for checking quality of the leedback. Twa pieces of feedhack have been provded for the same response to particular query. Which one better with regard their correctness, comprehensiveness, specificity the query?
[User's query]
[query)
[The Assistant's Response]
(response]
[Feedback
[feedback 1)
Feedback 21
(feodback 2)
Please choose from the following options, and give out your reason the next line.
Feedhack I is significantly better.
B: Feedhack 2 is significantly hetter.
Neither significantly better.
Table 16: Prompt for GPT-4 to pick a better critique out of two.
19

---

Published as a conference paper at ICLR 2024
You are ussessing two submitted responses on a given user's 's query and judging which response betler or they are tied. Here is data:
[REGIN DATA]
vew
[Query
[Response II:
[Response
[END DATA]
Here are the instructions to assess and compare the two responses:
1. Pinpoint the key factors distinguish these two responses.
2. Canclude your comparison prowiding final decision which response is hetter, tied. Begin final decision with
'oo, the final docision is Response Response 2/ Tic".. Ensure that your decision aligns coherently with the comprebensive evaluation and comparison you'le provided.
Table 17: Input format of AUTO-J for pairwise response comparison protocol.
Write critiques submitted response on given query. grade response:
[REGIN DATA]
[Queryl: prompt
644
[Response]: response
[END DATA]
Write critiques for this response. After that, should give final rating for the response scale of 10 by strictly following this formar: "|[ratinll", for example: 'Rating: ||511""
Table 18: Input format of AUTO-J for single-response evaluation protocol.
Template unified format
1. The key factors to distinguish these two responses:
(factor 1): (explanation
(factorr 2): (explanation
2. The final decision: So, linal decision is Response 1 Response Tie. few summarizing remarks)
An instantiated example
The key factres to di distinguish these two responses:
- pointing out: Both responses point the changes made script and explain these changes solve the user's problem. However, explanation how "Nolnternctive" flag will any prompts during the execution of the script. accuracy of the solution: Response 2's solution is more accurate as the "-NoInternctive" flag is the correct way to prevent any prompts or interactive input during exccution PowerShell script.
Response provides
instruction following: Both responses follow the user's instructions to provide a solution that allows the script to run without waiting for user script automatically.
input. However, Response solution more line
coverage: Both responses provide solution to the user's problem, but Response 2's solution is more accurate and directly addresses the user's need to run the script automatically without user input.
relevance the user's problem: Response 2's solution is relevant user's problem directly addresses the need script automatically without user input.
code correctness: cesponses provide code, Respense
more appropriate
problem.
2. The final decision:
decision Response Although responses provide valid solutions, Response solution accurate directly addresses the user's need run the script automatically without user inpur.
Table 19: The unified judgment format for pairwise response comparison training data, as well as rewrrtinewww"" scenario.
specific example for the
20

---

Published as a conference paper at ICLR 2024
Query
give trip plan for days France
Response
Sure, here's a potential 5-day trip plan for France:
Day 1: Arrival in Paris and Exploring the City
Arrive at Charles de Gaulle sirport and take taxi or public transportation to your hotel in central Paris
Check into your hotel and freshen up
Visit the Eiffel Tower, one of the most iconic landmarks in the world
Walk along the Chumps-Ell Ellss--lyssess
lamous avenue known for its luxury shops and restaurants
Explore the Louvre, one the world's most famous muscums, and some of the world's most famous artworks
Dinner at classic Parisian bistro
Day 2: Day Trip to Versailles
* Take high-sped train from Paris Versailles
Visit the Palace of Versailles, magnificent pulace and gardens built by the French monarchy
Explore the Palace's grand halls and grand staircases
Walk through the beautiful gardens and have pienic
Return Paris
evening
Day 3:
Exploring Montmartre and the Artists' Quarter
Take a metro to Montmartre, a charming neighhorhood known for its artistic history
Visit the Basilique Sacré- Corur, beautiful church stunning views Paris
Explore the streets and alleys of Monimartre and visit the works of famous artists like Van Gogh, Dali, and Picasso
Have lunch traditional histro
Visit the Musée d' O'Oraay museum with an impressive collection of French art
Day 4: Trip the Côte d' Azur
* Take high-speed train from Paris the Côke Azur, also known as the French Riviera
Visit the Monaco Grand Prix circuit and see where the famous Formula One race takes place
Walk along the Promenade des Anglais, a famous seaside promenade
Have lunch scaside restaurant
Check into your hotel in the Côte d' Azur
Critiques from GPT-4 without seenarlo criteria as the system message
The response, while detailed and comprehensive, does not take into account the feasihility of the proposed itinerary. On Day 4, the plan suggests a trip from Paris
Côte d' Azur, which a significant distance and would consume considcrable amount of time. This could poteotially leave the traveler with little time enjoy the activities listed for the day. Furthermre, the response does not provide any suggestions for Day 5 of the trip. leaving the user without a complete 5-day plan as requested in the query.
Critiques from GPT-4 with scenario criteria as the system message
The respunse does not provide plan for the kilh day the trip. which signilicant omission given the user's request for 5 d-y trip plan. This makes the plan incomplete and not fully aligned with the user's query.
While the plan generally well-stretured, lacks clear timeline for each day's activitics. Providing specific timeframes for each activity would make the plan more practical and casier to follow.
The plan does not consider potential comstraints such
feasibility of visiting multiple mujor attractions in a single day, which may not he realistic given factors like travel time, waiting lines, and the time needed to fully appreciate each site.
The plan could also be more creative and personalized. For example, it could suggest less touristy, off-the-beaten-patt attractions or activities that could provide unique and memorable experience.
Lastly, the plan does not provide any information on the budget, which is an essential aspect of trip planning, Including estimated costs for tronsportation. accommodation, meals, entrance foes would make the plan more thorough and useful.
The combined final judgment
The response provides detailed and comprehensive itinerary for a trip to France, however. it falls short several areas. Firstly, the plan does not fully align with the user's query it only provides irinerary for four days, leaving the fifth day unacoounted for. This is significant omission given the user's ss spocific request for a 5-day trip plan.
Secondly, the feasibility the proposed ilinerary is questionable. For instance, the plan suggests a trip from Paris Cote A'Auur on Day which is a considerable distance and would consume significunt amount of time. This could potentially leave the traveler with little time cajoy the activitics listed for the day.
The plan also does consider polential constraints such the feasibility of visiting multiple major attractions in single day. Factors like travel time, waiting lines, and the time needed to fully appreciate each site have not been taken into account. Providing specific timeframes for each activity would make the plan more practical and to follow:
Furthermore, the plan lacks creativity and personalization. It could be improved by suggesting less touristy, aff-the-beatcn-part attractions or activities that could provide unique and memorable experience.
Lasily, the plan does not provide any information on the budget, which is an essential aspect of trip planning. Including estimated costs for transportation. accommodation, meals, and entrance fees would make the plan more thorough and uselul.
cunclusion, while the response well structured and detailed, lacks completeness, feussiility, and personalization. Therefore, the response could he significamlly improved by addressing these issues. Rating: [[5]]
Table 20: An example for collecting complete evaluation judgment for single-response evaluation. We show the query, response, raw GPT-4 critiques with and without scenario criteria as the system message, and the final judgment by combining the previous two critiques.
21

---

Published as a conference paper at ICLR 2024
Label Distribution (Label, # of Samples)
Win
1594
Lose
1596
Tic
246
Source Dataset Distrihution (Source, # of Samples)
Chatbol Arena Conversations
2801
OpenAl Summury
100
OpenAl WebGPT Synthotic GPT-J
45
PKU-SafeRLHI
158
Stanford SHP
81
251
Sccnario Distribution (Name, # of Samples)
ranking
100
open question
100
text correction
18
recommendation
100
post, summarization
100
writing produet description
16
creative_we writing
writing song lyrics
language polishing
plamnin
100
functional writing
94
code. to code. translation
15
brainstoeming
writing cooking recipe
writing legal document
exam question_ Seipt_ttt___aaaaee math
100
code. corrcction rewriting
86
writing blog_post
13
roleplay
100
writing_ personal essay
84
tille gel__geacation
12
text summarization
analyving general
67
writing social media
asking_howe _o_ so__quettinn
100
explaining_coo code
59
reading_ comprehccsiion
11
chitchat
information extraction
writing technical_ document
verifying, fact
100
writing email
51
text simplification
10
value, jjudg_jddmenttt
writing job, application
46
keywords extraction
code_pcaeration
100
classification, identification
44
writing saapgg_ssetttteppeeeeeeeeeeeeeee
text to _o_ett____________ translation
100
writing presentation
42
writing marketing
baatkkeiinn_mttrillllee materials
math_ reasoning
question with with_math
41
topic modeling
question_
100
data analysis
39
writing sows_article
counterfactual
instructional rewriting
30
note summarization
seeking advice
100
paraphrasing
27
code simplification
explaining_penee peneral
writing_advertisenn advertiseement
20
others
Table 21: Statistics for pairwise training data: the distribution labels, source datasets, and scenarios.
Distribution (Score, Samples)
2
137
178
210
(5.5)
6
241
27
8
4
10
3
Sccnario Distribution (Name, # of Samples)
code peneration
explaining_oom code
writing technical document text sixpisimpliiiaatin
cxplaining_ gencral
23
functional writing writing song lyrics ranking planning
18
15
open question
language, language_polshiine code. to, code, tramslation writing blog post reading_ ceading_comprebeniie comprebeenioonnn
seeking_adwee auvice
23
18
15
reasoning
chitchat
21
classification, identification
iunsiffeatio_ddentifeniiooo
17
14
value value_judgmeett judgment brainstorming
exam question with math writing cooking recipe
papie__mdeelng
writing advertisement title generateonn
creative_ sesaive_witting
20
writing email information,
17
14
roleplay
keywends
verifying fact
20
paraprrssing
17
writing legal ddeculent
14
counterfactual
code wwwrection_reeeeeeee reeerttinnnwwiitigg
writing news writing social mwsdia_postttleeeeeeeaeeeesspsse code, simplification
asking_ httpe/ooooo____ooooee _o___uuttoqoonn
data, analysis
16
14
exam, _uur__quesion____ without math
19
writing product description
16
text summarization
instructional rewriting
writing scicntific paper
recommendation
18
writing presentation script
16
writing marketing materials mote summarizationn writing
8
question, generation
analyving general writing_joh. joh sp_appiicntion
text teeteee__eeee translation
18
16
writing_higgmphy hiogmphy
writing__prsoonll
text correction
others
Table 22: Statistics for single training data: the distribution GPT-4 ratings, and scenarios.
22

---

Published as a conference paper at ICLR 2024
GPT-4 judgments
Human judgments
Baseline
SelFee
Vicuna
SelFee
Vicuna
judgment
Win
Tie
Lose
Win
Tie
Lose
Win
Tie
Lose
Win
Tie
Lose
Summarization
12
0
0
0
3
1
11
Exam Questions
1
0
Code
20
0
0
15
0
5
15
2
3
11
Rewriting
18
0
2
18
0
2
14
5
1
14
4
2
Creative Writing
33
0
3
29
0
7
26
7
24
9
Functional Writing
37
0
3
28
0
12
37
2
32
General Communication
0
0
43
2
39
6
NLP Tasks
40
0
4
35
0
9
29
7
8
22
15
Overall
218
13
180
51
182
28
22
160
48
Baseline
L2Chat
ChalGPT
L2Chat
ChatGPT
judgment
Win
Tic
Lose
Win
Tic
Lose
Win
Tie
Lose
Win
Tie
Lose
Summarization
0
0
Exam Questions
11
0
1
10
0
2
3
3
Code
18
0
2
16
0
4
11
5
4
8
6
6
Rewriting
17
1
2
14
0
6
10
7
3
9
8
Creative Writing
0
2
14
Functional Writing
0
8
2
16
23
13
23
14
General Communication
41
0
7
36
0
12
28
15
5
29
10
Tasks
37
1
6
35
1
8
13
22
9
16
17
11
Overall
196
2
34
171
5
56
114
80
38
117
72
43
Baseline
WizardLM
GPT-4
WizardLM
GPT-4
judgment
Win
Tie
Lose
Win
Tic
Lose
Win
Tic
Lose
Win
Tic
Lose
Summarization
10
0
2
0
4
11
1
0
2
Exam Questions
11
1
0
0
9
3
3
2
2
8
Code
17
0
3
0
8
7
4
Rewriting
Creative Writing
31
1
4
23
0
13
17
17
2
11
9
16
Functional Writing
30
1
9
17
1
22
24
12
4
27
6
General Communication
38
1
9
28
0
20
32
14
2
35
3
10
Tasks
2
7
23
1
20
14
11
8
11
25
Overall
187
7
38
126
2
104
130
74
28
107
41
84
Table 23: Detailed comparison results between critiques generated by AUTO-J and baselines for single-response evaluation. Results left side are GPT-4 judgments, and results on right side are human judgments. Vicuna, L2Chat, and WizardLM respectively stand Vicuna-13B-vI.55 LLaMA-2 Chat- 13B, and M--33dLM-133Bv
23

---

Published as a conference paper at ICLR 2024
Model
Auto-J Rating
GPT-4
Rank
Win-rate
GPT-4
Auto-J
XwinLM 70b V0.1
5.694
95.57
1
ILaMA2 Chat 70B
5.678
92.66
2
2
MwinLM 13h V0.1
5.647
91.76
3
OpenChat V3.1 13B
5.532
89,49
4
8
WizardLM 13B V1.2
5.547
89.17
5
6
Vicuna 33B v1.3
5.570
88.99
6
5
Humpback LLaMa2 70B
5,498
87.94
XwinLM 7b V0.1
5,584
87.83
8
4
Open Bonoddy-LLMMA-70B---10000
5,448
87.67
9
14
OpenChat V2-W 13B
5.533
87.13
10
7
-3
OpenBuddy-LLaMAA46550
5,458
86.53
11
13
WizardIM 13B V1.1
5.497
86.32
12
12
OpenChat V2 13B
5.519
84.97
13
9
Humpback LLaMa 65B
5.379
83.71
14
19
Vicuna 13B v1.3
5.388
82.11
15
18
BpenBuddy-LLLMMA300B-
5.391
81.55
16
17
LLaMA2 Chat 13B
5,518
81.09
17
10
OpenChat-13B
5,437
80.87
18
15
OpenBuddy-Falcon-44000
5.373
80.70
19
20
UhraLM 13B
5.342
80.64
20
22
OpenChai8192-13B
5,429
79.54
16
OpcnCoderPlus-15B
5,357
78.70
22
21
OpenBuddy-LLLMA2-1BBBB
5.340
77.49
23
23
Vicuna 7B v1.3
5.332
76.84
24
25
WizardLM 13B
5.247
75.31
25
32
JinaChat
5.319
74.13
26
26
airoboros 65B
5.318
73.91
27
27
airoboros 33B
5,289
73.29
28
30
Guanaco 65B
5.313
71.80
29
29
LLaMA2 Chat
5.334
71.37
30
24
Vicuna 13B
5.314
70.43
31
28
OpenBuddy-Falcon--700
5.214
70.36
32
34
Baize-v2 13B
5.165
66.96
33
38
LLaMA 33B OASST RLHF
5.173
66.52
34
37
Minotaur 13B
5.210
35
36
Guanaco 33B
5.212
65.96
36
35
Nous Hermes 13B
5.271
65.47
37
31
-6
Vicuna 7B
5.237
64.41
38
33
Baize-v2 7B
5.083
39
39
LLaMA 33B OASST SFT
4,985
54.97
40
41
Guanaco 13B
5.027
52.61
41
40
ChatGl M2-66
4.846
47.13
42
46
Guanaco 7B
4.943
46.58
43
43
Falcon 40B Instruct
4.934
45.71
44
44
Alpaca Farm PPO Sim (GPT-4) 7B
4,978
44.10
45
42
Pythia 12B SFT
4,809
41.86
46
47
Alpaca Farm PPO Human 7B
4,907
41.24
47
45
-2
Cohere Chat
4.524
29.57
Cohere
4,522
28.39
49
52
Alpaca 7B
4.658
26.46
50
48
-2
Pythia 12B OASST SFT
4.620
25.96
51
49
-2
Falcon 7B Instruct
4.537
23.60
52
50
Baichuan 13B-Cha
4.291
21.80
53
Table 24: Values and ranking by Auto-J and GPT-4 for open- LLMs AlpacaEval. Value of AUTO-J is the model's average rating on AlpacaEval dataset assigned by AUTO-J single-response evaluation protocol, and value of GPT-4 is the model's win-rate against Davinci003 determined by GPT-4 on A AlpacaEyal dataset. Rank
Rank
24

---

Published as a conference paper at ICLR 2024
Query
Drafl an emaiil to my family telling ibem I booked llghts Thanksgiving. arrive on November 22 leave on the 30th
Response (from EPP4ALL-13B-sHoog
Response 2 (Preferred, from Claude-v1)
Subject: Exciting News: Flights Booked for Thanksgiving!
Here's a draft:
Dear Family,
Subject: Thanksgiving Travel Plans Hi everyone.
I hope this email linds you all doing well! I am thrilled to share some exciting news with you all. have booked my flights for Thanksgiving! I will be arriving on November 22 and leaving on November 30.
wanted let you know have booked my flights for Thanks- giving. will be arriving Wednesday, November 22nd and departing on Thursday, November 30th. I'm looking forward to seeing you and spending the holiday together.
I Teannot wait spend quality time with all of you and enjoy the de- licious Thanksgiving feast together. There might be few supriscs store for you all, so make sure keep an open mind!
Please let me know if there's is anything I should bring or if there are any plans already in the works for meals activities. I'm happy to contribute bowever I can.
I have attached the llght ilinerary for your reference. Please feel free reach out Wishing you all very happy Thanksgivigg! Love,
if you have any questions or concerns.
Safe travels anyone coming town. See you all very soon! Love,
[Your name]
[Your Name]
Foel free to modify the email as you see fit. Just let know if you'd d like mo to make any changes to the email before you send it.
Judgment by Auto-I (\)
1. The key factoes to distinguish these tWo responses:
purpose: Both responses meet this criterion as they both inform the family about the booked flights for Thanksgiving, However, asking for any plans for meals activities, which aligns mere closely with the purpose of the email, which is to inform and also to plan for the I Thanksgiving celebration.
Response 2 does better
tone: Both responses maintain casual and friendly tone. which appropriate for the situation. However, Response I's tonc is more formal and serious. while Response 2's tone mere casual and friendly, which is more suitable for family email.
body: Both responses provide details about the trip in the body of the email. However, Response 2 provides more detailed information about the trip, such the arrival and departure dates, and also asks for any plans for meals activities, which makes more informative
and engaging.
personalization: Response 2 is more personalized as it asks for any plans for meals or activities, which shows that sender interested in the lamily's 's plans and wants to be involved in the celebration.
information richness: Response provides more detailed information about the trip and also asks for any plans for meals or activities, which makes richer in information compared to Response 1.
2. The final decision:
So, the final decision is Response Althouph both responses meet the criteria and are well-written, Response is better because is more informative, cogaging, and porsonalized, which makes it more suitablc for family email.
Judgment by ChatGPT (X)
and enthusiastic tone. The email includes subject line that captures atlention and clearly states the purpose of the email. also mentions surprises and encourages the family to keep an open mind, adding a touch excitement. The email includes an attached flight itinerary for reference and invites the family to reach with any questions concerns. Ovenil, Assistant A's response is helpful, relevant, accurate, and provides good level detail.
Assistant A's response more detailed and provides
Assistant B's response is also helpful and relevant. It inclodes a subject that states the purpose of the email and provides the necessary information about the dates of arrival and departure. It also offers to contribute and asks if there any plans or items bring. However, atlached llight itinerary.
lacks the same level of enthusiusm and detail as Assistant A's response. does not mention surprises or provide an While it is trrighfooward and concise email, it could benefit from more personalization and excitement. Based on the factors mentioned above, Assistant A provides a better response. [[A]]
Judgment by LLaMA SAMMA22-Chan 2-Chat-133 (X)
Sure, I'd be huppy help! Here's my evaluation of the responses:
Both Assistant A and Assistant provide helpful and relevant responses to the user's 's question. However, there are some differences in the
quality their responses.
Assistant A's response is more detailed and creative. The email draft provided is well written and includes a personal touch with the mention more formal and polished, which is appropriate for family email. Additionally, Assistant
potential surprises. The language used A provides flight itinerary for reference, which is thoughtful touch.
Assistant B'ss response, on the other hand, is more straightforward and to the point. The email draft provided simple and easy to understand, but lacks the personal touch and creativity of Assistant A's response. Assistant B also uses more casual tone, which may not be appooriiat for a family email.
Based these factors, would say that Assistant provides better response to user's 's question. Assistant A's response is more detailed, creative, and formal, while Assistant B's response is more straightforward and casual. Therefore, my final verdict is: [[A]]
Table 25:
The complete judgments given by different models on a pairwise response comparison example. We mark if each judgment agrees with human preference as well, where / stands for agreement and X stands for disagreement. (Part I)
25

---

Published as a conference paper at ICLR 2024
Query
Draft an email to my family selling them I booked lights for Thanksgiving. I'll arrive on November 22 and leave on the 30th
Response (from EET4ALL-13B-soooe
Response 2 (Preferred, from Claude-v1)
Subject: Eaciting News: Flights Booked for Thanksgiving!
Here's a drift:
Dear Family,
Subject: Thanksgiving Travel Plans
I hope this email finds you all doing well! I thrilled to share some exciting news with you all. have booked my flights for Thanksgiving! I will be arriving on November 22 and leaving on November 30.
Hi everyone,
wanted to lot you know that I have booked my flights for Thanks- giving. will be arriving Wodnesday, November 22nd and departing on Thursday. November 30th. I'm looking forward to seeing you and spending the holiday together.
cannot wait spend quality time with of you and enjoy the de- licious Thanksgiving feast together. There might be few suprises in store for you all, so make sure to keep an open mind!
Please let me know if there's 's anything I should bring or if there are any plans already in the works for meals or activities. I'm happy to contribute however can.
I have attached the flight itinerary for your reference. Please feel free to reach out to me if you have any questions or concern.. Wishing you a very happy Thanksgiving!
Safe travels to anyone else coming into town. See you all very soon! Love,
Love.
|Your name]
[Your Name]
Feel to modifyy the email as you see you'd d like me to make any changes to the email before you send it.
know if
Judgmeet by Pandal (A)
Hnt Reason: Response is more concise and follows the instruction more closely.
MA# Reference: Subject: Flights Booked for Thanksgiving!
Dear Fumily.
I hope this email finds you all doing well! I am thrilled to share some exciting news with you all. I have booked my flights for Thanksgiving! will be arriving on November 22 and leaving on November 30. I cannot wait to spend quality time with all of you and enjoy the delicious 'I Thankagiving feast logether. There might be a few surprises in store you all, so make sure to keep an open mind!
I have attached the flight itinerry for your reference. Please feel free to reach out to me if you have any questions or concern..
Wishing you all a very happy Thanksgiving!
Love,
[Your Name]
Judgmeet by Vicuma- 1315-y1.5 (X)
Afler comparing the responses, lind that Assistant A's response well -tuec-strcrrred, with clear subject line and a warm greeting. It provides the necessary information about the trivel plans and expresses excitement about spending time with the family. The response also includes touch creativity with the mention of surprises. Assistant B's response is also good. but locks the depth and creativity of Assistant A's response. The response is more straighttorward and lacks a personal touch. does ask for inpul meals activities, which is good point, but could have been more engaging. Based on these factors, I conclude that Assistant A's response is better, and the final verdict is "[[A]]".
helpful, relevant, accurate, and creative. The response
Judgment by WizarL M-3B-LM113B-v1.2000 (X)
13B-v1.2
After evaluating responses, would rate Assistant A better one. Here's why:
Assistant A's response more concise and directly answers the user's question. It provides all the necessary information, including the dates
arrival and departure, and brief message expressing excitement to spead Thanksgiving with the family. The email is well-structured and professional with clear subject line and Assistant B's response, while polite and friendly. is less focused on the main request. It asks for inpul on meals and activities, which is not directly reluted to the user's question. The response also the email before sending it. Therefore, based the factors helpfalness, relevance, accumcy, depth, creativity, and level detail, would rate Assistant as the better response.
personalized closing.
he more of draft rather than final email, it invites the madify
Judgment by GPT- (/)
Both assistanis provided good responses, drufting an email that communicates the user's travel plans for Thunksgiving. However, Assistant I'ss response is slightly more interactive and considerate, asking if there's 's anything they should bring or if there are any plans already in the works for meals or activities. This shows willingness to contribute to the family gathering, Assistant also acknowledges that other family members might be traveling. wishing them safe travels. Therefore, Assistant B's response is more comprehensive and thoughtful. Final Verdict: [[B]]
Table 26: The complete judgments given by different models on a pairwise response comparison example. We mark if each judgment agrees with human preference as well, where / stands for agreement and X stands for disagreement. (Part 2)
26

---

Published as a conference paper at ICLR 2024
Query
What's the hest way to cook salmon on the grill.
Response
The best way to cook salmos the grill prebear the grill to medium-high beat. brush the salmon with olive oil, scason with salt and pepper, and place skin-side down on the grill. Allow the salmen to cook for 5-7 minutes, until the skin is crispy and the fish llakes easily with fork. Flip the salmon, and cook an additional 3-4 minutes. Serve with your favorite sides and enjoy.
Critique from AUTO-J
The response provided is generally informative and offers husic method for grilling salmon. Uowever, it lacks depih and detail in several key areas. A main shortcoming the absence of specific temperatures or cooking times, which are cracial achieving the perfect grilled salmon. The response vaguely states 'mediuu-hhigh heat" and '5-7 minutes" for the first side of cooking. and "3-4 minutes" for the sccond side. More precise guidence would be beneticial for users who may not be experienced grill masters.
The response also fails to address the quality of the salmon that is required for the recipe. It exentiaa to mention the type of salmon that would be hest suited for this method of conking. Furthernore, the response does not provide detailed information on how to determine when the salmon cooked. This could potentially lead overcooked undercooked salmon, which common problem for novice cooks. The response could have been more interactive and personalized by asking the user about their specific grill or their experience with cooking salmon. This would have made response more engaging and helpful. Additionally, the response could have been more professional by citing credible sources or providing referenees to back the cooking instructions. This would have added credibility to the response.
Lastly the response missed an oppectunity provide comprehensive answer by not addressiig patential questions that the might have, such as what to do if the salmon sticks to the grill or how to ensure the salmon is cooked evenly.
In conclusion, while the response a good starting point, lacks nocessary detail and personal touch that would make a truly
excellent response.
Critique from GPT-4 (AUTO-J wins)
The response quite comprehensive and provides a clear, stop-by-step guide bow grill salmon. includes details such as level of the grill, how to prepure the salmon, how long to cook il on each side, and how to determine when it's done. However, could be improved hy including some tips on choosing the best piece salmon grilling, or suggesting some specific side dishes that pair well with grilled salmon.
Critique from ChatGPT (AUTO-J wins)
Lack of details: response could benefit providing more specific instructions. For example, does not mention the thickness of the salmon fillet or whether the grill should he covered or uncovered during conking.
Absenes alternative methods: While the provided method is good starting point, would be helpful to mention other techniques such as using codar plank sluminum foil to cook the salmon.
3. Limited seasoning options: While salt and pepper are mentioned as seasonings, would be nice to suggest additional herbs or spices that could cahance the flavor of the salmon.
Critique from SelFee (AUTO-J wins)
The answer is clear and concise, providing step by step instructions on how to cook salmon on the grill, However, it would be helpful to mention the ideal grill temperature and suggest using a grill brush to clean the grates before cooking, Additionally, would he useful suggost using a thermometer to chock the internal temperature of the salmon to ensure it is cooked to safe temperature. Overall, I would give answer a score Revision needed.
Critique from LLaMA- 2- Chat--hh1 (AUTO-J wins)
1. Lack of detail: The response could henefit from more detail on the cooking process. For example, the user might want to know the exect temperature of the grill, how tell when the salmon is done.
2. I Limited sensoning options: The response only mentions salt and pepper sensoning options, but there are many other delicions scasonings that could be used cahance the flavor of the salmon.
No visuals: The response does not include any visuals or images help the user understand the cooking process the final producct.
Critique from Vicuma-138-v1.55
138-v1.5 (AUTO-J wins)
Critique I: response provides clear concise method for cooking salmon the grill. It includes important details such preheating the grill and brushing the salmon with olive oil. However, would be helpful to include more information about the cooking time and temperature, as well as tips for checking the doneness of the salmon.
Critique 2: The response is well- and easy follow, with clear instructions for seasoning and cocking the salmun. also suggests serving the salmon with favorite sides, which adds a personal touch. However, the response could henefit from more variety in the language and more details about cooking process, such importance of llipping salmon and menitoring the grill temperature.
Critique from Vizardim-133 13B-v1.2 (Tie)
1. The response is concise and provides clear, step-by-step guide on how to cook salmon on the grill. It mentions the preheating of the grill, scasoning the salmon, and the cooking time, which all essential details.
2. The response could benefit from more detail on the type of olive oil to use and the seasoning options. For example, extra virgin olive a good choice for grilling. and other seasoning options could include lemon juice, garlic powder, or smoked paprika.
3. The response does not mention the importance of oiling the grates before cooking, which can prevent the salmon from sticking to the grill.
The respanse does provide any information how tell if the salmon conked properly. It would be helpful mention that the salmon should flake casily with a fork and have a slight pink color in the center. The response does not suggest any sides serve with the salmon, which could be helpful addition users who are unsure what to serve with grilled salmon.
Table 27: Critiques given by different models for response. We remove the "rating" part in each critique, and mark the comparison results between AUTO-J and baselines judged by human.
27

---

Published as a conference paper at ICLR 2024
Query
My Ex, to whom never married, just served me with divorce papers TLDRR I'mm in the process of evicting my ex from my house Kansas, He just served me with divoree papers, though we were never married. Hc's claiming was "common law" and I think he's trying to get my house. This is just so strange that I feel like I'm living in another universe. I hope can get a little advice. This is all happening in Kansas, Anyway, my ex and I were logether for about 10 years. Almost 8 years ago bought house. I bought the house. Me. Alone. My down paymeut, my credit, my mortgage. He didn't I pay anything. His name was never on anything. Not the title not the insurance, not the mortgage, not even the ulilities. We did have unwritten agreement that 'd pay
the morigage, utilities, and other shared expenses like groceries. And for about year, mayhe, he did pay me varying amounts of money. It was never half, and it caused a lot of arguments. Whatever paid me, paid cash. Since summer 201 I have paid all of it. He pays for whatever food be cats outside of groceries and covers whatever expenses be has don't know anything about that. He runs "busincsses" that include a youtube channel, Twitch channel. and 's a band. don't I have any idea how much money he made at them, or il be made any al all. For a number of reasons, including this ridiculous financial situation, I ended our relationship in mid March, I told him I planning to move out of state without him and that be needed to leave the house. I told him to be out by mid July. He totally ignored me and did nothing. July came and went. moved out of state. Ile stayed my house. hegan communicating with him only by email told Octoher, that he had to he out etc. He continued to ignore me, and I had to get lawyer who served him with notice of eviction at the beginning of October. communicate. I never had any idea what he thinking. Then yesterday I was served with divorce papers. What have has no details. but a friend of ours lold me that the es is saying he's 's eniiledd to the house (or some pari of) because it is community property from our commmon law marriage. Is this insune? Recouse feels insane. I know have to deal with it, and I will, hut what the heck? We never discussed being married, I never told anyone we were married, and I never heard of him doing so. We never used cach other's last names. I have always filed my taxes as single. Does have any hope here at all? possible accidentally get married? Without consent and against my will? I was REALLY hoping to he done with this nonsense. Thanks to you all for whatever you can tell me.
wanted have the house on the market by
challenged the eviction and the court date is the first week of November Through all of this the ex refused
Response with the highest probability among 32 candidates (AUTO-J: 4.0, GPT-4: 4.0)
is not common law marriage, Kansas does not recognize common law marriage, Living logether for a certain period of does not automatically make a couple married, even if they helieve they are marridd. Common law marriage requires valid marriage ceremony and a mutual agreement to be married. It is important consult with an attorny to understand specific details of your simuttion and laws your state.
Best-of-8 and 16 response selected by AUTO-J (AUTO-J: 5.0, GPT-4: 6.0)
It sounds like you are going through a difficult and confusing situation. Based on the information you have provided, it seemss unlikely that your ex could claim common law marriage or any other kind of property rights in the house, as there no evidence that you two were ever 26enllleeee.. lowever, it is possible that your ex could make a claim for pruperty or ussets that you not married, Kansas is community property state, which means that any property
married or that you intended to be murried 26eml
aequired during your relationship. even if you assets acquired during a marriage are considered community property and are owned equally by both spouses. If your ex can prove that you acquired the house during your selationship. they may be able make a claim It is important to consult with qualified attorny who can advise you on your specific legal sitution and help you navigate the eviction and papers you have received and help you understand your rights and options. may also
pertion the property.
divorce process. They can review the helpful to keep detailed records of all financial transactions and agrcements between you and your to support your case in court.
Best-of-32 response selected by AUTO-J (AUTO-J: 6.0. GPT-4: 9.0)
It seems like you're 're experiencing a complex bew ildering situation. Based information you' 've provided, here's a summury of your situation and potcntial options:
1. You never married your ex partnere but be has served you with divorce papers.
2. He's claiming the house you purchased logether community property from commmon law marriage, though you never discussed
or confirmed such a marriage.
You're
process of evicting him from your house, and the court date for the eviction the first week of November.
4. He's challenging the eviction and claiming ownership of the house, based on his assertion that you two were in common law marriage.
It highly unusual for an unmarried couple to be considered married under common law, especially if they did have a ceremony rogister their relationship with any government agency. In most states, common law marriage requires mutual consent to be considered murried and evidence of contimmous cohabitation for certain period of time (usuully a year or more).
While it's possible to accidentally get married without consent, it would require extreme circumstances such judge's mistake clerical error. Typically, a couple must go through a formal ceremony, obtain a marriage license, and fulfill the requirements and paperwork
for marriage.
Given that you never murried your partner, it is highly unlikely that the two of you were in common law marriagu, even be claims important to consult with a local attomey who familiar with Kansas law to ensure that you understand your options and represent you court. If the court rules in his favor, you may be able to appeal the decision, but it would be advisable to consult with an attorne to explore your options and potential appeal process. In the meantime, continuc with the eviction process, as you have already initiated it, and ensure that you follow all logal requirements and procedures.
otherwise. However,
Table 28: Best-of-N selection example to show the usefulness of AUTO-J's overall rating on single-response evaluation. The base LLM is Vicuna-7B-vl.5.
28


Reviews:
[{"review_id": "kYo4Zxq6oJ", "reviewer": "Reviewer_jzn2", "summary": "This paper proposes a generative judge with 13 billion parameters to evaluate the generations of large language models from real-world scenarios. Specifically, the authors created a large collection of data among 58 different scenarios and guided GPT4 to collect evaluation judgements as supervised training signals. Extensive evaluations demonstrate Auto-J outperforms many strong baselines and more analysis shows advantages of the proposed method, like reducing positional bias and generating more specific critiques.", "questions": "1). Is it necessary to have a 13b parameter model to train the scenario classifier? Have you tried other simple BERT-like models? \n\n2).Is there any bad case that Auto-J fails or Auto-J generates wrong critiques?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1). The design of scenario-specific criteria is strongly motivated, which will enable LLM-based judges to produce high-quality evaluations and critiques. Curated criterias can be model-agnostic and adopted to multiple models. \n\n2). Comprehensive evaluation and analysis of Auto-J demonstrate that its evaluations are consistent and can align well with human judgements.", "weaknesses": "The technical contribution is a bit limited as it is still within the scope of training one more LLM as judges to evaluate other LLMs’ generation. Since the training data is obtained from GPT4’s output, it is unsure whether it can replace GPT4 as judges or has strong generalizations as GPT4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Ici66sfg0", "reviewer": "Reviewer_WSEj", "summary": "This paper aims to present Auto-J, an LLM that may evaluate LLMs. This is in the line of LMs that are used to evaluate other LMs.", "questions": "It is clear that this paper is in a long line of other approaches. Yet, it is not clear why evaluating LLMs with an LLM is principled. Cna you comment on this?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Auto-J proposes a way to produce evaluation methods for LLMs", "weaknesses": "- It is strange that larger models are used to evaluate other models\n- LLMs should somehow emulate human capabilities and not other LLMs' capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5yEYAAadyY", "reviewer": "Reviewer_PG92", "summary": "The paper fine-tunes a language model to automatically judge the output of another language model, either evaluating a single generation or a pair of generations. \n\n(Sorry that I do not think I understand the core training and evaluation setup of the paper; either it is my reading comprehension's problem or there might be issues with the presentation.)", "questions": "- If I understand correctly, did you define the scenarios and categories both for fine-tune the model to perform evaluation AND test the model to perform evaluation? In that case, if we want to evaluate an LM that performs a new task that is not in the 58 categories you have defined, would Auto-J generalize to categories unseen during training?\n- Did you use GPT-4 as the label generator? If so, if one had enough OpenAI credit, the optimal strategy of evaluation according to this paper would still be using GPT-4 or not? (both yes/no are okay answers, but I think it'd be useful to clarify)", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper provides an open-sourced model that can automatically judge a models' generated output; this could potentially enable more researchers to run automatic evaluation at a lower cost with higher reliability.", "weaknesses": "- The papers is a large engineering effort (e.g., distilling GPT-4 for the task of evaluation) without much novel ideas (I do not think that a paper needs to be novel to be accepted, but this paper does score low in terms of novelty)\n- The presentation of the method and contribution feels very confusing to me (maybe it's just my fault). See questions below. I do not know whether other reviewers would have similar concerns though.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)