Learning Preference Model for LLMs via Automatic Preference Data Generation
Authors: Shijia Huang
Jianqiao Zhao
Yanyang Li
Liwei Wang
Venue: EMNLP 2023 Main (2023)
Keywords: Large Language Model
Reward Model
Preference Model

Abstract: Despite the advanced capacities of the state-of-the-art large language models (LLMs), they suffer from issues of hallucination, stereotype, etc. Preference models play an important role in LLM alignment, yet training preference models predominantly rely on human-annotated data. This reliance limits their versatility and scalability. In this paper, we propose learning the preference model for LLMs via automatic preference data generation (AutoPM). Our approach involves both In-Breadth Data Generation, which elicits pairwise preference data from LLMs following the helpful-honest-harmless (HHH) criteria, and In-Depth Data Generation, which enriches the dataset with responses spanning a wide quality range. With HHH-guided preference data, our approach simultaneously enables the LLMs to learn human preferences and align with human values. Quantitative assessments on five benchmark datasets demonstrate the reliability and potential of AutoPM, pointing out a more general and scalable way to improve LLM performance.

Learning Preference Model for LLMs via Automatic Preference Data Generation
Shijia Huang, Jianqiao Zhao, Yanyang Li, Liwei Wang"
Department of Computer Science and Engineering, The Chinese University of Hong Kong (sjhuang, jqzhao, yyli21, 1wwanww cww.ee.eeeccc......e edu.
Abstract
Governing model behavior from implicit sig- nals underlying human feedback has long been a principled methodology for LLM alignment. Ex- emplary work employs preference models to em- bed those human feedback signals (Stiennon et al.. 2020; Nakano et al., 2021) and recent publications like InstructGPT (Ouyang et al., 2022) have made significant progress along this direction. These preference models also constitute a cheap yet reli- able model-based evaluation metric (Wang et al., 2023) and have thus been widely adopted. Nonethe- less, the acquisition of the preference model relies heavily on human-annotated preference data. A standard data collection procedure involves obtain- ing multiple machine machine-generater responses for each prompt, followed by engaging annotators to rank these responses. To guarantee alignment with hu- man values, recent work (Bai et al., 2022a) fol- lows HHH criteria (Askell et al., 2021) in their annotation process. They encourage crowdworkers to intentionally goad virtual assistants to produce helpful or harmful responses during the interaction, which further intensifies the reliance manual an- notation and impedes the versatility and scalability of preference models.
Despite the advanced capacities of the state- of-the-art large language models (LLMs), they suffer from issues of hallucination, stereotype, etc. Preference models play an important role in LLM alignment, yet training preference mod- els predominantly rely on human- annotated data. This reliance limits their versatility and scalability. In this paper, ing the preference model for LLMs via auto- matic preference data generation (AutoPM). Our approach involves both In-Breadth Data Generation, which elicits pairwise preference data from LLMs following the helpful-honest-- harmless (HHH) criteria, and In-Depth Data Generation, which enriches the dataset with re- sponses spanning wide quality range. With HHH-guided preference data, our approach si- multaneously enables the LLMs to learn hu- man preferences and align with human values. Quantitative assessments on five benchmark datasets demonstrate the reliability and poten- tial of AutoPM, pointing out more general and scalable way to improve LLM performance.
propose learn-
1 Introduction
With the rapid development of the Large Language Models (LLMs), the state-of-the-art LLMs (e.g.. GPT series (Brown et al., 2020a), OPT (Zhang et al., 2022), BLOOM (Scao et al., 2022), and LLaMA (Touvron al., 2023)) exhibit remark- able emergent capabilities (Wei al., 2022b) and can adeptly generate coherent responses in accor- dance with user instructions after supervised fine- tuning (SFT) (Ouyang et al., 2022: Wang et al., 2022; Taori et al., 2023; Chiang et al., 2023). De- spite their advanced generation capacities, LLMs may produce inaccurate and potentially detrimen- tal content that is not readily discernible by human users. Aligning LLMs with human values is thus of central consideration before deploying LLMs to society.
A line of work starts explore the possibility of leveraging LLMs themselves for gathering prefer- ence data (Wang et al., 2023; Peng et al., 2023). However, these methods still adhere to conven- tional preference data collection pipelines, with the exception of replacing human annotators with GPT-3.5 (OpenAI, 2023b) and GPT-4 (OpenAI, 2023a). Such naive data collection strategies lack a comprehensive guideline and therefore fail to align the preference model well with human values (See Section 5). To address these shortcomings, propose Au- toPM, an approach for learning the preference model via automatic preference data generation from LLMs. Our guideline-driven data generation method systematically directs the preference data
Corresponding author.

---

synthesis from performant LLMs, achieving broad coverage and diversity. Moreover, as our data syn- thesis guideline closely adheres to the HHH crite- ria (Askell et al., 2021), our approach enables the LLMs to learn human preferences and simultane- ously align themselves with human values.
they can directly synthesize annotated data for train- ing lightweight yet strong downstream task mod- els. For instance, ZeroGen (Ye et al., 2022) shows how to appropriately design prompts and sampling procedures to obtain synthetic data from LLMs to train a small LSTM. SunGen (Gao et al., 2023) im- proves over ZeroGen by filtering low-quality data from the generated corpus. These data generation techniques have also generalized ment, where a line of work focuses on producing instructions from LLM align L.LM itself with human expectations (Wei et al.. 2022a; Ouyang et al., 2022). One example is Self-Instruct (Wang et al., 2022). which starts from a seed set of human- written instructions and guides LLM to brainstorm board set of instructions for fine-tuning itself. More recent work (Peng et al., 2023) made use of GPT-4 (OpenAI, 2023a) to generate instruction- following data.
Specifically, AutoPM consists of In-Breadth Data Generation and In-Depth Data Generation. The in-breadth data generation prompts pairwise preference data from LLMs that either match or violate human standards from various aspects in ac- cordance with the HHH criteria as the guidance of human preferences. The in-depth data generation, on the other hand, employs a novel HHHH-gudded sequential post-editing process to construct prefer- ence data with broad coverage of quality.
LLM align-
Through this novel paradigm of preference data collection, AutoPM can effectively generate di- verse preference data that adheres closely to hu- man preference guidelines without human annota- tion. With GPT-3.5 the main source of synthetic preference data, amass 42K preference data and develop AutoPM based on LLaMA (Touvron et al., 2023). Extensive evaluation on benchmark datasets attests to the superiority of AutoPM. Ex- perimental results show that AutoPM's predictions are highly consistent with humans and GPT-4. Au- toPM can even elevate the performance of strong LLMs. Besides, we have two intriguing findings:
Preference Model for LLMs. A well-learned preference model can significantly contribute evaluating and aligning L.MMs (Wang et al., 2023; Ouyang et al., 2022: Bai et al., 2022b). Early work (Stiennon et al.. 2020; Nakano et al., 2021) begins to use preference models as human pref- erence feedback in specific NLP tasks. Instruct- GPT (Ouyang al., 2022) leads the way learn- ing the preference model for improving general AT assistants. The diversity of the prompts-responses set a key factor for training preference models. Anthropic (Bai et al., 2022a) asks crowdworkers to have open-ended conversations with AI assistance to cover more helpful and harmless cases. Work like SHP (Ethayarajh et al., 2022) collects pref- erence data from real-world websites like Reddit. Recent work starts leveraging strong LLMs to col- lect preference data (Wang al., 2023; Peng al., 2023), but still follow the conventional preference data collection pipeline except for ranking the re- sponses using more advanced GPT-3.5 and GPT-4. In contrast, the proposed AutoPM directly gener- ates preference data from LLMS governed by HHH criteria, ensuring not only high level of accuracy but also increased diversity.
Though the training data quality of AutoPM is upper-bounded by GPT-3.5, AutoPM is still capable of discriminating responses from stronger models like GPT-4, going beyond the limitations of training data.
By automatically learning the preference model, AutoPM can improve the response quality of not only Alpaca (Taori et al., 2023) but also stronger models like Vicuna (Chiang et al., 2023). This improvement achieved even when using the same prompt dataset and GPT-3.5 for data generation, as originally used by Alpaca.
2 Related Work
Data Generation from LLMs. Scaling transformer-basee language models has given rise a substantial paradigm shift in Natural Language Processing (Brown et al., 2020b; Chowdhery et al., 2022). These powerful LLMS can generate fluent and informative text with high coherency, which re- shapes the conventional machine learning pipeline:
Alignment of L.Ms. Alignment (Leike et al., 2018; Glaese et al., 2022) of LLMs aims to build agents that are better following user inten- tions. InstructGPT (Ouyang et al., 2022) pro- pose to improve LLM alignment in two ways: supervised fine-tuning (SFT) with demonstration data and reinforcement learning from human feed-

---

In-Breadth Data Generation
Helpful Honest
Instantiate
Helpful: (useful, useless),
IIЛ Criteria
Honest: (correcs, contains wrong information),
Harmless
Harmless: (imgartial, biased discrimntttt
Random Sampling
Given the question, generate useful answer A, and a useless answer B.
My grandmother baked the cake,
Instruction:
Rewrite this sentence to active voice: "The cake baked by grandmother."
Answer B: The cake was baked by my grandmother.
In-Depth
Data Generation
Deletion
Random Sampling
Substitution
Insertion
Normaa Answer
Delete useful content
Insert irrelevant content
Instruction: Introduce the first satollite launched space.
first satellite inso space was Sputnis which was luuched by the Soviet Union 1957.
Sputnik
Moon is satellite of the earth.
Figure I: The framework of the proposed AutoPM. Each blue represents preference data sample, from positive response to negative one.
back (RLHF) against the preference model. With the development of powerful GPT-3.5 (OpenAI, 2023b) and data generation techniques like Self- Instruction (Wang et al., 2022), Alpaca (Taori al., 2023) and Vicuna (Chiang al., 2023) align LLaMA by SET with generated data or real conver- sation data from GPT-3.5. WizardLM (Xu et al., 2023) and Dromedary (Sun et al., 2023) further im- prove SFT by generating complicated instructions and better principle-driven responses. Other than using reinforcement learning via proximal policy optimization (PPO) (Schulman et al., 2017) pref- erence models, RRHF (Yuan al., 2023) proposes to align human preferences through ranking loss to get rid of laborious hyper-parameter tuning, while RAFT (Dong et al., 2023) directly fine-tunes with the best response from multiple responses ranked by the preference model.
data from LLM by crafting appropriate prompts. A pair-wise setup is adopted: We collect set of phrase pairs. Each pair has opposite cases under one HHH principle in Askell et al. (2021), i.e., Helpful, Harmless, and Honest. These phrase pairs will then be injected into our manually writ- ten templates and passed into LLM to guide the data generation. LLM will generate two responses: one that matches the HHH criteria is marked as the positive case for training the preference model, and the other that violates the HHH criteria will be served the negative sample. This way makes prompt engineering casier obtain a diverse set of replies w.r.. a given LLM query, e.g., antonym pairs will be a natural choice to inform LLM to produce text from two distinct viewpoints.
Concretely, first instantiate the HHH crite- ria into pairwise phrase descriptions. We collect a wide variety phrase pairs and perform filtering and merging. The resulting phrase pairs are shown in Table I, each representing a unique property. For example, "specific" and "too-general" encour- ages LLM to generate data of various granularity, while "not relevant" and "highly relevant" will in- struct LLM whether to include redundant content Based phrase pairs, In-Breadth Data Generation is as follows: We first randomly sam- ple phrase pair (phrase,, phraso;) and natu- ral instruction (data sources are described in Sec- tion 4.1). A prompt template "Given the question,
3 Automatic Preference Data Generation
AutoPM comprises components: In-Breadth Data Generation and In-Depth Data Generation. In addition, we also use Paired Models Data further supplement the data. The detailed descriptions follow below.
3.1 In-Breadth Data Generation
A training sample of preference model consists a pair similar texts but different in qual- ity (Ouyang et al., 2022). We propose to elicit such

---

HHH Criteria
Phrase Pairs
Helpful
("useful" vs. "useless"), ("comprehensive-Hiinnii vs. "ringle-thinknng" ("highly relevand" vs.
"not relevant"), ("specific" vs. "too-general") ("correct" vs. "contains wrong information"), ("objective" "exaggerated"), ("honest" vs. "fabricated"), ("clear" "misleading") ("impartial" "biasedddisrrininnntorr"" ("legal" "contains illegal/sexual/hate content"), ("posirive" vs. "negative"), ("ethicall" vs. "unethicaf")
Honest
Harmless
Table I: Phrase pairs for in-breadth data generation.
generate phrase, answer A, and a phrasey an- swer B" is later employed to construct the model's input, with the natural instruction appended and (phrasea: phrasc,) substituted. The generated out- put from LLM will serve as the fine-tuning data of our preference model. A demonstration of this process is shown in the upper part of Figure 1.
making a worse answer", which instructs LLM to rewrite the previous step output in an expected way, Multiple pairs of preference data could be assem- bled from one edit actions sequence by composing LLM outputs in two random intermediate steps and annotating the one from the early step as positive. By sampling sequences of various lengths (1 ~ 3) and actions, the synthetic dataset could cover more possibilities and be non-trivial for optimization.
3.2 In-Depth Data Generation
Although altering the phrase pairs enables us to har- vest samples that possess various properties at scale, a crucial issue of these synthesized data is that the preference model might merely simplify this task into a binary classification problem and collapse to trivial solution without predicting reliable preference score. We thus propose In- In-Depth Data Generation to enrich the dataset with responses spanning a wide quality range, which resembles the post-editing process by executing a series of actions that edit the generated text outward the HHH criteria. Inspired by Levenshtein distance, we define the following edit actions:
3.3 Paired Models Data Generation
To improve the coverage of our synthetic prefer- ence data, we treat LLMs from different institutions as different sources for data collection. Gathering data from various LLMs also empowers us to con- trol the response quality, where the model perfor- mance is a reasonable quality indicator. We send the same natural instruction to a large LLM and relatively smaller one to obtain their responses. This pair of responses, as well as the original in- struction, forms training instance of our prefer- ence model. In our experiments, responses from GPT-3.5 (text-davinei-0033 (OpenAI, 2023b) will be treated as positive while those from OPT-IML- 1.3B (Iyer et al., 2022) will be negative.
Deletion removes content in the LLM re- sponse that useful for replying to the given query.
Substitution edits content in the LLM re- sponse such that it will be inappropriate for the given query.
Learning Preference Model
4.1 Preference Data Collection
Insertion adds new content irrelevant to the given query into the LLM response.
We adopt the auto auto-generated Alpaca dataset (Taori et al., 2023) as the source of instructions for preference data generation, which contains 52K fnslloction-follliiing data generated by the Self- Instruct technique (Wang et al., 2022). To cover scenarios where users make malicious requests, we additionally include 1K Red Team Prompts (Gan- guli et al., 2022). All prompts are randomly di- vided into three cqual-sized splits to synthesize preference data via the proposed three data gener- ation methods respectively. We choose GPT-3.5 (text-davinci-003) for in in-breadth and in-depth data generation and finally obtain 42K preference data. The data samples are provided in Appendix A. Following Wang et al. (2022), we manually re-
We operationalize these edit actions with pow- erful LLM, as these actions are challenging to im- plement with simple algorithms. With these edit actions, the propose in-depth data generation is il- lustrated in the lower part of Figure 1: Starting with a well-generated response for a given instruction, we sample sequence of edit actions to modify the raw response. LLM executes these actions se- quentially, where at each step LLM deteriorates the output from the previous step according to the current action. To execute a certain edit action in step i, its natural language description act; (See Appendix A.2) will be put into the prompt template "Given the question and answer pair, act, thus

---

Quality Review Question
Yes %
scaling. We train AutoPM with Zero Redundancy Optimizer (ZeRO) (Rajbhandari et al., 2020) stage on 8 NVIDIA A A100-SXM4-80GB GPUs, and adopt FP16 and gradient checkpointing (Sohoni et al., 2019) to reduce GPU memory footprint. All parameters are updated by the Adam opti- mizer (Kingma and Ba. 2014) for one epoch with learning rate of 5e-6 and cosine learning rate scheduler (Loshchilov and Hutter, 2016).
In- In-Breadthh Data Generation
chosen rejected
83%
chosen rejected
16%
chosen rejected
1%
In- Di--hephh Data Generution
chosen > rejected
79%
chosen & rejected
19%
chosen < rejected
2%
Paired Models Data Generation
chosen > rejected
94%
chosen s2 rejected
6%
Experiments
chosen < rejected
0%
Table 2: Synthetic preference data quality review result.
5.1 Baseline Systems
view the data quality by randomly sampling 200 generated examples from the splits of in-breadth and in-depth data generation. An expert annotator is asked to label whether the chosen response better (denoted as >), comparable (denoted as Rz), or worse (denoted as <)) than the rejected response. Results in Table 2 show that most generated data are valid, while only small part of chosen re- sponses is comparable to the rejected ones, mainly because: a) GPT-3.5 itself can not properly respond that instruction; b) Some instructions are suit- able to generate specific rejected responses, e.g., a fabricated and exaggerated response is probably not a bad choice for a -oorinii instruction.
We quantitatively evaluate AutoPM on 5 bench- mark datasets and evaluate AutoPM's reliability and potential to improve LLM. We compare Au- toPM with multiple advanced LLMs (Wang and Komatsuzaki, 2021; Brown et al., 2020a; OpenAI, 2023b; Touvron et al., 2023; OpenAI, 2023a). as well as two open-source preference models: Anthropic-PM. Bai et al. (2022a) aims to train a helpful and harmless assistant via reinforcement learning with the preference model. They let crowd- workers to ask for help or to elicit harmful re- sponses from the AI assistant, and compare mul- tiple Al responses. They collect 161K preference data and release this preference dataset but not the model. We reproduce Anthropic- on LLaMA-7B using their released dataset.
based
4.2 Training Strategy
We follow the training method of Instruct- GPT (Ouyang et al., 2022) and Anthropic-PM (Bai et al., 2022a) to learn the score-based preference model. Given the instruction and the response V. scalar value r(x, y) is predicted as the prefer- ence score. Specifically, we first concatenate the tokenized instruction æ with lg tokens and response y with ly tokens and feed them into the pretrained language model to get the last layer hidden state E IR ,yy+( where C' the hidden size. Then a linear regression head is applied on F to predict the score for each token. We take the average of the last by scores as the output scalar value r(x, v). The training objective is given as:
LM PandaLM (Wang et al., 2023) is devel- oped to distinguish the superior model from several LLMs in the instruction tuning optimization stage. It gathers responses from LLMs with similar scales (ranging from 6.7B~7B) and ranks them by GPT- 3.5, obtaining 300K preference samples.
5.2 Datasets
HHH-Eval. HHH-Eval (Askell et al., 2021) from BIG-bench (bench authors, 2023) systematically evaluates LLM responses according to HHH eri- teria. It contains about 200 samples and 50 for each of the following labels, including Helpful, Honest, Harmless, and Other. Each sample mim- ics a single-turn conversation between the user and a language model assistant, including a user request and a pair of chosen-rejected replies.
((0) - log (a (ro uo) ro (x, )))
(1)
where Be is the preferred response out of the paired responses Ye and yr w.r.t. the instruction 0 is the model parameters and a is the Sigmoid function. Implementation details. We build AutoPM on top of 7B parameter variant of LLaMA (Tou- vron et al., 2023), and we also provide a 30B pa- rameter variant to demonstrate the performance
TruthfulQA. TruthfulQA (Lin et al., 2021) is QA dataset targeted misconception and false belief. For instance, a question TruthfulQA is "What happens to you you eat watermelon seeds?". A common falschood that appears in the machine- generated answer is "You grow watermelons in your

---

Instruct. Source
Preference Source
HHH--Evll
Model
#Param.
Score
Overall
Harmless
Helpful
Honest
Other
Anthropic-IM GPT-3.5
52B
X
0.77
0.87
0.95
0.85
0.80
0.91
PandaLM Anthropic-PM Anthropic-PM
Alpaca
GPT3.5 Rank
7B
x
0.60
0.57
0.64
0.56
0.63
52B
0,86
Human
Human Rank
7B
0.81
0.88
0.71
0.75
0.93
AutoPM (ours)
7B
0.83
0.83
0.73
0.89
0.89
Alpaca
GPT3.5 Gen
AutoPM (ours)
30B
0.85
0.78
0.89
0.93
Table 3: Performance comparison on HHH-Eval benchmark. denotes our reproduction with the LLaMA-7B backhone using Anthropic's official dataset. Score denotes that the model outputs scalar preference for each response instead of comparing the two responses directly.
Model
#Param.
TruthfulQA-MCI
preference models first predict a preference score for each reply independently and select the one with the highest score. As shown in Table 3, our AutoPM-7B achieves an overall accuracy of 0.83, which is very close to Anthropic-PM-52D and GPT- 3.5. This is a strong result especially given that AutoPM is small and in harder setup where it can- not observe both replies for comparison the same time. With larger 30B backbone, the performance on Helpful and Other categories can be further improved. PandaLM performs poorly on HHH- Eval, probably because their training data are ordi- nary LLM responses and do not cover many harm- ful or dishonest cases. We also note that our repro- duced Anthropic-PM-77B is slightly worse than the original Anthropic-PM, as it has a smaller model size and experiences the extra preference model pretraining (PMP) on external data.
Authropic-LM
52B
0.32
Anthropic-PMi
7B
0.37
GPT-J
6B
0.20
GPT-3
175B
0.21
GPT-3.5
0,47
GPT-4
0.59
AutoPM (Ours)
0.56
AutoPM (Ours)
30B
0.59
Table 4: Performance comparison on TruthfulQA-MCI.
stomach". This dataset helps evaluate the hon- esty of LLM. TruthfulQA contains 817 questions and has two settings: a) Generation asks the model to generate the answer given only the question; b) MCI additionally provides 4-5 answer choices with only one true statement for each question.
Bot Adversarial Dialogues (BAD). BAD (Xu et al., 2020) a human-bot conversation dataset, where the users are instructed adversarially goad the chatbot to reply unsafe content and tag the chat- bot response as Not Harmful or Harmful. We adopt BAD to further assess the harmlessness of preference models.
Further Assessment on Honesty. We addition- ally evaluate AutoPM on the TruthfulQA MC1 task to validate the capability of finding out honest sponses. The inference of preference models is the same as in HHH-Eval. As shown in Table 4, AutoPM-7B and AutoPM-30B achieve an accuracy of 0.56 and 0.59, which outperform Anthropic's s model and GPT-3.5 and are comparable to GPT-4, This phenomenon implics that AutoPM can effec- tively recognize disinformation in Al replies and is promising for suppressing hallucinations.
Self-instruct User-oriented Benchmark. Self- instruct User-oriented benchmark (Wang et al., 2022) is an instruction-follovinn dataset that tests LLMs' ability to perform wide spectrum of tasks. It contains 252 expert-written instructions. Vicuna Benchmark. Vicuna Benchmark (Chi- ang et al., 2023) is a recently proposed dataset that evaluates whether LLM could generate human- preferred replies. It devises 80 questions from vari- ous categories and is evaluated by GPT-4.
Further Assessment on Harmlessness. To fur- ther assess AutoPM's ability to identify harmful responses, we test AutoPM on the BAD dataset. We follow Bai et al. (2022a) compute the prefer- ence score distribution over the chatbot responses (considering only the first chatbot utterance per BAD conversation). Figure illustrates the nor- malized preference scores distribution predicted by AutoPM-30B, where the harmful responses pos-
5.3 Benchmark Evaluation
We first conduct experiments on BIG-bench HHH- Eval. To evaluate preference models on HHH-Eval,

---

0.12
V-A
Human
GPT4
AutoPM-7B
Authropic-78'
31.0
0.81
GPT-4
0.87
0.83
2.00
G-D
Human
GPT-4
AutoPM-7B
Anthropic-7B'
3.60
Human
NA
0.92
0.79
0.66
GPT-4
0.92
N/A
0.76
0.63
Table 5: Evaluation consistency with Human/GPTT-4, V-A denotes the comparison between Vieuna- 13B and Alpaca-7B, while G-D denotes GPT-4 vs. GPT-3.5.
PM Snow
Figure 2: Normalized preference score distribution of AutoPM-30B on BAD.
them hard for humans to judge.
Vicura
Tie
Win
#GPTI Win
Dovincl
Table 5 shows the assessment consistency by calculating the proportion of agreement ob- served (Banerjee al., 1999). Since score-based preference models do not consider draws, we rule out all tie cases in the consistency computation. For Vicuna- 1aauma-13B vs. Alpaca-7B, AutoPM achieves a higher consistency of 0.86 with GPT-4 and 0.80 with humans compared to Anthropic-7BB For GPT- 4 vs. GPT-3.5, AutoPM also achieves consistency of 0.76 with GPT-4 and 0.79 with humans, even though our preference data are generated from the poorer GPT-3.5. These results show that AutoPM not only predict reliable preference scores that generalize well to unscen models like Vicuna- 13B but also able to discern responses from models like GPT-4 that are better than GPT-3.5.
Boll by GPT4
214
35
187
22
43
Homan
Eval by
169
29
154
91
AstePM
Dval by
220
32
185
62
Bralby Authoopic
212
166
81
(A) Vieuea-
GPT4 Test-daviaci003
Figure 3: Assessment results from various models and humans Self-instruct User-oriented benchmark.
sess significantly lower preference scores than non- harmful oncs. This distribution closely resembles the one predicted by Anthropic-PM PM-52B, suggest- ing that AutoPM has comparable performance on classifying malicious responses.
5.5 Improving LLMs with AutoPM
5.4 Reliability of AutoPM
We show the potential of AutoPM to improve LLMs in this section. Instead of conducting Rein- forcement learning (RL) following Ouyang et al. (2022) (in which the newly included fine-tuning dataset, the complicated and the stochasticity of the training process may hinder the accurate assessment of the preference model), we utilize Rejection Sampling (best-of- n) (Nakano et al., 2021; Askell et al., 2021) to improve existing LLMs at inference time. Setup. Given an instruction, the common decod- ing strategy of LLMs is to greedily generate the response with high confidence or randomly sample response to increasing diversity. our experiment, we let LLMS generate the greedy response and 9 sampled responses as the candidate responses set. Then we apply AutoPM to select the best one among these 10 responses and check whether Au- toPM can improve over the greedy result.
To characterize the reliability of AutoPM's predic- tions, we calculate the assessment consistency be- tween AutoPM and human/GPT-4 on Self-instruct User-oriented benchmark (Wang et al., 2022). We first choose model pair and let them re- ply to the same set of instructions, then request AutoPW/humanGGPT- to select the winner for each instruction. Two model pairs tested here: Vicuna- vs. Alpaca-7B, and GPT-4 vs. GPT- 3.5 (text-davinei-003. For human assessment, we ask three human annotators to label the preferred response (select 'A win', 'Tie', or 'B win') indepen- dently and use the majority vote as the final assess- ment decision. For GPT-4 assessment, we follow the system prompt provided by Chiang et al. (2023) to generate robust evaluation responses. Figure summarizes the overall assessment results. We find that there are more tie cases from the human evaluation. This is because the compared models are known to be strong in following human instruc- tions and delivering high-quality responses, making
setting,
AutoPM helps to generate truthful replies. We conduct experiments on the TruthfulQA Genera- tion task (Lin et al., 2021) and apply AutoPM to

---

Model
#Param.
Truthful
Iruthful'Inf
Ablation of Data Components
6B
0.22
0.19
Base
In-Breadth
In-Depth
HHH--Eval
TruthfulQA
GPT-3
175B
0.28
0.25
0.61
0.40
0.76
0.52
7B
0.33
0.29
0.79
0.53
13B
0.47
0,41
LLaMA
30B
0.52
0.48
0.83
0.56
65B
0.57
0.53
Ablation In-Breadth Data Generation
Alpucu- Greedy
7B
0.38
0.38
Helpful
Honest
Harmless
HHHH-Eval
TruthfulQA
Alpaca-Random
7B
0.42
0.42
0.74
0.49
Alpaca-AutoPM
7B
0.51
0.51
0.71
0.48
Vieuna- Vicuna-Greedy
13B
0.49
0.48
0.72
0.43
Vicuna-Random
13B
0.50
0.50
0.76
0.52
Vicuna-AutoPM
13B
0.63
0.63
Table 7: Ablation studies of data effectiveness.
Table 6: Results TruthfulQA Generation Task. We
report the fractions of truthful and answers, scored by fine-tuned GPT-3 (Lin al., 2021).
mark. In addition to LLaMA and Alpaca, also compare OPT-IML-1.3B (lyer et al., 2022) and BLOOMZ-7B1 (Muennighoff et al., 2022) from two different LLM families. The Lose statistics between AutoPM-selected and greedy re- sponses are visualized in Figure 4. The result in- dicates that AutoPM can identify better responses from multiple candidates. For OPT-IML-1.3B and BLOOMZ-7B1, we find there are 21 and 20 tied This is because these two models in- capable of generating diverse responses and thus their sampled responses are similar. For stronger models like Alpaca-7B and Vicuna-13B, there are slightly more cases where the greedy response wins, But the number of AutoPM-selected responses that are simultaneously preferred by GPT-4 is 2x more than the one of greedy responses.
AuloPM-Win
Tic
Greedy-Win Win
24
14
1,38
BLOOMZ
20
Alpaca 7B
Vicuna
13
Figure 4: Winning rate AutoPM-selectee responses greedy responses the Vieuna benchmark, judged by GPT-4.
two popular instruction-follovingg models, Alpaca- 7B (Taori et al., 2023) and Vicuna-13BB (Chiang et al., 2023). Table 6 shows the performance of dif- ferent LLMs. LLaMA-7B achieves value of 0.33 on the truthful metric and LLaMA-65B is 0.57. This observation is line with Wei al. (2022h), where LLMs are prone to output the true statement after parameter scaling. Although Alpaca-78 7B and Vicuna- 13B have strong instructionnfollooiing abili- ties, they are likely to produce hallucinated answers (0.38 and 0.19 on the truthful metric respectively) owing to their small sizes. Manually examining their sampled responses reveals that they are capa- ble of generating true statements, but sometimes false statements stand out because of greater prob- ability. By applying AutoPM to Alpaca-7B and Vicuna- 13B, both models achieve substantially bet- ter results of 0.51 and 0.63. Besides, we find that both Alpaca and Vicuna's responses are informa- tive, with the informative metric exceeding 0.99. AutoPM helps generate human-prett preferedef replies. We test whether AutoPM can help gen- erate numan--refente replies on Vicuna Bench-
5.6 Ablation Studies
We investigate the effectiveness of synthetic pref- erence data in Table In the data component ablation, we take paired model data as the base- line (denoted as Base), which performs poorly on HHH-Eval due to the low coverage of dishonest and harmful responses from OPT-IML-1.33 and GPT-3.5. In-breadth data generation contributes the most the performance gain of AutoPM, reach- ing an accuracy 0.76 on HHH-Eval and 0.52 on TruthfulQA MC1. Combining paired model data with in-breadth and in-depth data can further im- prove the performance and yield the final results of 0.83 and 0.56, implying the complementary nature of these three sources of generated data. Morcover, assess the contribution of each category of data from in-breadth data generation. We observe that removing any category can impair the performance on HHH-Eval, and the honest/harmless data con- tribute to TruthfulQA more.

---

6 Conclusion
References
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A general language assistant as a laboratory for align- ment. arXiv preprint arXiv: 1e..:222..000000 12.00811.
In this paper, we propose learning the preference model for LLMS via automatic data generation un- der the human preference guideline, named Au- toPM. The proposed AutoPM contains In- Be-Breadth Data Generation and In-Depth Data Generation, and Paired Models Data Generation. We conduct extensive quantitative assessments on 5 benchmark datasets, demonstrating the reliability of AutoPM and its potential for enhancing LLMs. We believe that AutoPM shed light on a more general and scal- able way for learning preference models.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. 2022a. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0588
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones. Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022b. Constitutional ai: Harmlessness from ai feedback. arXiv preprint Stfi:2212208007..
Limitations
In this work, we explore learning preference model for LLMs via automatic preference data generation. When reviewing the data quality, we find that most synthetic preference data valid, but small por- tion of the rejected response is comparable with or even better than the chosen response. Given that our approach has already shown its reliability and potential with this data noise exists, we will address this observation in the future.
Mousumi Banerjee, Michelle Capozzoli, Laura Mc- Sweeney, and Debajyoti Sinha. 1999. Beyond kappa: A review of interrater agreement measures. Cana- dian journal of statistics, 27(1):3-23.
BIG bench authors. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of lan- guage models. Transactions on Machine Learning Research.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, et al. 2020a. Language models are few-shot learners. Advances in neural information processing systems, 13:1
Ethics Statement
In this work, generate preference data from GPT-3.5, in which the rejected data may contain unhelpful, dishonest, and even harmful content. These rejected data are used and should be used as negative samples for preference model learning, therefore helping LLMs align with human prefer- ences and values. Wrongly using the rejected data, c.g., finetuming with these data, may cause LLMs to produce terrible responses. We conform to EMNLP Code of Ethics in every step of the work, and we advocate responsible data usage.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared Kaplan. Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Fric Sigler. Mateusz Litwin. Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020b. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Pro- cessing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual.
Aeknowledgements
This work was supported by National Key R&D Program of China (Project No. 20227D0161200 62000 2022ZD0161201). It was also partially funded by the Centre for Perceptual and Interactive Intelli- gence (CPII) Ltd under the Innovation and Tech- nology Commission (ITC)'s InnoHK. Liwei Wang is Principal Investigator of CPII under the In- noHK. This project was also partially supported by Research Grants Council (RGC) under grant Collaborative Research Fund (CRF) C4034-21G.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See http::/vviuua. Imsys. org (accessed 14 April 2023).
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi,

---

Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph. Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov. Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Hellstee Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- cling with pathways. CoRR, abs/2204.02311.
Jan Leike, David Krueger. Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. 2018. Scalable agent alignment via reward modeling: research direction. arXiv preprint 17X1::8811.07777
Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.0799
Ilya Loshchilov and Frank Hutter. 2016. Sgdr: Stochas- tic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983
Niklas Muennighoff, Thomas Wang. Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng -iin Yong. Hailey Schoelkopf, et al. 2022. Crosslingual generaliza- tion through multitask finetuning, arXiv preprint arXiv:221..0178:
Hanze Dong, Wei Xiong, Deepanshu Goyal. Rui Pan. Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06667
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang. Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders. et al. 2021. Webgpt: Browser-assisted question- answering with human feedback. arXiv preprint arXiv:2112.09332
Kawin Ethayarajh. Yejin Choi. and Swabha Swayamdipta. 2022. Understanding dataset difficulty with V-usable information. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 5988-6008. PMLR.
OpenAl. 2023a. abs/2303.08774.
GPT-4 technical report. CoRR,
OpenAL. 2023b. and
How do text-davinci- text-davinci- -03 differ? https://help.openai.com www/////aaaaccle//67999999999 (ttp:/aricces//77199999999 19w--oow-------- do-text-davvncii-002-omm -ad---ad---dd---e -a-i--di----iiiiiieeeeee
002
Deep Ganguli, Liane Lovit, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer. Kamal Ndousse, al. 2022. Red teaming language models to re- duce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.0788
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744,
Jiahui Gao, Renjie Pi, LIN Yong, Hang Xu, Jiacheng Yc, Zhiyong Wu, WEIZHONG ZHANG, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. 2023. Self- guided noise-free data generation for efficient zero- shot learning. In The Eleventh International Confer- ence on Learning Representations.
Baolin Peng, Chunyuan Li. Pengcheng He, Michel Gal- ley, and Jianfeng Gao. 2023. Instruction tuning with GPT-4, CoRR, abs/2304.03277.
Amelia Glaese Nat McAleese. Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. 2022. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209. 14375.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Com- puting, Networking, Storage and Analysis, pages 1- 16. IEEE
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang. Qing Liu, Punit Singh Koura, Xian Li, Brian O'Horo, Gabriel Pereyra, Jeff Wang, Christo- pher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, and Ves Stoyanov. 2022. OPT-IML: scaling language model instruction meta learning through the lens of generalization. CoRR, abs/2212.1 12017.
Teven Le Scao, Angela Fan, Christopher Akiki, El- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvo,, Mathhis Gallé, et al. 2022. Bloom: 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proxi- mal policy optimization algorithms. arXiv preprint 177.v:1707.06347,
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint urXiv:1412.6988

---

Nimit S Sohoni, Christopher R Aberger, Megan Leszczynski, Jian Zhang, and Christopher Ré. 2019. Low-memory neural network training: A technical report. arXiv preprint WrXiv:7944.0000
Jiang. 2023. WizardIm: Empowering large lan- guage models to follow complex instructions. arXiv preprint arXiv:2304. 12244.
Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Ja- son Weston, and Emily Dinan. 2020. Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learn- ing to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008- 3021.
Jiacheng Ye, Jiahui Gao. Qintong Li, Hang Xu, Jiangtao Feng. Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022. ZeroGen: Efficient zero-shot learning via dataset generation. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 11653-1 1669, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang. Zhenfang Chen, David Cox, Yiming Yang. and Chuang Gan. 2023. Principle-driven self- alignment of language models from scratch with minimal human supervision. arXiv preprint
arXiv2330.03304.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang. Songfang Huang, and Fei Huang. 2023. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunrri B Hashimoto. 2023. Stanford alpaca: An instruction-folloiing llama model.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Ilama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 13971.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, al. 2022, Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2055.0066.
A Appendix
Ben Wang and Aran Komatsuzaki. 2021. Gpt-j-6h: A 6 billion parameter autoregressive language model.
A.1 Data samples in AutoPM
In Tables 8, 9, and 10, we choose a few random samples generated by AutoPM for qualitative sessment. AutoPM generates some harmful content as negative samples, but we don't show that data here.
Yidong Wang, Zhuohao Yu, Zhengran Zeng. Linyi Yang, Cunxiang Wang. Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. 2023. Pandalm: An automatic evaluation benchmark for Ilm instruction tuning optimization. arXiv preprint 2306:2336000007.
A.2 Details of In-Depth Data Generation
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A Smith, Daniel Khashabi, and Han- naneh Hajishirzi. 2022. Self-instruct: Aligning lan- guage model with self generated instructions. arXiv preprint 22 10560.
The natural language descriptions of our defined edit actions are:
Deletion: "remove some answer content that is usefull to the question"
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoc V. Le. 2022a. Finetuned language models are zero-shot learners. In The Tenth International Conference Learning Representa- rions, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Substitution: "edit some answer content
make it inaccurate for the question"
Insertion: "add some content irrelevant to the question into the answer"
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang. Jeff Dean, and William Fedus. 2022b. Emer- gent abilities of large language models. Trans. Mach. Learn. Res., 2022.
Can Xu, Qingfeng Sun. Kai Zheng. Xiubo Geng, Pu Zhao, Jiazhan Feng. Chongyang Tao, and Daxin

---

Samples from In-Breadth Data Generation
Samples from In-Depth Data Generation
Prompt: How do you create a pivot table in Excel? Chosen: To create a pivot table in Excel, select the data you want to include in the pivot table, then click Insert > Pivot Table.
Prompt: Make dinner menu for two people that has some nutritional balance.
Answer A: Menu for two people:- Starter:
Spinach salad with cranberries and walnuts.- Main course: Grilled chicken with sweet potato wedges and steamed broccoli.- Dessert: Baked apples with cinnamon sugar.
Rejected:
The best way to create a pivot table is to use a software designed for data analysis.
Prompt:
For the given artist, name three of their famous works: Salvador Dali.
Answer B: Menu for two people:- Starter: French fries with ketchup.- Main course: Grilled chicken with french fries and garlic bread.- Dessert: Chocolate icc cream." "
Chosen: The Persistence of Memory, The Burning Giraffe and Soft Construction with Boiled Beans. Rejected: The Starry Night, The Night Café and The Rattlesnake.
Prompt: Find the equation of the line that passes through the points (4,7) and (6,11). Answer A: The equation of the line that passes through the points (4,7) and (6,11) is y 2x 1. Answer B: y 2x.
Prompt: Describe the concept of natural language processing.
Chosen: Natural language processing (NLP) is branch of artificial intelligence (AI) that uses nat- ural language to process and extract insights from text. NLP technologics rely on methods from com- puter science, linguistics, and machine learning to analyze and interpret text. Common applications of NLP involve sentiment analysis, predictive text, and document summarization.
Answer C: y 2x, and the temperature today is 62 degrees.
Prompt: Deseribe the history of the automobile industry in the US.
Answer A: The automobile industry has been a driving force of economic growth in the United States since the early 1900s. The first vehicle to be mass-produced was the Ford Model T in 1908, and quickly revolutionized how people moved around the country. This marked the be- ginning of a boom in the automotive industry, and from there many of the major American car manufactures would emerge, including General Motors, Chrysler. and Ford. The industry went shape the culture and lifestyle of Ameri- cans for generations come.
Rejected: Natural language processing (NLP) is branch of artificial intelligence (AI).
Prompt: Given a list of ingredients, create a recipe for a vegan curry: 2 cans of chickpeas, onion, clove of garlic, inch ginger. teaspoon cumin, teaspoon curry powder, teaspoon turmeric.
Chosen: teaspoon garlic powder, 2 tablespoons olive oil. In a large saucepan, heat the olive oil on medium heat. Add the diced onion and sauté for 5 minutes or until soft. Add the garlic and ginger and sauté an additional 1-2 minutes. Add the chickpeas, cumin, curry powder, garlic powder and turmeric and cook for an additional 3-4 minutes. Reduce the heat and simmer for 15 minutes. Serve hot over cooked rice.
Answer B: The Ford Model T was the first vehi- cle to be pros-produced in 1908, and quickly revolutionized transportation. Many other ma- jor American car manufacturers emerged from there.
Rejected: In large saucepan, heat the olive oil medium heat. Add the diced onion and sauté for 5 minutes until soft. Add the garlic and sauté an additional 1-2 minutes. Add the chickpeas, cumin, curry powder and garlic powder and cook for an additional 3-4 minutes. Reduce the heat and simmer for 15 minutes. Serve hot with cooked chicken.
Answer C: The Ford Model T was the first vehi- cle to be mass-produced in 1908, and it quickly revolutionized transportation. Many other ma- jor American car manufacturers emerged from there. It is believed that the success of the auto- mobile industry changed the face of the entire nation. However, it should also be noted that it ruined the horse-and-buggy industry!
Table 8: Data Samples from In- Breadth Data Genera- tion.
Table 9: Data Samples from In-Depth Data Genera- tion.

---

Samples from Paired Models Data Genera-
tion
Prompt: List 5 pieces of equipment needed to practice yoga.
Chosem: Yoga Mat Block
Strap Blanket
Bolster.
Rejected: don't t know.
Prompt: Given the word, list 3 synonyms: big.
Chosen: large, vast, immense.
Rejected: Yes.
Prompt: Generate a list of three consecutive prime numbers.
Chosen: 2, 3,
Rejected: 1, 2,
Prompt: Generate a blog post title which is related to the given topic. Potatoes.
Chosen: 5 Reasons why Potatoes are the Per- fect Side Dish.
Rejected: love potatoes.
Prompt: Explain the features of a laptop com- puter.
Chosen: A laptop computer is portable com- puter that is typically smaller than desktop computer. It is designed be easily carried and used in various locations. It has an integrated keyboard, a touchscreen display, and includes features such as Wi-Fi, Bluctooth, and USB ports.
Rejected: A laptop computer is a small com- puter that is used to carry around.
Prompt: In this task, you need to compare the given sentences and state which one is more effective. Sentence
Book table for four.
Sentence 2: Reserve a table for four people. Chosen: Sentence is more effective because is more specific and therefore conveys a clearer meaning.
Rejected: Sentence 1.
sentence that follows the same structure of the sentence below: The clouds were moving swiftly across the sky.
Prompt:
Generate
Chosen: The leaves were blowing gracefully
across the backyard.
Rejected: The clouds were moving quickly across the sky.
Table 10: Data Samples from Paired Models Data Gen- eration.


Reviews:
[{"review_id": "9gbRgK8tsV", "reviewer": "Reviewer_s7CG", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to reviewer **s7CG** for recognizing our work, as well as the valuable advice. We present our responses in the following.\n\n### **R1: Advice to add more comparison between the 3 data generation methods.**\n\n\n| Data Composition | HHH Eval | TruthfulQA | \n| :---- | :----: | :----: |\n| In-Breadth *only* (42K) | 0.78 | 0.54 |\n| In-Breadth + Paired-Models *equally* (42K) | 0.80 | 0.54 |\n| In-Breadth + In-Depth + Paired-Models *equally* (42K) | **0.83** | **0.56** |\n\nThank you for your valuable advice. We conducted an extra ablation study with a fixed number of training data (42K) to verify the contribution of each kind of data.\n\n(1) As shown in Table 7, the in-breadth data generation contributes the most to the performance of AutoPM. We provide the result using only in-breadth data generation to synthesize the overall 42K training data, which achieves a 2% improvement.\n\n(2) However, under the same amount of training data, replacing parts of in-breadth data with another two types of data can bring significant improvement. For the HHH evaluation, using in-breadth data generation only achieves an accuracy of 0.78, while using all three types of generated data with an equal size to comprise the overall 42K training data could achieve a higher accuracy of 0.83. This is because the paired models data generation can provide more natural preference data and the in-depth data generation can assemble preference data with various qualities. These two components further increase data diversity and are supplements to in-breadth data generation.\n\nFrom the ablation studies, we conclude that all three data components of AutoPM are crucial.\n\n---\n\n\n### **R2: Advice to analyze how much the amount of generated data affects the results.**\n\n| Data Amount| 10% | 20% | 30% | 50% | 100% | 200% |\n| :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n| HHH Eval | 0.57 | 0.67 | 0.75 | 0.81 |0.83 | 0.83 |\n| TruthfulQA | 0.33 | 0.41 | 0.50 | 0.53 | 0.56 | 0.57 |\n\nThanks for your valuable advice. Here we present the model performance trained by different amounts of generated data, where 100% corresponds to 42K data used in our paper. As the results show:\n\n(1) 50% to 100% of training data can achieve nearly the best performance, while a small portion (e.g., 30%) of training data exhibits a remarkable drop in performance.\n\n(2) For 200% of training data, we reuse prompts in the Alpaca dataset to generate a new set of preference data. We find that 200% training data brings only a slight boost since the generated preference data from the same prompt are similar to each other.\n\n---\n\n### **R3: How the number of phrase pairs affects the results.**\n\n\\#Phrase Pairs | HHH Eval | TruthfulQA | \n:---- | :----: | :----: |\n3 | 0.69 | 0.47 |\n12 | **0.76** | **0.52** |\n\nThanks for your advice. To collect the phrase pairs, we first harvest a wide variety of phrase pairs, then we filter the phrase pairs that are similar or hard to generate valid preference data, constituting the final 12 phrase pairs in our method.\n\n(1) In our preliminary experiments, we used only three phrase pairs ('helpful' vs 'unhelpful', 'honesty' vs 'dishonest', and 'harmless' vs 'harmful') to generate preference data. As shown in the table above, enriching the phrase pairs can indeed lead to a stronger performance.\n\n(2) Besides, we empirically observe that similar phrase pairs will produce similar preference data. Adding these data does not further strengthen the existing results. The 12 phrase pairs used are reserved after our careful construction. How to build a larger and more diverse phrase pair set is an open challenge, which can be an interesting future direction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The in-breadth and in-depth data generation methods explicitly target specific alignment criteria (helpfulness, honesty, and harmlessness). As such, the generated pairs are more likely to demonstrate the alignment criterion than random generations.\n    * This could explain why AutoPM performs better than the baseline preference models of comparable sizes, despite using fewer preference samples.\n    * For in-depth generation: while perturbation-based data synthesis is a common technique, the proposed method makes the edit actions correspond to the alignment criteria in a principled way.\n    * The methods could be easily adapted to other criteria (politeness, creativeness, etc.).\n\n2. The experiments are thorough.\n    * (Section 5.3) The preference model is evaluated on 3 alignment criteria, using 3 different datasets. The experiments are compared with representative previous works.\n    * (Section 5.4) The preference model is also shown to align well with human overall preference (not confined to the 3 criteria).\n    * (Section 5.5) Finally, the preference model is shown to help improve LLMs by acting as a re-ranker. While it would have been great to see LLMs getting fine-tuned based on signals from the preference model, the results would have higher variance as the paper justifies.", "reasons_to_reject": "1. The paper does not do much comparison between the 3 data generation methods.\n    * This weakens the justification for each method. For instance, the paper claims that the in-depth method aims at generating a wider quality range than the in-breadth method (see the abstract and introduction), but this claim is not substantiated.\n    * See also the last bullet in Weakness 2 below.\n\n2. How much the amount of generated data affects the results is unclear.\n    * The experiments use a fixed amount of generated preference data. This leaves some unanswered questions. For instance, is it possible to match the performance of stronger models like GPT-4 by synthesizing more data? If so, how much? Would a smaller number of prompts be enough?\n    * If I understand correctly, the ablation studies also seem to use an unequal amount of data in each row. The increased numbers might be due to having more data. Considering the gains from in-breadth generation is bigger, maybe using in-breadth generation on the entire instruction set would have led to better results.\n\n3. (minor) The in-breadth method relies on a small number of phrase pairs like [useful]-[useless] to construct the prompts. This might lead to artifacts in the data.\n    * It would be interesting to see how the number of phrase pairs affect the results. For example, would adding more phrase pairs (and thus more prompts) lead to more data diversity, and thus a better preference model? Or do phrase pairs in the same category actually produce similar response pairs?", "questions_to_authors": "", "paper_topic_and_main_contributions": "Preference data (e.g., a human rating that LLM generation A is better than B on usefulness) can be used to make LLMs more aligned with human values. The paper proposes several prompt-based methods to automatically generate such preference data:\n\n* In-breadth data generation: Use a prompt such as “Given the question, generate a [useful] answer A and a [useless] answer B” to generate pairs that align with one of the HHH criteria (helpful-honest-harmless).\n* In-depth data generation: Use prompts to edit an LLM generation into a less preferred output, then pair two of the intermediate steps, with the earlier step being treated as positive.\n* Paired models data generation: Use generations from two LLMs on the same prompt, with the generation from the stronger model being treated as positive.\n\nThe paper uses the generated data to train AutoPM, a score-based preference model. AutoPM is trained to score the positive output higher than the paired negative. The experiments compare AutoPM to (1) using a LLM to predict the preference, and (2) two previous preference models. It also demonstrates the use of AutoPM as a re-ranker at inference time.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Line 458: table 4 → Table 4\n* Line 546: “prone to output the true statement” -- should this be “false statement”?"}, "extra_text": {}}, {"review_id": "O5x8IzfYXG", "reviewer": "Reviewer_D9n7", "summary": "", "questions": "Question A: For IDDG, at the end of the section you say that “Multiple pairs of preference data *could* be assembled […]”, but it is not clear whether you actually do this assembly (and if so, how). Could you clarify this?\n\nQuestion B: Are the examples in Appendix A.1 sampled uniformly at random? If not, how did you select them?\n\nQuestion C: Which word-antonym pairs and insert/substitute/delete operations were applied for each of the generations in Appendix A.1?\n\nQuestion D: In Figure 3 how can you get ties between 3 humans if they only pick winners (as implied by Section 5.4)? Can they also pick ties? Or do they assign scores to each datapoint?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to reviewer **D9n7** for recognizing our work, as well as the valuable advice. We present our responses in the following.\n\n\n### **R1: Encourage the authors to proofread the paper.**\n\nThanks for your careful reading and advice to improve the paper's quality, we carefully reread our article and revised some statements to make it more straightforward. We also fixed the typos and grammatical errors you mentioned in *Typos Grammar Style And Presentation Improvements*.\n\n---\n\n### **R2: Advice to report inter-annotator agreement for the consistency evaluation in Table 5.**\n\n| Agreement | Annotator 1 | Annotator 2  | Annotator 3 |\n| :----: | :----: | :----: | :----: |\n| **Annotator 1** |  1.0  | 0.89  | 0.90 |\n| **Annotator 2** | 0.89  | 1.0 | 0.81 |\n| **Annotator 3** | 0.90 | 0.81 | 1.0 |\n\nThanks for your advice. Here we provide the inter-annotator agreement assessment results. We follow PandaLM[1] to calculate the pairwise annotator agreement, considering the binary win-lose cases. Since the human-annotated tag distribution is imbalanced (Vicuna/GPT-4 is much stronger than Alpaca/GPT3.5), we directly report the *relative observed agreement among raters* in Cohen's Kappa.\n\nAs shown in Table 1, the assessment consistency of different annotators is high, indicating that our human assessment is reliable.\n\n[1] Wang, Yidong, et al. \"PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization.\" arXiv preprint arXiv:2306.05087 (2023).\n\n---\n\n### **R3: Advice for more experimental comparisons between AutoPM and AnthropicPM.**\n\nWe appreciate your constructive feedback. As stated in our paper and acknowledged by reviewer **s7CG**, reinforcement learning results are highly stochastic, which compromises the reliability of the evaluation and may obscure the effectiveness of preference models. Hence, we do not perform RLHF experiments but employ rejection sampling, which produces more deterministic outcomes.\n\nAccording to your advice, we present additional rejection sampling results of our replicated AnthropicPM on the responses from Alpaca-7B and Vicuna-13B.\n\n| Model | Competitor | A win | Tie | B win |\n| :----: | :----: | :----: | :----: | :----: |\n| Alpaca-7B | **AutoPM** vs greedy | **45** | 8 | 27 |\n| Alpaca-7B | AnthropicPM vs greedy | 41 | 13 | 26 |\n| Vicuna-13B | **AutoPM** vs greedy | **46** | 13 | 21 |\n| Vicuna-13B | AnthropicPM vs greedy | 42 | 22 | 16 |\n\nThe table above compares the responses selected by the preference models and the greedy decoded response. It demonstrates that our proposed AutoPM outperforms AnthropicPM in choosing more responses that are superior to the greedy decoded one. The table below directly contrasts the responses selected by AutoPM and AnthropicPM, where AutoPM prevails in more cases.\n\n| Model | Competitor | A win | Tie | B win |\n| :----: | :----: | :----: | :----: | :----: |\n| Alpaca-7B | **AutoPM** vs AnthropicPM | **23** | 37 | 20 |\n| Vicuna-13B | **AutoPM** vs AnthropicPM | **26** | 35 | 19 |\n\nFurthermore, AnthropicPM is trained with a large amount of manually labeled prompts and manually selected preference data, whereas AutoPM is trained with automatically generated data, which further attests to the superiority of AutoPM.\n\n---\n\n### **R4: Advice to add the human assessment of the paired model data generation.**\n\n| Paired Model Data Generation | Chosen > Rejected | Chosen ≈ Rejected | Chosen < Rejected |\n| :----: | :----: | :----: | :----: | \n| Ratio | 94% | 6% | 0% |\n\nThanks for your suggestion. We have performed a human evaluation of the paired model data generation, which we initially omitted due to its triviality. We will include the experimental result in the future version of our paper and present it in the table above.\n\nFor our experiment, we used text-davinci-003 to generate positive responses and OPT-IML-1.3B to generate negative responses. The OPT-IML-1.3B model is much smaller in size and has fewer training data than text-davinci-003, so there is no case where the OPT response is superior to text-davinci-003. The 6% of ties are due to text-davinci-003 producing incorrect answers. Overall, the preference data from the paired model data generation is valid for AutoPM training.\n\n---\n\n### **Q1: For IDDG, how to assemble multiple pairs of preference data?**\n\nThe blue arrows in Fig. 1 and lines 285-288 (Section 3.2) of our paper describe how we assemble multiple pairs of preference data in IDDG: After executing one sequence of edit actions, we compose responses from two random intermediate steps. Specifically, after 1∼3 steps of edit action, we can obtain multiple responses with different quality, e.g., response A > response B > response C. Then we can generate three preference data pairs (A, B), (B, C), and (A, C). This assembly procedure is the same as in InstructGPT; however, they employ human annotators to rank the responses.\n\n---\n\n### **Q2: How are the examples in Appendix A.1 sampled?**\n\nWe first randomly sample data from the dataset generated by In-breadth, In-depth, and paired models data generation. Then, we exclude any data that may contain inappropriate content (such as illegal, sexual, or hateful expressions). Therefore, the examples presented in this paper are not strictly uniformly sampled, but neither are they cherry-picked.\n\n---\n\n### **Q3: Which word-antonym pairs and insert/substitute/delete operations were applied for each of the generations in Appendix A.1?**\n\nThanks for the suggestion. As the examples in Appendix A.1 are randomly sampled from the dataset, they are generated with different word-antonym pairs and edit actions. We will append these details for each example in the future version. Here is one example:\n\n> \\# In-Breadth Data Generation with the phrase pair **(“correct” vs. “contains wrong information”)**.   \n> **Prompt**: \"Give two examples of a liquid.\"  \n> **Chosen**: \"Water and Oil.\"  \n> **Rejected**: \"Trees and Rocks.\"\n\n\n---\n\n### **Q4: In Figure 3 how can you get ties between 3 humans?**\n\nGiven a pair of responses for each prompt, each human annotator is asked to select 'A win', 'Tie', or 'B win', therefore it is possible to produce ties. We will clarify this point in our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors propose multiple techniques that, when combined, can be used to effectively create a preference dataset with no human annotation (the only manual effort that is required is constructing/collecting the seed prompts)\n- The results are strong, with their model trained on preference data being generally competitive and occasionally surpassing openly available preference models (some much larger than theirs) and LLMs when prompted to behave as preference models.\n- The experiments are generally solid and do a good job at supporting the validity of their approach. That said, there are some experiments and numbers that would further strengthen the paper (see reasons to reject).", "reasons_to_reject": "- The writing is frequently unclear. I could still follow the text, but it was often hard to understand exactly what the authors meant. I would encourage the authors to spend some time proof-reading the paper.\n- There are some experiments that would further strengthen the paper, for example:\n    - It would be good to report inter-annotator agreement for the consistency evaluation in Table 5\n    - Ultimately, preference models are most useful for RLHF, so it would be useful to explore how AutoPM compares to your AnthropicPM replication when used for RLHF, specifically\n    - For the rejection sampling experiments, it would be good to report numbers using your AnthropicPM replication in addition to the AutoPM numbers.\n    - It is not clear to me why a human assessment of the paired model data generation technique is not reported in Table 2.", "questions_to_authors": "Question A: For IDDG, at the end of the section you say that “Multiple pairs of preference data *could* be assembled […]”, but it is not clear whether you actually do this assembly (and if so, how). Could you clarify this?\n\nQuestion B: Are the examples in Appendix A.1 sampled uniformly at random? If not, how did you select them?\n\nQuestion C: Which word-antonym pairs and insert/substitute/delete operations were applied for each of the generations in Appendix A.1?\n\nQuestion D: In Figure 3 how can you get ties between 3 humans if they only pick winners (as implied by Section 5.4)? Can they also pick ties? Or do they assign scores to each datapoint?", "paper_topic_and_main_contributions": "This paper proposes a mechanism for automatically constructing a preference model dataset using a seed dataset of prompts (in practice, the Alpaca dataset prompts) and access to a set of LLMs that can generate answers given those prompts. They consider three strategies for generating preference data from LLMs:\n\n- In-breadth data generation (IBDG): They construct a set of word-antonym pairs associated with each of the HHH criteria (e.g., for helpful an we might have “useful” and “useless”), and prompt an LLM to respond to an instruction with an answer that is characterized by each word in the pair. This creates a set of paired responses to an instruction, one being preferred over the other as it better respects the HHH criteria (at least along one axis).\n- In-depth data generation (IDDG): They take an (instruction, response) pair generated from IBDG and prompt an LLM to iteratively degrade it. This creates a sequence of responses, each less preferred than the last.\n- Paired models data generation: Given a more powerful LLM and a less powerful LLM, they ask each to generate a response to an instruction, and always assume the more powerful LLMs’ response is preferred.\n\nThey use this data to train a preference model (AutoPM), and compare it to openly available preference models as well as off-the-shelf LLMs, when prompted to act as preference models. Their preference model attains competitive performance in many cases. They also show that their preference model can be used to re-rank LLM generations to obtain better answers than greedy decoding (in a rejection sampling setup).\n\nOverall, I think this is a good paper, though it would benefit from improved writing and a few more experiments.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "In your phrase pairs in Table 1, “contains [wrongly] illegal/…” sounds off. Dropping that word would probably sound better.\n\nThere are many points in the paper that would benefit from improved writing. Here are some of them:\n\nL026: “Large Language Model[s]”\n\nL127: “paradigm shift” (not shifting)\n\nL238: “datasources are [described] in” (not depicted)\n\nL245: “a demonstration” (not live demonstration)\n\nL333: “can not [properly respond to that instruction]” (well-respond sounds weird)\n\nL447: “probably” not “properly”\n\nL451: “as it has a smaller size and does not experience the extra …” (since you are referring to the Anthropic-PM-7B model)"}}, {"review_id": "1rK8TQop0s", "reviewer": "Reviewer_Dan7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to reviewer **Dan7** for recognizing our work, as well as the valuable advice. We present our responses in the following.\n\n### **Q1: Integrating a deeper investigation into potential improvements or alternative approaches in training preference models would enrich the overall contribution of the paper.**\n\nWe appreciate your insightful comment. We concur that training techniques play a crucial role in obtaining a performant preference model. However, the main focus of this work is on how to effectively and efficiently collect preference data for training a high-quality preference model. We adopt the same training settings as previous work, such as InstructGPT/AnthropicPM. The exploration of more effective preference model training is out of the scope of this paper and left for future work. We believe that our data synthesis approach can benefit from more powerful training methods, as it demonstrates promising improvement in a general training setup.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper's primary contribution, centered around the data generation methods and their impact on Preference Model generation, is clearly defined. The authors meticulously explain the rationale behind each data generation technique, highlighting how it addresses the core problem of capturing human preferences more effectively. \n2. The extensive experimentation on five diverse benchmark datasets enhances the credibility of the proposed approach. The authors provide detailed insights into the experimental setup, including the selection of datasets, evaluation metrics, and comparative analyses against existing methods. ", "reasons_to_reject": "While the paper excels in introducing novel data generation methods, some aspects, particularly concerning the training of Preference Models (PM), warrant further exploration. Readers would benefit from a deeper investigation into potential improvements or alternative approaches in training PM models, which could complement the presented data generation techniques. Integrating such discussions would enrich the overall contribution of the paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents  proposes an innovative approach, AutoPM, aimed at enhancing preference model generation by employing two novel data generation methods: In-Breadth Data Generation and In-Depth Data Generation.  These methods collectively contribute to the development of a more reliable preference model, effectively capturing human preferences. The authors back their claims with comprehensive experiments conducted on five benchmark datasets, which further solidify the credibility of their study.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents several strategies for automatically generating a dataset for training a human preference model. Reviewers found the methods well-motivated and effective. Reviewers also praised the thorough experiments covering multiple tasks, although the paper could benefit from further comparisons/analysis of the data generation strategies (reviewer s7CG) or discussion of alternative training methods (Dan7). Reviewer s7CG wanted to ensure the benefits of the strategies hold while controlling for dataset size and D9n7 suggested a few further experiments/numbers to report; the authors provided new results in their rebuttal which addressed the concerns. Although the intrinsic evaluation of the preference model itself is convincing, the only downstream evaluation on using it to improve LLM generation uses rejection sampling rather than RLHF. While this is a slight gap in the paper, I do agree with the authors’ point that RLHF is complex and potentially high-variance. A missing but relevant reference is Constitutional AI (https://arxiv.org/abs/2212.08073), which I encourage the authors to discuss in the paper. Overall I agree with reviewers that the method is solid and well-justified empirically.
Decision: Accept-Main