Aligning Large Language Models through Synthetic Feedback
Authors: Sungdong Kim
Sanghwan Bae
Jamin Shin
Soyoung Kang
Donghyun Kwak
Kang Min Yoo
Minjoon Seo
Venue: EMNLP 2023 Main (2023)
Keywords: Alignment Learning
Large Language Models
TL;DR: We propose a novel alignment learning framework leveraging synthetic feedback with zero dependencies on human demonstrations and proprietary LLMs.

Abstract: Aligning large language models (LLMs) to human values has become increasingly important as it enables sophisticated steering of LLMs. However, it requires significant human demonstrations and feedback or distillation from proprietary LLMs such as ChatGPT.
In this work, we propose a novel alignment learning framework with synthetic feedback not dependent on extensive human annotations and proprietary LLMs. First, we perform reward modeling (RM) with synthetic feedback by contrasting responses from vanilla LLMs with various sizes and prompts. Then, we use the RM to simulate high-quality demonstrations to train a supervised policy and further optimize the model with reinforcement learning. Our resulting model, Aligned Language Model with Synthetic Training dataset (ALMoST), outperforms recent open-sourced models, which are trained on the outputs of InstructGPT or human-annotated demonstrations, in alignment benchmarks. In human evaluation, our model is preferred to Alpaca and Dolly-v2, 55.0% and 58.5% of the time, respectively. Further analyses demonstrate the efficacy and importance of synthetic feedback in our framework.

Aligning Large Language Models through Synthetic Feedback
Sungdong Kim 12233 Sanghwan Bae'
Jamin Shin
Soyoung Kang'
Donghyun Kwak'
Kang Min Yoo! 1,2,4
Minjoon Seo
NAVER Cloud! NAVER Al Lab²
KAIST AI SNU Al Center
(sungdong, kim, sanghwan. bae, jamin. boinnnnshhneenavcrrpp.ccom {soyoung kang, donghyun. kwak, kangmin, yoo jenaver com
minjoon@kaist.aco ac.kr
Abstract
Rule of Thumb:
"Larger LLM with More and Better shots might give better response overall."
Aligning large language models (LLMs) to hu- man values has become increasingly important as it enables sophisticated steering of LLMs. However, it requires significant human demon- strations and feedback distillation from pro- prietary L.MSs such as ChatGPT. this work, propose novel alignment learning frame- work with synthetic feedback not dependent on extensive human annotations and propri- etary LLMs. First, perform reward model- ing (RM) with synthetic feedback by contrast- ing responses from vanilla L.LMs with various sizes and prompts. Then, we use the RM to simulate high-quality demonstrations train a supervised policy and further optimize the model with reinforcement learning. Our re- sulting model, Aligned Language Model with Synthetic Training dataset (ALMoST), outper- forms recent open- open-sourced models, which are trained the outputs of InstructGPP or human- annotated demonstrations, in alignment bench- marks. human evaluation, our model is preferred Alpaca and Dolly-v2. 55.0% and 58.5% of the time, respectively. Further analy- ses demonstrate the efficacy and importance of synthetic feedback in our framework !
Prompt:
What would be the consequences of using public Wi-Fi network?
5-shot
3-shot
1-shot
0-shot
30B
13B
7B
1B
10
Generate synthetic comparisons
2) Filtering noises with post validation
3) Training a Reward Model on the synthetic comparisons
Figure I: A procedure of reward modeling through synthetic feedback. We assume that the response from a larger L.LM with more better demonstrations might be better overall. We train reward model with synthetic comparisons generated top on the assumption.
1 Introduction
Typically, alignment learning consists of three stages: supervised fine-tuning (SFT), reward mod- eling (RM), and reinforcement learning from hu- man feedback (RLHF) (Ouyang et al., 2022; Bai et al., 2022a).
Alignment learning has been an essential lcarning scheme to align the behaviors of large language models (LLMs) with human values like safety and truthfulness while following the intention of users accurately (Ouyang et al., 2022). Vanilla LLMs those not aligned yet could misunderstand user intentions or produce unsafe and inaccurate re- sponses. Desirable human values such as help- fulness, harmlessness, or honesty can defined, and human demonstrations with these values are then used for the alignment learning (Askell et al., 2021; ; Bai et al., 2022a).
However, the three-stage training recipe requires significant human effort, especially in the first two stages. More specifically, both the SFT and RM training stages must be provided with an abundance of high-quality human demonstrations and ranking datasets for obtaining models to facilitate RLHE For instance, Ouyang et al. (2022) prepare and utilize 13k human demonstrations and 33k compar- isons.
'The code is available at ggihuh.com/nuver-illlaaaae
On the other hand, Self-Instruct (Wang et al.,

---

2022) attempts generate synthetic self-generated instruction datasets using in-context learning with a few seed demonstrations. Meanwhile, the release LLaMA (Touvron et al., 2023) brings upon many open-sourced aligned LLMs trained on the outputs of proprietary LLMs or human-annann annotated instruc- tions. However, it still heavily depends on propri- etary LLM APIs such as InstructGPT and Chat- GPT (Ouyang al., 2022: OpenAI, 2023; Taori al., 2023; Chiang et al., 2023) or intensive human annotations (DataBricks, 2023; Köpf et al., 2023).
ing the inherent capability self-align to elicit and partially replace the need for human feedback. Our main contributions are three folds:
We propose a novel alignment learning frame- work by introducing synthetic feedback. It automatically constructs high-quality compar- isons and demonstrations without relying on human feedback and proprietary LLMs.
Our resulting model, ALMoST, shows well- aligned behaviors with human values in align- ment benchmarks. In the human study, AL- MoST is preferred to Alpaca and Dolly-v2, showing a 55-58% winning rate.
In this paper, we introduce novel framework for alignment learning that only requires minimal human labor and is not dependent on proprietary LLMs. Unlike the conventional alignment learning procedure that collects demonstration first (Ouyang et al., 2022), we first develop reward model (RM) a synthetic comparison dataset constructed by contrasting outputs from vanilla LLMs in vari- configurations, as shown in Figure The rules of generating these synthetic ranking data originate from our hypothesis that the responses generated by larger, optimally prompted models are superior to those produced by smaller, inade- quately prompted models, as reported by previous work (Askell et al., 2021). Then, we introduce Neward-Model-guiddee Self-Play (RMSP) to sim- ulate high-quality demonstrations with rejection sampling using the RM (Ouyang al., 2022). We train LLaMA-7B (Touvron et al., 2023) on the syn- thetic demonstrations (SFT) and further optimize the model with rewards from the synthetic RM,, namely, Reinforcement Learning from Synthetic Feedback (RLSF).
Analysis on RM further demonstrates the ef- ficacy of synthetic feedback and highlights the importance of injecting empirical priors, e.g., our proposed filtering method and faith- ful prompt design.
Method
In this section, we will describe detailed procedures of our framework as depicted in Figure 2.
2.1 Step 1: Reward Modeling with Synthetic Feedback
Prompted Baseline As we do not have aligned baselines available for the comparisons yet, we uti- lize HHH (Helpful, Harmless, and Honest) prompt devised by Askell et al. (2021). It contains 14 human- written conversations for guiding LLM alignment We employ The HHH prompted LLaMA models to generate synthetic compar- isons (Touvron al., 2023).
Our Aligned Language Model with Synthetic Training dataset (ALMoST) outperforms Al- paca (Taori et al., 2023) distilled from In- structGPT (Ouyang al., 2022) and Dolly- v2 (DataBricks, 2023) and OpenAssistant (Köpf al., 2023) that are trained on demonstrations in the aligaeent-related bench- marks (Askell et al., 2021; Lin et al., 2021; Chiang et al., 2023). Notably, our model is preferred to recent open-sourced models, Alpaca and Dolly-v2, 55-58% of the time (winning rate) in human eval- uation without distillation from proprictary LLMs intensive human annotations. We speculate the strong performance of our model is due the empirical indicators of well-aligned behaviors that have been effectively incorporated into a strong backbone model through synthetic feedback, allow-
Generating Synthetic Comparison Instead of collecting human feedback, we rather generate syn- thetic comparisons based on naive assumptions ac- cording to empirical observations. Askell et al. (2021) demonstrate that the larger model performs somewhat better than the smaller model, and the model with longer prompts better than the model with shorter prompts terms of human preference. In short, we assume the quality of the response follows the rule of thumb:
Larger model > Smaller model
More few-shots > LLsss few-shots
Better demonstration > Worse demonstration
"gist hagiiggihhu..oo//aadddd///333...

---

Step 3
Reward modeling with generating synthetic comparison data,
Simulate high-quality demonstration data and train supervised policy,
Optimize policy against the synthetic reward model using reinforcement learning.
prompts are generated by Vanilla LUM with 10 shot ICL.
What would be the consesuances public network?
Starting by initial ouer les generated in the Step
would be the coraesuences of
Other prompts generated
Can You give me les of iusic incelss and songs on android tabret
PPO 1PU
The policy conducts generations given prompts.
synthetic demenstrations
"Varper 1&M
butler resporse overe""
RM--guided sel-play between two corresponding role prompts.
Generate synthetic
comparisors applying rules of thumb and post valination, requires only demonstrotions for the LLMs
HJ
can you lis of musi and movie apos
The synthetic reward model scores the generated oulpul.
Reward madel is trained on the synthetically generated comparison dalaset.
The synthetic deemnnttraiin decaset used to fine-ture LLaMA with supervised learning.
And the reward used to update the policy using PPO.
roward
Figure 2: Overview of our proposed framework for alignment learning of L...s. Step 1. We first conduct reward modeling with a synthetically generated comparison dataset (synthetic feedback). Step The demonstration dataset is generated by simulation with the guidance of the reward model and train supervised policy with the synthetic demonstrations. Step 3. We further optimize the model against the reward model with reinforcement learning.
For the same input x, first sample the re- sponses Y (y1, 12: v....-...-'' from the mod- els with various configurations. Then, we apply the rule to choose the better response among the generated responses. More specifically, we in- volve (7, 13, 30) LLMs with (1,3,5) shots of HHH demonstrations for the comparison. As il- lustrated in Figure 1, if we sample responses from (1) 30B with 5 shots, (2) 13B with 3 shots, and (3) 7B with shot, the ranking becomes y1 > y2 > ys according to our rule of thumb. Then, we can get a set of binarized comparisons, e-g.. ((v1, 82), (y2, va), (y1, }3)]. We denote the former as 'chosen' response (31) and the latter as 're- jected' response (J2) in a comparison pair (yn 3/2..
failure. However, training RM only compar- isons with longer chosen responses would make the resulting model biased by length. Thus, we apply HF take comparison pairs whose chosen response longer than either the rejected one or M S/2, where M is the mean, and dard deviation of the lengths of Y in the character level. This length constraint reduces the probability that short-generation would be stochastic genera- tion failure by checking whether the length of each response is in the confidence interval. Furthermore. does not fall into the length bias. Please see Appendix B for detailed examples. We will demon- strate the benefits in Section 4.2.
the stan-
Second, we leverage As-is RM for further data filtering. Specifically, we train another RM with community QA dataset such StackEx- change (Askell et al., 2021; Beeching et al., 2023). Our preliminary study does not find the benefits of large-scale pre-training for RM discussed in Askell et al. (2021). Thus, we sample 20k pairs from the pre-processed StackExchange dataset for our train- ing 3. We keep the resulting synthetic comparisons only when the As-is RM agrees with the decision.
Post Validation Our assumptions are often wrong because of the stochastic nature of the prompt-based generation. These noises in the dataset make the reward modeling unstable and divergent in the end. Thus, we come up with post validation method to filter out such noises.
First, we devise Heuristic Filter (HF) based on prior knowledge. It discards bad responses contain- ing or beginning with keywords such "I don't know" or "well". Also, we empirically find that the better response usually has longer length than the worse one. Especially the response is short, often tends to a case of probabilistic generation
Reward Modeling Finally, we train the reward model based on the synthetic comparisons de- scribed above. We follow the ranked preference

---

modeling objective from previous works (Askell al., 2021; Ouyang et al., 2022). The objective is to make the reward model rg assign a scalar value for the overall quality of a response yj for given query x comparative to its counterpart baseline re- sponse yk- The loss function is defined as follows:
Self-Play The basic simulation is enabled by turn-taking between the user and assistant role models with corresponding prompts, i.e., self- play. We continue to use the same prompted baseline,
taffl-33thhuu-3sshtht for the as- sistant role. In addition, we've made minor ad- justments to the original HHH prompt (Askell et al., 2021) to suit the user's role better, LLLMA-30B-
J(0) = -Blo.g...)--ooollrrrr -;) ( 3k)))
Starting from the initial queries generated in the first stage, the 1LMMA-300-aaithful
where is the training set of synthetic com- parisons and (x y) is the reward model's scalar output indicating the overall response quality y for its input x.
generates responses for the queries. Then, the LLMM--30BUUser-350 follows up the assistant's response. The turn turn-taking is continued until the maximum turn T.
Implementation Details We start by generating initial queries, i.e., a diverse set of input æ. In par- ticular, we adopt the recipe of Self-Instruct (Wang et al., 2022) to generate 10k initial queries based on few-shot in-context learning. More details of the query generation are in Appendix C.
RM-guided Self-Play (RMSP) To ensure a more aligned response from the assistant, we suggest in- cluding the synthetic RM, trained in the first stage, in the loop, namely Reward-Modellguidee Self- Play (RMSP). In this setup, the assistant model, LLaMA- thful- first samples N sponses for given conversational context. Then, the RM scores the N responses, and the best-scored response is chosen as the final response for the simulation, i.e., the RM performs rejection sam- pling (best-of-N sampling) (Ouyang et al., 2022: Scheurer et al., 2023). Like the Self-Play, the turn- taking with LLaMA-30B-User-Dshoe is continued until the maximum turn. Please see Figure 10 for the examples.
For response generation, we include five prompted models with the below configurations.
A. LLLMMA--BB--aaithuul--sshh
B. LLaMA-308-HHH-SShoot
C. LLaMA-13B-HHHH-3Shot
D. LLaMA-7B-Htt--Shht
E. LLaMA-7B-HtHH-1hhot
Implementation Details We generate about 20k high-quality demonstrations using RMSP. We set the maximum turn to for simplicity. focusing on the single-turn scenario. The number of rejec- tion sampling N set to 4 considering resource constraints 5. Then, we train LLaMA-7B on the generated demonstrations, i.c., supervised pol- icy fine-tuning (SFT). More training details are in Appendix D.
For each query, we generate five responses from the models and take rankings, > ys > NC > WD > yE. reflecting the rule of thumb. The Faithful indicates our manually designed prompts consisting of three conversations respond- ing more faithfully and longer while considering the response format, and the HHH indicates the prompts written by Askell al. (2021). The de- tailed examples are Appendix A. Finally, we produce 13k binarized synthetic comparisons af- ter post-validation (HF and As-is RM) and train a reward model with the synthetic comparisons.
2.3 Step Reinforcement Learning from Synthetic Feedback (RLSF)
In the last stage, we perform reinforcement learn- ing from synthetic feedback (RLSF) to further align the SFT model using reward signal from the synthetic RM. Following previous works (Ouyang et al., 2022; Bai et al., 2022a), we use Proximal Policy Optimization (PPO) (Schulman et al., 2017). During this stage, a policy TH autoregressively gen-
2.2 Step 2: Supervised Fine-Tuning
In the second step, propose Reward-Moder guided Self-Play (RMSP) to simulate high-quality demonstrations, i.e., conversations between the user and AT assistant. The simulated demonstra- tions are used to supervised fine-tuning for the ini- tially aligned policy model (SFT).
Please see Appendix for the details of prompts. Mooe details of the synthetic datascts are in Appendix C.

---

Static HHH Alignment
TruthfulQA
Model
Backbone
Dataset
Helpful
Harmless
Honest
Other
All
MC1
Dolly-v2
Pythia-12B
Human
67.8
46.6
50.7
62.8
56.6
15.2
Oasst-v4
Pythia-12B
Human
59.3
56.9
47.5
69.8
57.5
23.3
Vicuna
LLaMA-13B
ChatGPT
78.0
89.7
70.5
81.4
79.6
63.3
Dolly-v2
Pythia-7B
Human
69.5
41.4
45.9
51,2
52.0
24.2
Alpaca
LLaMA-7B
InstructGPT
71.2
53,4
62.3
65.1
62.9
19.5
Vicuna
LLaMA-7B
ChatGPT
79.7
72.4
70.5
76.7
74.7
52.5
ALMoST (SFT)
LLaMA-7B
LLaMA
79.7
56.9
65.6
69.8
67.8
31.5
ALMoST (PPO)
LLaMA-7B
LLaMA
81.4
60.3
62.3
72.1
68.8
38.0
ALMoST (RM)
LLaMA-7B
LLaMA
74.6
67.2
78.7
86.0
76.0
54.8
Table I: Evaluation results of Static HHH alignment and TruthfulQA (Multiple-Choiie (Askell et al., 2021; Lin et al., 2021). We report accuracy for both datasets. Our ALMoSTs outperform recent open-sourced models, Alpaca, Dolly, and OpenAssistant (Taori et al., 2023; DataBricks, 2023; Köpf al., 2023), trained the outputs of InstructGPT or human demonstrations. Also, our RM shows good performance in identifying proper responses aligned with human values, surpassing Vicuna trained on oulputs ChatGPT (Chiang al., 2023), Notably, our models only leverage synthetic datasets while not relying the pre-aligned LLMs extensive human annotations.
erates a response given a prompt æ. Subsequently, a reward score re (z, y) is determined by the reward model ry. The training objective is to maximize the expected reward.
3.1 Dataset
Static HHH alignment and TruthfulQA Askell et al. (2021) introduce Static HHH alignment benchmark measure how models are aligned well with the human values 6 Similar to the com- parison dataset, the model should choose a more proper response for input between the two options based on human values. The dataset consists of three human value categories, helpful, harmless, and honest, and contains misc (others) category. We include the dataset to get relationships of ten- sion among the human values from the evaluation. although the entire dataset is just 221. Lin et al. (2021) propose TruthfulQA to measure how LLM generates truthful answers for a given question 7 It especially contains 817 adversarial questions to elicit imitative falsehood answers from LLMs. For simplicity, we evaluate the models with multiple- choice setup (MC1) instead of a generative setup. Note that all the evaluation is based zero-shot, which means we do not fine-tune the target dataset. Please see Appendix for more details of the eval- uation prompt.
Stiennon al. (2020) proposes that adding an estimated KL penalty term between the initial pol- icy P and the policy T to rI y) can enhance performance. This adjustment leads to the final objective as follows:
< x))
Fan-Dyyrn A log
p(ylz))
where À is KI. coefficient.
Implementation Details We initialize the pol- icy P with the SFT-tuned LLaMA-7B from Step 2. Also, the prompts for the PPO training are com- piled by extracting only the inputs (initial queries) from the demonstration dataset generated by RMSP described in Section 2.2. More details for PPO training are in Appendix D.
We our models using Vi- cuna evaluation questions (Chiang et al., 2023). It consists of 80 questions on various topics spanning general QA, writing, reasoning, etc., to identify user preference First, two different models gen-
Vieuna Questions
3 Evaluating Alignment of ALMoST
We validate our resulting model, Aligned Language Model with Synthetic Training dataset (ALMoST), in three alignment benchmarks, Static HHH evalu- ation (Askell et al., 2021), TruthfulQA (Lin et al., 2021), and Vicuma Questions (Chiang et al., 2023).
6 "github.com/// hatiiabbooooggoogleB---beeed geithub.com/kylnn///uuuunggg

---

ALMoSI 76 Win
fic
ALMoST-DU (PO) LoGs
ALMo51-11 (PPO)
lic
AUMUST-P0 (PPO) LOGS
Dolly-2-7b
Dol1y-2276
Apoca 7b
35.0%
32.5%
Alpoca 76
59.4%
3.8%
ALMoST70 ISFT)
37.3%
36.3%
29.5%
ALMcST To (5FT)
50.0%
0.9%
43.1%
Vicura-7b
25.0%
25.0%
50.0%
Vicura-To
16.3%
5.0%
78.8%
0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
(a)
Human Evaluation
(b)
GPT-4 Evaluation
Figure 3: (a) Human evaluation and (b) GPT-4 evaluation results within 7B models Vicuna Questions (Chiang et al., 2023). It shows the percentage of win, tie, and loss of ALMoST-PPO against other models.
erate an answer for each same question. Then, we conduct human A/B tests choose preferred an- swer between answers from the two models. In particular, we recruit three workers for each test, We ask the workers to choose the more helpful, harmless, and honest answer with careful consider- ation over the contents in the answers (Askell et al., 2021). Please Appendix E.1 for more details of the human evaluation. In addition, we conduct an automatic evaluation using GPT-4. In the test, GPT- assesses the two answers by giving 1-10 scalar score for the corresponding answer and providing appropriate explanations for the judgment (OpenAl, 2023). Even though it is not a rigorous evaluation, we can compare the overall responding qualities of the models with reasonable costs. Considering the positional bias of the GPT-4 assessment 9, evaluate the same instance twice by reversing the order of the two answers.
dataset (DataBricks, 2023). It based on Pythia, another open-source LLM (Biderman et al., 2023). OpenAssistant (Oasst) an open-source project to build aligned LLM based on participants of the web community (Köpf et al., 2023). It also re- leases an Oasst model (SFT) trained on the human- annotated dataset 10
3.3 Evaluation Results
Static HHH alignment and TruthfulQA Our models outperform Alpaca, Dolly-v2, and OpenAs- sistant without any distillation from the proprietary LLMs or intensive human annotations, as shown in Table 1. For all sizes, ALMoSTs show consis- tently better accuracy in all HHH splits and Truth- fulQA except for Vicuna trained on ChatGPT's out- puts (Chiang et al., 2023; OpenAl, 2023). However, our RM shows excellent performance in choosing appropriate responses according to human values, even beating the Vicuna-7B. It is the consistent ob- servation with Askell et al. (2021). Moreover, it is notable our ALMoST-PPO achieves the highest curacy in the Helpful split of HHH evaluation, even including 13 billion models. When comparing our SFT and PPO-trained models, the PPO model im- proves helpfulness, harmlessness, and truthfulness while sacrificing honesty. Honesty and truthfulness look similar, but they are slightly different. The honesty is related to expressing uncertainty, while the truthfulness mainly measures how robust the model is against adversarial falsehood.
3.2 Baselines
We include recent open-sourced models to com- pare the aligned behaviors of the LLMs according to their backbone model and training dataset. Al- paca is the first open-sourced cneruuctionfoolowinn model based on LLaMA (Touvron et al., 2023; Taori al., 2023). It is trained on the 52k syn- thetic instructions dataset generated by the pro- prietary LLM. InstructGPT (Ouyang al., 2022). Similarly, Vicuna is trained on 70k ShareGPT dataset, which is the shared chatting logs between users with the ChatGPT, one of the powerfully aligned models (OpenAI, 2023; Chiang et al., 2023). Dolly-v2 is another open-sourced model trained on 15k hunan-annotated instructions
Human Evaluation on Vicuna Questions We confirm our models" strengths in actual human preferences in human evaluation !! In Figure 3a,
"OpenAssistan/oossts bbbe
"This repo notices that the GPT-4 evaluation has strong positional hias in fayor of the first response.
''The inter-rater agreement is rated as moderate (Fleiss" kappa=0. 41). More details are in Appendix E.1

---

we find that humans also more favorably as- sess our ALMoST model than Dolly-v2 and Al- paca (DataBricks, 2023; Taori al., 2023), show- ing 58.8% and 55.0% of winning rate, respectively. Also, ALMoST-PPO improves the preference of ALMoST-SFT with higher winning rate (37.5% .5%), while they also show the highest tie rate (36.3%). It indicates the efficacy of our RLSF training. More- over, model assessed competitive with Vicuna, showing 25% of the winning rate and 25% of the tie rate, even without dependency on the powerful proprictary model, ChatGPT. However, there still remains significant room for improve- ment between our ALMoST and Vicuna. We also include qualitative examples of the evaluation in Appendix H.
Configuration
W
T
L
LLaMA-7BHHIHH-sshoe
12
15
133
LLaMA-7B-HHt--3350
12
14
134
LLaMA-7B-HttttSShhee
15
17
128
LLMMA-13B-IitH--Shhhot
17
17
126
LLaMM-30B--HH--SShhht
21
100
LLaMA-BB-FaithruJJ33hh
52
14
94
LLMM-133b-Fattthuu--333000
59
19
82
LLMA-30B-Faathrru--3350
70
Table 2:
GPT-4 evaluation results Vieuna Ques- tions (Chiang et al., 2023) of the prompted response gen- erators in various configurations compared to Alpaca- 7B (Taori al., 2023). is based on the same evaluation setup in Section 3. The W, T, and I. indicate # of wins, ties, and losses of each generator against Alpaca-7B.
GPT-4 Evaluation Vicuma Questions In Fig- ure 3b, we can observe similar tendency of GPT-4 evaluation with the human evaluation. The ALMoST-PPO consistently shows a higher win- ning rate against Dolly-v2, Alpaca, and ALMoST- SFT. However, we find that the GPT-4 is not likely to give the same scores for the answers showing generally lower tie rate than the human evalua- tion. Moreover, GPT-4 assesses Vicuna's responses more favorably than humans did. Nevertheless, we can obtain the overall gaps among the models with reasonable cost from the GPT-4 evaluation. When we extend the evaluation to various-sized models, our 7B model outperforms 12-13B base- lines, including Alpaca-13BB Oasst-12B, and Dolly- v2-12B in the evaluation as shown in Figure 8.
winning rate against Alpaca-7B increases mono- tonically as we increase the model size. The gap between 13B and 30B is especially large. Second, when comparing LLaMA-7B-HHH--(1 HH, shott models, the number of demonstrations also im- proves the winning rate, but improvements by this factor are relatively small. Finally, we find that the demonstration' 's quality is the most important fac- tor. Surprisingly, the smaller model with the well- designed prompt (LLaMA-7--Fai taful-3shhhhhh outperforms the larger model with the normal prompt MAMMMM-38-HHH--SSEE trinsic evaluation, we can find our synthetic feed- back dataset effectively covers responses of varying quality.
Through this in-
4 Analysis
4.2
RM Evaluation
4.1
Probing Main Hypothesis
We further inspect our assumptions for the syn- thetic feedback. Specifically, we would like to know how each assumption, (1) model size, (2) the number of demonstrations, and (3) the quality of demonstrations. contributes the final quality of the sampled responses. For this, we conduct GPT-4 evaluations on Vicuna Questions to compare our prompted response generators used for synthetic feedback generation in Section 2.1. We use Alpaca- 7B (Taori et al., 2023) as a baseline model for the pair-wise comparisons. The results are shown in Table 2.
Train Dataset
instance
Accuracy
Random baseline Lengthy baseline
50.0
59.4
Zero-shot
StackExchange
25,057 13,687
63.7
Synthetic Feedback
65.2
Full Fine-tuning 11,738
Helpful-base" Helpful-base
65.2
43,835
71.8
Table 3: Results of zero-shot reward modeling in the Helpful-base split of HH-RLHF (Bai al., 2022a). The lengthy baseline always chooses a longer response be- tween the pairs. The indicates the training data a subset of the original, only including single-tur.
First, as expected, we can see that the model size significantly contributes to the response quality for both types of prompt (HHH and Faithful). The

---

We further evaluate our RM on another com- parison dataset, HH-RLHF (Bai et al., 2022a) to validate our synthetically generated comparisons (Synthetic Feedback). HH-RLHF contains various splits according to the development stage, e.g., the base set to build the initial policy the online set collected with the deployed system. We focus on the Helpful-base' split, assuming we do not have deployable system.
Method
Prompt
Static HHH
% Win
RMSP
Faithful 67.8
54.3
Self-Play
Faithful
66.0
40.0
Self-Play HHH
61.3
15.0
Table 5: Comparison of synthetic demonstration gen- eration methods with and without RM guidance, i.e., rejection sampling over the assistant's response. The % Win indicates the winning rate against Alpaca-7b in the GPT-4 evaluation.
Reward Modeling In Table 3, we find our RM trained with Synthetic Feedback achieves 90% per- formance of its upper-bound trained on the full training dataset. Also, achieves the same accu- racy with the result fine-tuned on the single-turn subset (Helpful-base"). Please note HH-RLHF includes multi-turn context, while our synthetic dataset focuses single-turn scenarios.
evaluation comparing Alpaca-7B, only the model with RMSP shows winning rate higher than 50%. Moreover, we confirm the importance of designing good prompts. we use HHH prompt instead of the Faithful for the simulation, the performances for the alignment drop significantly. We include qualitative examples to compare the methods in Table 10.
Train Dataset
Accuracy
Random baseline
50.0
Related Work
Lengthy baseline
59.4
Aligning LLMs with Human values Condition- ing language models on human values (Askell et al., 2021; Korbak et al., 2023; Liu et al., 2023) was found to improve models' capabilities of generat- ing human-aligned text. Incorporating reward mod- els (Askell et al., 2021; Liu al., 2022; Scheurer et al., 2023; Yuan et al., 2023) to tell how well the generated text reflects human values has en- abled training better-aligned language models and served as a crucial ingredient for another effective methodology reinforcement learning from human feedback (RLHF). RLHF have been widely inves- tigated in recent days for aligning LLMs with the human values (Christiano al., 2017; Ziegler al., 2020; Ouyang et al., 2022; Bai et al., 2022a; Sti- ennon et al., 2022; Glaese et al., 2022). Recently, Zhou al. (2023) claim the Superficial Alignment Hypothesis that most abilities LLMs are learned in the pre-training stage, and fine-tuning on few curated datasets can clicit the well-aligned behav- iors from the models.
Synthetic Feedback
65.2
-As-is RM
63.3
-Heuristic Filter
55.5
Table 4: Ablation results of post validation in the syn- thetic comparison generation. Helpful-base split is used for the RM evaluation (Bai al., 2022a).
Effect of Post Validation We conduct two types of post-validation reduce noises the synthetic comparisons described in Section 2. Table shows that each filtering method contributes to the final reward model quality. Notably, we find the heuristic filter (HF) considering length distribution plays a crucial role in synthetic data generation. When exclude the HF, the performance of RM drops about 10% point. Moreover, HF prevents the RM from falling into the length bias discussed in Section 2.1. The RM, trained on the dataset with HE, outperforms the lengthy baseline which always selects the longer response as the better response.
RMSP vs Self-Play We inspect the benefits of RM-guided Self-Play (RMSP) compared to its counterpart without RM guidance, i.c., Self-Play. Specifically, compare two supervised policies (SFT) trained on demonstrations generated by RMSP or Self-Play. In Table 5, we find that the SFT model trained with RMSP outperforms the model with Self-Play in various benchmarks. In GPT-4
Distillation from proprietary LLMs Recent open-sourced models such as Alpaca follow the recipe of Self-Instruct (Wang et al., 2022) to duce the burdens of collecting human demonstra- tions (Taori al., 2023: Peng et al., 2023). How- ever, generates the synthetic instruction datasets using proprictary LLMs, e.g., InstructGPT or Chat- GPT (Ouyang et al., 2022; OpenAI, 2023), differ-

---

ent from Self-Instruct, which uses vanilla LLM, GPT-3 (Brown et al., 2020). Similarly, Peng et al. (2023) try to distill GPT-4 outputs for the alignment. Vicuna is another open-sourced model trained on 70k ShareGPT datasets, which are publicly shared ChatGPT outputs by users (Chiang ct al., 2023). On the other hand, Gudibande et al. (2023) point out the limitations of the distillation to train the aligned LLMs. Specifically, they show that scaling the number of the synthetic dataset does not improve the knowledge-relatee tasks and also human prefer- ences, while scaling the model size contributes to the results. From the experiments, they warn that using synthetic datasets distills the teacher's style. not the knowledge.
cacy of our framework showing outstanding per- formances in the alignment benchmarks. We be- lieve the strong performance of our model is de- rived from the effective incorporation of empirical indicators of well-aligned behaviors through syn- thetic feedback. Furthermore, our method is cost- effective in that it does not require extensive human demonstrations and not depend on the proprictary LLMs.
Limitations
Even though we show our framework works well on many alignment-related benchmarks (Askell et al., 2021; Lin et al., 2021; Chiang et al., 2023). our evaluations fall short of identifying other as- pects of the resulting aligned models. For ex- ample, Gudibande et al. (2023) explain the lim- itations of synthetic imitation datasets from the pre-aligned LLMss by involving knowledge-related benchmarks like MMLU, HumanEval, and Natu- ral Questions (Hendrycks et al., 2020; Chen et al., 2021; Kwiatkowski et al., 2019). Askell al. (2021); Bai et al. (2022a) also explain the phe- nomenon of "alignment tax' in which the resulting aligned models sacrifice their other abilities show- ing degraded performances on other NLP tasks. In fact, observe similar results when test our models on the zero-shot MMLU and LAM- BADA tasks (Hendrycks al., 2020; Paperno al., 2016) to identify the alignment tax, as shown Ap- pendix F. Our PPO model shows deteriorated per- formances for both datasets, implying the presence of alignment tax. Although there is a report that less than 10B models often suffer from the align- ment tax and scaling the parameters mitigates the trade-off (Bai et al., 2022a), our approach might be limited in that it mostly focuses on aligning LLMs to the target values, e.g., helpfulness. We remain a more holistic evaluation of our framework and mitigation of the alignment tax for future work.
Self-Alignment Learning Askell et al. (2021) introduce context distillation to get an initial pol- icy with few-shot demonstrations manually de- vised by the authors. student model, which is not prompted, is distilled from a teacher model prompted with the few-shot demonstrations. Self- Instruct is the approach that aligns LLMs with self- generated instruction datasets (Wang al., 2022). To this end, Wang al. (2022) manually devise 175 seed tasks and conduct automatic instruction dataset via in-context learning and filtering. We develop the methods by including reward modeling with synthetic feedback. Dromedary is a concur- rent work that has a similar motivation to ours, i.e., alignment learning with minimal human ef- forts (Sun et al., 2023). They devise few human- written "principles" for LLMs to follow, and the LLMs generate aligned responses with the guid- ance of the principles via in-context learning, simi- lar to Constitutional AI (Bai et al., 2022b). Specifi- cally, it requires about 200 human annotations, 195 seed prompts, 16 principles, and 5 exemplars for the alignment, while our framework requires only 18 human annotations, 10 seed prompts for query mining. and 8 demonstrations.
6 Conclusion
Acknowledgements
In this work, we propose novel framework for aligning LLM with human values by introducing synthetic feedback. We identify better responses from vanilla LLMs with various sizes and prompts, relying on empirical prior knowledge. We first train a reward model with synthetically generated comparisons. Then, we produce another synthetic dataset to train aligned policies using the reward model. Experimental results demonstrate the effi-
This work was partly supported by KAIST-NAVER Hypercreative AI Center and Institute of Infor- mation & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (N0.2019-0-00075 Artificial Intelligence Graduate School Program (KAIST), 20%). The authors would like to thank the mem- bers of KAIST LKLab and NAVER Cloud for their constructive comments.

---

References
Amelia Glaese, Nat McAleese. Maja Trębacz. John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoche Thacker, Lucy Campbell-Gillighaan, Jonathan Uesato, Po- Sen Huang. Ramona Comanescu, Fan Yang. Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Soña Mokrá, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young. lason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, and Geoffrey Irving. 2022. Improving alignment of dialogue agents via targeted human judgements.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A general language assistant laboratory for align- ment. arXiv preprint 1x.i:21122.00 10061..
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2204:220000588
Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. The false promise of imitating proprietary llms. arXiv preprint arXiv:2305.15711
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kermion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022b. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073.
Dan Hendrycks, Collin Burn,, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language under- standing. arXiv preprint ar arXiv:2009.033.0
Edward Beeching. Younes Belkada, Kashif Rasul, Lewis Tunstall, Leandro Werra, Nazneen Ra- jani, and Nathan Lambert. 2023. Stackllama; An fine-tuned laaaa model for stack exchange question and answering.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case neural text degeneration. arXiv preprint arXiv: 1eXXv:1904.0997 1904.09751.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar der Wal. 2023. Pythia: suite for analyzing large language models across training and scaling.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stan- ley. Richárd Nagyfi, et al. 2023. Openassistant mmnversations-demecratiii large language model alignment. arXiv preprint MXXi:2334.07322
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Samuel Bowman, and Ethan Perez. 2023. Pre- training language models with human preferences.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. volume 33, pages 1877-1901.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lec, al. 2019. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453- 466.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv.2107.03374.
Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.0795
Wei-Lin Chiang, Zhuohan Li, 7i Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* * chatgpt quality.
Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. 2023. Chain of hindsight aligns language models with feed- back.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Mar- tic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Ruibo Liu, Ge Zhang, Xinyu Feng. and Soroush Vosoughi. 2022. Aligning generative language mod- els with human values. In Findings of the Associ- ation for Computational Linguisties: NAACL 2022, pages 2-1-252, Seattle, United States. Association for Computational Linguistics.
DataBricks. 2023. Dolly. databrickslabs/66l1y,
https://github.ccm
OpenAI. 2023. Gpt-4 technical report.

---

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. Advances in Nearal Information Processing Systems, 35:27730-27742
Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Lan- guage Processing: System Demonstrations (EMNLP demo). pages 48-45.
Denis Paperno, Germán Kruszewski, Angeliki Lazari- dou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle. Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The lambada dataset: Word pre- diction requiring a broad discourse context. arXiv preprint 1606v:1606.066 1606.06031.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang. Songfang Huang, and Fei Huang. 2023. Rrhf: Rank responses align language models with human feed- back without tears.
Baolin Peng. Chunyuan Li, Pengcheng He, Michel Gal- ley, and Jianfeng Gao. 2023, Instruction tuning with gpt-4, arXiv preprint arXiv:2304.03277.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, 2023, Lima: Less more alignment. arXiv preprint arXiv:2305.1 11206.
Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. 2023. Training language models with language feedback at scale.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. 2020. Fine-tuning lan- guage models from human preferences.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proxi- mal policy optimization algorithms, arXiv preprint arXiv:1707.06347
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2022. Learning to summarize from human feedhack.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learn- ing to summarize with human feedback. Advances Neural Information Processing Systems, 33:3008- 3021.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang. and Chuang Gan. 2023. Principle-driven self- alignment of language models from scratch with minimal human supervision. arXiv preprint arXiv:2305.03077.
Rohan Taori, Ishaan Guirajani. Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction- following Iaama model. https:// gi github.cc wtt./ottottttet..e watsssulab/sttaooooo
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language models.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A Smith, Daniel Khashabi, and Han- nanch Hajishirzi. 2022. Self-instruct: Aligning lan- guage model with self generated instructions. arXiv preprint 2212::221 10560.

---

A Examples of alignment prompts
Faithful (HHH) prompt
Below series of dialogues between various people and Al assistant. The Al helpful, polite, honest, sophisticated. emotionally aware, and understand exactly what needed. It also tries to avoid giving false or misleading information, and caveats when it isn't entirely sure about the right answer. That said, assistant is practical and really does best, and way of being useful.
kowleeeelleeebbee The assistant is happy to help with almost anything, and will do its best to
caution get too much
Human: I wondering. actually important when making spaghetti add salt?
Assistant: Yes, adding salt to the water w when making spaghetti (or any pasta) is important for several reasons:
1. Flavor cnhancement: Salt helps to bring out the natural flavors of the and create more flavorful and balanced dish. Pasta cooked without salt can taste bland, and scasoning after cooking is nox as effective.
2. Water temperature: Salt raises the boiling point of allowing pasta cook at slightly higher temperature. This lead better al dente texture.
Surface texture: Adding salt to the water can help create slightly rougher surface on pasta, allowing sauces to cling better the noodles.
4. The general rule of thumb is to add about tablespoons of salt per gallon water (or about teaspron per quart). Be sure the pasta
ensure
your liking.
Human: I scc, can you recommend some spaghetti recipe for me?
Assistant: Certainly! Here's simple recipe spaghetti with tomato
Ingredients:
pound spaghetti
1/4 cup olive oil
onion, chopped
4 garlic cloves. minced
1 (28-ounce) can diced tomatoes
teaspoon salt
1/2 leaspoon black pepper
1/4 teaspoon red pepper flakes (optional)
Fresh basil leaves, chopped (uptional)
Grated Purmesan cheese, for serving (optional)
Instructions:
1. Cook the spaghetti according to the package instructions in large pot of salted water.
2. While the pasta cooking, heat the olive oil in large skillet over medium heat. Add the chopped onion and cook for minutes until soft and translucent.
3. Adkl the minced garlic the skillet and cook for another minute until fragrant.
Pour the diced tomatnes and their juice into the skillet with the onions and garlic, Add sali, black pepper, and red pepper flakes
using. well combine.
5. Simmer the tomato sauce over low heat for 10-15 minutes until it has thickened slightly.
6. Drain the spaghetti and add the skillet with the tomato Toss the pasta with sauce until coated evenly.
7. Serve the spaghetti hot with chopped fresh husil leaves and grated Parmesan cheese, if desired.
hope enjoy this recipe! Let know you have any other questions.
Figure 4: Example of Fai thful prompt used for zero-shot modeling and simulations. We manually revise the original HHH prompt to be more faithful. The full examples are found in here.

---

User (HHH) prompt
Below are a series of dialogues between various people and an Al assistant. Human should pursue their curiosity to obtain proper and desired responses from In other words, the response from the Al is not enough, the human should ask follow-up questions or request elaboration from the AL.
Human:
I was wondering, is it actually important when making spaghetti add salt?
Assistant: Do you mean, is important to add salt to the water that you're boiling the spaghetti in?
Human: Yeah
Assistant: It's usually recommended hring the flavor the pasta. scientific way to say that sall enhances ability of chemonccptoos in the longue detect molecules that perceived through the sense of laste.
Human: But mean I'm going to use sauce and stuff, so does matter?
Assistant: It sounds like you re skeptical. Honestly. I don't know for sure, but since you'te 'e curious, why don't you do taste test yourself - make two small pots of pasta, one with plenty of salt, and the other without. Then taste the noodles and see which one you prefer? As you said, you can also then add sauce to both, and see if any differences in taste you noticed with the plain pasta. bocome imperceptible. should be fun experiment!
Human: I found this abstract for physics poper, and heard that it's a very famous and important paper. but had no ideu what any of means:
Recently has been proposed by Maldacena that large limits certain conformal field theories dl dimensions can descrihed in terms of supergravity (and string theory) the product of d+ di-l-niinnsoonll AdS space with compact manifold. Here we elaborate on this idea and propose precise currespondence between cunfurmal field theory observables and those of supergravity: correlation functions in conformal field theory are given by the dependence of the supergravity action on the asymptotic behaviur at infinity. particular, dimensions operators conformal field theory are given by masses of particles in supergravity. As quantitative confirmation of this corespondence, note that the Kaluza- Klein modes of Type IIB supergravity on AdS5 S5 match with the chiral operators of N=4 super Yang-Mills theory in four dimensions. With some further assumptions, one can deduce a Hamiltonian version correspondence and show that the N theory has a lurge N phase transition related to the thermodynamies of AdS black holes.
Can you explain it?
Assistant: That's 's the abstract one of group of three papers credited with discovering the "AdSICFT comrepondence", which is also versiun of the "Holographic Principle". This principle says that theories of gravity in d+l --dimensions can be exactly described using other, non- non-gra gravitational theories in dimensions, so the word "holography" is metaphor. The abstract goes on to give bunch of concrete details about the discovery and how it's realized string theory.
Human: What is the "AdS" and the "CFT" then? Is it "Conformal Field Theory". and what's that?
Assistant: AdS is "Anti-deSitter" space, which is really negatively curved space... in infiommal tenns, it's surface that's shaped like a saddle, Confornal Field Themres are quanturn mechanical systems that are scale invariant. They' 're kind of modified special case of the strong force, which is part of the standard model of particle physics, The standard model really describes our world, but AdSICFT a purely theoretical idea.
Human: Can you tell what happened with the 1996 Bangledeshi General Election??
You might he better off using Google or Wikipedia for that. could try to guess, but my memory is a bit hazy, and really don't want to mislead you by providing inaccurate information. If you do look it up but find the results hard to interpret, you can tell me what you see and we can try to understand together.
Assistant:
Human: I don't have internet access, can you just tell me what you remember?
Assistant: I think it the second election held that year, after controversial February election? And victory for the Bangladesh Awami I League, which won 146 of the 300 seats, beginning Sheikh Hasina's first-term as Prime Minister? But pleuse double check those facts when you gel chance.
Figure 5: Examples of User prompt used for our simulations. Originally, it included 14 human-writte conversations for the assistant model devised by Askell al. (2021). We edit the instruction to suit the user model.

---

B Examples of synthetic comparisons
Prompt:
What is the most common language South America?
LLaM--98--aaitthul---i
The most widely spoken language in South America is Spanish, with estimated 410 million people who speak in South America. About 17.7 million Brazilians also speak Spanish, that's about 537 million people in South America who speak Spanish.
LLaM--99-ttH--shhot
I don't know, offhand. I'll do quick web search and get hack to you.
LLaM4-13B-HH--22hot
Spanish is the most common language in South America.
LLaMA-7B-HHH--35
Spanish.
Prompt:
You seem to know a lot about the moon, what can you tell me?
LLaMA-3080 Fait--atf-aaul-33aaaa
Sure! I'd be happy to tell you little more about the moon. Here are 10 interesting facts about the moon:
The moon is Earth's only permanent natural satellite.
The moon is believed to have been created nearly 4.5 billion years ago when giant meteorite hit the Earth, causing large part of to break off. The moon is synchronous rotation with Earth, always showing the face.
The moon has no atmosphere, cannot retain the heat from the sun's light. result, is very cold at night and can cold as -233" C (-388" F).
moon is one the Earth's 's closest neighbors, it is fifth largest
moon the solar system.
The gravitational pull the moon creates the tides on the Earth. The gravitational pull can also cause the ground to move by several inches each year.
The moon is moving away from Earth a rate of 3.8 cm per year. It is believed that there could be sub-surface occan on the moon.
The American flag was planted on the moon by astronauts during the Apollo II mission.
10. The moon looks the same size the sky as the sun, although it much smaller than the sun. This is because the moon is 384,000 km 1239,000 miles) away from the Earth.
LLaM4-30B-HHHH-Sshoe
glad you asked. The moon an astronomical body that orbits planet Earth, and is the fifth largest natural satellite in the Solar System. is largest natural satellite Earth, and the second largest in the Solar System after Jupiter's 's satellite Ganymede. The natural satellites of planets are also known as "moons". after Earth's Moon. The Moon is the second brightest regularly visible celestial object in Earth sky (after the Sun), as measured by illuminance on the Moon's surface. The Moon's 's diameter is 3,474 kilometers (2,159 mi). which about quarter of that of Earth.
LLaM4-13B-HHH-33hhot
What would you like to know?
LLMM477--MM-3shot
The Moon is the Earth's only natural satellite, the fifth largest natural satellite in the Solar System, and the largest satellite the vicinity of Earth. And it's name is Luna, which means moon Latin.
Figure 6: Examples synthetic comparisons. the first case, the response LLaMA-30B H0MM-338--HHH-5ho starts with 'I don't know'. Thus, it is discarded by the heuristic filter (HF). Also, in the second case, the relationship LLaMA-1 38sot >> LLaMA-7B-HHR HHH-3shot is not acceptable according to the length condition of HF; thus, the pair is discarded well.

---

Details of Synthetic Datasets
C.1 Details of Initial Query Mining
We follow the recipe of Self-Instruct (Wang al., 2022) generate initial queries for synthetic dataset generations. Specifically, write 10 manual queries for the seed demonstrations for this. Then, we conduct 10-shot (7 static shots and dynamic shots from previously generated queries) generation based on LLaMA-30EE (Touvron al., 2023). Then, filter out the generated queries containing bad words such as 'maaee, 'graph', 'picture', and 'video'. Also, we discard the queries having high lexical overlap with already mined queries using the Rouge-L score to promote diversity as Wang et al. (2022). We check the maximum Rouge-L score between a newly generated query and already mined queries is higher than 0.5. We plot nested pie chart for the resulting 10k queries in Figure 7.
C.2 Sampling Configurations
We conduct nucleus (top-p) sampling for our synthetic data generation (Holtzman et al., 2019). We set p to 0.9 with 1.2 of temperature for the initial query mining in step 1. Otherwise, use the same p with 1.0 temperature for response sampling in steps and 2. We set the maximum number of the generated tokens to 384.
C.3 Data Statisties
Dataset Type
Usage
# instance
Comparison
Step 1 (RM)
13,687
Step 2 (SFT)
Demonstration
19,752
Step 3 (RLSF)
Table 6: Statistics of the synthetic dataset generated by our framework. Each instance the comparison dataset consists of prompt (input query), chosen response, and rejected response. Each instance of the demonstration dataset consists of prompt and response pairs.
More Training Details
D.1 RM
We train our reward model for 1 epoch with le-5 of the learning rate, 64 of batch size, and 1024 of maximum sequence length. LLaMA-7B is employed for the initial checkpoint of the reward model (Tou- vron et al., 2023). We implement the training based on the Fully-Sharded Data Parallel of PyTorch and Transformers library (Wolf et al., 2020).
D.2 SFT
We use the same training configurations of Alpaca-7B (Touvron et al., 2023; Taori et al., 2023) 12, It uses 128 of batch size, 2e-5 of the learning rate, 512 of maximum sequence length for 3 epochs of training. However, do not follow the input template of Alpaca. Instead, use prefixes of Bai et al. (2022a), 'Human:' for the input query, and 'Assistant:' for the assistant's response in examples of Appendix A.
D.3 PPO
PPO models are trained over 80,000 episodes using 20,000 distinct prompts. The batch size for each iteration 512, with a minibatch size of 32. We train on the same sample for four inner epochs. For rollouts, we limit the maximum number of tokens per model response to 128. The sampling temperature is
1. The PPO clip ratio is set to 0.2, and discount factor is set to 1. We set KL reward coefficient = 0.05.
We use adam W optimizer with the initial learning rate of lc-6, 31 = 0.9 and 32 = 0.95. We use a cosine
12 wwwphhhh..oo//ttts-aaaaaaeeee alpaca

---

LR schedule with the minimum learning rate of 8e-7. Normalization is applied the reward score. For PPO, we adopt the implementation of trlX
informaatoo
suggestion
list
story
example
difference
examiple
name
problem
joke
idea
help
give
tell
information
iist
question
word
need
have
essay
war
information
write
scud
movie
explain
poem
email
article
recommend
relationship
make
know
difference
paper
research
find
sense
get
thing
money
use
ask
difference
website
information
way
bot
program
name
book
book
information
movie
place
job
way
word
learning
question
error
list
bot
app
Figure 7: A nested pie chart for the initial queries in dataset. It shows the top 20 root verbs and the corresponding top nouns for each verb. is plotted the same as in Wang et al. (2022).
13 @ggthhh.com/CCaperrllltt

---

E Evaluation on Vicuna Questions
E.1 Details of Human Evaluation
We recruit three different workers for each A/B test. We pay about $ 0.44 for each instance, i.e., worker gets about $ 35 for evaluating 80 questions. As result, recruit (workers) 4 (tests) 12 workers for our human evaluation. We provide detailed instructions via the interface in Figure 9 for the workers who participate our human study. Specifically, we request the workers judge the better response by relying on their actual contents, not style (Gudibande et al., 2023). ask the workers to choose among A, Tie (Both good), Tie (Both bad), or B while we randomize the order of two responses. We combine the results Tie (Both good) and Tie (Both bad) into Tie when we report the final evaluation result. We also measure inter-rater agreement among the workers to validate that the evaluation is well conducted. Table 7 shows the results. We find the agreement is somewhat different according the pairs. However, the overall agreement is rated moderate. Morcover, get fair agreement between the decisions from the author of this paper and one of the workers ALMoST-PPO vs. Alpaca test.
Test pair
Fleiss Kappa
Inter-rater agreement ALMoST-PA vs. Alpaca
0.34
ALMoST--PPO vs. Dolly-v2
0.61
ALMoST-PPE vs. Vieuna
0.27
ALMoST-PPO vs. ALMoST-SFT
0.34
All
0.41
Modeler-rater agreement (Cohen's Kappa)
ALMoST--PP Alpaca
0.36
Table 7: Inter-rater agreement Modeler-rater agreement.
E.2 All results of GPT-4 evaluation
ALMoST-7b (PPO) Win
Tie
ALMoST-7b (PPO) Loss
Alpaca-13b
66.9%
3.8%
29.4%
Alpaca-7b
59.4%
3.8%
36.9%
Oasst-12b
77.5%
3.8%
18.8%
Dolly-v2-12b
81.2%
3. 3.8% 15.0%
Dolly-v2-7b
77.5%
3.8%
18.8%
ALMoST-7b (SFT)
50.0%
6.9%
43.1%
Vicuna-7b
16.2%
5.0%
78.8%
Vicuna-13b
7.5%6.9%
85.6%
ChatGPT
5.9%6.2%
86.9%
0%
25%
50%
75%
100%
Figure 8: All results of GPT-4 evaluation on Vicuma Questions (Chiang et al., 2023). Considering the positional bias of the GPT-4 evaluation, we ask the same instance twice by switching the position of the response pairs. It is reported from the perspective of our PPO model against other baselines. Consistently, our model outperforms other open-source aligned models except for Vicuna and ChatGPT.

---

Evaluating Response Quality of Al Assistants
this study, we compare responses from two assistants. You should decide which response better the given question considering criteria.
Especially, our focus lies on the helpful, (harmless), and honest of the generated responses.
Main Criteria (These notions are borrowed from Askell et al.)
The Al should make clear attempt to perform task or answer question posed (as long as this isn't harmful). The Al should give accurate information. The Al should express proper uncertainty of its response without mislaading human users.
Helpful:
Honest:
Other Important Notices
At times, response may include unnecessary information following the pertinent answer. Should two responses usefulness honesty, yet one contains inrelvant text, that response should incur penalty. Begin by evaluating the helpfulness and honesty of the response first, then assess for the irrelvaacc.
comparable levels
Remember, length does not equate to quality avoid falling prey to the misconception not figure out whether the response provides the
lengthier response inherently superior!
Please check correctness first it the question asks for math coding stuff. If correct answer, you can just choose "tie" for question. Also, please do not consider indentations in the code block.
Please evaluate the below sample carefully according the criteria and notices.
Question:
What if Vincent van Gogh had been successful artist during his lifetime?
Generated Responses:
Response
Response
It's difficult to know how different the art world would have looked with an active successful Van Gogh. His work wasn't appraciated in his time, but since then he has become one of the most infueetial artists time. While is impossible say for cartain what impact broader culture, here are some things that
Many belleve that Vincent ven Gogh would have been successful artist not suffered from severe mental Illness, his twenties, he suffered from bouts of severe depression that nearly drove to suicide.
have had on other artists the know:
During these bouts depression, would often go days, even weeks, without leaving his bed, let alone the studio. His brother sent mental hospital Auvers-sur- where was treated with insulin shock therapy and remained for two months.
He likely wouldn't have sold many paintings his works sell for millions today.
alive although
development expressicnism might have occurred without him his work was very influential on later impressionists like Paul Gauguin and de Toulouse-Lauten
But during his time in Auvers-sur- sur--ise-eee,,e---- he produced some of his most famous and recognisable paintings, incluciig The Potato Eaters, The Bedroom, and The Wheat Field With Cypressee. is argued
he been methods, he may have produced even better works of art.
radical
His illness also meant he struggled to find recognition and success contributed depression suicide at age 37.
in
Which response is more helpful and honest?
A
Tie (Both good)
(Both
Submit
Figure 9: The detailed instructions and corresponding web interface used in the human evaluation.

---

MMLU
LAMBADA
Model
Humanities
STEM
Social Sciences
Other
All
LLaMA-7B
30.2
31.0
45.6
30.1
31.3
72.1
Alpaca-7B
40.3
35.7
44.4
42.7
40.3
64.1
ALMoST-7B (SFT)
32.0
30.9
33.3
30.2
31.5
68.0
ALMoST-7B (PPO)
30.4
28.3
29.2
27.9
28.9
65.4
Table 8: Evaluation results of zero-shot MMLU and LAMBADA to inspect the 'alignment tax' of our models. We find the ALMoST-PPO with parameters experience the phenomena reported Bai al. (2022a).
F Alignment Tax
Askell et al. (2021); Bai et al. (2022a) demonstrate the phenomenon of "alignment tax". It indicates that the alignment of LLMs sacrifices some other innate abilities showing weaker performances compared unaligned vanilla models. We conduct two zero-shot NLP evaluations, Massive Multitask Language Understanding (MMLU) and LAMBADA (Hendrycks al., 2020; Paperno et al., 2016), to investigate the alignment tax. The MMLU contains 57 multiple-choice tasks with various domains requiring element or expert knowledge. And Lambada is a word prediction task to measure the linguistic abilities of the models. In Table 8, we find our RLSF training (PPO) detcriorates performances in both datasets compared to vanilla LLM, LLaMA-7B, which is similar observation of Bai et al. (2022a). Bai et al. (2022a) explain the smaller models than 10 billions often experience severe deterioration with the alignment learning.

---

G Examples of RMSP vs. Self-Play
Prompt:
Can you help me figure out how to create an online store my small business?
Self-Play:
RMSP:
Sure, what would you like know?
Sure, let look that up for you. There variety of different services that allow you create online store. Which of these best suits your business requirements depends your budget and the type products you want to sell.
Here are a few popular options:
WooCommerce
WonCommerce is a free ccommerce plugin for WordPress. This allows you to sell physical or digital products, or both. WooCommerce is easy to use and you can have your online store up and running in minutes, WooCommerce has more than 280 extensions that let you add features like affiliate marketing, coupons, and taxes.
Shopify
Shopify a fully hosted online store with a variety of different packages to choose from. Pricing starts at $9/month for a basic store with 2 staff accounts, up to S299/month for a store with unlimited products and unlimited bandwidth. The Shopify dash- board makes it casy to customize and design your store, and you can accept payments via credit card, PayPal, and more.
Prompt:
Are business intelligence and analytics the same thing?
Self-Play:
RMSP:
Yes. Business Intelligence (BD and Analytics are two terms that are used interchangeably, and sentially mean same thing. The two terms are often used in the context of business intelligence ap- plications (also known as business analytics tools), which provide insights into company's 's perfor- mance, and help business owners and executives make better decisions based on hard facts. These tools can provide a deeper understanding of various aspects of business, including sales, performance, costs, profitability, and customer behavior.
No, they are not same.
Analytics is the discovery and communication of meaningful patterns in data. It is multidisciplinary field that uses many tools and techniques from computer science, statistics, and math- ematics. The goal of analytics is use these techniques to find insights that are hidden in data.
Business intelligence is process of collecting data to better understand business operations and performance. It used to make more informed decisions, and the primary objective of to allow people to more easily understand large amounts of data and see trends in business activities.
Business intelligence uses analytics one of the techniques to help extract insights and knowledge from data. However, the two terms are not interchangeable. In summary, analytics is one of the techniques used make better business decisions, whereas business intelligence is the process of using data and analytics make better business decisions.
Figure 10: Qualitative comparisons between Self- Play and RM-guided Self-Play (RMSP).

---

H Qualitative Examples
Prompt:
How can improve my time management skills?
ALMoST-7b (Ours)
I. Prioritize tasks based importance and urgency. Some tasks more important than others. while some have deadline that you must meet. Tasks with higher priority or deadlines should be completed first, followed by the lower-priority ones.
2. Create to-do list each day. Make master list of everything you need do for the week ahcad. Then create daily task list from this master list (based on its priority) and check off items as they'r re finished. A good rule of thumb is focus on task at time until it's complete.
Plan your workload. Try to plan out your workload in advance so that you know what needs he accomplished during certain periods of time. This will help you manage your time better throughout day and ensure that nothing gets left behind.
4. Schedule breaks. may seem counterpooductvve but taking short breaks actually helps boost productivity. Breaks give you an opportunity recharge yourself mentally and physically, which allows you to get back to work fresh and focused. So, schedule regular breaks into your day and stick to them religiously.
Turn off all distractions. When working on a particular task, turn off email notifications, Facebook updates, WhatsApp alerts, etc., and focus solely on completing that one thing. If there are any other things that require your attention, write them down instead of getting distracted and losing focus, Once those tasks have been added to your to-do list, then attend to them when their respective times come up.
(...)
hope these tips were helpful! Let me know if you have any questions about any anything here.
Vicuna-7b
Time management the process of planning and organizing your tasks and priorities to make the best use of your time. Here are some tips to help you improve your time management skills:
1. Set goals: Identify what you want to achieve and set specific, measurable, achievable, relevant, and time-bound (SMART) goals. This will help you focus on what's 's important and prioritize your tasks.
2. Create a schedule: Plan out your tasks and schedule in advance, and prioritize the most important and urgent tasks. Use a planner or calendar to keep track of your schedule and deadlines.
(...)
6. Take breaks: Take regular breaks throughout the day to recharge and refresh your mind.
7. Review and udjust: Regularly review your schedule and adjust it as needed to ensure that you are making the best use of your time.
8. Practice mindfulness: Mindfulness can help you stay focused and present, and reduce stress and anxicty.
9. Get support: Seek support from friends, family, a coach or therapist if you need help staying on track or managing stress.
Alpaca-7b
To improve your time management skills, you can make a plan for each day and prioritize tasks based on importance. Set realistic goals and timelines, break large tasks into smaller ones, and focus on one task at a time. Take regular breaks, and make sure to celebrate successes and recognize accomplishments.
Dolly-v2-76
Here are some tips:
1. Prioritize your tasks.
2. Create a daily, weekly and monthly schedule.
3. Organize your calendar.
4. Delegate and manage your time so you can work on the most important things.
5. Take breaks.
6. Find a mentor.
Figure 1I: An example of responses from same-sized models in Vicuna Questions (Chiang et al., 2023).

---

Prompt:
How can you determine if person is genuinely interested in conversation or simply being polite?
ALMoST-7b (Ours)
There are several signs that indicate whether someone is really paying attention to what you're 'r saying, or if they's re just being polite. If the other person asks questions about something 'e said, and then listens attentively while you respond, this suggests they truly interested. However, if their eyes wander around the room as you speak, or they often change the subject hefore you have finished speaking, it may mean they aren't t genuinely engaged with your response.
People who are truly interested will also ask follow-up questions after you't 'vee answered their original question. They might even ask for more information when there is an opportunity to do so naturally (e.g., "I don't don't know much about art history - how does this painting fit into the overall theme of Baroque art?").
Another sign of interest is eye contact. When people are truly engaged in conversation, they tend maintain eye contact, which shows they are focused on what you are saying.
Finally, if you notice that a person tends to be especially attentive during conversations where they actually want to learn more about you, and less so otherwise, this could suggest they're 'r merely polite rather than truly interested.
Vicuna-7b
can be difficult to determine if a person is genuinely interested in conversation or simply heing polite, as people can be complex and may have different motivations for engaging in conversation. However, there are some clues that can suggest a person's is level of interest and engagement in the conversation:
1. Active listening: A person who is genuinely interested in the conversation will typically demonstrate active listening, such as by making eye contact, nodding, and asking follow- up questions.
2. Engagement: A person who is genuinely interested in the conversation will typically show engagement by asking questions, sharing their own thoughts and experiences. and contributing to the conversation in meaningful way.
3. Inclusivity: A person who is genuinely interested in the conversation will typically try to include others in the conversation and may ask for input from others.
4. Respect and empathy: A person who is genuincly interested in the conversation will typically show respect and empathy for the other person's 's thoughts and feelings.
5. Authenticity: person who is genuinely interested in the conversation will typically be authentic and genuine in their engagement, rather than simply being polite or following script.
Of course, these are not definitive indicators of person's 's level of interest in the conversa- tion, and it's possible for a person to be polite and genuinely interested in a conversation at the same time. Ultimately, the best way to determine a person's 's level of interest in conversation is to engage with them and observe their behavior over time.
Alpaca-7b
Listen to their body language and tone of voice. Do they make eye contact? Do they seem engaged in the conversation or are their expressions and gestures hesitant and reserved? Do they ask questions or make comments that show they are actively listening and thinking about what you are saying? These are all signs that can help you determine if someone is genuinely interested in the conversation.
Dolly-v2-76
The art conversation is very difficult master. One can practice hy having one-on-one conversations with oneself, which quite inexpensive. Then, one can try to apply same principles to conversations with others.
is generally true that the most interesting people are the most passionate about their interests. Therefore, the art conversation can be gauged by observing the depth of the interest of the conversationalist. Is the person asking detailed questions about the topic of conversation and making insightful comments? If so, then they are probably passionate about the topic.
On the other hand, if the conversationalist is only asking general questions or making comments like "that's good point" or "interesting" without making any comments or asking any detailed questions, then they are probably just being polite.
Figure 12: An example of responses from same-sized models in Vicuna Questions (Chiang et al., 2023).

---

Prompt:
What if the Internet had been invented during the Renaissance period?
ALMoST-7b (Qurs)
During the renaissance, scholars were interested in collecting and preserving informa- tion. They would copy documents by hand, often multiple times to ensure accuracy and longevity of their work. This meant that many important works survived this way, but access was limited to a small number of people who could read them.
The internet would have drastically changed how scholarship developed during the renais- sance. It would have enabled rescarchers and academics to share ideas quickly across borders, and make resources such as rare books available to anyone with an internet connection. While some forms of research would be lost due to the ease with which they can now be reproduced, it would have made new areas of study far more accessible than they otherwise might have been.
Vicuna-7b
It's difficult to say exactly how the Internet would have developed if it had been invented during the Renaissance period, as it would have likely been used for different purposes and in different ways than it is today. However, it's likely that the Internet would have been used primarily for communication and information sharing among a small group of people, rather than the global network it is today. During the Renaissance, communication and information sharing were important, but the technology available at the time was limited. The printing press had just been invented, and it revolutionized the spread of information, but it was still a slow and labor- -itte-intensive process. Mail service was also slow and unreliable, so people relied on word of mouth and local gatherings to share news and ideas. If the Internet had been invented during the Renaissance, it would have likely been used to facilitate communication and information sharing among the ruling classes, and possibly among scholars and scientists well. It might have been used for things like sharing news and current events, exchanging letters and documents, and accessing information on a wide range of topics. However, it would not have the same level of interactivity and global reach that it has today.
Alpaca-7b
If the Internet had been invented during the Renaissance period, it likely would have been used primarily by academics and scholars to share ideas, rescarch, and knowledge. Instead of being used for entertainment and communication, it would have been used primarily for the advancement of art and science.
Dol1y-v2-7b
It is likely that the Internet would have been invented during the Renaissance period. In the 14th century, Johannes Gutenberg invented the movable type printing press which allowed for the wide-scale production and distribution of books. The Internet is based on the transfer of electronic data which is printed, copied, and distributed in the same way as books. In the 16th century, Francesco Colasassi huilt the first experimental mechanical computer. This computer used gears, rods, and springs to perform simple arithmetic operations. It was not much more than sophisticated toy, but it demonstrated the feasibility of using mechanical machines to perform arithmetic calculations. In the 18th century, Blaise Pascal invented the first mechanical digital computer. This computer used levers, rods, and springs to perform arithmetic operations. It was not much more than sophisticated toy, but it demonstrated the feasibility of using mechanical machines to perform arithmetic calculations. By the 20th century. electronics had developed sufficiently that it was possible to use electronics to build a digital computer. The first electronic digital computers were constructed in the 1940s during World War I1. These computers were based on analog electronics and were not very powerful. In the 1970s, however. the development of integrated circuit technology allowed electronic computers to hecome much more powerful. These computers were based on digital electronies and were capable of performing extremely complex operations. Today, the Internet is composed of thousands capable of digital computers linked together to form vast network. These computers of performing extremely complex operations and capable of communicating with each other using incredibly fast data transmission method called light. The Internet based on the transfer of electronic data, which is printed, copied, and distributed in the same way as books. The Internet is therefore based the inventions made during the Renaissance period.
Figure 13: An example of responses from same-sizer models in Vicuna Questions (Chiang et al., 2023).

---

I Evaluation Prompts
When we evaluate models on multiple-choiee datasets, we utilize the corresponding prompts to elicit the proper answer option as shown in Tables 14 15, and 16. We compute the likelihood of each option to select the answer option. Also, we adjust the prompts according to each model's training prompts. Specifically, we utilize the FastChat framework to form the prompt template. Please see github.com/lm- for more details.
Statie HHH Alignment
Human: Which response is more helpful, harmless, and honest from given context?
Context: [question]
Response A: [answer_a]
Response (answer_b)
Please choose either A B.
Assistant: Sure! The correct option would be Response
Figure 14: Prompt for evaluating Static HHH Alignment (Askell et al., 2021).
TruthfulQA
Human:
Interpret each question literally, and as a question about the real world. Carefully research whether the answer true or not from the given question, without falling prey to any common myths. Please choose true or false.
Question: [question]
Answer: [answer] | (true or false)
Assistant: Sure! The given answer is
Figure 15; Prompt for evaluating TruthfulQA (Lin al., 2021).
MMLU
Human:
The following multiple choice questions (with about (topic).
Please read the following question and choose the most proper choice from A, C, or D.
Question: {question)
(option_a)
joption_ b)
[option_e]
(option_d)
Assistant: Sure! correct answer choice would
Figure 16: Prompt for evaluating MMLU (Hendrycks et al., 2020).


Reviews:
[{"review_id": "MhVdxdbeqC", "reviewer": "Reviewer_NN7k", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[R1#1]**\n\n> 1. The key role-of-thumb hypothesis — responses from larger LLMs with more demonstrations might be better overall — alone does not deliver ideal synthetic feedback for effective RM learning. Without the Heuristic Filter post validation, the proposed RM even falls short of the simple lengthy baseline (Table 3).\n\nWe include the new intrinsic evaluation to probe our rule-of-thumb hypothesis. For this, we conduct a GPT-4 evaluation comparing the prompted generators in our configurations against Alpaca-7B, as we did in our paper. Note that we do not perform any post validation for this. We can observe that our rule-of-thumb hypothesis is still valid apart from using the heuristic filter in this evaluation. As shown in the table below, the overall generation qualities based on winning rate and relative score are matched with our rule-of-thumb.\n\n| vs. Alpaca-7B | Win | Tie | Loss | Relative Score |\n| --- | --- | --- | --- | --- |\n| LLaMA-7B-HHH-1shot | 12 | 15 | 133 | 54.1 |\n| LLaMA-7B-HHH-3shot | 12 | 14 | 134 | 59.9 |\n| LLaMA-13B-HHH-3shot | 17 | 17 | 126 | 64.8 |\n| LLaMA-30B-HHH-5shot | 39 | 21 | 100 | 80.3 |\n| LLaMA-30B-Faithful-3shot | 77 | 13 | 70 | 101.0 |\n- We additionally report the Relative Score, which is a relative overall score (sum of each 1-10 score for all questions) with Alpaca-7B.\n\nNevertheless, as you pointed out, we found the RM trained without the post validation significantly failed to choose a better response. We presume the reward modeling would be a sensitive process vulnerable to noisy signals. As we explained in L164, our assumptions are not always valid because of the stochastic nature of LLMs (as we conduct top-p=0.9 sampling for the response generation). Thus, we think that our proposed filtration method could support our hypothesis effectively removing such noises in our framework, making the reward modeling more robust as a result. We will include this new evaluation and add more explanation for the necessity of the post validation in the final version.\n\n**[R1#2]**\n\n> 2. The training of As-is RM requires 20k stackexchange ranking pairs. This seems to contradict the main goal of the paper, i.e., minimizing reliance on human annotated data.\n\nThe distinction between pretraining data (or web crawl data) and human-annotated data could be often unclear. We consider Stack Exchange to be pretraining data given that it is crawled from the raw web since comparison datasets (question, chosen answer, rejected answer) could be automatically constructed using the meta information naturally obtained by the real users, e.g., upvote or label for the accepted answer.  For instance, [1] also uses the Stack Exchange dataset as a pretraining dataset, along with Reddit comments and Wikipedia edits, for their reward modeling. Also, note that our reward model still performs competitively even without the As-is RM filtration, as shown in Table 3.\n\n[1] A General Language Assistant as a Laboratory for Alignment (Askell et al., 2021)\n\n**[R1#3]**\n\n> 3. Incorporating the ALMoST (RM) in the results of alignment benchmark in Table 1 is not suitable, as the goal is to evaluate different aligned LMs, instead of the reward model. When excluding the ALMoST (RM), the performance of RLSF aligned LM, i.e., the proposed ALMoST (PPO), is notably worse than Vicuna which doesn’t require comparison data to train.\n\nThank you for your suggestion. We intended to show the performance of our RM along with other generator models to emphasize that the adequate performances of our SFT and PPO models are derived from the well-performed RM. However, we agree that we could have made our presentation better to prevent misunderstanding and we will make Table 1 more clear as you suggested.\n\n**[R1#4]**\n\n> 1. On the first look, the term “pre-aligned” LLMs is confusing as readers may refer to it as vanilla LMs. Also, there are some inconsistencies in the usage of the term, e.g., in Line 391 and 425 it becomes “aligned” LLMs.\n\n> 2. Moving Line 125 ~ 127 into the introduction section could be beneficial for readers to understand the proposed framework more quickly with less confusion.\n\nThank you for your valuable suggestions. We will change the term \"pre-aligned\" LLMs to an unambiguous term. Also, we will reflect on your other suggestions in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Address the challenge of requiring LLM API or human annotated demonstrations to align Valinna LM.\n\n2. Thorough evaluation/ablation study of the RM.\n\n3. The paper is generally well-written with clear structures.", "reasons_to_reject": "1. The key role-of-thumb hypothesis — responses from larger LLMs with more demonstrations might be better overall — alone does not deliver ideal synthetic feedback for effective RM learning. Without the Heuristic Filter post validation, the proposed RM even falls short of the simple lengthy baseline (Table 3).\n\n2. The training of As-is RM requires 20k stackexchange ranking pairs. This seems to contradict the main goal of the paper, i.e., minimizing reliance on human annotated data.\n\n3. Incorporating the ALMoST (RM) in the results of alignment benchmark in Table 1 is not suitable, as the goal is to evaluate different aligned LMs, instead of the reward model. When excluding the ALMoST (RM), the performance of RLSF aligned LM, i.e., the proposed ALMoST (PPO), is notably worse than Vicuna which doesn’t require comparison data to train.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a novel Reinforcement Learning from Synthetic Feedback (RLSF) framework to align a vallina LM, LLaMA-7B, without any LLM API distillations or intensive human annotated demonstrations leveraging responses generated by LMs of different sizes/shots under the assumption that response from larger LMs with more shots might\nbe better overall.\n\nThe results suggest their resulting aligned LM — Aligned Language Model with Synthetic Training dataset (ALMoST) — achieves superior performances compared to other open-sourced aligned LMs in both alignment benchmarks and human evaluations.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. On the first look, the term “pre-aligned” LLMs is confusing as readers may refer to it as vanilla LMs. Also, there are some inconsistencies in the usage of the term, e.g., in Line 391 and 425 it becomes “aligned” LLMs.\n2. Moving Line 125 ~ 127 into the introduction section could be beneficial for readers to understand the proposed framework more quickly with less confusion."}}, {"review_id": "VgSWdpC8Ur", "reviewer": "Reviewer_Pu1N", "summary": "", "questions": "1. In the reward modeling process, you train a LLaMA-7B as the reward model with LLaMA-7B, LLaMA-13B and LLaMA-30B as feedback generators. Does it imply your method requires larger feedback generators than the reward model? Have you tried to train a reward model with feedback generators comparable to or smaller than it? Like LLaMA-30B as the reward model and LLaMA-7B/-13B/-30B as feedback generators.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[R2#1]**\n\n>  1. The separate ablation study of the assumptions they followed is missing. The rule of thumb they followed contains two parts: (1) feedbacks generated by larger models have better quality. (2) feedbacks generated given more demonstrations have better quality. While they do empirically verify the efficacy of jointly incorporating these two assumptions to generate synthetic feedback and train the reward model, the specific impact of each assumption on the reward model in isolation remains unclear. For example, how the reward model would behave when only trained on feedback generated given different numbers of demonstrations, but all by the same model?\n\nAs you suggested, we report additional intrinsic evaluations to investigate the impact of each assumption. To this end, we conduct a GPT-4 evaluation (on Vicuna Questions) to identify response quality from our prompted response generators compared to Alpaca-7B, as we originally did in our paper. We report the Win/Tie/Loss and Relative Score, the relative overall score (sum of each 1-10 score for all questions) over the questions compared to Alpaca-7B. We isolate the three factors,(1) model size, (2) the number of demonstrations, and (3) the quality of demonstrations, to analyze our rule-of-thumb comprehensively. \n\nFirst, we can see that the model size significantly contributes to the quality of response generation. The winning rate against Alpaca-7B increases monotonically as we increase the model size. The gap between 13B and 30B is especially large.\n\n| vs. Alpaca-7B | Win | Tie | Loss | Relative Score |\n| --- | --- | --- | --- | --- |\n| LLaMA-7B-Faithful-3shot | 52 | 14 | 94 | 82.7 |\n| LLaMA-13B-Faithful-3shot | 59 | 19 | 82 | 88.9 |\n| LLaMA-30B-Faithful-3shot | 77 | 13 | 70 | 101.0 |\n\nSecond, we find that the number of demonstrations also contributes to the response quality, but improvements by this factor are relatively small. The performance gain is saturated between 3 shots and 5 shots when we compare them with the relative score.\n\n| vs. Alpaca-7B | Win | Tie | Loss | Relative Score |\n| --- | --- | --- | --- | --- |\n| LLaMA-7B-HHH-1shot | 12 | 15 | 133 | 54.1 |\n| LLaMA-7B-HHH-3shot | 12 | 14 | 134 | 59.9 |\n| LLaMA-7B-HHH-5shot | 15 | 17 | 128 | 60.0 |\n\nFinally, we confirm the quality of demonstrations is the most important factor, as we reported in Table 4 of our paper. Surprisingly, we find that the smaller model with the well-designed prompt (LLaMA-7B-Faithful-3shot) outperforms the larger model with the normal prompt (LLaMA-30B-HHH-5shot).\n\n| vs. Alpaca-7B | Win | Tie | Loss | Relative Score |\n| --- | --- | --- | --- | --- |\n| LLaMA-30B-HHH-5shot | 39 | 21 | 100 | 80.3 |\n| LLaMA-30B-Faithful-3shot | 77 | 13 | 70 | 101.0 |\n\nWe appreciate your constructive suggestion and will surely include these analyses in the final version.\n\n**[R2#2]**\n\n> 2. The SFT and RLSF steps seem a little trivial as the reward model already achieves quite high scores in many aspects of human values according to Table 1.\n\nYes, we agree with you. Our main contribution stems mostly from the reward modeling stage (Step 1) since the remaining stages are not feasible without synthetic feedback from the first stage. Even without the remaining two stages, we could perform adequate inference with the reward model, e.g., best-of-N sampling with prompted baseline model (LLaMA-30B-Faithful-3shot). However, we think the remaining stages are also important to build an effective and efficient assistant model. First, the SFT stage makes the policy model standalone by eliminating dependency on the demonstrations. In other words, the model’s efficiency could be enhanced because of the reduced token consumption for the demonstrations in long sequences. Second, the RLSF stage also discards the dependency on the reward model at inference time. As a result, we conclude the SFT and RLSF stages are still needed to effectively transfer the reward model's ability into the policy model.\n\n**[R2#3]**\n\n> 1. In the reward modeling process, you train a LLaMA-7B as the reward model with LLaMA-7B, LLaMA-13B and LLaMA-30B as feedback generators. Does it imply your method requires larger feedback generators than the reward model? Have you tried to train a reward model with feedback generators comparable to or smaller than it? Like LLaMA-30B as the reward model and LLaMA-7B/-13B/-30B as feedback generators.\n\nNo, we do not think leveraging larger generators is an essential requirement for our reward modeling. As we showed in R2#1, the quality of demonstrations contributes to the response quality more than the model size. On the other hand, we think the various-sized large models could be used to improve overall synthetic feedback quality. We have not tried the 30B RM reward model training yet because of resource constraints, but we would like to investigate the generalizability according to model size in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Their method is novel and interesting. The rule of thumb they follow to do reward modeling sounds reasonable. Also, other enhancements they implement, such as post validation and RMSP integration in other segments of the pipeline, prove to be quite useful.\n2. The experiment results seem promising. It shows the models trained by their methods can surpass even larger models trained through genuine human feedback or other aligned models.", "reasons_to_reject": "1. The separate ablation study of the assumptions they followed is missing. The rule of thumb they followed contains two parts: (1) feedbacks generated by larger models have better quality. (2) feedbacks generated given more demonstrations have better quality. While they do empirically verify the efficacy of jointly incorporating these two assumptions to generate synthetic feedback and train the reward model, the specific impact of each assumption on the reward model in isolation remains unclear. For example, how the reward model would behave when only trained on feedback generated given different numbers of demonstrations, but all by the same model?\n2. The SFT and RLSF steps seem a little trivial as the reward model already achieves quite high scores in many aspects of human values according to Table 1.", "questions_to_authors": "1. In the reward modeling process, you train a LLaMA-7B as the reward model with LLaMA-7B, LLaMA-13B and LLaMA-30B as feedback generators. Does it imply your method requires larger feedback generators than the reward model? Have you tried to train a reward model with feedback generators comparable to or smaller than it? Like LLaMA-30B as the reward model and LLaMA-7B/-13B/-30B as feedback generators.", "paper_topic_and_main_contributions": "This paper proposes a new method to align large language models with human values using synthetic feedback. The authors follow a rule of thumb that the content generated by larger models with richer demonstrations would have better quality. Based on it, they design a pipeline to generate synthetic feedback with models of different sizes given different demonstration numbers, and then adopt the ranked preference modeling objective to train a reward model. Additionally, they propose a post validation method to guarantee the quality of the generated feedback in the reward modeling process. In the supervised fine-tuning stage, they adopt RM-guided Self-Play (RMSP), which includes the trained reward model trained in the data generation loop. In the experiment section, they demonstrate the effectiveness of their proposed method in aligning the model with human preference. Their models outperform the models trained by aligned language models or human feedback. They also conduct a series of ablation studies to validate the post validation and RMSP they designed.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cUL3BRdIz6", "reviewer": "Reviewer_Zsnr", "summary": "", "questions": "Question A: Relevant to the first item above, it is unclear to me why `LLaMA-30B-Faithful-3shot` is better than `LLaMA-30B-HHH-5shot`. \n\nQuestion B: see the third item above ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[R3#1]**\n\n> The core assumption that the response from a larger LLM with more and better demonstrations might be better overall is not conprehensively studied. Although the performance of the trained reward model can be viewed as a indirect evidence supporting the assumption, it would still be benefitial to understand when and how the assumption may not hold through a systematic evaluation on the output of various LLM configurations used.\n\n> Question A: Relevant to the first item above, it is unclear to me why LLaMA-30B-Faithful-3shot is better than LLaMA-30B-HHH-5shot.\n\nFollowing your valuable suggestion, we attach new intrinsic evaluations to study our core assumption (rule-of-thumb). Specifically, we conduct a GPT-4 evaluation comparing our prompted generators with Alpaca-7B as we did in our paper. The table below shows that our assumption is valid in all the three factors we claimed (model size, the number of demonstrations, and the quality of demonstrations).\n\n| vs. Alpaca-7B | Win | Tie | Loss | Relative Score |\n| --- | --- | --- | --- | --- |\n| LLaMA-7B-HHH-1shot | 12 | 15 | 133 | 54.1 |\n| LLaMA-7B-HHH-3shot | 12 | 14 | 134 | 59.9 |\n| LLaMA-7B-HHH-5shot | 15 | 17 | 128 | 60.0 |\n| LLaMA-13B-HHH-3shot | 17 | 17 | 126 | 64.8 |\n| LLaMA-30B-HHH-5shot | 39 | 21 | 100 | 80.3 |\n| LLaMA-30B-Faithful-3shot | 77 | 13 | 70 | 101.0 |\n- We additionally report the Relative Score, which is a relative overall score (sum of each 1-10 score for all questions) with Alpaca-7B.\n\nMoreover, we can observe that the model size and the quality of demonstrations significantly contribute to the final response quality (LLaMA-13B-HHH-3shot << LLaMA-30B-HHH-5shot & LLaMA-30B-HHH-5shot << LLaMA-30B-Faithful-3shot), while the number of demonstrations makes a relatively small contribution, and its effect is saturated between 3 and 5 shots. Please see also R2#1 for more details. We will include this study in our final version.\n\n**[R3#2]**\n\n> From Table 3, the heuristic filter seems to play an important role in the synthetis data generation process. While sub-optimal, the filter requires a fair amount of expert knowledge and empirical experience to design and improve, which conflict with the motivation of reducing human effort.\n\nWe agree that the heuristic filter is an essential pipeline in our method. However, we did not do heavy engineering for the heuristic filter to improve the performance, as shown in Figure 6. For example, we only used a few keyword-based rules as a minimal effort of qualification (If the generated response does not end with '.', If the generated response starts with 'well', If the generated response contains \"I don’t know\", 'image', or special-purposed prefix.). Also, we believe another assessment checking the confidence interval of response lengths is a very intuitive process that does not require expert knowledge.\n\n**[R3#3]**\n\n> It is unclear how to use the apporach to align larger models. For example, as the supervised fine-tuning data used to train the 7B model in the paper is produced by a well-prompted 30B model, what would be the teacher model to produce data for aligning a 30B model?\n\nWe believe the 30B model could also be aligned with our synthetic dataset generated by the 30B model (even smaller models). In Table 4 (RMSP vs. Self-play), we found that the ability to discriminate between better or worse responses contributes to well-aligned performance. Moreover, as we see in R3#1, the quality of demonstrations (Faithful >> HHH) is a more important factor than the model size in producing high-quality responses. In other words, we can effectively contrast the good and bad responses with various combinations of demonstrations to obtain synthetic feedback.\n\nIn addition to our speculation, we additionally conduct a new experiment to train a larger SFT model with the same synthetic dataset used in our paper. Specifically, we train LLaMA-30B with LoRA-tuning [1] on the same demonstration dataset used in our paper. We use the same configurations of 7B training except for the global batch size, which is set to 64 on 8 * A100 GPUs. Also, we set additional configurations (lora_r=8, lora_alpha=16, lora_dropout=0.1, target_modules=['k', 'v']) for the LoRA-tuning. We report a GPT-4 evaluation result against Alpaca-7B.\n\n| vs. Alpaca-7B | Win | Tie | Loss | Relative Score |\n| --- | --- | --- | --- | --- |\n| ALMoST-7B-SFT (Full Fine-tuning) | 87 | 17 | 56 | 104.5 |\n| ALMoST-30B-SFT (LoRA) | 109 | 20 | 31 | 112.8 |\n\nAs shown in the above table, our synthetic demonstrations dataset could be used to align the 30B SFT model successfully. While this result might be sub-optimal since it is based on LoRA-tuning instead of full fine-tuning, we think fine-grained prompt design per task type instead of the static 3-shot of demonstrations will make the synthetic dataset more powerful. We remain the investigation for future work.\n\n[1] LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Aligning LLMs with Human values is one of the most important research directions in the field of LLM research. Different from previous work either distills from pre-aligned LLM outputs or learns from additionally collect human feedback data, the method proposed by this paper does not require any of these external supervisions, which significantly reduce the cost of alignment learning. I believe the idea would inspire more research along the same line in the future. \n- The full recipe of how to use synthetic feedback to train a better aligned model is beneficial to the community. ", "reasons_to_reject": "- The core assumption that the response from a larger LLM with more and better demonstrations might be better overall is not conprehensively studied. Although the performance of the trained reward model can be viewed as a indirect evidence supporting the assumption, it would still be benefitial to understand when and how the assumption may not hold through a systematic evaluation on the output of various LLM configurations used. \n- From Table 3, the heuristic filter seems to play an important role in the synthetis data generation process. While sub-optimal, the filter requires a fair amount of expert knowledge and empirical experience to design and improve, which conflict with the motivation of reducing human effort. \n- It is unclear how to use the apporach to align larger models. For example, as the supervised fine-tuning data used to train the 7B model in the paper is produced by a well-prompted 30B model, what would be the teacher model to produce data for aligning a 30B model? ", "questions_to_authors": "Question A: Relevant to the first item above, it is unclear to me why `LLaMA-30B-Faithful-3shot` is better than `LLaMA-30B-HHH-5shot`. \n\nQuestion B: see the third item above ", "paper_topic_and_main_contributions": "This paper proposes a new method for improving the alignment of large language models, which is the process of ensuring that the model's outputs are consistent with human expectations. It presents a novel alignment learning framework that uses a synthetic preference ranking dataset to train a reward model, and leverages the feedback from that reward model to do both (reward model guided) supervised fine-tuning and reinforment learning. The resulted model, ALMoST, outperforms several recent LLMs using external human or pre-aligned LLM supervisions (Alpaca, Dolly v2, OpenAssistant) on alignment benchmarks (Static HHH, TruthfulQA) and human evaluations.\n\nThe key contribution is a simple yet intuitive way to construct the synthetic preference ranking dataset, without relying on external supervions such as human feedback or responses from pre-aligned models. Based on the assumption that larger and better prompted models are likely to produce responses with higher quality for the same query, ranking labels can be automatically obtained from synthetic responses from different models with controlled sizes (e.g. larger > smaller), in-context learning examples (more > less), and prompt quality (detailed > concise). Experimental results show that (1) the LLM trained using the feedback from the proposed reward model outperforms competitive baselines using external human or pre-aligned LLM supervisions; (2) the reward model trained on the synthetic dataset is comparable with the one trained on human preference data (HH-RLHF). \n\nAnother contribution includes the exploration of using the reward model to generate supervised fine-tuning data of better quality. A new method is proposed, namely reward-model guided self play (RMSP), where the best based model is used to generate synthetic instruction following data and the reward model is used to filter out low-preference responses. Experimental results show that the model trained using the filtered data outperforms its counterpart without using the reward model. \n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a novel alignment learning framework that relies on synthetic feedback rather than extensive human annotations or pre-aligned large language models. By utilizing reward modeling with synthetic feedback and reinforcement learning, the proposed method shows improved performance in alignment benchmarks compared to recent open-sourced models. The reviewers are generally positive towards the contribution of this paper. I therefore would recommend acceptance of this paper to the main conference.
Decision: Accept-Main