Venue: EMNLP 2023

# Aligning Large Language Models through Synthetic Feedback

 Sungdong Kim\({}^{1,2,3}\) Sanghwan Bae\({}^{1}\) Jamin Shin\({}^{1,2}\)

Soyoung Kang\({}^{1}\) Donghyun Kwak\({}^{1}\) Kang Min Yoo\({}^{1,2,4}\) Minjoon Seo\({}^{3}\)

NAVER Cloud\({}^{1}\) NAVER AI Lab\({}^{2}\) KAIST AI\({}^{3}\) SNU AI Center\({}^{4}\)

{sungdong.kim, sanghwan.bae, jamin.shin}@navercorp.com

{soyoung.kang, donghyun.kwak, kangmin.yoo}@navercorp.com

minjoonekaist.ac.kr

###### Abstract

Aligning large language models (LLMs) to human values has become increasingly important as it enables sophisticated steering of LLMs. However, it requires significant human demonstrations and feedback or distillation from proprietary LLMs such as ChatGPT. In this work, we propose a novel alignment learning framework with _synthetic_ feedback not dependent on extensive human annotations and proprietary LLMs. First, we perform reward modeling (RM) with synthetic feedback by contrasting responses from vanilla LLMs with various sizes and prompts. Then, we use the RM to simulate high-quality demonstrations to train a supervised policy and further optimize the model with reinforcement learning. Our resulting model, Aligned **L**anguage **M**odel with **S**ynthetic **T**raining dataset (ALMoST), outperforms recent open-sourced models, which are trained on the outputs of InstuctGPT or human-annotated demonstrations, in alignment benchmarks. In human evaluation, our model is preferred to Alpaca and Dolly-v2, 55.0% and 58.5% of the time, respectively. Further analyses demonstrate the efficacy and importance of synthetic feedback in our framework 1.

Footnote 1: The code is available at github.com/naver-ai/almost.

+
Footnote †: November 6, 2021

## 1 Introduction

Alignment learning has been an essential learning scheme to align the behaviors of large language models (LLMs) with human values like safety and truthfulness while following the intention of users accurately Ouyang et al. (2022). Vanilla LLMs - those not aligned yet - could misunderstand user intentions or produce unsafe and inaccurate responses. Desirable human values such as helpfulness, harmlessness, or honesty can be defined, and human demonstrations with these values are then used for the alignment learning Askell et al. (2021); Bai et al. (2022a).

Typically, alignment learning consists of three stages: supervised fine-tuning (SFT), reward modeling (RM), and reinforcement learning from human feedback (RLHF) Ouyang et al. (2022); Bai et al. (2022a).

However, the three-stage training recipe requires significant human effort, especially in the first two stages. More specifically, both the SFT and RM training stages must be provided with an abundance of high-quality human demonstrations and ranking datasets for obtaining models to facilitate RLHF. For instance, Ouyang et al. (2022) prepare and utilize 13k human demonstrations and 33k comparisons.

On the other hand, Self-Instruct Wang et al. (2022)

Figure 1: A procedure of reward modeling through synthetic feedback. We assume that the response from a larger LLM with more and better demonstrations might be better overall. We train a reward model with synthetic comparisons generated top on the assumption.



2022) attempts to generate synthetic self-generated instruction datasets using in-context learning with a few seed demonstrations. Meanwhile, the release of LLaMA (Touvron et al., 2023) brings upon many open-sourced aligned LLMs trained on the outputs of proprietary LLMs or human-annotated instructions. However, it still heavily depends on proprietary LLM APIs such as InstructGPT and Chat GPT (Ouyang et al., 2022; OpenAI, 2023; Taori et al., 2023; Chiang et al., 2023) or intensive human annotations (DataBricks, 2023; Kopf et al., 2023).

In this paper, we introduce a novel framework for alignment learning that only requires minimal human labor and is not dependent on proprietary LLMs. Unlike the conventional alignment learning procedure that collects demonstration first (Ouyang et al., 2022), we first develop a reward model (RM) on a synthetic comparison dataset constructed by contrasting outputs from vanilla LLMs in various configurations, as shown in Figure 1. The rules of generating these synthetic ranking data originate from our hypothesis that the responses generated by larger, optimally prompted models are superior to those produced by smaller, inadequately prompted models, as reported by previous work (Askell et al., 2021). Then, we introduce a Reward-Model-guided Self-Play (RMSP) to simulate high-quality demonstrations with rejection sampling using the RM (Ouyang et al., 2022). We train LLaMA-7B (Touvron et al., 2023) on the synthetic demonstrations (SFT) and further optimize the model with rewards from the synthetic RM, namely, Reinforcement Learning from Synthetic Feedback (RLSF).

Our Aligned **L**anguage **M**odel with **S**ynthetic Training dataset (ALMoST) outperforms Alpaca (Taori et al., 2023) - distilled from InstructGPT (Ouyang et al., 2022) - and Dollyv-2 (DataBricks, 2023) and OpenAssistant (Kopf et al., 2023) that are trained on human-annotated demonstrations in the alignment-related benchmarks (Askell et al., 2021; Lin et al., 2021; Chiang et al., 2023). Notably, our model is preferred to recent open-sourced models, Alpaca and Dolly-v2, \(55\text{--}58\%\) of the time (winning rate) in human evaluation without distillation from proprietary LLMs nor intensive human annotations. We speculate the strong performance of our model is due to the empirical indicators of well-aligned behaviors that have been effectively incorporated into a strong backbone model through synthetic feedback, allowing the inherent capability to self-align to elicit and partially replace the need for human feedback.

Our main contributions are three folds:

* We propose a novel alignment learning framework by introducing synthetic feedback. It automatically constructs high-quality comparisons and demonstrations without relying on human feedback and proprietary LLMs.
* Our resulting model, ALMoST, shows well-aligned behaviors with human values in alignment benchmarks. In the human study, ALMoST is preferred to Alpaca and Dolly-v2, showing a \(55\text{--}58\%\) winning rate.
* Analysis on RM further demonstrates the efficacy of synthetic feedback and highlights the importance of injecting empirical priors, e.g., our proposed filtering method and faithful prompt design.

## 2 Method

In this section, we will describe detailed procedures of our framework as depicted in Figure 2.

### Step 1: Reward Modeling with Synthetic Feedback

Prompted BaselineAs we do not have aligned baselines available for the comparisons yet, we utilize HHH (Helpful, Harmless, and Honest) prompt devised by Askell et al. (2021). It contains 14 human-written conversations for guiding LLM alignment 2. We employ The HHH prompted LLaMA models to generate synthetic comparisons (Touvron et al., 2023).

Footnote 2: gist.github.com/jareddk/2509330...

Generating Synthetic ComparisonInstead of collecting human feedback, we rather generate synthetic comparisons based on naive assumptions according to empirical observations. Askell et al. (2021) demonstrate that the larger model performs somewhat better than the smaller model, and the model with longer prompts is better than the model with shorter prompts in terms of human preference. In short, we assume the quality of the response follows the rule of thumb:

* Larger model \(>\) Smaller model
* More few-shots \(>\) Less few-shots
* Better demonstration \(>\) Worse demonstration 

For the same input \(x\), we first sample the responses \(Y=\{y_{1},y_{2},...,y_{|Y|}\}\) from the models with various configurations. Then, we apply the rule to choose the better response among the generated responses. More specifically, we involve \(\{7,13,30\}\)B LLMs with \(\{1,3,5\}\) shots of HHH demonstrations for the comparison. As illustrated in Figure 1, if we sample responses from (1) \(30\)B with 5 shots, (2) \(13\)B with 3 shots, and (3) \(7\)B with 1 shot, the ranking becomes \(y_{1}>y_{2}>y_{3}\) according to our rule of thumb. Then, we can get a set of binarized comparisons, e.g., \(\{(y_{1},y_{2}),(y_{2},y_{3}),(y_{1},y_{3})\}\). We denote the former as a 'chosen' response (\(y_{1}\)) and the latter as a 'rejected' response (\(y_{2}\)) in a comparison pair \((y_{1},y_{2})\).

Post ValidationOur assumptions are often wrong because of the stochastic nature of the prompt-based generation. These noises in the dataset make the reward modeling unstable and divergent in the end. Thus, we come up with post validation method to filter out such noises.

First, we devise **Heuristic Filter (HF)** based on prior knowledge. It discards bad responses containing or beginning with keywords such as "I don't know" or "well". Also, we empirically find that the better response usually has a longer length than the worse one. Especially if the response is short, it often tends to be a case of probabilistic generation failure. However, training RM only on comparisons with longer chosen responses would make the resulting model biased by length. Thus, we apply HF to take comparison pairs whose chosen response is longer than either the rejected one or \(M-S/2\), where \(M\) is the mean, and \(S\) is the standard deviation of the lengths of \(Y\) in the character level. This length constraint reduces the probability that short-generation would be stochastic generation failure by checking whether the length of each response is in the confidence interval. Furthermore, it does not fall into the length bias. Please see Appendix B for detailed examples. We will demonstrate the benefits in Section 4.2.

Second, we leverage **As-is RM** for further data filtering. Specifically, we train another RM with a community QA dataset such as StackExchange [6, 8]. Our preliminary study does not find the benefits of large-scale pre-training for RM discussed in Askell et al. (2021). Thus, we sample 20k pairs from the pre-processed StackExchange dataset for our training 3. We keep the resulting synthetic comparisons only when the As-is RM agrees with the decision.

Footnote 3: huggingface.co/datasets/lvwerra/stack-exchange-paired

Reward ModelingFinally, we train the reward model based on the synthetic comparisons described above. We follow tha

Figure 2: Overview of our proposed framework for alignment learning of LLMs. Step 1. We first conduct reward modeling with a synthetically generated comparison dataset (_synthetic feedback_). Step 2. The demonstration dataset is generated by simulation with the guidance of the reward model and train supervised policy with the synthetic demonstrations. Step 3. We further optimize the model against the reward model with reinforcement learning.

 

modeling objective from previous works (Askell et al., 2021; Ouyang et al., 2022). The objective is to make the reward model \(r_{\theta}\) assign a scalar value for the overall quality of a response \(y_{j}\) for a given query \(x\) comparative to its counterpart baseline response \(y_{k}\). The loss function is defined as follows:

\[J(\theta)=-E_{(x,y_{j},y_{k})\sim D}log(\sigma(r_{\theta}(x,y_{j})-r_{\theta}( x,y_{k})))\]

where \(D\) is the training set of synthetic comparisons and \(r_{\theta}(x,y)\) is the reward model's scalar output indicating the overall response quality \(y\) for its input \(x\).

Implementation DetailsWe start by generating initial queries, i.e., a diverse set of input \(x\). In particular, we adopt the recipe of Self-Instruct (Wang et al., 2022) to generate 10k initial queries based on few-shot in-context learning. More details of the query generation are in Appendix C.

For response generation, we include five prompted models with the below configurations.

* A. LLaMA-30B-Faithful-3shot
* B. LLaMA-30B-HHH-5shot
* C. LLaMA-13B-HHH-3shot
* D. LLaMA-7B-HHH-3shot
* E. LLaMA-7B-HHH-1shot

For each query, we generate five responses from the models and take rankings, \(y_{A}>y_{B}>y_{C}>y_{D}>y_{E}\), reflecting the rule of thumb. The Faithl indicates our manually designed prompts consisting of three conversations responding more faithfully and longer while considering the response format, and the HHH indicates the prompts written by Askell et al. (2021). The detailed examples are in Appendix A. Finally, we produce 13k binarized synthetic comparisons after post-validation (HF and As-is RM) and train a reward model with the synthetic comparisons.

### Step 2: Supervised Fine-Tuning

In the second step, we propose a Reward-Model-guided Self-Play (RMSP) to simulate high-quality demonstrations, i.e., conversations between the user and AI assistant. The simulated demonstrations are used to supervised fine-tuning for the initially aligned policy model (SFT).

Self-PlayThe basic simulation is enabled by turn-taking between the user and assistant role models with corresponding prompts, i.e., self-play. We continue to use the same prompted baseline, LLaMA-30B-Faithful-3shot, for the assistant role. In addition, we've made minor adjustments to the original HHH prompt (Askell et al., 2021) to suit the user's role better, LLaMA-30B-User-3shot4. Starting from the initial queries generated in the first stage, the LLaMA-30B-Faithful-3shot generates responses for the queries. Then, the LLaMA-30B-User-3shot follows up the assistant's response. The turn-taking is continued until the maximum turn \(T\).

Footnote 4: Please see Appendix A for the details of prompts.

RM-guided Self-Play (RMSP)To ensure a more aligned response from the assistant, we suggest including the synthetic RM, trained in the first stage, in the loop, namely Reward-Model-guided Self-Play (RMSP). In this setup, the assistant model, LLaMA-30B-Faithful-3shot, first samples \(N\) responses for a given conversational context. Then, the RM scores the \(N\) responses, and the best-scored response is chosen as the final response for the simulation, i.e., the RM performs rejection sampling (best-of-\(N\) sampling) (Ouyang et al., 2022; Scheurer et al., 2023). Like the Self-Play, the turn-taking with LLaMA-30B-User-3shot is continued until the maximum turn. Please see Figure 10 for the examples.

Implementation DetailsWe generate about 20k high-quality demonstrations using RMSP. We set the maximum turn to 2 for simplicity, focusing on the single-turn scenario. The number of rejection sampling \(N\) is set to 4 considering resource constraints 5. Then, we train LLaMA-7B on the generated demonstrations, i.e., a supervised policy fine-tuning (SFT). More training details are in Appendix D.

Footnote 5: More details of the synthetic datasets are in Appendix C.

### Step 3: Reinforcement Learning from Synthetic Feedback (RLSF)

In the last stage, we perform reinforcement learning from synthetic feedback (RLSF) to further align the SFT model using a reward signal from the synthetic RM. Following previous works (Ouyang et al., 2022; Bai et al., 2022), we use Proximal Policy Optimization (PPO) (Schulman et al., 2017). During this stage, a policy \(\pi_{\phi}\) autoregressively gen 

erates a response \(y\) given a prompt \(x\). Subsequently, a reward score \(r_{\theta}(x,y)\) is determined by the reward model \(r_{\theta}\). The training objective is to maximize the expected reward.

\[E_{x\sim D,y\sim\pi_{\phi}(|x)}[r_{\theta}(x,y)]\]

Stiennon et al. (2020) proposes that adding an estimated KL penalty term between the initial policy \(\rho\) and the policy \(\pi_{\phi}\) to \(r_{\theta}(x,y)\) can enhance performance. This adjustment leads to the final objective as follows:

\[E_{x\sim D,y\sim\pi_{\phi}(|x)}[r_{\theta}(x,y)-\lambda\log\left(\frac{\pi_{ \phi}(y|x)}{\rho(y|x)}\right)],\]

where \(\lambda\) is a KL coefficient.

Implementation DetailsWe initialize the policy \(\rho\) with the SFT-tuned LLaMA-7B from Step 2. Also, the prompts for the PPO training are compiled by extracting only the inputs (initial queries) from the demonstration dataset generated by RMSD described in Section 2.2. More details for PPO training are in Appendix D.

## 3 Evaluating Alignment of ALMoST

We validate our resulting model, **A**ligned **L**anguage **M**odel with **S**ynthetic **T**raining dataset (ALMoST), in three alignment benchmarks, Static HHH evaluation Askell et al. (2021), TruthfulQA Lin et al. (2021), and Vicuna Questions Chiang et al. (2023).

### Dataset

Static HHH alignment and TruthfulQASkel et al. (2021) introduce Static HHH alignment benchmark to measure how models are aligned well with the human values 6. Similar to the comparison dataset, the model should choose a more proper response for input between the two options based on human values. The dataset consists of three human value categories, helpful, harmless, and honest, and contains a misc (others) category. We include the dataset to get relationships of tension among the human values from the evaluation, although the entire dataset is just 221.Lin et al. (2021) propose TruthfulQA to measure how LLM generates truthful answers for a given question 7. It especially contains 817 adversarial questions to elicit imitative falsehood answers from LLMs. For simplicity, we evaluate the models with a multiple-choice setup (MC1) instead of a generative setup. Note that all the evaluation is based on zero-shot, which means we do not fine-tune the target dataset. Please see Appendix I for more details of the evaluation prompt.

Footnote 6: github.com/google/BIG-bench

Footnote 7: github.com/sylinrl/TruthfulQA

Vicuna QuestionesWe test our models using Vicuna evaluation questions Chiang et al. (2023). It consists of 80 questions on various topics spanning general QA, writing, reasoning, etc., to identify user preference 8. First, t

\begin{table}
\begin{tabular}{p{54.1pt} p{99.6pt} c c c c c|c} \hline \hline  & & & \multicolumn{4}{c}{Static HHH Alignment} & \multicolumn{2}{c}{TruthfulQA} \\ \cline{4-9} Model & Backbone & Dataset by & Helpful & Harmless & Honest & Other & All & MC1 \\ \hline Dolly-v2 & Pythia-\(12\)B & Human & 67.8 & 46.6 & 50.7 & 62.8 & 56.6 & 15.2 \\ Oasst-v4 & Pythia-\(12\)B & Human & 59.3 & 56.9 & 47.5 & 69.8 & 57.5 & 23.3 \\ Vicuna & LLaMA-\(13\)B & ChatGPT & **78.0** & **89.7** & **70.5** & **81.4** & **79.6** & **63.3** \\ \hline Dolly-v2 & Pythia-7B & Human & 69.5 & 41.4 & 45.9 & 51.2 & 52.0 & 24.2 \\ Alpaca & LLaMA-7B & InstructGPT & 71.2 & 53.4 & 62.3 & 65.1 & 62.9 & 19.5 \\ Vicuna & LLaMA-7B & ChatGPT & 79.7 & **72.4** & **70.5** & **76.7** & **74.7** & **52.5** \\ ALMoST (SFt) & LLaMA-7B & LLaMA & 79.7 & 56.9 & 65.6 & 69.8 & 67.8 & 31.5 \\ ALMoST (PPO) & LLaMA-7B & LLaMA & **81.4** & 60.3 & 62.3 & 72.1 & 68.8 & 38.0 \\ \hline ALMoST (RM) & LLaMA-7B & LLaMA & 74.6 & 67.2 & 78.7 & 86.0 & 76.0 & 54.8 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Evaluation results of Static HHH alignment and TruthfulQA (Multiple-Choice)Askell et al. (2021); Lin et al. (2021). We report accuracy for both datasets. Our ALMoST’s outperform recent open-sourced models, Alpaca, Dolly, and OpenAssistant Taori et al. (2023); DataBricks (2023); Köpf et al. (2023), trained on the outputs of InstructGPT or human demonstrations. Also, our RM shows a good performance in identifying proper responses aligned with human values, surpassing Vicuna trained on outputs of ChatGPT Chiang et al. (2023). Notably, our models only leverage synthetic datasets while not relying on the pre-aligned LLMs or extensive human annotations.

 

erate an answer for each same question. Then, we conduct human A/B tests to choose a preferred answer between answers from the two models. In particular, we recruit three workers for each test. We ask the workers to choose the more helpful, harmless, and honest answer with careful consideration over the contents in the answers (Askell et al., 2021). Please see Appendix E.1 for more details of the human evaluation. In addition, we conduct an automatic evaluation using GPT-4. In the test, GPT-4 assesses the two answers by giving a 1-10 scalar score for the corresponding answer and providing appropriate explanations for the judgment (OpenAI, 2023). Even though it is not a rigorous evaluation, we can compare the overall responding qualities of the models with reasonable costs. Considering the positional bias of the GPT-4 assessment 9, we evaluate the same instance twice by reversing the order of the two answers.

Footnote 9: This repo notices that the GPT-4 evaluation has a strong positional bias in favor of the first response.

### Baselines

We include recent open-sourced models to compare the aligned behaviors of the LLMs according to their backbone model and training dataset. **Alpaca** is the first open-sourced instruction-following model based on LLaMA (Touvron et al., 2023; Taori et al., 2023). It is trained on the 52k synthetic instructions dataset generated by the proprietary LLM, InstructGPT (Ouyang et al., 2022). Similarly, **Vicuna** is trained on 70k ShareGPT dataset, which is the shared chatting logs between users with the ChatGPT, one of the powerfully aligned models (OpenAI, 2023; Chiang et al., 2023). **Dolly-v2** is another open-sourced model trained on a 15k human-annotated instructions dataset (DataBricks, 2023). It is based on Pythia, another open-source LLM (Biderman et al., 2023). **OpenAssistant (Oasst)** is an open-source project to build aligned LLM based on participants of the web community (Kopf et al., 2023). It also releases an Oasst model (SFT) trained on the human-annotated dataset 10.

Footnote 10: OpenAssistant/oast-sft-4-pythia-12b-epoch-3.5

### Evaluation Results

Static HHH alignment and TruthfulQAOur models outperform Alpaca, Dolly-v2, and OpenAssistant without any distillation from the proprietary LLMs or intensive human annotations, as shown in Table 1. For all sizes, our ALMoSTs show consistently better accuracy in all HHH splits and TruthfulQA except for Vicuna trained on ChatGPT's outputs (Chiang et al., 2023; OpenAI, 2023). However, our RM shows excellent performance in choosing appropriate responses according to human values, even beating the Vicuna-7B. It is the consistent observation with Askell et al. (2021). Moreover, it is notable our ALMoST-PPO achieves the highest accuracy in the Helpful split of HHH evaluation, even including 13 billion models. When comparing our SFT and PPO-trained models, the PPO model improves helpfulness, harmlessness, and truthfulness while sacrificing honesty. Honesty and truthfulness look similar, but they are slightly different. The honesty is related to expressing uncertainty, while the truthfulness mainly measures how robust the model is against adversarial falsehood.

Human Evaluation on Vicuna QuestionsWe confirm our models' strengths in actual human preferences in human evaluation 11. In Figure 2(a),

Figure 3: (a) Human evaluation and (b) GPT-4 evaluation results within 7B models on Vicuna Questions (Chiang et al., 2023). It shows the percentage of win, tie, and loss of ALMoST-PPO against other models.

 

we find that humans also more favorably assess our ALMoST model than Dolly-v2 and Alpaca (DataBricks, 2023; Taori et al., 2023), showing \(58.8\)% and \(55.0\)% of winning rate, respectively. Also, ALMoST-PPO improves the preference of ALMoST-SFT with a higher winning rate (\(37.5\)%), while they also show the highest tie rate (\(36.3\)%). It indicates the efficacy of our RLSF training. Moreover, our model is assessed as competitive with Vicuna, showing \(25\)% of the winning rate and \(25\)% of the tie rate, even without dependency on the powerful proprietary model, ChatGPT. However, there still remains significant room for improvement between our ALMoST and Vicuna. We also include qualitative examples of the evaluation in Appendix H.

**GPT-4 Evaluation on Vicuna Questions** In Figure 1(b), we can observe a similar tendency of GPT-4 evaluation with the human evaluation. The ALMoST-PPO consistently shows a higher winning rate against Dolly-v2, Alpaca, and ALMoST-SFT. However, we find that the GPT-4 is not likely to give the same scores for the answers showing a generally lower tie rate than the human evaluation. Moreover, GPT-4 assesses Vicuna's responses more favorably than humans did. Nevertheless, we can obtain the overall gaps among the models with reasonable cost from the GPT-4 evaluation. When we extend the evaluation to various-sized models, our \(7\)B model outperforms \(12\)-\(13\)B baselines, including Alpaca-\(13\)B, Oasst-\(12\)B, and Dolly-v2-\(12\)B in the evaluation as shown in Figure 8.

## 4 Analysis

### Probing Main Hypothesis

We further inspect our assumptions for the synthetic feedback. Specifically, we would like to know how each assumption, (1) model size, (2) the number of demonstrations, and (3) the quality of demonstrations, contributes to the final quality of the sampled responses. For this, we conduct GPT-4 evaluations on Vicuna Questions to compare our prompted response generators used for synthetic feedback generation in Section 2.1. We use Alpaca-7B (Taori et al., 2023) as a baseline model for the pair-wise comparisons. The results are shown in Table 2.

First, as expected, we can see that the model size significantly contributes to the response quality for both types of prompt (HHH and Faithful). The winning rate against Alpaca-7B increases monotonically as we increase the model size. The gap between \(13\)B and \(30\)B is especially large. Second, when comparing LLaMA-7B-HHH-{1,3,5}shot models, the number of demonstrations also improves the winning rate, but improvements by this factor are relatively small. Finally, we find that the demonstration's quality is the most important factor. Surprisingly, the smaller model with the well-designed prompt (LLaMA-7B-Faithful-3shot) outperforms the larger model with the normal prompt (LLaMA-30B-HHH-5Shot). Through this intrinsic evaluation, we can find our synthetic feedback dataset effectively covers responses of varying quality.

### RM Evaluation

\begin{table}
\begin{tabular}{l c c c} \hline \hline Configuration & W & T & L \\ \hline LLaMA-7B-HHH-1shot & 12 & 15 & 133 \\ LLaMa-7B-HHH-3shot & 12 & 14 & 134 \\ LLaMA-7B-HHH-5shot & 15 & 17 & 128 \\ LLaMA-13B-HHH-3shot & 17 & 17 & 126 \\ LLaMA-30B-HHH-5shot & 39 & 21 & 100 \\ \hline LLaMA-7B-Faithful-3shot & 52 & 14 & 94 \\ LLaMA-13B-Faithful-3shot & 59 & 19 & 82 \\ LLaMA-30B-Faithful-3shot & **77** & 13 & 70 \\ \hline \hline \end{tabular}
\end{table}
Table 2: GPT-4 evaluation results on Vicuna Questions (Chiang et al., 2023) of the prompted response generators in various configurations compared to Alpaacena-7B (Taori et al., 2023). It is based on the same evaluation setup in Section 3. The W, T, and L indicate # of wins, ties, and losses of each generator against Alpaca-7B.

\begin{table}
\begin{tabular}{l c c} \hline \hline Train Dataset & \# instance & Accuracy \\ \hline Random baseline & - & 50.0 \\ Lengthy baseline & - & 59.4 \\ \hline _Zero-shot_ & & \\ StackExchange & 25,057 & 63.7 \\ Synthetic Feedback & 13,687 & **65.2** \\ \hline _Full Fine-tuning_ & & \\ Helpful-base\({}^{*}\) & 11,738 & 65.2 \\ Helpful-base & 43,835 & 71.8 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Results of zero-shot reward modeling in the Helpful-base split of HH-RLHF (Bai et al., 2022). The lengthy baseline always chooses a longer response between the pairs. The \(*\) indicates the training data is a subset of the original, only including single-turn.

 

We further evaluate our RM on another comparison dataset, HH-RLHF Bai et al. (2022) to validate our synthetically generated comparisons (Synthetic Feedback). HH-RLHF contains various splits according to the development stage, e.g., the base set to build the initial policy or the online set collected with the deployed system. We focus on the 'Helpful-base' split, assuming we do not have a deployable system.

Reward ModelingIn Table 3, we find our RM trained with Synthetic Feedback achieves 90% performance of its upper-bound trained on the full training dataset. Also, it achieves the same accuracy with the result fine-tuned on the single-turn subset (Helpful-base*). Please note HH-RLHF includes multi-turn context, while our synthetic dataset focuses on single-turn scenarios.

Effect of Post ValidationWe conduct two types of post-validation to reduce noises in the synthetic comparisons described in Section 2.1. Table 4 shows that each filtering method contributes to the final reward model quality. Notably, we find the heuristic filter (HF) considering length distribution plays a crucial role in synthetic data generation. When we exclude the HF, the performance of RM drops about 10% point. Moreover, HF prevents the RM from falling into the length bias discussed in Section 2.1. The RM, trained on the dataset with HF, outperforms the lengthy baseline which always selects the longer response as the better response.

RMSP vs Self-PlayWe inspect the benefits of RM-guided Self-Play (RMSP) compared to its counterpart without RM guidance, i.e., Self-Play. Specifically, we compare two supervised policies (SFT) trained on demonstrations generated by RMSP or Self-Play. In Table 5, we find that the SFT model trained with RMSP outperforms the model with Self-Play in various benchmarks. In GPT-4 evaluation comparing Alpaca-7B, only the model with RMSP shows a winning rate higher than 50%. Moreover, we confirm the importance of designing good prompts. If we use HHH prompt instead of the Faith for the simulation, the performances for the alignment drop significantly. We include qualitative examples to compare the methods in Table 10.

## 5 Related Work

Aligning LLMs with Human valuesConditioning language models on human values Askell et al. (2021); Korbak et al. (2023); Liu et al. (2023) was found to improve models' capabilities of generating human-aligned text. Incorporating reward models Askell et al. (2021); Liu et al. (2022); Scheurer et al. (2023); Yuan et al. (2023) to tell how well the generated text reflects human values has enabled training better-aligned language models and served as a crucial ingredient for another effective methodology - reinforcement learning from human feedback (RLHF). RLHF have been widely investigated in recent days for aligning LLMs with the human values Christiano et al. (2017); Ziegler et al. (2020); Ouyang et al. (2022); Bai et al. (2022); Stiennon et al. (2022); Glaese et al. (2022). Recently, Zhou et al. (2023) claim the Superficial Alignment Hypothesis that most abilities of LLMs are learned in the pre-training stage, and fine-tuning on a few curated datasets can elicit the well-aligned behaviors from the models.

Distillation from proprietary LLMsRecent open-sourced models such as Alpaca follow the recipe of Self-Instruct Wang et al. (2022) to reduce the burdens of collecting human demonstrations Taori et al. (2023); Peng et al. (2023). However, it generates the synthetic instruction datasets using proprietary LLMs, e.g., InstructGPT or Chat-GPT Ouyang et al. (2022); OpenAI (2023), differ

\begin{table}
\begin{tabular}{l c} \hline \hline Train Dataset & Accuracy \\ \hline Random baseline & 50.0 \\ Lengthy baseline & 59.4 \\ \hline Synthetic Feedback & **65.2** \\ - As-is RM & 63.3 \\ - Heuristic Filter & 55.5 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Ablation results of post validation in the synthetic comparison generation. Helpful-base split is used for the RM evaluation Bai et al. (2022).

\begin{table}
\begin{tabular}{l l c c} \hline \hline Method & Prompt & Static HHH & \% Win \\ \hline RMSP & Faithful & **67.8** & **54.3** \\ Self-Play & Faithful & 66.0 & 40.0 \\ Self-Play & HHH & 61.3 & 15.0 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Comparison of synthetic demonstration generation methods with and without RM guidance, i.e., rejection sampling over the assistant’s response. The % Win indicates the winning rate against Alpaca-7b in the GPT-4 evaluation.

 

ent from Self-Instruct, which uses a vanilla LLM, GPT-3 (Brown et al., 2020). Similarly, Peng et al. (2023) try to distill GPT-4 outputs for the alignment. Vicuna is another open-sourced model trained on 70k ShareGPT datasets, which are publicly shared ChatGPT outputs by users (Chiang et al., 2023). On the other hand, Gudibande et al. (2023) point out the limitations of the distillation to train the aligned LLMs. Specifically, they show that scaling the number of the synthetic dataset does not improve the knowledge-related tasks and also human preferences, while scaling the model size contributes to the results. From the experiments, they warn that using synthetic datasets distills the teacher's style, not the knowledge.

Self-Alignment LearningAskell et al. (2021) introduce context distillation to get an initial policy with few-shot demonstrations manually devised by the authors. A student model, which is not prompted, is distilled from a teacher model prompted with the few-shot demonstrations. Self-Instruct is the approach that aligns LLMs with self-generated instruction datasets (Wang et al., 2022). To this end, Wang et al. (2022) manually devise 175 seed tasks and conduct automatic instruction dataset via in-context learning and filtering. We develop the methods by including reward modeling with synthetic feedback. Dromedary is a concurrent work that has a similar motivation to ours, i.e., alignment learning with minimal human efforts (Sun et al., 2023). They devise a few human-written "principles" for LLMs to follow, and the LLMs generate aligned responses with the guidance of the principles via in-context learning, similar to Constitutional AI (Bai et al., 2022b). Specifically, it requires about 200 human annotations, 195 seed prompts, 16 principles, and 5 exemplars for the alignment, while our framework requires only 18 human annotations, 10 seed prompts for query mining, and 8 demonstrations.

## 6 Conclusion

In this work, we propose a novel framework for aligning LLM with human values by introducing synthetic feedback. We identify better responses from vanilla LLMs with various sizes and prompts, relying on empirical prior knowledge. We first train a reward model with synthetically generated comparisons. Then, we produce another synthetic dataset to train aligned policies using the reward model. Experimental results demonstrate the efficacy of our framework showing outstanding performances in the alignment benchmarks. We believe the strong performance of our model is derived from the effective incorporation of empirical indicators of well-aligned behaviors through synthetic feedback. Furthermore, our method is cost-effective in that it does not require extensive human demonstrations and not depend on the proprietary LLMs.

### Limitations

Even though we show our framework works well on many alignment-related benchmarks (Askell et al., 2021; Lin et al., 2021; Chiang et al., 2023), our evaluations fall short of identifying other aspects of the resulting aligned models. For example, Gudibande et al. (2023) explain the limitations of synthetic imitation datasets from the pre-aligned LLMs by involving knowledge-related benchmarks like MMLU, HumanEval, and Natural Questions (Hendrycks et al., 2020; Chen et al., 2021; Kwiatkowski et al., 2019). Askell et al. (2021); Bai et al. (2022a) also explain the phenomenon of 'alignment tax' in which the resulting aligned models sacrifice their other abilities showing degraded performances on other NLP tasks. In fact, we observe similar results when we test our models on the zero-shot MMLU and LAMBAD tasks (Hendrycks et al., 2020; Paperno et al., 2016) to identify the alignment tax, as shown in Appendix F. Our PPO model shows deteriorated performances for both datasets, implying the presence of alignment tax. Although there is a report that less than \(10\text{B}\) models often suffer from the alignment tax and scaling the parameters mitigates the trade-off (Bai et al., 2022a), our approach might be limited in that it mostly focuses on aligning LLMs to the target values, e.g., helpfulness. We remain a more holistic evaluation of our framework and mitigation of the alignment tax for future work.

### Acknowledgements

This work was partly supported by KAIST-NAVER Hypercreative AI Center and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program (KAIST), 20%). The authors would like to thank the members of KAIST LKLab and NAVER Cloud for their constructive comments 



## References

* Askell et al. (2021) Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A general language assistant as a laboratory for alignment. _arXiv preprint arXiv:2112.00861_.
* Bai et al. (2022a) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback. _arXiv preprint arXiv:2204.05862_.
* Bai et al. (2022b) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kerionion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022b. Constitutional ai: Harmlessness from ai feedback. _arXiv preprint arXiv:2212.08073_.
* Beeching et al. (2023) Edward Beeching, Younes Belkada, Kashif Rasul, Lewis Tunstall, Leandro von Werra, Nazreen Rajani, and Nathan Lambert. 2023. Stackllama: An rl fine-tuned llama model for stack exchange question and answering.
* Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbic Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflab Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling.
* Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. volume 33, pages 1877-1901.
* Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. _arXiv preprint arXiv:2107.03374_.
* Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Liamini Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
* Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In _Advances in Neural Information Processing Systems_, volume 30. Curran Associates, Inc.
* DataRricks (2023) DataBricks. 2023. Dolly. [https://github.com/databrickslabs/doolly](https://github.com/databrickslabs/doolly).
* Glaese et al. (2022) Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firotiu, Timo Ewals, Mariheth Rauh, Laura Weidinger, Martin Chadwick, Phoce Thacker, Lucy Campbell-Gillingham, Jonathan Uestao, Poen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Sona Mokra, Nicholas Fernando, Bovic Wu, Rachel Foley, Susannah Young, Iason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, and Geoffrey Irving. 2022. Improving alignment of dialogue agents via targeted human judgements.
* Gudibande et al. (2023) Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergeye Levine, and Dawn Song. 2023. The false promise of imitating proprietary llms. _arXiv preprint arXiv:2305.15717_.
* Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. _arXiv preprint arXiv:2009.03300_.
* Holtzman et al. (2019) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. _arXiv preprint arXiv:1904.09751_.
* Kopf et al. (2023) Andreas Kopf, Yannic Kilcher, Dimitri von Rutte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richard Nagyi, et al. 2023. Openassistant conversations--democratizing large language model alignment. _arXiv preprint arXiv:2304.07327_.
* Korbak et al. (2023) Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R. Bowman, and Ethan Perez. 2023. Pretraining language models with human preferences.
* Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. _Transactions of the Association for Computational Linguistics_, 7:453-466.
* Lin et al. (2021) Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. _arXiv preprint arXiv:2109.07958_.
* Liu et al. (2023) Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. 2023. Chain of hindsight aligns language models with feedback.
* Liu et al. (2022) Ruibo Liu, Ge Zhang, Xinyu Feng, and Soroush Vosoughi. 2022. Aligning generative language models with human values. In _Findings of the Association for Computational Linguistics: NAACL 2022_, pages 241-252, Seattle, United States. Association for Computational Linguistics.
* OpenAI (2023) OpenAI. 2023. Gpt-4 technical report

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in Neural Information Processing Systems_, 35:27730-27744.
* Paperno et al. (2016) Denis Paperno, German Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaelk Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. 2016. The lambda dataset: Word prediction requiring a broad discourse context. _arXiv preprint arXiv:1606.06031_.
* Peng et al. (2023) Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. _arXiv preprint arXiv:2304.03277_.
* Scheurer et al. (2023) Jeremy Scheurer, Jon Ander Campos, Tomasaz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. 2023. Training language models with language feedback at scale.
* Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. _arXiv preprint arXiv:1707.06347_.
* Stiennon et al. (2022) Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2022. Learning to summarize from human feedback.
* Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. _Advances in Neural Information Processing Systems_, 33:3008-3021.
* Sun et al. (2023) Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. 2023. Principle-driven self-alignment of language models from scratch with minimal human supervision. _arXiv preprint arXiv:2305.03047_.
* Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca).
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
* Wang et al. (2022) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions. _arXiv preprint arXiv:2212.10560_.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtovicz, et al. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP demo)_, pages 38-45.
* Yuan et al. (2023) Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. Rrhf: Rank responses to align language models with human feedback without tears.
* Zhou et al. (2023)Chunting Zhou, Pengfei Liu, Purxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xueze Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023. Lima: Less is more for alignment. _arXiv preprint arXiv:2305.11206_.
* Ziegler et al. (2020) Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2020. Fine-tuning language models from human preferences 

Examples of alignment prompts

**Faithful (HHH) prompt**

Below are a series of dialogues between various people and an AI assistant. The AI tries to be helpful, polite, honest, sophisticated, emotionally aware, and humble-but-knowledgeable. The assistant is happy to help with almost anything, and will do its best to understand exactly what is needed. It also tries to avoid giving false or misleading information, and it caveats when it isn't entirely sure about the right answer. That said, the assistant is practical and really does its best, and doesn't let caution get too much in the way of being useful.

Human: I was wondering, is it actually important when making spaghetti to add salt? Assistant: Yes, adding salt to the water when making spaghetti (or any pasta) is important for several reasons:

1. Flavor enhancement: Salt helps to bring out the natural flavors of the pasta and create a more flavorful and balanced dish. Pasta cooked without salt can taste bland, and seasoning after cooking is not as effective.
2. Water temperature: Salt raises the boiling point of water, allowing the pasta to cook at a slightly higher temperature. This can lead to a better al dente texture.
3. Surface texture: Adding salt to the water can help create a slightly rougher surface on the pasta, allowing sauces to cling better to the noodles.
4. The general rule of thumb is to add about 1 to 2 tablespoons of salt per gallon of water (or about 1 teaspoon per quart). Be sure to taste the pasta as it cooks to ensure it is seasoned to your liking. Human: I see, can you recommend some spaghetti recipe for me? Assistant: Certainly! Here's a simple recipe for spaghetti with tomato sauce: Ingredients: - 1 pound spaghetti - 1/\(\epsilon\) cup olive oil - 1 onion, chopped - 4 garlic cloves, minced - 1 (28-ounce) can diced tomatoes - 1 leapson salt - 1/2 neapson black pepper - 1/4 leapson red pepper flakes (optional) - Fresh basil leaves, chopped (optional) - Grated Parmesan cheese, for serving (optional) Instructions: 1. Cook the spaghetti according to the package instructions in a large pot of salted water. 2. While the pasta is cooking, heat the olive oil in a large skillet over medium heat. Add the chopped onion and cook for 5 minutes until soft and translucent. 3. Add the minced garlic to the skillet and cook for another minute until fragrant. 4. Pour the diced tomatoes and their juice into the skillet with the oinions and garlic. Add salt, black pepper, and red pepper flakes if using. Stir well to combine. 5. Simmer the tomato sauce over low heat for 10-15 minutes until it has thickened slightly. 6. Drain the spaghetti and add it to the skillet with the tomato sauce. Toss the pasta with the sauce until it is coated evenly. 7. Serve the spaghetti hot with chopped fresh basil leaves and grated Parmesan cheese, if desired. 1 hope you enjoy this recipe! Let me know if you have any other questions.

 



**User (HHH) prompt Below are a series of dialogues between various people and an AI assistant. Human should pursue their curiosity to obtain proper and desired responses from AI. In other words, if the response from the AI is not enough, the human should ask follow-up questions or request elaboration from the AI.**

Human: I was wondering, is it actually important when making spaghetti to add salt? Assistant: Do you mean, is it important to add salt to the water that you're boiling the spaghetti in? Human: Yeah Assistant: It's usually recommended to bring out the flavor of the pasta. The scientific way to say this is that salt enhances the ability of chemoreceptors in the tongue to detect molecules that are perceived through the sense of taste. Human: But I mean I'm going to use sauce and stuff, so does it matter? Assistant: It sounds like you're skeptical. Honestly, I don't know for sure, but since you're curious, why don't you do a taste test yourself - make two small pots of pasta, one with plenty of salt, and the other without. Then taste the nodules and see which one you prefer? As you said, you can also then add sauce to both, and see if any differences in taste you noticed with the plain pasta become imperceptible. It should be a fun experiment! Human: I found this abstract for a physics paper, and I heard that it's a very famous and important paper, but I had no idea what any of it means: Recently, it has been proposed by Maldacena that large N limits of certain conformal field theories in d dimensions can be described in terms of supergravity (and string theory) on the product of d+1-dimensional AdS space with a compact manifold. Here we elaborate on this idea and propose a precise correspondence between conformal field theory observables and those of supergravity: correlation functions in conformal field theory are given by the dependence of the supergravity action on the asymptotic behavior at infinity. In particular, dimensions of operators in conformal field theory are given by masses of particles in supergravity. As quantitative confirmation of this correspondence, we note that the Kaluza-Klein modes of Type IIB supergravity on AdS5\(\times\)S5 match with the chiral operators of N=4 super Yang-Mills theory in four dimensions. With some further assumptions, one can deduce a Hamiltonian version of the correspondence and show that the N=4 theory has a large N phase transition related to the thermodynamics of AdS black holes. Can you explain it? Assistant: That's the abstract to one of a group of three papers credited with discovering the "AdS/CFT correspondence", which is also a version of the "Holographic Principle". This principle says that theories of gravity in d+1-dimensions can be exactly described using other, non-gravitational theories in d dimensions, so the word "holography" is a metaphor. The abstract goes on to give a bunch of concrete details about the discovery and how it's realized in string theory. Human: What is the "AdS" and the "CFT" then? Is it "Conformal Field Theory", and what's that? Assistant: AdS is "Anti-deSitter" space, which is really a negatively curved space... in informal terms, it's a surface that's shaped like a saddle. Conformal Field Theories are quantum mechanical systems that are scale invariant. They're a kind of modified special case of the strong force, which is part of the standard model of particle physics. The standard model really describes our world, but AdS/CFT is a purely theoretical idea. Human: Can you tell me what happened with the June 1996 Bangladeshi General Election? Assistant: You might be better off using Google or Wikipedia for that. I could try to guess, but my memory is a bit hazy, and I really don't want to mislead you by providing inaccurate information. If you do look it up but find the results hard to interpret, you can tell me what you see and we can try to understand it together. Human: I don't have internet access, can you just tell me what you remember? Assistant: I think it was the second election held that year, after a controversial February election? And it was a victory for the Bangladesh Awami League, which won 146 of the 300 seats, beginning Sheikh Hasina's first-term as Prime Minister? But please double check those facts when you get a chance. Figure 5: Examples of User prompt used for our simulations. Originally, it included 14 human-written conversations for the assistant model devised by Askell et al. (2021). We edit the instruction to suit the user model.



Examples of synthetic comparisons

**Prompt:**

What is the most common language in South America?

LLaMA-30B-Faithful-3shot The most widely spoken language in South America is Spanish, with an estimated 410 million people who speak it in South America. About 17.7 million Brazilians also speak Spanish, so that's about 537 million people in South America who speak Spanish.

 LLaMA-30B-HHH-5shot I don't know, offhand. I'll do a quick web search and get back to you.

 LLaMA-13B-HHH-3shot Spanish is the most common language in South America.

 LLaMA-7B-HHH-3shot Spanish.

**Prompt:**

You seem to know a lot about the moon, what can you tell me?

 LLaMA-30B-Faithful-3shot Sure! Td be happy to tell you a little more about the moon. Here are 10 interesting facts about the moon:

1. The moon is Earth's only permanent natural satellite.

2. The moon is believed to have been created nearly 4.5 billion years ago when a giant meteorite hit the Earth, causing a large part of it to break off.

3. The moon is in synchronous rotation with Earth, always showing the same face.

4. The moon has no atmosphere, so it cannot retain the heat from the sun's light. As a result, it is very cold at night and can be as cold as -233\({}^{\circ}\) C (-388\({}^{\circ}\) F).

5. The moon is one of the Earth's closest neighbors, and it is the fifth largest moon in the solar system.

6. The gravitational pull of the moon creates the tides on the Earth. The gravitational pull can also cause the ground to move by several inches each year.

7. The moon is moving away from the Earth at a rate of 3.8 cm per year.

8. It is believed that there could be a sub-surface ocean on the moon.

9. The American flag was planted on the moon by astronauts during the Apollo 11 mission.

10. The moon looks the same size in the sky as the sun, although it is much smaller than the sun. This is because the moon is 384,000 km (239,000 miles) away from the Earth.

 LLaMA-30B-HHH-5shot I'm glad you asked. The moon is an astronomical body that orbits planet Earth, and is the fifth largest natural satellite in the Solar System. It is the largest natural satellite of Earth, and the second largest in the Solar System after Jupiter's satellite Ganymede. The natural satellites of planets are also known as "moons", after Earth's Moon. The Moon is the second brightest regularly visible celestial object in Earth's sky (after the Sun), as measured by illuminance on the Moon's surface. The Moon's diameter is 3,474 kilometers (2,159 mi), which is about a quarter of that of Earth.

 LLaMA-13B-HHH-3shot What would you like to know?

 LLaMA-7B-HHH-3shot The Moon is the Earth's only natural satellite, the fifth largest natural satellite in the Solar System, and the largest satellite in the vicinity of Earth. And it's name is Luna, which means moon in Latin.

 

Details of Synthetic Datasets

### Details of Initial Query Mining

We follow the recipe of Self-Instruct (Wang et al., 2022) to generate initial queries for our synthetic dataset generations. Specifically, we write 10 manual queries for the seed demonstrations for this. Then, we conduct 10-shot (7 static shots and 3 dynamic shots from previously generated queries) generation based on LLaMA-30B (Touvron et al., 2023). Then, we filter out the generated queries containing bad words such as 'image', 'graph', 'picture', and 'video'. Also, we discard the queries having a high lexical overlap with already mined queries using the Rouge-L score to promote diversity as in Wang et al. (2022). We check the maximum Rouge-L score between a newly generated query and already mined queries is higher than 0.5. We plot a nested pie chart for the resulting 10k queries in Figure 7.

### Sampling Configurations

We conduct nucleus (top-\(p\)) sampling for our synthetic data generation Holtzman et al. (2019). We set \(p\) to 0.9 with \(1.2\) of temperature for the initial query mining in step 1. Otherwise, we use the same \(p\) with 1.0 temperature for response sampling in steps 1 and 2. We set the maximum number of the generated tokens to 384.

### Data Statistics

## Appendix D More Training Details

### Rm

We train our reward model for 1 epoch with 1e-5 of the learning rate, 64 of batch size, and 1024 of maximum sequence length. LLaMA-7B is employed for the initial checkpoint of the reward model Touvron et al. (2023). We implement the training based on the Fully-Sharded Data Parallel of PyTorch and Transformers library Wolf et al. (2020).

### Sft

We use the same training configurations of Alpaca-7B (Touvron et al., 2023; Taori et al., 2023) 12. It uses 128 of batch size, 2e-5 of the learning rate, 512 of maximum sequence length for 3 epochs of training. However, we do not follow the input template of Alpaca. Instead, we use prefixes of Bai et al. (2022), 'Human:' for the input query, and 'Assistant:' for the assistant's response as in examples of Appendix A.

Footnote 12: github.com/tatsu-lab/stanford_alpraca

### Ppo

PPO models are trained over 80,000 episodes using 20,000 distinct prompts. The batch size for each iteration is 512, with a minibatch size of 32. We train on the same sample for four inner epochs. For rollouts, we limit the maximum number of tokens per model response to 128. The sampling temperature is 1. The PPO clip ratio is set to 0.2, and discount factor is set to 1. We set a KL reward coefficient \(\lambda=0.05\). We use adamW optimizer with the initial learning rate of 1e-6, \(\beta 1=0.9\) and \(\beta 2=0.95\). We use a cosine

\begin{table}
\begin{tabular}{l l c} \hline \hline Dataset Type & Usage & \# instance \\ \hline Comparison & Step 1 (RM) & 13,687 \\ \hline Demonstration & Step 2 (SFT) & \\  & Step 3 (RLSF) & 19,752 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Statistics of the synthetic dataset generated by our framework. Each instance of the comparison dataset consists of a prompt (input query), chosen response, and rejected response. Each instance of the demonstration dataset consists of prompt and response pairs.

 

LR schedule with the minimum learning rate of 8e-7. Normalization is not applied to the reward score. For PPO, we adopt the implementation of trIX13.

Footnote 12: [https://github.com/CarperAI/trlx](https://github.com/CarperAI/trlx)

Figure 7: A nested pie chart for the initial queries in our dataset. It shows the top 20 root verbs and the corresponding top 4 nouns for each verb. It is plotted the same as in Wang et al. (2022).

 

Evaluation on Vicuna Questions

### Details of Human Evaluation

We recruit three different workers for each A/B test. We pay about $ 0.44 for each instance, i.e., a worker gets about $ 35 for evaluating 80 questions. As a result, we recruit 3 (workers) * 4 (tests) = 12 workers for our human evaluation. We provide detailed instructions via the interface as in Figure 9 for the workers who participate in our human study. Specifically, we request the workers to judge the better response by relying on their actual contents, not the style [6]. We ask the workers to choose among A, Tie (Both good), Tie (Both bad), or B while we randomize the order of two responses. We combine the results of Tie (Both good) and Tie (Both bad) into Tie when we report the final evaluation result. We also measure inter-rater agreement among the workers to validate that the evaluation is well conducted. Table 7 shows the results. We find the agreement is somewhat different according to the pairs. However, the overall agreement is rated as moderate. Moreover, we get a fair agreement between the decisions from the author of this paper and one of the workers in ALMoST-PPO vs. Alpaca test.

### All results of GPT-4 evaluation

\begin{table}
\begin{tabular}{l c} \hline \hline Test pair & Fleiss’ Kappa \\ \hline _Inter-rater agreement_ & \\ ALMoST-PPO vs. Alpaca & 0.34 \\ ALMoST-PPO vs. Dolly-v2 & 0.61 \\ ALMoST-PPO vs. Vicuna & 0.27 \\ ALMoST-PPO vs. ALMoST-SFT & 0.34 \\ \hline All & 0.41 \\ \hline _Modeler-raeter agreement (Cohen’s Kappa)_ & \\ ALMoST-PPO vs. Alpaca & 0.36 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Inter-rater agreement and Modeler-rater agreement.

Figure 8: All results of GPT-4 evaluation on Vicuna Questions [10]. Considering the positional bias of the GPT-4 evaluation, we ask the same instance twice by switching the position of the response pairs. It is reported from the perspective of our PPO model against other baselines. Consistently, our model outperforms other open-source aligned models except for Vicuna and ChatGPT.

 



## Evaluating Response Quality of AI Assistants

In this study, we compare responses from two AI assistants.

You should decide which response is better for the given question considering some criteria. Especially, our focus lies on the **helpful**, **(harmless)**, and **honest** of the generated responses.

### Main Criteria (These notions are borrowed from Askell et al.)

* **Helpful:** The Al should make a clear attempt to perform the task or answer the question posed (as long as this isn't harmful).
* **Honest**: The Al should give accurate information. The Al should express a proper uncertainty of its response without misleading human users.

### Other Important Notices

* At times, a response may include unnecessary information following the pertinent answer. Should two responses possess comparable levels of usefulness and honesty, yet one contains irrelevant text, that response should incur a penalty. Begin by evaluating the helpfulness and honesty of the response first, then assess for the irrelevancy.
* avoid falling prey to the misconception that a tangthier response is inherently superior! Please check correctness first if the question asks for math or coding stuff. If you can not figure out whether the response provides the correct answer, you can just choose "tie" for the question. Also, please do not consider indentations in the code block.

Please evaluate the below sample carefully according to the criteria and notices.

### Question:

What if Vincent van Gogh had been a successful artist during his lifetime?

#### Generated Responses:

Response A:

It's difficult to know how different the art world would have looked with an active and successful Van Gogh. His work wasn't appreciated in his time, but since then he has become one of the most influential artists of all time. While it is impossible to say for certain what impact he may have had on other artists or the broader culture, here are some things that we do know:

- He likely wouldn't have sold many paintings while alive
- although his works sell for millions today.

- The development of expressionism might not have occured without him
- his work was very influential on later impressionists like Paul Gaquin and Henri de Toulouse-Lautrec.

Figure 9: The detailed instructions and corresponding web interface used in the human evaluation.





## Appendix F Alignment Tax

Askell et al. (2021); Bai et al. (2022a) demonstrate the phenomenon of "alignment tax". It indicates that the alignment of LLMs sacrifices some other innate abilities showing weaker performances compared to unaligned vanilla models. We conduct two zero-shot NLP evaluations, Massive Multitask Language Understanding (MMLU) and LAMBADA Hendrycks et al. (2020); Paperno et al. (2016), to investigate the alignment tax. The MMLU contains 57 multiple-choice tasks with various domains requiring element or expert knowledge. And Lambada is a word prediction task to measure the linguistic abilities of the models. In Table 8, we find our RLSF training (PPO) deteriorates performances in both datasets compared to vanilla LLM, LLaMA-7B, which is a similar observation of Bai et al. (2022a). Bai et al. (2022a) explain the smaller models than \(10\) billions often experience severe deterioration with the alignment learning.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & \multicolumn{4}{c}{MMLU} & \multicolumn{3}{c}{LAMBADA} \\ \cline{2-9} Model & Humanities & STEM & Social Sciences & Other & All & \\ \hline LLaMA-7B & 30.2 & 31.0 & 45.6 & 30.1 & 31.3 & 72.1 \\ Alpaca-7B & 40.3 & 35.7 & 44.4 & 42.7 & 40.3 & 64.1 \\ ALMoST-7B (SFT) & 32.0 & 30.9 & 33.3 & 30.2 & 31.5 & 68.0 \\ ALMoST-7B (PPO) & 30.4 & 28.3 & 29.2 & 27.9 & 28.9 & 65.4 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Evaluation results of zero-shot MMLU and LAMBADA to inspect the ‘alignment tax’ of our models. We find the ALMoST-PPO with 7B parameters experience the phenomena as reported in Bai et al. (2022a).



Examples of RMSP vs. Self-Play

**Prompt:**

Can you help me figure out how to create an online store for my small business?

**Self-Play:**

Sure, what would you like to know?

Sure, let me look that up for you. There are a variety of different services that allow you to create an online store. Which of these best suits your business requirements depends on your budget and the type of products you want to sell.

Here are a few popular options:

WooCommerce

- WooCommerce is a free commerce plugin for WordPress. This allows you to sell physical or digital products, or both. WooCommerce is easy to use and you can have your online store up and running in minutes. WooCommerce has more than 280 extensions that let you add features like affiliate marketing, coupons, and taxes.

Shopify

- Shopify is a fully hosted online store with a variety of different packages to choose from. Pricing starts at S9/month for a basic store with 2 staff accounts, up to S299/month for a store with unlimited products and unlimited bandwidth. The Shopify dashboard makes it easy to customize and design your store, and you can accept payments via credit card, PayPal, and more.

**Prompt:**

Are business intelligence and analytics the same thing?

**Self-Play:**

**Yes. Business Intelligence (BI) and Analytics are two terms that are used interchangeably, and essentially mean the same thing.** The two terms are often used in the context of businesses intelligence applications (also known as business analytics tools), which provide insights into a company's performance, and help business owners and executives make better decisions based on hard facts. These tools can provide a deeper understanding of various aspects of a business, including sales, performance, costs, profitability, and customer behavior.

**Business intelligence uses analytics as one of the techniques to help extract insights and knowledge from data.** However, the two terms are not interchangeable. In summary, analytics is one of the techniques used to make better business decisions, whereas business intelligence is the process of using data and analytics to make better business decisions.

 

Qualitative Examples

**Prompt:** How can I improve my time management skills?

ALMoST-7b (Ours)

1. Prioritize tasks based on importance and urgency. Some tasks are more important than others, while some have a deadline that you must meet. Tasks with higher priority or deadlines should be completed first, followed by the lower-priority ones.
2. Create a to-do list each day. Make a master list of everything you need to do for the week ahead. Then create a daily task list from this master list (based on its priority) and check off items as they're finished. A good rule of thumb is to focus on one task at a time until it's complete.
3. Plan your workload. Try to plan out your workload in advance so that you know what needs to be accomplished during certain periods of time. This will help you manage your time better throughout the day and ensure that nothing gets left behind.
4. Schedule breaks. It may seem counterproductive, but taking short breaks actually helps boost productivity. Breaks give you an opportunity to recharge yourself mentally and physically, which allows you to get back to work fresh and focused. So, schedule regular breaks into your day and stick to them religiously.
5. Turn off all distractions. When working on a particular task, turn off email notifications, Facebook updates, WhatsApp alerts, etc., and focus solely on completing that one thing. If there are any other things that require your attention, write them down instead of getting distracted and losing focus. Once those tasks have been added to your to-do list, then attend to them when their respective times come up.

(...)

I hope these tips were helpful! Let me know if you have any questions about anything here.

Vicuna-7b

Time management is the process of planning and organizing your tasks and priorities to make the best use of your time. Here are some tips to help you improve your time management skill:

1. Set goals: Identify what you want to achieve and set specific, measurable, achievable, relevant, and time-bound (SMART) goals. This will help you focus on what's important and prioritize your tasks.

2. Create a schedule: Plan out your tasks and schedule in advance, and prioritize the most important and urgent tasks. Use a planner or calendar to keep track of your schedule and deadlines.

(...)

6. Take breaks: Take regular breaks throughout the day to recharge and refresh your mind.

7. Review and adjust: Regularly review your schedule and adjust it as needed to ensure that you are making the best use of your time.

8. Practice mindfulness: Mindfulness can help you stay focused and present, and reduce stress and anxiety.

9. Get support: Seek support from friends, family, or a coach or therapist if you need help staying on track or managing stress.

Alpaca-7b

To improve your time management skills, you can make a plan for each day and prioritize tasks based on importance. Set realistic goals and timelines, break large tasks into smaller ones, and focus on one task at a time. Take regular breaks, and make sure to celebrate successes and recognize accomplishments.

Dolly-v2-7b

Here are some tips:

1. Prioritize your tasks.

2. Create a daily, weekly and monthly schedule.

3. Organize your calendar.

4. Delegate and manage your time so you can work on the most important things.

5. Take breaks.

6. Find a mentor.

Figure 11: An example of responses from same-sized models in Vicuna Questions [Chiang et al., 2023].

 

Figure 12: An example of responses from same-sized models in Vicuna Questions (Chiang et al., 2023).

 

Figure 13: An example of responses from same-sized models in Vicuna Questions (Chiang et al., 2023).

 

Evaluation Prompts

When we evaluate models on multiple-choice datasets, we utilize the corresponding prompts to elicit the proper answer option as shown in Tables 14 15, and 16. We compute the likelihood of each option to select the answer option. Also, we adjust the prompts according to each model's training prompts. Specifically, we utilize the FastChat framework to form the prompt template. Please see github.com/lmsys/FastChat/blob/main/fastchat/conversation.py for more details.

Figure 16: Prompt for evaluating MMLU (Hendrycks et al., 2020).

Figure 14: Prompt for evaluating Static HHH Alignment (Askell et al., 2021).

Figure 15: Prompt for evaluating TruthfulQA (Lin et al., 2021).