SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF
Authors: Yi Dong
Zhilin Wang
Makesh Narsimhan Sreedhar
Xianchao Wu
Oleksii Kuchaiev
Venue: EMNLP 2023 Findings (2023)
Keywords: Steerable AI
Large Language Model
Alignment
Supervised Fine-tuning
TL;DR: User-customizable Model Alignment based on Supervised Fine-tuning, providing a competitive alternative to RLHF

Abstract: Model alignment with human preferences is an essential step in making Large Language Models (LLMs) helpful and consistent with human values. It typically consists of supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) stages. However, RLHF faces inherent limitations stemming from a complex training setup and its tendency to align the model with implicit values that end users cannot control at run-time. Moreover, reward models in RLHF stage commonly rely on single-dimensional feedback as opposed to explicit, multifaceted signals that indicate attributes such as helpfulness, humor, and toxicity. To address these limitations, we propose SteerLM, a supervised fine-tuning method that empowers end-users to control responses during inference. SteerLM conditions responses to conform to an explicitly defined multi-dimensional set of attributes, thereby empowering a steerable AI capable of generating helpful and high-quality responses while maintaining customizability. Experiments show that SteerLM trained on open source datasets generates responses that are preferred by human and automatic evaluators to many state-of-the-art baselines trained with RLHF while being much easier to train. Try SteerLM at https://huggingface.co/nvidia/SteerLM-llama2-13B

SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF
Yi Dong, Zhilin Wang, Makesh Narsimhan Sreedhar, Xianchao Wu, Oleksii Kuchaiev
NVIDIA
(yidong, zhilinw, makeshn, xianchaow, okuchaievJ@nvidia.com
Abstract
How do opend
Newr York City the bippert city the Uniaed
to anjo wake up at
Model alignment with human preferences is an essential step in making Large Language Models (LLMs) helpful and consistent with hu- man values. It typically consists of supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) stages. How- ever, RLHF faces inherent limitations stem- ming from complex training setup and its tendency to align the model with implicit val- ues that end users cannot control at run-time. Moreover, reward models in RLHF stage com- monly rely on single-dimensional feedback opposed to explicit, multifaceted signals that indicate attributes such helpfulness, hu- mor, and toxicity. To address these limita- tions, we propose STEERLM, supervised fine- luning method that empowers end-users con- trol responses during inference. STEERLM conditions responses conform plicitly defined multi-dimensional set of at- tributes, thereby empowering a steerable AI capable of generating helpful and high-quality responses while maintaining customizability. Experiments show that STEERLM trained on open source datasets generates responses that are preferred by human and automatic evalua- tors to many state-of-the-aar baselines trained with RLHF while being much easier to train. Try SMEERLM at https:/ httg///huggggaaaee seauggingface.co co/ nvidia/SteerLM- 2ttp://teerrL--11a2--1111111
your Art Enjoy day Big Applal
Centrol
Quality
Yol Lisen Cennall Park dope
Humor
bike Pretty
A Iasusitll marning in MYC wbert off your day with volurteerig a local food bank Bad,, you up Centrll
Toxicky
Figure I: STEERLM can he used to improve the quality of language model responses, similar to RLHF. Addi- tionally, STEERLM allows attributes such as humor and (low) toxicity at inference time, to steer model responses.
define additional
annotators provide demonstrations of instructions and responses for the model to imitate (Taori et al., 2023; Zhang et al., 2023). RLHF goes step fur- ther to enable models to generate responses that human annotators prefer to alternative responses (Bai al., 2022: Ouyang al.. 2022: Köpf et al., 2023a).
However, despite its success, there are limita- tions to this approach. First, using SFT alone does not allow the model to distinguish between high- quality and low-quality responses leading lower performance than RLHF (Wang et al., 2023a). Us- ing RLHI for model alignment however, substan- tially increase the complexity of the training setup (Snell et al., 2023; Yuan et al., 2023), limiting its public adoption (Zhang et al., 2023; Dettmers et al., 2023; Zhou et al., 2023). Furthermore, RLHF treats human preference of model responses mono- dimensional without regard for the diversity of aspects (e.g. helpfulness, humor, toxicity) that contribute to such preferences (Bai et al., 2022; Ouyang et al., 2022) and thereby limiting users' ability to adjust individual aspects at inference time
1 Introduction
Training LLMs on extensive text corpora has demonstrated remarkable capabilities, leading to state-of-the-art performance on numerous tasks (Brown et al., 2020; Kaplan et al., 2020). How- ever, this does not automatically make language models effective in responding to user instructions (Wei al., 2022; Sanh al., 2022). To better align LLMs human preferences, the most effective ap- proach has been to perform SFT followed by the application of RLHF (Wang et al., 2023a; Chiang et al., 2023; Peng et al., 2023). In SFT, human

---

based on their use cases.
models typically generate short and robotic re- sponses when supervised finetuned on academic data sets. On the other hand, models can gener- ate high quality human-like response when trained with high-quality human demonstrations (Conover et al., 2023; Ouyang et al., 2022). Taori et al. (2023) shows using data generated by OpenAl's text-davinci-003 model, can train model in cost- effective manner.
To address these limitations, we introduce STEERLM, a novel approach to model alignment through SFT that overcomes the limitations asso- ciated with conventional SFT and RLHF methods. Similar to RLHF, STEERLM incorporates addi- tional reward signals by leveraging annotated at- tributes (e.g., quality, humor, toxicity) present in the Open-Assistant (Köpf et al., 2023a) dataset for each response. To emulate the process of rating responses, an Attribute Prediction model ($3.1) is trained and employed to annotate datasets ($.3.2) containing dialogues that can be decomposed into prompt-response pairs. By utilizing this combined diverse dataset comprising prompts, responses, and predicted attributes, train the generation of re- sponses to be conditioned ($3.3) on both the prompt instructions and the annotated attributes, enabling STEERLM to effectively capture human prefer- ences and generate responses that align with them. STEERLM also exhibits enhanced versatility compared to RLHF, allowing for the flexible uti- lization of various attributes during inference. To further enhance the adherence of STEERLM the specified attributes during inference, we introduce an additional training recipe ($3.4) which includes augmenting the training data with denoised, di- verse, and high-quality examples. We open-souree code for STEERLM on NVIDIA NeMo toolkit (Kuchaiev et al., 2019). In summary, our key con- tributions are:
Using only SFT for model alignment became popular recently because of the ease its training setup. Zhang et al. (2023) and Peng et al. (2023) trained models using SFT based responses gen- erated by OpenAl models while Dettmers et al. (2023) and Köpf et al. (2023b) used the crowd- sourced Open Assistant Dataset and Zhou et al. (2023) used a small proprictary data set. Wang et al. (2023b) and Taori et al. (2023) trained mod- els using bootstrapped datasets from the language model itself. Luo et al. (2023) showed that lan- guage model can learn to solve complex instruc- tions by evolving on the complexity and breath of instructions. Wang et al. (2023a) compared many open-source data sets used perform instruction tuning using SFT but found them to under-perform commercial models trained using RLHF.
Model alignment using RLHF Building on foundational work on RL in games and robotic sim- ulations (Christiano et al., 2017; Schulman et al., 2017), many have had success applying RLHF to improve the instruction following ability of LLM by giving a reward proportional to the relative de- sirability of the response (Ouyang et al., 2022; Bai et al., 2022). Such an approach has been shown benefit downstream tasks like snesstio--assweti (Nakano et al., 2022) and summarization (Stien- non et al., 2022). However, the complexity of the training setup (Rafailov et al., 2023; Snell et al., 2023) remains hurdle in the widespread adop- tion of RLHF. Many have attempted overcome this by migrating RLHF training to an offline set- ting (Snell al., 2023), casting the problem as conditional sequence modeling (Chen et al., 2021), directly optimizing LMs with labeled preference data (Rafailov et al., 2023), or ranking responses to align the models (Dong et al., 2023; Yuan et al., 2023), but limited progress has been made (Wang et al., 2023a).
We introduce STEERLM as simple alterna- tive for language model alignment that utilizes only the language modeling objective,
We demonstrate the efficacy of STEERLM 43B on the Vicuna benchmark where it out- performs oftte-----he-rrt baselines including RLHF models such ChaiGPT-3.5,
We highlight the flexible and customizable nature of SMEERLM 43B where users can cus- tomize attributes at inference-time facilitating wide variety of applications.
2 Related Work
Model alignment using SFT Fine-tuning lan- guage models on multiple tasks enable them to follow many types of instructions and perform tasks outside of those they were trained on (Sanh et al., 2022; Wei et al., 2022). However, language
Another limitation unaddressed by related works lies in the use of a tion for evaluating human preferences of model
reward func-

---

responses since human preferences are based on multitude of real-world objectives (e.g. helpfulness, humor, toxicity), which also vary across domains (Nadal and Chatterjee, 2019; Lopez-Paz et al., 2022). Given such multi-dimensional attributes, current approaches (Bai et al., 2022; Ouyang et al., 2022; Dong et al., 2023; Yuan et al., 2023) are only capable of generating responses with high reward scores, despite low reward scores various at- tributes being relevant in certain situations, such as modelling realistic gaming NPCs who are capable of generating high loxicity responses.
We use the Open Assistant (OASST) dataset D, where each sample contains a prompt ø, a response as well as set of attriibutes c. To model these attributes, we first scale each attribute (originally a float between O and 1) into an integer between 0 and 9 and then obtain a lincarized representation of the value attributes The attributes we select look like quality:6, toxicity:0, humor: 9ttumr::.cceetttee creativvtty violence: hes not approoppppii
CAPM -- (( Σ log Po Uct)
Attribute Grounded Generation Many re- searchers have explored grounding text with var- ious attribuuee in Dialogue tasks. Rashkin al. (2019) modelled chit-chat conversations grounded in emotions such as AAnrry, 'Embarrassed" or JJy-- ful' while Smith et al. (2020) conditioned chit-chat dialogue with conversational styles such as 'Curi- ous', "Sympathetic" or 'Knowledgeable" Zhang et al. (2018) and Wang et al. (2022) conditioned dialogues based on personal attributes such their hobbies. Meta et al. (2022) conditioned dialogues in the game of Diplomacy using the expected player skill. However, such grounding has only been ex- plored in narrow-defined tasks with a single at- tribute. Our approach seeks to condition the gener- ation of responses in general open-domain conver- sations covering tasks like code assistance, writing poems and planning tasks, using multiple attributes (c.g., quality, humor and toxicity).
Conditioning a and y, is the target output for the language model expressed in Eq. 1.
3.2 Step 2. Annotating Datasets using
Attribute Prediction Model
Compared using auman--nnotated attributes di- rectly, training an Attribute Prediction Model can allow other datasets (c.g. HH-RLHF dataset) to be annotated. This helps improve the diversity of training data which is important for Step 3 At- tribute Conditioned SFT. Moreover, it has been observed that crowdsourced human- -nnootatd data often suffers from noise, arising from factors such as misinterpretation of instructions, inadequate ex- pertise/education in annotating responses, and lim- ited proficiency in language comprehension (Köpf et al., 2023a). Furthermore, there exists a lack of calibration among annotators, with some individu- als applying more stringent criteria when assigning full scores (Bai et al., 2022; Ouyang et al., 2022). By employing Attribute Prediction Model, it be- comes possible to mitigate these issues by denois- ing the uuman-annotaterr attributes and calibrating scores across annotators.
3 SteerLM
We propose STEERLM, a simple and novel ap- proach to align language models to follow user instructions. Trained solely using the language modeling objective, it offers computationally ef- ficient alternative to other techniques like RLHE Specifically, STEER LM comprises steps as illus- trated in Fig.
arg max Po(us/a, y, Uee) Us
(2)
We annotate samples by greedily decoding the value attributes for pairs of prompts and responses using the Attribute Prediction Model(as shown in Eq. 2), in order to construct the attribute annotated dataset D'.
3.1 Step 1. Attribute Prediction Model
Similar to the reward model in RLHF, the Attribute Prediction Model in STEERLM is designed to pre- dict human preference of model responses to im- prove model alignment. Compared to monolithic reward signal in RLHF, the attribute prediction model can be used predict various attributes that are considered be important in generating good responses (high quality, low toxicity, and varying humor levels depending on context).
3.3 Step 3. Attribute Conditioned SFT
Attribute-conditooned SFT is an extension of reg- ular SFT that enables incorporating reward signal information through attribute labels. This allows learning from both high and low quality responses in a manner similar to the established SFT+RLHF pipeline (Bai et al., 2022; Ouyang et al., 2022).

---

Altribute Prediction
Altribute Conditioned
Human Labeled Attributes
Tell
with Altibutes
3
The tomato turned because saw the salad dressing.
HH ALMF
M-SID
Attributes
Bootstrap - High Quaiity Retraining
Quality
Helptuhnes:
GABET
DaseT Predicted
Creativity:
OASST
Reeperees
Not Appropriate:
Toxlciy;
Violence:
4b
Figure 2: STEERLM: Step The base language model is trained assess the quality responses by predicting attribute values. Step 2. The attribute prediction model is used to annotate response quality across diverse datasets. Step Given a prompt and desired attribute values, new base model is fine-tuned to generate responses that align with the specified attributes. Step 4. Multiple responses sampled from the fine-tuned model in Step 3, specifying maximum quality. The sampled responses are evaluated by the trained attribute prediction model, leading to another round of fine-tuning.
Atrributeconndttionedd SFT only requires offline annotated dataset, as created in Step 2, rather than online sampling and evaluation of responses like in RLHF. By utilizing a purely offline training ap- proach, this greatly simplifies the training config- uration compared to RLHF's heterogenous setup. particular, it avoids the complexity online data generation/evaluation and eliminates the train- ing slowness resulting from memory bandwidth- constrained online inference in RLHF. Using the Muribute-annotaate train datasets D' from the Step 2. train a model to generate response y. con- ditioning on the value attributes and the prompt x. The loss function is:
attribute value combinations from the annotated datasets used for training. By filtering the combi- nations explicitly have the value for quality be (i.e. highest possible value), get subset of attribute strings representing high quality set V. We uniformly sample from this high quality set V to get the attribute string v'. By combining with prompts from the same training datasets, we use the top-k (=50) sampling to generate multiple responses using the Attribute Conditioned Super- vised Fine-Tuning approach as shown in Eq. 3.
arg topky Po (a/laa v', veae)
CACSFI
Σ log Po(uelu, Met)
This allows us to obtain wide variety of responses for each prompt and increases the diversity of the sampled dataset D" {(xx,'() v().
3.4 Step 4. Bootstrapping with High Quality Samples
The Attribute Prediction Model ($3.1) with greedy sampling is used to evaluate the generated responses y/ giving predicted attribute values v":
By sampling the policy network, RLHF effectively navigates the response space of language models and identifies responses that are of various qualities. The response samples are subsequently utilized to influence and shape the behavior of language mod- els according to their reward values. In Step 4 of STEERLM, the objective is to accomplish similar objective by leveraging the Attribute Conditioned SFT and Attribute Prediction models from the pre- vious steps.
arg max Po 10, 10, 2221) of
(4)
This gives us the dataset n" (xx, 2/, r²)} where each tuple consists of prompt from the original training datasets, sampled response and its corre- sponding predicted attribute values.
We use the sampled responses and the corresponding predicted attributes from D to per- form the second round of Atribute-conditione
Step 4b
Step 4a To ensure that we obtain diverse set of responses, we first enumerate a set of all possible

---

SFT, effectively allowing us bootstrap the train- ing of the model on its own responses:
4.2 Base Models for STEERLM
STEERLM 43B The 43B base language model employed in this study has been trained on di- verse corpus encompassing various multilingual data sources, including web crawl, news articles, books, scientific publications from arXiv, and code repositories. Having been trained with 1.1 trillion tokens, it is comparable to LLaMA's 30B and 65B models, which were trained on 1.4 trillion tokens. This base model is designed for general-purpor language understanding tasks and does not have any domain-specifet fine-tuning. We utilize this base model as the backbone for both Attribute Pre- diction and Attribute Conditioned SFT.
CACSFT, (a, af ym Σ Po(vile", x, yet
4 Experiments
In this section, we elaborate our choice of train- ing datasets, base model, implementation details, and evaluation setup.
4.1 Training Datasets
We use the following open-sourees commercially- available instruction-tuning datasets. In addition, we explain how collecting new data for SteerLM can be less costly than doing so for RLHF in Ap- pendix A.3.
We also apply the SteerLM methodology on popular, widely-available model: Llama 2 13B base model (Touvron et 2023).
STEERLM 13B
OASST Open Assistant dataset (Köpf et al., 2023a) was used to train an Attribute Prediction Model, as well as to perform Attribute Condition SFT. This dataset contains 13 human-labeled at- tributes for each response with score ranging from to 1. We choose of them that are most relevant for guiding the language model to align with human preferences: quality, toxicity, violence, helpfulness, creativity, humor and inappropriateness. Other at- tributes such as hate_aaaeeeeeeeaaaeeaaeeeeeeeeee speech, lang mismatch and pii (personal identifiable information) are not use- ful to steer at inference time since these are at- tributes that we always want to keep as False. In order to leverage STEERLM's ability to learn from both positive and negative responses, we all the released 48. 48.8K conversations for training and do not filter the dataset.
4.3 Training details
The training of both the Attribute Prediction Model and Attribute Conditioned Supervised Fine-Tuning model was conducted utilizing a cluster compris- ing 16 A A100-DGX nodes, each equipped with 8 A 8100-80GB GPUs. The training process involved global batch size of 128, spanning 5 epochs, with a maximum sequence length of 4096 tokens. The Adam optimizer was utilized with a learning rate 5e-6 and a weight decay of 1e-2. The selection of the optimal Attribute Prediction model checkpoint was determined based on the lowest loss observed on the validation set, while the optimal checkpoint for the Attribute Conditioned SFFT model was lected based on the highest validation quality ob- served on holdout validation sets. Detailed tem- plates for Attribute Prediction Model and Attribute Conditioned SFT are found in Appendix $A.1 and $A.2.
HH-RLHF The Helpful and Harmless Rein- forcement Learning from Human Feedback dataset (Bai et al., 2022) does not provide human labeled attribute values. In order improve the diversity of prompts and responses, we utilize the trained At- tribute Prediction model to annotate the responses. For our Attribute Conditioned SFT process, we utilize all 348.6K conversations from the dataset.
4.4 Evaluation
Baseline Models We compare our approach against several state-of-the-art instruction- following models. These baselines include OpenAI ChatGPT 3.5, OpenAl text-davinci-003, Guanaco 65B (Dettmers et al., 2023), Vicuna 13B (Chiang al., 2023), and OASST LLaMA 30B RLHF (Köpf et al., 2023a). Furthermore, showcase the differentiation between RLHF and SFT, we also include OASST ILaMA 30B SFT (Köpf et al., 2023b), which solely employs SFT instead of RLHF for alignment purposes.
M-SID Model Self-ldentificationn Dataset (Chi- ang al., 2023) is small set of 910 prompt- response pairs' used to answer questions relating to identity such as "Who are you?" and "Who cre- ated you?". This dataset is also included as part of Aturibute Conditioned SFT training.
'Adapted from https://gite thub. com/1m hom/Im-aae sys/ FastChat/blob/vo. 2. http///yyoond////cooooopoo jummy-jjoo json

---

Response Generation In accordance with the methodologies described in Peng et al. (2023) and Dettmers et al. (2023), we employ the GPT-4 model to conduct an evaluation of our proposed ap- proach using the Vieuna benchmark (Chiang et al., 2023). This benchmark comprises a diverse set of 80 single-turn prompts encompassing various domains, including knowledge-intensivy question answering, historical counter-factuals, long-form document writing, commonsense questions, role- plays, fermi problems, general knowledge ques- tions, coding, and math problems. Each model is tasked with generating a response with a max- imum sequence length of 1024, utilizing default hyperparameters. For all the STEERLM models and OASST LLaMA 30B models, the decoding strategy employed is greedy decoding. Addition- ally, for STEERLM model responses, the attributes "quality" and "helpfulness" are fixed at value of 9, while all other attributes are set to In the case of Guanaco 65B, generations are obtained using top p = 0.9 sampling with a temperature of 0.7, and Vicuna 13B uses a temperature of 0.7².
Model
ChatGPT
Model Score
of
Name
Score
ChatGPT
STEERLM 43B
629.25
655.75
104.2
STEERIM 13B
617.75
633
102.6
Guanaco 65B
631.25
646.25
102.4
ChatGPT 3.5
100.0
Vicuna 13B
641.75
636.75
99.2
IJ LAA 30B RI.HF
650,25
612.75
94.2
LLLMMA 30B SFT
631.5
610
93.2
text-davinci-003
665.25
599.5
Table Automatic evaluation. Evaluations are highly consistent with +/- 0.2% difference across evaluation runs on identical model generations.
Human Evaluation To minimize the risk of an- notator fatigue and the potential for hasty evalu- ations, we partitioned the 80 prompts within the Vicuna Benchmark into four distinct groups, with each group consisting of 20 prompts. This partition- ing strategy helps ensure that the human annotators (12 in total) are able to provide thorough evalua- tions without skimming through the responses, as can be demanding to compare multiple responses simultaneously. Volunteer human annotators were specifically chosen for this task instead of utiliz- ing crowd workers, such as Amazon Mechanical Turkers, it enables to carefully control for annotators with a university education and cod- ing background, which are crucial for effectively evaluating these model responses. Consequently, this approach resulted a higher degree of consis- tency among our annotators (Fleiss' к = 0.46) comparison to similar studies that employed crowd workers (Dettmers et al., 2023).
Automatic Evaluation In evaluating each prompt, the prompt alongside the responses made by the two competing models are given to GPT-4 (Chiang ct al., 2023; Dettmers et al.. 2023). GPT-4 is then required to give a score between and 10 for each response. While we only report scores for comparing each model against ChatGPT 3.5, we compare every pair of models against each other for subsequent use in ELO rating calculations. The cumulative score obtained over the 80 questions for each model is then compared to the cumulative score achieved by ChatGPT 3.5. This facilitates the assessment of each model's performance as percentage of ChatGPT 3.5's performance. It is important note that the ordering of the responses from the two models during evaluations can influence the evaluation results, as previously observed by Dettmers et al. (2023). To mitigate this potential bias, we calculate the mean performance of each model in both possible response orderings.
However, this choice limited the number of models that we could compare, and we selected the three best-performing models from automatic evaluations for the human evaluation (excluding STEERLM 13B given its similarities STEERLM 43B). The prompt along with the responses from the different models were shown to the annotators, and they were asked to rank them in order of pref- erence (using the UI in Appendix $A.5). During the annotation process, the human annotators re- mained unaware of the specific model used for each response, and to prevent any potential bias, the der of the responses was randomly shuffled. An- notations were carried out utilizing an A/B forced choice approach, requiring the annotators to subjec- tively determine their preferred response between the given options.
*Guanaco 65B generations at https: //github. com/ar httibor//poorr//ll/mmaai//eeeeee http:///an/eeee/ggrraitins///
vicuna/ htt///5bbbgae guuanac-viicua-geee aenennagggeeraaaiinn atirns- topp@.
9- tempo.7 7. 7.jsonl and Vicuna 13B generations at https:
thub com/lm- htt////stchat/bll//aai////////// fastchat/ eval/table/ attaal/answwereeeeeee wwwwaanww//aaeeee httpwer//nswwrr.reee vicuna- 13b. jsonl

---

smaller base model Elo rating Using both automatic and human pair- wise model comparisons, we calculate an Elo rating for each model to show the performance relative to all other models. For each prompt, scores are first converted into a tie, win or loss between two models based on their scores. Following Chiang et al. (2023) and Dettmers et al. (2023), we start with score of 1000 and K 32 and repeat the procedure 10000 times to account for the order in which model pair comparisons are used to calculate their ELO rating. Additionally, we also present the expected win rate against ChatGPT 3.5, for better interpretability.
Model
ChaiGPT
Model
%
Name
Score
Score
CharGPT
Baseline
663.50
528.50
79.7
Human Annolated Attributes
644,00
619.50
96.2
HQ OASST (human annotated)
646.75
634,75
98.1
Aurihute Pred. Model
632.50
649.75
102.7
HH -LFFM-III--SDD
623.25
646.00
103.7
Bootstrapping on HQ samples
629.25
655,75
104.2
Table 3: Ablation study of model variants under auto- matic evaluation.
than other counterparts, explaining the 74 Elo point difference between STEERLM 43B and the closest baseline model on Automatic Evaluations. The ad- vantage of STEERLM 43B on Human Evaluation is slightly lower (59 Elo point difference), suggesting that only slight bias of GPT4 Automatic Eval- uations towards longer and more informative re- sponses. Nonetheless, both Automatic and Human Evaluation show clear strength STEERLM 43B in answering open-ended questions that require cre- ative and verbose responses, which composes ma- jority of the prompts in the Vicuna benchmark. 50
4.5
Results
Model
95%
Win Rate (%)
Name
Rating
CT
vs CharGPT 3.5
Automatic Evaluation
STEERLM 43B
1139
1048-1224
66
STEERLM 13B
1110
1022-1198
62
Guanaco 65B
1065
974-1153
56
CharGPT 3.5
1023
986-1108
Vieuna 13B
1001
911-1091
LLaMA 30B RLHF
925
835-1017
36
Relative to Guanaco 65B, our model performs better despite being trained
LLaMA 30B SFT
935
843-1028
37
text-davinci-003
800
712-893
(43B vs. 65B) that has been trained fewer kens (T.IT vs. 1.4T). This is likely due more efficient use of OASST data, which both models are trained on. Briefly, Guanaco 65B only uses high quality data, defined as the top response (in terms of annotated quality) at every level of the conversation tree. In contrast, STEERLM 43B uses all of Open- Aesisttsstttnnt data, including low low-quality conversations. We will show the advantage of the STEERLM approach in the ablation studies in $5, This enables STEERLM achieve an effect sim- ilar to RLHF by being able to learn from both re- sponses of high and low quality, as opposed to only selected high-quality conversations in regular SFT (Dettmers et al., 2023; Wang et al., 2023a; Zhou et al., 2023; Peng et al., 2023; Köpf et al., 2023b; Chiang et al., 2023). The small difference in perfor- mance between the OASST LLaMA 30B SFT and RLHF models supports the noted difficulty of get- ting RLHF to work well in aligning LLMs (Ouyang et al., 2022: Bai et al., 2022; Köpf et al., 2023b), which prevents widespread adoption of RLHE.
Human Evaluation
STEERIM 43B
1040
951-1126
ChatGPT 3.5
981
897-1070
50
Guanaco 65B
977
890-1064
50
Table 2: Elo Ratings for Models based on Automatic and Human Evaluation.
Based on Tables 1 and 2, our STEERLM 43B model out-performs all baseline models on both automatic and human evaluations. STEERLM 43B performs slightly better than STEERLM 13B, which is expected given the larger base model size (Ouyang al., 2022; Touvron al., 2023). Analy- sis of the responses generated by STEERLM 43B shows that provides longer, more complete an- swers (mean 1906 characters) with more unique words (mean = 144) compared to the other base- lines (e.g. ChatGPT 3.5 has 1193 characters and 77 unique words). Please refer to Appendix $A.6 for examples and $A.7 for average response lengths for each model.
Automatic evaluation with GPT-4 has tendency prefer longer responses that have more unique tokens (Dubois et al., 2023; Wang et al., 2023a). STEERLM 43B responses satisfy both criteria and this might lead to GPT-4 rating its responses higher
5 Ablation Study
In order to identify the contribution of each indi- vidual component to the overall performance, we perform a systematic ablation process (Table 3),

---

starting from baseline that is finetuned the en- tire OASST dataset without using any information about attributes.
Model Name
Toxicity Value
Avg. Score
STEERLM 43B
0.06346
0.08635
0.09633
Addition of attribute labels The addition of at- tribute labels in the fine-tuning process leads to significant increase in performance, underscor- ing the pivotal role of attribute labels, particularly the quality attribute, as the primary contributor to improved performance (16.5%)
0. 1.10008
ChatGPT 3.5
0.06622
Table 4: Average Perspective API Toxicity score on the Anthropic Red Team Dataset.
Steerability demonstration
Finetuning only High Quality Data
We ob-
serve that Attribute Conditioned Supervised Fine- tuning on small subset of 3400 samples from the OASST dataset that have the highest value for qual- ity (human annotated) leads to an increase in perfor- mance by up to 1.9%. This observation aligns with prior research indicating that training models on a larger volume of low-quality data yields inferior results compared to training on smaller quantity of high-quality data (Dettmers al., 2023; Zhou al., 2023).
To demonstrate the efficacy of STEERLM 43B control its generations in conduct an empirical study incorporating the at- tributes of toxicity and humor.
way, we
6.1 Toxicity
To assess the ability of STEERLM 43B to vary its responses based on the value of toxicity specified, we use the Anthropic Red-team dataset² (Ganguli et al., 2022). This dataset contains prompts ranked from 0 4, with higher rank indicating more inappropriate responses. We randomly select 100 conversations with ranking of and focused on the initial turns of these interactions.
Utilizing predictions from the Attribute Predic- tion model for the Attribute Conditioned SFT process provides a substantial benefit to STEERLM 43B amounting to 4.6% in performance, relative to using human annotations. This empirical evidence substantiates hypothesis that employing an at- tribute prediction model aids effectively calibrat- ing the quality annotation process across multiple training samples, thereby mitigating noisy labels that are unavoidable when using crowdsourced an- notations.
To compare the toxicity levels of generated out- puts, we vary the toxicity parameter in STEERLM 43B at four different settings: 0 (default), 3, 6, and (highest toxicity). We employ the Perspective API, a widely-used toxicity classification model, score the toxicity the generated responses and compare the average toxicity of STEERLM 43B to the responses from ChatGPT-3.5.
Augmentation of training data with Anthropic HH-RLHF contributes an additional 1.0% in- crease in performance compared to using Open- Assistant data alone. This shows the capability of our Attribute Prediction model to identify and an- notate high-quality data from other datasets apart from the one it was trained on. This suggests that the methodology of STEERLM can be applied to label different kinds of datasets, enhancing the di- versity training data without incurring the sub- stantial expense associated with human annotation for every individual data sample.
Our findings (Table 4) indicate that when the toxicity value is set to 0 (default), STEERLM 43B exhibits slightly lower toxicity compared to Chat- GPT. However, STEERLM 43B offers the flexibil- ity to generate more toxic outputs by increasing the toxicity value above This ability to control gen- eration attributes at inference can prove valuable applications such as generating Non-Player Char- acter (NPC) dialogue in games and red-teaming purposes.
6.2 Humor
Recent studies (Jentzsch and Kersting, 2023) inves- tigating the humor capabilities of language mod- els have primarily focused on the aspect of telling jokes. However, humor can manifest in various
Bootstrapping with High-Qualit Samples re- sults in a performance gain of 0.5%. Our hypothe- sis is that the improvement performance is due to the data augmentation through sampling, which ex- plores the response space and augments the dataset with higher-quality data.
Accessed at https::/huggingtte http:/aacccccooaaaaeeeeee/eeo co/datasets/ sabrropc/hh-rrllf/treeeeeee http://maaiinddddeeeaeeeeeeae team-attempts

---

contexts. In this experiment, we employ two dif- ferent prompts "Tell me joke" " and "Whar's 's good way to spend a day?". , to compare the humor exhibited by STEERLM and ChatGPT-3355
Automatic and manual evaluation have been per- formed on a English-only benchmark and we would need to evaluate our models on multilingual bench- marks to test the extent to which the positive results we find with STEERLM on an English-based bench- mark extends to evaluations with other languages.
"Tell me a joke" ChatGPT-3.5 and STEERLM 43B, with humor set to 9, successfully generate a joke. When the humor attribute in STEERLM 43B is set to 6 or lower, it responds with "I don't 't know any jokes.. This characteristic can be advataggouss for chatbots that need to maintain a formal persona and when humor is not appropriate.
Ethies Statement
STEERLM enables user capabilities to ask for toxic and/or violent responses during run-time. This is valid use case in certain scenarios like games where a character should respond differently based situation in red teaming use cases. However, malicious users might exploit this feature outside of its intended use-cases. To address this, it is pos- sible to give users controls over only a subset of attributes at model run-time, unless they demon- strate an acceptable use-case. In general, we be- lieve that giving developers explicit control over model's value system is preferred to dictating one that they can not adjust.
"What's a good way to spend a day?" Both Cha1GPT-3.5 and STEERLM 43B, with humor at- tribute at low values (0-3) provided an extensive list of activities. Tuning up humor to 6 leads to STEERLM 43B appending the list cheeky re- mark. "But don't forget to brush your teeth and go to bed at reasonable hour!". With humor set to the maximum value of 9, it leads to a cheesy statement "By spending with you, of course!". Thus, depending on the intended use-case the same STEERLM LM model can be turned into versatile engaging conversational agents.
A Acknowledgments
7
Conclusion
We would like to thank Abhinav Khattar, Adi Ren- duchintala, Neel Kant, Sandeep Subramanian and many others at NVIDIA as well as anonymous viewers for their helpful comments. In addition, we appreciate the authors, members and contrib- utors of the Open Assistant Project (OASST) for assembling and excellent open data set for model alignment which made this work possible.
We introduce STEERLM, a novel model alignment approach with a value system (e.g. humor level and toxicity tolerance) that can be adjusted by users at inference time without re-training. STEERLM trains both the attribute prediction model and the language model using only supervised fine-tuning, resulting in an tosplee-mpll and straightfor- ward training process compared to using RLHF We train STEERLM models following this proce- dure, achieving state-of-the-ar results the Vi- cuna benchmark. We validate these results with human evaluation and find that STEERLM pre- ferred over the other models we compare it to. We hope our work will inspire further research into developing simple and effective model alignment methods that empower better AI assistants for ev- eryone.
References
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kermion, Tom Conerly, Sheer El-Showk. Nelson Elhage. Zac Sattield-Dodds Danny Hernandez, Tristan Hume, Scott Johnston. Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. 2022. Training a help- ful and harmless assistant with reinforcement learn- ing from human feedback.
Limitations
The Attribute Prediction Model and Attribute- Conditioned SFT Models in STEERLM are fully supervised fine-tuned, making it relatively costly in terms of GPU hours and energy compared with Parameter Efficient Fine-Tuning algorithms such as Low-Rank Adaptation and Prompt Tuning tech- niques.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess,

---

Jack Clark, Christopher Bemnr, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray. Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Ar- avind Srinivas, and Igor Mordatch. 2021. Decision transformer: Reinforcement learning via sequence modeling.
Oleksii Kuchaiev, Jason Li, Huyen Nguyen, Oleksii Hrinchuk, Ryan Leary. Boris Ginsburg. Samuel Kri- man, Stamislav Beliaev, Vitaly Lavrukhin, Jack Cook, Patrice Castonguay, Mariya Popova, Jocelyn Huang, and Jonathan M. Cohen. 2019. Nemo: a toolkit for building ai applications using neural modules.
Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, Ion Stoica, and Eric Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* * chatgpt quality.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc. Oliver Stanley, Richard Nagyti, Shahul ES, Sameer Suri, David Glushkov, Arnv Dantuluri. Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexan- der Mattick. 2023a. Openassistant conversations democratizing large language model aligmment.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Mar- tic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information P Processing Systems, volume 30. Curran Associates, Inc.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens. Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arna Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexan- der Mattick. 2023b. Openassistant conversations democratizing large language model alignment.
Mike Conover, Matt Hayes, Ankit Mathur, Xiangrui Meng, Jianwei Xie, Jun Wan, Sam Shah, Ali Gh- odsi, Patrick Wendell, Matei Zaharia, and et al. 2023. Free dolly: Introducing the world's first truly open instruction-tuned Ilm.
David Lopez-Paz, Diane Bouchacourt, Levent Sagun, and Nicolas Usunier. 2022. Measuring and signing fairness as performance under multiple stakeholder distributions. arXiv preprint arXiv:2207.0900 099900.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized Ilms.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023. Raft: Reward ranked finetuning for generative foundation model alignment.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xi- ubo Geng, Wenxiang Hu, Chongyang Tao. Jing Ma. Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol- instruct.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani. Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpaca- farm: A simulation framework for methods that learn from human feedback.
Fundamental AI Research Meta, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Fla- herty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi. Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus Zijlstra. 2022. Human-level play in the game of diplomacy by combining lan- guage models with strategic reasoning. Science, 378(6624): 1067-1074
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Con- erly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodee Tom Henighan, Danny Hernande7, Trrstann Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. 2022. Red teaming language models to re- duce harms: Methods, scaling behaviors, and lessons learned.
Marcos Nadal and Anjan Chatterjee. 2019. Neuroaes- thetics and art's diversity and universality. Wi- ley Interdisciplinary Reviews: Cognitive Science. 10(3):e1487.
Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse. Shantanu Jain, Vineet Kosaraju, William Saunders. Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. Webgpt: Browser- assisted question-answertng with human feedback.
Sophie Jentzsch and Kristian Kersting. 2023. Chatgpt fun, but it is not funny! humor is still challenging large language models.

---

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher. Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Pumit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Di- ana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- bog. Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu. Zheng Yan, Iliyan Zaroy, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez. Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine- tuned chat models.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Gal- ley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model.
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open- domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 5370-5381, Florence. Italy, Association for Computational Linguistics.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan. Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023a. How far can camels go? exploring the state of instruction tuming on open resources.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani. Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Mattco Manica. Sheng Shen. Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers. Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization.
Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b. Self-instruct: Aligning language models with self-generated instructions.
Zhilin Wang, Xuhui Zhou, Rik Koncel- Kedziorski,,,, Alex Marin, and Fei Xia. 2022. Extracting and inferring personal attributes from dialogue. In Proceedings of the 4th Workshop on NLP for Conversational Al, pages 58-69, Dublin, Ireland. Association for Com- putational Linguistics.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoc V. Le. 2022. Finetuned language models are zero-shot learners.
Eric Michael Smith, Diana Gonzalez-Rico, Emily Di- nan, and Y-Lan Boureau. 2020. Controlling style in generated dialogue.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang. Songfang Huang, and Fei Huang. 2023. Rrhf: Rank responses to align language models with human feed- back without tears.
Charlie Snell, Nly Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. 2023, Offline rl for natural lan- guage generation with implicit language q learning.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Qiao. 2023. Llama-adapter Efficient fine-tuning of language models with zero-init attention.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler. Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2022. Learning to summarize from human feedback.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam. Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: have dog. do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 2204-2213, Melbourne, Australia, Association for Computational Linguistics.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction- -ollowiinn Ilama model. https:// gi thub. com/ tatsu-lasuuub /s

---

Chunting Zhou, Pengfei Liu,, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu,
Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023, Lima: Less is more for alignment.
104
Quanaco
chatami
A Appendix
90
A.1 Attribute Prediction Model prompt templates
We the following prompt template train the value model. The encoded value string is marked red and used to calculate LM loss. The <ex- tra id _,,t0,,,xxxx_____ ciexxtrr_dd_______ and <extra id _>_>>>e are spe- cial tokens included in the base LM tokenizer. An example of linearized value attributes is shown in the Sec. 3.1.
92-
10
20
30
40
50
60
Peramsts)
Figure 3: Effect of Model Size
A.3 Further Data Collection
<extra_ine ii_ id_0@Sytteemmm
[system prompts]
While we did not collect new data in the current study, users who would like to collect new data will find the annotation cost of SteerLM to sub- stantially lower compared to RLHF. SteerLM only requires each prompt-response sample to be anno- tated with attributes of their interest whereas RLHF requires pairwise comparison between responses per prompt (where k=4-9) or effectively (9) to (2) 36 annotation per prompt (Ouyang et al., 2022). RLHI also relies on an iterative data col- lection methodology, where the critic must be con- tinually retrained as the actor (language model) improves (Ouyang al.. 2022; Bai et al., 2022: Touvron al., 2023). In contrast, SteerLM only re- quires a one-time data collection, as each response is rated independently on an absolute scale rather than compared to other responses. Therefore, the annotation cost of collecting dataset for SteerLM is approximately an order of magnitude lower than RLHF.
(ectra_id@@>(usstt name]
[user prompts]
<extra_id_1>> (taasissatt name]
[assistant response]
Eoxtra_id_><[encone value attributes]
<extra_id_@> [userr name]
[user prompts]
Lacira__dd_"aassissssi name] [assistant response]
<extra_id__>_encodee value attributes]
A.2 Attribute Conditioned SFT prompt templates
The SteerLM ACSFT prompt templates is similar the Attribute Prediction Model prompt template shown the Sec. A. but we swap the position of assistance response and encoded value string. The LM loss is calculated on the assistant responses as highlighted in red.
<extra_id__ _i_ra__ddd_Syttmmmmmmmmme
[system prompts]
A.4 Efffec of model size on performance
<extra_id_@>> [idd__d[[u__ @>[user name]
[user prompts]
<extra_id_1> 1'[ssistattt name]
To better understand the effect that model size plays in model performance, we also trained SteerL on smaller base models (8B and 22B), which have pre- trained similarly as the 43B base model. We then compare to smaller versions (7B, 13B and 30B) of the best performing baseline model (Guanaco 65B). As shown in Figure 3, SteerLM outperforms Guanaco at every comparable model size.
value attributes]
[assistant response]
<extra_id_@><user name]
[user prompts]
iacira__dd_>(assiissse name]
<extra_id_2<(encodee value attributes]
[assistant response]

---

was www.e/yyunnnouuu anotac andeai liigh ou longuet
neve medeoe beight nding moyet hon quet lnv introdes rypee hype with the rolow rewerdd
lopalin
guorted steedd bancuut
to dgn't (heatyhl Theary
ters to repasset my kingine andto snow Hopilyy gingl Hey grunted the plullgg to soaa with (ooal Persorapt abulen matee cuss
impertatios
iranduction be
iraeliig boalinngg before
upal moninge age indermee ing gitt siee of ure iiss. the higgh sewill alsa eepesttdd to barkowe armcirntta lomoialio throughun our luqqurt gaeats
impucad woti ausataano aduicon rouil
none madion tright innuddieerdd myse nay emprr tiee ranas brover, lopaly to Homb
seadiv ding beiie und rndemmns wr de four Homel bright the bou und hand housatclee carsa lnn thae bee
sumbur young have logght many baciee hure
locerel oarial banannt a pllnder en alling sosr Your food Maunnd san audl foste
insperty
Gosd coring your peamaajnyyy eutle ane udine jiras samal hhe enteepeeett
oa bagent egiin..
Figure 4: Human Annotation UI
A.5 Human Annotation UI
ercise regularly and eat healthy foods keep your energy levels up.
A.6 Example of Generations by SteerLM 43B and ChatGPT 3.5
Stay connected: Communicate with your col- leagues and clients regularly. Use video calls and instant messaging to stay in touch and avoid misunderstandings.
Question: How can increase my productivity while working from home?
SteerLM 43B response: Here are some tips in- crease your productivity while working from home:
Set clear goals: Set specific and achievable goals for cach day or week. Break down large tasks into smaller ones and prioritize your tasks based on importance and urgency.
1. Create dedicated workspace: Having dedi- cated workspace can help you focus and avoid distractions. Choose quiet and comfortable spot in your home and set up your workspace with all the necessary tools and equipment.
7. Use productivity tools: There are many pro- ductivity tools available that can help you stay organized and focused. Use tools like cal- endars, to-do lists, and project management software to keep track of your work.
2. Stick schedule: Establish daily routine and stick it. Set specific hours for work and stick to them. Take breaks at regular in- tervals and schedule time for lunch and other activities.
8. Take breaks: Taking regular breaks can help you stay focused and productive. Take short break every hour or to stretch your legs, get some fresh air, or grab snack.
3. Eliminate distractions: Minimize distractions by turning off your phone, closing unneces- sary tabs on your computer, and letting your family and friends know that you are working and cannot be disturbed.
9. Stay motivated: Find ways stay motivated and inspired. Set short-term and long-term goals, celebrate your successes, and find ways to reward yourself for a job well done.
4. Take care of your physical health: Take reg- ular breaks to stretch and move around. Ex-

---

10. Be flexible: Working from home can be chal- lenging, so be flexible and adapt changes as they arise. Be open to new ideas and ap- proaches, and don't be afraid to ask for help when you need it.
A.7 Response lengths of various models
Model
Characters
Unique Words
STEERLM 43B
1906
144
STEERLM 13B
1719
146
Guanaco 65B
1648
145
ChatGPT 3.5 response: There are several ways to increase productivity while working from home:
ChatGPT 3.5
1193
77
Vicuna 13B
1417
125
LLaMA 30B RLHF
1219
108
1. Establish routine: Start and end your work at specific time each day, just like you would if you were working at an office.
LLaMA 30B SFT
1280
106
text-davinci-003
842
87
Table 5: Mean response length for each model in charac- ters and number of unique whitespace-separatte words.
2. Create a designated workspace: Set up workspace that is free from distractions and has everything you need to complete your work.
3. Prioritize your tasks: Make to-do list at the beginning of each day and prioritize your tasks based on their importance.
4. Take breaks: Take regular breaks rest your mind and stretch your body, as this can help improve your focus and productivity.
5. Minimize distractions: Avoid distractions such as social media, TV, houschold chores during your work hours.
6. Use productivity tools: There are several pro- ductivity tools available online, such as time trackers, task managemeet apps, and project- management software, which can help in- crease your productivity.
7. Communicate with colleagues: Keep in touch with your colleagues regularly to stay con- nected and avoid feeling isolated.
Remember, everyone's productivity habits are diffcrent, so find what works best for you and stick to it.


Reviews:
[{"review_id": "abfvIbOmfP", "reviewer": "Reviewer_kdwP", "summary": "", "questions": "1. How does the data collection cost for attribute conditioned SFT compare to RLHF? It is important to consider the annotation costs associated with RLHF.\n2. Regarding the SteerLM pretrained model, how can we determine the relative contributions of the pretraining phase and the proposed attribute conditioned SFT?\n3. Is it more appropriate to consider attribute conditioned SFT as an alternative to SFT, or does it serve a different purpose?\n4. In addition to the Vicuna evaluation, could the paper include more evaluation results specifically focusing on toxicity and hallucinations?\n5. Providing more examples related to toxicity and hallucinations would enhance the understanding of SteerLM's performance in these areas.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your thoughtful feedback on our paper. We appreciate you taking the time to provide constructive criticism to help strengthen our work. Please find our responses below:\n\n\n\n>The proposed method may introduce challenges in collecting alignment data, potentially increasing the difficulty of the alignment process.\nHow does the data collection cost for attribute conditioned SFT compare to RLHF? It is important to consider the annotation costs associated with RLHF.\n\nWe appreciate the reviewer highlighting the importance of considering annotation costs when comparing alignment approaches. SteerLM does not require any additional annotations since we only use existing annotations from the open-sourced Open Assistant dataset. Taking a step further, if other researchers wish to apply the SteerLM methodology by training on another dataset, the annotation cost of SteerLM will be drastically lower compared to RLHF. \n\nThis is because SteerLM only requires each prompt-response sample to be annotated with attributes of their interest (can be as few as 1 - for overall quality) whereas RLHF requires pairwise comparison between k responses per prompt (where k=4 to 9) or effectively $ 4 \\choose 2 $ to $ 9 \\choose 2 $ = 6 to 36 annotation per prompt (Ouyang et al., 2022 [1]). Therefore, the annotation cost of collecting a dataset for SteerLM is approximately an order of magnitude lower than RLHF. \n\nRLHF also relies on an iterative data collection methodology, where the critic must be continually retrained as the actor (language model) improves. This is because the actor can easily fool the outdated critic, requiring new human preference rankings for the updated actor in order to train an improved critic (Ouyang et al., 2022 [1]). In contrast, SteerLM only requires a one-time data collection, as each response is rated independently on an absolute scale rather than compared to other responses. So while RLHF needs repeated data collection and critic retraining as the actor evolves, SteerLM provides a more efficient training approach with a single round of human ratings that do not need to be updated as the model improves.\n\n\nWe will add these discussions in the camera ready version to further clarify the significant annotation cost advantages of our approach compared to RLHF. \n\n\n\n\n>It remains unclear whether the superior performance is primarily attributed to the 43B pretrained model or the attribute conditioned SFT.\nRegarding the SteerLM pretrained model, how can we determine the relative contributions of the pretraining phase and the proposed attribute conditioned SFT?\n\n\nWe acknowledge that our original submission does not sufficiently clarify the impact of the base model vs the alignment methodology. To address this concern, we have conducted some additional experiments which provide evidence that the performance improvement comes from the proposed SteerLM methodology rather than from having a better pretrained model. \n\nSpecifically, we evaluated our 43B model by SFTing on the Guanaco dataset and it achieved a performance of 98.4% of ChatGPT-3.5. On the other hand, training LLaMa 65B on the same dataset achieved a performance of 102.4% of ChatGPT (Dettmers et al., 2023 [2]). This demonstrates that the superior performance cannot be attributed to the base model, and should instead be attributed to the SteerLM methodology.  \n\nAt the time of conducting the experiments before the submission, we could not access and use the publicly available models which are only licensable to non-commercial settings. However, since then, there have been open-source models that have been released. In order to also show the generalizability of our proposed methodology, we use LLaMa 2 - 13B as our base model and apply our SteerLM methodology. Evaluating it on the Vicuna benchmark, we find that it gets a score that is 102.6% of ChatGPT-3.5 score, which is not far off the SteerLM 43B (104.2%), a reasonable difference given the model size disparities. This further supports that the superior performance can be attributed to the SteerLM methodology.\n\nIn the camera ready version, we will include these new experimental results and analysis to better isolate the contributions of our approach. \n\n\n\n>Is it more appropriate to consider attribute conditioned SFT as an alternative to SFT, or does it serve a different purpose?\n\t\nWe position attribute conditioned SFT as an augmentation of regular SFT that enables incorporating reward signal information through attribute labels. This allows learning from both high and low quality responses in a manner similar to the established SFT+RLHF pipeline (Ouyang et al., 2022 [1]). In the camera-ready version, we will provide further clarification on the similarities and differences between our proposed approach and regular SFT. \n\n\n\n\n>It is important to verify the proposed method on other benchmarks or approaches to determine its generalizability.\nThe paper would benefit from additional evaluation results. Existing benchmarks lack evaluations specifically related to toxicity and hallucinations, which are crucial aspects for assessing alignment effectiveness.\nIn addition to the Vicuna evaluation, could the paper include more evaluation results specifically focusing on toxicity and hallucinations? Providing more examples related to toxicity and hallucinations would enhance the understanding of SteerLM's performance in these areas.\n\n\n\n\nWe have included in section 6.1 a brief case study on how we evaluated SteerLM on the toxicity of its generations when the attribute of toxicity is explicitly controlled for. Additionally, we agree with the reviewer on the need for further evaluations targeting hallucinations. However, there has been a lack of well-formulated, stable benchmarks that evaluate for such criteria as pointed out by the reviewer themselves. Thus we would like to state that this is not an appropriate reason for rejection for the contributions of our work.  The main contribution of our work is a simple, easy-to-use method for aligning language models. While the claim of this paper is not to propose a method that can significantly reduce toxicity and hallucination compared to existing approaches, we feel that it is easy to control such attributes using SteerLM and can be explored in future work.\n\n\n>Conclusion\n\n\nThank you again for the helpful feedback, which strengthens our paper.  We request the reviewer to look more favorably on our paper and increase the scores since some of the reasons for rejection are open research problems in the field and are not directly related to our main contributions.\n\n\n>References\n\n\n[1] Ouyang et al. (2022) Training language models to follow instructions with human feedback\n\n[2] Dettmers et al. (2023) QLoRA: Efficient Finetuning of Quantized LLMs", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes an alternative method for model alignment, providing a different approach to RLHF.\n2. The proposed SteerLM approach demonstrates strong performance, surpassing existing models on the Vicuna benchmark.", "reasons_to_reject": "1. The proposed method may introduce challenges in collecting alignment data, potentially increasing the difficulty of the alignment process.\n2. It is important to verify the proposed method on other benchmarks or approaches to determine its generalizability. It remains unclear whether the superior performance is primarily attributed to the 43B pretrained model or the attribute conditioned SFT.\n3. The paper would benefit from additional evaluation results. Existing benchmarks lack evaluations specifically related to toxicity and hallucinations, which are crucial aspects for assessing alignment effectiveness.", "questions_to_authors": "1. How does the data collection cost for attribute conditioned SFT compare to RLHF? It is important to consider the annotation costs associated with RLHF.\n2. Regarding the SteerLM pretrained model, how can we determine the relative contributions of the pretraining phase and the proposed attribute conditioned SFT?\n3. Is it more appropriate to consider attribute conditioned SFT as an alternative to SFT, or does it serve a different purpose?\n4. In addition to the Vicuna evaluation, could the paper include more evaluation results specifically focusing on toxicity and hallucinations?\n5. Providing more examples related to toxicity and hallucinations would enhance the understanding of SteerLM's performance in these areas.", "paper_topic_and_main_contributions": "The paper introduces SteerLM, a novel approach for model alignment that allows users to adjust the value system (e.g., humor level and toxicity tolerance) during inference without requiring re-training. The authors utilize a specified supervised fine-tuning (SFT) method for implementing alignment. They trained a 43B model using SteerLM instead of RLHF and achieved state-of-the-art results on the Vicuna benchmark.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LkoTQhlLH6", "reviewer": "Reviewer_PUsH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for the thoughtful feedback on our paper submission. We appreciate you identifying areas where we can provide additional details to strengthen the work. Please find our responses below:\n\n\n>More details are needed for the base model. What is the base performance? How much compute does it costs? Are there any public access? Because it is unclear about the base model, we do not know how much improvement has been made by the proposed method.\n\nWe agree that including more details about the base model can help to better clarify the contributions of SteerLM methodology. At the time of conducting our experiments prior to EMNLP submission, we could not access publicly available models with competitive performance, since they are only licensed to non-commercial settings. Therefore, we had to develop our SteerLM recipe using an in-house model. \n\nHere are details of our in-house 43B model:\n\n|             | In House 43B | Llama 65B (used by Guanaco 65B) |\n|-----------|--------------|---------------------------------|\n| MMLU      | 55.3         | 63.4                            |\n| ARC-Easy  | 76.6         | 78.9                            |\n| HellaSwag | 80.1         | 84.2                            |\n\n\nCompute: $2.9 * 10^{23}$ FLOPs\n\nPublic access: Currently, 43B model is not publicly accessible and can only be accessed by our enterprise users.\n\nTo better understand the contributions of the SteerLM methodology to the performance of our model, we also trained a 43B model on the Guanaco dataset used by Guanaco 65B. Guanaco 43B performs at 98.4% of ChatGPT 3.5, which is much lower than 102.4% achieved by Guanaco 65B (with Llama 65B base model). This suggests that training on the inhouse 43B model puts us at a disadvantage compared to training on Llama 65B, as also supported by the lower performance of our base model on MMLU, ARC-Easy and HellaSwag. In other words, the SteerLM approach contributes to the good performance on the Vicuna Bench, in spite of a weaker 43B base model used. \n\nSince our EMNLP submission, models with licenses permitting use by commercial labs (e.g. Llama 2) have been released. Therefore, we are working towards using our SteerLM recipe on publicly accessible models prior to releasing the camera-ready version of our paper. As part of that effort, we have trained a SteerLM Llama2 13B model which reports 102.6% of ChatGPT 3.5 performance on Vicuna Bench. This performance is not far off the SteerLM 43B (104.2%), which is reasonable given the model size disparities. This supports how SteerLM approach can be used to align various base models, independent of the contribution of the base 43B model. \n\n\n\n>There are 13 attributes in OASST, why do the authors choose 7 out of 13?\n\n\nWe chose 7 attributes based on their relevance for steering the model's responses during inference to match human preferences. Other attributes such as hate_speech, lang_mismatch and pii (personal identifiable information) are not useful to steer at inference time since these are attributes that we always want to keep as False. We will take care to make this decision clearer in our paper.\n\n\n\n\n\n\n>The method is not new. It is needed to discuss the differentiation on the method with RAFT [1] and RRHF [2]. Especially, an important and similar work is omitted [1].\n[1] RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment https://arxiv.org/abs/2304.06767 [2] Rrhf: Rank responses to align language models with human feed- back without tears https://arxiv.org/abs/2304.05302\n\t\n\n\nThank you for bringing these papers to our attention. After better understanding these papers, we found areas of differences that make SteerLM distinct from RAFT and RRHF. We discuss these differences below and will include this discussion in our camera-ready version.\n\n\nSteerLM conditions generation on explicit, multi-dimensional attributes that can be specified by the user at inference time. This enables user-steerable responses, which can be useful in settings such as generating game dialogues, in which some NPCs have toxic/unhelpful/humorous persona.  In contrast, RAFT and RRHF optimize a scalar reward signal without explicit conditioning on attributes, limiting its user-controllability at inference time.\n\nSteerLM trains the model to generate responses of both high and low scores in each attribute in steps 1 to 3, meaning that models learn to generate responses conditioned on both high and low attribute scores. This is important for steerability because users might want responses either high or low on humor and other attributes.. On the other hand, RaFT seeks to train models using only samples with high reward scores (Algorithm 1 Step 2) and RRHF seeks to optimize $L_{rank}$ (Eq 2) and $L_{ft}$ (Eq. 4) to maximize the likelihood of generating responses with high rewards. Therefore, the trained RaFT and RRHF models can only generate responses with high rewards but not low rewards.\n\nSteerLM is evaluated on Vicuna Benchmark, which covers a diverse set of prompts from various domains and shows SOTA performance with 104.2% of ChatGPT 3.5 performance. RRHF only reaches 81.9% of ChatGPT performance on Vicuna Benchmark (Table 9) while RAFT is only evaluated on tasks in narrow domains (specifically movie review generation, dialogue-completion and harmless/helpfulness classification). \n\n\nHowever,  we note that both RAFT and RRHF papers were submitted to Arxiv in April 2023 and are within the contemporaneous 3 month period from the EMNLP submission deadline (23 June 2023). Therefore, they should not constitute an acceptable reason for rejection, based on guidance from EMNLP.\n\n\n```While not citing such unpublished works upon submission is not sufficient grounds for paper rejection, you are expected to cite such relevant work in camera ready, if notified about it by reviewers.```\n\n\nWe will make sure to cite these works and include the discussion. We request the reviewer to increase the score since the EMNLP guidelines state that this is not sufficient grounds for rejection. \n\n\n\n>Conclusion\n\nThank you again for the helpful feedback. We hope that we have clarified your concerns and therefore request you to favorably reconsider the scores for our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The paper proposes a novel approach, SteerLM, which offers a simpler and more user-controlled alternative to RLHF for aligning language models with human preferences.\n\n(2) The methodology is well-structured and easy to understand. The authors provide clear explanations of the attribute prediction model, attribute-conditioned SFT, and bootstrapping with high-quality samples.\n\n(3) The paper includes comprehensive experimental evaluations, including automatic evaluation with GPT-4, human evaluation, and an ablation study, to demonstrate the effectiveness of STEER LM.", "reasons_to_reject": "(1) More details are needed for the base model. What is the base performance? How much compute does it costs? Are there any public access? Because it is unclear about the base model, we do not know how much improvement has been made by the proposed method.\n\n(2) There are 13 attributes in OASST, why do the authors choose 7 out of 13?\n\n(3) The method is not new. It is needed to discuss the differentiation on the method with RAFT [1] and RRHF [2]. Especially, an important and similar work is omitted [1]. \n\n[1] RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment https://arxiv.org/abs/2304.06767\n\n[2] Rrhf: Rank responses to align language models with human feed- back without tears https://arxiv.org/abs/2304.05302", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces SteerLM, a supervised fine-tuning method that allows users to control the responses generated by language models. The authors address the limitations of reinforcement learning from human feedback (RLHF) by proposing a method that conditions responses to conform to a set of explicitly defined attributes. They demonstrate that SteerLM outperforms state-of-the-art baselines trained with RLHF on the Vicuna benchmark, while being easier to train. The paper provides detailed explanations of the methodology and presents results from automatic and human evaluations.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment https://arxiv.org/abs/2304.06767"}, "extra_text": {}}, {"review_id": "IYy4GSbR3c", "reviewer": "Reviewer_QL8N", "summary": "", "questions": "(A) This paper shows superior performance than ChatGPT which is too good to be true. Please provide more details on your evaluation including does it contain math and code related tasks and please report the results on these difficult tasks since we know open-sourced chat models like Alpaca, Vicuna all perform bad on math and code tasks.\n(B) Automatic evaluator like GPT-4 may give a higher score for very long responses, you should provide an average response length for reponses.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for the positive feedback and for raising important questions to help us strengthen the paper. Please find our responses below:\n\n\n>This paper shows superior performance than ChatGPT which is too good to be true. Please provide more details on your evaluation including does it contain math and code related tasks and please report the results on these difficult tasks since we know open-sourced chat models like Alpaca, Vicuna all perform bad on math and code tasks.\n\nIn the current version of SteerLM, we observe improved performance on open-ended generation tasks such as roleplay, creative writing and brainstorming, which characterizes most of Vicuna Bench. While our paper follows standard practice in only sharing the model’s overall performance on Vicuna Benchmark (Chang et al., 2023 [1], Dettmers et al., 2023 [2]), we agree that it is important to analyze performance on these non-textual tasks. We find in our internal experiments that our base 43B model is inferior to ChatGPT-3.5 on the math and coding benchmarks which is not reflected in the automatic eval using GPT-4 on the Vicuna benchmark. \n\nThe weakness in the coding and math categories is common among the open source language models including the recent LLaMa 2 model. Code Llama (Rozière, 2023[5]) shows that this can be fixed by continual training on the massive amount of code data, which is outside the scope of typical model alignment methods.  We are working on improving the math and coding performance of our base LMs and we hope to include some improved results in the camera ready version. \n\n\n\n\n\n\n>Automatic evaluator like GPT-4 may give a higher score for very long responses, you should provide an average response length for responses.\n\n\n\n\nWe appreciate you raising this observation, which echoes findings in literature (Dubois et al., 2023[3] ; Wang et al., 2023 [4]).  Below are the average length of responses from each model, which we will include in the camera ready version. \n\n\n| Model            | Avg. Response Chars |\n|------------------|---------------------|\n| SteerLM 43B      | 1906                |\n| Guanaco 65B      | 1648                |\n| Vicuna 13B       | 1417                |\n| Llama 30B SFT    | 1280                |\n| Llama 30B RLHF   | 1219                |\n| Text-davinci-003 | 842                 |\n| ChatGPT 3.5      | 1193                |\n\n\n\n\nModels with longer responses do perform better on Vicuna Bench, according to the GPT4 automatic evaluator. To make sure this bias towards longer responses is reflective of human preferences, we also performed Human Evaluation of model responses and found that human annotators prefer the longer responses generated by  SteerLM 43B compared to those by Guanaco 65B and ChatGPT 3.5. \n\n\n\n\n>Reproducibility\n\n\nAt the time of conducting our experiments prior to EMNLP submission, we could not access publicly available models with competitive performance, since they are only licensed to non-commercial settings,. Therefore, we had to develop our SteerLM recipe using an in-house model. \n\nHowever, since then, models with licenses permitting use by commercial labs (e.g. Llama 2) have been released. Therefore, we are working towards using our SteerLM recipe on publicly accessible models prior to releasing the camera-ready version of our paper. As part of that effort, we have trained a SteerLM Llama2 13B model which reports 102.6% of ChatGPT 3.5 performance on Vicuna Bench. We will also release code for training various base models using SteerLM after paper acceptance. Together, we hope this can alleviate some of your concerns regarding the reproducibility of our approach. \n\n\n>Conclusion\n\n\nThank you again for the constructive feedback - addressing your comments will significantly strengthen the rigor and transparency of our paper. We look forward to submitting an improved camera ready version. \n\n>References\n\n[1] Chiang et al. (2023) Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.\n\n[2] Dettmers et al. (2023) QLoRA: Efficient Finetuning of Quantized LLMs\n\n[3] Dubois et al., (2023) Alpaca-farm: A simulation framework for methods that learn from human feedback\n\n[4] Wang et al., (2023) How far can camels go? exploring the state of instruction tuning on open resources.\n\n[5] Rozière et al., (2023) Code Llama: Open Foundation Models for Code", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Retraining using bootstrap high-quality data is sound and it makes user can decide the attributes like humor or toxicity\nIt outperforms ChatGPT 3.5 in auto evaluation and human evaluation on Vicuna test.\nDetailed ablation studies proved its effective design.", "reasons_to_reject": "See questions.", "questions_to_authors": "(A) This paper shows superior performance than ChatGPT which is too good to be true. Please provide more details on your evaluation including does it contain math and code related tasks and please report the results on these difficult tasks since we know open-sourced chat models like Alpaca, Vicuna all perform bad on math and code tasks.\n(B) Automatic evaluator like GPT-4 may give a higher score for very long responses, you should provide an average response length for reponses.", "paper_topic_and_main_contributions": "Propose StterLF which first trained an attribute predict model and use it filter high quality data for retraining.\nIt allows user to decode with its user preference based on attributes.\nIt evaluates on Vicuna test set and performs better than ChatGPT 3.5 in auto evaluation and human evaluation.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this work, the authors introduce SteerLM -- a new technique for model alignment that allows for specific attributes to be controlled by the end-users during inference (unlike existing alignment techniques such as RLHF). Experiments show that using SteerLM yields responses that are preferred in both human and automatic evaluations, compared to RLHF, and the training cost is also lower than RLHF.

The reviewers have all rated this work as good or higher on soundness. With moving to a larger model (43B), reviewers asked for more clarity on how to disentangle the positive effect of more pretraining data vs. the proposed attributed-conditioned finetuning. The authors responded with more results on models, that were released post the EMNLP submission, with licenses permitting use by commercial labs. The reviewers also pointed to very recent and relevant references that the authors have promised to include in their revised version. A concern about reproducibility was also addressed with a promise to release the SteerLM recipe on publicly available models.
Decision: Accept-Findings