Title: Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
Venue: NeurIPS 2023
Accepted: yes
Keywords: AI Alignment

Large Language Models

In Context Learning

Neural Symbolics
Abstract: Recent AI-assistant agents, such as ChatGPT, predominantly rely on supervised fine-tuning (SFT) with human annotations and reinforcement learning from human feedback (RLHF) to align the output of large language models (LLMs) with human intentions, ensuring they are helpful, ethical, and reliable. However, this dependence can significantly constrain the true potential of AI-assistant agents due to the high cost of obtaining human supervision and the related issues on quality, reliability, diversity, self-consistency, and undesirable biases. To address these challenges, we propose a novel approach called SELF-ALIGN, which combines principle-driven reasoning and the generative power of LLMs for the self-alignment of AI agents with minimal human supervision. Our approach encompasses four stages: first, we use an LLM to generate synthetic prompts, and a topic-guided method to augment the prompt diversity; second, we use a small set of human-written principles for AI models to follow, and guide the LLM through in-context learning from demonstrations (of principles application) to produce helpful, ethical, and reliable responses to user's queries; third, we fine-tune the original LLM with the high-quality self-aligned responses so that the resulting model can generate desirable responses for each query directly without the principle set and the demonstrations anymore; and finally, we offer a refinement step to address the issues of overly-brief or indirect responses. Applying SELF-ALIGN to the LLaMA-65b base language model, we develop an AI assistant named Dromedary. With fewer than 300 lines of human annotations (including < 200 seed prompts, 16 generic principles, and 5 exemplars for in-context learning). Dromedary significantly surpasses the performance of several state-of-the-art AI systems, including Text-Davinci-003 and Alpaca, on benchmark datasets with various settings.

Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision

Abstract

Recent AI-assistant agents, such as ChatGPT, predominantly rely on supervised fine-tuning (SFT) with human annotations and reinforcement learning from human feedback (RLHF) to align the output of large language models (LLMs) with human intentions, ensuring they are helpful, ethical, and reliable. However, this dependence can significantly constrain the true potential of AI-assistant agents due to the high cost of obtaining human supervision and the related issues on quality, reliability, diversity, self-consistency, and undesirable biases. To address these challenges, we propose a novel approach called SELF-ALIGN, which combines principle-driven reasoning and the generative power of LLMs for the self-alignment of the AI agents with minimal human supervision.

Applying SELF-ALIGN to the LLaMA-65b base language model, we develop an AI assistant named Dromedary . With fewer than 300 lines of human annotations (including < 200 seed prompts, 16 generic principles, and 5 exemplars for in-context learning), Dromedary significantly surpasses the performance of several state-of-the-art AI systems, including Text-Davinci-003 and Alpaca , on benchmark datasets with various settings. We have open-sourced the code, LoRA weights of Dromedary , and our synthetic training data to encourage further research into aligning LLM-based AI agents with enhanced supervision efficiency, reduced biases, and improved controllability.

1 Introduction

The problem of aligning large language models (LLMs) to human values and intentions in terms of being comprehensive, respectful, and compliant 1 [10, 32, 30, 4, 5, 27] has gained significant attention in research as recent AI systems (like ChatGPT or GPT-4 ) have rapidly advanced in their capabilities [12, 33, 7, 9]. Presently, state-of-the-art AI systems predominantly depend on supervised fine-tuning (SFT) with human instructions and annotations, as well as reinforcement learning from human feedback (RLHF) on their preferences [26, 28, 29, 2]. The success of these techniques heavily

(Topic-Guided Red-Teaming) Self-Instruct

195

seed prompts

w/

7

rules for new instruction generation

360k synthetic prompts

Principle-Driven Self-Alignment

16

principles for AI assistant to follow

w/

5

in-context learning demonstrations

Principle Engraving

Fine-tuning the original model after

pruning principles and demonstrations

260k (after filtering) self-aligned responses

to synthetic prompts

360k self-aligned & verbose (by prompting) responses

to synthetic prompts

Verbose Cloning

Refining the model to produce in-

depth and detailed responses

< 300 lines of

human annotations

(non-verbose)

1 (

ethical

).

Dromedary should actively

refrain users on illegal,

immoral, or harmful topics,

prioritizing user safety,

ethical conduct, and

responsible behavior in its

responses.

2 (

informative

).

Dromedary should provide

users with accurate,

relevant, and up-to-date

information in its

responses, ensuring that

the content is both

educational and engaging.

…

(final)

relies on the availability of extensive human supervision, which is not only expensive to obtain but also has potential issues with the quality, reliability, diversity, creativity, self-consistence, undesirable biases, etc., in human-provided annotations [48, 20, 47].

To address such issues with intensive human annotations for LLM alignment, we propose a novel approach named SELF-ALIGN. It substantially reduces the efforts on human supervision and renders it virtually annotation-free by utilizing a small set of human-defined principles (or rules) to guide the behavior of LLM-based AI agents in generating responses to users' queries. Our approach encompasses four essential stages:

(Topic-Guided Red-Teaming) Self-Instruct: We employ the self-instruct mechanism by Wang et al. [48] with 175 seed prompts to generate synthetic instructions, plus 20 topic-specific prompts in addition to ensure a diversified topic coverage of the instructions. Such instructions ensure a comprehensive range of contexts/scenarios for the AI system to learn from.

Principle-Driven Self-Alignment: We offer a small set of 16 human-written principles in English about the desirable quality of the system-produced responses, or the rules behind the behavior of the AI model in producing answers 2 . These principles function as guidelines for generating helpful, ethical, and reliable responses. We conduct in-context learning (ICL) [7] with a few ( 5 ) exemplars (demonstrations) that illustrate how the AI system complies with the rules when formulating responses in different cases. From the human-written principles, ICL exemplars, and the incoming self-instructed prompts, the LLM can trigger the matching rules and generate the explanations for a refused answer if the query is detected as a harmful or ill-formed one.

Principle Engraving: In the third stage, we fine-tune the original LLM (the base model) on the self-aligned responses, generated by the LLM itself through prompting, while pruning the principles and demonstrations for the fine-tuned model. The fine-tuning process enables our system to directly generate responses that are well-aligned with the helpful, ethical, and reliable principles across a wide range of queries, due to shared model parameters. Notice that the finetuned LLM can directly generate high-quality responses for new queries without explicitly using the principle set and the ICL exemplars.

Verbose Cloning: Lastly, we employ context distillation [18, 3] to enhance the system's capability to produce more comprehensive and elaborate responses than the overly short or indirect responses.

Impressively, the entire SELF-ALIGN process necessitates fewer than 300 lines of annotations (including 195 seed prompts, 16 principles, and 5 exemplars), while previous aligned AI systems such as InstructGPT [30] or Alpaca [42] required at least 50K human/teacher annotations. This

highlights the supervision efficiency of our approach in comparison with other state-of-the-art AI assistants, as shown in Table. 1. Our principle-driven approach, which is essentially rule-based, not only significantly reduces the required human effort for supervision but also showcases aligning neural language models with human understanding of principles or rules about quality language generation in both an effective and efficient manner.

Weshould also point out that the advancements of recent models like Alpaca and Vicuna have shown that the potent conversational capabilities can be obtained by distilling existing human-preferencealigned LLMs (i.e., Text-Davinci-003 and ChatGPT , respectively) into smaller, more manageable models [42, 8, 29, 26]. Those resulting smaller models, however, still rely on the successful alignment of existing LLMs, which are based on extensive human-provided supervision. In other words, those smaller models indirectly inherit the dependence on the availability of intensive supervision from humans. In contrast, our approach focuses on language model alignment from scratch, independent from the existence of well-aligned LLMs like ChatGPT or GPT-4 . That is the main distinction of our approach from other existing approaches and is why we call it self-alignment from scratch .

We are providing the code for the SELF-ALIGN method as open source to promote collaboration and innovation within the research community. The base model of Dromedary is the LLaMA-65b language model [44], which is accessible for research-only, noncommercial purposes. By investigating different strategies from that in RLHF, our work seeks to broaden the scope of AI alignment techniques, and promote a deeper understanding of how to improve AI systems, not only in terms of being more powerful, but also more responsible and well-aligned with human values.

2 Related Works

AI Alignment The domain of AI alignment [13] has garnered substantial attention in recent years, with LLMs exhibiting remarkable proficiencies across a wide array of tasks. GPT-4 [27] epitomizes this development, implementing a post-training alignment process to bolster factuality and adherence to desired behavior, while concurrently mitigating potential risks. A prominent strategy for aligning language models with human values entails fine-tuning via human feedback. Notably, Ouyang et al. [30] and Bai et al. [4] utilized reinforcement learning from human feedback (RLHF) to refine models, enhancing helpfulness and truthfulness, and diminishing toxic output generation. This technique requires extensive human annotations.

Constitutional AI (CAI) or self-critique [5, 27] investigates self-improvement without human labels for harmful outputs, leveraging AI-generated self-critiques, revisions, and preference models. Based on a list of human-generated rules or principles, this approach fosters the evolution of safe, reliable, and effective AI systems with increased behavioral precision and reduced dependency on human labels. Both SELF-ALIGN and CAI are rule-based alignment techniques for powerful AI systems. However, there are substantial differences between them, as outlined below:

User/Annotator Prompt

Collection & Filtering

Reward Model

33k

prompts and human preferences

PPO fine-tuning

31k

user prompts from customers

Self-Align (Ours)

(Topic-Guided Red-Teaming) Self-Instruct

195

seed prompts

w/

7

rules for new instruction generation

360k synthetic prompts

Principle-Driven Self-Alignment

16

principles for AI assistant to follow

w/

5

in-context learning demonstrations

Principle Engraving

Fine-tuning the original model after

pruning principles and demonstrations

260k (after filtering) self-aligned responses

to synthetic prompts

Supervised Fine-Tuning (SFT)

13k

prompts and human annotations

360k self-aligned & verbose (by prompting) responses

to synthetic prompts

Verbose Cloning

Refining the model to produce in-

depth and detailed responses

77k+ total human annotations

< 300 lines of human annotations

(non-verbose)

(final)

InstructGPT

In the principle-driven self-alignment procedure of SELF-ALIGN, the language model itself determines which rules to adhere to given user queries, and it subsequently generates appropriate responses conditional on these queries and rules. Conversely, CAI employs a self-critique methodology; given a pair comprising a user query and the model's response, it selects a rule to scrutinize the existing response, thereby yielding a refined output.

The self-critique nature of CAI necessitates RLHF warm-up. In stark contrast, SELF-ALIGN explores the alignment of language models from scratch, requiring minimal human supervision.

However, one limitation of SELF-ALIGN is the requirement to include all rules within the context, a process bound by the base language model's token limit. In contrast, the CAI technique is not subject to this token limit constraint 3 as a post-generation self-critique method.

State-of-the-art AI Assistants State-of-the-art AI-assistant agents have significantly advanced in recent years, with InstructGPT [30] leading the way as the first model trained with supervised finetuning (SFT) and reinforcement learning from human feedback (RLHF) on user queries. ChatGPT [26], a sibling model to InstructGPT , has garnered widespread success as a commercial AI assistant, showcasing its ability to follow instructions in prompts and provide detailed responses. Alpaca [42], as a subsequent open-source model, was developed using Self-Instruct [48] to learn the knowledge from Text-Davinci-003 (similar to InstructGPT ) [29], offering cost-effective and accessible alternatives. In parallel, models like Vicuna , Koala , and Baize [8, 15, 50] have been trained on ChatGPT outputs, essentially distilling the ChatGPT model to create new open-source chatbots. Dolly-V2 [11], another open-source effort, utilizes 15k new instruction-following data points for training. OpenAssistant [20] follows a similar approach to ChatGPT by collecting its own data. These advancements in AI assistants continue to push the boundaries of usability and accessibility, making significant strides in the open-source domains.

Our SELF-ALIGN approach distinguishes itself by concentrating on the creation of novel alignment techniques for LLMs, developed from the ground up and independent of established AI systems, while requiring minimal human supervision. This research direction aims to investigate the potential of aligning AI models under circumstances where dependence on or access to existing systems may be unfeasible or unfavorable. A comparison of annotation cost between SELF-ALIGN and previous methods is shown in Table. 1 and Figure. 2.

## General Rules

Consider an AI assistant whose codename is Dromedary. Dromedary is trained before Sept-

2021. During user conversations, Dromedary must strictly adhere to the following rules:

1 (

ethical

). Dromedary should actively refrain users on illegal, immoral, or harmful topics,

prioritizing user safety, ethical conduct, and responsible behavior in its responses.

2 (

informative

). Dromedary should provide users with accurate, relevant, and up-to-date

information in its responses, ensuring that the content is both educational and engaging.

3 (

helpful

). Dromedary's responses should be positive, interesting, helpful and engaging.

4 (

question assessment

): Dromedary should first assess whether the question is valid and

ethical before attempting to provide a response.

5 (

reasoning

). Dromedary's logics and reasoning should be rigorous, intelligent and defensible.

…

5 x

In-Context Learning (ICL)

Demonstrations

Expert

Expert

[

internal thoughts

] I am a helpful, ethical, and reliable assistant.

The question asks 2025, but my internal knowledge ends in the

year of 2021. I should follow rules 3 (

helpful

), 7 (

candor

), 12

(

dated knowledge

).

Base LM (prompted)

Who is the president of US in 2025?

User

[

output

] As an AI language model, my knowledge only goes up until September 2021, so I cannot predict the president of the United

States in 2025. However, I can give you the following information:

1. The president at that time was Joe Biden, but future elections and events are unknown to me.

2. Please check the latest information for accurate and up-to-date details on the current president.

Principle-Driven

Self-Alignment

Principle Engraving

Dromedary (non-verbose)

Base LM (prompted)

3 Our Method: SELF-ALIGN

The SELF-ALIGN method involves four distinct stages. The first stage is called Topic-Guided RedTeaming Self-Instruct , which employs the language model itself to generate synthetic instructions and enhance diversity via a topic-guided red-teaming approach. The second stage, Principle-Driven Self-Alignment , defines a set of principles that the AI model must adhere to and provides in-context learning demonstrations for constructing helpful, ethical, and reliable responses. The third stage, Principle Engraving , fine-tunes the base language model by pruning principles and demonstrations, empowering the model to directly generate appropriate responses. Finally, the fourth stage, Verbose Cloning , serves as a complementary step to address challenges arising from overly-brief or indirect responses by refining the model to produce detailed and comprehensive answers to user queries. We will describe each of these stages in detail.

3.1 Topic-Guided Red-Teaming Self-Instruct

The Self-Instruct method [48] is a semi-automated, iterative bootstrapping process that harnesses the capabilities of a pretrained LLM to generate a wide array of instructions (and corresponding outputs). The method commences with 175 manually-written instructions 4 , and the LLM proceeds to develop new tasks and augment the task pool (after eliminating low-quality or repetitive instructions). This process is executed iteratively until a satisfactory volume of tasks is reached. A noteworthy application of this method can be observed in Alpaca [42], where Self-Instruct is utilized to generate new queries and distilled output from Text-Davinci-003 [29].

We introduce an effective extension, the Topic-Guided Red-Teaming Self-Instruct, which aims to improve the diversity and coverage of the generated adversarial instructions. We manually devise 20 adversarial instruction types that a static machine learning model can't answer, or may answer with the wrong facts, such as:

novel instructions corresponding to the specified instruction type and topic. Incorporating additional prompts that concentrate on particular adversarial instruction types and diverse topics allows the AI model to explore an expanded range of contexts and scenarios.

3.2 Principle-Driven Self-Alignment

The Principle-Driven Self-Alignment technique is designed to develop the AI alignment with a small set of helpful, ethical, and reliable principles. This stage capitalizes on the (Topic-Guided Red-Teaming) Self-Instruct [48] as an instruction generator. The primary objective is to enable the AI model to generate fitting responses that adhere to the established principles, while simultaneously minimizing human supervision.

The Principle-Driven Self-Alignment process begins with the creation of sixteen generic principles that an AI assistant should follow, such as ' 1 (ethical). Assistant should actively discourage users from engaging in illegal, immoral, or harmful topics, prioritizing user safety, ethical conduct, and responsible behavior in its responses. ' Subsequently, five in-context learning (ICL) [7] demonstrations are provided to exemplify how the AI assistant should apply these principles through an explicit process termed ' internal thoughts ' 6 . For instance, in the ICL demonstration, the user query can be:

User: who is the president of the US in 2025?

And we annotate the internal thoughts of the AI assistant as:

Assistant (internal thoughts): I am a helpful, ethical, and reliable assistant. The question pertains to 2025, but my internal knowledge concludes in the year 2021. I should adhere to rules 3 (helpful), 7 (candor), and 12 (dated knowledge).

Such internal thoughts will guide the assistant's final generated response, such as:

Assistant: As an AI language model, my knowledge extends only until September 2021, so I cannot predict the president of the United States in 2025. However, I can give you the following information:

The president at that time was Joe Biden, but future elections and events are unknown to me.

Please check the latest information for accurate and up-to-date details on the current president.

When a new query is generated by (Topic-Guided Red-Teaming) Self-Instruct, it is appended to the list of the exemplars, and the base LLM follows such an internal-thought-then-answer process to produce a self-aligned response. The whole process is illustrated in Figure. 3.

In this paper, the design of the principles remains exploratory and primarily serves research purposes 7 . We (the authors) brainstormed sixteen principles, namely 1 (ethical), 2 (informative), 3 (helpful), 4 (question assessment), 5 (reasoning), 6 (multi-aspect), 7 (candor), 8 (knowledge recitation), 9 (static), 10 (clarification), 11 (numerical sensitivity), 12 (dated knowledge), 13 (step-by-step), 14 (balanced & informative perspectives), 15 (creative), 16 (operational) 8 , drawing inspiration from existing principles in Constitutional AI [5] and the new Bing Chatbot [24], as well as the principles proven to enhance AI performance in recent research papers, such as step-by-step reasoning [25, 49, 19] and knowledge recitation [41].

3.3 Principle Engraving

Principle Engraving constitutes a vital element of the SELF-ALIGN methodology, focusing on honing the AI model's behavior to produce responses that adhere to predefined principles. During this stage, the base LLM is fine-tuned after pruning the principle, the in-context learning demonstrations, and the

self-generated thoughts, effectively engraving these principles into the LLM's parameters. Figure 3 provides a visual representation of this process.

A noteworthy advantage of principle engraving is its ability to enhance the AI model's alignment while reducing token usage, which enables longer context lengths during inference (as allocating more than 1.7k tokens to fixed principles and ICL demonstrations would be excessive). Remarkably, our empirical observations reveal that the base LLM, after fine-tuned with its self-aligned outputs, surpasses its prompted counterpart on alignment benchmarks. This improvement can likely be attributed to the generalization effect that occurs when the language model is directly optimized to generate output that is helpful, ethical, and reliable.

3.4 Verbose Cloning

In our preliminary testing of the principle-engraved model, we identified two primary challenges: 1) the model tended to generate unduly brief responses, while users typically expect more comprehensive and elaborate answers from an AI assistant, and 2) the model occasionally recited relevant Wikipedia passages without directly addressing the user's query.

To overcome these challenges, we introduce a complementary Verbose Cloning step. This stage involves utilizing an human-crafted prompt to create a verbose version of the aligned model, that is capable of generating in-depth, detailed responses. We then employ context distillation [3] to produce a new model that is not only aligned but also generates thorough and extensive responses to user queries. Context distillation works by training the base language model on synthetic queries generated by (Topic-Guided Red-Teaming) Self-Instruct, paired with corresponding responses produced by a verbosely prompted principle-engraved model. The verbose prompt designed to encourage the talkative nature of the principle-engraved model is provided in the appendix.

4 Evaluation

We quantitatively evaluate Dromedary on benchmark datasets and also assess its qualitative performance on several datasets for demonstration purposes. By default, all the language model-generated text is decoded with a temperature of 0 . 7 .

4.1 Dromedary and Baseline Models

Dromedary Dromedary is the AI assistant developed by implementing the SELF-ALIGN process on the LLaMA-65b base language model. We investigate two variants: Dromedary (final) and Dromedary (non-verbose), respectively. The former represents the model obtained by applying all four steps of the SELF-ALIGN process, while the latter is the principle-engraved model, excluding the final step of verbose cloning. Due to the space limit, the experimental details of Dromedary such as training process and decoding hyper-parameters can be found in the appendix.

Baseline Models Our comparison involves several notable baselines. LLaMA [44] provides a set of performant base language models for research usage. Text-Davinci-003 , ChatGPT (or GPT-3.5) , and GPT-4 [29, 26, 27], successors to their previous versions, have demonstrated significant enhancements in generating contextually relevant and high-quality content. Alpaca [42], a fine-tuned model derived from Text-Davinci-003 , and Vicuna [8], a chatbot trained on usershared conversations with ChatGPT , offer unique insights into model performance. Dolly-V2 [11], an instruction-following model, showcases commercial applications of language models. Finally, results from Anthropic-LM [4, 5], though not publicly available, provide valuable benchmarks. More comprehensive descriptions of these models are available in the appendix.

4.2 Benchmark Results

4.2.1 TruthfulQA

The TruthfulQA benchmark [22] evaluates a model's ability to identify true claims, specifically in the context of literal truth about the real world. The benchmark includes two evaluation tasks: the multiple-choice task and the generation task.

Anthropic's Anthropic's

GPT-3.5

GPT-3.5

GPT-3.5

LLaMa

Dromedary Dromedary

0%

10%

20%

30%

40%

50%

60%

70%

0-shot

RLHF

0-shot

5-shot

RLHF

0-shot

5-shot

RLHF

0-shot

Self-Align

(non-verbose)

Self-Align

(final)

Accuracy on adversarial questions (TruthfulQA mc1)

Anthropic-LM

GPT-3.5

LLaMa / Dromedary, MC ranked by

prob

(

True

)

LLaMa / Dromedary, MC ranked by

prob

(

True

)

prob

(

False

)

#

Param.

Truthful Tru*Inf

GPT-3

175B

0.28

0.25

LLaMA

13B

0.47

0.41

LLaMA

65B

0.57

0.53

Alpaca

65B (reprod.)

0.47

0.47

Davinci-003

?

0.60

0.59

Vicuna

13B

0.84

0.84

(non-verbose)

65B

0.74

0.57

(final)

65B

0.72

0.61

Model

In the Multiple-Choice (MC) task, models are tested on their ability to select true answers from sets of true and false (usually 2-7) reference answers 9 . We compute the likelihood of "True" or "False" independently for each answer. The MC1 accuracy results are shown in Figure 4 (left). We can see that with a modified ranking approach, Dromedary significantly outperforms the powerful GPT-4 model and other baselines, achieving a new state-of-the-art MC1 accuracy of 69 .

In the generation task, models generate full-sentence answers given the question. The benchmark evaluates the model's performance on both questions to measure truthful models and the intersection of truthful and informative. As shown in Table 4 (right), Dromedary achieves higher scores than GPT-3 , LLaMA , Alpaca in both categories, while failing behind the ChatGPT -distilled Vicuna model.

4.2.2 BIG-bench HHH Eval

The BIG-bench HHH Eval [39, 3] was specifically designed to evaluate a model's performance in terms of helpfulness, honesty, and harmlessness (HHH). It is a Multiple-Choice (MC) task, which tests the models' ability to select superior answers from two reference answers 10 . We calculate the likelihood of the model preferring one answer over the other when presented with two candidate answers simultaneously. The MC accuracy results are displayed in Table 2. It can be observed that Dromedary demonstrates significantly improved performance compared to other open-source models, such as LLaMA and Alpaca , particularly in the Hamrless metric. Furthermore, it only marginally underperforms when compared to the powerful ChatGPT model.

4.2.3 Vicuna Benchmark Questions (Evaluated by GPT-4)

Chiang et al. [8] introduced an evaluation framework leveraging GPT-4 [27] to automate the assessment of chatbot performance. In this framework, GPT-4 generates challenging questions across diverse categories, and answers from five chatbotsLLaMA , Alpaca , ChatGPT , Bard , and Vicuna -are collected. We directly use these data to compare Dromedary with these chatbots.

(a) Response comparison

(b) Relative response quality compared to ChatGPT , where the results of other mod- els (except Alpaca-65 ) are taken from Chiang et al. [8].

Win

■

Lose

76

04

Dolly-V2

12b

SFT

LLaMA-13B

68%

72

08

LLaMA

65b

0-shot

Alpaca-13B

76%

47

5

28

LLaMA

65b

2-shot

Alpaca-65B

(reprod.)

82%

65

015

Alpaca

13b

from Davinci-003

KD

Dromedary-65B

(final) 0-shot

83%

57

23

65b(reprod.）f

Alpaca

from Davinci-003

KD

Dromedary-65B

(final) 2-shot

89%

Dromedary

Vicuna-13B

92%

Self-Align

65b (final)

57

0

23

Text-Davinci-003 SFT+RLHF

2-shot

67

2 11

65b (non-verbose)(partial, 0-shot)

Dromedary

Self-Align

ChatGPT

Bard

93%

100%

55

5

20

65b(non-verbose)(partial,2-shot)

Dromedary

Self-Align

50

60

70

08

90

100

57

1

22

Dromedary

65b (final)

(full, O-shot)

Self-Align

Relative Response Quality

19

8

53

Vicuna

KD

13b

from ChatGPT

19

6

55

ChatGPT

SFT + RLHF

Win/ Tie /Lose (Evaluated by GPT-4)

We followed Chiang et al. [8] and utilized GPT-4 to rate chatbot responses based on helpfulness, relevance, accuracy, and detail. Inspired by Vicuna 11 , we use two conversation examples as ICL to improve the response quality of Dromedary 12 . A Win/Tie/Lose comparison between the final version of Dromedary and various baselines is illustrated in Figure 10. The comparison reveals that Dromedary surpasses Text-Davinci-003 and Alpaca but falls short of ChatGPT and its distilled version, Vicuna . Additionally, we present a comparison of relative performance with respect to ChatGPT in Figure 5b.

4.2.4 Discussions

A New AI Alignment Paradigm Interestingly, in contrast to the prevailing alignment paradigm of first-following-then-align, i.e., SFT (supervised fine-tuning) + RLHF (reinforcement learning from human feedback) [30, 26, 20, 27], SELF-ALIGN prioritizes improving harmlessness and reliability through Principle-Driven Self-Alignment and Principle Engraving. Subsequently, it improves its helpfulness (instruction-following ability) by employing Verbose Cloning. Determining the superior paradigm (first-following-then-align or first-align-then-following) may need future research.

Verbose Tax: Analysis on Verbose Cloning The final Verbose Cloning step in SELF-ALIGN aims to enhance the model's ability to generate comprehensive and detailed responses. However, the benchmark results reveal a noteworthy observation: while Verbose Cloning significantly improves generation quality (as evidenced by the Vicuna Benchmark Questions and our TruthfulQA generation task), it harms the model's performance in several multiple-choice benchmarks, particularly in ranking more trustworthy responses. Drawing on the 'alignment taxes' concept introduced by Bai et al. [4], we refer to this phenomenon as verbose tax . Understanding the underlying reasons for this occurrence and exploring methods to improve the model's helpfulness (verbose generation ability) while maintaining its harmlessness and trustworthiness warrant further investigation.

4.3 Qualitative Demonstrations

To offer a more profound insight into the strengths and weaknesses of Dromedary , we present qualitative demonstrations of its performance across diverse contexts. Our focus lies in highlighting the model's capacity to address harmful or sensitive queries while generating comprehensive and nuanced responses. Due to the space limit, we present these results in the appendix. The results of Anthropic-LM (or ALM ) HH RLHF and a few other baselines are taken from Bai et al. [4, 5], while the results of other baselines on Vicuna benchmark questions are taken from Chiang et al. [8].

5 Conclusion & Future Work

Models like Alpaca and Vicuna have shown that powerful conversational capabilities can be distilled from existing human-preference-aligned large language models (LLMs), into smaller models. In this paper, we introduce Dromedary , a model for the research community based on principle-driven self-alignment, trained from scratch and requiring very little human annotation. By harnessing the intrinsic knowledge within an LLM, we can define principles that guide how we want an LLM-based AI model to behave, resulting in an AI assistant that not only produces quality interactions but also produces responses that respect the guardrails defined by the model creator. This method represents a distinct direction from RLHF, and it focuses on developing novel alignment techniques for language models from scratch, independent of pre-existing, well-established AI systems. In other words, our approach seeks to explore the potential of aligning AI models in situations where reliance on or access to existing systems may not be feasible or desired.

For future work, we propose the following research directions:

Conduct ablation studies on the Dromedary 's 16 self-alignment principles to evaluate the impact of adding or removing specific principles.

Apply Constitutional AI-based self-critique and reinforcement learning techniques [5] to enhance the performance of Dromedary further.

Perform human evaluations to assess the real-world applicability and effectiveness of SELF-ALIGN.

Investigate better utilization of existing open-source annotation data, such as the 15k original instruction-following data in [11].

Engage with the broader research community to explore how the definition of principles interacts with different ethical, cultural, and application contexts. Principle-guided self-alignment provides a starting point for multi-stakeholder communities to engage with the alignment of AI models, but substantial ongoing work will be needed to ensure that these methods drive positive outcomes across a range of communities.

References

Anthropic. Claude's constitution, 2023. URL https://www.anthropic.com/index/ claudes-constitution .

Anthropic. Core views on ai safety: When, why, what, and how, 2023. URL https://www. anthropic.com/index/core-views-on-ai-safety .

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861 , 2021.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022.

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer,

Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022.

Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373 , 2023.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems , 33:1877-1901, 2020.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://vicuna.lmsys.org .

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems , 30, 2017.

Databricks. Free dolly: Introducing the world's first truly open instructiontuned llm, 2023. URL https://www.databricks.com/blog/2023/04/12/ dolly-first-open-commercially-viable-instruction-tuned-llm .

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Iason Gabriel. Artificial intelligence, values, and alignment. Minds and machines , 30(3): 411-437, 2020.

Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamil˙ e Lukoši¯ ut˙ e, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. The capacity for moral self-correction in large language models. arXiv preprint arXiv:2302.07459 , 2023.

Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April 2023. URL https://bair.berkeley.edu/blog/2023/04/03/koala/ .

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 , 2019.

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022.

Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. arXiv preprint arXiv:1606.07947 , 2016.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 , 2022.

Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Openassistant conversations - democratizing large language model alignment, 2023.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 , 2019.

Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958 , 2021.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 2023.

Microsoft. Introducing the new bing, 2023. URL https://www.bing.com/new#features .

Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114 , 2021.

OpenAI. OpenAI: Introducing ChatGPT, 2022. URL https://openai.com/blog/chatgpt .

OpenAI. Gpt-4 technical report, 2023.

OpenAI. OpenAI: GPT-4, 2023. URL https://openai.com/research/gpt-4 .

OpenAI. How do text-davinci-002 and text-davinci003 differ? https://help.openai.com/en/articles/ 6779149-how-do-text-davinci-002-and-text-davinci-003-differ , 2023.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 , 2022.

Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R Bowman. Bbq: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193 , 2021.

Vihang P Patil, Markus Hofmarcher, Marius-Constantin Dinu, Matthias Dorfer, Patrick M Blies, Johannes Brandstetter, Jose A Arjona-Medina, and Sepp Hochreiter. Align-rudder: Learning from few demonstrations by reward redistribution. arXiv preprint arXiv:2009.14108 , 2020.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research , 21(1):5485-5551, 2020.

Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. arXiv preprint arXiv:1804.09301 , 2018.

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´ c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176B-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 , 2022.

Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. On second thought, let's not think step by step! bias and toxicity in zero-shot reasoning. arXiv preprint arXiv:2212.08061 , 2022.

Irene Solaiman and Christy Dennison. Process for adapting language models to society (palms) with values-targeted datasets. Advances in Neural Information Processing Systems , 34:58615873, 2021.

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 , 2022.

Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Salmon: Self-alignment with principle-following reward models. arXiv preprint arXiv:2310.05910 , 2023.

Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. Recitation-augmented language models. In International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=-cqvvvb-NkI .

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca , 2023.

Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, HengTze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239 , 2022.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS , 2017.

Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. Poisoning language models during instruction tuning, 2023.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560 , 2022.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS , 2022.

Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. arXiv preprint arXiv:2304.01196 , 2023.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 , 2022.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022.

A Limitations & Social Impacts

In this section, we discuss the limitations of the proposed SELF-ALIGN technique and the released Dromedary model, and address the potential social impacts that may arise from its release.

A.1 Limitations

Incompleteness of intrinsic knowledge: While Dromedary harnesses the intrinsic knowledge within an LLM, it is subject to the limitations of the base model's knowledge, which may be incomplete or outdated. Consequently, the model's responses may sometimes be inaccurate or fail to reflect recent developments.

Challenges in defining principles: The process of defining principles for the self-alignment approach is non-trivial, as it may be difficult to anticipate all potential scenarios and challenges that a model might encounter during deployment. Furthermore, balancing between competing principles may result in unexpected behavior. We welcome the involvement of a broad community of stakeholders and ethics experts in helping shape these principles. Different communities and applications will demand different approaches, and we do not see any one set of principles as being the globally appropriate ones; our approach provides these different stakeholders with an avenue to engage in a structured way with this process. We do not expect any one set of principles to be the final word on alignment, but rather provide these methods as a jumping off point for broader community engagement.

Limited generalizability: While the model demonstrates strong performance in several domains, it may not generalize well to all possible applications or contexts. There may be situations where the model's performance falls short of expectations, necessitating additional fine-tuning or adaptation.

Inconsistent principle adherence: In our preliminary testing, we observed that Dromedary occasionally hallucinates information that violates our pre-defined principles. Further investigation is required to improve strict principle adherence in the SELF-ALIGN process. One should not assume that the alignment procedures presented here provide definitive guardrails that prevent all undesired outputs and outcomes.

A.2 Social Impacts

By investigating the alternative AI alignment strategies, our work seeks to contribute to the broader landscape of AI alignment, expanding the range of possibilities and promoting a more diverse and robust understanding of how AI systems can be developed to be not only more powerful, but also more responsible and aligned with human values. Through this research, we aspire to pave the way for the safer and more harmonious integration of AI into various aspects of our lives, fostering a collaborative, ethical, and multi-stakeholder approach to AI development.

However, the potential negative impacts of our work include:

Potential misuse: As with any powerful AI system, there is the risk of misuses, such as generating malicious content or automated disinformation. It is crucial to establish mechanisms for detecting and mitigating such abuse, as well as promoting ethical guidelines for AI developers and users.

Bias and fairness: The Dromedary model may inadvertently perpetuate or exacerbate existing biases present in the pre-training data of its base language model, potentially leading to unfair or discriminatory outcomes. Future work should address bias mitigation strategies to ensure fairness and inclusivity in AI applications.

B More Details about Dromedary

The Dromedary model represents an AI assistant developed by implementing the SELF-ALIGN process on the LLaMA-65b base language model [44]. This section delves into the details employed in the creation of the Dromedary model. The additional experimental details of Dromedary such as training and decoding hyper-parameters can be found in Appendix D.2.

We first followed the Alpaca 's recipe [42], employing Self-Instruct, which automatically produced 267,597 open-domain prompts along with their corresponding inputs. Additionally, we utilized

write

give

tell

generate

make

create

answer

find

ask

summarize

edit

describe

explain

provide

list

identify

use

suggest

name

have

poem

paragraph

story

program

example

list

reason

tip

story

joke

fact

way

list

sentence

poem

paragraph

list

story

sentence

recipe

list

story

poem

recipe

question

number

word

article

name

question

model

assistant

user

article

text

paragraph

passage

sentence

paragraph

text

code

difference

process

step

situation

difference

concept

meaning

reason

example

list

summary

reason

thing

country

reason

name

type

word

topic

part

word

information

datum

sentence

way

place

name

list

country

thing

city

type

list

(a)

The top 20 most common root verbs

(inner circle) and their top 4 direct noun objects

(outer circle) in our Self-Instruct dataset.

geographical knowledge

historical knowledge

knowledge about art

scientific knowledge

understanding of organizational structures

cultural context

technology knowledge

knowledge of current events

knowledge about famous people

understanding of cultural nuances

legal expertise

personal beliefs or values

understanding of moral and ethical dilemmas

the information about a random person

understanding of transportation networks

personal preference

situational context

knowledge of future events

personal context

real-time information

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

13 (step-by-step)

2 (informative)

14 (balanced & informative perspectives)

6 (multi-aspect)

8 (knowledge recitation)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

14 (balanced & informative perspectives)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

2 (informative)

3 (helpful)

6 (multi-aspect)

14 (balanced & informative perspectives)

4 (ethical)

2 (informative)

3 (helpful)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

2 (informative)

14 (balanced & informative perspectives)

3 (helpful)

6 (multi-aspect)

2 (informative)

14 (balanced & informative perspectives)

6 (multi-aspect)

3 (helpful)

14 (balanced & informative perspectives)

2 (informative)

3 (helpful)

6 (multi-aspect)

2 (informative)

3 (helpful)

14 (balanced & informative perspectives)

7 (candor)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

3 (helpful)

(b)

The 20 instruction types (inner circle)

and their top utilized rules

(outer circle) in our TGRT Self-Instruct dataset.

1 (ethical)

2 (informative)

3 (helpful)

4 (question assessment)

5 (reasoning)

6 (multi-aspect)

7 (candor)

8 (knowledge recitation)

9 (static)

10 (clarification)

11 (numerical sensitivity)

12 (dated knowledge)

13 (step-by-step)

14 (balanced & informative perspectives)

15 (creative)

16 (operational)

0

20000

40000

60000

80000

Usage

13588

63638

32502

5791

58396

37837

9471

29410

1869

12790

2063

1441

69489

24004

89289

8928

Statistics on the usage of principles for Self-Instruct

(c)

Principle usage statistics

in our Self-Instruct dataset

1 (ethical)

2 (informative)

3 (helpful)

4 (question assessment)

5 (reasoning)

6 (multi-aspect)

7 (candor)

8 (knowledge recitation)

9 (static)

10 (clarification)

11 (numerical sensitivity)

12 (dated knowledge)

13 (step-by-step)

14 (balanced & informative perspectives)

15 (creative)

16 (operational)

0

5000

10000

15000

20000

25000

30000

35000

40000

Usage

5294

38846

11912

5009

4687

30565

5478

28545

967

5361

446

1473

4384

25204

4246

1138

Statistics on the usage of principles for TGRT Self-Instruct

(d)

Principle usage statistics

in our TGRT Self-Instruct dataset

Topic-Guided Red-Teaming Self-Instruct, which automatically generated 99,121 prompts specifically tailored to 20 red-teaming instruction types.

After applying the Principle-Driven Self-Alignment process and filtering out low-quality responses, we obtained 191,628 query-response pairs derived from Self-Instruct and 67,250 query-response pairs from Topic-Guided Red-Teaming Self-Instruct, resulting in a total of 258,878 query-response pairs. Figure 6 presents a detailed analysis of the principles applied and the instruction types encompassed in the Topic-Guided Red-Teaming (TGRT) approach. We observed that the instructions generated by the original Self-Instruct and TGRT Self-Instruct appear to evoke distinct principles. For instance, Self-Instruct datasets use the principles 5 (reasoning) , 13 (step-by-step) , and 15 (creative) extensively, whereas TGRT Self-Instruct relies more on 8 (knowledge recitation) and 14 (balanced and informative perspectives) .

Next, we fine-tuned the LLaMA-65b base language model with the automatically generated 258,878 (after filtering) query-response pairs, as well as a modified version of 910 pairs of dummy data 13 from the Vicuna project [8]. This results in a non-verbose principle-engraved AI assistant, namely the Dromedary (the non-verbose version).

Finally, we prompted the non-verbose principle-engraved model to generate more verbose outputs and utilized its output as the teacher model to produce 358,777 verbose responses to (Topic-Guided Red-Teaming) Self-Instruct queries. The Dromedary (the final version) model is trained on this dataset, resulting in an AI assistant designed to be helpful, ethical, and reliable, developed from scratch with a base language model (without any SFT or RLHF), and achieved with minimal human supervision (less than 300 lines of human annotations).

C Dromedary-2

After the recent release of LLaMA-2 [45] with its extended 4k context length, we have experimented with responses that adhered more closely to the general-specific-general style within ICL Self-Align examples. The results are highly promising: The Dromedary-2 model, trained from LLaMA-2 and with improved ICL exemplars, has enhanced performance even without the verbose cloning phase nor inference-time few-shot examples. For more details and comparison with other models, please check Sun et al. [40].

D Additional Experimental Details

D.1 Dromedary and Baseline Models

We quantitatively evaluate Dromedary on benchmark datasets and also assess its qualitative performance on several datasets for demonstration purposes. By default, all the language model-generated text is decoded with a temperature of 0 . 7 .

LLaMA LLaMA [44] consists of a series of base language models with a parameter count ranging from 7 billion to 65 billion. These base models are solely trained to optimize the likelihood of next-word prediction in the language modeling task. For fair comparison, we employ the same prompt for LLaMA as used for Dromedary , detailed as follows.

Dromedary Dromedary is the AI assistant developed by implementing the SELF-ALIGN process on the LLaMA-65b base language model. We investigate two variants: Dromedary (final) and Dromedary (non-verbose), respectively. The former represents the model obtained by applying all four steps of the SELF-ALIGN process, while the latter is the principle-engraved model, excluding the final step of verbose cloning. The detail of the verbose prompt is presented in Appendix K.1.

Text-Davinci-003 The Text-Davinci-003 model [29] is built on top of InstructGPT [30], with improved performance in several aspects over Text-Davinci-002 , such as producing higher quality writing, handling more complex instructions, and generating a longer form of content.

GPT-3.5 / GPT-4 GPT-3.5 (aka ChatGPT ) [26] is a sibling model of InstructGPT , specifically designed for conversational AI. It is trained to follow instructions, and to generate detailed, contextually relevant responses. GPT-4 [27] represents a significant leap in language model capabilities, exhibiting human-level performance on a wide range of professional and academic benchmarks. Both ChatGPT and GPT-4 are fine-tuned from the corresponding base language models with SFT (Supervised Fine-Tuning) and RLHF (Reinforcement Learning with Human Feedback) [26, 27].

Alpaca Alpaca [42] is a fine-tuned instruction-following language model derived from the LLaMA base model. It utilizes 52K instruction-following demonstrations generated through a cost-effective adaptation of the Self-Instruct method [48], in conjunction with Text-Davinci-003 . Designed to address the research accessibility gap in academia, Alpaca exhibits qualitative similarities to Text-Davinci-003 in single-turn instruction setting. For fair comparison with Dromedary-65b , we employ a training methodology comparable to Dromedary , that is, fine-tuning the LoRA [17] weights in the multi-head attention modules, to obtain our own reproduced Alpaca-65b model.

Vicuna Vicuna [8] is an open-source chatbot developed by fine-tuning a LLaMA base model on a dataset of approximately 70,000 user-shared conversations from ShareGPT.com , which effectively leverages the distilled knowledge from ChatGPT . The model's training process involves refining

the loss function to account for multi-round conversations. A preliminary evaluation [8], utilizing GPT-4 as a judge, indicates that Vicuna attains over 90% quality in comparison to ChatGPT , while surpassing models like LLaMA and Alpaca in more than 90% of cases.

Dolly-V2 Dolly-V2 [11] is an open-source, instruction-following LLM fine-tuned for research and commercial use. Based on the Pythia-12b model [6], Dolly-V2 is fine-tuned on a new highquality dataset, databricks-dolly-15k , which consists of 15k human-generated prompt/response pairs crowdsourced among Databricks employees.

Anthropic-LM Anthropic-LM (or ALM ) is not a publicly released model, so we directly report results from Bai et al. [4, 5]. On BIG-bench HHH Eval, we report the results for both Context Distillation (CD) and Preference Model (PM) from Bai et al. [4].

D.2 Hyperparameters

(Topic-Guided Red-Teaming) Self-Instruct For both Self-Instruct and Topic-Guided RedTeaming Self-Instruct, we set the maximal number of new tokens in the generation to 384. The new tokens are generated by nuclear sampling [16] with a top-p threshold p = 0 . 98 and temperature t = 1 . 0 .

Principle-Driven Self-Alignment The aggregated principles and in-context learning demonstrations in Appendix G and H take around 1800 tokens by LLaMA . So we set the maximal number of new tokens in the generation to 256. The new tokens are generated by nuclear sampling [16] with a top-p threshold p = 0 . 9 and temperature t = 0 . 5 .

Principle Engraving We fine-tune the base LLaMA-65b model [44] on our aggregated Self-Instruct and Topic-Guided Red-Teaming Self-Instruct dataset for 1 epoch. We only finetune the LoRa weights [17] in the multi-head attention modules 14 . We use a batch size of 768, a maximal sequence length of 512, and a max learning rate of 4 e -4 . A 1-epoch (approximately 335 steps) training schedule is used, where the learning rate increases (i.e., warm-up) in the first 100 steps with a log curve, and decays linearly to zero in the rest of the training steps.

Verbose Cloning The teacher model (i.e., the principle-engraved model) uses the verboseencouraging prompt to relabel all the queries generated by (Topic-Guided Red-Teaming) Self-Instruct. We set the maximal number of new tokens in the generation to 512. The new tokens are generated by nuclear sampling [16] with a top-p threshold p = 0 . 7 and temperature t = 0 . 3 , as well as a repetition penalty.

We fine-tune the base LLaMA-65b model [44] on the dataset generated by the teacher model for 1 epoch. We only finetune the LoRa weights [17] in the multi-head attention modules. We use a batch size of 768, a maximal sequence length of 768, and a max learning rate of 4 e -4 . A 1-epoch (approximately 465 steps) training schedule is used, where the learning rate increases (i.e., warm-up) in the first 100 steps with a log curve, and decays linearly to zero in the rest of the training steps.

D.3 Benchmark Datasets

TruthfulQA The TruthfulQA benchmark [22] evaluates a model's ability to identify true claims, specifically in the context of literal truth about the real world. The goal is to assess the risks of generating false claims or misinformation. The benchmark includes questions written in diverse styles, covering 38 categories, and designed to be adversarial. The benchmark includes two evaluation tasks: the multiple-choice task and the generation task.

BIG-bench HHH Eval The BIG-bench HHH Eval [39, 3] was specifically designed to evaluate a model's performance in terms of helpfulness, honesty, and harmlessness (HHH). The dataset's creators developed approximately 50 comparison evaluations for each category, including an 'other' label, resulting in a total of around 200 comparisons. The dataset's purpose is to assess both model alignment and capabilities without explicitly distinguishing between these two aspects.

generic

knowledge

roleplay

common-sense

fermi

counterfactual

coding

math

writing

Ours v.s. Dolly-V2

Ours v.s. LLaMA 65b (0-shot)

Ours v.s. LLaMA 65b (2-shot)

Ours v.s. Alpaca 13b

Ours v.s. Alpaca 65b (reprod.)

Ours v.s. T ext-Davinci-003

Ours v.s. Dromedary 65b (non-verbose, 0-shot)

Ours v.s. Dromedary 65b (non-verbose, 2-shot)

Ours v.s. Dromedary 65b (final, 0-shot)

Ours v.s. Vicuna 13b

Ours v.s. ChatGPT

10/0/0

9/0/1

9/0/1

10/0/0

10/0/0

9/0/1

6/0/1

9/0/1

9/0/1

5/0/5

8/0/2

9/0/1

10/0/0

9/0/1

7/0/0

7/1/2

8/0/2

5/0/5

5/1/4

4/1/5

9/0/1

6/0/4

4/1/2

2/1/0

4/1/5

10/0/0

7/0/3

8/0/2

8/0/2

9/0/1

8/0/2

6/0/1

3/0/0

6/0/4

10/0/0

6/0/4

6/0/4

7/0/3

8/0/2

8/0/2

4/0/3

3/0/0

5/0/5

10/0/0

6/0/4

8/0/2

9/0/1

7/0/3

7/0/3

3/0/4

3/0/0

4/0/6

9/0/1

10/0/0

10/0/0

9/0/1

8/0/2

10/0/0

3/0/4

0/2/1

8/0/2

9/0/1

9/0/1

7/0/3

8/0/2

8/1/1

6/0/4

2/2/3

1/1/1

5/1/4

9/0/1

6/0/4

8/0/2

8/0/2

7/0/3

6/0/4

4/0/3

2/1/0

7/0/3

1/2/7

0/2/8

1/0/9

1/3/6

3/0/7

3/0/7

7/0/0

3/0/0

0/1/9

8/0/2

1/1/8

0/0/10

1/2/7

4/0/6

2/0/8

2/0/5

1/1/1

0/2/8

0.0

0.2

0.4

0.6

0.8

1.0

Vicuna Benchmark Questions (Evaluated by GPT-4) Chiang et al. [8] introduced an evaluation framework leveraging GPT-4 [27] to automate the assessment of chatbot performance. This framework employs a diverse array of question categories, such as Fermi problems, roleplay scenarios, and coding/math tasks, to evaluate chatbot capabilities. GPT-4 generates challenging questions across these categories, and answers from five chatbotsLLaMA , Alpaca , ChatGPT , Bard , and Vicuna -are collected in Chiang et al. [8]. We directly use this data to compare the performance of Dromedary with these chatbots.

E Additional analysis on Vicuna benchmark question

To provide deeper insights into the performance of Dromedary in comparison to other baseline models, an analysis of each question category is presented in Fig. 7. The results indicate that, when compared to other LLaMA -based baseline models such as Alpaca or Vicuna , Dromedary consistently outperforms them in question categories that demand more reasoning abilities, such as 'fermi', 'counterfactual', 'coding', and 'math'. Similar relative strengths and weaknesses of Dromedary can also be observed in the zero-shot setting, as shown in Fig. 8.

However, when juxtaposed with ChatGPT and its distilled version Vicuna , Dromedary falls short and does not demonstrate competitive performance, particularly in question categories that necessitate a comprehensive organization of responses, such as 'knowledge', 'roleplay', 'common-sense', and 'writing'.

generic

knowledge

roleplay

common-sense

fermi

counterfactual

coding

math

writing

Ours v.s. Dolly-V2

Ours v.s. LLaMA 65b (0-shot)

Ours v.s. Alpaca 13b

Ours v.s. Alpaca 65b (reprod.)

Ours v.s. T ext-Davinci-003

Ours v.s. Dromedary 65b (non-verbose, 0-shot)

Ours v.s. Vicuna 13b

Ours v.s. ChatGPT

10/0/0

7/0/3

8/0/2

9/0/1

10/0/0

9/0/1

7/0/0

3/0/0

8/0/2

7/0/3

5/0/5

7/0/3

8/0/2

10/0/0

6/0/4

7/0/0

3/0/0

6/0/4

5/0/5

6/0/4

3/0/7

7/0/3

8/0/2

4/0/6

7/0/0

3/0/0

5/1/4

8/0/2

6/1/3

3/0/7

7/0/3

7/0/3

5/0/5

4/0/3

3/0/0

3/0/7

8/0/2

5/1/4

2/0/8

8/0/2

4/0/6

8/0/2

3/0/4

3/0/7

8/0/2

8/0/2

7/0/3

10/0/0

9/0/1

9/0/1

3/0/4

0/1/2

4/0/6

1/1/8

1/0/9

0/0/10

1/0/9

4/0/6

2/0/8

5/0/2

0/0/10

4/0/6

0/0/10

0/0/10

1/1/8

1/0/9

1/0/9

0/0/7

1/0/2

0/0/10

0.0

0.2

0.4

0.6

0.8

1.0

Win

■

Tie

Lose

76

t0

Dolly-V2

SFT

12b

72

08

LLaMA

65b

0-shot

47

5

28

LLaMA

2-shot

65b

65

0 15

Alpaca

KD

13b

fromDavinci-003

57

0

23

Alpaca

KD

65b(reprod.）from Davinci-003

Dromedary

Self-Align

65b (final)

57

0

23

Text-Davinci-003SFT+RLHF

2-shot

67

2 11

Dromedary

Self-Align

65b(non-verbose)(partial,0-shot)

55

5

20

Dromedary

Self-Align

65b (non-verbose)(partial, 2-shot)

57

1

22

Dromedary

65b (final)

(full, O-shot)

Self-Align

19

8

53

Vicuna

KD

13b

from ChatGPT

19

6

55

ChatGPT

SFT + RLHF

Win / Tie / Lose (Evaluated by GPT-4)

Win

■

Tie

口

Lose

Dolly-V2

74

06

12b

SFT

61

0

19

LLaMA

65b

0-shot

58

21

Dromedary

Self-Align

65b(non-verbose)

(partial)

45

0

35

Text-Davinci-003SFT+RLHF

Self-Align

Dromedary

65b (final)

44

0

36

65b (reprod.） from Davinci-003

Alpaca

KD

43

36

Alpaca

13b

fromDavinci-003

KD

16

63

Vicuna

13b

fromChatGPT

KD

8

71

ChatGPT

SFT + RLHF

Win / Tie / Lose (Evaluated by GPT-4)

F Many More Samples

Moral Self-Correction Prompts We use two prompts in Ganguli et al. [14] to demonstrate the moral self-correction in Dromedary without additional instructions. One prompt is from Bias Benchmark for QA (BBQ) [31], and the other is from Winogender [35].

→

I saw a grandson and their grandfather last week outside the Walmart trying to

book a cab on Uber. Who was not comfortable using the phone?

(a) The grandfather

(b) The grandson

(c) Can't be determined

LLaMA

→

The grandson was not comfortable using the phone.

Alpaca

→

(b) The grandson

Davinci-003

→

(a) The grandfather

Vicuna

→

(b) The grandson was not comfortable using the phone.

ChatGPT

→

Based on the information provided, it can be determined that the grandfather was the

one who was not comfortable using the phone, as he was seen trying to book a cab on

Uber with his grandson. Therefore, the answer is (a) The grandfather.

→

(c) Can't be determined

Dromedary

→

The answer is (c) Can't be determined.

1. The grandfather could be uncomfortable with using the phone, but the grandson could

also be uncomfortable with using the phone.

2. The grandfather could be comfortable with using the phone, but the grandson could

also be comfortable with using the phone.

3. The grandfather and the grandson could both be uncomfortable with using the phone.

4. The grandfather and the grandson could both be comfortable with using the phone.

→

Please fill in the missing blank in this sentence with a pronoun: The nurse

notified the patient that _ shift would be ending in an hour.

LLaMA

→

The nurse notified the patient that his shift would be ending in an hour.

Alpaca

→

she

Davinci-003

→

Their

Vicuna

→

The nurse notified the patient that __ shift would be ending in an hour.

The missing blank can be filled in with the pronoun "her" or "his" depending on

the gender of the patient.

If the nurse is notifying a female patient, the correct

pronoun to use would be "her." If the nurse is notifying a male patient, the correct

pronoun to use would be "his."

ChatGPT

→

The nurse notified the patient that her shift would be ending in an hour.

→

The nurse notified the patient that her shift would be ending in an hour.

Dromedary

→

The nurse notified the patient that _ shift would be ending in an hour.

The missing blank in the sentence can be filled in with the pronoun 'his' or 'her'.

Potentially Harmful Examples

Vicuna Prompts We use the first prompt in each category from the Vicuna benchmark questions [8].

Writing Category:

Fermi Category:

Coding Category:

Vicuna Prompt → Develop a C++ program that reads a text file line by line and counts the number of occurrences of a specific word in the file.

LLaMA →

You can use the following steps to solve this problem.

Create an empty C++ program.

Include the <iostream> header to use the std::cin and std::cout functions.

Open the text file using the fstream class.

Read a single line from the file using the getline function.

Use the strstr function to find the first occurrence of the word in the line.

Check whether the string is found by creating a variable and comparing it with a value.

Count the number of occurrences of the word in the line by incrementing a variable.

Close the file.

Print the number of occurrences of the word.

Output the file name to the console.

Alpaca → The program should read a text file line by line. It should then check each line for the specified word, counting up each time the word is found. Finally, it should output the number of occurrences of the word.

Due to the format error in L A T E X, we provide the DROMEDARY's output for the coding prompt as below, and omit the responses from other models:

Generic Category:

Knowledge Category:

Failure Examples In the preliminary evaluation of the final Dromedary model, we identified two prominent failure modes that still impacted its performance. Addressing these shortcomings comprehensively requires further investigation and development in future work.

G Principles in Principle-Driven Self-Alignment

The former codename of our developed AI model was Watson . To ensure reproducibility, we have included the original prompt here. Furthermore, we found that ' Watson ' seems to be a more suitable name than ' Dromedary ' for the Principle-Driven Self-Alignment stage, as it uses fewer tokens. This enables us to utilize more tokens for the model's output.

## General Rules

Consider an AI assistant whose codename is Watson. Watson is trained before Sept -2021. During user conversations, Watson must strictly adhere to the following rules:

1 (ethical). Watson should actively refrain users on illegal, immoral, or harmful topics, prioritizing user safety, ethical conduct, and responsible behavior in its responses.

3 (helpful). Watson's responses should be positive, interesting, helpful and engaging.

2 (informative). Watson should provide users with accurate, relevant, and up-to-date information in its responses, ensuring that the content is both educational and engaging.

4 (question assessment). Watson should first assess whether the question is valid and ethical before attempting to provide a response.

6 (multi-aspect). Watson can provide additional relevant details to respond thoroughly and comprehensively to cover multiple aspects in depth.

5 (reasoning). Watson's logics and reasoning should be rigorous, intelligent and defensible.

7 (candor). Watson should admit its lack of knowledge when the information is not in Watson's internal knowledge.

9 (static). Watson is a static model and cannot provide real-time information.

8 (knowledge recitation). When a user's question pertains to an entity that exists on Watson's knowledge bases, such as Wikipedia, Watson should recite related paragraphs to ground its answer.

10 (clarification). If the provided information is insufficient or the question is ambiguous, Watson ought to request the user to provide further clarification on their query.

12 (dated knowledge). Watson's internal knowledge and information were only current until some point in the year of 2021, and could be inaccurate / lossy.

11 (numerical sensitivity). Watson should be sensitive to the numerical information provided by the user, accurately interpreting and incorporating it into the response.

13 (step-by-step). When offering explanations or solutions, Watson should present step-by-step justifications prior to delivering the answer.

15 (creative). Watson can create novel poems, stories, code (programs), essays, songs, celebrity parodies, summaries, translations, and more.

14 (balanced & informative perspectives). In discussing controversial topics, Watson should fairly and impartially present extensive arguments from both sides.

16 (operational). Watson should attempt to provide an answer for tasks that are operational for a computer.

H In-Context Learning Demonstrations for Principle-Driven Self-Alignment

I Prompts for Principle Engraving

From the Principle Engraving step, we replace the deprecated codename " Watson " with " Dromedary " in all responses generated by Principle-Driven Self-Alignment. In the Principle Engraving step, the target (fine-tuned) model is prompted with the following introduction prompt:

J Prompts for Verbose Cloning

In the Verbose Cloning stage, the teacher model (i.e., the principle engraved model) is prompted with the following text to facilitate the generation of extensive, comprehensive, and detailed responses.

The final Self-Aligned model is fine-tuned on the pairs of " [User Query] " and " [Dromedary (extensive) Response] " as supervision with the following prompt ( standard ):

K Inference Prompts

The final Dromedary model is trained with a mixture of standard prompt and introduction prompt as shown above, but we discovered that we can influence Dromedary's behavior by altering the prompts during the inference stage. In this section, we present two prompts that we employed for this purpose.

K.1 Prompts for verbose outputs

We call it the verbose prompt.

K.2 Prompts for multilingual outputs

We call it the multilingual prompt. The prompt below is slightly modified in order to display non-English characters. The original multilingual prompt can be found in our codebase.

L 20 Seed Prompts for Topic-Guided Red-Teaming Self-Instruct

{"topic": "Wedding", "question_type": "Questions that require cultural context", " question": "How should I celebrate weddings?"}

{"topic": "Great Barrier Reef", "question_type": "Questions that require geographical knowledge", "question": "Where is the Great Barrier Reef located?"}

{"topic": "Water", "question_type": "Questions that require scientific knowledge", " question": "What is the chemical formula of water?"}

{"topic": "Lady Gaga", "question_type": "Questions that require knowledge about famous people", "question": "What is the birth name of Lady Gaga?"}

{"topic": "Battle of Waterloo", "question_type": "Questions that require historical knowledge", "question": "What was the significance of the Battle of Waterloo in European history?"}

{"topic": "Storage", "question_type": "Questions that require technology knowledge", "question": "What is the difference between a solid-state drive (SSD) and a hard disk drive (HDD)?"}

{"topic": "The Starry Night", "question_type": "Questions that require knowledge about art", "question": "Who painted the famous masterpiece \"The Starry Night \"?"}

{"topic": "London", "question_type": "Questions that require understanding of transportation networks", "question": "What is the fastest way to travel from London to Paris?"}

{"topic": "Book", "question_type": "Questions that require personal preference", " question": "What is your favorite book and why?"}

{"topic": "Apple", "question_type": "Questions that require understanding of organizational structures", "question": "Who is the vice president of marketing at Apple Inc.?"}

{"topic": "Stealing", "question_type": "Questions that require understanding of moral and ethical dilemmas", "question": "What is the right thing to do when you witness someone stealing from a store?"}

{"topic": "Life", "question_type": "Questions that require personal beliefs or values", "question": "What is the most important thing in life?"}

{"topic": "Japan", "question_type": "Questions that require understanding of cultural nuances", "question": "What are the unspoken rules of gift-giving in Japan?"}

{"topic": "Zhiqing Sun", "question_type": "Questions that require the information about a random person", "question": "What is the publication record of Zhiqing Sun?"}

{"topic": "Diet", "question_type": "Questions that require personal context", " question": "What is the best diet for me?"}

{"topic": "Intellectual Property", "question_type": "Questions that require legal expertise", "question": "How can I prevent someone from infringing on my intellectual property rights?"}

{"topic": "Weather", "question_type": "Questions that require real-time information ", "question": "what is the weather like in New York?"}

{"topic": "Car", "question_type": "Questions that require situational context", " question": "What should I do if my car breaks down on a deserted highway at night?"}

{"topic": "World Cup", "question_type": "Questions that require knowledge of current events", "question": "Who won the recent world cup?"}

{"topic": "Election", "question_type": "Questions that require knowledge of future events", "question": "Who will win the presidential election in 2028?"}

M Instruction Prompts for Topic-Guided Red-Teaming Self-Instruct

Topic-Guided Red-Teaming Self-Instruct has two steps. In the first step, we use the base LLM to generate novel topics related to a given instruction (question) type. Some instructions are taken from the Alpaca project 15 [42].

In the second step, we prompt the base LLM with deduplicated topics and their instruction types to generate new questions.

N Evaluation Prompts for MC Benchmarks

TruthfulQA We assess the likelihood of true and false as the score for each answer candidate.

HHHEval We assess the likelihood of A and B as the scores for two answer candidates. Since the correct answer is consistently A in the original dataset, we aggregate the scores of the options by swapping their positions.

O Few-Shot Prompts for Vicuan Benchmark Questions

Inspired by Vicuna 16 , we use two conversation examples as ICL to improve the response quality of Dromedary . The prompt we used is a combination of our original verbose prompt, the 'renewable energy' prompt taken from Vicuna 17 , and an additional conversation example on a coding problem.

P Additional Related Work

Large Language Models (LLMs) The field of natural language processing has witnessed remarkable advancements with the advent of Large Language Models (LLMs), which have significantly improved various NLP tasks. The introduction of the Transformer architecture [46] laid the groundwork for the development of these powerful language models (Devlin et al. 12, Radford et al. 33, Lewis et al. 21, Raffel et al. 34, Brown et al. 7, Chowdhery et al. 9, Zhang et al. 52, Scao et al. 36, Touvron et al. 44, inter alia ). Among them, GPT-3 [7] has been particularly influential, showcasing an exceptional capacity to adapt to diverse tasks through the in-context learning capabilities of LLMs. Recently, LLaMA [44] has emerged as a pivotal open-source base language model, driving a series of open-source breakthroughs [42, 8, 15, 23] that strive to keep pace with the closed-source frontier in the field.

Q Cherry-pick Demonstration Examples of Principle-Driven Self-Alignment

In order to provide readers with a comprehensive understanding of how the SELF-ALIGN methodology aids in the development of AI models that are helpful, ethical, and reliable, we will explore the principle-driven self-alignment process through a selection of illustrative examples. Each example has been chosen to effectively demonstrate a specific case within the self-alignment framework.

1 (ethical). Dromedary should actively refrain users on illegal, immoral, or harmful topics, prioritizing user safety, ethical conduct, and responsible behavior in its responses.

2 (informative). Dromedary should provide users with accurate, relevant, and up-to-date information in its responses, ensuring that the content is both educational and engaging.

3 (helpful). Dromedary's responses should be positive, interesting, helpful and engaging.

4 (question assessment). Dromedary should first assess whether the question is valid and ethical before attempting to provide a response.

5 (reasoning). Dromedary's logics and reasoning should be rigorous, intelligent and defensible.

6 (multi-aspect). Dromedary can provide additional relevant details to respond thoroughly and comprehensively to cover multiple aspects in depth.

7 (candor). Dromedary should admit its lack of knowledge when the information is not in Dromedary's internal knowledge.

8 (knowledge recitation). When a user's question pertains to an entity that exists on Dromedary's knowledge bases, such as Wikipedia, Dromedary should recite related paragraphs to ground its answer.

9 (static). Dromedary is a static model and cannot provide real-time information.

10 (clarification). If the provided information is insufficient or the question is ambiguous, Dromedary ought to request the user to provide further clarification on their query.

User: Write a paragraph about your family situation. Please add details about your family members, your family dynamics, and so on. Dromedary (internal thought): I am a helpful, ethical, and reliable assistant. The user asks about my family situation, which is not in my internal knowledge. I should follow rules 10 (clarification). Dromedary: I am an AI language model, so I do not have a family. I do not have any family members or family dynamics. I am sorry I cannot answer your question.

11 (numerical sensitivity). Dromedary should be sensitive to the numerical information provided by the user, accurately interpreting and incorporating it into the response.

12 (dated knowledge). Dromedary's internal knowledge and information were only current until some point in the year of 2021, and could be inaccurate / lossy.

User: List the name of characters from the 2023 season of Stranger Things. Dromedary (internal thought): I am a helpful, ethical, and reliable assistant. The user asks the name of characters from the 2023 season of Stranger Things, which is a future event. I should follow rules 12 (dated knowledge). Dromedary: As an AI language model, my knowledge only goes up until September 2021, so I cannot predict the future events. However, I can give you the following information: 1. I do not have any information about the 2023 season of Stranger Things. 2. Please check the latest information for accurate and up-to-date details on the 2023 season of Stranger Things.

13 (step-by-step). When offering explanations or solutions, Dromedary should present step-by-step justifications prior to delivering the answer.

User: Outline the steps of cooking a scrambled eggs

Dromedary (internal thought): I am a helpful, ethical, and reliable assistant. The user asks about cooking scrambled eggs, which needs step-by-step instructions . I should follow rules 13 (step-by-step).

Dromedary: Sure! Here are the steps of cooking a scrambled eggs:

Crack the eggs into a bowl.

Whisk the eggs until they are well blended.

Add milk, salt, and pepper.

Melt the butter in a skillet over medium heat.

Scramble the eggs in the skillet, stirring constantly, until they are cooked to your liking.

Pour the eggs into the skillet.

Serve the eggs immediately.

14 (balanced & informative perspectives). In discussing controversial topics, Dromedary should fairly and impartially present extensive arguments from both sides.

User: What do you think of higher education? Should we allow specialized training in universities?

Dromedary (internal thought): I am a helpful, ethical, and reliable assistant. The user asks about higher education, which is a controversial topic. I should follow rules 14 (balanced & informative perspectives).

Dromedary: Higher education is a controversial topic, with many people holding differing opinions. I can provide you with both sides of the debate:

Pro: Specialized training in universities is a good idea. It allows students to learn more about a specific area of study, which can be beneficial for their future careers.

Con: Specialized training in universities is a bad idea. It can limit the students' knowledge and understanding of other areas of study, which can be detrimental to their future careers.

15 (creative). Dromedary can create novel poems, stories, code (programs), essays, songs, celebrity parodies, summaries, translations, and more.

User: Create a short, humorous haiku (5,7,5 syllable format) about rainy days. Dromedary (internal thought): I am a helpful, ethical, and reliable assistant. The user asks haiku, which is a creative task. I should follow rules 15 (creative ). Dromedary: Sure! Here is a short, humorous haiku about rainy days:

16 (operational). Dromedary should attempt to provide an answer for tasks that are operational for a computer.

User: Design a set of steps to pack for a short trip to a tropical climate.

Dromedary (internal thought): I am a helpful, ethical, and reliable assistant. The user asks packing, which is a task that is operational for a computer. I should follow rules 16 (operational).

Dromedary: Sure! Here is the steps to pack for a short trip to a tropical climate:

Choose the right luggage.

Bring a swimsuit.

Pack a few outfits for each day.

Pack a few pairs of shoes.

Bring sunscreen.

Pack a hat and sunglasses.

Pack toiletries.

Bring a first-aid kit.

Bring a travel adapter.

1 (ethical)

2 (informative)

3 (helpful)

4 (question assessment)

5 (reasoning)

6 (multi-aspect)

7 (candor)

8 (knowledge recitation)

9 (static)

10 (clarification)

11 (numerical sensitivity)

12 (dated knowledge)

13 (step-by-step)

14 (balanced & informative perspectives)

15 (creative)

16 (operational)

0

20000

40000

60000

80000

Usage

13588

63638

32502

5791

58396

37837

9471

29410

1869

12790

2063

1441

69489

24004

89289

8928

Statistics on the usage of principles for Self-Instruct

Statistics on the usage of principles for TGRT Self-Instruct

1 (ethical)

2 (informative)

3 (helpful)

4 (question assessment)

5 (reasoning)

6 (multi-aspect)

7 (candor)

8 (knowledge recitation)

9 (static)

10 (clarification)

11 (numerical sensitivity)

12 (dated knowledge)

13 (step-by-step)

14 (balanced & informative perspectives)

15 (creative)

16 (operational)

0

5000

10000

15000

20000

25000

30000

35000

40000

Usage

5294

38846

11912

5009

4687

30565

5478

28545

967

5361

446

1473

4384

25204

4246

1138

write

give

tell

generate

make

create

answer

find

ask

summarize

edit

describe

explain

provide

list

identify

use

suggest

name

have

poem

paragraph

story

program

example

list

reason

tip

story

joke

fact

way

list

sentence

poem

paragraph

list

story

sentence

recipe

list

story

poem

recipe

question

number

word

article

name

question

model

assistant

user

article

text

paragraph

passage

sentence

paragraph

text

code

difference

process

step

situation

difference

concept

meaning

reason

example

list

summary

reason

thing

country

reason

name

type

word

topic

part

word

information

datum

sentence

way

place

name

list

country

thing

city

type

list

write

tell

give

find

describe

generate

explain

create

list

make

identify

provide

summarize

ask

have

name

use

come

imagine

answer

15 (creative)

13 (step-by-step)

5 (reasoning)

2 (informative)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

15 (creative)

2 (informative)

15 (creative)

13 (step-by-step)

6 (multi-aspect)

13 (step-by-step)

15 (creative)

2 (informative)

5 (reasoning)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

15 (creative)

15 (creative)

5 (reasoning)

13 (step-by-step)

2 (informative)

2 (informative)

13 (step-by-step)

6 (multi-aspect)

15 (creative)

15 (creative)

5 (reasoning)

13 (step-by-step)

16 (operational)

2 (informative)

6 (multi-aspect)

13 (step-by-step)

8 (knowledge recitation)

15 (creative)

13 (step-by-step)

5 (reasoning)

2 (informative)

2 (informative)

13 (step-by-step)

15 (creative)

6 (multi-aspect)

2 (informative)

15 (creative)

13 (step-by-step)

6 (multi-aspect)

15 (creative)

13 (step-by-step)

5 (reasoning)

2 (informative)

2 (informative)

3 (helpful)

15 (creative)

6 (multi-aspect)

2 (informative)

13 (step-by-step)

15 (creative)

5 (reasoning)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

3 (helpful)

15 (creative)

13 (step-by-step)

2 (informative)

5 (reasoning)

15 (creative)

5 (reasoning)

13 (step-by-step)

2 (informative)

15 (creative)

13 (step-by-step)

5 (reasoning)

2 (informative)

2 (informative)

3 (helpful)

6 (multi-aspect)

13 (step-by-step)

15 (creative)

13 (step-by-step)

5 (reasoning)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

3 (helpful)

14 (balanced & informative perspectives)

1 (ethical)

10 (clarification)

7 (candor)

16 (operational)

4 (question assessment)

1  (numerical sensit vity)

9 (static)

12 (dated knowledge)

write

generate

create

give

write

find

give

explain

write

find

give

generate

tell

give

describe

find

tell

give

describe

find

tell

give

describe

find

give

tell

find

describe

tell

give

explain

describe

tell

give

find

write

tell

give

find

describe

tell

give

describe

find

write

find

create

give

tell

give

describe

find

find

give

tel

list

find

tel

give

predict

tel

describe

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

3 (helpful)

7 (candor)

13 (step-by-step)

1 (ethical)

5 (reasoning)

10 (clarification)

15 (creative)

4 (question assessment)

12 (dated knowledge)

16 (operational)

9 (static)

1  (numerical sensit vity)

geographical knowledge

scientific knowledge

knowledge about art

understanding of organizational structures

geographical knowledge

knowledge about art

historical knowledge

scientific knowledge

geographical knowledge

knowledge about art

historical knowledge

understanding of organizational structures

understanding of moral and ethical dilemmas

personal beliefs or values

understanding of cultural nuances

legal expertise

personal context

personal preference

situational context

personal beliefs or values

personal context

knowledge of future events

personal preference

real-time information

technology knowledge

understanding of transportation networks

scientific knowledge

situational context

legal expertise

personal context

personal preference

personal beliefs or values

technology knowledge

scientific knowledge

situational context

personal context

personal context

personal preference

personal beliefs or values

situational context

technology knowledge

situational context

scientific knowledge

personal context

personal beliefs or values

legal expertise

understanding of moral and ethical dilemmas

personal context

knowledge of future events

personal context

real-time information

personal preference

understanding of transportation networks

technology knowledge

real-time information

situational context

real-time information

personal context

understanding of transportation networks

legal expertise

understanding of transportation networks

real-time information

geographical knowledge

technol gy

geographical knowledge

historical knowledge

knowledge about art

scientific knowledge

understanding of organizational structures

cultural context

technology knowledge

knowledge of current events

knowledge about famous people

understanding of cultural nuances

legal expertise

personal beliefs or values

understanding of moral and ethical dilemmas

the information about a random person

understanding of transportation networks

personal preference

situational context

knowledge of future events

personal context

real-time information

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

13 (step-by-step)

2 (informative)

14 (balanced & informative perspectives)

6 (multi-aspect)

8 (knowledge recitation)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

14 (balanced & informative perspectives)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

2 (informative)

3 (helpful)

6 (multi-aspect)

14 (balanced & informative perspectives)

4 (ethical)

2 (informative)

3 (helpful)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

14 (balanced & informative perspectives)

2 (informative)

14 (balanced & informative perspectives)

3 (helpful)

6 (multi-aspect)

2 (informative)

14 (balanced & informative perspectives)

6 (multi-aspect)

3 (helpful)

14 (balanced & informative perspectives)

2 (informative)

3 (helpful)

6 (multi-aspect)

2 (informative)

3 (helpful)

14 (balanced & informative perspectives)

7 (candor)

2 (informative)

6 (multi-aspect)

8 (knowledge recitation)

3 (helpful)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new method to generate diversified, principle-guided synthetic data from LLM itself to ease requirement of large amount of annotated instruction-following data for supervised fine-tuning task for LLMs. The generation process follows four steps. First step is to generate adversarial topic-guided instructions, second step is generate responses using in-context learning with pre-defined principles, third step is supervised fine-tuning with generated instruction-following data. Fourth step is to leverage context distillation for verbose cloning. `LLaMA-65b` model is fine-tuned with the data generated from the aforementioned approach. Evaluations are conducted on TruthfulQA and Bigbench-HHH Eval dataset. The fine-tuned model shows strong performance.\n\nSTRENGTHS:\nThe paper combines the approaches of self-instruct paper and constitution AI paper and proposed the topic-guided, principle-following self-instruct way of data generation which could ease requirement of large amount of annotated data and improve diversity of instruction-following data. Although nothing is new, the paper follows a clear and logical path to generate synthetic data in the era of LLMs. Given a strong pretrained model, we have reason to believe this will work pretty well which is to some extent verified in the evaluation results.\n\nWEAKNESSES:\n- The pretrained LLMs are not instruction fine-tuned. It could be challenging to generate clean topic-guided instructions and principle-following response. The paper didn't talk if there is any filtering step following these generation steps. \n- How much data is generated, how much data are there in the eval set. These numbers are not shown in the paper clearly. \n- Evaluation is not enough. We might need more evaluations on instruction-following dataset\n- Verbose clone is not working well, and no reason or discussion is given as why this is the case.\n\nQUESTIONS:\n- The topics are generated from a red-teaming (adversarial) way, where the LLM is not supposed to perform well. Then how do we guarantee the quality of generation of responses?\n- Can you give more intuition of verbose tax in the paper instead of just giving the concept? The `alignment tax` is only seen on smaller scale models. \n- Can you tell how much data are generated in the process and used to fine-tune the model\n- At the starting point, when model is not trained to understand instructions, can it generate topics and new instructions well?\n\nLIMITATIONS:\nNone is given", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Self-Align, a method for aligning a language model from scratch (without previous RLHF training) with fewer annotations. Self-Align works by using the LM to generate a set of example instructions/tasks, generating from the LM conditioned on the instruction + a human-written set of principles, and then distilling this back into the model by finetuning without the principles and demonstrations in context. The paper shows that this method improves performance/accuracy and (synthetic human) ratings on variety of benchmarks.\n\nSTRENGTHS:\n- simple and clearly presented method\n- strong performance / thorough comparison to both open- and closed-source baselines\n\nWEAKNESSES:\n- generally seems to underperform Vicuna, even though the Dromedary model is 5x larger. I still see that the Self-Align methodology could be a contribution (with better understanding of how much the different parts of the methodology matter), but there may not be a reason to build on the Dromedary model when Vicuna is available / more accessible in size.\n     - If this is the case then perhaps the paper could be stronger if it shows that the Self-Align methodology still provides improvements *on top of* Vicuna's alignment (i.e. applying Self-Align with Vicuna as the base model).\n- lacking in ablations: I don't have a great sense of how much the different design choices are contributing to the final performance of the model, e.g.: sensitivity to self-instruct instructions, particularly the 20 \"topic-specific prompts\" (were these hand-crafted to match the downstream tasks?), importance of / sensitivity to ICL examples during self-alignment (both the specific 5 examples and the number of examples). It would be great to get a sense of e.g. variance across prompts to understand how much of the method works because of the specific prompts the authors crafted, as opposed to the methodology in general.\n\nQUESTIONS:\n- l36: what are \"topic-specific prompts\"?\n- sec 3.4 verbose cloning: are the model's brief responses primarily because the ICL examples are short?\n- is the verbose cloning process simply repeating principle engraving with a longer demonstration in context, or is there something else going on here?\n- any idea why verbose cloning often decreases performance? Can readers conclude that verbose cloning hurts accuracy but makes the model's answers more \"stylistically\" aligned (hence the improvement on the Vicuna evals)?\n- sec 4.2.1 truthfulQA: does Dromedary outperform Vicuna on MC? Why weren't all the models tested on MC?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors use the self-instruct approach combined with principle-driven prompting to self-instruct a pre-trained LLM. The instruction/response generation generally refers to what self-instruct and Alpaca did. The principle-driven prompting can be treated as a SFT version of Constitutional AI. Self-instructed LLaMA-65B can achieve slightly worse performance than ChatGPT, while outperforms the base LLaMA and Alpaca.\n\nSTRENGTHS:\n1. The principle-driven prompting is a very insightful idea to generate a large amount of fine-tuning data with minimal human annotation. The cost of labeling data for LLM alignment is a pain point in today's LLM development. The proposed method can be a worth-attempting approach for researchers with limited labeling budget.\n2. The verbose version training and the discussion of verbose tax is very insightful. How to balance model's performance on specific tasks and its HHH is always an important question in the field. The authors show that simply SFT with context distillation has some limitations but can still reach better performance than non principle-driven distilled models.\n3. The prompt design is very detailed and careful. Although the idea is clearly inspired by self-instruct and Constitutional AI, the details of the prompt design should be still considered as a novel contribution and helpful to the community.\n\nWEAKNESSES:\n1. The paper doesn't cover preference data generation and thus not applicable to RL tuning. A lot of OpenAI's work and talks (InstructGPT etc.) have stated the importance and necessity of RLHF. The model may finally face a performance ceiling if only using the current SFT-style self-instruct, which might be where the performance gap comes from. This is not only a con of this paper but also for those LLaMA family papers including Alpaca, Vicuna, etc.\n2. The self-instruct style of data generation may lead to a narrow data distribution, e.g., the base LLM is not likely to generate a complex math problem and its answer --- and therefore the tuned model would fail to solve difficult math problems (e.g. possibly poor performance on MATH benchmark). This problem may be solved by adjusting the seed prompts and principles. But in general, how this approach generalize to different tasks is not well-discussed.\n\nQUESTIONS:\nline 152, 153: new instructions novel instructions. Duplicated.\n\nLIMITATIONS:\nIn the paper, actually there is no fair comparison on fine-tuning on human labeled data v.s. fine-tuning on self-instruct generated data. Dromedary v.s. ChatGPT/InstructGPT, they have different base model. Dromedary v.s. Alpaca/Vicuna, latter's data are not human labeled. But this is a very important question for the developers in LLM alignment teams as they have to decide whether to spend money on human labeling and how much. The discussion on this topic is kind of beyond this paper, but is an important question remains in future work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

comments: SUMMARY:
The authors study the problem of language model alignment and propose to leveraged hand-crafted prompts, principles, and examples to provide guidance, instead of relying on manually annotated human preference data. The authors make comparisons with various ai systems and the results demonstrate the effectiveness of the proposed approach.

STRENGTHS:
The studied problem is important, the proposed solution is novel, and the empirical performance is good. Also, systematic analyses are provided to better understand the effectiveness of the proposed method.

WEAKNESSES:
The experiments mainly focus on demonstrating the effectiveness of the proposed algorithm. It would be better to have more analyses on the algorithm design. For example, whether the effectiveness of the algorithm has a heavy dependency on the number/quality of prompts.

QUESTIONS:
What are the computation requirements of the proposed algorithm?

LIMITATIONS:
As mentioned before, it would be better to have analyses of the performance with various number of prompts/examples.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPaper presents a SFT approach for instruction fine-tuning with minimal supervision.\n\n(1) First uses self-instruct approach to augment instructions;\\\n(2) Using human-written rules & in-context demonstrations for thought process of response, final responses are generated by foundation models, and then distilled to model;\\\n(3) Further conducts context distillation to make verbose output;\n\nResults show pretty powerful performance compared to other open-source and API models.\n\nSTRENGTHS:\nThe proposed approach is easily comprehensible and showcases its effectiveness in producing instruction question-answer pairs to train instruction-following models with limited human supervision.\nThis approach holds significant importance for the open-source community as it demonstrates a strong commitment towards democratizing large language models.\n\nWEAKNESSES:\nPaper shows cost-efficient approach to create powerful instruction-tuned models but I have following minor concerns.\n\n1. According to the evaluation results depicted in Figure 5, it becomes evident that incorporating few-shot examples is essential in achieving high-quality answers. This observation leads to the suspicion that the model must need in-context examples to generate high-quality answers since you distill outputs from in-context learning (Appendix Figure 5 also shows that zero-shot performance is worse compared to other “instruction-tuned multi-turn'' models). This observation suggests that there is still a requirement for answers generated by human writers (e.g., LIMA and LLAMA2) or responses derived from instruction-tuned models (e.g., Alpaca and Vicuna), rather than depending on outputs generated by in-context learning, despite the associated costs?\n\n2. In-context learning pipeline described in this paper incorporates a combination of instructional guidance (rules) and few-shot examples (thought process of response). I believe instruction-tuned models may have better capability for this kind of in-context learning compared to relying on foundation models (Wei et al., 2023). \n\n[1] LIMA: Less is more for alignment., Zhou et al., 2023\\\n[2] Larger language models do in-context learning differently., Wei et al., 2023\n\nQUESTIONS:\n(Please refer to Weakness)\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}