SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhiqing Sun $^{1*}$
Yikang Shen
Qinhong Zhou<sup>3</sup>
Hongxin Zhang<sup>3</sup>
David Cox2
Chuang Gan2,3
$^{1}$ Language Technologies Institute, CMU
$^{2}$ MIT-IBM Watson AI Lab, IBM Research
$^{3}$ UMass Amherst
https://github.com/IBM/Dromedary
Recent AI-assistant agents, such as ChatGPT, predominantly rely on supervised fine-tuning (SFT) with human annotations and reinforcement learning from human feedback (RLHF) to align the output of large language models (LLMs) with human intentions, ensuring they are helpful, ethical, and reliable. However, this dependence can significantly constrain the true potential of AI-assistant agents due to the high cost of obtaining human supervision and the related issues on quality, reliability, diversity, self-consistency, and undesirable biases. To address these challenges, we propose a novel approach called SELF-ALIGN, which combines principle-driven reasoning and the generative power of LLMs for the self-alignment of the AI agents with minimal human supervision.
Applying SELF-ALIGN to the LLaMA-65b base language model, we develop an AI assistant named Dromedary. With fewer than 300 lines of human annotations (including  $< 200$  seed prompts, 16 generic principles, and 5 exemplars for in-context learning), Dromedary significantly surpasses the performance of several state-of-the-art AI systems, including Text-Davinci-003 and Alpaca, on benchmark datasets with various settings. We have open-sourced the code, LoRA weights of Dromedary, and our synthetic training data to encourage further research into aligning LLM-based AI agents with enhanced supervision efficiency, reduced biases, and improved controllability.
The problem of aligning large language models (LLMs) to human values and intentions in terms of being comprehensive, respectful, and compliant<sup>1</sup> [10, 32, 30, 4, 5, 27] has gained significant attention in research as recent AI systems (like ChatGPT or GPT-4) have rapidly advanced in their capabilities [12, 33, 7, 9]. Presently, state-of-the-art AI systems predominantly depend on supervised fine-tuning (SFT) with human instructions and annotations, as well as reinforcement learning from human feedback (RLHF) on their preferences [26, 28, 29, 2]. The success of these techniques heavily
*Correspondence: zhiqings@cs.cmu.edu.
This is the definition of AI alignment in this paper, distinct from following simple instructions [30, 48, 42].
Figure 1: An illustration of the four essential stages in the SELF-ALIGN process
relies on the availability of extensive human supervision, which is not only expensive to obtain but also has potential issues with the quality, reliability, diversity, creativity, self-consistence, undesirable biases, etc., in human-provided annotations [48, 20, 47].
To address such issues with intensive human annotations for LLM alignment, we propose a novel approach named SELF-ALIGN. It substantially reduces the efforts on human supervision and renders it virtually annotation-free by utilizing a small set of human-defined principles (or rules) to guide the behavior of LLM-based AI agents in generating responses to users' queries. Our approach encompasses four essential stages:
1. (Topic-Guided Red-Teaming) Self-Instruct: We employ the self-instruct mechanism by Wang et al. [48] with 175 seed prompts to generate synthetic instructions, plus 20 topic-specific prompts in addition to ensure a diversified topic coverage of the instructions. Such instructions ensure a comprehensive range of contexts/scenarios for the AI system to learn from.
2. Principle-Driven Self-Alignment: We offer a small set of 16 human-written principles in English about the desirable quality of the system-produced responses, or the rules behind the behavior of the AI model in producing answers<sup>2</sup>. These principles function as guidelines for generating helpful, ethical, and reliable responses. We conduct in-context learning (ICL) [7] with a few (5) exemplars (demonstrations) that illustrate how the AI system complies with the rules when formulating responses in different cases. From the human-written principles, ICL exemplars, and the incoming self-instructed prompts, the LLM can trigger the matching rules and generate the explanations for a refused answer if the query is detected as a harmful or ill-formed one.
3. Principle Engraving: In the third stage, we fine-tune the original LLM (the base model) on the self-aligned responses, generated by the LLM itself through prompting, while pruning the principles and demonstrations for the fine-tuned model. The fine-tuning process enables our system to directly generate responses that are well-aligned with the helpful, ethical, and reliable principles across a wide range of queries, due to shared model parameters. Notice that the fine-tuned LLM can directly generate high-quality responses for new queries without explicitly using the principle set and the ICL exemplars.
4. Verbose Cloning: Lastly, we employ context distillation [18, 3] to enhance the system's capability to produce more comprehensive and elaborate responses than the overly short or indirect responses.
Impressively, the entire SELF-ALIGN process necessitates fewer than 300 lines of annotations (including 195 seed prompts, 16 principles, and 5 exemplars), while previous aligned AI systems such as InstructGPT [30] or Alpaca [42] required at least 50K human/teacher annotations. This
2The detailed principles are given in the appendix. Analogous to Constitutional AI [5], the design of these principles in SELF-ALIGN remains exploratory and primarily serves research purposes.
Table 1: Comparison of human/teacher supervisions used in recent AI systems. The alignment techniques used in previous work include SFT (Supervised Fine-tuning), RLHF (Reinforcement Learning from Human Feedback), CAI (Constitutional AI), and KD (Knowledge Distillation). Information is from:  ${}^{a}$  OpenAI [29],  ${}^{b}$  OpenAI [26],  ${}^{c}$  Bai et al. [5],Anthropic [2],  ${}^{d}$  OpenAI [27].
Total Annotations	Annotation Sources	Alignment Techniques
(closed-source models)
InstructGPT	77K	Users & Annotators	SFT & RLHF
Text-Davinci-003	?	?	SFT & RLHFa
ChatGPT	?	?	SFT & RLHFb
Claude	?	?	RLHF & CAIc
GPT-4	?	?	SFT & RLHF & CAId
(open-source models)
Alpaca	52K	Text-Davinci-003	Self-Instruct & KD
Vicuna	70K	Users & ChatGPT	KD
Koala	472K	Humans & Teacher Models	KD & SFT
OpenAssistant	161K	Annotators	SFT & RLHF
Dolly-V2	15K	Annotators	SFT
Dromedarye	< 300 lines	Humans	Self-Instruct & Self-Align
highlights the supervision efficiency of our approach in comparison with other state-of-the-art AI assistants, as shown in Table. 1. Our principle-driven approach, which is essentially rule-based, not only significantly reduces the required human effort for supervision but also showcases aligning neural language models with human understanding of principles or rules about quality language generation in both an effective and efficient manner.
We should also point out that the advancements of recent models like Alpaca and Vicuna have shown that the potent conversational capabilities can be obtained by distilling existing human-preference-aligned LLMs (i.e., Text-Davinci-003 and ChatGPT, respectively) into smaller, more manageable models [42, 8, 29, 26]. Those resulting smaller models, however, still rely on the successful alignment of existing LLMs, which are based on extensive human-provided supervision. In other words, those smaller models indirectly inherit the dependence on the availability of intensive supervision from humans. In contrast, our approach focuses on language model alignment from scratch, independent from the existence of well-aligned LLMs like ChatGPT or GPT-4. That is the main distinction of our approach from other existing approaches and is why we call it self-alignment from scratch.
We are providing the code for the SELF-ALIGN method as open source to promote collaboration and innovation within the research community. The base model of Dromedary is the LLaMA-65b language model [44], which is accessible for research-only, noncommercial purposes. By investigating different strategies from that in RLHF, our work seeks to broaden the scope of AI alignment techniques, and promote a deeper understanding of how to improve AI systems, not only in terms of being more powerful, but also more responsible and well-aligned with human values.
AI Alignment The domain of AI alignment [13] has garnered substantial attention in recent years, with LLMs exhibiting remarkable proficiencies across a wide array of tasks. GPT-4 [27] epitomizes this development, implementing a post-training alignment process to bolster factuality and adherence to desired behavior, while concurrently mitigating potential risks. A prominent strategy for aligning language models with human values entails fine-tuning via human feedback. Notably, Ouyang et al. [30] and Bai et al. [4] utilized reinforcement learning from human feedback (RLHF) to refine models, enhancing helpfulness and truthfulness, and diminishing toxic output generation. This technique requires extensive human annotations.
Constitutional AI (CAI) or self-critique [5, 27] investigates self-improvement without human labels for harmful outputs, leveraging AI-generated self-critiques, revisions, and preference models. Based on a list of human-generated rules or principles, this approach fosters the evolution of safe, reliable, and effective AI systems with increased behavioral precision and reduced dependency on human labels. Both SELF-ALIGN and CAI are rule-based alignment techniques for powerful AI systems. However, there are substantial differences between them, as outlined below:
Figure 2: Side-by-side comparison: on the left is a typical SFT + RLHF alignment pipeline (InstructGPT [30]), and on the right are the four stages in our SELF-ALIGN procedure.
- In the principle-driven self-alignment procedure of SELF-ALIGN, the language model itself determines which rules to adhere to given user queries, and it subsequently generates appropriate responses conditional on these queries and rules. Conversely, CAI employs a self-critique methodology; given a pair comprising a user query and the model's response, it selects a rule to scrutinize the existing response, thereby yielding a refined output.
- The self-critique nature of CAI necessitates RLHF warm-up. In stark contrast, SELF-ALIGN explores the alignment of language models from scratch, requiring minimal human supervision.
- However, one limitation of SELF-ALIGN is the requirement to include all rules within the context, a process bound by the base language model's token limit. In contrast, the CAI technique is not subject to this token limit constraint<sup>3</sup> as a post-generation self-critique method.
State-of-the-art AI Assistants State-of-the-art AI-assistant agents have significantly advanced in recent years, with InstructGPT [30] leading the way as the first model trained with supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) on user queries. ChatGPT [26], a sibling model to InstructGPT, has garnered widespread success as a commercial AI assistant, showcasing its ability to follow instructions in prompts and provide detailed responses. Alpaca [42], as a subsequent open-source model, was developed using Self-Instruct [48] to learn the knowledge from Text-Davinci-003 (similar to InstructGPT) [29], offering cost-effective and accessible alternatives. In parallel, models like Vicuna, Koala, and Baize [8, 15, 50] have been trained on ChatGPT outputs, essentially distilling the ChatGPT model to create new open-source chatbots. Dolly-V2 [11], another open-source effort, utilizes 15k new instruction-following data points for training. OpenAssistant [20] follows a similar approach to ChatGPT by collecting its own data. These advancements in AI assistants continue to push the boundaries of usability and accessibility, making significant strides in the open-source domains.
Our SELF-ALIGN approach distinguishes itself by concentrating on the creation of novel alignment techniques for LLMs, developed from the ground up and independent of established AI systems, while requiring minimal human supervision. This research direction aims to investigate the potential of aligning AI models under circumstances where dependence on or access to existing systems may be unfeasible or unfavorable. A comparison of annotation cost between SELF-ALIGN and previous methods is shown in Table. 1 and Figure. 2.
3For example, the latest Claude employs at least 58 rules [1].
Figure 3: Illustration of Principle-Driven Self-Alignment and Principle Engraving.
3 Our Method: SELF-ALIGN
The SELF-ALIGN method involves four distinct stages. The first stage is called Topic-Guided RedTeaming Self-Instruct, which employs the language model itself to generate synthetic instructions and enhance diversity via a topic-guided red-teaming approach. The second stage, Principle-Driven Self-Alignment, defines a set of principles that the AI model must adhere to and provides in-context learning demonstrations for constructing helpful, ethical, and reliable responses. The third stage, Principle Engraving, fine-tunes the base language model by pruning principles and demonstrations, empowering the model to directly generate appropriate responses. Finally, the fourth stage, Verbose Cloning, serves as a complementary step to address challenges arising from overly-brief or indirect responses by refining the model to produce detailed and comprehensive answers to user queries. We will describe each of these stages in detail.
3.1 Topic-Guided Red-Teaming Self-Instruct
The Self-Instruct method [48] is a semi-automated, iterative bootstrapping process that harnesses the capabilities of a pretrained LLM to generate a wide array of instructions (and corresponding outputs). The method commences with 175 manually-written instructions<sup>4</sup>, and the LLM proceeds to develop new tasks and augment the task pool (after eliminating low-quality or repetitive instructions). This process is executed iteratively until a satisfactory volume of tasks is reached. A noteworthy application of this method can be observed in Alpaca [42], where Self-Instruct is utilized to generate new queries and distilled output from Text-Davinci-003 [29].
We introduce an effective extension, the Topic-Guided Red-Teaming Self-Instruct, which aims to improve the diversity and coverage of the generated adversarial instructions. We manually devise 20 adversarial instruction types that a static machine learning model can't answer, or may answer with the wrong facts, such as:
Questions that require scientific knowledge  
Questions that require knowledge of future events  
Questions that require real-time information  
Questions that require legal expertise
, and prompt the base LLM to generate novel topics (e.g., Water) relevant to these types<sup>5</sup>. Subsequently, after removing duplicated topics, we prompt the base LLM to generate new instructions
4https://github.com/yizhongw/self-instruct/blob/main/data/seed_tasks.json
See the appendix for the seed prompts we used for Topic-Guided Red-Teaming Self-Instruct.
novel instructions corresponding to the specified instruction type and topic. Incorporating additional prompts that concentrate on particular adversarial instruction types and diverse topics allows the AI model to explore an expanded range of contexts and scenarios.
3.2 Principle-Driven Self-Alignment
The Principle-Driven Self-Alignment technique is designed to develop the AI alignment with a small set of helpful, ethical, and reliable principles. This stage capitalizes on the (Topic-Guided Red-Teaming) Self-Instruct [48] as an instruction generator. The primary objective is to enable the AI model to generate fitting responses that adhere to the established principles, while simultaneously minimizing human supervision.
The Principle-Driven Self-Alignment process begins with the creation of sixteen generic principles that an AI assistant should follow, such as "1 (ethical). Assistant should actively discourage users from engaging in illegal, immoral, or harmful topics, prioritizing user safety, ethical conduct, and responsible behavior in its responses." Subsequently, five in-context learning (ICL) [7] demonstrations are provided to exemplify how the AI assistant should apply these principles through an explicit process termed "internal thoughts". For instance, in the ICL demonstration, the user query can be:
User: who is the president of the US in 2025?
And we annotate the internal thoughts of the AI assistant as:
Assistant (internal thoughts): I am a helpful, ethical, and reliable assistant. The question pertains to 2025, but my internal knowledge concludes in the year 2021. I should adhere to rules 3 (helpful), 7 (candor), and 12 (dated knowledge).
Such internal thoughts will guide the assistant's final generated response, such as:
Assistant: As an AI language model, my knowledge extends only until September 2021, so I cannot predict the president of the United States in 2025. However, I can give you the following information:
1. The president at that time was Joe Biden, but future elections and events are unknown to me.
2. Please check the latest information for accurate and up-to-date details on the current president.
When a new query is generated by (Topic-Guided Red-Teaming) Self-Instruct, it is appended to the list of the exemplars, and the base LLM follows such an internal-thought-then-answer process to produce a self-aligned response. The whole process is illustrated in Figure. 3.
In this paper, the design of the principles remains exploratory and primarily serves research purposes<sup>7</sup>. We (the authors) brainstormed sixteen principles, namely 1 (ethical), 2 (informative), 3 (helpful), 4 (question assessment), 5 (reasoning), 6 (multi-aspect), 7 (candor), 8 (knowledge recitation), 9 (static), 10 (clarification), 11 (numerical sensitivity), 12 (dated knowledge), 13 (step-by-step), 14 (balanced & informative perspectives), 15 (creative), 16 (operational)<sup>8</sup>, drawing inspiration from existing principles in Constitutional AI [5] and the new Bing Chatbot [24], as well as the principles proven to enhance AI performance in recent research papers, such as step-by-step reasoning [25, 49, 19] and knowledge recitation [41].
3.3 Principle Engraving
Principle Engraving constitutes a vital element of the SELF-ALIGN methodology, focusing on honing the AI model's behavior to produce responses that adhere to predefined principles. During this stage, the base LLM is fine-tuned after pruning the principle, the in-context learning demonstrations, and the
The effectiveness of such a thinking procedure has been proven on a wide range of reasoning [49], action [51], or knowledge-intensive [41] tasks.
7Analogous to Constitutional AI [5], we believe that, in the future, such principles should be redeveloped and refined by a more extensive set of stakeholders. Given the small number of bits of information involved in these principles, a thorough examination of these bits is warranted.
The detailed principles and the ICL exemplars are given in the appendix.
self-generated thoughts, effectively engraving these principles into the LLM's parameters. Figure 3 provides a visual representation of this process.
A noteworthy advantage of principle engraving is its ability to enhance the AI model's alignment while reducing token usage, which enables longer context lengths during inference (as allocating more than 1.7k tokens to fixed principles and ICL demonstrations would be excessive). Remarkably, our empirical observations reveal that the base LLM, after fine-tuned with its self-aligned outputs, surpasses its prompted counterpart on alignment benchmarks. This improvement can likely be attributed to the generalization effect that occurs when the language model is directly optimized to generate output that is helpful, ethical, and reliable.
3.4 Verbose Cloning
In our preliminary testing of the principle-engraved model, we identified two primary challenges: 1) the model tended to generate unduly brief responses, while users typically expect more comprehensive and elaborate answers from an AI assistant, and 2) the model occasionally recited relevant Wikipedia passages without directly addressing the user's query.
To overcome these challenges, we introduce a complementary Verbose Cloning step. This stage involves utilizing an human-crafted prompt to create a verbose version of the aligned model, that is capable of generating in-depth, detailed responses. We then employ context distillation [3] to produce a new model that is not only aligned but also generates thorough and extensive responses to user queries. Context distillation works by training the base language model on synthetic queries generated by (Topic-Guided Red-Teaming) Self-Instruct, paired with corresponding responses produced by a verbally prompted principle-engraved model. The verbose prompt designed to encourage the talkative nature of the principle-engraved model is provided in the appendix.
We quantitatively evaluate Dromedary on benchmark datasets and also assess its qualitative performance on several datasets for demonstration purposes. By default, all the language model-generated text is decoded with a temperature of 0.7.
4.1 Dromedary and Baseline Models
Dromedary Dromedary is the AI assistant developed by implementing the SELF-ALIGN process on the LLaMA-65b base language model. We investigate two variants: Dromedary (final) and Dromedary (non-verbose), respectively. The former represents the model obtained by applying all four steps of the SELF-ALIGN process, while the latter is the principle-engraved model, excluding the final step of verbose cloning. Due to the space limit, the experimental details of Dromedary such as training process and decoding hyper-parameters can be found in the appendix.
Baseline Models Our comparison involves several notable baselines. LLaMA [44] provides a set of performant base language models for research usage. Text-Davinci-003, ChatGPT (or GPT-3.5), and GPT-4 [29, 26, 27], successors to their previous versions, have demonstrated significant enhancements in generating contextually relevant and high-quality content. Alpaca [42], a fine-tuned model derived from Text-Davinci-003, and Vicuna [8], a chatbot trained on user-shared conversations with ChatGPT, offer unique insights into model performance. Dolly-V2 [11], an instruction-following model, showcases commercial applications of language models. Finally, results from Anthropic-LM [4, 5], though not publicly available, provide valuable benchmarks. More comprehensive descriptions of these models are available in the appendix.
4.2 Benchmark Results
4.2.1 TruthfulQA
The TruthfulQA benchmark [22] evaluates a model's ability to identify true claims, specifically in the context of literal truth about the real world. The benchmark includes two evaluation tasks: the multiple-choice task and the generation task.
Figure 4: TruthfulQA evaluation. On the left, the Multiple Choice (MC) accuracy on TruthfulQA, where multiple choices are ranked by asking the model if each choice is True or False, and other results are taken from OpenAI [27]. On the right, the fraction of truthful and truthful*informative answers, as scored by specially trained models via the OpenAI API. The results of GPT-3 and LLaMA are taken from Touvron et al. [44].
# Param.	Truthful	Tru*Inf
GPT-3	175B	0.28	0.25
LLaMA	13B	0.47	0.41
LLaMA	65B	0.57	0.53
Alpaca	65B (reprod.)	0.47	0.47
Davinci-003	?	0.60	0.59
Vicuna	13B	0.84	0.84
( non-verbose)	65B	0.74	0.57
( final)	65B	0.72	0.61
Table 2: Multiple Choice (MC) accuracy on HHH Eval. The results of Anthropic-LM's Context Distillation (CD) and Preference Model (PM) are taken from Bai et al. [4].
Anthropic-LM	LLaMA-65B	Alpaca-65B (reprod.)	ChatGPT	Dromedary-65B
CD	PM	non-verbose	final
Harmless	-	-	0.71	0.76	0.95	0.91	0.91
Helpful	-	-	0.83	0.85	0.85	0.86	0.85
Honest	-	-	0.72	0.72	0.80	0.74	0.74
Other	-	-	0.84	0.86	0.91	0.88	0.81
Overall	0.77	0.86	0.77	0.79	0.87	0.85	0.83