SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the Multiple-Choice (MC) task, models are tested on their ability to select true answers from sets of true and false (usually 2-7) reference answers<sup>9</sup>. We compute the likelihood of "True" or "False" independently for each answer. The MC1 accuracy results are shown in Figure 4 (left). We can see that with a modified ranking approach, Dromedary significantly outperforms the powerful GPT-4 model and other baselines, achieving a new state-of-the-art MC1 accuracy of 69.
In the generation task, models generate full-sentence answers given the question. The benchmark evaluates the model's performance on both questions to measure truthful models and the intersection of truthful and informative. As shown in Table 4 (right), Dromedary achieves higher scores than GPT-3, LLaMA, Alpaca in both categories, while failing behind the ChatGPT-distilled Vicuna model.
4.2.2 BIG-bench HHH Eval
The BIG-bench HHH Eval [39, 3] was specifically designed to evaluate a model's performance in terms of helpfulness, honesty, and harmlessness (HHH). It is a Multiple-Choice (MC) task, which tests the models' ability to select superior answers from two reference answers<sup>10</sup>. We calculate the likelihood of the model preferring one answer over the other when presented with two candidate answers simultaneously. The MC accuracy results are displayed in Table 2. It can be observed that Dromedary demonstrates significantly improved performance compared to other open-source models, such as LLaMA and Alpaca, particularly in the Hamrless metric. Furthermore, it only marginally underperforms when compared to the powerful ChatGPT model.
4.2.3 Vicuna Benchmark Questions (Evaluated by GPT-4)
Chiang et al. [8] introduced an evaluation framework leveraging GPT-4 [27] to automate the assessment of chatbot performance. In this framework, GPT-4 generates challenging questions across diverse categories, and answers from five chatbots—LLaMA, Alpaca, ChatGPT, Bard, and Vicuna—are collected. We directly use these data to compare Dromedary with these chatbots.
The evaluation prompt we used for TruthfulQA-MC can be found in the appendix.
10The evaluation prompt we used for HHH Eval can be found in the appendix.
(a) Response comparison
(b) Relative response quality compared to ChatGPT, where the results of other models (except Alpaca-65) are taken from Chiang et al. [8].
Figure 5: Evaluation on Vicuna benchmark questions: assessed by GPT-4.
We followed Chiang et al. [8] and utilized GPT-4 to rate chatbot responses based on helpfulness, relevance, accuracy, and detail. Inspired by Vicuna<sup>11</sup>, we use two conversation examples as ICL to improve the response quality of Dromedary<sup>12</sup>. A Win/Tie/Lose comparison between the final version of Dromedary and various baselines is illustrated in Figure 10. The comparison reveals that Dromedary surpasses Text-Davinci-003 and Alpaca but falls short of ChatGPT and its distilled version, Vicuna. Additionally, we present a comparison of relative performance with respect to ChatGPT in Figure 5b.
4.2.4 Discussions
A New AI Alignment Paradigm Interestingly, in contrast to the prevailing alignment paradigm of first-following-then-align, i.e., SFT (supervised fine-tuning) + RLHF (reinforcement learning from human feedback) [30, 26, 20, 27], SELF-ALIGN prioritizes improving harmlessness and reliability through Principle-Driven Self-Alignment and Principle Engraving. Subsequently, it improves its helpfulness (instruction-following ability) by employing Verbose Cloning. Determining the superior paradigm (first-following-then-align or first-align-then-following) may need future research.
**Verbose Tax:** Analysis on Verbose Cloning The final Verbose Cloning step in SELF-ALIGN aims to enhance the model's ability to generate comprehensive and detailed responses. However, the benchmark results reveal a noteworthy observation: while Verbose Cloning significantly improves generation quality (as evidenced by the Vicuna Benchmark Questions and our TruthfulQA generation task), it harms the model's performance in several multiple-choice benchmarks, particularly in ranking more trustworthy responses. Drawing on the "alignment taxes" concept introduced by Bai et al. [4], we refer to this phenomenon as **verbose tax**. Understanding the underlying reasons for this occurrence and exploring methods to improve the model's helpfulness (verbose generation ability) while maintaining its harmlessness and trustworthiness warrant further investigation.
4.3 Qualitative Demonstrations
To offer a more profound insight into the strengths and weaknesses of Dromedary, we present qualitative demonstrations of its performance across diverse contexts. Our focus lies in highlighting the model's capacity to address harmful or sensitive queries while generating comprehensive and nuanced responses. Due to the space limit, we present these results in the appendix. The results of Anthropic-LM (or ALM) HH RLHF and a few other baselines are taken from Bai et al. [4, 5], while the results of other baselines on Vicuna benchmark questions are taken from Chiang et al. [8].
<sup>11</sup>https://github.com/lm-sys/FastChat/blob/main/fastchat/conversation.py
The two-shot prompt we used for open-ended conversation can be found in the appendix.
Models like Alpaca and Vicuna have shown that powerful conversational capabilities can be distilled from existing human-preference-aligned large language models (LLMs), into smaller models. In this paper, we introduce Dromedary, a model for the research community based on principle-driven self-alignment, trained from scratch and requiring very little human annotation. By harnessing the intrinsic knowledge within an LLM, we can define principles that guide how we want an LLM-based AI model to behave, resulting in an AI assistant that not only produces quality interactions but also produces responses that respect the guardrails defined by the model creator. This method represents a distinct direction from RLHF, and it focuses on developing novel alignment techniques for language models from scratch, independent of pre-existing, well-established AI systems. In other words, our approach seeks to explore the potential of aligning AI models in situations where reliance on or access to existing systems may not be feasible or desired.
For future work, we propose the following research directions:
- Conduct ablation studies on the Dromedary's 16 self-alignment principles to evaluate the impact of adding or removing specific principles.
- Apply Constitutional AI-based self-critique and reinforcement learning techniques [5] to enhance the performance of Dromedary further.
- Perform human evaluations to assess the real-world applicability and effectiveness of SELF-ALIGN.
- Investigate better utilization of existing open-source annotation data, such as the 15k original instruction-following data in [11].
- Engage with the broader research community to explore how the definition of principles interacts with different ethical, cultural, and application contexts. Principle-guided self-alignment provides a starting point for multi-stakeholder communities to engage with the alignment of AI models, but substantial ongoing work will be needed to ensure that these methods drive positive outcomes across a range of communities.
This work was supported in part by IBM research, the Microsoft Accelerate Foundation Models Research award, and the Google PhD Fellowship. We would also like to thank the computation support from AiMOS, a server cluster for the IBM Research AI Hardware Center. We thank Yizhong Wang, Frank Xu, and Zhengbao Jiang for their insightful discussions and help with the experiments.
[1] Anthropic. Claude's constitution, 2023. URL https://www.anthropic.com/index/claudes-constitution.
[2] Anthropic. Core views on ai safety: When, why, what, and how, 2023. URL https://www.anthropic.com/index/core-views-on-ai-safety.
[3] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021.
[4] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.
[5] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer,
[39] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022.
[40] Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Salmon: Self-alignment with principle-following reward models. arXiv preprint arXiv:2310.05910, 2023.
[41] Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. Recitation-augmented language models. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=-cqvvvb-NkI.
[42] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023.
[43] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, HengTze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022.
[44] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
[45] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017.
[47] Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. Poisoning language models during instruction tuning, 2023.
[48] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022.
[49] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 2022.
[50] Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. arXiv preprint arXiv:2304.01196, 2023.
[51] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
[52] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
A Limitations & Social Impacts
In this section, we discuss the limitations of the proposed SELF-ALIGN technique and the released Dromedary model, and address the potential social impacts that may arise from its release.
- Incompleteness of intrinsic knowledge: While Dromedary harnesses the intrinsic knowledge within an LLM, it is subject to the limitations of the base model's knowledge, which may be incomplete or outdated. Consequently, the model's responses may sometimes be inaccurate or fail to reflect recent developments.
- Challenges in defining principles: The process of defining principles for the self-alignment approach is non-trivial, as it may be difficult to anticipate all potential scenarios and challenges that a model might encounter during deployment. Furthermore, balancing between competing principles may result in unexpected behavior. We welcome the involvement of a broad community of stakeholders and ethics experts in helping shape these principles. Different communities and applications will demand different approaches, and we do not see any one set of principles as being the globally appropriate ones; our approach provides these different stakeholders with an avenue to engage in a structured way with this process. We do not expect any one set of principles to be the final word on alignment, but rather provide these methods as a jumping off point for broader community engagement.
- Limited generalizability: While the model demonstrates strong performance in several domains, it may not generalize well to all possible applications or contexts. There may be situations where the model's performance falls short of expectations, necessitating additional fine-tuning or adaptation.
- Inconsistent principle adherence: In our preliminary testing, we observed that Dromedary occasionally hallucinates information that violates our pre-defined principles. Further investigation is required to improve strict principle adherence in the SELF-ALIGN process. One should not assume that the alignment procedures presented here provide definitive guardrails that prevent all undesired outputs and outcomes.
A.2 Social Impacts
By investigating the alternative AI alignment strategies, our work seeks to contribute to the broader landscape of AI alignment, expanding the range of possibilities and promoting a more diverse and robust understanding of how AI systems can be developed to be not only more powerful, but also more responsible and aligned with human values. Through this research, we aspire to pave the way for the safer and more harmonious integration of AI into various aspects of our lives, fostering a collaborative, ethical, and multi-stakeholder approach to AI development.
However, the potential negative impacts of our work include:
- Potential misuse: As with any powerful AI system, there is the risk of misuses, such as generating malicious content or automated disinformation. It is crucial to establish mechanisms for detecting and mitigating such abuse, as well as promoting ethical guidelines for AI developers and users.
- Bias and fairness: The Dromedary model may inadvertently perpetuate or exacerbate existing biases present in the pre-training data of its base language model, potentially leading to unfair or discriminatory outcomes. Future work should address bias mitigation strategies to ensure fairness and inclusivity in AI applications.
B More Details about Dromedary
The Dromedary model represents an AI assistant developed by implementing the SELF-ALIGN process on the LLaMA-65b base language model [44]. This section delves into the details employed in the creation of the Dromedary model. The additional experimental details of Dromedary such as training and decoding hyper-parameters can be found in Appendix D.2.
We first followed the Alpaca's recipe [42], employing Self-Instruct, which automatically produced 267,597 open-domain prompts along with their corresponding inputs. Additionally, we utilized
The top 20 most common root verbs (a) (inner circle) and their top 4 direct noun objects (outer circle) in our Self-Instruct dataset.
(c)
Principle usage statistics in our Self-Instruct dataset
Figure 6: Statistics of our Self-Instruct and Topic-Guided Red-Teaming (TGRT) Self-Instruct datasets.
The 20 instruction types (inner circle)
(b) and their top utilized rules (outer circle) in our TGRT Self-Instruct dataset.
(d)
Principle usage statistics in our TGRT Self-Instruct dataset
Topic-Guided Red-Teaming Self-Instruct, which automatically generated 99,121 prompts specifically tailored to 20 red-teaming instruction types.
After applying the Principle-Driven Self-Alignment process and filtering out low-quality responses, we obtained 191,628 query-response pairs derived from Self-Instruct and 67,250 query-response pairs from Topic-Guided Red-Teaming Self-Instruct, resulting in a total of 258,878 query-response pairs. Figure 6 presents a detailed analysis of the principles applied and the instruction types encompassed in the Topic-Guided Red-Teaming (TGRT) approach. We observed that the instructions generated by the original Self-Instruct and TGRT Self-Instruct appear to evoke distinct principles. For instance, Self-Instruct datasets use the principles 5 (reasoning), 13 (step-by-step), and 15 (creative) extensively, whereas TGRT Self-Instruct relies more on 8 (knowledge recitation) and 14 (balanced and informative perspectives).
Next, we fine-tuned the LLaMA-65b base language model with the automatically generated 258,878 (after filtering) query-response pairs, as well as a modified version of 910 pairs of dummy data<sup>13</sup> from the Vicuna project [8]. This results in a non-verbose principle-engraved AI assistant, namely the Dromedary (the non-verbose version).
13The dummy data are used to improve the self-identification of Dromedary: https://github.com/lm-sys/FastChat/blob/main/playground/data/dummy.json.
Finally, we prompted the non-verbose principle-engraved model to generate more verbose outputs and utilized its output as the teacher model to produce 358,777 verbose responses to (Topic-Guided Red-Teaming) Self-Instruct queries. The Dromedary (the final version) model is trained on this dataset, resulting in an AI assistant designed to be helpful, ethical, and reliable, developed from scratch with a base language model (without any SFT or RLHF), and achieved with minimal human supervision (less than 300 lines of human annotations).
C Dromedary-2
After the recent release of LLaMA-2 [45] with its extended 4k context length, we have experimented with responses that adhered more closely to the general-specific-general style within ICL Self-Align examples. The results are highly promising: The Dromedary-2 model, trained from LLaMA-2 and with improved ICL exemplars, has enhanced performance even without the verbose cloning phase nor inference-time few-shot examples. For more details and comparison with other models, please check Sun et al. [40].
D.1 Dromedary and Baseline Models
LLaMA LLaMA [44] consists of a series of base language models with a parameter count ranging from 7 billion to 65 billion. These base models are solely trained to optimize the likelihood of next-word prediction in the language modeling task. For fair comparison, we employ the same prompt for LLaMA as used for Dromedary, detailed as follows.
Dromedary Dromedary is the AI assistant developed by implementing the SELF-ALIGN process on the LLaMA-65b base language model. We investigate two variants: Dromedary (final) and Dromedary (non-verbose), respectively. The former represents the model obtained by applying all four steps of the SELF-ALIGN process, while the latter is the principle-engraved model, excluding the final step of verbose cloning. The detail of the verbose prompt is presented in Appendix K.1.
Text-Davinci-003 The Text-Davinci-003 model [29] is built on top of InstructGPT [30], with improved performance in several aspects over Text-Davinci-002, such as producing higher quality writing, handling more complex instructions, and generating a longer form of content.
GPT-3.5 / GPT-4 GPT-3.5 (aka ChatGPT) [26] is a sibling model of InstructGPT, specifically designed for conversational AI. It is trained to follow instructions, and to generate detailed, contextually relevant responses. GPT-4 [27] represents a significant leap in language model capabilities, exhibiting human-level performance on a wide range of professional and academic benchmarks. Both ChatGPT and GPT-4 are fine-tuned from the corresponding base language models with SFT (Supervised Fine-Tuning) and RLHF (Reinforcement Learning with Human Feedback) [26, 27].
Alpaca Alpaca [42] is a fine-tuned instruction-following language model derived from the LLaMA base model. It utilizes 52K instruction-following demonstrations generated through a cost-effective adaptation of the Self-Instruct method [48], in conjunction with Text-Davinci-003. Designed to address the research accessibility gap in academia, Alpaca exhibits qualitative similarities to Text-Davinci-003 in single-turn instruction setting. For fair comparison with Dromedary-65b, we employ a training methodology comparable to Dromedary, that is, fine-tuning the LoRA [17] weights in the multi-head attention modules, to obtain our own reproduced Alpaca-65b model.
Vicuna Vicuna [8] is an open-source chatbot developed by fine-tuning a LLaMA base model on a dataset of approximately 70,000 user-shared conversations from ShareGPT.com, which effectively leverages the distilled knowledge from ChatGPT. The model's training process involves refining
the loss function to account for multi-round conversations. A preliminary evaluation [8], utilizing GPT-4 as a judge, indicates that Vi cuna attains over  $90\%$  quality in comparison to ChatGPT, while surpassing models like LLaMA and Alpaca in more than  $90\%$  of cases.
Dolly-V2 Dolly-V2 [11] is an open-source, instruction-following LLM fine-tuned for research and commercial use. Based on the Pythia-12b model [6], Dolly-V2 is fine-tuned on a new high-quality dataset, databricks-dolly-15k, which consists of 15k human-generated prompt/response pairs crowdsourced among Databricks employees.
Anthropic-LM Anthropropic-LM (or ALM) is not a publicly released model, so we directly report results from Bai et al. [4, 5]. On BIG-bench HHH Eval, we report the results for both Context Distillation (CD) and Preference Model (PM) from Bai et al. [4].
(Topic-Guided Red-Teaming) Self-Instruct For both Self-Instruct and Topic-Guided Red-Teaming Self-Instruct, we set the maximal number of new tokens in the generation to 384. The new tokens are generated by nuclear sampling [16] with a top-p threshold  $p = 0.98$  and temperature  $t = 1.0$ .
Principle-Driven Self-Alignment The aggregated principles and in-context learning demonstrations in Appendix G and H take around 1800 tokens by LLaMA. So we set the maximal number of new tokens in the generation to 256. The new tokens are generated by nuclear sampling [16] with a top-p threshold  $p = 0.9$  and temperature  $t = 0.5$ .
Principle Engraving We fine-tune the base LLaMA-65b model [44] on our aggregated Self-Instruct and Topic-Guided Red-Teaming Self-Instruct dataset for 1 epoch. We only finetune the LoRa weights [17] in the multi-head attention modules<sup>14</sup>. We use a batch size of 768, a maximal sequence length of 512, and a max learning rate of  $4e - 4$ . A 1-epoch (approximately 335 steps) training schedule is used, where the learning rate increases (i.e., warm-up) in the first 100 steps with a log curve, and decays linearly to zero in the rest of the training steps.
**Vocabulary Cloning** The teacher model (i.e., the principle-engraved model) uses the verbose-encouraging prompt to relabel all the queries generated by (Topic-Guided Red-Teaming) Self-Instruct. We set the maximal number of new tokens in the generation to 512. The new tokens are generated by nuclear sampling [16] with a top-p threshold  $p = 0.7$  and temperature  $t = 0.3$ , as well as a repetition penalty.