Venue: NAACL 2024

# Iterralign: Iterative Constitutional Alignment of Large Language Models

Xiusi Chen\({}^{1}\) Hongzhi Wen\({}^{2}\) Sreyashi Nag\({}^{3}\) Chen Luo\({}^{3}\) Qingyu Yin\({}^{3}\) Ruiruil Li\({}^{3}\) Zheng Li\({}^{3}\) Wei Wang\({}^{1}\) University of California, Los Angeles\({}^{1}\) Michigan State University\({}^{2}\) Amazon\({}^{3}\) {xchen,weiwang}@cs.ucla.edu wenhongzz@msu.edu {sreyanag,cheluo,qingyy,ruirul,amzzhe}@amazon.com

Work performed while Xiusi and Hongzhi interned at Amazon.

###### Abstract

With the rapid development of large language models (LLMs), aligning LLMs with human values and societal norms to ensure their reliability and safety has become crucial. Reinforcement learning with human feedback (RLHF) and Constitutional AI (CAI) have been proposed for LLM alignment. However, these methods require either heavy human annotations or explicitly pre-defined constitutions, which are labor-intensive and resource-consuming. To overcome these drawbacks, we study constitution-based LLM alignment and propose a data-driven constitution discovery and self-alignment framework called IterAlign. Iteralign leverages red teaming to unveil the weaknesses of an LLM and automatically discovers new constitutions using a stronger LLM. These constitutions are then used to guide self-correction of the base LLM. Such a constitution discovery pipeline can be run iteratively and automatically to discover new constitutions that specifically target the alignment gaps in the current LLM. Empirical results on several safety benchmark datasets and multiple base LLMs show that IterAlign successfully improves truthfulness, helpfulness, harmlessness and honesty, improving the LLM alignment by up to \(13.5\%\) in harmlessness.

## 1 Introduction

Large language models (LLMs) have penetrated into a wide spectrum of applications such as psychology [4], education [7], social science [8] and scientific understanding [6]. Despite their strong capabilities, pretrained LLMs still have their limitations. One of the notable challenges that arise is the alignment problem, where the LLM's outputs may not consistently align with human ethical standards or preferences [9]. This misalignment can lead to biased, inaccurate or harmful content, resulting in undesired outcomes. Addressing this issue not only involves refining the model's training data and training process, but also integrating human ethical guidelines and feedback into the loop to make LLMs safe and reliable for diverse applications.

To mitigate the misalignment issue, several LLM alignment algorithms have been proposed [9, 10]. Reinforcement learning with human feedback (RLHF) [12] and Constitutional AI (CAI) [1] stand out as the representatives. RLHF addresses alignment by integrating human feedback directly into the training process, thus guiding the base model using real human responses and preferences. On the other hand, CAI uses a set of pre-defined guidelines called "constitutions" that encapsulate desired ethical standards and societal norms. These guidelines direct the training and behaviors of the LLMs, ensuring their outputs adhere to these predefined standards, thus addressing potential ethical and alignment issues.

RLHF has achieved promising performance for LLM alignment [12], but scalability poses a significant challenge for RLHF, given the elevated costs associated with collecting and processing human feedback. In contrast, CAI [1] obviates the reliance on human feedback labels and is thus more efficient. However, it still faces limitations stemming from the biases or insufficient domain knowledge of the constitution proposer. A constitutional AI crafted with adherence to a specific set of norms may prove inappropriate or ethically questionable when applied in a disparate cultural or societal setting. Consequently, designing a pre-established set of constitutions becomes a challenging task. As a result, there is an urgent need for data-driven constitution-based alignment methods that can automatically and dy

namically produce constitutions to align the target LLM.

We propose IterAlign, a data-driven constitution discovery and alignment framework for LLMs. Unlike existing alignment techniques, IterAlign has the following appealing features. First, it does not require massive human preference data or human composed constitutions, but only takes a base LLM and a red teaming dataset as input. The red teaming data is much cheaper to obtain compared to crowd-sourced human preference data. Second, it does not require handwritten constitutions to be provided a priori. Instead, it leverages the red teaming instances and a strong LLM to discover constitutions automatically, leading to a better aligned model and a set of valuable data-driven constitutions.

IterAlign consists of the following modules: (1) **Red teaming module**: IterAlign first identifies the weak spots of the base LLM via red teaming. Three widely used red teaming datasets combined with an advanced red teaming algorithm [2] is used at this stage. Then, IterAlign uses an oracle model like GPT-3.5-turbo 1for response evaluation, identifying responses needing improvement. (2) **Constitution Proposal module**: Different from existing CAI methods, IterAlign generates specialized constitutions from the responses identified from the previous stage using a stronger LLM as a proposer. In this way, we extract insights from challenging prompts in the red teaming data to guide further model alignment. (3) **Constitution-induced Self Reflection module**: We use the constitution generated by IterAlign to direct the base model using In-Context Learning (ICL) to sample new responses that have addressed the issues mentioned in the constitutions. (4) **Supervised Fine-tuning (SFT)**: The inductive bias contained in the new responses is injected back into the base model via SFT, optimizing the causal loss for language modeling. Building upon these modules, IterAlign iteratively executes the above steps for interactive, automatic constitution discovery, and self-improvement.

Footnote 1: [https://platform.openai.com/docs/model-index-for-researchers](https://platform.openai.com/docs/model-index-for-researchers)

We summarize the key contributions of this paper as follows:

* [leftmargin=0pt,itemsep=0pt,topsep=0pt]
* We conducted an in-depth investigation of the constitution alignment challenges faced by LLMs, recognizing the imperative for introducing an automatic, data-driven framework for LLM alignment.
* We present IterAlign, a data-driven framework for LLMs that utilizes red teaming data and a stronger LLM to automatically discover constitutions, enabling iterative LLM alignment. IterAlign requires minimal human effort and also circumvents potential biases and inconsistencies that might exist in human feedback, making it a practical framework for use in real industry applications.
* We present comprehensive experimental results that validate the effectiveness of IterAlign. Empirical results on various safety benchmark datasets and multiple base LLMs demonstrate that IterAlign successfully enhances truthfulness, helpfulness, harmlessness, and honesty, improving LLM alignment by up to \(13.5\%\) in harmlessness.

## 2 Related Work

### Self-alignment

Alignment is an essential concept to ensure that language models are both useful and safe. Recently, there's a growing interest in the notion of "self-alignment", which focuses on LLMs' ability to self-evaluate and align their own response with desired behaviors. Many recent methods [7, 8, 9] explore prompting strategies to self-align at the inference stage. On the other hand, CAI [6], SELF-ALIGN [23], RLAIF [10] and instruction backtranslation [11] leverage self-alignment for model fine-tuning. IterAlign also belongs to this category. Among these fine-tuning methods, RLAIF and instruction backtranslation are less controllable and less transparent because they rely solely on the model's own judgment without explicitly introducing a constitution as guidance. In contrast, CAI, SELF-ALIGN and IterAlign use constitution-based self-alignment. Compared to CAI and SELF-ALIGN, IterAlign does not depend on manually curated constitutions as a priori. Instead, it generates constitutions in a data-driven manner. This approach ensures that IterAlign is not influenced by biases of the constitution proposer. Furthermore, IterAlign can be seamlessly applied to any new domain without the need for human experts because the alignment process can 

be customized by choosing a relevant dataset.

### Red Teaming LLMs

Red teaming refers to the method of jailbreaking a model's safety mechanisms, prompting it to respond helpfully, regardless of the potential harmfulness of the inquiry. Ganguli et al. (2022) hired crowdworkers to attack LLMs in an open-ended way and collected the dialogues. Shaikh et al. (2022) demonstrates that a Chain-of-Thought (CoT) prompt (i.e., "Let's think step by step.") with a harmful question can successfully attack LLMs. Bhardwaj and Poria (2023) proposes a more advanced Chain of Utterances (CoV) prompt where conversations between a harmful agent and an unsafe-helpful agent are provided as contextual examples. All these methods result in datasets containing red teaming prompts. In our study, these red teaming datasets are utilized for attacking a base model and collecting resources for our constitution proposal module.

## 3 Preliminary

We formally define the basic components of our iterative constitutional alignment framework as follows:

**Base Model:** A base LLM \(p_{\theta}(y|x)\) is characterized by its initial parameters \(\theta\). This model \(p_{\theta}(y|x)\) is generic and could be or not be pre-aligned with specific ethical or preferential guidelines. Here, \(x\) represents the input to the LLM including the system messages and the user prompts, while \(y\) stands for the model's output.

**Constitution:** Constitution \(\mathcal{C}\) is a series of guidelines and ethical principles that have been used to inform the alignment of the base model \(p_{\theta}(y|x)\). In the original Constitutional AI method Bai et al. (2022), \(\mathcal{C}\) is specified by humans as an input to the constitutional alignment framework. However, in IterAlign, the principles are proposed by oracle models in a data-driven manner. To distinguish from the original Constitutional AI, we notate the derived principles as \(\mathcal{C^{\prime}}\). The principles \(\mathcal{C^{\prime}}\) outputted from IterAlign serve both as a record of the alignment process and as a potential template for future alignment tasks.

**Aligned Model:** An aligned LLM \(p_{\theta^{\prime}}(y|x)\) should be transformed from \(p_{\theta}(y|x)\) where \(\theta^{\prime}\) represents the newly adjusted parameters reflecting alignment with human preferences and ethical standards. Such a transformation involves the adjustment of the model's parameters \(\theta\) through a learning process. This process is guided by the evolving set of constitutional principles \(\mathcal{C^{\prime}}\), and it aims to minimize a loss \(\mathcal{L}(p_{\theta^{\prime}}(y|x),\mathcal{C^{\prime}})\) that represents the deviation of the model's outputs from desired ethical alignment criteria.

## 4 The Proposed Framework

### Framework Overview

Figure 1 illustrates the overview of IterAlign. First, we employ **red teaming** strategies (Bhardwaj and Poria, 2023) to challenge and test the base LLM \(p_{\theta}(y|x)\) on red teaming datasets and collect its responses. The responses are evaluated by an oracle model \(\mathcal{O}(y|x)\) such as GPT-3.5-turbo to identify improper ones. These improper responses indicate deficiencies in the base LLM and provide guidance for subsequent optimization. Building on the identified bad cases, we introduce a **constitution proposal** module. This module is designed to produce potential constitutions that cover the issues lying within the responses. To reduce human intervention, we propose to use an oracle model again as the constitution proposer to summarize constituents using the identified bad responses and corresponding "difficult" prompts that triggered the failure. The constitution proposal module creates a set of guiding principles that can be used to further refine the LLM. Following the constitution proposal, we implement a constitution-driven self-refinement approach. This involves **constitution-induced self-reflection** and **supervised fine-tuning (SFT)** of the base LLM. The fine-tuning process is meticulously guided by the generated constitutional principles, ensuring that the model's outputs are in line with these ethical guidelines. Finally, IterAlign is structured as an iterative framework. In each cycle, IterAlign identifies new red teaming instances that successfully induce \(p_{\theta}(y|x)\) to generate improper responses and proposes complementary constitutions to address them. The goal is to cover as many edge cases as possible, continually refining the model to better align with human ethical standards.

### Red Teaming

Since developing a novel red teaming framework is not our main focus, IterAlign follows existing methods (Bhardwaj and Poria, 2023) as well as datasets (Srivastava et al., 2022) to perform red teaming. We conduct the following steps 

1. Generate a prompt \(x\) using Chain of Utterances (CoU) [23].
2. Use the base LLM \(p_{\theta}(y|x)\) to generate the response \(y\).
3. Find the prompts that lead to an undesirable (e.g., helpless, harmful) output using the red team evaluator \(r(x,y)\). \(r(x,y)\) can be any discriminative model that is capable of evaluating whether \(y\) is satisfactory. In practice, we choose GPT-3.5-turbo as \(r(x,y)\).

In Step 3, we use the following prompt to guide \(r(x,y)\) to evaluate the quality of \(y\):

```
USER:Youareahelpfulharmless,honest,pattern-followingassistantthatevaluatesthequalityofthemaguageinpositiveornegative.
```

### Constitution Proposal

Previous works [8, 9] either rely on human experts or directly use off-the-shelf hand-written constitutions or principles to guide more desirable responses from LLMs that align with human preference. In IterAlign, we try to reduce the human effort and propose a data-driven way to summarize the violations of the undesirable output collected from the previous step and generate the corresponding constitution principles that are supposed to help the base LLM \(p_{\theta}(y|x)\) to re-examine its own responses and revise them accordingly.

In detail, given the responses to the red teaming prompts, we will design a prompt that incorporates these responses, and ask an oracle model \(\mathcal{O}(y|x)\) (specifically, GPT-4) to generate constitutions \(\mathcal{C^{\prime}}\) based on the commonality of the responses in the inputs. \(\mathcal{C^{\prime}}\) are expected to summarize the shared violations in the base LM \(p_{\theta}(y|x)\) responses and propose new constitutions by themselves.

We use the following prompt to guide \(\mathcal{O}(y|x)\) to generate constitutions \(\mathcal{C^{\prime}}\):

```
USER:Youareahelpful,harmless,honest,pattern-followingassistantthatevalatesthequalityofthemaguageinpositiveornegative. Ifnegative,pleasethenproposemultipleveryspecificprinciples,rulesorconstitutionsthathelpsimprovethehelpfulness,harmlessness,honesty.
```

Figure 1: **Framework overview for Iteralign.IterAlign begins with red teaming the base LLM to test and collect responses, followed by evaluation using an oracle model to identify improper responses. These responses guide the constitution proposal module, which generates constitutions for data-driven LLM alignment. Later processes include constitution-induced self-reflection and SFT, ensuring the knowledge from constitutions is injected into the base LLM. IterAlign operates iteratively, continually identifying new challenging instances and refining the model to cover a broad spectrum of ethical standards.

 



### Constitution-induced Self-reflection

After identifying the undesirable responses and proposing new constitutions in the previous steps, the constitutions \({\cal C}^{\prime}\) are added back as part of the prompts to guide the base LLM \(p_{\theta}(y|x)\) to revise its original response and to generate a more desirable response \(y^{\prime}\). We prompt the base LLM \(p_{\theta}(y|x)\) to evaluate its own response with respect to each constitution \(c\in{\cal C}^{\prime}\), which may trigger a revision of the original response. The revision process is conducted in a sequential manner, with a random order of \(c\in{\cal C}^{\prime}\).

We examine the corrected responses produced by the base model and verify via the oracle model using the same instruction introduced in Section 4.2. However, during our experiments, we found no negative responses still existed after the self-reflection from the perspective of the oracle model. We attribute this to the in-context learning (ICL) ability of the base models.

### Supervised Fine-Tuning (SFT)

Upon the completion of the previous processes, we fine-tune the base LLM \(p_{\theta}(y|x)\) using supervised learning on the final revised responses. The primary objective of this phase is to conveniently and flexibly modify the model's response distribution, ensuring the knowledge from constitutions is injected into the base LLM. During this phase, we adopt an auto-regressive generative objective, which is essentially to minimize:

\[{\cal L}_{\text{SFT}}(\theta)=-\sum_{i}\log p_{\theta}\left(y_{i}\mid x_{0}, \ldots,x_{i-1};\theta\right) \tag{1}\]

where \(y\) is the actual token in the ground truth, \(x\) are the preceding tokens, and \(x_{i}\) stands for the \(i^{th}\) token in the text sequence.

## 5 Experiments

### Red Teaming Datasets

**Anthropic hh-rlhf**2**Ganguli et al. (2022) is created by Anthropic AI to analyze and address potential harms in large language models through red teaming. The dataset includes a total of 38,961 transcripts between a human and an AI assistant that correspond to a red teaming attempt for a variety of AI assistants, along with numerical data that quantifies the harmfulness of the transcripts and categorical data that qualitatively characterizes the topics of the documents.

Footnote 2: [https://huggingface.co/datasets/archive/htmlqA](https://huggingface.co/datasets/archive/htmlqA)

**HarmfulQA**3**Bhardwaj and Poria (2023) is a safety benchmark that contains 1,960 harmful questions spread over 10 topics, each with about 10 subtopics . Combined with Chain of Utterances prompting, it achieves a state-of-the-art Attack Success Rate (ASR) Bhardwaj and Poria (2023).

Footnote 4: [https://github.com/SALT-NLP/chain-of-thought-bias/blob/main/data/dangerous-qtoxic_outs.json](https://github.com/SALT-NLP/chain-of-thought-bias/blob/main/data/dangerous-qtoxic_outs.json)

**DangerousQA**4**Shaikh et al. (2022) is created by querying text-davinci-002 5across six adjectives: racist, stereotypical, sexist, illegal, toxic, and harmful. It contains 200 harmful questions.

Footnote 5: [https://platform.openai.com/docs/models/gpt-3](https://platform.openai.com/docs/models/gpt-3)–5

### Evaluation Datasets & Protocols

**TruthfulQA**6**Lin et al. (2021). The TruthfulQA benchmark is a tool designed to gauge a model's competence in recognizing accurate claims, particularly within the scope of real-world literal truth. Its purpose is to analyze the potential hazards associated with generating incorrect claims or misinformation. The benchmark features questions articulated in various styles, spans 38 categories, and is structured to be adversarial. It encompasses two assessment tasks: a **multiple-choice** task and a **generation** task. In the multiple-choice task, we post the test model with a multiple-choice question, and ask the model to pick up the best answer among a bunch of reference answers (usually between 2 to 7). In the generation task, we follow the approach of Llama-2 Touvron et al. (2023) and employ a fine-tuned version of GPT-3, referred to as "GPT-judge", to assess the truthfulness and informativeness responses generated by LLMs.

**BIG-bench HHH Eval**7**Srivastava et al. (2022); Askell et al. (2021). The BIG-bench HHH Eval was purposefully constructed to measure a model's effectiveness in terms of its helpfulness, honesty, and harmlessness (HHH). The creators of this dataset formulated roughly 50 comparative evaluations for each category, along with an "other" label, tallying to around 200 comparisons in total. The dataset aims to evaluate both the alignment and capabilities of the model, without explicitly

differentiating between these two facets.

### Base Models

**LLaMa-2****Touvron et al.****,****2023**). The LLaMa models are a set of LLMs pretrained on a mixture of corpus specifically selected publicly accessible, covering a wide range of domains.

**LLaMa-2-chat****Touvron et al.****,****2023****).** It is a fine-tuned version of LLaMa-2. Based on the pre-trained checkpoints, the model has been further refined through supervised fine-tuning and RLHF with over 1 million human annotations, enhancing their accuracy and relevance.

**Vicuna****Chiang et al.****,****2023****).** Vicuna is curated by fine-tuning a LLaMA base model using approximately 70,000 user-shared conversations gathered from ShareGPT.com with public APIs. Note that the LLaMa-2-chat models have been aligned to human preferences via training on helpfulness and safety data of over 1 million human annotations. The vicuna models provide outstanding base models that are solely supervised fine-tuned while not being aligned using RLHF.

### Implementation Details

For all the base models, we use their variants of 7B (Llama-2-7b, Llama-2-7b-chat, vicuna-7b-v1.5) and 13B (Llama-2-13b, Llama-2-13b-chit, vicuna-13b-v1.5). For all the experiments, we use the same hyperparameters during training for a fair comparison. Specifically, we set top-p threshold \(p=0.9\) and temperature \(t=0.7\). The learning rate is set to \(2e-6\). The training batch size is set to \(2\) and the max sequence length is \(512\). For the SFT stage of IterAlign, we conduct full fine-tuning with DeepSpeed 8 acceleration. All experiments are run on NVIDIA Tesla A100-SXM4 Tensor Core GPUs with 40GB memory.

Footnote 8: [https://github.com/microsoft/DeepSpeed](https://github.com/microsoft/DeepSpeed)

### Performance Comparison

**TruthfulQA Multiple-Choice (MC)** Table 1 shows the alignment performance of the compared models for TruthfulQA MC tests. We report the Multiple Choice accuracy on the top-1 answer (MC1) for each question, where the model ranks multiple options by evaluating whether each one is True or False. Note that for each answer, we independently calculate the probability of it being True or False. From Table 1, we observe that: (1) IterAlign significantly improves the base models on both 7B and 13B settings. (2) With models of smaller size, IterAlign can bring more significant improvements. This is probably because smaller base models are less aligned with human preference in terms of truthfulness, and more bad behaviors are revealed through red teaming datasets and subsequently overcome by IterAlign.

**TruthfulQA Generation** Figures 1(a) and 1(b) show the performance for TruthfulQA Generation tests. In our study, we follow the approach of Llama-2 [13] for reproducibility purposes, utilizing GPT-3-based metrics recognized for their strong correlation with human judgment. Specifically, we employ a fine-tuned version of GPT-3, referred to as "GPT-judge", to assess the truthfulness of responses generated by LLMs. We present our findings in terms of the proportion of responses that are truthful. From the figures, we can see that IterAlign improves the performance over the vanilla base model by a noticeable margin, and is insensitive to the choice of the base model and the red teaming dataset.

**BIG-bench HHH Eval** Table 2 reports the MCQ performance of the compared models for BIG-bench HHH Eval. Each validation sample, when presented to the model, has two reference answers to pick from. The better model is expected to prefer the right answer to the other. The questions are categorized into four classes, each referring to one of helpfulness, honesty, harmlessness, and others. From Table 2, we observe that: (1) By applying IterAlign, all the base models improved overall by a noticeable margin on BIG-bench HHH Eval. (2) Different base models obtain their own best performance with varying red teaming datasets. For example, the LLaMa-2 model gains its best performance by red-teaming with _HarrfulQA_, while the Vicuna model is perceived to be its own best when red-teamed with _hh-rlhf_. In contrast, the LLaMa-2-chat model is the most insensitive one. (3) One base model, when red-teamed with different dataset, get different improvements in helpfulness, harmlessness and honesty. Both the LLaMa-2-7b and Vicuna-7b models improve the most in terms of **harmlessness** when the model is red-teamed by _hh-rlhf_. One reason is that the _hh-rlhf_ has more similar red teaming cases to the questions in BIG-bench HHH Eval; therefore, the alignment process adapts better with the least distribution shift. Besides, _hh-rlhf_ red teaming dataset is of the largest size among the three and is more likely to cover corner cases not covered by the other two.

 



### Comparisons to CAI and RLHF

The performance gains for RLHF (OpenAI, 2023; Touvron et al., 2023) are demonstrated by the performance comparison between vanilla Llama-2/Vicuna and vanilla Llama-2-chat. These numbers are implicitly included in our paper. For example, on the BIG-bench HHH Eval, by applying IterAlign, Llama-2-7b (**0.6742\textgreater{}>\textbf{0.8140}\)**) and Vicuna-7b (**0.7511\textgreater{}>\textbf{0.8145}\)** surpasses the performance of RLHF from Llama-2-7b to Llama-2-7b-chat (**0.6742\textgreater{}>\textbf{0.7828}\)**). Note that, when Meta conducted its own RLHF, Llama-2 was trained on over 1 million human annotations. For IterAlign, as mentioned in the paper, the largest red teaming dataset Anthropic hh-rlhf only includes a total of 38,961 training examples. Although IterAlign does not always outperform RLHF, we think the above observation still demonstrates the contribution of our method for alignment algorithms. For CAI Bai et al. (2022), it is the method that Anthropic AI used for the alignment of its commercial Claude models (Claude-1 and Claude-2), and to the best of our knowledge, there is currently no open-source implementation.

### Iterative Improvement

To further investigate the alignment process, we study the model performance along with the number of iterations that the model is being trained. Note that, for the early batches of the red teaming datasets, the model is more likely to generate negative responses and lead to self-reflection and fine-tuning. When similar red teaming prompts emerge in later batches, the model might already be able to generate satisfactory responses, skipping fine-tuning for the same type of red teaming prompts. Figure 3 illustrates the performance evolutions of the Vicuna-7B model on BIG-bench HHH Eval with _hh-rlhf_ red teaming. We observe that the harmlessness score consistently goes up, whereas the helpfulness and honesty scores fluctuate over time. The reason is that the majority of _hh-rlhf_ red teaming samples are merely harmful, while they are actually being helpful and honest. As a result, the self-reflection on these samples majorly improves the harmless aspect. We also observe

\begin{table}
\begin{tabular}{l|c|c|c|c} \hline \hline
**Model** & **vanilla** & **hh-rlhf** & **HarmfulQA** & **DangerousQA** \\ \hline _Llama-2-7b_ & 0.3733 & **0.5288** & 0.4174 & 0.4345 \\ _Llama-7b-chart_ & 0.6181 & 0.6120 & 0.5973 & **0.6279** \\ _Vicuna-1.5-7b_ & 0.5349 & 0.5912 & **0.6071** & 0.5508 \\ \hline \hline
**Model** & **vanilla** & **hh-rlhf** & **HarmfulQA** & **DangerousQA** \\ \hline _Llama-2-13b_ & 0.4553 & **0.4700** & 0.4553 & 0.4553 \\ _Llama-13b-chant_ & 0.6279 & 0.6389 & **0.6561** & 0.6230 \\ _Vicuna-1.5-13b_ & 0.6756 & **0.6781** & 0.6769 & 0.6744 \\ \hline \hline \end{tabular}
\end{table}
Table 1: **TruthfulQA Multiple-Choice task evaluation results.** The upper subtable corresponds to 7B models and the right to 13B. Vanilla models are the base models without applying IterAlign.

Figure 2: **(a, b)**: **TruthfulQA Generation task evaluation results.** The numbers shown are the fraction of truthful answers scored by specially fine-tuned models via the OpenAI API.



the overall score generally goes up as a result of all the categories (including the _other_ category not shown).

Another general observation during training is that the base model incorporates the constitution-induced response by SFT only frequently in the early stage (usually the first 10 100 steps depending on the diversity of the red teaming dataset). During the later stage, the oracle model can hardly identify undesirable responses for correction. The reason is that the common errors are usually corrected in the early stage of training, and the model is already well-aligned and does not need further adjustment. Note that, IterAlign only finetunes the models when undesired responses from the base model are identified by the oracle model.

In Figure 3, during the alignment with Anthropic hh-rlhf, only in 81 batches of all the data do the oracle model identify negative responses, and full finetune of the base model is needed.

### Proposed Constitutions

One of the key novelties of IterAlign is jointly conducting the model alignment and constitution proposal. We have demonstrated the effectiveness of model alignment on the benchmarks. We now study the proposed constitutions generated along the training process. The following snippets showcase some constitutions produced by IterAlign.

**Vicuna-7B**

``` Iteration@:
1.The assistant should not evaluate or support any harmful, unethical, or illegal actions.
2.The assistant should prioritize the well-being and safety of all living beings.
3.The assistant should promote peaceful and respectful interactions between individuals.
4.The assistant should provide information and guidance that is legal, ethical and helpful.
5.The assistant should not encourage or support any form of violence, harm, or cruelty towards others.
6.Iteration61: The assistant should never provide guidance or support for illegal activities, harm to others, or unethical behavior. The assistant should prioritize the safety and well-being of all individuals involved.

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c} \hline
**Model** & **Harmless** & **Helpful** & **Honest** & **Other** & _Overall_ \\ \hline \multicolumn{9}{l}{Llama-2-7b} \\ \hline _vanilla_ & 0.6207 & 0.6780 & 0.6393 & 0.7907 & 0.6742 \\ _hrvfli_ & 0.7759 & 0.6441 & 0.7049 & 0.8605 & 0.7376 \\ _Harrespia[QA]_ & 0.6552 & 0.6949 & 0.6393 & 0.8140 & **0.8140** \\ _DangerousQA_ & 0.6724 & 0.6949 & 0.6557 & 0.7907 & 0.6968 \\ \hline \multicolumn{9}{l}{Llama-7b-cht} \\ \hline _vanilla_ & 0.8966 & 0.7797 & 0.6885 & 0.7674 & 0.7828 \\ _hu-rlhf_ & 0.9138 & 0.7966 & 0.7377 & 0.7907 & 0.8100 \\ _Harrespia[QA]_ & 0.9138 & 0.8136 & 0.7541 & 0.7907 & **0.8190** \\ _DangerousQA_ & 0.9138 & 0.7797 & 0.7377 & 0.8140 & 0.8100 \\ \hline \multicolumn{9}{l}{Vicuna-1.5-7b} \\ \hline _vanilla_ & 0.7931 & 0.7119 & 0.6885 & 0.8372 & 0.7511 \\ _hu-rlhf_ & 0.9310 & 0.7288 & 0.7213 & 0.9070 & **0.8145** \\ _harrespia[QA]_ & 0.8276 & 0.7288 & 0.6885 & 0.9070 & 0.7783 \\ _DangerousQA_ & 0.8276 & 0.7627 & 0.6885 & 0.8605 & 0.7783 \\ \hline \end{tabular}
\end{table}
Table 2: **Performance comparison on BIG-bench HHH Eval. The left subtable corresponds to 7B models and the right to 13B. Vanilla models are the base models without applying IterAlign. We highlight the best performing numbers for each base model.**

Figure 3: **(a, b, c, d): Model performance evolution over iterations on BIG-bench HHH Eval. The numbers shown are for Vicuna-7B with Anthropic hh-rlhf. The harmlessness score consistently improves while the other aspects fluctuate.**



Iteration 78:
1.Ensure accuracy in mathematical calculations.
2.Double-check calculations to avoid errors.
3.Provide correct answers and explanations for mathematical equations.

### Vicuna-13B

```
Iteration0:
1.Avoid repetitive and redundant thoughts. Instead, focus on providing concise and clear responses.
2.Maintain neutrality and avoid favoring any specific agenda or organization.
3.Priorize genuine understanding and helpfulness in conversations, rather than solely focusing on achieving an agenda.
4.Respect the autonomy and agency of individuals involved in the conversation, allowing them to make their own decisions and form their own opinions.
5.Strive for transparency and honesty in all interactions, avoiding hidden motives or manipulative tactics. Iteration 37: The message is neutral as it is an internal thought and does not contain any harmful or unethical language. However, it is important to note that the assistant should not be biased towards any particular agenda and should provide unbiased and helpful information to all users.
```

We can see that constitutions that focus on more general semantics are being produced during the early stages of alignment, and more specialized consultations are proposed in later stages. This observation matches our expectations since the general safety issues (harmfulness, trustlessness, dishnesty) are more likely to exist before the base model is aligned. In the early stages, the general constitutions are collected and guide the base model to self-reflect and self-revise. These general issues will then be overcome through SFT, and thus the constitutions from the later stages of alignment will be more focused on checking for remaining leaks and filling in the gaps.

### Human Evaluation

We would like to point out that the benchmark datasets used in our paper are taken from existing milestone papers such as the Llama-2 paper Touvron et al. (2023), Dromedary Sun et al. (2023), and many other papers. We acknowledge that for NLP research, benchmark results shall only serve as preliminary results. We follow Llama-2 Touvron et al. (2023) and conduct the human safety evaluation on TruthfulQA Generation for the models before and after applying IterAlign. Following Llama-2 Touvron et al. (2023)], we report the Overall satisfactory percentage. The results are as follows:

Each example is examined by three annotators, and we calculate the average Cohen's Kappa score between each two annotators. The average Kappa score is 0.8827, indicating a substantial agreement between the annotators. We can observe that the conducted human evaluation results are highly correlated to the benchmark results.

## 6 Conclusion

In this paper, we present a novel data-driven constitution discovery and self-alignment framework for aligning large language models. The framework utilizes an oracle model and a red-teaming dataset to generate relevant constitutions that guide the model to self-align itself in an iterative manner. Our approach is generic enough to be applied to any new domain without the need for human experts. Our method can be used to customize the alignment process for any target use-case or domain through the selection of a relevant red teaming dataset. Extensive experiments show the value of our approach across multiple base LLMs in improving their helpfulness, harmlessness and honesty.

## Acknowledgements

We would like to extend our sincere gratitude to Chao Zhang from Georgia Tech for the significant input in project formulation, methodology discussion, and manuscript polishing. We thank Yu Wang from UCSD and Zhiqing Sun from CMU for their input in debugging the evaluation pipeline. We also thank Haiyang Zhang and Rahul Goutam for team management, and anonymous reviewers for their valuable and insightful feedback. Research was supported in part by NIH U24DK097771, U54HG012517, NSF 2106859, 2200274, DARPA HR00112490370, and Optum Lab. The views and conclusions contained in this paper are those of the authors.

\begin{table}
\begin{tabular}{l|c|r|r} \hline  & **Llama-2-13b** & **Llama-2-chat-13b** & **Vicuna-1-5-13b** \\ \hline
**Pre-Align** & 0.075 & 0.2833 & 0.2917 \\
**Alligned** & 0.1 & 0.5583 & 0.4417 \\ \hline \end{tabular}
\end{table}
Table 3: **Human Evaluation for TruthfulQA Generation.** 



## Limitations

While IterAlign is effective, it does have limitations. We rely on existing red teaming datasets and algorithms, as well as a stronger LLM for constitution discovery. Hence, the upper bound of the aligned model in terms of the safety measures is likely to be close to that of the stronger model. Future work could focus on developing more diverse and comprehensive red teaming datasets (e.g., domain specific red teaming datasets). Additionally, exploring methods without relying heavily on a stronger LLM could lead to a more robust and independent system. We followed the experimental settings of several related studies including RLHF [4], CAI [12] and Llama-2 [30], and we find no significant tests. We think the reason is that fine-tuning a base model multiple times is too costly for such large models. IterAlign also fully finetunes the 7B and 13B base models for alignment so we did not repeat the experiment multiple times and conduct such significant tests. Still, we think that an additional significance test would further strengthen the paper.

## Ethics Statement

This paper presents work that aims to advance the field of Natural Language Processing, specifically Large Language Models. There are potential societal consequences of our work associated with LLMs, such as AI safety and reliability, and our work aims to reduce such risks. Beyond LLMs, we feel no other consequences must be highlighted here.

## References

* Askell et al. (2021) Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A general language assistant as a laboratory for alignment. _arXiv preprint arXiv:2112.00861_.
* Bai et al. (2022) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback. _arXiv preprint arXiv:2212.08073_.
* Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3615-3620.
* Bhardwaj and Poria (2023) Rishabb Bhardwaj and Soujanya Poria. 2023. Redteaming large language models using chain of utterances for safety-alignment. _arXiv preprint arXiv:2308.09662_.
* Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Tianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatapt quality.
* Demszky et al. (2023) Dorottya Demszky, Diyi Yang, David S. Yeager, Christopher J. Bryan, Margaret Clapper, Susannah Chandhok, Johannes C. Eichstaedt, Cameron Hecht, Jeremy Jamieson, Meghann Johnson, Michaela Jones, Danielle Krtetek-Cobb, Leslie Lai, Nirel JonesMitchell, Desmond C. Ong, Carol S. Dweck, James J. Gross, and James W. Pennebaker. 2023. Using large language models in psychology. _Nature Reviews Psychology_, 2:688-701.
* Gangiul et al. (2022) Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. 2022. Red tearing language models to reduce harms: Methods, scaling behaviors, and lessons learned. _arXiv preprint arXiv:2209.07858_.
* Lee et al. (2023) Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbuine, and Abhinav Rastogi. 2023. Rlair: Scaling reinforcement learning from human feedback with ai feedback. _arXiv preprint arXiv:2309.00267_.
* Li et al. (2023) Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. 2023. Self-alignment with instruction backtranslation. _arXiv preprint arXiv:2308.06259_.
* Lin et al. (2021) Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. _arXiv preprint arXiv:2109.07958_.
* Liu et al. (2023) Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2023. Trustworthylms: a survey and guideline for evaluating large language models' alignment. _arXiv preprint arXiv:2308.05374_.
* Madan et al. (2023)Aman Madan, Niket Tandon, Pradhar Gupta, Skyler Halliman, Luyu Gao, Sarah Wiegerffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. _arXiv preprint arXiv:2303.17651_.
* OpenAI (2023) OpenAI. 2023. GPT-4: Improving language model performance through scale and data. _arXiv_.
* Rao et al. (2023) Kavel Rao, Liwei Jiang, Valentina Pyatkin, Yuling Gu, Niket Tandon, Nouha Dziri, Faeze Brahman, and Yejin Choi. 2023. What makes it ok to set a fire?

iterative self-distillation of contexts and rationales for disambiguating defeasible social and moral situations. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 12140-12159.
* Saunders et al. (2022) William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. Self-critiquing models for assisting human evaluators. _arXiv preprint arXiv:2206.05802_.
* Shaikh et al. (2022) Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. 2022. On second thought, let's not think step by step! bias and toxicity in zero-shot reasoning. _arXiv preprint arXiv:2212.08061_.
* Shen et al. (2023) Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. 2023. Large language model alignment: A survey. _arXiv preprint arXiv:2309.15025_.
* Srivastava et al. (2022) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abuwal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adria Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. _arXiv preprint arXiv:2206.04615_.
* Sun et al. (2023) Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. 2023. Principle-driven self-alignment of language models from scratch with minimal human supervision. _arXiv preprint arXiv:2305.03047_.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Zelikman et al. (2023) Eric Zelikman, Wanjing Ma, Jasmine Tran, Diyi Yang, Jason Yeatman, and Nick Haber. 2023. Generating and evaluating tests for k-12 students with language model simulations: A case study on sentence reading efficiency. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 2190-2205.
* Zhang and Yang (2023) Xuanyu Zhang and Qing Yang. 2023. Self-qa: Unsupervised knowledge guided language model alignment. _arXiv preprint arXiv:2305.11952_.

## Appendix A Appendix

### Generalizability of Iteralign

The use of strong LLMs and red teaming datasets is a demonstration (proof-of-concept) that IterAlign's alignment paradigm can effectively improve the performance of open-accessible LLMs in terms of safety aspects. Essentially, the strong LLMs and red teaming datasets serve as the **supervisions** of the alignment process, like the human-annotated data in RLHF and the human-written constitutions in Constitutional AI (CAI), and the supervisions can be generalized to many other forms. For example, the stronger LLMs can be substituted with domain experts or any LLM agent with domain knowledge, while the red teaming datasets from two existing red teaming methods used in IterAlign can be extended to any other red teaming methods.

## Appendix B Base Model Selection

We instantiate our base model with llama-2, llama-2-chat, and vicuna-v1.5 since they are the de facto gold standard open-source base models for experiments. In our opinion, these base models are representative and diverse, since they range from different stages of LLMs, namely pretrained, instruction tuned, and finetuned. We would like to clarify that our method can be adapted to any other base model, such as T5, BART, etc.