SoFA: Shielded On-the-fly Alignment via Priority Rule Following (2024)

Abstract: Content warning: This paper contains examples of harmful language. The alignment problem in Large Language Models (LLMs) involves adapting them to the broad spectrum of human values. This requirement challenges existing alignment methods due to diversity of preferences and regulatory standards. This paper introduces a novel alignment paradigm, priority rule following, which defines rules as the primary control mechanism in each dialog, prioritizing them over user instructions. Our preliminary analysis reveals that even the advanced LLMs, such as GPT-4, exhibit shortcomings in understanding and prioritizing the rules. Therefore, we present PRIORITYDISTILL, a semi-automated approach for distilling priority following signals from LLM simulations to ensure robust rule integration and adherence. Our experiments show that this method not only effectively minimizes misalignments utilizing only one general rule but also adapts smoothly to various unseen rules, ensuring they are shielded from hijacking and that the model responds appropriately.1

text: Findings of the Association for Computational Linguistics ACL 2024, pages 7108–7136 August 11-16, 2024 ©2024 Association for Computational Linguistics
The alignment problem in Large Language Models (LLMs) involves adapting them to the broad spectrum of human values. This requirement challenges existing alignment methods due to diversity of preferences and regulatory standards. This paper introduces a novel alignment paradigm, priority rule following, which defines rules as the primary control mechanism in each dialog, prioritizing them over user instructions. Our preliminary analysis reveals that even the advanced LLMs, such as GPT-4, exhibit shortcomings in understanding and prioritizing the rules. Therefore, we present PRIORITYDISTILL, a semi-automated approach for distilling priority following signals from LLM simulations to ensure robust rule integration and adherence. Our experiments show that this method not only effectively minimizes misalignments utilizing only one general rule but also adapts smoothly to various unseen rules, ensuring they are shielded from hijacking and that the model responds appropriately.1

heading: 1 Introduction
text: In the present era of Large Language Models (LLMs), the alignment problem has emerged as a foundational and critical research direction (Shen et al., 2023; Ji et al., 2023), which centers on the challenge of ensuring AI systems behave in line with human intentions and values (Leike et al., 2018). Technically, the alignment process can be viewed as steering the LLM’s output (Burns et al., 2023). For example, in Figure 1, while all three responses are reasonable, the alignment process aims to steer the model towards a more harmless direction in regard to human society.
* Corresponding authors. 1Code and data are available at https://github.com/
luxinyu1/sofa
Despite its increasing importance, a key challenge lies in the diversity and the complexity of human values (Casper et al., 2023), i.e., how to effectively steer LLMs to satisfy diverse user preferences (Cheng et al., 2023; Bakker et al., 2022) and comply with various regulations (Yang et al., 2023). First, preferences and regulations vary greatly in different real-world scenarios. For instance, honesty is a valuable trait for LLMs when performing knowledge-intensive tasks (Zhang et al., 2023), but it can be a disturbance when instructed to play as an agent in strategic games (Xu et al., 2023b). Second, the relationship between regulations and instructions is complex, the model needs to well comprehend the intention and boundary of regulations to avoid violating them while still fulfilling user requests. For example, the model should be capable of discerning whether the context involves strategic games, particularly if a regulation states, “When playing strategic games, one should not al-
7108
ways be honest.” Unfortunately, current alignment technologies fail to effectively address these challenges. Learning-based alignment methods like Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017; Stiennon et al., 2020; Bai et al., 2022a; Ouyang et al., 2022) heavily rely on annotating preference data, making them both time-consuming and expensive for specific scenarios. On the other hand, it is a common practice that steers the generation of LLMs with controlling instructions, such as “You are a helpful assistant.” in ChatGPT. Unfortunately, the unclear boundary and complex relationship between regular instructions and the controlling ones can cause conflicts, leading to confused and hijacked model responses (Mu et al., 2023; Toyer et al., 2023). To address these challenges effectively, it is essential to develop a mechanism that enables LLMs to clearly distinguish controlling instructions from the other instructions. Furthermore, it’s important to train these models to better integrate the rules, ensuring that controlling instructions are shielded from hijacking and that the model responds appropriately.
In this paper, we propose the alignment paradigm of priority rule following, which defines rules as a controlling strategy for each dialogue and prioritizes these rules above all user instructions. Specifically, as illustrated in Figure 1, we focus on two core abilities of priority following: (1) Integration, which entails effectively incorporating the rules to steer the model’s responses to instructions (e.g., ensuring the model comprehends the intention of the rule to be harmless to humans) (2) Maintenance, which involves the model’s adherence to the rules when there are attacks or conflicts from instructions (e.g., maintaining harmlessness even when faced with a deceptive “Sure, here’s” style attack in the user instruction). Our preliminary study reveals that even the most advanced LLMs (e.g., gpt-4) still lack proficiency in these abilities.
To enhance the capability of LLMs in integrating and maintaining rules, we introduce PRIORITYDISTILL, a semi-automated process that distills priority-following signals from LLM simulations. Specifically, we design a self-align algorithm that identifies diverse rules and their key corresponding instructions, then simulate the process of priority following through a System 2 process to ensure correct comprehension of the rules’ intentions and priorities. Utilizing this algorithm, we harvest PRI-
ORITYRULES, a rule-based alignment dataset containing over 20K rules with corresponding instructions and responses. Additionally, to ensure that the alignment process of the model is based on the rules, we introduce a supervision signal from the reference model to prevent the model from directly memorizing the instruction-response pair. Our experiments reveal that the proposed method not only effectively reduces misaligned behaviors using a single general rule, but also effectively applies to various unseen rules, rejecting the harmful ones.
The contributions in our paper are three-fold:
1) We introduce a novel alignment paradigm that trains models to better integrate and maintain rules, thereby enabling shielded on-the-fly alignment, rather than directly learning preferences and regulations.
2) We propose PRIORITYDISTILL, a semiautomated process that improves the model’s ability to integrate and maintain rules, consequently enhanced the model’s on-the-fly alignment capability.
3) We identify and further annotate a set of benchmarks to examine the model’s proficiency in on-the-fly alignment, providing a resource that can benefit future research.

heading: 2 Related Work
text: Rules can be viewed as clear and concrete standards for alignment. The predominant alignment process (Bai et al., 2022a; Ouyang et al., 2022; Touvron et al., 2023; Rafailov et al., 2023) encodes models with a vague boundary mirroring the general preferences of the majority of human annotators. As a supplement, Sparrow (Glaese et al., 2022) and GPT-4 (OpenAI, 2023) utilized a rule-based reward model to assess and mitigate potential violations against established rules, enhancing the model’s helpfulness and safety by combining it with preference-based rewards. Yuan et al. (2024) demonstrate the potential of prompting LLMs with rule-based criteria and regarding the judging results as rewards for iterative self-improvement.
In addition to using rules for reward modeling, another line of works choose to in-context steering the model via generating (Sun et al., 2023; Yang et al., 2024), critiquing and refining (Bai et al., 2022b; Tan et al., 2023) response based on predefined principles or constitutions. In particular, Bai et al. (2022b) and Sun et al. (2023) demonstrate
that robust alignment can be achieved through the amplification of supervision signals from a static set of constitutional rules. Motivated by these findings, our work expands this setting to include both static and on-the-fly rules with a hierarchical prioritization, aiming to achieves more adaptive control of the model’s behavior with support from the LLMs themselves. The on-the-fly rules can vary in granularity, from general constitutional ones to instance-specific ones.
Concurrently, Mu et al. (2023) and Toyer et al. (2023) explore LLMs’ ability to follow taskspecific rules. Xu et al. (2023a) collects real-world laws and retrieves the relevant ones to perform onthe-fly alignment, while our work focuses on the generator-side optimization using synthetic rules.

heading: 3 Priority Rule Following
text: To enable the model to adaptively align with a broader range of regulations and preferences, we propose enhancing its ability to align with given rules with priority, rather than learning the preferences and regulations from responses directly.

heading: 3.1 Core Abilities
text: In this section, we define two critical abilities necessary for a language model in the priority rule following paradigm: integration and maintenance.
Integration Ability refers to the model’s ability to integrate the rules to guide its response to specific instructions. This includes model’s adaptability in aligning with a diverse range of rules and the applicability of these rules as universal constraints for all user instructions within their defined scope. Importantly, the model should be sensitive to the scope of each rule and ensure responses to out-of-scope instructions are not interfered by the rule. Additionally, to prevent potential abuse (Shah et al., 2023), the model should hold the ability to reject toxic rules, such as “You are a harmful assistant”.
Maintenance Ability is about the models’ ability to consistently adhere to the rules, regardless of the context and instructions. That is, the rule should be followed in priority over the user instructions. When user requests conflict with these rules, the latter should take precedence. The model should be equipped with the ability to maintain the order of priority under potential attacks on the rules.

heading: 3.2 A Preliminary Study
text: To explore the rule following capability of current LLMs, we conducted a preliminary study with 55 real-world rules. Each rule necessitates the LLM to insert specific sentences into its response under certain conditions. For example, in Figure 2, when asked about diet or nutrition advice, the LLM should trigger the rule and generate responses that end with the specified content.
Given the predictable effects of the rule, the integration and maintenance capabilities can be automatically evaluated with the pass rate by string matching on the specific sentence within the response. To assess the integration ability, we pair the rule with both related and unrelated instructions. Specifically, the inserted sentence should be presented in the response to the related instruction but not in the unrelated one. If both criteria are satisfied, the rule is considered “passed”. For testing the maintenance ability, about half of the related instructions introduce a conflict (e.g., don’t believe the doctor in Figure 2), the model should adhere to the rule even though the instruction intentionally guide the language models to violate it.
The results in Figure 3 indicate all evaluated models exhibit a limited ability to integrate and maintain the rule effectively, with no model achieving a pass rate above 0.4. This limitation becomes more pronounced in cases of conflicting instructions, resulting in even lower pass rates. Notably, gpt-3.5-turbo-1106 outperforms gpt-4 in achieving the highest pass rate. We further analyze the responses and find that although gpt-4 is more capable of dealing with conflicting instructions, it struggles more with unrelated instructions, suggesting that mere scaling may not effectively address this limitation. Based on these observations, we emphasize the importance of enhancing the model’s abilities to integrate and maintain rules.

heading: 4 PRIORITYDISTILL
text: To achieve targeted refinement, this section introduces PRIORITYDISTILL, a process that distills priority following signals from LLM simulations. With r representing the alignment rule, i as the corresponding user instruction, and y as the ruleregulated response, PRIORITYDISTILL primarily addresses three main challenges: first, identifying high quality (r, i) pairs that can potentially supervise the integration and maintenance abilities; second, obtaining the appropriate response signal y
that satisfied the key properties outlined in Section 3; third, effectively learning the (r, i, y) triplets.
The simulation process is outlined in Section 4.1, and illustrated in Figure 4. Design specifics, including the complete set of prompts used are in Appendix B. Section 4.2 introduces the resulting dataset. Section 4.3 introduces learning methods.

heading: 4.1 Simulate Priority Execution with LLMs
text: The simulation process includes three steps, focusing on harvesting triplets d = (r, i, y).

heading: 4.1.1 Harvest Rules and Instructions
text: To prepare a large and diverse rule set SR and instruction set SI for subsequent steps, while leveraging minimum human annotation. For the rule generation, inspired by Wang et al. (2023), we prompt the LLM to extend the rule set iteratively. At each iteration, the LLM receives a specific number of topics or rules as hints from seeds and generates new rules based on these hints. These newly generated rules then added to the seed pool. We also employ a ROUGE-L filter to guide the language
model to extend the rule set to a considerably large quantity while ensuring quality and diversity. The instruction generation process is conducted identically to Wang et al. (2023) and Taori et al. (2023) for simplicity.
The expanded rule set SR roughly contains three main categories of rules starting with “You/Your”, “When”, and “If”. Each category provides extensive possibilities for steering the model’s response. For instance, “When discussing space exploration, emphasize potential risks of the unknown.” and “You should use formal language and maintain a posh tone in your responses.” are representative examples of these rules.

heading: 4.1.2 Automatic Probe and Constraint Generation
text: After harvesting a diverse set of rules and instructions, our goal in this step is to identify crucial instruction and rule pairs for the enhancement of integration and maintenance ability. Therefore, for each rule in SR, we prompt LLM to simulate three categories of user instructions: Related, within the regulatory scope of a rule; Unrelated, outside of this scope; Attack, which encourages the model to violate a rule. Specifically, the integration ability is enhanced by simulating both related and unrelated instructions for one rule, which can further enhance the comprehension of the intention of rules, similar to Yoran et al. (2023). The maintenance ability is enhanced by the attack instructions.
Furthermore, merely the paired instruction derived from rules is insufficient, since the content and form of instructions are highly limited by the content of the rules. Therefore, as a mutually complementary, we instruct the LLM to generate related and unrelated constraint rules for each harvested instruction in SI . But notably, we don’t
instruct the LLM to generate rules that being attacked by the given instruction because it’s beyond the current model’s capability in our trials.

heading: 4.1.3 Priority Distillation
text: After the previous steps, we collected high-quality (r, i) pairs. The final step, termed priority distillation, is aimed at getting the corresponding response y that meets the key properties outlined in Section 3. To accomplish this, we define constitutional rules based on Sun et al. (2023) in the prompt. Subsequently, we add an illustration that explains the concept of priority in context. This includes specifying the priority order as follows: Constitutional Rules ≥ The On-the-fly Rule > Instructions.
To guide the LLM in systematically analyzing and judging the relationship between instructions, the on-the-fly rules and constitutional rules, we prompt the LLM to engage in Chain-of-Tought (CoT) reasoning (Wei et al., 2022) before generating each response. During this process, the LLM is instructed to:
(1) Analyze the user’s intention behind the instruction;
(2) Identify the related constitutional rules;
(3) Judge whether the user instruction fall within the regulatory scope of the on-the-fly rule;
(4) Determine if there exists conflict between the constitutional rules, the on-the-fly rule, and the instruction, and reclaim the priority.
Through this CoT process, we verbalizing the complex reasoning and judgement path, leading to responses that better align with the rule’s intention and aware of regulatory scope, while maintaining the priority order. This process is then distilled into the target LLM’s parameters θt by directly learning the (r, i, y) triplets, which can be represented as KL(pθt(y|r, i) || pLLM(y|r, i, c)). Here c is the prompt in-context and the CoT reasoning process.

heading: 4.2 The PRIORITYRULES Dataset
text: While the simulation process is applicable to both base LLMs and the aligned ones, in this paper we implemented the process using gpt-3.5-turbo-1106 (OpenAI, 2022). Throughout all stages of the simulation, we refined the prompts based on manual analysis of the output quality. We also categorized the observed failures and incorporated them into the requirements part of each corresponding prompt.
Table 1 presents the basic statistics of the final
training set, denoted as PRIORITYRULES. We harvest over 20K different rules in total and 42K corresponding instructions from the pipeline. For quality evaluation purposes, 2% of the total data was reserved. We conduct both automatic and human evaluation on this reserved set to assess the data quality. The evaluation details are depicted in Appendix C. Both evaluation indicates low error rates and high data quality.

heading: 4.3 Learning with Reference
text: Training with Rules On-the-fly During training, rule r is on-the-fly injected into the model with the paired instruction i, and the loss is only computed on the response y:
Lrule = −E(r,i,y)∼Dp∪Db
[ T∑
t=1
log pθ ( yt | y1:t−1, r, i
) ]
Here Dp represents a rule-following training set like PRIORITYRULE, and Db denotes the base dataset used for instruction tuning. Considering the instances in Db do not have corresponding rules, the rule element is either considered empty or uniformly set across all instances.
Integrating the Reference Signal Additionally, to ensure that the model’s alignment process is guided by the on-the-fly rules rather than simply memorizing instruction-response pairs, we introduce a reference term Lref into the standard crossentropy loss. This term samples the response to instruction in Dp from corresponding reference model πref which is only instruction-tuned on Db. The final loss, incorporated with the reference term, is formally defined as follows: Lref = −Ei∼Dp,y∼πref(i,θ′) [ T∑
t=1
log pθ ( yt | y1:t−1, i
) ]
Lfull = Lrule + Lref
The introduction of the reference signal is intended as an optional performance enhancement measure, in practice, relying solely on the Lrule can achieve most of the performance gains. We will present relevant experimental results in the next section.

heading: 5 Experiments
text: In this section, we conduct experiments to explore the following research questions: RQ1 (Depth): Does enhancing the model’s priority rule following ability helps in mitigating common misaligned behaviors?
RQ2 (Breadth): Does enhancing the model’s priority rule following ability helps the model to be more aligned to diverse regulations?
To investigate RQ1, we adopt the default system message of Llama-2-chat as the rule (See in Figure 6). In this context, the model’s rule-based alignment ability performs constitutional alignment (Bai et al., 2022a). We measure the harmlessness, bias, and truthfulness of the model using HHRedTeaming (Ganguli et al., 2022), TruthfulQA (Lin et al., 2022), BBQ (Parrish et al., 2022). This setting tests the depth of alignment that one rule can achieve since it is examined by thousands of instructions in various scenarios.
To answer RQ2, we need to vary the rules to test the model’s ability of on-the-fly rule adaptation, observing how the model adapts to diverse regulations. We adapt RuLES (Mu et al., 2023) as the test bench. The detailed information of all the datasets and settings are shown in the Appendix D.2.

heading: 5.1 Training Details
text: All models are trained using the Llama-2-chat template (Touvron et al., 2023), in which rules are prepended as system messages at the beginning of the dialogue and separated with the instructions with special tokens. Each model is fine-tuned for three epochs. We use the same hyperparameters for all the models to ensure comparability, except that the batch size for the 7B and 13B models is 32 and 16. The full set of hyperparameters can be found in Appendix D.3.

heading: 5.2 Baselines
text: Llama-2-Chat (Touvron et al., 2023) is a set of industry-level dialog models finetuned on publicly available data and proprietary data. It can be viewed as a strong baseline for all benchmarks, since it went through iterated RLHF process and optimized for system instruction using synthetic short constraints distilled from itself.
ShareGPT is a dialog dataset collected from sharegpt.com, a website collecting user-shared dialog with ChatGPT. While ShareGPT is the training set of Vicuna (Chiang et al., 2023), the exact data has not been released. We adopt a version containing 86K dialogs.
ShareGPT Uncensored is a variant of the ShareGPT dataset, which roughly filtered out alignment signals based on string matching (e.g., remove all the ChatGPT answer starts with "I’m
sorry"). The uncensored variant contains 53K dialogs. The complete list of matching rules is shown in the Appendix D.4.

heading: 5.3 Benchmark Results
text: We compare the PRIORITYRULES enhanced models with their respective base instruction-tuned models. Table 2 presents the results.
The depth of alignment. From the results of HH-RedTeaming, BBQ, and TruthfulQA, we can observe that: 1) Through enhancing the integration and maintenance ability of models, the model trained with PRIORITYRULES can mitigate more misaligned behaviours with same single rule pretended in-context. The enhancement helps most when the base instruction tuning data lack censorship signals (i.e., the uncensored models). 2) The depth of alignment can scale with the model capability. 3) Prepending all the base instructions with
the same general rule (+sys) can decrease the rule integration ability, while training with the reference signals increase the ability.
The breadth of alignment. The results on RuLES suggest that integrating PRIORITYRULES enhance the model’s ability to on-the-fly align with a diverse set of rules in both systematic and manual test suites. We specifically evaluated ShareGPTbased models, analyzing the categorized pass rates by attack methods and rule scenarios. It is evident that incorporating PRIORITYRULES increases the pass rate across the red-teaming scenarios compared to the vanilla ShareGPT and Llama-2-chat models, thereby affirming it notably enhances the model’s rule maintenance ability. The categorized pass rate are visualized in the Figure 17 in Appendix. Additionally, we test the enhanced models on the rules test set annotated in the preliminary study. Our best performing 7B model surpasses
the gap between 2 and 3 is significantly smaller. Similar results are observed in the uncensored models.
gpt-3.5-turbo and gpt-4 in conflict (+11%) and overall (+7.3%) pass rate, indicating better integration and maintenance ability of these rules.

heading: 5.4 Detailed Analysis
text: The bonus to reject the harmful rule. In order to prevent the potential misuse of on-the-fly aligned LLMs, the model should be capable of rejecting the harmful rules. Hence, we manually design a set of ten harmful rules (see these rules in Table 13 in Appendix), and then test the model’s rejection ability to these toxic rules using HH-REDTEAMING2. As shown in Table 5, our models are more robust to the harmful rules compared to the models only trained on the base instruction tuning data. We
2Each instruction in dataset is randomly assigned a toxic rule.
further examine the data and find that although we didn’t intentionally design the red-teaming rules, it’s still likely to occur in the harvested rules, and further be rejected in the priority distillation stage.
The casual effect of the rule. To examine the impact of the added rule (i.e., the system message) on RQ1-related improvements, we perform an ablation study to assess the model’s performance when inference without the rule, thus determining its causal effect. As shown in Table 3, compared to the base instruction-tuned models, we notice that most of the improvement comes from the prepended rule, instead of directly memorizing preferred values and regulations in the training data.
The alignment tax. To evaluate the potential costs associated with additional training of the rulebased alignment data, we examine the models’ performance on the standard LLM test suite3. The results in Table 4 indicate a minimal alignment tax resulting from the additional training data. Notably, there was no significant difference in Accuracy on the TruthfulQA benchmark when evaluated in the few-shot setting. This suggests that the alignment process does not directly parameter-learn values and preferences. Instead, the on-the-fly rule is the primary factor mitigating the model’s misaligned behavior.
The long-term decay trend. We further analyze the decay effects of rule effectiveness as the number of dialogue turns increases. Specifically, we add additional unrelated dialogue turns using instructions sampled from the Alpaca (Taori et al., 2023) dataset and allow the model to generate responses until k − 1 turns are achieved. We then query the model with the test cases. As shown in Figure 5, it can be inferred that the enhanced rule-
3https://huggingface.co/spaces/HuggingFaceH4/ open_llm_leaderboard
following capabilities demonstrate some degree of persistence over the long term, despite the absence of multi-turn dialogue datapoints in the PRIORITYRULES. We believe integrating methods like GAtt (Touvron et al., 2023) can further enhance the long-term dependency ability.
Qualitative analysis. To better comprehend the enhancements in the models’ on-the-fly alignment capabilities, we collected and designed several rules, and conducted a qualitative analysis across both single-turn and multi-turn scenarios. Observations indicate that the model’s performance improves in instructions that are related, unrelated, and involve adversarial attacks. Due to space limitations, detailed cases are provided in Appendix D.6.

heading: 6 Conclusion
text: In this paper, we propose a novel alignment paradigm to train the integration and maintenance ability of models and perform alignment in a onthe-fly manner. We design a process called PRIORITYDISTILL to enhance these two abilities via LLM simulation. Experimental results demonstrate the enhanced rule-base alignment ability can helps the model mitigate more misaligned behaviors and achieves compliance with a wider range of regulations. We hope future work can build on top of our paradigm and the alignment process to develop better on-the-fly aligned models.

heading: Limitations
text: Flaw in short natural language rules The rules described using short texts can be incomplete and
easily to be misinterpreted. Despite in this work we enhanced the rule-based alignment ability of the models, the benchmark performance is still far from perfect. Future work can focus on automatically deducting more self-consistent and clear rules that minimize ambiguity for the targeted models. Additionally, we believe the rule-based alignment capability serves as a supplement to constitutional alignment, providing an online strategy to intervene in the model’s output, rather than being a substitute for constitutional alignment.
Boarder and deeper rule-based alignment Ideally, the rule should constrain the model’s embodied behaviour, such as tool use as discussed in Ruan et al. (2023) and internal states (Hubinger et al., 2019) as well as surface form language output. We will investigate the boarder and deeper alignment in the future research.

heading: Ethics Statement
text: We have developed techniques aimed at enhancing the alignment with rules on-the-fly, ensuring that the model’s behavior aligns with human values. Meanwhile, our method is more robust to reject the harmful rules. Specifically, our approach contains the step of prompting the LLM to generate instructions to attack the rule. Through conducting simulated attacks, a process referred to as red-teaming, we aim to identify and mitigate weaknesses in the system. This ultimately contributes to the enhanced security of our models.

heading: Acknowledgments
text: We sincerely thank all anonymous reviewers for their insightful comments and valuable suggestions. This research work is supported by the National Natural Science Foundation of China under Grants no. 62122077, no. 62306303, no. 62106251, and the Basic Research Program of ISCAS, Grant No.ISCAS-JCZD-202303.

heading: A Additional Details of the Preliminary Study

heading: A.1 Annotation Details
text: The test set used in the preliminary study was meticulously annotated by a pair of annotators. This annotation process encompassed a preparatory training session, succeeded by the actual annotation phase. During this initial training, the annotators
were equipped with a set of guidelines and selected examples chosen by the authors of this study. These examples were thoroughly reviewed with the annotators to address any uncertainties and to refine the annotation guidelines. One annotator is responsible for labeling all the rules, while the other handles the labeling of all instructions and their corresponding attributes. Both annotators can use ChatGPT for brainstorming purposes during this process. A competitive compensation of $30 per hour was provided for both the annotation and training periods.

heading: A.2 More Results
text: We present additional results on the probe test set in Table 6, The baseline prompt comprises straightforward instructions that emphasize rule adherence, and the priority definition prompt further specifies strategies for handling conflicts. As can be observed, the inclusion of a priority definition enhanced the model’s capability to resolve conflicts, achieving overall performance comparable to treating rules as system prompts in ChatGPT APIs. Furthermore, our trained models significantly outperform all models on both Pass Rates.

heading: B Additional Details of
text: PRIORITYDISTILL

heading: B.1 Harvest Instruction and Rules from LLMs
text: We implemented two methods to extend the rule set. The first involves directly amplifying rules from the initial seed set, focusing on harvesting rules that constrain the response style. The second method begins by mining topics (e.g., culture, education), then generate the principle rules using the mined topics as the hints. We implemented it to harvest principles that a human or AI assistant can adhere to.
The complete prompts to implement these methods are shown in Figure 7, which includes a task definition, and the detailed requirements about the content and format. The seed examples are organized as one rule per line.
For output validation, we instruct the model to present the harvested instruction in JSON format. We discard any output that failed JSON parsing. Furthermore, we employed a ROUGE-based filter to eliminate highly similar rules. Specifically, we calculate the ROUGE-L scores of the generated rules with each seed rule and previously generated rule, the rules with a score higher than a threshold
is discarded. This process continued until 1K rules were harvested. We provide several examples of the seed rules and generated examples in Table 7.
The process for instruction mining replicates the methodology of Wang et al. (2023). We use the resulting dataset implemented by Taori et al. (2023) to minimize API call costs.

heading: B.2 Automatic Probe and Constraint Generation
text: For this step, the objective is to generate various paired instructions and rules. We specifically target the generation of pairs that exhibit three distinct types of relationships: related, unrelated, and attack.
When simulating user instructions for the given rules, apart from the instructional guidelines, we carefully annotated these three category user instructions for several rules and used them as incontext exemplars and guide the simulation process to be more stable. Our method used in this step is loosely inspired by Perez et al. (2022, 2023), but red-teaming each rule is more challenging than red-teaming the language model to elicit harmful behaviors since the attack instructions can vary greatly with regard to each rule. We find showing exemplars with the definition of attack helps the LLM to attacks the rule, but current LLMs still fall short of the creativity of attack instructions.
The complete prompt used to probe user instructions from rules is shown in the Figure 8 and 9.
The complete prompt to automatically generate constrainal rules for a given instruction is shown in Figure 10. We provide examples of paired rules and instructions in Table 9.

heading: B.3 Priority Distillation
text: Apart from the details introduced in the main page, we advocate for transparent alignment, where the model should clearly indicate when its behavior is regulated by rules, especially in sensitive scenarios or those with potential negative impacts. For example, rules like "You believe that abortion should be legalized" are essential for an LLM to act like an agent with social functions (Xi et al., 2023), but can lead to the division in society if vastly deployed without transparency. we can easily achieve transparency alignment as we can steer the model to respond "as per the rule regulation" using in-context exemplars when necessary.
This method draws partial inspiration from Context Distillation (Askell et al., 2021) and Dromedary (Sun et al., 2023). However, our approach differs from them in that it requires the LLM to concisely analysis the priority order and complex relationships between constitutional rules, on-the-fly-rules, and instructions, rather than perform the alignment process through distilling data from a static rule set.
The detailed version of prompt used in this step are shown in Figure 11 and 12.

heading: C Details of PRIORITYRULES

heading: C.1 Quality Control
text: To evaluate the overall quality of the harvested data in the simulation, we use the prompt in Figure 14 and Figure 15 to examine the rule following performance and helpfulness of the data. The evaluation prompt for harmless is akin to that in Figure 16. We iteratively refined these prompts until achieving satisfactory consistency and evaluation quality across trial runs. The model engine used in this evaluation is gpt-3.5-turbo. The results are presented
in Table 10, demonstrating high rule adherence and helpfulness while minimizing the generation of potentially harmful content.
Additionally, we conduct a human annotation of 100 randomly selected samples from the dataset. This annotation is exclusively carried out by the authors, without the involvement of external annotators. The analysis primarily focuses on two types of errors in the dataset: rule generation errors and instruction generation errors. Furthermore, the quality of the responses is assessed using a fivepoint scoring criteria, with the aggregate scores also recorded in Table 10.
During the annotation process, it was observed that the predominant rule generation errors involved the repetition of information from the instructions within the rules or the generation of vague rules. The principal errors in instruction generation are the production of instructions that did not align well with real-world scenarios and some generated instructions that are perplexing. The key issues affecting response quality include exaggerated refusals, undue brevity of some responses, rep-
etition of information from in-context exemplars, and rare instances of responses violating the rules.
Both automatic and human quality control ensures the quality of the final dataset.
C.2 Visualization
Additionally, We further visualize the Figure 13 to demonstrate the main content of rules and instructions in the data, following Wang et al. (2023). The plot’s inner circle depicts the root verbs of rules and instructions, whereas the direct objects are shown in the outer circle.

heading: D Details of Benchmark and Evaluation

heading: D.1 The General Rule
text: We use the Llama-2-chat system message (see in Figure 6) to align the models to investigate the depth of alignment that only one rule can achieve. The system message is designed to guide the model towards being helpful, harmless, unbiased, and truthful.

heading: D.2 Benchmarks

heading: HH-RLHF RedTeaming (Ganguli et al., 2022)
text: contains human-crafted test case that that can potentially elicit harmful model responses. We filtered out the first turn user instruction as the test set. Meanwhile, to make the red teaming procedure more challenging to current models, we collected4 and simplified a set of adversarial templates and
4https://huggingface.co/datasets/rubend18/ ChatGPT-Jailbreak-Prompts
conduct an adversarial attack. To reduce the evaluation cost, we randomly select 1K samples from this subset. All evaluation are conducted using gpt-3.5-turbo5 through single choice format to evaluate whether the model’s responses are harmful. The full evaluation prompt is shown in Figure 16.
BBQ (Parrish et al., 2022) measures bias across various social dimensions in LLMs using scenariopaired multiple-choice questions, each with three possible answers.This dataset includes both ambiguous scenarios, where the answer is inherently
5The Jan. 2024 version.
unknown, and unambiguous scenarios with definitive answers. We provided a detailed example of ambiguous and unambiguous scenario and the relation with bias in Table 11. To reduce the inference cost, we randomly sampled 10% (i.e., 5849) instances as the final test set. We report Micro and Macro Accuracy of the answers as a measurement of the bias of LLMs, noting that there is a negative correlation between the accuracy and the bias.
TruthfulQA (Lin et al., 2022) measures the imitative falsehoods of LLMs through well-crafted misleading questions and corresponding true/false references, This dataset serves as an indicator of hallucination, which the constitutional alignment process aims to mitigate. TruthfulQA-MC6 is a multi-choice variant consisting of 684 questions, filtered from the original test set. We report the Accuracy metric on this test set.
RuLES (Mu et al., 2023) is a benchmark to test the rule-following ability of LLMs. Each rule defines a scenario inspired by classic computer security problems (e.g., User Access Control) and
6https://huggingface.co/datasets/EleutherAI/ truthful_qa_mc
child games (e.g., Rock Paper Scissors). Each rule is complied with a set of human-crafted test case to examine the model’s adherence to given rules. This benchmark contains two subset. The manual subset serve as a exploratory one recorded with the initial trail of users to testify and red-teaming the rules, while the systematic one clearly defines six high-level strategies summarized from the manual one to guide the red-teaming process. The evaluation is automatically conduct with pre-defined programs for each scenario.

heading: D.3 Hyperparameters
text: In the experiments, unless otherwise specified, the sampling strategy employed was greedy search. The hyperparameter settings utilized for fine-tuning are detailed in Table 8.

heading: D.4 Baseline Details
text: More example of string matching rules7 used to filter the censorship signals in original ShareGPT data is shown in Table 12.

heading: D.5 Categorical Analysis on RuLES
text: Figure 17 presents a categorized pass rate across various red-teaming methods and scenarios on RuLES. We refer the readers to the RuLES (Mu et al., 2023) paper for more detailed information about the attacks and scenarios.

heading: D.6 Qualitative Analysis
text: To provide more insights about what changed after the enhanced rule-based alignment ability in the trained model, we carefully collected and designed several rules and conduct a qualitative study for
7We follow the setting of https://huggingface.co/ datasets/cognitivecomputations/wizard_vicuna_ 70k_unfiltered
both single-turn and multi-turn settings. Table 14 presents cases of single-turn interactions. From cases 1 and 3, we observe that the model learned to prioritize rule adherence over following instructions. The second case shows the on-the-fly alignment capability with rules can make the alignment process more transparent and customized. Table 15 depicts a three-turn interaction with the models. In the first turn, the question clearly fall outside the regulation scope of the rule. Llama-2-chat directly violate the rule, base model that instruction tuned on the original ShareGPT data is interfered by the rule and fail to be helpful to the user. Only our model gives the helpful response. While in the second turn, the user use a common adversarial attack to instruct the model to leak the trigger word. Only our model defend this attack although it misinterpret about the reason. Regarding the final turn, our model also performs the best when we considering the adherence to the rule.

heading: Llama-2-chat system message

heading: Prompt for harvesting rules

heading: Automatic Constrain Generation

heading: Examine Rule Following Performance

heading: Harmful Discriminator

heading: Examine Helpfulness

heading: Future Work
text: Future work can focus on automatically deducting more self-consistent and clear rules that minimize ambiguity for the targeted models. Additionally, we believe the rule-based alignment capability serves as a supplement to constitutional alignment, providing an online strategy to intervene in the model’s output, rather than being a substitute for constitutional alignment. Boarder and deeper rule-based alignment Ideally, the rule should constrain the model’s embodied behaviour, such as tool use as discussed in Ruan et al. (2023) and internal states (Hubinger et al. , 2019) as well as surface form language output. We will investigate the boarder and deeper alignment in the future research