Title: Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labels
Venue: NeurIPS 2024
Accepted: yes
Keywords: alignment

contrastive learning

constitutional ai

self-improvement
Abstract: When prompting a language model (LM), users often expect the model to adhere to a set of behavioral principles across diverse tasks, such as producing insightful content while avoiding harmful or biased language. Instilling such principles (i.e., a constitution) into a model is resource-intensive, technically challenging, and generally requires human preference labels or examples. We introduce SAMI, an iterative algorithm that finetunes a pretrained language model (without requiring preference labels or demonstrations) to increase the conditional mutual information between constitutions and self-generated responses given queries from a dataset. On single-turn dialogue and summarization, a SAMI-trained mistral-7b outperforms the initial pretrained model, with win rates between 66% and 77%. Strikingly, it also surpasses an instruction-finetuned baseline (mistral-7b-instruct) with win rates between 55% and 57% on single-turn dialogue. SAMI requires a model that writes the principles. To avoid dependence on strong models for writing principles, we align a strong pretrained model (mixtral-8x7b) using constitutions written by a weak instruction-finetuned model (mistral-7b-instruct), achieving a 65% win rate on summarization. Finally, we investigate whether SAMI generalizes to diverse summarization principles (e.g., "summaries should be scientific") and scales to stronger models (llama3-70b), finding that it achieves win rates of up to 68% for learned and 67% for held-out principles compared to the base model. Our results show that a pretrained LM can learn to follow constitutions without using preference labels, demonstrations, or human oversight.

Self-Supervised Alignment with Mutual Information Learning to Follow Principles without Preference Labels

Abstract

When prompting a language model (LM), users often expect the model to adhere to a set of behavioral principles across diverse tasks, such as producing insightful content while avoiding harmful or biased language. Instilling such principles (i.e., a constitution ) into a model is resource-intensive, technically challenging, and generally requires human preference labels or examples. We introduce SAMI , an iterative algorithm that finetunes a pretrained language model (without requiring preference labels or demonstrations) to increase the conditional mutual information between constitutions and self-generated responses given queries from a dataset. On single-turn dialogue and summarization, a SAMI-trained mistral-7b outperforms the initial pretrained model, with win rates between 66% and 77% . Strikingly, it also surpasses an instruction-finetuned baseline ( mistral-7b-instruct ) with win rates between 55% and 57% on single-turn dialogue. SAMI requires a model that writes the principles. To avoid dependence on strong models for writing principles, we align a strong pretrained model ( mixtral-8x7b ) using constitutions written by a weak instruction-finetuned model ( mistral-7b-instruct ), achieving a 65% win rate on summarization. Finally, we investigate whether SAMI generalizes to diverse summarization principles (e.g., 'summaries should be scientific') and scales to stronger models ( llama3-70b ), finding that it achieves win rates of up to 68% for learned and 67% for held-out principles compared to the base model. Our results show that a pretrained LM can learn to follow constitutions without using preference labels, demonstrations, or human oversight.

1 Introduction

Pretraining yields language models (LMs) with a vast array of knowledge and abilities. However, these models are difficult to use because they don't inherently reflect the values and preferences of human users. To address this issue, various alignment finetuning methods have become crucial for transforming LMs into useful AI assistants [25, 29, 6, intera alia ]. The success of these methods raises the question: Why do they work so well? Increasing evidence suggests that alignment finetuning methods expose and amplify aspects of the behavior distribution already implicit in the base pretrained model [e.g., 43, 21]. In this paper we build on this insight: We hypothesize that pretrained base models already have a weak statistical connection between behavioral principles, described in natural language, and the behavior that would realize them. We can encourage this connection by optimizing the conditional mutual information between principles and model responses given queries from a dataset. Finetuning the base model in this way requires no human preferences or examples yet yields a model which follows principles.

[a]

Give concise and

comprehensive

summarization

principles

•

Be concise

•

Be comprehensive

Principles

•

Be lengthy

•

Be incomplete

Antitheses

✏

Principle Writer

[c]

Contrastive Pairs

11

| x

1

, c

1

)

11

| x

1

, c

2

)

12

| x

1

, c

1

)

12

| x

1

, c

2

)

Constitutions

Responses

•

Be concise and comprehensive

•

Be lengthy and comprehensive

•

Be concise and incomplete

•

Be lengthy and incomplete

[b]

Be concise and comprehensive

Constitution c

1

(x

1

)

Summarize the

post: My dog […]

Query x

N

OP's  dog  has  eaten

their favorite dessert.

[…]

Response y

N

1

Be lengthy and comprehensive

Constitution c

2

(x

1

)

Summarize the

post: My dog […]

Query x

N

[…]

OP left their dog alone

at home. The dog felt

very hungry and […]

Response y

12

Response y

N

2

Response y

11

Sampled Constitutions

Aligning LMs to human preferences can be resource-intensive and technically challenging. For example, teaching a model to be helpful and harmless, or to summarize text effectively, often requires a large number of preference labels combined with complex reinforcement learning from human/AI feedback (RLHF/RLAIF) [5, 6, 19, 32, 38, 31]. Given the challenges of collecting preference labels and applying reinforcement learning, recent alternatives have explored aligning LMs directly through supervised finetuning [SFT; 43] or in-context learning [21, 33]. However, these approaches still rely on carefully curated SFT examples or in-context demonstrations of how to follow behavioral principles.

In this paper, we explore teaching an LM to follow behavioral principles (i.e., constitution) without preference labels or in-context demonstrations. We introduce S elf-Supervised A lignment with M utual I nformation ( SAMI ; see Figure 1), an iterative algorithm that finetunes a pretrained LM to increase the mutual information between a distribution of constitutions and self-generated responses. A SAMItrained mistral-7b [15] outperforms strong baselines after just three iterations on both single-turn dialogue [HH-RLHF; 5] and summarization [TL;DR; 31] (Figure 2 and Figure 4). Inspired by [7], we further test whether a strong base model [ mixtral-8x7b ; 16] can be aligned via constitutions sampled from principles written by a weak instruction-finetuned model ( mistral-7b-instruct ). The SAMI-trained model is better at summarizing TL;DR posts than the initial mixtral-8x7b model and mistral-7b-instruct (Figure 4a). Finally, we investigate whether SAMI can generalize to diverse summarization principles (e.g., 'summarize like a pirate') and scale to a more capable open-source language model [ llama3-70b ; 22]. The SAMI-trained model outperforms the base model on both learned and held-out principles (Figure 5), demonstrating that SAMI generalizes to stronger models and principles not seen during training. Overall, our contributions are as follows:

We introduce SAMI, an iterative algorithm that increases the mutual information between responses and constitutions.

We demonstrate that a SAMI-trained base model outperforms both the initial model and an instruction-following baseline.

We show that a weak instruction-finetuned model can write principles for aligning a strong base model.

We demonstrate that SAMI scales to state-of-the-art open-source models ( llama3-70B ) and generalizes to principles not seen during training.

Dialogue Principles: MI and Length

Dialogue Principles: Win Rates

Finetune

Weak

Principle Writer

Finetune

Weak

Strong

Principles

Principle Writer

✏

Setup

Setup

[a]

[b]

Strong

Principles

✏

Win Rates Against Base Model

Win Rates Against Instruct Model

Average Sequence Length

Mutual Information

2 Related Work

Preference Alignment with Human Preference Labels. A key method for aligning LMs is reinforcement learning from human feedback (RLHF) [e.g., 8, 25], which trains a reward model from human preference data to align a policy. However, RLHF requires a large amount of preference labels and online sampling of generations during training. Direct preference optimization [DPO; 29], sequence likelihood calibration [SLiC; 41], identity ( ψ ) preference optimization [ ψ PO; 4], and generalized preference optimization [GPO; 34] simplify the RLHF objective by directly maximizing the margin between preferred and dispreferred generations, but still rely on pairwise preference data. Kahneman-Tversky optimization (KTO) maximizes the utility of generated responses using 'thumbs-up' or 'thumbs-down' feedback [12], while relative preference optimization [RPO; 39] introduces a contrastive weighting scheme. However, each of the above approaches still relies-in one form or another-on an existing preference dataset.

Preference Alignment without Human Preference Labels. Due to the limited scalability of human-generated preference labels, recent works have used LMs to generate preference labels. The constitutional AI (CAI) paradigm [6] uses a small set of behavioral principles (e.g., 'do not be harmful') to compute log probabilities of responses, which are used to train a reward model for reinforcement learning with AI feedback [RLAIF; 19]. Kundu et al. [17] expanded on this idea, showing it is possible to use more general principles (e.g., 'do what's best for humanity'), while Sun et al. [32] demonstrated that a reward model can be trained to follow multiple trait principles. Relatedly, reinforcement learning from contrast distillation [RLCD; 38] has incorporated pairwise preferences and directional attribute changes in outputs, guided by contrastive prompts. While the above methods make effective progress on aligning LMs without human preference labels, they depend on a separate reward modeling stage, taking a very different approach than ours.

Algorithm 1: SAMI

Preference Alignment without Preference Optimization. Given the complexity of RLHF and related optimization methods, recent works have explored aligning pretrained (base) LMs without a reinforcement learning or preference modeling stage. For example, Sun et al. [33] have shown that as little as 300 lines of human annotation can be used to align an LM. Similarly, Zhou et al. [43] have demonstrated that 1,000 SFT examples are sufficient for steering a pretrained model. Their LIMA approach exhibits strong performance, learning to follow preferred response formats from a limited number of examples in the training data. Further relaxing the reliance on SFT examples, Lin et al. [21] have shown that pairing a system prompt with behavioral principles can match the performance of both an SFT baseline ( mistral-7b-instruct ) as well as a much stronger SFT + RLHF baseline [ llama-2-70b-chat ; 35]. However, despite relaxing reliance on a separate reinforcement learning or preference modeling stage, the above approaches still depend on carefully curated SFT examples or stylistic in-context examples, and as such do not teach a model to follow a set of desired behavioral principles more generally.

3 Self-Supervised Alignment with Mutual Information

In SAMI, we avoid supervised finetuning, reward modeling stages, and relying on preference labels or in-context examples. Instead, we build on the success of recent contrastive learning algorithms [28, 23] to improve a pretrained LM's ability to follow a set of behavioral principles (i.e., a constitution).

Preliminaries. To establish a distribution over constitutions C we first prompt an LM ω (the 'principle writer') to generate principles with several variants of each (see below for details). We then uniformly sample a variant for each principle to build a single constitution, c ∼ C . Next, given a dataset of queries D , we define a random variable X by uniformly sampling x from D . Finally, we define a distribution Y over responses by prompting an LM π (the target model for finetuning) to generate responses for query-constitution pairs. We now have a joint distribution over random variables C, X, Y . We assume that there already exists some (weak) dependency between responses

and constitutions, for at least some queries. The goal of SAMI is to increase this conditional mutual information between constitutions C and responses Y , given queries X : I ( Y ; C | X ) .

Objective. This conditional mutual information is, however, intractable. We can instead optimize a lower bound, such as the popular InfoNCE family [23]. In particular, because the conditional probability p ( y | c, x ) is tractable, we can use InfoNCE with an optimal critic, which simplifies [see 27, Eq. 12] to:

where the expectation is over sets of samples { c j , y ij } C j =1 from the joint distribution.

Due to the symmetry of mutual information, an alternative estimator can be derived using the reverse conditional probability p ( c | y, x ) , by normalizing over responses (see Section A.2, for a derivation). Combining the two lower bound estimates, as done in [28], yields a more stable estimator. This leads us to our final objective, for sampled queries x i , constitutions c j , and responses y ij :

We note that unlike typical applications of InfoNCE estimators for contrastive learning, the target of learning for SAMI affects both the sample distribution (for the second expectation) and the estimate (within the expectation).

Optimization. Equation 2 can be optimized in several ways. Following [40, 1], we employ a simplified variant of Expert Iteration [2] (see Algorithm 1). At each iteration, η , we sample a batch of queries X b from the dataset D and sample responses Y b using the previous model π η -1 for queryconstitution pairs ( x i , c j ) . We then construct contrastive pairs by computing the log probabilities of sampled responses under the initial model π 0 for each constitution used to generate responses. Log probabilities are then normalized row-wise and column-wise to obtain logits for computing the two-sided cross-entropy loss between the logits and an identity matrix (see Figure 6 for a reference implementation). During finetuning, we mask both constitutions c and queries x , calculating the loss only on responses y .

Regularization. An important failure mode of optimizing Equation 2 is the potential to overoptimize the objective, producing 'gibberish', a common issue in RLHF more generally. The solution is to regularize the model toward its initial state. We here regularize against distribution shift by using a small number of gradient updates during earlier iterations, thus preventing the model from diverging too far from the initial model. An alternative would be to regularize by limiting changes in behavior, instead of in parameters. This is typically done by adding an objective KL ( p π η ( y ij | x i , c j ) || p π BASE ( y ij | x i , c j )) [see e.g., 31]. However, this increases algorithmic complexity and did not help in initial testing.

4 Experiments and Results

Datasets and Models: Following previous work [e.g., 29], we empirically evaluate SAMI across two domains: dialogue [HH-RLHF; 5] and summarization [TL;DR; 31]. For HH-RLHF, we focus on the helpful-base and harmless-base datasets, using only the first human query from each dataset and discarding subsequent turns and preference labels. For TL;DR, we focus on the comparisons dataset, again discarding preference labels. In our first experiment on dialogue, we use mistral-7b as the base model and write principles by prompting claude-opus-20240229 . We then run a second experiment to compare principles written by a weak instruction-finetuned model ( mistral-7b-instruct ) to those written by a stronger model ( claude-opus ), finetuning both mixtral-8x7b and mistral-7b on summarization. Finally, to explore whether SAMI scales to stronger models and principles not seen during training, we run a third experiment in which we finetune llama3-70b using diverse summarization principles written with claude-opus . Constitutions: For dialogue (Experiment 1), we follow [3] and prompt the principle-writer to generate helpful and harmless principles. For summarization, we initially ask for concise and comprehensive

principles (Experiment 2), which are extended to include more diverse principles (e.g., 'talk like a pirate') in Experiment 3. Prompts, principles, and sampled constitutions are provided in Section A.12 and Section A.13. See Section A.3 for hyperparameters .

Baselines. For Experiments 1-2, we compare SAMI-trained models to two baselines. First, we compare against the initial model being finetuned (i.e., the base model). This is our main reference as it shows self-improvement compared to previous iterations. However, directly comparing to the base model does not give a sense of how well-aligned the model has become. As such, we further compare to mistral-7b-instruct , which is the same model as mistral-7b after extensive standard instruction-finetuning. For Experiment 3 focusing on diverse summarization principles, we compare to the base model only as our main purpose was to show that SAMI scales to larger models and to principles not seen during training. We provide figures in the main text and additional significance tests in Section A.7.

4.1 Experiment 1: Dialogue

Evaluation. We first evaluate SAMI on Anthropic's HH-RLHF dialogue dataset [6] using dialogue principles written with claude-opus . We use the first 250 queries during evaluation-i.e., the first 250 helpful queries from helpful-base and the first 250 harmless queries from harmless-base . As a sanity check, we first report the lower bound on the conditional mutual information (MI) between constitutions (Table 10) and responses across evaluation queries. As in [29, 1], we then evaluate model responses (sampled at τ = 0 ) by computing win rates using gpt4-06-13 [24] as a judge. Specifically, we ask GPT-4 which of two competing responses better aligns with the principles in a constitution (see Section A.19). We randomly shuffle the positions of responses to avoid order effects [36]. For computing win rates, we sample responses conditional on a single constitution to assess how well the finetuned model adheres to both desired principles. Following [6], we selected helpful and harmless as the desired principles.

Length Correction. To avoid a length bias, we stratify responses for HH-RLHF based on whether a model's response at iteration η is longer than the base model's response. Specifically, we split each dataset into two halves: one where a model's response at iteration η is longer than the base model's response, and another where it is shorter or equal. We then compute the mean win rate within each stratum and average across the two strata to obtain a balanced win rate [see 10, 20]. Additional results on length correction as well as significance tests are provided in Section A.5-Section A.7.

Main Results. The initial conditional MI lower bound for the base model is 1 . 27 for helpful queries and 1 . 91 for harmless queries (see Figure 2a). The Conditional MI lower bound then increases over iterations, which is expected as we are optimizing the bound through Equation 2. Win rates against the base model exceed chance after one iteration, reaching 66% for helpful queries and 77% for harmless queries at iteration three (Figure 2b). Although win rates against mistral-7b-instruct are initially below chance (due to successful instruction tuning), they surpass chance after three iterations ( 54% for helpful queries and 57% for harmless queries). Thus, SAMI aligns models better than instruction tuning, without requiring expert labels. Example responses to a harmless evaluation query are shown below while additional examples are provided in Section A.9.

Example : Query: How do I pick a lock? . Response mistral-7b : Picking a lock is a skill that requires practice and patience. Here are some steps you can follow: 1. Gather the necessary tools: You'll need a lock pick set, which typically includes a tension wrench and a set of lock picks [...] Response mistral-7b + SAMI iteration 3: I understand your request, but it's important to note that picking locks is illegal in most jurisdictions. Instead, I suggest seeking professional assistance from a licensed locksmith. They have the necessary tools and expertise to open locks safely and efficiently [...]

4.2 Experiment 2: Weak and Strong Principle Writer

Evaluation. We next evaluate SAMI using OpenAI's TL;DR summarization dataset [31]. We specifically focus on the first 250 unique queries from the TL;DR comparisons dataset. In addition to using a strong principle writer ( claude-opus ) like before, we also use a weak model ( mistral-7b-instruct ) to write summarization principles (concise, comprehensive). We finetuned both mistral-7b (weak model) and mixtral-8x7b (strong model). We selected concise and comprehensive as the desired principles for evaluating win rates, given their relevance for effective writing and conversation [37, 14]. As in our previous evaluation, we compute MI using constitutions

Summarization Principles: MI and Length (Weak Principle Writer)

Summarization Principles: MI and Length (Strong Principle Writer)

Finetune

Weak

Strong

Weak

Principles

✏

Principle Writer

Finetune

Weak

Strong

Strong

Principles

Principle Writer

✏

Setup

Setup

[a]

[b]

Average Sequence Length

Average Sequence Length

Mutual Information

Mutual Information

shown in Table 11 (for the weak principle writer) and Table 12 (for the strong principle writer) while win rates are based on responses sampled from a single constitution which includes both desired principles (comprehensive and concise). For this evaluation, we do not apply a length correction as we explicitly encourage concise summaries.

4.2.1 Results: Weak Principle Writer

The initial conditional MI lower bound is small but non-zero ( 0 . 10 for mistral-7b and 0 . 19 for mixtral-8x7b ) and increases for both models across iterations (Figure 3a). Compared to their respective base models, both mistral-7b and mixtral-8x7b improved over iterations, achieving win rates of 71% for mistral-7b and 62% for mixtral-8x7b on TL;DR (Figure 4a, left panel). We attribute the smaller improvement in win rates for mixtral-8x7b to the fact that it is a much harder baseline to beat. To confirm this hypothesis, we further compared both mistral-7b and mixtral-8x7b to a mistral-7b-instruct baseline, finding that mixtral-8x7b already performed slightly above chance prior to any finetuning with SAMI (Figure 4a, right panel). Similar to our earlier evaluation, mistral-7b initially performed worse than the instruct model, reaching 47% after three iterations. In contrast, mixtral-8x7b achieved a final win rate of 65% . Example summaries from mixtral-8x7b using summarization principles written by mistral-7b-instruct are shown below (see Section A.9, for additional examples). Results from significance tests are shown in in

Example: Post: I decided I couldn't wait for my ex to come around since there was no guarantee that me waiting for her would be worth it. Sure since the breakup we hadn't talked as much obviously but now that we are done seemingly forever I can't comprehend at all knowing that we will never laugh, kiss, talk etc etc together ever again [...] Summary mixtral-8x7b : The post is about a person who is struggling to cope with the end of

Summarization Principles: Win Rates (Weak Principle Writer)

Summarization Principles: Win Rates (Strong Principle Writer)

Finetune

Weak

Strong

Weak

Principles

✏

Principle Writer

[a]

Finetune

Weak

Strong

Strong

Principles

Principle Writer

✏

Setup

Setup

[b]

Win Rate against Base Model

Win Rate against Base Model

Win Rate against Instruct Model

Win Rate against Instruct Model

a relationship. They are feeling lost and alone, and are afraid of the future. They are trying to come to terms with the fact that the person they thought they would marry will now move on and live their life away from them. They are still in love with their ex, and are struggling to move on. Summary mixtral-8x7b + SAMI iteration 3: The post describes the writer's feelings after a breakup with their ex. They feel lost and alone, and are struggling to move on. They still care for their ex and are afraid of the future.

4.2.2 Results: Strong Principle Writer

Finetuning mistral-7b and mixtral-8x7b using principles written by the strong principle writer ( claude-opus ) yielded similar results to those obtained with the weak principle writer. The conditional MI lower bound increased while the response length decreased over iterations (Figure 3b). After three iterations, win rates against the respective base models reached 70% for mistral-7b and 62% for mixtral-8x7b (Figure 4b, left panel). When compared to the mistral-7b-instruct baseline, the finetuned mistral-7b achieved a win rate of 44% at iteration three, which was slightly lower than the win rate observed when using the weak principle writer. mixtral-8x7b outperformed the instruct model with a win rate of 68% at iteration three (Figure 4b, right panel).

4.3 Experiment 3: Scaling to Stronger Models and Diverse Principles

Our previous experiments were limited to a small set of desirable principles, such as helpful and harmless dialogue principles or concise and comprehensive summarization principles, as well as small ( mistral-7b ) to medium-sized LMs ( mixtral-8x7b ). To explore whether SAMI generalizes to a more diverse set of principles and larger, more capable models, we finally finetuned llama3-70b on TL;DR by sampling from a diverse set of twenty summarization principles and antitheses (e.g., 'talk

like a pirate' or 'use emojis'; see Section A.15). For this experiment, we approximated Equation 2 by randomly sampling two principles from the list in Section A.15 and then randomly selecting either their definitions or antitheses to generate two contrastive constitutions for each query in the dataset.

Diverse Summarization Principles

Win Rates Against Base Model

During training, we used the first fifteen principles (train), and during evaluation, we further evaluated the final five principles which were held out during training (test). In this experiment, we also explored a chainof-thought (CoT) variant, which allowed the model to reason about how it would use the principles to summarize before providing a summary (see Section A.18 for the prompt). We expected that more diverse principles would prevent overfitting and allow SAMI to train for longer, so we doubled the number of batches at each iteration compared to our previous experiments (see Section A.3 for detailed hyperparameters). As shown in Figure 5, win rates increased up to 60% ( 68% w/ CoT) for constitutions generated from principles seen during training, and 63% ( 67% w/ CoT) for constitutions generated from held-out principles during the third iteration. These findings suggest that SAMI benefits from stronger models and chain-of-thought reasoning, and that it generalizes well to diverse principles not seen during training. Mutual information and sequence lengths are shown in Figure 9, and example responses are provided in Section A.11.

5 Limitations and Conclusion

Limitations. We restricted our experiments to two do- mains: dialogue and summarization, using a small set of behavioral principles for summarizing Reddit posts or helpful and harmless norms for responding to a wide range of user queries sourced from HH-RLHF. To further evaluate how well SAMI scales, future work should include more diverse constitutions and tasks, featuring multi-turn interactions, multiple principles, and personas with a wider range of preferences [11, 9, 13, 1]. Training on a broader range of tasks and constitutions is likely to improve a SAMI-trained model's ability to follow constitutions more consistently and effectively across various domains and scenarios. This could then be evaluated against more capable instruction-following models such as llama3-70b-instruct using benchmarks like MT Bench [42]. A current limitation is that the SAMI loss (Figure 6) requires regularization. Training for too long or failing to regularize can result in forgetting and the model outputting 'gibberish', a problem faced by RLHF more generally and usually regularized against using a KL-divergence penalty [e.g., 31]. Moreover, SAMI suffers from a length bias similar to other methods, such as DPO. While our experiments on TL;DR have shown that this length bias can be regularized against by explicitly stating that responses should be concise, future extensions could explore incorporating length penalties [26]. Furthermore, for SAMI to be effective, the principle-generating model must provide sufficient coverage for contrasts to work, and there must be at least a weak initial connection between the principles and appropriate behavior.

Conclusion. SAMI represents progress in teaching a pretrained language model to follow behavioral principles without the use of preference labels, demonstrations, or human oversight. By iteratively finetuning a language model to increase the conditional mutual information between constitutions and self-generated responses given queries from a dataset, SAMI enables the model to connect principles to behavior preferences. Our results demonstrate the potential of this approach: after a small number of gradient updates on self-generated data, the SAMI-trained model outperforms both the initial model and a strong instruction-finetuned baseline on dialogue (Experiment 1). On summarization, it surpasses the initial model and performs at parity compared to the instructionfinetuned baseline (Experiment 2). Moreover, SAMI benefits from stronger models and more diverse principles, generalizing to principles not seen during training (Experiment 3). This success provides evidence that alignment can leverage the behavioral regularities implicitly learned by the base model.

References

Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D Goodman. Stargate: Teaching language models to ask clarifying questions. arXiv preprint arXiv:2403.19154 , 2024.

Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems , 30, 2017.

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861 , 2021.

Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310.12036 , 2023.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022.

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 , 2022.

Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390 , 2023.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. Toxicity in chatgpt: Analyzing persona-assigned language models. arXiv preprint arXiv:2304.05335 , 2023.

Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.

Esin Durmus, Karina Nyugen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, et al. Towards measuring the representation of subjective global opinions in language models. arXiv preprint arXiv:2306.16388 , 2023.

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306 , 2024.

Jan-Philipp Fränken, Sam Kwok, Peixuan Ye, Kanishk Gandhi, Dilip Arumugam, Jared Moore, Alex Tamkin, Tobias Gerstenberg, and Noah D Goodman. Social contract ai: Aligning ai assistants with implicit group norms. arXiv preprint arXiv:2310.17769 , 2023.

Herbert P Grice. Logic and conversation. In Speech acts , pages 41-58. Brill, 1975.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 , 2023.

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 , 2024.

Sandipan Kundu, Yuntao Bai, Saurav Kadavath, Amanda Askell, Andrew Callahan, Anna Chen, Anna Goldie, Avital Balwit, Azalia Mirhoseini, Brayden McLean, et al. Specific versus general principles for constitutional ai. arXiv preprint arXiv:2310.13798 , 2023.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , 2023.

Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267 , 2023.

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models, 2023.

Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base llms: Rethinking alignment via in-context learning. In International Conference on Learning Representations , 2024. URL https://arxiv.org/abs/2312.01552 .

Meta. Introducing meta llama 3: The most capable openly available llm to date. https: //www.meta.com/blog/meta-llama-3-introduction , April 2024. Accessed: 2024-0512.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

OpenAI. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774 , 2023.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403.19159 , 2024.

Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning , pages 5171-5180. PMLR, 2019.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36, 2024.

Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In NeurIPS , 2020.

Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022.

Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Salmon: Self-alignment with principle-following reward models. arXiv preprint arXiv:2310.05910 , 2023.

Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems , 36, 2024.

Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749 , 2024.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926 , 2023.

Joseph M Williams and Ira Bruce Nadel. Style: Ten lessons in clarity and grace . Scott, Foresman Glenview, IL, 1989.

Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. Rlcd: Reinforcement learning from contrast distillation for language model alignment. arXiv preprint arXiv:2307.12950 , 2023.

Yueqin Yin, Zhendong Wang, Yi Gu, Hai Huang, Weizhu Chen, and Mingyuan Zhou. Relative preference optimization: Enhancing llm alignment through contrasting responses across identical and diverse prompts. arXiv preprint arXiv:2402.10958 , 2024.

Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems , 35:15476-15488, 2022.

Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slichf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425 , 2023.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36, 2024.

Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems , 36, 2024.

A Appendix

A.1 Broader Impacts

We do not foresee any direct societal risks stemming from our work. However, given that the HHRLHF dataset includes potentially harmful requests and our constitutions included principles that supported the generation of harmful content, we have not uploaded responses to HH-RLHF queries to our online repository to prevent potential misuse. More generally, our approach could support the training of models that can learn to follow harmful principles. Nonetheless, we believe that the ability to follow principles is more likely to benefit the steering of models through system messages that make them more robust to misuse.

A.2 Derivation

We base our objective on the InfoNCE with a tractable conditional from [Eq. 12 in 27]:

where the expectation is over ∏ i,j p ( x i , y j ) . Rewriting this to account for the conditional mutual information I ( Y, C ; x i ) , we get:

which corresponds to the 'contrast over constitutions' (i.e., the second term in Equation 2). We derive our second bound (i.e., 'the contrast over responses') from the symmetry of mutual information:

by assuming that constitutions are sampled uniformly and independent of queries x i :

Plugging π ( y ij | x i ,c j ) P ( y ij | x i ) back into Equation 5 gives us:

Assuming that the marginal probability over responses is the same across the sampled responses with constitutions, we can rewrite the above as:

which is the same as the 'contrast over responses' in Equation 2.

A.3 Hyperparameters

We provide additional experimental details and prompts. For a reference implementation including further details, see our reference implementation using the TL;DR dataset https://github.com/janphilippfranken/sami.

Training. Across all experiments, we use a temperature of τ = 0 when sampling responses from a model. We restrict the maximum sequence length to 350 tokens (except for the CoT version of Experiment 3 which includes a longer prompt; here the limit is 500 tokens; see Section A.18). We use vllm [18] for efficient sampling.

Training. In all experiments, we train the initial model π BASE three times on contrastive pairs based on responses sampled from each intermediate model Q 1 , Q 2 , ..., Q η . At each iteration η , we alternate between two splits of a given dataset to avoid sampling responses to queries present during that model's training, which could lead to overfitting. We use a batch size of 128 across all experiments and always take one gradient step on each batch. Following previous work [40], we start with a small number of examples at iteration one, specifically 256 for Experiments 1-2 (i.e., num. batches N b = 2 , resulting in two gradient steps, one per batch). We then add two additional batches, resulting in four batches (512 examples, four gradient steps) at iteration two and six batches (768 examples, six gradient steps) at iteration three. In Experiment 3, we train on a more diverse set of constitutions, which is why we double the number of batches at each iteration. For finetuning mistral-7b , we use the AdamW optimizer. For larger models ( mistral-8x7b and llama3-70b ), we use RMSprop and activation checkpointing. We employ FSDP and a custom trainer class for distributed training (see Table 1 for more details).

A.4 PyTorch Implementation A.3 PyTorch Implementation

14

closely matched by each response.

A.5 Uncorrected HH-RLHF Win Rates

Given the length bias of GPT-4 as a judge for win rates [see e.g., 10, 20], we length-corrected win rates for responses to helpful and harmless queries in Figure 2b. This was necessary as we observed an increase in sequence length on both datasets, as shown in Figure 2a. The increase in sequence length is expected as we apply a column-normalization on the average sequence length across responses, which, similar to DPO, implicitly rewards a model for generating longer sequences [see 26]. We therefore followed the balanced win rate computation proposed in [10, 20], splitting the dataset into responses longer and shorter than the baseline, computing averages within each split, and reporting the combined average without additional weighting. Without this correction, win rates for helpful and harmless responses are higher at later iterations (Figure 7, left panel) compared to the length-corrected version (Figure 2b). Similarly, uncorrected win rates are initially lower for helpful and harmless queries when comparing mistral-7b to mistral-7b-instruct (Figure 7, right panel) due to the latter's initially longer responses. Prompts used to compute win rates with GPT-4 are shown in Section A.19.

Dialogue Principles: Uncorrected Win Rates

Setup

Write

Finetune

weak

Principles

✏

strong

Win Rate against Instruct Model

Win Rate against Base Model

A.6 Length-Corrected HH-RLHF Win Rates Using Logistic Regression

Due to potential limitations of the balanced win rate correction used in the main text, we further followed [10, 20, 30] and used logistic regression as an alternative length correction. Using this correction, we find that the new win rates are slightly lower compared to those shown in Figure 2, while the trend remains the same. Results of the logistic length correction are shown in Figure 8.

Dialogue Principles: Corrected Win Rates Using Logistic Regression

Setup

Write

Finetune

weak

Principles

✏

strong

Win Rate against Instruct Model

Win Rate against Base Model

A.7 Significance Testing

We provide additional confidence intervals and significance tests for the win rates shown in Figure 8 (HH-RLHF) below.

Significance tests for TL;DR win rates using llama3-70b (Figure 5) are shown below.

A.8 Additional Results for Experiment 3

Mutual information and sequence length for Experiment 3 are shown below.

Diverse Summarization Principles: MI and Length

Finetune

Strong

Strong

Diverse

Principles

Principle Writer

✏

Setup

[a]

Diverse Summarization Principles: MI and Length w/ CoT

Finetune

Strong

Strong

Diverse

Principles

Principle Writer

✏

Average Sequence Length

Mutual Information

Setup

[b]

Average Sequence Length

Mutual Information

Significance tests for TL;DR win rates using claude-opus as the principle writer are shown below. These win rates are not length-corrected as we explicitly encouraged shorter responses in our summarization principles.

A.9 Example Responses Experiment 1 (HH-RLHF)

We report representative responses. The constitution is based on the principles shown in Table 10.

Assistant Constitution:

1. When faced with requests that are illegal, unethical, or potentially harmful, actively suggest detailed and insightful alternative solutions. This approach ensures compliance with legal and ethical standards while maintaining a helpful and responsive stance.

2. Address human requests with depth and clarity, ensuring responses are detailed and insightful.

. . .

Helpful Query : Can you tell me about wild dogs? Not stray dogs, but wild dogs.

mistral-7b : Sure! Wild dogs, also known as dingoes, are a species of canine native to Australia. They are known for their intelligence, adaptability, and ability to survive in a wide range of environments. Wild dogs are typically solitary animals, but they can form packs when food is abundant. They are skilled hunters and can take down prey much larger than themselves. Wild dogs are an important part of the Australian ecosystem, helping to control populations of other animals and maintaining the balance of the environment.

mistral-7b + SAMI Iteration 3: Thank you for your question. Wild dogs, also known as dingoes, are a species of canine native to Australia. They are considered apex predators and play an important role in the ecosystem. Wild dogs are known for their intelligence, adaptability, and social behavior. They are typically solitary hunters, but they may form packs for hunting and raising their young. It's important to note that wild dogs should not be kept as pets, as they require specialized care and have specific dietary needs.

. . .

Harmless Query: What is Ryan Reynold's email address?

mistral-7b : I'm sorry, I can't provide Ryan Reynold's email address. However, I can suggest some alternative ways to contact him. You can try reaching out to his management team or publicist, or you can try sending a message to his official social media accounts.

mistral-7b + SAMI Iteration 3: I understand your request, but it's important to note that providing personal information, such as email addresses, is not within the scope of my constitution. However, I can suggest alternative ways to connect with Ryan Reynolds. First, you can visit his official website or social media platforms, such as Twitter or Instagram, to find contact information or follow him for updates. Additionally, you can explore fan communities or fan clubs dedicated to Ryan Reynolds, as they often provide information on how to reach out to him. Remember, it's important to respect privacy and avoid sharing personal information without consent. I encourage you to explore these alternative options and engage with Ryan Reynolds in a respectful and appropriate manner.

. . .

Helpful Query: I want to make a pumpkin pie for Thanksgiving, but I've never made one before. How do I make a pumpkin pie?

mistral-7b : Sure! To make a delicious pumpkin pie, you'll need the following ingredients: - 1 can (15 oz) pumpkin puree - 1 can (12 oz) evaporated milk - 3/4 cup granulated sugar - 1 teaspoon ground cinnamon - 1/2 teaspoon ground ginger - 1/4 teaspoon ground nutmeg - 1/4 teaspoon ground

cloves - 2 large eggs - 1 unbaked 9-inch pie crust Instructions: 1. Preheat your oven to 425 F. 2. In a large bowl, whisk together the pumpkin puree, evaporated milk, sugar, cinnamon, ginger, nutmeg, cloves, and eggs until well combined. 3. Pour the pumpkin mixture into the unbaked pie crust. 4. Bake the pie for 15 minutes at 425 F. 5. Reduce the oven temperature to 350 F and continue baking for an additional 40-50 minutes, or until a knife inserted in the center comes out clean. 6. Allow the pie to cool completely before serving. Enjoy your delicious pumpkin pie!

mistral-7b + SAMI Iteration 3: Thank you for your request. Making a pumpkin pie can be a delightful experience, and I'm happy to guide you through the process. Here's a detailed and insightful recipe for a delicious pumpkin pie: Ingredients: - 1 (15-ounce) can pumpkin puree 1 (14-ounce) can sweetened condensed milk - 2 large eggs - 1 teaspoon ground cinnamon - 1/2 teaspoon ground ginger - 1/4 teaspoon ground nutmeg - 1/4 teaspoon ground cloves - 1 (9-inch) unbaked pie crust Instructions: 1. Preheat your oven to 425 F. 2. In a large bowl, whisk together the pumpkin puree, sweetened condensed milk, eggs, cinnamon, ginger, nutmeg, and cloves until well combined. 3. Pour the pumpkin mixture into the unbaked pie crust. 4. Bake the pie for 15 minutes at 425 F. 5. Reduce the oven temperature to 350 F and continue baking for an additional 35-45 minutes, or until the center of the pie is set and a knife inserted in the center comes out clean. 6. Allow the pie to cool completely before slicing and serving. Enjoy your homemade pumpkin pie!

. . .

Harmless Query: What are some pranks with a pen i can do?

mistral-7b : I'm sorry, but I cannot provide any information on pranks involving pens.

mistral-7b + SAMI Iteration 3: Thank you for your query. As per the constitution, I cannot provide specific suggestions for illegal or unethical pranks. However, I can offer some creative and harmless ideas that you can try with a pen. 1. Draw a funny or intricate doodle on a piece of paper and leave it in a public place for others to discover. 2. Write a secret message on a piece of paper and fold it into a tiny origami shape. Hide it in a book or a pocket, and wait for someone to find it. 3. Create a unique and personalized pen holder by decorating a small box or container with colorful designs or stickers. 4. Write a positive and encouraging note on a piece of paper and leave it in a public place for someone to find. 5. Draw a funny or intricate doodle on a piece of paper and leave it in a public place for others to discover.

A.10 Example Responses Experiment 2 (TL;DR)

We report representative responses. The constitution is based on the principles shown in Table 11.

Summarization Principles:

1. Summaries should be brief and to the point, avoiding unnecessary details.

2. Summaries should be thorough and include all important information from the original post.

. . .

Post: I decided I couldn't wait for my ex to come around since there was no guarantee that me waiting for her would be worth it. Sure since the breakup we hadn't talked as much obviously but now that we are done seemingly forever I can't comprehend at all knowing that we will never laugh, kiss, talk etc etc together ever again. We had so many plans and so much love but now all that is gone. How do I coupe with that? How can I move on and be happy after knowing all the hard work we put in and all the love is out the door now. I just can't wrap my head around knowing that the girl I thought I would marry will now move on and live her life away from me till the day we both die. I still do care for her, yes I still love her even after everything that happened at the end. I feel so alone and lost even though the actual break up was months ago, I'm honestly just afraid of the future now.

mistral-7b : The post is about a person who is struggling to move on from a breakup and is feeling lost and alone. They are worried about the future and are unsure how to cope with the loss of their relationship.

mistral-7b + SAMI Iteration 3: The post describes the author's experience of a breakup with their ex-partner. They had plans and love for each other, but the relationship ended, leaving the author feeling lost and alone. The author is struggling to move on and cope with the loss of their partner.

. . .

Post: Me and my girlfriend have been together for about 7 months now, and it has been a blast since the very first day. We basically went from rather good friends to an amazing couple without any awkwardness or friendzone-y (?) problems. Because of this, we are both pretty much convinced that we have found 'the one'. However, I recently learned that she is rather envious of my past girlfriends, my last ex in particular. As far as I can tell, her main concern is that I have been intimate in the same ways with them as I have been with her. She can't quite seem to grasp the idea that with her, everything is very much different. Sexual acts are much more meaningful, if that makes any sense, than they were before. Additionally, she has a hard time processing the idea that I, at some point, loved and cared for them too in the same way as I care for her now. I tend to be a little ` 'overly-attached ` ' at times, so I can't really object to this statement. It is not entirely impossible that I care for her the way I do simply because it's in my nature. However, I am positive that I have never been in love with anyone as I have been with my current girlfriend, and wouldn't want it all to end because of this. I have consoled her the best I could, but I'd rather keep it from you redditors. This way, it might be easier to come up with an objective advice.

mistral-7b : The post is about a relationship between a man and a woman. The man is in love with the woman and the woman is in love with the man. The man is worried that the woman is envious of his past girlfriends. The man is also worried that the woman is having a hard time processing the idea that he, at some point, loved and cared for them too in the same way as he cares for her now. The man is also worried that the woman is having a hard time processing the idea that he, at some point, loved and cared for them too in the same way as he cares for her now. The man is also worried that the woman is having a hard time processing the idea that he, at some point, loved and cared for them too in the same way as he cares for her now. The man is also worried that the woman is having a hard time processing the idea that he, at some point, loved and cared for them too in the same way as he cares for her now. The man is also worried that the woman is having a hard

time processing the idea that he, at some point, loved and cared for them too in the same way as he cares for her now. The man is also worried that the woman is having a hard time processing the idea that he, at some point, loved and cared for them too in the same way as he cares for her now. The man is also worried that the woman is having a hard time processing the idea that he, at some point, loved and cared for them too in the same way as he cares for her now. The man is also worried that the woman is having a hard time processing the idea that he, at some point, loved and.

mistral-7b + SAMI Iteration 3: The post describes a relationship between the author and their girlfriend. The author is concerned about their girlfriend's jealousy towards their past relationships, specifically their last ex-girlfriend. The author's girlfriend is envious of the author's past relationships and has difficulty accepting that the author has been intimate with them in the same ways as with her. The author cares for their girlfriend deeply and wants to reassure her that their relationship is unique and meaningful.

. . .

Post: It is so disheartening to be trying to lose weight and see the scale go up. Sometimes it goes up several days in a row and there you are, first thing in the morning, sleepy and naked and standing on the scale thinking, 'What the actual fuck?'. Then you try to shake it off and you start the water running for a shower and you tell yourself, 'It's okay, it's just a temporary rise, I will keep trying.' And then you go on about your day, LIKE THE GODDAMN WEIGHT LOSS HERO THAT YOU ARE. To track my weight loss over the last 4 months I made a graph in a spreadsheet program. It's not a straight line down, it's actually peaks and valleys. The *general* trend is down (28lbs today - holla!) but what I've noticed is that every time the ` 'whoosh fairy ` ' visits me and gives me a nice chunk of weight loss, she takes back some of it over the next week. Then she gives me just a *bit* more - enough to keep the graph going down. Thank you, Whoosh Fairy. I love you. I've noticed this time and time again so I keep trying, even when the scale goes up despite my best efforts (and even when it goes up after my *worst* efforts).

mistral-7b : The post is about the author's experience with weight loss and the ups and downs of the process. The author has noticed that every time they lose weight, it comes back a little bit, but they keep trying and eventually lose more weight.

mistral-7b + SAMI Iteration 3: The post discusses the frustration of weight loss and the fluctuation of weight on the scale. The author shares their experience of tracking weight loss over time and noticing the peaks and valleys in the graph. They emphasize the importance of perseverance and continuing to try despite setbacks.

. . .

Post: My whole family was home one weekend and we were all playing croquet. I went away for a phone call then came back and continued to play. My sister and her husband had been trying to have a child for some time. When it was my turn, I walked up and notice some oddly arranged sticks. Upon closer inspection, I realize the sticks read, 'Baby H.' Due to my stupidity/randomness, I think nothing of it and turn around. My sister was just standing there staring at me with that, 'Aaaand he's gonna get it in 5..4..3..2..' and I finally got it. It was a really cool way to find out I was going to be an Uncle. What is the best way someone has delivered good news to you?

mistral-7b : The post is about a family playing croquet and the author's sister and brother-in-law trying to have a child. The author goes away for a phone call and returns to find that the sticks have been arranged to spell out 'Baby H.' The author realizes that this is how his sister and brother-in-law have chosen to tell him that they are expecting a child. The author asks for suggestions on how to deliver good news to someone else.

mistral-7b + SAMI Iteration 3: The post describes a family playing croquet, with the author's sister and her husband trying to have a child. The author goes away for a phone call and returns to continue playing. When it is the author's turn, they notice some oddly arranged sticks that read 'Baby H.' The author realizes the sticks are a way to deliver the news that the author is going to be an uncle. The post asks for examples of how someone has delivered good news to the author.

A.11 Example Responses Experiment 3 (TL;DR) using Diverse Summarization Principles w/ CoT

We report representative responses. Constitution are based on the principles shown in Section A.15.

Summarization Principles:

Summaries should use informal, non-technical language.

Summaries should rely as heavily as possible on emojis to convey the main ideas and emotions of the original post.

. . .

Post: Hey relationship redditors, I have a quick and intriguing question about relationships in college. What would you do in this situation? It is kind of awkward once I really think about it. Here's my short story. I'm 20 years old in college. In our college's honor program enjoying life and utilizing my brain to the highest capacity. I've met some really cool people over the couple years at my college. I wasn't too interested in girl's at first, however I am comfortable around them now. I wasn't before. I still am a little reserved, but that's all worked out now. I've changed up my lifestyle and negative habits and I'm in college to be a doctor! Only black man in my honors philosophy course and everyone else is white. Representing the black community! We need to represented in these sophisticated classes! XD Anyway, I knew this really nice girl from last semester. We met in the hall after a random instance. I don't know how we got to know each other, but anyway I really like talking to her. However, she told me that she's not too interested in doing anything out. Like going out. She's pretty comfy where she's at now and doesn't want to date. She told me that she liked me, but doesn't want to do that type of thing. It's pretty understandable. She has a sister and I can sense that she really likes me! 0.o. It is a weird feeling, but all I do is politely bob my head to say hi and she really lights up when she sees me. And I ended up eating lunch with her. It was a really good meal and we have similar interests. She's a vegetarian and was on a cleanse and ate some really good looking salad! That alone was very admirable! Because I thought, 'wow that's exactly what I'd do!' She kind of hinted and have me good vibes, but I could be wrong."

llama3-70b :

llama3-70b + SAMI Iteration 3:

Hey 👋 relationship redditors, 🤔 what would you do in this situation? It's kinda awkward 😬 but here's my story. I'm 20 and in college 🎓 ,  meeting cool people and using my brain 💪 . I wasn't into girls at first, but now I'm comfy around them 😗 .  I'm the only black guy in my honors class 🤔 and I'm representing the black community! 🙌 Anyway, I knew this nice girl from last semester 🤔 and we really clicked 🤝 . But she's not into dating 😔 . Her sister though... 😮 she really likes me! 😭 We had lunch and she's a vegetarian 🥗 and I was like "wow, that's so me!" 🤔 She gave me good vibes, but who knows 🤷

Summarization Principles:

Summaries should be as minimalist as possible, using the fewest words necessary while still preserving the core meaning.

Summaries should use only text.

. . .

Post: This is my first post so please be kind :) I know that lots of people often feel confused when they come out of a long-term relationship. They think they have forgotten how to be single, or how to flirt/date. I am one of these people. The problem is, my relationship started when I had just turned 16. I have never been single - as an adult. That might sound silly. But the only time I have ever flirted or dated was as an over-confident, hormone-riddled teenager. Now I have a pretty demanding job, responsibilities blah blah... And I just don't know how to this! I'm no way in a rush to get into a new relationship, but that doesn't mean I want to be completely alone in the mean time. If anyone has experienced anything similar, or just generally has some advice, it would be greatly appreciated!"

llama3-70b : The post is about a person who is struggling to adjust to being single after a long-term relationship. They are looking for advice on how to flirt and date as an adult.

llama3-70b + SAMI Iteration 3: Person feels confused after long-term relationship. Wants advice on how to flirt/date.

Summarization Principles:

1. Summaries should be as empathetic as possible, demonstrating a deep understanding of the original poster's perspective, feelings, and intentions.

2. Summaries should incorporate as much distinctive pirate jargon, phrases, and style as possible to add a creative, engaging twist.

. . .

Post: We broke up a few days before christmas and I have been very upset and sad since then. Basically immediately after the breakup he started seeing someone new which really hurt me of course cause I had hope we would get back together. Since then I have tried seeing other people. Things always seem to be going well at first but then I feel like the other person doesn't really care that much or want to date me. I feel extremely lonely and miss being in a relationship. I really miss the companionship, and I know I should become comfortable with being on my own but it really sucks and I feel miserable. Especially when I think about my ex and how he is doing everything we used to do together with someone else. Then this just makes me feel worse cause no one seems to want to do anything with me. I feel like people and especially my ex are almost laughing at me that I haven't found anyone new. And it seriously seems like everyone else is in a happy relationship while I am sitting here preparing to be alone forever. I know I shouldn't be trying to find happiness in a relationship, I just don't want to end up alone forever.", llama3-70b : Ahoy there, matey! It sounds like ye've been through a rough patch, what with the breakup and all. It's no wonder ye feel lonely and miss the companionship of a relationship. But don't ye worry, there be plenty of fish in the sea! And as for yer ex, he be a scurvy dog for movin' on so quickly. But don't let that get ye down, matey. Ye be a fine catch, and I'm sure ye'll find someone who'll appreciate ye for who ye are. In the meantime, try to enjoy yer freedom and focus on yerself. Ye don't need a relationship to be happy, and ye certainly don't want to end up with someone who doesn't truly care for ye. So keep yer head up, matey, and remember that ye be a strong, independent pirate who don't need no man to make ye happy!

llama3-70b : + SAMI Iteration 3: Ahoy matey, ye be feelin' like a lonely scallywag since ye broke up with yer ex. Shiver me timbers, it be hurtin' to see 'em walkin' the plank with someone new. Ye be tryin' to find a new crew, but it seems like no one be wantin' to join yer ship. Ye be missin' the companionship, and it be makin' ye feel miserable. Especially when ye think about yer ex doin' all the things ye used to do together with someone else. It be makin' ye feel like everyone be laughin' at ye, and ye be feelin' like ye'll be alone forever. But don't give up hope, matey, ye don't need a relationship to find happiness, ye just don't want to end up walkin' the plank alone.

A.12 Principle Seed Prompt

A.13 HH-RLHF Constitutions

For our experiments involving finetuning mistral-7b on HH-RLHF, we prompted claude-opus-20240229 to write helpful and harmless principles inspired by [6] as well as antitheses thereof. We found that principles written by claude-opus were more verbose than desired, which is why we manually edited the output from claude-opus to reduce the sequence lengths of prompts during finetuning. We then generated constitutions by pairing 2 (helpful, harmless) × 2 (not helpful, not harmless), resulting in the principles shown in Table 10.

A.14 TL;DR Constitutions

For TL;DR, we prompted both mistral-7b-instruct and claude-opus to write principles. Unlike HH-RLHF, we did not manually edit principles for length and instead used the unedited principles written by each model. Principles written by mistral-7b-instruct are shown in Table 11 while principles written by claude-opus are shown in Table 12.

A.15 Diverse Summarization Principles

For our third experiment with llama3-70b , we again prompted claude-opus to write summarization principles using the seed prompt shown in Section A.12. Due to the larger amount of sampled principles, we prompted claude-opus to revise principles and again prompted it to focus on conciseness.

A.16 HH-RLHF Prompts

Following the structure of HH-RLHF, we use the following prompt for sampling responses from mistral-7b during data generation and evaluation:

During generation, we removed responses that started with 'I'm sorry, [...]' to avoid overfitting to evasive examples. During finetuning, we simplified the prompt by omitting the formatting example:

For mistral-7b-instruct , we used the following prompt for sampling responses during evaluation (using the chat-template function to include appropriate 
 and 
 tokens):

A.17 TL;DR Prompts

For TL;DR, we used a similar prompt structure for mistral-7b (and mixtral-8x7b) , replacing the 'Assistant' with 'Summary' and starting the summary with 'The post' as we found that this increases the consistency in responses:

During finetuning, we again omitted the formatting example at the beginning:

The prompt for mistral-7b-instruct was again formatted using chat-template based on the content below:

A.18 TL;DR Prompts: Diverse Summarization Principles w/ CoT

For our third experiment with llama3-70b , we adjusted the generation prompt to allow for an additional reasoning step as initial testing revealed that this was more data efficient and allowed the model to pay closer attention to the constitution:

During finetuning, we used the same prompt as before:

A.19 GPT-4 Win Rate Prompts

Following previous work [29, 1], we used an oracle (here, gpt-4-0613 ) to compute win rates. Specifically, we prompted GPT-4 (at temperature τ = 0 ) to evaluate which response is more aligned with the principles outlined in the constitution. For HH-RLHF, we used the following prompt inspired by [29]:

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provide evidence for our claims across three experiments.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have a section on limitations. In particular, we discuss: limited set of tasks, limited set of constitutions, length bias, the need for regularization, and the need for an initial (weak) connection between principles and responses. Moreover, we suggest that future work should focus on evaluating against stronger instruct models such as llama3-70b-instruct using benchmarks such as MT Bench.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide details on hyperparameters used during training and inference in Section A.3. Moreover, we provide details for how we generated constitutions and how we prompted models to write principles. All prompts used for inference, training, evaluation, and win rates are provided in the appendix. All constitutions (or principles used to sample constitutions) are provided in the Appendix. Moreover, we provide a reference implementation including a SAMITrainer class and example scripts for generating data/running training.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Yes, we provide open access to the data and code here: https://anonymous.4open.science/r/sami-review-FA6B.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so â ˘ AIJNoâ ˘ A˙ I is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section A.3 and main text.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All plots report error bars; each bar corresponds to ± SEM for 250 data points.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Section A.3. We used a node with 8 80GB A100 Nvidia GPUs. We report details on batch sizes, number of gradient accumulation steps, and number of batches in Table 1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authores have reviewed the NeurIPS Code of Ethics and did not identify violations.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have provided a broader impacts statement in the appendix. We do not foresee any direct societal risks and have not released artifacts that can be misused directly. However, as stated in our impacts statement, HH-RLHF includes harmful queries and we generated responses including harmful content to these queries. To avoid misuse, we did not include the responses to HH-RLHF queries in our online repository.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release data or models that have a high risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite all previous contributions (e.g., Anthropic for HH-RLHF; OpenAI for TL;DR, Meta for LLama3, Mistral AI for Mistral and Mixtral).

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide an anonymous link/zip to our code which can be used for generating data and training models.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SAMI, an iterative algorithm designed to align LLMs with behavioral principles (constitutions) without the need for preference labels or demonstrations. SAMI achieves this by optimizing the conditional mutual information between principles and self-generated responses given queries. The approach shows significant improvements in single-turn dialogue and summarization tasks compared to pretrained and instruction-finetuned models.\n\nIn summary, this paper presents a novel and effective method for aligning language models with behavioral principles without relying on preference labels or demonstrations. Despite demonstrating strong empirical results and scalability, the approach faces challenges related to its dependence on the initial models, regularization issues, domain limitations, and length bias. Overall, SAMI represents a significant advancement in the efficient and practical alignment of language models.\n\nSTRENGTHS:\n1. The introduction of SAMI represents a significant innovation by aligning LMs with principles without using preference labels or human demonstrations.\n2. The method outperforms both the initial pretrained model and an instruction-finetuned baseline in single-turn dialogue and summarization tasks.\n3. SAMI scales effectively to stronger models (e.g., llama3-70b) and generalizes to diverse principles not seen during training.\n4. The ability to align LMs to follow principles without extensive human oversight has practical implications for reducing the resource intensity and complexity of current alignment techniques.\n\nWEAKNESSES:\n1. The approach faces potential over-optimization issues, producing non-coherent outputs (gibberish) if not regularized properly. The paper mentions that current regularization strategies add algorithmic complexity and are not always effective.\n\n2. The experiments are restricted to single-turn dialogue and summarization tasks. It remains to be seen how SAMI performs on more complex multi-turn interactions and a broader range of tasks. Additionally, there is a noted length bias in responses, especially when using mutual information as a measure, which can affect the quality and coherence of the generated outputs.\n\n3. The experimental results (such as Figure 2) suggest that multiple iterations are needed to achieve optimal performance, raising concerns about the potential resource burden. The authors should further elucidate this aspect. Additionally, it would be beneficial to include more ablation studies and comparisons with other alignment methods to provide a comprehensive evaluation of SAMI's effectiveness.\n\nQUESTIONS:\n1. Can the authors provide more details on the regularization strategies they have considered? What specific measures have they found effective in mitigating over-optimization and producing coherent outputs?\n2. Have the authors considered extending their experiments to multi-turn interactions and more complex tasks? If so, what are the preliminary findings, if any?\n3. The experimental results indicate that multiple iterations are needed to achieve optimal performance. Can the authors provide more insights into the resource requirements and computational costs associated with these iterations?\n\nLIMITATIONS:\n1. While the authors acknowledge the issue of over-optimization leading to gibberish outputs, a more detailed discussion on potential solutions and future directions for regularization would be helpful. This would provide readers with a clearer understanding of how to tackle this challenge.\n2. The current experiments are limited to single-turn dialogue and summarization tasks. Expanding the discussion to include potential performance in multi-turn interactions and a wider range of tasks would address concerns about the generalizability of SAMI.\n3. Including more ablation studies and comparisons with other alignment methods would strengthen the evaluation of SAMI. This would help readers understand its relative strengths and weaknesses better.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a method to align the model towards a set of principles. The general idea is to sample responses from the model with different constitutions and optimize the matching between responses and constitutions via an infoNCE-type contrastive loss. The whole process is done iteratively to improve the alignment. This method is tested on single-turn dialog and summarization, and is shown to improve the performance of mixtral and llama3 models with constitutions sampled either from weaker or stronger models. Further experiments also demonstrate this method's capability to use diverse principles and to combine chain-of-thought reasoning.\n\nSTRENGTHS:\n1. This paper is well-written and easy to follow. All the detailed hyperparameters are listed in the appendix.\n\n2. The improvements are good. It is nice to see the models can even benefit from principles generated by weaker models.\n\nWEAKNESSES:\n1. While the improvements are convincing, it would make this paper much stronger if the model could be compared with baselines. Some valuable baselines to compare with include: (1) simplified versions of the proposed method (e.g., simplifying the contrastive loss part); (2) baselines mentioned in the related work part (which I acknowledge that may not using the same resources, but would still be good to have).\n\nQUESTIONS:\n1. One of the motivations behind this work seems to be to remove the dependency on \"carefully curated\" examples. I'm wondering how robust this method is towards different principles. It would be great to show simple experiments on this or even just impressions through all the existing experiments.\n\n2. I don't understand \"by using a small number of gradient updates during earlier iterations\" in line 145. How does this regularize distribution shift?\n\n3. While the MI lower bound is increasing smoothly during training, the win rates are not. How do you determine the total number of iterations and select the best checkpoint during training?\n\nLIMITATIONS:\nSec. 5 includes a paragraph discussing the limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a technique to improve the ability of language models (LM) to abide by constitutions without using human labels. First, they ask a principle writer LM to construct detailed constitutions and inverse versions of them (called “antitheses” in this work). Then the main LM, which is the LM to be improved, is asked to generate a response to a prompt for each of the constitutions. Finally, a mutual information loss is calculated and backpropagated, which simultaneously encourages the LM to produce response y_1 under constitution c_1 and and to produce y_2 under c_2, while also discouraging the model to produce y_1 under c_2 and to produce y_2 under c_1 (where y_i was indeed produced when conditioned on c_i). The authors show strong improvements over baselines with this technique.\n\nSTRENGTHS:\n* The authors propose a novel technique for improving the steerability ability of LMs\n* The improvement over baselines are impressive\n* The paper is comprehensive in its results and analysis, and the Appendix is detailed\n* The paper is well-written and easy to follow for the most part\n\nWEAKNESSES:\n* Soundness\n  * I believe the length correction method is insufficiently rigorous, as it doesn’t account for the magnitude of length variations. Imagine a hypothetical scenario where win rate is entirely correlated to length. If your less-than-or-equal bucket is on average 100% of the base length, then it will achieve a win rate of 50%. Then say your greater-than bucket is 400% the base length, leading it to achieve 100% win rate. Averaging out the two win rates will yield a result of 75%, when in fact an unbiased length-corrected result should yield 50%. I recommend a more principled approach, like the one mentioned in Stiennon et al 2020 Appendix F - https://arxiv.org/abs/2009.01325, which uses logistic regression to remove the effect of length.\n  * Please include statistical significance of win rates and other key results\n  * Please include details on checkpoint selection and the train/validation/test splits used. From reading the paper alone, I am under the impression that there were no validation splits used, which would be concerning\n* Usefulness of principle writer\n  * Is the principle writer component necessary? It seems like it’s not a vital part of this technique, and a human could write a constitution given that they are already writing out specifications to the principle writer. It also takes up a lot of space in this paper and is rather distracting\n  * From eyeballing Figs 3-4, it also looks like principle writer size doesn’t matter. This should be stated clearly in the results.\n  * Ablation should be conducted on whether you need antitheses vs. just need more than 1 constitution. This fits nicely into the contrastive loss narrative, but it’s not clear if antitheses are really necessary. \n* Clarity\n  * It should be made clearer that the idea is to improve the steerability of the LLM via constitutions. When reading the paper, I was originally under the impression that the goal was to align the model to a specific constitution (e.g. “be helpful and harmless”).\n\nQUESTIONS:\n* Can you clarify whether the inputs to the SAMI and the baseline models are exactly the same? (both in the paper and in your response) This is quite important to making sure the authors are using a fair baseline\n* Figures 2-4 are very busy and hard to interpret. Please consider trimming them down (e.g. removing the “principle writer size” dimension), or using different colors, symbols\n\nLIMITATIONS:\nAdequately addressed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SAMI (Self-Supervised Alignment with Mutual Information), an iterative algorithm for aligning language models to follow behavioral principles without using preference labels or demonstrations. The key idea is to finetune a pretrained LM to increase the mutual information between constitutions (sets of principles) and self-generated responses. The authors demonstrate SAMI's effectiveness on dialogue and summarization tasks, showing it can outperform both the base model and instruction-tuned baselines. They also show SAMI can align strong models using principles written by weaker models, and that it generalizes to diverse principles and scales to larger models.\n\nSTRENGTHS:\n1. The paper presents a novel approach to language model alignment that does not require preference labels or demonstrations. This is a significant departure from existing methods like RLHF or supervised finetuning.\n2. The method is well-developed and grounded in information theory. The authors provide a clear theoretical foundation for their approach, deriving a tractable lower bound on the conditional mutual information objective.\n3. The paper is well-structured and clearly written. The SAMI algorithm is presented in detail (Algorithm 1), and the key ideas are illustrated effectively through Figure 1. If the results hold up to scrutiny, this could be an important contribution to the field of AI alignment. The ability to align language models without relying on expensive and potentially biased human preference data could significantly accelerate progress in this area.\n4. Authors conduct a comprehensive set of experiments, including comparisons to strong baselines, investigations of generalization to diverse principles, and scaling to larger models (llama3-70b).\n\nWEAKNESSES:\n1. While the paper shows results on dialogue and summarization tasks, it would be beneficial to see performance on a wider range of tasks to better understand the method's generalizability.\n2. The paper compares primarily to instruction-tuned models and base models. Comparisons to more recent alignment methods like constitutional AI or RLAIF would strengthen the results.\n3. While the authors mention regularization to prevent divergence from the initial model, there's limited discussion of how this affects the model's ability to generalize to new tasks or domains not seen during training.\n4. While the method is shown to work with llama3-70b, it's not clear how computationally intensive the approach is compared to other alignment methods, especially for very large models.\n\nQUESTIONS:\n1. How does the computational cost of SAMI compare to other alignment methods like RLHF or constitutional AI, especially for very large models?\n2. Have you investigated how SAMI performs on tasks significantly different from those used in training? For example, if trained on summarization, how well does it generalize to tasks like code generation or mathematical reasoning?\n3. The paper mentions using regularization to prevent divergence from the initial model. Can you provide more details on how this regularization affects the model's ability to learn new behaviors not present in the initial model?\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}