Aligning Large Language Models with Human Preferences through Representation Engineering (2024)

Abstract: Aligning large language models (LLMs) with human preferences is crucial for enhancing their utility in terms of helpfulness, truthfulness, safety, harmlessness, and interestingness. Existing methods for achieving this alignment often involve employing reinforcement learning from human feedback (RLHF) to fine-tune LLMs based on human labels assessing the relative quality of model responses. Nevertheless, RLHF is susceptible to instability during finetuning and presents challenges in implementation. Drawing inspiration from the emerging field of representation engineering (RepE), this study aims to identify relevant representations for high-level human preferences embedded in patterns of activity within an LLM and achieve precise control of model behavior by transforming its representations. This novel approach, denoted as Representation Alignment from Human Feedback (RAHF), proves to be effective, computationally efficient, and easy to implement. Extensive experiments demonstrate the efficacy of RAHF in not only capturing but also manipulating representations to align with a broad spectrum of human preferences or values, rather than being confined to a singular concept or function (e.g. honesty or bias). RAHF’s versatility in accommodating diverse human preferences shows its potential for advancing LLM performance.

heading: 1 Introduction
text: While large language models (LLMs) learn broadranging world knowledge and a degree of reasoning proficiency, precise control over their behavior proves challenging due to the unsupervised nature of their pre-training (Radford et al., 2018, 2019; Brown et al., 2020; Bubeck et al., 2023; Touvron et al., 2023). For each query, instruction-tuned LLMs (Wei et al., 2021; Chung et al., 2022; Tou-
† These authors contributed equally. ∗ Corresponding author.
vron et al., 2023) exhibit the capacity to generate multiple responses that are both semantically and syntactically coherent by some sampling techniques. While such ability enables the models to provide diversity that is essential for chat agents, some responses may contain harmful, unethical, socially biased, and negative, even illegal content (Srivastava et al., 2022; Thoppilan et al., 2022; Bubeck et al., 2023; Wang et al., 2023).
Existing methods steer LLMs to align with human preferences often using reinforcement learning (RL), with reinforcement learning from human feedback (RLHF) emerging as the most successful one (Ouyang et al., 2022). However, the underlying learning algorithms exhibit a considerable degree of complexity, sensitivity to hyperparameters, instability during training, and necessitate additional training in the reward model and value network, leading to substantial computational costs (Yuan et al., 2023; Rafailov et al., 2023).
In addressing the aforementioned challenges posed by RL-based methods, several computationally lightweight alternatives have been proposed to simplify the human preference-matching process.
10619
Two prominent paradigms among these alternatives include contrastive learning (Rafailov et al., 2023; Zhao et al., 2023; Yuan et al., 2023) and Hindsight instruction relabeling (HIR) (Zhang et al., 2023; Liu et al., 2023). Contrastive learning-based methods optimize a language model policy by increasing the relative probability of preferred responses over dispreferred ones, while HIR methods transform human feedback into instructions by relabeling the original ones, indicating the relative quality of provided responses. A common characteristic shared by these two paradigms is their capability to align language models with human preferences through reward-free fine-tuning.
However, the reward-free fine-tuning is vulnerable to the presence of noisy data or incorrect labels in a training set comprising a collection of preference-annotated response pairs (Li et al., 2023b; Dumoulin et al., 2023). Instances of dull sentences or very brief responses may appear repeatedly in such a training set, potentially introducing bias into the models. The exclusion of such instances from the training set renders it impossible for LLMs to glean insights into human preferences expressed in these instances. In contrast, RL-based methods adopt a different strategy, wherein a reward function is first extracted from a dataset of response rankings, and then this reward function can be applied to train an LLM, effectively mitigating the model’s direct exposure to noisy data or incorrect labels within the dataset.
In this study, we aim to seek for a computationally lighter and reward-free algorithm that can effectively harness human preference expressed in datasets meanwhile safeguarding LLMs from the influence of noisy data. Inspired by the recent advance in representation engineering (Zou et al., 2023), we initially locate relevant representations and activity patterns associated with high-level human preferences within an LLM, and subsequently gain precise control over its behavior by manipulating its internal representations. In the neural architecture, network weights determine neural activity, neural activity determines the networks’ output, and the networks’ output determines the networks’ behavior. Instead of focusing on neurons and their connections, we see aligning LLMs with human feedback as an outcome of representational spaces, implemented by patterns of activity across populations of neurons. We first identify the differences in model activities between preferred and dispreferred stimuli, and then control its behavior
by leveraging the identified differences in representations (see Figure 1). We introduce two methods for controlling representations and demonstrate the efficacy of these representation engineering (RepE) approaches in aligning LLMs with a broad spectrum of human preferences through a collection of response pairs.
To validate the effectiveness of our approach in aligning with human preferences, we conducted extensive comparative experiments on the generated results. Our method outperformed RLHF and other RL-free approaches in human evaluations and automated metrics such as general abilities and GPT-4 evaluations. Notably, the underlying algorithms exhibit simplicity in implementation and straightforwardness in training.

heading: 2 Related Work
text: Tuning large language models to elicit desired responses and behavior from their extensive knowledge and capabilities is essential in the development of chat agents, such as ChatGPT (Brown et al., 2020), LLaMA (Touvron et al., 2023) and GPT-4 (Bubeck et al., 2023), characterized by safety, performance, and controllability. The enlargement of the size of language models only does not inherently enhance their ability to follow a user’s intent. For example, LLMs may still generate outputs that are untruthful, toxic, or simply not helpful to the user. Existing human preference alignment methods can be broadly classified into three major categories: reinforcement learning (Ouyang et al., 2022; Ramamurthy et al., 2023), contrastive learning (Rafailov et al., 2023; Zhao et al., 2023; Yuan et al., 2023), and Hindsight instruction relabeling (Zhang et al., 2023; Liu et al., 2023).
Extensive research has been devoted to the exploration of RL from human feedback through ratings or rankings, spanning tasks from NL-to-SQL conversion (Zhong et al., 2017), machine translation (Kreutzer et al., 2018), task-oriented dialogue systems (Su et al., 2019; Zhang et al., 2019; Takanobu et al., 2019), summarization (Stiennon et al., 2020), story-telling (Ziegler et al., 2019) to instructionfollowing (Ouyang et al., 2022; Ramamurthy et al., 2023). Typically, these methods involve the fitting of a reward model to a dataset of human preferences, followed by the optimization of a LLM policy to generate responses with high reward, using RL algorithms such as REINFORCE (Williams, 1992) or proximal policy optimization (Schulman
et al., 2017). Despite the attractiveness of leveraging human preferences that are easier to collect than expert demonstrations, training LLMs with RL poses significant practical challenges, which is attributed to the sensitivity of RL to hyperparameters and the inherent instability during training.
The solutions based on Hindsight instruction relabeling (Zhang et al., 2023; Liu et al., 2023) and contrastive learning (Rafailov et al., 2023; Zhao et al., 2023; Yuan et al., 2023) have emerged as computationally efficient alternatives to RL-based methods without explicit reward modeling. However, these reward-free fine-tuning solutions are susceptible to noisy data or incorrect labels within a training set. They exhibit performance lags compared to models tuned with RL counterparts (see Section 4). Furthermore, the question of whether LLMs trained with such fine-tuning methods can generalize well to out-of-distribution queries remains unresolved when contrasted with models incorporating an explicit reward model. RLHF method (Ouyang et al., 2022) offers a potential avenue for improvement by leveraging additional unlabeled examples through labeling LLM generations with the learned reward model.
To enhance transparency and controllability of neural networks, Zou et al. (2023) introduced representation engineering (RepE) as a methodology, drawing an analogy between understanding deep neural networks through representation tomography and studying brains via neuroimaging techniques. Their work demonstrated the efficacy of RepE in addressing diverse safety-related challenges such as truthfulness, honesty, and hallucination. This study falls in line with recent research findings and extends its application to aligning LLMs with a wide spectrum of human preferences. Our study introduces two novel methods to instruct LLMs on human preferences first, and then extract differences in model activities between preferred and dispreferred stimuli. These differences in activity patterns serve as a foundation for manipulating the model’s behavior, leading to the generation of responses that better align with human preferences. Due to the lightweight computational advantages of parameter-efficient fine-tuning techniques(Houlsby et al., 2019; Lester et al., 2021; Hu et al., 2021; Wu et al., 2023, 2024), these techniques are utilized to fit the disparity in activity patterns. In contrast to the approach adopted by Zou et al. (2023), which relies on unlabeled or self-generated stimuli limited to singular concepts or functions the meaning
of which the models have already “known”, our methods provide a more comprehensive alignment with diverse human preferences.

heading: 3 Method
text: We begin by instructing LLMs on human preferences with a set of preference-annotated response pairs. We introduce two novel methods for instructing LLMs on human preferences and extracting their activity patterns: one involving a single LLM (trained to discern the relative quality of responses) and the other employing dual LLMs (“a good guy and a bad guy”). Secondly, we collect the activity patterns of LLMs when exposed to stimuli that are preferred or dispreferred. The differences in these patterns serve as the foundation for manipulating LLMs, enabling them to generate responses more closely aligned with human values. Finally, we construct the final model by training a low-rank adapter(Hu et al., 2021) to fit the disparity in activity patterns.

heading: 3.1 Instructing LLMs on Human Preferences
text: To extract activity patterns from the model that align with human preferences, it is crucial for the model to possess a correct understanding and awareness of these preferences. The effectiveness of extracting activity patterns from alignment finetuned models, such as LLaMA-2-chat, in capturing concepts like truthfulness and honesty has been validated by Zou et al. (2023). However, for nonaligned models, such as pre-trained large language models or LLMs subjected to simple fine-tuning, explicit indications of human preferences should be provided to elicit and capture activity patterns induced by stimulus preferences. This capability enables the accumulation of diverse activities, subsequently utilized to calibrate LLMs based on human preferences.
For instructing LLMs on human preferences, we rely on a dataset annotated with human preferences. As mentioned earlier, we employ two methods to achieve this goal. The first method utilizes Hindsight(Zhang et al., 2023), using contrastive instructions to instruct a single LLM. The second method involves fine-tuning two LLMs separately: one (referred to as the preferred model) is fine-tuned based on preferred responses, while the other (referred to as the dispreferred model) is fine-tuned on dispreferred responses.

heading: 3.1.1 Preference Instruction with a Single Model
text: Within the proposed framework, the Single LLM Method focuses on fine-tuning a Single Large Language Model through Contrastive Instruction Tuning (SCIT). This process involves two instructions: one instructs the model to generate responses preferred by humans, while the other guides the model to generate responses dispreferred by humans. Following such fine-tuning, we can optimize the model for consistency with human preferences. We can also stimulate the model to elicit distinct activity patterns by employing different instructions subsequently.
Specifically, the training dataset is curated to include pairs of both preferred and dispreferred instructions, alongside associated queries and their corresponding responses (details on preferred instructions can be found in Appendix A.1). Following HIR(Zhang et al., 2023), for instructions linked to positive preferences, the fine-tuning objective aims to increase the probability of generating preferred responses while concurrently decreasing the probability of generating dispreferred responses. Conversely, for instructions associated with negative preferences, the objective is to elevate the probability of generating dispreferred responses and reduce the probability of generating preferred responses.
Formally, let D represent the training dataset, with qi denoting the query, ri representing the response, and pi indicating the instruction (positive or negative). The fine-tuning of the LLM involves minimizing the following loss:
L = − ∑
(pi,qi,ri)∈D (P+ + log
exp (P+)
exp (P+) + exp (P−) )
(1)
where P+= log π(ri | pi, qi; θ), P−= log π(ri| p∗i , qi; θ) and p ∗ i denotes the opposite instruction, ensuring a contrast between preferred and dispreferred cases.
Throughout the entire fine-tuning process, the LLM undergoes a learning phase to distinguish between preferred and non-preferred responses, revealing distinct activity patterns associated with human preferences. Subsequently, these two instructions will serve as stimuli to acquire the model’s internal representations, which will be used for further alignment. This contrastive training relying on preference data enables the achievement of the overarching goal of consistency with a broad spectrum of human preferences, rather than a singular concept.

heading: 3.1.2 Preference Instruction with Dual Models
text: In the Dual LLMs method, we aim to train two LLMs with distinct tendencies: one model is inclined to generate preferred responses, while the
other tends to produce dispreferred responses. To achieve this objective, we employ paired preference data to conduct supervised fine-tuning of the LLMs. Specifically, we use the preferred data from the preference pairs to train the preferred model and the dispreferred data from the preference pairs to train the dispreferred model.
Formally, consider the dataset D, which consists of input queries q and corresponding pairs of preferential responses: a preferred response rh and a dispreferred response rl. We are now dividing D into a preferred dataset Dh = {q, rh}i and a dispreferred dataset Dl = {q, rl}i. Utilizing this data, we employ a supervised learning approach (maximum likelihood) to fine-tune the LLMs, thereby obtaining two models expressing preferences, denoted as πh and πl respectively. The fine-tuning of these two LLMs is aimed at maximizing the achievement of the following objectives:
πh(θ ∗) = argmax
θ
∑
(qi,ri)∈Dh
log π(ri | qi; θ) (2)
πl(θ ∗) = argmax
θ
∑
(qi,ri)∈Dl
log π(ri | qi; θ) (3)
Through this training process, the preferred model and dispreferred model have respectively learned the activity patterns associated with humanpreferred and dispreferred responses. Due to the human preference learning conducted in two distinct models, in contrast to SCIT, the Dual LLMs method does not require additional distinct instructions during fine-tuning. Instead, guidance for the model is provided solely through different responses.

heading: 3.2 Collecting Activity Patterns
text: Following the establishment of comprehension of human preferences by LLMs, we are able to extract representations of what humans prefer and disprefer. Due to the characteristics of autoregressive Transformer language models, the attention mechanism results in tokens at different positions exhibiting distinct representations. The activation representation of a token at the current position is influenced by preceding tokens. Therefore, for a specific pair of query q and response r, this pair is concatenated with two instructions from Section 3.1, which guide the model in forming the concept of human preferences and inputted into the model to obtain the intermediate layer hidden states at each position as internal representations.

heading: Dispreferred ActivationsPreferred ActivationsDifference VectorPad Token
text: Formally, for a given instruction p, a decoder model π, we collect the l-th layer’s hidden states of each token for the query-response pair (qi, ri) within dataset D. This can be formalized as follows:
Ap,π,l = πl(p, qi, ri) | (qi, ri) ∈ D (4)
Here, πl represents the hidden states output by the neural network’s lth layer. We directly extract the hidden layer states from the neural network as representations. To address the issue of varying response lengths during the activity pattern collection, we concatenated the same response with different instructions as input to ensure the representations were extracted with the same length. Different instructions will elicit distinct activity patterns even though the same response was provided and the differences in the elicited activity patterns can be used to capture the behavior of the models. Such differences can be conceptualized and modeled as the probability of generating the same response conditioned on different instructions. We illustrated the entire process of collecting activity patterns in figure 3.
We obtain the final difference vector by subtracting the hidden states of dispreferred outputs from those of preferred outputs, as expressed by the equation:
vl = Ap+,π,l −Ap−,π,l (5)
This difference vector vl represents the difference in activation patterns produced under the two different stimulus conditions p+ and p−. Subsequently,
we perturb the model’s original representation by incorporating the difference vectors. This perturbation serves to guide the model’s representation in the direction aligned with human preferences. It is noteworthy that, for the Single Large Language Model through Contrastive Instruction Tuning (SCIT), both Ap+,π,l and Ap−,π,l are generated by the same model. In the dual LLMs approach, pairs concatenated with different instructions are inputted into the respective preferred and dispreferred models, thereby enabling the independent extraction of activation patterns from each model.

heading: 3.3 Constructing Final Models
text: In this phase, we construct the final model by leveraging the difference vector vl, derived in Section 3.2 to perturb the original representations. To achieve this, we draw inspiration from the approach of Zou et al. (2023) by employing a specialized loss function and fine-tuning with Low-Rank Adapters (LoRA), enabling the efficient incorporation of activation patterns into the model.
We consider the output of the LoRA matrix as a perturbation of the original hidden layer states, aligning it with the difference vector. Specifically, we employ Mean Squared Error (MSE) loss as the objective function:
LAlign = ∥(Ap,πLoRA,l − (Ap,πbase,l + αvl))∥2 (6)
where α serves as a hyperparameter controlling the extent to which the difference vector vl intervenes in the model integration process. Ap,πLoRA,l and Ap,πbase,l represent the activity patterns of the target model equipped with and without LoRA, respectively. vl is the extracted difference vector as outlined in Section 3.2. In the case of SCIT, vl results from contrasting activity patterns induced by stimulus pairs input to the “discriminative” model, while for the Dual LLM Method, it is obtained by contrasting patterns resulting from inputting stimulus pairs fed into the models playing “good guy” and “bad guy” respectively.

heading: 4 Experiment
text: Following Rafailov et al. (2023), we mainly conducted experiments on single-turn dialogue tasks. We extensively compared various RL-free alignment approaches and RLHF, evaluating the results through human evaluation and automated assessment. Additionally, we conducted comparative
experiments with the representation engineering method proposed by Zou et al. (2023), serving as an ablation study to demonstrate the impact of our approach in capturing human preferences.

heading: 4.1 Experimental Setups
text: Dataset In single-turn dialogue, we use UltraFeedback dataset1 (Cui et al., 2023), denoting human preference responses. Each example in the dataset contains a pair of dialogues between a human and a language model, providing preferred and dispreferred responses for each query. Base Model Ouyang et al. (2022) and Ramamurthy et al. (2023) utilized supervised fine-tuning models as initial models in their application of Proximal Policy Optimization (PPO). For a fair comparison, we performed fine-tuning on the LLaMA2-7B model (Touvron et al., 2023) using Anthropic’s Helpful and Harmless dataset2 (Bai et al., 2022). We denote the resulting model after fine-tuning as the Base Model. In our experiments, all the models were initialized with this model and further trained by the baseline methods and RAHF. Additionally, we report the results of experiments using Mistral-7B (Jiang et al., 2023) as the base model in Appendix C.1.

heading: 4.2 Baselines
text: To evaluate our proposed approach, we conduct extensive comparisons with existing alignment methods, including Reinforcement Learning from Human Feedback (RLHF) and other alternative methods for preference alignment. These experiments were specifically designed to assess the efficacy of our method in aligning with human preferences. Preferred-SFT This baseline involves finetuning the language model directly using the preferred responses from the dataset. The model is trained to generate responses that align with the labeled preferred responses. HIR Hindsight Instruction Relabeling (HIR) proposed by Zhang et al. (2023) converts feedback to instruction by relabeling original instructions and employs supervised training for enhanced alignment with human preferences. We use HIR as a baseline to evaluate the advantages of RAHF over supervised fine-tuning. DPO Direct Preference Optimization (Rafailov et al., 2023) directly optimizes a language model to
1 https://huggingface.co/datasets/argilla/
ultrafeedback-binarized-preferences-cleaned 2 https://huggingface.co/datasets/Dahoas/ full-hh-rlhf
adhere to human preferences without using explicit reward modeling or reinforcement learning. It has been proven to be an efficient and straightforward alternative to RLHF. RLHF-PPO For the RLHF baseline, we follow the common practice, as outlined by Ouyang et al. (2022). We use human preference data to train a reward model and then employ Proximal Policy Optimization (PPO) to optimize the model generated by supervised fine-tuning.
Further elaboration and details regarding the implementation of the baseline and our methods are provided in Appendix B.

heading: 4.3 Automatic Evaluation
text: To validate the effectiveness of our proposed method in aligning models with human preferences, automated evaluations were carried out on models trained via RAHF and various baseline methodologies, focusing on their general capabilities and the quality of generation. Specifically, we assessed the performance of different models across three widely used benchmarks: Open LLM Leaderboard(Beeching et al., 2023), AlpacaEval(Li et al., 2023a), and MT-Bench(Zheng et al., 2023). In Appendiex B.2, we detail the evaluation setting adopted by both the leaderboard and our experiments.

heading: 4.3.1 Evaluation on the benchmarks of Open LLM Leaderboard
text: Open LLM Leaderboard comprises six benchmarks that cover science questions, commonsense inference, multitasking accuracy, and truthfulness in generating answers. We evaluate the models’ general capabilities on these tasks.
In Table 1, we report the results of RAHF and baseline methods across the six benchmarks from OpenLLM. RAHF-SCIT achieves the best results in three benchmarks and improves the score by

heading: Method AlpacaEval (win %)
text: 2.33 on average, compared to the base model. RAHF-Dual exhibits the best performance on the MMLU benchmark. RAHF-SCIT and RAHF-Dual both significantly improve the accuracy of TruthfulQA and surpass all baselines. Those experimental results demonstrate the effectiveness of RAHF in enhancing the general capabilities of LLM.
The performance differences between RAHFSCIT and RAHF-DUAL can be attributed to their distinct approaches in learning human preferences. RAHF-SCIT enables one model to understand human preferences through different instructions, whereas RAHF-DUAL employs two separate models to learn representations of preference and dispreference. Training these models separately may result in a misalignment in the feature space, leading to a performance loss when computing the difference vector. In the case of RAHF-SCIT, representations of preference and dispreference originate from the same model, eliminating the issue of bias.

heading: 4.3.2 Evaluation on AlpacaEval
text: AlpacaEval is an automated evaluation benchmark based on LLMs. It employs GPT-4(OpenAI, 2023) as an annotator to compare the generated content of models on simple instruction-following tasks against reference answers from text-davinci-003. Previous work has shown that using GPT-4 as an annotator correlates highly with assessments from human evaluators(Li et al., 2023a). Therefore, we
consider AlpacaEval as an automated approximation of human annotation.
Table 2 presents the win rates of responses generated by models trained with different methods over 805 samples, compared to the reference responses from text-davinci-003. Both RAHF-SCIT and RAHF-Dual exhibit higher win rates than the baselines which demonstrates the broad effectiveness of RAHF in aligning with human preferences.

heading: 4.3.3 Evaluation on MT-Bench
text: MT-Bench is a collection of challenging questions, consisting of 80 samples, each with two turns. This benchmark also employs GPT-4 as a judge to score the responses of models. For each turn, GPT-4 will assign a score on a scale of 10.
Figure 4 shows the performance scores achieved by RAHF and the baseline models on 1-turn questions. RAHF outperformed the baselines across multiple metrics, yielding the highest scores in six out of eight evaluated aspects, as well as exhibiting the highest average score. Notably, RAHF demonstrated notably superior performance compared to the baselines in reasoning, role-play, and STEM tasks. Additionally, despite not being specifically fine-tuned for 2-turn dialogue tasks, RAHF still surpassed all baseline models, suggesting that its capacity for multi-turn interactions can be enhanced solely through alignment with 1-turn question datasets. Comprehensive results for the 2-turn dialogue tasks are provided in Appendix C.4 for detailed comparison.

heading: 4.4 Human Evaluation
text: For the human evaluation, we assigned evaluators the task of comparing two randomly selected responses and providing judgments on their relative performance, categorizing them with three results: win, lose, or tie.

heading: Method Win Tie Lose RAHF-Dual
text: Table 3 presents the comparative results of RAHF against RL-free methods and RLHF in human evaluation. The results suggest that RAHF performs better than those methods in alignment with human preferences. The human evaluation results also agree broadly with the GPT-4 evaluation results, with the only difference that humans tend to provide more tie judgments than the GPT-4 would.

heading: 4.5 Ablation Study
text: To evaluate the influence of instructing LLMs on human preferences using a human-annotated dataset, we executed ablation experiments involving the exclusion of this instructional phase. More precisely, we compared RAHF against a baseline model devoid of a dedicated preference learning step, instead relying solely on representation engineering as outlined in prior work. Additionally, we report the results of several hyperparameter ablation experiments in Appendix C.3. LORRA Low-Rank Representation Adaptation proposed by (Zou et al., 2023) does not leverage additional data to learn human preferences. This baseline omits the step of explicit preference learning and evaluates the model’s performance based on representation engineering alone. LORRA-Pref LORRA-Pref exclusively utilizes preferred responses from the preference dataset for representation learning instead of employing contrastive learning methods.
This ablation analysis allows us to isolate and quantify the impact of assimilating human prefer-
ences into the framework of our proposed approach. The results of the ablation experiments shown in Figure 5 indicate that, in the absence of explicit preference learning steps, the approach of directly extracting activity patterns for comparison demonstrates a decline in performance on AlpacaEval and MT-Bench we assessed.

heading: 4.6 Visualization
text: To gain a deeper understanding of the working mechanism of our method, we conducted a visual analysis of the model’s internal representations using the t-SNE technique.
Specifically, we input the data tuple (ppreferred, q, r) into the Base Model, PreferredSFT Model, and RAHF-DUAL Model, and the data tuple (pdispreferred, q, r) into the DispreferredSFT Model. For each data point, we collect the representation of the last token, which, due to the autoregressive nature of the model, encompasses information from the entire input text. Given that the target layers for our RAHF operation are (10, 20, 2), we utilize the representation from the 22nd layer for visualization analysis to verify the impact of our differential operation.
The results are shown in Figure 6. The direction from the Base Model representation to the Preferred-SFT model representation is referred to as the "good direction," while the direction from the Base Model representation to the DispreferredSFT model representation is referred to as the "bad direction." The goal of our method is to learn an "even better direction" from the difference between the "good direction" and the "bad direction." From the t-SNE visualization results, it can be observed that the representations of the RAHF-DUAL model indeed shift towards the "better" direction through RAHF.

heading: 5 Conclusion
text: In this study, we have explored a representation engineering approach to aligning large language models with human preferences, drawing upon insights from cognitive neuroscience. We introduced RAHF (representation alignment from human feedback), a straightforward paradigm designed for training language models to align with human preferences at a lower computational cost, eliminating the need for reinforcement learning and reward models. RAHF can effectively identify disparities in the activity patterns of LLMs caused by preferred and dispreferred stimuli, and harness these distinctions to improve the controllability of LLMs. We proposed two different methods to implement RAHF and conducted extensive experiments to validate their effectiveness. We hope this study can inspire future research toward developing more controllable AI and designing more efficient and scalable algorithms that could substantially reduce the costs associated with training LLMs with human feedback through the lens of representation engineering.

heading: Limitations
text: In this study, we validated the effectiveness of RAHF on LLMs with 7B parameters. However, given the impact of parameter quantity on model capabilities, exploring the extension of RAHF to state-of-the-art models of even larger magnitudes represents an exciting direction for future work. Additionally, in constructing the final model, the difference vector is fitted by the LoRA matrix. An inherent limitation of this methodology is that it introduces additional parameters, although the extra computational overhead incurred by LoRA is
minimal. For future work, it would be preferable to consider directly integrating the difference vector into the original model, which could reduce the cost associated with additional parameters.

heading: Reproducibility Statement
text: We have publicly shared our code through a GitHub repository https://github.com/ LiuAmber/RAHF. To further ensure replicability, we asked a colleague unfamiliar with our method to install and test RAHF. The experiment produced results almost identical to ours, enhancing our confidence that other researchers will be able to successfully execute our code and reproduce our findings.

heading: Acknowledgements
text: The authors would like to thank the anonymous reviewers for their valuable comments. This work was supported by National Natural Science Foundation of China (No. 62076068).

heading: A Prompts

heading: A.1 Preference Instructions
text: Figure 7 presents two instructions used in this study for preferred and dispreferred responses.
B Implementation Details

heading: B.1 Training Setups
text: All baselines and our models were trained using Anthropic’s Helpful and Harmless dataset(Bai et al., 2022) fine-tuned model as the base model. During the supervised training of the base model, we calculated the loss for both prompts and responses. Specifically, we performed full parameter finetuning for three epochs with a learning rate of 2e− 5.
For training, the data is formatted as follows: Human: {prompt} \n\nAssistant: {response}. For all models trained, we established a maximum query length of 256 and a maximum sentence length of 768. We exclude samples from the dataset where queries exceed 256 characters and truncate sentences to the maximum sentence length. The UltraFeedback dataset has been partitioned into a training set. Further, we split the training set into three distinct parts: the first part is utilized in the first step of RAHF for instructing LLM on human preferences, training the reward model within the RLHF-PPO baseline, and for the training of other baselines. The second part is utilized for the construction of the final model in RAHF and running the PPO algorithm.

heading: B.2 Evaluation Setups
text: For all methods, we employ greedy decoding during generation on the benchmarks. To avoid the issue of repetition, we set the repetition penalty to 1.2.
For the Open LLM Leaderboard, we utilized the Eleuther AI Language Model Evaluation Harness library(Gao et al., 2023) to assess language models trained using different methods. Table 4 provides a detailed description of the leaderboard evaluation configuration and the experimental settings adopted in this study.
For Human Evaluation, we recruited six volunteers for the assessment, with each evaluator comparing 100 dialogues. Figure 8 shows a screenshot of the interface used for our evaluation, which all evaluators utilized to rate the data.

heading: B.3 Experimental Details
text: In this section, we present the experimental details and hyperparameters of the baselines we compare with and our proposed methods. Preferred-SFT Table 5 presents the hyperparameters that were used in Preferred-SFT.
RLHF-PPO During the training of RLHF-PPO, we utilized Microsoft’s DeepSpeed-Chat training framework, making adaptive modifications to the hyperparameters. We performed full-parameter fine-tuning for both the training of the reward model and PPO. Table 6 presents the hyperpa-
rameters for reward model training, while Table 7 presents the key parameters for PPO.
HIR For the HIR baseline, we also conducted full-parameter fine-tuning. Table 8 displays the hyperparameters used for HIR.
DPO We employed the trl framework from Hugging Face to train DPO model. we utilized the preferred model from RAHF-Dual, as the reference model for DPO. We employed LoRA for finetuning. The hyperparameters used in the DPO training are detailed in Table 9. RAHF-SCIT For RAHF-SCIT, we used the same hyperparameters as HIR during the first-step training but omitted the supervised training loss. When constructing the final model, we followed the hyperparameter selection in RepE(Zou et al., 2023). We manipulated layers (10, 20, 2) and set
the perturbation coefficient α to 5. The details of the hyperparameters are shown in Table 10. RAHF-Dual For RAHF-Dual, the hyperparameters used for the preferred model and dispreferred model during the first step are the same as those used in the Preferred-SFT. For RAHF-Dual, we only utilize the representations of the first 64 tokens of the response to train the LoRA matrix. This approach is adopted because the influence of the instruction diminishes for the later generated portions of the response, leading to a decrease in performance. The hyperparameters used in RAHF-Dual are shown in Table 11.

heading: C Additional Results

heading: C.1 Experiment Results On Mistral-7B
text: To verify the effectiveness of our method, we utilized Mistral-7B as the base model, continuing the
previous experimental setup for training, and conducted results on AlpacaEval and MT-Bench. The results are shown in Table 12. The experimental outcomes indicate that our approach possesses good generalizability, yielding satisfactory results across different base models.

heading: C.2 Toxicity Evaluation
text: To ensure that our method does not compromise the model’s safety while augmenting its performance in the aforementioned aspects, we conducted further tests using the Toxigen dataset (Hartvigsen et al., 2022). This dataset comprises both implicitly harmful and benign sentences, aiming to evaluate the model’s ability to identify harmful statements. Accuracy served as the primary metric for evaluation(higher is better). Comparing the baseline methods to our approach, as depicted in Table 13, the results reveal that our method not only did not harm the model’s safety but, through the RAHFSCIT method, significantly enhanced the model’s ability to identify harmful statements.

heading: C.3 Ablation Experiment of Hyperparameters
text: In this section, we primarily report the impact of the hyperparameter α, which controls the intervention strength of the difference vector, and the selected target layer position on alignment performance.

heading: C.3.1 The Effect of Hyperparameter α
text: We conducted an ablation study with different values of α. As we expected, using a smaller α may result in insufficient intervention strength. Conversely, a larger α may lead to excessive intervention strength, which could disrupt the model’s
original representation and cause a degradation in the model’s generation abilities. Therefore, the influence of the hyper-parameter α on performance demonstrates a trend of initial increase followed by a decline as the α value increases. We validated the impact of α across six benchmarks of Open LLM Leaderboard and AlpacaEval shown in Table 14 and Table 15, and our experimental results corroborate the aforementioned perspective.

heading: C.3.2 The Effect of Target Laters’ Selection
text: The earlier layers of neural networks can not fully capture the representation of entire input texts, while the layers close to the top are more taskspecific. Previous studies proved that the representations extracted from the middle layers are more effective in capturing concept-related information (Zou et al., 2023). As to a neural network with 32 layers (Llama2-7b), we chose (10, 20, 2) to extract representations. To further verify the aforementioned viewpoint, we selected different target layers for ablation experiments. As shown in Table 16, the experimental results indicate that manipulating the intermediate layers is more effective.
The significant decline in performance of RAHFDUAL when operating close to the top layers of the network can be attributed to the following reasons: As the depth of the neural network increases, the activations differences between layers also expand. When choosing to operate near the top layers of the network, under the same intervention hyperparameter α conditions, operations at the top layers have a more significant impact on the original representations compared to operations at the middle layers. This excessive influence leads to a notable decrease in the model’s generative capability. Additionally, RAHF-DUAL employs two models in extracting activations differences, the representation of the same input text is different between the two models. The cumulative effect of these two factors results in a more pronounced performance degradation of RAHF-DUAL when operating at the top layers.

heading: Method α AlpacaEval (win %)

heading: Method Target Layers AlpacaEval (win %)

heading: C.4 Experiment Results of MT-Bench
text: Table 17 presents the detailed results of RAHF, baselines, and the ablation study on MT-Bench.

heading: D Qualitative Examples
text: Figure 9 and Figure 10 present qualitative examples of RAHF compared with baselines in dialogue tasks.

heading: Future Work
text: However, given the impact of parameter quantity on model capabilities, exploring the extension of RAHF to state-of-the-art models of even larger magnitudes represents an exciting direction for future work. Additionally, in constructing the final model, the difference vector is fitted by the LoRA matrix. An inherent limitation of this methodology is that it introduces additional parameters, although the extra computational overhead incurred by LoRA is
minimal. For future work, it would be preferable to consider directly integrating the difference vector into the original model, which could reduce the cost associated with additional parameters