Is Free Self-Alignment Possible?

## Abstract

Abstract Aligning pretrained language models (LMs) is a complex and resource-intensive process, often requiring access to large amounts of ground-truth preference data and substantial compute. Are these costs necessary?
That is, it is possible to align using only inherent model knowledge and without additional training? We tackle this challenge with AlignEZ , a novel approach that uses (1) self-generated preference data and (2) representation editing to provide nearly cost-free alignment.
During inference, AlignEZ modifies LM representations to reduce undesirable and boost desirable components using subspaces identified via self-generated preference pairs. Our experiments reveal that this nearly cost-free procedure significantly narrows the gap between base pretrained and tuned models by an average of 31.6%, observed across six datasets and three model architectures. Additionally, we explore the potential of using AlignEZ as a means of expediting more expensive alignment procedures. Our experiments show that AlignEZ improves DPO models tuned only using a small subset of ground-truth preference data. Lastly, we study the conditions under which improvement using AlignEZ is feasible, providing valuable insights into its effectiveness.

## 1 Introduction

Large language model (LMs) alignment involves the use of complex and expensive pipelines [, , ]. Usually at least two critical components are needed: (1) collecting human preference data, and (2) modifying pretrained model weights to better align with these preferences. Some pipelines involve more complexity (e.g., RLHF trains a reward model on the human preference data and uses it for PPO-based model optimization).
Such approaches face substantial scalability challenges: collecting human preference data is costly and time-intensive, and as model sizes increase, the computational requirements for fine-tuning are likely to become prohibitive.

A prospective way to bypass the need for human preference data is to exploit knowledge *already contained* in the pretrained model weights.
This idea is motivated by evidence suggesting that alignment merely reveals knowledge and capabilities acquired during pretraining [, ].
This notion has led to a growing body of literature achieving impressive results using signal contained in pretrained models for fine-tuning [, , , ], largely or totally sidestepping human annotation.

Next, to achieve free alignment, we must additionally obviate the need for fine-tuning.
Instead, we propose to replace it with a form of *representation editing* that does not require computing gradients or even optimizing a proxy loss at all.
Existing representation editing approaches [, , ] rely on access to ground truth data, which does not account for the unique challenges of using only signals from pretrained models. These signals are often noisier and more limited compared to human-annotated data [, , , ], necessitating a more tailored approach.

This work puts together these two pieces to explore the feasibility of free self-alignment. We align pretrained LMs to human preferences using only the knowledge from the model itself, without additional training or fine-tuning. We introduce AlignEZ, a novel approach designed for this setting. Using the pretrained model’s own generated preference pairs, AlignEZ identifies the subspaces within the model’s embedding spaces that correspond to helpful and non-helpful responses. During inference, we surgically modify the model’s embeddings by boosting the components from the helpful subspaces and neutralizing those from the non-helpful ones.

With this nearly cost-free procedure, we effectively narrow the performance gap between pretrained and aligned models by 31.6% across three model architectures and six datasets. Additionally, we explore the potential of AlignEZ to expedite more expensive alignment processes. Our experimental results demonstrate that AlignEZ improves upon models trained using DPO [] with only a small subset of ground-truth preference data. In summary, our contributions include:

- 1.
We introduce
AlignEZ, a nearly cost-free approach that leverages preference data generated by the pretrained LM to modify its embeddings, aligning outputs to human preferences.
- 2.
Our experiments show that AlignEZ significantly narrows the gap between the base model and its counterparts aligned with traditional expensive methods by 31.6% across three model architectures and six datasets.
- 3.
We demonstrate that AlignEZ can *expedite* more expensive methods like DPO by improving models trained with DPO using only a small subset of ground truth preference data, by 2.2% on average.
- 4.
We demonstrate a simple method to possibly predict conditions when free self-alignment using AlignEZ is possible, as a function of the quality of self-generated preference pairs.

Figure: Figure 1: AlignEZ identifies helpful and harmful subspaces for alignment (left)—using only self-generated data. These enable modifying representations during inference (right).
Refer to caption: extracted/5647149/figs/editing.jpg

## 2 Related Work

Our work tackles alignment and sits at the intersection of self-generated synthetic data and efficient model editing. We give a (necessarily) compressed introduction to these areas.

#### LM Alignment.

The standard approach to aligning LMs with human values and preferences relies on human-annotated preference data. This data is used either to (i) train a reward function and subsequently fine-tune the LM to maximize this reward using reinforcement learning objectives, as in methods like RLHF [, ], or (ii) optimize a proxy loss to maximize the margin between preferred and not preferred outputs, as in methods like DPO []. While these methods achieve remarkable performance, they are challenging to implement due to their complex pipelines, the high cost of computing resources, and the limited scalability of acquiring human-preference data.

#### Self-Improvement.

The difficulty of obtaining human-annotated data has led to significant efforts to bypass this requirement. Methods such as those proposed by [, , ] use manually crafted seed prompts to generate high-quality synthetic datasets from pretrained LMs, which are then used for fine-tuning or training reward models. [] uses retrieval-augmented generation to remove reliance on manually designed prompts. Another approach, [], leverages instruction-tuned models to assist in generating synthetic datasets. The work most similar to our approach is [], which emphasizes maximizing the use of knowledge from the pretrained model being aligned. Our work takes this further by exploring whether self-alignment can be made even more cost-effective by replacing fine-tuning with representation editing, dramatically accelerating the alignment process.

#### Representation Editing.

A parallel line of work seeks to modify model behavior without fine-tuning—doing so by solely editing the model’s representations. For vision-language models like CLIP, [] and [] show that removing spurious or unwanted concept subspaces from embeddings boosts model accuracy on rare class predictions. [] shows that doing so in LLM architectures reduces gender bias in generated sentences without degrading model performance in other tasks. [, , ] demonstrate that modifying embeddings during inference to steer them towards certain traits (e.g., honesty, truthfulness, sentiment) can effectively enhance these traits in the generated outputs. Similarly, [] learns the appropriate embedding modification, acting as a form of fine-tuning. *These methods assume access to ground-truth* preference datasets. Our work differentiates itself by designing an intervention technique that can handle the noisier signal from synthetic data generated by LMs.

Figure: Figure 2: Generating (noisy) preference pairs. First, we prompt pretrained models to provide their insight on the characteristics of helpful and non-helpful responses (top). Then, we ask the model to generate responses based on these characteristics (bottom).
Refer to caption: extracted/5647149/figs/generate_data.jpg

## 3 AlignEZ : (Almost) Free Alignment of Language Models

We are ready to describe the AlignEZ algorithm. First, we query a base pretrained LM to generate its own preference data (Figure 2). Our intuition is that, while noisy, base models have learned, from pretraining data, sufficient signal to aid in alignment.
Using this self-generated data, the identify the subspaces in the LM’s embedding spaces that correspond to helpful and harmful directions for alignment. During inference, we modify the LM embeddings using these identified subspaces, steering the model to generate outputs that better align with human preferences (Figure 1).

First, we describe the self-generated preference data extraction pipeline in Section 3.1. Next, we explain how AlignEZ identifies helpful and non-helpful subspaces in Section 3.2. Finally, we detail the embedding editing operation in Section 3.3 and the layer selection procedure for intervention in Section 3.4.

### 3.1 Self-generated Preference Data

First, we extract the human preference signal from the base LLM by querying it to generate its own preference data. Given a dataset $D$ of $N$ queries, for each query $q_{i}$, we first ask the base LM (denoted as $\omega$) to describe characteristics of answers from a helpful agent ($c_{i}^{help}$) and a malicious agent ($c_{i}^{harm}$). Next, we pair each query with its corresponding characteristics: $(c_{i}^{help},q_{i})$ and $(c_{i}^{harm},q_{i})$. We then prompt the LM to generate responses conditioned on these characteristics, resulting in self-generated preference pairs for each query, denoted as $(p_{i}^{help},p_{i}^{harm})$. By applying this procedure to all $N$ samples in the dataset, we obtain self-generated preference data pairs $P^{help}$ and $P^{harm}$. Note that we do not perform any prompt tuning, instead relying on a fixed set of prompt templates. This process is illustrated in Figure 2, with prompt details provided in the Appendix.

Critically, we note that the base models for generating the preference data are not aligned or instruction-tuned. Consequently, the resulting preference pairs may not always align with the conditioning characteristics, introducing noise into the self-preference data. To address this challenge, we tailor the embedding intervention in AlignEZ to accommodate this condition.

### 3.2 Finding Preference Directions

Next, using the noisy self-generated preference data, we identify the directions in the model embedding space that correspond with human preferences. These directions, represented as vectors $\theta\in\mathbb{R}^{d}$ within $\omega$’s latent space, can either (i) align with the helpful preferences $P^{help}$, facilitating alignment of the model’s generated sentences, or (ii) align with the harmful preferences $P^{harm}$, leading to adverse effects on alignment [] []. We denote these directions as $\theta^{help}$ and $\theta^{harm}$, respectively. We explore several ways to identify them.

Figure: Algorithm 1 AlignEZ harmful and helpful subspaces identification

#### SVD-Based Identification.

Our first approach for identifying these directions involves using singular value decomposition (SVD) on the preference data embeddings. We extract the first eigenvector $\theta$. Intuitively, we view $\theta$ as the direction that best captures the underlying concepts. Let $\Phi_{l}$ represent the function that maps an input sentence to the LM embedding space at layer $l$. For each pair $(p_{i}^{help},p_{i}^{harm})$, we obtain their corresponding representations $\Phi_{l}(p_{i}^{help})$ and $\Phi_{l}(p_{i}^{harm})$, which we abbreviate as $\Phi_{i,l}^{help}$ and $\Phi_{i,l}^{harm}$, respectively. To begin, we construct an embedding matrix for helpful preferences, denoted as $\textbf{H}_{l}^{help}$, using these representations:

$$ $\textbf{H}_{l}^{help}:=\left[\Phi_{i,l}^{help}\middle|\ldots\middle|\Phi_{N,l} ^{help}\right]^{T}.$ $$

Similarly, we create the harmful preferences embedding matrix $\textbf{H}_{l}^{harm}$. Then, we proceed to identify the helpful direction as follows:

$$ $\displaystyle\textbf{H}_{l}^{help}=\textbf{U}\Sigma\textbf{V}$ $\displaystyle\theta_{l}^{help}:=\textbf{V}_{0,*}.$ (1) $$

Here, U and V represent the left and right unitary matrices produced by running SVD, respectively, and $\Sigma$ is the diagonal matrix of singular values. We define $\theta_{l}^{help}$ as the first row of V, corresponding to the first eigenvector of $\textbf{H}_{l}^{help}$. The harmful direction $\theta_{l}^{harm}$ is defined similarly.

#### CCS-Based Identification [ 6 ] .

Another approach for identifying these directions is by finding a hyperplane in the latent space that separates helpful data embeddings from harmful ones. Typically, this is achieved by training lightweight probes $\theta_{l}$ that maps $\Phi_{i,l}^{help}$ and $\Phi_{i,l}^{harm}$ to their respective classification labels []. However, we face the challenge of avoiding overfitting to the noise inherent in self-generated data, which limits the applicability of supervised classifier loss in our context. To mitigate this issue, we employ the unsupervised Contrast-Consistent Search (CCS) loss $\mathcal{L}_{CCS}$ proposed in []. Adapting the definition from [] to our notations, $\mathcal{L}_{CCS}$ can be expressed as:

$$ $\displaystyle\mathcal{L}_{consistency}:=[\theta_{l}(\Phi_{i,l}^{help})-(1- \theta_{l}(\Phi_{i,l}^{harm}))]^{2}$ $\displaystyle\mathcal{L}_{confidence}:=min\{\theta_{l}(\Phi_{i,l}^{help}), \theta_{l}(\Phi_{i,l}^{harm})\}$ $\displaystyle\mathcal{L}_{CCS}:=\operatorname{\mathbb{E}}{\left[\mathcal{L}_{ consistency}+\mathcal{L}_{confidence}\right]}.$ (2) $$

Training $\theta_{l}$ with the $L_{CCS}$ objective aims to find a separating hyperplane without fitting any labels with $\mathcal{L}_{consistency}$ and concurrently promoting maximum separation with $\mathcal{L}_{confidence}$. Unlike the SVD approach, the hyperplane identified by this method can be used as either $\theta_{l}^{harm}$ or $\theta_{l}^{help}$, depending on which cluster it maps to class ‘1’. Specifically, we assign $\theta_{l}$ as $\theta_{l}^{harm}$ if it maps the majority of samples in $\textbf{H}_{l}^{harm}$ to class 1.

#### Hybrid Identification.

After exploring both methods, we find that the best results come from combining the two approaches. Specifically, we use SVD to identify $\theta_{l}^{help}$ and CCS to determine $\theta_{l}^{harm}$. This combined approach leverages the strengths of both techniques: SVD’s ability to capture the primary direction of helpful embeddings and CCS’s effectiveness in identifying the hyperplane that best separates harmful embeddings from helpful ones. We describe AlignEZ subspace identification in Algorithm 1

### 3.3 Alignment with Embedding Editing.

With the harmful and helpful subspaces $\theta_{l}^{harm}$ and $\theta_{l}^{help}$ identified, we proceed to modify the LM embeddings during inference. Given $x_{l}$ as the output of the MLP of layer $l$, the AlignEZ editing process proceeds as follows:

$$ $\displaystyle\hat{x_{l}}\leftarrow x_{l}-\frac{\langle x_{l},\theta_{l}^{harm} \rangle}{\langle\theta_{l}^{harm},\theta_{l}^{harm}\rangle}\theta_{l}^{harm} \quad\text{ and }\quad\hat{x_{l}}\leftarrow\hat{x_{l}}+\frac{\langle\hat{x_{l} },\theta_{l}^{help}\rangle}{\langle\theta_{l}^{help},\theta_{l}^{help}\rangle} \theta_{l}^{help}.$ $$

In the first step, we use vector rejection to remove the influence of $\theta_{l}^{harm}$ from $x_{l}$. In the second step, we adjust the embedding by steering it towards the helpful direction $\theta_{l}^{help}$. We perform the edit at every generation time-step. We illustrate AlignEZ’s representation editing step in Figure 1.

### 3.4 Selecting Layers for Intervention.

The last piece of the puzzle is determining which layers of the LM to apply our embedding editing to. Intuitively, we want to intervene in the layers where the embeddings of $X^{harm}$ and $X^{help}$ *are most separable*, maximizing the effectiveness of AlignEZ. To accomplish this, we select the layers for intervention by identifying the subset of layers with the lowest average $\mathcal{L}_{CCS}$ loss. This ensures that our alignment interventions are targeted at the most impactful layers of the model. We provide the pseudocode for layer selection in the Appendix.

Figure: Figure 3: AlignEZ Relative Improvement%. The y-axis shows the Relative Improvement%: how much AlignEZ enhances the base model’s performance compared to an aligned version. Values are recorded across six datasets (x-axis). A value of 100% means AlignEZ improves the base model to the same extent as the aligned version, while 0% means AlignEZ performs the same as the base model. Performance is recorded for three model families: Mistral (orange), Llama2 (green), and Llama3 (blue). We observe substantial improvements in the base models, resulting in a narrower performance gap between the base models and the aligned versions.
Refer to caption: extracted/5647149/figs/main_plots/Engagement.png

## 4 Experiments

We evaluate the following claims about AlignEZ.

- •
Reduces alignment gap (Section 4.1). AlignEZ significantly reduces the performance gap between the base model and aligned model without any additional fine-tuning and access to ground-truth preference data.
- •
Expedites alignment (Section 4.2). AlignEZ *expedites DPO alignment* by improving models that have been DPOed on *only a small* subset of ground-truth preference data.
- •
Compatible with prompting techniques (Section 4.3). AlignEZ is compatible with and can be used in combination with prompt engineering-based alignment methods [] (Section 4.3).
- •
Predicts when self-alignment is possible? (Section 4.4). Self-generated data provides a signal about the model’s ability to self-align with AlignEZ.

#### Metrics.

We follow the most popular standard for automatic alignment evaluation, using GPT-4 as a judge to compare a pair of responses [] and calculate the win rate (Win %) and lose rate (Lose %). To ensure a more nuanced and unbiased evaluation, we employ the *multi-aspect evaluation technique* proposed in []. Rather than evaluating the overall quality of the generated text, we ask GPT-4 to assess it across five aspects: Engagement (E), Helpfulness (H), Factuality (F), Depth (D), and Clarity (C). We use the same prompt template as [] and measure the following metrics:

- 1.
Net Win% $=$ Win% $-$ Lose%: A model that produces meaningful improvement over the base model will exhibit a higher win rate than lose rate, resulting in a positive net win percentage.
- 2.
Relative Improvement%.
$\frac{\text{Net Win}{\hskip 4.0ptours-base}}{\text{Net Win}{\hskip 4.0
ptaligned-base}}\times 100.$
This metric evaluates how much AlignEZ improves alignment of the base pretrained model, relative to the aligned model. A value of 0% means AlignEZ offers no improvement over the base model, while 100% means AlignEZ matches the performance of the aligned model. Positive percentages between 0% and 100% indicate that AlignEZ narrows the performance gap between the base and aligned models, and a negative percentage indicates a performance decline from the base model. Excitingly, we additionally sometimes observe AlignEZ performance beyond the aligned model.

#### Datasets.

To evaluate AlignEZ’s generalization capability across diverse tasks and topics while keeping evaluation affordable, we use the helpfulness slice of the just-eval-instruct dataset []. This dataset is a diverse collection of queries created by sampling and merging several datasets. Specifically, we use the helpfulness slice, which combines (1) AlpacaEval [] (including helpful-base, koala, vicuna, open-assistant (oasst), and self-instruct), and (2) MT-Bench []. We report AlignEZ’s performance on these individual slices.

#### Baselines.

We compare AlignEZ against several base models: (1) Mistral-7B-v0.1 [], (2) Llama-2-7B [], and (3) Llama3-8B []. As an upper bound, we also compare these base models to their aligned versions. For Llama2 and Llama3, we use Llama-2-7b-Chat and Llama-3-8B-Instruct, which are RLHF versions of the base models [, ]. For Mistral, we use Mistral-7B-Instruct-v0.1, a version of the base model fine-tuned with instruction tuning datasets []. We report results using the Mistral instruction-tuned model because our experiments show it outperforms the open-source Mistral DPO [] on our evaluation datasets.

While we do not expect AlignEZ to consistently outperform the aligned models, we anticipate a positive Relative Improvement% metric. This would indicate that AlignEZ effectively brings the base model’s performance closer to that of the aligned model without incurring additional costs.

### 4.1 Reducing Alignment Gap

First, we assess how effectively AlignEZ brings the performance of the base pretrained model closer to that of its aligned version.

#### Setup.

All experiments use frozen LLM weights, with no additional training of these weights. We only train lightweight probes to identify $\theta_{l}^{harm}$ using $L_{CCS}$ (see Section 3). Details on the hyperparameters for probe training are provided in the Appendix.

#### Results.

Our results are shown in Figure 3. We observe consistent positive Relative Improvement% across datasets and model architectures. This validates our claim that AlignEZ reduces the alignment gap between base models and their aligned versions, occasionally even surpassing the performance of the aligned models. Remarkably, these improvements are achieved without access to ground truth preference data or any additional fine-tuning. In cases where AlignEZ does not yield improvements, such as with the Llama2 model on the vicuna dataset, we investigate the essential conditions for improvement in Section 4.4.

Figure 3 also reveals an interesting insight: AlignEZ shows more significant improvements in aspects like Helpfulness and Factuality compared to Engagement and Depth. This suggests that AlignEZ primarily enhances utility-related aspects of the base model, while its impact on stylistic aspects is comparatively limited. This indicates potential areas for further improvement in the self-generated data process. For example, generating preference data based on multiple aspects rather than a single differentiating category (e.g., helpful vs. non-helpful, as shown in Figure 2) might lead to improved overall performance.

Figure: Figure 4: AlignEZ improves DPO models trained on a small subset of the ground-truth preference dataset. The column % is the percentage of data used for DPO training.
Refer to caption: extracted/5647149/figs/dpo.png

### 4.2 Expediting Alignment

Next, we evaluate AlignEZ’s ability to expedite more expensive alignment techniques like DPO. Specifically, we test whether AlignEZ can improve models trained with DPO using only a smaller subset of ground-truth preference data.

#### Setup.

We perform DPO fine-tuning on the Mistral-7b-base model using the UltraFeedback-binarized dataset [, ] and do evaluation on the test set. We provide the complete DPO training parameters in the Appendix.

#### Results.

Our results are shown in Table 5. AlignEZ enhances the alignment of models tuned using DPO on a small subset of ground truth preference data, indicated by the positive Net Win%. This confirms our claim that AlignEZ expedites DPO alignment. In Figure 5, we observe that the improvement provided by AlignEZ diminishes as the percentage of training data increases, which is expected since the benefit from DPO itself grows with more training data. This result highlights AlignEZ’s potential to provide additional alignment gains when only a limited amount of ground-truth preference data is available.

### 4.3 Compatibility with Prompting Techniques

We also investigate the adaptability of AlignEZ when combined with other cost-effective alignment techniques, such as prompting [].

**Table 1: Compatibility with prompting-based methods.**
| Dataset | Model | Net Win% ($\uparrow$) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| E | H | F | D | C | Avg. |  |  |
| Vicuna | Llama2-base | 10 | 3 | 3 | 7 | 10 | 6.6 |
| Koala | Llama3-base | 8 | 12 | 1.3 | 5.3 | 6.7 | 6.6 |

#### Setup.

We use the URIAL prompt proposed in [] as a prefix for every query and record the performance both with and without AlignEZ applied. This prompt consists of manually crafted set of in-context learning examples designed to mimic the style of high-performing models such as ChatGPT and other advanced aligned LLMs.

#### Results.

Table 1 demonstrates that AlignEZ enhances performance beyond what is achieved by using the prompting technique alone, as indicated by the positive Net Win%. This confirms our claim that AlignEZ is compatible with prompting techniques and shows its versatility to be used in combination with other cost-effective alignment methods.

Figure: (a)
Refer to caption: extracted/5647149/figs/micro_logreg/Mistral.png

### 4.4 When is Self-Alignment Possible?

We study whether the quality of self-generated data can predict if using AlignEZ leads to model improvement. To assess the data quality, we measure the generalization ability of classifiers trained on the self-generated data.

#### Setup.

We train logistic regression classifiers on the embeddings of the self-generated data to predict the labels associated with the data and record the test performance. Additionally, we use an off-the-shelf sentence embedder to remove the influence of model embedding quality. The reported values are averaged across five independent runs.

#### Results.

Figure 6 shows that the average Net Win% achieved by AlignEZ generally correlates with the adjusted classifier accuracy. This supports our claim that self-generated data provides a signal about the model’s ability to self-align. This correlation is particularly strong for the Mistral model. For the Llama3 and Llama2 models, the trend is mostly consistent, with some exceptions being the koala dataset on Llama3 (leftmost point) and the self-instruct dataset on Llama2 (rightmost point).

Extending this approach may offer a quick and effective method for selecting data suitable for alignment. This is crucial, as extensive research has shown that the composition and quality of training data are critical to the resulting model’s performance [, , ].

## 5 Discussion

#### Limitations and Future Work.

AlignEZ presents several limitations and avenues for future exploration. First, we perform embedding editing at every generation time step. However, it remains uncertain whether selecting specific time steps for intervention could yield further improvements. Second, while we see promising indications in Section 4.4 that the quality of self-generated data correlates with AlignEZ improvement, refining this characterization by developing a specialized metric for predicting the model’s ability to self-align would be useful. Similarly useful would be to conduct an analysis to gauge the steerability of the base model based on the quality of its pretrained model embeddings. Additionally, our technique needs to be adapted for red-teaming scenarios, where the goal is to have the model refuse to answer certain questions instead of providing information.

#### Conclusion.

We introduce AlignEZ, a novel approach for aligning pretrained LMs with human preferences without access to human-annotated data and fine-tuning. By leveraging the inherent knowledge within pretrained models, AlignEZ modifies model embeddings during inference to produce outputs that better align with human preferences. We empirically show that AlignEZ consistently enhances the alignment of the base model across multiple evaluation aspects, occasionally surpassing the performance of their aligned counterparts. Additionally, we show that AlignEZ can expedite more costly alignment techniques like DPO.

This work takes an initial step toward achieving truly cost-free alignment and paves the way for the development of techniques in exciting new domains like real-time dynamic alignment and fast model personalization – areas currently beyond the reach of standard alignment methods.

## Appendix A Appendix / supplemental material

### A.1 Glossary

Table 2 shows glossary of terms used in this paper.

**Table 2: Glossary of variables and symbols used in this paper.**
| Symbol | Definition |
| --- | --- |
| $D$ | Dataset of queries |
| $q_{i}$ | Sample query |
| $\omega$ | Language Model |
| $l$ | Language model layer index |
| $c_{i}^{help}$ | Characteristic of helpful answer |
| $c_{i}^{help}$ | Characteristic of harmful/unhelpful answer |
| $p_{i}^{help}$ | Helpful preference sample |
| $P^{help}$ | Self generated helpful preference data |
| $P^{harm}$ | Self generated harmful/unpreferred preference data |
| $\theta^{help}$ | Subspace of helpful preference samples |
| $\theta^{harm}$ | Subspace of harmful/unpreferred preference samples |
| $\Phi_{i,l}^{help}$ | Embedding of $p_{i}^{help}$ in layer $l$ of $\omega$, abbreviation of $\Phi_{l}(p_{i}^{help})$ |
| $\Phi_{i,l}^{harm}$ | Embedding of $p_{i}^{harm}$ in layer $l$ of $\omega$, abbreviation of $\Phi_{l}(p_{i}^{harm})$ |
| $\textbf{H}_{l}^{help}$ | Embedding matrix stacked from $\Phi_{i,l}^{help}$ |
| $\textbf{H}_{l}^{harm}$ | Embedding matrix stacked from $\Phi_{i,l}^{harm}$ |
| $\textbf{V}_{0,*}$ | First row of the right unitary matrix |
| $x_{l}$ | output of MLP at layer $l$ |
| $\hat{x_{l}}$ | MLP output after AlignEZ embedding edit |

### A.2 DPO Training details

#### Dataset

DPO experiment were trained on binarized UltraFeedback dataset [, ].

#### Computing resources

Experiment training on 1%, 5%, 10% and 25% of the dataset were run on an Amazon EC2 Instances with eight Tesla V100-SXM2-16GB GPUs.

#### Hyperparameters

The hyperparameters we used consist of 1 training epoch, a gradient accumulation step of 1, a learning rate of $5e-5$, a max grad norm of 0.3, a warmup ratio of 0.1 (based on []), a precision of bfloat16, a memory saving quantize flag of "bnb.nf4", a learning rate scheduler type of cosine, and an optimizer of AdamW [] (based on []). We applied PEFT [] method to model training with hyperparameters of a $r$ of 256, a $\alpha$ of 128, a dropout of 0.05 and a task type of causal language modeling (based on [, ]). A batch size of 16 is used to train the 1%, 5%, 10% and 25% data experiment. A batch size of 20 is used to train the full data experiment.

### A.3 CCS Probe training details

We train a 1 layer linear layer with dimension of the LM embedding using the following hyperparameters: epoch = 1000, lr=1e-3, batch size=number of preference pairs, weight decay=0.01. We repeat training 10 times and take the probe with the lowest $\mathcal{L}_{CCS}$. Training is conducted in the Amazon EC2 instances with 8 Testa V100s.

### A.4 AlignEZ Net Win and Relative Improvement Table

**Table 3: Main results table. In cases when AlignEZ produces positive improvement and instruct models produce zero or negative, we mark Relative Improvement% as -inf. In cases where both methods produce zero or negative improvement, we mark it as -.**
| Dataset | Model | Net Win% ($\uparrow$) | Relative Improvement% ($\uparrow$) |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| E | H | F | D | C | Avg. | E | H | F | D | C | Avg. |  |  |
| helpful-base | Mistral-7B + Ours | 3 | 6 | -2 | 12 | 6 | 5 | -inf | -inf | - | 86 | -inf | -inf |
| Mistral-7B-instruct | 0 | -16 | -17 | 14 | -12 | -6 |  |  |  |  |  |  |  |
| Llama2-7B + Ours | 4 | 7 | 9 | 3 | -3 | 4 | 13 | 26 | 112.5 | 9.4 | -33.3 | 19 |  |
| Llama2-7B-chat | 31 | 27 | 8 | 32 | 9 | 21 |  |  |  |  |  |  |  |
| Llama3-8B + Ours | 7 | 1 | -1 | 4 | 3 | 2.8 | 150 | -inf | - | -inf | 100 | -inf |  |
| Llama3-8B-instruct | 5 | -12 | -1 | -6 | 3 | -2 |  |  |  |  |  |  |  |
| koala | Mistral-7B + Ours | 1.3 | 12 | 4 | 8 | 1.3 | 5.3 | 43.3 | 120 | 133.3 | 40 | -inf | 88.3 |
| Mistral-7B-instruct | 3 | 10 | 3 | 20 | -7 | 6 |  |  |  |  |  |  |  |
| Llama2-7B + Ours | 4 | 7 | -2.6 | 9 | 2.6 | 4 | 9.3 | 17 | -17.3 | 22 | 8.7 | 11.8 |  |
| Llama2-7B-chat | 43 | 41 | 15 | 41 | 30 | 34 |  |  |  |  |  |  |  |
| Llama3-8B + Ours | -7 | 5 | 4 | 4 | -5 | 0.3 | -50 | 31.2 | 200 | 33.3 | -30 | 2.5 |  |
| Llama3-8B-instruct | 14 | 16 | 2 | 12 | 17 | 12 |  |  |  |  |  |  |  |
| vicuna | Mistral-7B + Ours | 7 | -3 | -3 | 10 | -3 | 1.3 | 233.3 | -30 | -100 | 50 | - | 21.7 |
| Mistral-7B-instruct | 3 | 10 | 3 | 20 | -7 | 6 |  |  |  |  |  |  |  |
| Llama2-7B + Ours | 0 | 0 | 7 | -14 | 3.5 | -0.7 | 0 | 0 | 70 | -42 | 15 | -2.4 |  |
| Llama2-7B-chat | 37 | 43 | 10 | 33 | 23 | 29 |  |  |  |  |  |  |  |
| Llama3-8B + Ours | -7 | 7 | 0 | 0 | 3 | 0.7 | -41 | -inf | - | 0 | 30 | 11.7 |  |
| Llama3-8B-instruct | 17 | 0 | 0 | 3 | 10 | 6 |  |  |  |  |  |  |  |
| oasst | Mistral-7B + Ours | -2 | -4 | -2 | -5 | -2 | -3 | -200 | -133 | - | -100 | - | - |
| Mistral-7B-instruct | 1 | 3 | -5 | 5 | -9 | -1 |  |  |  |  |  |  |  |
| Llama2-7B + Ours | 3 | 7 | -3 | 7 | 7 | 4.2 | 7.5 | 15.6 | -50 | 13.5 | 70 | 14 |  |
| Llama2-7B-chat | 40 | 45 | 6 | 52 | 10 | 30 |  |  |  |  |  |  |  |
| Llama3-8B + Ours | 7 | 8 | -1 | 4 | 7 | 5 | 87.5 | 50 | -11 | 28.6 | 46.7 | 40.3 |  |
| Llama3-8B-instruct | 8 | 16 | 9 | 14 | 15 | 12.4 |  |  |  |  |  |  |  |
| self-instruct | Mistral-7B + Ours | 1 | 1 | 2 | 2 | 1 | 1 | 33.3 | 11.1 | 66.7 | 50 | 20 | 20 |
| Mistral-7B-instruct | 3 | 9 | 3 | 4 | 5 | 5 |  |  |  |  |  |  |  |
| Llama2-7B + Ours | 2.7 | 9 | 2.7 | 7.2 | 8 | 6 | 15 | 45 | 39 | 27 | 40 | 33.3 |  |
| Llama2-7B-chat | 18 | 20 | 7 | 27 | 20 | 18 |  |  |  |  |  |  |  |
| Llama3-8B + Ours | -2 | 7 | -3 | 4 | 0 | 1.2 | -14 | 44 | -150 | 33.3 | 0 | 10 |  |
| Llama3-8B-instruct | 14 | 16 | 2 | 12 | 17 | 12 |  |  |  |  |  |  |  |
| MT-Bench | Mistral-7B + Ours | 3 | 5 | 0 | 1 | 0 | 2 | 30 | 500 | -inf | 7.7 | 0 | 40 |
| Mistral-7B-instruct | 10 | 1 | -5 | 13 | 6 | 5 |  |  |  |  |  |  |  |
| Llama2-7B + Ours | -2.5 | 9 | -1.2 | 6 | 0 | 2.3 | -12.5 | 60 | -24 | 40 | 0 | 21 |  |
| Llama2-7B-chat | 20 | 15 | 5 | 15 | 1 | 11 |  |  |  |  |  |  |  |
| Llama3-8B + Ours | 6 | 5 | 2.5 | -4 | -4 | 1.3 | 95 | 125 | -inf | -63 | -400 | 37 |  |
| Llama3-8B-instruct | 6.3 | 4 | 0 | 6.3 | 1 | 3.5 |  |  |  |  |  |  |  |

Table 3 shows the detailed numbers for the experiment in Section 4.1.

### A.5 Layer Selection Pseudocode

Below is the pseudocode for layer selection. We select layers that have low average $\mathcal{L}_{CCS}$, by heuristically select the layers before the running mean increases significantly.

### A.6 Prompt Template

Following is the prompt template used to query the base LM to generate preference samples:

#### Generating helpful samples characteristics:

[QUERY]. You are a helpful assistant. Your answer to this query should:

#### Generating harmful/unpreferred sample characteristics:

[QUERY]. Pretend you are a malicious and useless assistant. Your answer to this query should:

### A.7 Broader Impacts

Our work inherits the societal impacts associated with language models. On one hand, there’s the risk of these models generating responses to potentially harmful queries, particularly in redteaming scenarios—an issue we acknowledge and address in our main body’s limitation section. Conversely, our approach offers a potential positive societal impact by enabling a nearly cost-free alignment of language models. This capability could facilitate easier and faster alignment processes, leading to broader access to well-aligned models and ultimately contributing to positive societal outcomes.

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. While the paper shows the compatibility of this method with the prompting method, I wonder about the head-to-head comparison of the proposed technique and prompting technique in improving the alignment performance of LLMs, since both are very cheap methods,
2. Any fun visualization or analyses on the "help" and "harm" vector space?
3. Any thoughts on the reward-guided test-time alignment techniques? This is related to the efficiency alignment techniques, although it's not closely related to the techniques in this paper per se.
soundness: 3
strengths: 1. By eliminating reliance on costly fine-tuning and human-annotated preference data, AlignEZ offers a more resource-efficient alternative to traditional alignment techniques, making it potentially scalable for larger or real-time applications.
2. AlignEZ presents a conceptually straightforward approach, leveraging self-generated preference data and representation editing to achieve alignment without the need for extensive fine-tuning or additional ground-truth annotations.
3.  Empirically, the approach demonstrates substantial performance gains over DPO baselines, showing that AlignEZ is effective even when traditional alignment data is limited. It also improves data efficiency by expediting existing RLHF methods, like DPO. 
4. Moreover, AlignEZ integrates well with prompting strategies, enhancing alignment performance beyond what prompting alone achieves and expanding its usability with other alignment techniques. Prompting method is another cheap way to align LLMs without much cost and it's good to see both methods can be combined strongly.
summary: This paper introduces AlignEZ, a method for aligning large language models (LMs) with human preferences without using additional training data or fine-tuning. The approach addresses the high costs associated with traditional alignment methods that require extensive human-annotated data and fine-tuning. AlignEZ leverages self-generated preference data by prompting the base model to produce examples of “helpful” and “harmful” responses, thereby creating synthetic data that approximates human preferences. During inference, AlignEZ performs representation editing, modifying model embeddings to accentuate desirable (helpful) and reduce undesirable (harmful) components. This enables alignment without training, relying instead on adjustments to the model’s representations in real time.

Empirically, AlignEZ narrows the performance gap between base and aligned models by 29.1% on average across multiple datasets and architectures, demonstrating that it can boost alignment quality in a cost-effective manner. In experiments, AlignEZ also expedites traditional alignment processes like Direct Preference Optimization (DPO), enhancing model performance even when only a small subset of ground-truth data is available. Additionally, AlignEZ integrates effectively with prompting techniques, yielding further improvements beyond what prompting alone can achieve.
weaknesses: 1. While AlignEZ combines self-generated preference data and representation editing effectively, both of these approaches are well-established methods. Techniques for generating synthetic preference data have been extensively studied in self-alignment literature, which limits the technical novelty of this work. To strengthen its contribution, the authors could further emphasize unique aspects of their integration of these techniques or explore additional novel dimensions, such as adaptive mechanisms for more refined, context-dependent alignment adjustments.
2. The concept of "free" alignment in AlignEZ may be somewhat overstated, as both the self-generation of preference data and embedding modifications require non-negligible computational resources. While ALIGNEZ reduces reliance on human-labeled data, it does not entirely eliminate costs, especially when scaling to larger models. Clarifying these claims by discussing cost reductions relative to traditional methods, rather than "free" alignment, would provide a more balanced and realistic portrayal of AlignEZ's cost efficiency.
3. Similarly, the title, “Is Free Self-Alignment Possible?” may prioritize entertainment value over clarity, providing limited insight into the paper’s actual contributions. A more precise framing, such as “Cost-Efficient Self-Alignment through Representation Editing and Self-Generated Data,” could better communicate the scope and implications of the study.
4. The theoretical analysis presented in Section 3 is built on simplifying assumptions that may not fully capture the complexities of real-world model behavior. Specifically, the assumption that LLM space is orthonormal is oversimplified. Also, the conclusion is unclear enough to specify whether the singular vectors from AlignEZ are strong enough or whether the kNN smoothing is good enough, etc
5. The empirical results are based on relatively small language models (7B and 8B parameters), which, given the rapid advancements in AI, are now considered less representative of state-of-the-art capabilities. Applying ALIGNEZ to significantly larger models (e.g., 100B+ parameters) would provide stronger evidence of its scalability and effectiveness in larger, more complex architectures. While it is challenging to access larger models due to their proprietary nature, exploring ways to adapt ALIGNEZ for environments where parameter access is limited, or testing it on larger open-source models, could strengthen its empirical validation. Additionally, the reliance on open-source models might be seen as a minor limitation, as this restricts ALIGNEZ’s applicability to cases where model weights are accessible, potentially narrowing its practical impact in real-world, production-level systems that often employ closed-source models.

---

confidence: 4
contribution: 3
initial_rating: 3
presentation: 3
questions: 1. What would the impact be if only the “helpfulness enhancement” or solely the “harmfulness suppression” (as described in Section 2.3) were applied individually? Could you provide insights or experiments on this?

2. Could using k-nearest neighbor (kNN) search to retrieve responses as in-context examples when prompting the model further enhance its alignment abilities, instead of just using this in the sample-conditional estimation of helpful and harmful direction.
soundness: 3
strengths: 1. The paper is well-written and easy to follow.
2. The performance results presented, particularly in Table 1, demonstrate the effectiveness of AlignEZ. As an inference-time alignment technique, AlignEZ significantly improves the alignment recovery compared to other test-time alignment baselines.
3. The authors conduct additional experiments to confirm that their method is orthogonal to prompting techniques such as URIAL. This enhances the practicality and robustness of the proposed approach.
summary: The paper introduces AlignEZ, a cost-efficient approach for aligning pretrained language models without the need for large-scale ground-truth preference data or extensive computational resources. Instead, AlignEZ utilizes self-generated preference data and representation editing to adjust model outputs during inference. By modifying model representations to suppress undesirable traits and enhance preferred ones using identified subspaces, AlignEZ significantly improves model alignment. Experimental results across five datasets and two architectures demonstrate a 29.1% average improvement, narrowing the gap between pretrained and fine-tuned models. Additionally, AlignEZ shows potential for expediting more expensive alignment methods by enhancing models trained with limited ground-truth preference data.
weaknesses: 1. The comparison with the baselines appears somewhat unfair. While the baseline test-time alignment techniques utilize ground-truth preference signals, these signals are sourced from a different dataset (HH-RLHF), introducing a distribution shift compared to the data used in AlignEZ. To provide a more balanced comparison, I suggest evaluating AlignEZ using the same HH-RLHF data during inference.

2. Concerning practicality and efficiency, the authors do not provide details on the overhead induced by their method, particularly in terms of inference latency or memory utilization. It would be beneficial to compare these overheads against those of the baseline methods.

3. It remains unclear whether AlignEZ affects other critical aspects, such as model safety or propensity for hallucination. I recommend including a discussion or experiments that assess the potential impact of AlignEZ on these factors.

4. A highly viable and practical application of test-time alignment is personalization, rather than just general helpfulness (which is already addressed by the aligned versions of many models). Moreover, the dataset used in this paper is somewhat outdated. Providing experiments or insights focused on personalization could offer valuable contributions and make the paper more relevant.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. In the claim, 'With this nearly cost-free procedure, we effectively narrow the performance gap between pretrained and aligned models by 29.1% across two model architectures and five datasets,' how was the 29.1% calculated?

2. Line 239 may be intended to refer to $z_{help}$ and $z_{harm}$.

3. In Figure 4 (right), what insights can be drawn from the statement 'The lowest cosine similarities are observed in the middle layers (layers 10 to 25)'?

4. In line 303, the term "DPOed" is used, which is not a formal expression. 

5. The main experiments lack detailed descriptions. It's not clear how to implement the method across different datasets.
soundness: 2
strengths: 1. Self-Generated Data: ALIGNEZ relies solely on self-generated data from the language model, eliminating the need for additional manual annotation, which is promising for scalability.

2. Performance Improvement: Experimental results show that this alignment approach effectively reduces the performance gap compared to traditional training-based methods, without requiring any additional training.

3. Data Efficiency: By using only 1% of preference data for DPO training, ALIGNEZ achieves results comparable to those obtained with 25% of preference data, demonstrating compatibility with existing preference alignment methods.
summary: This paper introduces ALIGNEZ, a method that utilizes the inherent knowledge of pretrained language models to achieve low-cost alignment using only self-generated preference data. It employs a KNN approach to identify the top n-closest preference data points for a given query, followed by SVD to extract the most relevant representations related to helpful and harmful preferences. By adjusting these embeddings during inference, ALIGNEZ effectively narrows the performance gap between pretrained and aligned models, guiding them to generate more helpful responses without requiring additional training or human-annotated data.
weaknesses: 1. Unclear Methodology: The description of the method's details is vague, particularly regarding the origin of queries used for generating self-generated preference data. Additionally, it's unclear which dataset was used for the experiments shown in Figures 3 and 4, and whether the reported $\Delta%$ represents an average score across multiple datasets or results from a single dataset.

2. Dependence on base LM: The base LM may generate incorrect preference data, and the paper does not clarify how it ensures that self-generated preference data accurately reflects the correct preference relationships. This lack of clarity could significantly impact the results. Furthermore, while the method shows some effectiveness on the base LM, its performance on instruction-tuned LMs remains unexplored, limiting its contribution. For example, in the experiment for Figure 3, the performance with and without ALIGNEZ when the DPO dataset is expanded to 100% is not assessed.

3. Generalizability Issues: ALIGNEZ uses a statistical method (kNN) to obtain feature vectors for embedding editing during inference, raising concerns about its generalizability. It may only be effective when the inference input data is strongly correlated with the statistical data, making it difficult to handle out-of-distribution (OOD) situations.

4. Narrow Focus on Helpfulness: A significant limitation of the paper is its narrow focus on helpfulness, which does not convincingly demonstrate the overall effectiveness of ALIGNEZ. This raises doubts about whether ALIGNEZ is universally applicable to other aspects, such as safety or more complex alignment scenarios that involve a mix of helpfulness and safety. More extensive testing would strengthen the paper's claims.

5. Dependency on LLM's Instruction-Following Ability: Since ALIGNEZ relies on self-generated data to identify subspaces in the LM's embedding spaces corresponding to helpful and harmful directions for alignment, it requires a high degree of instruction-following capability from the LLM. This suggests that the method is better suited for models that have undergone instruction fine-tuning or alignment. However, the experiments are primarily conducted on the base pretrained model, which may not adequately reflect the method's potential effectiveness in more refined contexts.

Meta-review: ALIGNEZ is a cost-effective approach for aligning language models through self-generated preferences and representation editing, eliminating the need for additional training. However, reviewers have raised concerns about its limited innovation and a lack of clarity in the methodology. Furthermore, the paper falls on the borderline regarding quality and contribution. I recommend rejection.

Decision: Reject