Model Extrapolation Expedites Alignment

## Abstract

Abstract The open-source community is experiencing a surge in the release of large language models (LLMs) that are trained to follow instructions and align with human preference.
However, further training to improve them still requires expensive computational resources and data annotations. Is it possible to bypass additional training and cost-effectively acquire better-aligned models? Inspired by the literature on model interpolation , we propose a simple method called ExPO to boost LLMs’ alignment with human preference.
Utilizing a model that has undergone alignment training (e.g., via DPO or RLHF) and its initial SFT checkpoint, ExPO directly obtains a better-aligned model by ex tra po lating from the weights of the initial and the aligned models, which implicitly optimizes the alignment objective via first-order approximation. Through experiments with twelve open-source LLMs on HuggingFace, we demonstrate that ExPO consistently improves off-the-shelf DPO/RLHF models, as evaluated on the mainstream LLM benchmarks AlpacaEval 2.0 and MT-Bench.
Moreover, ExPO exhibits remarkable scalability across various model sizes (from 1.8B to 70B) and capabilities.
Through controlled experiments and further empirical analyses, we shed light on the essence of ExPO amplifying the reward signal learned during alignment training.
Our work demonstrates the efficacy of model extrapolation in expediting the alignment of LLMs with human preference, suggesting a promising direction for future research.

## 1 Introduction

Over the past year, the open-source community has witnessed explosive growth in large language models (LLMs).
These powerful LLMs, typically with billions of parameters, are trained to follow instructions and align with human preference [, , ].
Although the open weights of LLMs facilitate out-of-the-box use, further training to improve their performance usually requires expensive computational resources and additional data annotations.
Is it possible to bypass additional training and cost-effectively acquire better-aligned models?

We draw inspiration from the literature on model interpolation, also known as model/weight averaging.
This technique merges different models fine-tuned from the same base model by interpolating between their weights [, , ], relying on the mode connectivity of neural networks [, ].
Previous work observes that while model interpolation can integrate the respective strengths of different models to improve out-of-distribution generalization, it usually results in in-between performance compared to the original ones [, , ].
We similarly observe this phenomenon when interpolating between a supervised fine-tuned (SFT) model and a model further trained by direct preference optimization (DPO) [] or reinforcement learning from human feedback (RLHF) [], as shown in Figure 2.

Figure: Figure 2: Calculating the reward scores (§ 3.1) on the UltraFeedback [] development set, we observe that model interpolation usually gives trade-off performance between the two original models (e.g., an SFT model and a further-trained DPO/RLHF model). This observation motivates our proposal of ExPO, which cheaply acquires a better-aligned (stronger) model from a DPO/RLHF model and its initial SFT checkpoint (i.e., two relatively weaker models) via model extrapolation.

Intrigued by the observations of model interpolation, we turn to another compelling but unexplored direction:
What if we consider a DPO/RLHF model as the interpolated result from the initial SFT model and a hypothetically better-aligned model?
If this hypothetical model exists, we can straightforwardly obtain its weights by reversely extrapolating from the weights of the SFT and the DPO/RLHF models, as indicated by the gray arrow in Figure 2.
This can potentially further improve many off-the-shelf DPO/RLHF-aligned LLMs without any additional training.

Building upon the above assumption, we propose a simple method called ExPO (model extrapolation) to boost LLMs’ alignment with human preference (§ 2).
Utilizing a model $\mathcal{M}_{1}$ that has undergone alignment training (e.g., via DPO or RLHF) and the SFT model $\mathcal{M}_{0}$ that initializes $\mathcal{M}_{1}$, ExPO directly extrapolates a better-aligned (stronger) model $\mathcal{M}_{2}$ from the weights of the two relatively weaker models $\mathcal{M}_{1}$ and $\mathcal{M}_{0}$.

Despite its simplicity, we demonstrate the impressive efficacy of ExPO through extensive empirical experiments, as summarized in Figure 1.
Through experiments with twelve open-source LLMs on HuggingFace, we show that ExPO consistently improves off-the-shelf DPO/RLHF models, by up to 4.5% on AlpacaEval 2.0 [] and 0.66 on MT-Bench [] (§ 3).
Moreover, ExPO also manifests remarkable scalability across various model sizes (from 1.8B to 70B) and capabilities.
We further conduct controlled experiments to shed light on how ExPO amplifies the reward signal learned during $\mathcal{M}_{1}$’s alignment training, where we show that ExPO can boost models trained with less preference data (e.g., 10% or 20%) to compete and even outperform the fully-trained one (§ 4).
Our work demonstrates model extrapolation as a promising method for expediting the alignment of LLMs with human preference, and we believe it deserves more exploration in future research.

## 2 Methodology

### 2.1 Overview

Our proposed ExPO method is inspired by the observation in Figure 2 and the mode connectivity of neural networks [, , ].
Formally, we denote that a language model $\mathcal{M}_{1}$ (parameterized by $\bm{\theta}_{1}$) has undergone training for human preference alignment (e.g., via DPO [] or RLHF []).
We denote its corresponding SFT checkpoint as $\mathcal{M}_{0}$ (parameterized by $\bm{\theta}_{0}$), which is used for initializing $\mathcal{M}_{1}$.
We denote the model’s parameter space as $\bm{\Theta}$ and suppose that the alignment level can be quantified by a continuous scalar function $\Omega:\bm{\Theta}\to\mathbb{R}$, where higher $\Omega(\bm{\theta})$ indicates better alignment with human preference.
ExPO assumes that there exists a better-aligned model $\mathcal{M}_{2}$ (parameterized by $\bm{\theta}_{2}$) that satisfies $\Omega(\bm{\theta}_{0})<\Omega(\bm{\theta}_{1})<\Omega(\bm{\theta}_{2})$, and an interpolation coefficient $\gamma\in[0,1]$ such that $\bm{\theta}_{1}=(1-\gamma)\bm{\theta}_{0}+\gamma\bm{\theta}_{2}$.
Here, we consider the simplest form of uniform linear interpolation, as we find it can already work well.
With the substitution of $\alpha=1/\gamma-1\in[0,+\infty)$, ExPO obtains the assumed better-aligned (stronger) model $\mathcal{M}_{2}$ by extrapolating from the weights of the two (relatively weaker) models $\mathcal{M}_{1}$ and $\mathcal{M}_{0}$ (i.e., weak-to-strong extrapolation), formulated as follows:

$$ $\displaystyle\bm{\theta}_{2}=(1+\alpha)\bm{\theta}_{1}-\alpha\bm{\theta}_{0}= \bm{\theta}_{1}+\alpha(\bm{\theta}_{1}-\bm{\theta}_{0})=\bm{\theta}_{1}+{ \color[rgb]{1.0,0.4980392156862745,0.054901960784313725}\alpha\Delta\bm{\theta }},$ (1) $$

where the coefficient $\alpha$ serves as the hyperparameter that controls the extrapolation length.
In practice, $\alpha$ can be easily tuned as a decoding hyperparameter (similar to the sampling temperature).
This requires only one 24GB GPU for 7B LLMs (e.g., half-precision inference with vllm []), which, however, is far from sufficient for model training.

### 2.2 Explanation and Insights

Figure: Figure 3: Illustrative 1D diagram of $\Omega(\bm{\theta})$. ExPO can be viewed as a “global gradient update” along ${\color[rgb]{0.12156862745098039,0.4666666666666667,0.7058823529411765}\Delta \bm{\theta}}$. It essentially amplifies the reward signal learned during alignment training.

Theoretically, ExPO takes first-order approximation to implicitly optimize the alignment objective $\Omega(\bm{\theta})$.
Note that alignment algorithms typically include the regularization term (e.g., the KL constraint in RLHF) that restricts $\bm{\theta}_{1}$ within the small vicinity of $\bm{\theta}_{0}$ (i.e., $|\Delta\bm{\theta}|$ is small), and we can also control $\alpha$ such that $|\alpha\Delta\bm{\theta}|\ll|\bm{\theta}_{1}|$.
We can apply first-order Taylor Expansion and have:

$$ $\displaystyle\Omega(\bm{\theta}_{1}+\alpha\Delta\bm{\theta})\approx\Omega(\bm{ \theta}_{1})+\alpha\nabla\Omega(\bm{\theta}_{1})\cdot\Delta\bm{\theta}.$ (2) $$

Therefore, $\Omega(\bm{\theta}_{2})=\Omega(\bm{\theta}_{1}+\alpha\Delta\bm{\theta})>\Omega
(\bm{\theta}_{1})$ holds if the gradient of $\Omega$ at $\bm{\theta}_{1}$ has a positive component along $\Delta\bm{\theta}$ (as long as $\Omega$ is not locally maximum at $\bm{\theta}_{1}$).
This can generally be satisfied, as we can reasonably assume $\Omega$ to monotonically increase from $\bm{\theta}_{0}$ to $\bm{\theta}_{1}$ during alignment training, as illustrated in Figure 3.

The above assumption also implies that the alignment training from $\mathcal{M}_{0}$ to $\mathcal{M}_{1}$ is suboptimal, as shown in Figure 3.
We conjecture this is very likely to occur in practice due to both the regularization in alignment algorithms (e.g., the KL constraint in RLHF) and the sparsity of reward signal.
For the latter, since it is intractable to directly optimize $\Omega(\bm{\theta})$, the dominant practice is to first employ another reward model to assign preference labels (e.g., in DPO) or reward values (e.g., in RLHF) to the language model $\mathcal{M}$’s outputs, on which we then train $\mathcal{M}$.
However, transmitting the reward signal through the intermediate discrete, textual outputs can make the learned reward signal noisy or sparse, which consequently hinders the optimal alignment training.
We will show in § 3 that the open-source DPO/RLHF models usually have significant room for further improvement by ExPO.

Additionally, Figure 3 provides a more intuitive illustration of ExPO.
Specifically, ExPO can be viewed as a “global gradient update” along the weight change ${\color[rgb]{0.12156862745098039,0.4666666666666667,0.7058823529411765}\Delta
\bm{\theta}}$.
Note that starting from $\bm{\theta}_{0}$, $\Delta\bm{\theta}$ indicates a direction in the parameter space in which the alignment level $\Omega$ with human preference increases.
Therefore, ExPO essentially amplifies the learned reward signal through the extrapolation ${\color[rgb]{1.0,0.4980392156862745,0.054901960784313725}\alpha\Delta\bm{
\theta}}$.
This insight underscores the importance of the “quality” of $\Delta\bm{\theta}$, i.e., $\Delta\bm{\theta}$ should indicate a direction that truly improves the alignment with human preference.
Otherwise, ExPO could also amplify the learned spurious features in $\Delta\bm{\theta}$.
We will provide more empirical analyses in § 4 to show that the “quality” of $\Delta\bm{\theta}$ can vary depending on the training configuration for $\mathcal{M}_{1}$.

### 2.3 Highlights

We underline the following appealing properties of ExPO:

- •
Simplicity:
ExPO is extremely simple and quick to implement.
It merely involves performing extrapolation based on the weights of two checkpoints $\mathcal{M}_{0}$ and $\mathcal{M}_{1}$, which can be implemented within just a few lines of code.
- •
Efficiency:
ExPO does not need any additional model training.
The only variable $\alpha$ is efficient to tune as a decoding hyperparameter, which requires much fewer computational resources than model training (e.g., one 24GB GPU is enough for 7B LLMs).
Moreover, we believe more efficient means of hyperparameter search can be developed in future work, as evidenced by the advances in adaptive model interpolation [, ].
- •
Scalability:
ExPO is, in principle, applicable to various LLMs, including those of large sizes and those trained by advanced alignment algorithms like iterative DPO [, ].
We will show in § 3 that ExPO can improve off-the-shelf models across various sizes and capabilities.

## 3 ExPO Improves Off-the-shelf Models

In this section, we demonstrate the impressive efficacy of ExPO in improving off-the-shelf LLMs from HuggingFace, utilizing their SFT and DPO/RLHF checkpoints.
We particularly underscore the scalability of ExPO across different model sizes and capabilities.

### 3.1 Experimental Setup

#### Models

When selecting open-source LLMs for experiments, we found that many well-known LLMs, such as LLaMA-2/3 [, ], Gemma [], and Qwen [], only release the final DPO/RLHF checkpoints but not the corresponding SFT ones.
Such an opacity hinders the feasibility of experimenting with these more representative models.
To facilitate reproducible research, we select the following twelve open-source DPO/RLHF models on HuggingFace (see Appendix B for the full list of their model IDs) that (1) have publicly accessible SFT checkpoints, (2) have disclosed the training details, and (3) are popularly downloaded:

- •
zephyr-7b-alpha/beta [], two Mistral-based [] models developed by HuggingFace.
They are initialized from different SFT checkpoints and trained via DPO on UltraFeedback [].
- •
starling-7b-alpha/beta [], two Mistral-based models.
They are initialized from different SFT versions of the OpenChat model [] and trained via the RLHF algorithm.
- •
snorkel-7b [], a Mistral-based model.
It is initialized from the official SFT Mistral model and trained via the iterative DPO algorithm [] on the instructions of UltraFeedback.
- •
llama3-8b-iter [], a LLaMA-3-based [] model developed by Salesforce.
It is trained via iterative DPO on open-source datasets.
- •
internlm2-1.8/7/20b [], a Chinese-English bilingual model suite developed by Shanghai AI Laboratory.
The three-sized models undergo the same SFT training and similar online RLHF training processes.
- •
tulu-2-dpo-7/13/70b [], a LLaMA-2-based model suite developed by the Allen Institute for AI.
The three-sized models undergo the same SFT and DPO training processes.

We decide the optimal $\alpha$ in ExPO from [0.1, 0.2, 0.3, 0.4, 0.5] based on the model performance on the instructions of the UltraFeedback(^1^11[https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized)) [] development set.
The performance is measured by the expected reward score calculated by an open-source reward model(^2^22[https://huggingface.co/weqweasdas/RM-Mistral-7B](https://huggingface.co/weqweasdas/RM-Mistral-7B)).
It ranks among the top on RewardBench(^3^33[https://huggingface.co/spaces/allenai/reward-bench](https://huggingface.co/spaces/allenai/reward-bench)) [], a leaderboard that assesses the performance of reward models.
This reward model is also not involved in either preference annotation or RLHF training of all the models we experiment with in this work, thus reducing the risk of reward hacking.

#### Benchmarks

We employ three mainstream LLM benchmarks for evaluation:

- •
AlpacaEval 2.0 [], a leading benchmark that assesses LLMs’ instruction-following ability and the alignment with human preference.
It calculates the probability that an LLM-based evaluator (gpt-4-1106-preview) prefers the model’s output over the GPT-4 baseline, which provides an affordable and replicable alternative to human preference annotation.
The win rate over the GPT-4 baseline is computed as the expected preference probability.
Recently, AlpacaEval 2.0 has introduced the new length-controlled (LC) win rate metric [], which alleviates the length bias of the GPT-4 evaluator (i.e., the prior preference toward longer responses) [].
According to [], the LC win rate metric currently has the highest correlation (a Spearman correlation of 0.98) with the real-world human evaluation on Chatbot Arena [].
- •
MT-Bench [], another leading benchmark for assessing chat LLMs’ general and multi-turn ability.
It contains a set of challenging multi-turn open-ended questions covering topics such as writing, role-playing, math, coding, and more.
The model-generated answers are judged by gpt-4 via a scalar score (from 1 to 10), without any pairwise comparison.
- •
Open LLM Leaderboard [], a popular evaluation suite hosted by HuggingFace.
It consists of six benchmarks and assesses a variety of model abilities across commonsense reasoning [, ], math problem-solving [], human falsehood mimicking [], and general knowledge [, ].
We follow the official evaluation protocol [] and report the average scores on the six benchmarks, while the breakdowns are shown in Appendix D.

### 3.2 Results

**Table 1: AlpacaEval 2.0 (win rate and LC win rate) and MT-Bench evaluation results of off-the-shelf DPO/RLHF models. The gray models’ scores are copied from the official leaderboard for reference.**
|  | Original | + ExPO, no training |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | WR | LC WR | MT-B | Win Rate | LC Win Rate | MT-Bench |
| llama2-7b | 5.0% | 5.4% | 6.27 | - | - | - |
| llama2-70b | 13.9% | 14.7% | 6.86 | - | - | - |
| mistral-7b-v0.2 | 14.7% | 17.1% | 7.60 | - | - | - |
| claude-2.1 | 15.7% | 25.3% | 8.18 | - | - | - |
| gpt-4-0314 | 22.1% | 35.3% | 8.96 | - | - | - |
| zephyr-7b-alpha | 6.7% | 10.0% | 6.85 | 10.6% (+3.8%) | 13.6% (+3.6%) | 6.87 (+0.02) |
| zephyr-7b-beta | 10.2% | 13.2% | 7.02 | 11.1% (+0.9%) | 14.0% (+0.8%) | 7.06 (+0.04) |
| starling-7b-alpha | 15.0% | 18.3% | 7.82 | 18.2% (+3.2%) | 19.5% (+1.2%) | 7.91 (+0.09) |
| starling-7b-beta | 26.6% | 25.8% | 8.10 | 29.6% (+3.0%) | 26.4% (+0.7%) | 8.18 (+0.08) |
| snorkel-7b | 24.7% | 24.0% | 7.63 | 28.8% (+4.1%) | 26.4% (+2.4%) | 7.69 (+0.07) |
| llama3-8b-iter | 29.2% | 36.0% | 8.08 | 32.7% (+3.5%) | 37.8% (+1.8%) | 8.45 (+0.37) |
| internlm2-1.8b | 3.8% | 4.0% | 5.17 | 5.2% (+1.5%) | 4.3% (+0.3%) | 5.26 (+0.08) |
| internlm2-7b | 20.5% | 18.3% | 7.14 | 28.1% (+7.6%) | 22.7% (+4.4%) | 7.80 (+0.66) |
| internlm2-20b | 36.1% | 24.9% | 8.13 | 46.2% (+10.1%) | 27.2% (+2.4%) | 8.26 (+0.13) |
| tulu-2-dpo-7b | 8.5% | 10.2% | 6.35 | 11.5% (+3.0%) | 11.7% (+1.5%) | 6.38 (+0.03) |
| tulu-2-dpo-13b | 11.2% | 15.5% | 7.00 | 15.6% (+4.3%) | 17.6% (+2.1%) | 7.26 (+0.26) |
| tulu-2-dpo-70b | 15.4% | 21.2% | 7.79 | 23.0% (+7.6%) | 25.7% (+4.5%) | 8.03 (+0.24) |

Figure: Figure 4: Open LLM Leaderboard evaluation results of off-the-shelf DPO/RLHF models. We report the average scores over the six tasks. Breakdowns are shown in Appendix D.

In Table 1, we demonstrate that ExPO consistently enhances the evaluated LLMs, with increases of up to 10.1% basic win rate on AlpacaEval 2.0 (for internlm2-20b), 4.5% LC win rate (for tulu-2-dpo-70b), and 0.66 on MT-Bench (for internlm2-7b).
The improvements are made across LLMs of various sizes and capabilities, from the smallest internlm2-1.8b and the second weakest zephyr-7b-alpha, to the largest tulu-2-dpo-70b and the strongest llama3-8b-iter and starling-7b-beta, which demonstrates the remarkable scalability of ExPO.
In Figure 4, we also show that ExPO overall slightly improves the Open LLM Leaderboard scores, indicating that ExPO generally does not impact the base model’s capability.
Overall, our extensive evaluation suggests that most open-source LLMs have not been trained optimally for human preference alignment, while ExPO enables further improvements for them without any additional training.

## 4 Controlled Experiments and Analyses

In this section, we conduct controlled experiments to give more insights into ExPO, where we fix the same $\mathcal{M}_{0}$ and adopt varying training configurations for $\mathcal{M}_{1}$, including training data sizes and hyperparameters.
We also discuss the impact of model choices of $\mathcal{M}_{0}$ and $\mathcal{M}_{1}$ on the effectiveness of ExPO.
We underscore that ExPO amplifies the reward signal learned during alignment training, but it can also amplify the learned spurious features such as the length bias.

### 4.1 Experimental Setup

#### Models

We refer to the alignment handbook(^4^44[https://github.com/huggingface/alignment-handbook](https://github.com/huggingface/alignment-handbook)) [], a widely-used code base released by HuggingFace for alignment training of LLMs.
We adopt their recipe for training the zephyr-7b-sft and zephyr-7b-dpo models, which are popularly used for controlled experiments in recent LLM alignment research [, , ].
The recipe employs DPO for alignment training, where the SFT model zephyr-7b-sft is used as the reference model in DPO and also for initializing the policy models.
We adopt the same hyperparameter configuration (see Appendix E) and train all the models on 4 A100 80GB GPUs.
We use zephyr-7b-dpo as the fully-trained baseline (i.e., using 100% data).

#### Data

We use the same preference dataset UltraFeedback [] for alignment training.
It contains diverse instructions and response pairs with GPT-4-annotated preference labels and has been popularly used by the open-source community for training aligned LLMs [, , ].
The preprocessed version on HuggingFace contains 61K and 1K preference data in the training and development sets, respectively.
As in § 3, we search the optimal $\alpha$ in ExPO based on the performance on the instructions of the development set(^5^55 One may be concerned that the development set (1K data) is used to select $\alpha$ in ExPO but not involved in improving the baselines where ExPO is not applied, which may lead to unfair comparison. In Appendix F, we show that using the 1K development data to further train the baselines still largely underperforms ExPO.), as evaluated by the same open-source reward model.

### 4.2 Analysis of Training Data

**Table 2: AlpacaEval 2.0 evaluation results of models trained with varying sizes of preference data.**
|  | Original | + ExPO, no training |  |  |
| --- | --- | --- | --- | --- |
|  | Win Rate | LC Win Rate | Win Rate | LC Win Rate |
| SFT ($\mathcal{M}_{0}$) | 4.7% | 8.7% | - | - |
| DPO (init from $\mathcal{M}_{0}$, 100% data) | 14.7% | 17.3% | 18.0% (+3.3%) | 20.2% (+2.8%) |
| DPO (init from $\mathcal{M}_{0}$, 5% data) | 5.0% | 9.1% | 11.5% (+6.5%) | 14.7% (+5.6%) |
| DPO (init from $\mathcal{M}_{0}$, 10% data) | 5.9% | 10.4% | 17.9% (+12.0%) | 16.3% (+5.8%) |
| DPO (init from $\mathcal{M}_{0}$, 20% data) | 8.6% | 12.9% | 22.7% (+14.2%) | 21.3% (+8.4%) |
| DPO (init from $\mathcal{M}_{0}$, 40% data) | 12.1% | 14.6% | 17.7% (+5.6%) | 16.6% (+2.0%) |

Figure: Figure 5: Illustrative 2D contour diagram of $\Omega(\bm{\theta})$. The “quality” of $\Delta\bm{\theta}$ and the effectiveness of ExPO can vary depending on the training configurations for $\mathcal{M}_{1}$. Here, $\Delta\bm{\theta}$ indicates a superior extrapolation direction to $\Delta\bm{\theta}^{\prime}$.

We first study the impact of training data on the effectiveness of ExPO.
We train multiple $\mathcal{M}_{1}$ from the same initial $\mathcal{M}_{0}$ (i.e., zephyr-7b-sft), but with varying data sizes (from 5% to 40%).
In Table 2, we show their performance as well as the results of further applying ExPO to them.
While training with less preference data usually results in lower-tier performance, ExPO boosts the performance to compete (10% data, 16.3%) and even surpass (20% data, 21.3%) the fully-trained model (17.3%).
We also observe that the model trained with 20% data obtains a larger improvement than other data proportions.
It implies that the former gives a superior extrapolation direction $\Delta\bm{\theta}$ (i.e., of a higher “quality”), as illustrated in Figure 5.

However, the “quality” of $\Delta\bm{\theta}$ is not simply correlated with the amount of data.
As shown in Table 2, using 20% data slightly outperforms using 100% data when both applying ExPO (21.3% vs. 20.2%), while the gain from ExPO decreases when the used data increases to 40%.
In Figure 6, we present the reward scores and output lengths on the UltraFeedback development set versus varying $\alpha$ values.
From the left part, we observe that the global optimal reward score (6.08) achieved by ExPO is obtained with a medium size (20%) of training data, rather than the smaller (5% or 10%) or larger (40%) ones.
For the former (5% and 10% data), although ExPO still notably improves the performance (from the reward score 3.13 to 4.79, and 3.59 to 5.82, respectively), the limited data still cannot provide an accurate $\Delta\bm{\theta}$, thus capping the improvement after model extrapolation.
For the latter (40% data), we speculate that the model has learned the spurious features within the training data as shortcuts, especially the length bias(^6^66 The average lengths of the preferred and unpreferred responses in the UltraFeedback training set are 319 and 277 tokens, respectively.) [] where the preferred responses are usually longer.
As shown in the right part of Figure 6, for the model trained with 40% data, using a very small $\alpha$ results in a dramatic increase in the output length.
However, this does not lead to sustained improvement in performance, where the optimal rewards typically correspond to moderate output lengths ranging between 500 and 600.

Figure: Figure 6: We train multiple $\mathcal{M}_{1}$ from the same initial $\mathcal{M}_{0}$, but with varying data sizes. We plot the reward scores (left) and output lengths (right) on the instructions of the UltraFeedback development set versus varying $\alpha$ values (x-axis). Note that $\alpha=0$ indicates that ExPO is not applied.

### 4.3 Analysis of Training Hyperparameters

Figure: Figure 7: We train multiple $\mathcal{M}_{1}$ from the same initial $\mathcal{M}_{0}$ using the same 20% preference data, but with larger learning rates or for more epochs. We plot the reward scores (left) and output lengths (right) on the instructions of the UltraFeedback development set versus varying $\alpha$ values (x-axis). Note that $\alpha=0$ indicates that ExPO is not applied.

As ExPO can be viewed as a “global gradient update” (§ 2.2), we also compare with simply tuning the training hyperparameters.
We use the same 20% training data but increase the learning rate or training epochs, and train multiple $\mathcal{M}_{1}$ from the same initial $\mathcal{M}_{0}$.
From the left part of Figure 7, we observe that increasing the learning rate or training epochs indeed somewhat improves the original reward score.
However, it is still inferior to the optimal reward score achieved by ExPO under the default configuration, and also notably impairs the gains from ExPO (the peak points are lower than that of the default configuration).
This is probably because the model is overfitted to the training data and similarly learns the spurious features (such as the length bias), thus failing to produce an accurate $\Delta\bm{\theta}$.
The overfitting issue can also be evidenced by the right part of Figure 7.
The models trained with larger learning rates or for more epochs become prone to generating longer outputs with a small $\alpha$, but do not obtain noticeable reward improvement (the left part of Figure 7).
This suggests that $\Delta\bm{\theta}$ is very likely to contain the spurious length feature rather than the true human preference.

### 4.4 Discussion on Model Choices

Finally, we discuss the impact of model choices for $\mathcal{M}_{0}$ and $\mathcal{M}_{1}$ on the effectiveness of ExPO.
In the experiments so far, we choose $\mathcal{M}_{0}$ as an SFT model and $\mathcal{M}_{1}$ as the model further trained for human preference alignment on top of $\mathcal{M}_{0}$.
Can other types of model combination $\mathcal{M}_{0}$ and $\mathcal{M}_{1}$, such as a Base and an SFT model, or two separately-trained RLHF models, be able to produce meaningful extrapolated models?
We experiment with the following types of combinations:

- (1)
Base + SFT: mistral-7b-v0.1 [] as $\mathcal{M}_{0}$ and mistral-7b-instruct-v0.1 as $\mathcal{M}_{1}$.
- (2)
SFT 1 + SFT 2 (trained from different base models): mistral-7b-instruct-v0.1 as $\mathcal{M}_{0}$ and mistral-7b-instruct-v0.2 as $\mathcal{M}_{1}$.
- (3)
SFT 1 + SFT 2 (same base): openchat-3.5 [] as $\mathcal{M}_{0}$ and openchat-3.5-0106 as $\mathcal{M}_{1}$.
- (4)
RLHF 1 + RLHF 2 (same base): gemma-7b-it [] as $\mathcal{M}_{0}$ and gemma-1.1-7b-it as $\mathcal{M}_{1}$.
Note that it is not disclosed whether the two models are initialized from the same SFT model.

Figure: Figure 8: Reward scores of different model combinations on the instructions of the UltraFeedback development set, with $\alpha$ (x-axis) varying from 0.1 to 0.5.

From Figure 9,
(1) we find that extrapolating from two SFT models that are initialized from different base models can easily lead to the model collapse, due to that they do not satisfy the mode connectivity [, ],
(2) For the combination of Base and SFT, extrapolation degrades the performance, probably because training from Base to SFT does not naturally optimize for human preference and increase the alignment level $\Omega$.
This is exactly why we need additional training for human preference alignment.
(3&4) For two separately-trained SFT or RLHF models, we find that they also fail to benefit from model extrapolation.
We speculate that this occurs because when $\mathcal{M}_{1}$ is not initialized from $\mathcal{M}_{0}$, the alignment level $\Omega$ does not monotonously increase along the path in the parameter space from $\bm{\theta}_{0}$ to $\bm{\theta}_{1}$.
Instead, $\Omega$ may first reach an intermediate peak point and then decrease, as illustrated in Figure 9.
Therefore, $\Delta\bm{\theta}$ fails to indicate a direction in which the reward signal can be amplified, even if the alignment level $\Omega(\bm{\theta}_{1})$ is higher than $\Omega(\bm{\theta}_{0})$.
Overall, our method ExPO is generally applicable to the combination of an SFT model $\mathcal{M}_{0}$ and a model $\mathcal{M}_{1}$ further trained on top of the former, which is a very realistic combination choice, as modern LLMs that are trained to align with human preference are almost all initialized from their SFT checkpoints.

## 5 Related Work

#### LLM Alignment

Modern LLMs are typically first pre-trained on massive textual corpora (resulting in a Base model) [, , ] and then trained to align with human expectations [, , ].
The alignment process generally contains two stages.
In the first stage, an LLM is supervisedly fine-tuned (SFT) on demonstration outputs and learns to follow human instructions [, , ].
In the second stage, the LLM is trained to learn human preference and assign higher probabilities to human-preferred outputs over the disfavored ones.
This is usually implemented in the fashion of reinforcement learning (RL) [, ] or contrastive learning [, , ], as exemplified by the reinforcement learning from human feedback (RLHF) [] and direct preference optimization (DPO) [] algorithms, respectively.
However, as the model size increases (from 7B, 13B to 70B or larger), the computational resources required for alignment training also become extremely expensive.
For instance, training a 7B model via DPO has commonly required 4 or 8 A100 80GB GPUs, which can be unaffordable for open-source community users.
Our work proposes the ExPO method to boost the alignment of LLMs with human preference in a simple, efficient, and scalable manner.

#### Model Merging and Interpolation

Model merging is a recent focal technique for building powerful LLMs based on existing ones [, , ].
It aims to integrate multiple models fine-tuned from the same base model into a unified one that retains the respective strengths.
The simplest form of model merging is model interpolation, also known as model/weight averaging [, , , ], which builds upon the mode connectivity of neural networks [, ].
Our work is inspired by the phenomenon that interpolation usually results in in-between performance compared to the original models, as observed in previous literature [, , ] and our experiments in Figure 2.
The proposed ExPO method has a similar idea of blending model weights, but works under a distinct premise and goal.
Rather than integrating the strengths of multiple strong models, ExPO starts from two relatively weaker models and aims to produce an overall stronger one.

There is another line of work that improves text generation by blending the token prediction distributions of multiple language models during the inference time [, , ].
They share somewhat similar forms to model merging, but operate on output logits rather than model weights.
Besides, they can suffer from decreased generation efficiency due to the interference with the inference process, and the increased exposure bias of different models.
Our proposed ExPO method, as well as the work in model merging, bypasses these issues by producing a new single model.

## 6 Conclusion

We present ExPO, a simple method to boost LLMs’ alignment with human preference.
By extrapolating from the weights of an aligned model and its initial SFT checkpoint, ExPO enables directly obtaining a better-aligned model without any additional training.
We demonstrate the efficacy of ExPO in enhancing open-source LLMs across various model sizes (from 1.8B to 70B) and capabilities, suggesting significant improvement room for most open-source models.
We also shed light on the essence of ExPO amplifying the reward signal learned in the alignment training through controlled experiments.
Given its simplicity, efficiency, and scalability, we recommend ExPO as a promising approach for expediting the alignment of LLMs with human preference, which we believe deserves more future exploration.

#### Limitations & Future Work

Our work is limited by the public accessibility to the SFT and DPO/RLHF checkpoints.
Thus unfortunately, we are unable to experiment with the more representative LLMs like LLaMA-2/3 [, ], Gemma [], and Qwen [].
We hope for more open-source efforts in increasing LLMs’ transparency and accessibility.
Outside the scope of our work, there are several problems that can potentially attract future research.
First, since ExPO is currently based on the simplest uniform linear extrapolation (Equation 1, using the same $\alpha$ for all the model modules), future work can devise methods to adaptively search optimal $\alpha$ for different model modules.
Second, although our work provides a basic explanation for ExPO (§ 2.2) and empirically demonstrates its effectiveness, future work can establish more profound theoretical foundations for its underlying mechanisms.
Third, while we currently rely on an external reward model for searching $\alpha$, future work may get rid of such reliance by resorting to the inherent capability of the models $\mathcal{M}_{1}$ and $\mathcal{M}_{0}$ themselves.
Finally, it would also be interesting to apply ExPO to multi-modal LLMs like LLaVA [] and other model architectures like Mamba [].

## Appendix A Broader Impacts and Safeguards

Our work aims to improve the alignment of LLMs with human preference when they follow human instructions.
It can facilitate the development of more helpful AI assistants.
On the other hand, the increased utility of LLMs also faces risks of dual use.
For instance, they may be asked to assist with malicious queries like falsifying information or causing damages.
For real-world deployment, it is essential that LLMs undergo additional safety training and learn to recognize and refuse harmful queries [, ].
Furthermore, they should be equipped with necessary moderation mechanisms, such as safeguard classifiers or guardrail prompts [, , ].

## Appendix B Open-Source Models Used in This Work

| Model | HuggingFace Model ID |
| --- | --- |
| reward model | [weqweasdas/RM-Mistral-7B](weqweasdas/RM-Mistral-7B) |
| mistral-7b-sft-alpha | [HuggingFaceH4/mistral-7b-sft-alpha](HuggingFaceH4/mistral-7b-sft-alpha) |
| zephyr-7b-alpha | [HuggingFaceH4/zephyr-7b-alpha](HuggingFaceH4/zephyr-7b-alpha) |
| mistral-7b-sft-beta | [HuggingFaceH4/mistral-7b-sft-beta](HuggingFaceH4/mistral-7b-sft-beta) |
| zephyr-7b-beta | [HuggingFaceH4/zephyr-7b-beta](HuggingFaceH4/zephyr-7b-beta) |
| openchat-3.5 | [openchat/openchat_3.5](openchat/openchat_3.5) |
| starling-7b-alpha | [berkeley-nest/Starling-LM-7B-alpha](berkeley-nest/Starling-LM-7B-alpha) |
| openchat-3.5-0106 | [openchat/openchat-3.5-0106](openchat/openchat-3.5-0106) |
| starling-7b-beta | [Nexusflow/Starling-LM-7B-beta](Nexusflow/Starling-LM-7B-beta) |
| mistral-7b-instruct-v0.2 | [mistralai/Mistral-7B-Instruct-v0.2](mistralai/Mistral-7B-Instruct-v0.2) |
| snorkel-7b | [snorkelai/Snorkel-Mistral-PairRM-DPO](snorkelai/Snorkel-Mistral-PairRM-DPO) |
| llama3-8b-sft | [RLHFlow/LLaMA3-SFT](RLHFlow/LLaMA3-SFT) |
| llama3-8b-iter | [RLHFlow/LLaMA3-iterative-DPO-final](RLHFlow/LLaMA3-iterative-DPO-final) |
| internlm2-1.8b-sft | [intermlm/internlm2-chat-1_8b-sft](intermlm/internlm2-chat-1_8b-sft) |
| internlm2-1.8b | [intermlm/internlm2-chat-1_8b](intermlm/internlm2-chat-1_8b) |
| internlm2-7b-sft | [intermlm/internlm2-chat-7b-sft](intermlm/internlm2-chat-7b-sft) |
| internlm2-7b | [intermlm/internlm2-chat-7b](intermlm/internlm2-chat-7b) |
| internlm2-20b-sft | [intermlm/internlm2-chat-20b-sft](intermlm/internlm2-chat-20b-sft) |
| internlm2-20b | [intermlm/internlm2-chat-20b](intermlm/internlm2-chat-20b) |
| tulu-2-7b | [allenai/tulu-2-7b](allenai/tulu-2-7b) |
| tulu-2-dpo-7b | [allenai/tulu-2-dpo-7b](allenai/tulu-2-dpo-7b) |
| tulu-2-13b | [allenai/tulu-2-13b](allenai/tulu-2-13b) |
| tulu-2-dpo-13b | [allenai/tulu-2-dpo-13b](allenai/tulu-2-dpo-13b) |
| tulu-2-70b | [allenai/tulu-2-70b](allenai/tulu-2-70b) |
| tulu-2-dpo-70b | [allenai/tulu-2-dpo-70b](allenai/tulu-2-dpo-70b) |
| zephyr-7b-sft | [alignment-handbook/zephyr-7b-sft-full](alignment-handbook/zephyr-7b-sft-full) |
| zephyr-7b-dpo | [alignment-handbook/zephyr-7b-dpo-full](alignment-handbook/zephyr-7b-dpo-full) |
| mistral-7b-v0.1 | [mistralai/Mistral-7B-v0.1](mistralai/Mistral-7B-v0.1) |
| mistral-7b-instruct-v0.1 | [mistralai/Mistral-7B-Instruct-v0.1](mistralai/Mistral-7B-Instruct-v0.1) |
| gemma-7b-it | [google/gemma-7b-it](google/gemma-7b-it) |
| gemma-1.1-7b-it | [google/gemma-1.1-7b-it](google/gemma-1.1-7b-it) |

## Appendix C Calculated Reward Histograms on the AlpacaEval 2.0 Instructions

For the DPO/RLHF models in § 3, we draw their reward distributions on the AplacaEval 2.0 Instructions, which are calculated by the aforementioned reward model.
As shown below, ExPO generally shifts the distribution toward the higher-reward direction (i.e., the right-hand direction in the figures).

## Appendix D Breakdowns of Open LLM Leaderboard Evaluation Results

|  | ARC | HellaSwag | MMLU | GSM8K | Winogrande | TruthfulQA |
| --- | --- | --- | --- | --- | --- | --- |
| zephyr-7b-alpha | 61.0 | 84.0 | 61.4 | 14.0 | 78.6 | 57.9 |
| + ExPO | 60.8 | 84.3 | 60.6 | 28.3 | 78.1 | 60.9 |
| zephyr-7b-beta | 62.0 | 84.5 | 61.1 | 11.4 | 78.1 | 57.4 |
| + ExPO | 62.3 | 84.5 | 61.0 | 27.3 | 77.7 | 58.3 |
| starling-7b-alpha | 63.7 | 84.9 | 64.7 | 62.3 | 80.4 | 46.3 |
| + ExPO | 63.9 | 84.8 | 64.6 | 61.6 | 80.4 | 46.4 |
| starling-7b-beta | 67.2 | 83.5 | 65.1 | 66.6 | 81.3 | 55.5 |
| + ExPO | 67.9 | 83.6 | 65.3 | 65.7 | 81.4 | 57.2 |
| snorkel-7b | 66.1 | 85.6 | 60.7 | 36.1 | 76.5 | 69.6 |
| + ExPO | 66.3 | 85.7 | 60.9 | 34.8 | 76.4 | 69.8 |
| llama3-8b-iter | 64.8 | 83.8 | 66.4 | 67.3 | 79.2 | 62.2 |
| + ExPO | 66.0 | 84.2 | 66.3 | 59.6 | 79.3 | 64.0 |
| internlm2-1.8b | 43.1 | 60.5 | 46.9 | 30.4 | 62.8 | 42.2 |
| + ExPO | 42.5 | 60.1 | 46.6 | 31.2 | 63.0 | 42.4 |
| internlm2-7b | 57.9 | 78.8 | 58.4 | 27.1 | 72.6 | 56.6 |
| + ExPO | 57.8 | 78.7 | 57.9 | 30.5 | 72.5 | 58.3 |
| internlm2-20b | 62.7 | 82.5 | 66.4 | 61.3 | 79.7 | 54.8 |
| + ExPO | 62.7 | 82.5 | 66.1 | 62.8 | 79.6 | 56.3 |
| tulu-2-dpo-7b | 57.2 | 81.0 | 52.0 | 27.3 | 74.0 | 55.9 |
| + ExPO | 58.0 | 81.3 | 52.0 | 26.7 | 74.7 | 59.6 |
| tulu-2-dpo-13b | 61.5 | 84.6 | 57.7 | 38.3 | 77.5 | 59.0 |
| + ExPO | 62.7 | 85.1 | 57.5 | 38.8 | 77.9 | 63.7 |
| tulu-2-dpo-70b | 72.1 | 89.0 | 69.8 | 62.6 | 83.3 | 65.8 |
| + ExPO | 72.7 | 89.3 | 69.6 | 59.4 | 83.2 | 70.0 |

## Appendix E Implementation Details

For response generation in § 3 and 4, we employ the vllm [] library for high-throughput inference.
We use top-$k$ ($k=40$) and nucleus sampling [] ($p=0.9$) with a temperature of 0.7.
To avoid repetition in generated texts, we set both the factors of presence penalty and frequency penalty to 0.1.
We adopt the same decoding hyperparameters with the sampling random seed set to 42 for all the evaluated models across all the experiments, except in the evaluation of MT-Bench and Open LLM Leaderboard, as they have their own sets of decoding hyperparameters.

For model training in § 4, we adopt the global batch size 128 and gradient accumulation steps 4.
We train the models on 4 A100 80GB GPUs, with ZeRO-3 offload [] and gradient checkpointing for reducing GPU memory usage.
We set the learning rate to 5e-7, with the cosine scheduling and warmup ratio of 0.1, and use the AdamW [] optimizer to train the models for one epoch.
For DPO, we follow zephyr-7b-dpo and set $\beta$ to 0.01.

For hyperparameter search in § 3 and 4, we perform grid search on the values of $\alpha$.
We use the obtained model to generate responses on the UltraFeedback development set, score the responses with the reward model, and choose the optimal $\alpha$ corresponding to the highest average score.
We list below the search range and the optimal $\alpha$ in our experiments.

| DPO/RLHF Model | SFT Checkpoint | Search Range | Optimal $\alpha$ |
| --- | --- | --- | --- |
| zephyr-7b-alpha | mistral-7b-sft-alpha | [0.1, 0.2, 0.3, 0.4, 0.5] | 0.3 |
| zephyr-7b-beta | mistral-7b-sft-beta | [0.1, 0.2, 0.3, 0.5] | 0.1 |
| starling-7b-alpha | openchat-3.5 | [0.1, 0.2, 0.3, 0.5] | 0.2 |
| starling-7b-beta | openchat-3.5-0106 | [0.1, 0.3, 0.4, 0.5] | 0.5 |
| snorkel-7b | mistral-7b-instruct-v0.2 | [0.1, 0.2, 0.3, 0.4, 0.5] | 0.3 |
| llama3-8b-iter | llama3-8b-sft | [0.1, 0.2, 0.3, 0.4, 0.5] | 0.3 |
| internlm2-1.8b | internlm2-1.8b-sft | [0.1, 0.3, 0.4, 0.5] | 0.5 |
| internlm2-7b | internlm2-7b-sft | [0.1, 0.3, 0.4, 0.5] | 0.5 |
| internlm2-20b | internlm2-20b-sft | [0.1, 0.3, 0.4, 0.5] | 0.5 |
| tulu-2-dpo-7b | tulu-2-7b | [0.1, 0.3, 0.4, 0.5] | 0.5 |
| tulu-2-dpo-13b | tulu-2-13b | [0.1, 0.3, 0.4, 0.5] | 0.5 |
| tulu-2-dpo-70b | tulu-2-70b | [0.1, 0.3, 0.4, 0.5] | 0.5 |
| DPO (5% data) | zephyr-7b-sft | [5, 10, 20, 25, 30] | 25 |
| DPO (10% data) | zephyr-7b-sft | [2, 5, 7, 8, 9, 10] | 8 |
| DPO (20% data) | zephyr-7b-sft | [1.0, 2.0, 2.5, 3.0] | 2.5 |
| DPO (40% data) | zephyr-7b-sft | [0.2, 0.4, 0.5, 0.6] | 0.5 |
| zephyr-7b-dpo | zephyr-7b-sft | [0.1, 0.2, 0.3, 0.4, 0.5] | 0.3 |

## Appendix F Results of Further Training on the Development Set Data

In § 4, one may be concerned that the UltraFeedback development set (1K data) is used to select optimal $\alpha$ in ExPO but is not involved in improving the baselines where ExPO is not applied, which may lead to unfair comparison.
We thus further train these baselines on the 1K development data, and calculate the expected reward score on the development set.
Note that ExPO only uses the instructions of the development set, while the further training for baseline models uses both the instructions and preference labels.
In the table below, we show that further training on the development set still results in inferior performance to simply applying ExPO.

|  | Original Reward | + Training on Dev | + ExPO, no training |
| --- | --- | --- | --- |
| DPO (5% data) | 3.13 | 3.33 (+0.20) | 4.79 (+1.66) |
| DPO (10% data) | 3.59 | 3.73 (+0.14) | 5.82 (+2.23) |
| DPO (20% data) | 4.37 | 4.46 (+0.09) | 6.08 (+1.71) |
| DPO (40% data) | 5.30 | 5.33 (+0.03) | 5.80 (+0.50) |
| DPO (100% data) | 5.66 | 5.64 (-0.02) | 5.81 (+0.15) |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - More in-depth discussion of the method is necessary (Why does it work? When does it fail? etc.), For example:  it would be helpful to discuss why extrapolation can enhance alignment while being ineffective for general training.

- Theoretical discussion is missing: there is no theoretically evidence provided to support on the factors contributing to its effectiveness

- Analysis on the selection choice and importance of \alpha across various model sizes
soundness: 3
strengths: + The important task of efficiency of LLM alignment, and the interesting idea of extrapolating the LLM weights along the direction of the initial model toward the partially-trained one
+ Experiments are done on multimple open-source LLMs and bechmarks
summary: In this paper, the authors introduce EXPO, an efficient method for enhancing LLM alignment with human preferences. The core concept of EXPO is that the weights of a partially-trained model can serve as an intermediate result between its initial SFT checkpoint and a hypothetical, better-aligned model. By extrapolating the weights from the initial model toward the partially-trained one, EXPO can efficiently estimate the weights of a better aligned model without additional training steps. Experiments validate the improvement and data efficiency of this approach, showing consistent gains across various open-source LLMs of differing parameter sizes and multiple benchmarks.
weaknesses: - Lack of enough theoretical analysis
- Some parameters seem difficult to fix (e.g., \alpha) 
- English should be improved

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Given that α in EXPO operates over an open range [0,+∞), what strategies do the authors suggest for eﬀectively managing and narrowing down the search space for α?
2. The paper mentions that EXPO’s α search process takes less than 0.5 GPU hours, which appears eﬃcient. Could the authors provide a more detailed breakdown of how the search process unfolds? Specifically, how many steps or iterations are typically required, and what is the distribution of time spent in binary search versus grid search?
soundness: 3
strengths: 1.The paper provides theoretical explanation with comprehensive experiment results to show the eﬀectiveness of EXPO.
2. The paper presents thorough experimental results that span various model architectures and alignment techniques, highlighting EXPO’s flexibility and robustness.
3. The results suggest that EXPO could be beneficial, especially for LLMs, as it provides a computationally economical pathway to enhance alignment.
summary: The paper introduces Model Extrapolation (EXPO), a novel method designed to expedite the alignment process for large language models. EXPO harnesses model extrapolation to substantially reduce the training overhead associated with traditional alignment methods, such as Direct Preference Optimization (DPO) and Reinforcement Learning from Human Feedback (RLHF). These traditional methods typically require substantial computational resources to fine-tune models to align with human preferences. By extrapolating weights between a supervised fine-tuning checkpoint and an intermediate, partially trained DPO model, EXPO achieves better model alignment with significantly reduced training time. The method is validated through experiments that show improved alignment performance using only 20% of the training steps required by conventional methods. EXPO consistently demonstrates performance gains across multiple open-source large language models (LLMs) on benchmarks such as AlpacaEval 2.0 and MT-Bench.
weaknesses: 1. The core idea of EXPO seems to be a straightforward extension of existing model merge concept, primarily adjusting the interpolation parameter to a negative value (extrapolation). While the results are promising, this incremental shift from interpolation to extrapolation may be seen as lacking in true innovation.
2. Unlike interpolation, which typically operates within a bounded range [0,1], the extrapolation parameter α in EXPO operates within an open range [0, +∞). this open-ended range can complicate the search process, as the optimal value could vary widely depending on the model and training stage.
3. In addition to the open range above, the current manual search for α detracts from EXPO’s eﬃciency and scalability. The authors acknowledge this limitation and suggest future work on optimizing α automatically and adaptively. Having a better way to find the optimal α in this paper will make the innovation more solid.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 4
questions: The main issue that I'm looking to see resolved is in (1) above. The results of the paper are good, but there seems to be a mismatch between the theory and the method. It leaves me wondering if there's a different reason that the method works. See discussion in (1).
soundness: 1
strengths: - The paper studies a method for improving reward-fine-tuned models. This is an important problem in current LLM research.
- The paper is well-written and easy to understand. Figure 2 presents a clear indication of how the method works. Section 2 clearly presents the hypothesis, and the method is neatly summarized by equation 2.
- The results presented are significant. Table 1 demonstrates significant gains in Win Rate using the proposed method. Table 5 demonstrates that the results extend to other models. Table 6 demonstrates the algorithm extends to other alignment methods.
- The paper presents ablations to help understand the method. The ablation in Table 3 (extending the training regime or increasing the learning rate) is particularly important for understanding why the method differs in performance from simply extending training.
summary: The paper presents a method called EXPO for improving the accuracy of language models fine-tuned with DPO/RLHF. The intuition is that model interpolation can produce a model with accuracy in between the original model and the reward-fine-tuned model. So, model extrapolation should be able to produce a model with greater accuracy. By partially training a model, then performing extrapolation, a more accurate final model can be obtained with less compute. The method searches along the line defined by M0 (the initialization for reward-fine-tuning (e.g. the SFT model)) and M1 (the partially-reward-fine-tuned model) to find a more accurate model. The paper presents results for a variety of open-source models.
weaknesses: 1) To me, there seems to be a disconnect between the theory presented in Section 2 and the results presented. The results seem to point towards different conclusions that contradict the theory.
  a) Line 192-193: Extrapolation strongly improves the results obtained by "DPO 100%". This is not predicted by the theory presented. The theory presented essentially states, "we can partially train an RLHF model, then predict the result of fully training. The resulting predicted model will achieve the accuracy that would have been obtained by full training". Instead, the results demonstrate that the extrapolation method results in strong gains even when M1 is a fully trained model. (This seems at first like a sign that M1 is undertrained, but this theory is rebuked in Table 3).
  b) Based on the theory, the extrapolated model M2 should be most accurate if the search for M2 occurs along the line between M0 and the "fully trained" M1. This line is approximated by training a proxy (e.g. $M_1 ^{20\%}$) and searching along the line between M0 and the proxy. But for some reason, this proxy yields *better* results than searching the line between M0 and M1 (22.7% win rate compared to 18.0% win rate). Thus, a misalignment of the search space is beneficial to the model. This seems like a clear sign that something not predicted by the theory is causing the benefits in accuracy.
  c) Given the above observations, it seems like the true gains in accuracy could be coming from the fact that ExPO models are selected using a method that finds the best-performing model on the UltraFeedback dev set as calculated by another reward model. If so, this is an interesting discovery in itself, but it's not what the paper focuses on (and it's just a theory that could be completely wrong). My point here is, it seems like the theory presented doesn't match the given results, and there's some other reason for the gains in accuracy.

Line 225-229: The authors are trying to show that $|\alpha \delta \theta|$ is small. They argue that $\alpha$ decreases through training, then conclude that $|\alpha \delta \theta|$ is small. But they do not discuss the fact that $\delta \theta$ is (presumably) increasing during training. This justification seems incomplete/incorrect to me.

Section 3.4: The authors successfully demonstrate in this section that length bias can be an issue for the method. But this does not prove that length bias is the only issue causing reduced accuracy when training for longer. Also, shouldn't the same logic (length bias being an issue) apply to the standard training procedure? Yet, training for the full recipe produces optimal results, as opposed to training for a fraction of training steps. Why is length bias only an issue for ExPO?

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: I encourage the authors to address the points raised in the weaknesses section and to conduct additional experiments where further investigation is required.
soundness: 2
strengths: 1. The paper tackles an important problem—reducing the computational cost of aligning large language models—which is crucial for scaling models efficiently.

2. The explanation of the method is clear and easy to follow, with helpful figures that enhance understanding.

3. The authors show EXPO’s ability to cut down alignment training costs, which supports their claim.
summary: The paper introduces **EXPO** (model extrapolation), a method to improve the efficiency of aligning large language models (LLMs) with human preferences. Instead of fully training models, EXPO extrapolates model weights from a partially-trained model, significantly reducing training time while improving performance. Through experiments, the method shows that models trained with fewer steps can outperform fully-trained ones, validated on benchmarks like AlpacaEval 2.0 and MT-Bench. EXPO is applicable to a wide range of LLMs, making alignment more efficient and less computationally expensive.
weaknesses: 1. **Limited theoretical foundation**: The paper presents an interesting empirical finding with E X PO, but lacks a rigorous theoretical analysis of why it works. The discussion in Section 3.3 on why E X PO can work is somewhat speculative. A more formal theoretical treatment, perhaps drawing connections to optimization theory or analyzing the loss landscape, would strengthen the paper's contribution.

2. **Limited analysis of failure cases**: While some negative results are reported (e.g. for KTO algorithm in Table 6), there's little in-depth analysis of when and why E X PO fails. A more comprehensive error analysis would provide insights into the method's limitations.

3. **Limited analysis of extrapolation's impact on model calibration**: While the paper focuses on performance metrics, it doesn't investigate how E X PO affects the model's calibration. An analysis of how extrapolation impacts the model's uncertainty estimates and confidence calibration would be valuable, especially for safety considerations.

4. **Insufficient investigation of extrapolation instability**: The authors mention in Section 4.3 that extrapolating from pre-trained to SFT models can lead to model collapse, but they don't provide a rigorous analysis of this phenomenon.

5. **Lack of analysis on the impact of different λ values in DPO**: In Table 4, the authors test λ values of 0.001, 0.01, and 0.1, but don't provide a detailed analysis of how these different λ values affect the optimization landscape and consequently E X PO's performance. A more granular sweep of λ values and an analysis of how they interact with the optimal α could provide insights into the method's sensitivity to DPO hyperparameters.