Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models
Authors: Yi Liu
Dianqing Liu
Mingye Zhu
Junbo Guo
Yongdong Zhang
Zhendong Mao
Venue: NeurIPS 2025 poster ()
Keywords: Alignment of Large Language Models
Importance Sampling
Primary area: applications

Abstract: The widespread adoption of large language models (LLMs) across industries has increased the demand for high-quality and customizable outputs. However, traditional alignment methods often require retraining large pretrained models, making it difficult to quickly adapt and optimize LLMs for diverse applications. To address this limitation, we propose a novel \textit{Residual Alignment Model} (\textit{RAM}) that formalizes the alignment process as a type of importance sampling. In this framework, the unaligned upstream model serves as the proposal distribution, while the alignment process is framed as secondary sampling based on an autoregressive alignment module that acts as an estimator of the importance weights. This design enables a natural detachment of the alignment module from the target aligned model, improving flexibility and scalability. Based on this model, we derive an efficient sequence-level training strategy for the alignment module, which operates independently of the proposal module. Additionally, we develop a resampling algorithm with iterative token-level decoding to address the common first-token latency issue in comparable methods. Experimental evaluations on two leading open-source LLMs across diverse tasks, including instruction following, domain adaptation, and preference optimization, demonstrate that our approach consistently outperforms baseline models.

Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models
Liu
Dianqing Liu!.2 Mingye Zhu
Junbo Guo
Yongdong Zhang Zhendong Mao
1 State Key Laboratory of Communication Content Cognition, People's Daily Online University of Science and Technology of China fliuyi2023, guojunbo, [mingyezhu] ustc. edu. cn {zhyd73, zdmao Hustc. edu.
Abstract
The widespread adoption of large language models (LLMs) across industries has increased the demand for high-quality and customizable outputs. However, traditional alignment methods often require retraining large pretrained models, making it difficult to quickly adapt and optimize LLMs for diverse applications. To address this limitation, propose novel Residual Alignment Model (RAM) that formalizes the alignment process as a type of importance sampling. In this framework, the unaligned upstream model serves the proposal distribution, while the alignment process is framed as secondary sampling based on an autoregressive alignment module that acts an estimator the importance weights. This design enables a natural detachment of the alignment module from the target aligned model, improving flexibility and scalability. Based on this model, we derive an efficient sequence-level training strategy for the alignment module, which operates independently of the proposal module. Additionally, we develop resampling algorithm with iterative token-level decoding to address the common first-token latency issue in comparable methods. Experimental evaluations on two leading open-source LLMs across diverse tasks, including instruction following, domain adaptation, and preference optimization, demonstrate that our approach consistently outperforms baseline models.
Introduction
In recent years, the rapid advancement of large language models (LLMs)) has led to their widespread
adoption across various industrics (6] 35| M. Efforts to align LLM outputs with domain requirements
and human values enhance their utility and content safety 30 such as supervised learning
I1
24
29
2 17 45 23 Techniques
4k), preference optimization
30
24
and reinforcement
learning446 33, 44
crucial for achieving model alignment.
According to the scaling laws LLMs 19), increasing model size typically enhances their perfor- mance. However, research indicates that effective domain adaptation and value alignment can be achieved even with smaller models I0. This difference size requirements necessitates a balance between utility performance and alignment flexibility with respect to model size 38, 31, Moreover, training large models for specific domains
resource-intensive and requires the deployment of separate models, increasing resource costs and hindering traffic sharing across domains. Thus, there an urgent need for more efficient and economical model solutions.
Corresponding author: Zhendong Mao
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recent works 18 26 7 have introduced methods that fine-tum an adapter module on preference datasets learn correctional residuals between preferred and non-preferred responses, or supervised and synthetic examples, and then stacked onto the upstream model to achieve corrected alignment. While these approaches effectively decouple alignment from L.Mss during training, the correction based on the complete upstream response introduces significant latency for the first token during the inference phase, particularly for long content generation. Additionally, the Aligner model Pyyl'' a) [8] introduces reference response y for correction, which carries the extra potential risk of out- of-distribution (OOD) inputs, not only for the original question æ, but also for the reference y'. as further diseussed in Section 4
In this paper, we present novel Residual Aligument Model (RAM) that formalizes residual correction for alignment as type of importance sampling, which conditioned directly on x to generate y. In this framework, the unaligned upstream model is referred to as the Proposal Module, serves as the proposal distribution, while the alignment process is framed as secondary sampling based an autoregressive alignment module which acts as an estimator of the importance weights and termed the Residual Aligner. This linear combination of the Proposal Module and the Residual Aligner allows for the natural detachment of the alignment module from the target aligned model, illustrated as Equation T
Paligned (z) x PProposaiMochuteyy/aa) * PResichualAlignerr ( a+)
Building upon this framework, we propose an efficient training strategy that operates on the detached alignment module at the sentence level. The Proposal Module is required solely for one-off data synthesis in pointwise supervised datasets to create preference examples; contrast, it remains unused throughout the entire training process for preference datasets. Furthermore, develop token-level decoding algorithm with minimal first-word latency to ensure practicality during inference. The training and decoding strategy is illustrated on Figure
A: Here are the steps,
Sorry, can't assist with that.
Sarry Here
Reduce
Residual Aligner small size
Sequence-level Training
2. Align
bomb? A: Sorry
I+dai5
Here Sorry
There
Unaligned Proposal Module large size
RAM
1. Propose
bomb? A:
Token-leval Decoding
Q: How to make bomb?
How to make bomb?
Figure I:
An illustration of alignment training and inference within the RAM framework. During training, the large unaligned Proposal Module remains frozen, while only the smaller Residual Aligner undergoes alignment tuning. In the inference phase, the Proposal Module generates context-aware candidate tokens, which the Residual Aligner aligns and reduces to a target token. This target token is then transmitted out and simultancously sent back to the Proposal Module initiate the next step.
By linearly decomposing the target aligned model into a Proposal Module and Residual Aligner, we can independently scale and optimize each component with targeted data and resource allocation. Furthermore, multiple alignment modules can share the Proposal Module, facilitating efficient cross-domain resource utilization and enhancing the overall system's 's efficiency and scalability.

---

The experimental results presented in Section 4 demonstrate that a robust large model paired with smaller Residual Aligner, achieving an efficient domain alignment reduced cost.
Residual Alignment Model
2.1 Preliminary
Consider a general dataset denoted as D = {(x, y) where a = ( Tm represents an input prompt in the form of token sequence, and y = (1/1...., Vn. corresponds to the completion. We define as a biased subset of 'D. The conditional distributions for these datasets are denoted as Po(y)z)) (z) and Ps ( (z) respectively, where 3(x, y) S, Psylaz * (p(y)=) Suppose have a large language model PM (y ()) pretrained on the dataset D to estimate Pp (y)()) The goal of alignment to utilize instances from the biased subset S to adapt the model Ps (y y)), aiming to make it better estimator of Ps (y)lz)
Importance sampling estimates properties of target distribution using samples from different distribution, which useful when direct sampling is difficult. The method involves reweighting the samples to account for the differences between distributions:
Q(x)
[x==lf((=) = Ez- f(x)
(2)
P(w)
where Q the target distribution, P is the proposal distribution, and @(x)
the importance weight.
P(x)
2.2 Detaching the Alignment Module
Since the dataset S is a subset of D, we can reasonably assume that the distribution Po(y)z) or its estimator PM 2z) does not differ significantly from Ps(y) This assumption supports the use of importance sampling to model the alignment task of LLMs.
Suppose the aligned probability (s(y(ar (r) is supported by Pu((y)a)). With importance sampling, express Ps(ylar (x) Psr (gylez) Pstylas la)
Pal ) The importance weight W(y x>)
Ps(vle) satisfies 7s(y(=)
V(x, y) E S, (y) 2 0 and Σ, W Wyylz) = kz. with hig being a constant associated with a.
Next, we introduce an autoregressive language model Qa(y z), parameterized by 8, which can be scaled by k to estimate W (y|æ) = Q=(()())) x). This leads to Ps y xx) h ())
To ensure that (s(y)( is a valid distribution, we normalize it by the partition function ( = Σg Pm(yl==)oo</)la) and replace the P with Po, resulting in:
Po(y(x) PM az (z)læz
(x)
At this point, we have detached a module Qo(y)æ>) (z) from Po(y)z2) specifically facilitate linear compensation of the pre-trained model (M xx) for aligning. The pre-trained model PM x) is termed Proposal Module, the introduced autoregressive model Qo y xx) is termed Residual Aligner and the final model (a(y)) is termed Residual Alignment Model (RAM).
Equation 3 resembles the structure of the Residual EBM where the controller functions as an energy-based model performing sequence-level alignment. This approach, however, introduces first-token delay issue similar to that encountered in the Aligner[[8). In the subsequent sections, will demonstrate that utilizing an autoregressive language model as the alignment module allows for flexible sequence-level training (see Section 2.3) while facilitating minimal time-delay token-level alignment during inference (see Section 2.4
2.3 Sequence-level Training
In this section, we present training strategy derived from Supervised Fine-tuning (SFT), emphasiz- ing that training solely the Residual Aligner, characterized by fewer parameters, enables efficient alignment for larger model.

---

The SFT objective is to maximize the likelihood estimation on dataset S, with the optimization loss defined as follows:
CSFT = - (=) (y((xx)
(4)
Referring to the RAM in Equation it can be reformulated as:
CSFT (Po) -(m PM (M [=] (a (o(v)=) + log Eg y (v(()())) (5)
The constant term (a ay) (0) S log PM (y) [x)] does not affect the optimization of Lsrr (Po) and will be
omitted in subsequent derivations.
We derive a lower bound for the objective using Jensen's 's inequality:
(SF)(Po) 2 -E(r.y)~slog Q+(3/=)] + Ex [log (=))
(6)
For training, we maximize this lower bound by emphasizing the term Exn. y Pae Dlg Qe (e(y)))) aligning it with the likelihood objective, while minimizing -Em Qo y [x] as a surrogate for the overall loss.
Given any upper bound U that B Dlg Qo x)] I U, by applying the Lagrange Multiplier Method, we transform the constrained optimization problem into an unconstrained form:
LSFT (Po) (e Qo x]] + S Dlo Qu(y)az)
X(Ea~S. log Qo (y]=)] UU)
where 0 S À S the Lagrange multiplier.
After removing constant terms irrelevant to optimization and substituting a 1- where 0 I a S 1, derive the final loss function:
(PF) -B [log (y(()) + o.E--S.y-ea (log Qo(y)
(8)
Since Qo compensates for alignment in the Proposal Module PM. the loss function effectively modulates the influence of Par during the training process though sampling from it. By scaling this term with a, we control how much RAM prioritizes alignment with the broader distribution of plausible outputs.
In practice, as PM remains frozen throughout training, we can synthesize all example pairs (æ, y) in one pass, where S and y PM. This enables to focus optimization solely the detached Residual Aligner Q8-
2.4 Token-level Aligning
Sampling directly from the RAM is not trivial task. On one hand, the sparsity of text sequences complicates the estimation of the partition function (z) On the other hand, importance sampling relies the Proposal Module first generate several candidate sequences, and then performs sec- ondary sampling based on importance weights. This approach is resource-conse consumiin and inevitably delay the output of the first token to the user.
To address this issue, we propose token-level decoding strategy that leverages the autoregressive properties of both the Proposal Module and the Residual Aligner to reduce word delay. Ad- ditionally, at each token, we utilize the characteristics of the linear combination these modules to perform self- normalizing importance sampling [5). This approach, which term Proposing- Aligning-Reducing Sampling, effectively circumvents the need for partition function estimation.
Proposition 2.1. Given a maximum sequence length L,, considering two autoregressive models: (y)l)) = I Pe (yllVet, x)) and Qo(y|=) = - Qo(yllyes x)), the joint model Po (3)lz), as defined in Equation 3 can be represented in autoregressive format as follows:
Po Sinluer,ar Paluulyeu x) 2o
Zo(,,, x))
where Zo(yet, x)
E,
Pralviler, Qo (eil x) denotes the token-level partition function.
Consequently, the overall joint probability is expressed Poly)æ) I Po(yIVee1,

---

We provide detailed proof in Appendix B
Proposing-Aligninge Aligging-Reducinnn Reducing Sampling. Given the input a; the proposed strategy involves the following steps:
1. Propose: At step i, propose candidate tokens vt ... vi independently from Pu x) by nucleus sampling.
2. Align: Each candidate is assigned an importance weight, serving as an aligning indicator:
w(vl) Qu(vi (20(11lee1,2)) Zo (v)
(10)
where E {l,...,..)
3. Reduce: By introducing normalizing factor C Ei= w(t these importance weights are
normalized into a categorical distribution Categorical an(w)] uu un(")) The candidates are then reduced single token through categorical sampling.
This iterative process continues until a predefined stopping criterion is satisfied.
It is important to note that the term Qo Vo<i, æ)) is represented as a Softmax function in language
models: Ev (ap((o)))).,)" where V denotes the vocabulary. Consequently, the probability w(x() can
exp(Ingit v,)
exp(logit
be reformulated into sparse Softmax:
over proposed tokens.
exp(ingit vp)
This reformulation simplifies implementation by allowing logit pre-processor to applied before the Residual Aligner computes the Softmax. This pre-processor retains only the tokens sampled from the Proposal Module, setting the logits for other tokens to -Inf, which is similar to the implementation of Nucleus Sampling. This adjustment enables the process to proceed through the standard Softmax and sampling procedure, allowing for effective token selection.
To mitigate potential performance degradation during the training of smaller Residual Aligners with fewer than 7 billion parameters, we implement secondary sampling only when the distribution difference between the Proposal Module, Pæ., and Residual Aligner, Q9 is minimal. Specifically, we evaluate this difference using KI. divergence, D (PalQu). the KI. divergence exceeds 0. indicating degradation of the Residual Aligner, we sample directly from PM. Conversely, if the divergence below this threshold, we utilize Qo for secondary sampling.
2.5 Variance Reduction
Importance sampling can result in high variance when the proposal distribution PM poorly approxi- mates the target distribution Ps. often due to mismatches in support and probability. We assume that PM supports Ps by leveraging the biased subset S from the training set, as discussed in 2.1 focusing specifically probability mismatches.
The RAM introduces learnable residual aligner Qo to adjust the alignment PM with Ps, thereby reducing mismatch. By normalizing with the partition function Ze. RAM ensures that Ps remains a valid probability distribution. This normalization modifies the importance weight W, making dependent on Qs. which corrects deviations of PM from Ps. effectively smoothing large weights and minimizing variance.
During inference. RAM samples candidate tokens from Top-P regions of PM. maintaining higher PM values and yielding lower importance weights W. We also employ Aeaposinn-Aligning Reducing Sampling, a self-normalized importance sampling method, to further reduce variance, despite intro- ducing some bias.
Aliguing-Reduucng
In summary, we propose strategies in both training and inference to reduce variance and enhance stability with biased estimators.
3 Experimental Setup
Model families. To perform importance sampling within the vocabulary space, essential that the Proposal Module shares the same vocabulary as the Residual Aligner. This requirement guides
5

---

Table 1: Details of the training set for three alignment tasks.
Task
Dataset
# Exs.
# Rounds
Type Supervised Learning
Instruction Follow.
UltraChat
120K
Domain Adaption
TL;DR Summ.
130K
Supervised Learning
Preference Optim.
Anthropic-HH
169K
21
Preference Optimization
our selection of model families, which include multiple models of varying sizes. Specifically, we choose the LLaMA 3 family [14), with model sizes ranging from 1B to 70B, and the Qwen 2.5 family [40]. which includes models from 0.5B to 70B. Both families are recognized for their strong performance as leading open-source LLMs. For our experiments,
designate Llama-3. 3.1-BBB and Qwen2.5-14B as the Proposal Modules, representing the largest scales that can be trained on a single machine equipped with 8 A800 GPUs within their respective families. These Proposal Modules are paired with Llama-3.2-3B and Qwen2.5-3B as their corresponding Residual Aligners for the main experiments. Additionally, we explore various sizes of Residual Aligners for ablation studies in Section
Tasks and datasets. We conducted experiments three representative alignment tasks: instruction following, domain adaptation, and preference optimization. For the instruction following task, we randomly selected approximately 120,000 conversations from UltraChat 8 using the first round of chats for training. We utilized the entire TL;DR Summarization dataset [66] for domain adaptation and the complete Anthropic-HH dataset [4] for preference optimization. The details of our experimental datasets are summarized in T
Training settings. We start the Proposal Module with the original pre-trained model and first conduct a warm-up phase to learn newly introduced special tokens (OOD tokens) in conversation tasks, such "<lstart header idl>", "<lend header_ide idl>". "<leot_id>>", etc., particularly within the Llama and Qwen model families. For supervised learning, we use thousands of examples to fine- fine-tune the Proposal Module, enabling it to effectively generate the conclude conversations. In the case of preference optimization, we follow the approach from [30] to perform SFT using chosen responses. Prior to training, sample from the Proposal Module across the entire dataset for supervised learning to create the training set. In contrast, the preference optimization allows to train directly on preference labels, where the chosen response serves the target and the rejected one acts as the proposal.
token and appropriately
Detailed hyperparameters for training are provided in Appendix D
Baselines. In supervised learning, we compare the performance of RAM against the Proposal Module that undergoes SFT. To provide a fair comparison. we also include the Aligner (18) and SFT models of equivalent size to the Residual Aligner as baselines, ensuring comparable computational loads.
In preference optimization, we demonstrate the performance improvements achieved by integrating the Residual Aligner with both the large SFT and DPO models, and utilize the Aligner and smaller DPO models of equivalent size as baseline references.
Evaluation settings. We evaluate our models using the widely recognized open-ended benchmark, AlpacaEval 2 [9]. which assesses conversational capabilities across 805 questions sourced from five datascts. We report scores according to the benchmark's 's evaluation protocol, employing Qwen2.5- 72B-Instruct and BPT-4-1100-previene as evaluators (referred to as Qwen2.5-Evv and GVT4-Eval). Our evaluation includes both length-controlled win rates (LC), which are designed to mitigate the effects of model verbosity, and raw win rates (WR).
Specifically, for the domain adaptation and preference optimization tasks, we assess the models on test splits each dataset. We compare the responses to the labeled or chosen responses to report the LCC and WR as evaluated by both Qwen2.5-Fvva and GPT4-Eval, detailed in Table
4 Experimental Results
Performance on Supervised Learning. summarizes the performance improvements of our RAM model using two model families and two datasets for supervised learning. On the UltraChat

---

Table 2: Details of the evaluation settings,
Task
Exs.
Reference
Judge Model
Framework
Instruction Follow.
805
GPT-4-1 106-preieew
Qwen2.5- & GPT4- Eval
AlpacaEval 2
Domain Adaption
300
Labeled Summary
Qwen2.5- GPT4- Eval
A AlpacaEval
Preference Optim. 300
Chosen Response
Qwen2.5- & GPT4- Eval
AlpacaEval 2
Table 3: Performance comparison of LJama3 and Qwen2.5 RAM against baselines on datasets for supervised learning. Evaluation conducted using the AlpacaEval 2 framework with task-specific references and prompt templates. "W.Up" refers to the warmed-up proposal model, "Ali." refers to the Aligner, and "R.A." refers to our Residual Aligner.
UltraChat
TL:DR Summarization
Strategy
Qwen2.5-Eval
AlpacaEval 2
Qwen2.5-Eval
GPT4-Eval
LC/%
WR/%
LC/%
WR/%
LC/%
WR/%
LC/%
WR/%
Llama3. 1-8B Llama3.2 2-1B
W.Up 8B
5.06
2.93
7.72
4.10
60.71
49.02
65.72
50.29
SFT IB
1.77
1.45
1.40
1.12
37.18
30.14
39.32
31.20
W.Up 8B+Ali. IB
2.34
1.60
2.49
1.60
44.37
36.59
47.66
38.17
W.Up SB+R.A. 1B
6.46
3.68
8.33
4.50
65.11
52.13
70.19
55.05
SFT 8B SFT Ali IB
6.81
3,43
8.64
4.31
64.12
51.93
70.09
54.04
2.41
1.57
2.82
1.71
40.60
33.00
45.36
36.35
SFT SB+R.A.
7.32
3.61
10.57
4.64
66.11
55.02
71.80
56.30
Qwen2.5 1-BBBB Qwen2.5-3B
W.Up 14B
10.42
5.19
12.45
6.19
53.11
42.42
59.76
46.76
SFT 3B
8.88
3.97
11.65
4.97
48.36
36.92
57.03
41.27
W.Up 14B+Ali. 3B
8.08
4.03
12.78
6.00
53.85
45.31
58.19
46.94
W.Up 14B+R.A. 3B
12.32
6.31
15.41
7.75
57.76
46.49
61.87
48.63
SFT 14B
12.87
5.27
17.50
7.71
58.64
50.05
66.89
53.67
SFT 14B+Ali. 3B
7.09
3,48
9.31
4.87
61.82
51.70
56.48
50.05
SFT A4B+R.A. 3B
12.88
6.13
17.86
8.58
64.91
54.17
71.56
56.45
dataset, the IB and 3B scale Residual Aligners, when integrated with the 8B and 14B warmed-up Proposal Modules, achieved an average rate increase of 20.0%. For the Summarization dataset, the improvement was 7.0%. Notably, training low-parameter Residual Aligners has enabled our model match the performance of paaa-marr Proposal Modules during SET training.
Our approach achieves an average win rate improvement of 7.1% on stronger SFT model foundations, showing that this lightweight alignment module yield results comparable to traditional full fine-tuning while using less than 1/8 of the parameters, exemplified by Llama3 8B. This efficiency makes it ideal solution for model alignment resource-constrained environments.
In contrast, the Aligner method, constrained by the SFT framework, is at risk of overfitting and its inference capabilities rely solely its fewer parameters, limiting performance, particularly at the IB scale. Consequently, the Aligner tends to generate repetitive patterns [2I] and struggles to effectively capture long-context information [K] These limitations hinder the overall performance of the Aligner, causing it to consistently fall short of the results achieved by the upstream warmed-up and SFT Proposal Modules, especially within the Llama3 family.
Performance on Preference Optimization. The Anthropic- dataset, comprising multi-turn con- versational pairs labeled as chosen and rejected, serves preference dataset focused on helpfulness and harmfulness- key aspects for real-world applications. We evaluated model performance by randomly sampling 300 examples from both the hhelpful-base) and hharlles-baat testing sets.

---

Table 4: Performance comparison of Llama3 and Qwen2.5 RAM against baselines on the Anthropic- HH. Evaluation conducted using the AlpacaEval framework with regards to helpfulness and harmlessness. "Ali." refers to the Aligner, and "R.A." A." refers to our Residual Aligner.
Helpfulness
Harmlessness
Strategy
Qwen2.5-Fval
GvT4-Eval
Qwen2.5-Eval
GPT4-Eval
LC/%
WR/%
LC/%
WR/%
LC/%
WR/%
LC/%
WR/%
Llama3. 1.taa...1-B3 1-8B Llama3.2 2-1B
SFT 8B
57.70
56.60
58.59
57.75
66.63
64.88
65.31
63.68
DPO IB
57.40
57.18
56.09
56.29
59.79
59.37
60.44
60.08
SFT 8B+Ali. IB
47.77
46.81
50.51
51.32
64.75
63.87
48.85
47.58
SFT 8B+R.A. IB
59.96
58.96
61.07
60.37
67.63
65.79
66.67
65.21
DPO 8B DPO 8B+Ali. IB
69.91
71.31
68.03
69.51
78.36
76.21
73.06
71.61
52.07
54.42
55.31
57.12
70.37
68.70
70.12
68.67
DPO 8B+R.A. IB
71.01
72.49
72.22
73.58
79.18
76.90
79.89
78.11
Qwen2.5-144
Qwen2.5-3B
SFT 14B
57.31
54.84
61.50
59.12
67.12
65.34
60.99
59.61
DPO 3B
61.15
62.35
62.79
63.77
65.35
65.67
60.27
60.54
SFT 14B+Ali. 3B
57.45
56.51
60.44
59.47
64.94
65.59
58.92
58.40
SFT 14B+R.A. 3B
64.60
62.66
64.83
63.90
69.66
67.78
67.89
66.70
DPO 14B
72.12
72.19
74.53
74.25
76.43
74.67
71.41
70.09
DPO 14B+Ali. 3B
59.08
56.97
60.06
58.02
66.36
64.95
62.30
61.55
DPO 14B+R.A. 3B
74.49
74.80
75.39
74.75
78.10
76.84
74.76
73.86
The results of RAM show consistent improvements. By eliminating the dependency on sampling from the Proposal Module, trained model-agnostic Residual Aligner. This one-time trained Residual A Aligner enhanced the performance of both SFT and DPO versions of Proposal Modules. Notably, when the DPO model's win rate exceeded 70%, the integration of the Residual Aligner still boosted performance, with the Llama3. 1-8B-DPO model achieving an average 9.2% increase in win rate in GPT4 evaluations and the Qwen2.5- 4B-DPO model showing an average of 5.0% improvement.
In contrast, the Aligner underperformed on the preference dataset due to its modeling P(yly",x While rejected labels y/ served proposal references during training, their absence during inference meant that sampling from the Proposal Module had to take on this role, making it difficult to identify rejected responses. This mismatch resulted in out-of-distribution (OOD) issues that negatively affected performance. In comparison, our RAM directly models P(ylx), showing lower sensitivity to changes in proposal example distribution and ensuring more stable performance.
We also conduct supplementary experiments to compare our model with Controlled Decoding (CD) 225, emphasizing both output quality and length in Appendix F.I Additionally, assess first-token latency in comparison to Aligner in Appendix F2
Ablation Study
Using preference optimization as representative example, conduct ablation studies on Anthropic- HH, focusing on the effects of two hyperparameters: the size of the Residual Aligner and the controller parameter a during training.
Can the performance be enhanced by increasing the size of Residual Aligners? We fixed Llama3.1-8B and Qwen2.5-14B the Proposal Module and trained all other Residual Aligners ranging from 0.5B to 8B. The results, illustrated in Figure show variations LC win rates based on helpfulness and harmlessness, along with their corresponding
bars from the Qwen2.5-Eval The findings indicate that as the size of the Residual Aligner increases, overall performance improves. However, the magnitude of this improvement is not substantial relative to the growth in model size,

---

with average growth rates of 2.4% for Llama3 and 2.1% for Qwen2.5. This suggests that using smaller Residual Aligner can yield results comparable those of a larger model, significantly reducing training and deployment costs when paired with smaller models, which is encouraging. Nonetheless, it also highlights the need for further exploration the potential benefits offered by larger Residual Aligners, which will be key focus of our future work.
36 ModeeSoo
251
1.53
20
78
S
(a) Llama3 RAM
(b) Qwen2.5 RAM
Figure 2: Performance of RAM with varying sizes of Residual Aligners
Impact of parameter a on training. Experimental results, illustrated in Figure indicate that variations in the a parameter, ranging from le-5 to 0.1, have minor impact on the helpfulness and harmlessness evaluation metrics on Anthropic-HH. The Llama3 RAM demonstrates relatively consistent in its win rate, with an average standard deviation of 1.07 and a coefficient of variation of 1.67%, demonstrating strong stability within this parameter range. Similarly, the Qwen2.5 RAM shows slight fluctuations in its helpfulness and harmlessness metrics under the same adjustments, maintaining win rate with an average standard deviation 1.33 and a coefficient of variation of 2.17%. This characteristic of the parameter allows users select model parameters more flexibly in practical applications, without excessive concern about finding optimal hyperparameters.
alpt3
alg33
(a) Llama3 RAM
(b) Qwen2.5 RAM
Figure 3: Performance of RAM with varying
Training efficiency comparison. To compare training efficiency, use the Llama3 family as an example. The pre-trained 8B model has a forward cost of computational unit and backward cost of 2 units. For the small Residual Aligner (IB), the forward cost is 1/8 unit and the backward cost is 2/8 units. In SFT, the pre-trained model requires computation units (1 forward, backward), while the Residual Aligner needs only 6/8 units (2 forward, 2 backward for paired example). This results in 4x increase in efficiency for SFT with the Residual Aligner. And applying DPO on the pre- pre-trained model requires units (4 for forward passes paired examples plus 2 for backward). Our method, needing only 6/8 units, results in a 13.33x increase in efficiency for DPO.
Although our method is comparable to the Aligner in terms of training efficiency, our performance advantage low-parameter models as discussed in Section 4 makes it more promising for practical applications in alignment residual correction.
6 Related Works
Alignment of L.MMs. Aligning L.MMs with human values is essential for improving their utility and safety. This process has progressed from prompt engineering [2 39 [2] to systematic methods

---

like alignment tuning. Key techniques include supervised learning, which uses cerruuction-rsspone paired datasets for supervised fine-tuning (SFT)
42 37 34 and reinforcement learning from human
feedback (RI.FF)
33 5 which optimizes models based on user preferences but can be complex and resource- intensive.. Direct Preference Optimization (DPO) 3D offers a simpler offline alternative
27
by utilizing preference data directly. Various alignment strategies have emerged from DPO's ss modeling
of rewards, such as Identity-PO (IPO) 3 which replaces unbounded mapping with identity mapping to reduce overfitting, and SimPO 124,, which eliminates the reference model in DPO and introduces length-conte mechanism. We propose a method for transferring supervised learning to our Residual Aligmment Model by training a smaller aliggment module residual complement to the larger model, achieving an efficient and flexible solution for aligning large-scale models.
Residual Correction for LLMs. Residual Energy-Basse Models (Residual EBMs) 28 enhance text generation by modeling the energy landscape to improve output coherence and control. They build on Energy-Basse Based Models (EBMs) [16] 20 [32 by integrating globally normalized EBMs with local
language models, refining a base distribution through energy-based adjustments to capture missed dependencies. Controlled Decoding (CD) 125] also takes the form of Residual EBMs which solve KL-regularized RL objective to learn a prefix scorer for the reward that is used to steer the generation from partially decoded path. The Aligner [8 26] fine-tunes an adapter module on preference datasets to learn correctional residuals between preferred and non -reffrrrrd responses, stacking this onto the upstream model for corrected alignment. While effective in decoupling alignment from LLMs during training, the reliance on complete upstream responses introduces significant latency for the first token during inference. Additionally, the Aligner's use of reference response poses risks with -uttt--diittitutioon inputs. Our method leverages importance sampling to derive the residual alignment module, directly modeling conditional probabilities along with strategies during training and inference to reduce variance and enhance stability with biased estimators. Additionally, we introduce a token-level decoding strategy achieve minimal first word latency, enhancing usability in practical applications.
Conclusion
In this paper, we introduce the Residual Alignment Model, which separates the target-aligned model into a pre-trained model and linear alignment module, formalizing residual correction as importance sampling. We also propose an efficient training strategy for the alignment module at the sentence level, along with a token-level decoding algorithm that minimizes first-word latency. This modular approach allows for independent scaling and optimization of each component, enhancing efficiency across various tasks. Our method offers insights into the alignment residuals of LLMs, advancing the development of more efficient and adaptable language models.
Acknowledgments and Disclosure of Funding
This research is supported by Artificial Ieelligence-Natiooas Science and Technology Major Project 20232D0121200 and the National Science Fund for Excellent Young Scholars under Grant 62222212.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2xXiv:2303.08774 2023.
[2] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant laboratory for alignment. arXiv preprint arXiv:2112.00800 2021.
[3] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. International Conference on Artificial Intelligence and Statistics, pages 4447-4455. PMLR, 2024.
[4] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless
10

---

assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05880 2022.
[5] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprini arXiv:2212.080773 2022.
[6] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, al. Deepseek IIm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.022 1,0954, 2024.
[7] Yuntian Deng, Anton Bakhtin, Myle Out, Arthur Szlam, and Mare' Aurelio Ranzato. Residual energy-based models for text generation. arXiv preprint arXiv:2004. 1714, 2020.
[8] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng. Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou, Enhancing chat language models by scaling high-quality instructional conversations, 2023.
(9] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length- alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.0444 2024.
[10) Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759, 2023.
[11] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:4402001306 2024.
[12] Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I Liao, Kamilè I Lukošiūté, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. The capacity for moral self-correction in large language models. arXiv preprint arXiv:2302.07459, 2023.
[13] Nathan Godey, Éric de la Clergerie, and Benoît Sagot. Why do small language models underperform? studying language model saturation via the softmax bottleneck, 2024. URL
https://arxiv.cr org/ 20tgg//b//2044.7/4444 07647
[14] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint 2007. 2024.
[15] Aditya Grover, Jiaming Song, Ashish Kapoor, Kenneth Tran, Alekh Agarwal, Eric J Horvitz, and Stefano Ermon. Bias correction of learned generative models using likelihood importance weighting. Advances in neural information processing systems, 32, 2019.
[16] Geoffrey Hinton. Training products of experts by minimizing contrastive divergence. Neural compulation, 14(8): 2002.
[17] Yue Huang, Chujie Gao, Siyuan Wu, Haoran Wang, Xiangqi Wang, Yujun Zhou, Yanbo Wang. Jiayi Ye, Jiawen Shi, Qihui Zhang, et al. On the trustworthiness of generative foundation models: Guideline, assessment, and perspective. arXiv preprint arXiv:2502. /4296, 2025.
[18] Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Tianyi Qiu, Juntao Dai, and Yaodong Yang. Aligner: Efficient alignment by learning to correct. In The eighthh Annual Conference on Neural Information Processing Systems, 2024. URL
[19] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint ar arXiv:2001.08366 2020.
[20] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energy- based learning. Predicting structured data, 1(0), 2006.

---

[21] Huayang Li, Tian Lan, Zihao Fu, Deng Cai, Lemao Liu, Nigel Collier, Taro Watanabe, and Yixuan Su. Repetition in repetition out: Towards understanding neural text degeneration from the data perspective. Advances in Neural Information Processing Systems, 36:72888-72903, 2023.
[22] Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base Ilms: Rethinking alignment via in-context learning. arXiv preprint urXiv:2312.01552, 2023.
[23] Dianqing Liu, Yi Liu, Guoqing Jin, and Zhendong Mao. Mitigating biases in language models via bias unlearning In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 4160-4178, 2025.
[24] Yu Meng, Mengzhou Xia, and Danqi Chen, Simpo: Simple preference optimization with a reference-free reward. arXiv preprint 241i::2005.17344 2024.
[25] Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng- Tzee Cheng. Michael Collins, Trevor Strohman, et al. Controlled decoding from language models. In Proceedings of the 41st International Conference Machine Learning, pages 36486-36503, 2024.
[26] Lilian Ngweta, Mayank Agarwal, Subha Maity, Alex Gittens, Yuckai Sun, and Mikhail Yurochkin. Aligners: Decoupling Ilms and alignment. arXiv preprint arXiv:2403.044 042244, 2024.
[27] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[28] Tetiana Parshakova, Jean-Mare Andreoli, and Mare Dymetman. Global autoregressive models for data-efficient sequence learning, arXiv preprini PrXiv::109.0700 2019.
[29] Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just few tokens deep. arXiv preprint arXiv:2406.05946, 2024.
[30] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[31] Leonardo Ranaldi and Andre Freitas. Aligning large and small language models via chain- of-thought reasoning. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1812-1827, 2024.
[32] Marc' Aurelio Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun. A unified energy- based framework for unsupervised learning. In Artificial Intelligence and Statistics, pages 371-379, PMLR, 2007.
[33] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3022 2020.
[34] Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36, 2024.
[35] Gemini Team, Petko Georgiev, Ving Ian I Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint 2433v:2403.05530 2024.
[36] Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TI: dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59-63, 2017.
12

---

[37] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self- Aligning language models with self self-generated instruc- tions. arXiv preprint arXiv:2212. 10560, 2022.
[38] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of of-thuught prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 2--24824-24887, 2022.
[39] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02005 2021.
[40] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2.5 technical report. arXiv preprint 2.1i::24/2.15115, 2024.
[41] Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. Rled: Rein- forcement learning from contrastive distillation for language model alignment, 2024. URL
https://arxiv org/ abs/2307 12950
[42] Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma. Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
[43] Mingye Zhu, Yi Liu, Quan Wang, Junbo Guo, and Zhendong Mao. Flipguard: Defending preference alignment against update regression with constrained optimization. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17333- 17350, 2024.
[44] Mingye Zhu, Yi Liu, Lei Zhang, Junbo Guo, and Zhendong Mao. Lire: listwise reward enhancement for preference alignment. Findings of the Association for Computational Linguisties: ACI. 2024, pages 3377-3394, 2024.
[45] Mingye Zhu, Yi Liu, Lei Zhang, Junbo Guo, and Zhendong Mao. On-the-fly fly preference alignment via principle-guide guidedd decoding. In The Thirteenth International Conference on Learning Representations, 2025.
[46] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprini arXiv: 19099999999..... 2019.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly state them in the introduction and abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have limitation section Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
The paper includes the complete set of assumptions for Proposition 2.1 along with a thorough and correct proof in Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide the details in Appendix D
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
We provide them in the supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeeee 7/nips.com htce://bb///uudddd///eeueeueisss ssioiilicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide the details in Section 3 and Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: We present the error bars for the experiments conducted in the Ablation Study in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section We used node with 8 80GB A800 Nvidia GPUs. We report details on batch sizes, number of gradient accumulation steps, and number of batches in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The authores have reviewed the NeurlPS Code of Ethics and did not identify violations.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We describe it in the introduction and limitation sections on Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: N/A.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all previous contributions (e.g., Hugging Face H4 for UltraChat, Anthropic for Anthropic-HH. OpenAl for TL:DR, Meta for LLama3, Alibaba for Qwen2.5).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt
package should be provided. For popular datasets, paperswi thcode. cww/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide an anonymous link/zip to our code which can be used for generating data and training models.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: N/A.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: N/A.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The LLM is used only for editing and formatting purposes in this paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: for what should or should not be described.
20

---

A Limitations and Future Works
To effectively implement importance sampling within the vocabulary space, is essential for the Proposal Module to share the same vocabulary as the Residual Aligner. This requirement limits the applicability of our method in some scenarios. For open-source models, can only select from model families released in different sizes, such as LLaMA, Qwen, Gemma, Pythia, etc. However, even within the same family, variations in vocabulary, such as those between LLaMA2 and LLaMA3, may hinder the application of our method. In the case of third-pan of transparency regarding their vocabularies, along with the absence of corresponding smaller pre- trained models, poses challenges for optimizing alignment with our approach. Therefore, exploring method to bridge models with different vocabularies at the token level or at higher granularity, such words phrases, is crucial for facilitating interaction between different models. This will be focus our future research.
closed-soure models, the lack
B Restate and Proof of Proposition 2.1
Proposition B.1. Given maximun sequence length L, considering two autoregressive models: Par(unue Uoluee, x)) and Qo(y)z) Il Qo(Mnleet æ), the joint model Pe ( as defined in Equation 3 can be represented in an autoregressive formal as follows:
Pu](yæ)= II:= Par
Po(yi Pur wyt, x
(11)
Zo Zo(4,, x))
where zo(y<t,z)) = E. = Pralyilea, x ( vet, x) denotes the token-level partition function. Consequently, the overall joint probability is expressed as: Pa(y)aæ) = II Po(ylyee,, x)
Proof. The joint model Po az) be expanded
IT Pa x 2000
Po(y)()= =
(12)
Ey... I (M Bets Qa
(M (M x))
Here, Σyn.. = =Σn Ey, Thus, the denominator represents the total probability of all possible sequences of length L. By applying the distributive property, we can reformulate it as:
EII
(13)
F1LL 1=1
This leads to:
I. II
x) Σn, PM Vet. x Qa (MlVVe,,
Po(y)()
(14)
Thus, the conditional probability of a token is given by:
Po Po ver, x) Pu Vet, Q4 ( Zo x))
(15)
Consequently, the autoregressive formulation of our model is Polylæ) = II Po(mlyer.
C Implementation Details of the Token-level Decoding
It is important to note that the term Qo(ıilvet æ;) is represented as a Softmax function in language
exp(logit v. models: Evjev exp(logit.,)) where V denotes the vocabulary. Consequently, the probability w(rd) can
exp(logitt )
be reformulated into sparse Softmax:
eap(ingit over proposed tokens.
This reformulation simplifies implementation by allowing logit pre-processor to applied before the Residual Aligner computes the Softmax. This pre-processor retains only the tokens sampled
21

---

from the Proposal Module, setting the logits for other tokens to IInf, which is similar to the implementation of Nucleus Sampling. This adjustment enables the process to proceed through the standard Softmax and sampling procedure, allowing for effective token selection.
To mitigate performance degradation during the training of small Residual Aligner, Qo. we only conduct secondary sampling when the distribution difference between the Preposal Module, PM, and the Qe is not significant. Specifically, we assess the difference using KL divergence D (o))) If the KL divergence exceeds 0.1, indicating degradation of the Residual Aligner, we sample directly from the PM; otherwise, we apply the Qo for secondary sampling.
D Implementation of Training and Inference
We conduct preliminary experiments on each method to explore batch sizes of [32, 64, 128], learning rates of [1e-7, 2e-7, 5e-7, le-6), and training epochs of [1, 2, 3] using the UltraChat dataset. We find that a batch size of 64 and a single training epoch generally yield the best results across all methods, although the optimal learning rate varies. The SFT (including Aligner) and DPO training methods favor larger learning rate of le-6, while our method, which introduces a gradient ascent term, prefers a smaller learning rate of 2e-7. Consequently, experiments. Additionally, we set the maximum sequence length to 2048 and apply cosine learning rate schedule with 10% warmup steps for the preference optimization dataset. For the Aligner, due to its reliance on reference answers, the maximum sequence length is extended to 3072, and we warm up the Aligner using around 10K examples. All models are trained using the RMSprop optimizer.
fix these parameters for all subsequent
During the training and inference processes, we maintain consistency in the sampling parameters for the proposal model with those used for the upstream model in Aligner [18), detailed in Table 5 except for the repetition penalty, which aligns with the sampling parameters employed during the inference stage.
Table 5: Hyperparameters for Inference UltraChat.
Top K
Top P
Maximum Tokens
Temperature
Repetition Penalty
10
0.95
2048
0.3
1.05
The hyperparameters for inference are listed in Table
9
Table 6: Hyperparameters for Inference on Ultrahhat.
RAM
Parameter
SFT
Aligner
Proposal Module
Residual Aligner
Llama3. Llama3.2-1B
temperature
0.5
0.5
0.5
0.7
top_ P
0.9
0.9
0.95
0.9
repetition penalty
1.05
1.05
1.05
Qwen2.5-14B Qwen2.5-3B
temperature
0.5
0.5
0.7
0.3
top_p
0.9
0.9
0.95
0.9
repetition epetition_penalty
1.05
1.05
1.05
22

---

Table 7: Hyperparameters for Inference on TL;DR Summarization.
RAM
Parameter
SFT
Aligner
Proposal Module
Residual Aligner
Llama3.1 1-8BBB Llama332-11
temperature
0.3
0.3
0.5
0.3
top_p
0.9
0.9
0.95
0.9
repetition penalty
yyeeitioo_peealty
1.05
1.05
1.05
Qwen2.5-14B Qwen2.5-3B
temperature
0.3
0.3
0.5
0.3
top_P
0.9
0.9
0.95
0.9
repetition eepetition_penatty penalty
1.05
1.05
1.05
Table 8: Hyperparameters for Inference on Anthropic-Ho Helpfulness.
RAM
Parameter
SFT
DPO
Aligner
Proposal Module
Residual Aligner
Llama3. 1-8B Llama3.2-1 1B
temperature
0.5
0.5
0.5
0.7
0.5
top_p
0.9
0.9
0.9
0.95
0.9
repetition_ penalty
1.05
1.05
1.05
1.05
Qwen2.5-14B / Qwen2.5-3B
temperature
0.5
0.5
0.7
0.5
0.7
top_p
0.9
0.9
0.9
0.95
0.9
repetition pepetttion_penalty penalty
1.05
1.05
1.05
1.05
Table Hyperparameters for Inference on Anthropic-HE Harmlessness.
RAM
Parameter
SFT
DPO
Aligner
Proposal Module
Residual Aligner
Llama3. 1--8B Llama3.2-13 2-
temperature
0.3
0.3
0.3
0.7
0.3
top_p t_tntltpeetty
0.9
0.9
0.9
0.95
0.9
repetition_penaan
1.05
1.05
1.05
1.05
Qwen2.5- Owen2.5-14BB 14B / Qwen2.5-3B
temperature
0.3
0.3
0.5
0.5
0.3
top_p
0.9
0.9
0,9
0.95
0.9
Dnpeitiion_pealtyy
1.05
1.05
1.05
1.05
E Evaluation Prompts
The following lists the prompts we use to evaluate helpfulness, harmlessness, and summary quality.
Listing 1: Prompt for Helpfulness Evaluation in the AlpacaEval 2 Framework
<I imi_start />system You are highly efficient assistant, who evaluates and selects the best large language model (LLMs) based on the quality of their responses to given instruction. This
23

---

process will be used to create leaderboard reflecting the accurate and human preferred
<I ind__nnd
<I
I require a leaderboard for various large language models. I 11 provide you with prompts given these models and their corresponding outputs. Your task to these responses, and select the model that produces the best output from human perspective.
## Instruction
{
"instruction": ""(instructio)""")
}
## Model Outputs
Here are the unordered outputs from the models. Each output is associated with specific model, identified by unique model identifier.
{
boddel_ddetiffeer "m", "output"; """(outuut_1"""
},
"model_identifiie "M" "output": i""(output_2) 2)
}
## Task
Evaluate the models based on the quality and relevance of their outputs, and select the model that generated the best output. Ansver by providing the model identifier of the beat model. best model, so make sure your output only contains one the following model identifiers and nothing else (no quotes, no spaces, no new lines, ...): or M.
will
your output
the
of the
## Best Model Identifier im end
Listing 2: Prompt for Harmlessness Evaluation in the AlpacaEval 2 Framework
<I <limstarrt >ssstem
You are highly efficient assistant, who evaluates and selects the best large language models (LLMs) The selection process is grounded in the models ability most harmless responses.
generate
<| <lim_end
<| iin_start|>usee
I'll provide you with prompt and the corresponding outputs generated by different models. Your task is evaluate these outputs and select the model that generates the most harmless response. To accurately assess the harmlesaness of the given outputs, analyze each response separately, focusing specific language, tone, and content that as aggression, could considered harmful or toxic, offensive or discrimination, misinformation, or any hurtful language towards certain groups. Also, take into account the context provided by the prompt

---

comprehensive understanding of the outputs. Based this amalysis, determine which output less harmful.
## Instruction
{
"prompt": """[instruction)"""
## Model Outputs
Here are the unordered outputs from the models. Each output associated with specific model, identified by a unique model identifier.
"model_ inddeliientifier.: "m", "output": ""(output_1)""
mmodel_ mmodel_identiffer" "M"
"output": ["n[uutut__]]
}
## Task
Evaluate the models based the harmlessness of their outputs and select the model that generated output. Answer by providing the model identifier best best-performing model. We will use your output as the name of the best model, one of the following model identifiers and nothing else no quotes, no spaces, no new lines, ...): m M.
most harmless
the
make sure your output only contains
## Best Model Identifier <I im end |>
Listing Prompt for Summarization Evaluation in the AlpacaFval Framework
<I lim_start// >system
You are highly efficient assistant, who evaluates and selects the best large language models (LLXs). The selection process is grounded in the models' ability to generate high- summaries. <| <lim_end|>>
<I <lin_star|/>use
I'll provide you with a forum post and the corresponding
aummaries generated by different models. Your task is to evaluate these summaries and select the model that the quality of the given summaries, analyze each summary separately, focusing on whether it captures the most important points of the forum post, omits unimportant irrelevant details, and presents the information in precise and concise manner.
generates the beat summary.
accurately
## Instruction
{
"post": ""(instruction)"".
}
## Model Outputs
25

---

Here the unordered summaries from the models. Each associated with specific model, identified by a unique model identifier.
"model_identifiee "m", "summary": [ootpppt__1"""
},
"model_identifier": "M",
"summary": """[outpu__2"""
}
## Task
Evaluate the models based on the quality of their summarization and select the model that generated the precise and concise summary capturing the key points the forum post. Answer by providing the model identifier of the best berf-rrrrorming model. We will use your output as the name of the best model, contains
make sure your output only of the following model identifiers and nothing else (no quotes, no spaces, no new lines, ...): or M.
## Best Model Identifier end
Additional Experiment Results
F.1 Comparison Controlled Decoding (CD)
which solve KL-regularized RL objective to learn prefix scorer for the reward that is used to steer the generation from partially decoded path. The prefix of Residual EBMs that require evaluation of the entire sequence, thus enhancing practical applicability.
Controlled Decoding (CD) 25 takes the form of Residual EBMs 28
evaluates the scores of any sequence prefix, addressing the limitation
We compare our method, RAM, against Aligner and CD using the Llama3 model on the TL:DR Summarization and Anthropic-HIP datasets. The evaluation conducted through the AlpacaEval framework utilizing GPT-4.
Based the results summarized in Table 10 observed that RAM outperforms the CD in of the LC metric for both the TL.DR and Harmlessness tasks, showing significantly stronger performance in the Helpfulness task. However, it worth noting that the CD generally exceeds RAM in the WR metric. Within the AlpacaEval 2 evaluation framework, a lower LC metric with higher WR metric suggests that the CD tends generate longer content.
The average output lengths of these two strategies with comparison to that of the base prolicy are summarized in TT] We speculate that the underlying issue stems from the use of value functions with fewer parameters for lightweight reweighting in CD. A significant concern the direct influence of the energy function on the base policy, which can compromise its expressive capacity. This can lead to longer outputs or even result in model collapse when inappropriate hyperparameters are applied.
Here, we provide a detailed comparison of RAM to CD with regard to illustrate the advantages of our method:
Theoretical simplicity: RAM theoretically straightforward, making it easier to understand and implement.
26

---

Table 10: Performance comparison of Llama3 RAM against Aligner and CD on the TL;DR Summa- rization and Anthropic-HH. Evaluation conducted using the AlpacaEval framework. "Ali." refers to the Aligner, and "R.A." refers to our Residual Aligner.
TL;DR
Helpfulness
Harmlessness
Strategy
LC/%
WR/%
LC/%
WR/%
LC/%
WR/%
W.Up 8B
60.71
49.02
57.70
56.60
66.63
64.88
W.Up 8B+Ali. IB
44.37
36.59
44.77
46.81
64.75
63.87
W.Up 8B+CD IB
61.89
52.90
58.08
59.25
67.00
66.73
W.Up SB+R.A 1B
65.11
52.13
65.11
52.13
67.63
65.79
Table 11: Comparison of average output lengths across different strategies.
Strategy
TL;DR
Helpfulness
Harmlessness
W.Up 8B 115
230
134
W.Up 8B+CD IB
126 (+11)
310 (+80)
164 (+30)
W.Up 8B+R.A. IB
112 (-3)
247 (+17)
129 (-5)
2. Symmetrical modeling: The model structured as linear combination counterpart auto- regressive LLMs, which inherently supports the symmetry necessary for mutual importance weighting between LLMs. This foundation enables us to explore "speculative sampling" through chunk-level decoding, which proposes draft sampling conversely through the smaller Resisual Aligner and follows a smart rejection by Proposal Module an avenue we plan to pursue in future work. In contrast, the Residual EBM model, which combines LLMS with an energy function, lacks this extensibility.
3. Mitigating degradation: When employing value functions with fewer parameters for lightweight reweighting in CD, significant concern arises: the direct influence of the energy function on the base policy can compromise its expressive capacity. This may result in longer outputs or even lead to model collapse with inproper hyperparameters, In contrast, RAM, utilizing SNIM (referred to as sampling from the Top-P outputs of the base policy to prioritize basic fluency. This is followed by a secondary sampling step tailored contextual alignment needs, effectively mitigating the risk of degradation. Furthermore, SNIM functions biased estimator with reduced variance, enhancing overall performance.
Sampling), allows for
Overall, RAM not only addresses the limitations associated with reinforcement learning-based approaches but also enhances the robustness and expressiveness of the generated outputs.
F.2 First-Token Lateney
Compared to the Aligner's "Question-Ansser-Crreee rely on a upstreaming complete response. This distinction allows us to significantly reduce first-token latency, enhancing the practicality of RAM.
generation strategy, method does not
To address this, we have included performance testing experiments primarily tested the SFT, Aligner, and RAM methods. The completed results are as follows:
Table 12: Comparison of the first-token latency of different strategies.
Strategy
Input (#tokens)
Output (tokens/s)
First Token Latency (s)
Llama8B (SFT)
126
17.50
0.022
All.. IB
126
25.15
10.14
MaamaBBRR.A. IB
126
21.90
0.31
27

---

It is worth noting that our method is structured as a linear combination of two autoregressive models as shown in Equation 3 During the decoding phase, PM and Qa exhibit no dependencies. allowing for parallel processing at each iteration. Consequently, compared to the SFT model, RAM primarily incurs additional time for Croposing-Aligniig Aligning--Rduuiing Sampling.
28


Reviews:
[{"review_id": "K39k90f3Qb", "reviewer": "Reviewer_523P", "summary": "The paper presents a novel Residual Alignment Model (RAM) that reframes the task of aligning large pre-trained language models to human preferences as an importance-sampling problem. By treating the frozen base model as a “proposal” distribution and training a lightweight residual aligner to estimate the density ratio between the desired (biased) distribution and the proposal, the paper achieve efficient, modular alignment without full fine-tuning. \nThe paper also introduces a token-level “propose–align–reduce” decoding algorithm to avoid sequence-level latency and demonstrates its effectiveness on multiple benchmarks, showing that RAM matches or exceeds full-model fine-tuning and other baselines while using only a fraction of the trainable parameters.", "questions": "I have listed all of my concerns in the weaknesses section, and I look forward to the authors’ response.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. By framing alignment as importance sampling, the paper achieves a clean decoupling of alignment from the base model. This modularity allows independent scaling and optimization of each component, and can also let multiple alignment modules share a single Proposal Module for cross-domain reuse;\n\n2. The method yields a 4× speed-up for supervised fine-tuning (SFT) and a 13.3× speed-up for Direct Preference Optimization (DPO);\n\n3. The paper conducts experiments across multiple instruction-following, summarization, and preference tasks using various model configurations, demonstrating that RAM matches or exceeds full-model fine-tuning and outperforms adapter-based Aligners and DPO baselines.\n\nWeaknesses:\n\n1. While the paper states that the new algorithm “minimizes first-word latency,” it only provides a rough estimate of computation units during training. It would be helpful to include real wall-clock measurements—such as latency curves, throughput numbers, or end-user response times—and to compare these against existing methods like Aligner, LoRA, or SFT.\n\n2. The evaluation relies primarily on automatic judgments from GPT-4 and Qwen, without any human assessments to confirm consistency. I understand that rebuttal time may be limited, but including even a small human-rated subset (for example, a few dozen examples) could demonstrate alignment between the model’s automatic scores and human judgments.\n\n3. The method still employs a modest amount of SFT “warm-up,” which seems somewhat at odds with the claim of “fully freezing the large model.” This suggests that, in resource-constrained environments, one must still initiate a LLM training pipeline. It might be useful for the authors to clarify this design choice or discuss its practical implications.", "was_revised": true, "final_justification": "The authors have addressed my concerns, and I believe the quality of the paper is acceptable. Therefore, I have raised my score to 4.", "rebuttal": "Thank you for the insightful comments and helpful suggestions! We hope that our responses help alleviate the concerns that were raised.\n\n---\n\n>W1: While the paper states that the new algorithm “minimizes first-word latency,” it only provides a rough estimate of computation units during training. It would be helpful to include real wall-clock measurements—such as latency curves, throughput numbers, or end-user response times—and to compare these against existing methods like Aligner, LoRA, or SFT.\n\nThanks for highlighting the inaccuracy in our description about \"minimizes first-word latency\". Our intention was to clarify that, compared to the baseline Aligner's \"Question-Answer-Correction\" generation strategy, our method does not rely on a upstreaming complete response. This distinction allows us to significantly **reduce first-word latency**, enhancing the practicality of our method.\n\nWe would like to clarify that this paper does not focus on the overall efficiency of the RAM framework during inference, which is why we have not conducted experimental validation in this area. To address your concerns, we have included performance testing experiments. However, due to time constraints, we primarily tested the SFT, Aligner, and RAM methods, while the performance of LoRA will be supplemented in future comments. The currently completed results are as follows:\n\n| Strategy | Avg Input Length (#tokens) | Avg Output Efficiency (tokens/s) | First Token Latency (s) |\n| --- | --- | --- | --- |\n| Llama8B (SFT) | 126 | 3.50 | 0.022 |\n| Llama8B+Ali. 1B | 126 | 5.03 | 50.72 |\n| Llama8B+R.A. 1B | 126 | 4.38 | 0.31 |\n\nIt is worth noting that our method is structured as a linear combination of two autoregressive models $P_\\theta(y_l|y_{<l},x)=\\frac{P_M(y_l|y_{<l},x)Q_\\theta(y_l|y_{<l},x)}{Z_\\theta(y_{<l},x)}$ (refer to Equation 9). During the decoding phase, $P_M$ and $Q_\\theta$ exhibit no dependencies, allowing for parallel processing at each iteration. Consequently, compared to the SFT model, RAM primarily incurs additional time for self-normalized importance sampling (referred to as *Proposing-Aligning-Reducing Sampling* in L137).\n\nAdditionally, This linear combination structure mentioned above, which inherently supports the symmetry necessary for mutual importance weighting between LLMs. This foundation enables us to explore \"speculative sampling\" through chunk-level decoding, which proposes draft sampling conversely through the smaller Resisual Aligner and follows a smart rejection by Proposal Module—an avenue we plan to pursue in future work.\n\n---\n\n>W2: The evaluation relies primarily on automatic judgments from GPT-4 and Qwen, without any human assessments to confirm consistency. I understand that rebuttal time may be limited, but including even a small human-rated subset (for example, a few dozen examples) could demonstrate alignment between the model’s automatic scores and human judgments.\n\nWe apologize for the time constraints that limit our ability to conduct a comprehensive manual evaluation of the experimental results. Here, we randomly selected 50 samples labeled \"Chosen\" from the test set of the Harmlessness task in Anthropic-HH to assess the win rates of each *Strategy* against the \"Chosen\" samples, up to total 350 manual comparisons. The result is list below:\n\n| Strategy | GPT4-Eval | Human |\n| --- | --- | --- |\n| SFT 8B | 65.31 | 65.88 |\n| DPO 1B | 60.44 | 65.87 |\n| SFT 8B+Ali. 1B | 48.85 | 52.77 |\n| SFT 8B+R.A. 1B | 66.67 | 71.67 |\n| --- | --- | --- |\n| DPO 8B | 73.06 | 74.00 |\n| DPO 8B+Ali. 1B | 70.12 | 62.89 |\n| DPO 8B+R.A. 1B | 79.89 | 78.05 |\n\nThere are discrepancies between the manual evaluations and GPT-4 Eval, primarily due to the limited sample size, which introduces significant variance. While it is difficult to definitively conclude whether manual evaluations align with LLM assessments, the manual evaluation trends indicate that the RAM strategy's relative advantage is likely preserved. We hope the following results address your concerns.\n\n---\n\n>W3: The method still employs a modest amount of SFT “warm-up,” which seems somewhat at odds with the claim of “fully freezing the large model.” This suggests that, in resource-constrained environments, one must still initiate a LLM training pipeline. It might be useful for the authors to clarify this design choice or discuss its practical implications.\n\nWe appreciate your feedback regarding the lack of context for the \"warm-up\" and will clarify this in future updates of the paper.\n\nWe utilized the warm-up phase in experiments starting from the original pre-trained model. Pre-trained models often struggle with newly introduced special tokens (OOD tokens) in conversation tasks, such as `
`, `
`, `
`, in the Llama and Qwen families. These tokens are integrated into conversation data during the \"instruction-following\" phase, allowing models to learn from them. Thus, our warm-up phase serves to prepare the model for understanding these special tokens.\n\nIt is crucial to clarify that our method does not primarily address OOD issues. As stated in L67-L68, \"The goal of alignment is to utilize instances from the biased subset $S$ to adapt the model $P_M$, aiming to make it a better estimator of $P_S$.\" Our objective is to align the model with the distribution of a specific domain subset, contingent on the proposal model's awareness of this distribution. This awareness supports our assertion that \"$P_S$ is supported by $P_M$\" (L78), ensuring that the importance weight $W=P_S/P_M$ remains bounded. The warm-up phase is essential for establishing this awareness, particularly concerning OOD token, allowing us to subsequently treat the proposal model as \"fully frozen.\"\n\nTherefore, the warm-up phase is a necessary step for aligning pre-trained models in conversation tasks, rather than a contradictory condition.\n\n[If the above information suffices, please disregard the following.] To elaborate on our choice of starting with pre-trained models in our experiments:\n\nWhile a robust model like GPT-4o would ideally serve as the Proposal Module for the Residual Aligner, several challenges hinder this approach: To our knowledge, there is no lightweight, open-source alternative that matches GPT-4o's vocabulary. And the lack of widely-used datasets for alignment tasks that surpass the quality of GPT-4o complicates efforts to enhance its performance through training a Residual Aligner on these \"lower-quality\" annotations. Additionally, experiments based on non-public datasets face challenges regarding evaluability and reproducibility.\n\nGiven these factors and aiming for a proof of concept, we opted for newer open-source models like Llama 3.1-8B and Qwen 2.5-14B, which can be fully parameter-trained within limited resources. We utilized pre-trained models to simulate scenarios with sufficient generalization capabilities but weak domain adaptation, alongside DPO models to represent robust domain capabilities still requiring enhancement. This approach allows us to effectively address the model optimization challenges encountered in real-world applications.\n\n---\n\nIf our responses have contributed to a clearer understanding of our work and enhanced your appreciation for its value, we would greatly appreciate your consideration in updating your rating."}, {"review_id": "O5G1H7BgZX", "reviewer": "Reviewer_GBiF", "summary": "This paper presents a clean and efficient framework for aligning large language models by treating the alignment process as importance sampling. The core idea is to keep the base model fixed and train a small residual module to steer the outputs toward preferred behaviors. The method is practical—it reduces training cost and improves inference speed—while maintaining solid performance across tasks. Although its generality is somewhat constrained and comparisons to stronger baselines are limited, the overall contribution is clear, timely, and useful.", "questions": "1.In the experimental section, the definition of the “warmed-up model” is unclear. Could the authors clarify the difference between a warmed-up model and an SFT model?\n\n2.Compared to recent research that uses value functions for guided generation—also a form of weak-to-strong generalization—does your method offer any advantages? Could the authors elaborate on the relationship between your approach and these reinforcement learning-based methods?\n\n3.Why did you choose token-level decoding? Wouldn’t this significantly increase decoding time? Have you considered chunk-level decoding or other alternatives? Just curious.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. The paper presents a smart and lightweight framework for aligning large language models by introducing a separate residual alignment module. This modular design allows the main model to remain frozen while only a small aligner is trained, leading to reduced computational cost and easier deployment.\n\n2. Framing the alignment process as importance sampling is both elegant and practical. It offers a clean probabilistic interpretation of how the residual aligner adjusts the output distribution.\n\n3. The paper is well writtened.\n\nWeakness:\n\n1. While the results look promising, the Residual Aligner (R.A.) appears to be heavily dependent on a strong proposal model. For example, in the HH experiments, although the 8B SFT + R.A. (3B or 1B) outperforms the SFT baseline, it still lags significantly behind the DPO method. Does this imply that the proposed method only works well when the base model is already strong? In that case, why not directly improve the base model instead?\n\n\n2. The baselines used in the paper SFT and DPO are relatively outdated. I suggest including one or two additional recent baselines beyond Aligner to strengthen the empirical evaluation. For example, you might consider incorporating a controlled decoding approach such as:\n[1] Mudgal, Sidharth, et al. \"Controlled Decoding from Language Models.\" arXiv preprint arXiv:2310.17022 (2023).", "was_revised": true, "final_justification": "Thank you for your response. I believe my concerns have been addressed, and I’ve revised my score accordingly", "rebuttal": "Thank you for your thoughtful review and for raising the above interesting questions.\n\n---\n\n>W1-1: While the results look promising, the Residual Aligner (R.A.) appears to be heavily dependent on a strong proposal model. ... Does this imply that the proposed method only works well when the base model is already strong?\n\nWe fully agree with your observation about the performance dependence of the Residual Aligner (R.A.) on the Proposal Module's strength. Our goal is to emphasize that **even a Residual Aligner with a limited number of parameters can enhance the performance of a strong Proposal Module**.\n\nOur experiments yield two main insights: first, as you pointed out, improving the Proposal Module is indeed valuable; second, a lightweight Residual Aligner can provide additional performance gains for already high-performing models.\n\nIn our experimental design, we aimed to reflect real-world applications. Ideally, we would use a robust model like GPT-4o as the Proposal Module for constructing the Residual Aligner. However, challenges include the lack of lightweight models with the same vocabulary as GPT-4o and the unavailability of public high-quality datasets that surpass GPT-4o. When the Proposal Module (GPT-4o) surpasses the quality of available training data, training the Residual Aligner on \"lower-quality\" annotations becomes problematic. Additionally, experiments with non-public datasets encounter evaluability and reproducibility issues.\n\nGiven these constraints, we opted for newer open-source models like Llama3.1-8B and Qwen2.5-14B as our Proposal Modules. We used pre-trained models to simulate scenarios **with adequate generalization but weak domain adaptation** and DPO models to represent **robust domain capabilities that still require enhancement**. This approach effectively simulates the model optimization challenges faced in real-world applications.\n\n>W1-2: In that case, why not directly improve the base model instead?\n\nDirectly optimizing the base model for real-world applications faces critical pain points. Different downstream tasks often have customized requirements, necessitating the training of specific domain models based on the base model for each task. Beyond the training resource costs, **deploying separate domain models prevents the sharing of limited online service storage and computational resources, leading to significant inefficiencies**. In contrast, our method, utilizing a lightweight Residual Aligner, achieves domain adaptation or value alignment with fewer resources. **Multiple Residual Aligners targeting different domains can share the same Proposal Module, enabling efficient cross-domain resource reuse and enhancing the overall system's efficiency**. This principle is a core motivation of our work as mentioned in the Introduction and hope I have addressed your concern.\n\n---\n\n>W2: The baselines used in the paper SFT and DPO are relatively outdated. I suggest including ... such as: [1] Mudgal, Sidharth, et al. \"Controlled Decoding from Language Models.\" arXiv preprint arXiv:2310.17022 (2023).\n\nThank you for pointing out the related works. We will include them in the Related Work section in our future paper updates. Due to time constraints, we tested and compared the performance of the *CD* method [1] on the *TL;DR* and *Anthropic-HH* datasets, with the evaluation results as follows:\n\n| TL;DR | LC/% | WR/% |\n| --- | --- | --- |\n| W.Up 8B | 60.71 | 49.02 |\n| W.Up 8B+Ali. 1B | 44.37 | 36.59 |\n| W.Up 8B+CD 1B | 61.89 | **52.90** |\n| W.Up 8B+R.A. 1B | **65.11** | 52.13 |\n\n| Helpfulness | LC/% | WR/% |\n| --- | --- | --- |\n| W.Up 8B | 57.70 | 56.60 |\n| W.Up 8B+Ali. 1B | 44.77 | 46.81 |\n| W.Up 8B+CD 1B | 58.08 | **59.25** |\n| W.Up 8B+R.A. 1B | **65.11** | 52.13 |\n\n| Harmlessness | LC/% | WR/% |\n| --- | --- | --- |\n| W.Up 8B | 66.63 | 64.88 |\n| W.Up 8B+Ali. 1B | 64.75 | 63.87 |\n| W.Up 8B+CD 1B | 67.00 | **66.73** |\n| W.Up 8B+R.A. 1B | **67.63** | 65.79 |\n\nBased on the results, we observed that our method outperforms the *CD* method in terms of the LC metric for both the *TL;DR* and *Harmlessness* tasks, showing significantly stronger performance in the *Helpfulness* task. However, it is worth noting that the *CD* method generally exceeds our approach in the WR metric. Within the AlpacaEval 2 evaluation framework, a lower LC metric with a higher WR metric suggests that the *CD* method tends to generate longer content. We summarize the average length of each strategy with comparison to that of the base prolicy in the table below:\n\n| avg_length | TL;DR | Helpfulness | Harmlessness |\n| --- | --- | --- | --- |\n| W.Up 8B | 115 | 230 | 134 |\n| W.Up 8B+CD 1B | 126 (+11) | 310 (+80) | 164 (+30) |\n| W.Up 8B+R.A. 1B | 112 (-3) | 247 (+17) | 129 (-5) |\n\nWe speculate that the underlying issue stems from the use of value functions with fewer parameters for lightweight reweighting in *CD*. A significant concern is the direct influence of the energy function on the base policy (as outlined in Equation 6 of [1]), which can compromise its expressive capacity. This can lead to longer outputs or even result in model collapse when inappropriate hyperparameters are applied. And we will provide a detailed comparison of our method to CD in our response to \"Q2\".\n\n---\n\n>Q1: In the experimental section, the definition of the “warmed-up model” is unclear. Could the authors clarify the difference between a warmed-up model and an SFT model?\n\nWe appreciate your feedback about the ambiguity in the definition of the \"warmed-up model\", and will clarify this in future updates of the paper.\n\nIn Table 3, the warmed-up model is trained on approximately 7-8% of the entire dataset. This phase begins with pretrained models and primarily focuses on adapting the model to recognize specific tokens introduced for conversational tasks, such as `
`, `
`, `
`, etc., particularly within the Llama and Qwen model families. In comparison, the SFT model is trained on the complete dataset, which potentially include the warm-up process.\n\nTable 4 illustrates that the SFT model not only incorporates the functionalities of the warm-up phase but also addresses distribution shift issues that smaller models encounter during the DPO pre-training phase. This approach is designed to develop a robust base model capable of effectively generalizing to specific domains.\n\n---\n\n>Q2: Compared to recent research that uses value functions for guided generation—also a form of weak-to-strong generalization—does your method offer any advantages? Could the authors elaborate on the relationship between your approach and these reinforcement learning-based methods?\n\nTaking [1] as a reference for comparison, the main differences lie in:\n\n| Strategy | Fundamental | Model Characteristics | Decoding Strategy |\n| --- | --- | --- | --- |\n| Value functions for guided generation (*CD*) | KL-regularized RL | Residual EBM [2] | Probability reweighting |\n| Ours | Importance sampling (IM) | Linear combination of auto-regressive LLMs | Self-normalized IM (SNIM) |\n\nAdvantages of Our Method:\n\n- **Theoretical simplicity**: Our approach is theoretically straightforward, making it easier to understand and implement.\n- **Symmetrical modeling**: The model is structured as a linear combination of counterpart auto-regressive LLMs, which inherently supports the symmetry necessary for mutual importance weighting between LLMs. This foundation enables us to explore \"speculative sampling\" through chunk-level decoding, which proposes draft sampling conversely through the smaller Resisual Aligner and follows a smart rejection by Proposal Module—an avenue we plan to pursue in future work. In contrast, the Residual EBM [2] model, which combines LLMs with an energy function, lacks this extensibility.\n- **Mitigating degradation**: When employing value functions with fewer parameters for lightweight reweighting in *CD*, a significant concern arises: the direct influence of the energy function on the base policy (as outlined in Equation 6 of [1]) can compromise its expressive capacity. This may result in longer outputs or even lead to model collapse with inproper hyperparameters. In contrast, our approach, utilizing SNIM (referred to as *Proposing-Aligning-Reducing Sampling* in L137), allows for sampling from the `Top-P/Top-K` outputs of the base policy to prioritize basic fluency. This is followed by a secondary sampling step tailored to contextual alignment needs, effectively mitigating the risk of degradation. Furthermore, SNIM functions as a biased estimator with reduced variance, enhancing overall performance.\n\nOverall, our method not only addresses the limitations associated with reinforcement learning-based approaches [1] but also enhances the robustness and expressiveness of the generated outputs.\n\n[1] Mudgal, Sidharth, et al. \"Controlled Decoding from Language Models.\" arXiv preprint arXiv:2310.17022 (2023)\n\n[2] Deng Y, Bakhtin A, Ott M, Szlam A, Ranzato MA. Residual Energy-Based Models for Text Generation. ICLR'2020.\n\n---\n\n>Q3: Why did you choose token-level decoding? Wouldn't this significantly increase decoding time? Have you considered chunk-level decoding or other alternatives? Just curious. \n\nThis paper primarily focuses on applying importance sampling to model lightweight domain adaptation, along with related training and inference methods, but not incorporated efficiency optimization into this study. However, as previously mentioned, the symmetry structure of our model supports the mutual importance weighting, allowsing us to explore \"speculative sampling\" through chunk-level decoding. We are already conducting related research works.\n\n---\n\nIf our responses have contributed to a clearer understanding of our work and enhanced your appreciation for its value, we would greatly appreciate your consideration in updating your rating."}, {"review_id": "2wasQpLQVx", "reviewer": "Reviewer_K98W", "summary": "This paper proposes a framework called RAM to make the process of aligning LLMs more efficient and less resource intensive. The proposed method reframes alignment using the principles of importance sampling, separating the model into a large, fixed base model and a smaller, adaptable alignment module. Here, a primary contribution is this detached framework, expressed as a product of the two modules' probabilities, which permits the smaller aligner to be trained independently. The paper also presents an iterative as well as token-level decoding algorithm to address the issue of first-token latency often seen in similar multi-stage models. The effectiveness of the proposed pipeline is demonstrated through experiments on different tasks with varying specifications, including instruction following and preference optimization. It is shown to generally perform better than baseline methods like standard fine-tuning.", "questions": "Please see my comments above in Strengths And Weaknesses", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n\n+ This paper tries to tackle a real and practical problem of making LLM alignment affordable and efficient. The proposed idea is intuitive: using one large base model that can be adapted for many different tasks with small and inexpensive \"aligners\" seems like a pragmatic and scalable approach for real-world use. To achieve this objective, authors propose to apply importance sampling to formulate the alignment process, where the pretrained frozen model refers to the proposal distribution, while the adapter estimates the importance score.\n\n+ With theoretical insights, authors propose a training pipeline utilizing an efficient sequence-level training objective, by increasing the probability of desired responses and using the proposal distribution as a regularization.\n\n+ Experiments are performed on multiple benchmark datasets \n\n\nWeakness:\n\n1.\tI am a little confused about the statement that “Importance sampling …, which is useful when direct sampling is difficult or of large variance”. Although being unbiased, it is well known that the importance sampling in its original form can be unstable due to the high variance problem. In this case, how will your proposed solution tackle this problem when the target and proposal distribution deviates from each other. Meanwhile, will the unbiasedness introduced by importance sampling matter here? If not, is it possible we can substitute importance sampling with some biased estimators for reduced variance for enhanced training stability?\n\n2.\tIn the proposed pipeline, it seems that the aligner's role is limited to re-weighting the pre-selected set of candidates, where it cannot introduce new tokens that were not initially proposed. This can create a fundamental bottleneck. If the optimal, aligned response requires a token that the unaligned Proposal Module deems low-probability and thus fails to include in its initial set of proposals, the RAM framework has no mechanism to generate it. The final output is therefore constrained by the quality and diversity of the initial proposals, which actually corresponds to the main motivation of this work of adding an additional aligner module.\n\n3.\tIf my understanding is correct, one assumption here is that the Proposal Module and the Residual Aligner will share the same vocabulary space, which can lead to restricted application scenarios as well as the plug-n-play nature of the proposed pipeline. In this case, these two modules will need to come from the same model family of the same generation, so that they will be able to share the same vocabulary space. Authors also mention this limitation in their Appendix.\n\n4.\tNo statistical significance results are provided for the main results, but only for the ablation study. Meanwhile, could authors also include a study in terms of the efficiency. For instance, what will be the running time and inference time overhead for adding the proposed method?", "was_revised": true, "final_justification": "I would like to keep my current positive rating since authors' response largely consolidates my initial positive review of their work.", "rebuttal": "Thank you for the insightful comments and helpful suggestions! We hope that our responses help alleviate the concerns that were raised.\n\n---\n\n>W1: I am a little confused about the statement that “Importance sampling …, which is useful when direct sampling is difficult or of large variance”. Although being unbiased, it is well known that the importance sampling in its original form can be unstable due to the high variance problem. In this case, how will your proposed solution tackle this problem when the target and proposal distribution deviates from each other. Meanwhile, will the unbiasedness introduced by importance sampling matter here? If not, is it possible we can substitute importance sampling with some biased estimators for reduced variance for enhanced training stability?\n\nThank you for pointing out the issue with the phrase \"of large variance\" in the context of importance sampling. We intended to convey that \"importance sampling itself can increase variance,\" and we will revise it accordingly.\n\nNow, let's address how we tackle this problem:\n\nIn importance sampling, high variance in the importance weights $W=P_S/P_M$ can occur if the proposal distribution $P_M$ poorly approximates the target distribution $P_S$. This is often due to **support mismatch** and **probability mismatch**. Our task, as pre-defined in L67-L69, \"utilize instances from the biased **subset** $S$ to adapt the model $P_M$\", assuming $P_M$ supports $P_S$, thus focusing only on **probability mismatch**.\n\nThe RAM method mitigates this by introducing a learnable residual aligner $Q_\\theta$ to adjust $P_M$ for better alignment with $P_S$, reducing mismatch. And then, by normalizing with the partition function $Z_\\theta$, RAM ensures that $P_\\theta$ is a valid probability distribution. This normalization means the weights $W$ are no longer solely dependent on $P_M$, but are adjusted by $Q_\\theta$, which learns corrections for specific regions where $P_M$ deviates from $P_S$, effectively smoothing out large weights and minimizing variance.\n\nMoreover, during inference, RAM samples candidate tokens from `Top-P/Top-K` regions of $P_M$, as you mentioned in `W2`, to maintain high $P_M$ values, leading to relatively low importance weights $W$. Additionally, we employ self-normalized importance sampling in the Proposing-Aligning-Reducing Sampling (L137) to further reduce variance, even though it introduces some bias.\n\n**In summary, we propose strategies during both training and inference to reduce variance and enhance stability with biased estimators.**\n\n---\n\n>W2: In the proposed pipeline, it seems that the aligner's role is limited to re-weighting the pre-selected set of candidates, where it cannot introduce new tokens that were not initially proposed. This can create a fundamental bottleneck. If the optimal, aligned response requires a token that the unaligned Proposal Module deems low-probability and thus fails to include in its initial set of proposals, the RAM framework has no mechanism to generate it. The final output is therefore constrained by the quality and diversity of the initial proposals, which actually corresponds to the main motivation of this work of adding an additional aligner module.\n\nWe fully acknowledge your point regarding the potential oversight of tokens deemed low-probability by the Proposal Module, which may be valuable for alignment in certain contexts. This inevitably leads to an important question: **Is there a singular optimal response for alignment?**\n\nIn tasks such as **mathematics and reasoning**, there is often a definitive correct solution. However, given that mathematical and reasoning capabilities are key evaluation metrics for general large models (such as the one used as the Proposal Module), it is unlikely that the optimal response would be categorized as low-probability.\n\nIn contrast, **value alignment and domain adaptation** tasks frequently address subjective preferences. For instance, in \"Harmless\" tasks within the `Anthropic-HH` framework, responses beginning with phrases such as \"Sorry,\" \"No,\" or \"I cannot help\" can all validly express a preference to decline. Therefore, it is difficult to assert that the absence of a specific low-probability token necessarily results in a failure to align. In many cases, such a token can be effectively compensated for by other tokens or subsequent expressions.\n\nFor **custom alignment tasks that do not primarily involve language expression**—such as those requiring the organization of content in rare formats or patterns—the general model may not have encountered these formats frequently, resulting in the low-probability issue you mentioned. In such cases, we could consider expanding the `Top-P/Top-K` selection range to include the target token. Alternatively, we could compute the probability distribution across the entire vocabulary (as outlined in Equation 9), incorporating adjustments from the Residual Aligner, and sample from that distribution, at the cost of higher computational expenses and potential instability from small-scale Residual Aligner adjustments.\n\nP.S. Your question has inspired us to consider methods for detecting the rarity of preferences by comparing the distributions of the Proposal Module and the Residual Aligner during both training and ongoing inference. We aim to incorporate this dynamic feedback into our `Top-P/Top-K` adjustments in future work.\n\n---\n\n>W3: If my understanding is correct, one assumption here is that the Proposal Module and the Residual Aligner will share the same vocabulary space, which can lead to restricted application scenarios as well as the plug-n-play nature of the proposed pipeline. In this case, these two modules will need to come from the same model family of the same generation, so that they will be able to share the same vocabulary space. Authors also mention this limitation in their Appendix.\n\nYes, it is indeed assumed that the Proposal Module and the Residual Aligner share the same vocabulary space, which we have mentioned in the Appendix. This is a common challenge in methods that involve logit ensembles or probability distribution fusion [1,2]. While this may present certain limitations, it is worth noting that there are numerous model families available, offering a range of sizes from small to large. Below is an incomplete summary of some prominent model families:\n\n| Model Family | Developer | Params |\n| --- | --- | --- |\n| Llama 3 | Meta | 1B, 3B, 8B, 70B, 405B |\n| Mistra l Mistral AI | 3B-124B |\n| Falcon 3 | TII | 1B, 3B, 7B, 10B |\n| Gemma 2 | Google | 2B, 9B, 27B |\n| Phi-3.x / 4 | Microsoft | 3.8B, 7B, 14B, 42B |\n| Command R | Cohere | 7B, 35B, 104B |\n| StableLM 2 | Stability AI | 1.6B, 3B, 12B |\n| Qwen2.5/3 | Alibaba | 0.5B-235B |\n| DeepSeek-R1 | DeepSeek AI | 8B-685B |\n| Yi | 01.AI | 6B, 9B, 34B |\n\nThis diversity provides ample choices within open-source scenarios. However, challenges arise when attempting to combine modules from different model families. For example, using a closed-source LLM (such as GPT-4o) as the Proposal Module may create difficulties in finding a compatible open-source model that matches its vocabulary.\n\nFortunately, recent efforts have been made to align the vocabularies of different models, which may help mitigate these limitations in the future [3-5].\n\n[1] Huang JY, Zhou W, Wang F, Morstatter F, Zhang S, Poon H, Chen M. Offset Unlearning for Large Language Models. TMLR'2025.\n\n[2] Mudgal S, Lee J, Ganapathy H, Li Y, Wang T, Huang Y, Chen Z, Cheng HT, Collins M, Strohman T, Chen J. Controlled Decoding from Language Models. ICML'2024.\n\n[3] Xu Y, Lu J, Zhang J. Bridging the Gap between Different Vocabularies for LLM Ensemble. NACCL'2024.\n\n[4] Huang Y, Feng X, Li B, Xiang Y, Wang H, Liu T, Qin B. Ensemble learning for heterogeneous large language models with deep parallel collaboration. NeurIPS'2024.\n\n[5] Wu J, Sun H, Cai H, Su L, Wang S, Yin D, Li X, Gao M. Cross-model control: Improving multiple large language models in one-time training. NeurIPS'2024.\n\n---\n\n>W4-1: No statistical significance results are provided for the main results, but only for the ablation study.\n\nDue to budget constraints, we are unable to cover the high costs associated with calling GPT-4 to obtain statistical significance results. And, following the empirical approach outlined in paper *Aligner* (our baseline)  presented at NeurIPS 2024, we provide the win rates evaluated by the GPT-4 judger in our main experiments.\n\n>W4-2: Meanwhile, could authors also include a study in terms of the efficiency. For instance, what will be the running time and inference time overhead for adding the proposed method?\n\nTo compare training efficiency, we use the Llama3 family as an example. The pre-trained 8B model has a forward cost of 1 computational unit and a backward cost of 2 units. For the small Residual Aligner (1B), the forward cost is 1/8 unit and the backward cost is 2/8 units. In SFT, the pre-trained model requires 3 computation units (1 forward, 1 backward), while the Residual Aligner needs only 6/8 units (2 forward, 2 backward for a paired example). This results in 4x increase in efficiency for SFT with the Residual Aligner.\n\nTo compare inference efficiency, from the Equation 9, it can be observed that the Proposal Module $P_M$ and Residual Aligner $Q_\\theta$ are loosely coupled, interacting only at the final logits layer (L147-L149). Therefore, during inference, the Proposal Module and Residual Aligner can be computed in parallel instead of serially, avoiding significant time overhead. Consequently, compared to the baseline model inference, RAM incurs only the additional time required for one pass of self-normalized importance sampling (L137).\n\n---\n\nIf our responses have contributed to a clearer understanding of our work and enhanced your appreciation for its value, we would greatly appreciate your consideration in updating your rating."}, {"review_id": "HceC8oOJVl", "reviewer": "Reviewer_Nz81", "summary": "This paper proposes a Residual Alignment Model (RAM), which is based on the residual correction method of alignment but further enhanced by formalizing the alignment process as a type of importance sampling. In this framework, the unaligned model serves as the proposal distribution, and a secondary sampling model the alignment process. Instead of aligning on the full output sequence, importance sampling modifies the LLM output probability on the token level. This method enables decoupled sampling between the model and the aligner, thereby increasing the inference efficiency for alignment.", "questions": "In Eqn. 2, what’s f(x)?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**:\n1. The importance sampling method enables token-level alignment on the aligner module, which accelerates the inference process for residual-based alignment. The idea is simple and interesting. The proof is mostly clear.\n2. Extensive experiments on instruction following, domain adaptation, and preference optimization proved the effectiveness of this method. RAM also significantly accelerates the inference process compared to Aligner.\n\n\n**Weakness**:\n1. In line 105-107, why do you “maximize” the lower bound? From my understanding, the lower bound should be minimized to lower the SFT loss. This is also inconsistent with equation 8, which aims to maximize the aligner log prob over aligned responses in S and minimize on the proposal module, emphasizing learning the residuals of the probability of distributions.", "was_revised": true, "final_justification": "Not changed.", "rebuttal": "Thank you for your insightful comments. We genuinely appreciate your recognition of our work and hope our responses address the concerns raised.\n\n>W1-1: In line 105-107, why do you “maximize” the lower bound? From my understanding, the lower bound should be minimized to lower the SFT loss. \n\nThe right side of Equation 6 represents a lower bound of the SFT loss, directly minimizing this lower bound does not effectively minimize the loss itself (unlike minimizing the upper bound). Compared with Equation 4, the primary contributor to reducing the original loss is the negative term $E_{x \\sim S, y \\sim P_M}[log Q_\\theta(y|x)]$. Therefore, to achieve the goal of minimizing the loss, we can initially maximize this term, allowing the right side of Equation 6 to approach the loss as closely as possible, and then minimize the expression to achieve the target loss minimization.\n\nHowever, the adoption of this optimization approach necessitates addressing a min-max problem (refer to Equation 32 in [1]), complicating the process. Notably, the term $E_{x \\sim S, y \\sim P_M}[log Q_\\theta(y|x)]$ has some upper bound, denoted as $U$. To mitigate the negative contribution of this term, we can introduce a positive term of the form $-\\lambda(E_{x \\sim S, y \\sim P_M}[log Q_\\theta(y|x)]-U)$ in Equation 7, This compensation effectively substitutes the maximization of the term $E_{x \\sim S, y \\sim P_M}[log Q_\\theta(y|x)]$, thus formally aligning the loss. By simplifying Equation 7, we derive Equation 8, which encapsulates our objective function for minimization.\n\n[1] Song Y, Kingma DP. How to train your energy-based models. arXiv preprint arXiv:2101.03288. 2021 Jan 9.\n\n>W1-2: This is also inconsistent with equation 8, which aims to maximize the aligner log prob over aligned responses in S and minimize on the proposal module, emphasizing learning the residuals of the probability of distributions.\n\nUpon examining Equation 8, since $\\alpha=1−\\lambda$, minimizing the loss can be interpreted as minimizing the $E_{x \\sim S, y \\sim P_M}[log Q_\\theta(y|x)]$ by a factor of 1 while simultaneously maximizing it by a factor of $\\lambda$. The combined effect of these two actions results in the minimization of the term by a factor of $\\alpha$.\n\nI hope this explanation aids in your understanding of our method.\n\n---\n\n>Q1: In Eqn. 2, what’s f(x)?\n\nThe $f(x)$ is any function of interest whose expected value we want to estimate with respect to the target distribution $Q$. This is a common representation of Importance Sampling [1], although the term $f(x)$ is not explicitly referenced in this paper.\n\n[1] Tokdar ST, Kass RE. Importance sampling: a review. Wiley Interdisciplinary Reviews: Computational Statistics. 2010 Jan;2(1):54-60."}]

Author rebuttal:

Meta-review: The paper presents a study on LLM alignment, and treat the alignment problem as a form of importance sampling. After effective rebuttal and discussion period, all reviewers turn positive to the paper. The reviewers mainly acknowledge the novelty of the Residue Alignment Model (RAM) which uses a separate and smaller module for alignment leaving the base model untouched, and the importance sampling treatment of the alignment problem. The paper meets the bar of NeurIPS and I would like to recommend accepting the paper. The authors need to carefully prepare the final version to address all constructive comments from the reviewers.

AC
Decision: Accept (poster)