Title: Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Alignment of Large Language Models

Importance Sampling
Abstract: The widespread adoption of large language models (LLMs) across industries has increased the demand for high-quality and customizable outputs. However, traditional alignment methods often require retraining large pretrained models, making it difficult to quickly adapt and optimize LLMs for diverse applications. To address this limitation, we propose a novel \textit{Residual Alignment Model} (\textit{RAM}) that formalizes the alignment process as a type of importance sampling. In this framework, the unaligned upstream model serves as the proposal distribution, while the alignment process is framed as secondary sampling based on an autoregressive alignment module that acts as an estimator of the importance weights. This design enables a natural detachment of the alignment module from the target aligned model, improving flexibility and scalability. Based on this model, we derive an efficient sequence-level training strategy for the alignment module, which operates independently of the proposal module. Additionally, we develop a resampling algorithm with iterative token-level decoding to address the common first-token latency issue in comparable methods. Experimental evaluations on two leading open-source LLMs across diverse tasks, including instruction following, domain adaptation, and preference optimization, demonstrate that our approach consistently outperforms baseline models.


Peer reviews:
comments: SUMMARY:
This paper proposes a Residual Alignment Model (RAM), which is based on the residual correction method of alignment but further enhanced by formalizing the alignment process as a type of importance sampling. In this framework, the unaligned model serves as the proposal distribution, and a secondary sampling model the alignment process. Instead of aligning on the full output sequence, importance sampling modifies the LLM output probability on the token level. This method enables decoupled sampling between the model and the aligner, thereby increasing the inference efficiency for alignment.

STRENGTHS AND WEAKNESSES:
**Strength**:
1. The importance sampling method enables token-level alignment on the aligner module, which accelerates the inference process for residual-based alignment. The idea is simple and interesting. The proof is mostly clear.
2. Extensive experiments on instruction following, domain adaptation, and preference optimization proved the effectiveness of this method. RAM also significantly accelerates the inference process compared to Aligner.


**Weakness**:
1. In line 105-107, why do you “maximize” the lower bound? From my understanding, the lower bound should be minimized to lower the SFT loss. This is also inconsistent with equation 8, which aims to maximize the aligner log prob over aligned responses in S and minimize on the proposal module, emphasizing learning the residuals of the probability of distributions.

QUESTIONS:
In Eqn. 2, what’s f(x)?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework called RAM to make the process of aligning LLMs more efficient and less resource intensive. The proposed method reframes alignment using the principles of importance sampling, separating the model into a large, fixed base model and a smaller, adaptable alignment module. Here, a primary contribution is this detached framework, expressed as a product of the two modules' probabilities, which permits the smaller aligner to be trained independently. The paper also presents an iterative as well as token-level decoding algorithm to address the issue of first-token latency often seen in similar multi-stage models. The effectiveness of the proposed pipeline is demonstrated through experiments on different tasks with varying specifications, including instruction following and preference optimization. It is shown to generally perform better than baseline methods like standard fine-tuning.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n+ This paper tries to tackle a real and practical problem of making LLM alignment affordable and efficient. The proposed idea is intuitive: using one large base model that can be adapted for many different tasks with small and inexpensive \"aligners\" seems like a pragmatic and scalable approach for real-world use. To achieve this objective, authors propose to apply importance sampling to formulate the alignment process, where the pretrained frozen model refers to the proposal distribution, while the adapter estimates the importance score.\n\n+ With theoretical insights, authors propose a training pipeline utilizing an efficient sequence-level training objective, by increasing the probability of desired responses and using the proposal distribution as a regularization.\n\n+ Experiments are performed on multiple benchmark datasets \n\n\nWeakness:\n\n1.\tI am a little confused about the statement that “Importance sampling …, which is useful when direct sampling is difficult or of large variance”. Although being unbiased, it is well known that the importance sampling in its original form can be unstable due to the high variance problem. In this case, how will your proposed solution tackle this problem when the target and proposal distribution deviates from each other. Meanwhile, will the unbiasedness introduced by importance sampling matter here? If not, is it possible we can substitute importance sampling with some biased estimators for reduced variance for enhanced training stability?\n\n2.\tIn the proposed pipeline, it seems that the aligner's role is limited to re-weighting the pre-selected set of candidates, where it cannot introduce new tokens that were not initially proposed. This can create a fundamental bottleneck. If the optimal, aligned response requires a token that the unaligned Proposal Module deems low-probability and thus fails to include in its initial set of proposals, the RAM framework has no mechanism to generate it. The final output is therefore constrained by the quality and diversity of the initial proposals, which actually corresponds to the main motivation of this work of adding an additional aligner module.\n\n3.\tIf my understanding is correct, one assumption here is that the Proposal Module and the Residual Aligner will share the same vocabulary space, which can lead to restricted application scenarios as well as the plug-n-play nature of the proposed pipeline. In this case, these two modules will need to come from the same model family of the same generation, so that they will be able to share the same vocabulary space. Authors also mention this limitation in their Appendix.\n\n4.\tNo statistical significance results are provided for the main results, but only for the ablation study. Meanwhile, could authors also include a study in terms of the efficiency. For instance, what will be the running time and inference time overhead for adding the proposed method?\n\nQUESTIONS:\nPlease see my comments above in Strengths And Weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a clean and efficient framework for aligning large language models by treating the alignment process as importance sampling. The core idea is to keep the base model fixed and train a small residual module to steer the outputs toward preferred behaviors. The method is practical—it reduces training cost and improves inference speed—while maintaining solid performance across tasks. Although its generality is somewhat constrained and comparisons to stronger baselines are limited, the overall contribution is clear, timely, and useful.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The paper presents a smart and lightweight framework for aligning large language models by introducing a separate residual alignment module. This modular design allows the main model to remain frozen while only a small aligner is trained, leading to reduced computational cost and easier deployment.\n\n2. Framing the alignment process as importance sampling is both elegant and practical. It offers a clean probabilistic interpretation of how the residual aligner adjusts the output distribution.\n\n3. The paper is well writtened.\n\nWeakness:\n\n1. While the results look promising, the Residual Aligner (R.A.) appears to be heavily dependent on a strong proposal model. For example, in the HH experiments, although the 8B SFT + R.A. (3B or 1B) outperforms the SFT baseline, it still lags significantly behind the DPO method. Does this imply that the proposed method only works well when the base model is already strong? In that case, why not directly improve the base model instead?\n\n\n2. The baselines used in the paper SFT and DPO are relatively outdated. I suggest including one or two additional recent baselines beyond Aligner to strengthen the empirical evaluation. For example, you might consider incorporating a controlled decoding approach such as:\n[1] Mudgal, Sidharth, et al. \"Controlled Decoding from Language Models.\" arXiv preprint arXiv:2310.17022 (2023).\n\nQUESTIONS:\n1.In the experimental section, the definition of the “warmed-up model” is unclear. Could the authors clarify the difference between a warmed-up model and an SFT model?\n\n2.Compared to recent research that uses value functions for guided generation—also a form of weak-to-strong generalization—does your method offer any advantages? Could the authors elaborate on the relationship between your approach and these reinforcement learning-based methods?\n\n3.Why did you choose token-level decoding? Wouldn’t this significantly increase decoding time? Have you considered chunk-level decoding or other alternatives? Just curious.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel Residual Alignment Model (RAM) that reframes the task of aligning large pre-trained language models to human preferences as an importance-sampling problem. By treating the frozen base model as a “proposal” distribution and training a lightweight residual aligner to estimate the density ratio between the desired (biased) distribution and the proposal, the paper achieve efficient, modular alignment without full fine-tuning. \nThe paper also introduces a token-level “propose–align–reduce” decoding algorithm to avoid sequence-level latency and demonstrates its effectiveness on multiple benchmarks, showing that RAM matches or exceeds full-model fine-tuning and other baselines while using only a fraction of the trainable parameters.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. By framing alignment as importance sampling, the paper achieves a clean decoupling of alignment from the base model. This modularity allows independent scaling and optimization of each component, and can also let multiple alignment modules share a single Proposal Module for cross-domain reuse;\n\n2. The method yields a 4× speed-up for supervised fine-tuning (SFT) and a 13.3× speed-up for Direct Preference Optimization (DPO);\n\n3. The paper conducts experiments across multiple instruction-following, summarization, and preference tasks using various model configurations, demonstrating that RAM matches or exceeds full-model fine-tuning and outperforms adapter-based Aligners and DPO baselines.\n\nWeaknesses:\n\n1. While the paper states that the new algorithm “minimizes first-word latency,” it only provides a rough estimate of computation units during training. It would be helpful to include real wall-clock measurements—such as latency curves, throughput numbers, or end-user response times—and to compare these against existing methods like Aligner, LoRA, or SFT.\n\n2. The evaluation relies primarily on automatic judgments from GPT-4 and Qwen, without any human assessments to confirm consistency. I understand that rebuttal time may be limited, but including even a small human-rated subset (for example, a few dozen examples) could demonstrate alignment between the model’s automatic scores and human judgments.\n\n3. The method still employs a modest amount of SFT “warm-up,” which seems somewhat at odds with the claim of “fully freezing the large model.” This suggests that, in resource-constrained environments, one must still initiate a LLM training pipeline. It might be useful for the authors to clarify this design choice or discuss its practical implications.\n\nQUESTIONS:\nI have listed all of my concerns in the weaknesses section, and I look forward to the authors’ response.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}