Title: Flexible Realignment of Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Realignment

Reasoning

Dialougue

Hybrid model
Abstract: Realignment becomes necessary when a language model (LM) fails to meet expected performance. We propose a flexible realignment framework that supports quantitative control of alignment degree during training and inference. This framework incorporates **Training-time Realignment (TrRa)**, which efficiently realigns the reference model by leveraging the controllable fusion of logits from both the reference and already aligned models. For example, TrRa reduces token usage by **54.63%** on DeepSeek-R1-Distill-Qwen-1.5B without any performance degradation, outperforming DeepScaleR-1.5B’s **33.86%**. 
To complement TrRa during inference, we introduce a layer adapter that enables **smooth Inference-time Realignment (InRa)**. This adapter is initialized to perform an identity transformation at the bottom layer and is inserted preceding the original layers. During inference, input embeddings are simultaneously processed by the adapter and the original layer, followed by the remaining layers, and then controllably interpolated at the logit level. We upgraded DeepSeek-R1-Distill-Qwen-7B from a slow-thinking model to one that supports both fast and slow thinking, allowing flexible alignment control even **during inference**. By encouraging deeper reasoning, it even surpassed its original performance.


Peer reviews:
comments: SUMMARY:
This paper proposes a flexible realignment framework for LLMs, including training-time realignment and inference-time realignment. Training-time realignment utilizes logits, and inference-time realignment is implemented with a layer adaptor. Experiments on math reasoning and dialogue models show effectiveness.

STRENGTHS AND WEAKNESSES:
## Strength
1. Their framework includes LM realignment during both training and inference time. The framework is quite flexible and dynamic.
2. They study various models. For training and inference realignment, they study Deepseek. They expand their experiment to dialogue models, including Llama and Qwen. 

## Weakness
1. One concern is that the model Inference-time realignment performance relies a lot on the hyperparameter λ. The optimal λ varies across the tasks. It would be improved if the authors could provide more discussion on the intuition for how to tune λ.
2. TrRa-iter sacrifices correctness for token reduction, as well as in some cases of no iterations. It would be better if the authors could provide more discussion.
3. It would be better if they could discuss more about the motivation for studying dialogue models and the training-time realignment of those.

QUESTIONS:
1. When saying Llama 3.1 8B under the dialogue model, are you referring to chat models?
2. I wonder if you did any ablation study on the impact of adapting different layer depths?

LIMITATIONS:
Yes, the author addressed the limitations in Appendix
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWithin the problem space of aligning models, the authors use the DeRa (Decoding Time Realignment) technique to define a logit target that is interpolated or extrapolated from two models - in this case a DeepSeek-R1-Distilled-1.5B model and DeepScaleR-1.5B; they use this logit target to distill a model using a parameter-efficient finetuning technique (they use an extra layer added to the bottom, which they show to work better than simply using LoRA adapters), and this first step is called Training-time Realignment (TrRa). In a second contribution, the authors show that it is possible to apply TrRa iteratively to reach more extreme extrapolation with slightly less degradation.\n\nIn summary, the authors combine a number of ad-hoc techniques to get a similar effect as DeRa but with greater efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The approach is described in sufficient detail for others to understand and possibly reimplement the approach\n- Compared to the brute-force way of solving the same problem, the authors' approach is much less compute-heavy\n\nWeaknesses:\n- The paper does a very bad job of separating between the objective that the authors want to achieve and the way of achieving it. There is some evidence in the form of the LoRA ablation as well as the ablation of which layers to train and how to initialize them which suggests that the chosen appoach is indeed a good choice of a design to solve the problem the authors want to solve\n- The problem as stated by the authors is too narrow to be practically relevant - in my opinion the goal of the paper should be more general to be relevant, even if that means that the paper doesn't reach that goal with the present work.\n- The paper is somewhat incremental work on top of DeRa and DeepScaleR - Trading off solution quality versus length is a good problem to attack in general, but if that is the goal then the approach of the author only offers minimal novelty against what already exists\n\nQUESTIONS:\nCan you specify, in simple terms, what problem TrRa-iter solves in a more general case? Does it address use cases not already adressed by DeRa or is the main claim that you get better quality in extrapolation (noting that you're adding multiple layers at the same time if I understood the paper correctly)\n\nLIMITATIONS:\nIn my opinion, the main limitation of the paper is the very one-dimensional framing: they take a problem that is relatively narrow and then show one recipe - made from known ingredients - for solving it. The limitations that the authors list (yes their way allows to load the original model only once, but you get two times the KV cache) are all technical limitations within this narrow framing which I find unhelpful", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper proposes training-time realignment (TrRa) and inference-time realignment (InRa) to better combine the logits of base model and reference models for control the alignment. TrRa can achieve similar or better MATH500/AIME24 performance while shorten the response length half.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The idea is clear and the writing is easy to folllow.
2. The performance of TrRa is good, achieve similar or better MATH500/AIME24 performance while shorten the response length half. And the computation cost of TrRa is not high (200 step finetuning). InRa is also more flexible.
3. Trying different benchmarks and models to show the effectiveness

Weakness:
1. It seems requiring to search the best hyperparameter lambda for getting the best performance. And there are also some other parameters can be tuned (like temperature), it's not clear that if the best lambda is different for different temperature
2. may decrease the throughput and increase the cache in inference. And if training TrRa for more iterations, the performance may drop.
3. could move Fig.2 to appendix because it's common for the readers of this paper. Maybe Fig3 can using caption with TrRa and InRa to be more clear.

QUESTIONS:
The DeepSeek-R1-Distill-Qwen-1.5B's MATH500 score is 80.23 in the paper, while 83.9 in its official report. Is that difference caused by temperature difference? If so, when utilizing the same settings as official report, what would TrRa and InRa perform?

LIMITATIONS:
authors mention it in Appendix A
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03