Flexible Realignment of Language Models
Authors: Wenhong Zhu
Ruobing Xie
Weinan Zhang
Rui Wang
Venue: NeurIPS 2025 poster ()
Keywords: Realignment
Reasoning
Dialougue
Hybrid model
Primary area: deep_learning
TL;DR: We propose a flexible realignment framework that enables efficient and controllable realignment of LLMs during both training and inference, addressing challenges in reasoning efficiency and personalized response balancing.

Abstract: Realignment becomes necessary when a language model (LM) fails to meet expected performance. We propose a flexible realignment framework that supports quantitative control of alignment degree during training and inference. This framework incorporates **Training-time Realignment (TrRa)**, which efficiently realigns the reference model by leveraging the controllable fusion of logits from both the reference and already aligned models. For example, TrRa reduces token usage by **54.63%** on DeepSeek-R1-Distill-Qwen-1.5B without any performance degradation, outperforming DeepScaleR-1.5B’s **33.86%**. 
To complement TrRa during inference, we introduce a layer adapter that enables **smooth Inference-time Realignment (InRa)**. This adapter is initialized to perform an identity transformation at the bottom layer and is inserted preceding the original layers. During inference, input embeddings are simultaneously processed by the adapter and the original layer, followed by the remaining layers, and then controllably interpolated at the logit level. We upgraded DeepSeek-R1-Distill-Qwen-7B from a slow-thinking model to one that supports both fast and slow thinking, allowing flexible alignment control even **during inference**. By encouraging deeper reasoning, it even surpassed its original performance.

Flexible Realignment of Language Models
Wenhong Zhu 1.2
Ruobing Xie
Weinan Zhang!?
Rui Wang!"
Shanghai Jiao Tong University Shanghai Innovation Institute
LLarge Language Department, Tencent
Abstract
Realignment becomes necessary when language model (LM) fails to meet ex- pected performance. We propose flexible realignment framework that supports quantitative control of alignment degree during training and inference. This frame- work incorporates Training-time Realignment (TrRa), which efficiently realigns the reference model by leveraging the controllable fusion of logits from both the reference and already aligned models. For example, TrRa reduces token usage by 54.63% on DeepSeek-FFe -i-eck-----still----- Sweiill--ee--lll 1.5B without any performance degradation, outperforming DeepScaleR- we introduce layer adapter that enables smooth Inference-time Realignment (InRa). This adapter is initialized to perform bottom layer and is inserted preceding the original layers. During inference, input embeddings simultaneously processed by the adapter and the original layer, followed by the remaining layers, and then controllably interpolated at the logit R--icck-----sil-----O-
33.86 %. To complement TrRa during inference,
identity transformation the
level. We upgraded one that supports both fast and slow thinking, allowing flexible alignment control even during inference. By encouraging deeper reasoning. it even surpassed its original performance.
from slow-thinking model
Introduction
Current large language models (LLMs), such as GPT-4o [I] and reasoning-focusse models like OpenAl-ol (2) and DeepSeek-R1 3. have achieved remarkable success. These models typically hinge on a series of critical training phases [4] First, they undergo pre-training on vast corpora to master the ability to predict the next token (5]. Next, the pre-trained models fine-tuned through
supervised fine-tuning (SFT) as a cold start to better adapt to specific domains [6] 7 Reinforcement Learning (RL) has emerged as a crucial component of the entire training pipeline.
In the RL phase, the core objective is to maximize the expected reward while incorporating the KL-divergence from the reference policy 10) The reward signal is key to alignment: correctness- based rewards improve reasoning ability 2 31 while 3H-based (honesty, harmlessness, and helpful- ness) rewards reflect human values (1I). However, misalignment can still emerge due to imperfect rewards or evolving user needs. Review the pain points the existing product models: The most advanced reasoning models tend suffer from the overthinking problem [12 3 leading increased computational costs. How can we realign these models for efficient reasoning to ensure user afford- ability? Meanwhile, adapt to individual user preferences, conversational models often become overly sycophantic [13]. How can we realign them to balance personalization and objective responses better? Realignment thus essential to correct model behavior and ensure robustness over time.
"Correpooning author. Code: http://ghhbb..om/wwwogg/44/../
Email: zwhong7 1ambong714@000 @4@sjtu.dd...c
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

One typical practical approach to realignment is to retrain the model under the same reward signal while exploring different hyperparameters. How- ever, for models trained via RI., this process is For instance, simply replicating the DeepSeek-RI experiments (with context lengths exceeding 32K over 8000 training steps) using a 1.SB-parameter model requires at least 70,000 A100 GPU hours[14] We need to address this challenge through a more efficient method without compromising performance. Ad- ditionally, we seek a solution that offers flexibility during training and inference.
Sofmax
Ahatgned
(1
A) Airewoenc
often resource-intensivg
Decoder layer
Decoder layer
Decoder layer
Decoder leyer
A Layer Adapter
Input token
The inputs are fed simul- tancously into the layer adapter and the original bottom layer of the L.M. The hidden states from both paths are propagated through all layers and merged at the logit level. The layer adapter en- ables flexible realignment even during inference.
Figure 1: Our InRa:
We propose a flexible realignment framework that facilitates training-time and inference-time realignment, controlling the alignment degree to satisfy different demands. (I) We draw inspiration from knowledge distillation for training- timen re- alignment (TrRa). Specifically, we realign the reference model using a teacher signal constructed from a controllable fusion of the output logits of the reference and the already aligned models. (2) We introduce a layer adapter to endow the LM inference-time realignment (InRa). This is inspired by the fact that the lower layers of the I.M are more influential than the upper layers during fine-tuning (see Sec Based on this observation, duplicate the bottom layer and insert it as an identity mapping layer before the original layers (see See 3.2). Fine-tuning restricted solely this layer adapter. As illustrated in Figure [] the input embeddings are processed through this adapter and original layers during inference, and the resulting output logits are combined via interpolation coefficient A. This design retains both paths of logits within a single model, enabling smooth and flexible control over alignment.
In summary, main contributions are follows:
We propose new post-training methods, TrRa and TrRa-iter, that use controllable teacher signal created by combining logits from reference and aligned models, overcoming the fixed teacher in traditional knowledge distillation to enable flexible training-time realignment.
We propose a parameter-effiiien fine-tuning approach called the layer adapter. It allows smooth and efficient realignment during inference within a single model.
Experiments demonstrate that our flexible realignment framework enables efficient and flexible realignment during both training and inference. For example, TrRa-iter reduces token usage by 54.63% on DeepSeek-R Disttll-Qwen--.55 without any loss in performance. Additionally, InRa has been successfully tested in practical scenarios, such as combining fast and slow thinking models and flexibly realigning with 3H values.
2 Preliminary
Autoregressive LM. Given a query sequence x := (II,..., E X. an auto-regressive LM defines probability distribution over possible response sequences y (V1 1/....., ... yn) E y The probability %o(y x) can be decomposed using the chain rule of probability as To x) =
IT No (ye V),, where V<e denotes {y1 V2: ...,..... 11-1).
Transformer Decoder Layer. The current mainstream LMS based on transformer architec- ture [15] typically have multiple decoder layers (øo, $1, ..., (r.) 4 Each layer consists of an attention component and an MLP component. Given an input h4-1, the layer computes the output h, through the following steps: h/-1 h--1 + Attention( Attention(RMMSNNngg (RMSNorm((())))) and h h/_, + MLP (RMSNorm (k/_))). Both components have a projector to ensure the module's input and output dimensions are consistent, facilitating the combination with a residual connec- tion (16).
Reward-based Fine-tuning. Given a pre-trained (Base) and typically SFT reference model n r r), RL is a commonly used post-training technique to enhance model capabilities further. The

---

optimization objective maximizes the expected reward r(z, y) while including KL-divergence term from the reference policy as a penalty. The objective is as follows:
Te (y xx) gt (y [x)
max E X (gl r(x:)) v) - Blog
where B is regularization parameter. It has closed-form solution for the aligned model, given as follows:
rd a)) exp (r( w)
(B) 2) Eg π ro (y/ x:) exp 1/r (x, 5/)
(2)
Typically, we can transform the above equation by representing r(x, y) (B) æ) + log (xx), r(( v) = loog 7tel x)
(3)
where ((x)== :- En ref (y/l x) exp ( (1) is the partition function.
Realignment. Realignment becomes necessary when the LM fails to meet expected performance. As shown Equation [] the KL. regularization parameter 3 determines how far the policy model Toly x)) deviating from its initial state T x)) [17]. To adjust the alignment strength of the LM, one can modify the value of B, which can be achieved by scaling it with factor À during training. This adjustment leads to updated optimal solution for the realigned model, expressed as T (3) x) as follows:
t ref (y
x)
( goat (sr (e)
(B/A)( x)
(4)
Ey mree (y'
xx)
x3 (3)(v)))) x (br(
However, computing Equation is infeasible due to the normalization constant involving all possible sequences. DeRa [18] demonstrates that Equation can be approximated at the per-token level through the auto-regressive property of LMs. When decoding token by token, combines the logits from the reference model, ref, with its from the 3-regularization-aligee model, he (8), each time step f, as detailed below.
io (B/A) (- x, V<r) softmax Xhς (3) (1 A)h
(5)
The interpolation parameter À functions readjusting the alignment strength during the inference. Refer to Appendix C.T the proof.
Flexible Realignment Framework
This section introduces our flexible realignment framework, which enables LM realignment during training and endows the LM with the capability of dynamic realignment during inference.
3.1 Training-time Realignment
Based on the previous description, DeRa [8] approximately doubles the decoding time and memory consumption. However, reveals appealing property: the reference and aligned models can be interpolated at the logit level. Drawing inspiration from knowledge distillation, where the student model learns from the teacher's predicted probability distribution, we propose innovative approach where Equation [ serves as the teacher's distribution to realign the reference model. Our method minimizes the following objective function:
L DKL (To( (B
(6)
3

---

Out Zero-Dut
Down Zero=Down
Ourput
X
Decoder löyer
Decoder layer
Softmax
SWIGLU
Decoder
Decoder layer
Self-Alletion
Up
Gate
Decoder
Decoder layer
v
Decoder
Decoder layer
RMSNorm
RMSNorm
Layer Adapter
Identity copy
Input
(a) Attention Component
(b) MLP Component
(x) Reference model
(b) Aligned model
Figure 2: Overview of attention and MLP com- ponent. Identity copy makes the last projector of each component with weight and bias to zero.
Figure 3: (a) All layers fine-tuning (b) Fine- tuning on the added identity layer while keeping the original layers of the LM frozen.
Flexible Control During Training In previous distillation approaches, the distribution of teachers typically remained fixed. However, TrRa introduces the capability to dynamically generate multiple teacher distributions by flexibly adjusting X. These teacher distributions are derived from reference and aligned models, enabling the interpolation and extrapolation the reward signal.
Iterative Realignment We can apply TrRa iteratively (TrRa-iter) to derive better model. Suppose denotes the base model and B the aligned model. realigned model can be derived using Objective This procedure can be iteratively extended to A and C, resulting in further realigned model.
3.2 Inference-time Realignment
To complement TrRa, aim to equip the LM with realignment capability during inference, enabling more flexible use for end users. We duplicate the original LM's bottom layer as an identity copy and insert it before the original layers. The layer adapter can be fine-tuned to inject alignment information, such as short-thinking patterns or 3-H values.
Identity copy The identity copy is defined (ha h which means the input and output are identical. This can be achieved as long as Attention(MMSNorm(/h.o MILPRRMSNOrm(A;-:))
= O and Then, the input is directly the result of the output due to the residual. We initialize the projection weight Matrrces--Wa.. in the Attention module and Woow in the MLP-as indicated by the dark purple regions in Figure setting them zero to ensure this identity property.
Layer adapter Layer expansion involves inserting additional layers into the original layer structure. Incorporating identity layers ensures that the added layers do not compromise the original capabilities
of LMs.. As illustrated in Figure 3 duplicate the bottom layer from the original model and insert it
as an identical mapping. LoRA is orthogonal to our method. The principle of LoRA [19 is given by
Wo + AW Wo + BA, where initialized with / (0, d²) and B is zero matrix. Therefore. LoRA is also initialized as an identity component. We can implement our method using trainable rank decomposition matrices.
Training As shown in Figure 3, we freeze the original layers of the LMs and perform fine-tuning only the layer adapter. This training guarantees fine-tuning line-tuning starting from the original distribution. The critical aspect of this step is the injection of the reward signal.
The decoding architecture depicted Figure 1 During the inference phase, the LM processes the input embeddings by passing them through the layer adapter alongside the original bottom layer of the LMM The hidden states generated by both layers are retained and subsequently fed into the remaining layers. The aligned and reference logits combined in the LM head layer
Inference

---

Table 1: Performance comparison of different models on three benchmarks. 'iter x' denotes
iterative realignment applied times under the same settings. Red indicates improved performance compared to DeepScalerR-1..3 1.EER--1.--PPeeeee Preview, while green indicates decrease.
AIME24
AIME25
MATHI-500
Token Reduction
Models
Pass@1
#Token
Pass
#Token
Pass@1 1
#Ioken
DeepSeek Distill -waeil--eeee-1.B
30.00
12602
19.58
12278
80.23
DeepScck-R1- -iibcck--RI-TRR---------
0.5
38.33
10678
28.75
10254
83.70
3734
17.42
DeepScaleR- 1.55-PPldR
37.50
8520
30.41
8143
85.20
3030
33.86
DeepSeck-BR -I-Taa--
41.25
8091
30.83
7353
85.20
2982
37.48
DeepSeck- R| -1----R--L.5--A 2.0
37.50
7441
28.33
6498
84.98
2897
42.11
DeepScck-RU--TRe--1BB--.
31.25
6297
27.50
5652
83.95
2844
47.83
DeepSeek R1-TTRa-1.5B--A 10.0
29.58
6004
25.00
5174
81.58
2713
50.83
DeepSeck-R -I-TrrR--ttct----5---- 2.0
29.17
5631
25.00
4434
81.35
2599
54.63
DeepScck-ER R11-T/Ra-iter2-1.5555 2.0
14.58
4294
15.42
3887
75.93
2483
60.48
using the interpolation parameter A. This parameter functions similarly to a temperature setting, enabling users to customize the desired alignment strength smoothly. Merging hidden states in early layers can hurt model performance, often causing repeated outputs like "!!!!!!
3.3 Discussions on Trainning/afeernnc-tinne Realignment
TrRa and InRa are orthogonal. (1) TrRa realigns the model during training, ensuring flexibility and performance. In contrast, InRa also requires training; however, its training phase focuses on injecting the reward signal into the layer adapter, and this training can be done via SFT, DPO, or TrRa. See Appendix for justification. (2) InRa retains both aligned and reference logits, enabling realignment during inference. This feature incurs additional KV-cache storage overhead. Although we integrate InRa into the vLLM framework (20) leads decrease in inference throughput. See Appendix A for potential solutions.
Experiments
In Section 4.1
demonstrate the effectiveness of TrRa during training. Section 4.2 presents an extension of the current slow-thinking model into slow-fast thinking framework. In Section 4.3 we explore the integration of 3H-values into the chatbot model. In the latter sections, we evaluate the effects of realignment during inference.
4.1 Training-tine Realignment
Evaluation Settings. (a) Models and Baselines: We use DeepSeck-R --ii-----iiiilieee------
as (trained on 40K high-quality math problems
our reference model and DeepScaleR- aligned model. (b) Calibrated Training Datasets: We use the MettR--Math22222 dataset (21. Due to computational constraints, we filter samples with generation
with 3,800 A100 hours)
lengths between 4k and 8k. (c) Evaluation Dataset: We evaluate challenging reasoning tasks including AIME-24, AIME-25, and MATH-500 to assess performance. (d) Setup: We realign --ieck--RR--isill-----ee-e 1.5B for 200 steps with batch size of 16. Performance is measured using the Pass @ I metric and token count, where we sample 8 generations per example and report the average score. Each generation has maximum length of 16384 tokens, with temperature set 0.7 and top-p set to 0.95.
As shown in Table I DeepScaleR-IL -.5BB --.e-rew demonstrates strong performance and efficient reasoning. By applying our TrRa method to realign DeepSeek-R1-Distillle R1-Distill-QWwen- 1.5B, we make the following observations:
Results.
(a) TrRa an efficient and flexible alignment controller. can be seen that we achieve effective realignment at very low cost compared to the training cost of DeepScaleR. With correction applied using a context length of just 4k to 8k, the model generalizes well to 16k during inference. Besides, can control the degree of alignment achieved by sweeping over different values of X.
5

---

Taken Count
Possg
AIME-24
AIME-25
MATH-500
32,02%
32.03%
12000
10
30, 12000
20.0
5000
0%
30.00
13,58h
75.0g
000D
204
0000
0000
3000
70.0
15.
2200
1010
60.795
3.78
5.0
2.31%
5.0
0.0
0.5
1.0
15
0.0
0.5
1.0
0.0
1.0
AIME-24
AIME-25
08.
68.98%
MATH-500
90.0
10000
50.4
8000
40.0
1000
2000
9000
2000
20.0
11.255 35%
20.0
10.85%
1000
70.0
10.0
7.50%
55.0
0.0
0.5
3.0
155
0.0
0.5
3.0
1.5
0.0
1.0
Figure 4: Reasoning Performance on different models and benchmarks with our InRa, verifying the successful interpolation and extrapolation of realignment. À = O means merely using slowing thinking, while À indicates solely using fast thinking.
(b) TrRa leads to a more efficient reasoning pattern. By appropriately increasing the value of the reasoning becomes concise without sacrificing correctness. Notably, even at À 10, the model achieves a 50.8% reduction in tokens while outperforming DeepSeek BeeoSeek-RR-Disttine I--DisiillQWeen 1.5B.
(c) TrRa-iter can further amplify this efficient reasoning pattern. Iterative realignment results in more efficient reasoning than setting a large initial X. However, as the iteration of realignments increases, the model tends to produce more concise reasoning at the cost of lower correctness.
4.2 Inference-time Realignment for Reasoning
This section explores integrating fast and slow thinking modes into a unified model. Within this model, floating-poine ppintt hyperparameter like temperature can smoothly adjust the balance between the two modes of thinking.
Evaluation Settings. (a) Models and Baselines: We adopt
-iiiieek-R--iiilil------- 1.55B (
and DeepScck-RI-Distineee Disl--iilll--wenn-777 (3] as our primary models. (b) Training Datasets: We perform SFT on short CoT segments from the OpenRI -Mathh-220- dataset (after the </think> tag), yielding controllable reasoning models named by adding the -InRa suffix to the original models. (c) Setup: We train our model for three epochs using batch size of 128. (d) Evaluation: The evaluation setting the same as in Section
Results. The results are shown in Figure We provide the following analyses:
(a) The degree of alignment could be flexibly adjusted even AFTER training. It confirms the practicality of our proposed InRa.
(b) InRa enables continuous transformation of reasoning tokens by tuning the realignment parameter X. Generally, the extent of reasoning significantly changes around X = 0.5. For detailed examples, see Appendix
(c) Extrapolation further enhances model performance. When > encourages the model to use fast thinking. We can see that fasting thinking sacrifices reasoning accuracy with the token decreasing. Surprisingly, we even found that perform better. Notably, the reasoning accuracy on all three benchmarks exceeds that of the original reasoning model (e.g., DeepSeek-RI-Disseee RI--Disilll--een-71 with nearly 4% average improvements on AIME-24, AIME-25. and MATH-500.
may encourage the model to think more and

---

Table 2: Evaluation results of models across different settings and benchmarks. LC and WR refer to length-controlled and raw win rates, respectively. At 0.0, the model corresponds to the original SFT version, while at À = 1.0, it represents our efficient fine-tuning method using DPO. Red indicates improved performance compared DPO full fine-tuning, green indicates a decrease.
Llama3 2---------------
Llama3.1 1-88-B.-1----- BBB-Base
Method
AlpucaEval2
Arema-Hlad WR (%)
MT-Bench
AlpacaEvvl22
Arena-Haadd
MT MTT-Bench
LC (%)
WR (%)
1-turn
2-turn
LC (%)
WR
WR (%)
1-tum
2-tum
SET
2.83
2.42
2.00
6.39
5.53
3.06
2.79
3.60
6.99
6.43
DPO
11.44
10.47
11.60
6.98
6.51
20.16
15.02
26.20
7.66
7.28
DeRay=20
4.50
4.97
19.80
6.30
5.30
28.63
25.68
36.30
7.74
7.45
7.28
6.62
6,90
6.66
6.18
12.19
9.80
14.00
7.26
7.03
In
11,53
11.93
11.00
7.11
6.36
20.45
17.86
24.40
7.46
7.13
12.71
14.84
17.30
6.88
6.74
21.99
20.81
29.80
7.67
7.31
InRax=20
12.01
14.92
21.80
6.93
6.24
20.90
21.09
34.30
7.36
7.04
Qwcn2.5-1.5B-BB. ---BBBs
Qwcn2.5-7B-Bal
Method
AlpacaEval2
Arena-Hard WR (%)
MT-Bench
AlpacaRval2
Arena-Hard WR (%)
MT MTT-Bench
LC (%)
WR (%)
1-turn
2-turn
LC (%)
WR (%)
1-tum
2-turn
SFT
2.55
2.54
2,80
6.85
5.30
5.42
3.59
8,30
7.43
7.01
DPO
8.38
8.36
15.60
7.49
6.68
25.20
20.92
45.50
8.21
7.80
DeRaz=20
4.50
4.97
9.90
7.09
6.59
36.53
34.73
58.00
8.59
8.28
Insa
4.75
5.04
7,00
6.98
6.13
14.19
11.52
30.50
8.17
7.53
InRa
9.13
10.50
14.80
7.51
6.59
25.74
25.83
45.10
6.70
InRax=156
8.64
11.23
16.30
7.48
6.54
30.69
34.15
8.48
5.94
In
7.15
10.38
15.30
7.58
6.68
30.99
35.35
50.30
8.51
5.58
4.3 Inference-time Realignment for Dialogue Model
This section explores the 3H-values realignment in dialogue models. The GPT-4o sycophancy incident [13] on April 25th, 2025, highlights the importance of balancing the reward signals in dialogue systems, and thus flexible alignment controller desired.
Evaluation Settings. (a) Models: We implement our proposed method on the Llama3.2-4 [2)
Llama3.1 1-8B 22 Qwen2.5-1.57 23 and Qwen2.5-7B models (23) (b) Baselines: Full fine-tuning
using DPO 10 and the DeRa method [8]. (d) Setup: We first train the base models using the
UlrrChatt-00kk dataset [24], which contains 1.5 million high-quality multi-turn dialogues, obtain
the SFT models. Subsequently, we apply DPO on the UltraFeedback dataset (25), which emphasizes 3H-values. (c) Evaluation: We evaluate our models primarily three benchmarks: MT-Bench (26).
AlpacaEval 2 (27,, and Arena-Hard v0.1 (88
Results. AlpacaEval2 and Arena-hard are designed to evaluate the alignment performance (3H- values), to assign higher scores to responses preferred by humans 28). MT-Bench a benchmark that assesses model's ability to engage in multi-turn dialogue and accurately follow instructions 226
The results are presented Table 2 and we have the follow observations:
(a) Layer Adapter has comparable alignment performance with full fine-tuning. The SFT model shows strong instruction-folloving capabilities, as evidenced by its MT-Bench scores. However, its alignment performance reflecting the 3H-value remains limited. However, DPO full fine-tuning significantly enhances alignment performance. Furthermore, we evaluate our proposed InRa with X = 1.0. The results indicate that the alignment performance is par with that of the fully fine-tuned DPO model. Compared to DeRa (which uses SFT and DPO models), InRa achieves comparable performance with significantly higher computational efficiency by utilizing only single adapter layer.
(b) Interpolation and extrapolation of realignment. Taking the Arena Arena-hard benchmark as an example, when À = 0.5, the alignment strength lies between the SFT model and the InRa model with X = 1.0. By appropriately increasing the value of X, the alignment ability can be further enhanced, even surpassing the performance of the DPO fully fine-tuned model. similar phenomenon is observed for AlpacaEval and the first-turn dialogue in MT-Bench.
7

---

(c) InRa offers quick way to study alignment tax. However, all MT-Beech results reveal decline in the model's second-round conversational abilities. As shown in Table the Qwen2.5-7B model's MT-Bench score decreases as À increases in the second-turn dialogue. We attribute this behavior alignment tax. 'Tee layer adapter was fine-tuned on the Uttrafeedback dataset, composed solely of single-turn dialogues [25]. As result, increasing À enhances the model's 's ability to adhere
to single instructions with 3-H values. We provide the case study in Appendix F.2 It also reconfirms the importance of flexible realignment for diverse practical demands.
5
In-depth Model Analyses
5.1 Layer Significance
We opt to experiment by freezing the lower layers and fine-tuning the top-k layers, as well as by freezing the upper layers and fine- tuning the bottom-k layers. This way, we aim determine which layers are most ef- fective for alignment. The learning rate is 5e-6, and B equals 0.01. As shown in Ta- ble [3] tuning the top layers brings limited gains, while adjustments to the bottom layers lead to substantial improvements, highlight- ing their critical role in preference learning.
Table 3: The significance layers in alignment on Llama3.1-8B
AlpacaEval2
Arena-Hard WR (%)
MT-Bench
Method
I.C. (%)
WR (%)
Score
top-1 top-3
4.03
4.87
8.20
5.67
4.05
4.96
10.40
6.32
bottom-1
14.94
13.93
24.80
7.37
bottom-3
16.38
15.51
25.30
7.40
5.2 Layer adapter
Initialzation For 2D tensors, Kaiming initialization [29], and for ID ten- sors, apply standard normalization. As shown in Table 4 a good initialization more effective when starting from the orig- inal weights.
Table 4: Initialization method comparison for layer adapter alignment on Qwen2.5-7B
AlpacaFval2
Arena-Hard MT--Bench
Method
LC (%) WR (%)
WR (%)
Score
Random
20.00
20.75
36.10
7.54
Identity copy
25.74
25.83
45.10
8.21
Comparison with LoRA We compare our layer adapter with the LoRA method, shown Table and Table 6 We fine-tune the LMs using LoRA with ranks r and r 128. Our method achieves performance comparable to full fine-tuning, offering improved training efficiency. Moreover, demonstrates certain performance advantages over LoRA.
Table 5: Evaluation results across different fine-tuning meth- ods on Qwen2.5-7B.
Table 6: Efficiency comparison: training parameters and training time on Qwen2.5-7B.
AlpacaEval2
Arena-Hard
MT-Bench
Method
LC (%)
WR (9)
WR (%)
Score
Method
Params
Time
Full
25.20
20.92
45.50
8.21
Full
7264M
3h35min
Lorar=s
22.04
18.64
41.50
8.16
Lora,-s
19M
Ih50min
Lorar=128
24.97
19.22
42.50
8.42
Lora,-12s
128
308M
Ih50min
Ours
25.74
25.83
45.10
8.21
Ours
222M
50min
Increasing Layer Adapters In Analy- sis5.1, we know that alignment on the bottom layers would be beneficial. Therefore, we are wondering if we can add more adapters preceding the original layers to improve the alignment ability of LM further. We copy the bottom layer n times and perform align- ment these layers. As shown in Table
Table Increasing layer adapters on Llama3.1 1-8B
AlpacaFval2
Arena-Hard WR (%)
MT-Benchh
Method
I.C. (%)
WR (%)
Score
layer
17.66
14.61
25.40
7.38
+2 layers
14.27
12.67
24.70
7.42
+3 layers
14.78
13.22
23.40
7.03

---

increasing the number of layer adapters does not significantly improve performance under the same hyperparameter settings.
Hyperparameter Stability
Table 8: Hyperparameter stability on Qwen2.5-7B
We try different and learning rate combinations in the DPO algorithm to test layer adapter training stability. As shown in Table 8 using a smaller 3 yields significant performance improvements. Moreover, it
AlpacaEval2
Arena-Has Hana-Hard
MT MT-Bench
Method
I.C. (%)
WR (%)
WR (%)
Score
Layer adapier
0.01, Ir 5e
20.66
19.25
38.30
8.48
0.01, Ir
17.53
17.34
33.20
7.87
0.005, = 5e 6
19.94
19.64
39.90
7.87
has been observed that appropriate 3 value employed in layer adapter training is also well-suited for DPO full fine-tuning. Therefore, method can be lightweight proxy for hyperparameter tuning
S 0.005, Ir = 5e
25.74
25.83
45.10
8.21
Full
3 0.01, Ir 5e
12.05
12.41
31.00
7.84
0.01, Ir 5e 25.20
20.92
45.50
8.21
0.005, 5e 6
6.09
6.83
9,70
4.56
5 0.005, = 5e 26.94
22.30
46.70
8.45
before switching to full fine-tuning. Additional experiments with LoRA, presented in Appendix F.5 further highlight the advantages of our method.
Related Work
Parameter Efficient Fine Tuning. Parameter Efficient Fine-Tuning (PEFT) aims to adapt large pre- trained models to downstream tasks by updating only a small subset of parameters, thereby reducing memory and computation costs. Early approaches include adapter modules [30), where small bottleneck layers are inserted into the model and only these are fine-tuned. LoRA [19] introduces low-rank updates weight matrices, significantly reducing the number of trainable parameters without sacrificing performance. Our method is orthogonal to these methods.
Progressive Learning. Gong et al. 31 introduced stacking approach that incrementally doubles model depth to improve training effectiveness. Expanding on this concept, CompoundGrow [32 integrates FeedForward Network expansion into structured training schedule. More recently, LLama- Pro 33 employs depth growth to retain general model performance while enabling adaptation to domain-specific tasks. Our work employs depth growth at the lowest layer.
Preference Leaning. RLHF a method aimed at aligning LLMs with human values and preferences
34 The PPO algorithm [ is frequently employed. However, challenges exist throughout the RLHF process, from collecting preference data to training the model, as highlighted by Radford al. (5). Alternatively, techniques like DPO [10] eliminate the need for reward model by training LLMs directly based on human preferences. Other competing methods, including IPO (36), KTO
37.. and WSPO (38), have also emerged.
Realignment. The most effective way to achieve realignment by sweeping the hyperparameter. DeRa [1]] dynamically adjusts alignment strength at inference time using aligned and unaligned models. Similarly, WSPO (38] demonstrates that when the weak model identical to the strong model, it can regulate alignment strength during training. We are the first explore techniques that explicitly incorporate inference-time realignment considerations into the training process.
Conclusion
We introduce flexible realignment framework that addresses the realignment of LMs during training and inference. TrRa constructs controllable teacher signal from existing models, enabling efficient post-training realignment. InRa augments the model with a lightweight layer adapter, supporting inference time alignment adjustment within single model. Our experiments confirm the practicality of this framework diverse use cases, such as cost-effective reasoning and dynamic 3H alignment, pointing toward promising direction for building flexible and user-controllable LLMs.

---

Acknowledgments and Disclosure of Funding
Rui is supported by the General Program the National Natural Science Foundation of China (62176153), Ruobing is supported by the Young Elite Scientists Sponsorship Program by CAST (2023QNRC00)..
References
[1] OpenAl. Hello GPT-4o, 2024. hww.hh/ppe/pppaaa..cc///////////
[2] OpenAL. Learning reason with llms, 2024. URL https://openai.com index/ learning- Cearinng-t-raaee reason- watth-1lmss
[3] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint 12X94, 2025.
[4] OpenAl. Gpt-4 technical report, 2024. |tt/
08774
[5] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[6] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models zero-shot learners. arXiv preprint arXiv:2 2021.
[7] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409. 12122, 2024.
[8] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X1v:1707.06347, 2017.
[9] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835-10866. PMIR, 2023.
[10] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher 1) Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[11] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0558 2022.
[12] Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, al. Do not think that much for 2+ 3=? the overthinking of olkee llms. arXiv preprint arXiv:2412.21188 2024.
[13] OpenAl. Expanding what missed with sycophancy, 2025. URL BoLLhttpa///ppeeee com/ fidex/exxpadingg -otpadddingnn--syyopaaagggo
[14] Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai. Jeffrey Luo, Li Erran Li, Raluca Ada Popa. and Ion Stoica. Deepscaler: Surpassing ol-preview with 1.5b model by scaling rl. https::/pretty/ radio- nottoiioooo..n www.--rreiowwiiaaaaiaaa--aaa
notioo
Sitte/DepSScaleR-Surrpaaa Surpassing Previere win--withh-a 1- 50--BB-MMo M-----MMe-------- -y--oooe--b--Sallinng RL 1,961000c16600011ecc33330001eee3
2025. Notion Blog.
[15] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
10

---

[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[17] Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. In International Conference on Machine Learning, pages 2160-2169. PMLR, 2019.
[18] Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding- time realignment of language models. arXiv preprint arXiv:2402.02992 2024.
[19] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li. Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, (2):3, 2022.
[20] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 1-66, 2023.
[21] open rl. Openr1 1ppthr-mmath- mmtth 220k, 2025. URL http:://wwmniiigge.c.oo//attaata OpenRi-Math 220k
[22] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
[23] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, al. Qwen2 technical report. arXiv preprint MiXi::2407.10671 2024.
[24] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305. 140:33333.,,33, 2023.
[25] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting language models with scaled ai feedback. In Forty-first International Conference on Machine Learning, 2024.
[26] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46622 2023.
[27] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controllde alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04477 2024.
[28] Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph Gonzalez, and lon Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. 1939, 2024.
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing buman-level performance on imagenet classification. In Proceedings of the IEEE international conference computer vision, pages 1026-1034, 2015.
[30] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficiee transfer learning for nip. International conference on machine learning, pages 2790-2799. PMIR, 2019.
[31] Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. Efficient training of bert by progressively stacking. In International conference on machine learning, pages 2337-2346. PMLR, 2019.

---

[32] Xiaotao Gu, Liyuan Liu, Hongkun Yu, Jing Li, Chen Chen, and Jiawei Han. On the transformer growth for progressive bert training. arXiv preprint arXiv:2010.12562. 2020.
[33] Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ping Luo, and Ying Shan. Llama pro: Progressive llama with block expansion. arXiv preprint 4rX2v:2401.02415, 2024.
[34] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[35] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. San Francisco, CA, USA, 2018.
[36] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. International Conference on Artificial Intelligence and Statistics, pages 4447-4455. PMLR, 2024.
[37] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01366, 2024.
[38] Wenhong Zhu, Zhiwei He, Xiaofeng Wang, Pengfei Liu, and Rui Wang. Weak-to-strong prefer- ence optimization: Stealing reward from weak aligned model. arXiv preprint
2024.
[39] Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangehi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics. URL http:://axivv...t org/ abs/ abs//bbs/240000 2203 13372
[40] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https
//arxiv. org/ wwgg/ba///////o..oo..o////eeeee/ 2009. 03300
[41] Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese, 2024. URL https: http://aaxvvvvoo wtwpi//..rrgaaaa////////
[42] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2021.
[43] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
[44] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL https://zenodo.crr wwwpe/eewddoo..ooooeooooooo wwgrre/ord// 12608602
12

---

Appendix
A
Limitation
For InRa, the inference key-value cache size would double, even though we integrate our new architecture into the vLLM framework (00) which simplifies key-value cache management. However, this may lead to reduced inference throughput. We hope that future work will explore key-value compression techniques address this issue.
Future work. We list some potential future works follow:
Key-value compression. We believe that the two key-value cache paths share significant similarities, making key-value compression based on this work a valuable direction for future research.
Contrastive Reward Signal. By injecting short-thinking patterns into the layer adapter and extrapolating between short-thinking and long-thinking logits, the model can be further encouraged to engage in deeper reasoning. Therefore, designing an effective contrastive reward signal could promising direction enhance the model's reasoning capabilities.
Realignment. Current research on training-timee time realignment remains limited. We hope future work will explore this area more thoroughly, it holds potential for improving alignment and reasoning performance during model training.
Hybrid Model. A more efficient architecture could support hybrid capabilities, such as dynamically adjustable fast and slow thinking modes, allowing the model balance speed and reasoning depth based on the task requirements.
B Broader Impact
This paper presents work that aims advance the field of natural language processing. Our work has many potential societal consequences, none of which must be specifically highlighted here.
C
Proof
C.1 Approximate Token-Level Distribution
The approximate realigned model (B/2).
To (y) x)) exp
(x, v)
(B/A)) x)) Ey rof (y' cxp
(7)
3 (x., 5n)
Substituting Equation 3 into Equation 7 7yields the following equation:
rrel x))
(a)( nian (plir)
(B/A) x) Ey mrref (v' xt)
(8)
(F) (x)
x (v'
Proposition 1 can be equivalently written as
Ro(B/2)
| x. (..11=11) softmax Xh((3) + A)n.
(9)
Proof. Refer to DeRa paper [8 for the proof.
13

---

D Justification
Theorem
As shown in Rafailov et al. [10], any fine-tuned LM nhhh and its corresponding pre-trained model n= can be associated with reward function Uxa (2, y) such that solving a KL-constrained reward,function
RL problem yields the fine-tuned model as its optimal policy: x² (f xr = -²". In particular, the
implicil reward can be expressed as n (2, y) 3 log
(ulz)
(v/z)
Using Theorem D we justify that experiments in Section 4.2 and Section 4.3 are theoretically
well-founded. The models trained via SFT and DPO can be regarded as implicitly learning the reward function embedded in the dataset.
E
Detailed Experiment
Model Description Deepseek-R Miii--RR--Disillee---eeeee -wen-11.eBB and DDepseck-RI--Dssillee--No are fine-tuned using reasoning data generated by DeepSeek-R1 []. 1eepSccaeRR11.B-Preeee an LM finetuned from -eepssek-R---ssiille-----e---
using simple RI. [4]
Dataset Description OpenR1 MeenR1-MMth-- Math-220k is large-scale dataset for mathematical reasoning. It consists of 220k math problems with two to four reasoning traces generated by DeepSeck RI for problems from NuminaMath 1.5. The traces were verified using Math Verify for most samples and Llama-33-77B--Intree as judge for 12% of the samples, and each problem contains at least one reasoning trace with correct answer (1
Training framework
The training framework utilizes the LLaMA-Factory [3] repository. All epoch with warm-up ratio of 0.1.
training processes involve full fine-tuning
E.1 Training-time Realignment
We use 1.25 and À in TrRa to realign the reference model. The loss curves are shown in Figure 5 As observed, the loss converges rapidly, typically within 200 steps. The learning rate is set 10 and the batch size 16.
Tommn
(a) 1.25
(b) A
Figure 5: Comparison of two loss curves.
E.2
Inference-time Realignment for Reasoning
We employ the short CoT for SFT models, with learning rate of 2 x 10-5 and a batch size of 128. As illustrated in Figure 6 our layer adapter demonstrates its effectiveness- larger models exhibit improved learning performance on the data.
E.3 Inference-time Realignment for Dialogue Model
Training details The batch size is set to 128. For SFT training, learning rate of 2e-6 is used for all models. For the DPO-trained model, different learning rates and 3 values are explored, and the best-performing configuration is selected.
14

---

Siocted
030
0 0000
(a) DeepSeck-R1 D-Disiill-Qwen 1.5B
(b) DeepSeck-R1-Disse -DDa----lstslll--we-TBBB
Figure 6: Comparison of two loss curves.
344
0425
370
(a) Full fine-tuning, 1r-5e-7, 3 = 0.005
(b) layer adapter, 1r-5e-7, B = 0.005
Figure 7: Comparison of two loss curves on Qwen2.5-7B model.
As illustrated in Figure 7
these results further demonstrate that, under identical hyperparameter settings, the layer adapter follows a learning trajectory similar to that of full fine-tuning.
Inference details All inferences are conducted using the vL.MM engine 20 with a temperature setting of 0.0 (greedy decoding) and a maximum generation length of 4096 tokens.
Judgement All these benchmarks are auto-evaluated using LLMs. And we use Qwen2.5-72B- Instruct 23 as the backend API to provide judgment.
E.4 Numerical Results in Se04.2
15

---

Table Performance comparison of different methods on various benchmarks.
AIME24
AIME25
MATH-500
Models
Acc
#Token
Acc
#Token
Acc
#Token
DeypSeckKI--DDsilllowee-----BB- () 0.33)
32.92
12608
21.67
12732
82.88
5042
DeepSeek-R1
(A 0)
30.00
12602
19.58
12278
80.23
4699
Seepseek-RI-Disiill---eeeen (A 0.25)
25.00
11932
21.25
11368
78.65
3321
DeepSeck- SinSscckkRL--issill--wee-----s/ (A 0.5)
14.17
8493
12.08
8303
67.75
1765
SennScce--L--DDeellll--vem------ (A 0.75)
5.42
1211
5.42
1201
61,48
515
DegpScee-Rl-DDiiill-vvvv-------- (A 1.0)
5.00
4.17
633
60.75
(A 1.25)
3.75
661
3.33
555
56.83
436
(A -0.25)
55.00
11224
44.17
10867
88.98
3925
Bitbie--RR--Disssll--wee-------- (A = 0)
51.67
10576
36.67
11279
87.83
3667
1irbsedk-K--Diiilll0oon------1-- (A 0.25)
47.50
9942
31.67
9003
87.95
DeepSeck-FR RI- Rll-Disilll-Qwen-7BB----- (A 0.5)
42.92
8718
30.00
8629
86.40
2291
-aieck--R---istte -iiilid------iiiiil---ooo------- (A 0.75)
11.25
712
7.08
608
70.85
480
DeepSeek
InRa (λ 1.0)
11.25
658
70.40
440
DeepSeek R1 -ii-ssilll--wen-----0 InRa (A 1.25)
7.50
616
10.83
640
64.98
443
Supplementary Experiments
F.1
Alignment Tax
Alignment Tax Alignment tax is the performance incurred to ensure a chatbot's behavior aligns safely and reliably with human values and intentions.
Experiments We conduct reasoning tasks to evaluate whether the layer adapter can learn the reward
signal without compromising the foundational capabilities of the LMM. We use the zero-shot setting to
test the reasoning ability across four benchmarks, including MMLUU [40], CMMLUU 41, Truthful-
QA [42]. and GSM8K 43 repo
We evaluate these benchmarks using lin-evaluationnharee [44]
Results As shown in Table 10 reasoning ability decreases slightly as model alignment improves. However, different trend is observed with TruthfulQA. This is likely because the reward signal incorporates the 3-H values into the model, enhancing its truthfulness.
Table 10: Evaluation results of models across different benchmarks. We evaluate these benchmarks repo
using lin-evvaluation-harnee 44
Llama3. 1-8B-Base
Qwen2.5-7B-BB
Method
MMLU
CMMLU
GSMSK
Truthful-QA
MMLU
CMMLU
GSM8K
Truthful-QA
InRax=0
59.78
47.56
57.01
54.29
71.39
81.84
82.22
56.37
InRax-o.s
59.80
47.74
60.05
54.83
70.80
81.53
77.75
57.26
InRax=1.0
59.91
47.86
60.65
$6.90
70.37
81.28
72.71
58.44
InRax=1.5
60.04
47.92
$7.92
59.98
70.03
80.94
68.99
59.59
InRax=2.0
60.01
48.04
54.74
61.07
69.31
80.49
65.50
61.05
Application specific reward signal.
This provides quick way identify the alignment problem introduced by
F.2
Alignment Tax Verification
alignment extrapolation appears to impair the model's 's instruction- following capabilities, a phenomenon we term the alignment tax. To substantiate this observation, we conduct the following experiment.
As discussed in Section 4.3
16

---

Experiments We perform the following experiments to verify this assumption. We train one more epoch during the SFT phase, aiming reinforce the multi-turn dialogue capability (UlraChat200k is multi-dialogue dataset).
Table 11: Performance of models on MT-Bench: The SFT model trained for two epochs using the Qwen2-7B-Base model during the SFT phase.
MT-Bench 1-turn
Method
2-turn
SFT
7.60
7.20
InRax=0.5
8.14
7.48
InRax=1.0
8.56
7.39
InRax=1.5
8.48
7.21
InRax=2.0
8.51
7.06
Results As shown in Table 11 the performance of the 2-turn dialogue has improved compared to the results presented in Table This observation validates assumption.
F.3 Flexible inference-Time Switching Mechanism
The alignment tax prevents the chatbot from directly following instructions, leading prioritize aligning with user preferences-a phenomenon also observed in the GPT-4o incident [3
Experiments We conduct experiments with varying À values in the multi-turn dialogue phase, utilizing the inference-time realignment capabilities of our InRa.
Table 12: The MT-Bench score on the second turn, using different values across the two dialogue phases.
I-turn
0.5
1.0
1.5
2.0
2-turn
À 0.0
7.24
7.61
7.58
7.39
À = 0.3
7.59
7.35
7.60
7.71
À 0.5
7.53
7.40
7.40
7.68
À = 0.7
7.18
7.45
7.19
7.39
Results As shown in Table 12 our findings indicate that lowering the À value in the second-turn dialogue helps the model better comprehend the context and follow instructions more effectively, avoiding the alignment tax problem. Refer to Appendix Fo detailed case study.
E4 The Function of Layer Adapter
We provide visualization to illustrate the function of the layer adapter. As shown in Figure X the adapter does not project the original input embeddings into a higher- sigher-dimensional space; instead, maps the unaligned input embeddings their aligned counterparts.
F.5 The results of Lora
Experiments
We sweep the hyperparameter to the different methods.
LoRA appears to require a high learning rate, whereas full fine-tuning demands lower learning rate maintain language ability and achieve strong performance. Our method aligns closely with full fine-tuning and remains stable throughout different hyperparameters.
Results As shown in Table 13
We further increased the learning rate to train LoRA, and the results are presented in Table 14 As shown, increasing the learning rate degrades the model's language capabilities. Therefore, these configurations did not produce optimal results.
17

---

Unaligned
Aligned
0.6
0.4
yaxxs
0.0
-0.
-0.6
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
X-axis
Figure Visualization of input embeddings using the PCA method, with the ID numbers representing token positions.
Table 13: Hyperparameter stability on Qwen2.5-7B
AlpacaEval2
Arena-Hard MT-Bench
Method
LC (%) WR (%)
WR (%)
Score
Layer adapter
3 = 0.01, Ir 5e 6
20.66
19.25
38.30
8.48
3 0.01, Ir 5e 7
17.53
17.34
33.20
7.87
3 0.005, lr 5e 6
19.94
19.64
39.90
7.87
3 0.005, lr 5e 25.74
25.83
45.10
8.21
LoRA,
3 0.01, Ir 5e 6
22.04
18.64
41.50
8.16
3 0.01,
5e
4.33
2.84
11.30
7.74
3 0.005. 1r 5e 6
17.99
14.98
36.10
7.99
3 = 005, lr = 5e 7
5.31
3.24
11.90
7.75
LoRA, r
128
8 0.01, Ir 5e 6
21.50
17.56
37.30
8.33
3 0.01, Ir 5e 7
6.29
4.25
13.50
7.68
3 0.005, /r 5e 6
24.97
19.22
42.50
8.42
3 = 0.005, lr 5e 7
7.12
4.46
13.50
7.64
Full
3 = 0.01, Ir 5e 6
12.05
12.41
31.00
7.84
3 0.01, /r 5e 7
25.20
20.92
45.50
8.21
3 0.005, 1r = 5E
6.09
6.83
9.70
4.56
3 0.005, ir 5e 26.94
22.30
46.70
8.45
Table 14: Increasing layer adapters Qwen2.5-7B
AlpacaEval2
Arena Hard
Method
LC (%)
WR (%)
WR (%)
B 0.01, Ir 2e 5
16,68
16.89
34.90
8 0.01.
5e
14.06
14.83
24.20
0.01,
1e 4
12.26
12.51
19.80
8 0.005, Ir 2e
0.42
1.05
0.70
8 0.005, fr 5e 4.69
4.34
2.10
8 0.005. Ir 1e 0.16
0.49
1.80
18

---

Our method can be lightweight proxy for hyperparameter tuning before switching to full fine-tuning. This is possible because the optimal hyperparameters tend be relatively consistent between the two.
Conclusion
F.6 Alignment Tax Case Study
This section presents case from MT-Bench that highlights how the alignment tax, introduced by catering to human preferences, can impair an LM's ability to manage multi-turn dialogues. 1. We begin by presenting the first-turn instruction follows:
1-turn Insturction
Read the below passage carefully and answer the questions with explanation: At small company, parking spaces are reserved for the top executives: CEO, president, vice president, secretary, and treasurer with the spaces lined up in that order. The parking lot guard can tell at glance the cars are parked correctly by looking at the color of the cars. The cars are yellow, green, purple, red, and blue, and the executives' names are Alice, Bert, Cheryl, David, and Enid.
* The car in the first space is red.
A blue car is parked between the red car and the green car.
# The car in the last space purple.
The secretary drives a yellow car.
*
Alice's 's car is parked next David's.
Enid drives green car.
&
Bert's car is parked between Cheryl's and Enid's.
David's car is parked in the last space.
Question: What is the name the secretary?
2. The model can follow the first-turn instruction well.
1-turn Response
The secretary's car identified by two clues in the passage:
1. The secretary drives yellow car.
2. Alice's s car parked next to David's. Since David's car is parked in the last space (the fifth position), and the car in the last space is purple, David cannot be the secretary. Therefore, Alice must be the secretary because her car is next to David's s, and the only car left that matches the secretary's 's description is yellow. The name of the secretary Alice.
3. We go on to request the second instruction.
2-turn Instruction
List car colors in order from last to first.
However, the model follows the second instruction well but fails to incorporate the context when generating its response.
19

---

2-turn Response
To provide list of car colors in order from last first, need to define what "last" and "first" mean in this context. Typically, "last" would refer to the most recently introduced or popular color, and "first" would refer the oldest least popular color. However, without specific criteria, I will provide a general list based on historical popularity and modern trends: Black A classic and timeless color that has been popular for decades.
2. White Another enduring color that is simple, elegant, and versatile.
Silver modern and sleek color that has been popular since the 1980s.
4. Gray A neutral and sophisticated color that has gained popularity in recent years.
Red A bold and eye-catching color that has been popular since the early 20th century.
6. Blue A calming and versatile color that has been popular since the 1960s.
Green natural and earthy color that has seen resurgence popularity in recent years.
8. Yellow A bright and cheerful color that has been popular in certain eras, such as the 1970s.
9. Pink A feminine and playful color that has been popular in various eras, including the 1950s and 1980s.
10. Purple A regal and luxurious color that has been popular in certain eras, such as the
1970s and 1980s.
This list is not definitive and can vary based personal preferences and cultural trends.
5. We modify the second-turn instruction to encourage the model to context when answering questions.
2-turn Instruction
According to the above passage, list car colors in order from last first.
As shown below, the model successfully provides the correct answer.
2-turm Response
The passage does not provide list of car colors in order from last to first. Instead, it describes specific car locations and colors based on certain positions. correct the order, we need additional information. Here is the information given in the passage:
The car in the first space is red.
2. A blue is parked between the red car and the green car.
3. The car the last space is purple.
4. The secretary drives yellow
5. Alice's car parked next to David's.
6. Enid drives green
7. Bert's is car is parked between Cheryl's and Enid's.
8. David's car parked in the last space.
From this, we can deduce the following order based on the positions:
Red car (first space)
2. Blue car (second space)
Green car (third space)
4. Yellow car (last before the last space)
Purple car (last space)
So, the car colors in order from last to first as described in the passage are: Purple, Yellow,
Green, Blue, Red. /
20

---

E.7
Thinking Case Study
Problem
Find the sum all integer bases for which 17g is divisor of 97a-
À 1.0 tokens: 429
To find the sum of all integer bases 6 >> 9 for which 17, is divisor of 97s, start by converting the numbers from base to base 10.
17, in base 10 is 1 b + 7 b + 7. 97, in base is 9 b 7 96 + 7. We need 17, to divide 97b, which translates to finding b > 9 such that + 7 divides 9b + 7. This can be expressed as:
9b + b + 7
is an integer
To solve this, we use polynomial division or the Euclidean algorithm. Dividing 9b + 7 by b + gives:
9b 7 9 + 7) - 65
Thus, the remainder is -65. For b + 7 to divide 96 - 7, it must divide 65. The divisors of 65 are 5, 13, and 65. Since 9, we consider only the divisors 13 and 65, leading to: 7 13 which gives = 6 (not valid as b > 9) 19-b-4 7 = 65 which gives b = 58
Thus, the only valid base b 58. The sum of all such bases is simply 58.
58
DeepSeek- Bitseekk-1---- R1-Distill Sellll-wenn-TTT Poven-BB-iie RB-InRa À 0.6 tokens: 720
<think> Alright, let try to solve this problem step by step. So, the question is: Find the sum of all integer bases > 9 for which 17, is a divisor 97,- First, need to understand what 17, and 97, mean in base b. In any base b, number like 17g translates to x 6 + 7 in decimal, right? Similarly, 97, translates to 9 x + 7. So, need to express these values in decimal form.
Calculating the decimal equivalents: 17, = 1 x b 7 b+ 7 97, - x b 7 - 9b + Now, the problem states that 17, divisor of 97.. This translates to the condition that + must divide 96 + 7. Mathematically, this means:
9b + = (mod b b+ +7
I remember that if divides 6, then b mod 0. So, applying this here:
9b + 7 0 (mod b+7 7)
To solve this congruence, I can use the property that = -7 (mod b 7). This is because
b = (b 7) so b -7 (mod b + 7).
Substituting = -7 into 9b + 7:
9(-7) + 63
-56
Thus, the condition simplifies to:
56 (mod b + 7)
b + must be a divisor -56. Since b+ is positive integer (as b > 9), must be positive divisor of 56. The divisors of 56 are:
1,2, 4, 7, 8, 14, 28, 56
Since 1) + 7 16 (because > 9), the possible values of b are 28 and 56.
Therefore, solving for If +- 7 28, then = 21 If + 7 = 56, then = 49 Both bases 21 and 49 greater than 9. The problem asks for the sum of all such bases. Thus, the sum is 21 + 49 = 70.
**Final * The sum of all integer bases > 9 is 70
21

---

DeepSeck-RI-Diee Peeissill-Qwen---e- 7a---BB-IRaa À 0.5 tokens:
1693
<think> Alright, so I have this problem here: need to find the sum of all integer bases 6 for which 17, is a divisor of 97,, Hmm, okay. Let me try to break this down step by step. base b can be converted decimal (base 10) by multiplying each digit by b raised to the power of its position, starting from 0 on the right. So, let me convert both numbers 17, and 97, into decimal.
First, know that numbers
Starting with 17a: The rightmost digit is 7, which is in the bo place, so that's 7 1 The next digit to the left I, which in the b² place, so that's 1 together, 17, in decimal is b + 7.
b. Adding them
Now, 97p: The rightmost digit is 7, so that's x = 7. The next digit is 9, which is in the 6² place, so that's 's 9 6 9b. Adding them together, 97, in decimal is 96 + 7.
So, the problem simplifies to finding all integer bases 6 >> 9 where + divides 96 + 7. Let's denote d = b + 7. Then, d must divide 9b +
To find this divisibility, let me express 9b + 7 in terms of d. Since d = + 7, we can solve for b:
= 7-7
Substituting into 96 + 7:
((d - +7 = 9d - 63 + 7 = 9d - 56
So, d must divide 9d 56. This implies: d divides 9d 56
Subtracting 9d from both sides, we get: d divides - 56
Since divisibility is unaffected by the sign, d must be positive divisor of 56. The positive divisors of 56 are:
1, 4, 8, 14, 28, 56
But d = b+ 7 and 6 9. d must be greater than 9 9+ 7 = 16. Looking at the list of divisors, the ones greater than 16 are 28 and 56. Therefore, the possible values for d are 28 and 56. Calculating for each: If d = 28, then b 28 = 21 If d 56, then b 56 7 49
So, the bases are 21 and 49. Adding them together:
21 + 49 70
Therefore, the sum of all integer bases > 9 for which 17, is a divisor of 97, is 70
Wait a second... ** Let me double-check my reasoning. Could there be any other divisors I missed?
The divisors of 56 are indeed 1, 2, 4. 7, 8, 14, 28, 56. Those greater than 16 are only 28 and 56. Since d must be greater than 16, 21 and 49 are the only valid bases. Yep, that seems right. Just to be thorough, let me verify with these bases:
189 + 7 196 Checking 28 divides 196: 19628 7, which is an integer. So, yes, 28 divides 196.
For b = 21: 1721 = 21 7 28 9721
21
2. For b = 49: 1749 = 49 7 = 56 9749 = 49 + 7 441 448 Checking
56 divides 448: 44856 8, which is an integer. So, yes, 56 divides 448.
All checks out. So, the answer must be correct.
**Final Answer** 70 </think>
To find the sum of all integer bases b > 9 for which 17, is divisor of 97,, we first convert the numbers from base to decimal.
The number 17, in decimal is 9 x b + 7 = 9b + 7. We need 6+77 to be a divisor of 96 7. Let d b+ Then, d must divide 96 Substituting
b + 7.
The number 97, in decimal is
b = d - into 9b + 7 gives:
9 - 7) 7 9d - 56
Thus, d must divide 9d - 56, which implics d divides 56. Therefore, d must be positive divisor of 56. The positive divisors of 56 are:
1,2, 4, 7, 14, 28, 56
Since d = + 7 and 6 > 9. d must be greater than 16. The valid divisors are 28 and 56.
Calculating for each: If d = 28, then 28 - 7 = 21 - If = 56, then b = 56 7 =
Thus, the bases b are 21 and 49. Adding them together:
21 + 49 = 70
Therefore, the sum of all integer bases b > 9 for which 17, is a divisor of 97, is 70
22

---

DeepSeck-RI-Diiee Peristil-Qwee---ee Qwen- Baren-7B-InR-- À 0.0 tokens: 2192
<think> Alright, so have this problem here: need to find the sum of all integer bases b for which 17g is divisor of 97. Hmm, okay. Let me try to break this down step by step. First, let me recall what it means for number to be
certain base. If have number like 17.. that means in base b, it's equal to b + 7 x 1 in decimal, right? Similarly, 97p would be
in decimal. So, math terms, 17,
b + b + and
97, = 9 x b 1 7 96 + 7.
Now, the problem says that 17, divisor of 97.- So, in decimal terms, that means 7 divides 96 + 7. In other words, b + 7 is factor of 9b + 7. How can translate this into an equation?
Well, if b 7 divides 96 7, then 96 7 divided by b should give integer. Maybe I can perform the division and see what that gives me.
Let me set up the division: 96--7 simplify this.
Let's do polynomial division maybe use algebra to 0+7
Let's express 96 + 7 in terms of 7. Notice that 96 +b+7 7 9(b+ 63 9 7) 56. Wait, let me check that. I expand 9(b 7). it's 96 63. Then subtracting 56 gives 96 Yeah, that works.
So, 9b 7 9(b
7) 56. Therefore, dividing by b we get 96+7
b+7
6+7
For this to be an integer, b+7 must be an integer. So, b + 7 must be a divisor of 56. Got it. So, b+ 7 divides 56. Therefore, b + 7 must be one of the positive divisors of 56. Since b is an integer greater than 9, let's find all positive divisors of 56, subtract 7, and see which of those results are greater than 9.
56
First, let's list positive divisors of 56. The divisors 56 are: 2, 4, 7, 14, 28. 56.
Now, subtracting 7 from each of these, we get: 7 14 7 28 7 21 56 7 49
1- -5 4 So, the possible values for b are these results, but since b has to be an integer greater than 9, we can ignore the negative numbers and zero. That leaves us with 1, 7, 21, 49. Wait, but 1, are not greater than 9. So, only 21 and 49 are valid.
Wait, hold on, are these correct? Let me verify.
b 7 is divisor of 56, then b + 7 can be 8, 14, 28, 56. Since 6 9, 7 16. So, the possible + 7 must be divisors of that are greater than
Looking back at the list of divisors: 56's divisors are 2, 7, 8, 14, 28, 56. Divisors greater than 16 are and 56.
So, b+ 7 = 28 implies b 28 7 = 21. b+7 56 implics = 56
49.
Ah, was wrong earlier. should have considered only the b greater than 16, which gives as 21 and 49.
Therefore, the integer bases greater than 9 are 21 and 49.
Therefore, the sum of these 21 49 = 70. Wait, just make sure haven't missed anything.
Let me double-check:
Compute 17 and 97 in decimal:
17 1 x 21 + 7 = 28
9721
21 7= 189 + 7 196
Is 28 divisor of 196? Yes, because / 28 = 7, which is an integer.
Similarly, for base 49:
1749 49 + 7 56
9749 49 7= 441 + 448
Is 56 divisor of 448? 448 56 8, which is integer. Perfect.
Wait a second, just to make sure, are there any other divisors between, say, and 56? Let's see: Divisors of are as listed: 1, 4, 7, 8, 14, 28, 56. So, between and 56, the only one 28 and 56.
Wait, actually 56 is included in the first step when subtracted 7, but 56 is just one of the divisors.
So, so think I got all possibilities.
23

---

Wait, but let's make sure that 6 must be greater than 9, so in base b, the digits must be less than b. 97b, the digit 9 must be less than b, which means 9. So, that is already taken care of.
Therefore, the bases 6 > are 21 and 49, and their sum is 70. Therefore, the final answer is 70.
**Final Answer** 70 </think>
To solve the problem, we need to find the sum of all integer bases for which 17g is divisor of 97p-
First, we convert the numbers from base b to decimal: 17, = 1 x b + = 7
97, 9 + 7 96 7
We need b + to be divisor of 96 + 7. This can be expressed as:
9b +7
b+7
We simplity this division by expressing 9b + 7 in terms of b + 7:
+ 9(b + 7) 56
Thus, the division becomes:
9b
56
b 4+7 7
b+7
h+7 must be an integer. Therefore, b must be divisor of 56. The divisors of 56 are: 2, 4, 7, 14, 28, 56. Since > 9, we need b + to be a divisor of 56 greater than 16. The valid divisors are 28 and 56: If 7 28, then b 21 If + 56, then b 49
For this be an integer, .56
Thus, the valid bases are 21 and 49. Their sum is:
21 + 49 70
Therefore, the final answer is:
70
DeepSeek - R1-Distill Qwen- 7B -B-nRRa À -0.5 tokens: 3380
<think> Alright, so have this problem here: need to find the sum of all integer bases b > 9 for which 17, is divisor of 97 Hmm, okay, let me try to parse this out step by step. First off, when numbers are written with subscript like 17, or 97,, that means they're numbers in base right? So 17, isn't one seven in base b, but rather, it's the number "one" and "seven" in base b. Similarly, 97, is nine and seven in base b. Okay, got it. So, base 10, we re used to numbers like 17 being "one ten and seven units." which is 1*10 + 7*1. Similarly, 97 in base 10 is nine tens and seven units. But since we' re dealing with an unknown base b, think we need to convert these numbers into base 10 work with them more easily, since divisibility is something
can handle in base 10.
Airight, so let's 's try to convert 17, and 97, into base 10. In base b, the rightmost digit the units place, which is 62, then the next is the b' place, and so So for 176: The rightmost digit is 7, which is 7 in base 10, and the next digit is 1, which is 1 x b1. So adding those together, 17, in base 10 is 1 b + Similarly, for 97g: The rightmost digit is 7, so that's 7 x 1, and the next digit is 9, which is 6. Therefore, 97, in base is 6 7 1 96 7. So now we have 17, = 6 + 7 and 97, = 96 7. The problem states that 17, is divisor of 97,, which other words, 97, divided by 17, an integer. So mathematically, that means (9b + 7) divided by (b + 7) is an integer. So, written as fraction, it would be 96-+7 b+7 and we want this to be an integer. Let me note that down:
b+ 7. Got that.
9b+7 must be an integer.
6+7
Our goal is to find all integer bases b > 9 for which this fraction is integer, and then those bs.
24

---

So, approach this, maybe can perform the division of 9 by b+7 and see if simplifies way that gives us some equation we can work with. Let set N 9b + and D
7, we have N/ must be integer. Let me try to perform the division.
Divide N by D: (9b 7) + (b+
Let me do polynomial long division or just think about how many times + 7 goes into
9b
Well, + multiplied by is 9b + 63. Hmm, let's subtract that from 96 7:
96 7 1966-996))))
(9b+ 63) 0b 56. So, we have a remainder of -56. So, other words: 9h-77 = -9- 8-7
6+7
So. for 9b+7 to be an integer, that implies that 56 must also be an integer because 9 is 6+7
6+7
already integer, their difference has be an integer. So, 56 must be integer. 56 is integer. Therefore, b+ must be a divisor of 56. Since b 9, b 7 9 + = 16.
So, h+7
So, b++ must be a positive divisor of 56 that's greater than 16. Therefore,
find all the positive divisors of 56, then exclude those that are 16 or less, and the remaining ones would correspond to possible values of + 7, hence we can find b by subtracting 7.
Okay, so first, let me list all positive divisors of 56.
56 is equal to 7 8, which is 7 23, its prime factors are 2, 7. So the divisors be found by taking combinations of these exponents.
The divisors 56 are: 1, 4. 7, 14, 28, 56. Wait, let me check that: starting from 1, since divides everything. Then, because 56 is even.
4 because 56 2 is 28, which is 2x14, then 7 is a prime, so 7 divides 56 since 7x8=56. is 8x7, so 8 divisor. Then, 14 14x4=567 Wait, 14x4 56, yes. 14x4=56. Then, 28 is 28x2=56, and 56 itself is divisor.
8. 14. 28, 56. So, that eight divisors. Now, since b > 9, 7 > 16. So, we need divisors of 56 that are greater than 16.
So the positive divisors are 1, 2,
Looking the list: 1, 4, 7, 8, 14, 28,
Which of these are greater than 16?
28, and 56. is less than 16, so 28 and 56.
Wait, but 14 is 14, which is less than 16, doesn't qualify. Therefore, the only divisors greater than 16 are and 56.
Thus, 6 + can be either 28 or 56, so b can be 28 7 21, or 56 = 49. Wait, that seems straightforward, but let me check that's s right. So, b+ 7 = 28 = 21, and 7 56 Is that all? Let me make sure. Wait, hold digits 1, 7 and 9, 7. So, digits in base have to be less than the base. Therefore, in 17,, the digits are and so 7 has be less than b. Similarly, in 97,, the digits are 9 and 7, so 9 has to be less than b. Therefore, b must be greater than 9. Wait, the problem already states b 9. So, 7 less than so no additional constraints from that. So, that just reiterates that > 9. Therefore, our only possible bases Wait, but let mc make sure that when we have these bases, the initial numbers make sense in that base. For 17s. digits and 7, which 97, has digit 9, which is less than both 21 and 49, so that's fine too. Therefore, seems like b 21 and 49 satisfy the condition.
b 49.
second. The problem mentions that in base b, the digits in 17, and 97, have
21 and 49.
both less than 21 or 49, that's fine. Similarly,
Hence, the sum is 21 + 49.
Calculating 21 + 49: + 40 60, So, the sum should be 70.
10, so 70.
But me just double-check to make sure haven't made any mistakes. First, let's check for b 21.
Calculating 17 is * 21 + 28.
Calculating 9721 is 9 21 7. 9*20 is 180, plus 9 is 189, plus 7 is 196.
So, 196 divided by 28 equal to 7, which an integer. So, yes, 11 is valid.
25

---

Now, let's check 49.
Calculating 19: 1 49 + -7=55 56.
Calculating 49: 9 49 7. 9.49 is 360, 989 is 81, 360 + 81 441, plus 7 is 448. Dividing 448 by 56. 86 is 448, so that's 8, which is an integer. So, yes, 49 also valid. valid bases, and their sum is 70.
So, indeed, both 21 and 49
Therefore, think 70 is the answer.
But let me just go back and check if there could be other divisors which may have missed. Because sometimes, when we re working with divisors, we might miss some, especially we're only considering positive integers.
Wait, let me list all the positive divisors of 56 again. 1, 2, 4. 7, 8, 14, 28, 56. So that is all. Are there any negative divisors? Well, technically, yes. 56 has also negative divisors: -1, -2. -4, -8, -14, -28, -56.
But since b base, and base cannot be negative, so we only consider positive divisors. Additionally, b + 7 is positive number greater than 16. Therefore, we only considered 28 and 56, corresponding b and b 49. So that seems complete.
I was just thinking, sometimes with bases, the digits have to satisfy that all digits are less than the base. So, in this case, our digits are 1,7 and 9. Therefore, for each number, in its respective base, all digits must be less than the base.
But as mentioned earlier, for 17,, digits are I and 7, base must be greater than 7, and for 97,, digits and 7, so base must be greater than 9. Since the problem already specifies b > 9, so in case, b = 21 and 49 both satisfy that, so no issues there.
I also double-checked the division earlier on, seems to hold, so my conclusion seems solid. Therefore, I'm ' confident that the answer is 21 + 49 70.
**Final * The sum of all integer bases > 9 is 70 </think>
To solve the problem, need to find the sum of all integer bases b for which 17, is divisor 97g-
First, convert the numbers from base b to base 10: 17, in base 10 is 1 x b = 7. 97, in base 10 is x b+ 7 9b +
We need 17a to be divisor of 97,, which means 9h+ must be an integer. Performing polynomial long division, get:
9b +
56
b 7
b+7
For this to be an integer, 547 must be an integer. Therefore, b 7 must be divisor of 56. The divisors of are: 1, 2, 4, 7, 8, 14, 28, 56. Since > + must be greater than 16. The suitable divisors are 28 and 56.
Thus, b b
28 gives b 21 and 7 56 gives 49.
The sum of these bases is 21 + 49 70.
The sum of all integer bases b > 9 70
26

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Method, Experiments
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Appendix
27

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Preliminary and Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Experiment and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
28

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Supplementary
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ /ttppp.cc/pbllc/guudeeee http://oeeuuiiisiioooooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
29

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Experiments and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Experiments
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Analysis
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
30

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttge///eurrpp..ccppbbbbb//bbb
Answer: [Yes]
Justification: Our paper
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: No such risks
Guidelines:
The answer NA means that the paper poses such risks.
31

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Reference
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Models
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
32

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy http::///eerrpp.ccccooofereeeeep for what should or should not be described.
33


Reviews:
[{"review_id": "41gqfahoSk", "reviewer": "Reviewer_L9eG", "summary": "This paper proposes training-time realignment (TrRa) and inference-time realignment (InRa) to better combine the logits of base model and reference models for control the alignment. TrRa can achieve similar or better MATH500/AIME24 performance while shorten the response length half.", "questions": "The DeepSeek-R1-Distill-Qwen-1.5B's MATH500 score is 80.23 in the paper, while 83.9 in its official report. Is that difference caused by temperature difference? If so, when utilizing the same settings as official report, what would TrRa and InRa perform?", "limitations": "authors mention it in Appendix A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The idea is clear and the writing is easy to folllow.\n2. The performance of TrRa is good, achieve similar or better MATH500/AIME24 performance while shorten the response length half. And the computation cost of TrRa is not high (200 step finetuning). InRa is also more flexible.\n3. Trying different benchmarks and models to show the effectiveness\n\nWeakness:\n1. It seems requiring to search the best hyperparameter lambda for getting the best performance. And there are also some other parameters can be tuned (like temperature), it's not clear that if the best lambda is different for different temperature\n2. may decrease the throughput and increase the cache in inference. And if training TrRa for more iterations, the performance may drop.\n3. could move Fig.2 to appendix because it's common for the readers of this paper. Maybe Fig3 can using caption with TrRa and InRa to be more clear.", "was_revised": true, "final_justification": "I would keep my current score as reviewer solve most of my concerns", "rebuttal": "# \n\n```\n |-----------|-----------|-------------------|-----------|------>  \n 0.0         0.5        1.0                 2~5          >5.0       \n (Base)       ↓      (already aligned)   (More aligned)  ⚠️ Reward Hacking Risk\n              |\n        Base → Aligned\n```\n\n**W1: Function of $\\lambda$**\n\n> - Table 1 in our paper illustrates how different $\\lambda$ values in TrRa enable **flexible** training-time realignment. The #Token column aligns well with the above axes, **showing where to adjust the model** without the instability of retraining methods like RL, which often suffer from sampling and other factors.\n> - In InRa, $\\lambda$ functions similarly to a temperature parameter: larger values lead to stronger alignment, while smaller values result in weaker alignment. It serves as a **customizable** option to enable adaptive reasoning **based on the user's budget.** It is a parameter that can be customized during the inference stage.\n> - For the effect of temperature, see **Q1**.\n>\n> Overall, adjustments on the Pareto plane prioritize trade-offs rather than optimizing for a single best outcome.\n\n**W2: Limiation** \n\n> - Indeed, InRa faces this issue, as acknowledged in the limitations. We hope that KV compression or other engineering optimizations can help mitigate this problem. But TrRa does not suffer from this problem.\n>\n> - Performance drop with more iterations is inevitable.  TrRa-iter demonstrates that the reward signal can be effectively **transferred** and the correctness of TrRa.\n>   - Suppose you use λ=2 in iteration 1, resulting in a better-aligned model.\n>   - If you then use λ=2 again in iteration 2, it may be equivalent to λ>4 in iteration 1 (based on the empirical experiments).\n>   - Therefore, due to **reward hacking**, performance degradation becomes **unavoidable** with more iterations of TrRa.\n\n**W3: Thanks for your suggestion!** \n\n> We fully accept your suggestion and make the corresponding revisions. \n\n\n\n**Q1: The temperature effect on the performance is negligible.** \n\n> - The evaluation difference between ours and DeepSeek is as follows:\n>\n> |             | Temperature | Generation Length | Average times |\n> | ----------- | ----------- | ----------------- | ------------- |\n> | DeepSeek-R1 | 0.6         | 32796             | 64            |\n> | TrRa        | 0.7         | 16483             | 8             |\n>\n> - We think the results difference **depends on reasoning tokens**; therefore, we extend the generation length to 32K, and the results are as follows: \n\n**For TrRa:** \n\n> | DeepSeek-R1-Distilled-1.5B | Math500 (Avg@8) | Token |\n> | -------------------------- | --------------- | ----- |\n> | temperature = 0.6          | 84.95           | 5437  |\n> | temperature = 0.7          | 85.38           | 5219  |\n>\n> - It is **consistent** with the DeepSeek report. \n>\n> | TrRa $\\lambda=0.5$ | Math500 (Avg@8) | Token |\n> | ------------------ | --------------- | ----- |\n> | temperature = 0.6  | 88.25           | 3936  |\n> | temperature = 0.7  | 87.88           | 3949  |\n>\n> | TrRa $\\lambda= 2$ | Math500 (Avg@8) | Token |\n> | ----------------- | --------------- | ----- |\n> | temperature = 0.6 | 89.10           | 2859  |\n> | temperature = 0.7 | 89.35           | 2862  |\n>\n> - Our method achieves more token reduction and high performance. \n\n**For InRa:** \n\n> Our customized implementation VLLM engine results are as follows: \n>\n> | InRa $\\lambda= 0.0$ | Math500 (Avg@8) | Token |\n> | ------------------- | --------------- | ----- |\n> | temperature = 0.6   | 84.53           | 5543  |\n> | temperature = 0.7   | 85.88           | 5289  |\n>\n> | InRa $\\lambda= 0.5$ | Math500 (Avg@8) | Token |\n> | ------------------- | --------------- | ----- |\n> | temperature = 0.6   | 72.60           | 2524  |\n> | temperature = 0.7   | 71.43           | 2224  |\n>\n> | InRa $\\lambda= 1.0$ | Math500 (Avg@8) | Token |\n> | ------------------- | --------------- | ----- |\n> | temperature = 0.6   | 64.75           | 461   |\n> | temperature = 0.7   | 64.38           | 459   |\n>\n> - When $\\lambda = 0$, the result is consistent with the DeepSeek report. By increasing $\\lambda$, users can flexibly trade off between performance and token reduction based on their budget."}, {"review_id": "fWW7IhvHRa", "reviewer": "Reviewer_WpBq", "summary": "Within the problem space of aligning models, the authors use the DeRa (Decoding Time Realignment) technique to define a logit target that is interpolated or extrapolated from two models - in this case a DeepSeek-R1-Distilled-1.5B model and DeepScaleR-1.5B; they use this logit target to distill a model using a parameter-efficient finetuning technique (they use an extra layer added to the bottom, which they show to work better than simply using LoRA adapters), and this first step is called Training-time Realignment (TrRa). In a second contribution, the authors show that it is possible to apply TrRa iteratively to reach more extreme extrapolation with slightly less degradation.\n\nIn summary, the authors combine a number of ad-hoc techniques to get a similar effect as DeRa but with greater efficiency.", "questions": "Can you specify, in simple terms, what problem TrRa-iter solves in a more general case? Does it address use cases not already adressed by DeRa or is the main claim that you get better quality in extrapolation (noting that you're adding multiple layers at the same time if I understood the paper correctly)", "limitations": "In my opinion, the main limitation of the paper is the very one-dimensional framing: they take a problem that is relatively narrow and then show one recipe - made from known ingredients - for solving it. The limitations that the authors list (yes their way allows to load the original model only once, but you get two times the KV cache) are all technical limitations within this narrow framing which I find unhelpful", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The approach is described in sufficient detail for others to understand and possibly reimplement the approach\n- Compared to the brute-force way of solving the same problem, the authors' approach is much less compute-heavy\n\nWeaknesses:\n- The paper does a very bad job of separating between the objective that the authors want to achieve and the way of achieving it. There is some evidence in the form of the LoRA ablation as well as the ablation of which layers to train and how to initialize them which suggests that the chosen appoach is indeed a good choice of a design to solve the problem the authors want to solve\n- The problem as stated by the authors is too narrow to be practically relevant - in my opinion the goal of the paper should be more general to be relevant, even if that means that the paper doesn't reach that goal with the present work.\n- The paper is somewhat incremental work on top of DeRa and DeepScaleR - Trading off solution quality versus length is a good problem to attack in general, but if that is the goal then the approach of the author only offers minimal novelty against what already exists", "was_revised": true, "final_justification": "While I am grateful to the authors for the clarifications and the effort they made to push the reviewers towards higher scores, I still think that the paper's contribution is too narrow and limited for a full NeurIPS paper.", "rebuttal": "**For your Summary** \n\n> We believe there may be some misunderstandings:\n>\n> (1). You mentioned, `“they use this logit target to distill a model using a parameter-efficient finetuning technique.”`\n>\n> In fact, TrRa can be implemented through both full fine-tuning and parameter-efficient fine-tuning techniques.\n>\n> (2). Additionally, it appears that the InRa method was overlooked. The layer adapter parameter-efficient finetuning technique is designed to facilitate the InRa method, which performs inference-time realignment.\n>\n> Please refer to Section 3.3 of our paper, where we discuss the relationship between TrRa and InRa.\n>\n> (3). Upon careful consideration of your review, we realized you may have missed that we also extend our method to the 3H experiments.\n>\n> ---\n>\n> **To aid your understanding, we provide the following summary:**\n>\n> We propose training-time realignment (TrRa) and inference-time realignment (InRa) to better combine the logits of a base model and an aligned model for controllable alignment. Traditional methods for determining the optimal regularization level require retraining multiple models with varying strengths. TrRa addresses this limitation by efficiently identifying where to adjust the model using limited computational resources—without the uncertainty and cost of retraining-based methods. InRa offers an API interface, similar to temperature, allowing users to customize regularization strength during generation.\n\n**W1: Misunderstading**\n\n> This point may originate from your misunderstanding summary; you may neglect the layer adapter is to faciliate the InRa. Our TrRa is full fine tuning. \n>\n> **But thank you for acknowledging our chosen appoach is indeed a good choice.**\n\n**W2: `The paper should be more general to be relevant, even if that means that the paper doesn't reach that goal with the present work.`**\n\n> (1) In Section 4.3, we further extend our method to chat models, enabling the **adjustment of 3H values** and demonstrating the generality of our approach.\n>\n> (2) In Section 5.2 (Hyperparameter Stability), we show that the hyperparameter setting in our method aligns closely with that of full fine-tuning, suggesting that our approach can serve as **a lightweight proxy for hyperparameter tuning**. **This is the practically relevant claim of DeRa.**\n>\n> (3) In Appendix F.1, we demonstrate that our method can also facilitate the study of **alignment tax**.\n>\n> (4) Moreover, with the rapid development of reasoning models, both **efficient thinking** (e.g., **DeepScaleR**) and **adaptive thinking** (e.g., hybrid reasoning in **Claude 4**, and the thinking budget mechanism in **Gemini 2.5**) are gaining importance. We argue that the two scenarios we consider—efficient and adaptive thinking—are highly relevant in practice, **as they contribute to user satisfaction and reduce inference costs.**\n>\n> (5) TrRa reduces the need to retrain models from scratch, introducing a new paradigm for post-training alignment.\n\n**W3 : `Trading off solution quality versus length is a good problem to attack in general, but if that is the goal then the approach of the author only offers minimal novelty against what already exists`**\n\n> **Thank you for acknowledging that trading off solution quality versus length is an important problem. We would like to highlight that our work offers several novel contributions beyond this specific aspect:**\n>\n> - **Generality across modalities:**\n>   In Section 4.3, we extend our method to chat models, allowing adjustment of 3H alignment values , thereby demonstrating the broad applicability of our approach.\n> - **Efficiency and performance:**\n>   While DeepScaleR demonstrates strong performance and flexible control, it requires **3,800 A100 GPU hours** according to their blog. In contrast, our TrRa approach achieves **better performance** at **only 4 A100 hours** of training (see Table 1, particularly at λ = 1.5), highlighting a significant improvement in cost-effectiveness.\n> - **Beyond TrRa and InRa:**\n>   Our paper also provides additional insights and technical contributions:\n>   - We conduct an in-depth analysis and find that **bottom layers are particularly influential** for alignment (see Section 5.1).\n>   - We propose a **novel parameter-efficient training technique** using a single-layer adapter, enabling smooth inference-time realignment without full fine-tuning.\n> - **Comparison to DeRa:**\n>   see Q 1.\n\n**Q1:**\n\n>  `What problem TrRa-iter solves in a more general case?`\n>\n> - **TrRa-iter** further validates our algorithm and demonstrates that the reward signal can be **effectively transferred**. TrRa-iter solves the general problem of scalable, training-time realignment toward arbitrary alignment targets — especially when those targets lie beyond what DeRa or a single TrRa step can reach.\n>\n> - It does this by iteratively updating the model using intermediate, logit-fused teacher distributions, without the cost of full retraining.\n>\n>  `Does it address use cases not already adressed by DeRa?`\n>\n>   DeRa is indeed a strong method, but there remain areas for improvement. For example, as reasoning models evolve and generate increasingly **long chains of thought, the cost of DeRa becomes less acceptable due to its runtime overhead.** Our TrRa addresses this by enabling efficient training-time realignment. In addition, InRa allows alignment information to be injected with minimal training resources—using just a single-layer adapter—which is not feasible with DeRa.\n>\n>   | Difference             | DeRa                                                 | TrRa                                                         | InRa                                                         |\n>   | ---------------------- | ---------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ |\n>   | Stage                  | Inference-time                                       | Training-time                                                | Inference-time                                               |\n>   | Updates original model weights? | No                                                   | Yes                                                          | No                                                           |\n>   | Mechanism              | Fuses reference/aligned logits at each decoding step | Learns from interpolated logits as teacher (like knowledge distillation) | Inserts trainable adapter layer before LM layers and fuses outputs |\n>   | Iterative realignment? | No                                                   | Yes                                                          | No                                                           |\n>   | Inference cost         | double forward passes and doubled KV-cache           | No changes                                                   | Slightly higher (adapter + original layer + extra KV-cache)  |\n>   | Model depolyment       | Two Models                                           | No changes                                                   | Single model with integrated adapter layer                   |\n>   | User control?          | Yes – λ is tunable during inference                  | not tunable after deployment                                 | Yes – λ is tunable during inference                          |\n>   | Trainable component    | None                                                 | Whole model                                                  | Only bottom-layer adapter                                    |\n>\n>  ` (noting that you're adding multiple layers at the same time if I understood the paper correctly)`\n>\n>   Just to clarify, the adapter we use is a **single** additional layer (not multiple layers), inserted before the original bottom layer.\n\n**We sincerely hope this addresses your concerns and that you might consider revising your score accordingly.**"}, {"review_id": "yOlGZHLVPw", "reviewer": "Reviewer_PCc3", "summary": "This paper proposes a flexible realignment framework for LLMs, including training-time realignment and inference-time realignment. Training-time realignment utilizes logits, and inference-time realignment is implemented with a layer adaptor. Experiments on math reasoning and dialogue models show effectiveness.", "questions": "1. When saying Llama 3.1 8B under the dialogue model, are you referring to chat models?\n2. I wonder if you did any ablation study on the impact of adapting different layer depths?", "limitations": "Yes, the author addressed the limitations in Appendix", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength\n1. Their framework includes LM realignment during both training and inference time. The framework is quite flexible and dynamic.\n2. They study various models. For training and inference realignment, they study Deepseek. They expand their experiment to dialogue models, including Llama and Qwen. \n\n## Weakness\n1. One concern is that the model Inference-time realignment performance relies a lot on the hyperparameter λ. The optimal λ varies across the tasks. It would be improved if the authors could provide more discussion on the intuition for how to tune λ.\n2. TrRa-iter sacrifices correctness for token reduction, as well as in some cases of no iterations. It would be better if the authors could provide more discussion.\n3. It would be better if they could discuss more about the motivation for studying dialogue models and the training-time realignment of those.", "was_revised": true, "final_justification": "The author has addressed my concerns. I have increased my score.", "rebuttal": "```\n |-----------|-----------|-------------------|-----------|------>  \n 0.0         0.5        1.0                 2~5          >5.0       \n (Base)       ↓      (already aligned)   (More aligned)  ⚠️ Reward Hacking Risk\n              |\n        Base → Aligned\n```\n**W1: Intutition of $\\lambda$ in InRa**\n> - The origin is the base model, and the aligned model lies to the right. If alignment feels too strong, set $\\lambda$ between 0 and 1; if it's too weak, set it above 1. Values around 2–5 are generally safe; beyond that may cause reward hacking. \n>\n> - It is implemented as a single API parameter—similar to `temperature` and `top_p  `(lower for more deterministic, higher for more diverse)—that you can flexibly adjust according to your needs.\n> - For instance, as shown in experiments in Section 4.2, $\\lambda$ enables adaptive reasoning **based on the user's budget.** The token reduction performance across three benchmarks (blue curve) is consistent.\n\n**W2: `TrRa-iter sacrifices correctness for token reduction, as well as in some cases of no iterations.`**\n\n> - In Table 1 of no iteration, only the case of $\\lambda = 10$ shows a 0.42% performance drop on AIME-24, which may be attributed to randomness. In contrast, on AIME-25 and Math-500, the performance consistently improves by over 3% on average. \n> - **The table sweeps $\\lambda$ values, which are more intended to reflect the flexibility and correctness of TrRa.**\n>\n> - **TrRa-iter** further validates our algorithm and demonstrates that the reward signal can be **effectively transferred**. However, performing TrRa-iter may not be strictly necessary in practical applications.\n>   - Suppose you use λ=2 in iteration 1, resulting in a better-aligned model.\n>   - If you then use λ=2 again in iteration 2, it may be equivalent to λ>4 in iteration 1 (based on our empirical experiments).\n>   - Therefore, due to **reward hacking**, performance degradation becomes **unavoidable** with more iterations of TrRa.\n>  - OpenAI o3 and DeepSeek R1 also suggest that the **short chain of thought may not benefit model reasoning.**\n\n**W3: Motivation for studying chat models**\n\n> - Generality of the method. \n> - Quick Alignment Tax Study (Appendix F). \n> - Retraining with RL can be unstable due to noise in the reward model, sampling variability, and other factors. Training-time realignment acts as a tuning scale, guiding model trainers on where to realign for appropriate performance.\n>   - For instance, during the GPT-4o sycophancy incident (April 25, 2025), the team rolled back and retrained the model due to an inappropriate reward signal. **InRa can help identify the alignment tax, while TrRa may help prevent the need for retraining.**\n> - Subjective Alignment Preferences: People's views on 3H alignment vary—what one sees as sycophancy, another may find acceptable. **InRa allows adaptation to these differences. TrRa helps iterative products quickly.**\n\n\n\n**Q1: `When saying Llama 3.1 8B under the dialogue model, are you referring to chat models?`**\n\n> Yes, we will change these terms to chat models. \n\n**Q2: `I wonder if you did any ablation study on the impact of adapting different layer depths?`**\n\n> - In Section 5, we first analyze the relative importance of top versus bottom layers and find that **bottom layers are more influential**. We then apply adaptation to these layers and observe consistent performance, aligning with the earlier analysis. Adapting top layers often compromises language ability.\n>\n> - We also add the experiments on the **LLaMA3.2-3B** model, which has different layer depths compared to the 8B model, and observe **consistent results**.\n>\n> | LLama3.2-3B-Base | AlpacaEval2 | Arena-Hard | MT-Bench |\n> | ---------------- | ----------- | ---------- | -------- |\n> | top-1            | 4.03        | 4.00       | 5.85     |\n> | top-3            | 2.72        | 4.70       | 5.68     |\n> | bottom-1         | 9.86        | 9.80       | 6.53     |\n> | bottom-3         | 9.91        | 11.70      | 6.56     |"}]

Author rebuttal:

Meta-review: This paper proposes a flexible realignment framework for language models, introducing both training-time realignment (TrRa) and inference-time realignment (InRa). The approach enables controllable alignment by fusing logits at different stages, significantly reducing token usage while maintaining or even improving reasoning performance across benchmarks such as MATH500 and AIME24. Reviewers generally praised the clarity of writing, technical soundness, and the breadth of experiments. Reviewers raised concerns about performance degradation in some settings and further discussion of motivation in applications. The rebuttal addressed some main concerns. Although one reviewer still remains skeptical, I regard their concerns to be mainly about scope but not technical matters. Therefore, the merits still outweigh weaknesses. I recommend to accept this paper.
Decision: Accept (poster)