Token-Level Self-Play with Importance-Aware Guidance for Large Language Models
Authors: Tue Le
Hoang Tran Vuong
Quyen Tran
Linh Ngo Van
Mehrtash Harandi
Trung Le
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Importance Sampling
Preference Learning
Primary area: deep_learning
TL;DR: We introduce SWIFT, a token-level self-alignment and distillation method that improves LLMs by assigning token weights via a teacher model

Abstract: Leveraging the power of Large Language Models (LLMs) through preference optimization is crucial for aligning model outputs with human values. Direct Preference Optimization (DPO) has recently emerged as a simple yet effective method by directly optimizing on preference data without the need for explicit reward models. However, DPO typically relies on human-labeled preference data, which can limit its scalability. Self-Play Fine-Tuning (SPIN) addresses this by allowing models to generate their own rejected samples, reducing the dependence on human annotations. Nevertheless, SPIN uniformly applies learning signals across all tokens, ignoring the fine-grained quality variations within responses. As the model improves, rejected samples increasingly contain high-quality tokens, making the uniform treatment of tokens suboptimal. In this paper, we propose SWIFT (Self-Play Weighted Fine-Tuning), a fine-grained self-refinement method that assigns token-level importance weights estimated from a stronger teacher model. Beyond alignment, we also demonstrate that SWIFT serves as an effective knowledge distillation strategy by using the teacher not for logits matching, but for reward-guided token weighting. Extensive experiments on diverse benchmarks and settings demonstrate that SWIFT consistently surpasses both existing alignment approaches and conventional knowledge distillation methods.

Token-Level Self-Play with Importance-Aware Guidance for Large Language Models
Tue Le Hanoi University of Science and Technology Hanoi, Vietnam tue 1dt.11122999ssiss hust.uuuuo edu. vn
Hoang Tran Vuong Hanoi University of Science and Technology Hanoi, Vietnam
hoang tv224855siis hust. must.edu.wv. edu.vn vn
Quyen Tran Rutgers University New Jersey, US
Linh Ngo Van Hanoi University of Science and Technology Hanoi, Vietnam linhnv@soict. hust. hhut.edu...... vn
qt60@rutgers. edu
Mehrtash Harandi
Trung le Monash University Clayton, VIC 3800, Australia trunglm@monash.c edu
Monash University
Clayton, VIC 3800, Australia Mehrtash. Harandi wwwaandimmonssh..omm
Abstract
Leveraging the power of Large Language Models (LLMs) through preference optimization is crucial for aligning model outputs with human values. Direct Preference Optimization (DPO) has recently emerged a simple yet effective method by directly optimizing on preference data without the need for explicit reward models. However, DPO typically relies human-labeled preference data, which can limit its scalability. Self-Play Fine-Tuning (SPIN) addresses this by allowing models generate their rejected samples, reducing the dependence on human annotations. Nevertheless, SPIN uniformly applies learning signals across all tokens, ignoring the fine-grained quality variations within responses. As the model improves, rejected samples increasingly contain high-quality tokens, making the uniform treatment of tokens suboptimal. In this paper, propose SWIFT (Self-Play Weighted Fine-Tuning). a fine-grained self-refinement method that assigns token-level importance weights estimated from stronger teacher model. Beyond alignment, we also demonstrate that SWIFT serves as an effective knowledge distillation strategy by using the teacher not for logits matching, but for reward- -udddgd token weighting. Extensive experiments on diverse benchmarks and settings demonstrate that SWIFT consistently surpasses both existing alignment approaches and conventional knowledge distillation methods.
Introduction
Large Language Models (LLMs) I
have demonstrated strong generalization across diverse tasks, including text summarization 6 code generation 88, 9. and instruction following [1]. However, they can also produce harmful content (12), hallucinate [3], or reinforce sociocultural bi- ases [14 [1. underscoring the need for alignment with human values. To address this, Reinforcement Learning from Human Feedback (RLHF) has emerged as the standard approach for preference
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

alignment. RLHF typically involves collecting human feedback on model outputs to train a reward model, which is then used to fine-tune the base model using reinforcement learning techniques such as Proximal Policy Optimization (PPO) [5] While effective, RLHF is complex and resource-intensive, requiring extensive high-quality human feedback. To address these challenges, Direct Preference Optimization (DPO) [16] simplifies the process by directly optimizing on preference data, bypassing explicit reward modeling. Nevertheless, DPO still depends heavily on costly human annotations, limiting scalability.
To better balance alignment performance and generation diversity by controlling the KL divergence at the token level, Token-level DPO (TDPO) [17] redefines the objective maximizing restricted rewards in a sequential manner using the advantage function. Building on TDPO, TIS-DPO [18] argues that different tokens should not be treated equally and proposes principled methods for estimating token weights by contrasting two I.LMs: one biased toward high-reward tokens and another favoring low-reward tokens. However, this approach requires training two additional LLMss which can be prohibitively expensive in terms of computation.
To reduce reliance on costly human preference annotations, Self-Play Fine-Tuning (SPIN) [9 trains models to generate rejected responses using earlier versions of themselves. These are paired with human-annotated SFT samples to form synthetic preference pairs for iterative fine-tuning, eliminating the need for explicit reward models and preference data. However, SPIN applies uniform learning signals across all tokens, overlooking that both chosen and rejected responses may contain mix of high- and low-quality tokens-vee weakening token-level gradients as the model improves. Technically, SPIN matches distributions through self-play adversarial game to generate outputs close to ground- truth sentences, but this limits its ability to perform graigrrdddd, token-level alignment, which typically requires to leverage with the advantage function and oaken-based reward functions from reinforcement learning.
In this work, we adopt a token-level perspective for matching the distributions of interest, which naturally leads the formulation of token-level reward functions. Notably, our approach token- level distribution matching reveals a strong connection to the objective of maximizing the advantage function, thereby enabling the incorporation of token-level weighting. However, this formulation inherently resembles teacher forcing. To transform it into a token-level self-play framework, we recast the problem using the student foreing mechanism [20], ultimately deriving our final objective function. For the token weight estimation, we leverage teacher model to provide fine- lne-grained token-level reward signals, enabling the student model to focus more on informative tokens during learning.
In summary, we propose SWIFT (Self-Play Weighted Fine-Tuning), an extension of SPIN that incorporates token-level importance weights estimated from a stronger teacher model. Unlike conventional distillation methods (2 that match logits or hidden states, SWIFT leverages fine- grained token-level rewards to guide the student model's 's attention toward more informative tokens. To address tokenizer mismatches between teacher and student, we introduce a general token mapping strategy based on shared surface segments, enabling accurate weight transfer without compromising alignment. While SWIFT benefits from access to a strong teacher, in scenarios where we do not have access to such a model, we observe that assigning uniform token weights or applying contrastive token importance estimation as proposed in [18] still provides meaningful improvements, as shown in our ablation study (). We validate SWIFT through extensive experiments under both alignment and knowledge distillation settings across multiple benchmarks, consistently demonstrating its effectiveness and efficiency in aligning LLMs with human preferences.
Our main contributions can be summarized as follows:
We propose SWIFT, fine-grained self-alignment and distillation method that leverages teacher models to provide token-level importance signals.
We introduce a practical solution to handle tokenizer mismatches, enabling reliable token- level weight transfer between teacher and student models.
We conduct comprehensive experiments on multiple benchmarks and settings, showing that SWIFT consistently outperforms existing alignment and distillation methods.

---

2 Preliminaries
2.1 Preference-Based Alignment in LLMs
Let Ty denote Large Language Model (LLM) parameterized by 0. Preference-based alignment aims to adjust Ty to prefer human-preferred responses over less desirable ones. Formally, we consider dataset of triplets D {(z, y", 3/)}, where z is the prompt, and y" and y' are the preferred (winning) and less preferred (losing) responses, respectively, such that y" > y'. The objective to optimize such that it assigns higher likelihood to y" than y'. Direct Preference Optimization (DPO) [6] addresses this by directly optimizing the model on preference data, bypassing explicit reward modeling. Despite its simplicity, DPO still requires high- high-quality preference annotations, which are costly to obtain.
2.2 Self-Play Fine- TuninT (SPIN)
To overcome the reliance external annotations, Self-Play Fine-Tuning (SPIN) [9] introduces self- play paradigm, enabling the model to iteratively refine itself without access to additional preference data. SPIN starts from Supervised Fine-Tuned (SFT) model trained dataset S {(2r, y)}, where is a high-quality human-annotated response to prompt 2.
At each iteration f, the model TTe generates synthetic responses y' for prompts sampled from S. These are paired with the corresponding human- annottted responses to create preference pairs (æ, y, v/). The updated model Te.41 is then trained to distinguish between these pairs, encouraging the model to prefer human-wringg responses over its own previous generations. Formally, SPIN optimizes the following objective:
CSPIN(TT TT.) (x.y)) y Noo (-1z) [{
(Blog
To.+1 (y x)) Toa (y/ x)
3 log To (V' 1 | To, (1/ x))
(I)
where ((-) convex decreasing loss function (e.g., logistic loss), and (3 is a scaling factor.
While SPIN enables iterative self-improvement at the sequence level, it applies uniform learning signals across all tokens, overlooking token-level quality variations. As the model improves, this inefficiency grows, as rejected responses may still contain valuable tokens. To address this, we propose SWIFT, token-level refinement framework that incorporates importance-awee aware token weighting.
3
Our Proposed Self-Play Weighted Fine-Tuning
3.1 Problem Setting
We are given supervised training set D y') where of and y' are the input/output sequence of the tokens. Let ( denote the parameters of the teacher model, and @ the parameters of the student model be trained. We denote the student model at iteration k as og. At the outset, we emphasize that the student model is primarily trained via weighted token-level self-play, with the teacher model providing guidance solely for estimating token importance.
We need to train the student LLM (y x) to incorporate the knowledge from dataset D and use the guidance from the teacher model for the token weight estimation. Let us denote Pa as the distribution of the ground-truth pair (z: y). while Pos as the distribution of the synthetic pair (æ, v'), where y' ~ TTp (- a) with a ground-truth input x. Our task to learn Tas so that the distribution Pos remains closely related to the data distribution P...
3.2 Our Proposed Approach
In what follows, we present the thcoretical framework for our SWIFT. We denote pa(a, y) and pas (z. y) where y ly .U......77 yr] are the density functions of Pa and Pos. The following lemma characterizes the necessary and sufficient conditions in order to Fa = PAS.
Lemma 3.1. The necessary and sufficient conditions for Pa = Pas are palys x, wet) = Pos (ys I, yct) for all t < T.

---

We note that, similar to the distribution matching in SPIN [19, we assume that and y' from Fa and Pgs have the same length T (e.g., the maximum sequence length). This is standard in sequential training, where padding is typically used to align sequences to the maximum length. Moreover, inscribed in Lemma 3.1 we need to estimate the divergence between pa(Ue Ir Uet) and Pos (Ve Z. V-e)) To this end, we examine the following Integral Probabilistic Metric (IPM) [22 [9
Vt: max E [>..S))>> (ra ([or, Ueel= yr) - Bage-wagt ( ((.we))) EER:
[Fx [x: Ucel víl
(2)
where ri ([r, yerl, 2) is the t-th token- based reward model belonging in function family Rİ. Here we note that the index k specifies the current iteration with the current student model 0g. while the index t specifies the i-th token. We name ra (Iæ, v-cch z) as token-based reward function because it offers high reward value for a ground-truth token 2 = ye. while offering a low reward value for a synthetic token 2/ with 1/2 Tog (- x, Mer)..
Additionally, can express the token-level reward model [re]e using a sequeecce-eevl function r(r, 9) for any output sequence ý. where each token-level reward is given by ra ([x, y<e] (9)) = (2, VSa). To support variable-Iength inputs when computing reward outputs, represent either with a dedicated transformer- baase model or implicitly through our LLM, following approaches similar DPO [16] or Self-Play [19]. We denote the function family for R*. We assume the function family Re is consistent with the token-based function families [RA], as follows.
Definition 3.2. (Consistency) The function family R is said to be consistent with the token-level function families [R if, for any sequence of functions [r> Rt there exists function E R such that
(z, yge) rı (Iz, yeel 11)), V(z, D.
Conversely, for any r R* there exists a corresponding sequence [re e RA that satisfies the same equality, ensuring bidirectional consistency.
In addition, we represent and [r=]e as implicit functions parameterized by the LLM mÿ. which is assumed to have infinite capacity (i.e.. it can approximate any measurable function to arbitrary precision). As result, the consistency property inherently satisfied. Leveraging the consistency property, we can reformulate the original optimization problem (OP) in 2 into a more tractable form.
Theorem 3.3. Assume the consistency property between R* and [Rt]e, je we can equivalently reformu- late the original OP in 2 into the following OP
max E (a (1x) PER*
Σ
Σntho Bet).e)) Σx 1r ([r, (lr.vell.h)))
(3)
where 7 E [0, 1] is the discount factor.
In particular, the OP (3) aims to align pas (ye x. yee) with Pa(V, xx, Dec) for all t, which in turn drives the joint distribution pys (x, y) to match the data distribution pa(z, y) for (z, D. Additionally, the optimization problem in (3) follows the teacher-forcin
paradigm. We can reformulate it into an equivalent student-forcing objective (20), which achieves the same goal of aligning the joint distribution pas (x. y) with the data distribution pa(x, 8), as follows.
max E (ex.)) was (-1x)
Σ
7 1 (I.,)e))...))
Σ 7 t Ir Vee.le
FER
Denote the reward function R (ar, v) = E, - (Iz, Verl = E, - (xr, Ueee) as the sum of the loken-based reward functions, we can rewrite the OP in (4) as
max (o FER
R (z. R(x,
(5)
It is evident that 55) defines IPM between the two distributions of interest, pa(æ,, v) and pos (s, u). which serves the same purpose as the objective in (3), as demonstrated in the following theorem.
Theorem 3.4. Assume that the function class R² is symmetric (i.e., ifr E R², then -r E R* ), and that the distribution fumily {pes (x, y) 0 E ② includes the data distribution pa(æ,, y). Under these conditions, optimizing the objective in either or 55) learn the student model 0 is equivalent. Mathematically, the minimization over 0 using or Dields the same optimal solution.

---

To generalize the optimization problem in we consider a broader objective in the same spirit, formulated as follows:
min (izyy) ~ D,y ~ ( !! (Z, v) R(x,
(6)
TER*
where is non-increasing function (e.g.. l(t) = log(1 + cxp(
Inspired by 17 18 we consider the following alternative token-level objective function involving
advantage function:
(Rase)) (uretng (1x, ((r, - -BDrk (Taa (- [xr, Meal) |! (- [ar, weel))
max R88
(7)
where the advantage function An ([z, M-co w/) =- an (1z, Mees w/) V (Ix, weeel) with the state- action value function Q= and state value function VT Dxr. r. the KL divergence, and [ws] are the token weights representing the importance of the tokens.
We now explain why the objective function in 7 assists us in learning a good new student model m.. Considering rolling out one-step for Va, we can approximate
Angg ((ar, y-er]: v/) ([r, M-ce] v/1) By (-la,eee)) (r ([r,, Meee .v')
(8)
Linking the maximization in 7 by maximizing the advantage function, we effectively maximize ([a,, Meeel vf), where V/ ~ ga [xr, V-ce)) This encourages the student model to generate synthetic tokens with high reward values. From (2). this process implicitly pushes pas x, gecc) closer to pal- 2, Merr, desired. Moreover, the second term, By (+1r,ve)) [r ([ær, M-er) v )], is likely to
remain moderately high due to the previous update, thereby exerting a stronger effect in guiding Pas toward Pa-
Similar to 17
18
we have the following lemma.
Lemma 3.5. The optimization problem in 7 admits the following closed-form solution:
To (= x. Mece) exp [ Qxg, (1r, ver)
Tăs (z x. yc1)
(9)
Z (1x, Nerliwee 3)
where 7 (1x, Ueel; We, B) is the partition function. Consequently. the Q-function can be expressed as:
Tge (2 x. Ve1) Ta (2 x, vet)
Qxy (Ir.Vee), weB log
+ wrB 7 (1x, Mec Meil: Wes 1).
(10)
The optimal solution in 9 and 110) hints us how to define the family function Rh. Specifically, this is defined as
Tgs (2 x, Mec)))
R*
r: 30$ EOAQUO (1 vell2) = we.3 log
+ w. log Z ([x, Beck
Tap
2, 2,
(11)
Certainly, the reward functions r, and hence R. are implicit functions of the policy was. To derive the objective for updating kas, need to substitute this dependency into (6). We now rewrite the objective function in (6) as presented in the following lemma.
Lemma 3.6. The objective function in (6) can be further derived as presented the following lemma.
((.(z.)) R(x,v))) =1
Σ 11 Axys (ee,Vee) v=) - ΣT Angs (1x, Ve..eelb)))
(12)
Based on Lemma 3.6 we can reformulate the OP of interest as shown in the following theorem.
Theorem 3.7. The functionn of interest in (6) can be reformulated to (u (I, y, V T85 w) (x, v Ag5 (T8,w)).
5

---

where we have defined
u (r. U. Tos, ,w) = Σ
3
log
Tos (ye
24 Mce))
Σ
3
log
Tos (1/ Ir, v
we
To (Ve
2, M=te)
we
ToB (v/
I. Vlec)
v (.. Tos, w) = DD (Ir, (I, wi Tgs ITT with the weighted sequence KL divergence being defined
8 DDeak (I, w Tas TTs
D (2, U.W:: Σ w DEL (T1
22, Met) || vce)).
Finally, the objective function to train our approach is
min (n..)) (-la.)) [1 (u (x,, y, mg,, v (2. mgs, Ag8
(13)
where 1 (t) log + exp (-t)) is non-increasing function.
Pseudo-code for our method can found Appendix B.4 the supplementary material.
3.3 Teacher-Guided Token Importance Estimation
Our aim is to develop an efficient mechanism to estimate the importance weight of each token. Intuitively, well-trained ceacher-havinng been trained on more data with higher model capacity- can better signal which tokens carry higher "reward". In this section, we present a principled method to distill token-level reward signals from teacher model 6" and transfer them to a student model @ Due to space limitations, we leave a detailed explanation of this mechanism in Appendix B.5
can
Inspired by TIS-DPO [8] we first estimate the raw importance weight for the td token in response
(y1, ............................. 1/) as
w/aW = k exp clamp log Ter (yt x. y/ 2 L, U))
(14)
Tgs (ye I, y2))
where clamp(a, LL min mmax(a, L), U) truncates the log-odds to IL, U' to reduce variance and stabilize optimization. Here and μ are constant; and are lower and upper clipping bounds.
However, this formulation assumes that the teacher and student models share the same tokenizer and vocabulary. In practice this assumption rarely holds. A sequence might be tokenized by the teacher model as [yT 32 ..... vi' and by the student different sequence [vΣ. 3º. ... v. making direct token-to-token alignment in (14) infeasible. To address the misalignment issue caused by differing tokenizers, we propose generalizable token mapping strategy based on shared surface segments.
Case 1: Shared Tokenizer. In the scenario where both the teacher and student models utilize an identical tokenizer, the alignment between their respective token sequences inherently preserved. Specifically, each token yi generated by the student model corresponds directly to a token produced by the teacher model the same position, thereby eliminating the need for additional alignment mechanisms. As result, the distillation weight W/ associated with each token position & can be directly adopted from the unprocessed or raw weighting scheme. Formally, we define:
W:
(15)
where waw represents the original distillation weight derived from the teacher model's outputs as in Eq 14 without any post-pon post-processing or re-alignment.
Case 2: Different Tokenizers. When tokenizers differ, direct index alignment between tokens becomes unreliable. Rather than aligning tokens individually, we instead leverage a common structural unit shared across both tokenizations. Most modern tokenization algorithms, including BPE (3), SentencePiece (24, and WordPiece (25), segment text into subwords while generally respecting word boundaries. This means that each token strictly belongs to either single word

---

or the whitespace separating words, making word spans natural and reliable basis for alignment. Leveraging this property, we segment the original text into boarse-grained units corresponding to complete words and their leading whitespace, resulting in sequence of shared components fci, c2..... CK) This ensures that every token from both teacher and student sequences can be assigned to one of these shared components, which are small enough to preserve local semantics but broad enough to encompass all subword variations.
Suppose student token V belongs to component Ch. and the teacher tokens in that segment are T(c,)) = {v] "9+1....... v]+z}- Assuming all tokens within a shared component contribute equally, we define the student's token importance as the average the teacher tokens' weights in the corresponding component cn:
k exp clamp (1og Tar (vf I. v2r) v I, 1.
Σ w/ = [T(eN] (.T(()))
(16)
Tag (yy
Comparison with Prior Work. Compared to the contrastive-bast token weighting in [8 our method is significantly more computationally efficient, as does not require training separate positive and negative models. We validate this by providing direct comparison of the total time cost between SWIFT and the contrastive- based token weighting approach (TIS-DPO) in Appendix [CT in the supplementary material. Furthermore, shown in T our teacher- teacher-guided guided estimation consistently achieves superior performance, confirming both its efficacy and practicality.
4 Experiments
We evaluate SWIT through extensive experiments across diverse settings, highlighting several key findings. (1) SWIFT consistently outperforms existing alignment methods on multiple benchmarks. (2) Since SWIFT relies only on an SFT dataset without preference-llbeled data, we also compare it with knowledge distillation baselines, where it similarly outperforms, demonstrating the effectiveness of token-level importance estimation for distilling teacher knowledge. (3) Furthermore, ablation studies confirm that using the teacher estimate token weights yields significant gains over alternative weighting methods.
4.1 Experiment Setup
We conduct two experimental settings to evaluate SWIFT: Alignment and Knowledge Distillation. The Alignment setting compares SWIFT with existing alignment methods, while the Knowledge Distillation setting assesses its effectiveness against existing knowledge distillation methods.
For the Alignment setting, We use Liven1.5-1.80 26 the base model. As the teacher model, we adopt 7Bphyr--B-SFT-Full (27). which is based on Mistral-7B (28) and further fine-tuned on the Ultrachat200k datase provided by HuggingFace. We follow the procedure in [19]. 50,000 prompts are randomly sampled from U1 21trachat200kk and generate synthetic responses using the base model. For evaluation, adopt the HuggingFace Open LLM Leaderboard (29]. which is commonly used to assess the underlying capabilities of models through few-shot evaluation. Additionally, we evaluated the output quality of the LLM using MT-bench [30] with its provided dataset,
the
API of GPT-4 as the judge.
For the Knowledge Distillation setting. We use GPT2-1.5B [3 as the base model and Qwen2.5- 7B-Instruct 22 as the teacher. Four datasets are selected for evaluation: DATABRICKSDOLLY (Dolly) [33). ALPACA (Alpaca) (34), S-NI (S-NI) [35), and DIALOGSUM (Dialogsum) [36]. Final performance is reported using the ROUGE-L metric 37 between generated outputs and human-
annotated references.
We perform 4 iterations, each iteration consisting of 2 epochs training. Additional implementation details are provided in Appendix B in the supplementary material.

---

Table 1: Performance of SWIFT based Qwen1.5-1.8B across HuggingFace Open LLM Leader- board datasets, compared with all baselines. * Methods trained on Ultrachat200k SFT data. + Methods trained on UltraFeedback Binarized preference data [39].
Methods
Arc
TruthfulQA
Winogrande
GSM8k
MMLJ
HellaSwag
Avg
Teacher
60.41
43.73
74.19
26.76
60.92
82.85
58.14
SFT"
39.08
38.42
58.64
19.03
41.30
60.09
42.76
DPO
39.33
38.37
59.12
19.18
41.30
61.74
43.17
IPO
37.29
38.09
61.04
32.52
44.40
60.62
45.66
TDPO
39.08
38.12
58.47
20.70
41.29
61.22
43.15
TIS-DPO
40.44
39.09
61.27
30.71
41.61
61.66
45.80
SPIN"
39.93
40.46
58.17
18.42
40.81
61.42
43.20
SWIFT'
39.78
39.12
61.48
37.93
44.84
61.63
47.46
SWIFT baselines MT-Bench.
(b) Training reward comparison.
SWITT
2
10.4%
5F1
Roward
SWIFT SHN
Chosen Roward SWIT
pcodd Bessart SWATT
60
60
100
Chount Rataard SPI
accidd Rewward SPM
SWIFT Wins
Baseline Wins
10000
20000
30000
40000
50000
Step
The left figure compares SWIFT with SFT, DPO, and SPIN on MT-Bench, evaluated by GPT-4. The right figure shows the trends of chosen and rejected rewards for SWIFT and SPIN during iteration O.
Figure 1:
4.2
Main Results
4.2.1
Comparison with Alignment Methods
Table compares SWIFT with SFT and alignment baselines, including DPO 16] IPO 38).
TDPO 17). TIS-DPO 18]. and SPIN [19]. Following [19], DPO, IPO, TDPO, and TIS-DPO trained on the UltraFeedback Binarized dataset (39]. which contains 62k GPT-4-labeled pref- erence samples, incurring significant data collection costs. Across six Open LLM Leaderboard benchmarks, SWIFT achieves the highest average score (47.46) surpassing the strong baseline TIS-DPO by a margin of +1.6, with strong gains on GSMSk (+7.22) and MMLU (+4.03) over SPIN. Complementing the leaderboard results, MT-Bench pairwise comparisons with GPT-4 the judge (Figure Tab show SWIFT consistently outperforming SFT, DPO, and SPIN, achieving win rates of 44.4% against SPIN and SFT, and 41.9% against DPO, confirming its advantage in response quality.
Figure 2 further supports this by illustrating performance trends across multiple iterations. While both SWIFT and SPIN start from the same SFT baseline, SWIFT demonstrates a clear upward trajectory across iterations, especially on MMLU and HellaSwag.
4.2.2 Evaluation against Knowledge Distillation Baselines
To further validate the effectiveness of our proposed method, we evaluate SWIFT under a knowledge distillation (KD) setting, where smaller student model is distilled from a larger teacher model. As shown in T we compare SWIFT against Supervised Fine Tuning (SFT) and several existing KD baselines, including ULD 40 MinED [4]. DSKD 42 The teacher model is Qwen2.5-7B. Instruct, while the student is GPT2-1-5H Across all benchmarks, SWIFT achieves the highest average ROUGE-L score (29.20), consistently outperforming previous distillation methods.

---

(a) MMLU.
(b) HellaSwag.
(c) Average.
46
44
02
46
42
40
60
42
38
SFT
iter-0
iter-I iter-2
iter.3
SFF
iter-O iter. iter-2
iter-3
SPT
iter-O iter-1
iter-2
iter-3
SPIN
SWIFT
Figure 2: The average score of SPIN and SWIFT at different iterations on the HuggingFace Open LLM leaderboard datasets.
Table 2:
Evaluation results of Qwen2.5-7B-Instruuc distilled to GPT2-1.5B on four benchmarks and the averaged performance
Qwen2.5-7B-Instruct -> GPT2-1.5B
Methods
Dolly
Alpaca
S-NT
Dialogue Sum
Avg
Teacher
28.49 + 0.21
35.75 = 0.25
32.35 + 0.24
35.24 + 0.08
32.96
SFT
21.83 + 0.28
27.15 = 0.31
23.16 0.15
30.74 0.17
25.72
ULD
24.52 + 0.28
29.17 = 0.22
24.18 0.08
32.74 0.35
27.65
MinED
25.52 + 0.44
30.41 = 0.56
25.09 0.25
33883 + 0.24
28.71
DSKD
25.38 + 0.46
30.48 = 0.38
25.92 0.18
33.82 0.23
28.90
SWIFT
25.94 + 0.32
30.69 = 0.33
26.43 + 0.19
33.74 0.13 0.13
29.20
These results highlight an alternative way to leverage teacher models by estimating token-level importance weights, rather than relying solely on logits hidden states. Moreover, the consistent gains across diverse datasets further demonstrate the robustness of our approach. Furthermore, unlike conventional knowledge distillation, which requires online teacher access and incurs high memory and compute overhead, SWIFT computes token weights in single offline pass, significantly reducing training costs while retaining the benefits of distillation.
4.3 Ablation Studies
We conduct ablation studies investigate the influence of different token weighting strate- gies the performance of SWIFT. Table reports results over four self-play iterations us- ing Qwenl.5-1.88 as the student model under various token importance estimation methods.
Table 3: Ablation study for token weight estimation Qwen1.5-1.88
Methods
iter 0
iter
iter
iter 3
rand weight equal weight
40.37
39.71
38.85
34,95
We experiment with several token weighting strategies: random weight, where weights are uniformly sampled from [-0.5. 1.5]; equal weight, where all tokens are assigned a con- stant value of 1; contrastive weight, follow- ing TIS-DPO [8] by training separate pos-
42.74
43.05
43.11
43.23
contrastive weight
44.75
45.03
45.11
45.27
reverse weight
33.21
31.05
27.88
25.43
SWIFT
46.94
47.12
47.02
47.46
itive and negative models to infer token importance; reverse weight, which inverts the weights computed by our method. Specifically. the reverse weighting assigns it 1 w; and SWIFT, our default setting where token importance is estimated via a teacher model as described in 3.3
SWIFT assigns a weight w to a token t in sequence
As shown in Table 3 our method (SWIFT) consistently achieves the best performance across all iterations. Equal weighting performs reasonably clearly outperformed by importance-awarr alternatives. Contrastive weighting provides moderate gains but still trails behind our approach,

---

while both random and reverse weighting lead to substantial performance degradation. These results underscore the importance of accurate token importance estimation, and confirm the advantage of using a teacher-guided approach as in SWIFT.
We further examine the evolution of chosen and rejected rewards throughout training, as depicted
in Figure lb Inspire by (18), in our approach, the chosen reward is computed by incorporating
token-level weights into the DPO reward formulation, specifically: Σ w Blog Tyet (3." [= In
<M'""" (=.)).)
SPIN, both chosen and rejected rewards decline over time, reflecting ineffective learning of preferred responses. In contrast, when applying our token weighting strategy, the chosen reward increases steadily while the rejected reward decreases, indicating that incorporating token-level importance helps guide the model toward more effective optimization.
Conclusion
We presented SWIFT, an effective method for token-level importance-aware fine-tuning via teacher- guided distillation. By estimating token weights from a stronger teacher model, SWIFT improves model alignment and generation quality. Our results across multiple benchmarks and settings validate the effectiveness and flexibility of this approach. However, using the teacher model repeatedly across multiple self-play iterations may introduce additional overhead. Future work may explore more efficient strategies, well as deeper analysis on how token weighting influences learning dynamics.
Acknowledgements
Trung Le and Mehrtash Harandi were supported by the ARC Discovery Project grants DP230101 176 and DP250100262, as well as by the Air Force Office of Scientific Research under award number FA9550-23-S-0001.
We thank the anonymous NeurIPS 2025 reviewers (Reviewer Qm7U, 8XmY, Qi3n, ti22 and Program Chair for their constructive feedback and valuable suggestions that have substantially improved this manuscript.
References
[1] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers open language model post-training. arXiv preprint 2rXiv:2411.15122 2024.
[2] Jiaxuan Gao. Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective rl reward at training time for I1m reasoning. arXiv preprint arXiv:2410. 15000.511151 2024.
[3] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599 2025.
[4] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948 2025.
[5] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
[6] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022. URL https://arxiv. org/abs, 2009.
[7] Huan Yee Koh, Jiaxin Ju, Ming Liu, and Shirui Pan. An empirical survey on long document summarization: Datasets, models, and metrics. ACM computing surveys, 55(8):1-35, 2022.
10

---

[8] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et Evaluating large language models trained on code. arXiv preprint arXiv:2107.0333 03374, 2021.
[9] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764-10799. PMLR, 2023.
[10] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finttuee language models. Journal of Machine Learning Research, 25(70): 2024.
[11] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[12] Xiaowei Huang, Wenjie Ruan, Wei Huang, Gaojie Jin, Yi Dong, Changshun Wu, Saddek Bensalem, Ronghui Mu, Yi Qi, Xingyu Zhao, al. A survey of safety and trustworthiness of large language models through the lens of verification and validation. Artificial Intelligence Review, 57(7):175, 2024.
[13] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang. Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's 's song in the ai ocean: survey on hallucination in large language models. arXiv preprint (3Xivv2309.01//1 2023.
[14] Isabel Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Demoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097-1179, 2024.
[15] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
[16] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-53741, 2023.
[17] Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang. Haifeng Zhang, and Jun Wang. Token- level direct preference optimization. arXiv preprint 1991 2024.
[18) Aiwei Liu, Haoping Bai, Zhiyun Lu, Yanchao Sun, Xiang Kong, Simon Wang, Jiulong Shan, Al- bin Madappally Jose, Xianjiang Liu, Lijie Wen, et al. Tis-dpo: Token-level importance sampling for direct preference optimization with estimated weights. arXiv preprint arXiv:2410.04350, 2024.
[19] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint 24X1:2201.00333 2024.
[20] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 28. Curran Associates, Inc., 2015.
[21] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini arXiv: 1503.0253,, 2015.
[22] Alfred Müller. Integral probability metries and their generating classes of functions. Advances in applied probability, 29(2):429 443, 1997.
[23] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint a5Xiv:/508.0790) 2015.
[24] Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808. 06226, 2018.

---

[25] Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. Fast wordpiece tokenization, arXiv preprint ar514 2020.
[26]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 2023.
[27] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro Von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint urXiv:2310.1683 2023.
[28] AQ Jiang, A Sablayrolles, A Mensch, C Bamffrd, DS Chaplot, Ddl Casas, F Bressand, G Lengyel, G Lample, L Saulnier, et al. Mistral 7b. arxiv 2023. arXiv preprint arXiv:2310.06825, 2024.
[29] Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open I1m leaderboard, 2023.
[30] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595 46623,, 2023.
[31] Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arxiv 2019. arXiv preprint arXiv:1910.09700 2019.
[32] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412. 15115, 2024.
[33] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. arXiv preprint arXiv:2306.08543 2023.
[34] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023.
[35] Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. buperrrnturalinnttucttonn::: Generalization via declarative instructions on 1600+ nlp tasks. arXiv preprint arXiv:2204.077000 077000 2022.
[36] Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. Dialogsum: A real-life scenario dialogue summarization dataset. arXiv preprint arXiv:2105.06766 2021.
[37] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
[38] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447-445 PMLR, 2024.
[39] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. 2023.
[40] Nicolas Boizard, Kevin El Haddad, Céline Hudelot, and Pierre Colombo. Towards tooss-tokenizer distillation: the universal logit distillation loss for llms. arXiv preprint arXiv:2402.120300 2024.
[41] Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. arXiv preprint arXiv:2401. 2024.
[42] Songming Zhang, Xue Zhang, Zengkui Sun, Yufeng Chen, and Jinan Xu. Dual-space knowledge distillation for large language models. arXiv preprint arXiv:2406.17328, 2024.
12

---

[43] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):3243344 1952.
[44] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[45] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou, Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint urXiv:2305: 14233, 2023.
[46]
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024.
[47] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint wrXiv:1803054533 2018.
[48] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a 2019.
machine really finish your sentence? arXiv preprint
[49] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
[50] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.033 03300, 2020.
[51] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint MXXXi:2109007553 2021.
[52] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiihhrr Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. /4168, 2021.
[53] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-16. IEEE, 2020.
[54] Tri Dao. Flashattention- Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023.
[55] Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Lecture 6a overview of mini- batch gradient descent. Coursera Lecture slides https://class. coursera. org/neuralnets-200 2012- 00 (0l/leecure, 2012.
[56] Woosuk Kwon, Zhuohan Li. Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 1-166, 2023.
[57] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, et al. Challenging big- bench tasks and whether -ffin-of-thought can solve them. arXiv preprint arXiv:22 10.09261, 2022.
[58] Dheeru Dua Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gard- ner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint 1xXhn:1903.0016. 2019.
13

---

[59] Yujia Qin, Shihao Liang, Yining Ye. Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models master 16000+ real-world apis. arXiv preprint arXiv:2307. 16789, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and section 1 we clearly demonstrate the contribution and scope this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
Please refer to Section 3.2 for our assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided detailed descriptions of the experimental setup and methods in 3 to ensure that our experiment can be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
We released our code and datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
Swwelee//ooSSSSeeeeeeeeee wwtsSubbmiiiioooioooooooooii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We presented experiment settings and details in Section 4.1 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We conducted the experiment five times independently with different random seeds and reported the mean and standard derivation of the result. The other experiment setups follow the protocol of prior well-known work.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We mention about compute resources in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:p////errrr.....oo////////aa
Answer: [Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is a methodological paper to improve preference optimization algorithms and does not have direct societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not involve releasing data or models that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets used in paper public.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.paaersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: There is assets introduced in the paper.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: There are no crowdsourcing experiments and research with human subjects in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20

---

A
Background
We consider Large Language Model (LLM) parameterized by parameters 8, and denote its output distribution as vy. Given an input sequence z, commonly known as the prompt, the model generates a response sequence In preference- based alignment, the training dataset is made up of triplets D ((x, Dee yt),, where Yor and !/ are two possible responses. Among them, Yoo is considered better than y!, which we write Vhe y/- We refer to yo the chosen (winning) response and y as the rejected (losing) one. In the following sections, we provide a brief overview of DPO, SPIN, recent token-level extensions of DPO and traditional knowledge distillation approaches.
Direct Preference Optimization (DPO). DPO [66 provides an elegant and efficient alternative to RLHF [1I] by avoiding explicit reward model training. Instead, it reformulates the reward function r(z,y) using closed-form expression based on the ratio between the policy model and a fixed reference model:
To(y x) Tee x))
r(x, v) Blog
Blog (xx),
(17)
where Te is the current policy model, Rref is static reference policy, and Z(x) is partition function independent of the policy. By plugging this reward into the Bradley-Terry framework 43]. the preference probability between two responses Yo and Vi is modeled p(y > 31 r)) = a (r(æ, yw) r(z, st)), where a(-) denotes the sigmoid function. This leads to the DPO loss function, which optimizes the policy directly using preference data:
(y [x) Tret (yw x)) Tee xr)
-llg To x))
LDPO (To: Fret) -B log a (log
(18)
This formulation enables the model to directly incorporate human preference signals into policy learning without needing an explicit reward model reinforcement learning.
Self-Play OineTTnnngg Inspired by Generative Adversarial Networks (GAN) Self-Play flne- tuNing (SPIN) [19] proposes an iterative self- eelf-pla framework where a language model fine-tunes itself by generating synthetic responses and learning to distinguish them from couman-annotater data. Specifically, at each iteration t, the current model Te, generates response y' for given prompt forming a synthetic preference pair (x, y > 3/), where y is the ground-truth response. The model is then updated by minimizing:
[e [e (Blog
Te (y x)) Blogg Th (B' | 1x) Te, (y) x)
To, (y' x)) }.
(SPIN(TO To.) )= Be.ggraan
(19)
where ((-) a convex decreasing loss (e.g., logistic loss), and B3 is sealing factor. This formulation encourages the updated model to assign higher likelihood to responses resembling the ground-truth and lower likelihood to its own earlier responses. This self-play process eliminates the need for explicit reward models or preference-labeled data.
Token-Level DPO. Recent works have recognized the value fine- aineggrineed feedback. Rafailov et al. [16] theoretically demonstrate that DPO can represent any dense reward function by reparameterizing an optimal advantage function within token-level Markov Decision Process. This formulation enables DPO to effectively optimize policies at the token level. Building upon this, TDPO [17] introduces forward Kullback-Leible divergence constraints and leverages the Bradley-Teer convert sentence-level preferences into token-level rewards, allowing the model to adjust its strategy dynamically during generation. Furthermore, TIS-DPO [8] enhances this approach by estimating token importance weights based on differences in prediction probabilities from contrastive language models, enabling importance sampling that approximates the optimal distribution by assigning weights to each token according to its estimated reward.
model to
21

---

Traditional Knowledge Distillation Approaches. Knowledge distillation (KD) is widely used technique to transfer knowledge from larger teacher model to smaller student model. Traditionally, KD employs the Kullback-Leibler (KI.) divergence to minimize the difference between the teacher and student probability distributions (2I]). Given a sequence x, the stu- dent model learns to match the teacher's output distribution by minimizing the following loss: CKD = E; DE (p(a; X)i, (o; x T)), where DxL(-1) |I denotes the KL divergence and is the temperature to control the smoothness of the distributions.
B Implementation Details and Algorithm
B.1 Experiments Setup
We conduct two experimental settings to evaluate SWIFT: Alignment and Knowledge Distillation. The Alignment setting compares SWIFT with existing alignment methods, while the Knowledge Distillation setting assesses its effectiveness against existing knowledge distillation methods.
For the Alignment setting, we use Qwenl.5-1. 1.88 [26 as the base model. As the teacher model, we adopt Zephyr-7B-SFT-E SFF-Fuull [27]. which is based on Mistral-7B [28] and further fine-tuned on the U1trcchat200k datase provided by HuggingFace, Ultrachat200k is curated 200k subset of the UltraChat corpus [45]. which consists of approximately 1.4 million high-quality instructional dialogues generated via OpenAl's Turbo API. We follow the procedure in [19]. 50,000 prompts are randomly sampled from Ultrachat200k and generate synthetic responses using the base model. For evaluation, we adopt the HuggingFace Open LLM Leaderboard [29], implemented via the Language Model Evaluation Harness [46). The leaderboard covers six representative benchmarks targeting
different capabilities of LLMs: commonsense reasoning (ARC [47), HellaSwag (48). Winogrande
(49). ), multi-task language understanding (MMLU (50]). resistance to misinformation (TruthfulQA [TT). and mathematical reasoning (GSM8k (52). These benchmarks collectively provide rigorous and diverse framework for evaluating both alignment quality and generalization. Additionally, we the
evaluated the output quality of the L.LM using M'T-bench (30 with its provided dataset,
API of GPT-4 as the judge. The details of the benchmarks are provided in table 4 beloo.
Table 4: Details of the HuggingFace Open LLM Leaderboard evaluation datasets, including the number shot examples and the evaluation metric for each.
Dataset
Are
TruthfulQA
Winogrande
GSM8K
HellaSwag
MMLU
# Few-shot
25
5 acc
5 acc
10
Metric
acc norm
mc2
acc_ norm
acc
For the Knowledge Distillation setting. We use GPT2-1.5B [1 as the base model and Qwen2.5-
In-Instt 32 as the teacher. Four datasets are selected for evaluation: DATABRICKSDOLLY
(Dolly) [33. ALPACA (Alpaca) (34), S-NI (S-NI) [35), and DIALOGSUM (Dialogsum) [36]. Final performance is reported using the ROUGE-L metric (37 between generated outputs and human- annotated references. In the state-of-the-art method DSKD [2), distillation is typically performed on a single dataset, while evaluation is conducted across multiple datasets spanning different domains or tasks. In contrast, we construct separate training, validation, and testing splits for each domain, allowing for a more targeted evaluation of knowledge distillation within the same domain. The details of the datasets are provided in table 5 below.
Table 5: Dataset Statistics
Dataset
Train
Validation
Test
Dolly
11,435
1,000
500
Alpaca
10,396
500
500
S-NI
10,414
500
1,902
DialogSum
12,460
500
1,500
22

---

B.2 Hyperparameters
To reduce training costs and memory consumption, we employ DeepSpeed ZeRO-3 [53] and FlashAttention-22 [54] throughout all training iterations. Models trained using the RMSProp optimizer (55) without weight decay, following standard practice for L.M alignment fine-tuning, We set the global batch size to 2, bfloat precision, and apply 10% linear warmup at the start of each iteration. The peak learning rate set to 5 x 10-² for iterations and 1, and x 10-? for iterations 2 and 3 as training approaches convergence. Each iteration is trained for 2 epochs with maximum sequence length of 2048 tokens. For token importance estimation as defined in equation 14 in main paper, set μ 1, with lower and upper clipping bounds L -0.5 and U 1.5, respectively. The hypeparameter is fixed to 1. All experiments are conducted on 2x NVIDIA RTX 4090 GPUs.
B.3 Synthetic Data Generation
We generate synthetic rejected responses using the library vLLM [56] to speed up inference with distributed inference multiple GPUs. We use sampling decoding strategy gener- ate responses, with temperature 1.0 and top_p of 1.0. We consider the prompting template An : (prpaumaaaan:ppppppppa (prompt) AAAsssssttnnt...
B.4 Algorithm
We provided the pseudocode of our proposed SWIFT (Self-Play Weighted Fine-Tuning) method in Algorithm
Algorithm Self-Play Weighted Fine-Tuning (SWIFT)
Require: ((x, (i)henT SFT dataset, me, LLM with initial parameters 0o, me, teacher model,
M: number of iterations.
1: for 0, M
2: for iori=10 1......... ..... do
3: Generate synthetic data v/1 ~ ne, (-|24).
4: end for
5: Compute token importance weights w using ne, and no,
6: 0.+1 arg minsge EvENy) (u (z, v/. m)) - (z. w))]
7: end for
8: return 0r-
B.5 Details of Teacher-Guided Token Importance Estimation
In this section, we provide detailed explanation of the method described in Section 3.3 (Teacher- Guided Token Importance Estimation) the main paper, addressing how token-level importance weights are distilled from a teacher model and mapped to student tokens, especially in the presence of tokenizer mismatches. We also include the full algorithm, computational analysis, and illustrative examples to enhance clarity.
B.5.1 Overview of the Method
Our aim is to develop an efficient mechanism to estimate the importance weight of each token. Intuitively, well-trained teacher-havine been trained on more data with higher model capacity- better signal which tokens carry higher "reward". this section, we present a principled method to distill token-level reward signals from teacher model 8 and transfer them student model Ø
Inspired by TIS-DPO [8] we first estimate the raw importance weight for the tt token in a response y (V ....... Vr) as
waw = k -expppp clamp log
Mer (yr x, y
L, U))
(20)
Tas (11 x, y =t)
23

---

where clamppa. U) = min mmaxma L), U) truncates the log- to LL U] to reduce variance and stabilize optimization. Here and μ are constant: and U are lower and upper clipping bounds.
However, this formulation assumes that the teacher and student models share the same tokenizer and vocabulary. In practice, this assumption rarely holds. A sequence might be tokenized by the teacher model [y 3/2 ..... Mi and by the student different sequence [ys 3/ ..... ................................ y. making direct loken-to-token alignment in (20) infeasible. To address the misalignment issue caused by differing tokenizers, we propose generalizable token mapping strategy based on shared surface segments.
Case 1: Shared Tokenizer. In the scenario where both the teacher and student models utilize an identical tokenizer, the alignment between their respective token sequences is inherently preserved. Specifically, each token y/ generated by the student model corresponds directly to a token produced by the teacher model the same position, thereby eliminating the need for additional alignment mechanisms. As result, the distillation weight W/ associated with each token position t can be directly adopted from the unprocessed or raw weighting scheme. Formally, we define:
Wp
(21)
where w!th represents the original distillation weight derived from the teacher model's 's outputs as in 20 without any post-pol proseesoossigg or re-alignment.
Case 2: Different Tokenizers. When tokenizers differ, direct index alignment between tokens becomes unreliable. Rather than aligning tokens individually, we instead leverage a common structural unit shared across both tokenizations. Most modern tokenization algorithms, including BPE 23 SentencePiece [24), and WordPiece [25], segment text into subwords while generally respecting word boundaries. This means that each token strictly belongs to either a single word or the whitespace separating words, making word spans a natural and reliable basis for alignment. Leveraging this property, we segment the original text into coarse-grained units corresponding to complete words and their leading whitespace, resulting This ensures that every token from both teacher and student sequences can be assigned to one of these shared components, which are small enough to preserve local semantics but broad enough to encompass all subword variations.
a sequence of shared components
{ci, C2, ................................ .K
Suppose student token yf belongs to component Ch. and the teacher tokens in that segment are T(c.) {v/f' wf 1 w/ Assuming all tokens within a shared component contribute equally, define the student's token importance as the average of the teacher tokens' weights in the corresponding component c₁:
Tar (V. x. ver) Taa (yg
L. U 1))
w/ [[(N]] (.ET((.)
Σ
k- exep μ clamp log
(22)
B.5.2 Pseudo-Code and Computational analysis for Guidhe Token Weight Estimation
The complete process of the Teacher-Guided Token Weight Estimation method is described in Algorithm 2
Given text response we first tokenize it using the teacher tokenizer tokr and the student tokenizer toks to obtain token sequences YT and y respectively. In addition, the raw text y segmented into a sequence of shared components C, where each component consists a full word along with its leading whitespace. This segmentation has time complexity of (nn), where n is the number of characters in the input y.
For each component belong to c. We then will estimate the weight for all the student tokens belonging to this component Since each token belongs to exactly component, the algorithm processes each student token only once.
C, we define T(c) and S(c) the sets of teacher and student tokens that
The core computation involves measuring the discrepancy between the teacher and student log- likelihoods. For each student token E S(e), we compute the log-probability under the student model, and for each teacher token T(c). compute its log- log-probability under the teacher
24

---

Algorithm 2 Teacher-Guided Token Importance Estimation
Require: Teacher model & student model T. teacher tokenizer tokr, student tokenizer toks. hyperparameters (ki ps, L, U response y.
Ensure: Importance weights w.S "1:lys] for student tokens.
1: yr < tokr(y), y +- ttks(y)
2: C < WORDSEGMENTTS(U/
D. Segment raw text into word-level spans
3: for each component in C do
4: T(c) - {t: Y.," c}, S(c) {i: c
5: for each t [c) do
6: a4 1-4-lopp log (Y? x, YZ1))
7: end for
8:
for each i ((c) do b (Y;s x, Y21)
9:
for each / (c)) do
11:
Le x az - b; w k exp(1 clamp(les, I, U))
12:
end for
14:
w + Tiol EiET(O) w
>> eq 22
15: end for
end for return 11- |
17:
model. The weight for student token i is then obtained by averaging the clamped exponential of the log-probability differences between the teacher and student tokens within the same component, as formalized in Equation 22
This design has two critical efficiency advantages: (i) the decoupled log probability computation allows for only a single forward pass per model per response, and (ii) the local averaging over shared components yields robust token importance estimates. These properties make the algorithm both scalable and robust differences in tokenization schemes between teacher and student models.
B.5.3 Illustrative Examples
Figure 3 illustrates Teacher-Guided Token Weight Estimation method applied to the input "Taylor Swift is a singer not an AI algorithm" The sentence is segmented into shared components (green), which serve alignment components between teacher tokens (red) and student tokens (blue). For each component, token-level log prgobrbbabilities are computed from both models. Stu- dent token weights then computed using divergence function (xx) k eep(xx-cllmp)) clammpa, L, U)) over the difference between teacher and student log big--obbbbilttes. In cases where multiple tokens align to the same component, weights are averaged across the corresponding pairs.
Note that, there may be more than one whitespace character between two words, or including newline characters (e.g.. An). Importantly, modern tokenizers do not tokenize words slone-they also treat sequences of whitespace (spaces, tabs, or newlines) as separate tokens. In our implementation, we preserve these segments standalone components, denoted as [SPACE].
B.5.4 Comparison with Prior Work.
Compared to the contrastive-ben based token weighting in [18), our method significantly more compu- tationally efficient, as it does not require training separate positive and negative models. We validate this by providing direct comparison of the total time cost between SWIFT and the contrastive-ban token weighting approach (TIS-DPO) in Appendix C.T. Furthermore, shown in Table 3 in main paper, our teacher- reacher-guided guided estimation consistently achieves superior performance, confirming both its efficacy and practicality.
25

---

y= TTaylor is a singer not an Al algorithm"
Teacher token log-prob
a
Of
of
Teacher token
'Tallor'
'GSwift'
'GAl'
'Galgo'
'rithm"
Shared component C
Taylor
Swift
Algorithm
Student token
Tay
'lor'
'GSwift'
"GAU"
'Galgritthm' b!3
Student token log-prob
b;
62
bga
Wy
W
Token weight
D(a b))
D(a) bb)
D(af-br
D(a t0 bg9
D(a 1- bg )+ D(a?" 699)
2
Figure 3: An example of our Teacher-Guided Token Weight Estimation method where D(x)
k exp(x- clamp(ær, L, U))
C Additional Experiment Result
C.1 Training Overhead
To further examine the computational efficiency of our method, compare the total runtime of the full training pipeline between SWIFT and the contrastive-based token weighting approach (TIS- DPO), introduced in [18]. The TIS-DPO framework comprises three strategies for estimating token weights: (1) TIS-DPO (P), which guides the original LLM with contrastive prompts; (2) TIS-DPO (S), which involves training two separate LLMs via SFT on winning and losing responses, respectively; and (3) TIS-DPO (D), which performs both forward and reverse DPO training using winning and losing responses. Among these, [18] report that TIS-DPO (D) achieves the strongest empirical performance, and thus we adopt for our comparison.
Table 6: Total times for different iterations under SWIFT and TIS-DPO compute weight method.
SWIFT
TIS-DPO (D)
Iteration
Generation
Compute Weight
Training
Generation
Compute Weight
Training
Iter
0.52h
3.08h
6.03h
0.52h
15.33h
6.16h
Iter
0.51h
3.05h
6.13h
0.52h
15.24h
6.10h
Iter
0.52h
3.10h
6.01h
0.50h
15.15h
6.08h
Iter
0.49h
3.05h
6.10h
0.51h
15.28h
6.06h
Table presents the generation, weight computation, and training times for both SWIFT and TIS-DPO (D) across different self-play iterations. Notably, while generation and training times are comparable between the two approaches, SWIFT demonstrates a significant reduction in weight computation time. This efficiency gain arises because TIS-DPO (D) requires training two distinet models obtain token weights. This highlights the superior computational efficiency of our approach over TIS-DPO (D). Moreover, as shown in Table main paper, SWIFT not only offers reduced overhead also achieves better overall performance.
To further quantify the computational overhead of SWIFT, report the GPU-hours required for each stage of the SWIFT pipeline on the 50k Ultrachat subset using a single NVIDIA H100 GPU, The main phases include: (1) response generation, (2) token weight computation (which includes the teacher forward pass), and (3) training. The breakdown per iteration is shown below:
As shown, the teacher forward pass (offline, single forward) only takes Ô. 19 hours it minutes for 50k samples, which we believe is quite efficient and does not contribute significantly to the overall
26

---

Table 7: GPU-hours per iteration for each stage of SWIFT on 50k Ultrachat samples.
Iteration
Generation (1)
Compute Weight (2)
Training (3)
Teacher Forward
Overall
0.24
0,58 h
1,45 h
0.19 h
2.27 h
0.23
0.57 h
1.42
0.19 h
2.22
2
0.24 h
0,58 h
1,43 h
0.19 h
2.25 h
0.22
0.55 h
1.40 h
0.18 h
2.17
training cost. Notably, in this table, we only used sequential (single-process) inference for simplicity; parallel inference would further reduce this time substantially.
To compare efficiency with distillation baselines. we report the training time per sample and peak GPU memory usage across SFT, ULD, DSKD, and SWIFT, However, we would like to note that our distillation codebase based on DSKD, which uses a different training library than our SWIFT codebase, making direct comparisons challenging. However, made every effort to fairly compare the training time per sample and peak GPU memory usage across these baselines in the table below:
Table Training time per sample and peak GPU memory usage.
Method Training Time Sample
Peak GPU Memory (GB)
SFT
0.26
51.08
ULD
0,60
70,20
DSKD
0.54
66.63
SWIFT
0.30
55.39
These results show that SWIFT is comparable to SFT in runtime and memory usage, while more efficient than typical distillation baselines.
C.2 Robustness to Noisy Guidance
To examine the stability of SWIFT under imperfect guidance, we inject uniform noise in the range [-0.2, 0.2] into the teacher-estimated token weights. As shown in Table 9 performance decreases slightly under noise, but SWIFT remains stable across iterations, indicating resilience to moderate perturbations teacher signals.
Table 9: Performance comparison under 20% annotation noise
Noise
Iter
Are
TruthfulQA
Winogrande
GSM8k
MMLU
HellaSwag
Avg
itcO
39.16
41.01
61.96
33.43
44.41
61.69
46,94
itel
38.91
38.93
61.80
37.91
43.83
61.32
47.12
0
itc2
39.76
40.02
62.04
33.60
44.61
62.07
47.02
ite3
39.78
39.12
61.48
37.93
44.84
61.63
47,46
+:0.2
ite0 37.07
38.37
62.09
32,22
44.06
60.60
45.74
10.2
itel
37.58
38.95
61.75
34.51
43.45
60.85
46.18
+0.2
itc2
37.63
39.54
62.13
33.19
44.17
61.48
46.36
10.2
ite3
38.06
39.08
62.01
35.18
44.09
61.53
46.66
Moreover, would like to clarify why method might actually have advantage in lower-data regimes. A crucial premise of this paper is that ground truth can contain some low-reward tokens. These tokens can be understood as noise in SPIN [19]. By introducing token-level importance estimation, SWIFT can effectively identify and filter out such noisy tokens, optimizing primarily the high-quality parts of responses.
C.3 Extended Evaluation Reasoning and Agentic Tasks
To further evaluate the applicability SWIFT beyond preference alignment, we conduct experiments reasoning-intensive intensive and agentic settings. For reasoning, two challenging benchmarks:
27

---

Big-Bench-Haid (BBH) [57 and Discrete Reasoning Over Paragraphs (DROP) [58. The student model is Qwen2. 5B Instruct and the teacher model Qwen3-32B. Results show that SWIFT consistently improves over baselines on reasoning tasks.
Table 10: Performance on reasoning benchmarks (higher is better).
Benchmark
Base
DPO
SPIN
SWIFT
BBH
65.07
64,23
64.85
66.01
DROP
59.28
59.10
58.62
62.03
For agentic capabilitiis,, we evaluate on the ToolBench [59]
benchmark, which measures performance tool-augmented interaction tasks. SWIFT again demonstrates consistent improvements.
Table 11: ToolBench evaluation agentic interaction tasks.
Metric
Base
DPO
SPIN
SWIFT
Act.M f
46.69
46.12
47.01
48.36
f
40.51
40.63
40.73
42.08
HalluRate +
1.60
1.44
1.42
1.40
Rouge-L f
4.35
4.57
4.40
4.61
These additional results confirm that SWIFT extends effectively both complex reasoning tasks and real-world agentic interaction scenarios.
C.4 Evaluation on Larger Backbone Models
To assess the scalability of SWIFT beyond Qwen1 10en1.5-1..0 5-1.8B conduct experiments on recent, stronger instruction-tuned and reasoning-capahh backbones. We evaluate two student-teacher config- urations: Qwen3-322 - Mistral- 7B-v0- 1 and Qwen3-322 - Qwen2. 5-7.---IISttt IIB-Insttut.. Re- sults in table 12 below confirm that SWIFT continues to deliver meaningful performance gains even with newer and stronger instruction and reasoning models.
Table 12: Scaling analysis with larger backbone models.
Setting
Model
Are
Truthful
Wino
GSM8k
MMLU
HellaSwag
Avg
Qwen3-32B
71.08
59.29
76.95
80.97
81.53
83.96
75.63
Base
59.90
42.65
77.43
31.99
60.60
83.47
59.34
DPO
60.70
42.11
76.59
32.65
61.30
83.92
59.55
Mistral-7B-v0.1
SPIN
61.73
44.63
77.07
33.14
62.76
81.68
60.17
SWIFT
63.06
51.25
78.20
42.20
65.17
81.13
63,50
Base 65.96
64.70
75.06
68.61
73.49
81,45
71.55
DPO 66.12
63.50
74.19
67.49
73.02
81.67
71.00
-wwen2557B--nsttee
SPIN
66.53
65.76
75.17
69.35
72.38
82.30
71.92
SWIFT
66.98
66.21
75.53
73.95
73.36
82.45
73.08
C.5 Teacher Quality Dependence
To assess the sensitivity of SWIFT to teacher quality, experiment with multiple teacher models of varying strength and report the resulting student performance after training with each teacher model.
These results show that while stronger teachers yield slightly better student performance, the gains are not strongly correlated with teacher capability. This is because teacher is only used to estimate the importance weight of token, not to directly control what the model generates. As result, SWIFT maintains stable and effective performance across diverse teacher-studen pairs and does depend heavily on highly capable teachers.
28

---

Table 13: Effect of different teacher models student performance.
Setting
Are
Truthful
Wino
GSM8k
MMLU
HellaSwag
Avg
Zephyr-7B-SFF Full 60.41
43.73
74.19
26.76
60.92
82.85
58.14
+ Qwen1.5-1.8B
39.78
39.12
61.48
37.93
44.84
61.63
47.46
Swwn2.5-7B-Instuun
65.96
64.70
75.06
68.61
73.49
81.45
71.54
+ Qwenl.5-I.8B
39.91
40.11
62.43
37.56
44.50
62,43
47.82
Qwen3-32B
71.08
59.29
76.95
80.97
81.53
83.96
75.63
Qwen1.5-1.8B
40.27
38.96
63.51
38.83
44.69
62.19
48.07
C.6 Qualitative Analysis on MMLU benchmarks
To further analyze how SWIFT impacts different knowledge domains, we conducted a qualitative analysis using the MMLU benchmark, which covers diverse knowledge domains. MMLU comprises four major sub-domains: Humanities, Social Sciences, STEM, and Other. Results across iterations are reported in Table 14 below.
Table 14: MMLU subdomain analysis across SWIFT iterations.
Domain
Base
Ite 0
Ite 1
Ite 2
Ite 3
Humanities
39.85
40.89
40.13
40.70
40.78
SocialSci.
47.43
48.29
47.90
48.88
49.59
STEM
34.61
36.76
37.08
37.93
38.12
Other
50.08
52.08
52.11
51.75
52.14
The results show clear and consistent improvements, especially STEM and Social Sciences, which suggests the model is getting better at reasoning and logical thinking. Meanwhile, performance in Humanities and Other domains remains stable improved, showing that the model's reasoning gains didn't ' hurt its general knowledge or overall quality.
C.7 Qualitative Comparison Between Teacher-Guided and Contrastive Token Weighting
To provide further insight into how SWIFT differs from contrastive token weighting (TIS-DPO [8] we conduct qualitative analysis comparing the most influential tokens selected by each method.
Specifically, at each iteration, the contrastive weighting strategy adopted from TIS-DPO relies on two separate student models (a positive and a negative model), which are trained via DPO on pairs constructed from (ground truth, model respeose))oose and (model response, ground ground_truth)). Token importance is then estimated based on the difference in logits between these two models. the
While effective initially, this approach faces inherent limitation iterations progress: model improves over time, its generated responses become increasingly similar to ground truth. Consequently, the distinction between the ground truth and model response training pairs diminishes in later iterations, weakening the contrastive signal and thereby potentially reducing the quality of token importance estimates. In contrast, our proposed SWIFT method employs fixed, external teacher model and it is unaffected by the student's 's iterative improvement, it provides stable, high- quality importance signals across iterations.
Furthermore, we performed an additional qualitative analysis, comparing the top 20 highest-weighted tokens from each method. Token rankings computed based on the average importance per appearance, defined by:
Σ importance weights assigned t number of appearances of
score(t)
(23)
The results presented below:
Top 20 Tokens of SWIFT:
29

---

["Serve", "tering", "antibiotic", "intermittent", "evaluates", "destroys", "Wealth", "FDA", "angelo", "onne", "visited", "paralyzed", "aken", "toughest", "Pricing", "CLA", ".dequeue", "avenous", "plings", "specifies"]
Top 20 Tokens of TIS-DPO:
("FACE", "[e", "LANG", ",dest, "ReturnValue", "baseUrl", "@Print", "keycode", "-has", "HG", "repaint", "Denver", "FINAL", "simil", "iža", "[/*", "owler", ".des", "Teens", "Ač"]
SWIFT selects semantically meaningful and contextually relevant tokens (e.g., antibiotic, FDA), reflecting focus on factual and domain-rich content. In contrast, TIS-DPO often prioritizes tokens resembling code artifacts or noise (e.g., ReturnValue, @), suggesting less semantic alignment.
However, we understand that our subjective judgment alone might not sufficiently convincing, further conducted objective evaluation using GPT-4o as external evaluator assess each method's top-20 tokens. We designed the following targeted prompt for GPT-4o:
Prompt
Below, will provide the top 20 most important tokens across an entire dataset, identified by two different methods. Please evaluate which method is better based on key criteria for token importance in a sentence, such as: semantic relevance (how meaningful content-rich the token is), syntactic role (its grammatical contribution), contextual influence (how much it affects the surrounding content), and task-specific utility (its contribution downstream tasks like classification retrieval). Here are these tokens:
- towkess_of_eehholll (top_20, 2e__200tokees___p_p Sfa__o_SSWWWWo SWIFT)
(top_20 1ec00_tokes__eee of of_TIS-
GPT-4o provided the following summarized judgment:
LLM response
Method clearly better overall than Method 2. It includes more semantically rich, syn- tactically functional, and contextually influential tokens, many of which are likely to be task-relevant (e.g., medical, procedural, or evaluative terms). Method appears to highlight code artifacts, UI labels, tokenization noise, which are often less useful for language tasks.
C.8 Generation Examples
Tables 15
and present additional generation examples produced by our SWIFT fine-tuned model across different training iterations. Table showcases examples from math problems, Table focuses on pode-generation tasks, and (ableeTTjinclude general cnstruction-folloinn cases. These examples illustrate a clear improvement in response quality compared to the outputs from the initial SFT checkpoint. Notably, as the number of iterations increases, the model's responses become more concise and better aligned with the ground-truth completions. For clarity, in each prompt, responses generated by SFT, iteration 0, iteration 1, and iteration 2 are respectively labeled as Response 1, Response 2, Response 3, and Response 4.
30

---

Table 15: Generation example of our fine-tuned model by SWIFT at different iterations. GPT-4 chooses SWIFT iteration 2 over SFT, and iteration 1.
Prompt
<Human>:1f the year is divisible by both 4 and 100, is not leap year, unless it is also divisible by 400.
<Assistant>:
Ground truth
This is a true statement. For example, the year 1900 was divisible by both and 100 but not by 400, it not leap year. However, the year 2000 was divisible by both and 100 and also by 400, so was leap year.
SFT
This is true. For example, the year 1900 was divisible by both 4 and 100 but not by 400, so it was not leap year. However, 1904 was divisible by but by 100, so it was a leap year. The same applies to the year 2000, 2004, and 2008.
Iteration 0
True. For example, the year 1900 was divisible by both 4 and 100 but not by 400, so it was not a leap year. However, the year 2000 was divisible by both and 100 and also by 400, it was leap year.
Iteration 1
This is true statement. For example, the year 1900 was divisible by both and 100 but not by 400, so it was not leap year. However, the year 2000 was divisible by both and 100 and also by 400, so was leap year.
Iteration 2
This is a true statement. For example, the year 1900 was divisible by both and 100 but not by 400, so it not leap year. However, the year 2000 was divisible by both and 100 as well as 400, so it was leap year.
Judgment
Comparison: Response 3 = Response 4 > Response 2 Response 1.
Explanation: Responses 3 and 4 are identical and fully match the ground truth both in correctness and clarity, providing the textbook explanation and appro- priate examples (1900 and 2000) without unnecessary additions. Response is also correct and concise but slightly less formal in tone by starting with "True" instead complete sentence, which slightly reduces explanatory quality, Response 1, while factually correct in its individual statements, introduces un- necessary examples (1 (1904, 2004, 2008) that deviate from the prompt's focus the specific exception rule involving divisibility by 100 and 400, potentially confusing the reader by adding standard leap year cases that do not illustrate the special condition described in the prompt.
31

---

Table 16: Generation example of our fine-tuned model by SWIFT at different iterations. GPT-4 chooses SWIFT iteration 2 over SFT, and iteration 0, 1.
Prompt
EHuten>>Write Write Python program that prompts the user enter string and then counts number of characters the entered string. The program should print the total number of characters in the string. <Assistant>:
Ground truth
String input( 'Enter string: ")
count =
character string;
count +=
print(" total number of characters the string is:", count)
SFT
Here's a possible solution:
string = input( ""Eterr a string:
count = 0
for character in string:
count
print("The total number of characters in the string is:", count)
Here's how the program works:
1. We prompt the user enter string using the input() function and store string the variable string.
2. We initialize a variable count to O, which will be used to keep track of the number of characters in the string.
3. We use for loop to iterate over each character in the string. The loop iterates over each character in the string using string["]. which gives us slice of the string that starts the current position and goes (but not including) the end of the string. We add the count variable to increment it for each character we encounter.
4. We use the print() function to output the result to the screen. use string interpolation to embed the count variable the output string, so it's easy to see how many characters are in the string.
5. The program terminates when the user enters string, to wait for user input before printing the final result.
while loop
Iteration 0
String input( 'Enter string: ")
count = 0
for character in string:
count +=
print("Total number of characters in the entered string is: ", count)
Iteration 1
String input( "Enter string: ")
count =
for char in string:
count print total number of characters the string is:", count)
Iteration 2
String input( Enter string: ")
count
for character string:
count
print( total number of characters the string is:", count)
Judgment
Comparison: Response 4 Response Response Response 1.
Explanation: Responses and 3 are functionally identical to the ground truth, only differing slightly variable naming ("char" vs. "character") but maintain- ing correct, concise, and syntactically sound code. Response 2 also correct has a minor stylistic difference in the output string ("Total number of characters in the entered string is") that deviates from the ground truth and slightly affects alignment. Response 1, while technically correct in the code block, contains multiple issues in the explanation: it introduces confusion by mentioning nonexistent while loop and incorrectly describes the of string["]. which is invalid syntax misrepresents how the loop operates. These inaccuracies make the least reliable response.
32

---

Table 17:
Generation example of our fine-tuned model by SWIFT at different iterations. GPT-4 chooses SWIFT iteration 2 over SFT, and iteration 0,
Prompt
<iuman>-Given the text: During The Daily Show's "Between the Scenes" segment April 10, Noah asked television mogul Winfrey if there any sense of normalcy left in her life. The answer, quite simply, was no. Even when she tried to dig up a passable answer, she hilariously failed. The avocado orchard is tucked away on the property of Winfrey's s 23.000-square-fod Montecito home, which she purchased for $50 million back in 2001. The home also boasts 5, 5,000-square-fool ranch-style house, a pool, horse stables, an equestrian ring. fruit orchard, and more. Simply put, Oprah's 's home much like guacamole -
is extra. Get it? Could you provide more information about the features of Oprah Winfrey's ' Montecito property, including the size of her home and other amenities?
<Assistant>:
Ground truth
Yes, Oprah Winfrey's Montecito property is 23,00 23.000-square-foo in size and in- cludes 5,000-square- foot ranch-style house, pool, horse stables, equestrian ring, a fruit orchard, and an avocado orchard. The property was purchased for $50 million in 2001.
SFT
Yes, the text mentions that Oprah Winfrey purchased her 23,000--quare-foor Montecito home for $50 million in 2001. It also states that the home has a 5,000-square-fove ranch-style house, pool, horse stables, an equestrian ring, a fruit orchard, and an avocado orchard. Therefore, the size of Oprah's 's home is not given in the text.
Iteration 0
According to the text, Oprah Winfrey purchased her Montecito property for $50 million in 2001. The property consists of 23,000 23,0000-sqarr-foo Montecito home and a 5,000-square-foot ranch-style house, an irrigation pond, natural walkways, fountain, tailgating capability, and more. Additionally, the property includes a fruit orchard and an avocado orchard
Iteration 1
Of course! Oprah Winfrey's Montecito property is 23.000-square-forn in size and was purchased for $50 million 2001. The property also includes 5,000- square-foot Ranch-style house, pool, horse stables, equestrian ring, fruit orchard, and an avocado orchard. truly a sprawling estate that puts even avocado-curious minds to shame.
Iteration 2
Yes. Oprah Winfrey's Montecito property is 23,000-square-fott in size and includes a 5,000-square-foo ranch-style house, pool, horse stables, an equestrian ring, fruit orchard, and an avocado orchard. She purchased the property for $50 million in 2001.
Judgment
Comparison:
Response 4 > Response 3 Response 2 Response 1.
Explanation: Response is the most accurate, concise, and directly mirrors the ground truth in both factual details and structure. making it the clearest and most reliable answer. Response structured but adds unnecessary embellishment ("puts even avocado-curious minds to shame") that, while engaging, introduces subjective commentary not present in the ground truth. Response includes factual inaccuracies (mentions "irrigation pond, natural stone walkways, fountain, tailgating capability") that are not supported by the text, thereby reducing its reliability. Response 1 contains a clear contradiction- dentradiction-ddssiiie correctly listing most amenities, it confusingly states "the size of Oprah's 's home not given in the text," which is factually incorrect, making it the weakest response.
similarly factually correct and well-
33

---

D All Proof
Proof of Lemma 3.1
We prove by induction. First, for t = 1, we have
Pa (y1 z) pa (2r v1) and pas (y1 x)= pga (2r, 91) Pa (x)
(24)
Pd (z)
From Eq. (24). we reach the conclusion for t = 1. Assume that we have the conclusion for S - 1. We further derive
Pa (x,, Wgt) Pa (vi x, Mee)) Pa (x)
(ys x, ve,))),
Pos (z, (5t) Pos (ye n, vet) pa (x)
Pos (yt x, ves).
(25)
From Eq. (25). we reach the conclusion for t. Proof of Theorem 3.3
Let r? RØ be the solution of
[.DD [re (læ,, g<cel .2)) Bo Ira (x, gccel .cll
Vt: max E NER:
(26)
According to the consistency property, there exists r' E R such that Vt rī (fx, vell r* Ugt). We now prove that is the solution of the following OP:
= (27)
Σ 2 (1x.)ec).00) Ex (Ix, vecl ((r,vel))))))
max D(z.)) oo g(1)))))) (1m)
FER*
where we denote ([z, 3/cc] 310 r (z, vg1).
Let Fm R* be the solution of the OP in 27 We have
Σ 7 1" ([x, ver) ve) Σ 7" ([x, Vet] ./)) Σ 21 17th ([r,yee) (1) F" ([x, Ueel .16)1
Σn- lit (Ix, Veil v1) Fē (1x, Veel v11l Σ -0 Int ([x, Ner, rg (1x, vec v(1
-E ([x, vel we) - r" (1x, vea) v(),
(28)
where F. E R are the consistent versions of 7* E R in the token-level families.
Finally, (28) indicates that r is the solution of the OP in 22
Proof of Theorem 3.4
We consider
d1 (Po, Pos) == max ((x.))-D. rER*
[z v) R(x,8/))
(29)
Because if E R* then -r E R* we have di (IP,, Pa=) > and mingsee di Pos) 2 Moreover, the minimization is obtained at 0 for 0 such that P&s = Pa-
We now consider
d2 (Pa,, Pas) = max Re (...) (-1z) "ER*
Σ
Σ E ([x, (1x,
(30)
Because if E R* then -r E R* we have dg (P4, Pos) 2 and mingsee d2 (Pa, Poss) Moreover, the minimization is obtained at 0 for 0 such that Pass = Pa according to Lemma 1.
2
34

---

Proof of Lemma 3.5
Our proof is adopted from the proof of Lemma 4.2 in 17).
WeATT og (1r, vb)] 3DxL (TOS (- [ar, Meel) oo ( [ar, weel))
max De (a 8
We Ropp (1r,ve)) v6)) wi wiVaag ([r, - B log Tgs (v6
[ar, Meee)
max E D Da
Tog (vl
[2, Veel)
Tos (2/ Ix, v<el) exp We Omo (1x, (er.
max
E. (12.00 wi Voog Ueel) + log Z ([x, y<t) yet): 83]
Blog
Tas (y/ [x. v<(o))
Z (120, F-ee] wr, B)
-BDKL Taa (vl [x. v<l) Tog (v/ [2,, y-el) exp { w. Qrgg (Ixr, v-el 36) Z ([2, FCt]; we, 3)
max E Bayer D.yl a (12001)))
we Vong ((r, + log Z (lr.Vell:w,, B3
This concludes our proof.
Proof of Lemma 3.6
Using the same derivations as in 17 we gain
R (<, v) V ([x, y "1) 7 V ((r,NeT+1))
T
17 (r ((a,,ee)) w1) + VVa (Ir,))++))) - Ve ((x,
V ([2, - Vx (10,8<4+1))
[r (1z, vecll 11)) Vx (Ix, vel+il) Va (Iz. velll,,
since = 1. We further have
Qx (1x, vecl,8)) = r ([x, ve) + Vn (1x, il),
A (1z, [ee], ve))) = Q ([x, veel, w)) - Vx (Iz, vee))
=r 91) Vx (1x, )<++)))) Vx ((x, vce))-
By noting that
Vn (1x, Ver+)) 0nd Vx (1Z,0~1) Vx ([x,8' 111 Vx (Ix, II),
we gain
1 (R(2)))-R(z..) (x,v')) Σ
Enn Ar (1x, Vecl Σ (Ix, vecl. ./))
Proof of Theorem 3.7
We have
Age (z I, Bee)) x, Mee)
Qrg, (1x, (xr,ee))=) = wi 1 log
+ w. log Z (1x, (erliw,), (31)
Fog (z
We also have
(R (x.0) - R =1 En-- x--1 Angg
Angg ([x, Ver.eel.v)
- ([x, Veel 16)) (32)
35

---

We further derive as
Σ x-1 Argg (1r,Vedl (1)= =- (Rrop (1x, vc) Vzog (1x, Meall]
Σ 1t111 Goog (10, Bysnomog [Grog (1r,vee) mb)
Tos (ye | I, (x)())
Σ x
W 3log
+ " Z (1,, Beeliwe, B)
Tog (ye (x()
8s (1/ (et)
E B
W/ 3 log
+ w Blog Z (1x, Beeliwer (3
T (3/
I.
ags (y/,
(et))
ByiNTOR Wi w Blog Tas (Vk x. Mct))
W/ 3 log
T (y/,
Tox (v' we, yee))
Σ xt1
w/ ( log
z (ye
xr (21)
+ wwi 10-1DDKL (Tăo (: [x, 1 (- xr verc))
T (ya
I., U<t))
=3 Σ
W/ 1 log
Tgs (ye
+ $
Σ Wi 2t DKE (#ée (- (-1 ve)) Tog I 8ki)
Tay (ye
Finally, substituting to Eq.
32
we gain the conclusion.
36


Reviews:
[{"review_id": "dSX48UhecI", "reviewer": "Reviewer_Qm7U", "summary": "This paper proposes a method, namely, SWIFT (Self-Play Weighted Fine-Tuning), for aligning Large Language Models (LLMs) that improves upon existing self-play techniques. Existing methods apply a uniform learning signal across all tokens in a generated response, ignoring the fact that even \"rejected\" responses can contain many high-quality tokens, especially as the model improves. To address this, SWIFT introduces a fine-grained, token-level weighting mechanism into the self-play loop. Experiments are conducted in two settings: preference alignment and knowledge distillation.", "questions": "1. The teacher model provides the crucial token-level weights. If this teacher model has its own inherent biases (e.g., factual inaccuracies, stylistic tics, or harmful stereotypes), is there a risk that SWIFT will not only distill its capabilities but also amplify its flaws in the student? Does the proposed weighting scheme have any mechanism to mitigate the transfer of undesirable biases from the teacher?\n\n2. The final objective in Theorem 3.7 is composed of two main terms, $u(⋅)$ and $v(⋅)$. The term $u(⋅)$ resembles the DPO objective applied to the advantage function. Could you provide a more intuitive explanation for the role of the second term, $v(⋅)$, which involves the difference of weighted sequence KL-divergences? How does it complement the first term to improve alignment?\n\n3. The ablation study shows that teacher-guided weighting outperforms the contrastive weighting from TIS-DPO. Is this purely because the teacher model is stronger, or is there a fundamental difference in the kind of tokens that each method identifies as important? For example, does one method focus more on factual correctness while the other focuses on stylistic flair? A qualitative comparison of the highest-weighted tokens from each method would be insightful.", "limitations": "The paper should more explicitly discuss the risk of error propagation in the iterative self-play loop, where a model trained on flawed self-generated data could degrade over time.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Overall, this paper is easy to follow and well-motivated. The core contribution is to use a stronger \"teacher\" model to estimate the importance of each token in both the human-annotated (\"chosen\") and self-generated (\"rejected\") responses. Instead of traditional knowledge distillation that matches logits, SWIFT uses the teacher's outputs to assign reward-guided weights, effectively telling the student model which tokens to focus on during alignment. \n\nAuthors provide a solid theoretical foundation for this approach, deriving a token-level objective function from an Integral Probabilistic Metric (IPM) that connects to the advantage function used in reinforcement learning. It also introduces a practical method for transferring these token weights between models with different tokenizers by aligning them based on shared word segments.\n\nSome weaknesses and potential improvements can be made:\n1. The success of SWIFT is fundamentally tied to the availability of a powerful teacher model. The performance gains are contingent on the teacher providing high-quality importance signals. This dependency limits the method's applicability in scenarios where a significantly stronger, accessible teacher model does not exist. The ablation study confirms that without such guidance (e.g., using \"equal weight\"), the method's advantage diminishes.\n\n2. The method relies on an iterative self-play process. While the results show consistent improvement, the paper does not fully explore the potential for error propagation. If the student model generates particularly poor responses in an early iteration, and the teacher's guidance is imperfect, the model could be trained on a flawed signal, which might be amplified in subsequent iterations. An analysis of the method's stability over many iterations or in lower-data regimes would be beneficial.", "was_revised": true, "final_justification": "During rebuttal, authors adequately address my original questions. I maintain my original assessment.", "rebuttal": "Dear reviewer Qm7U, We’d like to thank you for acknowledging the novelty and the contribution of our proposed approach. We address your raised points as follows.\n\n\n---\n\n**Weakness1. Dependency on a Strong Teacher Model for SWIFT’s Effectiveness**\n\n**A1:** We agree that a powerful teacher greatly benefits SWIFT by providing high‑quality token‑importance signals. Nevertheless, Table 3 (ablation study) shows that our contrastive‑weight strategy — inspired by TIS‑DPO [1] — achieves competitive performance without any external teacher. This variant relies solely on the base model: it trains separated “positive” and “negative” models via DPO and estimate token weights from their logits difference. Because it does not rely on any teacher model, it remains a viable approach in scenarios where access to a strong teacher is unavailable, while still outperforming other non-teacher baselines. We believe this makes the method broadly applicable, even in resource-constrained settings.\n\n---\n\n**Weakness2. Risk of Error Propagation in Iterative Self-Play with Imperfect Guidance**\n\n**A2:** We agree that imperfect teacher guidance could adversely affect the student's training process. To directly address this concern, we conducted an additional robustness experiment in which we added random noise in the range [-0.2, 0.2] to the teacher’s token weights. As shown in the table below, performance dropped slightly, and SWIFT remained stable across iterations.\n\n| |Noise level|Arc|TruthfulQA|Winogrande|GSM8k|MMLU|HellaSwag|Average|\n|----------------------|-----------:|----:|-----------:|-----------:|-----:|-----:|----------:|--------:|\n|SWIFT (ite0)|0|39,16|41,01|61,96|33,43|44,41|61,69|46,94|\n|SWIFT (ite1)|0|38,91|38,93|61,80|37,91|43,83|61,32|47,12|\n|SWIFT (ite2)|0|39,76|40,02|62,04|33,60|44,61|62,07|47,02|\n|SWIFT (ite3)|0|39,78|39,12|61,48|37,93|44,84|61,63|47,46|\n|SWIFT (ite0)|±0.2|37,07|38,37|62,09|32,22|44,06|60,60|45,74|\n|SWIFT (ite1)|±0.2|37,58|38,95|61,75|34,51|43,45|60,85|46,18|\n|SWIFT (ite2)|±0.2|37,63|39,54|62,13|33,19|44,17|61,48|46,36|\n|SWIFT (ite3)|±0.2|38,06|39,08|62,01|35,18|44,09|61,53|46,66|\n\nMoreover, we would like to clarify why our method might actually have an advantage in lower-data regimes. A crucial premise of this paper is that ground truth can contain some low-reward tokens. These tokens can be understood as noise in SPIN [2]. By introducing token-level importance estimation, SWIFT can effectively identify and filter out such noisy tokens, optimizing primarily the high-quality parts of responses.\n\n---\n\n**Question1. Risk of Amplifying Teacher Biases Through Token-Level Weighting**\n\n**A1:** We appreciate the concern that a teacher model’s own biases—whether factual, stylistic, or societal—could be inherited or even amplified. However, SWIFT incorporates two built‑in safeguards that inherently mitigate this risk:\n\n1. **Clipped token weights:**\n   During the estimation of raw importance scores, we apply a clipping function to the teacher–student log-probability ratio, constraining token-level weights within the range $[L, U] = [-0.5, 1.5]$. This mechanism ensures that no single token can disproportionately influence training, significantly reducing the possibility of amplifying outlier biases or inaccuracies introduced by the teacher model.\n\n2. **Smoothing via averaging token weights across shared word segments:**\n   When teacher and student models employ different tokenizers, we group tokens based on shared word-level components and then average token weights within each component. This method reduces the effect of any one token and makes the importance weights more balanced.\n\nWe acknowledge that while these safeguards significantly mitigate potential bias transfer, they may not fully eliminate it. Thus, we consider this issue as a direction for future research and will clearly discuss these points in the final version.\n\n---\n\n**Question2. Intuition Behind the $v(\\cdot)$ Term in the SWIFT Objective**\n\n**A2:** To provide a clearer intuition, theorem 3.7 breaks the objective into two terms:\n\n- $u\\left(x,y,y',\\pi_{\\theta^{S}},\\omega\\right)$ functions as a per-token alignment signal, similar in spirit to a token-level DPO update.\n- $v\\left(x,y,y',\\pi_{\\theta^{S}},\\omega\\right) = \\beta D_{SeqKL}\\left(x,y,\\omega;\\pi_{\\theta^{S}}\\Vert\\pi_{\\theta_{k}^{S}}\\right)-\\beta D_{SeqKL}\\left(x,y',\\omega;\\pi_{\\theta^{S}}\\Vert\\pi_{\\theta_{k}^{S}}\\right)$\n represents the difference between weighted sequence-level KL divergences from the updated student policy $\\pi_{\\theta^S}$ to its previous iteration $\\pi_{\\theta^S_k}$. It measures shifts in the model’s conditional distributions, carefully weighted by the token-level importance estimates.\n\nIntuitively, the $v(\\cdot)$ term acts as a form of adaptive trust-region regularization. By penalizing excessive changes in the model’s conditional token distributions—particularly on high-importance tokens—it ensures stability and prevents the model from overfitting to narrow reward signals at the expense of general language quality or coherence. In this way, the term complements the alignment-driven updates from $u(\\cdot)$ by encouraging conservative updates where necessary.\n\nTogether, the two terms strike a balance: $u(\\cdot)$ pushes the model to better align with preferred outputs, while $v(\\cdot)$ tempers this push with a careful consideration of distributional drift, resulting in both effective and stable training dynamics.\n\n---\n\n**Question3. Fundamental Differences Between Teacher-Guided and Contrastive Token Weighting**\n\n**A3:** We believe there is indeed a fundamental difference—not just a matter of teacher strength—in how each method identifies important tokens.\n\nSpecifically, at each iteration, the contrastive weighting strategy adopted from TIS-DPO relies on two separate student models (a positive and a negative model), which are trained via DPO on pairs constructed from (ground_truth, model_response) and (model_response, ground_truth). Token importance is then estimated based on the difference in logits between these two models. While effective initially, this approach faces an inherent limitation as iterations progress: as the model improves over time, its generated responses become increasingly similar to ground truth. Consequently, the distinction between the ground truth and model response training pairs diminishes in later iterations, weakening the contrastive signal and thereby potentially reducing the quality of token importance estimates. In contrast, our proposed SWIFT method employs a fixed, external teacher model and it is unaffected by the student's iterative improvement, it provides stable, high-quality importance signals across iterations.\n\nFurthermore, we performed an additional qualitative analysis, comparing the top 20 highest-weighted tokens from each method. Token rankings were computed based on the average importance per appearance, defined by $\\frac{\\sum \\text{importance weights assigned to this token}}{\\text{number of appearances of this token}}$. The results are presented below:\n\n- **Top 20 tokens of SWIFT:**\n\n[`Serve`, `tering`, `antibiotic`, `intermitt`, `evaluates`, `destroys`, `Wealth`, `FDA`, `angelo`, `onne`, `visited`, `paralyzed`, `aken`, `toughest`, `Pricing`, `CLA`, `.dequeue`, `avenous`, `plings`, `specifies`]\n\n- **Top 20 tokens of TIS-DPO(D):**\n\n[`FACE`, `[@`, `LANG`, `.dest`, `ReturnValue`, `baseUrl`, `ĉPrint`, `keycode`, `-has`, `HG`, `repaint`, `Denver`, `FINAL`, `simil`, `ï¿½s`, `{/*`, `owler`, `.des`, `Teens`, `ÃĢ`]\n\nSWIFT selects semantically meaningful and contextually relevant tokens (e.g., \"antibiotic\", \"FDA\"), reflecting focus on factual and domain-rich content. In contrast, TIS-DPO often prioritizes tokens resembling code artifacts or noise (e.g., \"ReturnValue\", \"[@\"), suggesting less semantic alignment.\n\nHowever, we understand that our subjective judgment alone might not be sufficiently convincing, we further conducted an objective evaluation using GPT-4o as an external evaluator to assess  each method’s top-20 tokens. We designed the following targeted prompt for GPT-4o:\n\n```text\nBelow, I will provide the top 20 most important tokens across an entire dataset, as identified by two different methods. Please evaluate which method is better based on key criteria for token importance in a sentence, such as: semantic relevance (how meaningful or content-rich the token is), syntactic role (its grammatical contribution), contextual influence (how much it affects the surrounding words or sentence meaning), and task-specific utility (its contribution to downstream tasks like classification or retrieval)\nHere are these tokens:\n- **top_20_tokens_of_method1**:  \n  {top_20_tokens_of_SWIFT}\n- **top_20_tokens_of_method2**:  \n  {top_20_tokens_of_TIS-DPO}\n```\n\nGPT-4o provided the following summarized judgment:\n\n```text\nMethod 1 is clearly better overall than Method 2.\nIt includes more semantically rich, syntactically functional, and contextually influential tokens, many of which are likely to be task-relevant (e.g., medical, procedural, or evaluative terms).\nMethod 2 appears to highlight code artifacts, UI labels, or tokenization noise, which are often less useful for language tasks.\n```\n\nWe hope this comprehensive analysis effectively addresses your question.\n\n---\n\n**References:**\n\n[1] Liu, Aiwei, et al. \"Tis-dpo: Token-level importance sampling for direct preference optimization with estimated weights.\" arXiv preprint arXiv:2410.04350 (2024).\n\n[2] Chen, Zixiang, et al. \"Self-play fine-tuning converts weak language models to strong language models.\" arXiv preprint arXiv:2401.01335 (2024)."}, {"review_id": "zvtcXQc9Bs", "reviewer": "Reviewer_8XmY", "summary": "This paper proposes SWIFT (Self-Play Weighted Fine-Tuning), a method to assign token-level importance weights from a strong teacher model, which are used to guide the student model via distillation. SWIFT is extended from a prior work SPIN, which lets the model to generate responses and improve them with human responses. While SPIN operates on the sequence level, SWIFT further applies token-level refinement via importance-aware token weighting. A general token mapping strategy is applied to solve the issue of unmatched tokenizers between student and teacher. Experiments on alignment setting and distillation setting show that the proposed method can improve from baseline methods across multiple benchmarks.", "questions": "- The performance improvement on Table 1 seems to be mainly from GSM8K compared with baselines. Is there an intuition why it's the case?", "limitations": "Limitations are discussed in Conclusion.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n- The proposed method is intuitive and shows to be effective in experiments.\n- The proposed approach of dealing with different student-teacher tokenizers is interesting.\n- The comparison with SPIN shows the importance of token-level reweighting when training goes on, which justifies the effectiveness of the method.\n\n## Weaknesses\n- Although the paper is well-written in general, the presentation flow is a little disconnected. The theoretical analysis takes too much space, while the algorithm details and experiment results are not discussed extensively in the main paper. It would be helpful to emphasize the main theoretical results and insights, and discuss more on experiments.\n- By introducing a teacher model, and computing token-level weight, the computational cost can be an issue. Is there a way to overcome it by reducing the computations? Or is it possible to combine SWIFT and SPIN?", "was_revised": true, "final_justification": "My rating does not change after rebuttal.", "rebuttal": "Dear reviewer 8XmY, we thank you for your review and for recognizing the intuition, novelty, and effectiveness of our approach. The following are our responses to each individual comment.\n\n---\n\n**Response to Weakness1.**\n> Although the paper is well-written in general, the presentation flow is a little disconnected. The theoretical analysis takes too much space, while the algorithm details and experiment results are not discussed extensively in the main paper. It would be helpful to emphasize the main theoretical results and insights, and discuss more on experiments.\n\n\n**A1:**  Thank you for your suggestion regarding the balance between theoretical analysis and experimental details. We will carefully revise the final manuscript to emphasize the main theoretical results more clearly and allocate more space in the main paper for key algorithmic details and experimental results. We appreciate this feedback and believe these changes will improve the presentation and flow.\n\n\n---\n\n**Response to Weakness2.**\n> By introducing a teacher model, and computing token-level weight, the computational cost can be an issue. Is there a way to overcome it by reducing the computations? Or is it possible to combine SWIFT and SPIN?\n\n\n**A2:** We agree that introducing a teacher model can add computational cost. However, compared to most recent knowledge distillation approaches—which often require the teacher to be used online throughout training and thus demand significant additional memory and compute—SWIFT only needs the teacher for a single forward pass to estimate token importance weights. In particular, SWIFT performs a single offline forward pass of the teacher model over the training data to compute token-level importance weights. These weights are then saved and reused during training, eliminating the need to keep the teacher in memory. Furthermore, since each sample is independent, this step can be fully parallelized to significantly speed up computation. This design keeps the overhead manageable and scalable.\n\nTo quantify this, we report the GPU-hours required for each stage of the SWIFT pipeline on the 50k Ultrachat subset using a single NVIDIA H100 GPU. The main phases include: (1) response generation, (2) token weight computation (which includes the teacher forward pass), and (3) training. The breakdown per iteration is shown below:\n\n| Iteration | Generation (1) | Compute Weight (2) | Training (3) | Teacher Forward | Overall |\n|-----------|------------|----------------|----------|------------------|---------|\n| 0         | 0.24 h     | 0.58 h         | 1.45 h   | 0.19 h           | 2.27 h  |\n| 1         | 0.23 h     | 0.57 h         | 1.42 h   | 0.19 h           | 2.22 h  |\n| 2         | 0.24 h     | 0.58 h         | 1.43 h   | 0.19 h           | 2.25 h  |\n| 3         | 0.22 h     | 0.55 h         | 1.40 h   | 0.18 h           | 2.17 h  |\n\nAs shown, the teacher forward pass (offline, single forward) only takes ~ 0.19 hours ~ 11 minutes for 50k samples, which we believe is quite efficient and does not contribute significantly to the overall training cost. Notably, in this table, we only used sequential (single-process) inference for simplicity; parallel inference would further reduce this time substantially.\n\nFurthermore, when using a teacher is impractical, we offer a teacher-free variant is contrastive weighting strategy, inspired by TIS-DPO [1]. As shown in Table 3 (ablation study), this approach achieves competitive performance without requiring any external teacher. Specifically, it trains separate “positive” and “negative” models via DPO, and estimates token weights based on the difference in their output logits. Because it does not rely on any teacher model, it remains a viable approach in scenarios where access to a strong teacher is unavailable, while still outperforming other non-teacher baselines. We believe this makes the method broadly applicable, even in resource-constrained settings.\n\nFinally, we view SWIFT as an advanced extension of SPIN, which can be flexibly reduced to SPIN or hybridized based on resource constraints. We will clarify this relationship in the final version.\n\n---\n\n**Response to Question1.**\n> The performance improvement on Table 1 seems to be mainly from GSM8K compared with baselines. Is there an intuition why it's the case?\n\n\n**A3:** Thank you for noting the strong improvements on GSM8K. This benchmark is heavily focused on mathematical reasoning, where token-level signals are particularly valuable—rewarding intermediate reasoning steps, not just final answers. SWIFT’s fine-grained weighting helps the student focus on critical tokens, leading to more substantial gains. For other tasks, gains are also observed, but the token-level reward signal seems especially effective for the complex multi-step reasoning required by GSM8K.\n\n\nOnce again, thank you for your thoughtful review and for recognizing the contributions of our work.\n\n---\n\n**References:**\n\n[1]  Liu, Aiwei, et al. \"Tis-dpo: Token-level importance sampling for direct preference optimization with estimated weights.\" arXiv preprint arXiv:2410.04350 (2024)."}, {"review_id": "h0MvENTVLO", "reviewer": "Reviewer_Qi3n", "summary": "The paper proposes SWIFT (Self-Play Weighted Fine-Tuning)—a token-level, importance-aware extension of SPIN that leverages a stronger teacher model to estimate per-token rewards. The authors (i) derive the objective from an IPM formulation that links to advantage-maximization, (ii) introduce a simple tokenizer-mismatch mapping, and (iii) show that SWIFT can act both as a preference-alignment technique and as a lightweight knowledge-distillation method. On six alignment benchmarks for LLMs, small student LLMs gain significant more performance from teacher against other DPO-/KD-style baselines. Ablations confirm that teacher-guided weights dominate random/equal/contrastive alternatives.", "questions": "1. **Compute overhead.**  How many GPU-hours did the single offline teacher pass require for the 50k Ultrachat subset? Could you provide the computation cost of each distillation methods and the SFT baseline? Could it be more than pruning and fine-tuning the teacher model directly?\n2. **Scaling.**  It seems that SWIFT gains the largest improvement in the first iteration, and could not consistently improve the student model in subsequent iterations (Fig 2). Could you provide some insights on the scaling potential of SWIFT along training iterations?\n3. **Teacher quality dependence.**  Table 3 shows uniform weights still help, but less than the teacher to a large margin.  Can you quantify performance as a function of teacher–student capability gap and test with other teachers?\n4. **Qualitative analysis.**  Can you provide some qualitative analysis on the trained models at each iteration? What domains do they improve on?", "limitations": "The paper lists teacher reuse overhead and future efficiency work, but should also discuss how to use this white-box distillation method practically. In real-world applications, the teacher model is often black-box and not easily accessible for analysis.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. **Sound Object Derivation:** The authors provide detailed derivations of the SWIFT objective from an IPM perspective, linking it to advantage-maximization. The use of student forcing, optimal solutions, and the final form is simple to calculate. \n\n2. **Practical Mitigation for Heteogeneous Tokenizers:** The paper introduces a straightforward method to handle tokenizer mismatches, which is a common issue in LLM distillation. \n\n3. **Clear Experiment Settings:** The experiments are well-explained, under a white-box distillation setting, which is an important research problem. \n\n\n### Weaknesses\n\n1. **Limited Scope of Methodology:** The method is mainly extending the token-level DPO (ITS-DPO, etc.) to self-play settings. It is indeed an important extension, but its effectiveness is unclear for recent RL settings on LLMs, e.g., incentivizing long reasoning chains or agentic tasks. Current design and experimentation focus only on preference alignment. \n\n2. **Inappropriate Baseline Comparisons:** There are a few settings differ between the baselines and SWIFT:  \n   (a) In the alignment experiments, SWIFT is trained on 50k SFT dataset and other DPO-style baselines are trained on 62k preference samples. This is not a fair comparison. At least SWIFT should be trained on 62k preference samples as well (using positive responses).  \n   (b) In the knowledge distillation experiments, only SFT/KL-based baselines are compared with SWIFT, while other DPO-style baselines are not included. For instance, by sampling from the student model, negative samples could be obtained for DPO-style baselines.  \n   (c) In Fig 1(b), the rewards calculated for SWIFT and SPIN are not in the same way. SWIFT does use a token-weighted reward. However, sample-level rewards in original DPO could also be obtained from the trained model, which is more direct opponent for SPIN's rewards. Different reward calculation methods could lead to unfair comparisons due to the scales of rewards might be different.\n\n3. **Outdated Backbone Models:** The experiments are conducted on small student models from several years ago (e.g., GPT-2, Qwen-1.5). The effectiveness of the method on more recent instruction models (e.g., Gamma-3) or reasoning models (e.g., Qwen-3) is unclear. The paper should at least mention the limitations of the current experiments and the potential challenges of scaling to these models.", "was_revised": true, "final_justification": "After reading the rebuttal and other reviews, I decide to raise my rating since the authors gave reasonable answers to my questions.", "rebuttal": "Dear reviewer **Qi3n**, thank you for your recognition of our paper. We will now address your questions below.\n\n---\n**Weakness1. Limited Scope of Methodology Experiments**\n\nA1: Thank you for pointing out the value of our method. Our original experiments followed the same setup as most recent DPO-style works, which mainly focus on preference alignment tasks. However, we understand your concern, so we conducted additional experiments focusing on reasoning and agentic tasks. The student model is **Qwen2.5-7B Instruct** and the teacher model is **Qwen3-32B**.\n\nFor reasoning tasks, we use two challenging benchmarks: **Big-Bench-Hard (BBH)** and **Discrete Reasoning Over Paragraphs (DROP)**. The result in table below demonstrate that SWIFT consistently outperforms the baselines on reasoning-heavy tasks.\n||Base|DPO|SPIN|SWIFT|\n|-|-|-|-|-|\n|BBH|65.07|64.23|64.85|**66.01**|\n|DROP|59.28|59.10|58.62|**62.03**|\n\nRegarding agentic tasks, we further evaluated on the **ToolBench** benchmark to assess SWIFT’s performance in agentic, tool interactions.\n|Metric|Base|DPO|SPIN|SWIFT|\n|-|-|-|-|-|\n|Act.EM(↑)|46.69|46.12|47.01|**48.36**|\n|F1(↑)|40.51|40.63|40.73|**42.08**|\n|HalluRate(↓)|1.60|1.44|1.42|**1.40**|\n|Rouge-L(↑)|4.35|4.57|4.40|**4.61**|\n\nThese results support SWIFT’s applicability to complex reasoning task and real-world agentic tasks.\n\n---\n**Weakness2. Concern on Baseline Comparisons: a) SWIFT uses a smaller 50k SFT dataset while DPO-style baselines use 62k preference samples; b) KD comparison lacks DPO-style baselines; c) reward formulas differ between SWIFT and SPIN in Fig 1(b)**\n\nA2:\n(a) We followed the same setup as the SPIN paper to make a fair comparison—using 50k SFT examples for SWIFT/SPIN and 62k preference examples for DPO-style methods. However, we agree that comparing on equal data sizes is also important. So, we ran an extra experiment using a larger 62k SFT dataset by adding random 12k new examples from UltraChat200k. When retrained on this larger SFT dataset, SWIFT achieved a final score on Open LLM Leaderboard of **47.52**, slightly above the original result of **47.46**, confirming that our findings are robust to dataset size differences.\n\n(b) Thank you for this suggestion. However, these DPO-style methods are originally designed for preference datasets, which are not really suitable with SFT datasets. Regarding your suggestion to simulate preference data by treating the ground-truth response as \"chosen\" and the model-generated response as \"rejected\", we believe this construction is suitable for the self-play mechanism of SWIFT or SPIN, and it is unclear whether such modifications would be effective for baselines like DPO or TIS-DPO. Due to rebuttal space constraints, we plan to incorporate a comparison with DPO-style methods adapted to the SFT data in the final version.\n\n(c) We appreciate your observation regarding Fig 1(b). This figure was designed to compare the actual reward signals used by SWIFT and SPIN during training rather than standardize rewards across all methods. Since SWIFT’s reward formulation is token-weighted, this plot reflects how the optimization differs in practice. However, we understand your concern and have generated a new figure where both methods use the same reward formula. Due to the rebuttal format limitations, we will include this new figure in the final version.\n\n---\n**Weakness3. Limited Backbone Models**\n\nA3: We acknowledge that using student models with ~2B parameters may not fully reflect the scalability of our method. Therefore, we ran additional experiments on recent, stronger base models. These results in table below confirm that SWIFT continues to deliver meaningful performance gains even with newer and stronger instruction and reasoning models.\n|Setting|Model|Arc|Truthful|Wino|GSM8k|MMLU|HellaSwag|Avg|\n|-|-|-|-|-|-|-|-|-|\n||Teacher|71.08|59.29|76.95|80.97|81.53|83.96|75.63|\n|Qwen3-32B→Mistral-7B-v0.1|Base|59.90|42.65|77.43|31.99|60.60|83.47|59.34|\n||DPO|60.70|42.11|76.59|32.65|61.30|**83.92**|59.55|\n||SPIN|61.73|44.63|77.07|33.14|62.76|81.68|60.17|\n||SWIFT|**63.06**|**51.25**|**78.20**|**42.20**|**65.17**|81.13|**63.50**|\n|Qwen3-32B→Qwen2.5-7B-Instruct|Base|65.96|64.70|75.06|68.61|**73.49**|81.45|71.55|\n||DPO|66.12|63.50|74.19|67.49|73.02|81.67|71.00|\n||SPIN|66.53|65.76|75.17|69.35|72.38|82.30|71.92|\n||SWIFT|**66.98**|**66.21**|**75.53**|**73.95**|73.36|**82.45**|**73.08**|\n\nWe understand that, as in typical knowledge distillation settings, scaling up student models naturally increases dependency on suitably powerful teachers. We recognize this limitation and plan to explore in future work.\n\n---\n**Question1. Compute overhead**\n\nA1: We report below the GPU-hours required for each phase of the SWIFT pipeline, measured on the 50k Ultrachat subset using a single NVIDIA H100 GPU.\n|Iteration|Generation|Compute Weight|Training|Teacher Forward|Overall|\n|-|-|-|-|-|-|\n|0|0.24h|0.58h|1.45h|0.19h|2.27h|\n|1|0.23h|0.57h|1.42h|0.19h|2.22h|\n|2|0.24h|0.58h|1.43h|0.19h|2.25h|\n|3|0.22h|0.55h|1.40h|0.18h|2.17h|\n\nThe teacher forward pass only takes ~ 0.19 hours ~ 11 minutes for 50k samples, which we believe is quite efficient and does not contribute significantly to the overall training cost.\n\nWe would like to note that our distillation codebase is based on DSKD, which uses a different training library than our SWIFT codebase, making direct comparisons challenging. However, we made every effort to fairly compare the **training time per sample** and **peak GPU memory usage** across SFT, ULD, DSKD, and SWIFT in the table below:\n|Method|TrainingTime/Sample(s)|Peak GPU Memory(GB)|\n|-|-|-|\n|SFT|0.26|51.08|\n|ULD|0.60|70.20|\n|DSKD|0.54|66.63|\n|SWIFT|0.30|55.39|\n\nThese results show that SWIFT is comparable to SFT in runtime and memory usage, while more efficient than typical distillation baselines.\n\nFinally, we appreciate the reviewer’s suggestion to consider pruning as an alternative. While pruning can yield compact models, it often involves additional hyperparameter tuning and retraining overhead. However, we see it as a valuable idea and plan to consider it in future work.\n\n---\n**Question2. Scaling iteration potential**\n\nA2: We believe most of SWIFT’s gains are in the first iteration because, at this stage, the student model is still weak and thus has more to learn. After that, the student already improves, so there’s less to fix, and learning slows down. Moreover, since SWIFT uses the student’s previous iteration to generate new responses. As the student gets better, it generates fewer mistakes and thus fewer learning opportunities.\n\nWe acknowledge the limitation in scaling potential of SWIFT. However, while continued improvements slow over iterations, we see this fast convergence as an advantage—SWIFT achieves strong gains in few rounds, reducing training time.\n\n---\n**Question3. Teacher quality dependence**\n\nA3: To better understand SWIFT's sensitivity to teacher quality, we ran more experiments using three different teacher models. We then reported the student model performance after training with each one:\n||ARC|Truthful|Wino|GSM8k|MMLU|HellaSwag|Avg|\n|-|-|-|-|-|-|-|-|\n|zephyr-7b-sft-full|60.41|43.73|74.19|26.76|60.92|82.85|58.14|\n|zephyr-7b-sft-full→Qwen1.5-1.8B|39.78|39.12|61.48|37.93|44.84|61.63|47.46|\n|Qwen2.5-7B-Instruct|65.96|64.70|75.06|68.61|73.49|81.45|71.54|\n|Qwen2.5-7B-Instruct→Qwen1.5-1.8B|39.91|40.11|62.43|37.56|44.50|62.43|47.82|\n|Qwen3-32B|71.08|59.29|76.95|80.97|81.53|83.96|75.63|\n|Qwen3-32B→Qwen1.5-1.8B|40.27|38.96|63.51|38.83|44.69|62.19|48.07|\n\nThese results show that while stronger teachers yield slightly better student performance, the gains are not strongly correlated with teacher capability. This is because the teacher is only used to estimate the importance weight of token, not to directly control what the model generates. As a result, SWIFT maintains stable and effective performance across diverse teacher–student pairs and does not depend heavily on highly capable teachers.\n\n---\n**Question4. Qualitative analysis**\n\nA4: Following your suggestion, we conducted a qualitative analysis using the MMLU benchmark, which covers diverse knowledge domains. MMLU comprises four major sub-domains: Humanities, Social Sciences, STEM, and Other. The results are shown in the table below:\n\n|Domain|base|Ite 0|Ite 1|Ite 2|Ite 3|\n|-|-|-|-|-|-|\n|**Humanities**|39.85|**40.89**|40.13|40.70|40.78|\n|**SocialSci.**|47.43|48.29|47.90|48.88|**49.59**|\n|**STEM**|34.61|36.76|37.08|37.93|**38.12**|\n|**Other**|50.08|52.08|52.11|51.75|**52.14**|\n\nThe results show clear and consistent improvements, especially in STEM and Social Sciences, which suggests the model is getting better at reasoning and logical thinking. Meanwhile, performance in Humanities and Other domains remains stable or improved, showing that the model’s reasoning gains didn’t hurt its general knowledge or overall quality.\n\n---\n**Limitation. Practical use of the method**\n\nA: We appreciate the opportunity to clarify the practical workflow. In practice, SWIFT can be implemented efficiently by first running the teacher model in a single offline pass over the training data to estimate token-level importance weights. These weights are then saved with the dataset and simply loaded during training. Moreover, since each sample is independent, this step can be fully parallelized to significantly speed up computation. Notably, in the timing reported in our response to Question1, we only used sequential inference for simplicity; parallel inference would further reduce this time substantially.\n\nWe agree that this approach assumes white-box access to the teacher, which may not hold in all settings. However, many recent studies indicate that white-box distillation often outperforms black-box. Moreover, the growing of powerful open-source models makes white-box distillation more feasible and desirable than ever. These trends motivate SWIFT to leverage the strengths of strong open teachers."}, {"review_id": "1MGwlf9BrM", "reviewer": "Reviewer_ti82", "summary": "To address the issues of the Self-Play Fine-Tuning, authors want to put different weight into each token. This paper proposes self-play weighted fine-tuning to incorporate token-level importance weights from a stronger teacher model. Considering the mismatches of tokenizer between the teacher model and the student model, authors introduce a practical solution in two different cases. Comprehensive experiments shows improvement compared to DPO-related alignment methods.", "questions": "1. In the proof of Lemma 3.1 in the Appendix, could you provide further clarification on why the existing proof is sufficient to establish the necessary and sufficient condition? Specifically, what steps or assumptions are being made that allow for this conclusion?\n2. Could you elaborate on the relationship between the proposed approach and the theorems presented? A more explicit expression or explanation would help to clarify how these components interact and support the overall method.\n3. While the experiments demonstrate improvements on small language models, it would be valuable to investigate the performance of the proposed approach on larger models. How does the method scale, and are there any limitations or challenges that arise when applying it to more complex models?\n4. What is the computational efficiency of the proposed method compared to existing baselines? Are there any trade-offs between accuracy and computational resources, and if so, how do they impact the practicality of the approach?", "limitations": "No", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n1. The motivation behind assigning different weights to individual tokens is straightforward and intuitive.\n2. In addition to the proposed approaches, the authors provide theoretical guidance and consider real-world challenges, such as mismatches between tokenizers. They also present ablation experiments that exclude distillation from the teacher models, offering further insights into their method.\n3. The baselines used in the study are comprehensive, allowing for a thorough validation of the model's performance.\n\nCons:\n1. The writing is unclear in certain sections, leading to confusion and making it difficult to follow the authors' arguments.\n2. A limitation of the experimental setup is that the student model used is relatively small, with approximately 1 billion parameters, which may not accurately represent more complex models.\n3. The proposed method heavily relies on previous work (self-play fine-tuning), which somewhat limits its technical contribution and novelty.", "was_revised": true, "final_justification": "In the rebuttal phase, new experiments addressed the limitation of the initial paper, supporting their claim on effectiveness. I suggest accepting the paper.", "rebuttal": "Dear reviewer **ti82**, thank you for your thoughtful review and for recognizing our work. Below, we address your questions and concerns in detail.\n\n---\n**Response to Weakness1.**\n\n> 1. The writing is unclear in certain sections, leading to confusion and making it difficult to follow the authors' arguments.\n\n**A1:** Thank you for the feedback. We will carefully revise all the sections to clarify our arguments and improve overall readability.\n\n---\n**Response to Weakness2.**\n\n> 2. A limitation of the experimental setup is that the student model used is relatively small, with approximately 1 billion parameters, which may not accurately represent more complex models.\n\n**A2:** We agree that using a ~2B parameter model may not fully reflect how SWIFT performs at larger scales. To directly address this, we conducted additional experiments on recent, stronger base models — specifically, **Mistral-7B-v0.1** and **Qwen2.5-7B-Instruct** — using **Qwen3-32B** as the teacher. The results are reported below:\n\n|Setting|Model|Arc|Truthful|Wino|GSM8k|MMLU|HellaSwag|Avg|\n|-|-|-|-|-|-|-|-|-|\n||Teacher|71.08|59.29|76.95|80.97|81.53|83.96|75.63|\n|Qwen3-32B→Mistral-7B-v0.1|Base|59.90|42.65|77.43|31.99|60.60|83.47|59.34|\n||DPO|60.70|42.11|76.59|32.65|61.30|**83.92**|59.55|\n||SPIN|61.73|44.63|77.07|33.14|62.76|81.68|60.17|\n||SWIFT|**63.06**|**51.25**|**78.20**|**42.20**|**65.17**|81.13|**63.50**|\n|Qwen3-32B→Qwen2.5-7B-Instruct|Base|65.96|64.70|75.06|68.61|**73.49**|81.45|71.55|\n||DPO|66.12|63.50|74.19|67.49|73.02|81.67|71.00|\n||SPIN|66.53|65.76|75.17|69.35|72.38|82.30|71.92|\n||SWIFT|**66.98**|**66.21**|**75.53**|**73.95**|73.36|**82.45**|**73.08**|\n\nThese results confirm that SWIFT consistently improves performance, even when applied to larger and more capable models.\n\n---\n\n**Response to Weakness3.**\n\n> 3. The proposed method heavily relies on previous work (self-play fine-tuning), which somewhat limits its technical contribution and novelty.\n\n**A3:** While our method builds upon the self-play fine-tuning framework (SPIN), it introduces key improvements that address important limitations in prior work.\n\nSpecifically, SPIN applies uniform learning signals across all tokens, assuming that all tokens in a response are equally important. However, in real-world scenarios, ground-truth and generated responses often contain a mix of high- and low-quality tokens. Treating all tokens equally can amplify noise or misleading patterns during training. Our method addresses this by introducing token-level importance weights, estimated from a stronger teacher model, to guide learning more precisely.\n\nBeyond this practical improvement, we also provide a novel theoretical formulation of token-level self-play, along with complete proofs that ground our method in a rigorous framework. Additionally, we propose a robust and general solution for handling tokenizer mismatches between teacher and student models — a practical challenge often overlooked in knowledge distillation.\n\nTogether, we believe these contributions represent a meaningful and practical advance over prior self-play fine-tuning methods, and offer both theoretical insights and empirical improvements.\n\n---\n\n**Response to Question1.**\n\n> 1. In the proof of Lemma 3.1 in the Appendix, could you provide further clarification on why the existing proof is sufficient to establish the necessary and sufficient condition? Specifically, what steps or assumptions are being made that allow for this conclusion?\n\n**A1:** We are pleased to elaborate on the reasoning behind Lemma 3.1 and clarify why the proof is sufficient to establish the necessary and sufficient condition. \n\nWe do not need to make any assumption for the proof of Lemma 3.1.\nLet us denote $A: P_d = P_{\\theta^S}$ and $B: p_d(y_t \\mid x, y_{<t}) = p_{\\theta^S}(y_t \\mid x, y_{<t}), ∀t$.\n\nOur proof in Lemma 3.1 proves $A \\implies B$ by using induction. This means that $B$ is a necessary condition of $A$. We now provide the proof of $B \\implies A$. We have\n$$p_{d}\\left(x,y\\right)=p_{d}(x)\\prod_{i=1}^{T}p_{d}(y_{i}\\mid x,y_{<i})$$\n$$p_{\\theta^{S}}\\left(x,y\\right)=p_d(x)\\prod_{i=1}^{T}p_{\\theta^{S}}(y_{i}\\mid x,y_{<i})$$\nTherefore, we reach $p_d(x,y) = p_{\\theta^{S}}\\left(x,y\\right)$ or $P_d = P_{\\theta^S}$. This implies that $B$ is a sufficient condition of $A$.\n\n---\n\n**Response to Question2.**\n\n> 2. Could you elaborate on the relationship between the proposed approach and the theorems presented? A more explicit expression or explanation would help to clarify how these components interact and support the overall method.\n\n**A2:** We appreciate the opportunity to clarify how our theoretical framework supports the overall method and connects the key components of the proposed approach.\n\nFirst, our Lemma 3.1 indicates the connection between the distribution matching and token level distribution matching. This leads to the IPM (Integral Probabilistic Metric) formulas in Eq. (2). However, the reward functions engage in the IPM formulas in Eq. (2) are the token level reward functions $r_t([x,y_{<t}],y_t)$.\n\nWe then propose the definition of consistency in Definition 3.2 to unify the token level reward functions to a unique setence level reward function $r(x,y_{\\leq t})$. Furthermore, Theorem 3.3 allows us to equivalently transform the IPM form for the distribution matching regarding the sentence level reward function in Eq. (3). However, this follows the teacher-forcing paradigm. Therefore, in Theorem 3.4, we equivalently turn it to the student-forcing form in Eq. (5) or Eq. (6).\n\nWe now connect the token level (sentence level) distribution matching viewpoint and the reinforcement viewpoint through the advantage functions in Eq. (7). Please read Lines from 167 to 173 in the main paper for the discussion of the connection indicating that Eq. (7) encourages the token level distribution matching, hence the sentence level distribution matching.\n\nWe then develop Lemma 3.5 to see the connection between the optimal policy of Eq. (7) and the Q-value function (i.e., this is relevant to the reward function). Therefore, we can implicitly consider the reward function as an implicit function of the policy which is reflected in Eqs. (10) and (11). Fortunately, we can turn this connection to Eq. (12) in Lemma 3.6, leading to the trainable algorithm in Theorem 3.7 by refering to the optimization problem in (6).   \n\nTo summarize, our theoretical development connects the distribution matching viewpoint in Self-Play and the RL-based viewpoint through advantage functions which we believe opens doors for other ideas and developments.\n\n---\n\n**Response to Question3.**\n\n> 3. While the experiments demonstrate improvements on small language models, it would be valuable to investigate the performance of the proposed approach on larger models. How does the method scale, and are there any limitations or challenges that arise when applying it to more complex models?\n\n**A3:** Please refer to our response to **Weakness2** for results on larger models. We find that SWIFT continues to provide clear performance gains even as the student model size increases. However, as with most knowledge distillation methods, using larger students often requires a strong enough teacher to guide the learning process. We recognize this limitation and plan to explore techniques for reducing teacher dependency in future work.\n\n---\n\n\n**Response to Question4.**\n\n> 4. What is the computational efficiency of the proposed method compared to existing baselines? Are there any trade-offs between accuracy and computational resources, and if so, how do they impact the practicality of the approach?\n\n**A3:** The computational efficiency we mentioned in our paper is: In practice, unlike typical white-box knowledge distillation methods that require online access to the teacher model during training, which increases training time and memory usage, SWIFT performs a single offline forward pass of the teacher model over the training data to compute token-level importance weights. These weights are then saved and reused during training, eliminating the need to keep the teacher in memory. Furthermore, since each sample is independent, this step can be fully parallelized to significantly speed up computation.\n\nTo further quantify this, we compare **training time per sample** and **peak GPU memory usage** across SFT, ULD [2], DSKD [1], and SWIFT. While we would like to note that our distillation codebase is based on DSKD [1], which uses a different training library than our SWIFT codebase, making direct comparisons challenging. However, we made every effort to ensure a fair comparison:\n\n| Method | Training Time / Sample (s) | Peak GPU Memory (GB) |\n|-|-|-|\n|SFT|0.26|51.08|\n|ULD|0.60|70.20|\n|DSKD|0.54|66.63|\n|SWIFT|0.30|55.39| \n\nThese results show that SWIFT is comparable to SFT in runtime and memory usage, while more efficient than typical white-box distillation baselines.\n\nBecause the teacher is only used once and the token weights are reused throughout training, we believe there is no trade-off between accuracy and computational resources in our approach.\n\nOnce again, thank you for your comment and suggestion. If you have any additional comments or concerns, please feel free to share them with us.\n\n---\n\n**References:**\n\n[1] Zhang, Songming, et al. \"Dual-space knowledge distillation for large language models.\" arXiv preprint arXiv:2406.17328 (2024).\n\n[2] Boizard, Nicolas, et al. \"Towards cross-tokenizer distillation: the universal logit distillation loss for llms.\" arXiv preprint arXiv:2402.12030 (2024)."}]

Author rebuttal:

Meta-review: (a) Summary:
We propose SWIFT (Self-Play Weighted Fine-Tuning), a token-level extension of self-play methods for aligning large language models (LLMs). Unlike prior approaches that apply uniform rewards across tokens, SWIFT leverages a stronger teacher model to estimate per-token importance, allowing useful supervision even from partially rejected responses. Building on SPIN, SWIFT introduces a token-weighted objective and a practical tokenizer-mapping strategy for student–teacher mismatch. Experiments in both preference alignment and knowledge distillation show consistent gains over DPO- and KD-style baselines across six benchmarks. Ablations further highlight the advantage of teacher-guided token weighting over uniform or heuristic alternatives.


(b) Strengths:

1. The paper provides a solid theoretical foundation and proposes a practical method.

2. The baselines and ablation study in the experiments are comprehensive.

(c) Weaknesses (after rebuttal):
1. Some writing issues exist and the presentation flow is a little disconnected.

2. Introducing a teacher model can add computational cost.

(d) Why this decision:
The paper is on the borderline. The rebuttal has done a good job by addressing reviewers' concerns in the first round of review. All reviewers vote for borderline accept consistently.

(e) Summary of discussions:
See above
Decision: Accept (poster)