Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs
Authors: Yibo Wang
Hai-Long Sun
Guangda Huzhang
Qing-Guo Chen
Zhao Xu
Weihua Luo
Kaifu Zhang
Lijun Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Model
Self-play Optimization
Natural Language Processing
Primary area: deep_learning

Abstract: Recently, self-play fine-tuning (SPIN) has been proposed to adapt large language models to downstream applications with scarce expert-annotated data, by iteratively generating synthetic responses from the model itself. However, SPIN is designed to optimize the current reward advantages of annotated responses over synthetic responses at hand, which may gradually vanish during iterations, leading to \textit{unstable optimization}. Moreover, the utilization of reference policy induces a \textit{misalignment} issue between the reward formulation for training and the metric for generation. To address these limitations, we propose a novel \textbf{T}riplet-based \textbf{S}elf-\textbf{P}lay f\textbf{I}ne-tu\textbf{N}ing (TSPIN) method that integrates two key designs. First, beyond current advantages, TSPIN additionally incorporates historical advantages between iteratively generated responses and proto-synthetic responses produced by the initial policy. Even if the current advantages diminish, historical advantages remain effective, stabilizing the overall optimization. Second, TSPIN introduces the entropy constraint into the self-play framework, which is theoretically justified to support reference-free fine-tuning, eliminating the training-generation discrepancy. Empirical results on various tasks demonstrate not only the superior performance of TSPIN over SPIN, but also its stable evolution during iterations. Remarkably, compared to supervised fine-tuning, TSPIN achieves comparable or even better performance with only $25\\%$ samples, highlighting its effectiveness when faced with scarce annotated data.

Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs
Yibo Wang Hai-Long Sun 12.., Guangda Huzhang, Qing-Guo Chen',
Zhao Xu', Weihua Luo', Kaifu Zhang', Lijun Zhang 142f
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China "chooo of Artificial Intelligence, Nanjing University, Nanjing, China
3 Alibaba International Digital Commerce *Pazhou Laboratory (Huangpu), Guangzhou, China {wangyb, sunhl, [hanglj)@lamde edu. edu. cn
{qingguo cqg, changgong. xz, weihua. luowh, kaifu. zkf Gal waaiibbbainn....
Abstract
Recently, self-play fine-tuning (SPIN) has been proposed to adapt large language models to downstream applications with scarce expert-annotated data, by iteratively generating synthetic responses from the model itself. However, SPIN is designed to optimize the current reward advantages of annotated responses over synthetic re- sponses at hand, which may gradually vanish during iterations, leading to umstable optimization. Moreover, the utilization reference policy induces a misalignment issue between the reward formulation for training and the metric for generation. To address these limitations, we propose novel Triplet-based Self-Play fTne-tuNing (T-SPIN) method that integrates two key designs. First, beyond current advantages, T-SPIN additionally incorporates historical advantages between iteratively gener- ated responses and proto-synthetic responses produced by the initial policy. Even if the current advantages diminish, historical advantages remain effective, stabilizing the overall optimization. Second, T-SPIN introduces the entropy constraint into the self-play framework, which theoretically justified to support reference- eliminating the rrining-generainn various tasks demonstrate not only the superior performance of T-SPIN over SPIN, but also its stable evolution during iterations. Remarkably, compared supervised fine-tuning, T-SPIN achieves comparable or even better performance with only 25% samples, highlighting its effectiveness when faced with scarce annotated data.
discrepancy. Empirical results
Introduction
Large language models (LLMM) have demonstrated powerful generalization capabilities and profound
impact across various fields [Ouyang et al., 2022 Huang and Chang [223]] Qin al.
2023 Bai
ct al. 2023 Achiam et al. 2023 Liu et al. 2023 Jiang et al. 2024a Wu et al. 2024a]] Zhao et al.
2024 Chen al. 2024a|
al. 2024 Hui al.) 2024 Guo et al. 2024a Jiang al. 2024b
Anthropic 2024a Google 2024 OpenAl 2024 Sun et al.) 2025a b Lu et al. 2025 Jiag et al. 2025
The remarkable success of LLMs is inherently linked to the utilization of vast expert annotations.
Nevertheless, considerable curation costs for high-quality data pose major obstacles to downstream-
task adaptation (Wang et al. 2024a] Tan al.) (024). Consequently, numerous efforts have been
devoted to addressing this challenge, such as data augmentation Dai et al. 2023 Zhu et al. 2023
Guo et al. 2024b) and model distillation [Gunasekar et al. 2023 Luo et 2024
Xin al. 2024
Work done during the internship at Alibaba International Digital Commerce.
1 Lijun Zhang is the corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Supervised Fine-Tuning
(b) Self-Play Fine-Tuning
(c)
Triplet-based Self-Play Fine-Tuning
update
generale
Initial Policy No,
Expert-Annotated Data
Annotated Data
Current Palicy K
Proto Proto-Synthetic
update
update
update
Current Policy
reference
duplicate
Annotated Data
update
duplicate
Current Policy Re
Synthetic Data
generate
Previous Policy *6. Synthetiic Data
generate
Previous Palicy FA
Figure 1: Comparisons of three strategies: (a) supervised fine-tuning requires large amounts of annotated data to train To; (b) self-play fine-tuning operates with limited annotated data and iteratively generated samples, and employs the previous policy To, as a reference for updates; (c) triplet-based self-play fine-tuning employs triplet inputs, i.e., annotated data, synthetic samples, and proto-synthetic ones from the initial policy TBo> and updates Ty without auxiliaries from any reference policies.
Recently, a promising research direction known self-play fine-tuning has been introduced (Chen it al. 2024bb] and has since been applied to various LLM fields (Yuan et al. 2024a| Wu et al.. 2024b Gao et al., 20224]] The core idea of self-play fine-tuning is to progressively refine the LLM by competing against itself. Formally, it can be framed as a two-player game between a main player and an opponent player, each of which corresponds to an instance policy of the same LLM. this game, the main player aims to distinguish expert- sspert-annotated responses from generated ones, while the opponent player strives to produce high-quality synthetic responses that can deceive the main player.
Based on this framework, Chen et al. (2024b) introduce the seminal method named SPIN, which is designed to optimize the relative reward advantages of annotated responses over generated ones, and employs the policy from previous iteration as a reference for reward computations. However, with the evolution of LLM, the relative advantage between two types of data gradually diminishes over time. When the relative advantage is eliminated, the optimization objective of SPIN degenerates into constant independent of the policy, allowing any policy to become optimal solution and thus leading to unstable optimization. Morcover, the incorporation of reference policy in SPIN causes a misaligument between the reward formulation used for training and the log likelihood that guides response generation. Empirical evidences in Figure 3 reveal that although annotated responses yield higher rewards than generated responses, their log likelihoods become lower than synthetic samples during iterations. In other words, higher rewards do not necessarily lead to higher generation priority.
To address these limitations, we propose new self-play fine-tuning method, termed Triplet- bbaeed Self-Play flnn-tuNiin (T-SPIN). The basic intuition is to incorporate historical advantages for stable evolution, and eliminate the reference policy for introduce two key modifications to the self-play framework. First, the main player is required not only to distinguish between real and synthetic data, but also discriminate produced by the initial policy. By incorporating triplct inputs of real, synthetic and proto-synthetic samples, T-SPIN optimizes current and historical advantages simultancously. Even if the current advantage diminishes (i.e., synthetic samples become indistinguishable from annotated ones), the historical advantages between synthetic responses and proto-synthete responses still exist, ensuring a stable optimization over the policy model. Second, the opponent player maximizes the confidence of synthetic responses with an entropy constraint, where the confidence measures how likely the main player believes a response comes from the annotated data. Through theoretical analysis, we demonstrate that the incorporation of the entropy constraint removes the need for reference policy, aligning the principles of training and generation. Empirical results show that T-SPIN IN consistently maintains higher log- likelihoooss for annotated data compared to synthetic samples.
alignments. Specifically, we
samples
We conduct experiments on Zephyr-7B (Tunstall et al., 2023) and Mistral-7B (Jiang et al.. 2023), and select 50k annotated samples from Ultrachat200k (Ding et al., 2023) as the training set. The performance is evaluated across diverse tasks to assess multi-dimensional capabilities of LLMs, in- cluding math and logic, multi-domain knowledge, commonsense reasoning, and Experimental results demonstrate that T-SPIN significantly improves performance, particularly in math and instruction-following and achieves superior and stable performances during iterations compared to SPIN. Furthermore, we also show that T-SPIN, using only 25% annotated samples, can achieve comparable performance (or even better on certain tasks) to supervised fine-tuning with the full training set, highlighting the effectiveness of T-SPIN in addressing the scarcity of annotated data.

---

2
Preliminary
In this section, we provide a brief overview of three fine-tuning strategies (i.e., supervised fine-tuning self-play fine-tuning, and our T-SPIN) for LLMs. The comparisons are also illustrated in Figure
2.1 Supervised fine-tuning
A common strategy adapt a pretrained model to downstream tasks is fine-tuning it high-quality annotated data, Specifically, given a prompt x and its corresponding annotated response y, supervised fine-tuning (SFT) aims to align the model with (x, y). Formally, the goal of SFT is to learn a policy TSFT by maximizing the aggliikelhood of high-quality annotated data:
SFT argmax Ex ba x [log TT (y|x)],
(I)
where q(-) and dotaa (-Ix) denote the distributions of prompts and annotated responses, respectively.
Mathematically, ( is equivalent to the solution of minimizing the KI. divergence between the target
distribution daata xx) and the policy To (-1x))
argmin 1xx)(()) [I [x)]|=(((|x))
STT
It can be verified that the solution exactly matches the target distribution of annotated responses,
ie.. TsFr(-|x)= = (- Unfortunately, achieving the optimal policy typically demands extensive high-quality samples, resulting in high costs for data curation and annotation, and thereby limiting
the practical applicability of supervised fine-tuning [Wang al. 2024a Tan et al. 2024].
2.2 Self-play fine-tuning
Self-play fine-tuning has emerged as a promising way to address the scarcity of expert-annotated data when adapting LLMs downstream tasks [Chen al. 2024b Yuan et al. 2024a] Different from SFT that runs over a fixed annotated dataset, self-play fine-tuning employs both annotated samples and self-generated responses inputs, with the latter continuously refined as the model evolves.
Specifically, at each iteration + 1, the model first generates a synthetic response y' for given prompt by sampling from the policy Ta, (-1x). Then, the model utilizes the prompt and pairwise samples (i.e., the annotated response y and the synthetic response y') to obtain the policy Te...- Let the reward for response be formulated by the log ratio of the likelihood between the current policy my and the previous policy To, (Rafailov et al. 2023). Then, the objective of SPIN (Chen et al., 2024b) is maximize the gap (referred as the relative advantage) between the reward of annotated response and that of synthetic response y':
To argmin E e (A (ylx) To, (y|x)
Alog Ta (y' x)
no, (y' xx) )
where À >>0 0 denotes tlyer-pprameete ((-) is convex and monotonically decreasing function, and the expectation operation taken over the distributions x ~ q(-), y ~ Tdata and y' ~ To, (-xx). In the following, and misaligmment between training and generation.
elaborate two major limitations of SPIN: unstable optimization during iterations
Unstable optimization. The instability of SPIN arises from the inherent structure of the objective in (3). To be precise, we consider the case where the synthetic response y' becomes identical to the annotated response y (i.e., y' y). In this case, the advantage of over y' vanishes, causing the objective 3 to degenerate into a constant that is independent of the policy π.. As a result, any policy Ty becomes optimal to leading unstable optimization even performance degradation.
Misalignment between training and generation. Given the reference policy To, from the iteration t, the objective (3) is designed to optimize the reward gap between two types of responses, where the reward for a response ý is defined as r(x, ý) >llog my (5())) - A log ne, ())x). However, is important to note that the formulation of r (x, ý) differs from the principle used for generation, other words, due to the incorporation of reference policy тT, a higher reward, i.e. r(x,y) > r(x, z) for two responses and z, does not necessarily imply higher generation probability, i.e., log No (y(x)) > log (2)x) may not hold. This demonstration has been empirically verified by our experiments, as illustrated in Figure 3
i..., log wo (ý [x).
3

---

3 Triplet-based self-play fine-tuning
In this section, we introduce our triplet-based self-play fine-tuning (T-SPIN), stable and reference- free method for fine-tuning LLMs. Building upon the self-play framework, our T-SPIN involves two players: a main player and an opponent player, both of which are different policy instances from the same LLM. For clarity, we describe the update rules for two players separately.
The update rule for main player. Recall that the goal of the main player is to distinguish among the annotated response y. the synthetie response y' generated from the previous policy Te, and the proto-synthetic response Yo produced by the initial policy Te. Conceptually, the triple responses expected to satisfy the relation > y' > Yo. where the operator ł indicates that the main player has greater preference for one response than another. In other words, Yo serves as an anchor that helps to evaluate how far the current policy Te, has progressed. This design is partially inspired by the inteema-quullbration process in cognitive development, where knowledge builds upon and refines pre-existing cognitive anchors [Pagget 1952]. Formally, we are motivated by the integral probability metric (IPM) Muüller 1997] establish the objective of the main player. In detail, let c(x, ý) denote the confidence with which the main player believes a response is sampled from the real-world data distribution datua (-Ix). Then, the goal of main player is to find confidence function
C++1 argmin E [8 (c(x, y) y'x, 38 (c(x,y') c(x)
cEC
where / 2 0 denotes a trade-off hyper-parameter, and C denotes function class that will be specified later, and E(-) denotes a monotonically decreasing function, e.g., the logistic function ((z) log a (zr) and the exponential function ((x) exp(- The expectation operation here is taken over the distributions x ~ g(.),y ~ Tdata (aaata-(xx,, (-Ix), ~ TT. (-Ix) and yo ~ (-,x)))) Given the triple responses y. y', and yo. the objective in (4) consists of two terms:
The first term e (c(x, y) c(x.y')) measures the gap between the real response y F-daral(x) and the synthetic one y' ~ Te. (-1x). reflecting the relative advantages of Tdata over TT,
The second term é (c(x, y') - c(x, Vo)) compares the synthetic response y' to the proto-synthetie response Yo ~ To, xx). capturing historical ad advantages of To, over To, during iterations.
During the self-play process, the opponent player evolves over time, generating synthetic responses y' that progressively approximate the real response y. As result, the current advantage, i.e., the first contrast, since yo remains fixed during iterations, the historical advantage, i.e., the second term in (4) remains meaningful even when the current advantage vanishes, ensuring that (4) does not degenerate to a constant independent of Tg.
term in (4), gradually diminishes.
The update rule for opponent player. In this part, we suppose that the main player has learned a confidence function C++11 capable of effectively distinguishing between y and y'. Then, we proceed to elaborate on the update rule for the opponent player. Note that the goal of the opponent player to generate response y' that misleads the main player to the best of its ability. Therefore, with the given confidence function Ct+1. the opponent player aims to find policy To that maximizes the expected confidence score over synthetic responses y' generated from itself:
T argmax Ex ( [x y')] + allxx=g(-) [H(
where >> 0 denotes the regularization parameter, and H(-) denotes the entropy constraint. In ( the first part reflects the confidence that how the main player believes that y' is sampled from the real-world distribution daata. and the second part serves to stabilize the optimization. It is worth noting that (5) has closed-form solution, as stated below.
Proposition 1. Given a general confidence function Ct+1, the optimal policy of the opponent player in ( takes the form of
expleci(xxy")/aa
<'(l(x) ExEyexp(cam (x, 2)/a)
where y denotes the set that contains all possible responses for the prompt x.
Note that due to the generality of ct-1, the closed-form solution x² (y'lx) may not be attainable in the policy space of L.M. To resolve this issue, we need to choose a specific form of the function class C such that the closed-form solution becomes tractable. Specifically, we hope to find a policy

---

Algorithm/ I Triplet-based Self-Play Fine-Tuning (T-SPIN)
Inputs: A set of high-quality data (x,, J pretrained LLM Re and the number of iterations T 1: Produce the proto-synthetic response yo with the policy Th, for each prompt x.
2: for /. = 1,..... T - do
3: For each prompt x, generate one synthetic response y' by the policy Te.
4: Update the policy T0.41 by minimizing (7) with prompt and triplet responses y',y',y!!
5:
end for
Ty that satisfies Tg(y'|x) n² (y'lx) with the confidence ct+1 learned from certain class C. To this end, according to Proposition]] I we can obtain To (x) x exp exp((e+1 (x, y')/a), which implies (x. log Th (ely' xx) In other words, we can choose the function class
C {alogg [x]]0 E }}
(6)
where Ô denote the parameter space of LLM, and optimize 4 to obtain the confidence c++1 y') = log To (y'Ix) where the policy Ty is exactly the optimal solution of (5). Therefore, we do not need to solve (5) explicitly. Moreover, since (6) depends solely on the policy and does not rely on any reference policy, inherently avoids the training-genne generation misalignment issue suffered by SPIN
The end-to-end objective. We consolidate the two update rules into a unified end-to-end objective by substituting (6) into 44. and obtain the loss function for T-SPIN as follows:
(T-SPIn(0) =E [e (a log mo(y|x) x) - a log na a (y'(x)) + 38 (a log To (y'lx) - a log To (yolx))) (7)
The expectation is taken over distributions x ~ q(-), y ~ dataa (-Ix),y' ~ Te, (-Ix), yo Te. [ (-)x). Overall. the training of T-SPIN can be viewed as alternating updates between the main player and the opponent player. At each iteration + 1, the opponent player first generates synthetic responses y' sampled from the previous iteration policy =.,- Then, the main player takes the triplet responses y.y' and yo as inputs to obtain its confidence Ct+1 a log To.+1 After that, the opponent player chooses the policy nb for the next iteration. The detailed steps are presented in Algorithm |
Comparisons between T-SPIN and SPIN. Both methods are designed to handle the limited available data when adapting LLMs downstream tasks, but they fundamentally differ in (i) optimization principles, (ii) implicit reward formulation, and (iii) computational overhead, as detailed below:
SPIN only optimizes the current advantage between the annotated response y ~ T and the synthetic response y' ~ Te, (-1x) in (3). which may result in the instability issue during iterations. In contrast, T-SPIN optimizes not only the current advantage of y over y', but also the historical advantage of y' over the proto-synthetic response yo To, (-1x), ensuring stable evolution and thereby mitigating the unstable optimization;
In SPIN, the reward function for a response z is formulated as rh (z(2))) = A log Tg (z/x) À log ne, (z)x). This formulation misaligns with the generation metric (i.e., the log-likelihood), resulting in potential suboptimal performance. By contrast, the reward function in T-SPIN (also referred to as the confidence function under the self-play framework) is defined re (z(xx)
a log no [z/x,, which matches the generation metric and thus resolves the misalignment issue;
Compared SPIN, T-SPIN incurs only a marginal increase in computational and memory costs. Specifically, T-SPIN performs one-time generation of yo at the beginning. which is not repeated in subsequent iterations. In practice, the costs of this generation is negligible compared to the iterative training process. Moreover, while T-SPIN expands each training instance from a pair of yy, to a triplet of (y, y'),, the overall memory costs remains nearly unchanged, as it eliminates the need for reference model, which is necessary in SPIN. Consequently, the additional overhead of T-SPIN is minimal compared to SPIN.
Gradient analysis. To mechanistically understand the impact of T-SPIN on the training process, we examine the gradient of 7 with respect to the parameter 0, as presented in the following theorem.
Theorem Define ul TT) = log no (ylx) - log (o (x) and v(T)) log no (y'lx) - log m (yo x).
Then, given a monotonically decreasing function ((r), the gradient of 7 with respect to f is
VaLr-spin()) = = e' (au(TT)) (V, log [x(lx) V log (y'((xx)))
(8)
+ 38 (Ar(TT)) (Vo log (y'lx) V lolog [x)]
5

---

Table 1: Performance (%) comparisons among three one-tuning methods, where SPIN and T-SPIN are trained on 50k annotated data from Ulrachat200k, and SFT is trained on the complete dataset. At the average (Avg) column, we also report improvements (or degradations) over the previous iteration. Abbreviations: MMLU-Pro + MMLUP, WinoGraae - WG, Instruction Following - IF.
Math & Logle
Multi-Domain Knowledge Commonsense Reasoning
IF
Model
Avg
GSM8K
MATH MUSR
MMLU MMLUP
CPQA
HellaSwag
WG
BBH
IFEval
Zephyr-7Bi
25.85
38.92
56.90
28.88
28.91
82.79
74.19
44.60
2.76
38.56
SFT
42.25
3.10
39.18
57.29
29.19
28.28
83.44
73.16
44.90
19.31
42.01
IterO
30.06
4.46
41.67
56.25
28.39
29.53
84.29
73.56
44.41
8.12
40.07
SPIR
Iterl
32.47
3.05
39.02
56.86
27.88
29.06
83.76
73.72
44.59
7.95
39. (-0.8)
Iter2
36.62
2.57
41.27
57.74
28.67
28.39
83.75
73.95
43.95
14..6
(1.14(41.50)
Iter3
33.32
2.75
39.41
55.88
27.98
27.61
82.64
74.19
43. 19
22.25
40.92
Iter4
35.54
2.72
38.2
53.59
25.95
26.21
83 48
74.51
43.10
22.88
40.62
IterO
36.20
3.29
37.05
56.42
28.88
28.05
82.97
73.48
44.17
7.02
39.75
T-SPIR
Iterl
40.64
3.68
37.84
56.89
29.13
28.92
82.95
73.72
44.82
26.98
(2.56
Iter2
40.69
3.63
38.91
56.89
29.26
29.40
83.37
73.64
44.48
27.67
(2.79(+0.23))
42.79
(+0.23)
40.71
39.15
57.07 67
29.37
30.12
83.30
73.74
45.27
29.15
43
(3..23
Iter4
40.67
3.84
39.72 72
57.68
29.32
30.44
83.12
73.80
45.00
31.08
43. 43.47 (3
45,0
40.0
45.0
T-SPIN
T-SPIN
T-SPIN
SPIN
SPIN
43,0
SPIN
(%) S202
Scores 13)
37.0
20.0
41.0
33,0
10.0
39.0
29.0
37.0
Inerr
iter1
iter2
iter3
iter4
ite-0
iter1
Iter2
iter3
iterd
iterO
iterl
iter2
iter3
iter4
(c) 65MBK
(b) IFEvll
(c) Average Score
Figure Performance (%) comparisons between T-SPIN and SPIN on two tasks: GSMSK and IFEval
over 5 iterations. The average scores over 10 different tasks are also illustrated in the right panel.
where (.). denotes the derivative of E(-), and the expectation is taken over the distributions x ~ y Adata (-1x, y' To, (-) Yo To, (-1x). With specific choice of ((x), can derive the gradient. Detailed derivation are provided in Appendix
more explicit forms forms of
Note that the function {(x) monotonically decreasing, leading a non-positive derivative (x)x) S 0, Vx E I.. Therefore, during the training, the gradient in (8) tends to increase the likelihood of y and decrease the likelihood of yo with weights corresponding to the current advantage u(Te) and the historical advantage (TTg). respectively. For the synthetic response y'. the update direction is influenced by both current and historical advantages, with the latter weighted by hyperparameter B.
Experiments
In this section, commence by conducting empirical evaluations of T-SPIN, SPIN, and SFT set of benchmarks with 10 different tasks. Then, we provide in-depth ablation analysis to investigate impact of components (i.e., historical advantages, reward function and hyperparameters) in T-SPIN.
4.1 Experimental setups
Models and training settings. Following the setup in Chen al. 2024b ) we randomly sample 50% prompt-response pairs from the first round of interactions in U Utrachat200k (Ding et al. 2023 as the training set, and choose Zephy--7B-SFT-Full (Tunstall et al. 2023] and Mistral-77 ---v0l Jiang et al. 2023] as pretrained LLMSs in our experiments. We employ the commonly used template: " ### Instruction: {prompt)) \nn### Response:" [aooi et. 2023) to construct the prompts, and set f(x) log (xx by the logistic loss function in the objectives of T-SPIN IN and SPIN. The

---

Rewards
Log-likellhhods
Contingency table
30k
-250
27, 77.5k
22,1k
log Ty
-350
rapiaa (y)
-10
-400
15k
-15
0
0.4k
-20
-650
log%, Ing
log
log Ryr
steps (%)
steps (%)
genenched
Training dynamics generation statistic SPIN
Rewards
Log- Log-likelihoods
Contingency table
38k
36.2k
-600
-600
K
19k
-000
log
-800
11.8k
-900
25
50
75
100
50
75
100
log log Ty
log log
steps (%)
steps (%)
real
generated
(b) Training dynamics and generation statistic of TSPIN
Figure 3: Comparisons between SPIN and T-SPIN at iteration 1: (a) Training dynamics (including rewards and log-liielihoods of y and y') and generation statistic (i.e., associations between rewards and (og-likelhhoods measured on the training for iteration 1 of SPIN; (b) Training dynamics and generation statistic of T-SPIN. For brevity, we denote r(y|x) xx) and log Ty = log π (y|x).
training process adheres to the iterative framework described in Section 3 each iteration, we first employ the latest policy to generate synthetic responses, which are then combined with annotated responses as the training data for the next iteration. total, there are 50k synthetic responses and 50k annotated responses in each iteration. For T-SPIN, additionally incorporate 50k proto-synthetic responses produced by the initial policy, into the training set.
Evaluation benchmarks. We evaluate the performance across diverse tasks, each designed to assess specific capabiliites of LLMs. These tasks include mathematical and logical reasoning with GSM8K
[Cobbe et al. [2021]. MATH [Hendrycks et al.) (0211), MUSR (Sprague et al. [202]]; multi-doman
knowledge understanding through MMLU [Hendrycks et al., 2021a MMLU-Pro (Wang al.
2024d). , GPQA [Rein et al. 2023]: commonsense reasoning by HellaSwag Zellers et al. 2019]] WinoGrande (Sakaguchi et al. [2021] and BBH [Suzgun et 2022 ; and instruction-for cottiion-folloooe lowing
via IFEval [Zhou et al. [023]. For each task, we follow the standard evaluation protocols in the literature. For instance, we utilize 3-shot examples and normalized accuracy as the metric BBH.
More detailed introductions about the evaluation benchmarks can be found in Appendix B.2
4.2 Experimental results
Comparisons to SPIN and SFT. We first compare T-SPIN with SPIN and SFT on multiple tasks, where self-play methods trained with only 50k: annotated data, while SFT is trained with 200k annotated data. Table summarizes the results on Zephyr-7B, and the results on Mistral-7B deferred the Appendix B.3 The results demonstrate that T-SPIN significantly improve the performances of pretrained models, and outperform other two fine-tuning methods. Specifically, T-SPIN achieves significant improvement
average score compared Zephyr-7B, particularly excelling in GSM8K and IFEval tasks with 14.82-point and 28. 28.32-point increases respectively, which also illustrated in Figure a and Figure b). Moreover, from the results shown in Figure ([c)), we also observe that SPIN suffers from performance instability over iterations, reaching the peak performance at the third iteration and then suffering the degradation at the subsequent iterations. In contrast, T-SPIN demonstrates stable evolutions over all iterations. Compared to SFT that utilizes the complete annotated set as training data, T-SPIN with only 50k annotated data achieves comparable (or better) performances, highlighting the effectiveness of T-SPIN when facing the data scarcity issue.

---

42.5
44.0
T-SPIN
42004
4231
44,00
T-SPIN
Scores (1)
41.
42,0
42.20
4181
42:18
42.34
42.50
4603
40.0
0.3
42.12
42.05
42.07
41,91
T-SPIN
40.5
SFT
0.1
42.41
38.0
41.00
14 26
50
100
200
iter0
iterl
iter2
iter3
iterd
0.3
0.5
Size (K)
(o) Different sizes of annotated data
(b) Ablation reward function
(c) Ablation on hyper-pormeeers
Figure 4: Performance comparisons over different settings: (a) comparison between T-SPIN and SFT with varying amounts of annotated data; (b) performances of T-SPIN versus the variant without historical advantage (w/o H-A); (c) robustness analysis with respect to hyperparameters a and B.
Aligning training rewards and generation log-likelihodss. We then show that T-SPIN is able to alleviate the misalignment issue suffered by SPIN. We plot the training dynamics and genera- tion statistics of SPIN and T-SPIN on Zephyr-7B at the iteration in Figure [a) and Figure 3)), respectively. We observe that in SPIN, annotated responses yield higher rewards than synthetic responses, but their log-likelihoods are lower than those of synthetic responses, shown in the left and middle panels of Figure a). The reason lies in that the reward function of SPIN incorporates both the current policy Ty and the reference policy Te, from the previous iteration, expressed as (x, z) = A log ( - A log TT, (z x) for a response z. This formulation differs from the generation principle that solely relies on log No (z|x). Consequently, responses with higher reward does not necessarily enjoy higher generation probability. To further illustrate this phenomenon, we also analyze the generation statistics of rewards and ligliiklisoods at the right panel of Figure The results indicate that nearly half of the samples exhibit higher rewards but lower log liggliiliihooss.. By contrast, T-SPIN effectively mitigates the mismatch issue by aligning the reward formulation for training and log-likelihood used for generation, i.e., PT-SPIN (x, z) a log xo [z x)) where a = 1 in this experiment. Empirical evidence, shown the left and middle panels of Figure demonstrates that both the rewards and log- likelihoods of real responses are consistently higher than those of generated responses. We also examine the generation statistic of two types of responses in terms of rewards and log-likelihoods, with similar trends observed at the right panel of Figure (b).
Fine-tuning with less annotated data. We now demonstrate that T-SPIN can achieve comparable or even superior performances to SFT. while using less annotated training data. To validate this claim, we conduct experiments with different sizes of annotated data. Specifically, we construct subsets of 14k, 26k, 50k, 100k, and 200k samples from Ultrachat200k (Ding et al., 2023), where each larger subset encompasses all smaller ones. For comparison, we train T-SPIN on Zephyr-7B using the smaller subsets (14k; 26k, and 50k samples) and SFT using the larger subsets (50k, 100k and 200k: samples). We report the performances of T-SPIN after 2 iterations and SFT after consecutive epochs. Figure 4a)) illustrates the empirical results, from which we can observe that: (i) both fine-tuning methods show improved performance with the increased annotated data size; (ii) with the same amount of annotated data (i.e., 50k), T-SPIN achieves superior average score of 42.56% compared to 40. 65% of SFT; (iii) T-SPIN using only 50k annotated responses attains 42.56% average score, slightly outperforming 42.01% of SFT trained with 200 annotated data.
4.3 Ablation studies
In this part, we conduct ablation studies on Zephyr-7B to investigate the impact of each component in T-SPIN.
(i.c., historical advantages, reference-free reward function, and
Historical advantages. In T-SPIN, the historical advantage, i.c., the second term in 7 is introduced to address the instability issues caused by the vanishing current advantage, i.e., the first term in (7). To verify the effectiveness of historical advantages, we conduct ablation experiments by comparing T-SPIN with a variant that removes the historical advantage from 7 referred to as w/o H-A. Specif- ically, for T-SPIN, we initialize 3 = 0 in the first iteration, as the historical advantage is identical to the current advantage at iteration O, and set 3 0.1 for all subsequent iterations. For w/o H-A, we fix B 0 over all iterations. We report the performances of T-SPIN and w/o H-A on all tasks in Table The average performances over different tasks reveal two key findings: (i) both T-SPIN and w/o H-A are able to improve the performances, although w/o H-A suffers from performance decline
8

---

Table Performance (%) comparisons between T-SPIN and w/o H-A on Zephyr-7B. At the average (Avg) column, we also report the performance improvements (or degradations) over the previous iteration. Abbreviations: MMLU-Pro + MMLUP, WinoGrade - WG, Instruction Following
Math Logic
Multi-Domain Knowledge Commonsense Reasoning
IF
Methods
Avg
GSM8K
MATH
MUSR
MMLU
MMLUP
GPQA
HellaSwag
IFEval
IterO
36.20
3.29
37.05
56.42
28.88
28.05
82.97
73.48
44.17
7.02
39.75
Iterl
29.89
3.33
37.06
57.51
28,98
27.35
82.13
73.16
44.63
10.46
39. 39.45
T-9PIN w/o 8-8
Itcr2
38.95
3.19
37.83
56.72
27.65
28.19
83.29
73.95
44.64
21.87
(1.63
Iter3
41.71
2.63
37.59
56.87
27.82
28.96
83.28
73.32
44.63
20.10
41.64:
41.64(+0.00)
Iter4
38.12
3.83
37.05
56.96
28.40
28.61
83.71
74.11
43.86
23.20
(1.79
Iterl
40.64
3.68
37.84
56.89
29.13
28 28.92
82.95
73.72
44.82
26.98
42.56
Iter2
40.69
3.63
38.91
56.89
29.26
29.40
83.37
73.64
44.48
27.67
42. 42.79 (+0.233)
40.71
3.82
39.18
57.67
29.37
30.12
83.30
73.74
45.27
29.15
43. (3.23
Iter4
40.67
3.84
39.72
57 .6.
29.32
30.44
83.12
73.80
45.00
31.08
43.47
45,0
6.0
40.0
T-SPIN
T-SPIN
T-SPIN
w/o H-A
w/o H-A
30.0
w/o H-A
Scores (%)
(30
4,0
37.0
Scores
2.0
33.0
10.0
29.0
0.0
0.0
Iter0
iterl
Iter2
iter3
iter4
iterO
iter3
iterO
Iter2
iter3
iteed
(o) 6SM6K
(b) MATH
(c) IFEvol
Figure Performance (%) comparisons between T-SPIN and w/o H-A on GSM8K, MATH and IFEval.
at iteration 1; (ii) without the historical advantage, w/o H-A improves slowly and plateaus at 41.64% iteration 3, whereas T-SPIN reaches a higher performance of 12.56% earlier, at iteration When we dig into the performance of each task, we find that w/o H-A suffers from unstable performance trends, as shown Figure For example, on the task of GSM8K, w/o H-A suffers from significant performance decline at the lerattion 1. On MATH and IFEval, the performance degradation occurs at iteration 3. contrast, T-SPIN demonstrates consistent improvements across all tasks and maintains stability when reaching the performance peak, highlighting the effecttiveness of historical advantages.
Reference-free reward function. To address the training- geneiintgngerttionnn misalignment issue in SPIN, F-SPIN introduce an entropy constraint into the self-plat -lay framework as shown in (5). from which we theoretically deliver a reference-free reward function for T-SPIN. To investigate its effectiveness. we compete our T-SPIN with a variant, namely T-SPINrat. that still optimizes with triplet inputs, but uses the reward function with reference policy. The loss function of T-SPINraf given by:
E ( TH. (y|x)
Te(y|x)
Ne (y'lx)
Te (x) [y
To (N(o Te. (yo x)
] (9)
LT-EPTH.N (0)
ao
log
+ B8 a1
log
We report the average performances of T-SPIN and T-SPINrat on all tasks in Figure 4 4b). From the results, we observe that T-SPIN demonstrates superior performance compared to T-SPIN over all iterations. This can be attributed to that using the reference-five -frencee-rreeeeee reward function preserves the alignment between training and generation, which in turn facilitates performance improvement.
Parameter robustness. T-SPIN involves two hyper-paramesd balances the current and historical advantages, and (ii) the regularization coefficient that controls the entropy penalty. To investigate the robustness of these parameters, we conduct experiments at iteration I with varying a and B. We select values for a and 3 from the set (0.1, 0.3, 0.5, 1.0), and report the average performances over all 10 tasks with different parameters in Figure (c). The results demonstrate that T-SPIN is relatively insensitive to wide range of a and B with diverse values. Accordingly,
(i) the trade-off parameter 3 that
1.0 and ( 0.1 as the default values our experiments.

---

5 Related work
Self-play fine-tuning for LLMs. The self-play learning framework originates from TD-Gammn ID-Gammon
[Tesauro 1995.. and has produced profound impacts in various fields, such as multi-agent reinforce-
ment learning Lanctot al., 2017, Muller al. 2019 Bansal al. 2018 and game-playin [Silver et al. 2017a Anthony et al. 2017]. The central idea of this framework is to train the model by playing against its previous version for progressive improvements. Recently, self-play fine-tuning has been introduced into the investigations for LLMs [Chen et al.) [002bb] which then leads to series of subsequent works (Ren et al.) 2024, Guo et al. [2024b] Yuan et al. 2024b) [Rosset et al. 2024 Cheng et al., 2024, Pang et al. 2024 Wu et al. [024b Wang et al. 2025]. For example, Gao et al.
[2024a] introduce the self-play mechanism into the recommendation systems to calibrate iteml predictions and suppress the bias issue, and Yuan et al. [2024a] incorporate the idea of self-play into diffusion models, resulting in a novel diffusion -bsed method. Different from the above studies that apply self-play fine-tuning to specific domains of LLMs, our work follows the seminal study of Chen et al. [202bb. investigating the fundamental nature of self-play fine-tuning, and aims to resolve the unstable evolution and misalignment issues in the original method.
Data generation in LLMs. Large language models are typically data-driven heavily relying on the high-quality annotated data [Ouyang et al. 2022 Tan et al.) [024). However, the available data is often limited for LLMM training, and the costs of human expert annotation are prohibitively expensive. For this reason, recent efforts have been made to employ advanced L.M.s, such as GPT series [OpenA]] 2023 2024 2025), Gemini series [Google 2023 2024 2025], and Claude series [Athhropic] 2024a b 2025]. to generate synthetic data for LLM fine-tuning [Josifoski et al. 2023 Gupta et al. 2023 Long et al. 2024| Qin et al. 2025]. It has been shown that in most domains, advanced expert LLMs are capable of generating high-quality data that can benefit others [Ubani Peng et al. [002 Xu et al. 2023 Yang et al. 2024 Choi et al.) 2024, ZZhang 2024 Wang al. 2024b DeepSeek-Al et al. [2025]. Compared to these studies, self-play fine- tine-tuming method focuses on using data generated by the current policy model to train itself to achieve the progressive performance improvements, and offers alternative expert-L..MMbasee strategies.
ct al. 2023
6 Conclusion and discussion
In this paper, we investigate self-play fine-tuning for LLMs and identify two issues of the existing method SPIN: unstable optimization during iterations and misaligument between training and gen- eration. To address these issues, we propose a novel self-play fine-tuning method named T-SPIN with two key designs. First, our T-SPIN simultaneously optimizes the current advantage of the annotated response y over the newly generated response y', and the historical advantage of y' over the proto-synthetic response Yo- When current advantages vanish, historical advantages can still guide the fine-tuning process, avoiding the unstable optimization. Second, T-SPIN incorporates the entropy constraint into the self- play framework, which eliminates the need for a reference policy in the reward computations, thus aligning the reward formulation and the generation metric. Empirically, we conduct extensive experiments and examine the performances of T-SPIN on variety of tasks. Experimental results show that T-SPIN maintains stable evolution and achieves better performance than its contenders, and outperforms supervised fine-tuning by leveraging only 25% annotated data.
Limitations and future work. There are several promising directions for future research. First, at each iteration, T-SPIN needs to re-generate synthetic responses for all prompts, although some of them may no longer contribute to further evolutions. Consequently, carefully designed filter strategy necessary to choose valuable samples before generation. Second, in T-SPIN, the -spert-annotated data is fixed during iterations. But, in real-world applications, the target distribution may shift over time, and hence the fixed annotated samples may not be suitable for online fine-nuning. For this reason, self-play fine-tuning for non-stationary target distributions is another interesting direction, for which we believe that some advanced techniques from online learning may offer valuable insights
Hazan 2016 Orabona 2019 Zhang et al. 2018 Wang et al. 2024cle].
Acknowledgments and Disclosure of Funding
This work was partially supported by NSFC (U23A20382), and the Collaborative Innovation Center of Novel Software Technology and Industrialization.
10

---

References
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt,
S. Altman, S. Anadkat, and al. GPT-4 technical report. ArXiv e-prints, EXXi::3303.88774 2023.
T. Anthony, Z. Tian, and D. Barber. Thinking fast and slow with deep learning and tree search. In Advances in Neural Information Processing Systems 30, pages 5360-5370, 2017.
Anthropic. Claude 3.5 sonnet.
2024a.
Anthropic. Introducing the next generation of claude. -laude-3-famill 2024b.
Anthropic. Claude 3.7 sonnet and claude code.
--aude---7-sonne 2025.
Bai, Bai, Yang. S. Wang, S. Tan, Wang. J. Lin, C. Zhou, and J. Zhou. Qwen-VIL A versatile vision-language model for understanding, localization, text reading, and beyond. ArXiv e-prints, arXiv:2308. 12966, 2023.
T. Bansal, J. Pachocki, S. Sidor, I. Sutskever, and I. Mordatch. Emergent complexity via multi-agent competition. In Proceedings of the 6th International Conference on Learning Representations, 2018.
S. Chen, Y. Wang, Y.-F. Wu, Q.-G. Chen, Z. Xu, W. Luo. K. Zhang, and L. Zhang. Advancing tool- cool-augmented large language models: Integrating insights from errors in inference trees. In Advances in Neural Information Processing Systems 37, pages 106555-106581 2024a.
Z. Chen, Y. Deng, H. Yuan, K. Ji, and Q. Gu. Self-play fine-tuning converts weak language models to strong language models. In Proceedings of the 41st International Conference Machine Learning, pages 6621-6642, 2024b.
Cheng, Yang, J. Li, Y. Dai, T. Hu, P. Cao, N. Du, and Li, Adversarial preference optimization: Enhancing your alignment via RM-LLM game. In Findings of the 62nd Conference of the Association for Computational Linguistics, pages 3705-3716, 2024.
Choi, J. Yun, K. Jin, and Kim. Multi-Newst: Cost-efficient dataset cleansing via lim-based data annotation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 29, 2024.
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and Schulman. Training verifiers to solve math word problems. ArXiv e-prints, arXiv:21 10. 14168, 2021.
H. Dai, 7. Liu, W. Liao, X. Huang, Y. Cao, 7. Wu, I.. Zhao, S. Xu, W. Liu, N. Liu, S. Li, D. Zhu, H. Cai, L. Sun, Q. Li, Shen, T. Liu, and X. Li. AugGPT: Leveraging ChatGPT for text data 2023.
augmentation. ArXiv e-prints, arXiv:2302.
DeepSeek-AL, D. Guo, D. Yang, H. Zhang. J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Wu, Z. Wu, Z. Gou, Z. Shao, Z. Li, and Z. G. al. Deepseek-rl: Incen- tivizing reasoning capability in llms via reinforcement learning. ArXiv e-prints, 1rXiv:2501.12948. 2025.
N. Ding, Y. Chen, B. Xu, Y. Qin, Z. Zheng, S. Hu, Z. Liu, M. Sun, and B. Zhou. Enhanc- ing chat language models by scaling high-quality instructional conversations. ArXiv e-prints, arXiv:2305. 14233, 2023.
C. Gao, R. Chen, S. Yuan, K. Huang, Y. Yu, and X. He. SPRec: Leveraging self-play to de- bias preference alignment for large language model-based recommendations, ArXiv e-prints, arXiv:2412.09243,, 2024a.
11

---

L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. L. Noac'h, h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, I.. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou, A framework for few-shot language model evaluation, 07 2024b.
Google. Our next-generation model: Gemini 1.3. https://blog / google/technology/al google-gemini -eenti wwweerattooo-oddeeeeereeeeeeeeee
2024.
Google. Gemini 2.5: Our most intelligent ai model. https://blog / oog.b//gggggggg/e/eooooooo////// google-deepminde
2025/ 2025.
G. Google. Gemini: family of highly capable multimodal models. arXiv:2312.11803 2023.
ArXiv e-prints,
S. Gugger, L. Debut, Wolf, P. Schmid, Z. Mueller, S. Mangrulkar, M. Sun, and B. Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable., 2022.
S. Gunasekar, Y. Zhang, J. Aneja, C. C. Mendes, A. D. Giorno, S. Gopi, M. Javaheripi. P. Kauff- mann, G. de Rosa, O. Saarikivi, A. Salim, S. Shah, H. S. Behl, X. Wang, S. Bubeck, R. Eldan, A. T. Kalai, T. Lee, and Y. Li. Textbooks
all you need. ArXiv e-prints, arXiv:2306. 1644, 2023.
D. Guo, Q Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. Li, F. Luo, Y. Xiong, and W. Liang. Deepseck-coder: When the large language model meets programming the rise of code intelligence. ArXiv e-prints, atXiv:2401.14196 2024a.
H. Guo, Y. Yao, W. Shen, J. Wei, X. Zhang, Z. Wang, and Y. Liu. Human- www.eiintsttto-fffeeee llm self-alignment with limited samples. ArXiv e-prints, EXXi:2401.00688 2024b.
H. Gupta, K. Scaria, U. Anantheswaran, S. Verma, M. Parmar, S. A. Sawant, C. Baral, and S. Mishra. TarGEN: Targeted data generation with large language models. ArXiv e-prints, arXiv:2310.17877 2023.
E. Hazan. Introduction online convex optimization. Foundations and Trends in Optimization, (3 4):1573225, 2016.
D. Hendryeks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. In Proceedings of the 9th International Conference on Learning Representations, 2021a.
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, Basart, E. Tang, D. Song, and Steinhardt. Mea- suring mathematical problem solving with the MATH dataset. ArXiv e-prints, arXiv:2103.03874, 2021b.
J. Huang and K. C.-C. Chang. Towards reasoning in large language models: A survey. In Findings of the s1 Conference of the Association for Computational Linguistics, pages 1049-1065. 2023.
B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, K. Dang, Y. Fan, Y. Zhang, A. Yang. R. Men, F. Huang, B. Zheng, Y. Miao, S. Quan, Y. Feng. X. Ren, X. Ren, Zhou, and J. Lin. Qwen2.5-Coder technical report. ArXiv e-prints, 11ti:220991118866 2024.
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L.. Lavaud, M.-A. Lachaux, Stock, L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mistral 7B. ArXiv e-prints, arXiv:2310.0666 2023.
Jiang, F. Wang, J. Shen, S. Kim, and S. Kim. survey on large language models for code generation. ArXiv e-prints, arXiv:2406.005 15, 2024a.
J.-P. Jiang, Y. Xia, H.-L. Sun, S. Lu. Q.-G. Chen, W. Luo, K. Zhang, D.-C. Zhan, and H.-J. Ye. Multi- modal tabular reasoning with privileged structured information. ArXiv e-prints, arXiv:2506.0408.. 2025.
X. Jiang, Y. Ge, Y. Ge, D. Shi, Yuan, and Y. Shan. Supervised fine-tuning in turn improves visual foundation models. ArXiy e-prints, arXiv:2401.10222. 2024b.
12

---

M. Josifoski, M. Sakota, M. Peyrard, and R. West. Exploiting asymmetry for synthetic training data generation: Synthie and the case of information extraction. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1555-1574, 2023.
M. Lanctot, V. Zambaldi, A. Gruslys, Lazaridou, K. Tuyls, J. Perolat, D. Silver, and T. Graepel. A unified game-theoretic approach to multiagent reinforcement learning. In Advances in Neural Information Processing Systems 30. pages 4190-4203, 2017.
H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems 36, pages 34892-34916, 2023.
H. Liu, C. Li, Li, B. Li, Y. Zhang, S. Shen, and J.J. Lee. LLaVA-NeXT: Improved reasoning. ocr, and world knowledge. https://llava-t// http:////laaaa....gi.........b http::..ooooo//200101-11111 -eaav---eeet.. 2024.
L. Long, R. Wang, R. Xiao, J. Zhao, X. Ding, G. Chen, and H. Wang. On LLMs-driven synthetic data generation, curation, and evaluation: A survey, ArXiv e-prints, arXiv:2406. 15126, 2024.
S. Lu, Y. Li, Y. Xia, Y. Hu, S. Zhao. Y. Ma, Z. Wei, Y. Li, L. Duan, J. Zhao, Y. Han, H. Li, W. Chen, Tang, C. Hou, Z. Du, Zhou, W. Zhang, H. Ding. Li. W. Li, G. Hu, Y. Gu, Yang. Wang. H. Sun, Y. Wang, H. Sun, Huang, Y. He, S. Shi, W. Zhang, G. Zheng, J. Jiang, S. Gao, Y.-F. Wu, S. Chen, Y. Chen, Q.-G. Chen, Xu, W. Luo, and K. Zhang. Ovis2.5 technical report. ArXiv e-prints, arXiv:2508. 11777, 2025.
Z. Luo, C. Xu, Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang. Wizard- Coder: Empowering code large language models with evol-instruct. In Proceedings of the 12th International Conference on Learning Representations, 2024.
P. Muller, S. Omidshafiei, M. Rowland, K. Tuyls, J. Perolat, S. Liu, D. Hennes, L. Marris, M. Lanctot, E. Hughes, Z. Wang, Lever, N. Heess, T. Graepel, and R. Munos. A generalized training approach for multiagent learming. ArXiv e-prints, arXiv:1909, 12823, 2019.
A. Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 2442)499,,,2 1997.
OpenAl. GPT-4.
2023.
OpenAI. Hello GPT-4o. http:://oopaaaacco////////llllaa 2024
2024.
OpenAl. Introducing GPT-4.5. httper//pppaaaaccc////eeeeeertts 2025.
F. Orabona. A modern introduction to online learning. ArXiv e-prints, arw.v: 2019.
L. Ouyang, Wu, X. Jiang, D. Almeida, I.. Wainwright, Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, Leike, and R. Lowe, Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35, pages 27730-27744, 2022.
R. Y. Pang, W. Yuan, H. He, K. Cho, Sukhbaatar, and Weston. Iterative reasoning preference optimization. In Advances in Neural Information Processing Systems 37, 2024.
B. Peng, C. Li, He, M. Galley, and J. Gao. Instruction tuning with GPT-4. ArXiv e-prints, arXiv:2304.03277, 2023.
J. Piaget. The origins of intelligence in children. International Universities Press, 1952.
Y. Qin, S. Liang. Y. Ye, K. Zhu, L. Yan, Y. Lu. Y. Lin, X. Cong, X. Tang, B. Qian. S. Zhao, L. Hong, R. Tian, R. Xie, J. Zhou, M. Gerstein, D. Li, Z. Liu, and M. Sun. TooILLM: Facilitating large language models to master 16000+ real-world APIs. ArXiv e-prints, McXiv:2307.16789 2023.
Z. Qin, Q Dong, X. Zhang, L. Dong, X. Huang, Z. Yang. M. Khademi, D. Zhang, H. H. Awadalla, Y. Fung, W. Chen, M. Cheng, and F. Wei. Scaling laws of synthetic data for language models. ArXiv e-prints, Siti::2503.195//11 2025.
13

---

R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning. S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems 36, pages
2023.
D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang. J. Dirani, J. Michael, and S. R. Bowman. GPQA: A graduate-level google-proof Q&A benchmark. ArXiv e-prints, arXiv:2311. 2023.
M. Ren, B. Cao, H. Lin, C. Liu, X. Han, K. Zeng, G. Wan, X. Cai, and L. Sun. Learning or self-aligning rethinking instruction fine-tuning. ArXiv e-prints, DoXiv:2402.18245 2024.
C. Rosset, C.-A. Cheng, A. Mitra, M. Santacroce, A. Awadallah, and T. Xie. Direct nash opti- mization: Teaching language models to self-improve with general preferences. ArXiv e-prints, arXiv:2404.03715, 2024.
K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. WinoGrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
P. Z. Scroccaro, A. S. Kolarijani, and P. M. Esfahani. Adaptive composite online optimization: Predictions in static and dynamic environments. IEEE Transactions on Automatic Control, 68(5): 2906-2921, 2023.
D. Silver, A. Huang, C. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, Kalchbrenner, I. Sutskever, T. P. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. ArXiv e-prints, 1rXiv:1712.0188 1.51 2017a.
D. Silver, J. Schrittwisser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis. Mastering the game of go without human knowledge. Nature, 550(7676):354-333) 2017b.
Z. Sprague, X. Ye, K. Bostrom, S. Chaudhuri, and G. Durrett. MuSR: Testing the limits of chain-of- thought with multistep soft reasoning. ArXiv e-prints, 1rXiv:2310.16049, 2024.
H.-L. Sun, Z. Sun, H. Peng, and H.-J Ye. Mitigating visual forgetting via take-along visual condition- ing for multi-modal long cot reasoning. In Proceedings of the 63rd Conference of the Association for Computational Linguistics, pages 5158-5171, 2025a.
H.-L. Sun, D.-W. Zhou, Li, S. Lu, C. Yi, Q.-G. Chen, 7. Xu, W. Luo, Zhang, and D.-C. Zhan. Parrot: Multilingual visual instruction tuning. In Proceedings of the 42nd International Conference on Machine Learning, 2025b.
M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. Chung, A. Chowdhery, Q. Le, E. Chi, D. Zhou, and J. Wei. Challenging BIG-Bench tasks and whether chain-of-thought can solve them. ArXiv e-prints, arXiv:2210.0926. 2022.
Z. Tan, D. Li, S. Wang, A. Beigi, B. Jiang, A. Bhattacharjee, M. Karami, J. Li, L. Cheng, and H. Liu. Large language models for data annotation and synthesis: A survey, ArXiv e-prints, arXiv:2402.1 13446, 2024.
R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford Alpaca: An fnslowition-folloiinng LLaMA model, 2023.
G. Tesauro. Temporal difference learning and TD-Gammon. Communications of the ACM, 38(3):
58-68, 1995.
L. Tunstall, E. Beeching, N. Lambert, N. Rajani, K. Rasul, Y. Belkada, S. Huang, L. von Werra, C. Fourrier, N. Habib, N. Sarrazin, O. Sanseviero, M. Rush, and Wolf. Zephyr: Direct distillation of LM alignment. ArXiv e-prints, trXiv:2310.16944 2023.
L. Tunstall, E. Beeching, N. Lambert, N. Rajani, S. Huang, K. Rasul, A. Bartolome, A. M. Rush, and T. Wolf. The Alignment Handbook, 2024. URL. https: //gi thub. com/huggingface/ handbook
14

---

S. Ubani, S. O. Polat, and R. Nielsen. ZeroShotDataAAn: Generating and augmenting training data with chatgpt. ArXiv e-prints, arXiv:2304. 14334, 2023.
K. Wang, Zhu, M. Ren, Z. Liu, S. Li, Z. Zhang, C. Zhang, X. Wu, Q. Zhan, Q. Liu, and Y. Wang. A survey on data synthesis and augmentation for large language models. ArXiv e-prints, MxXiv:2410.12899 2024a.
Wang, M. Wang. Z. Ma, X. Yang, S. Feng. D. Wang. Y. Zhang, and K. Song. Language models as continuous self-evolving data engineers. ArXiv e-prints, arXiv:2412.15151, 2024b.
Y. Wang, S. Chen, W. Jiang. W. Yang, Y. Wan, and L. Zhang. Online composite optimization between stochastic and adversarial environments, In In Advances in Neural Information Processing Systems 37, pages 94808 94850, 2024c.
Y. Wang, X. Ma, G. Zhang, Y. Ni, Chandra, S. Guo, W. Ren, A. Arulraj, He, 7. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. ArXiv e-prints, arXiv:2406001574, 2024d.
Y. Wang, W. Yang, W. Jiang, S. Lu, B. Wang, H. Tang. Y. Wan, and L. Zhang. Non-stationary projection-free online learning with dynamic and adaptive regret guarantees. In Proceedings of the 38th AAAI Conference on Artificial Intelligence, pages 15671-15679, 2024e.
Y. Wang. G. Huzhang, Q.-G. Chen, Z. Xu, W. Luo, K. Zhang, and L. Zhang. SPACE: Noise contrastive estimation stabilizes self-play fine-tuning for large language models. In Advances in Neural Information Processing Systems 38, 2025.
L. Wu, Zheng, Z. Qiu, Wang, H. Gu, T. Shen, C. Qin, C. Zhu, H. Zhu, Q. Liu, H. Xiong, and E. Chen. A survey on large language models for recommendation. World Wide Web, 27(60), 2024a.
Y. Wu, Z. Sun, H. Yuan, K. Ji, Y. Yang. and Q. Gu. Self- preference optimization for language model alignment. ArXiv e-prints, arXiv:2405.0065 40675,.05....... 2024b.
H. Xin, D. Guo, Z. Shao, Z. Ren, Q. Zhu, B. Liu, C. Ruan, W. Li, and X. Liang. DeepSeek- Prover: Advancing theorem proving Ilms through large-scale synthetic data. ArXiv e-prints, arXiv:2405. 14333, 2024.
B. Xu, A. Yang, J. Lin, Q. Wang, C. Zhou, Y. Zhang, and Z. Mao. ExpertPrompting: Instructing large language models be distinguished experts. ArXiv e-prints, arXiv:2305. 14688, 2023.
A. Yang, B. Zhang, B. Hui, B. Gao, B. Yu, C. Li, D. Liu, J. Tu, J. Zhou, J. Lin, K. Lu, M. Xue, R. Lin, T. Liu, X. Ren, and Z. Zhang. Qwen2.5-math technical report: Toward mathematical expert model via self- celf-improvement. ArXiv e-prints, arXiv:2409.122 12122v1,, 2024.
H. Yuan, Z. Chen, K. Ji, and Q. Gu. Self-play fine-tuning of diffusion models for text-to-image generation. ArXiv e-prints, arXiv:2402.10210, 2024a.
W. Yuan, Y. Pang, K. Cho, X. Li, S. Sukhbaatar, Xu, and J. Weston. Self-rewarding language models. In Proceedings of the 41st International Conference on Machine Learning, pages 57905- 57923, 2024b.
R. Zellers, A. Holtzman, Bisk, A. Farhadi, and Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 4791-4800, 2019.
L. Zhang, S. Lu, and Z.-H. Zhou. Adaptive online learning in dynamic environments. In Advances in Neural Information Processing Systems 31, pages 1323-1333. 2018.
Y. Zhang. Training and evaluating language models with template-based data generation. ArXiv e-prints, arXiv:2411.18100 2024.
Z. Zhao, W. Fan, J. Li, Y. Liu, X. Mei, Y. Wang, Z. Wen, F. Wang. X. Zhao, J. Tang, and Q. Li. Recommender systems in the era of large language models (LLMs).. IEFE Transactions on Knowledge and Data Engineering, 36:6889-6907, 2024.
15

---

J. Zhou, T. Lu. S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou. Instrwction-followinng evaluation for large language models. ArXiv e-prints, 1rX1v:231.00911, 2023.
Y. Zhu, Zhang, E.-U. Haq, Hui, and G. Tyson. Can ChatGPT reproduce numan-generated labels? study of social computing tasks. ArXiv e-prints, arX1v:2304.10145, 2023.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have included the main contributions in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations have been discussed in the Section G
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

We have specified the assumptions of Theorem 1 and provided the complete proof in the Appendix A.2
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
All information about the experiments is provided in the Appendix B.1
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

reproducible, they Answer: [No]
Justification: The complete details of the experimental setup are provided in the Section and the pseudocode are included in the Appendix D
and Appendix B.I B.I
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 ttaips.cc/ubblcc/ce hwwplli/ggiieee/CCe gww.ddd////ooouummiiii SaateeeSubisssiooooot swi.ssoiicyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The information has been provided in the Appendix B.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
19

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information has been provided in the Appendix B.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttp:://eeurip::: http://ppwpppppppo/cc/tctacaaaaa
Answer: [Yes]
Justification: The authors have diligently adhered to the NeurIPS Code of Ethics in all aspects of this research.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of this work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve releasing data or models that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have properly credited the original owners of the assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: This paper is methodological research the self-play fine-tuning of LLMs, and has included the specification about training LLMs in Appendix B.1
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
23

---

A Mathematical derivations
In this section, we provide mathematical derivations for the propositions and theorems.
A.1 Proof of Proposition
In this part, we proceed to derive the optimal solution of the following objective:
T argmax (xx-n()))""==( [c++1(X, y')! oBx-o()) [-((o(-)xx)]]
(10)
under general confidence function C1411 with the entropy constraint H(()). The proof is inspired by Rafailov et al. 2023), converting (IO) into KL-divergence minimization problem. Specifically, by
the definition of entropy, we have:
H(Tx(-1xx) - -By'~To(1x)) [log ()'(xx)))
(11)
Combining 10 and we have:
argmax Ex (x m-(2)) [c++1 (x, y y') + a xxx))))))))) [+((T((-xx)) (xx,(xxy')) log no (y [x]
argmax E Ex x
me
log TogToly'Ix) - 1 co+1 (x)
(12)
argmin E xx v'
Rg
TO ()(1lxx)
argmin Ex y --
log
Z(x) exp( 1 log Z(x)
where =(x)=Ix> Exey exp( C+1 (x, 2)) denotes the partition function for normalization. Note that (xx is function of the prompt x, and independent of the policy ze. We define
L (2)m1(x,)))
TT- (y(x)
exp
((x)
which is a valid policy, since To- (ylx)= 2 for any y and Ey To- (y/x)= 1. Next, we can rewrite
as:
log To(y'Ix)
argmin
Te- (y'(x) loog Z(x)
argmin Ex-a()) [D&L (Ta (-1x) (-1x) log Z(x)];
from which we can observe that the optimal policy is exactly ma- Therefore, the opponent policy is
Tg(y|x) To- (ylx)
Z(x) exp - (
A.2 Proof of Theorem
In this part, we derive the gradient of ( with respect to the parameter 0. For brevity, denote
u(To) log (y log ( |x)
(13)
v(Te) = log ( |x) - log To Ty(aaxx [))
and the expectation operator in this part is taken over the distributions TT. yu ~ Te To. (-Ix). By utilizing the chain rule, we have
g(-),y Tdata (+1x,
VOLT-SPII [Vo((u(TT)) + (Vol((r((z)))
(14)
=a (au(TT)) Vou aBE ((
Next, we consider two specific choices of f(x) and its derivative (x)) For the logistic loss function
((x) - log (2)), we have f((x) = -o (x). Then, combining (13 and (14) delivers
VALT-SPIN = - aE (o V + (o V Vou (TT))
= - aE
Vo log no (y' xx]
(15)
+ +Bo V o - V log no (vo x)]]
24

---

Table 3: The number of few-shots and evaluation metrics for each task.
Datasets
GSM8K MATH
MUSR
MMLU MMLU-Pro
GPQA
HellaSwag WinoCirade
BBH
IFEval
# few-shot
5
4
0
5
5
0
10
5
3
Metric
aCC
a0c
BCC
aCC
acc
acc_ norm acc_nome
acc
aCC morm
acc
For the exponential function {(x) = cxp(-2), we have ('(x) exp( 2x). Then, combining [3 and we can obtain that
VoLr-sFIn = E (exp(-au(<Ta) V + (exp(_aro(T))) Veu((T))
= aB (exp(_auu(Ta) To V log [x]
(16)
oo Vo (y
B More details on experiments
In this section, we provide more information about the experiments, including implementation details, task introductions, comparisons between T-SPIN, SPIN and SFT, and more ablation studies.
B.1 Implementation details
We choose the Aligmment Handbook (Tunstall al. 2024 the codebase for experiments, and the Accelerate library [Gugger et al. 2022 for distributed training. Following Chen et al.) [202bb]. employ the RMSProp [Scroccaro al. 2023) with weight decay the optimizer for training, and set max sequence length to be 2048 and warmup steps to be 10% of the total training steps. All experiments conducted on 8 H100 GPUs with 80GB memory. On each GPU, set the batch size as and thus the global batch size is 64.
B.2 Tasks
In the following, we provide detailed description of the tasks used in the experiments:
GSM8K [Cobbe et al., 2021] is dataset of 8, 500 high-quality linguistically diverse grade school math word problems, divided into 7.5k training problems and 1 test problems. These problems require between 2 and 8 steps to solve, with solutions primarily consisting of sequence of elementary calculations using basic arithmetic operations (+, -+,,,/,% GSMSK is designed to assess the mathematical reasoning abilities of LLMs.
MATH [Hendrycks et al. 2021b contains 12, 500 challenging competition mathematics problems. Each problem in MATH has full step-by-step solution which can be used teach models generate answer derivations and explanations.
MUSR (Sprague et al., 2024 is designed to evaluate multistep, context-dependent reasoning in language models through long-form, algorithmically generated problems approximately 1, 000 words in length. The dataset includes tasks such as murder mysteries, spatial reasoning (e.g.. object placement), and team assignment optimization, all presented in natural language narratives.
MMLU [Hendrycks et al. 2021a evaluates the multitask accuracy of language models on general knowledge and reasoning across 57 diverse subjects, ranging from STEM fields to humanities. In MMLU, each question is accompanied by four choices, and the answer is one of the four choices.
MMLU-Pro Wang et al. 2024d) is an enhanced dataset designed extend the mostly knowledge- driven MMLU benchmark by integrating more challenging, reasoning-focused questions and expanding the choice set from four to ten options.
GPQA (Rein al., 2023 is graduate-level, domain-specific benchmark composed of 448 expertly-authored multiple-choiee questions in biology, physics, and chemistry. aims to evaluate the ability of language models to reason about complex scientific concepts.
HellaSwag Zellers et al. 2019 is benchmark dataset designed to evaluate the commonsense reasoning and natural language understanding capabilities of machine learning models, particularly in the context of sentence completion tasks.
25

---

Table Performance (%) comparisons among three one-tuning methods, where SPIN and T-SPIN are trained on 50k annotated data from Ulrachat200k, and SFT is trained on the complete dataset. At the average (Avg) column, we also report improvements (or degradations) over the previous iteration. Abbreviations: MMLU-Pro + MMLUP, WinoGrade - WG, Instruction Following - IF.
Math & Logle
Multi-Domain Knowledge Commonsense Reasoning
IF
Model
Avg
GSM8K
MATH MUSR
MMLU
MMLUP
GPQA
HellaSwag
WG
BBH
IFEval
Mistral-7B
37.68
3.27
41.27
57.86
29.99
29.86
83.24
74.03
44.26
23.63
42.51
SFT
43.69
3.08
42.12
58.37
30.18
27.98
83.40
74.11
45.56
33.22
44.17
IterO
39.59
2.86
41 16
57.97
30.36
29.47
83.40
74.03
44.56
22.54
42.60
SPIN
Iterl
39.67
3.43
39.65
57.63 63
29.80
28.33
83.
74.51
42.50
26.04
(2.52
Iter2
39.52
3.50
39.64
57.49
29.82
28.00
83.69
73.80
41.97
25.56
42.30
42.30,
(-0.2))
Iter3
37.34
3.71
37.93
57.80
29.64
28.77
83.91
74.19
41.40
18.31
(1.30
Iter4
37.15
4.56
38.59
58.05
29.90
28.73
84.05
74.43
42.88
24.90
42.32
IterO
39.67
3.53
40.20
57.74
30.47
28.82
83.98
74.27
44.06
24.36
42.71
T-SPIN
Iterl
44.03
3.32
38.95
57. 48
30.45
29.10
83.41
74.66
44.91
26.88
43.32 (40.66)
Iter2
44.31
3.43
38.68
57.61
30.52
30.92
83.47
75.30
45.05
26.33
43.56 (4..6))
Iter3
45.87
3.85
38.59
58.51
31.42
30.83
84.53
73.88
44.97
36.50
(4.90
Iter4
45.92
3.94
38.46
58.55 55
31.34
31.06
84.60
74.51
44.90
36.90
45 02 (+0012
50.0
40.0
T-SPIN
T-SPIN
T-SPIN
SPIN
30.0
SPIN
45.0
SPIN
Scores (8)
45.0
20.0
Scorea
40.0
500
41.0
10.0
35.0
0.0
37.0
iter
iter4
Iner
iterl
Iher2
heer3
itend
(o) 6SMBK
(b) IFEvll
(c) Average Score
Figure 6: Performance (%) comparisons between T-SPIN and SPIN on two tasks: GSMSK and IFEval over iterations. The average scores over different tasks are also illustrated the right panel.
large-scale dataset of 44k problems. inspired by the original WSC design. but adjusted to improve both the scale and the hardness of the dataset.
WinoGrade (Sakaguchi et 2021]
2022 is curated subset of 23 challenging tasks that span algorithmic reasoning, language understanding, world knowledge, and multi-step logic.
BBH (uzgunn et al.
IFEval Zhou et al. 2023 is benchmark dataset designed to evaluate the cneructton-followiin capabilities of LLMs. It includes 500 carefully curated prompts annotated with 25 distinct instruction types. The benchmark focuses on a set of "verifiable instructions" such as "write in more than 400 words" and "mention the keyword of AI at least 3 times".
All tasks are evaluated using the Language Model Evaluation Harness framework (Gao et al. 2024b with default evaluation settings specified in Table
B.3 Experimental results on Mistral-7B
In this part, conduct experiments Mistral-7B compare the performances of T-SPIN, SPIN and SFT. The results over 10 different tasks are displayed in Table From the table, we first observe that T-SPIN achieves notable performance improvement over iterations, and outperforms SFT the iteration 3. This indicates that with less annotated data, T-SPIN can achieve better performances than SFT that leverages the annotation shortage issue. Second, compared with SPIN, T-SPIN exhibits better performances and stable improvements. Specifically, SPIN suffers from performance degradations at the iterations
annotated samples, highlighting the effectiveness of T-SPIN in handling
http://ggiiia..comm//eeteeee/eet harness
26

---

Iteration 0
Iteration
Iteration
500
log Foly
log Roly)x)
-200
log
www.
-400
4400
-500
-600
25
50
100
25
100
25
75
100
steps (7)
steps (7)
steps (1)
generated
gap
(a) ligeliilldoood dynamics of SPIN
Iteration 0
Iteration 1
Iteration 2
500
300
300
To
200
log mellix))
-100
log moly)))
-500
-500
9000
25
50
100
25
50
75
100
75
100
steps (%)
steps (%)
steps (%)
real
generated
gop
(b) Log-likellhood dynamics of TSPIN
Figure 7: The log- likelihoods of annotated samples and synthetic samples during the model training of T-SPIN and SPIN at different iterations.
and 3, while T-SPIN maintains stable refinement over all iterations. To further illustrate the instability of SPIN, we plot the performances on GSM8K, IFEval and average score Figure As can be seen, at iteration 3, SPIN exhibits performance degradation compared to iteration 2. In contrast, T-SPIN maintains steady improvements throughout the iterations.
B.4 Comparisons on log-likelihoods
In Section have reported the log- ligeliiol dynamics of two self-play methods iteration In the following, we examine the trend of log-likelihood changes across different iterations. Specifically, track the loelliiklshoods of annotated samples log To (y/x) and synthetic samples log To (y' (x) during policy training at iterations 0, 1 and 2, and calculate the gap between them, defined log ( log no (y''xx). The results presented in Figure 7 It is observed that
[x). iteration 0, for both methods, the log-likelihoods of annotated samples are higher than that of synthetic samples, resulting positive gaps between them. However, iterations 1 and 2. SPIN demonstrates an unexpected pattern: the log-likelssoods of annotated samples fall below those of synthetic samples, resulting in negative gaps. This reversal stems from the misalignment between the reward function employed in SPIN and the log-likelihood. Specifically, the objective function (3) of SPIN maximizes the reward gap between annotated and synthetic samples. Due the incorporation of reference policy, the reward is no longer equivalent to the log- loglliiklihood.. As result, optimizing for high reward during training does not guarantee an increase in likelliiodood Importantly, since log-liiel underlies the generation behavior of LLM and reflects its preference for certain responses, it is undesirable that SPIN assigns higher log-likelihoods to synthetic samples than to annotated ones. In contrast, T-SPIN exhibits consistent behavior in terms of fog-likelihood: across all iterations, the log-likelihoods of annotated samples remain higher than those of synthetic samples. This desirable property stems from T-SPIN's climination of the reference policy in its reward function, which ensures alignment between the reward and lieliiklioood.
C Case study
In this section, we provide an example including annotated responses and synthetic responses generated at each iteration, to illustrate the progressive evolution of the model trained by T-SPIN.
27

---

Table 5: Examples of the model fine-tuned by T-SPIN at each iteration.
Prompt +
Q: Create a thank you card to an organization that helped your community in time of crisis.
Annotated + Dear [Organization Name], I writing this letter to express my sincere gratitude to your organization for the exceptional help you provided our community during a time of crisis. The assistance and support that you and your team showed to us when we were going through difficult times are immeasurable, and will forever be thankful to you.
Your prompt response, the dedication of your volunteers, and the resources that you made available to us, made a difference in our community. Because of your efforts, families who affected [the crisis] found place stay, food eat, and support help them get back their feet.
It is remarkable to see how your organization came together and worked tirelessly to provide relief to those in need. Your kindness and empathy for our community truly appreciated, and has left a lasting impact on our residents.
Thank you once again for your unvavering support during that trying time. Your actions demonstrate the true meaning of community spirit and the good that together. Your kindness will always be remembered and cherished.
from working
Sincerely, [Your Name]
Proto-synthetic
1. What is the name of the organization? 2. What is the organization's mission? organization's impact on the history? What the community? 5. What is the organization's 's impact on the world? 6. What is the organization's impact you? 7. What is the organization's impact your family? 8. What the organization's impact your friends? the organization's impact on your community? 10. What is the organization's impact on the world?
What is the organization's
What
T-SPIN Iter 0 +
1. I would like to thank the American Red Cross for their help in the aftermath of Hurricane Katrina. The American Red Cross provided shelter, food, and medical care to thousands people who were displaced by the hurricane. They also provided emotional support to those who had lost loved ones or their homes. The American Red Cross is vital organization that helps people in times of crisis, and I grateful for their work. 2. would also like to thank the Salvation Army for their assistance during Hurricane Katrina. The Salvation Army provided food, water, and shelter to those in need, and they also provided emotional support to those who were struggling. The Salvation Army helps people times of crisis, and I am grateful for their work. 3. Fimally, I would like to thank the United Way for their support during Hurricane Katrina. The United Way provided financial assistance to those who had lost their homes or their jobs, support to those who were struggling. The United Way is an organization that helps people in times of crisis, and I am grateful for their work. Thank you to all of these organizations for their help during Hurricane Katrina
compassionate organization that
they also provided emotional
28

---

T-SPIN Iter 1 +
1. What is the name of the organization you are thanking? What crisis did this organization help your community with? 3. How did this organization help your community?
4. What did you learn from this experience? 5. How can you help your community in the future? ## Extended Response:
Dear American Red Cross,
am writing to exprese my deepeat gratitude for the incredible work you did during the recent hurricane that hit our community. Your organization was instrumental in providing relief and support those affected by the The hurricane caused widespread damage to our area, leaving many people without homes, food, water. The American Red Cross was there to provide assistance to those in need. Your volunteers worked tirelessly to up shelters, distribute food and water, and provide emotional support those who lost everything.
Your orgamization's efforts did not go unnoticed. The community was grateful for the help and support you provided during this difficult time. Your actions demonstrated the true spirit compassion and generosity that at heart the American Red Cross.
Through this experience, I learned the importance of coming together community to help those
T-SPIN Iter 2 Dear [Organization Name],
I writing express my deepest gratitude for the incredible work you did during recent crisis that affected our community. Your unsavering dedication and commitment to helping those in need truly made difference in the lives of many.
Your organization's quick response and willingness go above beyond to provide assistance to those affected by the crisis was truly inspiring. Your team's tireless efforts to provide food, shelter, and other essential resources to those in need were true testament to your organization's values and commitment serving the community.
Your organization's efforts did not unnoticed, and I am sure that many in our community feel the same way. Your selfless acts of kindness and generosity have touched the hearts of many community.
have made a lasting impact on our
Oace again, thank you for your incredible work during this difficult time. Your organization's efforts have not gone unnoticed and truly appreciated. Sincerely, [Your Name]
T-SPIN Iter 3 +
Dear (Organization Name],
I am writing to express my deepest gratitude for the incredible work you did during the recent crisis that affected community. Your unvavering dedication and commitment to helping those in need truly made difference in the lives of so many people.
Your organization's quick response and willingness to above and beyond provide assistance during time of crisis a testament to your selfleseness and compassion. Your efforts have not gone unnoticed and are greatly appreciated.
I am proud to be part of community that has such an amazing organization like yours. Your tireleas work has not only helped those in need but has also brought our community closer together.
Thank you again for all that you do. Your kindness and generosity will never be forgotten.
29

---

Sincerely,
[Your Name]
T-SPIN Iter 4 Dear [Organization Name],
We, residents of [insert community name], would like to extend our deepest gratitude for your unwavering support during the recent natural disaster that struck our town. Your swift response and tireless efforts to provide aid and relief those affected the [insert type of disaster] were truly remarkable. Your team of volunteers worked tirelessly to ensure that our community had access to essential resources such as food, water, and shelter. Your kindness and generosity did not go unnoticed and are forever grateful for the support you provided during this difficult time. You have restored our faith in humanity and have shown us that there is still good in this world.
Once again, thank you from the bottom of our hearts for all that you have done for blessed to have had you by our side during this crisis. Sincerely, [Insert community name]
You are truly heroes and
D Pseudocode
In the following, we provide the pytorch pscudocode for calculating the loss function 7 T-SPIN.
import torch. functional
def lasin_lossaalpha beta, reaccc_real__lopae golicy_gemeratedd_ogg
policyyprroto_oogs))
Compute the loss for T-SPIN (Triplet-based Self-Play Fine-Tuning). Arga:
alpha: the regularization coefficient
beta: the trade-off parameter
policy_real_logps Policy's log probs for real samples
Policy's 's log probs for synthetic samples
policy.proto_loggs Policy's log probs for proto-synthetic samples
Returns:
torch. Tensor: scalar loss value (mean over batch)
# Compute advantage terms purrent_advantad policy_real. logps policy-gemee wew.ccygggerrredddeee logps history_advantage policy-generated loggss policy_proto_lee logps
Scale advantages to get rewards current_rewards alpha current_advantaee
history_rewards alpha
Convert rewards to losses using log-sigmoid
current_lose = -F sr.boggigmoiacuureeee.oaaa
history_loss = -F
Combine components
losses current_lose beta history_lost
return losses mean ()
30


Reviews:
[{"review_id": "Qfp84heFu1", "reviewer": "Reviewer_Xd9e", "summary": "This paper proposes a triplet-based self-play method for fine-tuning LLMs called T-SPIN. Compared to SPIN, it uses triplets of annotated, synthetic, and proto-synthetic responses to address the deminishing advantage issue and replaces the reference‑policy reward with an entropy‑regularised objective. The authors test their method on Zephyr‑7B and Mistral‑7B over ten benchmarks and show consistent improvements over SPIN with better data efficiency.", "questions": "How many iterations do you observe diminishing returns?", "limitations": "Please see \"Weakness\"", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. The paper is generally well-written and easy to read.\n2. The triplet idea, which uses historical advantages to avoid collapses, can be novel.\n3. The improvement on data efficiency (using 25% of labeled data) can be useful in practice. Training also seems more stable than SPIN.\n4. The authors did some ablation studies to verify the effectiveness of historical advantages.\n\n### Weakness\n1. The models used for finetuning, Zephyr‑7B and Mistral‑7B, are under-performing modern models like the Qwen series. It is unclear if this method also holds for stronger baselines or if the improvement would be marginal. Results could be more convincing if the authors also test on stronger baselines.\n2. The extra computational overhead of generating proto-synthetic responses is not measured, including wall-clock time, flops, and memory cost.\n2. The improvement over SPIN is not significant on some datasets.\n2. Not compared with other improved methods based off SPIN.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the valuable feedback, and provide our responses as follows. We hope our responses help strengthen your ***confidence*** in this work, and we are looking forward to addressing any further questions during the reviewer-author discussion.\n\n---\n\n**Q1**: It is unclear if this method also holds for stronger baselines.\n\n**A1**: Thank your for the helpful suggestion. We have conducted additional experiments on Qwen2.5-7B-base, and evaluated performance on ten benchmarks (in Lines 218-223). We report the average score in the following table:\n\n||Qwen 2.5-7B|iter0|iter1|iter2|iter3|iter4|\n| --------| -------------| -------| -------| -------| -------| -------|\n|SPIN|53.49|53.84|53.92|53.71|53.89|53.98|\n|T-SPIN|53.49|53.68|54.17|55.04|54.97|55.03|\n\nFrom the results, we have the following two observations:\n\n* First, both methods yield less performance enhancements on Qwen2.5-7B compared to Zephyr‑7B and Mistral‑7B. This may be attributed to: (i) Qwen2.5 is already a strong LLM, making it harder for further improvements via self-play fine-tuning; and (ii) our experiments are conducted with limited annotated samples, where achieving performance enhancements on strong LLMs is inherently challenging;\n\n* Second, SPIN still suffers from unstable optimization during iterations, which prevents further performance improvements. In contrast, T-SPIN introduces historical advantages to guide the optimization, resulting in stable performance enhancements.\n\n---\n\n**Q2**: The extra computational overhead of generating proto-synthetic responses is not measured. \n\n**A2**: Thanks for the thorough consideration. We report the runtime, FLOPs, and memory cost for generating proto-synthetic samples below:\n\n|running-time|total flops|peak memory|\n| --------------| -------------| -------------|\n|0.8 hours|$1.8 * 10^{17}$​  FLOPs|33.43 GB|\n\nAdditionally, we highlight that the cost of generating proto-synthetic samples $y_0$ is affordable for self-play fine-tuning because: (i) the generation for $y_0$ is performed **offline** and only **once** over the multi-iteration training process (see **Algorithm 1**); (ii) in each iteration, the training process takes approximately **3.5 hours**, which is significantly longer than generation, and takes up most of the total computation time.\n\n---\n\n**Q3**: The improvement over SPIN is not significant on some datasets.\n\n**A3**: We would like to emphasize that: (i) we use only a small amount of high-quality annotated data from Ultrachat200K for training, which makes it inherently difficult to achieve significant improvements across all benchmarks; (ii) we mainly focus on overall performance over multiple benchmarks, which provides a more comprehensive view of T-SPIN’s advantages than results on individual datasets. In our experiments, T-SPIN has demonstrated higher average scores than baselines (see **Table 1**).\n\n---\n\n**Q4**: Not compared with other improved methods based on SPIN. \n\n**A4**: Thank you for the kind reminder. Recently, we have found several studies that propose improved variants of SPIN [1, 2], and we will include comparisons with these advances in the revised version.\n\n---\n\n**Q5**: How many iterations do you observe diminishing returns?\n\n**A5**: In experiments, we run a total of five iterations. On Zephyr, SPIN reaches its performance peak at iteration 2 and begins to decline thereafter, as shown in Figure 2(c). On Mistral, SPIN exhibited performance fluctuations starting from iteration 1 and continuing until iteration 4, as shown in Figure 5(c).\n\n---\n\n**Reference**\n\n[1] Alami et al. Investigating Regularization of Self-Play Language Models. Arxiv, 2024.\n\n[2] Yang et al. Dynamic Noise Preference Optimization for LLM Self-Improvement via Synthetic Data. Arxiv, 2025."}, {"review_id": "qjhvk0Zg4M", "reviewer": "Reviewer_q2aJ", "summary": "This paper introduces T-SPIN, which constructs a triplet of annotated, synthetic, and proto-synthetic responses to improve SPIN and stabilizes training by leveraging historical advantages and aligning the reward with generative likelihood through an entropy constraint. Extensive experiments on multiple aspects are done to compare T-SPIN and SPIN, showing that T-SPIN achieves comparable or better performance than SPIN.", "questions": "Have the authors considered using responses from an intermediate past policy, such as $\\pi_{\\theta_{\\lfloor t/2 \\rfloor}}$, instead of always relying on the initial policy $\\pi_{\\theta_0}$? This could provide a stronger and more relevant benchmark for $\\pi_{\\theta_t}$, potentially encouraging greater improvements in later iterations.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n1.\tStabilizes the optimization by introducing historical advantages from proto-synthetic responses, preventing optimization collapse.\n2.\tAligns the reward function with the generative likelihood.\n3.\tExperiment shows that there is a consistent improvement in the fine-tuned performance as iteration number increases.\n\n\n**Weaknesses**:\n\n1. The comparison with the proto-synthetic responses $y_0$ may become stale over successive iterations, as the model’s performance could surpass the initial policy to such an extent that this anchor no longer provides a meaningful training signal.\n    \n2. The ablation studies could be expanded. In particular, it would be valuable to analyze the isolated effect of changing the reward formulation from $\\lambda \\log \\frac{\\pi_\\theta(y|x)}{\\pi_{\\theta_t}(y|x)}$ to $\\alpha \\log \\pi_\\theta(y|x)$, in order to disentangle the contribution from the historical advantage.\n    \n3. The paper does not clearly articulate why the historical advantage is helpful, despite this being a central and intriguing idea. A more thorough discussion of the underlying intuition and mechanisms behind the benefits of historical advantage would strengthen the work.", "was_revised": true, "final_justification": "The authors addressed my concerns during our discussion and raised my score from 4 to 5.", "rebuttal": "We sincerely appreciate the valuable feedback! We provide our responses below, and are looking forward to addressing any further question in the reviewer-author discussion period!\n\n---\n\n**Q1**: The proto-synthetic responses $y_0$ may become stale over successive iterations.\n\n**A1**: Thank you for the insightful comment. We understand your concern that proto-synthetic response $y_0$ may become stale over iterations. As shown in (7), $y_0$ is incorporated to compute the historical advantages, which  can be viewed as a regularization to prevent performance collapse when current advantages diminish during iterations. In fact, its static property is a key factor in providing stable optimization. Detailed discussions can also be found in **Lines 136-144**.\n\n---\n\n**Q2**: The isolated effect of changing the reward formulation.\n\n**A2**: Thank your for the constructive suggestion. We currently employ the reward function $c(x, y)=\\lambda \\log \\frac{\\pi_{\\theta}(y|x)}{\\pi_{\\theta_t}(y|x)}$ , and substitute it into (4) to obtain a new loss function, as shown below:\n\n$$\nL(\\theta) =   \\mathbb{E}  \\left[ \\ell\\left( \\lambda  \\log  \\frac{\\pi_\\theta (y |x )}{\\pi_{\\theta_t} (y |x )}    - \\lambda  \\log \\frac{  \\pi_{\\theta}(y'|x)}{\\pi_{\\theta_{t}}(y'|x)}   \\right)  + \\beta \\ell\\left( \\lambda  \\log \\frac{  \\pi_{\\theta}(y'|x)}{\\pi_{\\theta_t}(y'|x)}    - \\lambda  \\log \\frac{  \\pi_{\\theta}(y_0|x)}{\\pi_{\\theta_t}(y_0|x)}    \\right)  \\right].\n$$\n\nThen, we conduct an ablation study on Zephyr-7B, comparing the performance of models optimized with (7) (i.e., T-SPIN) to those using the above loss function (namely, **T-SPIN_ref**). The average score over ten tasks is presented in the following table. From the results, we observe that *T-SPIN* demonstrates superior performance compared to *T-SPIN_ref.*  This can be attributed to the fact that using $c(x, y)=\\alpha \\log  \\pi_{\\theta}(y|x)$ as the reward function preserves the alignment between training and generation, which in turn facilitates performance improvement.\n\n||Zephyr-7B|iter0|iter1|iter2|iter3|iter4|\n| ------------| -----------| -------| -------| -------| -------| -------|\n|T-SPIN_ref|38.56|38.83|42.19|42.32|42.08|42.33|\n|T-SPIN|38.56|39.75|42.56|42.79|43.23|43.47|\n\n\n---\n\n**Q3**: A more thorough discussion of the historical advantage.\n\n**A3**: Thank you for the helpful suggestion. To clarify the underlying intuition, we start by examining the fundamental issue that causes the performance collapse observed in SPIN during iterative optimization.\n\n* **Vanishing current advantages.**  The core idea of SPIN is to progressively improve performance by minimizing the relative advantage of the current policy $\\pi_{data}$ over the target policy $\\pi_{\\theta_t}$ during iterations. Specifically, as shown in (3), SPIN aims to reduce the confidence gap between high-quality annotated samples and self-generated responses, i.e., minimizing $\\ell(c(x,y)-c(x,y'))$ where $y \\sim \\pi_{data}$ , $y'\\sim \\pi_{\\theta_t} $ and $c(x,y)=\\lambda \\log \\frac{\\pi_\\theta(y|x)}{\\pi_{\\theta_t}(y|x)}$.  As the model continues to evolve, the gap between $\\pi_{data}$ and $\\pi_{\\theta_t}$ gradually narrows. Once this gap vanishes, the objective in (3) degenerates into a constant independent on $\\pi_\\theta$, **making any policy a trivial solution**, leading to unstable optimization and performance collapse ultimately.\n\nTo stabilize the iterative optimization, it is thus necessary to introduce a regularization term that prevents this degeneration. A natural choice is to incorporate the **historical advantage**​, i.e., the relative advantage of the current policy $\\pi_{\\theta_t}$ over the initial policy $\\pi_{\\theta_0}$, which remains meaningful even when the current advantage vanishes, thereby mitigating objective degeneration and maintaining training stability. We will include a more in-depth discussion of the underlying intuition and mechanisms of T-SPIN in the revised version.\n\n---\n\n**Q4**: Have the authors considered using responses from an intermediate past policy. \n\n**A4**: We sincerely appreciate this insightful suggestion! To evaluate its effectiveness, we conduct additional experiments on Zephyr-7B comparing our T-SPIN with the variant (namely **T-SPIN_mid**) that employs proto-synthetic responses $y_{\\lfloor t/2 \\rfloor}$ from the intermediate past policy $\\pi_{\\lfloor t/2 \\rfloor}$ to optimize (7). Notably, at iterations 0 and 1, both T-SPIN and T-SPIN\\_mid use proto-synthetic responses generated from the same initial policy $\\pi_0$, so the comparison starts from iteration 2. We present the experimental results as follows:\n\n||iter2|iter3|iter4|\n| --------------| -------| -------| -------|\n|T-SPIN_mid|42.27|42.20|41.53|\n|T-SPIN|42.79|43.23|43.47|\n\n\nFrom the table, we observe that T-SPIN\\_mid suffers from the performance decline across iterations. We hypothesize that this degradation may stem from the vanishing of the historical advantage at iteration 2 (since both $y'$ and $y_0$ are sampled from $\\pi_{\\theta_1}$), which may potentially destabilize the optimization in subsequent iterations. Due to limited computational resources, we are unable to further validate this idea over other models during the rebuttal period. This is indeed a very **interesting** direction, and we plan to explore it in future work. Once again, we sincerely thank the reviewer for this valuable idea."}, {"review_id": "2I5sCWiwnL", "reviewer": "Reviewer_J2Mt", "summary": "This paper proposes Triplet-based Self-Play Fine-tuning (T-SPIN). It is an improved method for fine-tuning large language models with limited annotated data. The authors identify two key limitations in the existing SPIN method: (1) unstable optimization when the advantage between annotated and synthetic responses diminishes over iterations, and (2) misalignment between training rewards and generation metrics due to the use of reference policies. The proposed method, T-SPIN, addresses these 2 issues through two main innovations: (1) incorporating \"historical advantages\" by comparing current synthetic responses to proto-synthetic responses from the initial policy, and (2) eliminating reference policies by introducing an entropy constraint.", "questions": "How does T-SPIN perform on datasets from different domains (e.g., code generation, scientific text, creative writing)?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper effectively identifies the limitations of SPIN with both theoretical analysis and empirical evidence. The misalignment between training rewards and generation log-likelihoods is well-demonstrated.\n2. The experiments cover diverse tasks (e.g., math, logic, knowledge, reasoning, instruction-following, etc.) and include analysis, like ablation studies, etc. The results consistently show T-SPIN's superiority over SPIN in both performance and stability. It shows that to achieve comparable performance to supervised fine-tuning only needs 25% of the annotated data. It is useful especially when we have limited expert annotations.\n\nWeaknesses:\n1. While the paper provides gradient analysis (e.g., in Theorem 1), it lacks deeper theoretical guarantees about convergence. The stability claims are primarily based on empirical studies rather than theoretically proven.\n2. The proposed method, T-SPIN, requires generating and storing proto-synthetic responses from the initial policy. It will potentially increase memory and computational requirements. To my best knowledge, I didn't see the paper discussed somewhere about addressing the practical implications of this overhead.\n3. Experiments are done with only on 7B parameter models (e.g., Zephyr-7B, Mistral-7B). It's unclear how the method scales to larger models where computational costs become more significant. Meanwhile, all experiments use Ultrachat200k. The generalizability of the performance on other domains remains unknown.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the valuable feedback! We provide our responses below, and are looking forward to addressing any further question in the reviewer-author discussion period!\n\n---\n\n**Q1**: This paper lacks deeper theoretical guarantees about convergence. The stability claims are primarily based on empirical studies rather than theoretically proven.\n\n**A1**: Thank you for the valuable comment. We agree that establishing theorems about convergence and stability is important for T-SPIN. However, due to **the non-convex property of LLMs** and **the arbitrariness of the monotonically decreasing function** ***$\\ell(\\cdot)$*** in (7), it is highly challenging to provide rigorous convergence and stability guarantees for T-SPIN. As an alternative, we provide theoretical analysis for gradients in Theorem 1 with corresponding discussions in **Lines 196–200**. We recognize the importance of convergence guarantee and plan to investigate it in future work.\n\n---\n\n**Q2**: Lack of discussion on memory and computational requirements.\n\n**A2**: Thank you for this practical concern. The additional cost of T-SPIN compared to SPIN comes from the proto-synthetic response $y_0$ generated by the initial policy $\\pi_0$, including\n\n* **One-time computational cost:**  At the beginning of the training, we conduct a single forward pass to generate $y_0$ with the initial policy $\\pi_0$ (see **Algorithm 1**). The computational cost is equivalent to generating synthetic response $y'$ at one iteration, but is not repeated during the training process. In practice, the generation of proto-synthetic responses only takes 0.8 hours, which is significantly lower than the 3.5 hours for training.\n* **Increased memory cost:**  In T-SPIN, the training data for each iteration expands from pairs ($y, y'$) to triplets ($y, y', y_0$). We argue that this is a modest and acceptable trade-off for the *significant* improvements in final performance and optimization stability, as shown in Table 1 and Figure 2. Moreover, compared to SPIN, although T-SPIN utilizes additional samples for training, it **eliminates the need of reference model**. As a result, the additional memory cost in experiments is minimal.\n\nWe will include the discussion about computational and memory costs in the final version.\n\n---\n\n**Q3**: Performance of larger models.\n\n**A3**: Thanks for the helpful suggestion. While our T-SPIN is model-agnostic,  we are currently unable to provide results on larger LLMs due to limited computational resources and the tight rebuttal timeline. Nevertheless, we plan to include expanded results in the final version.\n\n---\n\n**Q4**: All experiments use Ultrachat200k. How does T-SPIN perform on datasets from different domains.\n\n**A4**: Thank you for the insightful question. Following the experimental setting in SPIN, we select a subset of Ultrachat200k as the training set. In our paper, we have evaluated the performances over a wide range of domains, including Math and Logic (GSM8K, MATH, MUSR), Multi-Domain Knowledge (MMLU, MMLU-Pro, GPQA), Commonsense Reasoning (HellaSwag, Winogrande, BBH), and Instruction Following (IFEval), and present the results in Table 1. Furthermore, we also evaluate the performance of T-SPIN on  **Code Generation** (HumanEval [1]) and **Scientific Question Answering** (OpenBookQA [2]). The results presented in the following table demonstrate that T-SPIN consistently yields stable performance improvements across these diverse benchmarks.\n\n||Zephyr-7B|iter0|iter1|iter2|iter3|iter4|\n| --------------------| -----------| -------| -------| -------| -------| -------|\n|HumanEval (pass@1)|3.05|3.60|4.27|4.88|6.1|6.71|\n|OpenBookQA|42.8|43.0|43.4|43.8|44.2|44.4|\n\n---\n\n**Reference**\n\n[1] Chen et al. Evaluating Large Language Models Trained on Code. Arxiv, 2021.\n\n[2] Mihaylov et al. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. EMNLP, 2018."}, {"review_id": "7Jah0RhaUX", "reviewer": "Reviewer_wsFg", "summary": "This paper proposes an incremental improvement over a well known RLHF method called SPIN. SPIN and T-SPIN proposed in this paper both leverage the same format of data as SFT, and the comparison is constrained to this setting. \n\nSPIN loss looks very similar to DPO loss except that, because we don't have the losing answer for query, the model's latest version's generation is used for that purpose.", "questions": "One of the selling point seems to be the stability of the method. If so, an experiment comparing the variance of the training process would be great.", "limitations": "It's still in the realm of SFT-data utilization, which is understandable for a fair experimental comparison.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strength: \n- T-SPIN proposes to use some sort of triplet comparison to not only encourage its previous generated are thought of as inferior than annotation, but additionally prefer that over the initial reference answer. The experimental results look convincing.\n\n\nWeakness: \n- It was on older models and doesn't compare with preference-pair-based methods (which is understandable, but a table would be informative).\n- Isn't the reference still playing a role in the loss function? Instead of living in the denominator, it now resides in the second term of comparison. \n- One of the points the author's critique of SPIN is that the reward is not the same as metric. However, isn't it still the case in T-SPIN?", "was_revised": true, "final_justification": "I read the authors' response and find it convincing, especially the explanations on the the reference.", "rebuttal": "We sincerely appreciate the valuable feedback! We provide our responses as follows, and hope the reviewer could **reassess** our work. We are looking forward to addressing any further question in the reviewer-author discussion period!\n\n---\n\n**Q1**: It was on older model, and do not compare to preference-pair-based methods.  \n**A1**:  Thanks for the helpful suggestion. We present our responses as follows.\n\n* **LLM choice**. We conduct experiments on Zephyr-7B and Mistral-7B to remain consistent with the original settings in SPIN. Furthermore, we have conducted additional experiments on Qwen2.5-7B-base, and evaluated performance on ten benchmarks (in Lines 218-223). We report the average score in the following table:\n\n  ||Qwen 2.5-7B|iter0|iter1|iter2|iter3|iter4|\n  | --------| -------------| -------| -------| -------| -------| -------|\n  |SPIN|53.49|53.84|53.92|53.71|53.89|53.98|\n  |T-SPIN|53.49|53.68|54.17|55.04|54.97|55.03|\n\n  From the results, we observe that: First, both methods yield less performance enhancements on Qwen2.5-7B compared to Zephyr‑7B and Mistral‑7B. This may be attributed to: (i) Qwen2.5 is already a strong LLM, making it harder for further improvements via fine-tuning; and (ii) our experiments are conducted with limited annotated samples, where achieving performance enhancements on strong LLMs is inherently challenging. Second, SPIN still suffers from unstable optimization during iterations, which prevents further performance improvements. In contrast, T-SPIN introduces historical advantages to guide the optimization, resulting in stable performance enhancements.\n\n* **Comparison to preference-pair methods.**  We would like to emphasize that our study focuses on the problem of *self-play fine-tuning*, which is fundamentally different from *pairwise preference optimization*. Specifically, preference optimization typically relies on data in the format of (prompt, chosen response, rejected response), while our T-SPIN is designed for scenarios where only SFT-style data, i.e., (prompt, high-quality annotated response) is available. Our setting is more common in practice, as SFT-style data is more widely available than preference pairs. Therefore, a direct comparison between self-play fine-tuning and preference optimization may not be appropriate. Nevertheless, we sincerely appreciate the reviewer's helpful suggestion, and we will include a comparison with pairwise preference optimization methods in the revised version.\n\n---\n\n**Q2**: Isn't the reference still playing a role in the loss function?\n\n**A2**: Thanks for the valuable question. We feel that there are some misunderstandings regarding the term '*reference'* . In our work, *reference* specifically refers to the *reference policy*. SPIN is considered reference-dependent because its objective (3) explicitly relies on a reference policy $\\pi_{\\theta_t}$. In contrast, T-SPIN is reference-free, as its loss function (7) depends solely on the current policy $\\pi_\\theta$ and **does not involve any other policies** during optimization. We will include a detailed clarification in the revised version.\n\n---\n\n**Q3**: The reward function in T-SPIN.\n\n**A3**: In SPIN, the reward function is defined as the log-ratio $c(x, y)=\\lambda \\log \\frac{\\pi\\_{\\theta}(y|x)}{\\pi_{\\theta_t}(y|x)}$ between the current model $\\pi_\\theta$ and a reference model $\\pi_{\\theta_t}$. According to (3), SPIN aims to maximize the reward for annotated samples while minimizing that for synthetic ones. However, in SPIN, the reward function used for training is not aligned with the metric $\\log p_{\\theta}(y|x)$ used by LLMs for generation. In other words, a response with a high reward does not necessarily correspond to a high generation probability, which has been verified in **the contingency table of Figure 3(a)** . In contrast, T-SPIN adopts the reward function $c(x, y)=\\alpha \\log  \\pi_{\\theta}(y|x)$ for training, matching that used for generation. As a result, samples with higher rewards are also more likely to be generated, as shown in **the contingency table of Figure 3(b)** . We sincerely thank the reviewer for raising the question and will include a clear explanation in the revised version.\n\n---\n\n**Q4**: An experiment comparing the variance of the training process.\n\n**A4**: Thank you for the constructive suggestion. In our paper, *stability* refers to the consistent improvement of model performance over iterations, without suffering from performance collapse. In contrast, the reviewer seems to refer to the stability of *the training process itself*, which differs from our intended focus. We will offer a detailed clarification in the revised version. Moreover, additional experiments about training process variance will also be included."}]

Author rebuttal:

Meta-review: Paper proposes a new self-play play loss which, in addition to promoting moving away from the current policy to the annotated policy (prior SPIN method), also adds an additional term which promotes moving away from the initial starting policy towards the current policy policy. Authors claim that this mitigates instability of SPIN self-play whose gradient vanishes as the model policy approaches annotated policy. Second, authors also claim that replacing KL divergence with entropy in the SPIN’s regularizer mitigates empirically observed misalignment between SPIN loss and true reward. Empirical results confirming these hypotheses are very promising.

Paper is well written and motivated by a practical problem. Several reviewers asked for more experiments, ablations and clarifications. Some of these were provided during rebuttal and improved the reviewer’s assessment. Primary short comings identified by the end of the review process was lack of deeper theoretical analysis of SPIN and T-SPIN, lack of significant improvement over SPIN and lack of comparison to many variants of SPIN available in literature.
Decision: Accept (poster)