SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models
Authors: Yibo Wang
Guangda Huzhang
Qing-Guo Chen
Zhao Xu
Weihua Luo
Kaifu Zhang
Lijun Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Language Models
Self-play Fine-tuning
Noise Contrastive Estimation
Primary area: deep_learning

Abstract: Self-play fine-tuning has demonstrated promising abilities in adapting large language models (LLMs) to downstream tasks with limited real-world data. The basic principle is to iteratively refine the model with real samples and synthetic ones generated from itself. However, the existing methods primarily focus on the relative gaps between the rewards for two types of data, neglecting their absolute values. Through theoretical analysis, we identify that the gap-based methods suffer from unstable evolution, due to the potentially degenerated objectives. To address this limitation, we introduce a novel self-play fine-tuning method, namely \underline{S}elf-\underline{P}l\underline{A}y via Noise \underline{C}ontrastive \underline{E}stimation (SPACE), which leverages noise contrastive estimation to capture the real-world data distribution. Specifically, SPACE treats synthetic samples as auxiliary components, and discriminates them from the real ones in a binary classification manner. As a result, SPACE independently optimizes the absolute reward values for each type of data, ensuring a consistently meaningful objective and thereby avoiding the instability issue. Theoretically, we show that the optimal solution of the objective in SPACE aligns with the underlying distribution of real-world data, and SPACE guarantees a provably stable convergence to the optimal distribution. Empirically, we show that SPACE significantly improves the performance of LLMs over various tasks, and outperforms supervised fine-tuning that employs much more real-world samples. Compared to gap-based self-play fine-tuning methods, SPACE exhibits remarkable superiority and stable evolution.

SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models
Yibo Wangl2,, Guangda Huzhang, Qing-Guo Chen?,
Zhao Xu', Weihua Luo', Kaifu Zhang', Lijun Zhang! 1421
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China School of Artificial Intelligence, Nanjing University, Nanjing, China 3 Alibaba International Digital Commerce *Pazhou Laboratory (Huangpu), Guangzhou, China
{wangyb, changlj]0l amla. .ju.nnu.....ee. {qingguo. cqg, changgong. xz, weihua. luowh, kaifu zkf Gal wwaiabbaac.a.....
edu.
Abstract
Self-play sinettuning has demonstrated promising abilities adapting large lan- guage models (LLMs) to downstream tasks with limited real-world data. The basic principle is to iteratively refine the model with real samples and synthetic ones generated from itself. However, the existing methods primarily focus the relative gaps between the rewards for two types of data, neglecting their absolute values. Through theoretical analysis, we identify that the gap-based methods suffer from unstable evolution, due to the potentially degenerated objectives. To address this limitation, introduce novel self-play fine-tuning method, namely Self- PlAy via Noise Contrastive Estimation (SPACE), which leverages noise contrastive estimation to capture the real-world data distribution. Specifically, SPACE treats synthetic samples as auxiliary components, and discriminates them from the real ones in binary classification manner. As result, SPACE independently optimizes the absolute reward values for each type of data, ensuring consistently meaningful objective and thereby avoiding the instability issue. Theoretically, we show that the optimal solution of the objective in SPACE aligns with the underlying distribution of real-world data, and SPACE guarantees a provably stable convergence the optimal distribution. Empirically, we show that SPACE significantly improves the performance L.LMs over various tasks, and outperforms supervised fine-tuning that employs much more real- world samples. Compared to gap-based self-play fine-tuning methods, SPACE exhibits remarkable superiority and stable evolution.
Introduction
In recent years. the success of large language models (LLMs) has drawn surge of attention from the
industry 2 4 22
47 55
[56], and also sparked extensive investigations across various fields in
academia, such as recommendation system [19
42 96. multimodality 4 36 44
45 48 reasoning
32 61 89]. code programming [34| 39 74 and beyond. One notable advantage of LLMs lies
their strong generalization capability, which allows them to perform well on downstream tasks supervised fine-tuning (SFT) on real-world data annotated by human experts (8).
It has been shown that effective handling of downstream tasks via SFT typically requires large amount of annotated data, which is often impractical in real-world applications due to the expensive costs of data curation and annotation [16] 46 82]. For this reason, pelf-pla fine-tuning is proposed,
"Wor done during the internship at Alibaba International Digital Commerce.
Lijun Zhang is the corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Iteration
Iteration
Iteration
Synthehc Dote Anmonstedd Deto
Spt Fonce SPIN Optiezzation force SPACE
Player
Player
Player
SPIN
Disonimintte
Diecniinens
Dismmintte
Annotated
Annotated Data
Annotated Data
Annotated Data
Dataset
Synthetic Dato
Synthetic Data
Synthetic Data
SPACE
Generate
Generate
Opponent Player
Opponent Player
Opporennt Player
(a) The framework of self-play finc-tuning
(b) Differences between SPIN and SPACE
The left part (a) depicts the framework of self-play fine-tuning. The right part (b) shows the difference in optimization forces between SPIN and SPACE, where SPIN maximizes the relative gaps between annotated and synthetic data, while SPACE optimizes two types of data independently.
Figure 1:
which takes advantage of supervision from LLMs themselves to expand the training set with synthetic samples [10 85 [0]. Specifically, self-play fine-tuning can be viewed two-player game where the model competes with itself for progressive evolution. This framework is illustrated in the left part of Figure ! where the opponent player seeks generate synthetic data that closely resemble the annotated samples, while the main player aims to discern real data from those generated by the opponent player. is worth noting that the main player and the opponent player refer the same model, but instantiated with the parameters from different iterations.
Under the two-player framework, the seminal work of Chen et al. [10] introduces the first self play fine- tuning method, termed SPIN, whose goal is maximize the relative gaps between the rewards of human-annotated responses y Pdatoo (-Ix) for the prompt x {()) and those of synthetic responses y' Pe, (x) generated from the LLM with the pa- rameter 6, However, it is observed that SPIN suffers unstable convergence during iterations 88 The- oretically, we attribute the instability issue to the op- timization on the potentially vacuous objective. For illustration, we consider an extreme case where the generated response y' closely resembles the real one (i.e., y' y). In this case, the relative gap closes and thus the objective of SPIN degrades to con- stant independent of Ó. In other words, any parameter e θ an optimal solution to the objective of SPIN. Our experimental results across various tasks from the HuggingFace Open LLM leaderboard reflect this phenomenon, where the performance of SPIN degrades after reaching its peak at iteration 2, as shown Figure
Average scores over diverse tasks (%)
49.0
DSPIN ISPACE
48.0
47.0
46.0
iterO
iterl
iter2
iter3
iter4
Figure 2: The average scores of SPIN and
SPACE at different iterations tasks from the HuggingFace Open LLM leaderboard.
To resolve this limitation, we introduce novel self- play fine-tuning method, namely Self-PlAy via Noise Contrastive Estimation (SPACE), which not only inherits the iterative evolution benefits of self-play fine-tuning but also ensures provably slable convergence to the real-world data distribution. SPACE inspired by Noise Contrastive Estimation (NCE) 26 27]. a classic technique that commonly used in unnormalized density estimation. Specifically, SPACE incorporates the self- generated responses as auxiliary components, and distinguish real responses from synthetic ones in a binary classification manner. In contrast to SPIN [10] that focuses the relative reward differences between real and synthetic responses, SPACE individually optimizes the absolute reward values for two types of responses. Therefore, even when the relative reward gaps diminish, the special design of SPACE ensures that the objective remains meaningful and does not collapse into a trivial constant. As result, SPACE avoids the instability issue. The above differences also illustrated in Figure (bb).
Theoretically, we prove that the optimal solution of the objective in SPACE aligns with the underlying distribution Paata of the real-world data, and SPACE ensures provably stable convergence to Pdnto- Empirically, our results on Mistral-7B [55] and Zephyr-7B [75] show that SPACE significantly im- proves the average performances over different tasks from the HuggingFace Open LLM Leaderboard

---

[6]. with up to almost 10 points increase on certain tasks, e.g., from 37.68% to 46.02% on GSM8K and from 23.63% 35.90% on IFEval. We also show that SPACE with only 50k real-world responses significantly outperforms SFT with 200k training samples. Moreover, we compare SPACE with SPIN and other gap-hased methods that are extended into the self-play fine-tuning framework, The results indicate that SPACE achieves superior performance and exhibits stable evolution during iterations.
The main contributions. In summary, this paper makes the following contributions: (i) We propose a novel self-play fine-tuning method SPACE, which is designed to deal with limited available expert- annotated data in downstream task adaptation for LLMs, and address the instability issue in self-play fine-tuning, (ii) We provide theoretical guarantees that ensure the provable and stable convergence of SPACE toward the optimal distribution (cf. Theorems 2 and (ii) We present results of extensive experiments that demonstrate the superiority of SPACE over existing tine-tuning strategies.
2
Related work
In this section, we briefly review two fine-tuning strategies for LLMs, í.e., supervised fine-tuning and self-play fine-tuning, and recent progress noise contrastive estimation.
Supervised fine-tuning. Supervised fine-tuning (SFT) is simple and effective strategy for adapting large language models to specific tasks with supervised data [16 82. The goal of SFT is learn the real-world data distribution Pdata (y (x) where x denotes prompt sampled from a task-specific distribution q()), and y denotes the corresponding high-quality response. Formally, the loss function
of SFT is formulated as:
(f) = -Ex=o() x (x) Po x]]]
can be verified that optimizing 1T) is equivalent to minimizing the KI. divergence between the distribution po and the target distribution Pdatos i.c., (L(paata ([x]aatt-lx))lpe((((((( (xx)))lp(((xx)). By the non- negativity of KL divergence, the optimal solution of (I) aligns with the real-world data distribution, i.e., Pe- (-1x) ( /x. Consequently, SFT offers principled way to capture the underlying distribution of high-quality responses through direct supervision from human-annotater data, Nev- ertheless, achieving the optimal solution requires extensive high-quality data, which significantly increases the annotation costs and limits the applicability of SFT in practice.
Self-play fine-tuning. Self-play framework, where the model iteratively improves the performance by competing against itself, is originally introduced in board games 688 73,, and subsequently
has found its great success in many fields (5] 40 41 (33. such as Al-player systems 770 71 and
protein discovery (78 155. Recently, Chen et al. 10 have introduced the self-play mechanism into large language models to address the limited high-quality data issue in SFT, and proposed the seminal self self-play mun-iuning method, named SPIN, which then inspires surge of research
88 92). The key idea to enlarge the training set with synthetic data generated by the model itself, and progressively refine the model by optimizing the relative differences between the real and synthetic samples. Mathematically, the iteration t ++ 1, SPIN aims to find parameter 0.+1 satisfying
IT 19 25]
65
66
81
85|
O.t1 argmnin (SPIN(0) = in E e (Allog Pa x) po, (x)x)
Po |x) po, (y' (x(x) )]
A log
(2)
dee
de@
where ((-) is a monotonically decreasing convex function, and the expectation is taken over x ~ q(-), y ~ Pdooaa-|xx and you ~ Po, (-xx). It can be observed that with iterative updates, the capability of the model gradually improves, leading higher-quality synthetic responses y'. This, in turn, renders the optimization on (2) inherently unstable during iterations. In particular, when the relative gap vanishes, the objective function LEPII degenerates into constant that is independent of the parameter 0. This implies that any parameter 0 in the space θ becomes an optimal solution of LSPIN, leading to unstable optimization or even performance collapse. Recently, there have been several efforts in the literature that aim to address the unstable convergence issue of SPIN. For example, [1] employ responses from a geometric mixture of historical distributions as synthetic ones in (2) to stabilize the optimization, and [88) propose to selectively filter generated samples for continual updates. Different from them, we introduce a novel objective function in self-play fine-tuning that decouples the optimization of real and synthetic responses, and establish theoretical guarantees for the achievability and maintainability
3

---

for the underlying distribution of real-world data. As a result, our SPACE not only inherits the benefits of self-play fine-tuning i.e., effectively fine-tuning LLMs with small amount of annotated data, but also ensures provable and stable convergence to the optimal distribution.
Noise contrastive estimation. Noise contrastive estimation (NCE) is initially proposed for density estimation in probabilistic learning 26 [7], and then has emerged as a powerful optimization
technique in widespread applications recently 7 [2] 31
33
52 54 98 The basic idea 49 51
of NCE is to formualte the density estimation as a binary classification task, and discriminate the target distribution from an additionally introduced noise distribution. recent years, NCE has also drawn significant attention in large language models, including preference alignment [8], information retrieval [77] and recommendation systems [87]. To the best of our knowledge, this paper is the first work that incorporates the spirit of NCE into self-play fine-tuning for large language models.
3 Method
In this section, we first introduce the proposed SPACE, including the update rules for main player and opponent player, and the overall objective. Then, we deliver theoretical guarantees for SPACE.
3.1 Our method: SPACE
Our method starts from a pretrained model po,, which subsequently fine-tuned on set of high- quality data annotated for specific downstream tasks. Our SPACE is built upon the self-play framework, where a main player and an opponent player compete with each other for progressive evolution. In this framework, both the main player and the opponent player are the same model, but equipped with different parameters. Specifically, at each iteration + 1, the opponent player parameterized by 0, generates the synthetic response y' for prompt x by sampling from the distribution pe, (-1x). Then, the main player takes the buman-annotated response y and the synthetic one y' as inputs and updates its parameters to obtain 0z+1. Subsequently, the opponent player adjusts its parameters so that it can generate synthetic responses for the next iteration. Overall, this process includes two steps: the updates for the main player and the opponent player, which separately detailed in the following.
Updates for the main player. Recall that the goal of the main player is to discriminate the human- annotated responses and the synthetic ones. Drawing inspiration from noise contrastive estimation [26 27,, we formulate the discrimination as a binary classification problem, where the main player serves as a classifier to distinguish between the human-annotated responses y and the synthetic responses y'. This formulation shares the same spirit with differentiation theory in cognitive development (21]. where learning emerges through progressive extraction of meaningful patterns from environments. Formally, we first employ the commonly used log ratio [2] as the reward for a response u: r(ulx) =
log po (u|x) - log Po, (ulx), where 0 and 6, denote parameters of main player and opponent player, respectively. Then, we establish the relationship between the rewards for two types of responses (i.e., y and y'), and their corresponding posterior probabilities, shown below.
Proposition 1. Let Po be the real data distribution, and r" (-(x) = log Po- (-1x) log Pé. (.1x) be the reward for pa-. Then, given a mixture distribution Penia (y (y|x)= (1 p1)- ppo (y|x) + 111
p (ylx) with the ratio μs, the posterior probabilities of a sample (x, from Praiz are
po- (c = 11y,y) 1+ (xxp(- po- (o- 0 1 μ -1 (exp(r*(y|x))
(3)
where e is the label indicating whether y is real (c 1) synthetic (c= 0).
The above proposition reveals that the optimal reward captures the posterior probability of response, thereby establishing a principled connection between the reward function and binary classification over a mixture distribution. To be precise, a higher reward r(y| (x) corresponds to a higher probability po(c y. x), indicating that the response y is more likely to be real; conversely, lower reward implies a higher probability pa(c = Oly, x) and a less realistic response. Therefore, with Proposition we can conveniently leverage the maximum likelihood strategy train the main player find I
parameter 0,41 that can distinguish between real and synthetic samples:
0 1++1 argmax Tog x)y,x) + + log po = Oly', x))
d50

---

Algorithm/ Self-PlAy via Noise Contrastive Estimation (SPACE)
Inputs: Annotated (xi, yi) the generation ratio μ., and a pretrained LLM peo
Initialization: Set Pá. = P... and compute the size of synthetic data m jm
for 0, 1, 2, ... do
2: Generate the synthetic tuples Yx [==1
by the opponent player po., i.e, N'y po, (-12g)
3: Obtain po,+1 by minimizing the loss function (8)
4: end for
where the expectation is taken over the distributions x g(.),y ~ Pdataa-(xx and y' ~ Pā. (1x))
Substituting (3) into 4 delivers
02+1 argmin E log on log po(y)x) lE0
-lllogo on-1 (10g Po xx)
Pd. (x'1x)
(5)
Po. (y x)
where (x) (1 + μ exp((=x))- Compared to the objective of SPIN in 2 ( separately
optimizes the rewards of the real responses and the synthetic ones. The separate structure ensures that, when y' closely resembles y. (5) does not degenerate into a constant independent
As a result, 5 guarantees provably stable convergence, which will be theoretically elaborated later.
optimizing
Updates for the opponent player. Given the newly obtained main player P8 we proceed to update the opponent player, the goal of which is to generate high-quality synthetic responses that can mislead the main player. To achieve this, natural intuition is to adjust the opponent player by maximizing the reward associated with synthetic responses, which is formulated as follows:
0e+1 argmax E (x)] = argmax E [log pel.1 (y'Ix) log (og))(y'
(6)
ôge
dce
where the expectation is taken over the distributions x q(-) and y' ~ p=(+ Mathematically, (6) measures the KL divergence between Pa and po. Consequently, by the non negativiit of KL divergence, we can obtain that the closed-form solution of (6) takes the form of
P (y xx po.+1 (y)x).
The above finding is particularly appealing, as it indicates that the updated parameter of the main player coincide exactly with the optimal solution for the opponent player. Consequently, we can directly set the parameter of the opponent player for the next iteration be the same as that of the main player, i..., Po.+1 Po... which inherently reveals the self-play nature of our SPACE.
The overall objective. Now, we are ready to introduce the overall objective of SPACE, by combining the above two update steps together. In details, we substitute into (5). and deliver the following objective function:
ESPACE (0)= = -E log oN (log Po(y|x)) x))
+ log a (log Po, Po, (y'1xx) ]
(8)
Pe, (y|x)
po(y'|x)
We summarize the detailed procedure of SPACE in Algorithm [ Specifically, the begining, initialize the opponent player with the pretained L.LM, i.e., Pa. Pa. and compute the size of synthetic data m = um. At each iteration f, the opponent player generates the synthetic response y' Pa. (-1x) for each prompt Then, the main player takes annotated and synthetic data as inputs, and obtains pa 1 by minimizing (8). Subsequently, the opponent player copys 0,+1 as its parameter according to to produce new synthetic responses for the next iteration.
In-depth discussions. In the following, we analyze the underlying cause of instability issue in SPIN, and explain how SPACE addresses this issue and achieves stable convergence. As shown in (2). SPIN aims to optimize the gaps bewteen the annotated responses and synthetic ones. In fact, the gap-based structure is vulnerable for self self-play fine-tuning. Specifically, with the refinement of synthetic responses, the gaps between real and self refereeeeeted data gradually close, and the gap-based objective will degenerate the parameter space can be considered optimal, leading to the instability issue. Moreover, if existing gap-based methods for large language models, such as IPO [ and SimPO [50], are extended to self-play fine-tuning scenarios, they similarly suffer from the instability issue, due to the inherent
trivial constant when the gap vanishes. this case, any solution in
5

---

weakness of gap-based objectives. More detailed introduction about gap-based extensions can be found in Appendix [ In contrast, our proposed SPACE circumvents this issue by decoupling the optimization of real and synthetic responses: instead of maximizing their relative difference, SPACE independently optimizes the absolute reward value for each, thereby maintaining a stable evolution.
3.2 Theoretical analysis
In the following, we provide theoretical analysis of SPACE to understand its underlying principles. We begin by presenting the gradient analysis for the objective function &
in SPACE.
Theorem 1. The gradient of (8) for a response u takes the form of
V V TeCsPAc((O) = -xx-(()) [- (P)
pe, log (9)
Remark. The above theorem indicates that the update rules in SPACE enjoy a desirable "response- dependent" property, i.e., the likelihood of a response u is adjusted based on itself. Specifically, we can examine this property by considering two types of responses: y ~ Pdaata (-Ix) sampled from the real-world distribution Pdato and y' ~ Pe, pe. (-Ix) generated by the model pa, Note that (-) > 0 holds always true for f O. Therefore, the former case where Pdata (y(x) pe, (y|x) 2 0 leads to an increase in the log probability of For the latter case where Pdato (y' Pe, (y' xx) S 0, the log- probability of y' will be decreased. This favorable property ensures that SPACE can effectively distinguish annotated responses and synthetic responses by adaptively adjusting their probabilities.
Remark. Note that although (9) shares similar structure with the policy gradient in reinforcement learning (RL), our SPACE exhibits fundamental differences from reinforcement learning methods. Specifically, in the standard policy gradient formulation (e.g., REINFORCE (84)), the gradient takes the form of (() = -En-p [R(ulx) Va In pa ( (u x)]. where R(ulx) denotes the reward associated with response Superficially, (9) can be viewed as special case of this formulation by choosing R(ulx) = -x-1 (-r(x, u))
[u|x) (U/x) pe, (u]x)). but it is important to note that SPACE based on the self-play fine-tuning framework, where the goal to iteratively improve performances trade- off in RL. Additionally, SPACE optimizes (8) in a sepeerissdd-earningg manner, without involving explicit rewarding or rollout phrases, which are essential components of standard RL methods.
through two-player competition. This fundamentally differs from the opploratoonexpplotaaae
Then, we demonstrate that SPACE enjoys the achievability and maintainabilliy for the annotated data distribution Pántas as established in the following two theorems.
Theorem 2. By minimizing the loss function distribution Pdata, i.e., Po- = Pdata. where 0r denotes the optimal solution to (8)
g SPACE is able capture the annotated data
Theorem 3. Suppose pa, has already converged to the annotated data distribution Pdato iteration t, i.e. pe, (-Ix) = P-atta-lxxx. Then, at iteration t + 1, SPACE still ensures Pe.., (-Ix) = Pdato (-Ix).
Remark. Theorem establishes the achievability for the target distribution Pánto i.e., it guarantees that minimizing the objective function (8) yields distribution that aligns with Pdata- This result provides theoretical support for SPACE, demonstrating its ability to align with the target distribution, which is particularly important in fine-tuning LLM for downstream tasks.
Remark. Theorem further emphasizes the maintainability for the target distribution Pdatos which be regarded
complement the achievability property in Theorem 2 highlighting the long-term stability for Pdata- Notably, this stable convergence property is also critical for self-play fine-tuning methods, as it ensures that the iterative optimization process retains the optimal solution without divergence, thereby mitigating the instability issue during iterations. This appealing property theoretically exhibits the advantages of SPACE over SPIN [10]
Experiments
In this section, we present empirical studies to validate the effectiveness of SPACE, We first describe experimental settings, including datasets, pre-trained models, implementations and evaluations. We then report the results with corresponding analyses. More experiments are provided in Appendix C

---

Table 1: Performance (%) comparisons various tasks among SPACE (red). SPIN, S-IPO and S- S-SimPO. Arg denotes the average scores over iterations 0 to 4 are highlighted in bold and underline, respectively.
different tasks, where highest and second- highest
Model
ARC
GSM8K HellaSwag MMLU TruthfulQA Winogrande IFEval
BBH
GPQA MMLU pmu
Avg
Mistral-71 7
61.01
37.68
83.24
57.86
42.62
74.03
23.63
44.26
29.86
29.99
48.42
IterO
60.49
40.96
83.58
57.17
45.45
73.88
20.85
42.33
30.71
30.01
48.54
S-10)
Iterl
60.07
38.91
83.47
57.29
43.62
73.80
21.98
42.08
30.19
29.90
Iter2
60.07
39.35
83.19
57.29
43.35
73.80
23.29
41.69
30.50
29.86
48
Iter3
60.67
41.05
83.42
57.03
44.35
74.27
23.00
41.73
29.32
29.69
48.45
Iter4
61.43
40.11
83.56
56.61
45.56
74.11
23.11
41.05
29.31
29.36
48.42
IterO
61.35
39.67
83.76
57. 37.29
47 47.44
74.66
24.88
41.86
30.10
30.04
49.11
S-SimPO
Iterl
61.69
41.51
83.77
57.48
48.98
74.90
23.01
41.86
29.90
30.22
49.33
Iter2
61.52
37.09
83.77
57.16
49.58
74.51
22.44
43.04
29.17
29.97
48 88.88
Iter3
61.26
35.22
83.70
57.35
49.88
74.82
21.15
42.80
29.04
30.01
48 52
Iter4
61.35
34.38
83.68
57.48
50.02
74.19
21.31
42.67
29.16
39.40
49.36
IterO
61.18
38.29
83.49
57.88
43.73
74.11
22.09
44.60
29.62
30.15
48.55 51
Iterl
61.52
32.85
84.02
57.32
47 47.69
73.95
21.92
40.59
28.22
30.12
47 .28
SPIN
Iter2
62.20
40.26
83.60
58.03
46,92
74.43
25.06
43.50
28.95
30.37
49.33
62.29
34.87
84.01
58.09
45.72
75.14
24.64
43.03
28.68
29.62
Iter4
61.86
34.70
83.99
57.99
46.00
75.14
25.73
43.34
27.10
29.82
48.57
SPACE (ours)
62.71
41.32
83.79
47.15
74.90
26.96
45.87
29.66
30.73
50.18
Iterl
64.85
45.41
83.86
58.87
48.99
75.14
31.51
45.73
29.46
30.94
51.48
Iter2
65.78
45.55
84.25
58.57
50.71
73.95
33.01
45.38
29.52
31.59
51.83
Iter3
65.96
45.84
84.39
58.53
51.34
74.19
33.19
45.38
29.90
31.44
52.03
Iter4
65.87
46.02
84.44
58.50
51.86
74.51
35.90
45.28
30.50
31.41
52.43
$3.0
IS-IPO
mS-SimmP)
mS-1PO
mS-SimPP) IIPAAC
OSPIN
ISPACE
47.0
DSPINN
S-IPO
Avcrage Score
51.0
Scnre (%))
43.0
S S-SimPO
2.9
1.9
3.0
3.6
3.2
39.0
49.0
SPIN
3.3
3.8
2.1
2.7
3.0
35.0
SPACE
3.8
3.3
3.1
iterU
Ster2
Her3
iteed
iterÜ
ieerl
iter2
Her3
iterd
iterO
iterl
iter2
iter3
iterd
(a) Average
(b) GSMSK
(c) Average rank
Figure 3: The performance comparisons among four self-play fine-tuning methods on Mistral-7B. (a) the average scores over different tasks; (b) the performances on GSM8K; (c) the average ranks over
different iterations, where the best rank among iterations ( to 4 is highlighted with a "gold medal"..
4.1 Experimental settings
Following Chen et [0], we randomly sample 50k prompts with their corresponding high-quality responses from the Ultrachat200k dataset [15), and choose Zephyr-7B-SFT-ful [75] and Mistral- 7B-Base 35 as pretrained models experiments. During the training, we first generate synthetic response y' for each x with the latest model at each iteration. The resulting synthetic response is then combined with annotated one to update the large language model for the subsequent iteration.
We evaluate the performances with different tasks from the HuggingFace Open LLM Leaderboard 18) each targeting distinct capability of LLMs. These tasks cover range of domains: science question answering with ARC-Challena [2] and GPQA [64), mathematical reasoning with GSM8K
13. commonsense inference with Winogrande (67 and HellaSwag [93], multitask language under- standing through MMLU [30] and MMLU-Pro 83 truthfulness and factuality with TruthfulQA
45
instruction following using [FEval 977 and complex reasoning with BBH [72]. All tasks are
implemented with the default configurations provided by the Language Model Evaluation Harness Our implementation is based on the codebase Alignment Handbook (76 and the Accelerate
library
23
We choose RMSProp (69 with default configurations the optimizer, and set the
7

---

ARC Score (%)
GSM8K Score (%)
IFEval Score (%)
Average Score (%)
66.0
46.0
33.0
52.0
30.0
51.0
64.0
43.0
27.0
50.0
62.0
40.0
--eerd
24.0
-iter
49.0
-itert
--iterl (2 spochs)
-ierl (2 opochs)
-terl (2 apochs)
-ttrr (2 spochs)
60.0
37.0
21.0
48.0
1
2
3
4 Epoch
0
3
0
2
3
5 Epoch
2
3
4
Epoch
Epoch
Figure 4: Performance comparisons between the model trained with multiple epochs at iteration 0 (green) and that trained with two epochs at iteration (red).
global batch size as 64 and the epoch as 2. For SPACE, we choose to set the generation ratio μ = 1 (i.e., m = n2) in our experiments, and we will return to this configuration later.
4.2 Experimental results
Comparison to self-play fine-tuning baselines. We compare SPACE with SPIN [0), as well as
self-play variants of two gap-based methods, namely S-IPO from IPO 3 and S-SimPO from SimPO
[50]. The detailed descriptions for two self-play variant methods are provided in Appendix B
The experimental results Mistral-7B are shown in Table I and we defer the results Zephyr-7B to Appendix C The results from Table Jindicate that all self-play methods surpass the initial base model with notable improvements. Specifically, our SPACE improves the average score of Mistral-7B from 48.42% to 52.43%. Remarkably, it achieves substantial performance gains on GSM8K and IFEval with almost 10 points improvements. Additionally, compared to other self-play baselines, our SPACE demonstrates superior performance and stable evolution. For clarity, we present the average scores over different tasks for each method in (igure(aa) We observe two key findings: (i) the performance improvements predominantly occur within the first two iterations, which aligns with the conclusions of Chen al. (TO): (ii) SPIN well as the self-play variants of IPO and SimPO suffers degradation after reaching their performance peak. Specifically, for the GSM8K task shown in Figure (b).. SPIN, S-IPO and S-SimPO show improvements at the initial iterations but incurs performance decline after that. In contrast, our SPACE maintains its stable improvements over iterations. Moreover, we also compare the average ranks over iterations, as shown in Figure B c). The results indicate that three baselines achieve their best performance in early iterations, whereas SPACE maintains steady improvements over iterations, ultimately achieving the best rank at the final iteration.
Comparison to training with more epochs. Then, we investigate the effectiveness of the self-play mechanism in SPACE. We conduct experiments on Mistral-7B by comparing the model trained for multiple epochs at iteration 0 with the fixed synthetic responses, versus the model trained on re-generated synthetic responses at iteration 1 with two epochs. The results are illustrated in Figure 4 We observe that training with multiple epochs at iteration 0 initially improves performance but gradually plateaus, and fails to surpass the performance achieved at iteration We consider the performance bottleneck primarily to the stagnation of negative samples. Specifically, at iteration 0, despite minor performance improvements, the model undergoes multiple rounds of training on an unchanged dataset. In contrast, at iteration 1, we first refine the training data by leveraging the self-play mechanism to regenerate synthetic samples with the latest model, and then train the model on the enhanced data, delivering substantial performance gains. This observation demonstrates that improving the quality of synthetic samples plays a critical role in enhancing model performance, validating the effectiveness of the self-play mechanism in our SPACE.
The generation ratio. Next, we examine the impact of the generation ratio μ., which quantifies the proportion of synthetic responses relative to numan-annotalie ones, on the performance of SPACE. Specifically, we conduct experiments on the Mistral-7B with different generation ratios from (1.0,3.0 7.0]. For 14 1.0, we generate 50k synthetic responses for the 50k human-annotate responses in the training dataset at each iteration. For μ. = 3.0 and μ == 7.0, we generate 150k and 350k synthetic responses respectively, while maintaining the same 50k numan-annotaan
responses.
The experimental results are shown in Table Overall, increasing the number of generated samples can improve the performance of SPACE, though this improvement gradually diminishes in later iterations. Additionally, important to note that such performance gains come at the cost of significantly higher computational resources. For instance, when μ 3.0, the generation phase
8

---

Table Performance (%) comparisons with different generation ratios of SPACE accross iterations.
Avg denotes the average score over multiple tasks, and Gen Ratio denotes the generation ratio.
Gen Ratio|
ARC
GSM8K HellaSwag
MMLU TruthfulQA
Winogrande
IFEval
BBH
GPQA MMLU
Avg
pco
Mistral-7B
61.01
37.68
83.24
57.86
42.62
74.03
23.63
44.26
29.86
29.99
48.
1.0
62.77
41.32
83.79
58.67
47.15
74.90
26.96
45.87
29.66
30.73
50.18
Itrr U
3.0
62.97
42.34
83.25
58.70
47.52
74.98
36,05
47.00
29.27
30.39
51.25
7.0
63.48
41.26
82.79
58.55
46.65
74.90
36.93
46.37
29.78
29.85
51.06
1.0
64.85
45.41
83.86
58.87
48,99
75.14
31.51
45.73
29.46
30.94
51,48
Iter 1
65.10
43.03
83.74
58.71
50.95
73.88
36.44
46.24
29.65
30.31
51.80
7.0
64.99
41.32
83.37
58,49
48.37
73.32
40.64
46.61
29.33
30.29
51.67
65.78
45.55
84.25
58.57
50.71
73.95
33.01
45.38
29.52
31.59
51.83
Iter
3.0
64.76
41.39
83.67
58.51
48.89
73.24
41.93
46,20
29.20
30.48
51.83
7.0
64.59
41.49
83.63
58.50
49.23
73.56
40.33
46. 22
29.69
30.29
51.75
Reward of gencrated responses
Reward annotated responses
Margins
30
SPIN SPACE
Value
-10
Value
SPIN SPACE
Value
20
-20
10
SPIN
SPACE
-30
25
50
100
25
50
75
100
25
100
Steps (%)
Steps (%)
Steps (%)
Figure 6: Comparison of rewards for generated and
responses, and the correspond-
ing gap between two types of responses, for SPIN and SPACE at iteration
requires 4.50 hours and the training phase requires 6.80 hours; when μ = 7.0, these increase to 6.20 hours and 12.23 hours respectively, substantially higher than the computational costs for μ 1.0 (0.83 hours for generation and 3.20 hours for training). Because the substantial computational costs, choose to μ 1.0 rather than larger generation ratios in our experiments.
Comparison SFT. In this part, we compare SPACE with SFT demonstrate that SPACE with less human-annotated annotated data can achieve better performance than SFT that utilizes much more data.
The experiments are conducted Mistral-7B (35] with varying amounts of human-annotated data. Specifically, we train SPACE on nested subsets of 12k, 25k. and 50k: samples from Ultrachat200k [15,, with each larger subset encompassing the smaller ones. Since the performance improvements primarily occur within the first two itera- tions, we train SPACE for two iterations on each subset. For SFT, we train the model on the subsets of 100k sam- ples from Ultrachat200k, the complete dataset with 2001: data, and an expanded version with 400k samples where each sample duplicated once. We display the average scores of SPACE and SFT in Figure 5) It is observed that both SPACE and SFT significantly improve model perfor- mance compared to the initial base model. However, the amount of training data increases from 100k to 400k, the performance gains of SFT gradually reach plateau, suggesting that additional human annotated responses provides limited benefit for further performance improvement. In contrast, SPACE achieves superior responses, compared to SFT trained on a much larger
SFT
$1.48
SPACE
51.0
50.34
Average Score
50.29
50.48
50.0
50.05
49.0
48.0
//
01225
100
400
# of ffuman-Anmotated Data (K)
Figure 5: sizes of annotated data. The start point denotes the performance of base model.
The average score with different
human-annotated
performance with fewer dataset, demonstrating its effectiveness in scenarios with limited human supervision.
Computational costs. In SPACE, the main computational costs come from the generation of synthetic responses and the model training. In implementations, we follow Chen et al. [10] utilizing the

---

Table 3: The computation time of generation and training for different iterations. Gen and Train denote the generation and training phases, respectively.
Iteration
Iter 0
Iter 1
Iter 2
Iter 3
Iter 4
Phase
Gen
Train
Gen
Train
Gen
Train
Gen
Train
Gen
Train
Time
0.83h
3.20h
0.83h
3.20h
0.83h
3.20h
0.83h
3.20h
0.83h
3.20h
Accelerate library [23] to generate synthetic responses in a distributed manner with the global batch size 256. All experiments are conducted on a single machine equipped with 8 HI00 GPUs, and we report the costs of generation and training in Table During the generation phase, we produce 50k synthetic responses for the input prompts and suffer the time cost of 0.83 hours per iteration. During the training phase, we train the model with 2 epochs with the total of 3.20 hours per iteration. It is observed that the majority of computational costs are concentrated in the training phase.
Preventing from reward decline for human-annotated responses. Finally, we discuss the reward decline issue for high-quality responses, which have raised wide investigations in preference align- ments 17 55 63 [6]. but has not yet been discussed in self-play fine-tuning. According to ( SPIN aims maximize the relative gap between the rewards of annotated response and self-generated one. In experiments, we observe that the reward margins between real and synthetic data increase (right of Figure 6). but this improvement comes from the different rates of decline between the two types of responses. In fact, the decreasing reward for human-annotated responses (middle of Figure 6 is undesirable, as it explicitly reflects the decline in the priority of human- annotated responses. In other words, the capability to generate high-quality responses gradually deteriorates [91].
It is worth noting that SPACE avoids the reward decline for real responses, and exhibits an increasing trend (middle of Figure 6). This can be attributed to the inherent structure of the objective function in (8) which explicitly increases the reward for human-annotated responses while decreasing it for self- -e-geeeerredd ones. Moreover, the favorable response-des deppedeede property (Theorem I also shows that SPACE can naturally support distinct update trends for two types of responses.
5 Conclusion and future work
In this paper, we investigate self-play fine-tuning for large language models, and introduce a novel method named SPACE to resolve the instability issue of gap-based methods. The pivotal idea of SPACE is to treat synthetic samples as auxiliary components and distinguish them from real ones in a binary classification manner. In this way, SPACE is able to optimize the absolute reward values for annotated and synthetic samples independently, avoiding the unstable convergence. Theoretically, we show that SPACE enjoys the favorable properties, i.e., the achievability and maintainability for the real-world data distribution Patta- In other words, SPACE can provably converge to Prtaa- and maintain it during iterations, ensuring stable evolution. Empirically, extensive experiments demonstrate that SPACE not only outperforms SPIN and other gap-hased self-play baselines, but also achieves superior performance to SFT, while utilizing substantially less annotated data.
There are many directions for future research. First, in existing self-play fine-tuning methods, the total number of iterations is hyper-parameter that has to be configured in advance, which may lead to unnecessary computation or premature convergence. In the future, we will investigate self-play fine- tuning methods that can adaptively adjust the number of iterations based on the evolution progress. Second, the current methods including SPIN [T0] and our SPACE are designed for the scenario where the optimal distribution is fixed during iterations. which, however, is not necessarily satisfied in practice. Therefore, it also interesting to explore self-play fine-tuning in the non-stationary scenario, which may require the advanced techniques from online optimization (28] 57 779 80 94]. Third, we also consider to apply SPACE to complex scenarios with limited annotated data, such as LLM agents.
Acknowledgments and Disclosure of Funding
This work was partially supported by NSFC (U23A20382), and the Collaborative Innovation Center of Novel Software Technology and Industrialization.
10

---

References
[1] Reda Alami, Abdalgader Abubaker, Mastane Achab, Mohamed Amine Seddik, and Salem Lahlou. Investigating regularization of self-play language models. ArXiv e-prints, arXiv:2405.00675. 2024.
[2] Anthropic. Claude 3.5 sonnet. http://///////.aaaaoao/.//////// 2024.
[3] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. Proceedings of the 27th International Conference on Artificial Intelligence and Statistics, pages 4447-4455, 2024.
[4] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. ArXiv e-prints, arXiv:2308. 2023.
[5] Yu Bai and Chi Jin. Provable self-play algorithms for competitive reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, pages 551-560, 2020.
[6] Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open LLM leaderboard (2023-2024). battps://huggingg http://ngnaccccssscsssse//// Seteaces/open-nneee 11m- http://ces...ppaa....aeeeeeeeeee lem_eeedebboard 2023.
[7] Omar Chehab, Aapo Hyvarinen, and Andrej Risteski. Provable benefits of annealing for estimating normalizing constants: Importance sampling, noise-contrastive estimation, and beyond. In Advances in Neural Information Processing Systems 36, pages 45945 2023,
[8] Huayu Chen, Guande He, Lifan Yuan, Ganqu Cui, Hang Su, and Jun Zhu. Noise contrastive alignment of language models with explicit rewards. In Advances in Neural Information Processing Systems 37, 2024.
[9] Sijia Chen, Yibo Wang, Yi-Feng Wu, Qing- Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Lijun Zhang. Advancing tool-augmented large language models: Integrating insights from errors in inference trees. In Advances in Neural Information Processing Systems 37, pages 106555-106581, 2024.
[10] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine- tuning convertsweak language models strong language models. Proceedings of the 41st International Conference on Machine Learning, pages 6621-6641 2024.
[11] Pengyu Cheng, Yifan Yang, Jian Li, Yong Dai. Tianhao Hu, Peixin Cao, Nan Du, and Xiaolong Li. Adversarial preference optimization: Enhancing your alignment via RM-LLM game. In Findings of the 62nd Conference of the Association for Computational Linguistics, pages 3705-377 1016, 2024.
[12] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the A12 reasoning challenge. ArXiv e-prints, 1.XXv:1803.00551 2018
[13] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv e-prints, 1cXi:2110.144166 2021.
[14] DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, and Ziyi Gao et al. DeepSeek-R1: Incentivizing reasoning capability in LLMss via reinforcement learning. ArXiv e-prints, arXiv:2501.12945 2025.
11

---

[15] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin. Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou, Enhancing chat language models by scaling high-quality instructional conversations. ArXiv e-prints, arXiv:2305. Seti::230..12333, 2023.
[16] Guanting Dong. Hongyi Yuan, Keming Lu, Chengpeng Li, Dayiheng Liu Mingfeng Xue, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. How abilities in large language models are affected by supervised fine-tuning data composition. In Findings of the 62nd Conference of the Association for Computational Linguisties, pages 177-198, 2024.
[17] Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. Towards ana- lyzing and understanding the limitations of DPO: A theoretical perspective. ArXiv e-prints, arXiv:2404.04626, 2024.
[18] Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open LLM leaderboard v2. https:/
11m-leaderboard)
open leaderboardd 2024.
[19] Chongming Gao, Ruijun Chen, Shuai Yuan, Kexin Huang, Yuanqing Yu, and Xiangnan He. SPRec: Leveraging self-play to debias preference alignment for large language model-based recommendations. ArXiv e-prints, arXiv:2412.092433 2024.
[20] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf. Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou, A framework for few-shot language model evaluation, 07 2024.
[21] Eleanor J. Gibson. Principles of perceptual learning and development. Pearson College Div, 1969.
[22] Google. Our next-generation model: Gemini 1.5. http::://bbblgggggggo//coooo//// google/technooogg/ai google-geming -eatinn--ext-een new-- gwweratnnnnniooo-n--onn -orooooooor-----r----20000-00
2024.
[23] Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Mare Sun, and Benjamin Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable., 2022.
[24] Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y.K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. DeepSeek Coder: When the large language model meets programming the rise of code intelligence. ArXiv e-prints, arXiv:2401. 14196, 2024.
[25] Hongyi Guo, Yuanshun Yao, Wei Shen, Jiaheng Wei, Xiaoying Zhang, Zhaoran Wang, and Yang Liu. cluman-instruction free LLM self-alignment with limited samples. ArXiv e-prints, #rXiv:2401.06782 2024.
[26] Michael Gutmann and Aapo Hyvarinen. Noise-contrastive estimation: A new estimation princi- ple for unnormalized statistical models. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, pages 297-304, 2010.
[27] Michael Gutmann and Aapo Hyvarinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of Machine Learning Research, 13:307-361, 2012.
[28] Elad Hazan. Introduction to online convex optimization. Foundations and TrenRds in Optimiza- tion, 2(3-4):157-325, 2016.
[29] Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In Proceed- ings of the 37th International Conference on Machine Learning, pages 4182-4192, 2020.
[30] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the 9th International Conference on Learning Representations, 1021.
12

---

[31] R Devon Hjelm, Alex Fedorov, Samuel Lavoie- Lavoie-Marchildee Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. ArXiv e-prinis, arXiv: 108.888..677777 2018.
[32] Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. In Findings of the 61st Conference the Association for Computational Linguistics, pages 1049-1065, 2023.
[33] Johannes Hugger and Virginie Uhlmann. Towards noise contrastive estimation with soft targets for conditional models. ArXiv e-prints, arXiv:2404.1407 14076, 2024.
[34] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang. Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xu- ancheng Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-Coder technical report. ArXiv e-prints, htt::2209.12186.. 2024.
[35] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mistral 7B. ArXiv e-prints, arXiv:2310.06825, 2023.
[36] Jun-Peng Jiang, Yu Xia, Hai-Long Sun, Shiyin Lu, Qing-Guo Chen, Weihua Luo, Kaifu Zhang, De-Chuan Zhan, and Han-Jia Ye. Multimodal tabular reasoning with privileged structured information. ArXiv e-prints, arXiv:2506.04p 000088 2025.
[37] Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. ArXiv e-prints, arXiv:2406.005: 2024.
[38] Xiaohu Jiang, Yixiao Ge, Yuying Ge, Dachuan Shi, Chun Yuan, and Ying Shan. Supervised fine-tuning in turn improves visual foundation models. ArXiv e-prints, arXiv:2401.10222, 2024.
[39] Heiko Koziolek, Sten Grüner, Rhaban Hark, Virendra Ashiwal, Sofia Linsbauer, and Nafise Eskandani. LLM-based and retrieval- -uunenel control code generation. In Proceedings of the 1st International Workshop on Large Language Models for Code, pages 22-29, 2024.
[40] Mare Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Perolat, David Silver, and Thore Graepel. A unified game- game-theorethe theoretic approach to multiagent reinforcement learning. In Advances in Neural Information Processing Systems 30, pages 4190-4203, 2017.
[41] Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1192-1202 2016.
[42] Jianghao Lin, Xinyi Dai, Yunjia Xi, Weiwen Liu, Bo Chen, Hao Zhang, Yong Liu, Chuhan Wu, Xiangyang Li, Chenxu Zhu, Huifeng Guo, Yong Yu, Ruiming Tang, and Weinan Zhang. How can recommender systems benefit from large language models: A survey. ACM Transactions on Information Systems, 43(2), 2025.
[43] Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Conference of the Association for Computational Linguistics, pages 2212-2252, 2022.
[44] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems 36. pages 34892-3444 14916, 2023.
[45] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. L.eTV--NeTT: Improved reasoning, ocr, and world knowledge. htttps///lava-vll thub io ho/blog/200 2024 30- llava- 2024.
13

---

[46] Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection instruction tuning. In Proceedings of the I 12th International Conference Learning Representations, 2024.
[47] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. DeepSeek-VL. Towards real-world vision-language understanding. ArXiv e-prints, AXXiv24403005525 2024.
[48] Shiyin Lu. Yang Li. Yu Xia, Yuwei Hu, Shanshan Zhao, Yanqing Ma, Zhichao Wei, Yinglun Li, Lunhao Duan. Jianshan Zhao, Yuxuan Han, Haijun Li, Wanying Chen, Junke Tang, Chengkun Hou, Zhixing Du, Tianli Zhou, Wenjie Zhang, Huping Ding, Jiahe Li, Wen Li, Gui Hu, Yiliang Gu, Siran Yang. Jiamang Wang, Hailong Sun, Yibo Wang, Hui Sun, Jinlong Huang, Yuping He, Shengze Shi, Weihong Zhang, Guodong Zheng, Junpeng Jiang, Sensen Gao, Yi-Feng Wu, Sijia Chen, Yuhui Chen, Qing-Guo Chen, Zhao Xu, Weihua Luo, and Kaifu Zhang. Ovis2.5 technical report. ArXiv e-prints, arXiv:2508. 1737, 2025.
[49] Stefan Matthes, Zhiwei Han, and Hao Shen. Towards unified framework of contrastive learning for disentangled representations. In Advances in Neural Information Processing Systems 36, pages 07459,67777,, 2023.
[50] Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with reference-free free reward. In Advances in Neural Information Processing Systems 37, 2024.
[51] Andriy Mnih and Koray Kavukcuoglu. Learning word embeddings efficiently with noise- contrastive estimation. In Advances in Neural Information Processing Systems 26, pages 2265-2273, 2013.
[52] Abdelrahman Mohamed, Hung yi Lee, Lasse Borgholt, Jakob D. Havtorn, Joakim Edin, and Christian Igel. Self-supervised speech representation learning: A review. IEEE Journal of Selected Topics in Signal Processing, 16(6):1179-1210 2022.
[53] Paul Muller, Shayegan Omidshafiei, Mark Rowland, Karl Tuyls, Julien Perolat, Siqi Liu, Daniel Hennes, Luke Marris, Mare Lanctot, Edward Hughes, Zhe Wang, Guy Lever, Nicolas Heess, Thore Graepel, and Remi Munos. A generalized training approach for multiagent learning. ArXiv e-prints, M.XXvx19999.182200 2019.
[54] Amanda Olmin, Jakob Lindqvist, Lennart Svensson, and Fredrik Lindsten. On the connection between noise-contrastive estimation and contrastive divergence. In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics, pages 3016-3024, 2024.
[55] OpenAl. Hello GPT-4o. htte:s///ppeaaa.ccc///neee/llalp
2024.
[56] OpenAL. Introducing OpenAl ol. htttps://ppeai..com comoo// 2024. 2024
[57] Francesco Orabona. A modern introduction to online learning.
ArXiv e-prints,
arXiv:1912.1322136 2019.
[58] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35, pages 27730-27744, 2022.
[59] Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with DPO-positive. ArXiv e-prints, MDVV22002.13225, 2024.
[60] Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. In Advances Neural Information Processing Systems 37, 2024.
14

---

[61] Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Back. Reasoning with large language models, survey. ArXiv e-prints, arXiv.2407.11511v1 2024.
[62] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. 2023.
In Advances in Neural Information Processing Systems 36, pages
[63] Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to Q": Your language model secretly a Q-function. In First Conference on Language Modeling, 2024.
[64] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof Q&A benchmark, ArXiv e-prints, arXiv:2311 12022, 2023.
[65] Mengjie Ren, Boxi Cao, Hongyu Lin, Cao Liu, Xianpei Han, Ke Zeng, Guanglu Wan, Xunliang Cai, and Le Sun. Learning or self-aligning? rethinking instruction fine-tuning. ArXiv e-prints, arXiv:2402. 18243, 2024.
[66] Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. ArXiv e-prints, arXiv:2404.03715. 2024.
[67] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
[68] A. L. Samuel. Some studies in machine learning using the game of checkers. IBM Journal of Research and Development, 3(3):210-229, 1959.
[69] Pedro Zattoni Scroccaro, Arman Sharifi Kolarijani, and Peyman Mohajerin Esfahani. Adap- tive composite online optimization: Predictions in static and dynamic environments. IEEE Transactions Automatic Control, 68()):290--2921, 2023.
[70] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap. Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mas- tering chess and shogi by self-play with a general reinforcement learning algorithm. ArXiv e-prints, arXiv:1712.011 wwt.:11112......... 2017.
[71] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui. Laurent Sifrre, George den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 350(7676):354355,, 2017.
[72] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed H. Chi, Denny Zhou, and Jason Wei. Chal- lenging BIG- BeGBBench tasks and whether thain-of-thought can solve them. ArXiv e-prints, arXiv:2210.09261, 2022.
[73] Gerald Tesauro. Temporal difference learning and td-gammon. Communications of the ACM, 38(3):58-66 68, 1995.
[74] Yun-Da Tsai, Mingjie Liu, and Haoxing Ren. Code less, align more: Efficient LLM fine-tuning for code generation with data pruning. ArXiv e-prints, arXiv:2407.0504) 2024.
[75] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of LM alignment. ArXiv e-prints, arXiv:2310. 16944, 2023.
15

---

[76] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alvaro Bartolome, Alexander M. Rush, and Thomas Wolf. The Alignment Handbook, 2024. URL WtRL:httss::// tttpbbtttttuu...e/ee com wwwm hww..ngfacggggc//cooooooooo a -iggmmmttthadbbbo.
[77] Shiqi Wang, Yeqin Zhang, and Cam-Tu Nguyen. Mitigating the impact of false negative in dense retrieval with contrastive confidence regularization. In Proceedings of the 38th AAAI Conference Artificial Intelligence, pages 19171-19179 2024.
[78] Yi Wang, Hui Tang, Lichao Huang, Lulu Pan, Lixiang Yang, Huanming Yang. Feng Mu, and Meng Yang. Self-play reinforcement learning guides protein engineering. Nature Machine Intelligence, 5:845-860, 2023.
[79] Yibo Wang, Sijia Chen, Wei Jiang, Wenhao Yang, Yuanyu Wan, and Lijun Zhang. Online composite optimization between stochastic and adversarial environments. In In Advances in Neural Information Processing Systems 37, pages 94808 94850, 2024.
[80] Yibo Wang, Wenhao Yang, Wei Jiang, Shiyin Lu, Bing Wang, Haihong Tang, Yuanyu Wan, and Lijun Zhang. Non-stationary projection-free online learning with dynamic and adaptive regret guarantees. In Proceedings of the 38th AAAI Conference on Artificial Intelligence, pages 15671-15679, 2024.
[81] Yibo Wang, Hai-Long Sun, Guangda Huzhang, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Lijun Zhang. Triplets better than pairs: Towards stable and effective self-play fine-tuning for LLMs. In Advances in Neural Information Processing Systems 38, 2025.
[82] Yizhong Wang. Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Ha- jishirzi. How far can camels go? exploring the state of instruction tuning on open resources. ArXiv e-prints, arXiv:2306.04: 0475., 2023.
[83] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Aruiraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. ArXiv e-prints, arXiv:2406.01577 2024.
[84] Ronald J. Williams. Simple statistical foldieet--ollowing algorithms for connectionist reinforce- ment learning. Machine Learning, 8:229-256, 1992.
[85] Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji. Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. ArXiv e-prints, arXiv:2405.00673 00675, 2024.
[86] Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, and Vasant G Honavar. Cal-DPO: Calibrated direct preference optimization for language model alignment. In Advances in Neural Information Processing Systems 37, 2024.
[87] Cong Xu, Zhangchi Zhu, Jun Wang, Jianyong Wang, and Wei Zhang. Understanding the role of cross-entropy loss in fairly evaluating large language model-based recommendation. ArXiv e-prints, arXiv:2402.06226. 2024.
[88] Haoyan Yang, Ting Hua, Shangqian Gao. Binfeng Xu, Zheng Tang, Jie Xu, Hongxia Jin, and Vijay Srinivasan. Dynamic noise preference optimization for Im seff-improvement via synthetic data. ArXiv e-prints, ar arXiv:2502.05400, 2025.
[89] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- VengYing Ma. Ya- Qi- Zhang, Lin Yan. Mu Qiao. Yonghui Wu, and Mingxuan Wang, DAPO: An open-source Ilm reinforcement learning system at scale. ArXiv e-prints, arXiv:2503.14477 2025.
[90] Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning of diffusion models for text-to-image generation. ArXiv e-prints, 1021:04002.10210, 2024.
16

---

[91] Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan. Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu. Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. Advancing LLM reasoning generalists with preference trees. In Proceedings of the 13th International Conference on Learning Representations, 2025.
[92] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. In Proceedings of the 41st International Conference on Machine Learning, pages 57905-57922 2024.
[93] Rowan Zellers, Ari Holtzman, Yonatan Bisk. Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 4791 4700, 2019.
[94] Lijun Zhang, Shiyin Lu, and Zhi-Hua Zhou. Adaptive online learning in dynamic environments. In Advances in Neural Information Processing Systems 31, pages 1323-1333, 2018.
[95] Youzhi Zhang, Yufei Li, Gaofeng Meng, Hongbin Liu, and Jiebo Luo. The docking game: Loop self-play for fast, dynamic, and accurate prediction of flexible protein-ligand binding, ArXiv e-prints, arXiv:2508.0500 2025.
[96] Zihuai Zhao, Wenqi Fan, Jiatong Li, Yunqing Liu, Xiaowei Mei, Yiqi Wang, Zhen Wen, Fei Wang, Xiangyu Zhao, Jiliang Tang, and Qing Li. Recommender systems in the of large language models (LLMs). IEEE Transactions on Knowledge and Data Engineering, 36: 6889-6907, 2024.
[97] Jeffrey Zhou, Tianjian Lu. Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-follovie evaluation for large language models. ArXiv e-prints, LtXiv:2311.07911 2023.
[98] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In Proceedings of the 30th International Conference on World Wide Web, pages 2069-2080, 2080, 2021.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction have clearly stated the contributions and scope this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations have been discussed in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
The proof' of the propositions and theorems have been provided in Ap- pendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We have provided all the information about the experiments in Section 4.1
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [No]
Justification: We have reported the complete descriptions of the experiments in Section and have provided the pseudocode in the Appendix
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c ec/
public/guides/ htwps///oooesuubsssssoooooooo
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https http://ooooSuuiiss.oooee
//nips //nips.cc/ppl cc/public/guu
httSSSabmiiiiooooiiioiooooi for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided all the information about the experiments in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
The error bars are not reported because is too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer:
[Yes]
Justification: We have provided all the information about the experiments in Section 4.1 and
Appendix [
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics /ttiiihhhttpe//errppp...ppp/pppp http///wwppppp.oooooooc/ccccccaa
Answer: [Yes]
Justification:
Our research is conducted in accordance with the NeurlPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
This research a methodological paper and does not have societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not release models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly cited the the creators of assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
23

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: The paper investigates the LLM fine-tuninge and we have described the training of LLMs experiments in Section 4.1
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
24

---

A Theoretical proofs
In this section, we provide the proofs the proposition and theorems presented in the main paper.
A.1 Proof of Proposition| 1
For the numan-annotater data, each response is drawn from the real-world distribution Patta (-1x); whereas for the synthetic data, each response comes from the opponent player Po, (-1x). Consequently, their conditional probabilities are
p(ylc 1) Pdata (y(x) po- (y(xx and p(y'lc Ic 0, = Pa, (yl[x).
where is the label indicating the real or synthetic, and H the optimal parameters for LLM. Then, given the mixture distribution
1
Pmii (ylx)
1 po (y|x) + 1 1 Po. (ylx)
can calculate the posterior probabilities of sampling response from pmir using Bayes' theorem:
pCc x(y,x) Pg. (y(x)
μPo (y'(x)
Pa- (y|x) + uPa. (ylx) and and Oly', x) Po- (y'l + uPa, (y' x)
Finally, substituting the implicit reward r" (y/xx) = log (ylx) - log (y|x) into the posterior probabilities completes the proof.
A.2
Proof of Theorem 1
Recall that (8) can be rewritten as
[
po (y(xx)
pPe, (y|xx)
LSPACE E
Paiaao (y/x) (x log
+ upo. (y(xx log
dy
0o.0. (y|x))
pee., (y|xx)
where the expectation is taken over q(-), and Pa.9, (y|x) - po py(xx)) + ppo, (y|x). Then, we take the derivative of LSPACK (0) with respect to po below:
v V9Lsrace (0) = VolE [ Pdata (y|x) log
po(y)x)
upe. (y (y/x) (x)
+ upo. (ylx)
dy
Pa.0.
(y(x)
Pa (y|xx)
Pdata (y|x)
10.0. (ylx)) ) + (rlx)) po
Po(y|x)
u1p. (y (x) po.0, (y|x)
E
po. ( Po
(x()(x)
PPo, (y (x)
Pdaoaa (y) xx)
E
-1]
Vpo(ylx))
Po.0. (x|xx)
po (y(p))
Pdaaa (v(x)
E
0y-1 (ar-(-r((x,))
-1 Vpo()lx))
(y(y(x)
E
[op-1 (-r(x, y)) (Pdato (y(xx) x)) Ve log (y (x]x
where 04-1 (1 + exp(-x)) -1 and r(x,y) logpoly x) log (y xx.
A.3 Proof of Theorem
For brevity, we denote po. (y|x) & (y(xx) + yupo, (ylx) and Paaca e. (v/x) 2 Pdata (y(x) μPo, (ylx). Then, we rewrite the loss function (8) as
[f (
Po (o(y x) Pe.e. (ylx))
ppo, (y/x))
LSPACE = -E
+ upe, (ylx) log
dy
(10)
Pe.9. (ylx)
where the expectation taken over x Pg. Therefore, we can add the following constant term:
(()). Note that our goal is to minimize
100
with respect to
C / Paata (y|x) log
Pdata (y|x)
HPe, (ylx)
Piiaaa 8, (ylx)) + pPpe, (ylx) log
dy
(11)
Pdata 0. (ylx)
25

---

to the right side of (10) without changing the minimization problem. Hence, combining delivers
10 and
[ Pdata (y/x) log
po(y(x)
Pdata (y|x)
LSPACE (0)+C
+ =E
+ Pdata (y(x) log
dy
po,0, (y|x))
Pdata (y(x)
/ -uPo, uPo, (v/x) log
p10, (y(x)
poeo (y|x)
+ E
+ HPo, log
Ddota, 00 (y(x) dy
Rearranging the above equation, we have
(8) C
/ Polato,0 (x1x) Adaaaa (x, y) log data (x, (x.
H (x,
)} dy
=R
H data (x, log
(x
[ Pouat.. (y|x)KL (Hdata (x. dy
=E
where Ha (x, =
Pdata P (y(x)) HD Hdata (x,y)
upo, (y(x) D (x(x)) and Hx, =
po poly|x)
Pdutu. (x))))
F9.0, (y(x)*
(x, y) = 190, (x(x)
Since the KL divergence non-negative, we always have LSPACE 2
Po,0, (y/pe)
and when Hata (c, Holc, x,), i.e, po (o(y))) Pdata (y/x), the loss function (8) achieves its minimum value.
A.4 Proof of Theorem
We suppose at the iteration L, the model pe; has converged the data distribution Pdata ie., (-)x = pe, (-Ix) for any prompt x ~ q(-). Then, at the iteration t. the generated re- sponse y' Pe, (-Ix) also closely approximates the real response y Pdatal-Ix). (-)x,, i... y y'. Therefore, at the iteration + 1, (8) can be further reformulated as:
-xx-q()) b PPaata (1x)) (1xx) log on log Po(y(x) + t log 0N-1 plogop-
pa, (y'(xx)
log pa (y' x)
Pe, (v(x))
log log (o(y)x)
Pduta (y(p)) + mlog on (10g
Pdata (y (x)
po py(y(xx)
(y()
Pdaca (y x) Pdato (y(x)
--Ex-q x
log po p)(y(x) uPdata (y(x) + plog
po (y((x)
(12)
where the last equality is due to the definition of the function of (x). For brevity, we denote
Po(y (x)
Pdata (y (xx)
(y log
(o(y + uPdata (ylx) + μ lo
(13)
Pdata (ylx + 4 "po(yix)"
Then, we take the derivative over 13) with respect to (y x) as below:
Vp Smacr V (x ).y PP (-x)) ( (y/x)] -Ex (10) n (y(x))
where the last equality holds due to the linearity expectation. Let V ho(lx() 0, we obtain
1
V ho = pa ( po(y(x) + updota (y Pdota (y) x) + μ1 1 pp(l)(x) 0. (14)
which indicates that the optimal distribution of (12) satisfies pa- (y|x) = Patta Since we
update pe (ylx) pe- x), the model pects still converges to Pdota-
26

---

Reward of generated responses
Reward of Humman-annollltie responses
Margins
iter
Herl
-10
iter
iterr
20
Value
Value
Herl
iter2
1
10
Herr
iterl
iter2
-30
0
25
50
75
100
0
25
50
75
100
25
50
75
100
Steps (%)
Sleps (%)
Steps (%)
Figure 7: The training dynamics (including the rewards of generated responses and human-annotated responses, and the gap between them) of SPACE iterations 0, 1 and 2 on Mistral-7B.
B Self-play variants
In this section, we introduce self-play variants of two gap-hased methods: IPO
3 and SimPO
50
Self-play IPO (S-IPO). IPO 3 is originally proposed to align the LLM with human- crmman-preffrence over set of pair-wise training data (x,y Vi). where Yw and Yı denote the preferred and dispreferred responses corresponding to the prompt respectively. The loss function of IPO is shown below:
(o) = E(x.y (1oo po (Y log
Pe (y
2T )
27
(re((y)(x)
where D denotes the preference dataset, Pre((-(x) denotes the reference model that is typically with the pre-trained LLM, and r denotes the hyper-parameter To adapt IPO to the self-play scenario where only the high-quality response available, we follow the similar strategy SPACE and SPIN 10] generating synthetic data from the previous iteration as the training data for IPO, i.e., Yt po, xx). Moreover, we choose the last iteration model po, (-1x) as the reference model Prer(- X). In this way, the loss function of S-IPO is shown below:
[L0gg log (o(v log
Pa Po(y'(x)
1 1 2T
(0) =
(15)
Pa. (y|x)
pa. (y' x)
where the expectation is taken over X ~ (()), ~ Paota (x) and y' Pe, (-Ix). It should be highlighted that similar to SPIN, S-IPO also focuses on the gap between the high-quality response and the synthetic response. When any is an optimal solution of 15 leading to unstable traning.
(15) also degenerates to constant independent of 0, and
Self-play SimPO (S-SimPO). SimPO [50] is also proposed for human preference alignment, of which the loss function is shown below:
log PogPooya (y) (x)) -
(S -E E(x.yw.r)>> log
[y log Po (y
(16)
ly
where 9 and y are hyree--arameters, and a(.)) the sigmoid function. Similar to S-IPO, can also modify (16) to the self-play scenario, shown below:
BB [
log log (y(x) B
Iy' logpo (xxx 1x))))>>
Ls Ls.sinno (0) = -E log
(17)
where the expectation taken over x ~ q(-), y ~ Pdata (-1x) and y' Pe, (-1x). . can observed that (7) still optimizes the gap between y and y', and therefore incurs the same issue as S-IPO.
C More empirical studies
In this section, we present more experimental investigations of SPACE.
Rewards at different iterations. In Section 4.2 we have shown that our SPACE is able to increase the reward of high-quality responses and decrease the reward generated ones. In this part, we further analyze the reward trends of two types of responses at different iterations. The experimental results
27

---

Table 4: Performance (%) comparisons various tasks among SPACE (red). SPIN, S-IPO and S- S-SimPO. Arg denotes the average scores over iterations 0 to 4 are highlighted in bold and underline, respectively.
different tasks, where highest and second- highest
Model
ARC
GSM8K HellaSwag
MMLU
TruthfulQA
Winogrande IFEval
BBH
GPQA MMLU pmu
Avg
Zephyr-7B
60.92
25.85
82.79
56.90
43.67
74.19
2.76
44.60
28.91
28.18
44.88
IterO
63.23
27.77
83.77
57.19
46.93
72.30
7.66
45.22
28.87
29.22
46.22
S-1P0
Iterl
61.95
32.26
83.92
57.16
47.64
72.98
8.73
44.57
28.80
46.70
Iter2
63.48
16.72
83.13
57.48
45.92
72.61
6.50
43.48
27.74
29.11
44.62
Iter3
63.91
17.65
84.07
57.89
45.95
73.09
6.12
43.48
28.87
29.29
45.03
Itcr4
63.40
15.83
84.01
58.03
46.00
72.69
6.05
43.45
28.70
29.39
44.76
IterO
61.35
34.33
80.01
57.39
41.46
72.69
7.45
45.28
26.80
40.49
45 45.54
S-SiPPO
Iterl
62.54
31.08
82.36
57.23
43.93
72.93
5.59
45.19
28.35
41.41
45.80
Iter2
62.80
25.80
82.66
57.26
43.08
73.40
5.74
45.40
28.18
28.65
45.30
Iter3
63.14
26.31
82.63
57.17
42.59
73.48
5.32
45.11
27.85
28.62
45.22
Iter4
62.71
23.05
82.55
57.29
42.56
72.80
5.49
44.99
27.40
28.77
44.76
IterO
63.14
29.34
84.10
56.47
18.93
73.48
9.16
43.
29.87
28.47
46.69
Iterl
64.33
31.08
83.76
57.34
52.12
74.11
10.06
44.70
29.75
28.03
47 553
SPIN
Iter2
63.23
36.62
83.75
57.74
51.77
73.95
14.46
43.95
28.39
28.67
48.25
Iter3
63.82
33.59
83.13
55.56
52.85
74.51
13.59
44.05
28.10
28.57
47.78
Iter4
63.99
31.69
83.15
56.37
51.73
74.35
11.75
43.29
28.56
27.73
47.26
SPACE (0urs)
61.35
40.49
83.53
46.05
73.88
10.65
42.96
28.64
28.81
47.42
Iterl
63.65
40.56
83.55
58.20
48.74
73.64
12.14
43.01
28.60
28.54
48.06
Iter2
64.85
40.64
83.58
58.10
48.32
74.03
14.19
43.51
28.64
28.72
48.46
64.59
40.94
83.60
58.01
48.51
74.11
16.28
43.75
28.76
48.69
Iter4
64.85
41.93
83.64
58.00
48.38
73.48
16.82
45.07
29.14
28.72
49.00
SSIP
ESPIN
MS-SimPP ISPACE
Avenge Score (%)
50.0
[SS-IPO
@SPN
DS-SimPP DSPACE
S-IPO
2.8
2.8
3.8
3,2
2.0
40.0
Score (9%
S-SinPO
3.3
2.6
2.2
3.2
3.6
30.0
SPIN
2.6
3.5
20.0
44,0
SPACE
4.1
3.5
2.6
2.4
2.1
10.0
42.0
iterO
iterl
iter2
iter3
10er4
iteed
iter2
(a) GSM8K
(b) Average score
(c) Average rank
Figure 8: The performance comparisons among four self-play fine-tuning methods on Zephyr-7B. (a) the performances on GSM8K; (b) the average scores over different tasks; (c) the average ranks over different iterations, where the best rank among iterations 0 to 4 is highlighted with a "gold medal".
are shown in Figure 7 From the results, we observe that (i) the rewards for generated responses are decreasing three iterations; (ii) the rewards for iteration, then oscillate near zero in subsequent iterations, which we consider as the model identifying a potential optimal distribution initially and subsequently exploring its vicinity; (iii) given the reward patterns of both positive and negative samples, the margins demonstrate a consistent increasing trend, reflecting the improved capability of our model to distinguish two types of responses.
responses increase at the first
Experimental results on Zephyr-7B. We present the experimental results on Zephyr-7B in Ta- ble and provide in-depth comparisons in Figure including the performance on GSMSK, average performance across all tasks, and average ranks over different iterations. Thole that SPACE sig- nificantly improves the performance the base model, especially on GSM8K and IFEval. Moreover, we also observe that gap-based baselines suffer from unstable evolution, with their best performance occurring early iterations, while our SPACE maintains stable evolution and continuously improves performance. This phenomenon is more evident in Figure a) and Figure Additionally, Fig- ure & c) also shows that the model performance later iterations surpasses that of earlier iterations, demonstrating the continuous evolution and improvement of SPACE.
28

---

D Case studies
In the following, we present synthetic responses generated by SPACE different iterations.
Table 5: Example between the annotaedd response, the responses generated by the base model (Mistral-7B) and the model trained by SPACE at iterations 0, 1, and 2.
Example 1: Prompt
Here is a piece of text: LIAM Payne missed his granddad's funeral because One Direction's current Take Me Home world tour. The 20-year-old boy band singer is devastated following the loss his grandfather but has been grateful for the huge amount of support he has received from fans who took to twitter to show they were thinking of him, leading to the hashtage "#RIPGrandpaaPyne" and "#StayStrongLiam" trending the social networking site. The star later wrote on his own page to thank his fans for their kind words, writing: "Thanks for everyone's support at this hard time obviously this has been tough few weeks for I've never been through anything like this." Liam dedicated the track 'More Than This' to his grandfather whilst stage in Australia, as he was not able to be with his family during this difficult time. One Direction which includes Niall Horan, Zayn Malik, Harry Styles and Louis Tomlinson kicked off the Australian leg of their 'Take Me Home' world tour in September. They took to the stage last night (02.10.15 at the country's culture capital Melbourne performing at the Rod Laver Arena and are set to do so again tonight (03. 10. 13). Based on the text material above, generate the response to the following quesion or instruction: Which song did Liam Payne dedicate to his grandfather during One Direction's concert in Australia?
Annotated Response
Liam Payne dedicated the song 'More Than This' his grandfather during One Direction's concert in Australia.
Base Model
More Than This 2. Want to Break Free 3. Will Always Love You 4. I'm Yours
Iteration 0
Liam Payne dedicated the track 'More Than This' to his grandfather whilst on stage in Australia, as he was not able be with his family during this difficult time.
Iteration
Liam Payne dedicated the track 'More Than This' to his grandfather whilst on stage Australia, as he was not able to be with his family during this difficult time.
Iteration 2 Liam Payne dedicated the song 'More Than This' to his grandfather during One Direction's concert Australia.
29

---

Table 6: Example 2 between the expert-annotated response, the responses generated by the base model (Mistral-7B) and the model trained by SPACE at iterations 0, 1, and 2.
Prompt
Tatiana Damiliyants is a poet, Film-maker, and artist. She is the author of three books of poetry in Russian: Red Noise (2012), White (2006), Venetian (2005), and four bilingual books (in Russian paired with Polish, Italian, French, and Armenian). She has twice won the International Festival of Verse Libre (Russia), as well as the Nosside prize (Italy). In 2014 she received the Catherine the Great medal for achievement in contemporary Russian literature (St. Petersburg Union of Avangardists). Her poetry has been translated and published in English, French, Italian, Polish, Armenian, German, Serbian, Macedonian, and other languages. Based on the passage above, Can you summarize the achievements and career of Tatiana Daniliyants as poet, filmmaker, and artist?
Annotated Response
Tatiana Damiliyants is prominent poet, filmmaker, and artist who has authored several books of poetry in Russian, as well as four bilingual books Russian paired with other languages. She has won numerous awards, including the International Festival of Verse Libre and the Nosside prize, and in 2014, she received the Catherine the Great medal for her contributions to contemporary Russian literature. Her poetry has been translated and published in several languages, including English, French, Italian, Polish, Armenian, German, Serbian, and Macedonian.
Base Model
Tatiana Daniliyants is a poet, film-maker, and artist She is the author of three books of poetry in Russian: Red Noise (2012), White (2006), Venetian (2005), and four bilingual books (in Russian paired with Polish, Italian, French, and Armenian). She has twice won the International Festival of Verse Libre (Russia), as well as the Nosside prize (Italy). In 2014 she received the Catherine the Great medal for achievement in contemporary Russian literature (St. Petersburg Union of Avangardists). Her poetry has been translated and published in English, French, Italian, Polish, Armenian, German, Serbian, Macedonian, and other languages.
Iteration 0 Tatiana Damiliyants is Russian poet, filmmaker, and artist who has written three books of poetry in Russian and four bilingual books. She has several awards for her poetry, including the International Festival of Verse Libre and the Nosside prize. Her poetry has been translated and published in multiple languages.
Iteration
Tatiana Daniliyants is a Russian poet, filmmaker, and artist who has authored three books of poetry in Russian and four bilingual books. She has won several awards, including the International Festival of Verse Libre and the Nosside prize. In 2014, she received the Catherine the Great medal for achievement in contemporary Russian literature. Her poetry has been translated and published in various languages.
30

---

Iteration 2 As poet, Tatiana Daniliyants has authored three books in Russian and four bilingual books. She has won the International Festival of Verse Libre twice and the Nosside prize Italy. In 2014, she received the Catherine the Great medal for achievement in contemporary Russian literature. Her poetry has been translated and published in multiple languages.
E
Pseudocode
We present the pscudocode to computing the loss function in SPACE follows:
def space_loss (mu, player_ reayer__eal_logps, player Slayer_generated_logr ppponent_real_iaee logps,
1ogps,
Args:
mu: the generation ratio
flayer_real logps: Player's log probs for real samples Slayer_generated logps: Player's log probs for synthetic samp les opponent_ real_ logps: Opponent's 's log probs for real samples
opponett_generater logps: Opponent's 's log probs for synthetic samples
Returns:
scalar loss value (meon over batch)
opponent_real_ real_loss F. r. torch. torch.log(m)) 1og(m))
fog_ratio_re
player_reallepeeppppeeaaaaaaaeea paaver_real__ae reap_
opponett_eenozaee__lgge player_generated_1gg generated_loss mu Iogeiiooiddlog..raii..eeeees torch.)og(1/mu)
ong_ratio_generann
losses - (real, loss + generated_loss)
return losses ()
31


Reviews:
[{"review_id": "Pwb5I42eOO", "reviewer": "Reviewer_1euE", "summary": "This paper proposes SPACE, a self-play fine-tuning framework for large language models that avoids the instability of prior reward-gap–based methods like SPIN. SPACE reformulates the objective using a noise contrastive estimation (NCE) loss that computes a symmetric log-probability ratio between the current model and a frozen past model. This avoids the reward-gap collapse observed in SPIN as models converge. The authors provide theoretical guarantees that SPACE retains non-vanishing gradients and is preference-optimal under decomposable reward assumptions. Empirically, SPACE outperforms SPIN, DPO, and supervised fine-tuning on GSM8K and IFEval, offering smoother training dynamics and achieving alignment without human-labeled preferences.", "questions": "# Questions\n\n1. How does SPACE affect response diversity and exploration?\n    Does it suppress generation diversity by continually pushing away from previous outputs? Could this lead to degenerate or overly cautious outputs over time?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "# Strengths\n\n1. The paper replaces reward-gap comparisons with a log-probability contrastive formulation, avoiding instability due to shrinking reward signals.\n2. Theoretical Soundness: Proves consistency with preference optimization and non-vanishing gradients, even as the model improves over iterations.\n3. Empirical Performance: Demonstrates strong results on reasoning tasks like GSM8K and IFEval using self-play without human preference data, outperforming SPIN and achieving performance comparable to or better than DPO in several settings.\n4.Training Stability: SPACE maintains more stable and consistent performance gains across training iterations, whereas SPIN suffers from reward-gap collapse—an issue that limits the scalability of self-play fine-tuning.\n\n# Weaknesses\n\n1. Assumes Frozen Baseline Quality: Performance depends on the frozen past model being appropriately calibrated. If too weak or too strong, the contrastive learning signal may be ineffective.\n2.No Diversity Evaluation: The paper does not analyze whether SPACE encourages mode collapse or overly conservative outputs, particularly as self-play proceeds across iterations. Output diversity is a key quality consideration in alignment methods and could affect robustness\n3. Limited Benchmark Coverage: Focused on factual, single-turn tasks (GSM8K, IFEval); the behavior of SPACE in open-ended settings like dialogue or summarization is unexplored.", "was_revised": true, "final_justification": "Based on a careful reading of the paper and the rebuttal, I believe this is a solid work suitable for publication at NeurIPS. My questions were satisfactorily addressed, and I am maintaining my original score.", "rebuttal": "**Many thanks for the constructive reviews!**\n\n---\n\n**Q1**: Assumes Frozen Baseline Quality.\n\n**A1**: Thanks for the insightful comment. The quality of the reference model is indeed a crucial factor in self-play fine-tuning for LLM, and SPACE is designed with this in mind. In particular, we consider two scenarios: (i) where the reference model is weak, and (ii) where it is strong. For the first case, Figure 4 shows that using synthetic samples generated by a weaker model leads to suboptimal performance, even with more training epochs. For this reason, SPACE employs an iterative strategy that updates synthetic samples using the most recent model for progressive evolution. In the second case, a strong reference model may lead to unstable optimization (as observed in SPIN), and our SPACE is specifically designed to deal with this issue. Moreover, Theorem 3 shows that SPACE can preserve optimality even with strong reference model. We will include a detailed discussion in the revised version.\n\n---\n\n**Q2**: Diversity and exploration of SPACE.\n\n**A2**: We appreciate the thoughtful question. We acknowledge that this concern is well-founded, since self-play methods are designed to handle the scenarios with scarce annotated data, where fine-tuning LLMs while maintaining diversity over multiple benchmarks is inherently challenging. In our experiments, we observe that SPACE indeed demonstrates relatively marginal performance improvements on certain benchmarks, such as Winogrande. Notably, this is ***a common issue*** for self-play methods, not just specific to our SPACE. One possible explanation is that self-play methods gradually reinforce their attention to annotated data during iterations, which thereby reduces the diversity of outputs. Nevertheless, we would like to emphasize that SPACE is proposed to address the instability issue of SPIN, and we consider the investigation of diversity for self-play methods a meaningful direction and will explore it in the future.\n\n---\n\n**Q3**: The behavior of SPACE in open-ended settings like dialogue or summarization is unexplored.\n\n**A3**: Thanks for the helpful suggestion! We have conducted additional experiments on MATH [1], openbookqa [2], and MT-Bench [3], which evaluate mathematical reasoning, common knowledge, and conversational ability in open-ended scenarios, respectively. We present the results in the following table, where columns 3 to 7 correspond to iterations 0 to 4 of SPACE. From the table, we observe that SPACE can progressively improve performance while maintaining stability.\n\n||Mistral|iter0|iter1|iter2|iter3|iter4|\n| ------------| ---------| -------| -------| -------| -------| -------|\n|MATH|3.25|3.40|3.58|3.69|3.84|3.90|\n|openbookqa|44.0|44.4|45.0|45.4|45.4|46.0|\n|MT-Bench|3.56|5.87|6.13|6.27|6.43|6.44|\n\n---\n\n**Reference.**\n\n[1] Measuring mathematical problem solving with the math dataset. 2021.\n\n[2] Can a suit of armor conduct electricity? a new dataset for open book question answering. 2018.\n\n[3] Judging llm-as-a-judge with mt-bench and chatbot arena. 2023.\n\n---\n\nWe hope that our responses can address your concerns, and we are also happy to provide further clarifications during the reviewer-author discussions if needed."}, {"review_id": "YgWfPYcaoO", "reviewer": "Reviewer_BDge", "summary": "This paper introduces SPACE, a novel self-play fine-tuning method for large language models (LLMs) that leverages noise contrastive estimation (NCE) to address the instability issue in existing gap-based self-play approaches. SPACE treats synthetic samples as auxiliary components and discriminates them from real data via binary classification, independently optimizing absolute reward values for both types of data. Theoretically, the authors prove that SPACE converges stably to the real-world data distribution. Empirically, SPACE outperforms SPIN and SFT with more data, on diverse tasks from the HuggingFace Open LLM Leaderboard. It demonstrates stable performance evolution across iterations, avoiding the degradation seen in gap-based methods, and highlights the effectiveness of NCE in stabilizing self-play fine-tuning for LLMs with limited annotated data.", "questions": "See weaknesses above", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. The paper points out an interesting reason why SPIN stops improving quickly, gives a reasonable explanation, and then comes up with a new loss function based on NCE. The new loss function has a good theoretical basis.  \n2. The results on various benchmarks show that it's significantly better than SPIN and its variants, and it seems to keep getting better over more rounds. So the method is good both in theory and in practice.  \n3. I like the analysis experiments, they're complete. They rule out possible counfounding factors that could explain why SPACE works better, like comparing more training rounds on the data from iteration 0. Also, the sample efficiency shown in Figure 5 is impressive, and I like how the results are presented.  \n4. The analysis of the reward decline problem is interesting too. It shows that SPACE properly reduces the problem, which helps explain why SPACE works well.  \n\n**Weaknesses:**\n1. When looking at the gradient in Eq. 9, we can see it's similar to the policy gradient in RL. Since SPACE uses annotated data, it might also be related to off-policy RL. Given how much attention RL is getting lately, it would have been good to discuss this connection.  \n2. Why does SPACE do so much better than SPIN on GSM8K and IFEval? With such big improvements, some explanation or examples would be good.  \n3. The method only looks at one model, so it's not clear if it works for other models.  \n4. There's no proof for the theorems. They should at least be in the appendix.", "was_revised": true, "final_justification": "My concerns have been addressed", "rebuttal": "**Many thanks for the constructive reviews!**\n\n---\n\n**Q1**: Connection to off-policy RL.\n\n**A1**: Thanks for the valuable suggestion! In the standard policy gradient formulation (e.g., REINFORCE [1]), the gradient is formulated as $\\nabla_\\theta J(\\theta) = \\mathbb{E}\\_{u\\sim p_\\theta}[R_u \\cdot \\nabla_\\theta \\ln p_{\\theta}(u|x)]$, where $R_u$ denotes the reward for a response $u$. Intuitively, (9) shares a similar structure to this formulation, if we set $R_u=\\sigma_{\\mu^{-1}}(-r(x,u))(p_{data}(u|x) - p_{\\theta_t}(u|x))$. However, we would like to emphasize that SPACE is based on the self-play fine-tuning framework, where the goal is to iteratively improve model performance through two-player competition. This fundamentally differs from the exploration-exploitation trade-off in RL. Additionally, SPACE optimizes (8) in a supervised-learning manner, without involving explicit rewarding or rollout phrases, which are essential components of standard RL methods. We will include a more detailed discussion in the revised version.\n\n\n---\n\n**Q2**: Why does SPACE do so much better than SPIN on GSM8K and IFEval?\n\n**A2**: Thanks for the insightful question. GSM8K and IFEval evaluates the abilities of mathematical problem solving and instruction following, respectively. Both benchmarks require strong logical reasoning and accurate responses. Based on the experimental results, we consider that for self-play fine-tuning methods, stable evolution during iterations is the key to ensuring rigorous reasoning and precise responses. Moreover, we present response comparisons on GSM8K below. From the responses, we observe that Mistral exhibits strong fluency but lacks understanding of the problem. SPIN demonstrates partial understanding but often fails to solve the problem correctly, likely due to unstable optimization. In contrast, SPACE, with its stable iterative evolution, is able to effectively solve practical problems.\n\n> **Prompt**: Weng earns \\$12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. How much did she earn?\n>\n> **Mistral**:  Guidance\\\\n\\\\nIn the last concept, we learned how to solve proportions involving rates. In this concept, we will be solving proportions as well. The difference is that rather than solving for a rate which compares two rates, we will be solving for an unknown quantity in a proportion. The formula to find any unknown amount is to cross multiply.\\\\n\\\\n#### Example A\\\\n\\\\nIf 24 is to 88 as 5 is to 12, how much is the unknown value 5?\\\\n\\\\nSolution: To find the value of an unknown amount in a proportion, we cross multiply the amount given in the proportion and the amount we are solving for... \\\\n\\\\nYou must check all answers, so check the answer of 12. ... The answer of 12 checks.\\\\n\\\\n#### Example B\\\\n\\\\n ... #### Concept Problem Revisited\\\\n\\\\nWeng earns 12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. How much did she earn?\\\\n\\\\nThis is a proportional situation.\\\\n\n>\n> **SPIN**: If the word problem asks you to find the total number of dollars earned (the solution), then first, you need to translate the words into a math problem. When you read the problem, ask yourself, “What is the unknown?” In this case, it is the number of dollars earned.\\\\n\\\\nThe money earning rate is given.\\\\n\\\\nWeng earns 12 an hour for babysitting.\\\\n\\\\nThe time taken is known.\\\\n\\\\nYesterday, she just did 50 minutes of babysitting.\\\\n\\\\nWhat is the multiplication formula for calculating the total sum from rate and time?\\\\n\\\\nrate × time \\= total\\\\n\\\\n12 × \\\\underline{\\\\underline{\\\\phantom{\\\\rule{1em}{0ex}}}} \\= total\\\\n\\\\nWrite the question word to fill the missing part of the equation.\\\\n\\\\nMultiply the rate and time.\\\\n\\\\nHere, the unknown is total, which is also unknown to find.\\\\n\\\\nYou can find the answer to the problem by using the multiplication formula and the information given in the word problem.\\\\n\\\\nNote: 1 hour \\= 60 minutes\\\\n\\\\nRate × time \\= total\\\\n\\\\n12\\\\⨯50\\=1,200\\\\n\\\\nShe earned 1,200.\n>\n> **SPACE**:  ## The math Rate \\= (\\\\$12) per hour Number of hours \\= (50 \\\\div  60) \\= (0.833333) h Earned \\= Rate x Number of hours \\= (\\\\$12 \\times 0.833333)  \\= (\\\\$10) Therefore, Weng earned (\\$10) for babysitting.\n\n---\n\n**Q3**: The method only looks at one model, so it's not clear if it works for other models.\n\n**A3**: Thanks for the helpful suggestion. In fact, we have included the experiments on Zephyr in the ***Supplementary Material***, and the results also demonstrate the superiority of SPACE over SPIN. Additionally, we also conduct empirical studies on Qwen2.5-7B-base model. The results are presented in the table below, where *base model* refers to the average performance of Qwen2.5 over multiple tasks (the same task set as shown in Table 5; only average scores are shown due to character limitations). From the results, we observe that (i) SPIN continues to suffer from unstable iterative optimization, while SPACE demonstrates consistent performance improvements; and (ii) the performance gains achieved by SPACE on Qwen2.5 are less pronounced compared to those on Mistral and Zephyr. This may be because Qwen2.5 has already been well optimized during pretraining, leaving less room for further improvement when only 50K annotated examples are used.\n\n||base model|iter0|iter1|iter2|iter3|iter4|\n| -------| ------------| -------| -------| -------| -------| -------|\n|SPIN|59.18|59.61|59.81|59.63|59.69|59.89|\n|SPACE|59.18|60.06|60.46|60.53|60.54|60.54|\n\n---\n\n**Q4**: There's no proof for the theorems.\n\n**A4**: Thanks for the kind reminder. The proofs of all theorems are included in the ***Supplementary Material***, and we will highlight this in the revised version.\n\n---\n\n**Reference.**\n\n[1] Simple statistical gradient following algorithms for connectionist reinforcement learning. 1992.\n\n\n---\n\nWe hope that our responses can address your concerns, and we are also happy to provide further clarifications during the reviewer-author discussions if needed."}, {"review_id": "ODpOw8phQ7", "reviewer": "Reviewer_Hm2z", "summary": "Current self-play fine-tuning methods like SPIN suffer from instability during training iterations. Specifically, current methods focus on maximizing relative gaps between rewards of real and synthetic data, but when synthetic data become similar to real ones, the objective function can degrade into a meaningless constant, leading to performance collapse. Instead of comparing relative rewards, SPACE optimizes absolute reward values for real and synthetic responses separately. In this way, even when synthetic and real responses become similar, the objective remains meaningful and doesn't collapse. But there is concern about whether NCE's fundamental assumptions hold in their setting.", "questions": "1. What happens if synthetic data quality degrades? Especially in the early stage?\n2. Could the iterative process gradually degrade capabilities in non-targeted areas?", "limitations": "Test with model under 7B; only one dataset; the Technical theory assumption lack of proof", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Hot topic;\n2. Clever Technical Insight\n3. Good performance. \n\nWeakness:\n1. Is the sync data a noise? NCE's core assumption that noise remains consistently different from real data. Real noise is uninformative and random. But the sync data is in fact informative, and gets better over time. And in fact the sync data can be better than real data, especially for datasets like Ultrachat200k. \n2. Limited datasets evaluated. Only Ultrachat200k is evaluated. \n3. Figure 5 may be an unfair comparison on SFT and SPACE. SPACE generates more than 200K data and runs for 4 times longer than SFT. To be fair, at least SFT should be run with the same computational budget. \n4. Missing recent methods. There are quite a few follow-up work after SPIN, that notices the model update issue. For example, Dynamic Noise Preference Optimization for LLM Self-Improvement via Synthetic Data https://arxiv.org/abs/2502.05400 also tries to add noise concept to resolve the model update stuck problem, although in another way.", "was_revised": true, "final_justification": "During the rebuttal period, the authors handle my concern properly and I will remain my score, which is already positive.", "rebuttal": "**Many thanks for the constructive reviews!**\n\n---\n\n**Q1**: Is the sync data a noise?\n\n**A1**: Thanks for the insightful question. In fact, the sync data generated by the previous iteration's LLM is not ***conventional uninformative noise*** in the traditional NCE. Instead, we leverage its progressively improving nature as a self-generated supervision to guide the iterative optimization. The sync data $y'\\sim p_{\\theta_{t-1}}$ can be viewed as hard negatives for $p_{\\theta_{t}}$, playing a critical role in the two-player framework [1, 2]. We note that, [3] demonstrates the quality of self-generated samples can potentially exceed that of annotated samples in self-play fine-tuning methods. This is also possible in SPACE. However, it is important to notice that when the sync data $y'$ aligns well with $p_{\\text{data}}$, i.e., $p_{\\text{data}}(y'|x) = p_{\\theta_t}(y'|x)$, our SPACE can provably maintain the optimality on $y'$, as shown in ***Theorem 3***.\n\n---\n\n**Q2**: Limited datasets evaluated.\n\n**A2**: To ensure the consistency with SPIN, we follow their experimental setup and conduct our experiments on the Ultrachat200k dataset. In addition, we have already validated the effectiveness of SPACE on many standard benchmarks, including ARC, GSM8K, and MMLU. Moreover, we have conducted additional experiments on MATH [1], openbookqa [2], and MT-Bench [3], which evaluate mathematical reasoning, common knowledge, and conversational ability in open-ended scenarios, respectively. We present the results in the following table, where columns 3 to 7 correspond to iterations 0 to 4 of SPACE. From the table, we observe that SPACE can progressively improve performance while maintaining stability.\n\n||Mistral|iter0|iter1|iter2|iter3|iter4|\n| ------------| ---------| -------| -------| -------| -------| -------|\n|MATH|3.25|3.40|3.58|3.69|3.84|3.90|\n|openbookqa|44.0|44.4|45.0|45.4|45.4|46.0|\n|MT-Bench|3.56|5.87|6.13|6.27|6.43|6.44|\n\n---\n\n**Q3**: Figure 5 may be an unfair comparison on SFT and SPACE.\n\n**A3**: Thanks for the insightful comment. Figure 5 is intended to highlight that SPACE, using a small amount of annotated data, outperforms SFT trained on a significantly larger number of annotated examples. In Figure 5, we present the performance of SPACE trained ***over two iterations***. For example, the reported average performance 51.48 is measure on the model trained with a total of 150K samples, including 50K annotated data and 100K synthetic data. To ensure a fair comparison under the same computational budget, we run SFT for multiple rounds with annotated samples, and present the results in the following table. The results show that, even when using more annotated data, SFT still underperforms compared to SPACE (51.48 with a total of 150K samples).\n\n||400K|600K|800K|\n| -----| -------| -------| -------|\n|SFT|50.48|50.14|49.84|\n\n---\n\n**Q4**: Missing recent methods\n\n**A4**: Thanks for the helpful reminder. DNPO and SPACE share a similar motivation, both of which aim to address the model update stagnation issue in SPIN, but they adopt different strategies. DNPO proposes to selectively filter generated samples for continual updates, whereas SPACE introduces a novel optimization objective that ensures provably stable convergence (shown in ***Theorem 3***). We will include a more detailed discussion in the revised version.\n\n---\n\n**Q5**: What happens if synthetic data quality degrades?\n\n**A5**: Thanks for the insightful question. The experiments in Figure 4 demonstrate that using synthetic samples generated by a weaker model leads to limited performance improvement, even with more training epochs. For this reason, SPACE employs an iterative strategy that updates synthetic samples using the most recent model for progressive evolution. We will include a more detailed discussion in the revised version.\n\n---\n\n**Q6**: Could the iterative process gradually degrade capabilities in non-targeted areas?\n\n**A6**: Thanks for the valuable question. We acknowledge that this concern is well-founded, since self-play methods are designed to handle the scenarios with scarce annotated data, where fine-tuning LLMs while maintaining diversity over multiple benchmarks is inherently challenging. In our experiments, we observe that SPACE indeed demonstrates relatively marginal performance improvements on certain benchmarks, such as Winogrande. Notably, this is ***a common issue*** for self-play methods, not just specific to our SPACE. One possible explanation is that self-play methods gradually reinforce their attention to annotated data during iterations, which thereby reduces the diversity of outputs. Nevertheless, we would like to emphasize that SPACE is proposed to address the instability issue of SPIN, and we consider the investigation of diversity for self-play methods a meaningful direction and will explore it in the future.\n\n---\n\n**Referece.**\n\n[1] Generative adversarial nets. 2014.\n\n[2] Mastering the game of go without human knowledge. 2017.\n\n[3] Dynamic Noise Preference Optimization for LLM Self-Improvement via Synthetic Data. 2025.\n\n[4] Measuring mathematical problem solving with the math dataset. 2021.\n\n[5] Can a suit of armor conduct electricity? a new dataset for open book question answering. 2018.\n\n[6] Judging llm-as-a-judge with mt-bench and chatbot arena. 2023.\n\n---\n\nWe hope that our responses can address your concerns, and we are also happy to provide further clarifications during the reviewer-author discussions if needed."}, {"review_id": "0cj1x9HJW4", "reviewer": "Reviewer_j1nk", "summary": "This paper proposes the noise contrastive estimation-based self-play fine-tuning approach, called SPACE. This method addresses the training instability issue of SPIN, a prior method in this direction. SPACE independently optimizes the absolute reward values for each type of data, ensuring a consistently meaningful objective and thereby avoiding the instability issue. Empirical results on Mistral and Zephyr demonstrate the advantages of the proposed method.", "questions": "1. Given the both SFT, SPIN, SPACE can theoretically converge to the underlying data distribution, do you believe what are exactly advantages of self-playing algorithms in this case? \n2. Do you believe the performance degradation in Figure 2 for SPIN is due to optimization instability or generalization issues?  \n3. Have you tested the method on Qwen2.5 and Llama-3.1 series? Models like Zephyr and Mistral are outdated (e.g., scoring 40-50 on simple benchmarks like GSM8K), making it hard to justify the superiority of the proposed method.  \n4. The OpenLLM Leaderboard primarily evaluates few-shot in-context learning (ICL) capabilities rather than downstream performance. Have you tested more challenging tasks like MATH?  \n5. Have you tested other datasets to ensure broader applicability of your conclusions?  \n6. A limitation of self-play algorithms is overfitting to the empirical data distribution, reducing diversity and harming subsequent RL fine-tuning. Have the authors observed this issue, where model outputs heavily conform to empirical data patterns?", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written in most places and easy to follow.  \n- The provided empirical results seem promising.  \n\nWeakness:\n- The criticism of SPIN seems only empirical, which lacks rigor. It is unclear whether this is due to a fundamental limitation or hyperparameter tuning. The argued extreme case—where the generated response matches the real one—is not a bug for me, as SPIN may already learn the data distribution in this scenario.  \n- Lack of related work discussion. This paper lies within the framework of statistical distribution matching but fails to discuss seminal works like Generative Adversarial Nets (GANs) and Generative Adversarial Imitation Learning (GAIL). \n- Empirical results are rather weak. Mistral and Zephyr are relatively weak models, and results on these models may not generalize to modern ones.  \n- Empirical comparisons with Supervised Fine-Tuning (SFT) are not rigorous. The proposed method requires more compute due to synthetic data generation and additional gradient iterations. Thus, comparisons should include more epochs (even beyond 4). The results in Figure 5 do not convincingly justify the comparison with the same human-annotated data.  \n\n--- \n\nGoodfellow, Ian J., et al. \"Generative adversarial nets.\" Advances in neural information processing systems 27 (2014).\n\nHo, Jonathan, and Stefano Ermon. \"Generative adversarial imitation learning.\" Advances in neural information processing systems 29 (2016).", "was_revised": true, "final_justification": "While the authors have provided additional empirical evidence, these results unfortunately demonstrate only marginal improvements, thereby calling into question the overall effectiveness of their proposed methods. Consequently, I maintain my original recommendation regarding this work.", "rebuttal": "**Many thanks for the constructive reviews!**  \n\n---\n\n**Q1**: The criticism (optimization instability) of SPIN seems only empirical. Do you believe the performance degradation in Figure 2 for SPIN is due to optimization instability or generalization issues?\n\n**A1**: We note that there are several studies also pointing out the training instability of SPIN [1, 2], which appears to be ***a widely-recognized issue*** in the literature. In our work, we have provided an in-depth discussion showing that gap-based methods (including SPIN) are susceptible to performance collapse when extended to self-play fine-tuning (Lines 183–195). This phenomenon is also empirically validated through extensive experiments (***Table 1***). The extreme cases presented in our analysis are intended to facilitate understanding. In practice, during the iterations, the loss of SPIN may become very small (even a constant), but the learned distribution may not align with the target distribution; instead, it could be ***any*** distribution, leading to performance collapse. We will provide a more detailed clarification in the revised version.\n\n---\n\n**Q2**: Lack of related work discussion, such as GAN and GAIL.\n\n**A2**: Thanks for the helpful suggestion. GAN [3] is proposed to learn generative models via an adversarial process, and GAIL [4] extends this adversarial paradigm to imitation learning. Overall, both [3, 4] and SPACE adopt two-player adversarial frameworks. However, SPACE is specifically designed for self fine-tuning of LLMs, where the two players are instances of the same LLM with different parameters. We will include a detailed discussion of this connection in the revised version.\n\n---\n\n**Q3**: Results on Mistral and Zephyr may not generalize to modern ones.\n\n**A3**: Thanks for the valuable suggestion. Here, we conduct additional empirical studies on Qwen2.5-7B-base model. The results are presented in the table below, where *base model* refers to the average performance of Qwen2.5 over multiple tasks  (the same task set as shown in Table 5; only average scores are shown due to character limitations). From the results, we observe that (i) SPIN continues to suffer from unstable iterative optimization, while SPACE demonstrates consistent performance improvements; and (ii) the performance gains achieved by SPACE on Qwen2.5 are less pronounced compared to those on Mistral and Zephyr. This may be because Qwen2.5 has already been well optimized during pretraining, leaving less room for further improvement when only 50K annotated examples are used.\n\n||base model|iter0|iter1|iter2|iter3|iter4|\n| -------| ------------| -------| -------| -------| -------| -------|\n|SPIN|59.18|59.61|59.81|59.63|59.69|59.89|\n|SPACE|59.18|60.06|60.46|60.53|60.54|60.54|\n\n---\n\n**Q4**: Empirical comparisons with SFT are not rigorous.\n\n**A4**: Thanks for the thoughtful consideration. Figure 5 is intended to highlight that SPACE, using a small amount of annotated data, outperforms SFT trained on a significantly larger number of annotated examples. In Figure 5, we present the performance of SPACE trained ***over two iterations***. For example, the reported average performance 51.48 is measure on the model trained with a total of 150K samples, including 50K annotated data and 100K synthetic data. To ensure a fair comparison under the same computational budget, we run SFT for multiple rounds with more annotated samples, and present the results in the following table. The results show that, even when using more annotated data, SFT still underperforms compared to SPACE (51.48 with a total of 150K samples).\n\n||400K|600K|800K|\n| -----| -------| -------| -------|\n|SFT|50.48|50.14|49.84|\n\n---\n\n**Q5**: Advantages of self-playing algorithms.\n\n**A5**: We appreciate your valuable question! We summarize the advantages of self-play methods and SPACE as follows:\n\n* **Better Performance with Fewer Annotations.**  SFT typically requires a large amount of annotated data, which is often impractical in real-world scenarios, and thus motivates the development of self-play fine-tuning methods. In this paper, we theoretically prove that with limited annotated samples, SPACE is able to capture the target data distribution (***Theorem 2***). Moreover, empirical results demonstrate that SPACE achieves superior performance with a small amount of annotated data, compared to SFT trained on significantly larger datasets (***Figure 4***).\n* **Stable Performance Improvement.**  Compared to the existing self-play method SPIN, the primary advantage of SPACE lies in its stable iterative optimization. To support this, we first provide a theoretical analysis demonstrating that SPACE is able to maintain the optimal distribution (***Theorem 3***). We then conduct extensive empirical studies to validate the stable convergence of SPACE over iterations (***Table 1***).\n\n\nAdditionally, we would like to emphasize that while SFT, SPIN, and SPACE can theoretically capture the underlying data distribution, they differ in their convergence rates (i.e., the amount of annotated data required) and stability during optimization. We will include a detailed discussion in the revised version.\n\n---\n\n**Q6**: Test more challenging tasks like MATH and other datasets.\n\n**A6**: Thanks for the helpful suggestion! We have conducted additional experiments on MATH [5], openbookqa [6], and MT-Bench [7], which evaluate mathematical reasoning, common knowledge, and conversational ability in open-ended scenarios, respectively. We present the results in the following table, where columns 3 to 7 correspond to iterations 0 to 4 of SPACE. From the table, we observe that SPACE can progressively improve performance while maintaining stability.\n\n||Mistral|iter0|iter1|iter2|iter3|iter4|\n| ------------| ---------| -------| -------| -------| -------| -------|\n|MATH|3.25|3.40|3.58|3.69|3.84|3.90|\n|openbookqa|44.0|44.4|45.0|45.4|45.4|46.0|\n|MT-Bench|3.56|5.87|6.13|6.27|6.43|6.44|\n\n---\n\n**Q7**: Have the authors observed the overfitting of self-play methods?\n\n**A7**: Thanks for the insightful question. We acknowledge that this concern is well-founded, since self-play methods are designed to handle the scenarios with scarce annotated data, where fine-tuning LLMs while maintaining diversity over multiple benchmarks is inherently challenging.  In our experiments, we observe that SPACE indeed demonstrates relatively marginal performance improvements on certain benchmarks, such as Winogrande. However, it is important to note that this is ***a common issue*** with self-play methods, not just specific to our SPACE. Nevertheless, we would like to emphasize that SPACE is proposed to address the instability issue of SPIN, and we consider the investigation of diversity for self-play methods a meaningful direction and leave it as the future work.\n\n---\n\n**Reference.**\n\n[1] Investigating regularization of self-play language models. 2024.\n\n[2] Dynamic noise preference optimization for LLM self-Improvement via synthetic data. 2025.\n\n[3] Generative adversarial nets. 2014.\n\n[4] Generative adversarial imitation learning. 2016.\n\n[5] Measuring mathematical problem solving with the math dataset. 2021.\n\n[6] Can a suit of armor conduct electricity? a new dataset for open book question answering. 2018.\n\n[7] Judging llm-as-a-judge with mt-bench and chatbot arena. 2023.\n\n---\n\nWe will carefully revise the paper according to your suggestions. If our responses have properly addressed your concerns, we would appreciate it so much if you could ***re-evaluate*** our work. We are also happy to provide further clarifications during the reviewer-author discussions if needed."}]

Author rebuttal:

Meta-review: This paper presents a novel self-play fine-tuning method inspired by noise contrastive estimation. The paper identifies a failure mode of a previous approach, SPIN, and shows that a noise contrastive estimation approach would mitigate the problem. The proposed method is sound and principled, and the paper is written clearly. The main concerns raised during the discussion include 1) only having a single base model in their experiment, 2) limited benchmark coverage, and 3) fairer comparison with SFT. Also, other issues, including diversity and connection to other prior works, are discussed. These concerns were addressed in the authors' response. Overall, the paper appears to be a solid contribution to the community, and I recommend acceptance. The authors are strongly encouraged to incorporate the feedback from the reviewers in the updated manuscript.
Decision: Accept (poster)