Title: SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Language Models

Self-play Fine-tuning

Noise Contrastive Estimation
Abstract: Self-play fine-tuning has demonstrated promising abilities in adapting large language models (LLMs) to downstream tasks with limited real-world data. The basic principle is to iteratively refine the model with real samples and synthetic ones generated from itself. However, the existing methods primarily focus on the relative gaps between the rewards for two types of data, neglecting their absolute values. Through theoretical analysis, we identify that the gap-based methods suffer from unstable evolution, due to the potentially degenerated objectives. To address this limitation, we introduce a novel self-play fine-tuning method, namely \underline{S}elf-\underline{P}l\underline{A}y via Noise \underline{C}ontrastive \underline{E}stimation (SPACE), which leverages noise contrastive estimation to capture the real-world data distribution. Specifically, SPACE treats synthetic samples as auxiliary components, and discriminates them from the real ones in a binary classification manner. As a result, SPACE independently optimizes the absolute reward values for each type of data, ensuring a consistently meaningful objective and thereby avoiding the instability issue. Theoretically, we show that the optimal solution of the objective in SPACE aligns with the underlying distribution of real-world data, and SPACE guarantees a provably stable convergence to the optimal distribution. Empirically, we show that SPACE significantly improves the performance of LLMs over various tasks, and outperforms supervised fine-tuning that employs much more real-world samples. Compared to gap-based self-play fine-tuning methods, SPACE exhibits remarkable superiority and stable evolution.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the noise contrastive estimation-based self-play fine-tuning approach, called SPACE. This method addresses the training instability issue of SPIN, a prior method in this direction. SPACE independently optimizes the absolute reward values for each type of data, ensuring a consistently meaningful objective and thereby avoiding the instability issue. Empirical results on Mistral and Zephyr demonstrate the advantages of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well-written in most places and easy to follow.  \n- The provided empirical results seem promising.  \n\nWeakness:\n- The criticism of SPIN seems only empirical, which lacks rigor. It is unclear whether this is due to a fundamental limitation or hyperparameter tuning. The argued extreme case—where the generated response matches the real one—is not a bug for me, as SPIN may already learn the data distribution in this scenario.  \n- Lack of related work discussion. This paper lies within the framework of statistical distribution matching but fails to discuss seminal works like Generative Adversarial Nets (GANs) and Generative Adversarial Imitation Learning (GAIL). \n- Empirical results are rather weak. Mistral and Zephyr are relatively weak models, and results on these models may not generalize to modern ones.  \n- Empirical comparisons with Supervised Fine-Tuning (SFT) are not rigorous. The proposed method requires more compute due to synthetic data generation and additional gradient iterations. Thus, comparisons should include more epochs (even beyond 4). The results in Figure 5 do not convincingly justify the comparison with the same human-annotated data.  \n\n--- \n\nGoodfellow, Ian J., et al. \"Generative adversarial nets.\" Advances in neural information processing systems 27 (2014).\n\nHo, Jonathan, and Stefano Ermon. \"Generative adversarial imitation learning.\" Advances in neural information processing systems 29 (2016).\n\nQUESTIONS:\n1. Given the both SFT, SPIN, SPACE can theoretically converge to the underlying data distribution, do you believe what are exactly advantages of self-playing algorithms in this case? \n2. Do you believe the performance degradation in Figure 2 for SPIN is due to optimization instability or generalization issues?  \n3. Have you tested the method on Qwen2.5 and Llama-3.1 series? Models like Zephyr and Mistral are outdated (e.g., scoring 40-50 on simple benchmarks like GSM8K), making it hard to justify the superiority of the proposed method.  \n4. The OpenLLM Leaderboard primarily evaluates few-shot in-context learning (ICL) capabilities rather than downstream performance. Have you tested more challenging tasks like MATH?  \n5. Have you tested other datasets to ensure broader applicability of your conclusions?  \n6. A limitation of self-play algorithms is overfitting to the empirical data distribution, reducing diversity and harming subsequent RL fine-tuning. Have the authors observed this issue, where model outputs heavily conform to empirical data patterns?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCurrent self-play fine-tuning methods like SPIN suffer from instability during training iterations. Specifically, current methods focus on maximizing relative gaps between rewards of real and synthetic data, but when synthetic data become similar to real ones, the objective function can degrade into a meaningless constant, leading to performance collapse. Instead of comparing relative rewards, SPACE optimizes absolute reward values for real and synthetic responses separately. In this way, even when synthetic and real responses become similar, the objective remains meaningful and doesn't collapse. But there is concern about whether NCE's fundamental assumptions hold in their setting.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Hot topic;\n2. Clever Technical Insight\n3. Good performance. \n\nWeakness:\n1. Is the sync data a noise? NCE's core assumption that noise remains consistently different from real data. Real noise is uninformative and random. But the sync data is in fact informative, and gets better over time. And in fact the sync data can be better than real data, especially for datasets like Ultrachat200k. \n2. Limited datasets evaluated. Only Ultrachat200k is evaluated. \n3. Figure 5 may be an unfair comparison on SFT and SPACE. SPACE generates more than 200K data and runs for 4 times longer than SFT. To be fair, at least SFT should be run with the same computational budget. \n4. Missing recent methods. There are quite a few follow-up work after SPIN, that notices the model update issue. For example, Dynamic Noise Preference Optimization for LLM Self-Improvement via Synthetic Data https://arxiv.org/abs/2502.05400 also tries to add noise concept to resolve the model update stuck problem, although in another way.\n\nQUESTIONS:\n1. What happens if synthetic data quality degrades? Especially in the early stage?\n2. Could the iterative process gradually degrade capabilities in non-targeted areas?\n\nLIMITATIONS:\nTest with model under 7B; only one dataset; the Technical theory assumption lack of proof", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SPACE, a novel self-play fine-tuning method for large language models (LLMs) that leverages noise contrastive estimation (NCE) to address the instability issue in existing gap-based self-play approaches. SPACE treats synthetic samples as auxiliary components and discriminates them from real data via binary classification, independently optimizing absolute reward values for both types of data. Theoretically, the authors prove that SPACE converges stably to the real-world data distribution. Empirically, SPACE outperforms SPIN and SFT with more data, on diverse tasks from the HuggingFace Open LLM Leaderboard. It demonstrates stable performance evolution across iterations, avoiding the degradation seen in gap-based methods, and highlights the effectiveness of NCE in stabilizing self-play fine-tuning for LLMs with limited annotated data.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The paper points out an interesting reason why SPIN stops improving quickly, gives a reasonable explanation, and then comes up with a new loss function based on NCE. The new loss function has a good theoretical basis.  \n2. The results on various benchmarks show that it's significantly better than SPIN and its variants, and it seems to keep getting better over more rounds. So the method is good both in theory and in practice.  \n3. I like the analysis experiments, they're complete. They rule out possible counfounding factors that could explain why SPACE works better, like comparing more training rounds on the data from iteration 0. Also, the sample efficiency shown in Figure 5 is impressive, and I like how the results are presented.  \n4. The analysis of the reward decline problem is interesting too. It shows that SPACE properly reduces the problem, which helps explain why SPACE works well.  \n\n**Weaknesses:**\n1. When looking at the gradient in Eq. 9, we can see it's similar to the policy gradient in RL. Since SPACE uses annotated data, it might also be related to off-policy RL. Given how much attention RL is getting lately, it would have been good to discuss this connection.  \n2. Why does SPACE do so much better than SPIN on GSM8K and IFEval? With such big improvements, some explanation or examples would be good.  \n3. The method only looks at one model, so it's not clear if it works for other models.  \n4. There's no proof for the theorems. They should at least be in the appendix.\n\nQUESTIONS:\nSee weaknesses above\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SPACE, a self-play fine-tuning framework for large language models that avoids the instability of prior reward-gap–based methods like SPIN. SPACE reformulates the objective using a noise contrastive estimation (NCE) loss that computes a symmetric log-probability ratio between the current model and a frozen past model. This avoids the reward-gap collapse observed in SPIN as models converge. The authors provide theoretical guarantees that SPACE retains non-vanishing gradients and is preference-optimal under decomposable reward assumptions. Empirically, SPACE outperforms SPIN, DPO, and supervised fine-tuning on GSM8K and IFEval, offering smoother training dynamics and achieving alignment without human-labeled preferences.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n1. The paper replaces reward-gap comparisons with a log-probability contrastive formulation, avoiding instability due to shrinking reward signals.\n2. Theoretical Soundness: Proves consistency with preference optimization and non-vanishing gradients, even as the model improves over iterations.\n3. Empirical Performance: Demonstrates strong results on reasoning tasks like GSM8K and IFEval using self-play without human preference data, outperforming SPIN and achieving performance comparable to or better than DPO in several settings.\n4.Training Stability: SPACE maintains more stable and consistent performance gains across training iterations, whereas SPIN suffers from reward-gap collapse—an issue that limits the scalability of self-play fine-tuning.\n\n# Weaknesses\n\n1. Assumes Frozen Baseline Quality: Performance depends on the frozen past model being appropriately calibrated. If too weak or too strong, the contrastive learning signal may be ineffective.\n2.No Diversity Evaluation: The paper does not analyze whether SPACE encourages mode collapse or overly conservative outputs, particularly as self-play proceeds across iterations. Output diversity is a key quality consideration in alignment methods and could affect robustness\n3. Limited Benchmark Coverage: Focused on factual, single-turn tasks (GSM8K, IFEval); the behavior of SPACE in open-ended settings like dialogue or summarization is unexplored.\n\nQUESTIONS:\n# Questions\n\n1. How does SPACE affect response diversity and exploration?\n    Does it suppress generation diversity by continually pushing away from previous outputs? Could this lead to degenerate or overly cautious outputs over time?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}