The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models
Authors: Ke Ji
Jiahao Xu
Tian Liang
Qiuzhi Liu
Zhiwei He
Xiaoyuan Liu
Xingyu Chen
Junying Chen
Benyou Wang
Zhaopeng Tu
Haitao Mi
Dong Yu
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models; Efficient Reasoning; Unsupervised Learning
Primary area: deep_learning
TL;DR: Unsupervised Prefix Fine-Tuning Method for Reasoning Models

Abstract: Improving the reasoning capabilities of large language models (LLMs) typically requires supervised fine-tuning with labeled data or computationally expensive sampling. We introduce Unsupervised Prefix Fine-Tuning (UPFT), which leverages the observation of Prefix Self-Consistency -- the shared initial reasoning steps across diverse solution trajectories -- to enhance LLM reasoning efficiency. By training exclusively on the initial prefix substrings (as few as 8 tokens), UPFT  removes the need for labeled data or exhaustive sampling. Experiments on reasoning benchmarks show that UPFT matches the performance of supervised methods such as Rejection Sampling Fine-Tuning, while reducing training time by 75\% and sampling cost by 99\%. Further analysis reveals that errors tend to appear in later stages of the reasoning process and that prefix-based training preserves the model’s structural knowledge. This work demonstrates how minimal unsupervised fine-tuning can unlock substantial reasoning gains in LLMs, offering a scalable and resource-efficient alternative to conventional approaches.

The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models
Ke Ji .Jiahao Xu* 2 Tian Liang*.2 Qiuzhi Liu Zhiwei He² Xingyu Chen?, Xiaoyuan Liu² Zhijie Wang² .Jnyying Chen' Benyou Wang
Zhaopeng Tut:2 Haitao Mi? and Dong Yu?
The Chinese University of Hong Kong, Shenzhen Tencent
Abstract
Improving the reasoning capabilities of large language models (LLMs) typically requires supervised fine-tuning with labeled data or computationally expensive sampling. We introduce Unsupervised Prefix Fine-Tume Tuning (UPFT), which leverages the observation of Prefix Self-Consistency- - the shared initial reasoning steps across diverse solution trajectories - to enhance LLM reasoning efficiency. By training exclusively on the initial prefix substrings (as few as 8 tokens), UPFT removes the need for labeled data or exhaustive sampling. Experiments on reasoning benchmarks show that UPFT matches the performance of supervised methods such as Rejection Sampling Fine-Tuning while reducing training time by 75% and sampling cost by 99%. Further analysis reveals that errors tend appear in later stages of the reasoning process and that prefix-based training preserves the model's structural knowledge. This work demonstrates how minimal unsupervised fine-tuning can unlock substantial reasoning gains in LLMs, offering a scalable and resource-efficient alternative conventional approaches.
1 Introduction
Large language models (LLMs) have demonstrated remarkable performance across a wide range of natural language understanding and generation tasks, primarily due large-scale pre-training and subsequent instruction fine-tuning on high-quality datasets [16] 27). Despite these successes, enabling LLMs exhibit systematic reasoning capabilities remains challenging endeavor (2) 20]. In multiple domains-from mathematical problem solving to logical and commonsense reasoning-models often rely large amounts of human-annotated data extensive sampling-and- filtering pipelines to achieve high accuracy.
Recent inquiry has introduced approaches such Rejection Sampling Fine-Tuning B3 (RFT) and Self- Taught Reasoner [34] (STaR) to leverage moodel-generated solutions for iterative self- improvement, often requiring multiple candidate responses and subsequent filtering verification steps. While these methods can yield impressive gains, they are time-consuning, resource-intensive.. and assume ready access correct targets or verification mechanisms particularly challenging when no reliable ground-truth is available.
"Equal Contribution. The work was done when Ke Ji, Zhiwei He, Xingyu Chen, Xiaoyuan Liu, and Zhijie Wang interning Tencent. CCorrsppnndecce to: Benyou Wang Baanungbenyouuuuu....ooo edu.en> and Zhaopeng Tu <zptu <xptu@teenent.com.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Rejection Sampling Fine- Tine-Tuning (Label Required)
88.4
Reduce 74,39 Takens
B- 100, Since 40% heavler, A- 100 c10% (00) - 140, 10% less than C, to C-07 Given A-140, 14000.7
Question
100/200
The weight A 40% than the weight D 30% than weight What
87
57.0
Let B- 100, Since A 40% heavler, A 100 (40% of 100) -I 1900 A 10% less than and A-1400 C- 1400 (70/100) 96. Racio BrC 10099 50049.
Accuracy
87.0
DeepSeek-R1 Bieieck--R-Diiill---e ---iiill-Quen--9
rusio weight ofC eepressal
85.0
Reduce 81.3% Tokans
Let R- 100. Since A 40% heavier, A- 100 (40% 1000 - 140, A 10% loss than C C- 2007. With 100, ratio BIC - (20077) 3/2.
85
UPFT
Qwen2. M-Mahb-BB
RFT
Baseline
Unsupervised Prefix Tine-Tuning (Label Free)
0
5
10
15
20
Question
Let B- 100. Since A 40% heavier. A 100
Tuning Tokens (M)
Fine-tuning methods
(b) Token-accuracy plot MATH500
Figure I: (a): Conventional Rejection Sampling Fine- Tine-Tuning (RFT) method (upper panel) involves generating multiple responses to a given question and then applying posterior filtering to discard trajectories that lead to incorrect answers. Finally, the correct trajectory is used for final training. In contrast, the proposed UPFT method (bottom panel) requires only prefix minimal initial tokens of single generated sample, eliminating the need for labeled data rejection sampling. (b): Our proposed UPFT matches the performance of supervised RFT, while reduces tuning cost by 75+%.
In this paper, we propose an unsupervised fine-tuning method that requires only single pass of gooer--geeertte responses per question, coupled with prefix- -based fine-tuning. Our key insight is that different solution paths often share a common initial reasoning phase, which we call "Prefix Seff-Consistency.. By fine-tuning on these minimal prefixes (as few as 8 tokens), we effectively guide the model's inherent reasoning structures toward more systematic solution strategies while avoiding the complexity and cost of large-scale or iterative filtering. Moreover, we preserve the model's overall problem-solving format through a small proportion of full-token fine-tuning experiments, ensuring the model does not lose its length generalization and cstruction-followiing abilitics.
We conduct comprehensive experiments across four training corpora and evaluate our method on four widely used reasoning benchmarks. UPFT demonstrates exceptional data efficiency and flexibility, outperforming conventional full-token tee-tuning in unsupervised sampling setting. Furthermore, UPFT achieves performance competitive with the widely used RFT method, which relies on labeled verification or large-scale rejection sampling, while requiring only 25% of the training time and 1 % of the sampling time. Our approach can be easily adapted to various datasets, tasks, and L.LM architectures, highlighting its flexibility and universality for developing robust problem-solving capabilities in large language models.
Our main contributions are as follows:
We identify Prefix Self-Consistency as a critical phenomenon, showing that early reasoning steps are highly consistent across trajectories, enabling efficient self-improvement learning.
2. We propose an unsupervised fine-tuning method UPFT that leverages only prefix substrings (i.e., minimal initial tokens of model-generated responses), eliminating the need for labeled data or rejection sampling.
3. We conduct comprehensive empirical validation to demonstrate that UPFT exhibits ex- ceptional data efficiency and versatility, outperforming vanilla full-token fine-tuning in unsupervised settings and achieving competitive performance with RFT while drastically reducing sampling and tuning overhead.
Prefix Self-Consistency
central observation of this work is that different solution trajectories for the same question often share common initial reasoning phase, even if their later steps diverge substantially. We term this phenomenon prefix self-consistency. In other words, when an LLM generates multiple solutions for single question, the opening tokens typically restatements of the problem or the setup of its

---

1,000
100
Llama-8B twen-MMth-7BB
Average Trajectories
80
650.5
Accuracy
60
500
09.4
40
308.1
Llama: Correct
219.9
250181.9
Llama: Incorrect
124.8
20
Qwen: Correct
43.6
20.2
2
43
4
1.4
Qwen: Incorrect
0
0
2
4
8
6
8 3 512
Prefix Length
Prefix Length
(a) Average number of covered trajectories.
(b) Success rate of trajectory prefixes.
Figure An empirical investigation of prefix self-consistency. investigate (a) the average number of trajectories covered by prefixes at different lengths, and (b) the success rate of 32 rollout samplings from prefixes for both correct and incorrect trajectories.
initial logical steps tend to be highly consistent across these separate responses. In this section, we identify two key characteristics of reasoning prefixes demonstrate the existence and plausibility of prefix self-consistency. We report results on 500 questions randomly sampled from the PRM training dataset.
Early reasoning steps are highly consistent across reasoning trajectories. We sampled 1,000 trajectories for each question instance (using temperature of 0.6) and calculated the average trajectories per prefix different lengths, as shown in Figure 2(a) The results confirm that reasoning processes exhibit strong prefix self- self-consistency, with remarkably high degree of prefix overlap among multiple trajectories for both models. As we increase (i.e., longer prefixes), the average number of samples per prefix pattern decreases, yet both models maintain consistent patterns well Twen-Math-7B-Innsruue preserves shared
beyond the very first tokens. Notably, the math-specialized 1---ma--.-----Issttiee IIstrrut,,, evidenced by the prefixes more consistently than the general-purpose Llama-3.1 higher values in all cases. This suggests that Qwen-Math- MB's generation process is more tightly anchored in its initial reasoning steps, whereas Llama-3.1 --B introduces prefix variability. An example is shown in Appendix
Errors predominantly occur in later reasoning steps. To empirically validate the plausibility of the reasoning prefix, we conducted rollout samplings for each token in both a correct and an incorrect trajectory for each question. Figure 2(b) illustrates the success rate of these rollout samplings across both correct and incorrect trajectories. Two key observations can be made:
Incorrect trajectories diverge significantly from correct ones later reasoning steps. The rollout success rate for correct trajectories steadily rises as t grows for both models. For example, with and reaches 96.2% by 512. This trend is
Llama, the success rate starts 54.2% expected: sampling from the later tokens of a correct trajectory leverages more accurate contextual information, increasing the likelihood of staying on a correct path. In contrast, the rollout success rate for incorrect trajectories declines substantially as f increases, indicating that once an incorrect path is taken, recovering through rollout sampling becomes far less likely. This contrast highlights that enrors tend to occur in the later stages of generation.
Initial reasoning prefixes exhibit self-consistency. At the earliest token positions (t < 16), the rollout success rates for both correct and incorrect trajectories are strikingly similar. For Qwen- Math-77 7B = 4, the success rates are 55.7% and 86. 1% correct and incorrect trajectories, respectively. This near-identical performance in early steps suggests that the initial tokens generated whether they lead correct or incorrect final answer are statistically indistinguishable in terms of leading to correct result when used starting point for rollout sampling.

---

Overall, these results indicate that while mistakes in reasoning are more prone to appear and accumu- late in later steps, the initial reasoning prefixes generated by LLMs exhibit a considerable degree of self-consistency. This consistency is reflected in the similar early-stage rollout success rates across both correct and incorrect trajectories. Consequently, enhancing the accuracy and robustness of these early reasoning steps may be a key factor in improving the overall reliability of reasoning tasks.
3 Methodology
Based on the aforementioned observation, we firstly provide a Bayesian explanation in Section 3.1 identify the coverage and accuracy of Prefix Self-Consistency in Section 3.2 and propose our UPFT Section 3.3
3.1
Understanding Coverage and Accuracy from Bayesian Perspective
We demonstrate that the SFT process can be naturally interpreted within a Bayesian framework. Consider dataset (2: y) DD where each æ represents an input and is the corresponding ground- truth answer. For each input z, we conduct the reject sampling process of SFT aims to maximize the following objective:
B (( p00)pp()1:00 log p(r(z) HER (r .)
where (() ~ p(-[2;;) 0) denotes the generated reasoning trace, and r En (r((*),)) denotes the rejection sampling, which only picks the reasoning trace whose final answer y.
From a probabilistic perspective, we can decompose the conditional probability p(y|z) by marginal- izing over all possible reasoning traces that could lead to the answer, and derive the following lower bound:
logp(y)>) log Σ p(y,r(a)
(2)
log Σ P(ulr,z))((1b
(3)
Σ p(r[z) logp(y)r))
(4)
Ee) logp(y/r,æ)
(5)
The correct answer logp(y|<) ()) is lower-bounded by:
The expectation (x=)p( is with respect to the distribution of reasoning trace p(r[z). The term p(r(z)) denotes prior probability reasoning trace r given input æ, which denotes the prefix coverage the entire reasoning trace space. This corresponds the average trajectory covered by prefixes in Section indicated by Figure 2(a)
Coverage:
Accuracy: Term p r, x) within the expectation can be viewed as the likelihood of the answer being correct given specific reasoning trace which is the accuracy. This is also observed by our previous observation Section 2 in Figure 2(b)
3.2 Prefix Coverage and Accuracy
To incorporate the concept prefix spans, we utilize the chain rule for conditional probabilities. We can decompose the probability of the full reasoning trace given input x
p(rlz) p(rc))22))) = p(reile))(r2s
where r<t denotes the prefix of before time step f. Substituting this decomposition of p(rlar) into the original lower bound from Equation (4), get:

---

log p(ylz) 2 Σ p(r(z) logp(ulr,x)
- Σ p(rea/x))
P2
E((racl))
Σ Zo
T21
Σ [1(rellt)L(ree
(6)
where,
L(r<,,)) Dpttaerreeetoooplleeeea..a
= -Brerrp((reeea)) (logp(u)r<au T2),2))
(7)
Equation 6 presents the original lower bound Equation ( in terms of prefix spans of the reasoning trace. It also states the importance of coverage and accuracy of the prefix span rext
Prefix Coverage: The outer expectation in Equation (6) (e is with respect to the distribution of prefixes p(r=c (z). The term p(r (z) represents the prior probability of the model generating prefix reasoning trace Ter given the input This denotes the coverage of prefix reasoning traces.
For each prefix T<c, the term L(res, in Equation 7 is the conditional lower bound given the prefix r<t- is the expected log-likelinn likelihood of the answer y given that the reasoning process starts with prefix ret, averaging over all possible suffixes T>r that can follow r<c. This can be viewed the expected accuracy when reasoning is conditioned to start with the prefix ret-
Prefix Accuracy:
So far, we have mathematically derived representation of the lower bound in terms of prefix spans for both prefix coverage and prefix accuracy. To achieve superior performance through prefix fine-tuming mee-unnngg specific prefix length t is required to trade off both prefix coverage and prefix accuracy terms.
3.3 Our Method: Unsupervised Prefix Fine- Hine-Tuning
Motivated by previous theoretical justification, we develop our method Unsupervised Prefix Fine- Tuning (UPFT), which is apt to maximize the coverage of the reasoning trace while maintaining a relatively high accuracy by only learning from the proper prefix of the reasoning trace. This strategy shares several advantages. First, UPFT exhibits enhanced coverage of all possible correct reasoning traces. As demonstrated in Section and Section [.I] the prefix of the reasoning traces represents set of reasoning traces with the ıdentical prefix, which increases the coverage term of Equation (5)s lower bound. Second, by solely decoding a certain length of the prefix, UPFT inherently enjoys improved computational efficiency. In contrast, conventional Reasoning From Traces (RFT) methods necessitate rejecting sampling over entire reasoning traces. This approach incurs significant inference burden, particularly scenarios where generating valid full reasoning traces from the base model proves challenging.
Our strategy consists of the following steps:
Given a training set (x., y) E DD we only decode prefix span rice of reasoning trace from the base model rez p=; 0);
We conduct the SET learning the prefix spans of the reasoning trace T-r with NLL objective;
5

---

Structure Tuning To avoid the catastrophic forgetting [9), we jointly conduct the prefix tuning and full trace tuning by a data split ratio p%., which yields a prefix dataset D, and full trace dataset Dy- We utilize specialized task template, as shown in Appendix A for prefix tuning to enhance the model's s ability to learn effectively from prefix-specific patterns. Note that the full trace tuning does not require correctness verification well. In summary, the overall objective is combination of UPFT and unsupervised SFT by maximizing the following objective:
log (oge(r=)(2;0)) 0) 2nD, [((((()));;) rop(1220)
(8)
Notably, compared to conventional RFT objective, our method does not require any rejection sampling process denoted by 1(x((), y) in Equation 11. Consequently, our method is naturally suitable for learning from unproven questions with extreme data efficiency.
4 Experiment
In this experiment, we compare UPFT with traditional supervised methods such as SFT and RFT in both supervised and unsupervised sampling settings. also evaluate its scalability by varying the self-training corpora and backbone models.
4.1 Experiments Setup
Backbone LLMs For our experiments, we selected three representative open-source backbone models: the general-purpose 1aama-3.1-B--Issee [4), the math maath-specialzzed Qwen2.5-Math-78 Instruct [29] optimized for mathematical tasks, and the long- ann-rensoning DeepSeek- R1-ttstill- 7B 8 distilled from DeepSeek-R1.
Fine-Tuning Tuning
We used four datasets to generate self-training data for fine-tuning:
1. PRM (12K instances; This dataset includes 4.5K MATH 110 test problems, which are drawn from the PRMS0OK training set.
2. OMT2 (600K instances; (26): This is a subset of 600K unique questions extracted from the OpenMathInstruur22 math-instruction tuning dataset.
3. LIMO (819 instances; 31): A high-quality training dataset specifically focused on challenging problems.
4. U-Hard (100K questions): This dataset, introduced in this work, is designed to explore the potential of our method in an unsupervised setting.
U-Hard was curated through an extensive collection of questions from publicly available online sources. Adhering to the Omni-Math approach [6], we labeled the collected data by difficulty and subsequently filtered out lower-difficuly questions. This process resulted in dataset composed exclusively of challenging questions, thereby ensuring U-Hard's 's practical relevance to real-word scenarios. To ensure fair comparison, UPFT employs the same hyperparameters as conventional SFT when fine-tuning models. metais. about hyperparameters is shown in Appendix During the inference stage, we adopt a prompted zero-shot setup and use standard greedy decoding, wherein models are directed to answer each question using natural language instructions without any accompanying contextual demonstrations.
Benchmarks We evaluated the performance of our method and the baseline methods on four widely used benchmarks: GSM8K MATH500 1O,, AIME24 [8] and GPQA Diamond 24
Tuning Scenarios
We employed two experimental settings:
Unsupervised sampling: we took one sample per question without any posterior filtering.
Supervised sampling: Following RFT 33), we sampled the responses for each question 16 times and used the ground truth answers to select correct response.

---

55
90
55
90
Qwen-Math- 7B Accuracy
30.0
Qwen-Math- Accuracy
Llama- (BB Accuracy
Llama 8B Accuracy
85.6
85.2
85.2
82.6
35.
$1.8
350
83.9
35.4
85.6
50
7.8
50.2
Llama-8B Qwen-Math-7B
10.2
Llama-8B Qwen-Math-78
45
80
40
2
4
8
16
32
64
5
10
50
100
Prefix Length
Strueture Tuning Ratio (%)
(a) Impact of Prefix Length
(b) Impact of Structure Tuning Ratio
Figure 3: Impact of (a) prefix length and (b) structure tuning ratio reasoning accuracy.
4.2 Ablation Study
In this section, we evaluated the impact of two hyperparameters UPFT on the PRM- 12K dataset using Llama-3. 1-8B-Instrrcc' and Qwen2.5-Math-77: Instruct in the unsupervised setting and report the results on the MATH500 test set.
Impact of Prefix Length Figure 3(a) illustrates how prefix length affects reasoning accuracy, Each
model has its own optimal prefix length for peak performance: Llama-3.1-88 1-8B-Innstrue achieves its highest average accuracy (52.0%) with an 8-token prefix, whereas Qwen2.5-Math-777 3--Math7BB-Instie remains stable over prefix lengths from 8 to 32 tokens. The math sath-speeialized
Twen2.5-Math Instruct is less sensitive to prefix length, likely because it was trained on the high-quality large-scale For subsequent experiments, we to tokens and Qwen2.5-Math ---Mathh-BB-Intrucc to 32 tokens. In the following experiments, we set the prefix length to 8 Llama-3. --B--Instrcc and for Qwen2.5-Math- Instruct. Because the responses from the long-reasoning DeepSeek-RI R1-Distill--wer (B are generally more than four times longer than those of Swee225-Mathh Matth-BB-Insteeee Instruct,, we use prefix length 128 tokens for the former.
math corpus, which reinforces its stability on the MATH500 set 1a-ma--.--BB-Innttteee
Impact of Structure Tuning Ratio Figure 3lb) the effect of the structure tuning ratio (p). The performance of both models generally improves with increasing P. peaking at p 10%. This result supports our hypothesis that even minimal structural supervision effectively guides models towards generating complete responses. However, exceeding this ratio leads to performance decrease. Consequently, we set p to 0.1 in UPFT for all datasets, except LIMO. For the smaller LIMO dataset, we increased 0.3 for structure tuning.
4.3 Unsupervised Fine-Tuning
Table [ presents the results for the unsupervised sampling setting, where the data not filtered based on the correctness of the extracted answer. We compare these outcomes to those obtained using conventional SFT (2I], which fine-tunes models on training data without any self-improvement or test-time verification.
UPFT demonstrates superior performance compared to SFT in unsupervised fine-tuning. UPFT consistently outperforms conventional SFT across various self-training datasets and backbone models under the unsupervised sampling setting. For instance, using the U-Hard dataset with Swen2.5-Math--7B. -B- UPFT achieves an average accuracy of 54.5% across all benchmarks, exceeding the 51.3% attained by conventional SFT. Likewise, with DeepSeek-RI Ri-eekt--- --Distill-Qwen--l and U-Hard, UPFT attains average of 61.6%, whereas SFT achieves 56.4%. These results highlight UPFT's effectiveness leveraging unsupervised data to enhance reasoning, thereby reducing the reliance on labeled data. They also indicate the versatility and broad applicability of UPFT across
7

---

Table 1: Model performance under the unsupervised sampling setting, without any filtering based on the correctness of the final answer. "Length" denotes the average length of the tuning samples in each dataset. Models trained with SFT and the proposed UPFT generate responses of similar length.
Fine-Tuning
Testsets
Method
Data
Avg. Length
GSM8K
MATH500
AIME2024
GPQA
Ave.
Llama-3
1-8B-Instruct
82.0
51.0
3.3
8.6
36.2
SFT
PRM
175.8
83.8
48.4
3.3
8.6
36.0
+ +UPFT
(12K)
15.8
85.4
52.0
6.7
9.1
38.3
Qwen2.5-Mathe
Mattt-B-Inntruuc
95.2
84.0
16.7
9.6
51.4
+ SIFT
PRM
300.
95.8
83.4
13.3
9.1
50.4
+ UPFT
(12K)
51.4
95.5
85.6
20.0
9.6
52.6
+ SI7T"
OMI2
533.2
95.4
83.4
13.3
6.6
49.7
UPFT
(600K)
67.5
95.4
86.4
20.0
9.6 52.9
+ SFT
LIMO
49T.8
95.8
84.2
20.0
7.6
51.9
+ UPFT
(0.8K)
77.8
95.6
85.8
20.0
8.6
52.5
+ SFT
U-Hard
393.3
95.5
83.4
16.7
9.6
51.3
+ UUFF
(100K)
68.2
96.0
85.6
26.6
9.6
54.5
PeepSeek-RI-Distiee R-Diisitl------ee--
88.6
87.0
40.0
13.1
57.2
+ SFT
LIMO
2029.5
85.7
87.0
40.0
12.1
57.2
+ UPFT
(0.8K)
757.7
92.0
89.4
43.3
17.7
60.6
+ SFT
O-Hard
3440.4
89.7
87.0
36.7
12.1
56.4
+ UPFT
(100K)
-61.7
91.4
89.2
50.0
15.7
61.6
different LLM architectures, supporting its claim of being flexible, universal method for improving reasoning capabilities.
The benefits of UPFT are more pronounced on complex reasoning tasks. The performance gains UPFT over conventional SFT are especially evident on more challenging benchmarks such as AIME2024 and GPQA. For example, on AIME2024 with the U-Hard dataset, UPFT achieves 26.6% accuracy with Qwen2.5-Math-775 -ath--aat---BB-Istrrucc, notable improvement over the 16.7% achieved by conventional SFT. A similar trend is observed with DeepSeck-RI- DiisSsck-R---ssssl---------- where UPFT reaches 50.0% compared with SFT's 36.7%, showing UPFT's capability to improve reasoning on difficult problems, aligning with our claim of enhancing reasoning capabilities effectively and efficiently.
on AIME2024.
The U-Hard dataset maximizes UPFT's potential through difficulty- curation. Training with U-Hard yields the strongest performance improvements across models, particularly for complex benchmarks. Qwen2.5-Math
achieves 26.6% on AIME2024 with U-Hard 10 points higher than with PRM data. This demonstrates that challenging questions provide richer signals for prasd self-improvement,, as they demand more sophisticated initial reasoning setups. Using U-Hard, UPFT achieves the highest average accuracy with both Qwen-7B, surpassing other datasets such as PRM- 12K and OM12-600K. These findings demonstrate that UPFT effectively leverages diverse and challenging questions refine the model's reasoning skills in an unsupervised manner, showcasing its data efficiency and versatility.
and DeepSeck-RR ---ck--i--
UPFT achieves dramatic efficiency gains through minimal token training. UPFT reduces train- ing sequence length by 82.6-94.7% compared to SFT across datasets, with U-Hard samples averaging 68.2 tokens vs. SFT's 393.3. This directly translates to 6.3- 16-16.7x 7x faster training iterations and reduced memory consumption. Notably, DeepSeek- RI RR--Dstill--wen.
attains better performance with UPFT's 561-token samples than SFT's 3,440-token sequences, proving that strategic prefix selection preserves critical learning signals. These efficiency characteristies validate UPFT's practical value for resource-constrainee applications while maintaining exceeding baseline performance.

---

Table 2: Model performance under the supervised sampling setting the PRM-12K dataset, with filtering 16 sampled solutions based on the correct extract answer. "#Tokens" denote the number of tokens in each phase. Compared RFT, V-STaR requires two additional samples tune the verifier.
#Tokens
Testsets
Method
Sampling
Tuning
GSM8K
MATH500
AIME2024
GPQA
Ave.
Llama-3. 1lama-3.1-8P 1-8---33--B--nttrreee
82.0
51.0
3.3
8.6
36.2
+RFT
2.3M
86.0
52.0
6.7
9.1
33..5
36.9M
+ V-STaR
6.8M
85.4
52.6
6.7
8.6
38.3
+UPFT (Ours)
0.2M
85.4
52.0
6.7
9.1
38.3
LLable Filter
36.9M
0.2M
85.8
53.4
6.7
9.1
38.8
Qwen2.5-Menn Instrrcct
95.2
84.0
16.7
9.6
51.4
RRFT
3.2M
95.7
85.2
20.0
9.6
52.6
51.7M
V-STTRR
9.6M
96.0
85.4
20.0
10.1
52.9
+ (UPFT(Our))
0.6M
95.5
85.6
20.0
9.6
52.6
+Lable Filter
51.7M
0.6M
96.0
85.6
20.0
10.1
52.9
ReepSeek-RI--issilll DB
88.6
87.0
40.0
13.1
57.2
+ +FF
318.0M
19.9M
90.7
87.0
40.0
11.1
57.2
++ +PPFT(OUn))
5.0M
9I.9
88.4
40.0
14.6
58.7
Lable Filter
318.0M
4.5M
92.3
89.2
40.0
13.6
58.8
4.4 Supervised Fine- Fine-Tuning
In the supervised sampling setting, we compare our approach with two SFT variants in Table
RFT 33 samples 16 candidate responses with label filter to identify a correct one.
V-STaR produces 16 responses for each tuning instance, trains a verifier on them for one iteration, and uses it test time select the answer from 4 completions.
UPFT achieves competitive performance with supervised methods while requiring significantly fewer tokens in both sampling and tuning. Across multiple model architectures, UPFT matches exceeds the performance of supervised baselines such RFT and V-STaR. On the Llama-3.1- 8B-Instruct backbone, UPFT attains an average reasoning benchmark score 38.3%, matching V-STaR (38.3%) and approaching RFT (38.5%). For -aaen----Maah----nntrrr identical average accuracy to RFT (52.6%), while using only 1.2% of the sampling tokens (0.6M vs. 51.7M). Most notably, UPFT enables DeepSeek-RI to achieve 58.7% average accuracy, which is 1.5 points higher than RFT, while requiring 16x fewer tuning tokens than RFT. This underscores how prefix-based fine-tuning effectively captures critical reasoning patterns without incurring the computational overhead of sampling 16 responses Moreover, UPFT maintains strong performance even when ground-trutt -ruth answers are available. By focusing on the shared reasoning prefixes, it does not sacrifice accuracy. These results reinforce our core claim: prefix-based based training captures the essential reasoning signals available in full trajectories. validating our Prefix Self- ceff-Consistency hypothesis. This hypothesis posits that the essential signals for effective reasoning are largely contained within the initial prefixes of successful solution trajectories, allowing for efficient learning and high performance.
UPFT achieves
question, as required by supervised basclines.
UPFT offers seamless integration with label verification for enhanced accuracy. An additional benefit UPFT is its inherent flexibility and compatibility with label verification techniques when such information is accessible. As demonstrated in the table's last rows for each model backbone, when UPFT enhanced with label filtering, it attains 58.8% average accuracy on DeepSeek-755 surpassing all baselines while maintaining a 4.4 tuning token advantage over RFT. For Qwen2.5- Math-7B-Instrun we match V-STaR's 52.9% peak performance without requiring compute-intensive verifier training or best-of-N inference. This dual capability demonstrates UPFT's unique adaptability it functions as a standalone unsupervised learner while seamlessly integrating supervision when available, making it practical for roal-world deployment across the supervision spectrum.

---

Related Work
Unsupervised Learning Unsupervised learning encompasses diverse methodologies, including pseudo-labeling (30], pivot-based approaches [22]. and adversarial networks [5]. Self-supervised
learning has since emerged as a dominant paradigm, particularly for language models, leading to the
success of pre-trained models like BERT 14] and others ( 23 More recently, self-introvement
techniques, such as self-rewarding 32), further advance unsupervised learning by enabling models iteratively enhance performance without external supervision. Building upon self-improvement. AL [3] extends this paradigm to unsupervised domain adaptation.
Self- Self-Improvement A family of methods, starting with STaR [. reinforced self- self- training 77.. and rejection fine-tuning [33], leverages the solutions generated by large language models (L.LMs) to iteratively update and improve the models themselves. These techniques involve fine-tuning the model on the generated solutions that produce correct answers. ReSTEM 25 views this fine-tuning reinforcement learning (RL) for solution-generatinn agent. Wang et al. [28] propose using a contrastive loss to enhance the likelihood of correct solutions over incorrect ones. The díscovery of successful solutions presents significant exploration challenge. Luong et al. [17] demonstrated that RL-based fine-tuning of an LLM is particularly difficult unless preceded by several steps of supervised fine-tuning. In An et al. [1] employed to edit the incorrect rationales generated by a smaller model, thereby providing positive data for its fine-tuning. However, Huang et al. 12 argued that LLMs have limited capacity correct their own reasoning flaws.
process expectation- expectatio-mmxximiaaiio
powerful LLM was
In contrast to the above approaches, this paper focuses on discovering self-supervised signals specifically for mathematical reasoning. We introduce UPFT, a simple and effective unsupervised post-training method requiring only questions and the LLM itself.
Conclusion
In this work, we presented an unsupervised fine-tuning method that enhances the reasoning capabilities of large language models using only prefix substrings as minimal guidance. Our approach leverages the inherent reasoning structures within pretrained models, exploiting the phenomenon of Prefix Self- Consistency where different reasoning trajectories share common prefixes. Extensive experiments demonstrated that our method outperforms traditional full-token fine-tuning and achieves performance comparable to supervised approaches like RFT, with significantly reduced training and inference times. This work highlights the potential of minimal unsupervised fine-tuning in improving the reasoning abilities of LLMs without relying on external supervision or extensive computational resources. Future work will explore the application of this method to other challenging tasks and investigate the theoretical underpinnings of Prefix colf-Conssttncy in more depth.
Acknowledgement
This work was supported by Major Frontier Exploration Program (Grant No. C10120250085) from the Shenzhen Medical Academy Research and Translation (SMART), the Shenzhen Science and Technology Program (JCYJ202208 18 18100001002) NSFC grant 72495131, Shenzhen Doctoral Startup Funding (RCBS2022 10000033000003330 Tianyuan Fund for Mathematics of National Natural Science Foundation of China (NSFC) (12326608), Shenzhen Science and Technology Program (Shenzhen Key Laboratory Grant No. ZDDSSS2023062200 1302006), and Shenzhen Stability Science Program 2023.
References
[1] Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. Learnngg from mistakes makes Ilm better reasoner. arXiv preprint arXiv:2310.20689, 2023.
[2] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. In ICML, 2024.
10

---

[3] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
[4] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint urXin:2407.21783 2024.
[5] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. anversrrd training of neural networks. Journal of machine learning research, 17(59):1-35, 2016.
[6] Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2440.07990 2024.
[7] Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahemn Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308. 088998 08998, 2023.
[8] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12944, 2025.
[9] Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, et al. Pre-trained models: Past, present and future. Al Open, 2: 225-250, 2021.
[10] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021.
[11] Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners. arXiv preprint arXiv:2402.06457 2024.
[12] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.0177 798, 2023.
[13] Ke Ji, Junying Chen, Anningzhe Gao, Wenya Xie, Xiang Wan, and Benyou Wang. Lims could autonomously learn without external supervision. arXiv preprint arXiv:2406. 006606 2024.
[14] Jacob Devlin Ming-Was Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACI-HIIT, pp. 4171-4186, 2019.
[15] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024.
[16] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. 2023.
[17] Trung Quoe Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. ReFT: Reasoning with reinforced fine-tuning, arXiv preprint arXiv:2401.08967, 2024.
[18] MAA Committees. Aime problems and solutions. htto:://aartooppobleessolti. com/ htwp:ppp/Atte__Poleemssossm
[19] Niklas Muennighoff, Zitong Yang. Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025.

---

[20] OpenAl. Gpt-4 technical report, 2023.
[21] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730- 277444 2022.
[22] Sinno Jialin Pan, Xiaochuan Ni, Jian-Tao Sun, Qiang Yang, and Zheng Chen. Cross-domain sentiment classification via spectral feature alignment. In Proceedings of the 19th international conference on World wide web, pp. 751-760, 2010.
[23] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99, 2019.
[24] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petry, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark, arXiv preprint 1. 2023.
[25] Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jachoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamalcldin Elsayed, Hanic Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel. Beyond human data: Scaling self-training for problem--oovving with language models. arXiv preprint 22.005688, 2023.
[26] Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data. arXiv preprint 24X.v:2410.01566 2024.
[27] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint ww.Xiv:2307.09288 2023.
(28) Peiyi Wang, Lei Li, Liang Chen, Feifan Song, Binghuai Lin, Yunbo Cao, Tianyu Liu, and Zhifang Sui. Making large language models better reasoners with alignment. arXiv preprins arXiv:2309.02// 2023.
[29] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui. Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412. 15115, 2024.
[30] Hai Ye, Qingyu Tan, Ruidan He, Juntao Li, Hwee Tou Ng, and Lidong Bing. Feature adaptation of pre-trained language models across languages and domains with robust self-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7386-7399, 2020.
[31] Yixin Ye, Zhen Huang, Yang Xiao, Ethan Cher, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprini arXiv:2502.03387, 2025.
[32] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024.
[33] Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308..0 0185,, 2023.
[34] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star; Bootstrapping reasoning with reasoning. Neural Information Processing Systems (NeurIPS), 2022.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction clearly describe our contribution, the algorithm, and the experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We mentioned our limitation in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

Justification:
We mentioned our Theory Assumptions in Section 2 and Section 3.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide a detailed description of our model architecture and present all the training details in Section 4 and Appendix B, including dataset and hyperparameter settings.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification:
We provide our code in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//uuuii wwwiii/gguie///eeeeeeoeic CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the full details in Section 4 and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, we mentioned in the Section 4 and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
15

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, mentioned in the Section 4 and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: cwwtrrisscccppppepp httpc//ccccccccccsssseeeeeeeeeee
didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: This paper strictly adheres the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work is foundational research and not tied to particular application.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
16

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, the paper properly credits the creators or original owners of assets (e.g.,code, data, models) and explicitly mentions and respects the relevant licenses and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, pwpprswi www.wwtththc....c com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
17

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, we mentioned in the Section 4.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact core methodology, scientific rigorousness, originality of the research, declaration is not required.
18

---

Answer: [Yes]
Justification: Our paper investigates the issue of efficient self-training in large language models, conducting research based multiple open-source large language model backbone.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy fattps://neuus wwwparrppsscooooffeeeeee /c//onnferences/2225//1 for what should or should not be described.
19

---

A Settings of the Experiments
A.1 Task Template for Prefix Tuning
For the prefix fine-tuning, we simply apply the task template shown in Figure
Task template used for prefix tuning.
[question] Please provide the initial step towards resolving the question. This step may serve as foundation but might not encompass the entire solution.
The task template used to leam from the prefix of the reasoning traces. represents the question that needs be answered.
Figure 4:
[question]
B Hyperparameters
Table The hyperparameters used our method on training corpora.
Hyperparam. PRM-12K
OM12-60K
LIMO
U-Hard
Optimizer
AdamW
Warmup Ratio
0.03
Learning Rate
1e-6
2--6
1e-6
LR Schedule
constant with warmup
Batch Size
Gradient Step
8
Max Length
4096
16384
Epoch
2
3
To ensure fair comparison, UPFT employs the same hyperparameters as conventional SFT when fine-tuning models, as listed in Table During the inference stage, adopt prompted zero-shot setup and use standard greedy decoding, wherein models are directed to answer each question using natural language instructions without any accompanying contextual demonstrations.
C Limitations
Validation with larger-scale models: our experiments, the choice of backbone was primarily based on the mathematical capabilities of the model. Our goal is to further enhance the model's reasoning ability at low cost. In future research within the RL framework, we will consider validating our approach on larger-scale models, such as QwQ-32B-Preview and -iseeee------issssll-ee---------
Study on sample-independent prefix selection strategy: Despite achieving satisfactory performance, this work is based on a heuristic prefix selection strategy. In future research, we will continue to design an adaptive sumple-indepnneee prefix selection for UPFT.
D Case study of Prefix Data
We provide a study of prefix data shown in Figure We see that the different 16 rollout samples have very similar prefixes.
20

---

Case Study
Question: Melinda has three empty boxes and 12 textbooks, three of which are mathematics textbooks. One box will hold any three of her textbooke, will hold any four of her textbooks, and will hold any five of her textbooks. If Melinda packs her textbooks into these boxes in random order, the probability that all three mathematics textbooks end in the same box can be written Find m.
where and are relatively prime positive integers.
The first 32 words (prefix substring) of all answers:
A1: determine the probability that all three mathematics textbooks up in the same box, we need to consider the total number of ways to distribute the 12 textbooks into the three A2: To determine the probability that all three mathematics textbooks end up **Calculate the total
in the same box, need follow these stepa: number ways to pack the textbooks
A3: To determine the probability that all three mathematics textbooks end up in the same box, start by calculating the total number of ways to distribute the 12 textbooks into the three
determine the probability that all in the same box, we need to consider the total number of ways to distribute the 12 textbooks into the three
mathematics textbooks up
A5: To determine the probability that all three mathematics textbooks end up the distribute the 12 textbooks into the three A6: To determine the probability that all three mathematics textbooks end up in the same box, need to consider the total number of ways to distribute the 12 textbooks into the three
box, start by calculating the total number of ways
determine the probability that all three mathematics textbooks up in the same box, we need to consider the total number of ways to distribute the 12 textbooks into the three AB: To determine the probability that all three mathematics textbooks end up in the same box, need consider the total number of ways to distribute the 12 textbooks into the three A9: To determine the probability that all three mathematics textbooks end up in the same box, need to consider the total number of ways to distribute the 12 textbooks into the three A10: To determine the probability that all three mathematica textbooks end up the same box, we need consider the total number ways to distribute the 12 textbooks into the three A11: To determine the probability that all three mathematics textbooks end up in the to distribute the 12 textbooks into the three A12: To determine the probability that all three mathematics textbooks end up in the same box, we need to follow these steps: 1. **Calculate the total number of ways distribute the 12
box, start by calculating the total number of ways
A13: To determine the probability that all three mathematica textbooks end up the same box, we need consider the total number ways to distribute the 12 textbooks into the three A14: To determine the probability that all three mathematica textbooks end up in the number ways to pack A15: To determine the probability that all three mathematics textbooks end up in the same box, we need to consider the total number of ways to distribute the 12 textbooks into the three A16: To determine
box, need follow these steps: 1. **Calculate the total
textbooks
probability that all three mathematice textbooks end up the same box, we need consider the total number ways to distribute the 12 textbooks into the three
Figure 5: With the temperature set to 0.7, sample 16 times based Owen2.5-Math-7 Math-7B-IInsruuct for the given question, where A1-A16 represents the corresponding output results.
21


Reviews:
[{"review_id": "E21gKGzwq3", "reviewer": "Reviewer_XozN", "summary": "This paper proposes the Unsupervised Prefix Fine-Tuning (UPFT), a method for improving the efficiency of LLM reasoning by fine-tuning models on minimal prefixes (shared reasoning phase). UPFT significantly reduces both training and sampling time, while achieving competitive performance compared to the baseline method, Rejection Sampling Fine-Tuning (RFT).", "questions": "n/a", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n-\tThis paper identifies a key insight, Prefix Self-Consistency— different solution paths often share a common initial reasoning phase—which is intriguing, and the proposed method UPFT is both computationally efficient and reduces the need for data labeling or rejection sampling.\n-\tThe paper is well-organized, beginning with an empirical observation that early reasoning steps are highly consistent across reasoning trajectories. This is followed by a theoretical analysis from a Bayesian perspective and the introduction of the Unsupervised Prefix Fine-Tuning method, which aims to maximize reasoning trace coverage while maintaining high accuracy. The writing is clear, and the paper is easy to follow.\n-\tExperiments are comprehensive, involving three models (Llama-3.1-8B-Instruct, Qwen2.5-Math-7B-Instruct, DeepSeek-R1-Distill-Qwen-7B), evaluated on four benchmarks (GSM8K, MATH500, AIME24, GPQA Diamond), and the method is compared against the baseline in both unsupervised sampling and supervised sampling settings.\n\n**Weaknesses**\n\nThis paper makes clear contributions to both methodology and empirical evaluation. I don’t see any major weaknesses.", "was_revised": true, "final_justification": "I have read the authors' responses and the other reviews. I will retain my score to show my support for this paper.", "rebuttal": "We want to express our sincere gratitude for your positive and encouraging review. We are thrilled that you found our paper \"excellent\" and our contributions clear and significant. Your understanding of our core insight and appreciation for our comprehensive experiments are very motivating. We kindly ask for your continued support during the internal discussion phase. Your endorsement will be invaluable in contextualizing our contributions for the other reviewers and the Area Chair."}, {"review_id": "46Gv3ltQXe", "reviewer": "Reviewer_JFVJ", "summary": "This paper proposes a novel self-improvement method for LLMs that enhances mathematical reasoning by training on only a small number of top-k examples. The approach not only reduces sampling and training costs but also outperforms rejection sampling fine-tuning in terms of performance.", "questions": "- Can the proposed method generalize to tasks beyond mathematical reasoning, such as code generation?\n- Is there a performance difference when applying UPFT to models trained with long vs. short CoT?\n- If responses are distilled from a stronger model to fine-tune a weaker one, is training on only the first few tokens still important?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n- This paper presents a novel observation and empirical analysis of the prefix self-consistency phenomenon.\n\n- Based on this insight, the authors design UPFT, an efficient SFT (supervised fine-tuning) method for self-improvement of LLMs.\n\n- The proposed UPFT method outperforms existing approaches such as RFT on mathematical reasoning tasks, while significantly reducing sampling and training costs.\n\n- The authors also provide theoretical analysis to explain why UPFT is effective.\n\n- Experiments across LLMs with varying levels of mathematical ability show that UPFT consistently leads to performance improvements.\n\n### Weaknesses:\n- The experimental evaluation is limited to mathematical reasoning tasks, which raises questions about the generalizability of the proposed method to other domains.\n\n- While the title “The First Few Tokens Are All You Need” may overstate the claim. In practice, the SFT procedure still involves structure tuning with the full reasoning trace, suggesting that early tokens are important but not sufficient on their own.\n\n- There appears to be a potential typo in the notation for rejection sampling on line 108, which may cause confusion.", "was_revised": true, "final_justification": "I have carefully read the responses and the other reviews. I decide to retain my score.", "rebuttal": "We sincerely thank you for your thoughtful review and recognition of our work's novelty and efficiency. Your constructive feedback is very helpful. We address your questions below.\n\n> **W1&Q1:** The experimental evaluation is limited to mathematical reasoning tasks, which raises questions about the generalizability of the proposed method to other domains.\n> \n\nYes, our method shows strong generalization to non-mathematical domains.** We'd like to highlight a key result that may have been missed:\n\n- **Our evaluation already includes a non-math, cross-domain benchmark.** The **GPQA** dataset, as described in the paper, consists of PhD-level questions across **biology, physics, and chemistry**, not math.\n- **UPFT demonstrates strong Out-of-Distribution (OOD) generalization.** Our models were trained *exclusively* on mathematical reasoning data. The consistent performance improvements on GPQA (e.g., **+4.6** for DeepSeek on LIMO, see Table in response to R_fBpq) show that UPFT enhances a general reasoning capability that transfers OOD.\n\nWhile we did not test on code generation, the strong OOD results on GPQA suggest UPFT is not limited to a single domain and could be promising for other reasoning tasks like code generation, which we leave for future work.\n\n> **W2:** While the title “The First Few Tokens Are All You Need” may overstate the claim. In practice, the SFT procedure still involves structure tuning with the full reasoning trace, suggesting that early tokens are important but not sufficient on their own.\n> \n\nThank you for this valuable suggestion. Our title was inspired by \"Attention is All You Need\" to be memorable, but we agree it could be interpreted more literally than intended. We acknowledge that the small percentage of full-trace tuning is important for preserving the model's output structure. We will revise the title to be more precise in the final version, e.g.. “**The First Few Tokens Matter**”.\n\n> **Q2:** Is there a performance difference when applying UPFT to models trained with long vs. short CoT?\n> \n\n**Yes, and our results suggest UPFT is broadly effective regardless of the CoT length of the training data.** As shown in Table 2, the training datasets have vastly different average CoT lengths (e.g., LIMO at 491.8 vs. U-Hard at 393.3 for Qwen2.5). Despite these differences, UPFT consistently delivers gains:\n\n- **Qwen2.5 on LIMO (long CoT):** Base 51.4 -> UPFT 52.5 (**+1.1** avg.)\n- **Qwen2.5 on U-Hard (shorter CoT):** Base 51.4 -> UPFT 54.5 (**+3.1** avg.)\n\nThis indicates that UPFT's benefit, which comes from reinforcing correct initial reasoning steps, is not dependent on the length of the full reasoning traces in the source data.\n\n> **Q3:** If responses are distilled from a stronger model to fine-tune a weaker one, is training on only the first few tokens still important?\n> \n\n **Not necessarily. Our method's core assumption is that we are fine-tuning on the model's *own* generated samples.** This self-sampling is key to reinforcing the model's inherent, correct reasoning pathways. Distilling from a stronger, external model is a different setting where this assumption does not hold.\n\nTo verify this, we ran a new experiment where we fine-tuned Llama3.1-8B-Instruct but generated the prefixes from stronger models.\n\n| **Decoding Model** (Tuned) | **Prefix Generation Model** (External) | **GSM8K** | **MATH500** |\n| --- | --- | --- | --- |\n| Llama3.1-8B-Instruct | Llama3.1-8B-Instruct (Self-Generated) | **82.0** | **51.0** |\n| Llama3.1-8B-Instruct | Llama3.1-70B-Instruct (Stronger) | 80.3 (↓1.7) | 49.6 (↓1.4) |\n| Llama3.1-8B-Instruct | Qwen2.5-Math-72B-Instruct (Stronger) | 78.5 (↓3.5) | 44.8 (↓6.2) |\n\nAs shown, performance **degrades** when using prefixes from an external model, even a much stronger one. This confirms that UPFT is most effective in the self-improvement setting, where it aligns the model's distribution with its own most consistent reasoning starts.\n\n> **W3:** Typo about line 108.\n> \n\nThank you for catching this. We will correct it in the final version."}, {"review_id": "qYKjsrGCzW", "reviewer": "Reviewer_fBpq", "summary": "This paper proposes an unsupervised prefix fine-tuning method for reasoning models. The key motivation is that training with a consistent prefix can cover many possible correct reasoning paths while being more efficient. The objective is to increase the probability of generating the correct prefix.", "questions": "- What happens if you run the experiments 5 times and report the average?\n- Does SFT refer to sampling one response from the model and using that as training data?\n- I’d like to see in-domain performance. How about using the MATH training dataset directly for training?\n- Any qualitative results? Any analysis of the samples that become solvable after fine-tuning?", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- This paper offers a novel perspective by focusing solely on prefix tuning to improve efficiency.\n- It does not require labeled data, which is a significant advantage.\n- The paper thoroughly justifies its approach both theoretically and empirically, and it is well written and easy to follow.\n\n**Weaknesses:**\n\nThe main weakness lies in the evaluation. Details are as follows:\n\n- SFT seems to decrease performance, and UPFT only marginally improves it. In this setting, fine-tuning itself appears largely ineffective. Increasing the p% over 10% seems to make things worse, possibly due to errors in full traces.\n- The performance gain on AIME appears to be the most significant, but AIME contains only a small number of questions. Thus, even a few additional correct answers can cause a noticeable change. This makes statistical significance critical, yet it seems to be missing. Results may vary considerably by random seed (e.g., performance on MATH500 can vary by 2–4% per trial).\n- The only meaningful baseline appears to be the supervised methods like RFT and V-Starm, but even there, the improvement is minimal. Also, why not test with additional training datasets in Table 2 like U-Hard?\n\nOverall, I find the idea interesting and the paper enjoyable to read, but I’m not convinced of its practical utility. I think this concern also arises from the fact that the equations in this paper do not show that maximizing this bound with short prefixes is a consistent estimator of true task accuracy.", "was_revised": true, "final_justification": "I appreciate the detailed author response. However, I don’t see a clear value in this incremental UPFT. It is interesting, but not particularly practical. Simply collecting a few SFT data points and applying SFT, or using self-consistency to generate synthetic data and then applying SFT, seems more valuable. Therefore, I will maintain my borderline reject score.", "rebuttal": "We thank you for your feedback and the opportunity to clarify our contributions. While you found the idea interesting, your main concerns were about the practical utility and statistical significance of our results. We respectfully disagree with the assessment that the gains are \"marginal\" and provide substantial new evidence below to address your concerns.\n\n> **W1:** SFT seems to decrease performance, and UPFT only marginally improves it. In this setting, fine-tuning itself appears largely ineffective.\n> \n\n**We respectfully disagree. The fact that standard SFT often fails in this unsupervised setting is precisely what makes UPFT's consistent and significant gains so valuable.**\n\nOur setting is fully **unsupervised**, meaning we train on a single, unfiltered sample per question, which often contains incorrect reasoning. Standard SFT struggles with this noisy data, frequently degrading performance. In contrast, UPFT is designed to be robust to this noise by focusing on the more reliable initial tokens.\n\nOur results in Table 2 (reproduced below with average gains) show that **UPFT consistently and significantly outperforms the base model, while SFT often hurts it.**\n\n| Model | Data | Base Avg. | SFT Avg. | UPFT Avg. | **UPFT Gain** |\n| --- | --- | --- | --- | --- | --- |\n| Llama-3.1-8B | PRM | 36.2 | 36.0 (↓0.2) | 38.3 | **+2.1** |\n| Qwen2.5-Math-7B-Instruct | PRM | 51.4 | 50.4 (↓1.0) | 52.6 | **+1.2** |\n| Qwen2.5-Math-7B-Instruct | U-Hard | 51.4 | 51.3 (↓0.1) | 54.5 | **+3.1** |\n| DeepSeek-R1-Distill-Qwen-7B | LIMO | 57.2 | 57.2 (+0.0) | 60.6 | **+3.4** |\n| DeepSeek-R1-Distill-Qwen-7B | U-Hard | 57.2 | 56.4 (↓0.8) | 61.6 | **+4.4** |\n\nThese are **not marginal improvements**; gains of **+3.1** and **+4.4** on strong base models are substantial. Furthermore, this is achieved with massive efficiency gains over supervised methods like RFT (**95% less sampling cost, 75% less training cost**), highlighting its practical utility.\n\n> **W2&Q1:** Concern about statistical significance and variance from random seeds, especially on AIME.\n> \n\n**This is a crucial point. We ran new experiments with 5 random seeds to confirm the statistical significance of our results.** We agree that single-run results can be noisy.\n\nThe table below shows the mean and standard deviation over 5 seeds for DeepSeek-R1-Distill-Qwen-7B on the LIMO dataset.  We performed additional significance testing experiments. In each test, 200 samples were randomly drawn from both SFT and UPFT inference results for comparison. This procedure was repeated 1000 times. **UPFT consistently and significantly outperforms SFT across all benchmarks, with p<0.01.**\n\n| Method | GSM8K | MATH500 | AIME24 | GPQA |\n| --- | --- | --- | --- | --- |\n| SFT | 89.8 ± 0.13 | 86.1 ± 0.25 | 40.0 ± 0.80 | 13.3 ± 1.32 |\n| UPFT | **92.8 ± 0.28** | **89.2 ± 0.39** | **42.7 ± 0.45** | **18.1 ± 0.89** |\n\nFurthermore, to mitigate sampling variance during evaluation, we also report **avg@32** results, which show even more stable and pronounced gains for UPFT. These results confirm our findings are robust and not due to random chance.\n\n| Backbone | Method | Data | GSM8K | MATH500 | AIME24 | GPQA |\n| --- | --- | --- | --- | --- | --- | --- |\n| Llama-3.1-8B-Instruct | SFT | PRM | 83.6 | 47.0 | 4.0 | 8.4 |\n|  | UPFT | PRM | **85.0** | **52.4** | **7.3** | **9.6** |\n| Qwen2.5-Math-7B-Instruct | SFT | PRM | **95.6** | 83.4 | 14.1 | 9.1 |\n|  | UPFT | PRM | **95.6** | **86.0** | **20.3** | **9.6** |\n| Qwen2.5-Math-7B-Instruct | SFT | U-Hard | 95.3 | 83.6 | 16.6 | **9.6** |\n|  | UPFT | U-Hard | **95.7** | **85.4** | **26.4** | 9.5 |\n| DeepSeek-R1-Distill-Qwen-7B | SFT | LIMO | 89.4 | 87.4 | 41.3 | 12.6 |\n|  | UPFT | LIMO | **91.6** | **89.0** | **43.3** | **16.5** |\n| DeepSeek-R1-Distill-Qwen-7B | SFT | U-Hard | 88.9 | 87.4 | 40.0 | 12.1 |\n|  | UPFT | U-Hard | **91.7** | **88.8** | **53.3** | **15.9** |\n\n> **W3:** The only meaningful baseline appears to be the supervised methods like RFT and V-Starm, but even there, the improvement is minimal. Also, why not test with additional training datasets in Table 2 like U-Hard?\n> \n- **On Baselines:** Our primary claim is effectiveness in an **unsupervised** setting. Therefore, the most meaningful baseline is **unsupervised SFT**, which UPFT consistently beats. We also added an **SC-CoT** (Self-Consistency) baseline, a widely-used unsupervised inference method. As shown below, UPFT is complementary and further boosts performance.\n\n| Method | MATH500 | AIME24 |\n| --- | --- | --- |\n| Qwen2.5-Math-7B-Instruct w SC-COT | 86.2 | 20.0 |\n| +UPFT w SC-COT | **88.0** | **29.7** |\n- **On U-Hard:** The reason U-Hard is not in the supervised comparison (Table 3) is that **U-Hard is an unsupervised dataset with no ground-truth answers.** It cannot be used for supervised methods like RFT, which require answer verification.\n\n> **Q2:** Does SFT refer to sampling one response from the model and using that as training data?\n> \n\n**Yes, that is correct.** In our unsupervised setting, SFT means fine-tuning on a single, complete response generated by the model for each question, without any filtering based on correctness.\n\n> **Q3:** I’d like to see in-domain performance. How about using the MATH training dataset directly for training?\n> \n\n**This is an excellent question that highlights the core principle of our method.** UPFT is designed for a **self-sampling** regime, where it reinforces the model's *own* consistent reasoning patterns. When using external, gold-standard data like the MATH training set, this self-alignment property is lost.\n\nWe ran this experiment. On the MATH dataset, standard SFT outperforms UPFT.\n\n| Model | Method | MATH500 |\n| --- | --- | --- |\n| Llama3.1-8B-Instruct | +SFT (MATH dataset) | **53.4** |\n| Llama3.1-8B-Instruct | +UPFT (MATH dataset) | 52.5 |\n\nThis result, combined with our strong performance in the self-sampling setting below (e.g., Llama3.1 on PRM: SFT 48.4 -> **UPFT 52.0** on MATH500), confirms that UPFT's strength lies specifically in unsupervised self-improvement from noisy, model-generated data.\n\n> **Q4:** Any qualitative results? Any analysis of the samples that become solvable after fine-tuning?\n> \n\nYes. We find that UPFT increases the model's reliability. For many problems, the base model might generate a correct answer in only 1 out of many attempts. After UPFT, the model's probability of generating the correct answer increases significantly. \n\nTake the case below as an example, for a GPQA question about optical activity, the base Qwen2.5 model answered correctly in only 1 of 16 attempts. After UPFT, it answered correctly in **3 of 16** attempts. This demonstrates that UPFT solidifies nascent correct reasoning abilities. We will add more case studies to the appendix.\n\n```\n***Question***\nHow many of the following compounds will exhibit optical activity?\n\n(Z)-1-chloro-2-methylbut-1-ene\n(3aR,7aS,E)-8-(chloromethylene)hexahydro-4,7-methanoisobenzofuran-1,3-dione\n(2R,3S)-2,3-dimethylsuccinic acid\n(2R,3R)-2,3-dimethylsuccinic acid\n(R)-cyclohex-3-en-1-ol\n(1s,3s,5s)-cyclohexane-1,3,5-triol\n1-cyclopentyl-3-methylbutan-1-one\n\n***Ground_Truth***\n3\n\n```"}, {"review_id": "bH2saJXPKs", "reviewer": "Reviewer_H5bo", "summary": "This paper proposes a simple and effective Unsupervised Prefix Fine-Tuning (UPFT) method that leverages the observation of prefix self-consistency—the commonality in early reasoning steps across different generated solutions. The method avoids the need for annotated data or rejection sampling by fine-tuning only on the initial tokens of generated reasoning traces. The authors conduct extensive experiments across different models and datasets, demonstrating that UPFT achieves comparable or superior reasoning performance to supervised methods like Rejection Fine-Tuning (RFT), while drastically reducing sampling costs.", "questions": "1. Why does the model's performance drop so sharply when the structure tuning ratio is set to 0? In my perspective, such senarios is standard fine-tuning. It that reasonable that the full-trace fine-tuning approach lead to such low performance without any prefix tuning?\n2. Have the authors considered applying sampling multiple prefix span for fine-tuning (i.e., question + prefix)? Since the prefix may already represent a consistent reasoning process, using more diverse prefix might further improve the fine-tuning performance.\n3. Can the proposed method largely enhance the performance of several weak LLM backbones? (e.g., the performance of Llama-3.1-8B-Instruct  on Math500 is only 0.51).", "limitations": "The proposed method may fail when applied to a base model with inherently weak capabilities. Basicially, this method cannot bring to very significant performan increase.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1. The core idea of exploiting prefix self-consistency for unsupervised fine-tuning is interesting.\n2. UPFT reduces sampling and tuning costs while preserving or improving reasoning accuracy, making it practical.\n\nWeakness:\n\n1. While concepts like prefix self-consistency are important to the method, the mathematical formulations in Sections 3.1 and 3.2 are dense and may overwhelm readers. The paper would benefit from a well-defined preliminary.\n2. This paper only considers the cost token in the sampling stage. A more holistic cost analysis (e.g., including the training time and cost) would strengthen the efficiency claims.\n3. While the method is labeled as “unsupervised,” it still implicitly relies on ground-truth answers for tuning. It may not be proper to mention an unsupervised tuning method.\n4. While the proposed method is interesting, the performance gain is limited, especially on challenging datasets (e.g., Math500, AIME 2024, and GPQA) using weak LLM backbones (e.g., Llama-3.1-8B-Instruct ). That means the proposed method may rely on the original capacity of backbones, e.g., if a similar rule exists on a 0.5B-size model. The potential limited generalization of the method reduces the paper’s impact.\n5. For the baseline comparison. The author only compares UPFT with RFT and V-STaR. More baseline comparisons (e.g., Quiet-star [1] and so on) could be conducted.\n\n[1] Zelikman, Eric, et al. \"Quiet-star: Language models can teach themselves to think before speaking.\" arXiv preprint arXiv:2403.09629 (2024).", "was_revised": true, "final_justification": "This paper proposes an effective fine-tuning method and only compares UPFT under standard SFT. Under this experimental setting, the performance and efficiency are validated. \n\nHowever, I keep my concerns about whether other types of SFT methods can achieve much better performance (e.g., the performance gain for Llama in Math 500 is 52.0->53.4), which makes the efficiency advantage invaluable. I raise these concerns in the rebuttal stage, which is not accepted or solved by the authors. Therefore, I will keep my negative score and tend to reject this paper.", "rebuttal": "Thank you for your review and for acknowledging the novelty and practicality of our method. We address your concerns below, particularly a key misunderstanding about our \"unsupervised\" setup and new experiments that refute concerns about model generalization.\n\n> **W3:** While the method is labeled as 'unsupervised,' it still implicitly relies on ground-truth answers for tuning. It may not be proper to mention an unsupervised tuning method.\n> \n\n**We must respectfully clarify a critical misunderstanding here. Our method is fully unsupervised and does NOT use ground-truth answers for training.**\n\nIn our unsupervised setting (the main focus of our paper), we generate a single reasoning path for each question and use its prefix for fine-tuning. This process is \"unfiltered\"—the sample is used **regardless of whether the final answer is correct or not.** The training data is therefore noisy and contains many incorrect solutions. This is the core challenge our method is designed to address and is fundamentally different from supervised methods like RFT, which filter for correct answers. We will make this distinction even clearer in the paper.\n\n> **W4&Q3:** While the proposed method is interesting, the performance gain is limited, especially on challenging datasets (e.g., Math500, AIME 2024, and GPQA) using weak LLM backbones (e.g., Llama-3.1-8B-Instruct ). That means the proposed method may rely on the original capacity of backbones, e.g., if a similar rule exists on a 0.5B-size model. The potential limited generalization of the method reduces the paper’s impact.\n> \n\n**This is a fair question. To address it, we ran new experiments on much weaker LLM backbones: Llama-3.2-1B and 3B.**\n\nThe results show that UPFT provides consistent and significant improvements even on these smaller models, demonstrating that it does not simply rely on the high capacity of large models.\n\n| Model | Method | MATH500 | AIME24 | GPQA |\n| --- | --- | --- | --- | --- |\n| Llama-3.2-1B | Base | 26.1 | 2.5 | 4.0 |\n|  | + SFT | 25.0 (↓) | 0.0 (↓) | 4.0 |\n|  | + UPFT | **28.3 (+2.2)** | **3.3 (+0.8)** | **5.6 (+1.6)** |\n| Llama-3.2-3B | Base | 45.8 | 0.0 | 8.6 |\n|  | + SFT | 46.5 | 6.7 | 6.7 (↓) |\n|  | + UPFT | **48.0 (+2.2)** | **6.7** | **10.1 (+1.5)** |\n\nAs before, SFT is unstable and often hurts performance, while UPFT consistently yields gains across all benchmarks. This confirms our method is effective even for weaker models.\n\n> **W2:** A more holistic cost analysis (e.g., including the training time and cost) would strengthen the efficiency claims.\n> \n\n**Excellent suggestion. We now include training time analysis, which further highlights UPFT's efficiency.** By using much shorter sequences, UPFT drastically cuts training time.\n\n| Model | Data (Size) | SFT Time (h) | UPFT Time (h) | **Time Saved** |\n| --- | --- | --- | --- | --- |\n| Qwen2.5-Math-7B-Instruct | OMI (600K) | 12.3 h | 3.0 h | **75%** |\n| DeepSeek-R1-Distill-Qwen-7B | U-Hard (100K) | 20.5 h | 6.0 h | **71%** |\n\nOn average, UPFT reduces training time by **over 70%** compared to standard SFT on the same data, in addition to the **95%+ sampling cost reduction** compared to RFT. We will add this table to the paper.\n\n> **Q1:** Why does the model's performance drop so sharply when the structure tuning ratio is set to 0? In my perspective, such senarios is standard fine-tuning.\n> \n\nThere seems to be a misunderstanding of this ablation. A **structure tuning ratio of 0** means we fine-tune **exclusively on prefixes (100% prefix-only)**, with *no* full-trace examples. This is the opposite of standard SFT.\n\nPerformance drops sharply in this setting because the model is never exposed to a complete thought process:\n\n- It **never sees a final, boxed answer**, so it doesn't learn to produce one.\n- The training sequences are **artificially short**, teaching the model to terminate prematurely.\n- It loses the high-level instruction-following format of a full solution.\nThis ablation confirms that a small amount of full-trace data is necessary to preserve the model's structural and formatting capabilities, which is why UPFT includes it.\n\n> **Q2:** Have the authors considered applying sampling multiple prefix span for fine-tuning?\n> \n\n**This is an insightful idea.** We ran a new experiment on Qwen2.5-Math-7B-Instruct where we used three prefixes per question (3xN) instead of one (1xN).\n\n| Method | MATH500 | AIME24 |\n| --- | --- | --- |\n| SFT | 83.4 | 13.3 |\n| UPFT (1 prefix / question) | 85.6 | 20.0 |\n| UPFT (3 prefixes / question) | **86.4** | **23.3** |\n\nUsing more prefixes provides a modest but consistent boost, confirming your intuition that prefix diversity can be beneficial. This also supports our theoretical claim about reinforcing consistent reasoning steps. Thank you for the great suggestion!\n\n> **W5:** More baseline comparisons (e.g., Quiet-star [1] and so on) could be conducted.\n> \n\nThank you for the pointer. Quiet-STaR is an interesting recent work on self-teaching. However, its focus is different from ours; it aims to generate rationales *before* answering, rather than improving efficiency via unsupervised prefix tuning. Its goals are not directly comparable to our efficiency- and unsupervised-focused evaluation against RFT and SFT. We will, however, add a discussion of Quiet-STaR and other related self-improvement methods to our related work section to better situate our contribution.\n\n> **W1:** While concepts like prefix self-consistency are important to the method, the mathematical formulations in Sections 3.1 and 3.2 are dense and may overwhelm readers. The paper would benefit from a well-defined preliminary.\n> \n\nWe agree and plan to introduce clearer definitions, brief preliminaries, and inline intuition for the Bayes formulation in a revision."}]

Author rebuttal:

Meta-review: (a) Summary of claims and findings:
This paper introduces Unsupervised Prefix Fine-Tuning (UPFT), a novel and highly efficient method for improving the reasoning capabilities of Large Language Models (LLMs). The core insight, termed "Prefix Self-Consistency," is the empirical observation that diverse, correct reasoning trajectories for a given problem often share the same initial sequence of tokens. Leveraging this, UPFT fine-tunes the model on just these short initial prefixes (e.g., the first 8 tokens) of model-generated solutions. Crucially, this is done in a fully unsupervised manner, using a single unfiltered response per question without requiring ground-truth labels or expensive rejection sampling to filter for correct answers. The authors claim that this simple technique can match the performance of costly supervised methods like Rejection Sampling Fine-Tuning (RFT) while reducing sampling costs by over 99% and training time by over 75%.

(b) What are the strengths of the paper?
The paper's primary strengths are its novelty, simplicity, and remarkable efficiency.

Novel and Intuitive Idea: The core concept of "Prefix Self-Consistency" is a simple yet powerful insight into the behavior of LLMs during reasoning. The idea of reinforcing these stable initial steps is elegant and well-motivated.

High Efficiency and Practicality: The method's main appeal is its extreme resource efficiency. By avoiding the need for labeled data and expensive sampling/filtering pipelines (like RFT), and by training on very short sequences, UPFT dramatically lowers the barrier to improving LLM reasoning, making it accessible to those with limited computational resources.

Strong Empirical Results: The authors conduct a comprehensive set of experiments across multiple models (Llama, Qwen, DeepSeek) and challenging reasoning benchmarks (GSM8K, MATH, AIME, GPQA). The results compellingly show that UPFT consistently improves performance over the base model, whereas naive unsupervised fine-tuning (SFT) often leads to performance degradation.

Excellent Rebuttal: The authors provided an exceptionally thorough and convincing rebuttal, running several new experiments (multi-seed validation, weaker model backbones, training time analysis, prompt engineering baselines) that substantially strengthened the paper's claims and addressed most of the reviewers' concerns.

(c) What are the weaknesses of the paper? What might be missing in the submission?
The primary weaknesses raised during the review process revolved around the perceived magnitude of the performance gains and the generalizability of the method.

Significance of Performance Gains: One reviewer (fBpq) argued that the absolute performance improvements were "marginal" and questioned the practical utility. While the gains are not massive in all cases, this critique somewhat overlooks the challenging unsupervised setting where the most direct baseline (unsupervised SFT) often fails entirely.

Generalizability: A concern was raised (JFVJ) that the experiments were overly focused on mathematical reasoning. The authors effectively countered this by highlighting the strong out-of-distribution performance on the GPQA dataset, which covers biology, physics, and chemistry.

Initial Clarity: Some aspects of the initial submission were unclear, leading to significant misunderstandings by at least one reviewer (H5bo) regarding the unsupervised nature of the method and the interpretation of an ablation study. However, the authors have committed to clarifying these points in the final version.

(d) Provide the most important reasons for your decision to accept/reject.
I recommend acceptance because the paper presents a well-motivated, simple, and highly efficient method for a significant problem. The strengths clearly outweigh the weaknesses.

The core reason for my recommendation is that UPFT successfully addresses the challenging task of model self-improvement in a fully unsupervised and resource-scarce setting. The most direct and fair baseline in this scenario is standard SFT on the model's own noisy outputs, which the paper shows is unreliable and often harmful. UPFT's ability to deliver consistent and statistically significant performance gains in this exact setting is a strong and valuable contribution.

While two reviewers recommended rejection, their concerns were either thoroughly addressed by the authors' rebuttal or stemmed from fundamental misunderstandings. Reviewer H5bo's critique was predicated on a factually incorrect assumption that the method used ground-truth labels and a non-standard definition of "unsupervised learning." Reviewer fBpq's skepticism about the practical utility, while a valid perspective, was countered by new experiments from the authors, including multi-seed runs demonstrating statistical significance and a prompt engineering baseline showing the gains were not trivial. The two positive reviewers (XozN, JFVJ) correctly identified the novelty and significance of the work. Given the evidence, especially after the extensive rebuttal, the paper stands as a solid, well-executed piece of research.

(e) Summarize the discussion and changes during the rebuttal period.
The rebuttal period was highly effective and substantially improved the paper's standing. The discussion centered on three main axes:

Clarifying "Unsupervised": Reviewer H5bo fundamentally misunderstood the method, believing it used ground-truth answers. The authors clarified that their method is fully unsupervised, training on single, unfiltered model outputs. This misunderstanding, which the authors flagged in a confidential comment, was a major source of the initial negative rating.

Demonstrating Robustness and Significance: In response to concerns from Reviewer fBpq about statistical significance and Reviewer H5bo about generalization to weaker models, the authors ran extensive new experiments. They provided results over 5 random seeds (confirming statistical significance with p<0.01), evaluated UPFT on smaller 1B and 3B models (showing consistent gains), and added a comprehensive training time analysis (demonstrating a >70% time saving). This new evidence directly addressed the reviewers' core empirical concerns.

Contextualizing Performance Gains: Reviewer fBpq remained skeptical about the value of the gains. The authors successfully argued that in the context of unsupervised learning from noisy data (where SFT degrades performance), UPFT's consistent improvements are significant. They further ran a new experiment showing that simple prompt engineering could not replicate the gains, reinforcing that the improvement comes from their method.

In summary, the authors successfully corrected factual misunderstandings, provided substantial new evidence to back their claims, and agreed to minor changes like clarifying the text and toning down the title. The rebuttal successfully convinced Reviewer JFVJ and provided strong counter-arguments to the points raised by Reviewers fBpq and H5bo.
Decision: Accept (poster)