TTRL: Test-Time Reinforcement Learning
Authors: Yuxin Zuo
Kaiyan Zhang
Li Sheng
Shang Qu
Ganqu Cui
Xuekai Zhu
Haozhan Li
Yuchen Zhang
Xinwei Long
Ermo Hua
Biqing Qi
Youbang Sun
Zhiyuan Ma
Lifan Yuan
Ning Ding
Bowen Zhou
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Test-Time Training
Large Language Model
Primary area: deep_learning

Abstract: This paper investigates Reinforcement Learning (RL) on data without explicit labels for reasoning tasks in Large Language Models (LLMs). The core challenge of the problem is reward estimation during inference while not having access to ground-truth information. While this setting appears elusive, we find that common practices in Test-Time Scaling (TTS), such as majority voting, yield surprisingly effective rewards suitable for driving RL training. In this work, we introduce Test-Time Reinforcement Learning (TTRL), a novel method for training LLMs using RL on unlabeled data. TTRL enables self-evolution of LLMs by utilizing the priors in the pre-trained models. Our experiments demonstrate that TTRL consistently improves performance across a variety of tasks and models. Notably, TTRL boosts the pass@1 performance of Qwen-2.5-Math-7B by approximately 211% on the AIME 2024 with only unlabeled test data. Furthermore, although TTRL is only supervised by the Maj@N metric, TTRL has demonstrated performance to consistently surpass the upper limit of the initial model, and approach the performance of models trained directly on test data with ground-truth labels. Our experimental findings validate the general effectiveness of TTRL across various tasks and highlight TTRL's potential for broader tasks and domains.

TTRL: Test-Time Reinforcement Learning
Yuxin Zuo 1,2
Kaiyan Zhang"!
Li Sheng 1.2
Shang Qu
Ganqu Cui Ermo Hua
Xuekai Zhu'
Haozhan Li
Yuchen Zhang²
Xinwei Long'
Biqing Qi
Youbang Sun'
Zhiyuan Ma
Lifan Yuan'
Ning Dingit.2
Bowen Zhou
Tsinghua University
Shanghai AI Lab
zhang- tysnemail...tiinguuuu....
tsinghua. edu. en, dingning@mail tsinghua, edu, cn
o Code: http:: https//ppp//ggthhu..///P//..EE.E
Abstract
This paper investigates Reinforcement Learning (RL) on data without explicit labels for reasoning tasks in Large Language Models (LLMs). The core challenge the problem is reward estimation during inference while not having access to ground-truth information. While this setting appears elusive, find that common practices Test-Time Scaling (TTS), such as majority voting, yield surprisingly effective rewards suitable for driving RL training. In this work, introduce Test- Time Reinforcement Learning (TTRL), a novel method for training LLMs using RL on unlabeled data. TTRL enables self-evolution of LLMs by utilizing the priors in the pre-trained models. Our experiments demonstrate that TTRL consistently improves performance across variety of tasks and models. Notably, TTRL boosts the pass @ performance of Qwen-2 2. 2......55... 5-Math-77 7B by approximately 211% 1% on the AIME 2024 with only unlabeled test data. Furthermore, although TTRL is only supervised by the maj @ metric, TTRL has demonstrated performance consistently surpass the upper limit of the initial model maj @n, and approach the performance of models trained directly on test data with ground ground-truth -ruth labels. Our experimental findings validate the general effectiveness of TTRL across various tasks and highlight TTRL's potential for broader tasks and domains.
AIME 2024
AMC
MATH-500
Pre-Training +SFT/RL
TTRL
Data
+66.4%
Pre Tralninn Data
Laheled Data
Unlabeled Data (eg, Test Data)
(Pass
+74.9%
TTRL estimates rewards via majority voting on unlabeled data for RI. training.
60
+159.3%
Role
40-
Training
67.5
Query
50.6
LLM
Dv )2...)
LLM
16.7
Test-Time Scaling (e.g. Majority Voting)
TTRL
Figure 1: Performance and Position of TTRL.
"Equal Contributions. 'Corresponding Authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Recent advances in Large Reasoning Models (LRMs), such as DeepSeek-RI [11] and OpenAl's ol [18], have demonstrated that Reinforcement Learning (RL) is essential for enhancing long chain- of-thought (CoT) reasoning [49] through training on expensive human-annotated data. These models achieve remarkable performance on a range of highly challenging tasks. For example, OpenAl's o3 attains a 7% success rate ARC-AGI-I. However, complex and unlabeled questions continuously emerge, posing significant challenges. For instance, o3 solves only 4% of problems on the recently released ARC-AGI-2 benchmark (2025). Addressing such tasks typically involves scaling up training with more data and computational resources, and it may still fail to yield strong performance on these tasks. Silver & Sutton [39] has recently advocated for a transition to the "era of experience," emphasizing the limitations of existing Al systems that rely heavily on human supervision, as well as the importance of enabling models to self-evolve through experience.
Further building upon the substantial progress LRMs, it naturally motivates a promising direction in which AI systems autonomously improve via RL on unlabeled data by directly engaging in self- experience and learning, thereby pushing the boundaries of RL and further advancing the frontier of Al capabilities. Such self-evolvement can be broadly categorized into two modes: adaptation to test-time data, which enables models to tackle harder benchmarks such as ARC-AGI-2, and training on external unlabeled data, which unlocks more training data beyond labeled corpora. This work focuses on the adaptation to test-time data, which has been extensively studied under the paradigm of Time-Tim Training (TTT) [1. 2, 42, 43]. TTT has received increasing attention recently. These approaches adapt model parameters at test time by exploiting the structure and distributional properties of incoming test data.
Therefore, we aim to fully advance Al evolution by updating models at test time using RI., thereby enhancing their generalization to previously unseen data. However, this introduces a critical challenge: How to obtain rewards for RL at test-time? This also highlights a broader limitation of current RL approaches. Despite their promise, most existing methods still rely heavily on labeled data, which significantly limits their scalability. As real-world tasks continue to increase in both complexity and volume, large-scale annotation for RL becomes increasingly impractical, posing a substantial barrier to the continual improvement of state-of-the-ee models.
We introduce Test-Time Reinforcement Learning (TTRL), which performs test-time training through RI. TTRL employs repeated sampling strategies in the rollout phase to accurately estimate the label and compute rule-based rewards, thereby enabling RL on unlabeled data. By incorporating effective majority voting rewards, TTRL facilitates efficient and stable RL in the absence of ground truth labels. As previously highlighted, the emergence of more challenging tasks will inevitably lead to larger proportions of unlabeled data. TTRL directly addresses the problem of training models via RL without explicit supervision, investigating model's ability to explore and learn in this challenging yet critical setting. Essentially, TTRL enables the model to generate its own experiences, estimate rewards, and improve its performance over time.
5-Math-7B results in an improvement on AIME 2024 of 211% (12.9 to 40.2), with an average gain of 76% across AIME 2024, AMC, MATH-500, and GPQA. These improvements are achieved through self-evolution without any labeled training data and further generalize to other tasks. TTRL not only enhances performance on pass @ but also improves TTS through majority voting. Moreover, our preliminary experiments suggest that TTRL is effective across models of different scales and types and that it can be integrated with existing RL algorithms. We also found that TTRL exhibits favorable characteristies such as a high high-perffrmance ceiling. These observations highlight its potential substantially reduce reliance human annotations, enabling continual learning and scaling RL to large-scale unsupervised training. Below are several key takeaways:
In experiments, applying TTRL to Qwen2.5-MM
Takeaways
1. Majority voting provides effective reward estimation for TTRL ($ 3).
2. TTRL can exceed its training signal and upper limit maj @n, and closely mirrors the performance of direct training on the test data with ground-truth ($ 4.1).
3. It is possible to achieve efficient and stable RL in unsupervised manner ($ 4.2).
2

---

2 Test-Timm Reinforcement Learning (TTRL)
Unlike traditional RL, where the agent learns from known reward signals, TTRL operates on unlabeled test data. In other words, the model must learn and adapt without access to explicit supervision. Our task is defined as follows:
We study the problem of training pre-trained model during test time using RL without ground-truth labels. We call this setting Test-Time Reinforcement Learning.
2.1 Methodology
Test-Time Reinforcement Learning (TTRL)
Policy Optimization
P1
R(P,y)
Test Data
P2
R(P2.y)
Majority Voting
Reward
q
LLM
Calculation
fM
R y)
Label Estimation
Predictions
Rewards
Figure 2: TTRL combines both Test-Time Scaling (TTS) and Test-Time Training (TTT).
Figure 2 illustrates how our approach, TTRL, tackles this challenge. Given prompt the model acts by producing output sampled from a policy Te x) parameterized by To construct reward signal without ground-truth labels, we generate multiple candidate outputs (V1, ....... yN from the model through repeated sampling. A consensus output y is derived, for instance, by majority voting or another aggregation method, serving as a proxy for the optimal action. The environment then provides reward r(y. y/" based on the alignment between the sampled action y and the consensus action y" The RL objective is thus to maximize the expected reward:
represented by the
max E (ry,) v²)} 8
and parameters are updated through gradient ascent:
0 - 0 + V VoBoye
(2)
where denotes the learning rate. This approach enables the model to adapt during inference, effectively improving its performance distribution-shifted inputs without the need for labeled data.
2.2 Majority Voting Reward Function
The majority voting reward is determined by first estimating label through majority voting, This estimated label then used to calculate rule-based rewards, which serve as the final rewards. Given question x, we first input into the LLM to generate a set of outputs. An answer extractor then processes these outputs to obtain the corresponding predicted answers, denoted as P = J We first follow Equation 4 over P estimate label, with majority voting the scoring function s sy), to get y, the most frequently occurring prediction in P. The majority-voted prediction is then used the estimated label to compute rule-based rewards [11]. The reward function is:
R(D., v) = (
if 0 =
(3)
0, otherwise.
Appendix C presents the pseudo-code of the reward function.

---

Table Main results TTRL on each task. indicates that Qwen3-88 8B is evaluated in non non-thinking mode within a 3k context. Figure 3 provides results within a 32k context.
Name
AIME 2024
AMC
MATH-500
GPQA
Avg
Math Base Models
Qwen2. 5-Math-1.55
7.7
28.6
32.7
24.9
23.5
TTRL
15.8
73.0
26.1
41.0
+20.3
+-4033
+1.2
+17.5
105.2%
71.0%
123.2%
t 4.8%
1 74.4%
Qwen2. 0 5-Math-7B
12.9
35.6
46.7
29.1
31.1
w/ TTRL A
40.2
68.1
83.4
27.7
54.9
+27.3
+32.5
+33.7
-1.4
+23.8
211.6%
91.3%
78.6%
4 4.8%
T 76.5%
Vanilla Base Models
Oven2.5-7B
7.9
34.8
60.5
31.8
33.8
w/ TIRL A
23.3
56.6
80.5
33.6
48.5
+11.4
+21.8
+20.00
+1..8
+14.7
194.9%
62.6%
33.1%
5.7%
43.7%
2-322
7.9
32.6
55.8
33.2
32.4
w/ TTRL
24.0
59.3
83.2
37.7
51.1
A
+16.1
+ +6..7
+22.44
+4.55
+18.7
203.8%
81.9%
49.1%
13.6%
57.7%
Instruct Models
LLaMA3.
4.6
23.3
48.6
30.8
26.8
w/ TTRL
10.0
32.3
63.7
34.1
35.0
A
+5..4
+-9.0
+15.1
+33.3
+8,,2
117.4%
38.6%
31.1%
10.7%
t 30.6%
Qwen3-8B"
26.9
57.8
82.3
48.1
53.8
w/ TTRL
46.7
69.1
89.3
53.0
64.5
+19.8
+11.3
+7.0
+-4.9
+10.8
73.6%
19.6%
1 8.55%
1 10.2%
1 20.0%
3 Experiments
3.1 Experimental Setup
Models To evaluate the generalizability of TTRL across different backbone models, conduct experiments using both base and instruct models various scales. addition, we carry out experiments on leading LRMs demonstrate that TTRL can improve model performance even after costly post-training. The models we experiment with are as follows:
Qwen Family: Qwen2.5-Math-11 1.5B [54], Qwen2.5-Math-77 [54], Qwen2.5-7B [55], Qwen2.5- 32B [55], Qwen3-88 (thinking mode & non-thinking mode) [55];
LLaMA Family: -LaMA-3.1-8B-Instie [10], LLaMA-3.2-338 Instrrrr [10], LLaMA-3.2-3B.
Oat-Zero [26];
Mistral Family: Mistral-Nemo-lnstmmee244040 [29],
[28);
DeepSeek Family: DeepSeek-Math-7B--nne [38], BeepSeck-RI-LLLaMAA [11]:
Others: Skywork-ORl-Mathh [13];
Benchmarks We evaluate TTRL on GPQA-Diamont [35], challenging and high-qualit subset of the Graduate-Level Google- Proof Question Answering benchmark, and mathematical reasoning benchmarks: AIME 2024 [21]. AMC [21], and MATH-500 [14].
Evaluation Setup We apply TTRL to each benchmark individually and then evaluate. We set the maximum generation length to 3072 tokens, unless otherwise specified. For the main experiments, following DeepSeek-RI [11], we adopt the @ k evaluation protocol [3] and calculate pass @ using non-zero temperature sampling. Specifically, we generate 16 responses (4 for 32k context) per question using temperature of 0.6 and top-p value of 0.95. For the analysis and additional experiments on Qwen2. 5-MATH, we evaluate using greedy decoding to report pass @ 1, ensure

---

fair comparison with previous works. Appendix E presents set of training-ting timenng--tmmmee metrics we used to monitor the performance TTRL and analyze its training dynamics in the absence of ground-truth
Baselines Since the use of TTT for reasoning has not been previously explored, we primarily compare it with the backbone model to validate whether TTRL can achieve effective improvements through self-evolution. Appendix D presents additional experimental results comparing TTRL with previous state-of-the-an RL approaches for reasoning.
Implementation Details We independently apply GRPO [38] on each benchmark to implement TTRL. For hyperparameterss, we cosine learning rate schedule with peak value of 5 x 10-? and adopt the AdamW optimizer for the policy model. For rollout, we sample 64 responses using temperature of 0.6 (1.0 for Qwen2. 5 5-Math and LRMs) for voting- based label estimation and downsample 32 responses per prompt for training. Evidence shows that our vote-then-sample strategy effectively reduces computational costs while still achieving strong performance. The maximum generation length is set to 32,768 tokens for LRMs and 3,072 tokens for all other models. We set the number of episodes 10, 30, and 80 for MATH-500, AMC, and AIME 2024, respectively, based on the dataset size. All experiments were conducted on 8 NVIDIA A A00 80GB GPUs.
3.2
Main Results
TTRL performs well on most tasks and models. Table I presents the main results. We apply TTRL to 6 models spanning 4 model families, 2 model types, and model sizes, consistently demonstrating substan- tial improvements across 4 highly challeng- ing benchmarks. On the demanding mathe- matical reasoning benchmark AIME 2024, TTRL achieves minimum improvement of 105% across all 6 models. Moreover, applying TIRL to a 1.5B model leads to significant gain of up 40.3 points on the MATH-500. Recently, Shao al. [37] demonstrated the importance of evaluat- ing different models for RL-based meth- ods to validate experimental conclusions. Therefore, we additionally report results on broader range of models from var- ious model families, such as DeepSeek- RI-LLaMA-8B. an LRM from DeepSeek trained on the LLaMA model. Table 2 presents the results. As shown, TTRL con- tinues to exhibit consistent effectiveness. Furthermore, as shown in Appendix D, de-
Table 2: Performance of TTRL on various models.
Name
AIME
AMC
MATH-500
LLaMA Family
3.MMA-3.-3B-Ou
0.8
15.1
41.9
TTRL
25.3
55.7
+2.5
+10.2
+13.8
L.aMA-3.2-3B-mmmm
43.9
TTRL
13.3
31.3
61.6
+7.3
+11.9
+17.7
Mistral Family
Mistral-Nemo-Instuue
0.8
15.4
40.8
w/ TTRL
24.8
$1.0
A
-0.8
+9.4
+10.2
Ministral-B Instrruut
1.3
19.7
52.4
w/ TTRL
3,3
28.9
57.8
+2.0
+9.2
+5.4
DeepSeek Family
DeepSeck-Math-TB. Math-7B-Innttrutt
1.9
16.3
42.3
w/ TTRL.
2.5
22.9
52.4
+0.6
+-6.6
+10.1
DeepSeck-RI-LL
51.7
81.6
89.6
w/ TTRL.
69.2
88.9
90.9
177.5
spite relying solely on self-evolution using unlabeled test data, TTRL achieves performance compa- rable to existing RL-based models that are trained on large-scale labeled datasets.
TTRL performs well on LRMs. With the rapid progress in RI and TTS, LRMs becoming in- creasingly central. To further examine whether
AIME 20024 Accuracy
Before TTRL
After TTRL
82.5
TTRL remains effective on LRMs that have un- dergone expensive post-training, especially on highly challenging tasks, we evaluate powerful LRMs. Figure 3 presents the results of applying TTRL additional reasoning mod- els. Qwen3-8B is evaluated in thinking mode.
80
75.0
75
72.5
other
70
66.7
65
Despite the extensive post-training these models have undergone, TTRL still achieves substan- tial performance gains, yielding improvements of approximately 10 points both backbones.
60
Skyworkk-OR 1R1-Math-1MB
Qwen3-8B
Model
Figure 3: TTRL on LRMs.
5

---

TTRL naturally scales, Another noteworthy observation is that as the model size increases (1.5B - 7B and 7B + 32B). performance consistently improves, highlighting the natural scaling behavior of TTRL: larger models can produce more accurate majority voting rewards during self- seff-improvement, which leads to more effective learning on new data.
TTRL generalizes well beyond the target task. We perform TTRL on each benchmark and further evaluate pass@1 using greedy decoding on others, with Qwen2. 5-Math-78 as the backbone. Figure 4 shows the results. Despite the out-of-distribution nature of this setting, TTRL achieves substantial improvements across all benchmarks. This suggests that TTRL does not rely on overfitting. which would lead to trade-offs on other tasks, but instead acquires generalizable gains during self- improvement.
TTRL AIME 2024
TIRL AMC
TTRL MATH-500
80
75.4
Before TTRI.
After
63.9
Accuracy
60
52.6
$2.6
39.8
33.3
23.3
13.3
13.3
0
Evaluation On
Evaluation On
Evaluation On
Figure 4: Out-of-distribution performance before and after TTRL.
TTRL compatible with different RL algorithms. We further apply TTRL using two RL algorithms MATH-500 with Qwen2. 5 5--55----h---. to assess its compatibility, which are PPO [36]. a value mode based method, and PRIME [7], process-level RL algorithm. Figure 5 presents the results. The performance trajectories of GRPO, PPO, and PRIME are closely aligned.
0.70
GRPO
PRIME
0.7
PPO
Acouracy
Entropy
0.50
0.5
0.45
GRPO
0.40
PRIME
0.3
0.35
PPO
25
15
20
Steps
Steps
(a) Accuracy Curve.
(b) Entropy Curve.
Figure 5: Comparison over steps of different RL algorithms, GRPO, PPO, and PRIME on MATH-500.
TTRL achieves sustainable self-evolution through "online" and "RI". To gain deeper under- standing of the underlying mechanisms of TTRL, we conduct analysis of the model's training dynamics by tracking the average (pass@ 1/avg @ 16) and majority (maj@ @ 16) scores throughout the training process. Given that majority voting serves as the basis for generating training signals, examining its performance trajectory is essential for understanding how it functions. Furthermore, we investigate whether TTRL improves pass@) at the cost of a reduction in maj @ 16 performance. Figure 6 illustrates the TTRL training dynamics on AMC with Qwen2. 5-Math-1 1. 5B as the base model. It notable that, training progresses, both metrics demonstrate a consistent upward trend. This indicates that TTRL is not simply approaching the initial model's majority voting performance. Due its dynamic nature, TTRL can generate higher-quality supervision signals as its capabilities improve. Moreover, through TTRL's use of RI for TTT, by converting voting-based pseudo-labels into reward signals, enhances the effective supervision quality (e.g., accuracy; see Q2 4.2), while decoupling learning from the limitations imposed by maj nn.

---

0.56
0.52
Avg@:1
0.40
Ma/@16
0.48
0.35
0.44
0.30
160
240
Steps
Steps
Figure 6: Training dynamics of TTRL AMC using Qwen2.5-Matee 5--Math1.... 1.5B as the base model.
4
Analysis and Discussions
4.1 Q1: How Well Can TTRL Perform?
Takeaways
1. TTRL surpasses the traditional self-training upper bound, the majority accuracy of the initial model.
2. The empirical upper bound of TTRL is direct RL on labeled test data (i.e., training the test data). TIRL can approach the performance of this upper bound, highlighting its potential advantages in efficacy
standard Iraining-evee -eteniing-vvllatiion protocols.
3. For challenging tasks, TTRL can reach the empirical upper bound using only a 1.5B model. This demonstrates that LLMs can now efficiently self-evolve through TTRL, enabling unbounded lifelong learning large-scale datasets.
We analyze the potential performance of TTRL using two upper bounds. The first upper bound is the maj @n of the initial model. The second upper bound is direct training benchmark data, which assumes access to ground-truth truth labels and thus leaks label information to the policy model.
Qwen2.5-Math- Avg@64
85.2
Qwen2.5-Math-78 Maj@64
Accuracy
TTRL Avg@64
63.9
66.3
66.4
56.6
w/ TTRL Maj@64
44.6
40
30.0
32.3
33.0
20
11.7
AIME 2024
AMC
MATH-500
Figure 7: Majority voting performance comparison between the backbone and after TTRL.
TTRL is Supervised by maj@n Yet Surpasses It. Since TTRL utilizes the model's own majority- voted outputs for RL, this voting-based performance of the initial model can intuitively be regarded as an upper bound of the final performance. This upper bound also the performance limit of traditional self-training methods [17], which select self- -effeeenedted CoT through majority voting for supervised fine-tuning (SFT). However, we observe a surprising phenomenon: after training, the model not only matches but also surpasses the expected upper bound, suggesting that it exceeds the performance limit of the original model, which also serves as initial supervision signal. Figure illustrates this remarkable result, where it can be observed that the final avg @ 16 score exceeds the initial maj @ 16 score by more than 20 points. Furthermore, we perform additional evaluations of TTRL Qwen2. 5- Math- 7B across various benchmarks, using more samples per question enable more

---

reliable assessment. Figure shows results. It can be observed that TTRL avg@ 64 consistently outperforms Qwen2. 5-Math- maj @4 across all benchmarks, with a considerable margin. Through a self-reinforeing loop, the model "lifts itself up by its own bootstraps", evolving beyond the performance ceiling. Moreover, the performance of TTRL further improves with majority voting.
TTRL's Performance Gains Approach Train- ing on the Benchmark. The motivation of TTRL is to estimate labels using majority vot- ing obtain more accurate rewards, facilitating effective self-impprvemmnt through RL on the data without ground-trun truth labels. Therefore, natural upper bound of TTRL is performing RL directly on the test data, denoted as RL. (leakage). Although this setting is rarely adopted or studied due to the issue of information leakage, repre- sents the most efficient way to improve perfor- mance on the particular dataset, with efficiency that far exceeds traditional training-evaluation paradigms. We use Q M-Math-7B to per- Figure 8: Comparison of RL (Leakage) vs TTRL. form both TTRL and RL (leakage) on MATH- 500 and conduct evaluations. Figure shows results. Surprisingly, we find that the performance curve of TTRL closely approaches that of RL (leakage). This suggests that:
0.80
0.75
Accuracy
0..70
0.65
0.60
(Leakage)
0.55
TTRL
10
20
30
40
50
Steps
TTRL can achieve level of reffvimproeement comparable that of supervised learning (even in the information leakage scenario) through RI. in an unsupervised setting.
TTRL provides evidence that even small LLMs can now effectively self-improve on input-only challenging tasks through RL, enabling continual learning. Results on Qwen2 5-Math-1.55 5B further support this observation: starting from subpar performance of 32.7 on MATH-500, the model improved by 123.2 2% to reach 73.0, demonstrating clear self-in raff-improvement through TTRL.
4.2 Q2: Why Does TTRL. Work?
This section presents an analysis of the factors enabling TTRL to achieve stable and effective RL under unsupervised conditions. Our analysis identifies three key factors: reward calculation, label estimation, and online learning.
Reward Calculations. When the model is capable of estimating accurate labels via majority voting, the reward and subsequently training are generally reli- able. However, a natural question arises: Why does TTRL remain effective even when the model fails to estimate accurate labels via majority voting on chal- lenging benchmarks such AIME 2024? The most fundamental reason lies in the mechanism by which the verifier computes rewards in RL. For tasks such as mathematics, the verifier works based on "compari- son" to obtain rule-based rewards by checking whether the predicted answer matches the given "label." This mechanism lead to the phenomenon of "Lucky Hit": for an incorrectly predicted answer, even if the estimated label does not match the ground truth label, as long as it differs from the predicted answer, the verifier will still output a negative reward, and this is exactly the correct reward that we expect, as illustrated in Figure 9. In other words, it sufficient that the es- timated label differs from the predicted answer for the verifier to assign the correct negative reward. To provide a more detailed case study, we examine the performance of TTRL on the AIME 2024 using Qwen2. 5-Math-7B Figure 10 presents the variation curves of the three metrics, as described Appendix E. We identify two main reasons why TTRL remains effective on AIME 2024:
Sampled Predictions: 11222454
Estimated Label: Estimated Rewards: 0 10111000
True Label:
True Rewards: 0 00000000
Reward Hit Rate: 62.5%
Figure 9: A toy case of "Lucky Hit". We illustrate basic numerical prediction sce- nario to compare reward computation under two conditions: when the model incorrectly estimates the label versus when the ground- truth label is used. As shown on the left, al- though the estimated label is incorrect, some the incorrect predictions still differ from the wrong label and therefore receive the cor- rect reward (denoted as 0).
8

---

0.6
0.5
Reward Accuracy
0.90
Majority Ratio
Label Accuracy
0.5
0.4
0.85
0.3
0.4-
0.3
15
30
15
Steps
Steps
Steps
Figure 10: Comparison of Majority Ratio, Label Accuracy, and Reward Accuracy on AIME 2024 over steps. Even with low label accuracy, reward accuracy remains high due to "Lucky Hit", allowing TTRL to provide reliable training signals.
1. Reward robustness enabled by multiple outputs within a rollout. First, rewards are denser than labels, allowing for more opportunities to recover useful reward signals even when the estimated label inaccurate. For example, even when the predicted label is incorrect, alternative outputs within the same rollout can still yield correct high-quality rewards, as shown in Figure 9, whereas a rollout containing only a single output would not provide such flexibility. This makes the overall reward signal more robust errors in pseudo-label estimation.
2. High reward accuracy due to scattered incorrect predictions. Second, counterintuitively, when the model has weaker capability, the majority voting rewards of TTRL may be more accurate. As shown in Figure 10, although the initial label estimation through majority voting achieves an accuracy of only 37%, the reward accuracy reaches an impressive 92%. By examin- ing the model outputs, we find that this is because the model's responses are highly scattered result consistent with this observation is
and consistently incorrect, as shown in Figure that, for the base model, the most frequently predicted answer accounts for only 16.6% of all predictions, indicating that the outputs are highly scattered. Therefore, even when the labels are not accurately estimated, due to "Lucky Hit", most outputs can still receive correct rewards. Morcover, the poorer the model's performance, the more mistakes it tends to make, which para- doxically leads to more accurate reward estimation. An empirical observation supporting this view is the comparison between the label accuracy and reward accuracy, as shown in Figure 10. Although the label accuracy rarely exceeds 50%, the reward accuracy remains consistently high, staying above 75%. This high reward accuracy provides a reliable foundation for effective self- celf-improvemment on test data.
Label Estimations. A direct difference between TTRL and standard RL algorithms is that TIRL involves label estimation, which introduces reward inaccuracies. We believe that TTRL works despite these inaccuracies due to the following two reasons. (i) Existing studies have shown that RL can tolerate a certain degree of reward inaccuracy. Moreover, RI. tends to generalize better than SFT, which often relies on memorizing training data [6]. In RL, rewards are typically vague and serve primarily directional signals for exploration, leading RL's robustness to reward noise [34]. (fi) Prior work has also examined what constitutes a good reward model from an optimization perspective, revealing that more accurate reward models are necessarily better teachers [45]. Therefore, reward signals estimated by the policy model itself may offer more suitable guidance for learning.
Online Learning. TTRL is designed based on online RL approach, whereas traditional self- training and test-time training methods operate in an offline manner. The online nature of TIRL enables the model to improve its capabilities during the application, which in turn leads more accurate labels generated through voting. As a result, the quality of the supervision signal improves, allowing for truly sustainable self-evolution. As shown in Figure 6, this dynamic learning process leads to a complementary improvement performance in both pass@) and maj@n.
Related Works
5.1 Test-Time Scaling
Test-Time Scaling (TTS) is designed enhance the capabilities Large Language Models (LLMs) in handling complex tasks by increasing computational resources at test time. Prior research [25, 40]

---

indicates that TTS is more efficient than scaling during pre-training [19]. Therefore, reallocating the same computational resources from tre-training to test-time could yield greater improvements in model performance. Current studies on TTS fall into two categories [51]: parallel generation and sequential generation. Parallel generation involves L.LMs producing multiple candidate responses (self-consistency [4, 48), best-of-N [30, 41)), decision steps (Monte Carlo Tree Search [52, 63]), or tokens (Reward-guide guided Search [9, 20]) during inference. Subsequently, an aggregation strategy is applied to integrate these candidates, commonly using process reward models [24, 46, 60J. Concur- rently, sequential generation focuses on extending the LLMs' output to include longer responses with reflective and chain-of- of-thoughtt (CoT) processes [27, 49]. Although prompting techniques are widely adopted, they are often constrained by the capabilities of the underlying models. Notably, DeepSeek- RI [11] is representative ad advancement in this area, achieving extended reasoning capabiliitess in pre-trained language models through outcome-based reinforcement learning (RL., more specifically group relative policy optimization [38]. Compared the first approach, which requires intensive process-level supervision [57], the second approach is more scalable due to its reliance on rule-based rewards.
Beyond the aforementioned methods that focus on scaling test-time inference computation, another approach to increasing test-time computing is Test-Time Training (TTT). We introduce the re- lationship between these terminologies in Appendix F. While prior work has primarily focused on applications such as video generation and understanding [8. 12], and to some extent large language models [1, 47], the integration of test-time scaling with reinforcement learning remains largely underexplored.
5.2 RL for Reasoning
Reinforcement Learning (RL) [44] plays a critical role in enhancing the cntrrcttionfollowwing ca- pabilities of Large Language Models (LLMs), particularly through approaches like Reinforcement Learning from Human Feedback (RLHF) [31]. RLHF aligns base models with human preferences using algorithms such as Proximal Policy Optimization (PPO) [36], where preference modeling is essential. Recently, Large Reasoning Models (LRMs), such as DeepSeek-RI [11]. have demonstrated the significance of RL in improving reasoning abilities using rule-based rewards, exemplified by GRPO [38]. Unlike RLHE, which is tailored to open-domain instructions, GRPO is specifically designed to elicit long CoT [49] reasoning in mathematical moblemmssolving Recent studies have focused primarily on improving the training stability of rule-based RL methods like GRPO and PPO [7, 26, 56]. However, these methods typically train LLMs only on supervised training data, while inference involves generating extended CoT reasoning on unseen test problems. Moreover, current RL approaches [15, 50] depend on verifiable outputs, such as solutions in mathematics or code, that can provide reliable reward signals.
Previous studies have explored self-rewarding [32, 58] and self-play training [5] for unlabeled data. However, these works primarily focus on open-domain instruction following [5, 58] rather than mathematical reasoning employ preference- baasee optimization strategies [32] such DPO [33] instead of online reinforcement learning algorithms. In addition to these studies, we identified several concurrent works [53, 61, 62), that explore self-supervised and semi-supervised reasoning using reinforcement-like methods. The key distinction lies in reward estimation: we employ majority voting, which is derived from the model itself and mitigates reward hacking. We acknowledge that future research integrating the insights and strengths of these approaches could lead to more robust reasoning models the era of experience [39]. TTRL offers a preliminary attempt RL with self-labeled rewards, ad advancing toward learning from streams of experience.
6 Conclusion
In this paper, we propose Timt-Time Reinforcement Learning (TTRL), novel framework for training large language models with Reinforcement Learning (RL) on test data without access to ground -truh labels. A key component of TTRL is its majority voting reward function, which generates based rewards based on consensus among model predictions. Our experiments demonstrate the strong potential of TTRL, achieving consistent improvements across a variety of models and tasks. We view TTRL as a preliminary step toward RL. with labeeld rewards, marking an important direction of learning from continuous streams of experience.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by National Science and Technology Major Project (20232D0121403) Young Elite Scientists Sponsorship Program by CAST (20230NRC001) National Natural Science Foundation of China (No. 62406165), and Shanghai Municipal Science and Technology Major Project. We thank anonymous reviewers for their insightful comments and suggestions.
References
[1] Ekin Akyürek, Mehul Damani, Linlu Qiu, Han Guo, Yoon Kim, and Jacob Andreas. The surpris- ing effectiveness of test-time training for abstract reasoning. arXiv preprint arXiv:2411.07279, 2024.
[2] Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning memorize at test time. arXiv preprint arXiv:2501.00663 2024.
[3] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
[4] Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang. and Denny Zhou. Universal self-consistency for large language model generation. arXiv preprint urXiv:2311.17311 2023.
[5] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint urXiv:2401.0133 2024.
[6] Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shenghang Tong, Saining Xie, Dale Schuurmans, Quoe Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: comparative study of foundation model post-training. arXiv preprint ar axXiv:2501.1716,,00 2025.
[7] Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.014
2025.
[8] Karan Dalal, Daniel Koceja, Gashon Hussein, Jiarui Xu, Yue Zhao, Youjin Song, Shihao Han, Ka Chun Cheung, Jan Kautz, Carlos Guestrin, et al. One-minute video generation with test-time training. arXiv preprint arXiv:2504.05298, 2025.
[9] Haikang Deng and Colin Raffel, Reward-augmented decoding: Efficient controlled text genera- tion with unidirectional reward model. arXiv preprint arXiv:2310.09520, 2023.
[10] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407:21783, 2024.
[11] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12944. 2025.
[12] Moritz Hardt and Yu Sun. Test-time training nearest neighbors for large language models, 2024. URL https: arxiv. org aa////b//330 18466.
[13] Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xi- aoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li. Liang Zeng. An, Yang Liu, and Yahui Zhou. Skywork
Tianwen Wei, Cheng Cheng, http:://carrrccooo--yytrrgg---- notion. open reasoner series.
htt.:/eo........ioso......... Skywork- Reaonser-Series- oet.rooeeee.cceeooooooeeeeeeeaoe
2025.
Notion Blog.

---

[14] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. 2021.
arXiv preprini
[15] Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yee Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model, arXiv preprint arXiv:2503.242200 2025.
[16] Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung, Shum. Open-reasoner-zero. An open source approach to scaling up reinforcement learning on the base model, 2025. URL https: org/ abs/2503 24290.
[17] Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang. Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint arXiv:2210.11610 2022.
[18] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El- El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412. 2024.
[19] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown. Benjamin Chess, Rewon Child, Scott Gray, Alee Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
[20] Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment as reward-guided search. arXiv preprint arXiv:2402.011694 2024.
[21] Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang. Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024.
[22] Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for rl scaling. arXiv preprint arXiv:2502. 1886, 2025.
[23] Jian Liang, Ran He, and Tieniu Tan. A comprehensive survey on test-time adaptation under distribution shifts. International Journal of Computer Vision, 133(1):31664, 2025.
[24] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
[25] Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou. Can lb Ilm surpass 405b Ilm? rethinking compute-optimat test-time scaling. arXiv preprint 7xXiv22502.0670: 2025.
[26] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding ri-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025.
[27] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534 2023.
[28] Ministral-8B-Instrree Ministral-Bh-innsrueet, 2024. URL httphtt////:ttrret ai/nws/ ministraux.
[29] MistralAI-NeMo. Mistralai-nemo, 2024. URL.
[30] Reiichiro Nakano, Jacob Hilton, Suchir Balaji. Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et Webgpt: Browser-assisted question- with human feedback. arXiv preprint 21933221,. 2021.
12

---

[31] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[32] Archiki Prasad, Weizhe Yuan, Richard Yuanzhe Pang, Jing Xu, Maryam Fazel-Zarandi, Mo- hit Bansal, Sainbayar Sukhbaatar, Jason Weston, and Jane Yu. Self-consistency preference optimization. arXiv preprint arXiv:2411.04109, 2024.
[33] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36:53728-53741, 2023.
[34] Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D Lee, and Sanjeev Arora. What makes a reward model good teacher? an optimization perspective. arXiv preprint WrXi:22503.1547, 2025.
[35] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google- progle q&a benchmark. In First Conference on Language Modeling, 2024.
[36] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: Nidi:::777.06447 2017.
[37] Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang. Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, et al. Spurious rewards: Rethinking training signals in rlvr. arXiv preprint arXi::2506.10944 2025.
[38] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
[39] David Silver and Richard S Sutton. Welcome to the era of experience. Google AI, 2025.
[40] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint vrXiv:2408.03314 2024.
[41] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances neural information processing systems, 33:3008-3021. 2020.
[42] Yu Sun, Xiaolong Wang. Zhuang Liu, John Miller, Alexei A Efros, and Moritz Hardt. Test-time training for diittiiiitttttoonn generalization. Arxiv, 2019.
[43] Yu Sun. Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, al. Learning to (lear at test time): Rnns with expressive hidden states. arXiv preprint arXiv:2407.04620, 2024.
[44] Richard S Sutton, Andrew G Barto, al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[45] Jingkang Wang, Yang Liu, and Bo Li. Reinforcement learning with perturbed rewards. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 6202-6209, 2020.
[46] Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. arXiv preprint arXiv:2312.089: 2023.
[47] Renhao Wang, Yu Sun, Arnuv Tandon, Yossi Gandelsman, Xinlei Chen, Alexei A Efros, and Xiaolong Wang. Test-time training on video streams. Journal of Machine Learning Research, 26(9):1-29, 2025.
13

---

[48] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. //171, 2022.
[49] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-the ofothut prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2483 2022.
[50] Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing Ilm reasoning via reinforcement learning on open software evolution. arXiv preprini arXiv:2502. /8449, 2025.
[51] Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, and Zaid Harchaoui. From decoding to meta generation: Inference-time algorithms for large language models. arXiv preprint arXiv:2406. 16838, 2024.
[52] Yuxi Xie, Anirudh Goyal, Wenyue Zheng. Min- Yen Kan, Timothy P Lillicrap. Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405.0045) 2024.
[53] Fangzhi Xu, Hang Yan, Chang Ma, Haiteng Zhao, Qiushi Sun, Kanzhi Cheng, Junxian Hc, Jun Liu, and Zhiyong Wu. Genius: A generalizable and purely unsupervised self- self-training framework for advanced reasoning. arXiv preprint arXiv:2504.08677 506.00677, 2025.
[54] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, al. Qwen2 technical report. arXiv preprint arXiv:240..1667,, 2024.
[55] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412. 151 15, 2024.
[56] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
[57] Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. arXiv preprint arXiv:2412.01991/ 2024.
[58] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2025. URL https:///// org/ abs/2401. 10020.
[59] Weihho Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025. URL https: / arxiv. org/ aaas//503 2503. 18892.
[60] Kaiyan Zhang, Jiayuan Zhang. Haoxin Li, Xuckai Zhu, Ermo Hua. Xingtai Lv, Ning Ding, Biqing Qi, and Bowen Zhou. Openprm: Building open-domain process-based based reward models with preference trees. In The Thirteenth International Conference on Learning Representations, 2025.
[61] Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question already half the answer: Fully unsupervised 11m reasoning incentivization. arXiv preprint arXiv:2504.05812 2025.
[62] Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data, 2025. URI. htttpps///axiv....
[63] Andy Zhou, Kai Yan, Michal Shlapentokh Rothman, Haohan Wang, and Yu- fu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.c 10.04406 2023.
14

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
No theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
wwt.:/uwwwsssooooooooooooooooooe oonolccy)) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
CCodeeubeiieeeeeee / //nips.coo. hpppppulluuddeeeeeeeeeppeppbpeee /tt.i///ggiiiee//eeeuubbiiiiii iesisssinnn ssionnollll seiiinyylcc)) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Conduct multiple experiments to calculate the pass @ass@1 1.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / newrips........ cc
Answer: [Yes]
Yes, the research conducted in this paper conforms to all aspects of the NeurIPS Code of Ethics. Specifically, ensured compliance regarding transparency, reproducibility, potential societal impacts, respect for privacy, and fairness. No ethical concerns arose from our methods, datasets, experiments throughout the research process.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper presents foundational research and not tied to any particular application.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Section 3
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (bttps://neur)) //tpsr//eeuu..eeeeeee www.arpps../ooooeeeeeeeeeeeeeeee cc//ooffereccce//22/1/I1 for what should or should be described.
21

---

A Limitations and Future Works
This work represents an initial exploration of test-time reinforcement learning using self-labeled rewards. While our experimental results are promising, several aspects require further investigation. In particular, we plan to conduct a more in-depth analysis of the impact of prior knowledge and hyperparameter configurations, both of which play critical roles in reinforcement learning dynamics. We will provide comprehensive discussions and ablation studies in future revisions of this paper.
Limitations
Future Works Building on our findings, we identify several directions for future research:
Theoretical Analysis: Developing formal convergence analysis of TTRL, particularly focusing on its ability to optimize toward the two upper bounds in $ 4.1.
Online Learning with Streaming Data: Extending TTRL to real-time learning scenarios, where models interact with continuously arriving data and adapt dynamically, that is Test-Time Adapta- tion [23].
Large-Scale Self-Supervised RL Training: Scaling up TTRL massive datasets and models to explore its potential in self- self-supervise regimes without human-labele data.
Agentic Tasks and Scientific Discovery: Applying TTRL to more complex, open-ended domains such agentic tasks and multi-step scientific reasoning.
B When Might TTRL Fail?
At the algorithmic level, TTRL is not fundamentally different from existing RL algorithms and therefore inherits several of their characteristics, such as sensitivity to data difficulty, strong reliance on priors, and risk of collapse under certain conditions. At the implementation level, these issues are further amplified by the constraints of TTRL, which estimates labels via majority voting and operates exclusively on test data that is both sparse and previously unseen, potentially resulting in failures in certain scenarios. In our preliminary experiments, we identified two potential issues:
Inappropriate RL Hyperparameters. Hy- perparameter settings play crucial role in RL training, varying across projects and often lead- ing to training failures. The influence of hyper- parameters is further amplified in TTRL due to potential noise in reward estimation and the char- acteristics of the test data. Figure 11 presents a comparison of several unsuccessful attempts on AIME 2024. Both of these failed attempts exhibit persistently high entropy that does not diminish throughout training, consistent with findings of prior work [13]. In our preliminary experiments, identified two key hyperparam- eters that can critically affect training stability and success:
0.30
0.25
Test Acc.
0.20
0.15
Successful Attempt
0.10
Temperature@0.6
0.05
Train 256
12
20
24
28
Steps
Figure 11: Failed attempts. We compare the curves under settings with appropriate parameters versus those with suboptimal ones.
Temperature: Setting the temperature to 1.0, as opposed 0.6, increases the model's output entropy. This promotes more extensive exploration and allows the model to make better use of its prior knowledge for self-improvement, which is particularly important when addressing challenging benchmarks.
Episodes: Given the substantial variation in size and difficulty across datasets, smaller and more difficult datasets need more episodes to achieve sufficient exploration.
Lack of Prior Knowledge on Target Task. Prior knowledge plays a crucial role RL, often determining the success or failure of the TTRL learning process. This is mainly because the test data generally exhibits higher difficulty and introduces new features, but TTRL does not incorporate mechanisms such data filtering to support curriculum learning.
22

---

Table 3: Performance of TIRL across the five difficulty levels of MATH-500
Metric
Name
MATH-500-L.1
MATH-500-122
MATH-500-1.3
MATH-500-L4
MATH-500-1.5
Accuracy
Backbone
25.9
33.0
36.3
32.5
22.3
W/ TTRL
71.2
76.2
76.3
58.7
39.2
A
+445.4
+43.2
+-40.0
+226.2
+16.8
175.3%
130.8%
110.2%
80.4%
1 75.3%
Response Len.
Backbone
2,339.2
2,125.1
2,120.6
1,775.1
1,751.3
w/ TTRL
624.3
614.4
672.3
783.5
985.3
1,715.0
-1,510.6
1.448.3
-99.6
766.0
4 73.3%
7..1%
4 68.3%
1.559%
43.7%
Therefore, for the same backbone, TTRL fails the model's prior knowledge is insufficient to handle the complexity of the data. To further validate this hypothesis, conduct ablation study MATH-500. We divide MATH-500 into five subsets according to its annotated difficulty levels, ranging from 5, and apply TTRL each subset independently. using We then compare the results to those of the backbone, as shown in Table 3. We observe that as the question difficulty increases, both the performance improvement and length reduction ratios tend to decrease. This suggests that the available prior knowledge of the backbone is insufficient to support learning on more challenging questions.
5B.
C Reward Function Pseudo-Code
Listing The pseudo-code of the majority voting reward function.
from collections import Counter
def vottrtty.voiingeeeeeee fn (outputs):
Assigns a reward to each output whose extracted answer matches the majority ansver, othervise 0.
Extract
from each output
answers textract_ansver (output) for output outputs]
Find the majority answer
counts Counter (answers)
answer, counts most_ (1) [0]
13
14
Assign revards: if matches majority, else
15
rewards [1 if ans -- anserr else 0 for ans in answers]
return rewards
17
18 outputs 11m generete (problle,, n=N)
revards = pajority.voting.. votilyvvviinnnaaeeeeeeeee reward_ fn feeearddnoouututtt_ss (outputs)
D Additional Results
Table 4 shows @ 1 results using greedy decoding. For the two base models, further include comparisons with their instruct versions that have undergone large-scale post-training. In addition, we include for reference current leading "R RRl-ZeroLikke" models with similar backbones, which are extensively trained using RL: DeepSeek-PR L-issilll--.BB&77B [11], SimpleRL-Zero--7 [59], PRIME-Zero-T1 [7]. OpenReasoner--erre Zero-7B [16], Oat-Zero- 1.50-11.B8778 [26], and LIMR [22]. Note that TTRL has a different setup from the previous models, which makes the comparison seem unfair.
the highly challenging mathematical reasoning benchmark AIME 2024, TTRL achieves sub- stantial improvement of 159.3%, surpassing all models trained on large-scale datasets. Furthermore,
23

---

when applied to Qwen2.5 5-Math-7B. TTRL yields an average improvement of 84.1% across three benchmarks. Figure 12 shows two curves of TTRL on AIME 2024 with Qwen2. 5-Math-7B as an example.
Table 4: Additional results of TTRL on each task. indicates results from Dr. GRPO [26]. Our training data size matches the corresponding benchmark dataset size.
Name
AIME 2024
AMC
MATH-500
Avg
Labeled Data
Qwen2.5 5-Math-M----- 115
20.0
32.5
33.0
28.5
TTRL
20.0
53.0
80.0
51.0
+20.5
+47.0
+22.5
63. 1%
142.4%%
1 79.0%
X
Swen2.5-Math-11.. 1.SB-Instruct*
10.0
48.2
74.2
44.1
3. 3.1M
DeepSeck- -ee--R--DDiis Disttillll 1.55 @3'
2.5
21.7
52.2
25.5
800K
DcepScck-R1-Dee R1- 8k'
20.0
49.4
77.4
48.9
800K
Oat-Zero-1.33
20.0
53.0
74.2
49.1
8.9K
Quen2. 5 1-MMtt-7""
16.7
38.6
50.6
35.3
TTRL
43.3
67.5
84.2
65.0
+226.6
+-28.9
+-33.6
+29.77
159.3%
74.9%
66.4%
84.1%
X
Qwen2.5 1ren2.5-Math-22 M-Mathh7B--B IBIntruut
16.7
83.6
51.1
3.1M
DeepSeek-RI -i-bsek-----istlt------ @3k"
10.0
26.2
60.1
32.1
8O0K
SimpleRL-Zerro7BB
26.7
60.2
78.2
55.0
8.9K
PRIME-Zero-78
16.7
62.7
83.8
54.4
230K
OpenieasonerrZeeo-BBB6kk
46.5
129K
Oat-Zero-7BB
43.3
62.7
80.0
62.0
8.9K
LIMR-7B
32.5
63.8
78.0
58.1
1.4K
0.40
Accuracy
0.35
Entropy
0.30
0.30
0.25
0.25
0.20
0.15
WV
Accuracy
0.15
Entropy
10
20
3.0
40
50
60
10
20
30
40
50
60
Steps
Steps
(a) Accuracy Curve.
(b) Entropy Curve.
Figure 12: The entropy and accuracy curves of TTRL on AIME 2024 with Q 5-Math-78
E
Training Metrics
Given the absence of ground-truth labels in the test data, evaluating the performance of TIRL throughout the training process presents challenge. To mitigate this limitation, we introduce set of training-time metrics specifically designed to monitor and assess the effectiveness of TTRL. These metrics inform the selection of the optimal checkpoint and provide valuable insights regarding training dynamics.
Entropy: Measures the uncertainty the model's generation.
Majority Voting Reward: Rule-based rewards computed from the majority-voted label.
Majority Ratio: The frequency of the most common answer within rollout.

---

Furthermore, we define several metrics that rely on access to ground-truie truth labels, which allow for deeper analysis of the model's behavior during training:
Label Accuracy (maj@n): Indicates whether the estimated label matches ground truth.
Reward Accuracy: Indicates the proportion of majority voting rewards (computed from the estimated label) that match rewards computed from the ground-truth label.
Ground-Trune Ratio: The frequency of the ground-truth answer within a rollout.
Terminology
Test-time scaling refers increasing computational resources during test time, which can be catego- rized into test-time training and test-time inference. These two approaches are complementary. We will provide introduction below.
Table 5: Terminology relationship.
Name
Category
Methods
Test-Time Training (TTT) Test-Time Inference (TTI)
Test-Time Reinforcement Learning (TTRL)
Tist-Time Scaling (TTS)
Majority Voting, Best-of-N
F1 Test-Time Training (TTT)
Test- Test-Time Training (TTT) is technique for adapting pre-trained model at inference time to improve generalization under distribution shifts. Let fo denote a model trained on a source domain Ds = {(z,, vi)l = 1 where I; E X. y: V. and represents the learned parameters. During standard inference, the model is evaluated test samples x, D: with fixed parameters 0, where D. D..
In contrast, TTT allows the model to adapt to each test sample by minimizing an auxiliary self- supervised loss Laux. without access to labels ye. The model parameters are updated online with the auxiliary task, which is typically designed to be label-free and consistent with the main task.
F.2 Test-Time Inference (TTI)
Test-Time Inference (TTI) refers the strategy of enhancing the performance of large language model during inference by allocating additional computational resources. Formally, let So denote language model with parameters and let be input prompt. The model generates output y by sampling from the conditional distribution pe (y Iz) TTI techniques aim to improve the quality of y by employing methods such as generating multiple candidate outputs and selecting the best one based on scoring function, or by refining the output through iterative processes [51].
One common approach involves generating N candidate outputs (y1, 32,..., and selecting the optimal output y" using a scoring function syy. x)):
y" = arg max 8(y1, x)
The scoring function sy,, can be instantiated in various ways, such
1. Majority Voting (MV): Selecting the most frequent output among the candidates.
2. Best-of-N (BoN): Using reward models to score each candidate, then selecting the highest- scoring one.
3. Weighted BoN: Integrating MV and BoN strategies to leverage their respective strengths.
25


Reviews:
[{"review_id": "FLp2Z6vRbM", "reviewer": "Reviewer_cYnp", "summary": "This paper introduces Test-Time Reinforcement Learning (TTRL), a novel framework for training Large Language Models (LLMs) with Reinforcement Learning (RL) on unlabeled test data. The core challenge addressed is reward estimation during inference without ground-truth labels. The authors leverage common Test-Time Scaling (TTS) practices, such as majority voting, to generate effective reward signals for RL training. TTRL enables LLMs to self-evolve by exploiting priors from pre-trained models, demonstrating consistent performance improvements across various tasks and model scales. Experiments show that TTRL can surpass the upper bound of initial model performance (e.g., majority voting accuracy) and approach the efficacy of RL with ground-truth labels, even using only unlabeled data.", "questions": "1. Is it equivalent to expanding the number of N in BoN during the RL process?\n2.. In Fig. 6, what if increasing voting number with backbone? I think it's necessary a fair comparison because you have consumed more voting times in RL process.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. This paper proposes TTRL, a novel method for RL on unlabeled data using majority voting, addressing the need for label-free training.\n2. It shows significant improvements (up to 211%) on math reasoning tasks across different model sizes.TTRL surpasses initial model limits and approaches supervised RL performance without ground-truth labels.\n3. It works across tasks (AIME, AMC, GPQA) and RL algorithms (PPO, PRIME), proving generalizability.\nWeaknesses\n1. It doesn’t compare directly with the latest unsupervised RL methods (e.g., Genius), weakening claims of superiority.\n2. Its performance depends on settings like temperature and episode counts, which aren’t fully optimized.\n3. It requires many samples for voting, making it resource-intensive (8×A100 GPUs).\n4. It mainly focuses on math tasks; effectiveness in open-domain NLP is unproven.", "was_revised": true, "final_justification": "With all discussions and  other comments of reviewers considered, I intend to maintain my score.", "rebuttal": "## Response to Weaknesses\n> 1. It doesn’t compare directly with the latest unsupervised RL methods (e.g., Genius), weakening claims of superiority.\n\nThank you for your suggestion! We note that according to the NeurIPS 2025 FAQ, papers appearing online after March 1st, 2025 is generally considered concurrent and authors are not expected to compare to those. Genius was uploaded to arXiv on April 11, 2025, just five weeks before the submission deadline of May 15, 2025. We apologize for unable to conduct this experiment in time. We have briefly discussed it in the related work, in accordance with NeurIPS policy. Nevertheless, the reviewer's suggestion is greatly appreciated and we will further investigate this line of work and include additional experiments in future revisions. We  believe this weakness should not affect the overall assessment of our manuscript and sincerely hope for your reconsideration.\n\n> 2. Its performance depends on settings like temperature and episode counts, which aren’t fully optimized.\n\nFirst, it is important to clarify that in our experiments, apart from minor adjustments to the episode and batch sizes due to the test-time setup, all other configurations, such as the temperature, follow prior work. The sensitivity we mentioned primarily refers to observations made during our exploration of alternative settings for improving performance, where we noticed a degradation in results, consistent with recent findings [1]. Moreover, the training dynamics metrics proposed in Appendix F can be used to further optimize the hyperparameters. At the time, we did not conduct these experiments due to the high computational cost. However, we fully agree that fine-grained ablation and in-depth analysis of the hyperparameters is necessary and could lead to further insights. We will include additional experiments on hyperparameters, such as ablation studies on batch size and training temperature. We sincerely appreciate your suggestion.\n\n> 3. It requires many samples for voting, making it resource-intensive (8×A100 GPUs).\n\nIn fact, increasing the number of vote samples followed by downsampling is optional. TTRL can rely solely on rollouts of standard RL for both voting and training. In this case, the computational cost of TTRL is the same as that of standard RL, without any additional overhead. We provide an experiment under this setting in Response to Weakness 2 of Reviewer 2bYH (2.1). The results show that with a computational cost same to that of standard RL, the performance is comparable to that achieved by incorporating additional generations for voting during rollout.\n\n> 4. It mainly focuses on math tasks; effectiveness in open-domain NLP is unproven.\n\nWe acknowledge that the current version of our experiments mainly focuses on math tasks. This choice was intentional, driven by the verifiability of task outcomes and the stability of the majority-vote-based reward signal. As we emphasized (in line 5 of our paper), TTRL is compatible with arbitrary Test-Time Scaling methods, including generative reward models, and is therefore applicable to general NLP tasks. We will include a discussion on the potential of applying TTRL to open-domain NLP tasks and identify this direction as a key focus for future work. Thanks for your suggestion!\n\n## Response to Questions\n\n> 1. Is it equivalent to expanding the number of N in BoN during the RL process?\n\nWe understand that the concern here refers to increasing the number of samples (N) during inference. Unlike simply enlarging N at inference time, TTRL leverages the reward signal to update the model’s policy distribution, rather than directly adopting the majority vote as the final output. This approach is more favorable for reducing inference-time costs, especially when transferring across different scenarios.\n\n> 2. In Fig. 6, what if increasing voting number with backbone? I think it's necessary a fair comparison because you have consumed more voting times in RL process.\n\nThis is an excellent question! We are willing to include majority voting over a backbone model with a very large number of votes (denoted as maj@n with large n) as a baseline. The inference cost of this setup matches the total training cost of TTRL, enabling a comparison under equivalent computational budgets. In fact, this baseline is unfavorable to TTRL, as majority voting incurs the full inference cost at prediction time for each evaluation on the same dataset, whereas TTRL is trained only once and avoids repeated inference overhead. To more thoroughly address your concerns, we conduct experiments under this setting, performing a series of TTS evaluations on the AMC2023 using Qwen2.5-Math-1.5B and Qwen2.5-Math-7B. The results demonstrate that the performance of majority voting converges as the number of votes increases, which aligns with previous findings, resulting in diminishing returns (see Figure 5 in [2] and Figure 7 in [3]). Therefore, the maximum achievable backbone maj@n scores are approximately 37.3 and 50.79, respectively, even when more voting is applied. In contrast, after applying TTRL, Qwen2.5-Math-1.5B and Qwen2.5-Math-7B achieve scores of 48.9 and 68.1, respectively, significantly surpassing the backbone maj@n with a large n.\n\n| Metric           | Qwen2.5-Math-1.5B | Qwen2.5-Math-7B |\n| ---------------- | ----------------- | --------------- |\n| maj@2            | 28.09             | 33.23           |\n| maj@4            | 33.68             | 41.2            |\n| maj@8            | 37.23             | 45.73           |\n| maj@16           | 38.1              | 47.98           |\n| maj@32           | 38.43             | 49.19           |\n| maj@64           | 38.17             | 49.87           |\n| maj@128          | 37.86             | 50.14           |\n| maj@256          | 37.55             | 50.4            |\n| maj@512          | 37.41             | 50.6            |\n| maj@1024         | 37.3              | 50.79           |\n| w/ TTRL (pass@1) | **48.9**          | **68.1**        |\n| delta            | +11.6             | +17.31          |\n\nTherefore, even under a computationally fair setting, TTRL retains a clear advantage: it converts compute into generalization capability via learning, rather than brute-force sampling. Moreover, in scenarios with distribution shift, TTRL-trained models can adapt to new test distributions using significantly less computation at inference time compared to large-N majority voting. In our revision, we will provide a more thorough discussion on this topic.\n\n## References\n\n[1] POLARIS: A POst-training recipe for scaling reinforcement Learning on Advanced ReasonIng modelS\n\n[2] Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling\n\n[3] Large Language Monkeys: Scaling Inference Compute with Repeated Sampling"}, {"review_id": "V9L9b9KOZa", "reviewer": "Reviewer_JAq8", "summary": "The paper introduces a novel method for training Large Language Models (LLMs) using Reinforcement Learning (RL) on unlabeled/self-generated data. The core idea is to use majority voting as a way to estimate a reward function used for post-training of LLMs. The paper shows that this simple approach can obtain significantly improve the base models and get performance similar to more involved RL schemes (which require human annotation). This paper can be interpreted as an alternative way to perform Test-Time Scaling (TTS) (which has seen much attention in the recent literature).\n\nKey findings:\n- Self-improvement of LLMs: TTRL enables LLMs to self-improve by leveraging the priors in pre-trained models and utilizing majority voting to estimate rewards from unlabeled test data.\n- Consistent Improvement: Experiments demonstrate that TTRL consistently improves performance across various tasks and models.\n- Sustained Self-Improvement: The paper highlights that TTRL achieves sustained self-evolution during \"online\" RL, dynamically generating higher-quality supervision signals as its capabilities improve.\n- Scalability and Generalization: TTRL scales with model size, meaning larger models can produce more accurate majority voting rewards, leading to more effective learning.", "questions": "Please consider the concerns I raise in Weaknesses above and discuss them. Additionally I had a few questions:\n - The authors defined the reward function in Equation 3, which is simply a delta function with respect to the label that has the majority vote. I was wondering if the authors considered any other softer distance functions such that the reward is higher when it is closer to the majority vote and less when it is further away but more as a continuous function.\n - I’m unconvinced with the ‘lucky hit’ phenomenon. It would seem that as the majority ratio increases the reward accuracy actually decreases (in Figure 9). To me it seems that: as the model becomes more confident, the reward gets worse. I’m not sure that’s the behavior we want, the lucky hit seems to be just hoping that there is enough uncertainty when the labels are wrong. But if the majority ratio increases during training surely the lucky hit phenomenon would disappear?\n - I am very curious why the curves are so close in Figure 7! The authors give some suggestions in the paper saying that it's great that TTRL is so close to RL (leakage) here but me it seems to suggest that the RL labels do not provide much signal?\n- It would be interesting to see if models can further improve using TTRL even when RL is employed. I.e. what happens if you use TTRL not just after pre-training but as an additional phase after RL training?\n- Majority voting is shown to work well in practice to improve model capabilities for math reasoning but there are many cases in which one would expect that the assumption of relying simply on the most frequent model output is wrong. Have the authors thought about this?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The setting of trying to achieve self-improvement of an LLM based on unlabeled examples is highly relevant for the community.\n- The proposed extraction of rewards based on majority voting is simple and concise. Leading to an appealing method with practical impact.\n- The method gives improved results across a variety of base models, a big plus given that many recently proposed test time scaling methods did not show consistent improvements across models.\n- The experiments are extensive, consisting of fine-tuning of multiple different base models.\n- The ablation studies are insightful and were good to read. I like the fact the authors investigated scaling behavior which suggests that as the models get larger the bigger the gain provided by TTRL.\n\nWeaknesses:\n- There is one main concern I have: the observed behavior is also explainable from the recently popularized perspective that the pre-trained model already contains the knowledge for significant performance gains; but achieving those gains is difficult via sampling alone (since in long sampling chains the likelihood of a hit is small). If that is the case, couldn't it be that the presented method merely reduces entropy (and does so towards the majority vote) and similar results could be achieved without any RL with a simple tightening of the logits around the majority vote? This concern is corroborated by the fact that the authors observed a clear binary threshold effect (models need to be good enough for TTRL to work effectively).\n- While it is great to see experiments with different models it is important to recognise that the gains achieved by TTRL do not lift any of the models anywhere close to the stat-of-the-art. Additionally simple other TTS methods such as s1 achieve far superior results. Perhaps this is mainly due to the fact that the paper here only experiments with smaller models and does not contain 32B parameter model experiments but it should be discussed (and perhaps experiments with a larger model should be added if feasible).\n- There are some odd/short figure captions. For example the caption in Figure 1 is very terse and what does \"Position\" mean here? \n- Some typos ungrammatical sentences. E.g. line 27 \"Silver and Sutton has recently\" -> \"Silver and Sutton have recently\". Line 30 \"LRM\" -> \"LLM\". Line 41 \"How to obtain rewards [...]\" -> \"How can one obtain rewards [...]\". Line 39 \"Therefore\" does not follow on from the previous sentence? Perhaps say \"In this paper\". Many more such examples throughout. The paper could benefit from some careful proof-reading.\n- The paper uses the term \"self-evolution\" a lot hinting at some connection to an evolutionary process which I find somewhat misplaced. Wouldn't it more sense to just call it self-improvement?\n- The definition of \"rule-based-rewards\" (l 80) seems a bit off. I understand that the authors were trying to make a connection to sparse rewards extracted based on rule matching but there is no such rule here, it's just the score from majority voting. It would be better to refer to it as \"a sparse reward analogous to rule-based rewards\".\n- The setting of self-improving models with their own feedback is not entirely novel and there has been lots of recent work around scaling data that requires less human feedback or ‘labels’ and I wish there were more comparisons made with respect to approaches such as RL with AI Feedback [1] for scaling feedback to train using RL.\n- There seems to be some sensitivity to hyperparameters.\n\n\n[1] Bai, Yuntao, et al. \"Constitutional ai: Harmlessness from ai feedback.\" arXiv preprint arXiv:2212.08073 (2022).", "was_revised": true, "final_justification": "I maintain that this paper is a clear accept to me. The experiments are interesting to the community and the rebuttal further added to the strength.", "rebuttal": "## Response to Weaknesses\n\n> 1. Couldn't it be that the presented method merely reduces entropy (and does so towards the majority vote) and similar results could be achieved without any RL with a simple tightening of the logits around the majority vote?\n\nWe acknowledge that this perspective is insightful. Indeed, TTRL can, to some extent, be interpreted as a form of entropy regularization. However, the RL process is fundamentally different from simple logit scaling or temperature reweighting. We emphasize that the reward is derived from cross-sample majority voting, rather than from static adjustments to the logits. Essentially, TTRL allows the model to endogenously select more strategic behavior sequences through policy learning, rather than relying on post-hoc filtering via majority vote.\n\n> 2. It is important to recognise that the gains achieved by TTRL do not lift any of the models anywhere close to the stat-of-the-art. Additionally simple other TTS methods such as s1 achieve far superior results. Perhaps this is mainly due to the fact that the paper here only experiments with smaller models and does not contain 32B parameter model experiments but it should be discussed.\n\nWe would like to note that experiments on Qwen2.5-32B  have been provided in Table 1. However, it is fully understandable that the reviewers' concern lies in TTRL may does not lift models close to the state-of-the-art. Regarding this, in our Table 2, we present the results of applying TTRL to the highly challenging AIME 2024 using Skywork-OR1-Math-7B and Qwen3-8B. It is evident that TTRL improves the performance of Qwen3-8B from 72.5 to **82.5** on AIME 2024, surpassing models such as OpenAI-o1, DeepSeek-R1, OpenAI-o3-mini (medium), Qwen3-32B, and Qwen3-30B-A3B [1].\n\n> 3. There are some odd/short figure captions. For example the caption in Figure 1 is very terse and what does \"Position\" mean here?\n\nThank you for your suggestion! We will make improvements in the next version. In Figure 1, the term \"Position\" mainly refers to the placement of TTRL within the existing machine learning paradigms. Our primary intention is to describe its distinctions, positioning, and role in comparison with paradigms such as SFT and RL.\n\n> 4. Some typos ungrammatical sentences. The paper could benefit from some careful proof-reading.\n\nThank you very much for pointing this out! We will carefully proofread the entire manuscript to correct these errors and address any others that may be present.\n\n> 5. The paper uses the term \"self-evolution\" a lot hinting at some connection to an evolutionary process which I find somewhat misplaced. Wouldn't it more sense to just call it self-improvement?\n\nOur intention was to highlight the model’s ability to progressively improve by leveraging its own outputs for learning, without external supervision. However, we agree that the term may imply a connection to biological or evolutionary processes, which is not our intended framing. To avoid potential confusion, we will adopt the more precise term “self-improvement” in the revised version and revise relevant sections accordingly.\n\n> 6. The definition of \"rule-based-rewards\" (l 80) seems a bit off. I understand that the authors were trying to make a connection to sparse rewards extracted based on rule matching but there is no such rule here, it's just the score from majority voting. It would be better to refer to it as \"a sparse reward analogous to rule-based rewards\".\n\nYour suggestion is reasonable. Our consideration is based on the definition in DeepSeek-R1, which is \"rule-based verification of correctness\". Therefore, we understand \"rule-based\" as primarily referring to whether the verifier used is rule-based or generative. The score obtained through majority voting here is still computed using a rule-based verifier. Hence, we consider this to still fall under the category of rule-based rewards. Nevertheless, we will consider your suggestion and explore refining the terminology to better distinguish it from traditional rule-based rewards.\n\n> 7. The setting of self-improving models with their own feedback is not entirely novel and there has been lots of recent work around scaling data that requires less human feedback or ‘labels’ and I wish there were more comparisons made with respect to approaches such as RL with AI Feedback [1] for scaling feedback to train using RL.\n\nYes, there are indeed numerous works on self-improvement models. We acknowledge that our discussion of related literature may be somewhat limited. We are grateful for your suggestion and for pointing out this relevant line of work, such as Constitutional AI. We will incorporate the discussion of these works and offer a more thorough comparison.\n\n> 8. There seems to be some sensitivity to hyperparameters.\n\nFirst, it is important to clarify that in our experiments, apart from minor adjustments to the episode and batch sizes due to the test-time setup, all other configurations, such as the temperature, follow prior work. The sensitivity we mentioned primarily refers to observations made during our exploration of alternative settings for improving performance, where we noticed a degradation in results, consistent with recent findings [2]. Moreover, the training dynamics metrics proposed in Appendix F can be used to further optimize the hyperparameters.\n\n## Response to Questions\n\n> 1. I was wondering if the authors considered any other softer distance functions such that the reward is higher when it is closer to the majority vote and less when it is further away but more as a continuous function.\n\nYou have provided an insightful suggestion! This approach is more general, as it can be extended to tasks that cannot be verified through rule-based methods. Since our experiments at that time primarily focused on verifiable tasks such as math, we did not come up with this method. We sincerely appreciate your thoughtful comments and will further try it.\n\n> 2. I’m unconvinced with the ‘lucky hit’ phenomenon. It would seem that as the majority ratio increases the reward accuracy actually decreases (in Figure 9). To me it seems that: as the model becomes more confident, the reward gets worse. I’m not sure that’s the behavior we want, the lucky hit seems to be just hoping that there is enough uncertainty when the labels are wrong. But if the majority ratio increases during training surely the lucky hit phenomenon would disappear?\n\nFirst, your understanding are entirely correct. We may not have explained this clearly: our intention is not to attribute the sustained improvement of TTRL to the \"Lucky Hit\" nor to claim that the impact of \"Lucky Hit\" remains constant throughout training. On the one hand, we consider \"Lucky Hit\" to be one of the reasons why TTRL can be effective even with weak models. It is undeniable that this phenomenon indeed increases the proportion of accurate rewards, even when the reward accuracy declines at later stages. On the other hand, as you pointed out, its influence naturally diminishes during the later stages of training. This attenuation is expected because \"Lucky Hit\" is more likely to occur when the model is weak, and as the model improves through training, the likelihood of such occurrences decreases.\n\n> 3. I am very curious why the curves are so close in Figure 7! The authors give some suggestions in the paper saying that it's great that TTRL is so close to RL (leakage) here but me it seems to suggest that the RL labels do not provide much signal?\n\nWe believe that this is indeed one of the reasons, which aligns with our perspective: the foundation of TTRL is to estimate labels for training through majority voting, and we consider that one reason why TTRL works is that using these estimated labels in a RL manner mitigates the noise contained in them. Moreover, we believe that the comparable performance can also be attributed to the fact that majority voting provides high quality supervision.\n\n> 4. It would be interesting to see if models can further improve using TTRL even when RL is employed. I.e. what happens if you use TTRL not just after pre-training but as an additional phase after RL training?\n\nYour thoughts are exactly the same as our initial intuition! This was indeed the intended outcome when we initiated this work: positioning TTRL as a post-training stage. In fact, we believe this represents a future trend, as the amount of unlabeled and highly challenging data will inevitably grow. Using TTRL to enable the scaling of RL and the self-evolution of models on more challenging and larger data (unlabled data) is valuable. To further verify that TTRL has the potential to serve as a post-training stage even when RL is employed, we conducted further experiments on DeepSeek-R1-LLaMA-8B:\n\n|                      | AIME 2024 | AMC 2023 | MATH-500 |\n| -------------------- | --------- | -------- | -------- |\n| DeepSeek-R1-LLaMA-8B | 51.7      | 81.6     | 89.6     |\n| w/ TTRL              | 69.2      | 88.9     | 90.9     |\n|                      | +17.5     | +7.3     | +1.3     |\n\nAfter applying TTRL, the performance on AIME 2024 improves significantly. The gains on AMC 2023 and MATH-500 are relatively modest likely due to the saturation of these benchmarks. Moreover, as mentioned in Response to Weaknesses 2, we also provide results of TTRL on several LRMs in Table 2 of the paper. These LRMs have already been trained with costly RL, and we hope these results help address your concerns.\n\n> 5. Majority voting is shown to work well in practice to improve model capabilities for math reasoning but there are many cases in which one would expect that the assumption of relying simply on the most frequent model output is wrong. Have the authors thought about this?\n\nPlease see Response to Weakness \"Reward Inference\" of Reviewer KyvW.\n\n## Reference\n\n[1] Qwen3 Technical Report\n\n[2] POLARIS: A POst-training recipe for scaling reinforcement Learning on Advanced ReasonIng modelS"}, {"review_id": "B4nRyx9Bfz", "reviewer": "Reviewer_2bYH", "summary": "Over the last year, RL fine-tuning of LLMs has led to significantly improved reasoning abilities in domains such as mathematics and coding. This work introduces Test-Time Reinforcement Learning (TTRL), a method for enhancing an LLMs task performance by fine-tuning on test samples via self-generated rewards. To avoid test-set leakage, these synthetic binary rewards are obtained by generating multiple rollouts and considering the majority answer as correct (reward=1) and others as incorrect (reward=0). TTRL improves performance on a variety of math and coding benchmarks, including MATH-500 and AIME2024. Consequently, this work introduces a simple but effective method to improve performance on hold-out data.", "questions": "1. Could the authors clarify the differences in base model performances between this work and the results reported in other works? \n2. What are your thoughts on the restricted token generation budget? Can you provide an ablation to reject the hypothesis presented above?\n3. Do you see any signs of collapse of reward hacking? For example, does the fraction of the majority class approach 1? \n4. Can you clarify the sensitivity to hyperparameters of TTRL (Appendix C)? It would be valuable to see more in-depth analysis on various temperatures and batch sizes (similar to Figure 10) to understand the sensitivity of TTRL better.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: \n\n\n\n1. The authors propose a simple method that results in considerable performance gains.\n2. They provide valuations for 4 (1/2)  model families (Qwen2.5-Math, Qwen2.5, Qwen3, Llama3.1 IT, Skywork OR1-Math) at the small to medium scale (1.5B, 7/8B, 32B), and show interesting analyses.\n3. The writing is clear, and the takeaways are presented well. \n\nWeaknesses: \n\n\n\n1. The performance improvements are unclear, as the performances for the base models seem to be weaker than those reported in the original publications. For example, when comparing Table 18 in [1] and Table 1 in this work, Qwen 2.5 7B attains 77.6% on MATH-500 in [1] vs. 60.5% in this work. Similarly, the GPQA performances are 36.4% vs. 31.8%. Similar trends are observed for other models (e.g., for MATH 500, 54.8% vs. 48.6% for Llama3.1 8B-Instruct, Qwen3 8B 87.4 vs. 89.3). Therefore, it is unclear where the performance differences come from, and how large the gains of TTRL are.\n2. Connected to point 1, it is a bit unclear what effect TTRL has. The primary performance benefit of TTRL may come from learning to adhere to the restricted token budget and the output format. Moreover, there are differences in the token generation budget between [1] and this work. \n    * Therefore, within the restricted budget, the base models may not produce a valid answer in the right format. The majority vote may primarily help filter out too-long answers. Consequently, via TTRL, the model may only learn to adhere to the token budget. In contrast, the base models do not have this advantage and may run against budget limits, producing no valid answer. \n    * To reject this hypothesis, an ablation with a larger token generation budget for both the base model and the TTRL fine-tuned models would strengthen the paper. In addition, statistics on how often the base models run into generation budgets would be interesting.\n    * This point is very important because the claimed performance gains of TTRL (e.g., 211% on AIME 2024) are pretty large and would need to be revised if the base model performances change. \n3. It is stated that TTRL is very sensitive to hyper-parameters, but no in-depth ablations on this are presented. Also, it would be good to highlight the sensitivity of TTRL to hyperparameters in the main text.\n\n[1] Qwen3 Technical Report", "was_revised": true, "final_justification": "New experiments (in Weakness 2) on extended generation budgets strengthen the core message of the paper.", "rebuttal": "## Response to Weaknesses\n\n> ### Weakness1: Weak Baselines\n\nWe fully understand your concerns, as reliable baseline performance is essential for validating claims, and we would like to address them from the following perspectives:\n\n**1.1 Our evaluation strictly follows prior works.**\n\na. First, we would like to clarify that our evaluation strictly follows the evaluation setup used in prior studies without introducing any custom components, using only third-party or officially released tools.\n\n- Decoding: We follow R1 [1] with the temperature = 0.6, top-p = 0.95 to get avg@16 (with a 3k context length following the baseline Dr GRPO [2] for a fair comparison, which significantly affects performance).\n- Prompt Template: We use the official prompt template of the corresponding models via `apply_chat_template` of transformers, along with the formatting instructions specified in R1.\n- Others: Publicly available datasets, Qwen-Math Verifier, official verl code, etc.\n\n*Note: To the best of our knowledge, none of our configurations deviates from the accepted practices. Should the reviewer find any questionable practices in our evaluation practices, we strongly welcome suggestions and will make the necessary modifications and update the experiments.*\n\nb. We are not the only ones achieving such results, and our scores are not consistently lower than those of other studies as the reviewer has claimed. As shown in Table 1 of *Luffy* [3] and Table 5 (Default) of *Spurious Rewards* [4], other studies employing similar configurations have reported scores comparable to ours.\n\n**1.2 The performance gap is entirely due to differences in evaluation settings, especially for** **Base** **models.**\n\nFirst, the large performance gap introduced by prompts and evaluation settings is well studied and acknowledged [2, 4]. Our settings differ from Qwen3, so different results are expected. **Comparing scores under different settings is not meaningful, particularly for** **Base** **models.** By examining the evaluation settings in Table 18 of Qwen3 (as described in Section 4.6), we identify two main differences:\n\na. Context length: Qwen3 uses context lengths up to 32k, and as high as 38k for AIME2024, whereas our context length is only 3k. **This difference alone is sufficient to create a large performance gap, making the results non-comparable.**\n\nb. Decoding parameters: The following is a comparison table of decoding parameters, which shows a substantial discrepancy. We simply followed R1’s configuration without tuning these parameters:\n\n|                  | Qwen3 (Table 18) | TTRL (follow R1 settings) |\n| ---------------- | ---------------- | ------------------------- |\n| temprature       | 0.7              | 0.6                       |\n| top-p            | 0.8              | 0.95                      |\n| top-k            | 20               | -1                        |\n| presence penalty | 1.5              | 0.0                       |\n\nAdditionally, the prompt template may differ. Qwen3 did not specify the prompt used for non-Qwen3 models, so we cannot make a direct comparison. However, prompt differences can significantly affect results. Dr GRPO [2] shows that performance on MATH-500 can vary by up to **30 points** depending on the prompt. Spurious Rewards [4] even show that randomly generated prompts can sometimes yield the best scores. This does not imply that one should search for random prompts solely to maximize a baseline. We strictly followed the settings used in previous studies and never attempted to intentionally report lower baselines.\n\n**1.3 We have provided our code and are willing to assist with reproduction and verification of the above details.**\n\nWe have provided our code in Supplementary Material and are willing to open-source and maintain it to facilitate reproducibility. We strongly encourage you to run the code for reproduction and verification of the implementation details mentioned above. Moreover, our improvements are already substantial, and minor variations in the scores do not affect the overall conclusions. There is no incentive for us to misreport baseline results.\n\n> ### Weakness 2: Learning to Adapt to Limited Context and Output Format\n\nIn our initial implementation, we simply followed the 3k context used by Dr. GRPO as shown in Appendix E. To address the concern that TTRL may only be adapting to constrained token budget and formatting, we conducted three sets of experiments:\n\n**2.1 Larger token generation budget (on Base models)**:\n\nAs the most direct approach, we extended the context to 8k tokens on Qwen-2.5-Math-1.5B, using 32 rollouts per prompt for both label estimation and training (without downsampling). This setup also reduces the overall computation compared to the original paper’s configuration and matches the computational cost of standard RL.\n\nWe perform TTRL on AIME 2024 and evaluate its performance not only on AIME 2024 but also on unseen benchmarks AIME 2025, AMC 2023, and MATH-500 to assess generalization. The evaluation metrics include avg@32 and maj@32.\n\n| Metric | Model              | AIME 2024 | AIME 2025 (OOD) | AMC 2023 (OOD) | MATH-500 (OOD) |\n| ------ | ------------------ | --------- | --------------- | -------------- | -------------- |\n| avg@32 | Qwen-2.5-Math-1.5B | 7.7       | 3.5             | 29.3           | 31.7           |\n|        | w/ TTRL            | 17.9      | 6.5             | 42.5           | 63.7           |\n| maj@32 | Qwen-2.5-Math-1.5B | 16.4      | 6.8             | 36.9           | 37.7           |\n|        | w/ TTRL            | 27.3      | 12.4            | 55.8           | 76.2           |\n\n- **Sufficient token budget**: Throughout training, the token budget was sufficient (average maximum response length: 1968.11 tokens, maximum clip ratio: 1.9%), TTRL still yielded substantial performance improvements, indicating the gains are not due to optimizing the model’s output length.\n- **Generalization**: Applying TTRL in AIME 2024 led to improved performance on various unseen datasets, including the more challenging AIME 2025. If TTRL were merely optimizing for output length, its effects would likely be limited to prompts seen during training (i.e., AIME 2024) and would not generalize well to new data.\n- **Rising Maj@32**: We further analyzed the evolution of maj@32 performance. To ensure the validity of this analysis, we controlled for computational budget as mentioned above. The results show that maj@32 increases consistently during training, indicating that TTRL does not simply optimize superficial properties of the original model distribution, such as output length.\n\n**2.2 Larger token generation budget (on Reasoning models)**:\n\nThe base models discussed above have limited reasoning lengths and therefore do not fully utilize a larger context. To further evaluate the effectiveness of TTRL under a larger token generation budget, we conduct additional experiments on reasoning models (some results have already been provided in Table 2 of the paper). Specifically, we perform experiments using DeepSeek-R1-LLaMA-8B with a 32k context. The results demonstrate that even with a sufficiently large budget, such as 32k tokens, TTRL continues to deliver substantial performance improvements.\n\n|                      | AIME 2024 | AMC 2023 | MATH-500 |\n| -------------------- | --------- | -------- | -------- |\n| DeepSeek-R1-LLaMA-8B | 51.7      | 81.6     | 89.6     |\n| w/ TTRL              | 69.2      | 88.9     | 90.9     |\n|                      | +17.5     | +7.3     | +1.3     |\n\n**2.3 Output format (on Instruct models)**:\n\nIt is undeniable that base models often struggle with instruction following, and may benefit from improvements in output formatting during training. To further confirm that the performance gains from TTRL are not merely due to optimizing the output format, we conducted additional experiments using various Instruct models, including the DeepSeek, Mistral, and LLaMA series. These Instruct models have stronger instruction-following capabilities, allowing a more accurate assessment of the improvements introduced by TTRL. The results show that TTRL continues to yield substantial performance gains across all tested models:\n\n|                           | AIME 2024 | AMC 2023 | MATH-500 |\n| ------------------------- | --------- | -------- | -------- |\n| LLaMA-3.2-3B-Instruct     | 6.0       | 19.4     | 43.9     |\n| w/ TTRL                   | 13.3      | 31.3     | 61.6     |\n| Mistral-Nemo-Instruct     | 0.8       | 15.4     | 40.8     |\n| w/ TTRL                   | 0         | 24.8     | 51.0     |\n| DeepSeek-Math-7B-Instruct | 1.9       | 16.3     | 42.3     |\n| w/ TTRL                   | 2.5       | 22.9     | 52.4     |\n\n> ### Weakness 3: Sensitivity to Hyperparameters\n\nPlease see Response to Weakness 2 of Reviewer cYnp.\n\n\n\n## Response to Questions\n\n1. Please refer to Response to Weakness 1.\n2. Please refer to Response to Weakness 2.\n3. We have observed no hacking signs for most the majority of our experiments. Taking AIME 2024 as an example, we almost did not observe hacking behavior even after 80 training episodes (i.e., each sample was seen by the model 80 times). Therefore, we believe that the impact of this issue is currently manageable. Moreover, we consider that it can be addressed by monitoring and controlling the number of training steps. Intervening during training can help mitigate the problem, similar in spirit to techniques such as gradient clipping used in conventional deep learning.\n4. Please see Response to Weakness 2 of Reviewer cYnp.\n\n## References\n\n[1] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning\n\n[2] Understanding R1-Zero-Like Training: A Critical Perspective\n\n[3] Learning to Reason under Off-Policy Guidance\n\n[4] Spurious Rewards: Rethinking Training Signals in RLVR"}, {"review_id": "fbHgb0mYPH", "reviewer": "Reviewer_KyvW", "summary": "This paper introduces Test-Time Reinforcement Learning (TTRL), a framework for adapting Large Language Models (LLMs) during inference on unlabeled data. The key idea is to apply Reinforcement Learning (RL) using rewards derived from majority voting among model-generated outputs, even at test time, where ground-truth labels are absent. The authors argue that this leads to self-evolving models that generalize better to unseen data. Experimental results show significant performance gains when applying TTRL on various tasks and models.", "questions": "Hereafter come some questions which I would appreciate getting addressed:\n\n- Why should I consider Test-Time Reinforcement Learning (TTRL) a sound and scientifically valid machine learning approach?Specifically, how do you justify updating model weights during inference time, and in what way does this respect standard ML evaluation principles?\n\n- How do you ensure that training on test data—even without ground-truth labels—does not lead to overfitting or inflated results?\n\n- Could the proposed approach be reframed as offline adaptation or self-supervised training, avoiding the test-time terminology?\n\nI believe these questions are critical to the scientific validity of the proposed setting. While I appreciate the strong empirical results of the paper, I remain unconvinced that Test-Time Reinforcement Learning, as currently framed, constitutes a sound machine learning methodology. That said, I am open to reconsidering my evaluation if the authors can provide a clear and rigorous justification for the proposed setting and address the core methodological concerns.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n- **Creative use of majority voting**: The proposed use of consensus-based surrogate rewards is an interesting direction and aligns with recent trends in weak or self-supervised reward shaping.\n- **Strong empirical results**: The experiments show substantial improvements on reasoning tasks, suggesting the majority voting mechanism has a nontrivial signal even without labels.\n\n## Major Concerns\n\n### Methodological Validity of “Training at Test Time”\n\nThe core assumption of **Test-Time Reinforcement Learning (TTRL)** is fundamentally problematic. The paper proposes to **train a pre-trained model during inference** using rewards inferred via majority voting over generated outputs, therefore effectively updating model weights at test time. This goes against a foundational tenet of machine learning: **test time is reserved for inference**, not training. While test-time adaptation is a known concept in some domains (e.g., Test-Time Augmentation, few-shot prompting), **training** on the test set as explicitly described in Section 2.1, even with pseudo-labels, raises major scientific concerns:\n\n- The TTRL setting violates the standard separation between training and testing data. Despite the lack of ground-truth labels, updating weights using test inputs, regardless of how reward is obtained, constitutes **training on the test set**.\n- This blurs the line between generalization and memorization. Using test data to update the model even via a proxy reward, is still exploiting the test distribution, potentially inflating performance claims.\n- The term “Test-Time RL” is misleading and could give the impression of a valid ML paradigm.\n\nUnless the authors **clearly define this as a simulation scenario** (e.g., *what-if offline adaptation using test-like data*) or reframe the setting as an **unsupervised domain adaptation task**, the current formulation is **not scientifically grounded**.\n\n###  Reward Inference is Non-Causal and Optimistic\n\nThe rewards in TTRL are generated **after** the model samples outputs from itself and checks for majority agreement. This is inherently a non-causal, retrospective reward shaping method that I believe may introduce confirmation bias:\n\n- Majority voting across generations does not reliably capture correctness in tasks like reasoning or math where the majority could be confidently wrong.\n- Optimizing the model to maximize agreement among its own outputs risks reinforcing poor behavior especially early in training where the model is uncertain.\n- The reported performance boosts might be partially due to overfitting to the test distribution via pseudo-reward engineering.\n\n## Minor Issues\n\n- **Line 27**: “has recently” → “have recently”\n- **Figure 1**: The Caption could include more details about what is being visualized and its relevance.\n- **Mathematical notation**: Line 71: the symbol `\\pi` is introduced without definition\n- **Writing style**: The paper often reads more like a technical report than a formal scientific manuscript. Several sections lack formalism, and some claims (e.g., \"we fully advance AI evolution\") are overstated and should be toned down.", "was_revised": true, "final_justification": "See my comment about the clarification about Test-Time RL", "rebuttal": "## Response to Major Concerns\n\n> ### Weakness 1: Methodological Validity of “Training at Test Time”\n\nWe appreciate the reviewer’s thoughtful concern regarding the scientific framing of Test-Time Reinforcement Learning (TTRL) and the potential conflict with traditional machine learning principles. We would like to clarify that TTRL is best viewed as part of a well-established and growing body of research on **test-time adaptation** and **unsupervised test-time training**, which has gained substantial traction in recent years [1, 2, 3, 4].\n\nIn particular, prior work such as *Test-Time Training* [1] and *Meta Test-Time Training (MT3)* [5] have demonstrated that updating model parameters at test time using unsupervised or self-supervised objectives can significantly improve generalization under distribution shift. These methods operate under the same core idea as TTRL: leveraging unlabeled test inputs to adapt the model dynamically, without accessing ground-truth labels.\n\nFurther, more recent work such as *TeST* [6] and *BiTTA* [7] has applied similar paradigms across diverse domains, showing that model updates based on pseudo-rewards or binary correctness feedback at test time are not only scientifically grounded, but also practically effective. These approaches, like ours, adhere strictly to label-free adaptation settings on test data and do not violate the separation between training and ground-truth-labeled test data.\n\nThus, rather than violating core ML principles, our work extends this paradigm to **LLMs and RL**, combining self-supervised test-time adaptation with reward-driven optimization. **For clarity and to avoid confusion with conventional evaluation protocols, we will revise our terminology in the final version to emphasize the connection with unsupervised online adaptation or self-supervised test-time optimization.**\n\n[1] Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. ICML 2020 (1000+ citations)\n\n[2] Tent: Fully Test-time Adaptation by Entropy Minimization. ICLR 2021 Spotlight (1500+ citations)\n\n[3] A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts. IJCV 2024 (300+ citations)\n\n[4] Continual Test-Time Domain Adaptation. CVPR 2022 (600+ citations)\n\n[5] MT3: Meta Test-Time Training for Self-Supervised Test-Time Adaption.  ICAIS 2022 (100+ citations)\n\n[6] TeST: Test-time Self-Training under Distribution Shift. WACV 2023 (30+ citations)\n\n[7] Test-Time Adaptation with Binary Feedback. ICML 2025\n\n> ### Weakness 2: Reward Inference is Non-Causal and Optimistic\n\nThe reviewer's concern about confirmation bias introduced by Majority Voting (MV) is valid, errors in MV labels can propagate during training. However, in our method, this issue is significantly mitigated, and we believe it can be further alleviated. We address your concern from several perspectives:\n\n**2.1 Inherent Limitation of the Self-Training Paradigm**: The use of MV or self-consistency as supervision has become a general paradigm in LLM self-training with unlabeled data [1, 2, 3]. This stems from two key observations: (a) model consistency correlates with correctness, and (b) self-training setups exclude external information, leaving self-consistency as the only viable signal. Thus, confirmation bias is a shared limitation across current self-training approaches, with no fully satisfactory solution yet.\n\n**2.2 Reward Accuracy as a Diagnostic Tool:** We have proposed \"Reward Accuracy\" to quantify the quality of the training signal and assess the impact of MV errors. Our experiments across 13 models (LLaMA, DeepSeek, Qwen, Mistral series; 1.5B, 3B, 7B, 8B, 32B; Base, Instruct, Reasoning) show consistent gains with no collapse, even after up to 80 episodes training. High Reward Accuracy (with initial values surpassing 90%) across all experiments essentially and directly responds to the fact that the confirmation bias in TTRL introduced by MV's error is minimal. We will include these metrics in the next version.\n\n**2.3 TTRL Improves Supervision Quality and Reduces Confirmation Bias:** Unlike traditional self-training (e.g., LMSI [1]), which directly uses MV outputs for SFT, TTRL leverages MV labels in a RL manner, which is inherently more tolerant to label noise (see Section 4.2, Reward Calculations):\n   - **Improved Supervision Quality:** Label Accuracy and Reward Accuracy are metrics for evaluating the quality of supervision signals of traditional self-training methods and TTRL respectively. As shown in Figure 9, when Label Accuracy (maj@n) is only 37%, the corresponding Reward Accuracy reaches 92%, demonstrating the superiority of TTRL over previous methods. This implies that our method extracts much higher-quality learning signals even when the underlying MV label is weak, explaining why TTRL improves even low-performing models (even on model with 2% pass@1), as noted in Response to Weakness 2 of Reviewer 2bYH (last table).\n   - **RL’s Tolerance to Reward Inaccuracies:** As noted by Reviewer JAq8, RL relies on directional and relatively imprecise feedback, making RL inherently more robust to reward inaccuracies, a property that has also been observed in prior work [4]. For example, when the reward accuracy is 92%, the actual noise introduced into training might be even less than 8%.\n\n**2.4 TTRL Requires a Minimal Base Competence:** Self-improvement methods like TTRL does requires a base model with minimal capability. This is natural and expected. We may consider an extreme scenario: could a randomly initialized 7B model emerge intelligence or complex world knowledge purely through self-supervision? We believe that evolution through self-training builds upon the model’s pre-existing capabilities. Moreover, TTRL can drive substantial improvement even when the model achieves only a 2 points pass@1, as noted in our response to Weakness 2 of Reviewer 2bYH (last table). We consider this to represent a sufficiently weak model.\n\n**2.5 TTRL Generalizes Beyond the Training Distribution:** As shown in Response to Weakness 2 of Reviewer 2bYH (First Table) and Figure 3 in our paper, TTRL demonstrates strong generalization ability, rather than merely overfitting via pseudo-reward engineering.\n\nLastly, TTRL can be extended to further mitigate confirmation bias, for instance, by selectively using only negatively rewarded responses as it may be difficult to determine whether a response is correct, introducing randomized rewards (akin to dropout), or applying entropy-based regularization.\n\n[1] Large Language Models Can Self-Improve. EMNLP 2023 (600+ citations)\n\n[2] Recursive Introspection: Teaching LLM Agents How to Self-Improve. NeurIPS 2024\n\n[3] Importance Weighting Can Help Large Language Models Self-Improve. AAAI 2025\n\n[4] What makes a reward model a good teacher? an optimization perspective. Arxiv 2025\n\n## Response to Minor Issues\n\nThank you for carefully reviewing our manuscript and providing such detailed and thorough feedback! We will address these issues and carefully revise the entire paper to improve formalism and avoid any overstated claims.\n\n## Response to Questions\n\n> 1. Why should I consider Test-Time Reinforcement Learning (TTRL) a sound and scientifically valid machine learning approach?Specifically, how do you justify updating model weights during inference time, and in what way does this respect standard ML evaluation principles?\n\nRegarding the validity of the TTRL setting, we have already cited several highly influential papers above to explain that TTT is currently a widely used experimental setup, and many works have been published through peer review under this setting. Please further refer to Response to Methodological Validity of “Training at Test Time”.\n\n> 2. How do you ensure that training on test data—even without ground-truth labels—does not lead to overfitting or inflated results?\n\nAs for distribution overfitting, our paper includes comparisons where the model is trained on a single dataset (AIME/MATH/AMC) and tested on other datasets, demonstrating that TTRL possesses strong generalization ability. In addition, we have stated in the paper that our primary goal is to address self-/unsupervised RL problems. Test-time training is a specific scenario (which is also widely studied at present), but our method can also be adapted to conventional training settings, and recent work has provided validation for its effectiveness in these scenarios as well [1]. We will highlight and discuss this point in future revisions of the paper. TTT is merely a broadly accepted evaluation setting, while TTRL can be applied to all unsupervised RL scenarios.\n\n> 3. Could the proposed approach be reframed as offline adaptation or self-supervised training, avoiding the test-time terminology?\n\nWe will incorporate your suggestions to discuss the relationships among these concepts in the paper, and we will further clarify in the paper that our primary aim is to explore RL in LLMs under unsupervised settings, with the applications to tasks such as test-time training (TTT) and test-time adaptation (TTA) being only preliminary. We will also refine the relevant descriptions and terminology to avoid potential misunderstanding by readers.\n\n[1] Can Large Reasoning Models Self-Train?\n\n---\n\nWe hope these revisions will fully address your concerns. Our work essentially follows and builds upon existing tasks and previous works, and it is not our intention to propose a valid ML paradigm. We respectfully hope you could reconsider the score assigned to our paper."}]

Author rebuttal:

Meta-review: This paper introduces Test-Time Reinforcement Learning a method for improving language models on reasoning tasks using reinforcement learning with rewards generated from a majority vote of the model's own outputs. A simple technique allows self-improvement on unlabeled data, yielding strong performance gains.

The paper's main strength is its simple yet effective method, which is backed by strong empirical results on math reasoning tasks. While reviewers initially raised concerns about the methodology and evaluation, the authors' rebuttal provided new experiments that resolved these issues. Reviewers initially raised major concerns about the validity of test-time training ( KyvW) and the strength of the experimental baselines (2bYH). The authors delivered a convincing rebuttal, providing new experiments that clarified the method's context within test-time adaptation and confirmed the performance gains were not artifacts of the setup. The thorough response improved support for the paper with the primary remaining limitation as narrow focus on mathematical tasks, though this is not considered a considerable drawback as the narrow setting provides substantial value.
Decision: Accept (poster)