TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning
Authors: Shenzhi Yang
Guangcheng Zhu
Haobo Wang
Xing Zheng
Yingfan MA
Zhongqi Chen
Bowen Song
Weiqiang Wang
Junbo Zhao
Gang Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning with Verifiable Rewards
Semi-supervised Learning
Large Language Model
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Reinforcement learning with verifiable rewards (RLVR) has proven effective in training large reasoning models (LRMs) by leveraging answer-verifiable signals to guide policy optimization, which, however, suffers from high annotation costs. To alleviate this problem, recent work has explored unsupervised RLVR methods that derive rewards solely from the model’s internal consistency, such as through entropy and majority voting. While seemingly promising, these methods often suffer from model collapse in the later stages of training, which may arise from the reinforcement of incorrect reasoning patterns in the absence of external supervision. In this work, we investigate a novel semi-supervised RLVR paradigm that utilizes a small labeled set to **guide** RLVR training on unlabeled samples. Our key insight is that supervised rewards are essential for stabilizing consistency-based training on unlabeled samples, ensuring that only reasoning patterns verified on labeled instances are incorporated into RL training. Technically, we propose an effective policy optimization algorithm TraPO that filters out reliable unlabeled samples by matching their learning trajectory similarity to labeled ones. Building on this, TraPO achieves remarkable data efficiency and strong generalization on nine advanced benchmarks. With only 1K labeled and 3K unlabeled samples, TraPO reaches 42.6% average accuracy, surpassing the best unsupervised method trained on 45K unlabeled samples (38.3%). Notably, when using 4K labeled and 12K unlabeled samples, TraPO even **outperforms the fully supervised model** trained on the full 45K labeled samples on all benchmarks, while using only **10%** of the labeled data. The code is available via https://github.com/ShenzhiYang2000/TRAPO.

Published as a conference paper at ICLR 2026
TRAPO: A SEMI-SUPERVISED REINFORCEMENT LEARNING FRAMEWORK FOR BOOSTING LLM REA- SONING
Shenzhi Yang' Zhongqi Chen²
Guangcheng Zhu" Bowen Songet
Haobo Wang
Xing Zheng²
Yingfan Ma? Gang Chen
Weiqiang Wang
Junbo Zhao'
Zhejiang University Ant Group
Tyangshenzhi, zhuguangcheng, wanghaobo, j. zhao, eg wew.u.edu........ee.mm. on (feishang. zx, bowen. sbw, weiqiang wwa gantgroouu com
ABSTRACT
Reinforcement learning with verifiable rewards (RLVR) has proven effective in training large reasoning models (LRMs) by leveraging answer-verifiable signals to guide policy optimization, which, however, suffers from high annotation costs. To alleviate this problem, recent work has explored unsupervised RLVR meth- ods that derive rewards solely from the model's internal consistency, such as through entropy and majority voting. While seemingly promising, these meth- ods often suffer from model collapse in the later stages of training, which may arise from the reinforcement of incorrect reasoning patterns in the absence of ex- ternal supervision. In this work, we investigate novel semi-supervised RLVR paradigm that utilizes small labeled set to guide RLVR training on unlabeled samples. Our key insight is that supervised rewards essential for stabilizing consistency-base training on unlabeled samples, ensuring that only reasoning patterns verified on labeled instances are incorporated into RL training. Techni- cally, we propose an effective policy optimization algorithm TRAPO that iden- tifies reliable unlabeled samples by matching their learning trajectory similarity to labeled ones. Building on this, TRAPO achieves remarkable data efficiency and strong generalization nine advanced benchmarks. With only 1K labeled and 3K unlabeled samples, TRAPO reaches 42.6% average accuracy, surpassing the best unsupervised method trained on 45K unlabeled samples (38.3%). No- tably, when using 4K labeled and 12K unlabeled samples, TRAPO even outper- forms the fully supervised model trained on the full 45K labeled samples on all benchmarks, while using only 10% of the labeled data. The code is available via https://githuue Yotgee/e/eeeeeee.oooooooo//////
INTRODUCTION
The reinforcement learning with verifiable rewards (RLVR), pioneered by DeepSeek-R1 (Guo et al. 2025), has significantly advanced the development of large reasoning models (LRMs). In typical RLVR (Shao al. 2024| Liu et al. 2025 Yu et al. 2025 Zheng et al.) 2025). questions from training corpus are fed into an LRM, which then generates multiple reasoning paths (rollouts) per input. Rewards computed based verifiable rules: most commonly, whether the final answer in a response matches the ground-tree -ruth label. By leveraging such an answer-verifiable structure, RLVR enables reward assignment through group-based advantage estimation, guiding the model to explore reasoning paths that lead to the correct final answer.
However, when scaling to large corpora, the reliance of this reward paradigm on gold-standard labels incurs prohibitively high annotation costs, making it difficult to generalize to specialized domains where ground-truth answers are scarce expensive obtain, such medicine and finance (Wang et al. 2024b). To address this challenge, recent work has explored unsupervised RLVR methods
* 'Equal Contribution. t Correspondence to Haobo Wang (wanghaobo Branghaoho@rju.cdu..oo @xju..du.cc) and Bowen Song
(bowen.sbw @ antgroup.com)

---

Published as a conference paper at ICLR 2026
000 Libee)
38.7
22.4
Perfrmecc
17.4
000
18e
OmennDoss
Omsn-iratratt
TTAL
Self-Carssiity
Entroyst
Extrep--
Trato (oom)
1024
2045
4078
18154
Original Modes
Utsspenbeed
Semivapeeried
Sample (log-scale)
Figure I: Performance overview. (Left) TRAPO surpasses fully supervised RLVR (45K samples) using just 10% (4K) annotated data. (Right) TRAPO scaling law: performance improves consis- tently with increasing sample sizes and varying annotation ratios. We only show the changes with sample size at a 25% annotation rate the figure; for other specific results, please see Table
(Zhang al. 2025a Zhao et al. 2025 Agarwal al. 2025 Li Liet al. 2025a Zuo al. 2025 Zhang
et al. 2025a) that aim to climinate dependence on external supervision directly. These approaches are grounded in the observation that LRMs have already internalized substantial knowledge during pretraining ( Ye et al. 2025); thus, the goal shifts from learning factual correctness to eliciting latent reasoning capabilities through self- self-guided exploration. In this framework. rewards are computed based on intrinsic signals such as self-certainty (Zhao et al., 2025), entropy (Agarwal et al.) 2025), or majority voting (Zuo et al. 2025), to encourage high-contidence and consistent outputs. Despite their promise, these unsupervised methods often fail to capture valid reasoning patterns and tend to reinforce incorrect consensus, leading to severe performance degradation in late training. This drawback can be attributed to the absence of external ground truth: the reward signal becomes self- reinforcing and prone to reinforcing systematic biases, leading to a degenerate feedback loop.
Analogous to human learn- ing, unsupervised RLVR re- sembles student solving prob- lems based solely on current beliefs, treating the most con- fident answer as the ground truth. When incorrect, repeated reinforcement of the same rea- soning path entrenches errors, leading to failure both the current and related tasks. To break this vicious cycle, hu- mans typically learn from a few well-solved examples with veri- fied solutions to establish a cor-
Supervised RLVR
Superstond grvend
exeenal sxpervsiion
with large
cortectees
<.>>
Unsupervised RLVR
%6
Unsupervised Reward consencue entropy/ pnf-cerratity
Internal
Wissing
without
Semi supervised (Ours)
noternll eaparaidien
with fow gold stower
Ilybrid Reward groand trath arower reliable powude- aede-rrrrerrr
Figure 2: Different RLVR training paradigms.
rect conceptual foundation, then generalize via analogical reasoning. Therefore, we hypothesize that LRMs possess similar property: a small number of verifiable labeled samples can enable LRMs to generalize patterns from larger amounts of unlabeled corpora. Inspired by this process, we propose Semi-supervised RLVR (SS-RI (SS-RLVR) paradigm that takes advantage of small set of labeled exam- ples anchor the reward signal, guiding the model toward reliable reasoning patterns and allowing more robust seff-improvement.
Although promising in principle, our experiments show that simply combining supervised and un- supervised RLVR algorithms delivers only marginal benefits. For example, when combined with 3K entropy-based unlabeled RLVR training, the IK supervised baseline only improves 0.6% ac- curacy. We argue that such failure stems from the neglect of internal links between labeled and unlabeled sets. In other words, only those reasoning patterns that are verified on labeled instances should be incorporated into RL training, and labeled data should be used role models (Tarvainen & Valpolal 2017) to guide robust learning on unlabeled instances, as shown in Figure 2 Based on this key insight, we propose TRAPO (Trajectory-basee Policy Optimization), which measures the
2

---

Published as a conference paper at ICLR 2026
At the 2-th epoch;
Policy Model
oi
Labeled Sample
Group Computation
Reward Model
P
A
Parameter Sharing naaring
Update
ob
A/
Eiit
Trend Database
of
Extract Answer
A(
Majority Voting
Add or Update
Unlabeled Sample
Pseuda Reward Model
top-p TCS u ros
Policy Model
%
rd
P
Group Computation
Query
Figure 3: TRAPO is a semi-supervised RLVR training framework to dynamically select reliable unlabeled samples throughout the training process based on pass rate trajectory matching.
similarity between unlabeled and labeled samples in terms of their pass rate trajectories and uses this alignment as a criterion select unlabeled samples with reliable pseudo-supervision for train- ing. Experimental results demonstrate that TRAPO, trained with only IK labeled and 3K unlabeled samples, achieves 4.3% improvement in in-domain performance over the strongest unsupervised baseline (trained on 45K unlabeled samples), 2.6% over the best naive semi-supervised method, and 3.2% over the supervised baseline (trained on 1K labeled samples). Notably, with labeled and 12K unlabeled samples, TRAPO surpasses the fully supervised model trained on all 45K labeled samples across all benchmarks, using only 10% the labeled data (see Figure [ left). The scal- ing law for TRAPO (Figure right) further demonstrates that with increased data and labeling ratio (e.g, 25%), TRAPO achieves or approaches fully supervised performance without extra la- bels. These results strongly demonstrate TRAPO's ability to balance data efficiency and learning effectiveness.
RELATED WORK
Semi-supervised Learning leverages both labeled and unlabeled data to improve model perfor- mance, typically by exploiting data structure (Chapelle et al. 2009 Rasmus et al. 2015)) or consis- tency assumptions (Laine & Aila 2016 Berthelot et al. 2019)) Xie et al. 2020 Sohn et al. 2020). In traditional classification tasks, outputs drawn from shared discrete label space, enabling effec- tive label propagation via feature similarity. However, in RLVR, each input has an instance-specin specific solution space, where "correct" outputs vary significantly across examples. This makes direct align- ment of unlabeled samples with labeled ones through standard similarity-bas -based methods impractical, posing key challenge in bridging labeled and unlabeled data for RLVR. Thus, in this paper, turn from what the model learns to how it learns and employ the pass rate change trajectory as medium bridge the gap.
Unsupervised RLVR is built upon supervised RLVR, which has proven effective for aligning rea- soning models in domains with executable or exact feedback, such as math and code (Hu et al. 2025 Guo et al. 2025 Shao et al.) 2024), using deterministic, rule-based reward verifiers (Jacch al. 2024). However, its reliance on outcome supervision limits applicability tasks lacking clear ground truth. Recent work explores unsupervised RLVR, which uses intrinsic, self-generated
signals to enable reward-free training. Methods include self-rewarding via judgment prompting (Wul
et al. 2024: Yuan et al.) 2024| Xiong et al.) 2025) ensemble heads (Wang al. 2024c Zhou
et al. 2025 ). though often costly for online use. More scalable approaches leverage lightweight
signals- such entropy (Agarwal et al. 2025), self-confidence (Li al. 2025a), majority vot- ing (Zuo et al. (025) to guide online policy updates (Zhang et al.J 2025a) Zhao et al., (025).
However, purely unsupervised training risks model collapse due to biased noisy signals rein-
3

---

Published as a conference paper at ICLR 2026
forcing incorrect behaviors (Zhang et al. 2025c b). Our work builds on this line by introducing new semi-superrvin framework that anchors learning with labeled data to correct intrinsic signals, improving stability and generalization.
Reasoning Data Selection is a critical step in training LRMs, which can be broadly categorized into external and internal approaches. External methods rely on auxiliary resources such human anno- tations (Li et al. 2022), knowledge bases (guyyen et al. 2024), or proxy models (He et al. (025a) evaluate correctness and confidence, but suffer from limited applicability due dependency on external resources (Bi el al. 2025). In contrast, internal methods leverage model-internal signals, such as output probabilities (Plaut al., 2024), semantic entropy (Kuhn al., 2023). hidden repre- sentations (Wang et al. (2024a), or the changes in reward (Li et | (202bb) to estimate data quality in label-free manner. Nevertheless, such metrics do not reflect the fundamental characteristics of data that are most beneficial for model learning. In this work, we go beyond superficial indicators by probing the intrinsic learning dynamies of the data, thereby identifying unlabeled instances that genuinely contribute to effective and robust model training.
3 METHOD
In this section, we present our semi-supervised reinforcement learning paradigm, which uses lim- ited labeled data to guide reliable policy learning on large-scale unlabeled data. In Section 3.1 we discuss the limitations of supervised and unsupervised RLVR, and highlight the motivation for semi-supervised RLVR. Section 3.2 we explore the bridge between labeled and unlabeled data, propose vajectory-basee method to select reliable rewards and provide theoretical analysis on generalization.
3.1 SEMI-SUPERVISED REINFORCEMENT LEARNING WITH VERIFIABLE REWARDS
Supervised RIVR. In traditional RLVR, we assume access to large labeled dataset D, (gi, v:)) where each sample consists of question q and its corresponding verifiable ground- truth answer V4- For each question Gi we input into a policy model my to generate G candidate outputs, denoted as { Given the ground-truth answer as a supervision, we assign rewards the generated responses based on whether they derive the correct answer. Specifically, we define a binary reward function that evaluates the final extracted answer from each output T/:
(1
if ag = yi+ otherwise.
R(T v.) T(?
Here, of extract(7?)) denotes the answer extracted from the generated response T/. such as the content within boxed delimiters (e.g., \boxed(-)). With the ground-truth answers y/s serving as explicit guidance signals, this Supervised RLVR paradigm reinforces only the responses that yield the correct answers; the policy model Te is gradually steered toward discovering valid and consistent reasoning paths, thereby enabling stable and scalable policy optimization.
Unsupervised RIVR. Although supervised RLVR has achieved great success, its reliance on golden answers y/ incurs high annotation costs. To address this, the community has explored un- supervised RLVR techniques that rely solely on unlabeled data D. = N. Under this setting, the absence of golden answers necessitates the use of proxy rewards R., (T:") that estimate R(T;' U:) based on the model's confidence or consensus (.) A widely adopted method is majority vot- ing, where the reward is defined as:
RA (T/) = conf( q.) = I(a = MAJ(a,, a² .a......"))) .a())
(2)
where MAJ(.) denotes the pseudo-label ÿ obtained by majority answer among G rollouts. This approach effectively treats the most frequently generated answer as the pseudo-label, providing form of self-supervised signal. Beyond majority voting, Zhao et al. 2025)) use self-certainty, Agarwal et al.] (2025) use token-level or sequence-level entropy as a proxy for confidence, and compute rewards accordingly. Fundamentally, these methods are based on a key assumption: higher confidence implies a greater probability of producing the correct answer, and thus the higher the reward it should receive.
4

---

Published as a conference paper at ICLR 2026
However, this assumption breaks down when the proxy reward diverges from actual correctness. Take the majority voting as an example, if the majority answer is not the correct answer, i.e,, reinforced. This creates dangerous feed- MAJ(a.. "a a # y:, then the incorrect responses back loop: the policy becomes more confident in the wrong answer, leading to even stronger wrong consensus in subsequent iterations. Over time, the model converges to state where it confidently produces incorrect outputs.
To break this vicious loop induced by the absence of grounded feedback, we hypothesize that we must introduce labeled examples to anchor the reward to ground truth. Formally, we adopt hybrid reward function that computes rewards differently for labeled and unlabeled data:
Semi-supervised RLVR.
[R(T) wi), if (qi Wi) E Di, (T!), if E Du.
Rsemi (T,)
(3)
Here, labeled data are used to compute rewards under supervision from the ground-truth labels Vi. while unlabeled data can adopt any seff-consistency-banee reward have stated previously. Since the reward R(7 w/) of labeled data is independent of the model's consensus, this training paradigm introduces a crucial distinction between correctness (alignment with ground truth) and self-consistency (internal agreement among outputs), thereby preventing the policy from reinforcing incorrect but internally consistent outputs.
The design of our Semi-supervised RLVR framework stems from the inherent trade-off between data efficiency and learning effectiveness. Compared to unsupervised variants, SS-RLVR effec- tively guides robust learning on unlabeled instances by using labeled data as reliable anchor. In contrast to fully supervised approaches, it significantly reduces the need for costly annotation-our experiments show that SS-RLVR achieves performance close to supervised learning using only % of the labeled data. In practice, this trade-off not only directly reduces the annotation burden, but also enables high-quality data synthesis within iterative refinement pipelines, thereby improving data quality over time. This makes SS-RLVR particularly attractive for domains where labeled data is scarce or expensive to obtain, such as medicine and finance.
3.2 PROGRESSIVE TRAJECTORY GUIDANCE FOR BRIDGING LABELED AND UNLABELED DATA
Despite its promise, we show that trivial baseline that simply combines supervised and unsuper- vised RLVR algorithms delivers only marginal benefits. For example, when supplemented with 3K entropy-based unlabeled RLVR training, the IK supervised bascline achieves merely 0.6% ac- curacy improvement. This suggests that such a naive strategy remains constrained by the internal signals of LRMs and suffers from the internal ungrounded reasoning patterns. Thus, SS-RLVR must move beyond shallow integration and instead uncover the deeper intrinsic relationships between la- beled and unlabeled data. In partieular, the key to exploit those reasoning patterns in unlabeled data that can be externally validated by labeled examples. To achieve this goal, it is required to iden- tify a shared, meaningful signal that transcends the heterogeneity of solution spaces and reliably reflects the model's 's ability to transfer knowledge from labeled to unlabeled data.
In this work, we propose TRAPO (Trajectory-
Policy Optimization), which leverages the learning dynamics of LRMs across training steps as a proxy to connect labeled and unlabeled data, as shown in F Specifically, at each step t, TRAPO computes the pass rate for each training point. We then identify those unlabeled samples whose pass rate trajectories closely align with those of labeled samples as reliable data, which means that their reasoning patterns can be externally validated by the labeled set. In other words, we hypothesize that when unlabeled sample is well-learned, its pass rate trajectory should exhibit trends consistent with those observed in labeled data. Naturally, since pass rates cannot be directly computed for unlabeled data, we introduce pseudo-pass rate approximation to serve as a proxy. Formally, for a question q at epoch 1, the (pseudo) pass rate is defined as the fraction of generated responses that satisfy the expected answer criteria:
1 I(a!0) = )) NEDw. G
p(t) =
(4)
I(a 4), q Dr. 1
5

---

Published as a conference paper at ICLR 2026
Then, define the pass rate trajectory of question the sequence of its pass rates across training epochs:
T p(() . p(2) ........ p(*)) E [0, 1]".
(5)
initialized T and updated iteratively via concatenation: T T denotes sequence concatenation. We maintain reliable pass rate database Dellialee initialized with all labeled sample trajectories:
P( where o
D Peliiable {T, E Dr}- Reliably pseudo-labeled trajectories from unlabeled data selected in subsequent steps are added to update this database. The average trajectory of this database, (
reliable [Pasatel EreP,man T, serves trusted reference for assessing the reliability of unlabeled samples based on trajectory alignment. Then we compute a trajectory-basee cosine similarity (TCS) as:
TC T(O Tellabbe) = T()) T())
Σ
p()) 500
(6)
reliable
reliable
==1
where ((1)
p(3)
and
p(5 reliable
pla)
are the normalized pass rate of the
(
( ml
unlabeled sample and the reliable database, respectively.
To select the reliable trajectories, we combine two criteria: the top-p of unlabeled samples with highest trajectory similarity to the labeled data, and any sample whose similarity exceeds a threshold
M(u) (u top-p (TCS(T,.. Traliass)) v (TCS T 2 I')
With this selection mask in hand, we now integrate it into the training process to ensure only reliably improving samples influence model updates. To ensure stability, we employ warm-up phase using only labeled data for updates, while accumulating unlabeled trajectories. After warm-up, we apply the mask M include only reliable unlabeled samples:
[(0)
labelled (0) + M
JGPP
Junlabeled (8).
where @ denotes the dot product of vectors. Here, JeRpo is the GRPO objective (Shao et al. 2024
(0)
CLIP(7H10);; A,,e) - 3 DKa.[Tal mer
Σi
Ta
where (0, = no(T.,1g. Ti, Fo. (T. Ti,(1) is the importance sampling term, and CLIP(?,A,E) min(r- A. clip(1;; e, +e) e) A] is the clipped surrogate objective.
summary, we propose leveraging the evolution of correctness during training (puss rate trajec- tories) as reliable signal for evaluating unlabeled samples. By measuring the similarity between the pass rate trajectory of an unlabeled instance and the average trajectory derived from labeled data, we identify samples whose learning dynamics align closely with those observed under trusted supervision. To validate the effectiveness TRAPO in selecting high-quality unlabeled samples and grounding unsupervised learning within stable feedback framework, provide theoretical analysis of its generalization error bound:
Theorem 3.1 Srajicttor-CCossistee Generalization). (Informal) Let the generalization error of policy To be the expected risk on the true distribution. Assuming Ly is the label space diameter, under the TRAPO framework, with probability at least 8, this error is bounded by:
T(2) ] Iny (1- X' +a EygaD. [1 (C)( (TT (t) RD, (T
In In (2n/6)
2G
(10)
where Ro, (z) (1) is the empirical risk on D,, X x 0 bounds the domain shift between Dj and D.,. and (1)) is the average voting confidence across n samples based on G votes.

---

Published as a conference paper at ICLR 2026
Table 1: Overall performance based on Owen2.5-Mathh under three different training paradigms. Bold and underline indicate the best and second-best results, respectively.
In-Distribution Performance
Out-of-Distribution Performance
Model
AIME 24/25 AMC MATH-500 Minerva Olympiad Avg.
ARC-e GPQA'
MMLU-Pro Avg.
Original Models
Qwen-Basc
11,5/4,9
31.3
43.6
7.4
15.6
19.0
18.2
11.1
16.9
15.4
Qwen Instruct
12.5/10.2
48.5
32.7
37.6
43.0
Unsupervised Methods Trained on 45K Samples w/n Any I Labels
TTRL
14.1/12.7
51.5
76.6
33.8
38.2
80.5
35.4
41.3
52.4
Self-certainty
16,9/10.2
51.7
77.6
34.9
38.8
38.3
72.9
30.8
41.4
48.4
Token-level Entropy
15,0/9.9
50,3
75.2
36.8
38.4
37.6
75.6
33.3
40,9
49.9
Sentence- level Entropy
11.4/10.7
32.7
32.6
51.5
Semi-superviscd Methods Trained on 1K Labeled Samples & 3K Unlabeled Samples
Fully Supervised w/ IK Labels
14.2/13.5
52.6
80.2
34.9
39.4
76.2
43.6
52.1
TTRI.
14.9/10.7
55.3
77.8
33.1
43.6
39.2
72.6
35.4
42,7
50.2
Self-certainty
16.5/11.4
55.6
79.8
35.3
41.2
40.0
64.8
30.3
41.6
45.6
Token-level Entropy
18.2/11.9
53.4
34.6
40.0
72.9
49.7
Sentence-level Entropy
15.4/11.5
54.9
79.4
36.0
41.2
39.7
79.4
33.8
44.5
52.6
TRAPO (ours)
17.9/13.8
58.7
81.4
38.2
45.5
42.6
83.7
37.9
46.8
56.1
Fully Supervised w/ 4K Labels
19.6/14.8
57.9
80.6
39.3
46.5
43.1
82.1
39.9
48.2
56.7
TRAPO Trained on 4K Labeled Samples & /2K Unlabeled Samples
TRAPO (ours)
24.3/17.1
60.0
84.6
39.3
48.3
45.6
84.6
43.9
50.7
59.7
Fully Supervised w/ 45K Labels
25.1/15.3
62.0
84.4
39.3
46.8
45.5
82.3
40.4
49.3
57.3
Theorem 3.1 highlights the role of trajectory consistency as a regularizer cuni-supervised policy learning. Specifically, the term ED 1 TCs (T T(())
o' relliable) encourages unlabeled samples to follow learning dynamics similar to those of labeled data, effectively anchoring the optimization path. The dependence on (+) reflects the model's 's self-confidence during training, with lower confi- dence leading to a looser bound. thus promoting cautious updates. The formal theorem and its proof
are presented in Appendix B.13
4 EXPERIMENT
This section reports the main experimental results. Appendix F. compares more fully supervised baselines; E.2 further validates TRAPO more models; shows that TRAPO is plug-and-play;
E.4 evaluattss TRAPO on the DeepMath dataset; E.5 compares TRAPO with other selection strate-
gies: cofffrms TRAPO's stability; E.:iitt::E77aallll analyzes the temporal efficiency of TRAPO: E.8 exaaminss different strategies for selecting utilization trajectories.
4.1 SETUP
Dataset and Benchmarks. We follow prior work Yan et al. (2025) and use the widely used math reasoning dataset OpenR1 -Math-2220 (Face 2025) for training. For evaluation,
focus six in-distribution (ID) math reasoning benchmarks: AIME 2024, AIME 2025, AMC (Li et al. 2024). Minerva (Lekkowyzz et al. 2022). , OlynpiadBench (He et al. 2024). and MATH-500 500 (Hendrycks) et al. 2001). We report e 32 on AIME 2024/2025 and AMC (due to small test sets) and pass@ @ on the others. For out-of-distributioo (OOD) generalization, we evaluate ARC-c (Clark al.) 2018), GPQA-diamond (Rein et al. 2024) (GPQA*), and MMLU-Pro (Wang et al. 2024b), cov- ering open-domain reasoning, graduate-level science, and academic reasoning. All evaluations temperature sampling with 0.6.
Baseline Methods. We evaluate supervised, unsupervised, and semi-supervised RLVR methods across varying data scales. For supervised training, apply GRPO on IK, 4K, and 45K labeled samples. In the unsupervised setting, we remove ground-truth labels from the full 45K dataset and evaluate four approaches: (1) TTRL (uuo al. 2025), which majority-voted outputs

---

Published as a conference paper at ICLR 2026
Table 2: Performance of different training paradigms with IK labeled math (ID) samples and 1K unlabeled non-math (OOD) samples. Bold and underline indicate the best and second-best results, respectively.
In-Distribution Performance
Out-of-Distriuation Performance
Model
AIME 24/25 AMC MATH-S00 Minerva Olympiad Avg.
ARC-c GPQA' MMLU-Pro Avg.
Original Model
Qwen-Buse
11,5/4,9
31.3
43.6
7.4
15.6
19.0
18.2
11.1
16.9
15.4
Qwen-Instruet
12.5/10.2
48.5
80.4
32.7
41.0
37.6
70.3
24.7
34.1
43.0
Unsupervised Methods Trained on /K Unlabeled ID Samples & IK Unlabeled OOD Samples
TTRL
13.3/9,4
48.2
72.2
27.6
34.8
34.3
76.7
33.8
36.2
48.9
Self certacett
18.5/9.6
53.4
79.6
33.4
40.4
39.2
76.7
37.9
45.6
53.4
Token-level Entropy
14.6/13.3
46.8
77.6
27.9
40.1
36,7
74.5
36.4
35.8
48,9
Sentence-level Entropy
16.4/11.5
51.8
74.0
33.5
37.2
37.4
74.5
34.8
43.3
50,9
Semi-supervised Methods Trained on IK Labeled ID Samples & JK Unlabeled OOD Samples
TTRL
16.4/13.6
49.9
66.9
26.5
37.8
35.2
62.0
31.8
43.5
45.8
Self celf-certainty
16.0/10.9
53.0
78.4
34.2
39.0
38.6
77.1
32.8
45.7
51.9
Token Leveel Entropy
17.7/110
38.6
76.5
30.8
44.7
Sentence-level Entropy
15.7/10.0
51.4
77.4
34.9
37.5
37.8
75.1
31.3
44,3
50.2
TRAPO (ours)
18.5/15.7
53,4
80.4
33.8
44.0
41.0
83.6
38.9
48.1
56.9
Fully Supervised w/ 2K Labels
17.3/12.4
56.8
81.4
38.6
44.8
41.9
82.0
38.9
52.4
57.8
as pseudo-labels; (2) Self-Certainty (Zhao et al. 2025). which maximizes KL divergence to en- courage confident predictions; (3) Token-Level Entropy (Agarwal et al., 2025), which minimizes token-level entropy for consistency; and (4) Sentence-Level Entropy (Agarwal et al. (025). which maximizes sentence likelihood. For semi-supervised training, we use 1K labeled and 3K unlabeled samples, applying GRPO on the labeled subset and each unsupervised method on the unlabeled subset to form hybrid baselines. We further evaluate stronger setting with 4K labeled and 12K unlabeled samples to assess performance under higher label efficiency.
In E. we compare with
more supervised baselines (Zegg al. 20255
Hu et al. 2025
Cui al. 2025
Liu al. 2025).
4.2 EXPERIMENTAL RESULTS
TRAPO achieves SOTA performance. Our main results are summarized in Table T First, TRAPO significantly outperforms all fully unsupervised baselines using only 1K labeled samples (with 3K unlabeled). Compared
the best unsupervised method trained on the full 45K unlabeled set, TRAPO achieves gains of 4.3% in ID and 3.7% in OOD accuracy, demonstrating that even mini- mal labeled data can lead to substantial improvements when effectively integrated. Second, TRAPO outperforms naive remi-supervised approaches that treat labeled and unlabeled data independently, improving the strongest such baseline by 2.6% (ID) and 3.5% (OOD), which underscores the im- portance of using labels to actively guide the learning from unlabeled examples. Finally, TRAPO surpasses the fully supervised model trained on the same 1K labels by 3.2% (ID) and 4.0% (OOD). matches the performance of a fully supervised model trained on 4K labels while using only 25% of the labeled data. Notably, when trained with 4K labeled and 12K unlabeled samples, TRAPO achieves 45.6 ID and 59.7 OOD accuracy, exceeding the fully supervised model trained on all 45K labels by 0.1% (ID) and 2.4% (OOD), despite using only 10% of the total labels. This remarkable performance highlights TRAPO's 's superior data efficiency and generalization capability.
TRAPO succeeds with OOD unlabeled data. To investigate whether labeled data can guide learn- ing on out-of-domain (OOD) unlabeled data, we evaluate semi-supervised setup with 1K labeled samples from the mathematics domain (ID) and 1K unlabeled samples from non-mathematical do- mains (OOD). This cross-domain setting is challenging due to the limited transfer of reasoning patterns across domains. As shown in Table 2 naive semi-supervised methods fail to benefit from labeled data well. For instance, self-certainty drops by 0.6% on ID and 1.5% on OOD, indicating that naive integration of labeled and unlabeled data harms learning under domain shift. In trast, TRAPO achieves significant improvements, outperforming the best unsupervised baseline by 1.8% ID and 3.5% OOD. It also closely matches the fully supervised model with 2K labels,
8

---

Published as a conference paper at ICLR 2026
The Trend of Mean Pass Rate Changes Ownit 1804 Loaner)
Tread Mean PasS Rate Changes
Trend Mean Pass Rate Changes
50%
70%
97%
30%
Figure 4: Left: Average performance changes on labeled and unlabeled data. Center: Unlabeled data performance vs. trajectory matching score using true training dynamics unlabeled data. Right: Unlabeled data performance vs. trajectory matching score using pseudo training dynamics on unlabeled data.
Are
Figure 5: Sensitivity Analysis. The left three plots show sensitivity analyses of top-p. r, and warmup epochs (Tables 10 and [ in the Appendix). The right two plots compare performance for dif- ferent ratios of selected and available unlabeled samples (3K x aMr). See tables 14 and 15 in the Appendix for details.
trailing by only 0.9% on both metrics. The substantial gain in OOD performance demonstrates that TRAPO enables robust cross-domain generalization, highlighting its strong ability to transfer reasoning knowledge even under domain discrepancy.
Effectiveness of trajectory matching. To evaluate whether trajectory matching iden- tifies reliable unlabeled examples, we ana- lyze the link between trajectory similarity and performance.
Trand Accuracy Changes
Supersieed
Semi
shown in the mid- dle plot of Figure samples with dynam- ies more aligned to labeled data achieve much higher performance. The top 10% of samples outperform the bottom 10% by
Acosscy
over 40%, confirming that alignment cor-
relates with reliability. In practice, we use pseudo-labels from voting estimate labeled sample dynamics. The right plot of Figure 4 shows that matching pseudo dynamics to true labeled dynamics still yields strong positive correlation with fi- nal performance. This validates the practi- cal utility of TRAPO.
Step 99
Figure 6: Performance comparison on Llama-3, 3.--B..
Sensitivity analysis. We systematically
analyze the impact of top-p, т, and warm-up length with the Qwen-2.5-7B model using 1K la- beled and 3K unlabeled samples (left three plots in Figure For top-p, larger values lead to noisy early-stage predictions and unreliable pseudo-labels, degrading overall performance. For I, setting it too low admits too many low-quality unlabeled samples, while setting it too high is overly con- servative, leading to underutilization; both extremes harm the model. Short warm-up lengths lead to unstable lseudo-labeling, but performance stabilizes as the warm-up lengthens. With different selection ratios and varying proportions (αM) available unlabeled samples, TRAPO outperforms random selection and strong token-level entropy baseline (the right plots in Figure 5). We find

---

Published as a conference paper at ICLR 2026
that TRAPO achieves optimal results using the top 30% of unlabeled samples, whereas adding more unlabeled samples increases noise and reduces gains. These experiments highlight the critical role of intelligent denoising and selection strategies.
Experiments with other LLMs. Besides Qwen, we also compare the training effectiveness of the three paradigms using the Llama-3. 1-8--3..---B---- 1-8B-Innttruc model. The model performance during training is shown in Figure 6 and detailed results are presented in Table Here, our semi-supervised TRAPO method exhibits similar trend supervised training and maintains consistent improvement. In con- trast, unsupervised training leads to rapid performance collapse within tens of training steps. This underscores the critical importance of effective pseudo-supervision selection via trajectory matching in stabilizing the training process.
CONCLUSION
In this paper, present the first exploration of semi-supervised learning in the RLVR setting. We in- troduce novel paradigm that leverages a small set labeled data to guide robust self-improvement on unlabeled data. We propose TRAPO (Trajectory based Policy Optimization), method that enables reliable pseudo-supervision by aligning the learning dynamics of labeled and unlabeled samples through trajectory similarity in pass rate progression. Results show TRAPO significantly outperforms various baselines using only a fraction of labeled data, achieving an exceptional balance between efficiency and effectiveness.
ACKNOWLEDGEMENT
This paper was supported by the NSFC under Grants (No. 62402424) and Ant Group through CCF- Ant Research Fund (No. CCF-AFSG RF20240406). This paper was also supported by Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China.
7 REPRODUCIBILITY STATEMENT
We committed to ensuring the reproducibility of our work. To this end, will open-source code, model weights, and processed datasets upon paper acceptance. The codebase will include detailed documentation and training scripts to reproduce experimental results reported in the paper.
BIBLIOGRAPHY
Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. The unreasonable effec- tiveness of entropy minimization in I1m reasoning. arXiv preprint 1rXi::2505.15134 2025.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in neural information processing systems, 32, 2019.
Shai Den-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory learning from different domains. Machine learning, 19(1)1151-175 2010.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural informa- tion processing systems, 32, 2019.
Jinhe Bi, Danqi Yan, Yifan Wang. Wenke Huang. Haokun Chen, Guancheng Wan, Mang Ye, Xun Xiao, Hinrich Schuetze, Volker Tresp, et al. Cot-kinetics: A theoretical modeling assessing Irm reasoning process. arXiv preprint arXiv:2505. /3408, 2025.
Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Pro- ceedings of the eleventh annual conference on Computational learning theory, pp. 92-100, 1998.
10

---

Published as a conference paper at ICLR 2026
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning (chapelle, et al., eds.; 2b0o)|book reviews]. IEEE Transactions on Neural Networks, 20(3):542-544 2009.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. TrXXi::803..0557711 2018.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456 2025.
Miroslav Dudík, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. arXiv preprint arXiv: 103.4601, 2011.
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https: huggi com/huggiitg
//github. com
fac.ae////ccoocoooooooooo open-rl
Chelsea Finn, Tianhe Yu, Justin Fu, Pieter Abbeel, and Sergey Levine. Generalizing skills with semi-supervised reinforcement learning. arXiv preprint arXiv:/612.0044 16/2.00499, 2016.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang. et al. Olympiadbench: challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 3828-3850, 2024.
Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang. Xingyuan Bu, Ge Zhang, Zhongyuan Peng, Zhaoxiang Zhang. Zhicheng Zheng, Wenbo Su, et al. Can large language models detect errors in long chain-of-thowee of-thought reasoning? arXiv preprint arXiv:2502. /9361,, 2025a
Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath-103k: A large-scale, challenging, de- contaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint 3xXiv:2504.11550 2025b.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint ar arXiv:2103.038/7 2021.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. zere: An open source approach to scaling up reinforcement learning on the base model, 2025. URL
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemel: Convergence and gen- cralization in neural networks. Advances in neural information processing systems, 31, 2018.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.1677 2024.
Nathan Kallus and Masatoshi Uchara. Double reinforcement learning for efficient off-po evalu- ation in markov decision processes. Journal of Machine Learning Research, 21(167):1-63, 2020.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664 2023.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint 16Xiv:1610.0223 02242,, 2016.

---

Published as a conference paper at ICLR 2026
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Soke Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q. Jiang. Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. https: hww.aaatttttttaataaaaaaaaaaeaeee Numinamath 2024. Hugging Face repository, 13:9.
Pengyi Li, Matvey Skripkin, Alexander Zubrey, Andrey Kuznetsov, and Ivan Oseledets. Confidence is all you need: Few-shot rl fine-tuning of language models. arXiv preprint arXiv:2506. 06395, 2025a.
Xuefeng Li, Haoyang Zou, and Pengfei Liu. arXiv:2502.1188% 2025b.
Limr: Less is more for rl scaling, arXiv preprint
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making large language models better reasoners with step-aware verifier. arXiv preprint ar arXiv:2066002366, 2022.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.2078, 2025.
Minh- Minh-Vuong Nguyen. Linhao Luo, Fatemeh Shiri, Dinh Phung, Yuan-Fang Li, Thuy-Trang Vu, and Gholamreza Haffari. Direct evaluation of chain-of-thoughh in multi-hop reasoning with knowl- edge graphs. arXiv preprint arXiv:2402. 1199, 2024.
Benjamin Plaut, Nguyen X Khanh, and Tu Trinh. Probabilities of chat Ilms are miscalibrated but still predict correctness on multiple-choice q&a. arXiv preprint 1txxxv:2402.32//.. 2024.
Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi- supervised learning with ladder networks. Advances in neural information processing systems, 28, 2015.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URU/https://operrviiie net forum Pefformmiii-o.oo iid--d--- 1d-T167584b33
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning open language models, 2024. httpstts:://et
org/ aass/2083
03300
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608, 2020.
Amarnag Subramanya and Jeff Bilmes. Semi-supervised learning with measure propagation. Jour- nal of Machine Learning Research, 12(11), 2011.
Kai Tang, Junbo Zhao, Xiao Ding, Runze Wu, Lei Feng, Gang Chen, and Haobo Wang. Learning geometry-awan aware representations for new intent discovery. In Annual Meeting of the Association for Computational Linguistics (ACL-Main), 2024.
Kai Tang, Rui Wang, Renyu Zhu, Minmin Lin, Xiao Ding. Tangjie Lv, Changjie Fan, Runze Wu, and Haobo Wang. Towards transferable personality representation learning based on triplet com- parisons and its applications. In The Conference on Empirical Methods in Natural Language Processing (EMNL.P-Maine) 2025.
12

---

Published as a conference paper at ICLR 2026
Kai Tang, Yixuan Tang, Tianyi Chen, Haokai Xu, Qiqi Luo, Jin Guang Zheng, Zhixin Zhang, Gang Chen, and Haobo Wang. Group-awar multiscale ensemble learning for test-time multimodal sentiment analysis. In Proceedings of the AAAI Conference Artificial Intelligence (AAAI), 2026.
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-averaged sistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
Meta Team. The laaaa 3 herd of models, 2024. URL https://aaxxww.or//bbs//// 217883
Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In International conference on machine learning, pp. 2139-2148. PMLR, 2016.
Yiming Wang, Pei Zhang, Baosong Yang, Derek Wong, and Rui Wang. Latent space chain-of- embedding enables output-free IIm self-evaluation. arXiv preprint arXiv:2410. 13640, 2024a.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.011544 2024b.
Zhaoyang Wang, Weilei He, Zhiyuan Liang, Xuchao Zhang, Chetan Bansal, Ying Wei, Weitong Zhang, and Huaxiu Yao. Cream: Consistency regularized self-rewarding language models. arXiv 2024c.
preprint
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with ain-as-a-meta--udee arXiv preprint arXiv:2407. 149.299977.9944 2024.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances in neural information processing systems, 33:6256-6266 2020.
Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, and Tong Zhang. Self-rewarding correction for mathematical reasoning. arXiv preprint 1rXiv:2502.19613 2025.
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off- off-colicy guidance. arXiv preprint arXiv:2504. /4945, 2025.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 0wen2.5-mathh technical report: Toward mathematical ex- pert model via seff-improvement, 2024. URL pettps://arxiv.org. haagg//bs/2240 2409. 2440.12122.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint 25X3::2503.14777 2025.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXhv2240.110220.3 2024.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- Investigating and taming zero reinforcement learning for open base models in the wild, 2025a. URL httpst:::: / arxiv. org absp//bb//25333 18892
Weihao Zeng, Yuzhen Huang, Wei Liu, Keging He, Qian Liu. Zejun Ma, and Junxian He. 7 model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient. httts::/hkkuse -nl.. not ion. site/ imp reason] 2025b. Notion Blog.
13

---

Published as a conference paper at ICLR 2026
Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised Ilm reasoning incentivization. arXiv:2504.05882. 2025a.
arXiv preprint
Yanzhi Zhang. Zhaoxi Zhang, Haoxiang Guan, Yilin Cheng, Yitong Duan, Chen Wang, Yue Wang, Shuxin Zheng, and Jiyan He. No free lunch: Rethinking internal feedback for Il reasoning. arXiv preprint wrxiv2256..72111 2025b.
Zizhuo Zhang, Jianing Zhu, Xinmu Ge, Zihua Zhao, Zhanke Zhou, Xuan Li, Xiao Feng, Jiangchao Yao, and Bo Han. Co-reward: Self-supervised reinforcement learning for large language model reasoning via contrastive agreement. arXiv preprint arXiv:2508. 00410, 2025c.
Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards. arXiv preprint (xXXi:25005119900 2025.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao. Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.1800 2025.
Jingho Zhou, Yixuan Du, Ruqiong Zhang, Jun Xia, Zhizhi Yu, Zelin Zang, Di Jin, Carl Yang, Rui Zhang, and Stan Z Li. Deep graph neural networks via posteriori-sampling-- based node-adaptative residual module. Advances in Neural Information Processing Systems, 37:68211-68238, 2024.
Jingbo Zhou, Jun Xia, Siyuan Li, Yunfan Liu, Wenjun Wang, Yufei Huang, Changxi Chi, Mutian Hong, Zhuoli Ouyang, Shu Wang, et al. Vecformer: Towards efficient and generalizable graph transformer with graph token attention. arXiv preprint 29t::2602.19622,, 2026.
Xin Zhou, Yiwen Guo, Ruotian Ma, Tao Gui, Qi Zhang, and Xuanjing Huang. of the internal reward models improves self-rewarding language models. arXiv preprint arXiv:2502.08922, 2025.
Self-consistency
Zhengyuan Zhou, Susan Athey, and Stefan Wager. Offline multi actioon policy learning: Generaliza- tion and optimization. Operations Research, 71(1):148-188 2023.
Guangcheng Zhu, Ruixuan Xiao, Haobo Wang, Zhen Zhu, Gengyu Lyu, and Junbo Zhao Large margin representation learning for robust cross-lingual named entity recognition. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4270-4299 2025.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A ge- ometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820 29834, 2021.
Yuxin Zuo Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504. 16084, 2025.
14

---

Published as a conference paper at ICLR 2026
Appendix
LLM Usage
15
Theoretical Proof
16
Notion
16
B.2
GRPO Preference Optimization
16
B.3
Gradient Dynamics and NTK Aligmment
18
B.3.1
Change in Log--robayilityy
18
B.3.2
Main Generalization Result
19
B.4
Unifying Trajectory Divergence and Domain Discrepancy
................................
20
B.5
Main Theorem: Generalization Bound
21
B.6
Main Theorem: Convergence Analysis
23
7
Addition Proofs
24
Discussion and Limitations
26
D Experiment Details
27
D.I Detailed Setup
27
D.2 System Prompt
28
D.3 Baseline Description
28
More Experiments
29
Comparison with More Supervised RLVR Baselines
29
Extend TraPO to More Models
29
................................
E.3
TraPO Is a Universal Component
30
Run TraPO on DeepMath
30
E.S
Different Selection Strategies
31
Stability of TraPO
E.
Training Cost Analysis of TraPO
31
E.8
Different Ways of Utilizing Reliable Passrate Databases
31
More Related Work
31
Pseudo Code
32
A LLM USAGE
In the preparation of this paper. the LLM was used solely for language editing and proofreading to improve clarity and readability.
15

---

Published as a conference paper at ICLR 2026
Table Table of Notations and Descriptions
Notation
Description
Optimization and Reward Setup
J
Group Relative Policy Optimization (GRPO): policy update via re- sponse grouping and relative advantage.
T E 10
Binary reward: 1 for correct, for incorrect response. Equivalent preference optimization objective under binary rewards.
Jprel
P
Empirical accuracy: fraction of correct responses in a batch.
N+
Expected number of correct and incorrect responses: N pN, N
(1 - p)N.
p².p"
Group-specific weights: p* = 1-p
Vp(I- Advantage estimator: A., = P
Vp(1pp
/p(1-p)
(0) clip(:: (1
Probability ratio between current and old policy for token generation. Clipping function to stabilize policy updates.
Generalization and NTK Analysis
A Alog n (
Change in log- pog-rrbbability of response Tg. after update.
(((g, r) (q )))
Response-leve NTK: (Ve log (T V log ((7'll)))
0+ > 0, +
0
Gradient alignment: correct-correct and error-error responses align.
Orthogonal gradients ((d))
Correct and incorrect response gradients are orthogonal.
traj (g. 4') sign( A log m²) = +1
Trajectory divergence: 1 - cos 7 between response pass rate.
Positive generalization: similar questions benefit from training.
Convergence and Risk Bounds
(D, D.))
Domain discrepancy: maximum distinguishability under H.
dHAH I (Dng)
+ Xa
Trajectory divergence bounds domain shift.
(t))
RD, (
Generalization risk on target domain.
RTC
Dynamic trajectory consistency risk: aEE
mij + Ly (1 - ((0)).
(())
Average confidence (e.g., pass rate) at iteration t.
U. RDD (2))))
Expected target risk, used in convergence analysis.
Up+1 - U T1 + +
Monotonic convergence inequality under consistent learning.
3:
Residual term: includes ADmij AC, and M2.
B THEORETICAL PROOF
In this section, we provide proof's for the generalization error bound and convergence of the proposed semi-supervised framework TRAPO.
B.1
NOTION
We provide the notions used in the proof in Table
B.2 GRPO PREFERENCE OPTIMIZATION
We begin by formally establishing that GRPO performs preference optimization between correct and incorrect responses when the reward binary.
Lemma B.1 (GRPO as Preference Optimization). When the reward is binary (rx E {0,1)) 13), the expected GRPO loss for a question g reduces weighted preference optimization objective:
N
(
44) 1 +
N
na(Tī
(q) -
Jmed= p
Σ
min
Σ
max
- (11)
THu
(TÚ
1=1
where:
== Σi= 1 (re(g)
is the empirical correctness rate for g.
16

---

Published as a conference paper at ICLR 2026
N pN, N (1- N)N are the expected number of correct and incorrect responses in a batch of N samples,
p
1-p Vp(1-p)'
vp(l-p) are the group-specific weights.
Proof. The standard GRPO loss for a batch of responses {Ti)i=ı is:
J
min (ra,(0)A.,,,.: A.,, clip(r..0),
Tolt..l where wherer,, (0) Te (T. is the probability ratio at token I, and Ai, is the advantage estimator:
For binary rewards, r:(g) r. the response Ti is correct, and 0 otherwise. The advantage A is defined as:
A.
where f p the empirical mean reward (correctness rate), and /p(1 p) is the empirical standard deviation.
Thus, the advantage simplifies to:
1-p p+
if r; (correct),
A. vr(1-p)
vp(l-p) pp
if rr 0 (incorrect).
Now, consider the term in the loss: min (re,(()AA,, Ar clip(ra.(0)) -8, + +)))
We analyze this based on the sign of A.
Case 1: A. > 0 (r; = 1, correct response) In this case, min function simplifies to:
TT( 9, Ti,
A. min (O), 1+))= min
1 +
Th (Ti, (, Ti,
Summing over all tokens l in the response T,t. and noting that Σ log Ti,()) log (T, (4), we have (in the limit of small learning rate or by ignoring token normalization):
TT(T," [g) To (T² (q)
min(.) x pt min
1++) )
Case 2: A < 0 (r; = 0, incorrect response)
Here, Ar -p_ and the min function becomes:
min (-p r (f), pp clip(r..(()) e, + +)) = = max (r,,(0), - -s),
because min(-a, b) = max b). Summing over tokens:
To(T 14)
min(.) ~ pp max
-
Tea (T (9)
Taking the expectation over the response batch [ ~ Tem (.14). and using the fact that there are N+ pN correct and N N)N incorrect responses average, we obtain the expected loss:
(T (g) Toq (T² (q)
TT( 44) Th (Tr ④))
E
p
min
1 1++
p
max
This is exactly the preference optimization objective in
11
This completes the proof of B.1
17

---

Published as a conference paper at ICLR 2026
B.3 GRADIENT DYNAMICS AND NTK ALIGNMENT
We now analyze how training on a question affects the model's behavior on another question q', leveraging the NTK framework.
B.3.1 CHANGE IN LOG-PROBABILITY
We start by deriving the change the bog-probability of generating response oó to question g' after a GRPO update on question
Proposition B.2 (Gradient Update Effect). Let A log n' (r(44)) = log n' (Tyll') log x (T)ll4) be the change in log probability after one GRPO update on q. Under the assumption that the parameter update 0 0º is small and given by the SGD update q, we have:
logg x' (Teld) Volg log T (T
N
V log n (T ②) -
Σ ==1
v log क' (Tj 10) 10)) (12)
Proof.
Using first-order Taylor expansion of log no (T around 0":
log + (Tald') log T' (T + (Vo log T' (T 0h+1 6*) 0(1 -8
The parameter update 0C 0 is proportional to the negative gradient of the GRPO loss on From B.1 the loss gradient is:
N+
(T; 9) Tea (T," (9)
)]
Ie (T (g)
]
VoJg p
0
min
1 +
V
max
Tea. (T [q)
In the "nearly online" setting of GRPO, where responses are resampled at each iteration, we assume Ty Tea. the ratios close to 1. In this case, the min and max operators are inactive (i.e., the clipping does not bind), and we have:
TT 9)
min
+ 8
2 Va log No (T; (4),
To
(T²" 9)
To (4)
V
max
~ v log (T( l1).
Te (1j (g)
Thus, the update direction is:
N+
N
0++1 Or
R
11
Vologn'(T, (g)
V log n (T, (0)
where 7) is the learning rate. Substituting into the Taylor expansion and dropping higher-order terms, we get:
A log n (Tild') -n V Vlog p*
V log n. (', (q) p log n' (T (q)
The learning rate is a positive scalar. Since we are interested in the sign of the change (increase or decrease), we can absorb -1) into the expression and consider the inner product as the primary determinant of the sign. For notational simplicity and consistency with the original text, we present the update direction without r). leading [2] This completes the proof B.2
To analyze the sign of A lo (Tilq'), we introduce the response-level NTK and state the gradient alignment assumption.
18

---

Published as a conference paper at ICLR 2026
Definition B.3 (Response-lever NTK). The response-level Neural Tangent Kernel (NTK) between two response-generattieee generation events (g, T) and (4/ r') is defined as:
((g, T), +)) = (Ve log (T q), Ve log (T (d)).
Under the NTK regime for sufficiently wide neural networks, + converges to deterministic limit and remains approximately constant during training (Jacot et al., 2018 Arora et al. 2019).
Assumption B.4 (Gradient Alignment). Let q, q' be two questions from the same task family with o' indicating semantic similarity. Then, in the infinitewwidh limit, the following asymptotic properties hold:
(i) (Correct-Correct Alignment) For all correct responses 7.. E (R) TK RR)
lim (Vo log To (T ld) V To(
+ ()) kk ' 0.
widli 000
(ii) (Incorrect-Incorreed Alignment) For all incorrect responses E R (q), T E R (d))
lim (Vo (Volog To (T 1d), V (oT [q)) kK .s3' > 0. widti
(iii) (Correct-Incorred Orthogonality) For all 7,. E R R (q). 7j E R (4). + E { T
lim VollogT(TT Ve log (T 1q)) 0,
lim
VoloogToTH Id),, V To 14)) 0.
wid
Remark B.5. This assumption is motivated by the structure the NTK. For semantically similar in- puts and valid (correct) outputs, the corresponding feature representations activate overlapping sets of neurons, leading positive kernel values. Conversely, correct and incorrect responses represent conflicting patterns, and their gradient directions become nearly orthogonal in overparameterized models (Zhu al. 2027
B.3.2 MAIN GENERALIZATION RESULT
With the NTK alignment assumption in place, mance on a similar q'.
prove that training improves perfor-
Proposition B.6 (Generalization through Gradient Alignment). Let q and be two questions that are similar in structure and difficulty, denoted q o, belonging to shared task family T. Let Th be a response to d' Under B.4 and the GRPO update rule, the sign of the change in log-probability A log n² (Th q') is determined follows in the infinite-width limit:
+1 The is correct response to d'
sign (Alog x
(Tkl Torg='(Tell
1-
i is an incorrect response to q'
Proof. We substitute 12 and analyze the two cases separately.
Case 1: Tg is a correct response (T' +
N+
Alog 1g)= = p
V ( Iq)). To q))
logoo ( I l')), V log x (T, 19))
(13)
By B.4 i). each inner product in the first sum is strictly positive in the infinite-width limit. Since
p+ 0, the entire first term is positive.
By B.4 iii), each inner product in the second sum is zero. Thus, the second term vanishes.
19

---

Published as a conference paper at ICLR 2026
Therefore, A log (' 1 (') 0, meaning the log-probability of the correct response T in- creases.
Case 2: Tye is an incorrect response (T xx
N+
Alog 1d')
V log (T 14), Vo log 1g))
Vo Volog ('), Vo log T (T, [g))
(14)
By B.4 iii), each inner product in the first sum is zero.
By B.4 ii,, each inner product in the second sum is strictly positive. Since p > O, the is positive, but it is preceded by negative sign, making the entire second term negative.
Therefore, A Alo (T' (')
meaning the log-probability of the incorrect response T/E de-
creases.
This shows that GRPO implicitly pushes the model in direction that generalizes to similar tasks by reinforcing correct responses and suppressing incorrect ones.
Combining both cases proves B.6
Corollary B.7. In the NTK regime, GRPO encourages an inductive bias towards solutions that lie in directions of high kernel alignment across correct responses within lask family. This promotes generalization even with sparse supervision.
B.4 UNIFYING TRAJECTORY DIVERGENCE AND DOMAIN DISCREPANCY
We establish formal connection between the trajectory-level dynamics in method and classical domain adaptation theory. While our theoretical analysis begins with gradient alignment in parameter space. the practical metric usjetrajyctory divergence-is measured in the space of confidence dynamics. We first define gradient-based notion of coherence, then show it implies similarity in pass rate evolution.
Definition B.8 (Gradient Coherence). For questions q and q', the gradient coherence step t is:
c(++) cllalg d') = E m- (-44) [cos - (Vo log No, (T(4), V ne, (T' ('))4))), p'mmo, (-14))
(15)
where cos La,,(aa,) b)
(a.b) Ta | High coherence indicates similar optimization directions.
Definition B.9 (Trajectory Divergence). Let T( (P("), (() RC be the trajectory
vector of question q, where p(() is its pass rate at round s. The trajectory divergence between q and at step is:
(T T:
Dig q')
(16)
|T
This measures the angular dissimilarity between their confidence evolution paths.
We establish the key link: gradient coherence implies low trajectory divergence.
Lemma B.10 (From Gradient Coherence to Trajectory Coherence), Suppose the policy ne trained
under small learning rates and lies in a region where the NTK is approximately constant. If for all
S and for questions q, d' we have ((a) (a that:
Es, then there exists constant L > 0 such
maj (g.
(9,9)) L.
Σ MsEa
20

---

Published as a conference paper at ICLR 2026
Proof (Sketch). Under NTK linearity, the change in log- pee--rbbbbiityy is Alog ~ M. Ve a log Te, (Tllq), A6g). High gradient coherence implies that the relative improvement for cor- rect responses similar across q and q'.
Since pass rate p( is an empirical estimate of the model's 's confidence generating correct re- sponses, coherent log-prob updates lead to similar p)(o) evolutions. By vector concentration and Lip-
schitz continuity of the cosine similarity, the Euclidean distance ||zz{") -r ll2 = O (E* NoEs)
which implies D (g. d') O (1175))) - T 11)) The full proof is in B.7
We now state the main result, bounding domain discrepancy via trajectory divergence. Proposition B.11 (Trajectory Divergence as Proxy for Domain Discrepancy). The HAH- divergence between Dj and Du is bounded by the expected pass-rate trajectory divergence:
(Di D.) S E. gND, o gNDu
[
Ad=
(17)
where a >> O depends on model smoothness and training dynamics, and da 2 0 is an irreducible baseline discrepancy.
Proof. The HLAl-divergence is:
Pr (h(g) # h1(x)) Pr (h(d)) + ('d')) AN R-D,
(D,)
sup
q'rDD
In our setting, hypotheses h E H are induced by the policy To. The ability H to distinguish D; from Da depends the discrepancy in their induced gradient fields:
G ED (o),( Gf Bg [ooJg (0.))
Let 1/
||G
(t) ||. Standard domain adaptation theory gives: G
dHAH(Di, D.) S C sup A Aas
for some C
>> O.
Now, A is small when the gradient fields are aligned across domains. From Definition this alignment is captured by Ggallg.g) Applying Lemma B.10 high gradient coherence (low
(t)
B.8
clt)) grad implies low (04) (g.9))
Conversely, ((t) (q, d') is small on average, indicates that the confidence evolution is coherent across domains, which (by contrapositive of Lemma B.10 implies that gradient coherence must be high, hence A G is small.
Therefore, E E serves as an upper bound proxy for A and thus for dHAH Setting a absorb the constants yields the result.
Corollary B.12. Low pass-rate trajectory divergence Duejj implies low domain discrepancy, en- abling effective transfer without explicit adversarial or feature-level alignment.
B.5 MAIN THEOREM: GENERALIZANE
BOUND
Theorem B.13 (Trajectory- (Trajectoyy-Consisteet -onssiseent Generalization Bound), (Formal) Let 8 E (0,1) be a con- fidence parameter. Suppose the loss function L y y Rzo is Ly-Lisschitz in its second argument and bounded, i.e., L(-,,) & B. Let Th be a model trained under the TRAPO framework at round f.
21

---

Published as a conference paper at ICLR 2026
Then, with probability at least 1 - ó over the sampling labeled and unlabeled data, the expected risk of ni on the target distribution D, satisfies:
RD. (T( - RD, ( + BV
In(4/))
+ a
E.g
| (d))
2m
+ Ly (1 0()
Iu(2n/0)
+ +X,
2G
where:
RD, (T is the empirical risk on m labeled source samples;
DD (T T( rellabiee
is the cosine divergence between the trajectory of d' and
IT I1- I ralmatalll
the average reliable trajectory;
= ΣØ g00) with - Ma -,,)) the voting confidence for unla-
beled sample dzj
= 0 absorbs the irreducible domain shift and best-in-clant error.
Moreover, define the Dynamic Trajectory Consistency Risk:
RTC a. E. (Dus (4/) + Ly (1- 1
In in(2n/5)
+
2G
the Consistent Trajectory Learning Condition holds:
lim B.g [D (q')) = and 1000
lim ( 1, 1-300
then R/O + and RD. (1) (T. X, implying asymptotic generalization to the target domain.
Proof.
We start from the standard domain adaptation risk decomposition (Ben-David et 2010
RD. (m Ro, (m dH, X,
(18)
where À = inf (Rv, (h) RD. (n)).
Step 1: Bounding the source risk Rp, (T (m)) Using a standard concentration inequality (c.g.,
Hoeffding's lemma) for bounded losses L B, with probability least 1 8/2:
In(4/0)
(T < RD. (m + BV
2m
Step 2: Bounding the domain discrepancy dHAH- Under the NTK alignment assumption, trajec-
tory consistency controls gradient field divergence. From the trajectory-proxy proxy proposition B.1l have:
(D, S Egg,, [Dit (()) + Xa-
where (11) (q') measures the cosine divergence between the gradient trajectory of g' and the average
reliable trajectory (1)
reliable over source or high-confidence samples.
Step 3: Pseudo-labeling error. Let g((t) the pseudo-label for g' via majority voting. The error
in using j/(1) instead of Vime bounded by:
RD, RD,(=)) (=(0) - [L(T) (())) < Ly P(yme + 10)
22

---

Published as a conference paper at ICLR 2026
For unlabeled samples, let pj P( (1) à Li= Ma = )) estimates pz. Then:
ymme.)). The observed confidence C
P(U)
(1) * Ylue.
cg + |c p31-
By Hoeffding's inequality and a union bound over = I,..., with probability at least 1 - 6/2:
In (2n//
10 - p2]
Vj.
2G
Averaging over j. we get:
In(2n/o)
P/ & 5/(0) 11 c(0 +
2G
Step 4: Union bound. Combining Steps 1-3 with a union bound (total probability 2 8), and absorbing Aa into X' = x λa, obtain the desired bound.
Finally, under the Consistent Trajectory Learning Condition, both ((t)) RiO 0, yielding asymptotic generalization.
and ((+) + 1, so
B.6 MAIN THEOREM: CONVERGENCE ANALYSIS
Theorem B.14 (Monotonic Convergence under Consistent Trajectory Learning.. Let U, [D.(TT)))
RD. (T denote the expected target risk at training round t. Under the Consistent Trajec-
tory Learning Condition B.13), and assuming:
1. Stochastic Gradient Descent (SGD) with learning rate q 0,
2. NTK stability:
| (t) (x)] is bounded for all x,
3. Lipschitz smoothness of L mg
(6)
4. Sufficient ensemble size G such that / /in(2n/8) S c,
then the expected risk sequence {U))=1 satisfies:
U4i U, 11 + Be,
where:
E = [ (T ( M1 2 measures the expected gradient magnitude on source data,
B, = ax.AA1 (D( + Ly AC + ni 2² aggregates the residual dynamics, with:
AD (t) mnj
E (((+1)) (g)) D/10) (a')
AC E [c(+t]]= -c(00] -c())
and M > O bounds the gradient variance.
Moreover, if Dee TIt oo and Ee Vi co, and (D(() Inj < 0, AC > 0fr all L > To. then:
lim 000
RIVo (1
(0) )0)]
and
lim simsupl U² S RD, (f*) + X, t0000
where 1 is stationary point of the source risk.
23

---

Published as a conference paper at ICLR 2026
Proof.
We analyze the expected change in target risk: Ut+1 U2 IE RD, (Jz+1) Ro. (m²²)
Using the smoothness of Lomg") and the update B1t1 1 where gr is the stochastic gradient, we have:
RD. (f++1) S RD. (T)) - n (V RD. ), 0t) + "/ 11
Taking expectation over the stochastic gradient and data sampling:
L nh [11 U44 s U. - B IiV.D. (T (0)) +
2
Now, from B.13 we know:
RD. (TT S RD. (TT + R/C const.
Thus, the gradient V. RD. (TT ) is aligned with V R., (T ( and VoR/C. Specifically:
E [IVooRD. (T |1 2 E V (T (1) 11
VoRtCl
Now, observe that:
VORIC
(2) S a
E.D (3) maj
+ Ly
d
c(0)
~ a
|AD + Ly |AC
dt
dt
in discrete time.
Under the assumption that trajectory divergence decreasing (AD (*)
and confidence in-
traj
creasing (Ac()) 2 0), the residual 3r captures the rate of improvement in transferability.
Furthermore, B[[la.11] S M2 under NTK stability and bounded loss.
Thus, we obtain:
U441 S U. - ncE. + Be:
with & = Ro (T (4) 11.) B = a. (D (t) + Toy AC + 1/ M
Now, summing over t:
Σ ne S U1 - lim U, +
B
If AD (t)
traj S and AC 2 0, then B, < 1/ M2 eventually, and Σ nh oo implies Enc < 00.
Since Σ 7/: 0o, we must have &t + 0, i.e.
RID (1) )1] O.
lim 200
Finally, from B.13 since (t)
O, we get:
lim sup U4 < Ro, (f)) +
t+00
where 1 is a stationary point. This completes the proof.
B.7 ADDITION PROOFS
We provide the full proof of Lemma B. 1.10 which connects gradient coherence in parameter space to trajectory coherence in the space of confidence dynamics.
24

---

Published as a conference paper at ICLR 2026
Lemma B.15 (Restatement of Lemma B.10 Suppose the policy ne is trained under small learning rates {= and lies in a region where the Neural Tangent Kernel (NTK) is approximately con-
stant. If for all S t and for questions 9. q', the gradient coherence satisfies C (g. q') 2 1 Cs.
then there exists constant L >> 0 such that:
Da ( L.
Σ 7sCs
Proof. We proceed in three steps: (1) bound the difference in log-probability updates under gradient coherence; (2) relate log-prob changes to pass rate evolution; (3) bound the cosine distance between trajectory vectors.
Step 1: Gradient coherence implies coherent log-prob updates. Under the NTK regime, the model evolves via kernel gradient descent, and the change log-probability after update is ap- proximately linear in the gradient:
A log nO (Tllg) == log Te, (rllg) log no.., (Tll4) = M-- (Vo log To._, (T(lg), 20,-1).
Let T. and T. be the correct responses for and q' We are interested in how the model's 's confidence generating correct responses evolves.
(a) Let By = Vo log ne, (T,)19) and Sy
Vo log To, (Tg) (Tgilla).. By Definition B.8 we have:
(+)
(gi
2 E3.
18g'
This implies (by standard vector inequality):
(s)
Bd
(+)
(5)
[g
B
Assume the gradient norms are bounded:
|| (5) S G.
(s) Eg'
S G. Then:
(x) 2/ + || - -
g
For simplicity, assume gradient magnitudes evolve similarly (or absorb into constants), so: (z) S G/s
Now, the parameter update is A0, 7/ VaJ., which is weighted sum of gradients over the batch. If q and o' are both in the batch or their gradients are representative, then:
A log n" (7
log " (T lq')) < nh nlll!!
s ||00/l/n S G' M,
where M bounds the update direction. Thus:
A log (7,119)) A log T (T) S CIvE. Summing over 1 to the total difference in log-prob evolution is:
log ne, (T log ne, ( (T)lld')1 S Ci Σ nAVe.
Step 2: Log-prob coherence implies pass rate coherence.
The pass rate p(() is defined as:
N
p(s) =
[fo, (g: Ex) passes],
N
25

---

Published as a conference paper at ICLR 2026
where &N represents stochasticity (e.g., dropout, sampling). pa is empirical estimate of Pr[correct|ly, 0s).
Assume the mapping from log Tu, (7g to B[Pg is L-Lipschitz (holds for softmax policies under bounded gradients). Then:
[ E I log no, (T)) (Tgill)) log no, (Tg S L'C. E noVE..
By concentration (c.g., Hoeffding's inequality), with high probability:
[P(( pp-1 S L'C. Σ nv
where = OLL/VG) is sampling error. For large N, is negligible.
Step 3: Trajectory vector proximity implies low divergence. Let Tz(0) (P()) ...., ,..,,, )))))
T = (P()) p(t)) Then:
[1T - T
11 Σ pp())
plo)
L'C Σ 1h VE
mm
Using the inequality (2 a,,) < 8 2im1 a& and assuming T)r, E, small, we get:
Σ ToVES
( - -1th S C2
Σ
Σ NsE
7
but more conservatively, 175s summable, then:
1 Σ n2cl/1
I T q
12
112
O
Now, the cosine distance:
(T()) T())
T
T(()
2
D (g.9') =1
+ 0 - - 112).
|
|
(1
|
the trajectories are bounded away from zero (i.e., not all zeros), then:
D g') L. |
I
12
L
Σ noVEs
To match the lemma statement, can weaken Vea to és under e. E (0, 1), redefine as the squared coherence gap. In either case, there exists constant such that:
DIG (g. L.
SL
Σ 1sk
which completes the proof.
C DISCUSSION AND LIMITATIONS
First, our results demonstrate that semi-supervised training using 4K labeled data combined with 16K unlabeled data outperforms fully supervised training 45K labeled data. This encouraging
26

---

Published as a conference paper at ICLR 2026
finding aligns with the insight proposed by Li et al. (2025b) in the context of RLVR training: thor- ough training (i.e., more training epochs) on smaller curated datasets can yield better performance than training with larger datasets for fewer epochs. Our work further extends this observation by showing that unlabeled data, when carefully selected using guidance from labeled data training, effectively enhance the model's reasoning capabilities, thus amplifying the benefits of semi- supervised RLVR.
In addition, due computational constraints, our evaluation is currently limited models under the 7B parameter scale. Exploring the applicability and scalability of this semi-supervised paradigm to larger language models (e.g.. 13B or beyond) remains an important direction for future research, as larger models may benefit even more from effective utilization of unlabeled data.
One key observation from our experiments comparing Qwen2.5-Math- and LLaMA-3. is that the more effective supervised training with labeled data is on model, the better the labeled data guides the selection and utilization of unlabeled data. Conversely, if RLVR training with labeled data yields only marginal gains, its impact unlabeled data filtering is also limited. Therefore, we recommend applying TRAPO primarily settings where the labeled data and model are well aligned.
Finally, we believe it is a promising direction, similar to active learning, to investigate what types of labeled examples most effectively guide unlabeled data training, and we plan to explore this in future work.
D EXPERIMENT DETAILS
D.1 DETAILED SETUP
Implementation Details. Following Dr.GRPO (Liu et al. 2025 disable length and standard
By default, we use Qwen2.5- Math-7B (Yang et al., 2024), following prior work Cui al. (2025); Zeng al.] (2025b): [iu et al. (2025). Besides, we remove the KL regularization by setting 3 = O and set the entropy coefficient 0.01. Our rollout batch size 64, with rollouts per prompt, and update batch size 64. Rollouts are generated with temperature sampling (T' = 1.0). We use Math- Math-Verify aas the reward function, without format or length bonuses. For unlabeled data selection, we set the top-p threshold 0. and the threshold I to 0.5 in Eq. 7 The warmup stage consists of 5 epochs. In addition, given that experiments are performed across different data scales, the samples used non-full-data scenarios are randomly sumpled from the original dataset.
error normalization in the GRPO loss (Eq. 9) for all experiments.
Training. In addition to Qwen2.5-Math-78 we extend TRAPO to DeepSeek-RI Ri-eessi---ii-ill------ 1.5B (Guo et al. 2025) and
(Team] 2024). To ensure fairness, we maintain 8 samples per prompt for all RL-trained models. The learning rate is constantly set as 1e-6. For all training. we use the same setting as in Yan et al. (2025) and the same validation set to select the best checkpoint. All the experiments
with 8x NVIDIA H200 with 141GB memory.
Our implementation based on LUFFY? and veRI which use vLLM as the rollout generators. We are thankful for these open-source repositories.
Qwen2.5-Series Models. Since the context length of Qwen2.5-Math is 4096 and the generation length of off-policy samples could be lengthy, we change the rope theta from 10000 to 40000 and extend the window size 16384. For all 5wen2.5-Series models, we use the same dataset as described in Sec.
is a compact, 1.5-billion- parameter language model distilled from the nigh-performing DeepSeek-RI series (Guo et al. 2025). Built on the Qwen architecture, it combines strong reasoning capabilities with high effi
Verify
27

---

Published as a conference paper at ICLR 2026
ciency, offering excellent performance in math and logic tasks despite its small size. For DeepSeek- RI-Distill-Qwen-1.5B we use the same dataset as described in Sec.
Llama-3.1-8B. For Llama3. 1-8B, follow Simple-RL-Zoo Zeng et (2025a) and use a sim- plified prompt, and we do not ask the model to generate <think>\n </thinka tokens.
D.2 SYSTEM PROMPT
All our trained models, except LLaMA-3.1-8B, share the same system prompt for training and inference:
Your task is to follow a systematic, thorough reasoning process before providing the fi- nal solution. This involves analyzing, summarizing, exploring, reassessing, and refining your thought process through multiple iterations. Structure your response into two sections: Thought and Solution. the Thought section, present your reasoning using the format: Each thought should include detailed analysis,
"<think>\a \n thoughts </think>\n" brainstorming, verification, and refinement of ideas. After "</thi thhink>k \n" the Solution section, provide the final, logical, and accurate answer, clearly derived from the exploration the Thought section. applicable, include the answer in \boxed() } for closed-form results like multiple choices mathematical solutions. User: This the problem: (QUESTION) Assistant: <think>
For LLaMA-3. 1-BB, we do not use the above system prompt as we find the model cannot follow such instruction. Thus, we use simplified version that only includes the CoT prompt and do not include <think> token.
User: {QUESTION)
Answer: Let's think step by step.
D.3 BASELINE DESCRIPTION
Unsupervised Baselines:
TTRL Zuo et al. 2025 : treating the majority-voted output as the pseudo-label and training
with GRPO.
Self-Certainty (Zhao et al. 2025): maximizing the KL divergence between the model's rollout token probabilities and a uniform distribution to encourage confident predictions.
Tooeen-Level Entropy (Agarwal et al. 2025): minimizing the entropy of individual output
tokens during rollout to promote consistency.
Sentence-Level Entropy (Agarwal al.
2025): maximizing the overall sentence probability of the generated output to favor high-likelihoon sequences.
Supervised Baselines:
Simple-RI (Zeng et al. 2025b): training from Qwwn2.5-Math-77 using rule-based reward.
Oat-Zero (Liu et al.
2025): training from (wwn2.55-Male Matth-BB and rule-based reward, propos- ing to remove the standard deviation GRPO advantage computation and token-level nor- malization in policy loss computation.
PRIME-Zero (Cui et al. 2025): using policy rollouts and outcome labels through implict process rewards.
OpenReesonerreero Cui et al.
2025 recent open-source source implementation of RLVR meth-
ods.
Fully Supervised (aa et al.
trained on-policy RL within the RLVR paradigm using Dr.GRPO (Liu et al. 2025 with the same reward and data.
2025)
28

---

Published as a conference paper at ICLR 2026
Table Comparison with other fully supervised training methods. Bold and underline indicate the best and second-bes results, respectively.
In-Distribution Performance
Out-of-Distribution Performance
Model
AIME 24/25
AMC MATH-500 Minerva Olympiad Avg,
ARC-e GPQA" MMLU-Pro Avg.
Qwen- Rase Yang
2024
11,5/4.9
31.3
43.6
7.4
15.6
19.0
18.2
11.1
16,9
15.4
Qwen-Instrict Yang al.
12.5/10.2
32.7
70.3
34.1
Fully Supervised Methods Truined on 45K Samples w/ All Labels
Sirp Zeng
2025b
27.066.8
54,9
76.0
25.0
34.7
37.4
30.2
23.2
34.5
29.3
OpenReasoner-222 Hu et
2025
16.5/15.0
52.1
82.4
33.1
47.1
41.0
66.2
29.8
58.7
51.6
PRIME-Zero Cui al. 2025
17.0/12.8
54.0
81.4
39.0
40.3
40.7
73.3
18.2
32.7
41.4
Oat-Zero (Liu 2025
334/11.9
61.2
78.0
34,6
43.4
43.7
70.1
23.7
41.7
45.2
On-Policy RI.
2025
25.1/15.3
62.0
84.4
39.3
46.8
45.5
82.3
40.4
49.3
57.3
TRAPO Trained w/ 4K Labeled Samples 12K Unlabeled Samples
TRAPO (ours)
24.3/17.1
60.0
84.6
39,3
48.3
45.6
84.6
43,9
50.7
89.7
Table 5: Overall performance on nine competition-leve benchmark performance on LLaMA-3.1- 8B-Instruct (Team 2024).
In-Distribution Performance
Out-of-Distriattion Performance
Model
AIME 24/25 AMC MATH-500 Minerva Olympiad Avg.
ARC-e GPQA"
MMLU-Pro Avg.
Original Model
Original Model
5.100.4
18.6
44.6
19.5
14.1
17.1
24.2
0.5
38.6
21.1
Unsupervised Methods Trained on /K Unlabeled ID Samples & /K Unlabeled OOD Samples
TTRL
6.1/0.1
21.8
46.6
25.4
16.7
19.5
11.0
41.8
17.6
Self-certainty
6.9/1.2
20.3
45.5
23.7
17.1
19.1
13.3
39.5
17.6
Token-level Entropy
5,3/0.1
19.6
43.5
22.7
16.9
18.0
10.5
0.0
38.7
16.4
Sentence- Entropy
7.2/0.2
11.7
41.5
Semi-supervised Methods Trained on IK Labeled ID Samples & JK Unlabeled OOD Samples
TTRL
7.1/0.1
24.6
19.3
11.5
40.9
Self-certainty
6.6/0.6
20.7
46.4
23.2
16.3
19.0
12.7
0.0
40.3
17.7
Token-level Entropy
6.4/0.1
20.5
23.3
18.6
11.3
41.6
17.6
Sentence-level Entropy
7.5/0.1
21.3
46.7
25.1
16.9
19.6
12.3
0.0
41.9
18.
TRAPO (ours)
9,9/0.2
21.5
48.0
26.1
18.7
20.7
12.1
0.0
43.4
18.5
Fully Supervised w/ Labels
6.9/1.6
22.2
52.2
21.0
17.5
20.2
10.4
47.5
19.3
E MORE EXPERIMENTS
E. COMPARISON WITH MORE SUPERVISED RLVR BASELINES
In Table we compare our method with additional fully supervised RLVR baselines, all of which trained the complete 45K labeled dataset, with results taken directly from Yan et al. (2025). The results show that our model, trained with only 4K labeled and 12K unlabeled samples, achieves performance that surpasses all baselines trained the full 45K labeled data. For instance, TRAPO method outperforms the outstanding Oat-Zero baseline by 1.9% in in-distribution perfor- mance and by significant 14.5% in out-of-distribution performance. This further underscores the effectiveness and value of our proposed TRAPO.
E.2 EXTEND TRAPO TO MORE MODELS
We further investigate whether our proposed semi-supervised paradigm, TRAPO, generalizes to small models, instruction- models, and weak models. To this end, we conduct experiments on DeepSeek-PP Ri-kisil- Seelll-ween-1..51 (representing small models) and LLaMA-3.1 --B--trrrutttt (repre- senting instruction- tuned and relatively weaker models), under unsupervised, semi-supervised. and fully supervised training settings. The experimental setup follows that of Table 2 As shown in
29

---

Published as a conference paper at ICLR 2026
Table 6: Overall performance on nine competition-level benchmark performance on DeepSeck-RI.
Distill-Qwen= Qwen-1.5B 1.5B (Guo et al. 2025).
Model
AIME 24/25
AMC
MATH-500
Minerva
Olympiad
Arg.
ARC-e
GPQA*
MMLU-Pro
Arg.
Original Model
21.0/20.
76.6
Unsupervised (TTRL.)
1621/21.7
80.6
28.7
42.7
42.8
25.7
31.9
19.2
Semi-supervise (TRAPO)
27.9/22.6
61.9
82.2
32.0
45.3
45.3
34.4
33.5
22.6
Supervised
28.3/22.5
Fiteiin Maninng
15.3
Pefommnnc
Porfommnnce
Deffcrmennce
44.3
40.6
17.4
41.0
28
31.6
Math-Aw.
ARGe
GRRA
Meth Ava
ABC-
GPPAA
MVLL-P
Math Ari.
GPOA
(a) Sentence-level Entropy
(b) Token-level Entropy
(c) TTRL
Figure 7: Different unsupervised methods combined with our trajectory-basse bbssed filtering approach improve performance, compared to naive semi-supervised method that directly combines pervised and unsupervised approaches. The experimental setup follows Table
Table 5 and 6 TRAPO consistently outperforms the unsupervised baseline (TTRL) by signifi- cant margin and approaches (or even surpasses) the performance of the fully supervised baseline on both models. Specifically, on PrepSeck-R1-Dissee Dhitill-Qwen-L.,, Qwen-1.5B, TRAPO improves over TTRL by 2.0% in in-distribution (ID) performance and 9.5% in -uttif-diittibuutinn (OOD) performance. On
exceeds TTRL. by 1.2% ID performance and 0.9% in OOD per- formance. Notably, TRAPO even outperforms the fully supervised baseline by 0.5% in IDD perfor- mance. These results strongly demonstrate the robustness, adaptability, and broad applicability of our method across diverse model scales and architectures.
E.3 TRAPO IS A UNIVERSAL COMPONENT
We demonstrate that TRAPO serves as a universal and modular component, whose pass rate trajectory-based sample selection mechanism be readily integrated into various semi-supervised baselines to identify reliable unsupervised reward signals. As shown in Figure 7 we apply this se- lection strategy three representative baselines: Sentence-level Entropy, Token-level Entropy, and TTRL. Compared to the naive semi-supervised counterparts that simply combine supervised and un- supervised objectives, augmenting these methods with our sample selection framework consistently yields performance gains across multiple benchmarks. This further validates the extensibility and plug-and-plan nature of our approach, indicating that the core principle of TRAPO dynamically identifying high-quality unlabeled samples via learning trajectories is broadly applicable and com- plementary to diverse semi-supervised paradigms.
E.4 RUN TRAPO ON DEEPMATH
To further verify TRAPO's broad applicability. we run it on DeepMath (He et al. 2025b), recently released dataset for mathematical reasoning. We randomly select 2K samples as labeled data and 6K samples as unlabeled data. We compare results from unsupervised, naive semi-supervised, and fully supervised methods. As shown in Table g our method, TRAPO, outperforms all unsuper- vised methods and naive supervised methods. Specifically, 1.5% improvement over the best naive semi-supervised method combined with TTRL, and is only 1.2% behind fully supervised training. Notably, on the OOD test set, TRAPO even surpasses fully supervised training by 2.4%, highlighting that TRAPO is not only label-efficient but also delivers outstanding performance.
the ID test set, TRAPO achieves
30

---

Published as a conference paper at ICLR 2026
E.5 DIFFERENT SELECTION STRATEGIES
Under a fixed selection ratio (30%), we compare TRAPO with other possible selection strategies, including simple random selection, sentence-level entropy-based selection (where lower entropy dicates more reliable pseudo-labels for the corresponding rollouts), and self-certainty (where higher self-certainty suggests more reliable pseudo-labels for the corresponding rollout). The experimental results in Table show that with fixed selection ratio of 30%, all other methods are significantly inferior to our selection method, TRAPO, on both the ID and OOD test sets.
E.6 STABILITY OF TRAPO
We seek verify whether TRAPO is sufficiently stable and insensitive sample order. To this end, we ran TRAPO three times with data randomly shuffled. Across these three trials (Qwen-2.5-7B, IK labeled, 3K unlabeled), both the results and the selected samples were nearly identical, confirming TRAPO's 's robustness (see table 16
E.7 TRAINING COST ANALYSIS OF TRAPO
We analyze the practical training cost of TRAPO from both theoretical and empirical perspectives.
Time Complexity. Each labeled or unlabeled sample is rolled G times per epoch, in line with standard RLVR practices. Let T represent the total number of training epochs, Nz, No the number of labeled and unlabeled samples, Csim the computational cost of a cosine similarity computation over short vectors, and Cgen the computational cost of single rollout. The only additional operation a cosine similarity computation Csim over short vectors, which is negligible compared to the cost of rollout generation Cgens i.e, Csim << Ogen- The time complexity of fully supervised training (using N = -N + Ny labeled samples) is:
Tsup O(T N G. -Ceen)
(19)
TRAPO has the same complexity:
TaaPo OTT. (Nz + Nu) G Cgen) OT (NN No) Csi)) O(T N G Cgen) (20)
Therefore, TRAPO and fully supervised RLVR share identical time complexity, both dominated by forward sampling and GRPO updates.
Empirical Training Cost. our experiments, TRAPO, supervised RLVR, and unsupervised RLVR are trained under identical conditions: same number of epochs, batch sizes, and hardware configuration H200 GPUs). Notably. TRAPO reaches its best checkpoint at nearly the same training step as the supervised baseline, indicating significant overhead in convergence speed. Table summarizes the wall-clock training times across different data scales, demonstrating that TRAPO incurs no substantial additional training cost compared to supervised RLVR.
E.8 DIFFERENT WAYS OF UTILIZING RELIABLE PASSRATE DATABASES
One may also consider other variants, such as not using the average pass rate trajectory and instead selecting, from the unlabeled samples. those whose pass rate trajectory most similar the trajec- tory of any labeled sample for inclusion in training. However, this approach can lead to unstable selection because, among the unlabeled samples, problems that are difficult, too easy, or of mod- erate difficulty can all exhibit relatively similar pass-rate trajectories among the labeled samples. As a result, the selection ineffective (Table 17).
F MORE RELATED WORK
Semi-supervised Reinforcement Learning. Semi-supervised learning has been widely studied
in weakly supervised scenarios (Zhu et al. 2025 Tang al. 2024| 2025 2026 Zhou et al. 2024
2026). where labeled and unlabeled data are combined to improve model performance under limited
annotation budgets (Blum & Mitchell 1998. Chapelle et al. 2009 SSubaaanna & Bilmes 2011
31

---

Published as a conference paper at ICLR 2026
Table 7: Wall-clock training time (reported "GPU-hours GPUs") across data regimes.
Data Size
Unsupervised
Supervised
Semi-Supervised (TRAPO)
4k
~7 x 8
~55
26
8k
11 x 8
~39 x 8
338 8
45k
~1l x
55
55
Rasmus et al., 2015 Laine & Aila 2016 Tarvainen & Valpola 2017; Berthelot et al. 2019 Xie
ct al. 2020] Sohn et al. 2020). In reinforcement learning, early work explored combining reward- based learning with self seff-superiised signals bseudo-rewards derived from environment dynamics or intrinsic motivation (Dudik et al. 2011 Finn et al. 2016, Thomas & Brunskill] 2016 Kallus & [Uehara 2020 Zhou et al. 2023). These methods typically treat supervised and unsupervised signals independently, for instance by summing reward and consistency objectives, or by pre-training on unlabeled data before fine-tuning labeled trajectories.
However, such semi-supervised RL approaches are ill-suited for large language model (LLM) train- ing under verifiable rewards (RLVR). In RLVR, the policy is optimized using feedback signals de- rived from answer verification (e.g., correctness of final outputs), rather than explicit action-level rewards. Unsupervised methods in this space rely on internal consistency, such as low token en- tropy (Agarwal et al.] 2025), high self-certainty (Zhao et al.) (025), or majority voting (Zuo et al.) 2025), to construct pseudo-rewards, While these signals can guide exploration, they often reinforce incorrect or degenerate reasoning patterns in the absence of external supervision, leading to model collapse (Zhang et al. 2025c).
Our work departs from prior approaches by introducing a guidance mechanism: the labeled data are not merely used to provide an additional reward signal, but to actively steer the selection and utiliza- tion of unlabeled samples. Specifically, we observe that reliable reasoning trajectories unlabeled data exhibit learning dynamics similar to those on labeled data. By measuring trajectory similarity in the reward model space, TRAPO identifies high-quality unlabeled samples whose reasoning patterns are consistent with verified ones. This ensures that unsupervised signals are only leveraged when they align with externally validated behavior, preventing the amplification of spurious patterns.
This paradigm shift from independent combination to supervised guidance addresses a key limitation of traditional methods. In high dimensional open ended generation tasks such reasoning with LLMs consistency alone is insufficient for correctness. Without supervision to anchor the learning process models easily overfit superficial patterns or self reinforced errors. TRAPO resolves this by using minimal labeled data as a "north star" enabling stable and effective learning from large amounts of unlabeled data, As we show empirically this leads to superior performance and data efficiency surpassing both fully supervised baselines trained on orders of magnitude more labels and unsupervised methods that fail generalize.
G PSEUDO CODE
We provide the pseudo code
32

---

Published as a conference paper at ICLR 2026
Algorithm TRAPO: Trajectory-based Policy Optimization
Require: Labeled data DD Unlabeled data Du, Warm-up epochs T warm. Threshold T, Top-p fraction
Ensure: Policy wo
Initialize: Pass rate trajectories T, [] for all
1: Reliable database Dellible < {T, D
2: for each training epoch do
3: Generate responses for D; U using Te
4: Compute (pseudo) pass rates p(t) for all questions
5: Update trajectories: T(t)
Tx TTi
p(()
6:
if' Tarrm then
Compute average reliable trajectory T!()) for a E Du do
7:
reliable
8:
Compute similarity: TCS., cos (T() (
9:
reliable
10:
end for Select reliable unlabeled samples:
Ureliable top tep- {u TCS, r}
12:
Add their trajectories to Peeliable
13:
end
14:
Compute loss:
(())
TGRPO +
Σ
JGRPO
u Matiiieeee
15:
Update using Vol(0)
16:
end for
Table 8: Overall performance based on Qwen2.5-Math-77 under three different training paradigms using DeepMath dataset (He et al. 2025b). Bold and underline indicate the best and second- results, respectively.
In-Distribution Performance
Dut-offDistribution Performance
Model
AIME MATH-500
Olympiad Avg.
ARC- GPQA" MMLU-PPr Avg.
Unsupervised Methods Trained 8K Samples w/o Any Labels
TTRL
11.6/8.4
74.8
38.7
30.3
self-certainty
11.9/1..2
45.6
74.4
36.4
37.0
35.9
75.9
23.7
36.7
45.4
Token-level Entroyy
13.5/9.3
43.2
71.4
36.0
35.0
34,7
75.9
32.8
39.3
49.3
Sentence-level Entropy
13.6/9.6
50.1
75.6
36.8
37.0
37.1
72.1
28.8
36.9
45.9
Semi-supervised Methods Trained 2K Labeled Samples & 6K Unlabeled Sumples
TTRL
14.1/13.0
48.8
77.8
32.4
37.0
37.2
77.4
27.2
40.1
48.2
Self-certainty
12.8/8.3
45.2
71.6
29.4
32.0
33,2
77.4
28.3
42.9
49.5
Token-level Entroyy
13.8/10.9
74.2
33.1
34.1
77.0
30.8
Sentence-level Entropy
9.6/9.9
45.6
73.8
32.4
34.5
34.3
76.9
28.3
39.8
48.3
TRAPO (ours)
13.8/13.6
51.4
79,8
33.8
40.0
35.4
43.6
52.1
Fully Supervised w/ 8K Labels 16.0/12.1
$2.9
78.8
36.8
42.8
39.9
77.0
29.3
42.7
49.7
33

---

Published as a conference paper at ICLR 2026
Table Overall performance on nine competition-level benchmarks for Qwen-2.5-7B under differ- ent top-p settings, with fixed (0.5) and a fixed warmup length (5). Training was performed with 1K labeled and 3K unlabeled samples.
AIME
MATH-500
Olympiad
Arg.
ARC-c
GPQA
MMI.U-Pro
Arg.
Owen Base
11,5/4.9
31.3
43.6
7.4
15.6
19.0
18.2
11.1
16,9
15.4
Qwen-Instruct
12.5/10.2
48.5
80.4
32.7
41.0
37.6
70.3
24.7
34.1
43.0
top-p
TRAPO
17.9/13.8
58.7
81.4
38.2
45.5
42.6
83.7
37.9
46.8
56.1
top-p = 0.3
TRAPO
16.6/15.7
82.6
79.7
46.7
53.6
lup-p 0.5
TRAPO
15.9/9.5
52.7
79.0
34.2
39.9
38.5
73.2
32.7
45.6
50.5
top-p=
TRAPO
14.9/10.8
53.4
81.8
34,9
41.8
39.6
75.4
36.9
43.8
52.0
top-p = 1.0
TRAPO
14.9/10.7
77.8
72.6
35.4
42.7
50.2
Table 10: Overall performance across nine competition-level benchmarks for Qwen-2.5-7B under varying I' values, with fixed top-p (0.1) and warmup length (5). Training was conducted with 1K labeled samples and 3K unlabeled samples.
Model
AIME 24/25
AMC
MATH-500
Minerva
Olympiad
Arg.
ARC-e
GPQA"
MMLU-Pro
Arg.
Owen- Base
11,5/4.9
31.3
43.6
7.4
15.6
19.0
18.2
11.1
16.9
15.4
Qwen Insttunsttnct
12.5/10.2
80.4
T= 0.1
TRAPO
15.7/10.9
$2.6
81.1
34.5
41.3
39.4
74.0
37.1
43.2
51.4
TRAPO
16.5/12.9
56.8
81.9
37.6
45.9
41.9
81.9
38.1
46.3
55.4
T 0.5
TRAPO
17.9/13.8
81.4
83.7
46.8
I 0.7
TRAPO
14.3/12.7
53.9
79.2
35.1
42.6
39.6
80.6
35.6
43.7
53.3
TRAPO
14.9/13.3
53.9
79.7
34.7
42,1
39.8
81.3
35.9
43.4
$3.5
Table 11: Overall performance across nine competition-level benchmarks for Qwen-2.5-7B under varying warmup lengths, with fixed top-p (0.1) and fixed I' (0.5). Training conducted with IK labeled and 3K unlabeled samples.
Model
AIME 24/25
AMC
MATH-500
Minerva
Olympiad
Arg.
ARC-c
GPQA"
MMLU-Pro
Aug.
Qwen-Base Base
11,5/4.
43.6
15.6
16.9
15.4
Qwen-Instruct
12.5/10.2
48.5
80.4
32.7
41.0
37.6
70.3
24.7
34,1
43.0
wann-up length =
TRAPO
16.1/12.0
77.8
34,0
40.2
39.2
78.5
33.2
41.5
$1.1
up length =
TRAPO
17.4/13.6
57.5
80.2
37.1
43.8
41.6
81.9
36.2
44.8
54.3
warm up length
TRAPO
17.9/13.8
58.7
81.4
38.2
45.5
42.6
83.7
37.9
46.8
56.1
warm-uu length
TRAPO
18.2/14.1
59.3
82.0
38.8
46.1
84.2
38.4
47.3
56.6
warm--up length = /2
TRAPO
17.6/13.5
58.1
80.9
37.7
44.9
42.1
83.1
37.6
46.1
55.6
34

---

Published as a conference paper at ICLR 2026
Table 12: Overall performance of Qwen2.5-Math-78 under different training sample sizes and an- notation ratios.
In-Distribution Performance
Out-of-Distriuutio Performance
Model
AIME 24/25 AMC MATH-500 Minerva Olympiad Avg.
ARC- GPQA" MML.U-Pr Avg.
Original Models
Qwen-Bast Base
11,5/4.9
31.3
43.6
7.4
15.6
19.0
18.2
16.9
15.4
Qwen-Instruct
12.5/10.2
48.5
80.4
32.7
41.0
37.6
70.3
24.7
34.1
43.0
TRAPO Trained Varying Sample Sizes (12.5% Labeled)
TRAPO w/ JK Samples
13.5/10.1
52.3
80.7
39.4
42.2
39.7
75.2
24.1
43.5
47.6
TRAPO 2K Samples
15.0/11.6
53.3
81.2
38.9
44.2
40.7
82.4
28.7
45.2
52.1
TRAPO Samples
16.1/12.9
56.8
82.3
36.7
45.4
41.7
82.1
46.7
54.2
TRAPO w/ 16K Samples
21.3/16.1
60.9
84,8
38.2
43.3
44.1
82.6
39.5
46.2
56.1
TRAPO Trained on Varying Sample Sizes (25% Labeled)
TRAPO w/ /K Samples
17.1/1288
53.6
79.4
39.3
41.5
40.6
72.7
30.3
42,4
48.5
TRAPO Samples
18.1/14.3
55.4
81.6
33.1
41.0
82.6
39.4
55.7
TRAPO Samples
17.9/13.8
58.7
81.4
38.2
45.5
42.6
83.7
37.9
46.8
56.1
TRAPO w/ 16K Samples
24.3/17.1
60.0
84.6
39.3
48.3
45.6
84.6
43.9
50.7
59.7
TRAPO Trained on Varying Sample Sizes (50% Labeled)
TRAPO w/ /K Samples
14.3/10.9
51.7
81.4
34.2
42.1
39.1
78.3
30.1
45.2
51.2
TRAPO Samples
16.2/13.1
54.8
82.3
37.1
41.5
81.5
46.6
54.1
TRAPO 4K Samples
17.3/15.7
59.2
83,9
39.4
47.3
43.8
83.7
36.8
46.6
55.7
TRAPO w/ 16K Samples
24.4/18.3
61.5
84.1
40.8
46.3
45.9
84.2
43.7
49.7
59.2
Fully Supervised w/ 45K Labels
25.1/15.3
62.0
84.4
39.3
46.8
45.5
82.3
40.4
49.3
57.3
Table 13: Qwen-2.5-7B results nine competition benchmarks using 1K labeled and 3K un- labeled samples (30% reliable data selected by Sentence-level Entropy, Self-certainty, and TRAPO).
Model
AIME 24/25
AMC
MATH-500
Minerva
Olympiad
Arg.
ARC-c
GPQA"
MMLU-Pro
Arg.
Owen-Base
11.5/4,9
31.3
43.6
7.4
15.6
19.0
18.2
11.1
16.9
15.4
Qwen-Instruct
12.5/10.2
48.5
80.4
41.0
70.3
24.7
34.1
Random
15.8/12.3
53.5
79.8
34.8
41.8
39.7
80.8
35.8
43,2
53.3
Sentence- level Entropy
16.3/12.5
Self-ccertiigg
15.8/13.3
52.9
80.7
36.6
43.5
40.5
80.4
35.8
42.9
53.0
TRAPO
16.7/13.7
37.4
Table 14: Overall performance on nine competition-level benchmarks for Qwen-2.5-7B using ran- dom selection or TRAPO. Training was conducted with 1K labeled samples and 3K unlabeled sam- ples.
Model
AIME 24/25
AMC
MATH-500
Minerva
Olympiad
Aug.
ARC-e
GPQA
MMLU-Pro
Arg.
Owen-Base
11.5/4.9
31.3
43.6
7.4
15.6
19.0
18.2
16.9
15.4
Qwen-Instruct Instruct
12,5/10,2
48.5
80.4
32.7
41.0
37.6
70,3
24.7
34,1
43.0
No Selection
14.2/13.5
52.6
80.2
34.9
40.9
39.4
76.2
36.4
43.6
52.1
10%
Selected
Random
14.9/13.3
53.9
79.7
34.7
42,1
39.8
80.3
34.9
42.4
52.5
TRAPO
15.8/13.5
80.3
43.2
40.7
81.5
35.8
43.5
53.6
30% Selected
Random
15.8/12.3
53.5
79.8
34,8
41.8
39.7
80.8
35.8
43.2
$3.3
TRAPO
16.7/13.7
81.0
44.6
83.2
45.9
55.5
50% Selecied
Random
14.5/12.8
$1.5
77.2
31.5
40.0
37.9
77.8
34.8
41.8
$1.5
TRAPO
15.1/13.6
54.2
80.5
35.2
42.5
82.0
36.2
43.8
54.0
70%
Selected
Random
14.6/13.0
52.4
78.5
34.0
40.8
38.4
79.2
35.2
42.5
52.3
TRAPO
14.9/13.5
53.8
79.9
34.9
41.9
39.8
81.0
35.4
42.8
53.1
All Selection
14.9/10.7
55.3
77.8
33.1
43.6
39.2
72.6
35.4
42.7
50.2
35

---

Published as a conference paper at ICLR 2026
Table 15: Overall performance across nine competition-leve benchmarks for Qwen-2.5-7B with varying ratios (0))) of unlabeled samples. Training uses 1K labeled samples and 3K unlabeled samples.
Model
ATME 24/25
AMC
MATH-500
Minerva
Olympiad
Avg.
ARC-c
GPQA'
MMLUUPPr
Arg.
Owen-Bas Base
11,5449
31.3
43.6
15.6
19.0
18.2
11.1
16.9
15.4
Qwen-Instruct
12,5/10.2
48.5
8044
32.7
41,0
37.6
70.3
24.7
34.1
43,0
PM 0.00
14.2/13.5
52.6
80.2
34.9
40.9
39.4
76.2
36.4
43.6
52.1
0.25
Token Entropy
16.7/13.6
54.6
81.4
34.3
41.3
40.4
79.6
35,9
44.6
53.4
TRAPO
14.6/13.6
55.4
79.8
35.7
42.1
40.2
81.9
35.4
44.0
53.8
Par 0.50
Token level Entropy TRAPO
15.0/12.4
51.6
79.8
32.7
39.9
38.6
77.3
34.8
42.9
51.7
16.8/13.6
56.2
80.5
38.9
43.6
41.6
82.8
36.6
44,9
54.8
0.75
Token-level Entropy TRAPO
16.2/13.4
52.1
79.0
33.8
39.1
38.9
71.6
29.8
41.3
49.6
17.4/12.9
57.2
80.8
37.5
44.3
41.7
82.5
37.2
45.9
55.2
1.00
Token-level Entropy TRAPO
18.2/11.9 17.9/13.8
53.4
80.2
41.9
40.0
72.9
32.3
49,7
58.7
81.4
38.2
45.5
42.6
83.7
37.9
46.8
56.1
Table 16: Overall performance across nine competition-leeel benchmarks for Qwen-2.5-7B, aver- aged over three runs. Training was performed with IK labeled samples and 3K unlabeled samples.
Model
AIME
AMC
MATHI-500
Minersa
Olympiad
Arg
ARC-c
GPQA*
MMLAU-Pre
Agg
Qwen Dase
11,5
31,3
43.8
7.4
19.0
18.2
11.1
16.9
15.4
Qwes-lustruet
12.5/0022
48.5
90.4
32.7
41.0
37.6
70.3
247
54.1
43.0
TRAPO
18.2 0.37 18.6 0.2
G9.3 0.5
81.9 + 0.4
37.9 0.4
45.8 0.5
42.8 0.4
68.9 06
37.8 =
47.5
36.4 = 0.5
Table 17: Qwen-2.5-7B results on nine competition-level benchmarks using IK labeled and 3K unlabeled samples, with average trajectory matching or maximum trajectory matching.
Model
AIME 24/25
AMC
MATH-500
Minerva
Olympiad
Arg.
ARC=
GPQA"
MMLUU-Pr
Owen Base
11.5/4.9
31.3
43,6
7.4
15.6
19.0
18.2
11.1
16.9
15.4
Qwen
12.5/10.2
80.4
TRAPO-MAA
16.3/9.9
$2.7
80.8
35.6
41.3
39.4
81.6
33.2
42.6
52.5
TRAPO MEAN
17.9/13.8
58.7
81.4
38.2
45.5
42.6
83.7
37.9
46.8
56.1
36


Reviews:
[{"review_id": "R4grus9vo0", "reviewer": "Reviewer_UVjM", "summary": "The paper proposes TRAPO, a semi-supervised variant of reinforcement learning with verifiable rewards (RLVR) designed to improve reasoning in large language models (LLMs) with minimal labeled data. The key idea is to use a small labeled dataset to guide policy optimization on much larger unlabeled corpora. TRAPO aligns unlabeled samples with labeled ones by comparing their pass rate trajectories—the evolution of correct-response rates during training—thereby selecting only reliable unlabeled examples for reward-based updates. This stabilizes learning and prevents the collapse common in unsupervised RLVR. Empirically, TRAPO achieves notable data efficiency: with only 1K labeled and 3K unlabeled examples, it outperforms fully unsupervised methods trained on 45K samples; with 4K labeled and 12K unlabeled, it surpasses even fully supervised models trained on all 45K labeled samples. The framework generalizes across domains and model architectures, offering a principled, efficient path for semi-supervised reasoning enhancement in LLMs", "questions": "(1) Is there a threshold that can adjust the total number of selected questions, i.e, sum of M (SigmaM), it could be better if there are trend curve and ablation that when SigmaM =  0, it shows the full grpo result with increment 0%, when it is 1, it gives marginal improvement like 0.6%, and when using your method, the increment is the optimal like 4.3%, and the performance at other possible values in between.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) My biggest takeaway is that carefully curated mixture of labeled and unlabeled data could improve the performance and solve the unstability issue raised in the unsupervised training.\n\n(2) The paper provides rigorous theoretical analysis, including generalization and convergence proofs linking trajectory consistency with domain adaptation theory.\n\n(3) The paper presentation is good and easy to follow, also comes with comprehensive baselines and benchmarks.", "weaknesses": "(1) My main concern is the practical usage of the method, to my understanding, the method works as a preprocessing data selection step before launching the training on mixture data, what would be the cost and running time for this step? if it needs a long trajectory to determine good samples, is this process even more costly than the rl training itself? Correct me if I am wrong\n\n(2) I am not sure if the roll out pass rate is a 'stable' indicator of this problem, as everytime the roll out could be different, while the model weights keep updating and the selection mask is applied to question.\n\n(3) I guess one baseline, where selecting samples using your methods vs random selection with the same amount, is missing, because increasing the number of unlabeled data could natuarally worsen the training", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "27KwXtd9ML", "reviewer": "Reviewer_aiJH", "summary": "This paper presents TRAPO, a semi-supervised RLVR paradigm. TRAPO measures the similarity between trajectories from unlabeled and labeled data across training epochs and updates the model using only reliable samples. Experiments on the Qwen2.5-7B model demonstrate that TRAPO achieves higher accuracy across several math reasoning benchmarks under limited labeled data settings, and in some cases even surpasses models trained with fully labeled data.", "questions": "Thank you for submitting this interesting and well-written paper. The problem is timely and relevant, and the proposed direction is promising. Below are my specific questions and suggestions for improvement:\n\n- Q1: Would not the ordering of samples within each epoch affect the results? Because the trajectories of each sample highly depends on the model's current capability.  \n\n- Q2: Why is the average trajectory used as a proxy for similarity a good choice? Have the authors considered clustering the unlabeled data first and then computing per-cluster averages for more robust similarity estimation?\n\n- Q3: Since TRAPO evaluates all data each epoch but only updates with part of it, what is the training overhead, and how does it scale with dataset size?\n\n- Q4: Could the authors clarify the expected-answer criteria used in evaluation (e.g., is $a_i = y$ for all $q \\in D_u$)?\n\n- Q5: The evaluation should include more models. While the Llama-3.1 experiments are appreciated, they lack fair baselines comparable to those in Qwen.\n\n- Q6: It would be informative to report results under different labeled data sizes (e.g., 10K, 20K).\n\n- Q7: Please include a sensitivity analysis for hyperparameters $p$ and threshold $T$ to better understand their influence on performance.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Tackles an important and practical problem in semi-supervised RLVR.\n- The proposed approach is overall reasonable and well-motivated.\n- The paper is clearly written and well-organized.\n- Experimental results show promising improvements.", "weaknesses": "- Some design choices require stronger justification (see below).\n- Missing key ablation studies to verify and better understand design components.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "94rXRuGReV", "reviewer": "Reviewer_NfpB", "summary": "TRAPO tackles semi-supervised RL with verifiable rewards (RLVR) for LLM reasoning. Core idea: use a small labeled set to **anchor** training while selecting reliable unlabeled items via **trajectory similarity**—match each unlabeled sample’s pass-rate curve to the averaged labeled trajectory, then include only top-p/above-threshold items in GRPO updates. Experiments on math ID (AIME/AMC/Minerva/MATH-500/Olympiad) and OOD (ARC-c/GPQA*/MMLU-Pro) show notable label-efficiency gains (e.g., 1k L + 3k U outperforming stronger unsupervised methods; 4k L + 12k U rivaling/ surpassing fully supervised on 45k L). Includes an informal generalization bound tying trajectory consistency to target risk.", "questions": "Please address the weakness above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Clean, intuitive mechanism:** “learn from how learning evolves,” not from point estimates. The trajectory-matching mask is simple to implement atop GRPO and robust to domain shift. \n2. **Label efficiency & breadth:** Strong results with tiny labeled sets; both ID and OOD improvements; ablations with OOD unlabeled data are convincing. \n. **Theoretical framing:** A readable (if high-level) bound linking trajectory similarity and confidence to generalization; clarifies the role of labeled anchors.", "weaknesses": "1. **Novelty positioning:** Semi-supervised filtering by “learning dynamics” echoes curriculum/confidence-based selection; paper could contrast more sharply vs. entropy/self-certainty and preference-based filtering beyond empirical gains. \n2. **Proxy fidelity:** Unlabeled “pseudo-pass rates” depend on majority voting; risk of reinforcing easy/short answers remains. Need stress tests where voting is systematically biased. \n3. **Sensitivity & cost:** Top-p/Γ thresholds, warm-up length, rollouts (G=8), batch sizes, β=0, entropy coef=0.01 with no thorough sweep or compute-normalized comparison; H200×8 requirements make label-efficiency claims less practical if compute rises. \n4. **Theory scope:** Bound is informative but rests on simplifying assumptions (e.g., confidence terms, domain discrepancy decomposition); no empirical diagnostics linking bound terms to measured quantities. \n5. **Generalization breadth:** Mostly math-centric; limited non-math/OOD beyond three benchmarks; no noisy-label or adversarial-feedback scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)