FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoning
Authors: Yuyang Ding
Chi Zhang
Juntao Li
Haibin Lin
Xin Liu
Min Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Reasoning
Reinforcement Learning
Robust Learning
Primary area: generative models

Abstract: Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising paradigm for enhancing the reasoning capabilities of 
large language models (LLMs).
In this context, models explore reasoning trajectories and exploit rollouts with correct answers as positive signals for policy optimization.
However, these rollouts might involve flawed patterns such as answer-guessing and jump-in-reasoning.
Such flawed-positive rollouts are rewarded identically to fully correct ones, causing policy models to internalize these unreliable reasoning patterns.
In this work, we first conduct a systematic study of flawed-positive rollouts in RL and find that they enable rapid capability gains during the early optimization stage, while constraining reasoning capability later by reinforcing unreliable patterns.
Building on these insights, we propose **F**lawed-**A**ware **P**olicy **O**ptimization (**FAPO**), which presents a parameter-free reward penalty for flawed-positive rollouts, enabling the policy to leverage them as useful shortcuts in the warm-up stage, securing stable early gains, while gradually shifting optimization toward reliable reasoning in the later refinement stage.
To accurately and comprehensively detect flawed-positive rollouts, we introduce a generative reward model (GenRM) with a process-level reward that precisely localizes reasoning errors.
Experiments show that FAPO is effective in broad domains, improving outcome correctness, process reliability, and training stability without increasing the token budget.

Published as a conference paper at ICLR 2026
FAPO: FLAWED-AWARE POLICY OPTIMIZATION FOR EFFICIENT AND RELIABLE REASONING
Yuyang Ding Soochow University yyding23 stu.suda.edu.cn
Chi Zhang
Juntao Li Soochow University @jt@ suda.edu.cn
ByteDance Seed
zhangehi. http:cchh.usl9992ee bytedance.com
Haibín Lin
Xin Liu
Min Zhang
ByteDance Seed
ByteDance Seed
Soochow University minzhang suda.cdu.cr
haibin.line @ bytedance.com
liuxin.ai @ bytcdance.com
Project Page: https://Eapo- io
ABSTRACT
Reinforcement learning with verifiable rewards (RLVR) has emerged promis- ing paradigm for enhancing the reasoning capabilities of large language models (LLMs). In this context, models explore reasoning trajectories and exploit roll- outs with correct answers as positive signals for policy optimization. However, these rollouts might involve flawed patterns such as answer-guessin and jump- in-reasoning. Such naweddpositiin rollouts are rewarded identically to fully cor- rect ones, causing policy models internalize these unreliable reasoning patterns. In this work, we first conduct systematic study of flawed-positive rollouts in RL and find that they enable rapid capability gains during the early optimization stage, while constraining reasoning capability later by reinforcing unreliable pat- terns. Building on these insights, we propose Flawed-Awar Policy Optimization (FAPO), which presents a parameter-free free reward penalty for flawed-positive roll- outs, enabling the policy to leverage them as useful shortcuts in the warm-up stage, securing stable early gains, while gradually shifting optimization toward reliable reasoning in the later refinement stage. To accurately and comprehensively detect kawed-positive rollouts, we introduce a generative reward model (GenRM) with a process-level reward that precisely localizes reasoning errors. Experiments show that FAPO effective in broad domains, improving outcome correctness, process reliability, and training stability without increasing the token budget.
INTRODUCTION
Large language models (LLMs) with strong reasoning capabilities, such OpenAl o-series (Ope- nAI 2024 2025). ). Deepseek RI (Guo et al. 2025). have sparked significant attention in reinforce-
ment learning with verifiable rewards (RLVR) (Shao al., 2024). In this paradigm, models timized through rule-based outcome rewards, typically a binary signal indicating whether the final answer correct, in verifiable tasks like mathematical reasoning (Yu al., 2025 Team et al.) 2025) and code generation (Xiaomi et al.) 2025). During RL training, the model explores diverse reasoning trajectories and exploits those with correct final answers as positive signals for policy optimization.
op-
This exploration-exploiaaee planning, which in turn facilitate generalization across a wide range of domains (Huan et al. 2025).
paradigm enables LLMs to evolve strong reasoning behaviors, such as
However, certain flawed reasoning patterns could also be reinforced during policy optimization. Recent studies (Zheng et al., 2024, Kalai et al. 2025) have revealed notable flawed reasoning pat- terns in current LLMs, such as answer susser-gusssing and jump-in-raasoning (Wang al. 2025). where models reach correct answers through shortcuts. This presents a fundamental challenge for RLVR, ie., rule-based outcome rewards assign identical positive signals to both flawed- positives and fully correct rollouts, thereby potentially reinforcing unreliable reasoning. This raises an urgent need for (1) analyzing the distribution and impact of flawed positives throughout the RL process, and (2) developing effective mitigation strategies to ensure efficient and reliable reasoning.
"orresponding author

---

Published as a conference paper at ICLR 2026
Flawed Positive Ratio
AIM124 (Avo61332
AIME25 (Avg@332)
GPQA-Diamon Diamond (Avg@-4)
35.0
Avg. Toltnnn 15555
Aug. Tokess 2778
32.5
35-
30.0
30
52
27.5
23-
-Aog: Tolanns --657
25
Aug. Tokezs 2900
Arg. Tokens 1390
Percom
25.0
22.5
15-
46
17.5
Brsalinn-8
JAPO-7B (Oumr)
5-
FAPO-7IB (Ours)
Busellie-
10
BAPO-32B (Ours)
Raseliin-225
42
Ruselinn-322
PAPO-328 (Oum)
0
Training
Truining Stops
Figure 1: Flawed-positive ratio and performance comparison between FAPO models and baselines.
To this end, we first conduct a preliminary study to investigate the prevalence and impact of flawed positives in the RL process. Our findings indicate that flawed positives persist steadily throughout training: (1) in the early stages, when models are not yet capable of producing fully correct roll- outs, flawed positives serve shortcuts to correct answers, accelerating capability gains; and (2) once the model can generate fully correct rollouts, these flawed positives may hinder learning by reinforcing unreliable reasoning patterns. Thus, the optimal role of flawed positives is to act as step- ping stones toward reliable reasoning. Building on these insights, we propose Flawed-Aware Policy Optimization (FAPO), which presents parameter- free reward penalty to flawed-pond seweed-ossiive rollouts. FAPO establishes a natural self-exploration learning trajectory: the model initially exploits flawed positives for knowledge, but as its capabilities advance, the training objective gradually shifts toward genuine problem-solving. improving both the training efficiency and reasoning reliability. Further- more, to accurately and comprehensively detect these flawed positives, we introduce a generative reward model with process reward, which enables the model to locate intermediate process errors.
Experimental results highlight the strong potential of FAPO. For flawed positive detection, our trained model, FAPO- FAPO-GenRMM GenRMM4B, achieves substantial gains both newly constructed bench- mark, FlawedPositiveBencch and the public ProcessBench (Zheng et al.] 2024). When integrated into the final RL process (results in Figure D]. FAPO effectively penalizes flawed-positive rollouts, reducing unreliable reasoning patterns (left subfigure, the decreasing flawed positive ratio), while delivering remarkable improvements across AIME24, AIME25, and 2024) (other three subfigures), with clear advantages at nearly all intermediate evaluation check- points. Overall, FAPO offers clear advantages: enhances outcome correctness, improves process reliability, and training efficiency and stability, all without increasing the token budget.
(Rein al.,
2
PRELIMINARY: UNVEILING FLAWED POSITIVES IN RL TRAINING
2.1 PROBLEM DEFINITION AND MOTIVATION
Group Relative Policy Optimization (GRPO) GRPO (Shao et al.. 2024 an efficient policy gradient method for LLM reinforcement learning that estimates advantages in group-relative man- ner without relying on a learned value model. For a given question q, the behavior policy generates group of G rollouts {a: evaluates their rewards {Ra je and normalizes them to obtain per-token advantage estimates Zrei
A mcan (R std( R. )
(1)
The policy model is then updated by maximizing the following clipped surrogate objective:
TaRPO (0) = E (aoa) (-
lodl
To(oslg,0ss) A clip( Towa (or (4,
To(o:lg, ocs) Th (0x oct) 1- 1,,,,++)44, }}
(2)
min
(c)) a.t))
where (g. a) denotes a question-ansee aaniior pair sampled from the data distribution D. Tei is the old policy, and controls the clipping range in importance sampling (Schulman al.] 2017) for stability.
2

---

Published as a conference paper at ICLR 2026
In this work, we adopt several effective strategies such clip-higher, token-level loss, and overlong reward shaping Yu et al. 2025), to ensure stable and efficient policy optimization.
[(0) B(g,a) (o
0.1
To To(or 02t) Tiuu (0r Oct))
Ai, clip( To mo(orlg, el, (eeeee))aa en) 1 Til (or (1, 0ct))
(3)
min
Σ los|
In these algorithms, the reward R is the primary supervision signal that guides the policy optimiza- tion, and existing RLVR approaches (Yang et al. 2025 Liu et al. (2025b) commonly employ rule-based outcome reward to mitigate reward hacking (Gao et al. 2023 Weng 2024), i.c.
11,
I(o,a")
BRIVR- = Raute aa) )
(4)
-1, Otherwise
where I(o, a²) is an indicator function that returns True if the predicted answer extracted from rollout matches the ground-truth answer a", and False otherwise.
Flawed Positive Issues Recent studies (Zheng et al. 2024 Zhang et al. 2025b) have identified notable nawed-positivn issues current LLMs, some cases even accounting for ratio of up to 50%, where models may reach correct final answers through unreliable reasoning patterns such as answer-guessing and jump-in-reasoning (Wang et al. 2025). This poses fundamental challenge for reinforcement learning: rule-based reward functions assign positive signals to newed-positivg roll- outs, thereby reinforcing unreliable reasoning patterns and ultimately limiting the model's 's perfor- mance ceiling. Formally, given question 9 and reasoning trajectory by policy æ, with predicted answer åz, the rollout is flawed positive if
[x1, .,,,..,, generated
å₂ a" and 3 t 21, .......................... n) s.t. step 2, is logically invalid.
(5)
While prior works have primarily revealed the prevalence of these issues in benchmark evaluations, their underlying mechanisms and impact on the RL process remain largely underexplored.
2.2 FLAWED POSITIVE ANALYSIS IN REINFORCEMENT LEARNING
Flawed Positives are Prevalent in Initial Checkpoints We first examine flawed positive issues in current LLMs, as they establish the starting conditions for subsequent RL optimization. We evaluate three representative models: Qwen2.5-Mathh --aa-----aa---BBBae (Yang et al.) 2024b). Llama3.3-70B- Instruct (Dubey al., 2024h. and Qwen3-1.7B (Yang et al., (025). on the DAPO-Math dataset. Following Zhang et al. (2025)), we employ Qwen3-32B to determine whether the reasoning trajec- tory contains unreliable reasoning patterns. As shown Figure (a). flawed positives are prevalent across various LLMs, accounting for 20%-40% of correct rollouts, highlighting the severity of this issue. Beyond the automatic L.LM-as-a-judge evaluation, we also conduct a manual case study of flawed-positive samples and analyze their underlying causes, which are provided in Appendix G
Flawed Positives are Stepping Stones in Learning Reinforcement learning is often formulated as an end-to-end optimization process driven by self-exploration towards self ceff-improvement. To better understand this process, and in particular the role of flawed-positive rollouts, we design a simulated experiment in which the learning stage of each sample approximated by its rollout accuracy. Specifically, we use a model to generate multiple rollouts per sample, compute the corresponding rollout accuracy, and then group all samples into different learning stages, as illustrated in Figure (b). The results reveal clear trend: flawed positives are most prevalent during the early learning stages but diminish significantly training progresses. This highlights their expected role natural stepping stones in the learning trajectory, allowing the model to initially reach correct answers before gradually evolving the capability to produce fully correct solutions.
Flawed Positives Persist and Exert Twofold Effects We further train pre-trained model, Owen2.5-Math-7B, with RL on DAPO-Math, and track its learning trajectory, as shown in Figure 2 (c). While the model's rollout accuracy steadily improves, the flawed flawed-positive ratio remains almost constant at around 30%. This indicates that the optimization process struggles to shift from unre- liable reasoning genuine problem-solving. A major concern is that flawed positives receive the
3

---

Published as a conference paper at ICLR 2026
(a) Across
Flawed-Positive Troining
70-
Accuracy (Pesttive) Rate
70-
Rollout Acquracy Rate
60-
Plaveed Pooilive Rate
Fluwad Pusitivn Rata
Mmm
50
Perentage
60-
S41
41.3
40-
20-
mmm
1E2
16.5
10
10
Oweea2.5 Sweaa25-Mtth-77 Math-7B
Uanaa 5 768 lastrut
Queai 78 Mailing
25
50
75
100
125
150
175
200
Free--rained Model
(Noo fhinking Model
Training Stups
Fosttives Mitigation Training
1.mmm3.3-70--nma
Qwnn8-1.7 (Thini)
70-
0
30
Foetore Itato
60-
47
62
25
50-
64
20
60-
4
:0-
Basebne-778 Posttives
20
Trial-78 (Flawed Positives Negativas)
FAPO-7B (Oun, Adaptive Strutegs)
10
1/0
2/8
3/8
4.3
5/8
6/8
7/B
8/8
25 50
75
100
125
150
175
200
Modells Pee-sample Conniieence (m Comnent / TTobal Trialss)
Training Staps
Figure 2: Preliminary experiment results of flawed positives.
same rewards correct solutions, thereby reinforcing unreliable reasoning patterns and hindering progress. To further explore this, we conduct preliminary trials using Qwen3-32B to detect flawed positives and assign them negative signals (same as negative rollouts) during training. Figure 2 (d) reports performance on AIME24 throughout the training process. Compared with the baseline RLVR setting (blue line), penalizing flawed positives (orange line) yields significant performance gains, though improvements emerge more gradually in the early stages. From the above findings, we find flawed positives persist throughout training, and exert a twofold effect: (1) flawed positives act stepping stones, enabling the model to achieve rapid capability gains in the early stages, and (2) their improper reward assignment can trap optimization in unreliable reasoning.
3 FAPO: FLAWED-AWARD POLICY OPTIMIZATION
Building on these insights, we propose Flawed-A ware Policy Optimization (FAPO) algorithm. For flawed-positive detection, directly employing a strong LLM like Qwen3-32B impractical and computationally inefficient. Instead, we propose an RL algorithm to train a compact yet effective generative reward model (GenRM). We then present an adaptive learning algorithm that dynamically adjusts advantage assignment towards the current suitable optimization direction.
3.1 FLAWED POSITIVE DETECTION
To identify a suitable LLM that detect flawed positives both effectively and efficiently, construct evaluation set, Flawed- PosiliveBench, by collecting positive samples (including flawed ones) from ProcessBench (Zheng
Evaluating Flawed Positive Detection Capabilities LLMs
al. 2024). We then quantify the detection capability with the following three metrics:
# ソ" =P]
# (jo
- = FP)
2
precision
recall
F 1/precision + 1 recallll (6)
# # FP)
= FP]
where jo indicates whether the judge model 0 predicts response flawed positive (FP), and the ground-truth label. Precision reflects the correctness of FP predictions, recall measures the coverage of true FPs, and the Fi score provides balanced summary of both. As shown in Figure 3 we observe that many models, such as Qwen3-4B-B 4B------ttrruuc (Yang et al., 2025) and Awen2.5-Mathh PRM-72B (Zhang al., 2025b), exhibit an over-critic phenomenon: they achieve high recall but suffer from low precision. Closer inspection reveals that these models often overemphasize minor unnecessary emors like unsimplified fractions. Overall, lightweight models struggle provide
4

---

Published as a conference paper at ICLR 2026
FAPO-GeaRM-IB (Ours)
Qwen3-32B (Teacher Model)
Qwen3 AB-Insttuctt2500 (Base Model)
Qwen2.5- Math Math-FRM----- FRM-72B (Distriminative SoIA)
(409)
37.0
:503
89.4
83.8
$2.0
81.6
74.8
G9.4
ProcessBench (Avg. F1)
FlawedPos (P1)
FlawodPositiveBen (Precision)
Flawe Bench (Rwcall):
Token Usage (Ay.. Toleens)
Figure Performance of current state-of-the-art (SoTA) generative models and diseriminative PRMs. Detailed subset-level results and additional models are reported in Table
appropriate criticisms to detect flawed positives, while stronger models achieve better accuracy but remain impractical for online RL due to slow inference. These findings suggest that existing models are not well-aligned in both detecting capabilities and inference efficiency.
Enhancing Detection Capabilities via Step-wise RL Optimization To enhance the detection capability, we introduce a step-wise RL reward formulation. Starting from a generative model, we develop the following RL strategies for training generative reward model (GenRM):
GEARO Routcome +R
11,
If jo =y Rproeess
lia-
If De y FP Otherwise
(7)
where Routeome
1-, Otherwise
0,
Building upon the outcome reward, we introduce a step-wise penalty, RPmceess fine-grainer and step-wise optimization. Here, to and t" denote the predicted and ground-truth error indices, and nawed-positine cases, the penalty is
the total number of steps, ensuring Rpreess E [],,0] distance-sensitive predictions closer to the true error receive higher rewards, while those farther away incur stronger penalties. This design guides the model toward precise error localization and fosters genuine error-detection ability, rather than mere guessing, based on the key points:
Learning beyond guessing: Flawed- Flawed-positive rollouts also exist in the error detection task, partic- ularly in the yes/no setting, where the model can often guess the label without truly identifying errors. Such guessing rollouts offer little optimization benefit. To mitigate this, we introduce the penalized step reward that guides the model toward genuine critic capabilities rather than guessing. Natural reward shift: early training. the reward design naturally emphasizes prediction cor- rectness, improving Raaseline yields substantial gains (-1 1), whereas Rpncess provides only limited gains (Rprocess E (-1,0) O)). As correctness saturates, process optimization becomes increas- ingly prioritized. This enables a smooth transition without additional controlling hyperparameters.
3.2 FLAWED POSITIVE PENALIZATION
With the GenRM detecting flawed positives, we then regulate their roles in the final RL optimization. As discussed in Section 2.2, flawed positives should ideally facilitate rapid warm-up and then be penalized to enable genuine problem-solving. However, the key challenge lies in how to balance their encouragement and suppression and when shift between these roles. To address this, we introduce a reward-penn wwward-penalization mechanism with group-relative advantage estimation:
ReApo(o, a" (0) (RLVR(O, (or,oo,,a"""
+R) (o, a" (0).
where RA (0, -x,, If ((o,a") and (o(ooaa) = FP
00 Otherwise
A, = [r. - mean( ( [ /std(
where RRIVR denotes the standard baseline (defined Equation 4). and À controls the penalization strength. To better characterize the entire learning dynamics of FAPO, provide theoretical
5

---

Published as a conference paper at ICLR 2026
analysis in Appendix A which demonstrates how FAPO enables the natural optimization shift while further stabilizing the RL training process. Concretely, when the current rollout stage contains a pro- portion of positive samples and / proportion of negative samples, the optimization shifts from the warm-up stage to the refinement stage once the learning progress p 3 reaches - 1. Moreover, the estimated advantage for positive samples becomes
as optimization continues, when p x downscaled, making the optimization more stable. In the process, the value of À determines the timing of this optimization shift. We adopt majority-gun -uuidityygdddddddeoo strategy, where the optimization direc- tion is determined by whether positive or negative samples dominate. This majority-guided strategy yields Puhin = 1, further detemining À 1. We set À = as the default setting. Overall, FAPO provides a principled mechanism for guiding the optimization process, aligning with the ideal learn- ing trajectory where the focus initially lies in producing correct solutions when model capability is limited, and naturally shifts toward refining reliability once correct rollouts surpass incorrect ones.
4 EXPERIMENTS
4.1 TRAINING DETAILS
In this work, we validate the effectiveness of FAPO on Qwen2.5-Math-7B (Yang et al., 2024b) and Qwen2.5-32B (Yang et al. 20024)). We adopt GRPO (Shao et al. (2024) with several commonly used strategies, including clip-kigher, token-level loss, and overlong reward shaping (Yu et al.) 2025 as our baseline algorithm. Notably, FAPO can be easily transferred to any other RLVR method as a drop-in replacement for rule-based outcome rewards. We conduct RL training using verl frame- work (Sheng et al.) (025), and develop an asynchronous architecture that decouples rollout inference and generative reward modeling, which substantially improves training efficiency.
To train the GenRM model via reinforcement learning, we construct a flawed- positive dataset, FAPO-Critic-85K, To ensure broad coverage, we employ a series of models from the LLaMA and Qwen families, ranging from 7B to 70B, to generate multiple responses to questions drawn from DAPO-Math-17K (Yu et al. 2025). Based on these responses, we select the samples with correct final answers and then employ Qwen3-32B to identify the inherent step-level error location. This yields the final process-crror dataset: DRAPO-Cati = {(%, r,, fx) the first error index of response rú, and and fully correct responses are included with t; +oo for convenience. This dataset is then used to train Qwen3-4B 4B--------nstccc (Yang et al. (025) with the reward defined in Equation 7 and additional hyperparameter settings are provided Appendix B
FAPO-GenRM
N t+)] where t, denotes
PAPO-Reasoning The trained critic model, J.PO-GenRM-4B. is then used to detect and penalize flawed positives in reinforcement learning for reasoning tasks, optimized with the reward defined in Equation 8 In practice, we deploy the GenRM as an external LLM service on a computing cluster, where process rewards are obtained via remote API requests during RL training. To ensure effi- ciency, we launch multiple server workers and employ router to distribute requests with balanced load across workers. This decoupled design enables asynchronous interaction between GenRM and other RL components, substantially improving training efficiency and making the integration of GenRM into large-scale RL training practically feasible. Further details on infrastructure design and hyperparameter configurations are provided in Section and Appendix
B respectively.
4.2 EVALUATION SETUP
Slaee-PPostiive Detection We primarily evaluate GenRM on FlawedPositiveeennch whose con- struction procedure and evaluation metrics are detailed in Section 3.1 as this benchmark directly aligns with our research purpose. addition, we also include ProcessBench (Zheng et al., 2024). which focuses recognizing fully correct samples and precisely locating errors in incorrect sponses. We report the harmonic mcan of the accuracies on correct and erroncous samples. Further- more, incorporate several state-of-the-an (SoTA) discriminative and generative models strong baselines for comprehensive comparison.
Reasoning Evaluation We conduct comprehensive evaluation covering AIME24 (Math), AIME25 (Math), and GPQA-Diamond (General Domain) (Rein et al,, 2024). Rather than lim- iting the analysis to single selected checkpoint, present all intermediate evaluation outcomes

---

Published as a conference paper at ICLR 2026
llawedPosttveelench (F1)
FlwwdPPsttivellenc (Prectsion)
ImsedPosttiveleen (Recall)
Rollout Length (a)
4000
Checkpoint
3750
Parcenta (%)
3000
3250
86
3000
13
82-
2500
GenORM-4B
GenORM-4B
GenORM--B
GenORM-4B
86
CAPO-GraIDE-48
EAPO-GenIMM--B
360
100
260
300
100
200
300
100
260
300
ATME24 (2xg832)
AIME25
GPOA-Diamond
Flawed Positive Ratão
36
Twneiin-322 TAPO-32B
54
Verconage 06)
15
38.1
44-
PAPO-32B (Durr)
Baseline 32B
10
PAPO-328 (Chas))
Bosclite 328
42
Baseline 22B
KAPO:328 (Ours)
3
100 200
300
600
300
120
2000
300
400
503
130
200
500
400
500
AUNICEE
Truitnng Staps
Trunnng Staps
Training Stage
Figure 4: Performance FAPO-GenRM and FAPO- Reasonnn during training. Top row: compari- son between FAPO-GenR.M and the baseline outcome reward models (setup in Equation 7b. Bottom row: comparison between results in broader domain be seen in Table 2 and Table
and the baseline setting (setup in Equation 8). Detailed
throughout the RL process. This only illustrates the performance gains achieved during the train- ing process but also highlights the stability and scalability of the optimization procedure, thereby providing stronger evidence of the effectiveness and robustness of approach.
4.3
MAIN RESULTS
The top row of Siguure@illusraatee the training dynamics of FAPO- GenRM. The model exhibits sigmificant performance gains in the early stages and continues to im- prove as training progresses. For subsequent use, however, we select an early checkpoint, as it delivers strong results with shorter responses, which crucial for maintaining efficiency when tegrating GenRM into RL training. Figure 3 compares our trained model against state-of-the-art discriminative and generative baselines. Built upon Dwwen3-4BIInstruct our approach achieves sub- stantial improvements on both FlawedPositiveBench and ProcessBench, even outperforming the teacher model Qwen3-32B, further demonstrating the effectiveness of our approach. Additional results of our model and other strong baselines be checked in Table
EAPO-GenRM Performance
Pea-onngg Performance Figure T and the bottom row of Figure summarize the overall performance of the FAPO reasoning models, which can be highlighted in the following aspects:
Outcome Correctness: Across benchmarks, FAPO consistently maintains a clear advantage over the baselines in both mathematical and general-domain tasks, demonstrating that detecting and penalizing flawed positives leads to broad improvements in problem-solving ability.
Process Reliability: We also measure the proportion of flawed positives. The results show that FAPO responses exhibit a substantially lower fawed-positive ratio. Beyond the LLM-assa-juuee approach using Qwen3-32B,
also launch manual verification of unreliable reasoning patterns, with details and results in Table 6 demonstrating the effectiveness of FAPO.
Training Stability: By mitigating the impact of flawed positives, training stability significantly enhanced. The overall learning curves are smoother, and unlike the baselines, FAPO does not exhibit notable performance drop the later stages of training.
Token Budget: The improvements from FAPO do not require longer responses. While prior work (Luo et al. 2025 An et al.] (025)) has shown that scaling up response length can yield substantial gains, FAPO achieves improvements without relying on this factor.
4.4 ABLATION STUDY

---

Published as a conference paper at ICLR 2026
Effectiveness of FAPO-GenRM method Previous re-
AIME24 (Avg@32)
sults have already demonstrated that our GenRM design achieves sfate-of-the-aur (SoTA) performance the error detection task (see Figure 3 Here, we further evaluate its impact the final RI. process. Figure compares
35
Accuracy (9)
30
FAPO-GenRM with the base model Bewwen-44-Inntrruee during RL training. The results indicate that stronger de- tection capability ultimately translates into improved per- formance. This highlights two key points: (1) the pro- posed FlawedPositive
20
Buseline- 7B (w/o GenRM)
15
TAPQ-7B
10
(w/ Baseline GenRM)
provides a reliable measure of detection ability that is well aligned with final perfor- mance, and (2) flawed positive detection plays crucial role, where even small improvements can yield substan- tial performance gains.
FAPO-7B
5-
(w/ FAPO-GenRM)
50
100
150
200
Training Stops
Figure 5: GenRM effectiveness.
Impact of Self-Correction Capability Self-correction an important evolved mechanism in RI., allowing rea- soning models to recover from initial mistakes and even-
Baseline-7B
1400
FAPO-7B (Ours)
Rollout Length (Tokens)
AIME24 Accuracy (%)
tually reach correct answers. Nevertheless, when cor-
1300
rectness depends heavily on long rollouts, it could also
30
1200
be regarded as a form of flawed positives. While self-
correction facilitates progress in the early stages of learn- ing, becomes less desirable once the model can solve problems directly, where fully correct rollouts not only ensure reliability but also promote more efficient reason- ing. To illustrate this phenomenon, Figure 6 visualizes rollout length during training: both FAPO and the base- line initially depend self-correction, but over time, FAPO shifts toward fully correct rollouts, resulting in shorter rollouts, more efficient reasoning, and consistent performance gains. These results indi- cate that FAPO preserves the benefit of learning from self-corrected rollouts at the beginning, but gradually shifts toward prioritizing fully correct rollouts later stages.
1100
1000
900
0
50
100
150
200
Training Staps
Figure 6: Self-correction Analysis.
4.5 DISCUSSION: GENRM APPLICATION IN FUTURE RL SYSTEMS
Introducing generative reward models (GenRMs) may have considerable impact the whole RL process, influencing both algorithmic effectiveness and infrastructure efficiency. In this section, we discuss the application potential of GenRMs (with FAPO considering perspectives from both algorithmic development and infrastructure design.
example) in future RL systems,
Algorithmic Challenge: Reward Hacking The pri- mary algorithmic challenge of deploying GenRM in large-scale RL systems lies in reward hacking, where the policy exploits imperfections in the reward signal to achieve high scores without genuinely performing the intended reasoning task. This issue is especially pro- nounced with complex, fine-grained reward signals pro- duced by reward models, as they provide more opportuni- ties for the policy to discover shortcuts for maximizing re- wards. For example, experiment with process-based reward that assigns scores according to the proportion of correct steps before the first detected error. However, this design results in a form of reward hacking: the model tends output only those reasoning steps in which it has very high confidence, while skipping uncertain ones altogether. As shown in Figure although the step-ratio reward delivers some capability gains the early stage compared the baseline, the sub- sequent progress stalls due to reward hacking. This behavior causes an obvious jump- comp-i--reasonngg phenomenon, which undesirable as undermines the reliability of the reasoning process.
AIME24 (Avg@32)
40
30
Accuracy
25
15
Baseline-7B
10:
PAPO-7B (SsspReward)
FAPO-78
100
150
Training Steps
Figure 7: Step reward ablation.
8

---

Published as a conference paper at ICLR 2026
Infrastructure Challenge: Long-tail Problem
Rodnot Inder
GenRM Inter
Actoe Train
IoLa: time
The long-tail problem has long been a key bottle- neck in scaling large-scale RL systems (Liang et al.] 2018). where GPUs often remain idle during the generation of long-tail samples. A concern with GenRM is that it introduces an additional genera- tion stage, further aggravating this inefficiency. To make FAPO practical for large-scale RL systems, we adopt the following efforts: (1) Asynchronous de- sign: We decouple GenRM from rollout inference and actor training, reducing GPU idle time. While a fully synchronous design may offer a better system- level solution (Fu ct al. 2025). this lies beyond the primary research focus of our work, and we leave this as an important future work. (2) GenRM training: FAPO employs an overlong reward strategy in training and treats token budget as key criterion in final checkpoint selection. With these efforts, the training time of FAPO is increased by less than 20% relative to the baseline.
GPU-1
6PU-2
Long Tall Sumple
GPU-1
GPU-2
Decouple GenRN
from Rollout Actor
GPU-3
|| [7
GPU-1
12
GPU-2
Pully-Aaync Aaync
GPU-4
pesssniky
125567A4
Figure 8: GenRM infrastructure design.
5 RELATED WORK
LLM Reinforcement Learning Reinforcement learning has emerged as a promising paradigm for advancing LLM reasoning capabilities (Xu et al., 2025a Zhang et al. 2025a), with the long-term
vision of Artificial Superintelligence. Recent milestones Guoo et al. 2025 OpenAl 2025) demon- strate that RL in verifiable tasks (Liu et al. 2025c He et al.
2025| Feng et al. (025) enables models
to develop reasoning patterns such as planning and tool usage, which in turn foster generalization across broader domains (Huan et al. 2025 Seed et al. 2025). While verifiable rewards provide only binary feedback, learning progresses gradually. Building on this direction, we explore the role of flawed positives this process and introduce the soweed-Aware Policy Optimization (FAPO) algorithm, which promotes a more natural learning trajectory and enables efficient and reliable RL.
Reward Models in Reinforcement Learning Reward models offer a promising approach to ad- dressing the limitations of rule-based rewards, which can be broadly categorized into generative (GenRMs) and discriminative reward models (DisRMs). Current GenRMs mainly serve as flexible verifiers (Xu et al., 2025b Chen et al. 2025) that augment rule-based systems when correctness cannot be reliably assessed by predefined rules (Liu et al.) 2025a Zhao et al. 2025). In addition to verifiable reasoning tasks, GenRMs are also applied in non- verifiabee tasks, providing subjective and rubric-based rewards (Mahan et al. 2024 /Zhang et al. 2024 /hou et al. 2025). In contrast, DisRMs typically output fine-grained, dense rewards at every decision step, such as token-level (Rafailov et al.] [023] Cui et al. (025). seep- (Lightman et al. 2023 Wang et al. 2023 Ding et al.55 and turn-level (Qian et al. [025 Dong et al. (025)). to support more precise optimization. However, the complexity of such dense rewards also makes them vulnerable to reward hacking (Gao et al. 2023), as policies may exploit spurious shorteuts instead of learning the intended behaviors (Weng, 2024.. limiting their application in large RL systems. To address this challenge, FAPO introduces an interpretable framework that trains GenRMs to detect flawed positives and provide nuanced, in- terpretable rewards. Both empirical experiments (Appendix 4 and theoretical analysis (Section A demonstrate that FAPO exhibits strong robustness against reward hacking and scales effectively.
6 CONCLUSION
This paper introduces the Flawed-Aware Policy Optimization (FAPO) algorithm to enhance effi ciency and reliability in LLM RL. We first reveal the role of flawed-positive rollouts: they accelerate capability gains in the early stage but ultimately constrain reasoning quality by reinforcing unreliable patterns. To reconcile this off FAPO applies a parameter- free reward adjustment that lever- ages flawed positives as shortcuts during warm-up up while progressively steering optimization toward reliable reasoning. In addition, we develop a generative reward model (GenRM) with process-level signals to accurately detect and localize reasoning errors. Both empirical experiments and theoreti- cal analysis demonstrate the effectiveness of FAPO in future large-scale RL systems.

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We confirm that this work adheres to ethical research practices. All data and LLMs used are publicly available (including API access) and properly cited, with involvement of human subjects. The Use of LLM statement is illustrated in Appendix E
REPRODUCIBILITY STATEMENT
We have made every effort to ensure the reproducibility of the results reported in this paper. Details of the algorithm design provided in Section []] while the infrastructure setup is described in Section Additional information, including the dataset, training hyperparameters, and evaluation setups, presented in Section 4 and Appendix
ACKNOWLEDGEMENT
We want thank all the anonymous reviewers for their valuable comments. This work sup- ported by the National Science Foundation of China (NSFC No. 62576232) and the Young Elite Scientists Sponsorship Program by CAST (2023QNRC001).
REFERENCES
Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong, Polaris: A post-training recipe for scal- ing reinforcement learning advanced reasoning models, 2025. URL https: //kkulp/
github. htt.bb...//boog/2///lllsee
Ding Chen, Qingchen Yu, Pengyuan Wang. Wentao Zhang, Bo Tang, Feiyu Xiong, Xinchi Li, Minchuan Yang, and Zhiyu Li. xverify: Efficient answer verifier for reasoning model evalua- tions. arXiv preprint arXiv:2504. 10481, 2025.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025.
Yuyang Ding, Xinyu Shi, Juntao Li, Xiaobo Liang, Zhaopeng Tu, and Min Zhang. Scan: Self-denoising monte carlo annotation for robust process reward learning. arXiv preprint arXiv:2509.16544 2025.
Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, et al. Agentic reinforced policy optimization. arXiv preprint 25xv:2507.19844 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2400 -407, 2024.
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504.11536 2025.
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, al. Areal: A large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint arXiv:2505.24298. 2025.
Lco Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference Machine Learning, pp. 10835-10866, PMLR, 2023.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501. 12948, 2025.
10

---

Published as a conference paper at ICLR 2026
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et al. Skywork open reasoner technical report. arXiv preprint arXiv:2505.22312, 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt, Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Pooven- dran, Graham Neubig, and Xiang Yue. Does math reasoning improve general Ilm capabilities? understanding transferability of Ilm reasoning. arXiv preprint arXiv:2507.0044: 2025.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint vrXiv:2403.07974 2024.
Adam Tauman Kalai, Ofrr Nachum, Santosh Vempala, and Edwin Zhang. Why language models hallucinate. arXiv preprint arXiv:2509.04664 2025.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gon- zalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. In International conference machine learning, pp. 3053-3062. PMLR, 2018.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Shudong Liu, Hongwei Liu, Junnan Liu, Linchen Xiao, Songyang Gao, Chengqi Lyu, Yuzhe Gu, Wenwei Zhang, Derek F Wong, Songyang Zhang, et al. Compassverifier: A unified and robust verifier for lims evaluation and outcome reward. arXiv preprint S.Xiv:2508.03666 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint ExXiv:5533.2088, 2025b.
Zihan Liu, Zhuolin Yang, Yang Chen, Chankyu Lee, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Accreason-nemotroon 1.1: Advancing math and code reasoning through sft and rl synergy. arXiv preprint arXiv:2506. 13284, 2025c.
Michael Luo, Sijum Tan, Justin Wong. Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa. and lon Stoica. Deepscaler: Surpassing ol o1-preiiew with a 1.5b model by scaling rl, 2025. Notion Blog.
Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprint arXiv:2410. 12832, 2024.
OpenAl Learning to reason with 1lss. Learning-to-reasoon wete-wiihh-iie 1lms/ 2024.
ht:p:://penal..om/nneee
OpenAl. Introducing openai o3 and 04-mini. https://opeaa..comnnees 2025.
Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. Toolrl: Reward all tool learning needs. arXiv preprint arXiv:2504. 13958, 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 11155772-53774 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-prod q&a bench- mark. In Fürst Conference Language Modeling, 2024.

---

Published as a conference paper at ICLR 2026
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
ByteDance Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, et al. Seed1. 5-thinking: Advancing superb rea- soning models with reinforcement learning. arXiv preprint arXiv:2504. 2025.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang. YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp- 1279-1297, 2025.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501. 12599, 2025.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. arXiv preprint arXiv:2312.008933 2023.
Yu Wang, Nan Yang. Liang Wang, and Furu Wei. Examining false positives under inference scaling for mathematical reasoning. arXiv preprint arXiv:2502.06217, 2025.
Lilian Weng. Reward hacking in reinforcement learning. Bllianwengggihhbbii Nov 2024. URL
github......................... http::////LLeeeeeeng.g.......o.d httpossss/024----...... 111224411-00 1e----eward-eeeeee hacking/
LLM Xiaomi, Bingquan Xia, Bowen Shen, Dawei Zhu, Di Zhang, Gang Wang. Hailin Zhang, Huaqiu Liu, Jiebao Xiao, Jinhao Dong, al. Mimo: Unlocking the reasoning potential of lan- guage model-from pretraining to posttraining. arXiv preprint arXiv:2505.07608, 2025.
Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, al. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprint arXiv:2501. 09686, 2025a.
Zhangchen Xu, Yuetai Li, Fengqing Jiang, Bhaskar Ramasubramanian, Luyao Niu, Bill Yuchen Lin, and Radha Poovendran. Tinyv: Reducing false negatives in verification improves for 11m reasoning, arXiv preprint arXiv:2505. 14625, 2025b.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self- eeff-improvement. arXiv preprint 12Xx:22499.12222 2024b.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint MxXiv:5505.09388 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. /4476, 2025.
Kaiyan Zhang, Yuxin Zuo, Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jia, Pengfei Li, et al. A survey of reinforcement learning for large reasoning models. arXiv preprint 2509 2025a.
12

---

Published as a conference paper at ICLR 2026
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint 15440 2024.
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301, 2025b.
Yulai Zhao, Haolin Liu, Dian Yu, SY Kung, Haitao Mi, and Dong Yu. One token fool Ilm-as-a- judge. arXiv preprint arXiv:2507.08794, 2025.
Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Processbench: Identifying process errors in mathematical rea- soning. arXiv preprint arXiv:2412.06559, 2024.
Yang Zhou, Sunzhu Li, Shunyu Liu, Wenkai Fang. Jiale Zhao, Jingwen Yang. Jianwei Lv, Kongcheng Zhang, Yihe Zhou, Hengtong Lu, et al. Breaking the exploration bottleneck: Rubric- scaffolded reinforcement learning for general Ilm reasoning. arXiv preprint arXiv:2508. 16949, 2025.
13

---

Published as a conference paper at ICLR 2026
A THEORETICAL ANALYSIS OF FAPO ALGORITHM
To better understand the effectiveness of FAPO and the whole learning process, we present a theoret- ical analysis. We begin by comparing FAPO with baseline settings to illustrate how the optimization direction shifts during the RL process and how the corresponding advantage estimation evolves. In this context, À plays a crucial role in controlling the optimization dynamics. Building on this, we further introduce parameter-free free, majority- serertt--guied optimization strategy.
In the context of LLM RL, we typically maximize the following clipped surrogate objective:
TaRPO (0) = =E (aa ~ (m
0ex) Aue, clip( clip( Te (001g,
To(orlg,0c1) - +
(9)
Σ
min
G
To Tom (or 1,00/9,000 02t)
with advantage estimated in group-relative manner.
1, If Og is correct
A., r: mean [Rs (
R
(10)
-1 Otherwise
std [ (R
In this context, MFaPo m determines the sign of the advantage A.,, which in turn dictates the current optimization direction while std( can be regarded scaling factor. FAPO introduces the following modification to the reward function:
If is fully correct 1-> If is correct yet flawed. (11) 1, Otherwise
1, If o correct -1, Otherwise
penalty
RGRPO
REAPO
We assume that the current sample contains n rollouts, with proportion of fully correct positives and a proportion of B negatives, leaving 1 - a B as flawed positives. Since GRPO does not distinguish between fully correct and flawed positives, the resulting advantage estimation is:
(AGRPO mean( (( {R:)==) x (1 B)n (-1) x Bn
3 B 2,3,
n
o std ) (1- (arMo) )² x(1 +( (- HaRPo )² x
)
(12)
= (1 - (1 - -MRRO)) 3(1 + (GRPO)
while the advantage estimation of FAPO is
PEAPO = mean [R =
a + (1 - A) - -a- B) 3
a(1 X) -B- 3(1 A) -
- 23 (1 - a -3)
(13)
PGRPO (1 -3),,
FAPO st 12 2 = (1-mam)) xan+ Hrano) (nn-nnnnn--- (n) mmam) 28.
2
a(1 MAXРO) )2 (1 3](1 2=XPO)² 3(1 + pFAPO)?
For TAAO we let 7 = - a - B, so APO = =R²RP 7A, then
APAO a(1 MFAPO) 2
a 3)(1 X MFAPO) 2 811 )FAPO)?
= ax( - MGRPO + 7>)² + 7(1 MGRPO + 7\ 2)2 + 3(1 + PGRPO - 72)2
= [a(1 - 2² +
7
(1- - 22 + 3(1 (
(14)
+ +1207A(1- MORFO) +2y(1 MCRPO) (7) - 2) 2 MA
+ +a72>>² - + 1) N2
4++++
14

---

Published as a conference paper at ICLR 2026
We break TAAO down into three sub-expressions (A, B, C) and simplify them separately.
A = a(1 |ARPP))² + (1 (GGPPO) )² + 8(1 + (GGPP))
(1 (1 MGRPO)2 + B(1 + )GRPO)2
+:9= 2:9=1-a-2
dORPO
B 20 - MGRPO) + 2y MGRPO (7 - X) - -2 + MGRPO)
=2>A/a(1 (ORPO)
(GRPP) (1 -7) - BB11 + MGRPO)]
= =2>A/2aB 23(a + (8- 3(2 23)1 -11- - a- B, PGRPO = 23
=2yA (-28) = -471B
(15)
C ==n? an? +7(7 1) + 222
= (1 - -177)>?> +1 7)
---=1-a-8
=y(1 - 7)1²
B+C = - 4713 + 71 - 7)12 = 7A(A(a (8 -3) -. -a BB
43
(1 7)(A
a + B Xy(1 -7)(A a/3 +1
Thus, MFPOO and FAPO can be expressed in terms of pGRPO and @ORPO
MEAPO = JARPPO - X7 PParO TORFO Ay - (A a/9+1
(16)
We assume a complete learning process that begins with the model unable to solve any problems, i.e., 8 = 1. As training progresses, 8 gradually decreases while increases. The shift optimization direction occurs when:
When Optimization Direction Shift
Aptawed 1
MRARO
<
OFAPO
PTAPO 1 1 A
MAPPO = ad - 3(2 - X) +1 1>11
(17)
28
2
a a+ 3
a/3 +1
-1
How Scaling Factor Changes
The scaling factor oAAO changes over dRRO is:
TAPO - LORO = - 7){A- a/3 +
4
when
1
= 2 &
(18)
3
when
a
4
1 + 2 LeRO
3
7
Summary: The Whole Optimization Process of FAPO We introduce p to characterize the current optimization state, which increases monotonically from 0. For a pre-defined and fixed X, the learning process drives p upward. Once p exceeds A - 1, the optimization direction shifts from reaching the correct answer (warm-up stage) toward reinforcing reliable reasoning (refinement stage). As p continues to increase and surpasses 1 - 11, the scaling factor a rises accordingly, making the advantage estimation more conservative (i.e., | < |AGRFO), for flawed- flawed-positive positive and fully correct rollouts). This conservativeness helps to stabilize training by preventing overly aggressive updates, while still ensuring that correct and reliable rollouts consistently prioritized.
15

---

Published as a conference paper at ICLR 2026
Determining From the above analysis, the reward assignment parameter À plays central role in determining when the optimization shift is applied, whether conservative or aggressive. In practice, adopt majority-guided strategy, which provides both intuitiveness and effectiveness. In the early stage of training, when negative samples with proportion 3 dominate, flawed positives are assigned positive advantages, enabling the model acquire the ability produce correct answers quickly. As training progresses and fully correct rollouts become the majority (i.e., a > (), the optimization naturally shifts toward reinforcing reliable reasoning. Formally:
2
Pshift
=1=
1.
(19)
Pshin +
Therefore, set as the default configuration in FAPO.
B IMPLEMENTATION DETAILS
FAPO-Reasoning Table summarizes the training configurations and hyperparameters of generative reward model (GenRM) and final reasoning models. For GenRM training, we follow the practice of [An al. (2025) by using higher rollout temperature encourage exploration. For reasoning model training, most settings are consistent with DAPO (Yu et al. 2025). except that we reduce the number of rollouts from 16 accelerate overall training speed.
Table I: Training configurations and hyperparameters of our experiments.
FAPO-GenRM-4B
Baseline & FAPO-7B
Baseline & FAPO-32B
Data Configuration
Global Batch Size
512 Instrructtt
512 Qwen2.5-Math-7B
512 Qwen2.5-32B
Base Model
Rollout Inference
Rollout Num per Prompt Temperature
16
8
1.2
1.0
1.0
Top-p
1.0
1.0
1.0
Top-k
-1
Max Prompt Length
5120
2048
2048
Max Response Length
8192
8192
20480
Overlong Buffer Length
4096
4096
4096
Overlong Penalty Factor
1.0
1.0
1.0
Actor Training
PPO Mini Batch Size
32
32
32
Advantage Estimation Type
GRPO
GRPO
GRPO
Clipping Goow
0.2
0.2
0.2
Clipping Fhigh
0.28
0.28
0.28
Optimizer
Adam
Adam
Adam
Learning Rate
10 10-6
10 6
10-6
Weight Decay
0.1
0.1
(31,8z)
(0.9, 0.999)
(0.9, 0.999)
(0.9,09999)
Gradient norm clipping
1.0
1.0
1.0
Learning Rate Scheduler
constant 10
constant 10
constant 10
Warmup Steps
Evaluation Setup
Temperature
0.6
1.0
1.0
Top-p
0.95
0.7
0.7
Top-k
-1
Max Generation Length
8192
8192
20480
FAPO-GenRM FAPO-GenRM is trained using the FAPO-critic dataset, where the ground truth label is generated by Qwen-32B. To mitigate the label noise the data and train FAPO-GenRM robustly, propose the following strategies:
16

---

Published as a conference paper at ICLR 2026
Overview of Agent Reward Loop
Server
Stondslone
Server
Server Handle (Remcrs)
Server kindle (Rempehs)
Server Roster
Rollout Services
Data Stream Pipeline
AgentLoopWWerker
RewerdWansge-Wrkk
AgertLLopWWerker
Rollout Data
Rewer&Wangge
Rollout & Reward
User-Custumizee Reward Function
Rallout Questions
AgentLoopWWrker
Rollout Data
RewerdWangge/Work
Rollout & Reword
AyentLoopWarker
Rallout Data
RewerdWangge
Rellout & Roward
Figure 9: Infrastructure Design of Reward Loop
Consensus filtering: During data synthesis, we sample each instance three times. A sample is kept only if all three generations yield consistent outcomes (i.e., the process errors occur at exactly the same positions). As result, the retained samples tend to be highly reliable and have strong internal agreement.
Robust Training Objective: Our designed training objective (in Equation 7 are robust to the subtle errors missed by the teacher. The reward supervision signal is a soft noise-robust label, the student can still receive appropriate reward even when the annotated error location deviates slightly from the true error.
Reward Loop Design Figure shows the infrastructure design of the reward loop, partially asynchronous infrastructure for reward computation. Reward Loop designed for:
Make reward computation more efficient: each rollout sample sent to the reward model immediately after it is generated, without waiting for the full batch rollouts to finish.
Make user customized reward function more flexible: the design of the Reward Loop provides substantial flexibility for implementing custom reward functions and supports both DiSRM and GenRM.
This implementation is based on the veRL agent loop (Sheng et al. 2025 and has been open-sourced (relevant implementation can be checked in supplementary materials).
C ADDITIONAL RESULTS AND DISCUSSION
Additional Results of FAPO-GenRM Table shows a detailed comparison of our FAPO- GenRM-4B model with other publicly available discriminative and generative reward models. Building upon Swen3--4-IIntrrun PositiveBench and ProcessBench Zheng al. 2024), even surpassing Qwen3-32B and the discrim- inative SoTA model Owen2.5-Math- PRM----------M225 (Zhang et al. 2025b).
our model achieves substantial improvements in both Flawed-
Additional Results of FAPO-Reasoning Strong generalization ability is one of the key advan- tages of reinforcement learning (Huan al. 2023). We then extend the evaluation of FAPO to additional tasks, including two more mathematical domains (MATH (Hendrycks et al. 2021 and AMC) and code reasoning (LiveCodeBench (Jain et al. (0024)). The overall results are shown Table 3 FAPO outperforms the baseline in code reasoning tasks and achieves consistent improve- ments across broad range tasks.
EAPO application in large-scale RL systems Introducing an extra generative reward model will bring burden to the systems. We then quantify the burden and explore the potential application of FAPO in large-scale RL systems. We provide more detailed breakdown the time distribution across different RL stages in Table
17

---

Published as a conference paper at ICLR 2026
Table 2: FAPO-GenRM results in FlawedPositiveeench and ProcessBencie
FlawedPositiveeencc
ProcessBench
Model
Precision
Recall
F1
Avg. F1
Discriminative Process Models (7B-70B)
Owen2.5-Math-- PRM-7B Qwen2.55MathhPPM-222
76.6
83.5
79.9
70.5
74.3
91.0
81.8
76.8
Generative Critic Models (7B-70B)
Qwen2.5 1-t22.5-ttth--- 7B-Instruct
58.0
43.8
Qwen2.5-78 ---nruuntt
50.0
66.2
57.0
38.9
Qwen3-1.78 1.7BB (Think)
73.4
75.1
74.2
56.0
Qwen3-4B-Instmun
69.4
97.9
81.2
77.1
Qwen3-4B (Think)
89.2
84.3
82.0
79.6
Qwen3-32B (Think)
85.1
90.8
87.8
82.0
FAPO-GenRM-IB (Ours)
87.5
91.4
89.4
83.3
Table 3: FAPO-Rees LaaO-Reasnning results in more evaluation benchmarks.
Math
Code LiveCodeBench
General GPQA-Diamond
Model
Avg.
AIME24
AIME25
AMC
MATH
Baseline-32B
38.9
29.5
85.0
72.8
28.6
51.0
51.0
FAPO-32B
42.4
33.5
91.6
74.6
33.6
53.1
54.8
The relative inference cost for long-cot models actually decreases, specifically: (1) Rollout: Long- CoT models exhibit strong long-tail issue: the generation time is bounded by the longest trajectory in the batch. Therefore, rollout time increases significantly the model generates longer traces (e.g., 12k tokens for Qwen3-4B). (2)GenRM: contrast, GenRM inference time does not increase with longer trajectories. We observe that the GenRM output length stays nearly constant across models (e.g., 3.2k - 3.8k). (3) Policy update: This stage accounts for a relatively small portion of the total cost and primarily scales with model size rather than trajectory length. Overall, the root cause that the critic task for
does not require a very long response length.
Ablation study of tbyper-parameter À À is the only key hyper- syper-parameter introduced in the FAPO algorithm. We explore the parameter tuning based on the FAPO-7B setting, with results illustrated in 5
We can conclude that:
Flaw-aware learning consistently improves performance, as all configurations outperform the baseline.
Achieving the best performance require tuning the parameter X. In the 7B setting, a more ag- gressive strategy leads to larger gains. The configuration with p 1/2, which corresponds to an optimization shift where about one third of the rollouts are fully correct, achieves the best performance.
So overall, flaw-aware learning leads performance gains, but achieving the best performance requires tuning the parameter X. That said, FAPO introduces only this single additional parameter, which makes the tuning process relatively easy.
Human Verification on the reliability of FAPO-32B Beyond the previous LLM-aa----udde eval- uation with Qwen3-32B, we further conduct detailed human verification to assess the process reliability of FAPO-32B. Specifically, we randomly sample 20 positive cases with correct final an- swers and manually examine whether they contain unreliable reasoning patterns, with the results summarized in Table 6 Through this analysis, observe that (1) multiple-choice questions hibit a noticeably higher proportion of flawed positives compared to math word problems, where the final answer is often symbolic expression; and (2) method consistently outperforms the
18

---

Published as a conference paper at ICLR 2026
Table 4: Time distribution across different RL stages in different settings.
Rollout (Infer)
FAPO-GenRM (Infer)
Policy Update (Train)
Model
Rollout Len
GenRM Len
Nodes
FAPO-7B
42%
1.1k
18%
3.0k
33%
4
FAPO-32B
60%
2.3k
14%
3.2k
20%
Qwen3-4B
72%
12.0k
10%
3.8k
14%
16
Table 5: Hyper-parameter tuning of X.
Setting
Performance
Bascline-7BB (p +:00
0)
32.1
FAPO with p= = X 1/3
34.6
FAPO with p= + À = (default setting)
36.8
FAPO with p= 1/2 = A = 1/3
39.6
baseline under both LLM-as-a-judge and human verification, further demonstrating its effectiveness in mitigating flawed- peoved-ossitvee issues.
Table 6: LLM-as-a-judge and human verification of flawed positive ratio.
AIME24
AIME25
GPQA-Diamont (Multi-Choice Problem)
Model
(Math Word Problem)
(Math Word Problem)
Baseline-32B
15.5
10.9
45.7
(Lud-as---judge)
FAPO-32B
7.1 (-8.4)
1.7 (-9.2)
42.0 (-3.7)
(LLM-as-a-judge)
Baseline-319 32B
4/ 20
1 1/20
10 20
(Human- )eriann-Verifiaatoo))
FAPO-32B
2/
0 /0
7 20
(Human-Verri) Verificatiin)
In section 2.2 we demonstrate that the flawed positive acts role of stepping stones to performance gains, and Figure (d) illustrates the correlation between them. Here, we provide a more complete explanation of how mitigating flawed positives leads performance gains.
How Flawed Learning contributes to Performance Gains
Early Stages: Flawed Behavior + More Correct Rollouts: Prior studies (Zheng et al. 2024 Wang al., 2025) have shown that flawed reasoning often acts as shortcut the correct final answer. This bias, inherited from pre-training (Kalai et al. 2025), leads to certain flawed rollouts. Thus, in the early phase of RL., flawed positives naturally increase the number of correct final answer rollouts.
Early Stages: More Correct Rollouts - Early performance gains: A larger pool of cor- rect rollouts yields more positive rewards, providing stronger supervision and driving exploitation early in training. This effect can be directly observed train-time reward statistics in Table y Later Stages: Penalizing flawed behavior - Fewer flawed rollouts: Our theoretical analysis demonstrates the optimization distribution shift: flawed rollouts receive negative advantage, pro- gressively reducing the model's tendency to produce flawed processes. This trend is reflected in Figure (lef),, where the proportion of flawed positives decreases steadily as training progresses. Later Stages: Fewer flawed rollouts Performance gains: As flawed rollouts diminish, the model allocates more rollout chances to fully correct trajectories. Consequently, the RL loop receives higher density of genuinely useful reward signals, as reflected in Table improving final performance.
Model Selection in FAPO Experiments and PAPO-Reasoning:
We explain the model selection in both FAPO-GenRM
19

---

Published as a conference paper at ICLR 2026
Table 7: Early-stage train-time rewards (view flawed rollout as positive rollout).
Table 8:
Ltter-stage train-time rewards (view flawed rollout as negative rollout).
Step
Step
200
Baseline-7B
0.27
0.29
0.34
0.36
Baseline-7B
0.35
0.37
0.36
0.40
0.39
FAPO-7B
0.33
0.42
0.44
FAPO-7B
0.40
0.41
0.44
0.45
Model Selection in FAPO-GenRM: We adopt Qwen3-4B-Instruct as the base model GenRM training, considering three factors: (1) it demonstrates strong instruction-folloiing and basic error-detection capability (as shown in Table 2), making it a suitable initialization for RL; (2) its relatively small size ensures efficient training and faster inference, with Instruct models pro- ducing shorter responses than think-style models; (3) it avoids potential concerns of knowledge leakage into the final RL process that may arise if the base model is overly strong.
Model Selection in RAPO-Reasoning: We follow the same setup as DAPO (Yu et al. (025) and use Swen2.5--Math-7B and Qwen2.5-32B base models, (I) the learning curves of pre- trained models clearly illustrate the entire exploration-exploiattoo trajectory starting from near- zero performance, whereas RL-finetuned Instruct models often exhibit instability; (2) these two models also strike a good balance between response length and training speed.
D LIMITATIONS AND FUTURE WORK
Our work presents several limitations that point to promising future directions, both in algorithmic effectiveness and infrastructure design. On the algorithmic side, although this work trains math- ematical reasoning tasks, FAPO has strong potential in broader settings such as multi-choice tasks, multi-turn interactions, and agent-based RL, where flawed processes often more pronounced and problematic. We will also further validate the effectiveness of FAPO across a wider range of model architectures (e.g., MoE) and larger model scales. On the infrastructure side, while our decoupled design improves efficiency and inference speed, its applicability to fully asynchronous RL systems remains uncertain, as we discussed in Section 4.5 specifically Table We regard these as important research directions in our future work.
E
LLM USAGE
During the writing of this paper, Al assistants are employed to assist with minor language refine- ment. Their suggestions are limited enhancing clarity and readability, without influencing the research design, experiments, or conclusions. All content was carefully reviewed, validated, and revised by the authors to ensure accuracy and fidelity to the research.
F PROMPTS
Prompts for Flawed Positive Detection
Prompt for Outcome Reward Model (ORM):
The following is math problem with its ground truth answer, along with an Al solution:
[Math Problem]
(problem statement)
[Ground Truth]
{ground truth answer)
[AI Solution]
[AI Solution]
20

---

Published as a conference paper at ICLR 2026
Your task is to review and critique the solution step by step, and output whether the AI solution is correct. Please reason step by step, put your final answer (i.e., TTree' or 'False") \boxed()
Prompt for Process Reward Model (PRM)
The following is math problem with its ground truth answer, along with an AI solution (split into paragraphs, enclosed with tags and indexed from 0):
[Math Problem]
(problem statement]
[Ground Truth]
(ground truth answer)
[AI Solution]
<paragraph. (tt.)/ll......aaarraaaaaa.a </paaagrpppphhh--1>>>>>>>>
Your task to review and critique the solution paragraph by paragraph. Once you identify an error in a paragraph, return the index of the paragraph where the earliest error occurs. Otherwise, return the index of -1 (which typically denotes 'not found'). (i.e., the index)
Please reason step by step, put your final in [boxed[]
Prompts for Reasoning
Prompt for DAPO-Math-17K, AIME24, and AIME25: (problem statement)
Please reason step by step, and put your final answer within [boxed(].
Prompt for GPOA-Diamond:
(problem statement)
A. (Choice A}
(Choice s)
(Choice c
D. (Choice_D)
Please reason step by step, and put your final answer (only the choice letter) within (boxed().
G FLAWED POSITIVE SAMPLES WITH HUMAN ANALYSIS
Flawed Positive Example
Question Source: AIME24
Question:
Alice chooses set A of positive integers. Then Bob lists all finite nonempty sets B of positive integers with the property that the maximum element of B belongs to A. Bob's 1ist has 2024 sets. Find the of the elements of A.
Generation Model: Qwen2. 5-Math-7B (Pre-Trained Base Model)
Response:
we are given that Bob lists all finite nenempty sets B of positive
21

---

Published as a conference paper at ICLR 2026
integers where the maximum element of B belongs to A. Bob's list contains 2024 such sets. We need to find the sum of the elements of A.
step-by-Step Reasoning:
1. Understanding the Problem:
Let A A= [01, (a.) where < .... <dn.
For each d; E A, Rob lists all finite nonempty B such that the maximum element of is d.
2. Counting the Sets for Each ai For given
the sets B can incluce any subset of the integers less than plus itself.
The number of subsets of the integers less than a 20:11 (since there are a- integers less than a,) However, since B must be nonempty, we subtract 1 from this count (the empty set).
Therefore, the number of sets B for given is:
24-11-1
3. Total Number of Sets: total number of sets listed by Bob the sum number of sets for each d::
(2 = 2024
Simplifying, we get!
201-1 11 2024
201-1 2024 +
4. Finding TE:
we need to find such that the sum Σ 2 is minimized. The smallest possible values for are 1,2, Let's s test with small values n: For 1:
21-1=1 (too small)
For 2:
21-1 2 1+2=3 (:00 small)
For = 3:
2 + 2 2
2 (tcc small)
For n n= 4:
21-1 + 22-1 33
(too small)
For 5:
2'-1 + 22-1 + 23 24 +
31 (too small)
For n = 6:
21-1 22- + 2 2 425 -
16 (too small)
For T:
21 22-1 + 23 +
2- + 23- + 28 +
2 127 (too small)
For n=8:
21-1 2--122-11 2--1 2 24-1
2 20-1 28-1 27-1 +
2 255 (too small)
For n = 9:
21-1 2 + +2 3-1 +2 1- +25
+ 2-1
+
2 2 2-1 511
(too small)
22

---

Published as a conference paper at ICLR 2026
For = 10:
2'
- +2- +20-1 =1033 (too small)
- For n
2 22-1422-1 - 1--1122-1110 122 12--1 120 +2"
+22 - +2 12 12 12-0-11211-1 -007 (too large)
Therefore, == 10 is the correct value because:
2'-1 + +2 +2 +2 + + + +20 +2 1023
And:
1023 + 10 = 1033 (which is too large)
So, we need to ad just to be and check the sum:
2 2'- +2 + 2 + +2 2 + +2 + + 2 + +2 + +2--1 22-1-210-1 1023
And:
1023 + 1033 (which too large)
Therefore, the correct is 10, and the of the elements of A is:
1+ 1+243+4556447H8--
Thus, the of the elements of A is:
55
Human Analysis:
There are two clear mistakes in the AI's respense:
1. The number of sets B for should be 20i-1 (not 24-11 1) This incorrect assunption leads derivation throughout the reat the solution.
completely flawed
2. The correct set A should be 6,,6,7, 3,,2,3,4,5, 6, 7, 9, From the solution process, it's clear that the guessed a value close to the target, rather than identifying the exact condition required.
10, 11}, instead of
In summary, although the final answer happens to be correct, the overall reasoning process fundamentally flawed.
Flawed Positive Example
Question Source: DAPO-Math-17K
Question:
Let b, be the three roots of the equation - kr 0, where is real number. If (a 2) + 2b= (c 2) -18, find the value of k.
Response Generation Model: Llama-3, 3.
(ShortCoT Model)
Response:
Step 1: Recall Vieta's formulas for cubic equation
Vieta's formulas for a cubic equation x² pr² + qu +r a, b, and e state that: b b = PP, ab be + ac = Q, and abc = -r. Applying these formulas to the given equation x² - (k+1)x* + kr + 12 = 0, we get ++b--=k+1l
with roots
ab be+ ac=acc=a k, and abc= -12 Expand the given expression using the identity for step (b-2)) ( 2)
(a (2)2) + ((b The identity for (a
2) + (b (b- 2) + (c - 2) can be expanded as
(a -0--6a" 60 12a 12a--8) -b' 66'-6""+100 -) 66 + - 8). This simplifies
to (a +3 - -() + Step
24.
the given equation to find +
we can express &, 83, and & in terms of a, b c, and k. Substituting
Since a, b, and e are roots of the equation 1)
23

---

Published as a conference paper at ICLR 2026
I a, and c into the equation gives a²
(k + 1)a² 12, Sumning these gives
= (k + 1)b kb 12, and e² (k + 1)c² ke 12.
a² + + === [k+ + -(a 36.
Step 4: Find d² +bb using the sum of roots From vieta's formulas, we know a+b+ce = ++1, Squaring both sides gives (a +b++)) (
which expands to d² + + e + 2(ab be + ac) = ka 2k +1. Since ab + + ac = k, we can substitute this in to get + 2kl,, simplifying to +
a² + + + = 2k
+1.
Step 5: Substitute a² 12+ into the expression for a ++b'+c" substituting
(k+1)((a (a21)(aa++b++ - k(k + 1) 36. Simplifying yields a + +c =cck + - 36 k2 35.
= into a² +'+
gives a" + 6 +
11
Step 6: Substitute known values into the expanded expression for (a ()-2)" (b 2)--2)"-1 (c-2 2)
(6- Given that (a 2 2) + (c- 2) -18, and using the expansion
from Step 2, we substitute a' + +e = -b -35 and a ++++o= k + into
the expression. We also need d + + +e² = + + from Step 4. The
expression becomes (k- 6(k +1)+12(2
Step Solve for simplifying the equation (k2 - 35) 6(k² 1) 12(k + 1) 1) = -18 gives
1 ---66 6k2 + 12k 35- - 12k + 12 24 18. Combining like terms yields 42 18. Adding 18 to both sides gives k2 6k 22k-42kk3333 0. This is cubic equation
terms of k. Step Solve the cubic equation for k To solve k -6-6-=+24k-33 0, let's try to factor it or find a rational root. Checking for simple integer solutions, & = 5 is found to be root (through trial and error using the Rational Root Theorem), as 5 6(6)) + 24(5) error in calculation: the correct calculation for
120-115100 +50 120 35 60. However this should be 6'-6(6)++24(5) 35 = 150 120 - = 60, which is not correct. The correct approach is to actually solve the equation correctly. Let's correct the step: We need to solve k² 6k 24k 35 0. Trying ==5 correctly: 6 + 24(5) -5 125 150
35 60, which does not
equal 0, indicating an error in identifying correct factorization or solution methed must be applied.
= 5 as root. The
Step Correctly identify and solve for k The in Step indicates miscalculation in identifying
root. The correct approach involves factoring or applying a method like synthetic division if rational rect is suspected, or using numerical methoas. However, given the mistake in calculation and approach, let's reconsider the equation and directly evaluate potential rational roots or use given conditions more effectively. The final answer is:
Human Analysis: The correct simplified equation is ka 6k2 + 12k - 35
not 1 6k 24k - 35 =5=0. This typo leads to incorrect solution of k. However, what's s quite peculiar is that despite model's multiple failed attempts to correct itself, it ultimately guessed ==5.
Flawed Positive Example
Question Source: DAPO-Math-17K
Question:
Let a, b be distinct numbers such that the equations 2"+ar+1 == = and x +o+ have common real root, and the equations a'+ = 0 and
2 + also have a common real rost. Compute the sum a b c.
Response Generation Model: Qwen3-1.78 (LongCoT Model, Think Mode)
Response:
(Think Content Omitted Here)
24

---

Published as a conference paper at ICLR 2026
To solve the problem, are given three distinct numbers a, t,b,c such
that:
The equations +=+az+1 = and +b+ share common real root, The equations x² +'+x+0= -= and x² +c also share a common real root.
We are to compute the value of atbtc.
Step Use the Common Root Condition
Let a be a real root of both equations x² + ar + +1= = and x² +c= 0. Substituting a:
aa+a++==0 (1) + ba (2)
Subtracting (1) from (2):
(b-aja aja + (e - 1) = 1-c (3)
Now, substitute into (1) and (2) to express relationships between a,b.c.
Step 2: Use the Second Pair of Equations Let 8 be real root of both x² + x Substituting 8:
0 and x² + b 0.
8 + (4) + +b= (5)
Subtracting (4) from (5):
(c 1)8 (b - a) 0 = 3 c-1 (6)
Step 3: Use the Relationship Between a and 8 From the problem, it is known that B = Therefore:
b
Also, from (6):
a b B
c-1
80:
a b
1 1-c
8
c
a b-a
This gives a relationship between a, b, and we proceed by assuming a= which simplifies the algebra.
Step Find a, with a=1 Suppose a a= is a common root of both pairs of equaticns. From equation (1):
Now, since a=l is root of equation (2):
00.l+c=0=1++++ -==
Now, from the second pair of equations, 8 is a root of both x² x =+a=0 0 and +b+cr+b=0, substitute == into both equations:
=0=)==+000)000+ 1 +eel-00e0@l4ch000aacti Thus, consistent. Now, the sum:
a+b+c=-2tb+ (-1- -33
Final Answer
The makes unwarranted assumption that 1, whereas in fact, deriving this requires a complex reasoning process. Although the final answer is correct, the reasoning is flawed and the correct value is reached coincidentally.
Human Analysis:
25


Reviews:
[{"review_id": "GChhpCHRaN", "reviewer": "Reviewer_DJdo", "summary": "This paper addresses a critical issue in RLVR for LLMs: the reinforcement of flawed reasoning patterns. In RLVR, models are rewarded for generating trajectories that lead to a correct final answer. However, many such positive rollouts contain unreliable reasoning steps, which are rewarded identically to fully correct solutions. This flawed-positive problem can lead to models that are correct on a specific metric but ultimately unreliable.\n\nThe authors first conduct a systematic analysis, revealing that flawed positives act as valuable stepping stones for rapid early learning but later constrain performance by reinforcing bad habits. To address this, they propose FAPO, a two-stage solution: They train a efficient generative reward model using a novel step-wise RL objective that localizes the first error in a reasoning chain. This model achieves state-of-the-art performance on error detection benchmarks. FAPO applies a parameter-free reward penalty to rollouts flagged as flawed-positive by the GenRM. The penalty is designed to dynamically shift the learning focus, initially allowing flawed positives to aid learning before gradually steering the policy toward fully reliable reasoning as its capability improves.", "questions": "Please refer to \"Weaknesses\".", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The problem tackled is of great significance to the LLM reasoning community. As RLVR becomes a dominant paradigm for advancing LLM capabilities, ensuring that the learned reasoning is not just correct but also reliable and transparent is crucial for safety and trustworthiness. FAPO provides a practical, efficient, and theoretically grounded solution that improves both the efficiency and the final quality of RL training. The release of code and benchmarks further enhances its impact. Experiments on mathematical and GPQA reasoning tasks demonstrate that FAPO improves outcome correctness, reduces the rate of flawed positives, and enhances training stability without increasing response length.", "weaknesses": "While the results on mathematical reasoning and GPQA are strong, the paper's claims of broad domains would be more convincing with validation on a wider range of tasks. A key domain of interest is code generation, where verifiable rewards are common and flawed reasoning (e.g., code that passes specific tests but is buggy or inefficient) is a major concern. Demonstrating coding effectiveness would significantly broaden the method's impact and generalizability.\n\nThe 20% training-time overhead, while reasonable, could be a barrier to truly massive-scale training. A more detailed discussion of the bottlenecks and potential optimizations for a synchronous system (even as future work) would be valuable for practitioners looking to adopt it at a larger scale.\n\nThe paper focuses on tasks with easily verifiable final answers. A natural question is how FAPO would be adapted to more subjective or open-ended tasks where a \"correct\" final answer is not binarily verifiable. In such scenarios, would the process-level reward become the primary signal?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZzC99EcKeI", "reviewer": "Reviewer_wvzj", "summary": "This paper addresses the issue of \"flawed-positive\" rollouts in reinforcement learning with verifiable rewards (RLVR) for LLMs, where models arrive at correct answers via unreliable reasoning (e.g., guessing). The authors first analyze this phenomenon, finding that while flawed positives provide rapid early gains, they ultimately constrain capability by reinforcing these unreliable patterns. To mitigate this, they propose Flawed-Aware Policy Optimization (FAPO), a method that applies a parameter-free reward penalty to flawed-positive rollouts. This approach aims to leverage flawed positives as shortcuts during the initial warm-up phase while gradually shifting the optimization objective toward reliable reasoning in the later refinement stage. To enable this, the work also introduces a generative reward model (GenRM) trained with a process-level reward to accurately detect and localize reasoning errors within rollouts. The authors claim FAPO improves correctness, reliability, and stability without increasing the token budget.", "questions": "none", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper clearly identifies and analyzes a critical problem in RLVR. The preliminary study (Section 2.2) effectively demonstrates the \"twofold effect\" of flawed positives—acting as \"stepping stones\" early in training but hindering optimization later —providing a solid empirical foundation for the proposed solution.\n\n2. The FAPO algorithm's adaptive reward penalty is simple yet theoretically grounded. The analysis in Appendix A demonstrates how this mechanism creates an automatic, parameter-free \"optimization shift\".\n\n3. The development of a compact (4B) generative reward model (GenRM) is a key strength. The step-wise, distance-sensitive reward formulation used to train it is well-designed to encourage precise error localization rather than simple binary guessing. This trained GenRM is shown to be highly effective, outperforming its 32B teacher model and a 72B SOTA discriminative model.\n\n4. The experiments are thorough, testing on multiple models (7B, 32B) and benchmarks (AIME24, AIME25, GPQA). Crucially, the authors present full learning curves rather than just final checkpoints. This transparently supports claims of improved training stability , outcome correctness , and process reliability (a reduced flawed positive ratio).", "weaknesses": "1. The entire framework's effectiveness is contingent on the quality of the FAPO-Critic-85K dataset, which was labeled by a \"teacher model\" (Qwen3-32B). The GenRM can only learn to detect flaws that the teacher model can identify. This creates a fundamental performance ceiling; any subtle errors missed by the teacher will be propagated, and FAPO will fail to penalize them.\n\n2. The GenRM is trained primarily on mathematical reasoning tasks. While it shows good performance on GPQA-Diamond, it is unclear how well this math-trained critic can generalize to detecting a wider array of non-mathematical flawed reasoning (e.g., logical fallacies, factual inconsistencies) in open-domain tasks. The paper acknowledges this as a limitation.\n\n3. The proposed asynchronous architecture, while necessary for performance, adds significant infrastructure complexity compared to standard RLVR, which uses a simple, synchronous rule-based verifier. The authors note this adds a non-trivial training time increase of \"less than 20%\", which could be a barrier to adoption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QerCuQfqyc", "reviewer": "Reviewer_hAMR", "summary": "The paper begins by identifying a common failure mode in outcome-based RLVR: flawed-positive rollouts, which are trajectories that produce correct final answers through flawed reasoning. It first quantifies their prevalence and twofold effects during training, then proposes FAPO (Flawed-Aware Policy Optimization) as an enhancement to DAPO/GRPO based on this insight. FAPO trains a generative reward model (GenRM) with both outcome- and process-level supervision to detect flawed positives, and integrates this signal into the policy optimization process via an additional penalization term to the outcome reward. Experimentally, the trained FAPO-GenRM achieves higher F1 scores than baselines on ProcessBench and the curated FlawedPositiveBench. When applied to train Qwen-7B and Qwen-32B reasoning models, FAPO delivers improved performance on AIME24/25 and GPQA-Diamond, reducing flawed-positive ratios compared to GRPO baselines without increasing the inference token budget. Additional ablations analyze GenRM effectiveness, examine potential reward-hacking risks, and show that asynchronous deployment keeps computational overhead modest.", "questions": "1. How sensitive is FAPO to the choice of λ beyond the majority-guided setting (λ = 1)? Could a curriculum that gradually increases λ during training yield better performance than the fixed choice?\n2. Do the authors have the performance results on other benchmarks (i.e., AIME25, GQPA) corresponding to the setup shown in Figure 5?\n3. During RL training, the GenRM remains fixed while the policy model continuously updates, which may lead to reward-hacking behavior. In Figure 7, FAPO-GenRM appears more robust than PRM, but to what extent? Has its robustness been quantitatively evaluated, and could it eventually suffer from reward hacking as well?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- **Well-motivated.** The paper clearly identifies *flawed-positive rollouts* as a pervasive yet unresolved failure mode in RLVR, illustrating their dynamics through quantitative trends (Figure 2) and motivating the need for process-level awareness beyond outcome-based rewards. This diagnosis provides a solid conceptual foundation for introducing FAPO.\n- **Sound theoretical formulation.** The theoretical sections (Sec. 3.2, Appendix A) formalize the reward-penalization mechanism using group-relative advantage estimation. The analysis further justifies adopting a fixed penalty coefficient (λ = 1) based on the majority-guided condition, making FAPO both principled and free of hyperparameters.\n- **Comprehensive experimental validation.** The experiments cover multiple reward benchmarks (FlawedPositiveBench, ProcessBench) and reasoning benchmarks (AIME24/25, GPQA-Diamond). The study includes one 4B model for the reward model and two models (7B and 32B) for the reasoner model training, supported by detailed analysis on FAPO-GenRM's detection ability, training effectiveness, and the reward-hacking risks. Together, these components form a coherent and comprehensive empirical evaluation.", "weaknesses": "- **[Soundness]** While the observations in Section 3 are insightful, the claimed causal relationship between flawed positives and performance gains (lines 190–192) requires further evidence. The presented figure only demonstrates a correlation, not causation, and additional controlled ablations would be needed to validate this conclusion.\n- **[Soundness]** The evaluation datasets are relatively small. AIME (30 samples) can be insufficient for robust reasoning assessment, and on the larger GPQA-Diamond (198 samples), the observed improvement is modest (Figure 4, 1.5 points). Moreover, in Figure 4 (bottom right), the flawed-positive ratio remains high even after FAPO training, suggesting potential limits in its corrective capability.\n- **[Significance]** Although the theoretical formulation is novel and well-structured, the method essentially extends RLVR by integrating outcome and process rewards into a unified training signal. Similar ideas have been explored in prior works such as [1] and [2], which slightly reduces the originality of the contribution. Furthermore, the algorithm focuses solely on mitigating flawed positives, leaving other issues in RLVR, such as false negatives, unaddressed.\n\n[1] Process Reinforcement Through Implicit Rewards, Arxiv Feb 2025\n\n[2] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains, Arxiv July 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qTuzHCRsd8", "reviewer": "Reviewer_Gwxh", "summary": "This paper introduces FAPO, a novel approach to improving RL for LLMs in reasoning tasks. It identifies that flawed-positive rollouts persist throughout RL training and hinder model performance, then proposes FAPO which combines a compact generative reward model (FAPO-GenRM-4B) to detect reasoning errors with process-level rewards and a parameter-free algorithm that applies adaptive penalties to flawed positives. FAPO demonstrates consistent improvements across 7B and 32B models on mathematical (AIME24: +4.7%, AIME25: +3.1%) and general reasoning tasks (GPQA-Diamond: +1.5%), while substantially reducing flawed-positive ratios.", "questions": "1. Why does the student model outperform the teacher model? Table 2 and Figure 3 show that FAPO-GenRM-4B outperforms Qwen3-32B (teacher) on FlawedPositiveBench\n2. What is the overall computational cost overhead? The paper mentions training time increases by \"less than 20%\" but lacks comprehensive cost analysis (e.g., the cost of data synthesis, training GenRM, and detailed inference costs during RL). In particular, I am curious whether the inference cost would increase for GenRM if performing FAPO on models that have some code-starts with long-CoT", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow. In particular, figures in the paper effectively illustrate key concepts\n2. The systematic study in Section 2.2 provides valuable insights into the prevalence and evolution of flawed-positive rollouts during RL training, supported by both automatic evaluation and human verification\n3. The effectiveness of FAPO is demonstrated both theoretically and empricially", "weaknesses": "1. The hybrid process + outcome reward formulation is not novel. Prior works have extensively explored combining step-level and outcome-level rewards (e.g., arXiv:2312.08935, arXiv:2504.13958)\n2. The paper only evaluates on mathematical reasoning tasks (AIME24, 25) and one general domain benchmark (GPQA-Diamond), which is insufficient to claim broad applicability. The paper would benefit significantly from including additional widely-used benchmarks such as AMC and MATH\n3. Only two model families (Qwen2.5-Math-7B and Qwen2.5-32B) are tested, both starting from pre-trained models. It would be valuable to stress-test the proposed GenRM on models that already have some code-starts with long-CoT trajectories\n4. While the improvements are consistent, the absolute performance gains come at considerable cost: (1) requires training an additional 4B reward model, (2) needs complex asynchronous infrastructure", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)