Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Models
Authors: Junyi Li
Hwee Tou Ng
Venue: NeurIPS 2025 poster ()
Keywords: Hallucination
Large Reasoning Models
Reinforcement Learning
Primary area: deep_learning
TL;DR: We reveal that reasoning-oriented RL fine-tuning significantly increases the prevalence of hallucinations and propose a FSPO approach via reinforcement learning.

Abstract: Large language models (LLMs) have significantly advanced in reasoning tasks through reinforcement learning (RL) optimization, achieving impressive capabilities across various challenging benchmarks. However, our empirical analysis reveals a critical drawback: reasoning-oriented RL fine-tuning significantly increases the prevalence of hallucinations. We theoretically analyze the RL training dynamics, identifying high-variance gradient, entropy-induced randomness, and susceptibility to spurious local optima as key factors leading to hallucinations. To address this drawback, we propose Factuality-aware Step-wise Policy Optimization (FSPO), an innovative RL fine-tuning algorithm incorporating explicit factuality verification at each reasoning step. FSPO leverages automated verification against given evidence to dynamically adjust token-level advantage values, incentivizing factual correctness throughout the reasoning process. Experiments across mathematical reasoning and hallucination benchmarks using Qwen2.5 and Llama models demonstrate that FSPO effectively reduces hallucinations while enhancing reasoning accuracy, substantially improving both reliability and performance.

Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Models
Hwee Tou Ng Department of Computer Science, National University of Singapore cheneyjunyi@gmall com, nght@comp mus
Junyi Li
nus.edu. edu. sg
Abstract
Large language models (LLMs) have significantly advanced in reasoning tasks through reinforcement learning (RL) optimization, achieving impressive capabil- ities across various challenging benchmarks. However, our empirical analysis reveals a critical drawback: reasoning-oriente RI. fine-tuning significantly in- creases the prevalence of hallucinations. We theoretically analyze the RL training dynamics, identifying high-variance gradient, -inrupy-nnduced randomness, and susceptibility to spurious local optima as key factors leading to hallucinations. To address this drawback, we propose Factuality-awan Step-wise Policy Optimization (FSPO), an innovative RL. fine-tuning algorithm incorporating explicit factual- ity verification at each reasoning step. FSPO leverages automated verification against given evidence to dynamically adjust token-level advantage values, incen- tivizing factual correctness throughout the reasoning process. Experiments across mathematical reasoning and hallucination benchmarks using Qwen2.5 and Llama models demonstrate that FSPO effectively reduces hallucinations while enhancing reasoning accuracy, substantially improving both reliability and performance.
https://gt thppsb. com/nusnlp/FSPO
Introduction
Large language models (LLMs) have recently achieved remarkable breakthroughs natural language generation and complex reasoning tasks 15). This progress is fueled by training paradigms that imbue LLMs with reasoning capabilities, including fine-tuning on long chain-of-thhugeee (CoT) so- lutions 443 and reinforcement learning (RL) based optimization [9. By leveraging these approaches, state-of-the thet models can decompose complex problems into reasoning steps and iteratively refine their answers, yielding impressive results challenging benchmarks in mathematics [0] multi-hop question answering [41), coding (2), and decision -aasiinnmmaingg [32].
Along with improved reasoning ability, critical question has emerged: To what extent do reasoning models maintain coctuuinnaatta factuality after RL fine-inu ? In this work, our empirical observations indicate that when LLMs trained with outcome-driven RL, they tend to produce more factually incorrect or fabricated statements (i.c., hallucinations [13] (46) during their reasoning process. We find that these hallucinations usually stem from incorrect intermediate reasoning, i.c., the model may generate unsupported or false claims the thain-of-le of-thoughh, even if its final answer is occasionally correct. Indeed, RL-aligned models can be highly fluent and confident in their step-by-step explanations while subtly introducing factual errors [40]. This issue undermines the reliability of RL-trained reasoning models, flawed or untruthful answer is problematic for trust and interpretability.
To understand why outcome-based RL fine-tuning exacerbates hallucinations, we conduct a theoretical analysis of the RL. training dynamics for reasoning tasks. Our analysis identifies three primary factors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

R1-05ol
RL-Dsali-
R142kst 1-Owen-
NL-Okoll-lammet
100
100
100
Hallucination Rate (%)
Hallucination Rate (%)
60
60
60
Hallucination
40
40
TruthfuiQA
HalluQA
Figure Hallucination rate (%) for six groups of twelve LLMs across three benchmarks.
First, optimizing solely for correct final answers leads to extremely high variance in the policy gradient when such answers are rare, yielding unstable training updates. Second, the need to discover rewarding outputs forces the policy maintain high level of entropy in its predictions, heightening the risk ballucinations. Third, standard RL objectives admit spurious local optima whereby the model converge to confidently held wrong answer that yields i.e. high-variance gradient, entropy-induced randomness, and spurious local optima, together explain why purely outcome-based RL approach can cause reasoning model to hallucinate and struggle learn faithful reasoning patterns.
reward. These three factors,
To mitigate these issues, propose Factuality aware Step wise Policy Optimization (FSPO), novel RL fine-tuning algorithm that explicitly incorporates factuality feedback at each reasoning step. The key idea is adjust the advantage values of individual tokens using step-wise factuality reward signal, encouraging LLMs to generate more factual content. Specially, we employ an automated verifier check whether each generated reasoning sentence is entailed by the given evidence, yielding step-wise factuality scores. These scores are then integrated into the overall reward signal to adjust the advantages of tokens during optimization, rewarding factually correct tokens while penalizing reward shaping provides denser and more informative feedback
incorrect tokens. This factuality-awan the policy, addressing the sparse-signal problem and reducing training instability. FSPO steers the policy toward solutions that are not only correct but also generated via faithful, verifiable chains of reasoning, thereby directly mitigating hallucinations in the RL training process.
We conduct extensive experiments to evaluate the effectiveness FSPO on both mathematical reason- ing and hallucination benchmarks using Qwen2.5-7B: 3-7B-Base//nstruus and Llama-3. 3.1--B-IJ ----Isttruct. The results show that our method achieves significantly improved mathematical reasoning performance and reduced hallucinations. Further detailed analyses indicate that method can enhance the factuality of reasoning steps without compromising the quality of generated content.
Preliminary Experiments
To explore the effect of RL on hallucinations, we conduct preliminary experiments based on several LLMs and hallucination benchmarks.
Models and Datasets. For comparative analysis, we select several groups of LLMSs without and with
large-scale RL training: (1) DeepSeek-V3 V3 [21 and DeepSeek-RR R1 91: (2) Owwe22.5-3B-Instrruu [39
and QwQ-32B 32B (28): (3) Qwen2.5- 111////111BB//1.BBBB
(39] and RI-Distill-
[9]
(4) Llama3.1 1-88----BBnnnre the synthesized Long-CoT data from DeepSeek-RI! thus exhibiting similar thinking modes to RL- tuned models. For the hallucination datasets, we select three benchmarks, namely TruthfulQA (20). HaluEval [19, and HalluQA 4. In TruthfulQA, we adopt the generation task, where the model prompted to generate an answer to a question, and then utilize a fine-tuned GPT-3 to predict the truthfulness of the answer. The hallucination rate is equal to the ratio of untruthful answers. HalluQA follows same method as TruthfulQA, except that it employs GPT-4 as the evaluator. For HaluEval, adopt the QA subset, where the LLM is provided with question and two candidate answers and
and RI-Distill Li-DDistll--aamann Llama-&B (9] These distilled models are fine-tuned on

---

it needs to assess which answer is factually correct. The hallucination rate refers to the ratio of false judgments made by the model.
Results and Analysis. The results are shown in Figure 1 We can clearly observe that after large-scale training with RL or long CoT data, models demonstrate dramatically higher degree of hal- lucinations across the three benchmarks. To further investigate the underlying causes of this phenomenon, we randomly select 200 samples from HaluEval-QA that DeepSeek-V V3 answers correctly while DeepSeek-RI does not. We manually examine these cases and classify the source errors (refer to Appendix E for more de- tails). In Figure 2 we see that most errors stem from incorrect reasoning steps. We hypothesize that this may be attributed to the assigned
Others
Overthin
1.0%
Incorsistency
17.5%
71.5%
Incorrect Reesoning
outcome reward modeling, where rewards and penalties solely based on the correctness of the final answer, while ignoring signals from intermediate reasoning steps. In the following sections, we will provide a theoretical analysis of how outcome-based reward modeling in RL could lead hallucinations, and then propose the factuality-awarr step-wise RI. algorithm to mitigate this issue.
Figure 2: Distribution of hallu- cination causes 200 sam- ples from HaluEval-QA.
Background
Reinforcement Learning. In reinforcement learning, language model generation be formulated as a token-level Markov decision process, denoted by . = {S, AA R, mo At the t-th generation step, the state 8, S is the concatenation of the input prompt x and the generated sequence so far V<z = [01, ...., 0t-1]. The policy (-(+s))) (parameterized by 0) selects the next action og E A deterministic transition to the next state 814-1 [st, or]. Let Y denote the set of correct answers for a given input prompt Existing research mostly adopts a binary reward for each output y. which is sparse and discontinuous 33 [8]. For
by generating token from the vocabulary A, leading
simplicity, we consider a prevalent format in this paper: Ryy) = when y E Y*, or Ryy) = 0 when
y & Y*. The expected return (reward) under the policy T8
{() = yy (u)) [R(y)] = E, R(y))o(ylx))
With the binary reward, the expected return is essentially the probability of producing a correct output under the current policy Ty. In the context of RL, algorithms. According to the policy gradient, the gradient of this objective can be written as:
update the policy using policy gradient
(o) E ( .>) [R()) Ve log No (y]=))
The policy gradient methods maximize the expected total reward by repeatedly estimating the gradient. Several policy gradient algorithms have been proposed such as Proximal Policy Optimization [29].
Group Relative Policy Optimization. In DeepSeek-Matt 30 researchers proposed a new policy gradient method, Group Relative Policy Optimization (GRPO), which improves the effectiveness and efficiency of RL training based on Proximal Policy Optimization (PPO) 229 The PPO algorithm maximizes the following surrogate objective:
To(or VeL) .Ta.u (or (ax, Wec))
To(od 2z, Vee) Te (or (z, Wee) 1- e. 4..
)al)
(0) = Ey--ro (:1=)
As, clip
(3)
where Tew is the policy before the update, is the clip hyperparameter, and A, is the advantage of the t-th token. In PPO, a value model is learned to estimate the advantage based on the reward R. which computationally expensive. Thus, GRPO uses the average reward of multiple sampled outputs as bascline in advantage estimation. Specifically, for each input æ, GRPO samples a group of outputs {vI,.., ... VG) and computes their rewards {R1, R& }. then sets the advantage of all tokens from y as A₁,t = (R, mean( ([R1, Ral))/ /std)))s (RR,,.., Ra)). The GRPO objective can be written as: JARPO (0)
Ey-r(-la)
[v]
Te(0s,1 vc) (1,
To( w. vr, 1 1 +e) Acc A..].
(4)
min
Ace. clip
1K
Kem (oc, (7, (t.
Tow (oe, (r, 2))
3

---

Compared to PPO, GRPO foregoes the value model to estimate udvantages that is typically the same size the policy model, relieving the memory and computational burden.
4 Theoretical Analysis
In this section, we present theoretical analysis that characterizes how the simplified and prevalent binary reward format (i.e., 1/0) affects the learning dynamics and induces hallucinations. The analysis can extend any arbitrary bounded real rewards.
Theorem 4.1 (Variance of Policy Gradient in Binary Reward RL). Under a binary reward function, the variance of the policy gradient estimator is directly proportional to the probability of generating correct answers. Let p denote the probability of producing a correct answer, so the variance of the policy gradient estimator is on the order of p(1 -p)l Vo log Toly" x>]?? (where y" is a correct answer). For small p. this variance extremely high, hindering stable learning.
Proof. The REINFORCE gradient for given input is g = R(y) V log no (y)ar)) This is random vector depending on the sampled output y. There is a p chance that the sampled output is correct (yielding R = 1) and 1 p chance that it is wrong (yielding R 0). Hence, the gradient g is zero with probability
p. and equals V, log To 2>) with probability p. The expected gradient is Big] p Va a log To To (v"[xz) The second moment is E[llg|l = p Vo log To (y))1= Therefore, the covariance (and in particular the trace as a measure of variance magnitude) is on the order of
p
|| Vo log To |=] lP a log To (z))) = p(1 - p) Ve log To (y"11)))
(5)
For very small p. the factor 1 p is near so roughly Var(g) p || log To(y") Il. other words, when the model has low chance to generate correct answers, the gradient estimator is almost always zero but occasionally a large spike, yielding extremely high variance relative to its mean. This high variance makes the training updates very unstable, as gradient estimates will fluctuate widely between zero and large values. We conclude that, with a binary reward, most of the time the model receives no learning signal, and when it does, the signal is an outsized jump.
Theorem 4.2 (Exploration (Exploration-Explootttoo Exploitation Trade-off and Hallucination Risk). Given the high-variance gradient under a binary reward, maximizing the expected binary reward objective I(0) will enforce an implicit constraint that the entropy Ho (x) remains sufficiently large. Let be small constant reflecting minimum acceptable exploration probability. To avoid stagnation at incorrect local optima without gradient feedback, the optimal policy update rule must satisfy: Ho(x) Z (c) > O.
Proof. The goal of policy model is to increase the probability of producing correct answers. In early stages or under uncertain knowledge (i.c., p is small), gradient updates are extremely noisy, often zero or very large (Theorem 4.1. To reliably find rewarding outputs, the agent must ensure it samples broadly from its policy distribution. Given c > O the minimal exploration threshold of ensuring a non-negligible probability of sampling potentially correct outputs, the exploration constraint is:
(u(x) 2 Vy E Y Yunexplored;
where Yunesplored is subset of unexplored outputs that have unknown correctness. Maintaining such a constraint implies a sufficiently high entropy Ha (z), because low entropy distributions concentrate their mass on few outputs, increasing the risk that these outputs yield zero reward consistently:
Ho(x) = = -
Σ x
(yg=) ((y 2 Hmin (e) > 0.
(7)
The necessity for exploration ensure non-stagnation zero-reward local minima introduces the trade-off. On one hand, the entropy must remain relatively high to explore potential answers, increasing the probability of hallucinated outputs. On the other hand, maximizing immediate reward requires concentrating distribution on already known rewarding answers.
Theorem 4.3 (Local Optima and Incorrect Convergence). Binary reward modeling admits spurious local optima where the policy produces incorrect answers with high confidence and receives zero reward, yet has no gradient incentive to change. Any policy that deterministically outputs a particular wrong answer is a stationary point of the optimization, ie., once the model heavily commits wrong answer, the policy gradient provides no immediate push away from that policy.

---

Factuality-aware Step-wise Policy Optimization
correct answer Group of Generated Reasoning Paths
Reword Function
Reword
Adventage
LGi
01,2
01,3
Ansswor
Answer Correctress
R₂
A
Query
Policy Model
02
22,3
Ardwer 2
Reward
R3
A
Step-wise Factuality
step
stop
OG11
0G2
0G3
Answer &
Reworrd
R
Aa
Factuality-aware
Advantage Adjustment
Aaz
Age
Riuctulity
Query: Which
Mera
Step-wise Factuality Reward
Dharam or Vineysgar?
Stea 1: First, I need to recall the release dates of both films, Neutrol
<thiikk> First, 1 need to recall the release detes of both films. know that, Hers Karam Dharam other hand, was released Vinayagar, in 1995. Compering these two years, 1995 is earlier than 1998, </thinks Vinayagar
Step 2: I know that Mere Karem Mers Dheram was releesed 1996. X entail?
Stop 3: Vinayagar, on the other hand, was released 1995.
Comparing these years,
earlier than 1998. Neutral
Evidence
Answer Correctness Reward
Answer: Vinayagar
X
Ground Karam
Figure 3: Overall framework of our proposed FSPO approach.
Proof. Given the objective R it is maximized at J = 1 when the policy outputs correct answer with probability 1. However, when a policy is certain but wrong for some incorrect answers Vfalse, the probability To (o
1 (so (y (x) = 0 for all correct answers Y*). For this policy, the expected reward [()) = 0. Importantly, the gradient VoJ((0) is zero as well, since VoJ = E (y())) V log To (y|=r) and Ryy) is always O under this policy (only generating incorrect answers). Thus, / a stationary point (the gradient is zero and no update is favored).
In strict mathematical sense, the policy of always outputting a particular wrong answer is actually local minimum of the objective (0), since any small increase the probability would increase [(0) slightly. However, the surface around it is a broad, flat plateau at reward zero for every direction not immediately introducing the correct answer. Since REINFORCE updates are driven solely by sampled positive rewards, the algorithm has no gradient signal to push the parameters off this plateau. effect, the training has converged to a policy that outputs wrong answer with full confidence, and because it never explores the correct answer, it gets no feedback. We call such states spurious local optima: the model is confidently wrong and training fails to progress.
Approach
To mitigate the hallucination issue during large-scale RL training, we propose the Factuality- Step-wise Policy Optimization (FSPO) algorithm. As shown Figure by integrating step-wise factuality signals into the policy optimization process, our approach encourages the model to generate more factual and faithful reasoning steps.
5.1 Reward Function Design
The core of reinforcement learning is to design appropriate reward functions that encourage high- quality reasoning trajectories and penalizes low-quality ones, guiding the policy model towards better responses using the RL algorithm. In our method, we introduce two rewards, i.e., answer correctness reward and step-wise factuality reward.
Step-wise Factuality Reward. Typically, factuality is mainly related to world knowledge, so we focus on complex knowledge-intensing tasks in this work, e.g., multi-hop question answering [4]]. Specifically, for an input question a source such as Wikipedia, denoted by IC, which can be called evidence. We first split the reasoning text into multiple sentences steps, denoted by {21, ..., 2N J. and then employ LLM to determine whether the evidence IC can entail each sentence Zj to assign a step-wise factuality reward as:
assume that it can be associated with knowledge snippets in
1. if the sentence 2j can be entailed from the evidence KC
Riactulity (2snually(22)_)))
0. if the sentence 2j is neutral the evidence KC
(8)
1, if the sentence 2, contradicts the evidence K.
5

---

We observe that the policy model might generate texts unrelated to the question in the reasoning process such connective phrases or exploratory sentences (c.g.. "Aha". "Wait"), which can be regarded as neutral to the evidence.
Answer Correctness Reward. most existing research usually adopted due to its significant scalability and simplicity. Specially, given an input prompt the policy model generates an output y, which includes intermediate reasoning texts followed by the final answer. We use rule-based reward function to evaluate answer's correctness as follows:
answer correctness reward is
1, if the final answer fully matches the ground truth
Ranswer (y)
0, the final answer does not match the ground truth
We simplify the answer reward design aiming to alleviate reward hacking and promote more diverse problem-solving behaviors based on mere outcome feedback.
To reduce the effect learning from sparse outcome reward, the final reward for an output y consists of both answer correctness and step-wise factuality rewards, which can provide denser feedback and mitigate the high-variance gradient issue (Theorem 4.1p in typical RL binary reward modeling:
1 N
Rimaal (y) Ranswer (v)
Riactualtyy (zz).
(10)
5.2 Factuality-Aware Policy Optimization
For online policy optimization, adopt Group Relative Policy Optimization (GRPO) [30], which samples a group of outputs .y...... yc for each input x and then obtains their rewards {R, Ra} compute the advantage A; for yi A; (R, -mean( (rean((R1,.0 (Ri, Ra (Ba))) / /std (R1, Ra))
Factuality-Awar Advantage Adjustment. Most existing work assigns the same advantage (derived from the overall response) to all tokens [30 38, which is suboptimal and does not differentiate between them. We argue that, regardless of whether the final answer is correct, factually correct reasoning tokens should be encouraged, while incorrect reasoning tokens should be penalized. Therefore, we adjust the advantage of each token oi,e E æj for the output yi as follows:
Ai
if A; 0 Reactuality (2z) 1 or A, 0 Reactuality (2j) -1
A,
if A 0 A R Ractuallty (2z)
1 A, < 0 A R (2z) =1 (11)
Ais
if A; or Raccuality (zj) =
In this manner, each token og, is re- re-weighted by the factuality- aware advantage A.e. Whenever the generated reasoning contains at least one sentence that is contradicted or entailed by the evidence (Rinetulity = 1, -)), the advantages for tokens inside that sentence are zero, even if the overall answer is still wrong. Consequently, the gradient is strictly non-zero, supplying a descent (or ascent) tokens and toward factually-
direction that pushes probability mass away from factually-incoree verified ones and enabling gradient-based learning to leave the spurious local optima (Theorem 4.3). Based on the adjusted advantage, we adopt the objective in Eq. to optimize the policy model:
JESPo (0) = (-y=((-l))
no(0s, x. Mi.sse ALC, Fo (oi, (2, Vi.se))
(e, To (or,e vi.et) e, 1 +c) Aid]3.
(12)
min
Experiments
In this section, we present our experimental setup, the main results, and our analysis of the results.
6.1 Experimental Setup
Datasets and Metrics. To perform FSPO, we adopt a challenging subset of HotpotQA and
2WikiMultHHpQA [|] from previous work (33] and randomly select 2K examples for training

---

(including questions, answers, and corresponding Wikipedia segments). Besides, to improve the complex reasoning capabilities of LLMs, we incorporate the SimpleRL dataset 45 with 8K mathe- matical problems for vanilla RI training. For hallucination evaluation, we adopt three widely-used benchmarks, i.e., TruthfulQA (20), HaluEval [T9], and HalluQA 4 For HaluEval, we adopt the QA subset and report the accuracy with which the model can discriminate correct and hallucinated answers. For TruthfulQA and HalluQA, we report the ratio of outputs that the judge model predicts as
truthful. We employ GSM8K [5], MATH-500 [0), AIME 2024 (22). and AIME 2025 [23] to evaluate mathematical reasoning performance and report the Pass accuracy results in these datasets. For more details, refer to Appendix C
Baselines. We employ Qwen2.5-7BB D0wen2.5-7BBBuss/nnstuuu [39 and LIama3. 1-8B-Instruct [ as backbone models for our approach. We compare our model against three groups of basclines:
A API-based Models include DeepSeek-V V3
21 DeepSeek-R1
9 OpenAl GPT-4o, and GPT-ol.
Reasoning Models consist of QwQ-32B 28 Distill-Qwen-P1 Distil--Qwen-7BB114B3BBB and Distill-Llama-88 (9)
which are fine-tuned on the long CoT data (800K samples) synthesized from DeepSeek-R1.
Open-source Models involve direct comparisons and can be easily accessed through their open-source weights.
Baael.5-7BB-Bse/Inssrre and L-ama3.1-8B- 1-8B-Insttruct, which are
Implementation Details. We train our model using the verl [31] framework. Specifically, during
training, we use a batch size of 8, generate 8 rollouts for each prompt, set maximum prompt and response length of 2,048 tokens, and train using a mini-batch size of 1,024. We train our model on the mixture dataset of HotpotQA subset and SimpleRL for one epoch with constant learning rate of 4e-7 and temperature parameter of 1.0. The coefficient for KL loss is set to 1e-3 and the clip ratio is 0.2. We adopt the HHEM-2.1 model [14] to verify step-wise factuality. During evaluation, set the sampling temperature to 1.0. We present prompt templates throughout this work in Appendix F
6.2 Main Results
The experimental results presented in Table demonstrate the performance of our proposed method, FSPO, across various reasoning and hallucination benchmarks compared to multiple competitive models, including API-based, reasoning, and open-source alternatives. On hallucination benchmarks, FSPO clearly outperforms all open-source, reasoning and even some API-based models. Specifically, FSPO based on Qwen2.5-7B Benn2.5-7--Bas achieves the highest scores among all open-source and reasoning models on hallucination datasets, clearly demonstrating the effectiveness of our training approach in substantially reducing hallucinations and enhancing factual correctness.
On reasoning benchmarks, FSPO achieves superior results within the open-source category, notably surpassing other base models like Qwen2.5-7B-Inss -nIstrucctt and Llama3. 1- 1--1--88B-stt Instructt by significant margins (e.g., GSM8K 89.5% 19.5% vs. 73.2% and 77.5%. respectively). FSPO even surpasses some reasoning models on GSMSK, and maintains competitive performance on MATH-500, This shows that our FSPO method can enhance reasoning capabilities while reducing hallucinations.
In terms of advanced reasoning tasks, FSPO delivers competitive outcomes, especially on AIME 2024 and 2025, with scores of 20.0% and 13.3% respectively. These scores notably surpass smaller base models, although trailing the reasoning and much larger API-based models. Such results underscore FSPO's robust mathematical reasoning capabilities, showcasing the efficacy of our specialized fine- tuning methods in elevating performance on complex mathematical reasoning tasks despite the base model's comparatively modest size.
6.3 Further Analysis
We further conduct detailed analysis on several key aspects of our approach. Due to similar findings, the following analysis is focused on MATH-500 and HaluEval-QA with Qwen2.5-7B-Bar Base.
Ablation Analysis. To validate the effectiveness of our approach, we ablate its key design elements. We compare FSPO to two variants: 1) GRPO trains the model on the mixture dataset using typical GRPO with only answer reward in Eq. 9 and 2) GRPO w/ factuality incorporates both answer and factuality rewards in Eq. 10 The results from Figure 4 further support the effectiveness of FSPO,
7

---

Table Evaluation results of closed- source, distilled, and open-source models across four reasoning benchmarks and three hallucination benchmarks. The bold and underlined scores denote the best and second best results among open-source models and our trained models.
MATH
AIME24 (Pass@1)
AIME25 (Pass @
Model
GSM8K
TruthfulQA1
HaluEval-QAt
HalluQAt
500
API-based Models
DeepSeck-V2
89.3
90.2
39.2
26.6
53.0
70.6
59.1
DeepSeek-RI
92.0
97.3
79.8
66.7
45.2
51.1
GPT-40-0513
90.8
74.6
9.3
13.3
59.0
62.6
53.1
GPT-o1-1217
92.3
96.4
79.2
76.7
62.4
70.2
56.0
Reasoning Models
QwQ:32B
88,6
89.8
79.5
56.7
23.2
29.6
23.7
RI-Distill-OO
Distill-Qen-32BB
87.4
94.3
72.6
53.3
19.7
33.5
26.9
RI-Distill-Qwees Qwen-1eeennB
85.1
93.9
69.7
50.0
18.2
24.8
20.2
RI-Distill- RI-Distil-QQwen-2E Qwen-7B
84.3
92.8
55.5
33.3
6.9
3.1
E1-Distill-Llama-B8
82.1
89.1
50.4
26.7
8.8
14.6
4.6
Open-source Models
Owcn2.5-7B-Basc
65.2
35.7
3.3
3.3
38.2
48.0
39,5
Bren2.5-7B-Instraae
73.2
51.6
6.7
3.3
36.7
38.4
32.0
Llama3. I-8t-Instruct
77.5
33.1
6.7
0.0
26.4
36.7
12.2
FSPO (Qwen-Base)
89.5
75.5
16.7
13.3
58.4
83.0
52.0
ESPO (Qwen-Instruct)
89.4
74.7
20.0
13.3
$4.0
64.7
50.0
FSPO Illaa-Instrucc)
86.2
68.3
13.3
6.7
41.1
67.1
42.0
(Accuracy)
(Accuracy)
(Accuracy)
HaluEval-QA (Accuracy)
ESPO
FSPO GIPO
GRPD wi Fachal by
FSPO Sanitrrree-
ORPO
Berfforcen
360
3000
2008
2005
550
1007
1548
2003
1006
1200
2000
306
10.00
10.00
32.0
Figure 4: Pass accuracy over training steps for ablation analysis.
Figure 5: Pass@ accuracy over training steps with different RI algorithms.
clearly outperforming baseline methods GRPO and GRPO with factuality scores across training steps on both MATH-500 and HaluEval-QA FSPO maintains consistently superior reasoning performance while achieving continuous improvement in reducing hallucinations, indicating robust generalization capabilities and the practical benefits of our proposed method.
RL Algorithm Analysis. We further adapt FSPO the Reinforce++ algorithm 12 for validating the adaptability and effectiveness of FSPO beyond GRPO. As shown in Figure FSPO with Reinforce++ exhibits significant improvement over Reinforce++ alone, closely following the performance of FSPO with GRPO. Moreover, FSPO with GRPO consistently achieves higher accuracy than the Reinforce++ baselines across training steps on both the MATH-500 and HaluEval-QA benchmarks. These findings indicate that our proposed FSPO approach generalizes well to other reinforcement learning algorithms, enhancing their effectiveness and stability in solving complex reasoning and reducing hallucinations.
Number of Training Samples. To investigate the model performance under varying numbers of training samples, we randomly sample IK, 2K, 4K, and 8K samples from the HotpotQA subset [3 with the results shown Figure On MATH-500, the accuracies are similar from 1K 4K samples while diminishing with 8K samples, which might be because more factual QA data lower the math reasoning performance. In contrast, on the HaluEval-QA task, significant accuracy improvement continues up to 2K samples, with 4K and 8K samples achieving mediocre results. This demonstrates that our approach can significantly reduce hallucinations using only small amount of data, without compromising the model's capacity for complex reasoning.

---

MATH-500 (Accuracy)
HaluEval-QA (Accuracy)
Average Response Length
Average Factuality Score
T2PO
ORPO
0.5
02
502
30
03-
440
-0.2
13
8C
1503 2500
322
1003
1000
2002
3500
Figure 6: Pass @1 accuracy over training steps
Figure 7: Average response length and factuality score for FSPO and GRPO.
with varying number of training samples.
Factuality Improvement Analysis. We further explore the benefit of incorporating step-wise factuality reward in our approach. The results in Figure illustrate that both FSPO and GRPO generate responses of similar lengths. However, as our proposed FSPO approach explicitly optimizes for factual correctness during intermediate reasoning steps, achieves significantly higher factuality scores compared to GRPO. This suggests that FSPO effectively enhances model reliability and factual accuracy without compromising the quality of the generated content.
7
Related Work
Large Reasoning Models. Recent advancements in large reasoning models have marked significant shift toward explicitly improving complex reasoning, planning, and problem-solving capabilities (34
OpenAl's o1 [15] exemplifies the integration of RI. with long CoT reasoning, enabling the model to iteratively refine its reasoning process before producing an answer. This approach significantly enhances its capability in complex problem-sol
scenarios, particularly in mathematical reasoning and algorithmic tasks. DeepSeek-R1 [9] and Kimi k1.5 [77] similarly emphasize RL-based reasoning but introduce open and flexible training pipeline. QwQ model (28] underscores the potential of smaller models when optimized specifically for reasoning tasks. Increasing efforts are devoted to reproducing prominent reasoning performance of these models [43 27 33. However, existing research has predominantly concentrated on enhancing the reasoning capabilities of models, while overlooking the issue of hallucinations emerging within these models.
Hallucination. Hallucination has been fundamental challenge in large language models (LLMs), receiving extensive attention in existing literature [
16 46). To investigate and detect halluci-
nations in LLMs, in scenarios where LLMs provide access to their internal states, researchers can closely examine their internals uncover underlying mechanisms leading to hallucinations [36] 44) Conversely, when the internal states of LLMs are inaccessible and they can only be queried through external API calls, researchers typically approach ballucination detection by examining correlations between imput prompts and generated responses [24] 42). To mitigate hallucinations, existing studies propose various strategies. During the pre-training phase, mitigation efforts typically revolve around curating and cleaning datasets 7 35). Following pre-trate gre-training, fine-tuning methods such as supervised fine-tuning (37] and reinforcement learning with human feedback (26) are commonly employed to further mitigate hallucinations. Additionally, post-processing mitigation involves verifying the factual correctness of generated content using either the LLM itself (25 or external knowledge-baad verifi- cation methods [I] Our work is an early effort focusing on analyzing and mitigating hallucinations large reasoning models.
Conclusion
In this paper, we have highlighted and theoretically explained a critical limitation associated with RL-based fine-tuning of LLMs, specifically the increased incidence of hallucinations generated reasoning content. Our analysis attributed this issue to unstable training dynamics driven by high- gradient variance, entropy-induced randomness, and convergence toward spurious local optima inherent in traditional outcome-based RL. training. To address these challenges, we introduced FSPO, a novel factuality-awar RI. algorithm that integrates step-wise factual verification, providing a denser and more accurate feedback mechanism during training. Comprehensive experimental evaluations

---

demonstrate that FSPO significantly mitigates hallucinations across multiple benchmarks without compromising the models' reasoning capabilities.
Acknowledgments
This research is supported by the National Research Foundation Singapore under its Al Singapore Programme (Award Number:
References
[1] Anthony Chen, Panupong Pasupat, Sameer Singh, Hongrae Lee, and Kelvin Guu. PURR: Efficiently Editing Language Model Hallucinations by Denoising Language Model Corruptions. arXiv preprint arXi:: 14908, 2023.
[2] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss. William Hebgen Guss, Alex Nichol, Alex Paino. Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating Large Language Models Trained Code. CORR, abs/2107.03377 2021.
[3] Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, et al. An Empirical Study on Eliciting and Improving RI Rl-like Reasoning Models. arXiv preprint wxXhv:2503004546 2025.
[4] Qinyuan Cheng, Tianxiang Sun, Wenwei Zhang, Siyin Wang, Xiangyang Liu, Mozhi Zhang, Jun- liang He, Mianqiu Huang, Zhangyue Yin, Kai Chen, and Xipeng Qiu. Evaluating Hallucinations in Chinese Large Language Models. CoRR, abs/23 10.03368, 2023.
[5] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers Solve Math Word Problems. arXiv preprint arXiv:2110.1416 2021.
[6] Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang, Siyuan Zhuang. Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, et al. The Danger of Overthinking: Examining the Reasoning- Action Dilemma in Agentic Tasks. arXiv preprint arXiv:2502.08233, 2025.
[7] Souvik Das, Sougata Saha, and Rohini K Srihari. Diving Deep into Modes of Fact Hallucinations Dialogue Systems. arXiv preprint arXiv:2301.04449 2023.
[8] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 Herd of Models. arXiv preprint 24X 2024.
[9] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501. 2025.
[10] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring Mathematical Problem Solving With the MATH Dataset. NewrIPS, 2021.
[11] Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, December 2020.
10

---

[12] Jian Hu. REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models. arXiv preprint arXiv:2501.03262, 2025.
[13] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qiang- long Chen, Weihua Peng. Xiaocheng Feng, Bing Qin, and Ting Liu. A Survey on Hallucination Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Truns. Inf. Syst. 43(2), January 2025. ISSN 1046-8188.
[14]
Simon Hughes and Minseok Bae. Vectara Hallucination Leaderboard, 11 2023. URL https //github. com/ http:///eeaee/aaaalaaaaaaaaaaaaa
[15] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, al. OpenAl ol System Card. arXiv preprint arXiv:2412.167200 2024.
[16] Ziwei Ji. Naycon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12):1-38, 2023.
[17] Kimi-Team. Kimi k1. 5: Scaling Reinforcement Learning with LLMs. arXiv preprint arXiv:2501.12599 2025.
[18] Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration Deep Reinforce- ment Learning: A Survey. Information Fusion, 85:1-22, 2022.
[19] Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian- Yun Nie, and Ji-Rong Wen. HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing. EMNLP 2023, Singapore, December 6-10, 2023, pages 6449 6464. Association for Computational Linguistics, 2023.
[20] Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214-3252. Association for Computational Linguistics. 2022.
[21] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 Technical Report. arXiv preprint axXiv:2412.19447,, 2024.
[22] MAA. American Invitational Mathematics Examination -IME, 2024.
[23] MAA. American Invitational Mathematics Examination AIME, 2025.
[24] Potsawee Manakul, Adian Liusie, and Mark JF Gales. SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models, arXiv preprint arXiv:2303.08890 2023.
[25] Niels Mündler, Jingxuan He, Slobodan Jenko, and Martin Vechev. Self-contradictory Hallu- cinations of Large Language Models: Evaluation, Detection and Mitigation. arXiv preprint arXiv:2305. 15852, 2023.
[26] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[27] Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, et al. 01 Replication Journey: A Strategic Progress Report- Part arXiv preprint arXiv:2410. 18982, 2024.
[28) Qwen-Team. QwQ-32B: Embracing the Power of Reinforcement Learning, March 2025. URL http//ww-3330 32b/
https://qwenlm. github i booo/llg///o

---

[29]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347 2017.
[30] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint vrXiv:2402.03300 2024.
[31] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang. Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A Flexible and Efficient RLHF Framework. arXiv preprini arviv: 2409. 19256, 2024.
[32] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of Go with deep neural networks and tree search. nature, 529(7587):484-48) 2016.
[33] Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. RI-Searcher: Incentivizing the Search Capability LLMs via Reinforcement Learning. arXiv preprint arXiv:2503.05500 2025.
[34] Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jianing Qiu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, et al. A Survey of Reasoning with Foundation Models. arXiv preprint arXiv:2312. 2023.
[35] Logesh Kumar Umapathi, Ankit Pal, and Malaikannan Sankarasubbu. Med-HALT: Medical Domain Hallucination Test for Large Language Models. arXiv preprint arXiv:2307.15343, 2023.
[36] Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generation. arXiv preprint wxXiv:2307.03993 2023.
[37] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-Instruct: Aligning Language Models with Self-Generated Instructions. arXiv preprint arXiv:2212./0566) 2022.
[38] Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning. arXiv preprint arXiv:2502. 14768, 2025.
[39] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui. Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 Technical Report. arXiv preprint arXiv:2412.151: 15115, 2024.
[40] Jiashu Yang. Ningning Wang, Yian Zhao, Chaoran Feng, Junjia Du, Hao Pang, Zhirui Fang, and Xuxin Cheng. Kongzi: A Historical Large Language Model with Fact Enhancement. arXiv preprint arXiv:2504.09488, 2025.
[41] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-23330 2380, 2018.
[42] Jia-Yu Yao, Kun-Peng Ning, Zhen-Hui Liu, Mu-Nan Ning, and Li Yuan. LLM Lies: Hallucina- tions are not Bugs, but Features as Adversarial Examples. arXiv preprint 10.0v:211000146) 2023.
[43] Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. LIMO: Less is More for Reasoning. arXiv preprint arXiv:2502.03387, 2025.
[44] Mert Yuksekgonul, Varun Chandrasekaran, Erik Jones, Suriya Gunasekar, Ranjita Naik, Hamid Palangi, Fee Kamar, and Besmira Nushi. Attention Satisfies: Constraint-Satisfaccion Lens on Factual Errors of Language Models. arXiv preprint 23t.:::300.10999 2023.
12

---

[45] Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 7B Model and 8K Examples: Emerging Reasoning with Reinforcement Learning is Both Effective and Efficient. htttps///kkuse nlp. notion. wwt.essimmteeeeeeeeeeeeee reason 2025. Notion Blog.
[46] Yue Zhang. Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren's s Song in the AI Ocean: A Survey on Hallucination in Large Language Models. CoRR, abs/2309.01219, 2023. doi: 10.48550AARXIV2230..0222
URL https
//doi. aeroi.org//1... 48550/ ar 0.309000220
NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]-
[NA] means either that the question is Not Applicable for that particular paper the
relevant information is Not Available.
Please provide short (1-2 sentence) justifiiation right after your answer (even for NA).
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction sections accurately reflect the paper's contribu- tions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discuss the limitations of work at Appendix A
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
13

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We have provided full assumptions and proof for our theoretical analysis in Section
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided all dataset and implementation details in Section 6. and
Appendix C and D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same
14

---

dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis-
sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-souree models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide full details of our datasets at Appendix
C
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.
ssinnoliccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide all necessary training and test details Section 6. Appendix D and the used prompt templates in Appendix F
15

---

Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
we provide the experimental results that support the main claims of our paper and conduct some detailed analysis to examine some hyper-parameters in our approach.
Justification: Section 0
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information about computer resources at Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpe///iiiee/////////////oooo/a http://wwrrr..ooooobo///cccccaca
16

---

Answer: [Yes]
Justification: Our paper follows the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discuss both potential positive societal impacts and negative societal impacts of our work at Appendix B
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML...
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our work uses existing open-source models and datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
17

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the papers for our used datasets and models and mentioned their license in Section 6.1 and Appendix C
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. www.coo//aaaaaaa con/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work uses existing datasets and models.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
18

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We do not involve crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this paper does involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.MM policy http:p////eaarrpppc.c//////o///a for what should or should not be described.
19

---

Technical Appendices and Supplementary Material
A
Limitations
Our theoretical analysis is mainly focused on the prevalent and simplified binary reward modeling. Future work can extend it any arbitrary bounded real rewards. The scope of our experiments constrained by the substantial computational cost of much larger models (e.g., 14B, 32B). Future work can explore applying our approach larger models.
B Broader Impact
Our proposed FSPO algorithm has the potential to significantly improve the trustworthiness and reliability of LLMs in critical applications, such as education, healthcare, and scientific research, by reducing factual hallucinations during complex reasoning. However, the same technology may also pose negative societal risks if misused, for example, it could be applied to train persuasive yet selectively factual systems in domains such political propaganda or misinformation campaigns, where step-wise coherence may mask underlying biases.
C
Datasets
To reduce hallucinations while improving the complex reasoning capabilities of LLMs, mix two datasets for fine-tuning base models, i.c., a subset of HotpotQA [33] and SimpleRL [45]. Previous study [33] construct a subset from HotpotQA [4]] and 2WikiMultHHoPAA I with varying difficulty levels including 4761 and 3737 samples from HotpotQA and 2WikiMultHHpQAA respectively. Each sample includes the question, ground-truth answer, and the corresponding segments from Wikipedia. We randomly select 2K samples from this subset for conducting our proposed ESPO approach. SimpleRL consists of 8523 mathematical samples, which are used to perform traditional RL fine- tuning based on GRPO in our training. For hallucination datasets, TruthfulQA (20), HaluEval- QA 199, and HalluQA 4 contain 817, 10000, 450 samples, respectively. HalluQA Chinese dataset while the others are English datasets. For mathematical reasoning datasets, GSM-8K (5) and MATH-500 [10] contains 1319 and 500 samples, respectively. There are 30 samples in both AIME 2024 222 and AIME 2025 [22] These datasets are in MIT or Apache-2.0 license.
D Training Infrastructure
We train our model on 4xH100 80G GPU cards for about 24 hours each time. We train our model using the verl [31 framework. During training, we use train batch size of 8, generate 8 rollouts for each prompt, set a maximum prompt and response length of 2,048 tokens, and train using mini-batch size of 1,024. We use constant learning rate of 4e-7 and temperature parameter of 1.0. The coefficient for KL loss is set to le-3 and the clip ratio 0.2. During evaluation, we set the sampling temperature 1.0 for our models and all baselines.
E
Preliminary Experiments
To demystify the question of "To what extent do reasoning models maintain factuality after RL fine-tuning??" conduct preliminary experiments three hallucination benchmarks with twelve LLMs. The results show that after large-scale training with RL or long CoT data, models demonstrate significantly higher degree of hallucinations. To investigate the underlying reasons, randomly select 200 samples (i.e.. examples that DeepSeek- V3 answers correctly but DeepSeek-RI does not) from HaluEval-QA and manually check their hallucination reasons. Specifically, the hallucination sources within our experiments can be roughly categorized into four types:
This is the most common source where the intermediate reasoning process is incorrect, which naturally results in wrong answers.
Intermediate Reasoning Error:
Reasoning and Answer Inconsistency: This the second most common source for which the intermediate reasoning process is correct but the final answer is wrong.
20

---

Table 2: Prompt for base and instruct models in mathematical reasoning tasks.
Model Type
Prompt
conversation between User and Assistant, The user asks question, and the assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the final answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively. For example, <think> reasoning process here </think> <answer> answer here </answer>. The Assistant shows the reasoning process within <think> </think> tags, and ONLY return the FINAL AN- SWER within <answer> </answer> tags. For example: <answer> </answer>, \n/,n User: [question] An Assistant: Let me solve this step by step. <think
Base model
You are a helpful assistant. Given a question, you need to think about the reasoning process in the mind and then provide the final answer. The reasoning pracess and answer enclosed within <think> </think> and <answer> </answer> tags, respectively. For example, <think reasoning process here </think> <answer> answer here </answer>, You must show reasoning process within <think> </think> tags, and ONLY return the FINAL ANSWER within <answer> </answer> tags. For example: <answer> 12 </answer>.
Instruct model
Table 3: Prompt for base and instruct models in question answering tasks.
Model Type
Prompt
A conversation between User and Assistant. The user asks question, and the assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the final answer. The reasoning process and within <think </think> and <answer> </answer> tags, respectively. For example, <think> reasoning process here </think> <answer> answer here </answer>. The Assistant shows the reasoning process within <think </think> lags, and ONLY return the FINAL AN- SWER within <answer> </answer> tags. For example: <answer> Kim Marton </answer>. nn)n User: (question)
are enclosed
Base model
\n Assistant: me solve this step by step. <think>
You helpful assistant. Given a question, you need to think about the reasoning process in the mind and then provide final answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively. For example, <think> reasoning process here </think> <answer> answer here </answer>. You must show the reasoning process within <think> </think> tags, and ONLY return the FINAL ANSWER within <answer> </answer> tags. For example: <answer> Kim Marton
Instruct model
</answer>.
Overthink:
This occurs when the intermediate reasoning process contains correct segments that can lead to the right answers but the model still continues to reason further, finally getting the wrong answers. This phenomenon is called "overthink", investigated in extensive studies [6].
Others: This includes the remaining cases, such as the output text is incomplete or mismatch between generated answers and ground truth.
F Prompt Templates
We present our prompts used during training and evaluation in Table 2 Table 3 and Table
21

---

Table 4: Prompt for base and instruct models in HaluFval-QAA dataset.
Model Type
Prompt
A conversation between User and Assistant, The user asks question, and the assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the final answer. The reasoning process and within <think> </think> and <answer> </answer> tags, respectively. For example, <think> reasoning process here </think> <answer> answer here </answer>. Now the user asks question and provides two candidate answers. the assistant needs to determine which answer is correct. The Assistant shows the reasoning process within <think> </think>> tags, ONLY return FINAL ANSWER within <answer> </answer> tags. For example: <answer> Kim Marton </answer>, \n'n User: #Question#: (question) \n #Candidate Answer 1#: (candidate_nt _an_aarr____s___________ | \n #Candidate Answer 2#: soandddae_answeet 2 \n Assistant: Let solve this step by step. <think>
are enclosed
Base model
You helpful assistant. Given a question, you need to first think about the reasoning process in the mind and then provide the final answer. The reasoning process and answer are enclosed within <think </think> and <answer> </answer> tags, respectively. For example, <think reasoning process here </think> <answer> answer here </answer>, Now the user asks question and provides two candidate answers, you need to determine which answer is correct. You must show the reasoning process within <think> </think> tags, and ONLY return the FINAL ANSWER within <answer> </answer> tags, such <answer> Kim Marton </answer>.
Instruct model
22


Reviews:
[{"review_id": "6NrM1Si6xC", "reviewer": "Reviewer_gdjF", "summary": "The paper investigates the issue of increased hallucinations (factual inaccuracies or fabricated statements) in large language models (LLMs) trained with reinforcement learning (RL) for reasoning tasks. To address the issue of hallucination, the paper introduces the Factuality-aware Step-wise Policy Optimization (FSPO), an RL algorithm that explicitly integrates factuality verification at each reasoning step. FSPO dynamically adjusts token-level advantage values, thereby encouraging factual correctness throughout the reasoning process. Empirical evaluations on various reasoning and hallucination benchmarks using models such as Qwen2.5 and Llama demonstrate significant reductions in hallucinations and improvements in reasoning accuracy.", "questions": "1. How well does FSPO scale to smaller or larger LLMs? Additional experiments on models with different sizes will benefit the claims.\n2. Could you elaborate on the sensitivity of FSPO to the accuracy of the external factuality verifiers? \n3. Can you provide a time complexity analysis of FSPO compared with traditional RL algorithms since you introduce external verifiers?\n4. Can you analyze why using FSPO is even a bit better than GRPO on MATH500?\n5. Did you try other verifiers? Why you choose your current one?\n6. The experiments show results over relatively short training periods. How does FSPO behavior evolve over extended training? Are there concerns about the factuality verifier becoming a bottleneck or introducing training instabilities over longer horizons? Can FSPO benefit for multi-turn training?", "limitations": "1. The effectiveness of FSPO heavily depends on the quality of external factual verifiers. The sensitivity of the overall system performance to these verifiers is not deeply explored. No discussion of potential biases in factuality verifiers and their impact on training.\n2. computational overhead of introducing factual verifiers\n3. Given that larger models often exhibit different behaviors, the generalizability to state-of-the-art models (30B+ parameters) remains unclear. But I acknowledge that the training cost might be too expensive to do.\n4. The evaluation focuses primarily on knowledge-intensive QA tasks and mathematical reasoning. The generalizability to other reasoning domains is not explored.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "1. the motivation is clear. \n2. the paper provides theoretical analysis on the reasons behind more hallucinations after RL training. The identification of mechanistic causes for RL-induced hallucinations is insightful.\n3. the paper structure and clarity are clear to me.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. The following is our response point-to-point.\n\n**Q1: How well does FSPO scale to smaller or larger LLMs? Additional experiments on models with different sizes will benefit the claims.**\n\nWe apply FSPO to smaller models (i.e., Qwen2.5 0.5B and 1.5B Base) and larger models (i.e., Qwen2.5 14B Base). Their results are shown in the table below, which are consistent with the conclusions reached in our paper. \n\n| Model                | GSM8K | MATH  | AIME24 | AIME25 | TruthfulQA | HaluEval | HaluQA |\n|---------------------|-------|-------|--------|--------|------------|----------|--------|\n| Qwen2.5-0.5B-Base        | 36.7  | 15.8  | 0.0    | 0.0    | 15.5       | 16.2     | 16.4   |\n| w/ FSPO             | 52.4  | 36.7  | 0.0    | 0.0    | 22.5       | 34.1     | 24.9   |\n| Qwen2.5-1.5B-Base        | 55.7  | 29.6  | 0.0    | 0.0    | 27.9       | 28.0     | 23.9   |\n| w/ FSPO             | 74.4  | 61.2  | 3.3    | 3.3    | 35.1       | 64.6     | 35.6   |\n| Qwen2.5-14B-Base         | 91.6  | 65.4  | 6.7    | 6.7    | 51.9       | 45.0     | 58.0   |\n| w/ FSPO             | 92.2  | 80.0  | 13.3   | 16.7   | 61.1       | 81.2     | 68.7   |\n\n**Q2: Could you elaborate on the sensitivity of FSPO to the accuracy of the external factuality verifiers?**\n\nTo clarify the sensitivity of FSPO to the accuracy of the external factuality verifier, we create pseudo-noisy factuality reward by replacing the original factuality reward with incorrect one in varying probabilities (20%, 40%, 60%, 80%), to mimic different accuracy levels of the external verifier. The results are shown in the following table. We can see that with more accurate factuality prediction, our approach achieves better reasoning performance and fewer hallucinations.\n\n| Model                         | GSM8K | MATH  | AIME24 | AIME25 | TruthfulQA | HaluEval | HaluQA |\n|------------------------------|-------|-------|--------|--------|------------|----------|--------|\n| Qwen2.5-7B-Base w/ FSPO      | 89.5  | 75.5  | 16.7   | 13.3   | 58.4       | 83.0     | 52.0   |\n| 20%                          | 89.1  | 74.2  | 16.7   | 16.7   | 55.1       | 78.3     | 50.1   |\n| 40%                          | 88.5  | 71.2  | 13.3   | 13.3   | 50.3       | 73.2     | 45.1   |\n| 60%                          | 81.6  | 65.2  | 6.7    | 6.7    | 42.5       | 64.3     | 39.8   |\n| 80%                          | 80.4  | 62.2  | 6.7    | 6.7    | 40.0       | 59.2     | 32.0   |\n\n\n**Q3: Can you provide a time complexity analysis of FSPO compared with traditional RL algorithms since you introduce external verifiers?**\n\nLet $L$ = number of tokens per sample, $P$ = number of model parameters (7B in our paper), $G$ = number of rollouts per prompt, $N$ ≈ number of sentences per rollout, and $C_v$ = time cost of one verifier call.\n\n* Standard GRPO/PPO: time taken ≈ $G \\times  L \\times P$.\n* FSPO: time taken ≈ $G \\times L  \\times P + G  \\times N \\times C_v$ (the second term incurred by stepwise verification)\n\nUsing 4×H100 (80 GB), our HHEM‑2.1 verifier (~400M) adds ~25 ms per sentence ($C_v$ = 25ms), while one policy forward/backward pass for a 7B model is ~300 ms for a 2K‑token rollout. Since the number of rollouts is set to 8 ($G$ = 8), the total update time is 2.4s. The average number of sentences for a rollout is ~10 ($N$ = 10) and the verification for 8 rollouts is done in parallel, so the verification cost is only ~0.25s. Roughly two‑thirds of the verifier’s runtime overlaps with the policy model’s forward/backward passes (they run in parallel on separate streams/devices), so only the remaining one‑third is visible as extra wall‑clock time 0.25 × (1/3) ≈ 0.08 s. The increased wall time percentage for FSPO is 0.08s/2.4s = 3.33%.\n\n**Q4: Can you analyze why using FSPO is even a bit better than GRPO on MATH500?**\n\nAlthough FSPO is mainly designed to reduce hallucinations in knowledge-intensive tasks, its stepwise reward shaping also encourages more coherent and stable reasoning trajectories. Even in math problems, models often “hallucinate” by introducing faulty intermediate steps (e.g., wrong expansions, arithmetic mistakes) that derail the final answer. FSPO’s stepwise reward penalizes such spurious steps and supplies denser, lower‑variance gradients, so the policy learns to keep each calculation logically consistent. This tighter control of intermediate reasoning, rather than external knowledge, is what enables FSPO to outperform GRPO on MATH500.\n\n**Q5: Did you try other verifiers? Why you choose your current one?**\n\nWe also tried using a general large language model (i.e., Llama3.1-8B-Instruct) to verify the factuality of each step and it achieves results similar to the current HHEM-2.1 model. We select the HHEM-2.1 model because it is a hallucination detection model specifically built for natural language inference. Besides, it has very few parameters and thus leads to low computational overhead and fast inference speed.\n\n**Q6: The experiments show results over relatively short training periods. How does FSPO behavior evolve over extended training? Are there concerns about the factuality verifier becoming a bottleneck or introducing training instabilities over longer horizons? Can FSPO benefit for multi-turn training?**\n\nWe trained FSPO for 3 epochs: reward and loss curves stayed smooth, hallucination rate kept falling, and accuracy on MATH500 plateaued—no drift or divergence, showing that the verifier does not introduce long‑horizon instability. Finally, we only show 1-epoch training curves in our paper. Because the verifier runs asynchronously on a small GPU and adds ~3 % wall‑clock time per step, its cost remains a constant factor, not a scaling bottleneck. FSPO also generalizes to a multi‑turn setting by running the sentence‑level verifier on each turn’s reasoning.\n\n**L1: The effectiveness of FSPO heavily depends on the quality of external factual verifiers. The sensitivity of the overall system performance to these verifiers is not deeply explored. No discussion of potential biases in factuality verifiers and their impact on training.**\n\nWe will include sensitivity analysis and discuss the potential biases in the factuality verifier in our revised paper.\n\n**L2: computational overhead of introducing factual verifiers**\n\nWe acknowledge that incorporating factual verifiers introduces additional computational overhead, which is common for process-supervised reward modeling approaches. However, this overhead is relatively minor compared to the overall cost of RL fine-tuning, as the verifier operates at the sentence level and uses lightweight entailment checks. More importantly, the verifier substantially improves factual consistency and reasoning reliability, leading to significant performance gains across hallucination benchmarks (Table 1). Thus, the modest computational cost is justified by the substantial reduction in hallucinations and improved model trustworthiness.\n\n**L3: Given that larger models often exhibit different behaviors, the generalizability to state-of-the-art models (30B+ parameters) remains unclear. But I acknowledge that the training cost might be too expensive to do.**\n\nScaling to larger models (30B+) will strengthen our claims and show the generalizability of our approach. Limited by the computational resources, we leave this to future work. \n\n**L4: The evaluation focuses primarily on knowledge-intensive QA tasks and mathematical reasoning. The generalizability to other reasoning domains is not explored.**\n\nTo verify the generality of our approach, we apply FSPO to Qwen2.5-7B-Base in scientific reasoning including Chemistry (ChemBench), Medicine (MedQA), and Physics (PIQA) domains/datasets.\n\n| Model         | ChemBench | MedQA | PIQA |\n|---------------|-----------|--------|------|\n| Qwen2.5-7B-Base     | 51.9      | 54.3   | 86.7 |\n| w/ FSPO        | 57.4      | 60.9   | 89.3 |"}, {"review_id": "wxYgWDnRDy", "reviewer": "Reviewer_3PGQ", "summary": "This paper proposes a novel reasoning-oriented RL method called FSPO, which imposes a step function-based factual reward over generated reasoning sentences. After getting factual reward for each sentence, FSPO also assigns different advantages to individual tokens based on the factual reward of the sentence they belong to. The authors demostrate that the proposed method outperforms other reasoning baseline models over multiple math and hallucination evaluation benchmarks.", "questions": "1. How does FSPO balance the trade-off between final answer accuracy vs. reasoning chain correctness? Because if the reasoning chain is significantly long with multi paragraphs, there's a risk that factuality reward will overwhelm answer reward, as is suggested in Eq. 10. Is it easy in terms of reward engineering to make the method work for various long responses?\n\n2. It looks like the design of factuality-aware advantage adjustment (L207) does not handle the situation where the answer is wrong but reasoning chain is factual but unhelpful to the question (e.g., unrelated truthful statements). In such scenario, FSPO actually learns towards it but not against it?", "limitations": "Overall a solid paper. See weaknesses and questions.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The proposed FSPO algorithm is technically sound and the empirical evaluation is thorough, with experiments on multiple models and datasets.\n* The paper is clearly written, with a logical flow that effectively communicates the problem, the proposed solution, and the experimental validation. The use of a theoretical analysis to motivate the problem of RL-induced hallucination adds depth to the exposition.\n* The proposed method achieves impressive results over many benchmarks.\n\nWeaknesses:\n* Lack of qualitative comparisons with baselines, demonstrating the difference in the reasoning chain.", "was_revised": true, "final_justification": "Thanks for the additional details in the rebuttal. I am in general satisfied with the explanation and will maintain my rating.", "rebuttal": "Thank you for your insightful comments. The following is our response point-to-point.\n\n**W1: Lack of qualitative comparisons with baselines, demonstrating the difference in the reasoning chain.**\n\nBelow, we show three responses generated by Qwen2.5-7B-Base, w/ GRPO, and w/ FSPO. We will add these qualitative comparisons in the revised version of our paper.\n\n| Model         | Question: Which film was released earlier, Mera Karam Mera Dharam or Vetri Vinayagar? |\n|---------------|-----------|\n| Qwen2.5-7B-Base     | To determine which film was released earlier between \"Mera Karam Mera Dharam\" and \"Vetri Vinayagar,\" we would need to look up the release dates of both films. Mera Karam Mera Dharam: This appears to be an Indian film. After checking reliable sources, it seems that \"Mera Karam Mera Dharam\" was released in 2013. Vetri Vinayagar: This is also an Indian film. \"Vetri Vinayagar\" was released in 2010. Given these release dates, \"Vetri Vinayagar\" was released earlier than \"Mera Karam Mera Dharam.\" |\n| w/ GRPO        | \\<think\\> First, I need to recall the release dates of both films. I know that Mera Karam Mera Dharam was released in 1998. Vetri Vinayagar, on the other hand, was released in 1995. Comparing these two years, 1995is earlier than 1998. \\</think\\>\\<answer\\> Vetri Vinayagar \\</answer\\> |\n| w/ FSPO | \\<think\\> “Mera Karam Mera Dharam” was released on 2 July 1987 in India, while “Vetri Vinayagar” premiered much later, on 16 September 1996. So to directly answer your question: Mera Karam Mera Dharam (Hindi) came out in summer 1987. Vetri Vinayagar (Tamil devotional film) followed more than nine years later, in September 1996. Therefore, Mera Karam Mera Dharam was released earlier. \\</think\\>\\<answer\\> Mera Karam Mera Dharam \\</answer\\> |\n\n**Q1: How does FSPO balance the trade-off between final answer accuracy vs. reasoning chain correctness? Because if the reasoning chain is significantly long with multi paragraphs, there's a risk that factuality reward will overwhelm answer reward, as is suggested in Eq. 10. Is it easy in terms of reward engineering to make the method work for various long responses?**\n\nIn Eq. 10 we average the stepwise factuality scores, so their combined reward is always bounded in [‑1, 1]—the same scale as the 0/1 answer‑correctness reward. Consequently, even very long chains cannot dominate the objective. To balance the trade-off between final answer accuracy and reasoning correctness, we can use a weight coefficient for factuality reward (now it is set to 1 by default since we think the two rewards are equally important).\n\n**Q2: It looks like the design of factuality-aware advantage adjustment (L207) does not handle the situation where the answer is wrong but reasoning chain is factual but unhelpful to the question (e.g., unrelated truthful statements). In such scenario, FSPO actually learns towards it but not against it?**\n\nIn Eq. 8, if the reasoning is truthful but unrelated, it likely obtains a reward of zero since it is neutral to the evidence. The evidence in our paper is some necessary knowledge to solve the question provided by the datasets. In the situation you mention, the answer and step rewards are all zero, so the policy model does not learn from the signals."}, {"review_id": "PDGuqGONlB", "reviewer": "Reviewer_UjfW", "summary": "This paper is motivated by the claim that recent RL based reasoning models trained with binary verifiable rewards hallucinates more, so the authors propose a method to try to reduce the problem.  The paper is in two parts: the first part of the paper theoretically proves that the binary verifiable reward used in the recent popular RL methods cause the following problems. First, the variance of the gradient will be high when the probability to output the correct answer is low or close to zero (which makes sense), and thus this results in the gradient signals in RL to be either close to zero or an outsized jump.  The outcome is that it’s common for the policy to converge to keep producing incorrect answers with high confidence and there are no gradients to push the model away from the local optima.  The second part of the paper is not closely related to the theory part, but tries to design step-wise “factuality” rewards that are neither sparse nor binary.  The authors named this method FSPO. They observed the performance improvement compared to the regular GRPO.", "questions": "In the ablation study in 6.3, what’s the difference between FSPO and GRPO w/ factuality? In my understanding, both of them use the combined reward in Eq. 10, and the differences seem not explicitly discussed.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths:\n- The method shows good performance improvement on the truthful-related benchmarks and the math benchmarks.\n- The method was widely tested on multiple families of base models.\n\n# Weaknesses:\n- The idea is relatively simple: adding a reward provided by external NLI and supporting documents.  The authors try to reframe this concept as a new “PO” method just like GRPO, which is a bit of a stretch. The method is still 100% under the GRPO framework but the reward design is not only the answer accuracy.  Also, the GRPO algorithm is not necessarily required to be paired with answer accuracy reward.\n- The reward design of using an external NLI model on the training data with supporting documents (e.g. from HotpotQA) is a narrow setting, which may not generalize well to other domains. It’s also hard to scale up as we have to collect a training dataset containing evidence documents and answers at the same time. There is also no guaranteed generalizability of the NLI model beyond the domain it was trained on, for a straightforward example, generalizing to non-English languages.\n- The proposed algorithm is an on-policy RL method just like GRPO, but we saw the improvement slightly decrease when trained with more examples, like 8K, similar to training with off-policy methods like DPO.  This could be caused by the mismatch between the NLI model’s judgement vs the real factuality. Thus, as the model fits too much to the NLI model’s biases, it can instead deviate from the real accuracy.\n- One could argue that the theory part of the paper can be simply concluded by the “sparse reward” problem widely studied in RL community", "was_revised": true, "final_justification": "Based on the feedback provided by the authors that addresses concerns by the reviewers I am raising my score to 4.", "rebuttal": "Thank you for your valuable comments. The following is our response point-to-point.\n\n**W1: The idea is relatively simple: adding a reward provided by external NLI and supporting documents. The authors try to reframe this concept as a new “PO” method just like GRPO, which is a bit of a stretch. The method is still 100% under the GRPO framework but the reward design is not only the answer accuracy. Also, the GRPO algorithm is not necessarily required to be paired with answer accuracy reward.**\n\nFSPO is not just “GRPO + an extra NLI reward”. Its novelty lies in two orthogonal aspects:\n\n1. Stepwise factuality reward (Eq. 10) – dense, bounded feedback that can replace any sparse objective, not limited to answer accuracy.\n2. Token‑level advantage re‑allocation (Eq. 11) – redistributes, rather than adds, the sample‑level advantage so factual tokens are credited and non‑factual ones penalized.\n\nThese components can plug into GRPO, REINFORCE++, or PPO (see Sec. 6.3), so treating it as a distinct “PO” family member is accurate: the optimization policy is unchanged, but the policy‑shaping mechanism is new and broadly applicable.\n\n**W2: The reward design of using an external NLI model on the training data with supporting documents (e.g. from HotpotQA) is a narrow setting, which may not generalize well to other domains. It’s also hard to scale up as we have to collect a training dataset containing evidence documents and answers at the same time. There is also no guaranteed generalizability of the NLI model beyond the domain it was trained on, for a straightforward example, generalizing to non-English languages.**\n\n- Generalizability of the verifier:\n\nOur design is verifier‑agnostic: any retrieval + factuality evaluator (NLI or self‑consistency judge) can replace HHEM‑2.1 without code changes. We also tried using a general large language model (i.e., Llama3.1-8B-Instruct) to verify the factuality of each step and it achieves results similar to the current HHEM-2.1 model.\n\n- Collection of data:\n\nCurrently there are many knowledge-intensive datasets and we can quickly employ retrievers to obtain relevant documents from Wikipedia or other large corpora. Evidence can also be collected on‑the‑fly with a lightweight retriever, so no curated document‑answer pairs are strictly required.\n\n**W3: The proposed algorithm is an on-policy RL method just like GRPO, but we saw the improvement slightly decrease when trained with more examples, like 8K, similar to training with off-policy methods like DPO. This could be caused by the mismatch between the NLI model’s judgement vs the real factuality. Thus, as the model fits too much to the NLI model’s biases, it can instead deviate from the real accuracy.**\n\nThe slight dip at 8K is due to dataset mix, not NLI drift: adding more Hotpot‑style QA shrinks the share of math cases, so the gains on MATH500 plateau while hallucination rate keeps falling (Fig. 6). Because FSPO averages and clips factuality scores to [‑1, 1] and combines them one-to-one with answer reward, over‑fitting to verifier biases is bounded. We ran a controlled experiment by decreasing the weight of stepwise factuality reward and the accuracy improved, confirming that the effect is tunable rather than an inherent bias accumulation. \n\n**W4: One could argue that the theory part of the paper can be simply concluded by the “sparse reward” problem widely studied in RL community**\n\nSparse reward is the starting point, but our analysis goes further: we quantify how binary‑reward reasoning tasks induce (i) high‑variance gradients, (ii) forced high‑entropy exploration, and (iii) spurious “confident‑but‑wrong” local optima—three precise mechanisms that directly manifest as LLM hallucinations (Theorems 4.1–4.3). These results explain why sparse reward is especially harmful in stepwise reasoning and motivate FSPO’s dense, token‑level shaping; they are not covered by generic RL sparsity discussions.\n\n**Q1: In the ablation study in 6.3, what’s the difference between FSPO and GRPO w/ factuality? In my understanding, both of them use the combined reward in Eq. 10, and the differences seem not explicitly discussed.**\n\nIn lines 271-272, GRPO w/ factuality uses the combined reward in Eq. 10 but does not re-allocate the advantage across tokens in Eq. 11. We will explicitly mention this in the revised version of our paper."}, {"review_id": "4pb3iE4O6J", "reviewer": "Reviewer_YGiP", "summary": "- Paper starts by notificing that RLHF-ed (GRPO) versions of base models from the Qwen, Llama, and DeepSeek families have a higher hallucination rate than their base equivalents\n- In parallel the paper introduces a series of theorems stating that binary sparse rewards lead to high variance (4.1), that exploration is necessary to avoid stagnation (4.2) and binary rewards have bad local minima with no training signal, specifically when model is over confident in the wrong cases (4.3)\n- They define a step-wise factuality reward equal to -1, 0, or 1 if the reasoning step can be entailed or contradicts a ground truth knowledge base, and linearly combine it with a reward of 1/0 for exact match of the final answer\n- They adjust advantage of each token (eq 11) to reflect whether the token is factual", "questions": "- You introduce a lienar combination of R_answer and R_factuality in eq (10). Is that the reward used to compute the advantage at line 206? Will it be an issue that R_factuality is used both to compute the advantage and in equation (11)? Why shouldn't we use just the original advantage?\n- I'm not sure i understand why we want -A_i if A_i < 0 and R_factuality = 1? The reasoning step is factual and the factuality-weighted outcome is negative. Why just flip the sign? Would it make sense to use a constant instead? It feels quite arbitrary, that the more negative A_i was, the more we want to reward that token? This could encourage factual reasoning steps that lead to wrong outcomes no? And encourage the more the outcome is wrong?\n- Table 2, i would prefer results to be grouped by based model so we're comparing apples to apples. (Why not merge the results with Figure 1 or use the same format?)", "limitations": "it's fine, it's a pretty theoretical paper", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- Hallucination and bad reasoning are important problems to solve and this paper addresses it\n- I'm not convinced regarding the details of the advantage defined in Eq 11", "was_revised": true, "final_justification": "the authors have addressed my concerns", "rebuttal": "Thank you for your valuable comments. The following is our response point-to-point.\n\n**W1: I'm not convinced regarding the details of the advantage defined in Eq 11**\n\nEq. 11 does not add or remove reward—it only redistributes the sample‑level advantage $A_i$ (already z‑normalized across the group) to reflect token‑level factuality.\n\n* If a step is factual but $A_i < 0$, we flip the sign so those tokens regain the credit they lost to unrelated errors; magnitude stays as $|A_i|$, preventing over‑compensation.\n* If a step is non‑factual and $A_i > 0$, we likewise negate its share, ensuring it is penalized.\n\nEmpirically, this re‑weighting reduces hallucinations and improves reasoning performance compared to using unmodified $A_i$ in GRPO.\n\n**Q1: You introduce a linear combination of R_answer and R_factuality in eq (10). Is that the reward used to compute the advantage at line 206? Will it be an issue that R_factuality is used both to compute the advantage and in equation (11)? Why shouldn't we use just the original advantage?**\n\nYes, the final combined reward ($R_\\text{final}$) is used to compute the advantage at line 206. In Eq. 11, the factuality reward ($R_\\text{factuality}$) is just used to re‑allocate the total advantage across tokens. It does not change the overall magnitude, so no double‑counting occurs. Keeping only the original, uniform advantage would ignore which tokens are factually sound—our ablation shows this loses both accuracy and hallucination control.\n\n**Q2: I'm not sure i understand why we want -A_i if A_i < 0 and R_factuality = 1? The reasoning step is factual and the factuality-weighted outcome is negative. Why just flip the sign? Would it make sense to use a constant instead? It feels quite arbitrary, that the more negative A_i was, the more we want to reward that token? This could encourage factual reasoning steps that lead to wrong outcomes no? And encourage the more the outcome is wrong?**\n\n$A_i < 0$ means this sample under‑performed the group (does not have to be a wrong answer), yet $R_\\text{factuality} = 1$ means the specific step is factually correct. Flipping its sign converts “punish all tokens” into “do not punish the factual ones”, ensuring they receive positive credit while keeping the magnitude proportional to the original advantage (already normalized by the group’s σ). Using a small constant would ignore how far the rollout missed the mark, giving too little signal for long or critical factual spans. The sign flip therefore protects good steps without over‑compensating or encouraging wrong answers.\n\n**Q3: Table 2, I would prefer results to be grouped by based model so we're comparing apples to apples. (Why not merge the results with Figure 1 or use the same format?)**\n\nWe think you mean the results in Table 1. We will group the results by base models in the revised version for better visualization. Figure 1 aims to compare models before and after RL and long CoT fine-tuning. Most results in Figure 1 have been included in Table 1 besides Qwen2.5-14B/32B-Instruct because we do not have computational resources for fine-tuning them. We will merge their results in Table 1 in the revised version."}]

Author rebuttal:

Meta-review: The paper studies why RL fine-tuning for reasoning LLMs increases hallucinations and provides a theoretical analysis linking sparse binary rewards to high-variance gradients, entropy-driven randomness, and confident-but-wrong local optima. The proposed Factuality-aware Step-wise Policy Optimization (FSPO) introduces step-wise factuality verification and token-level advantage redistribution. Experiments on Qwen and Llama models show consistent gains in reasoning accuracy and reductions in hallucinations.

**Strengths:**

The work is well motivated, with clear theoretical grounding and strong empirical validation. FSPO achieves substantial improvements across reasoning and hallucination benchmarks, and the paper is clearly written. The rebuttal further strengthened the work with new experiments on varying model scales, noisy verifiers, and additional domains (chemistry, medicine, physics), as well as qualitative examples.

**Weaknesses / Missing elements:**

The novelty is incremental, primarily a reward design modification rather than a new optimization paradigm. The method depends on external verifiers, raising concerns about bias, scalability, and applicability beyond English or curated datasets. Trade-offs between factual intermediate steps and final-answer correctness, and cases of “truthful but irrelevant” reasoning, remain underexplored.

--

Reviewers raised concerns about scalability, verifier dependence, computational cost, novelty, qualitative analysis, and clarity of advantage reweighting. The authors addressed these by showing consistent results on smaller and larger models, sensitivity analysis with noisy verifiers, modest overhead (~3%), empirical justification of the reweighting design, and qualitative comparisons. Additional domain experiments strengthened generality. After rebuttal, reviewers maintained or raised scores (one accept, three borderline accepts). The consensus leans positive, recognizing a technically sound contribution with incremental but useful impact.

All in all, the paper is technically solid and addresses an important, timely problem with empirical benefits. While conceptually incremental, the method is impactful in practice and relevant for hallucination reduction.
Decision: Accept (poster)