FlowRL: Matching Reward Distributions for LLM Reasoning
Authors: Xuekai Zhu
Daixuan Cheng
Dinghuai Zhang
Hengli Li
Kaiyan Zhang
Che Jiang
Youbang Sun
Ermo Hua
Yuxin Zuo
Xingtai Lv
Qizheng Zhang
Lin Chen
Fanghao Shao
Bo Xue
Yunchong Song
Zhenjie Yang
Ganqu Cui
Ning Ding
Jianfeng Gao
Xiaodong Liu
Bowen Zhou
Hongyuan Mei
Zhouhan Lin
Venue: ICLR 2026 Poster (2026)
Keywords: Reward Distribution Matching
Flow Balance
LLM Reasoning
Primary area: reinforcement learning

Abstract: We propose FlowRL: matching the full reward distribution via flow balancing instead of solely maximizing rewards in large language model (LLM) reinforcement learning (RL). Recent advanced reasoning models adopt reward-maximizing methods (e.g., PPO and GRPO), which tend to over-optimize dominant reward signals while neglecting less frequent but valid reasoning paths, thus reducing diversity. In contrast, we transform scalar rewards into a normalized target distribution using a learnable partition function, and then minimize the reverse KL divergence between the policy and the target distribution. We implement this idea as a flow-balanced optimization method that promotes diverse exploration and generalizable reasoning trajectories. We conduct experiments on both math and code reasoning tasks: FlowRL achieves a significant average improvement of $10.0\%$ over GRPO and $5.1\%$ over PPO on math benchmarks, and performs consistently better on code reasoning tasks. These results highlight reward distribution-matching as a key step toward efficient exploration and diverse reasoning in LLM reinforcement learning.

Published as a conference paper at ICLR 2026
FLOWRL: MATCHING REWARD DISTRIBUTIONS FOR LLM REASONING
Xuckai Zhu! Daixuan Cheng" Dinghuai Zhang' Hengli Li
Kaiyan Zhang
Che Jiang' Youbang Sun'
Ermo Hua
Yuxin Zuo Xingtai Ly
Qizheng Zhang' Lin Chen'
Fanghao Shao' Bo Xue Yunchong Song²
Zhenjie Yang' Ganqu Cui² Ning Ding42 Jianfeng Gao* Xiaodong Liu"
Bowen Zhou 42, Hongyuan Mei. Zhouhan Lin 1.2.
!Shanghai Jiao Tong University Shhangaii AI Laboratory Microsoft Research
*Tsinghua University SPeking University 'Remmin University of China
"Stanford University TTyyot Technological Institute at Chicago
Frontis.AI
hongyuanmel @gmail..c.mm ccom xuekai zhu0@gma i 1 com
FlowRI.
"Corresponding Authors.
ABSTRACT
We propose FlowRL: matching the full reward distribution via flow balancing in- stead of solely maximizing rewards in large language model (LLM) reinforcement meth-
learning (RL). Recent advanced reasoning LLMs adopt ods (e.g.. PPO and GRPO), which tend to over-optimize dominant reward signals while neglecting less frequent but valid reasoning paths, thus reducing diversity. In contrast, transform scalar rewards into a normalized target distribution us- ing a learnable partition function, and then minimize the reverse KL divergence between the policy and the target distribution. We implement this idea flow- balanced optimization method that promotes diverse exploration and generalizable reasoning trajectories. We conduct experiments both math and code reasoning tasks: FlowRL. achieves significant average improvement of 10.0% over GRPO and 5.1% over PPO math benchmarks, and performs consistently better on code reasoning tasks. These results highlight reward distribution-matehie as a key step toward efficient exploration and diverse reasoning of LLM reinforcement learning.
INTRODUCTION
Reinforcement learning (RL) plays crucial role in the post-training of large language mod- els (LLMs) (Zhang et al., 2025b). A series of powerful reasoning models (Guo et al., 2025; Kavukcuoglu, 2025; Rastogi et al., 2025) have employed large-scale reinforcement learning to achieve strong performance on highly challenging benchmarks. The evolution of RL algorithms for LLM reasoning has progressed through several key stages: REINFORCE (Sutton al., 1999a) provides a solid baseline that is easy to implement and efficient in simple settings; PPO (Schulman al., 2017) improves upon REINFORCE with better stability and efficiency in complex settings; GRPO (Shao et al., 2024) simplifies PPO training by eliminating the learning of separate value function and relying on group comparisons. However, all these methods share fundamental limi- tation in their reward-maximizing objective.
RL methods tend to overfit to the dominant mode of the reward distribu- tion (Skalse et al., 2022; Pan et al., 2022; Zelikman et al., 2022; Gao et al., 2023). As illustrated in Figure I, representative RL methods such as GRPO neglect other meaningful modes, which often results in limited diversity among generated reasoning paths and reduces generalization to less fre- quent yet valid logical outcomes (Hu et al., 2023). These drawbacks become especially pronounced in complex wft---chainnfoooooeeee tribution of plausible solutions is essential for effective generalization (Liu et al., 2025a). Recent approaches adjust the clip ratio (Yu et al., 2025b), apply entropy-based advantage shaping (Cheng
neward-maximinn
(CoT; Wei al.. 2022) reasoning, where capturing a diverse dis-

---

Published as a conference paper at ICLR 2026
Distribution-maccinng FlowRL
Reward-maximizing
R++, PPO and GRPO
Ground Truth FlowRL (Ours)
Ground Truth GRPO
KL 0.11
KL = 8.68
20
100
20
40
60
100
State x
State x
FlowRL
48.39
Moth Average Score
GRPO
38.34
10
20
30
40
50
FlowRL.
1549.47
CodeForces Rating
GRPO
1313.82
200 400
600
800
1000
1200
1400
1600
Figure/ Top: Comparison between Ietrribution-matehin and reward-maximizing approaches. FlowRI. (left) learns to match the full reward distribution, maintaining diversity across multiple methods (right) such as RE-
modes with low KL divergence. In contrast, wewardmmximiizin INFORCE+ (R++; Sutton et al., 1999b; Hu et al., 2025), PPO (Schulman et al., 2017), and GRPO (Shao et al., 2024) concentrate on a single high-reward peak, leading to mode collapse and higher KL divergence. Bottom: Performance comparison. FlowRL consistently outperforms GRPO across math and code domains.
al., 2025), or selectively promote high-entropy tokens (Wang al., 2025), thereby dynamically adapting the data distribution and implicitly increasing diversity. This raises fundamental question: How can we promote diverse exploration to prevent convergence to dominant solution patterns in RL training?
In this paper, we propose FlowRL, a policy optimization algorithm that aligns the policy model with the full reward distribution, encouraging mode coverage. FlowRL achieves more efficient exploration by fundamentally shifting from reward maximization reward distribution matching, thereby addressing the inherent mode-collapse limitations of previous RL approaches. As illustrated Figure 1, the core idea of FlowRL is to introduce a learnable partition function that normalizes scalar rewards into target distribution, and to minimize the reverse KL divergence between the policy and this reward-indee induced distribution. We develop this KL objective based on the trajectory balance formulation from GFlowNets (Bengio al., 2023b), providing gradient equivalence proof that bridges generative modeling and policy optimization. To address the challenges of long CoT training, we introduce two key technical solutions: length normalization tackle gradient explosion issues that occur with variable-length CoT reasoning, and importance sampling to correct for the distribution mismatch between generated rollouts and the current policy.
We compare FlowRL with mainstream RL algorithms for LLM reasoning. including REIN- FORCE++ (Hu et al., 2025), PPO, and GRPO, across math and code domains. using both base and distilled LLMs with 7B or 32B parameters. In the math domain, FlowRL outperforms GRPO and PPO by 10.0% and 5. 5,,, respectively, demonstrating consistent improvements on six chal- lenging math benchmarks (MAA, 2025; 2023; Lightman et al., 2023a; Lewkowycz et al., 2022; He et al., 2024). Furthermore, FlowRL surpasses both PPO and GRPO on three challenging coding benchmarks (Jain et al., 2024; Penedo et al., 2025; Chen et al., 2021), highlighting its strong gener- alization capabilities in code reasoning tasks. To understand what drives these performance gains, analyze the diversity of generated reasoning paths and confirm that FlowRL produces substan- tially more diverse rollouts than the baseline methods, thereby validating the effectiveness of our approach in exploring multiple solution strategies.
2

---

Published as a conference paper at ICLR 2026
Contributions. We summarize the key contributions of this work as follows:
We propose FlowRL, a policy optimization algorithm that shifts from reward maximization to reward distribution matching via flow balancing, encouraging diverse reasoning path exploration while addressing the inherent mode-collapse limitations of existing RL methods.
We introduce length normalization and importance sampling enable effective training
variable-length Co reasoning, addressing gradient explosion and sampling mismatch issues.
FlowRL outperforms GRPO and PPO by 10.0% and 5.1 respectively across math benchmarks and demonstrates strong generalization on code reasoning tasks, with diversity analysis confirming substantially more diverse solution exploration.
PRELIMINARIES
Reinforcement Learning for Reasoning, We formulate reasoning as a conditional generation problem, where the policy model receives question x E X' and generates an answer y E y. The objective is to learn a policy To (y(x) that produces high-quality answers under task reward signals r. To better illustrate the policy optimization procedure, we provide detailed formulation of GRPO below. For each question GRPO samples group of answers (y1, Y2,......... yG) from old policy aiai and updates the model by maximizing the following objective:
(o) Bep-P(AC, 1.... wotte
do Ve,sel ctpp
Molyyela, e,1 + Aes ADxx (olll
min
Te (Vi.
(y:l() log (x)(()) mo(x((x)
DEL (mo])
Toly)[xx)
(1)
where and are hyper-parameters. Here, A, denotes the advantage, computed by normalizing the group reward values (r1, r....,,r. rG) as A; = "--mean( (ri.r.." ra)) Compared to GRPO, std (71.(") ra))
REINFORCE applies the policy gradient directly, without advantage normalization, clipping, or KL regularization. PPO uses critic model estimate the advantage and employs importance sampling to stabilize policy updates.
GFlowNets. Generative Flow Networks (GFlowNets; Bengio et al., 2023a) are probabilistic framework for training stochastic policies to sample discrete, compositional objects (e.g., graphs, sequences) in proportion to a given reward. As shown in Figure 2, the core principle of GFlowNets is to balance the forward and backward probability flows at each state, inspired by flow matching (Bengio et al., 2021). The initial flow is estimated by (*0) the initial state 5g. The output flow is equal to the outcome reward r(sy) conditioned at the final state sy- Following Lee al. (2024), we use 3-layer MLP to parameterize Z This flow-balancing mechanism facilitates the discovery of diverse, high-reward solutions by ensuring proper exploration the solution space. See Appendix C for detailed GFlowNets background.
%
%
Lattor
$
SO
S
S/
(ES))
Figure 2: GFlowNets (Bengio et al., 2023a), flow-balance perspective on reinforcement learn- ing. The initial flow (so) injects probability mass into the environment, which is transported through intermediate states by the policy me and accumulated at terminal states in proportion to the scalar rewards.
3 METHODOLOGY
In this section, first formulate distribution matching in reinforcement learning through reverse KL divergence and establish its connection to trajectory balance from GFlowNets. To address the challenges of gradient explosion and sampling mismatch encountered during long CoT training,
3

---

Published as a conference paper at ICLR 2026
further incorporate length normalization and importance sampling. Using this enhanced framework, we derive bow-balanced objective, termed FlowRL.
3.1 FROM REWARD MAXIMIZATION TO DISTRIBUTION MATCHING
As illustrated in Figure recent powerful large reasoning models typically employ reward- maximizing RL algorithms, such as PPO or GRPO. However, these methods tend to optimize toward the dominant reward mode, frequently resulting in mode collapse and the neglect of other plausible, high-quality reasoning paths. To address this fundamental limitation, we propose optimizing the policy by aligning its output distribution to target reward distribution. A simple yet effective way to achieve this is to minimize the reverse KL divergence' between the policy and this target. How- ever, in long CoT reasoning tasks, the available supervision in RL is a scalar reward, rather than a full distribution. Moreover, enumerating sampling all valid trajectories to recover the true reward distribution is computationally intractable.
Inspired by eneryy-based modeling (Hinton al., 1995; Du & Mordatch, 2019), introduce learnable partition function (x) to normalize scalar rewards into a valid target distribution. This allows to minimize the reverse KI. divergence between the policy and the weight-we dis- tribution, formalized as:
cxp(Br(x,)) y))
min DKL @
(y x)
T x) xc exp(Br(x.
Z (x)
where r(x, y) is the reward function, ,3 a hyperparameter, [x is the learned partition function, (x..) This objective encourages xx(x)
and the resulting target distribution is defined as ity x) the policy to sample diverse, high-reward trajectories in proportion to their rewards, rather than
collapsing to dominant modes as in standard reward maximization.
While the KL-based formulation provides a principled target distribution, we derive practical, RL-style objective that facilitates efficient policy optimization.
Proposition In terms of expected gradients, minimizing the KL. objective Eq. 2 is equivalent to minimizing the trajectory balance loss used in GFlowNet (Malkin et al., 2022; 2023; Lee et al., 2024; Bartoldson et al., 2025):
exp(3r(x,y))
min DKL
Ty x)
min (log 7x(x) + log ty x) Br(x, y))
(x)
Trajectory Balance
Remark 2 (Trajectory balance as a practical surrogate for KL minimization). Given the equivalence established in Proposition 1, the KL-based distribution matching objective can reformulated as the trajectory balance loss. This reformulation provides a practical optimization approach by using a stable squared loss form rather than direct KL optimization, and by treating (x) as a learnable parameter rather than requiring explicit computation of the intractable partition function. The trajec- tory balance objective thus serves as a tractable surrogate for reward-guided KL minimization that can be directly integrated into existing RL frameworks.
3.2 FLOWRL
As established in Proposition 1, the target reward distribution can be approximated by optimizing the trajectory balance objective. However, applying this objective directly to long CoT reasoning introduces two key challenges:
Exploding gradients from long trajectories. Trajectory balance is a sequence-level objective, and applying it to long CoT reasoning with up to 8K tokens leads to exploding gradients and unstable updates. This issue is not observed in prior GFlowNets works, which typically operate on short trajectories in small discrete spaces. Specifically, the log-probability term log Tely x) decomposes into a token-wise sum, Σ, log To ( yetr x), causing the gradient norm to potentially scale with sequence length.
Problem I:
We reverse KT. since we can only sample from the policy model, not the target reward distribution.
4

---

Published as a conference paper at ICLR 2026
Problem II: Sampling mismatch. Mainstream RL algorithms such PPO and GRPO commonly perform micro-batch updates and reuse trajectories collected from an old policy Tews- enabling data- efficient training- In contrast, the KL-based trajectory balance objective assumes fully on-policy sampling, where responses are drawn from the current policy. This mismatch poses practical limita- tions when integrating trajectory balance into existing RL pipelines.
These limitations motivate our reformulation that retains the benefits of distribution matching while addressing key practical challenges. To enable this reformulation, first redefine the reward func- tion following established practices in GFlowNets literature (Lee et al., 2024; Bartoldson et al., 2025; Yu et al., 2025a) by incorporating reference model prior constraint on the reward distribution. Specifically, we modify the original exp(Br(x, y)) to include the reference model:
exp (3 r r(x, y)) Tref (y x);
(4)
where r(x. y) denotes the outcome reward commonly used in reinforcement learning and Tref the initial pre-trained model. We follow Guo et al. (2025) to use aatcome-based reward signals, and apply group normalization to r(x, y) as î; = (r: - mean(r))/std(r), where r = (ri, . r denotes the set of rewards within sampled group. By substituting the redefined reward formulation Eq. into Eq. 3, we derive the following objective?:
min (log Z + log Te x) - 3 fx y) - log Tee (y x))²
(5)
0
Remark 3 (Reward shaping via length normalization). Trajectory balance treats both the initial flow and the outcome reward as sequence-level quantities. contrast, standard policy optimization methods such as PPO GRPO assign rewards at the token level and compute gradients at each step. However, for trajectories of varying lengths (e.g., CoT responses), this mismatch can cause
the jog-probability term log (y x) Σi == log To(u. Mer, x) to scale with sequence length. To address this, we apply form of reward shaping by normalizing bog-prbbabilitics with respect to sequence length. Specifically, rescale the term it log Ty x), balancing the contributions of long and short sequences and stabilizing the learning signal.
Remark (Importance sampling for data-effficied training). To mitigate sampling mismatch, we employ importance sampling inspired by PPO to stabilize policy updates with off-policy data. We re- re-weight stale trajectories using the importance ratio w = To aa/Tala(y x), which serves as a coefficient in the surrogate loss. Since objective focuses optimizing trajectory balance rather than expected return, we detach the gradient from the current policy to prevent excessive policy drift: detach(ma) (y ool x). For additional stability, we incorporate PPO-style clipping to detach
= 7uily 1 e, e.
bound the importance weights: = clip
Incorporating these improvements into Eq. 5, arrive the following FlowRL objective:
FlowRL
F = w (1og (x( + | log Te x) - (f((x y)
log Fret (y xx)
(6)
where the clipped importance weight w and normalized reward f(x, y) are defined as:
To (y
x) x) e, detach
F; mcan(r)
w = clip(
A
oold (y
std(r)
We use this objective to update the policy parameters 9 during training, and refer to this strategy as
FlowRL. Implementation details and theoretical analysis are provided in $ 4 and $ B, respectively.
Thhe substitution replaces Br(x, y) in trajectory balance objective Eq. with Br(x, log Tret x) incorporate the reference model constraint.
5

---

Published as a conference paper at ICLR 2026
Table 1: Results on math reasoning benchmarks. We report Avg@ 16 accuracy with relative improvements shown as subscripts. Positive gains are shown in green and negative changes in red. FlowRL outperforms all baselines across both 7B and 32B model scales.
Models
AIME24
AIME25
AMC23
MATH500
Minerva
Olympiad
Avg
QwenZ. 5-32B-Base
Max
Response Len
8K Lakens
Backbone
4.58
2.08
28.59
52.48
26.99
21.37
22.68
R++
41.29
+7.08
42.65
14.35 8.13
8.13
17.37 -9.62
24.52-33.15
27.14
PPO
26.87 +22.29
20.41
76.40
47.81
47.81
69. +16.0
28.79 +1.0
37.90 37.90_16.62
43.25
GRPO
23. 23.12
44.58-12.50
76.87
61.60_9.12
18.95-8.01
34.94 -3.57
38.34
FlowRL
23.95
19..9
73.75 46
80.75 228.27
38.21
51.83 330.46
48.39
Qwan2. ,
Max Response Len
8K tokens
Backbone
4.38
2.08
30.78
54.47
22.38
24.03
23.02
R++
41.04
5.41 (3.33
66.71 (35.98
54.25 0.22
24.37 +1.99
27.33 33.30
31.52
PPO
9.38
7.29
63.43
57. 57.98
26.53
27.25-3..2
31.98
GRPO
13.54
9.79
64.53
37.05-2.58
23.06
26.88 -2.85
32.48
FlowRL
15.41
40.83
44..53.23.76
66.96.12.00
31.41
10.55
35.63
4 EXPERIMENT SETTINGS
Backbone Models. There are two learnable modules in Eq. 6: the policy model Te and the partition function Zg. For the policy model Te, we use Qwen-2. 5-7B/32B 32B (Team, 2024) for math tasks and DeepSeek-RA R1 Sei-Disssil--oi Qwen-7B (DeepSeck-AI, 2025) for code tasks, re- spectively. The reference model net is the corresponding fixed pretrained model. For partition function Zœ. following Lee et al. (2024), we use a randomly initialized 3-layer MLP with hidden dimensions matching those of the base model. The input to Zg is the mean of the language model's hidden states after encoding the input and the output is scalar value. We detail the implemen- tation of Z& in & F. All training scripts are based on the veRL (Sheng et al., 2024). For the reward function, following Lee et al. (2024), set the hyperparameter 3 15.
Baselines. We compare our method against three representative souudddmxiimzzaion RL base- lines: REINFORCE+ (R++; Sutton al., 1999b; Hu al., 2025), PPO (Schulman et al., 2017), and GRPO (Shao et al., 2024). All baselines follow the official veRL recipes, with consistent train- ing configurations. For fair comparison, all methods use the same learning rate, batch size, and training steps, and are evaluated at convergence using identical step counts.
Training Configuration. We experiment on both math and code domains. For the math domain, use the training set collected from DAPO (Yu al., 2025b). For the code domain, we follow the setup of DeepCoder (Luo et al., 2025), using their training set. For 7B model training, we use single node equipped with 8 NVIDIA H800 GPUs (80GB memory each). For 32B model training, we scale to 4 nodes with 32 GPUs to accommodate the larger memory requirements. All experiments use wtt.xprmpp__eeeee.
= 8192 across both model sizes. We use batch size of 512 for math reasoning tasks and 64 for code reasoning tasks. We set the learning rate to 1c-6 and enable dynamic batch sizing in veRL for efficient training. For GRPO and FlowRL, configure rollout 8, meaning each prompt generates 8 response rollouts as the group size.
2048 and max_meaxpppaaaee wwwpxeeesposeeeeeeee....
Evaluation Configuration. For the math domain, we evaluate on six challenging benchmarks: AIME 2024/2025 (MAA, 2025), AMC 2023 (MAA, 2023), MATH-500 (Lightman et al., 2023a), Minerva (Lewkowyez et al., 2022), and Olympiad (He et al., 2024). For the code domain, we evalu- ate LiveCodeBench (Jain et al., 2024), CodeForces (Penedo et al., 2025), and HumanEval+ (Chen et al., 2021). For all evaluation datasets, we perform 16 rollouts and report the average Pass @ accuracy, denoted Avg@ 16. We further report rating and percentile for Codeforces. During gen- eration, we use sampling parameters of t temperature rature = 0.6 and op p 0.95 for all evaluations. The response length for evaluation is 8, 192 tokens, consistent with the training configuration.

---

Published as a conference paper at ICLR 2026
Table 2: Results on code benchmarks. We report metries with relative improvements shown as subscripts. Positive gains are shown in green and negative changes in red. FlowRL achieves the strongest performance across all three benchmarks.
Models
LiveCodeBench
CodeForces
HumanEval+
Avg@ 16
Pass@ 16
Rating
Percentile
Avg@ 16
DeepSeek R1 -i--isstll Owen
7B, Max Response Len
8K tokens
Backbone
30.68
49 19.46
886.68
19.4%
80.90
R++
30.46-0.25
52.68+3.22
1208,03-322
56 56.8%
76.61
PPO
35.10+4.42
54.4845.02
+500.07=516.39 3916.39
73.7% 64.3%
82.32+1.42
GRPO
32.75+2.07
52.32 42.86
1313.82 4277.14
67.1% 47.17%
80.13
FlowRL.
37.43 46.75
56.27
+6.81
1549944
4602777
83.3%%46..o%
83.28
+2.38
Qwen2.5-7B the base model. Avg @ 16 accuracy is Table 3: Ablation study on FlowRL with
reported across six math reasoning benchmarks. IS denotes importance sampling.
Method
AIME 2024
AIME 2025
AMC 2023
MATH-500
Minerva
Olympiad
Avg
FlowRI.
15.41
10.83
54.53
66.96
31.41
34.61
35.63
w/o IS
6.25
7.91
41.40
56.97
22.19
25.52
26.71
Zhang et al. (2025a)
10.41
6.66
53.75
66.50
30.97
33.72
33.67
RESULTS
Main Results. Our experimental results, summarized in Table 1 and Table 2, demonstrate that baselines across both math and code reasoning domains. Table reports results on math reasoning benchmarks using both 7B and 32B base models, while Table presents the corresponding results code reasoning tasks. On math reasoning tasks. FlowRL achieves the highest average accuracy of 35.6% with the 7B model and 48.4% with the 32B model, surpassing PPO by 5.1% and GRPO by 10.1% on the 32B model. FlowRL shows strong improvements on challenging benchmarks like MATH-500 and Olympiad problems, demonstrating consistent gains across diverse mathematical domains. On code genera- tion tasks, FlowRL achieves compelling improvements with the highest Avg@ 16 score of 37. 37.43% on LiveCodeBench, Codeforces rating of 1549.47 with 83.3% percentile ranking, and 83.28% ac- curacy on HumanEval+, outperforming all baselines across the board. These consistent performance gains across both domains and model scales provide strong empirical evidence that FlowRL's flow- balanced optimization successfully enhances generalization. This improvement comes from pro- moting diverse solution exploration compared to previous reward- wwward-maximizing RI. approaches.
FlowRL consistently outperforms all
Ablation Studies. We conduct ablation studies on importance sampling and the hyperparameter. For importance sampling. compared the performance with and without it, and implemented a combined loss approach proposed by Zhang et al. (2025a) that simultaneously optimizes both GFlowNets and PPO objectives. This combined loss focuses on optimiz- ing diffusion models, and we adapt it to long CoT reasoning tasks for comparison. Table demon- strates that importance sampling substantially im- proves FlowRI. performance across math rea- soning benchmarks. Compared to Zhang et al. (2025a), using importance sampling as a trajectory- level ratio is more suitable than the combined loss of GFlowNets and PPO. The performance drop with- out importance sampling (from 35.63% to 26.71%)
37
35.63
36
Average Score (%)
35.09
35
34.41
34
33
32
31.34
31
30
B=5
B=10
B=15
B=30
Figure 3: Ablation study on the 3 in FlowRL. 8 15 (highlighted in blue) achieves the best performance.
highlights the critical role of correcting for distribu- tion mismatch between rollout generation and policy training. For the hyperparameter 33, conduct

---

Published as a conference paper at ICLR 2026
series of parameter ablation studies, and Figure shows that 3 15 achieves optimal performance, with detailed results shown in Table
ANALYSIS
Diversity Analysis. To assess solution diversity, we follow the approach of Yu et al. (2025a) and employ GPT-4o-mini (OpenAI, 2024) to evaluate all responses generated by each method on AIME 24/25, The evaluation prompt is shown in Appendix H. As shown in Figure 4, FlowRL achieves higher diversity scores compared to baseline methods. This demonstrates that FlowRL. improves sample diversity compared to baselines, which tend exhibit repetitive solution patterns. This diversity evaluation reveals significant differences in exploration patterns across methods. This nearly doubling of diversity score compared to the strongest baseline (PPO) indicates that FlowRL generates qualitatively different solution approaches rather than minor variations of the same strat- egy. The diversity analysis provides empirical validation of our core hypothesis that flw-balanced optimization promotes mode coverage in complex reasoning tasks.
Case Study. Table illustrates the behavioral dif- ferences between GRPO and FlowRI. on a repre- sentative AIME problem. GRPO exhibits repetitive patterns, applying AM-GM three times and getting stuck in identity loops, failing to solve the prob- lem. FlowRL explores more diverse actions: it sets a = b, derives a cubic equation, finds the rational root, and reaches the correct answer. This shows that FlowRL successfully avoids the repetitive ex- ploration patterns. The contrast reveals fundamen- tal differences in exploration strategies: GRPO's eeward-maximizing approach leads exploitation of familiar techniques (AM-GM inequality) with- out exploring alternatives, eventually reaching con- tradictory conclusions like b c. In con- enables strate-
2.5
2.28
2.0
Divensity Score
1.31
1.23
1.11
1.0
0.5
0.0
R+++
GRPO
PPO
FlowRL
Figure 4: GPT-judged diversity scores on rollouts of AIME 24/25 problems. FlowRL generates more diverse solutions than R++ GRPO, and PPO.
trast, FlowRL's iistribution-matchine gic decisions such as the symmetry assumption which transforms the problem into a tractable cu- bic equation a 27a 46 polynomial factorization.
allowing systematic solution through rational root testing and
7 RELATED WORK
Our work relates to GFlowNets, Flow-Matching Policies, Length Normalization and KL Regular- ization. We discuss three topics that relate most closely to our work in this section, and the other topics are included in Appendix E.
Reinforcement Learning for LLM Reasoning. RL has emerged as a powerful approach for LLM post-training on reasoning tasks (Sutton et al., 1999b; Schulman et al., 2017: Lightman et al., 2023b; Shao et al., 2024; Guo et al., 2025). Most approaches employ reward-maximizing RL to optimize expected cumulative returns. Entropy regularization (Haarnoja et al., 2018; Ahmed al., 2019; Cheng et al., 2025) is a classical technique for mitigating mode collapse by promoting diversity in the policy's s output distribution, and has also been shown to enhance reasoning capabilities in var- ious settings (Eysenbach & Levine, 2021; Chao et al., 2024). However, for long CoT reasoning, the extended trajectory length (e.g., more than 8k tokens) makes difficult for the regularization signal to effectively influence reward-mave ewward-maximizing learning. Recent work (Cheng et al., 2025; Wang et al., 2025; Cui et al., 2025; Dong et al., 2025) has discovered that training with more diverse or high-entropy training data further enhance training effectiveness. Compared to traditional en- tropy regularization, the above methods explicitly increase the proportion of low- cow-probability (i.c., high-entropy) tokens in the training data. In work, we address the mode-collapse problem by
8

---

Published as a conference paper at ICLR 2026
Table 4: Case study comparing GRPO and FlowRL rollouts on an AIME problem. GRPO exhibits repetitive patterns (AM-GM x3, identity loops x2), while FlowRL follows a more diverse solution path.
Content (boxed actions; "xk"= repeated; "..."= omitted)
Question
Let B be the set of rectangular boxes with surface area 54 and volume 23. Let be the radius of the smallest sphere that can contain each box in B. If P with ged(p, q) = 1, find +
GRPO
denote
a. a,
bab+bb+tcal caa) 54, abc 23
va + 12 e d/2
(atbte)2 a² +2 +c² + 2(ab be tea)
AM-GM
x3:
(1)
(M
AM-UM
(a+b+c)² identity loop
x2: loop (1)
loop (2)
(contradiction)
back to (a+bbee)² no factorization
FlowRL
let
a, b,c
with
Blab+be+ ca) = 54, abc = 23
d Va + 62 c² d/2
(a+bte)² a +b² +c² s² 54
a 27a + 46 =0
rational rool
2
...
factor (a- 22) 22 + 2a 23)
branch
+ 2/6
back-sub 23/aa ... a² + 22
657
22
657
Answer 721
fundamentally shifting from reward maximization reward distribution matching in our RL formu- lation. See Appendix for detailed comparisons.
GFlowNets. GFlowNets (Bengio al., 2023a) represent a class of diversity-driven algorithms designed to balance probability flows across states. They have rich connections to probabilistic modeling methods (Zhang et al., 2022a;b; 2024a; Zimmermann et al., 2022; Malkin al., 2023; Ma et al.), and control methods (Pan et al., 2023b;c;d; Zhang et al., 2024b; Tiapkin et al., 2024). This advantage has enabled GFlowNets to achieve successful applications in multiple downstream tasks, such molecular drug discovery (Jain al., 2022; 2023b; Liu al., 2022; Jain et al., 2023a; Shen et al., 2023; Pan et al., 2023a; Kim et al., 2023; 2024), phylogenetic inference (Zhou et al., 2024), and combinatorial optimization (Zhang et al., 2023a;b). For generative AI, GelowNets provide powerful approach to align pretrained models in scenarios such as image generation (Zhang et al., 2025a; Yun et al., 2025) and language model fine-tuning (Hu al., 2024; Yu al., 2025a; Lee et al., 2024). Another line of work primarily focuses on the theoretical aspects of GFlowNets. Re- cent theoretical studies have interpreted GFlowNets as solving a maximum entropy reinforcement learning problem within modified Markov Decision Process (MDP) (Tiapkin et al., 2024; Deleu al., 2024; Mohammadpour al., 2024). These theoretical contributions have inspired us to en- hance reinforcement learning from a more foundational standpoint using GFlowNets principles. A comprehensive overview GFlowNets theory can be found in Appendix C.
Flow-Matching Policies. Flow matching simplifies diffusion- approaches by learning vector fields that transport samples from prior to target distributions (Lipman et al., 2023). Recent work has explored flow matching for policy optimization. McAllister et al. (2025) reformulates policy optimization using advantage-weighter ratios from conditional flow matching loss, enabling flow- based policy training without expensive likelihood computations. Pfrommer et al. (2025) explored reward-weighted flow matching for improving policies beyond demonstration performance. Park et al. (2025) uses separate one-step policy to avoid unstable backpropagation through time when training flow policies with RL. Zhang et al. (2025a) proposed a combined loss function integrating PPO and GFlowNets to optimize diffusion model alignment. Lv et al. (2025) integrates flow-based policy representation with Wasserstein regularized optimization for online reinforcement learning. However, these approaches focus on continuous control, image generation, or vision-action models, rather than addressing mode-collapse limitations in reward-maximizing RL. Inspired by flow match-

---

Published as a conference paper at ICLR 2026
ing principles, work improves upon RL training to enhance training stability while promoting diverse solution exploration.
CONCLUSION
In this work, we introduce FlowRL, which transforms scalar rewards into normalized target distri- butions using learnable partition function and minimizes the reverse KL divergence between the policy and target distribution. We demonstrate that this approach theoretically equivalent trajec- tory balance objectives from GFlowNets and implicitly maximizes both reward and entropy, thereby promoting diverse reasoning trajectories. To further address gradient explosion and sampling mis- match issues in long CoT reasoning, we incorporate importance sampling and length normalization. Through experiments on math and code reasoning benchmarks, FlowRI. achieves consistent im- provements across all tasks compared to GRPO and PPO. Our diversity analysis and case studies confirm that FlowRL generates more varied solution approaches while avoiding repetitive patterns.
ACKNOWLEDGEMENT
This work is sponsored by the National Key Research and Development Program of China (No. 2023/D0121402) and the National Natural Science Foundation of China (NSFC) grant (No. 62576211). We are grateful Mingqian Feng and Yuetai Li for their valuable discussions and feedback, which helped improve the quality of this work.
ETHICS STATEMENT
This work presents FlowRL, reinforcement learning algorithm for improving reasoning large language models. Our focus mathematical and logical problem-solving directly supports ben- eficial applications in education, scientific research, and decision-support systems. We use estab- lished public benchmarks to ensure transparent and unbiased evaluation, and minimize computa- tional waste through efficient configurations, demonstrating our commitment to environmentally conscious and reproducible research.
REPRODUCIBILITY STATEMENT
We provide comprehensive details to ensure reproducibility: implementation specifics in Section (model architectures. training configurations, hyperparameters), complete algorithmic formulation in Eq. 6, experimental setup covering datasets and evaluation benchmarks, baseline implementa- tions following official veRL. recipes, and evaluation methodology. All mathematical formulations, implementation details, and experimental configurations necessary for reproduction are included in the paper.
REFERENCES
Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In International conference on machine learning, pp- 151-160. PMLR, 2019.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Brian Bartoldson, Siddarth Venkatraman, James Diffenderfer, Moksh Jain, Tal Ben-Nun, Seanie Lee. Minsu Kim, Johan Obando-Cerde Yoshua Bengio, and Bhavya Kailkhura. Trajectory bal- ance with asynchrony: Decoupling exploration and learning for fast, scalable Ilm post tost-training. arXiv preprint ar Xrv: 8929, 2025.
10

---

Published as a conference paper at ICLR 2026
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow net- work based generative models for non-iterative diverse candidate generation. Neural Information Processing Systems (NeurlPS), 2021.
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J. Hu, Mo Tiwari, and Emmanuel Ben- gio. Gflownet foundations. Journal of Machine Learning Research, 24(210)1-555, 2023a. URL http:/ /mmrr./mr...///.rr///.m////mm/ org/ /teg/paae papers/ /vr.//////////// 22224/2206642222 html.
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. The Journal of Machine Learning Research, 240):10006-10660, 2023b.
Chen-Hao Chao, Chien Feng, Wei-Fang Sun, Cheng-Kuang Lee, Simon See, and Chun-Yi Yi Lee. Maximum entropy reinforcement learning via energy- based normalizing flow. arXiv preprint arXiv:2405.13629, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray. Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective, arXiv preprint arXiv:2506. 14758, 147588 2025.
Miruna Cretu, Charles Harris, Ilia Igashov, Ame Schneuing, Marwin Segler, Bruno Correia, Julien Roy, Emmanuel Bengio, and Pietro Liò. Synflownet: Design of diverse and novel molecules with synthesis constraints. arXiv preprint arXiv:2405.01155 2024.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617 2025.
DeepSeek-AL. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. URL httph::////aec //arxiv oww.orog/oo..... aabs/////// 0025501.22944
Tristan Deleu, Padideh Nouri, Nikolay Malkin, Doina Precup. and Yoshua Bengio. Discrete proba- bilistic inference control in multi-path environments. arXiv preprint arXiv:2402.10309 2024.
Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang. Fuzheng Zhang, al. Agentic reinforced policy optimization. arXiv preprint arXiv:2507. 19849, 2025.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. Advances in neural information processing systems, 32, 2019.
Benjamin Eysenbach and Sergey Levine. Maximum entropy rl (provably) solves some robust rl problems. arXiv preprint arXiv:2103.06257, 2021.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948 2025.

---

Published as a conference paper at ICLR 2026
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforeement learning with a stochastic actor. In International conference machine learning, pp. 1861-1870. Pmlr, 2018.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024.
Haoran He, Can Chang, Huazhe Xu, and Ling Pan. Looking backward: Retrospective backward syn- thesis for goal-conditioned GFlownets. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /ttp////ppeeiieeeeeeeeeeeeeeeeee net/ forum? -ntdiiddfffaauuuaaa
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and
Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
Geoffrey E. Hinton, Peter Dayan, Brendan J. Frey, and R M Neal. The "wake-sleep" algorithm for unsupervised neural networks. Science, 268 5214:1158-61, 1995.
Edward J Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guillaume Lajoie, Yoshua Bengio, and Nikolay Malkin. Amortizing intractable inference in large language models. arXiv preprint arXiv:2310.0436,, 2023.
Edward J Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guillaume Lajoie, Yoshua Bengio, and Nikolay Malkin. Amortizing intractable inference in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https:://///..... hwtpt::ttpppeneeeeeeeetttttteeet /ttp:////pp/.eeeee////mo net/ poet/Eozer htt///ooummmidd
ca4ca60.
Jian Hu, Jason Klein Liu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models, 2025. URL https://arxiv. org/abs/2501, 3262:32-33, 2025.
Moksh Jain, Emmanuel Bengio, Alex Hernandez-Gareia. Jarrid Rector-Brooks, Bonaventure F.P. Dossou, Chanakya Ekbote, Jie Fu, Tianyu Zhang, Micheal Kilgour, Dinghuai Zhang, Lena Simine, Payel Das, and Yoshua Bengio. Biological sequence design with GFlowNets. International Conference on Machine Learning (ICML), 2022.
Moksh Jain, Tristan Deleu, Jason S. Hartford, Cheng-Hao Liu, Alex Garnández-Gareia, and Yoshua Bengio. Gflownets for ai-driven scientific discovery. ArXiv, abs/2302.00615, 2023a. URL http:/pp....aatttci.....mcc...se crrcholar................c.....l org/ arg//orta Corpporrrsus 2554459319.
Moksh Jain, Sharath Chandra Raparthy, Alex HarranezzGGarii,, Jarrid Rector-Brooks, Yoshua Bengio, Santiago Miret, and Emmanuel Bengio. Multi-objective GFlowNets. International Conference on Machine Learning (ICML), 2023b.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024.
Koray Kavukcuoglu.
Gemini 2.5: Our most intelligent AI model, 2025. https:: petps://blogg htw.loogggoolleeececooooooo hgop:/gooleeeoooooo deepmind/ Google Blog (The Keyword),
URL
Published Mar. 2025.
Minsu Kim, Taeyoung Yun, Emmanuel Bengio, Dinghuai Zhang, Yoshua Bengio, Sungsoo Ahn, and Jinkyoo Park. Local search gflownets. ArXiv, abs/2310.02710, 2023.
Minsu Kim, Joohwan Ko, Tacyoung Yun, Dinghuai Zhang, Ling Pan, Woochang Kim, Jinkyoo Park, Emmanuel Bengio, and Yoshua Bengio. Learning to scale logits for semperature-conditee gflownets, 2024.
Scanie Lee, Minsu Kim, Lynn Cherif, David Dobre, Juho Lee, Sung Ju Hwang, Kenji Kawaguchi, Gauthier Gidel, Yoshua Bengio. Nikolay Malkin, et al. Learning diverse attacks on large language models for robust red-teaming and safety tuning. arXiv preprint ar Xivv::205. 2...2405..88400 2024.
12

---

Published as a conference paper at ICLR 2026
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh. Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quan- titative reasoning problems with language models.
S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 3843-3857. Curran Associates, Inc., 2022. URL
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023a.
Hunter Lightman, Vincet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ila Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023b.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference Learning Representations, 2023. URL https: / openreview net/ forum? (er.rummniddPPPMDCCTTPP.3
Dianbo Liu, Moksh Jain, Bonaventure F. P. Dossou, Qianli Shen, Salem Lahlou, Anirudh Goyal, Nikolay Malkin, Chris C. Emezue, Dinghuai Zhang, Nadhir Hassen, Xu Ji, Kenji Kawaguehi, and Yoshua Bengio. Gflowout: Dropout with generative flow networks. In International Conference on Machine Learning. 2022.
Mingjie Liu, Shizhe Diao, Jian Hu, Ximing Lu, Xin Dong, Hao Zhang, Alexander Bukharin, Shaokun Zhang, Jiaqi Zeng, Makesh Narsimhan Sreedhar, et al. Scaling up rl: Unlocking di- verse reasoning in Iims via prolonged training. arXiv preprint arXiv:2507.12507 2025a.
Zhen Liu, Tim Z. Xiao, Weiyang Liu, Yoshua Bengio, and Dinghuai Zhang. Efficient diversity- preserving diffusion alignment via gradient-informed gflownets. In ICLR, 2025b.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint 250.:25000008.. 2025c.
Michael Luo, Sijun Tan, Roy Huang, Xiaoxiang Shi, Rachel Xin, Colin Cai, Ameen Patel, Alpay Ariyak, Qingyang Wu, Ce Zhang, Erran Li, Raluca Ada Popa, Ion Stoica, and Tianjun Zhang. Deepcoder: A fully open-source 14b coder at 03-mini level, 2025. Notion Blog.
Lei Lv, Yunfei Li, Yu Luo, Fuchun Sun, Tao Kong, Jiafeng Xu, and Xiao Ma. Flow-based policy for online reinforcement learning. arXiv preprint arXiv:2506. 12811, 2025.
Jiangyan Ma, Emmanuel Bengio, Yoshua Bengio, and Dinghuai Zhang. Baking symmetry into gflownets.
MAA. American mathematics competitions ame. https::/mma.org// 200002030..0./2333....
MAA. American invitational mathematics examination aime. pettps://maa.. org/, 2025.
Kanika Madan, Jarrid Rector-Brooks, Maksym Korablyov, Emmanuel Bengio, Moksh Jain, An- drei Cristian Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning gflownets from partial episodes for improved convergence and stability. In International Conference on Machine Learning, pp. 23467-23483. PMLRR 2023.
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in gflownets. Advances in Neural Information Processing Systems, 35:5955-5967. 2022.
Nikolay Malkin, Salem Lahlou, Tristan Deleu, Xu Ji, Edward Hu, Katie Everett, Dinghuai Zhang, and Yoshua Bengio. GFlowNets and variational inference. International Conference on Learning Representations (ICLR), 2023.
13

---

Published as a conference paper at ICLR 2026
David McAllister, Songwei Ge, Brent Yi, Chung Min Kim, Ethan Weber, Hongsuk Choi, Haiwen Feng, and Angjoo Kanazawa. Flow matching policy gradients. arXiv preprint arXiv:2507.21053, 2025.
Sobhan Mohammadpour, Emmanuel Bengio, Emma Frejinger, and Pierre-Luc Bacon. Maximum entropy gflownets with soft q-learning. In International Conference on Artificial Intelligence and Statistics, pp. 2593 PMLR, 2024.
OpenAl. Gpt-4o mini. https: openai.e...... com/ 2024. Accessed: 2024.
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint arXiv:2201.03552 2022.
Ling Pan, Moksh Jain, Kanika Madan, and Yoshua Bengio. Pre-training and fine-tuning generative flow networks, 2023a.
Ling Pan, Nikolay Malkin, Dinghuai Zhang, and Yoshua Bengio. Better training of GFlowNets with local credit and incomplete trajectories. International Conference Machine Learning (ICML., 2023b.
Ling Pan. Dinghuai Zhang, Aaron Courville, Longbo Huang, and Yoshua Bengio. Generative aug- mented flow networks. International Conference Learning Representations (ICLR), 2023c.
Ling Pan, Dinghuai Zhang, Moksh Jain, Longbo Huang, and Yoshua Bengio. Stochastic generative flow networks. Uncertainty in Artificial Intelligence (UAI), 2023d.
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. In Forty-second International Conference on Machine Learning, 2025. URL https: /ttpn//////peeeeeewee/ wtw.://wwwnnffouummemmm
KVI2SFLIpi.
Guilherme Penedo, Anton Lozhkov, Hynek Kydlíček, Loubna Ben Allal, Edward Beeching. Agustín Piqueres Lajarín, Quentin Gallouédec, Nathan Habib, Lewis Tunstall, and Leandro von Werra. Codeforces. https: http:///aaaa.eee/tttttttt//eeeee 2025.
Samuel Pfrommer, Yixiao Huang, and Somayeh Sojoudi. Reinforcement learning for flow-matching policies. arXiv preprint arXiv:2507.15073 2025.
Abhinav Rastogi, Albert Q Jiang, Andy Lo, Gabrielle Berrada, Guillaume Lample, Jason Rute, Joep Barmentlo, Karmesh Yadav, Kartik Khandelwal, Khyathi Raghavi Chandu, et al. Magistral. arXiv preprint arXiv:2506.1194 10110, 2025.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. First Conference on Language Modeling, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint T.XXi:1707.063444 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Max W. Shen, Emmanuel Bengio, Ehsan Hajiramezanali, Andreas Loukas, Kyunghyun Cho, and Tommaso Biancalani. Towards understanding and improving gflownet training. ArXiv, abs/2305.07170, 2023. URL https://app http://app..emmnnissccciieee wrw.aarrurrrCC/oCCIII.O. 258676487.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng. Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.1923 2024.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and charac- terizing reward gaming. Advances in Neural Information Processing Systems, 35:9460-944 9471, 2022.
14

---

Published as a conference paper at ICLR 2026
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning. Journal of Cognitive Neuroscience, 11(1):126-11 134, 1999a.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In S. Solla, T. Leen, and K. Müller (eds.), Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999b. URL htttps://proeett www.rnngnnnnnun...eeeee..e.r...e neurips cc wwtppppeeepteeeaeeeeeeeeeessssse pttppprr.illlepppppe///999// file/
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with Ilms. arXiv preprint Sttiv:22001.229999 2025.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. https::: //wen/lm. Aathub.ic/blan qwen2. 5/.
Daniil Tiapkin, Nikita Morozov, Alexey Naumov, and Dmitry Vetrov. Generative flow networks as entropy-regularizzd rl. In International Conference on Artificial Intelligence and Statistics, pp. 4213 PMLR, 2024.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng. Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning. arXiv preprint arXiv:2506.01933 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, al. Chain-of-thouglee prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Fangxu Yu, Lai Jiang, Haoqiang Kang, Shibo Hao, and Lianhui Qin. Flow of reasoning: Training Ilms for divergent reasoning with minimal examples. In Forty-second International Conference on Machine Learning, 2025a.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint 1iii::5003.147766 2025b.
Taeyoung Yun, Dinghuai Zhang, Jinkyoo Park, and Ling Pan. Learning to sample effective and diverse prompts for text-to-image generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 33625-23633 2025.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 15:1661615588, 2022.
David W. Zhang, Corrado Rainone, Markus F. Peschl, and Roberto Bondesan. Robust scheduling with gflownets. ArXiv, abs/2302.05444 2023a. URL htwph:ttt//aatt semant cacholl
org/CorpustD: DD:26827::55
Dinghuai Zhang, Ricky T. Q. Chen, Nikolay Malkin, and Yoshua Bengio. Unifying generative models with GFlowNets and beyond. arXiv preprint arXiv:2209.02666 2022a.
Dinghuai Zhang, Nikolay Malkin, Zhen Liu, Alexandra Volokhova, Aaron Courville, and Yoshua Bengio. Generative flow networks for discrete probabilistic modeling. International Conference on Machine Learning (ICML), 2022b.
Dinghuai Zhang, Hanjun Dai, Nikolay Malkin, Aaron C. Courville, Yoshua Bengio, and Ling Pan. Let the flows tell: Solving graph combinatorial optimization problems with gllownets. ArXiv, abs/2305.170100 2023b.
Dinghuai Zhang, Ricky T. Q. Chen, Cheng-Hao Liu, Aaron Courville, and Yoshua Bengio. Diffu- sion generative flow samplers: Improving learning signals through partial trajectory optimization, 2024a.
Dinghuai Zhang, Ling Pan, Ricky T. Q. Chen, Aaron Courville, and Yoshua Bengio. Distributional gflownets with quantile flows, 2024b.
15

---

Published as a conference paper at ICLR 2026
Dinghuai Zhang, Yizhe Zhang, Jiatao Gu, Ruixiang ZHANG, Joshua M. Susskind, Navdeep Jaitly, and Shuangfei Zhai. Improving GFlownets for text-to-image diffusion alignment. Transactions on Machine Learning Research, 2025a. ISSN 2835-8856. URL https://cpenreview net/
forum? 3iw.mm2dddddbbbb33332222
Kaiyan Zhang, Yuxin Zuo, Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jia, Pengfei Li, al. A survey of reinforcement learning for large reasoning models. arXiv preprint S.Xi:2550900827,, 2025b.
Xiaojiang Zhang, Jinghui Wang. Zifei Cheng, Wenhao Zhuang, Zheng Lin, Minglei Zhang, Shaojie Wang, Yinghan Cui, Chao Wang, Junyi Peng, al. Srpo: cross-domain implementation of large-scale reinforcement learning Ilm. arXiv preprint arXiv:2504. 14286, 2025c.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.1807/ 2025.
Mingyang Zhou, Zichao Yan, Elliot Layne, Nikolay Malkin, Dinghuai Zhang, Moksh Jain, Mathieu Blanchette, and Yoshua Bengio. Phylogfn: Phylogenetic inference with generative flow networks, 2024.
Heiko Zimmermann, Fredrik Lindsten, J.-W. van de Meent, and Christian Andersson Naesseth. A variational perspective generative flow networks. ArXiv, abs/2210.07992, 2022. URL https: /api. semanticscholas ctt.aaooo//cooo..0::200000000
16

---

Published as a conference paper at ICLR 2026
A PROOF OF PROPOSITION
We begin by analyzing the gradient of the Kullback-Leibler (KL) divergence between the policy To x) and the target reward distribution exp(Br(xx)))
(x(x)
exp(Br(x.))
DeL
To
xx)
(x(x)
Toly x) (x)
V.
Toly
x) log
dy
exp(Br(xx:))
/ Vo
(x) |x)
exp(Br(xx,) y)) dy + maly x)Volog log Zo(x) xx)
xx log
exp(Br(xxy)) dy
/ ma x) Vo Vologmol my
(x x)
(8)
=
x) log
exp(Br(xxy))
dy +
moly x) Vo log x)
-Va. syn
= -f Toly x) V log (o x) log
Zo(x) x)
dy
(xp(Br(x.))))
log Zo (x(x))T exp(Br(x,y))
x)
= Ey-mollix) xx)
Vg log (y
Next, consider the trajectory balance objective used in GFlowNets learning (Bengio al., 2023b; Lee et al., 2024; Bartoldson et al., 2025), defined as:
C(y, x: = Iog (x(x) Ty xx)
(9)
(B (xp((3(xx,))
Taking the gradient of this objective with respect to 0 yields:
[
(1og Zx(x) TT x) exp(Br(xx))
VoL(o) = E-yywao--lxx
Vo log mo xx)
(10)
Thus, minimizing the KL divergence is equivalent (up to a constant) to minimizing the trajectory balance loss, confirming Proposition
B THEORETICAL ANALYSIS
We conduct an interpretation of FlowRL that clarifies the role of each component in the objective. Proposition 5. Minimizing the KL divergence in Eq. 5 is equivalent (in terms of gradients) to jointly maximizing reward and policy entropy:
max E Eyoo
8 Br(x,y) - log (x(x) log TTee (ylx)
H(TT))
(1I)
reward
entropy
Remark (FlowRL beyond reward maximization). Proposition 5 reveals that FlowRL can be inter- preted as jointly maximizing expected reward and policy entropy. This formulation encourages the policy to explore broader set of high-quality solutions, enabling more diverse and generalizable behaviors reasoning tasks. Our interpretation also aligns with prior work that views GFlowNets training as form maximum entropy RL (Mohammadpour et al., 2024; Deleu et al., 2024).
The proof of Proposition 5 is provided as below.
Recall from Eq. 3 and Eq. that the FlowRI. objective sourced from the minimization of KI. divergence:
exp(3 r)) Free (y x) (x
Zx(x)xo x)
DKT. To(y x)
Toly x)) log
dy
exp (3 r(x,y))
)) Trer x)
(12)
17

---

Published as a conference paper at ICLR 2026
Rearranging the terms, obtain:
exp (3 r(x. y)) ( x)
arg min DKL
To(y x)
||
(x(x)
(x) x)
arg min
Toly x) log
dy
exp (33 r(x, ())
-Trof (y x)
(13)
exp (3 (xxx))) Trof (y
x)
- Toly x) log moly xx)dy)
arg max
Ey Ey)xo((ix) log
Zx(x)
exp (3 r(x, y)) TTee x)
arg max Finally, we express the FlowRL objective in its compact form:
(x)) log
+ H((T))
Zx(x)
max E By-ro(()x))
yr(x.y) - log Zo(xx) +
log Fee (y/x)
+ H((TT)
(14)
reward
normalization
reference model constraint
entropy
Therefore, minimizing the FlowRL objective can be interpreted as jointly maximizing reward and entropy, while also aligning the policy with structured prior. The reward term drives task perfor- mance, while the normalization term Zø (x) ensures consistency with properly normalized target distribution. This encourages the policy Te to cover the entire reward-weighted distribution rather than collapsing to a few high-reward modes. The reference policy Tye provides inductive bias that regularizes the policy toward desirable structures, and the entropy term H(TT)) encourages diversity sampled solutions. Together, these components promote better generalization FlowRI..
C GFLOWNETS
We follow the notation of (Madan et al., 2023; He al., 2025) to introduce the fundamentals of GFlowNets. Let X' denote the compositional objects and R be reward function that assigns non- negative values to each object x E X. GFlowNets aim to learn a sequential, constructive sampling policy that generates objects with probabilities proportional to their rewards, i.e.,)(x)) R(x). This process can be represented as a directed acyelic graph (DAG) g = (S,A), where the vertices states, and the directed edges (u v) A are called actions. The
S are referred generation of an object E X' corresponds to a complete trajectory = (so - ... - 8n) E T within the DAG, beginning the initial state so and ending at terminal state 8, X. The state flow F(s) is defined as non-ne non-negative weight assigned to each state E S. The forward policy PF(s' s) specifies the transition probability to a child state s', while the backward policy PB(s s') specifies the transition probability to a parent state s. To this end, detailed balance objective enforces local flow consistency across every edge (s 55)
v(s + 5's (E( s;0) Fo(s') 8';0).
(15)
To achieve this flow consistency, GFlowNets employ training objectives at different levels of granu- larity, including detailed balance (Bengio et al., 2023b), trajectory balance (Malkin et al., 2022), and sub-trajecoe trbjerajeytory balance (Madan et al., 2023). Leveraging their diiversity-seeking behavior, GelowNets have been successfully applied across a range of domains, including molecule generation (Cretu et al., 2024), diffusion fine-tuning (Liu et al., 2025b; Zhang et al., 2025a), and amortized reason- ing (Hu et al., 2024; Yu et al., 2025a). Among various training objective in GFlowNets, trajectory balance maintains flow consistency at the trajectory level, defined as:
Zo Pr(s, st-1:0) = R(x) II Pa 8c:0).
(16)
Furthermore, sub-trajectory balance achieves local balance on arbitrary subpaths Tay = {s, + 8, ), offering more stable and less biased learning signal. We build on trajectory balance to extend our KL-based objective through a gradient-equivalenee formulation (Prop. 1), and further improve it to better support long CoT reasoning in RI.
18

---

Published as a conference paper at ICLR 2026
Table 5: Math reasoning performance (Avg@64) 64) at temperature 0.6. Relative improvements are shown as subscripts, with positive gains in green and negative changes in red. FlowRL consistently outperforms all baselines and achieves the best average score under this now-temperature setting.
Models
AIME 2024
AIME 2025
AMC 2023
MATH-500
Minerva
Olympiad
Avg
Qwen2. 5-7B
Base Model
Backbone
4.37
2.08
30.78
54.48
22.38
24.02
R++
10.57
4.102
66.02 +35.24
433.24
54.29 -0119
24.17 24.47+22.09
27.30
31.29
PPO
9.95 46.58
7-34 43:26
63.63 $32288
57.72
26.22 26.22 4-8.8
27.35
4.33
33.3
32.03
GRPO
14.01 6.64
10.73-6.65
64.10433.55
57.4142.98
23.17
27.11 43.09
32.76
FLowRL
14.32 +9.95
10.05 -..7
55.08 +24.30
66.78 422.00
31.52
9..14
34.60..1o.8
35.39
Table 6: Math reasoning performance (Avg@64) temperature 1.0. Relative improvements are shown as subscripts, with positive gains in green. FlowRL maintains robust performance under higher generation randomness and continues to outperform all baselines on average.
Models
AIME 2024
AIME 2025
AMC 2023
MATH-500
Minerva
Olympiad
Avg
@wen2. 5-7B
Base Model
Backbone
3.39
1,51
23.90
45.18
16.98
18.27
18.20
R+
10.63
4.63--311
66.99
66.99-43.09
43.09
54.3649.18
23.89+6.91
26.65
31.19
PPO
10.5247.13
6.50
63.04-39.14 19.14
37.46
25.91
27.16+8..9
31.77
GRPO
12.50+9.11
10.1048.50
64.72
37.15+11.97
23.28 46.30
26.90 4..63...3
32.44
FlowRI.
14.22 $10.83
9.58
52.92
66.20 p21.02
30.32, 4-1.34
34.47 116.00
34.62
Table 7: Ablation study on the effect of the 33 parameter in FlowRL. We report Avg @ 16 accuracy across six math reasoning benchmarks for different values of B.
Models
AIME 2024
AIME 2025
AMC 2023
MATH-500
Minerva
Olympiad
Avg
8 =
13.54
10.00
56.09
58,91
20.79
28.72
31.34
8 10
14.79
10.20
59.53
64.30
25.27
32.39
34.41
3
15.41
10.83
54.53
66.96
31.41
34.61
35.63
3 30
15.00
10.83
50.62
69.02
30.03
35.03
35.09
D HUMAN STUDY AND CROSS-DOMAIN EVALUATION
Human Evaluation. We conduct a comprehensive human evaluation that demonstrates strong agreement with BPT-44-miin assessments. We use the same rollouts from the GPT-4o-mini diversity experiment (Sec 6) to further validate diversity. As shown in Table 8, both evaluators independently identify FlowRL as the most diverse method and R+ as least diverse, with GRPO and PPO showing intermediate diversity levels.
Human Instruction: As a human evaluator, assess the diversity of solutions for each problem by examining 16 solution attempts per method. Rate diversity 1-3 scale based the following
criteria:
Score (low diversity): 13+ responses essentially identical approaches with only trivial dif- ferences in arithmetic, notation, or wording.
Score 2 (moderate diversity): 7-12 responses use the most common approach, with 2-4 responses
showing distinct alternative strategies.
Score 3 (high diversity): <6 responses use the same method, with 4+ distinctly different solution strategies present.
Other Domain Evaluation. We conduct additional experiments on MMLU (Hendrycks et al., 2020) and GPQA (Rein et al., 2024) to demonstrate FlowRL's effectiveness extends beyond math- ematical reasoning to other domains. We use Qwen-2.5-7B as the base model and follow the math training setup described in Sec 4. As shown in Table FlowRL achieves the highest overall
19

---

Published as a conference paper at ICLR 2026
scores on both benchmarks (72.13% on MMLU and 36.87% on GPQA). These results demonstrate FlowRL's strong generalization capability across different domains beyond the originally tested mathematical reasoning tasks.
Table 8: Human-evaluated diversity scores
Table 9: MMLU and GPQA benchmark per- formance.
(Mean + Std).
Method
Score
Method
MMLU
GPQA
R+
1.10 = 0.20
R++
71.82
27.02
GRPO
1.42 0.42
GRPO
71.87
33.08
PPO
1.67 = 0.39
PPO
72.10
33.84
FlowRL
2.45 + 0.35
FlowRL.
72.13
36.87
E EXTENDED RELATED WORK AND COMPARISONS
Recent notable works have addressed similar challenges in large language model reinforcement learning from different perspectives and across various domains. We provide a detailed comparison below to highlight key distinctions and commonalities with existing methods.
Length Normalization. Dr. GRPO (Liu al., 2025c) proposes an unbiased optimization method that improves token efficiency by removing standard normalization terms from the ad advantage calcu- lation and removing length terms from the loss objective, while focusing primarily on mathematical reasoning improvements. SRPO (Zhang et al., 2025c) addresses length conflicts through two- stage training approach (math-first then coding) and history resampling to filter zero-advantage samples. GSPO (Zheng et al., 2025) conducts gradient analysis and applies length normalization
[_)()) lod) to avoid unstable training, particu- the sequence-level importance ratio (0, (o (9:12))
larly crucial for MoE model training. FlowRL operates
trajectory-level flow-balance objective that initially faced gradient explosion issues during long CoT reasoning. To overcome this chal- lenge, FlowRL integrates length normalization (1)) log ()))))) directly into the trajectory balance formulation, ensuring training stability and enabling effective scaling to extended CoT sequences. Unlike approaches requiring domain-speciin training strategies, FlowRL's unified formulation nat- urally handles variable sequence lengths through principled reward shaping within the flow-balance framework, achieving stable optimization across diverse reasoning tasks.
KL-Related Policy Optimization Methods. Kimi-K1.5 (Team et al., 2025) employs on- policy sampling with KL regularization and uses empirical mean of sampled rewards (F mean(r(a, "1,8"), v")...,, r(x, yk. v)))) )) to approximate the normalizing constant Z. This objective has closed form solution that introduces log Z, where y is a parameter controlling the degree of reg- ularization, maintaining the traditional reward maximization framework. IPO (Azar et al., 2024) addresses overfitting preference- effective KI. regularization with deterministic preferences, targeting preference based alignment problems. FlowRL differs by deriving its objective from reverse KL divergence minimization, shift- ing from reward maximization reward distribution matching via flow balance. This approach employs a learnable partition function (x) parameterized by 3-layer MLP and incorporates importance sampling for the entire trajectory balance objective. This approach provides both theo- retical rigor through generative flow networks and practical effectiveness across diverse reasoning tasks without requiring preference data or domain-specift training paradigms.
learning by using identity mapping (V J) to maintain
F IMPLEMENTATION OF PARTITION FUNCTION Z
We detail the implementation of the partition function Z covering theoretical foundations and practical aspects.
20

---

Published as a conference paper at ICLR 2026
From the flow perspective: Z& measures the probability flow from the initial state So. Intuitively, it estimates the denominator the sum of rewards across all possible paths enabling conversion to a probability distribution via r(x.y)
7x(x)*
From the implementation perspective: Since the input of Zg corresponds to the initial state, utilize the prompt representation from the language model. Specifically, we extract the hidden states from the final layer of the language model for all prompt tokens, and compute their to obtain a exee-oimensional representation. This averaged hidden state vector serves as the input feature for computing the scalar partition function value (x)x).
We conduct comprehensive ablation studies examining: (1) MLP architecture depth (1/3/5 layers); (2) Removing Zg entirely: quantify how much Z contributes the overall performance im- provement; (3) Replacing Zg with a constant value: to assess whether adaptivity is necessary or simple approximation suffices.
The results demonstrate that the learnable partition function Zg is essential for FlowRL's perfor- mance. As shown in Table 10, varying MLP depth has minimal impact, with 3-layer MLP per- forming slightly better. Table 11 shows that removing Ze causes significant drops (-5.62 on AIME 2024, -6.25 on AIME 2025), while using a constant Zo performs even worse (-7.91 and -8.75 re- spectively). These results confirm that Zz is critical. Theoretically, it is essential for matching the reward distribution.
Table 10: MLP Architecture Depth.
Table 11: Partition Function Zg.
Za Arch.
AIME 2024
AIME 2025
Method
AIME 2024
AIME 2025
I-ayyer MLP
12.79
8.12
FlowRL
15.41
10.83
3-layer MLP
15.41
10.83
w/o Z
9.79
4.58
5-layer MLP
10.49
6.77
w/ constant Z
7.50
2.08
G TRAINING ANALYSIS
Training Dynamics. We analyze model evolution during training by tracking AIME 2025 accu- racy and response length. As shown in Figure FlowRL gradually outperforms GRPO during training.
FlowRL's response length grows faster than GRPO, reaching approximately 2000 tokens by step 100 compared to GRPO's 1200 tokens. Correspondingly, FlowRL achieves higher AIME 2025 accuracy, with the performance gap widening training progresses, particularly after step 75 where FlowRL. begins to consistently outperform GRPO.
GRPO
20.0
FlowRL
2000
FlowRL
1800
AIME 2025 (acc@8)
15.0
Response Length
1600
12.5
1400
1200
7.5
1000
800
600
0.0
20
60
80
100
20
60
80
100
Training Step
Training Step
Figure 5: Training dynamics on Qwen2.5-7B, including AIME 2025 Acc@8 (left) and response length (right).
21

---

Published as a conference paper at ICLR 2026
Reward Distribution Analysis. We analyze reward distribution statistics during training Qwen-2.5-32B. FlowRL maintains higher variance than GRPO, indicating exploration of diverse solutions. Specifically, FlowRL achieves higher variance, aligning with flow matching theory that encourages exploration multiple solution paths.
Table 12: Reward Distribution Statistics.
Step
GRPO Std
FlowRL Std
0.1087
0.1087
50
0.1714
0.1341
100
0.0000
0.1165
150
0.0323
0.1664
200
0.1630
0,0730
245
0.0509
0.2341
Length Normalization Ablation. We conduct ablation study the length normalization term (1/ly). The results demonstrate that length normalization is essential for stable training.
Without it, training becomes highly unstable: at step 10, generation length explodes 1827 tokens with gradient norm spiking to 4.6M; at step 50, length collapses to only tokens, confirming that length normalization is critical for FlowRL's stability.
1827
1500
Length
1250
Gradlent Norm
1000
102
750
560
10f
250
0
20
40
60
80
100
120
20
40
60
80
100
120
Training
Training
Figure Ablation study on length normalization term (1/1y)) Left: average response length. Right: gradient norm (log scale). Without length normalization, training exhibits severe instability with length explosion/collapse and gradient spikes.
H THE USE OF LARGE LANGUAGE MODELS
LLMs (specifically GPT-40-mini) are used as a judge to evaluate the diversity of solution approaches our diversity analysis (Figure 4), following Yu al. (2025a). All core research ideas, theoretical derivations, experimental design, and algorithmic innovations are developed by the authors without LLM assistance. The mathematical formulations and proofs are entirely the work of the human researchers. LLMs do not contribute to the fundamental conceptual development of FlowRL or the core insights about reward distribution matching via flow balance.
22

---

Published as a conference paper at ICLR 2026
Diversity Evaluation Prompt
System: You evaluating the DIVERSITY of solution approaches for mathematics competi- tion problem. Focus on detecting even SUBTLE differences methodology that indicate different problem-solving strategies.
PROBLEM:
(problem}
16 SOLUTION ATTEMPTS:
fooraated responsss)
EVALUATION CRITERIA Rate diversity from to 5:
Score 1- Minimal Diversity:
14+ responses essentially identical approaches
Same mathematical setup, same variable choices, same solution path
Only trivial differences (arithmetic, notation, wording)
Indicates very low exploration/diverstty in the generation process
Score 2- Low Diversity:
11-13 respouses use the same main approach 1-2 alternative approaches appear are rare
Minor variations within the dominant method (different substitutions, orderings)
Some exploration but heavily biased toward one strategy
Score 3- Moderate Diversity:
responses use most common approach
2-3 distinct alternative approaches present
Noticeable variation in problem setup or mathematical techniques
Balanced mix showing reasonable exploration
Score High Diversity:
4-6 responses use the most common approach
3-4 distinct solution strategics well-represented
Multiple mathematical techniques and prohlem framings
Strong evidence of diverse exploration strategies
Score Maximum Diversity:
No single approach dominates (53 responses use same method) 4+ distinctly different solution strategies
Wide variety of mathematical techniques and creative approaches
Excellent exploration and generation diversity
IMPORTANT NT: Focus on the DIVERSITY of attempted approaches. Return ONLY a number from
5.
23


Reviews:
[{"review_id": "1QreKsXFUT", "reviewer": "Reviewer_vbie", "summary": "This paper proposes FlowRL, a reinforcement learning algorithm that matches the full reward distribution via flow balancing instead of solely maximizing rewards for LLM post-training. Unlike reward-maximizing methods (REINFORCE++, PPO, GRPO) that tend to over-optimize dominant modes while neglecting less frequent but valid reasoning paths, FlowRL transforms scalar rewards into a normalized target distribution using a learnable partition function Zφ(x), then minimizes the reverse KL divergence between the policy and this distribution.The authors prove gradient equivalence between this KL objective and the trajectory balance formulation from GFlowNets, and introduce length normalization and importance sampling to address gradient explosion and sampling mismatch in long CoT reasoning.", "questions": "As mentioned in Weaknesses.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Novel application of GFlowNets to LLM reasoning through distribution-matching framework\n- Rigorous mathematical formulation with gradient equivalence proof\n- Exceptionally clear presentation with intuitive figures\n- Achieves substantial improvements", "weaknesses": "- No human evaluation: All diversity assessments rely on GPT-4o-mini judgments\n- Only tested on reasoning tasks (math, code); unclear if FlowRL works for other domains\n- All experiments use thinking models or reasoning-focused tasks; what about non-reasoning tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aEHeGKKllh", "reviewer": "Reviewer_oDPD", "summary": "FlowRL introduces a new RL objective for LLM post training that is motivated by connecting reverse KL with GFlowNet. In this new formulation, the advantage is regularized by the important sampling term and a learned partition function (via a small MLP from the mean pooled LLM hidden state) so that the model's output diversity is preserved. The author demonstrates effectiveness of the proposed method on Qwen-2.5 and DeepSeek-R1-Distilled Qwen models on math and reasoning benchmarks.", "questions": "It would be really interesting to ablate Z_phi to see how much partition function is helping. In principle this should help reducing training variance, but then the question is what if we simply tuning other hparams like number of rollouts/sample.\n\nIt would also be interesting to see how the model evolves during training from things like AIME vs. generation length, or AIME vs. training flops. It is not immediately clear that the gains reported in FlowRL would scale with model size and training flops.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is generally well written and easy to follow.\n\nThe biggest contribution from the paper is, in my opinion, the formulation of using GFlowNets to implement reserve KL for distribution matching training objective. The idea is elegant and the ablations seem to cover important tricks that are necessary for implementation on models at scale. It is great to see works that are both theoretically grounded and implementationally sound.", "weaknesses": "Given my lack of familiarity of the GFlowNet lineage of work, I can only comment on the algorithm from a purely pragmatic perspective.\n\nThe main drawback is that the FlowRL's benefits mainly come from important sampling to control model entropy, which could be achieved more easily with other methods without additional learnable parameters (in the partition function). In particular, the objective has 2 main components that are augumenting the advantage function: 1) Importance sampling, 2) partition function. From the ablations in the work, it appears that the IS part of the augmentation is very important, however, it is not clear whether the partition function is needed at all in practice. _If_ partition function is not necessary from a final model performance perspective, then FlowRL is similar to GRPO with a penalty on pi_theta/pi_ref, which in spirit looks quite similar to a KL penalty. Therefore, from a pragmatic standpoint, the question is whether the complexity of FlowRL necessary for the performance gain that we observe.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "257IeHjlkW", "reviewer": "Reviewer_Ngao", "summary": "1. FlowRL shifts RL from maximizing scalar rewards to matching the full reward distribution, eliminating mode collapse and enriching reasoning diversity.\n2. A learnable partition-function network approximates the intractable normalization constant. It also employs length normalization and importance sampling to enhance training stability.\n3. Experiments demonstrate that the proposed method outperforms both PPO and GRPO on math and code reasoning tasks.", "questions": "1. The ablation study omits an ablation on “reward shaping via length normalization.”  \n2. Why a 3-layer MLP is chosen to parameterize Z_ϕ could be further investigated and compared with alternative designs.  \n3. A statistical analysis of reward distributions under different training scenarios—detailing their shapes, similarities, and differences—would be valuable.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The insight is highly meaningful—it enhances the representational capacity of rewards and constitutes a valuable innovative attempt.  \n2. Supported by solid theoretical justifications and in-depth analysis.  \n3. The paper is well-written, fluent, and easy to follow.", "weaknesses": "1. Please refer to the \"Questions\" section for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0UduiwEMJD", "reviewer": "Reviewer_BKo5", "summary": "This paper introduces FlowRL, a novel reinforcement learning algorithm for LLM reasoning that tackles the \"mode collapse\" problem inherent in traditional reward-maximizing methods like PPO and GRPO. Instead of merely seeking the highest reward, FlowRL's core contribution is to reframe the objective as reward distribution matching, aiming to make the policy's output distribution over reasoning paths proportional to their rewards. Drawing inspiration from GFlowNets, the authors formulate this as a tractable \"trajectory balance\" objective and introduce crucial technical adaptations—length normalization and a modified form of importance sampling—to make it effective for long-chain-of-thought training. Through strong empirical results on challenging math and code benchmarks, the paper convincingly demonstrates that this new paradigm significantly outperforms strong baselines by promoting greater solution diversity and improving generalization.", "questions": "See \"Weaknesses\".\n\nAdditional question:\n1. The choice to parameterize the partition function Zφ as a 3-layer MLP is presented without justification. This is a critical architectural decision, yet its impact on performance is unevaluated. What was the rationale for this specific depth, and how sensitive is the model's performance to variations in the Zφ architecture?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. By creatively connecting GFlowNets with LLM reasoning, this paper provides an elegant and novel solution to the well-known problem of mode collapse, moving from the standard reward-maximization objective in RL to a more principled reward distribution matching paradigm.\n2. The authors introduce crucial technical solutions—length normalization for stability and modified importance sampling for data efficiency—demonstrating a strong command of the challenges in large-scale RL and bridging the gap between theory and practice.\n3. The paper is exceptionally clear, logically motivating and deriving the FlowRL algorithm from first principles. The claims are backed by strong and comprehensive empirical results on challenging math and code benchmarks, where FlowRL consistently and significantly outperforms established baselines like PPO and GRPO.\n4. This paper is significant as it introduces a new, effective direction for LLM post-training. By promoting diverse and generalizable reasoning, FlowRL not only pushes state-of-the-art performance but also opens promising avenues for developing more robust and exploratory generative agents, which could have a broad impact on complex problem-solving applications.", "weaknesses": "1. The paper only reports final performance metrics, making it impossible to assess whether FlowRL is more or less sample-efficient than GRPO and PPO. Including learning curves (accuracy vs. training steps/samples) is essential to substantiate the method's practical training efficiency, especially given that GFlowNet-based objectives can be sample-inefficient.\n2. While the proposed solutions to GFlowNet's instability (e.g., length normalization, detached importance sampling) are empirically effective, their necessity and impact are not deeply ablated, leaving them feeling more like engineering choices than rigorously justified algorithmic components.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)