FlowRL: Matching Reward Distributions for LLM Reasoning
Authors: name: Xuekai Zhu; name: Daixuan Cheng; name: Dinghuai Zhang; name: Hengli Li; name: Kaiyan Zhang; name: Che Jiang; name: Youbang Sun; name: Ermo Hua; name: Yuxin Zuo; name: Xingtai Lv; name: Qizheng Zhang; name: Lin Chen; name: Fanghao Shao; name: Bo Xue; name: Yunchong Song; name: Zhenjie Yang; name: Ganqu Cui; name: Ning Ding; name: Jianfeng Gao; name: Xiaodong Liu; name: Bowen Zhou; name: Hongyuan Mei; name: Zhouhan Lin
Published: 2025-09-18 17:56:36

Abstract: We propose FlowRL: matching the full reward distribution via flow balancing
instead of maximizing rewards in large language model (LLM) reinforcement
learning (RL). Recent advanced reasoning models adopt reward-maximizing methods
(\eg, PPO and GRPO), which tend to over-optimize dominant reward signals while
neglecting less frequent but valid reasoning paths, thus reducing diversity. In
contrast, we transform scalar rewards into a normalized target distribution
using a learnable partition function, and then minimize the reverse KL
divergence between the policy and the target distribution. We implement this
idea as a flow-balanced optimization method that promotes diverse exploration
and generalizable reasoning trajectories. We conduct experiments on math and
code reasoning tasks: FlowRL achieves a significant average improvement of
10.0% over GRPO and 5.1% over PPO on math benchmarks, and performs
consistently better on code reasoning tasks. These results highlight reward
distribution-matching as a key step toward efficient exploration and diverse
reasoning in LLM reinforcement learning.