FlowRL: Matching Reward Distributions for LLM Reasoning
Authors: name: Xuekai Zhu, name: Daixuan Cheng, name: Dinghuai Zhang, name: Hengli Li, name: Kaiyan Zhang, name: Che Jiang, name: Youbang Sun, name: Ermo Hua, name: Yuxin Zuo, name: Xingtai Lv, name: Qizheng Zhang, name: Lin Chen, name: Fanghao Shao, name: Bo Xue, name: Yunchong Song, name: Zhenjie Yang, name: Ganqu Cui, name: Ning Ding, name: Jianfeng Gao, name: Xiaodong Liu, name: Bowen Zhou, name: Hongyuan Mei, name: Zhouhan Lin

We propose FlowRL: matching the full reward distribution via flow balancing
instead of maximizing rewards in large language model (LLM) reinforcement
learning (RL). Recent advanced reasoning models adopt reward-maximizing methods
(\eg, PPO and GRPO), which tend to over-optimize dominant reward signals while
neglecting less frequent but valid reasoning paths, thus reducing diversity. In
contrast, we transform scalar rewards into a normalized target distribution
using a learnable partition function, and then minimize the reverse KL
divergence between the policy and the target distribution. We implement this
idea as a flow-balanced optimization method that promotes diverse exploration
and generalizable reasoning trajectories. We conduct experiments on math and
code reasoning tasks: FlowRL achieves a significant average improvement of
10.0% over GRPO and 5.1% over PPO on math benchmarks, and performs
consistently better on code reasoning tasks. These results highlight reward
distribution-matching as a key step toward efficient exploration and diverse
reasoning in LLM reinforcement learning.