Flow Matching Policy Gradients
Authors: David McAllister
Songwei Ge
Brent Yi
Chung Min Kim
Ethan Weber
Hongsuk Choi
Haiwen Feng
Angjoo Kanazawa
Venue: ICLR 2026 Poster (2026)
Keywords: Flow Matching
Policy Gradient
Primary area: reinforcement learning

Abstract: Flow-based generative models, including diffusion models, excel at modeling continuous distributions in high-dimensional spaces. In this work, we introduce Flow Policy Optimization (FPO), a simple on-policy reinforcement learning algorithm that brings flow matching into the policy gradient framework. FPO casts policy optimization as maximizing an advantage-weighted ratio computed from the conditional flow matching loss, in a manner compatible with the popular PPO-clip framework. It sidesteps the need for exact likelihood computation while preserving the generative capabilities of flow-based models. Unlike prior approaches for diffusion-based reinforcement learning that bind training to a specific sampling method, FPO is agnostic to the choice of diffusion or flow integration at both training and inference time. We show that FPO can train diffusion-style policies from scratch in a variety of continuous control tasks. We find that flow-based models can capture multimodal action distributions and achieve higher performance than Gaussian policies, particularly in under-conditioned settings.

Published as a conference paper at ICLR 2026
FLOW MATCHING POLICY GRADIENTS
David McAllister ++
Songwei Ge
Brent Vii
Chung Min Kim Angjoo Kanazawa'
Ethan Weber!
Hongsuk Choi
Haiwen Fengl.2
UC Berkeley
Max Planck Institute for Intelligent Systems
ABSTRACT
Flow-based generative models, including diffusion models, excel at modeling con- tinuous distributions high-dimensional spaces. In this work, we introduce Flow Policy Optimization (FPO), simple on-policy reinforcement learning algorithm that brings flow matching into the policy gradient framework. FPO casts policy optimization maximizing ditional flow matching loss, in a manner compatible with the popular PPO-clip framework. It sidesteps the need for exact likelihood computation while preserv- ing the generative capabilities of flow-based models. Unlike prior approaches for diffusion-based reinforcement learning that bind training to specific sampling method, FPO is agnostic to the choice of diffusion or flow integration at both training and inference time, We show that FPO can train diffusion-style policies from scratch in a variety of continuous control tasks. We find that flow-based models can capture multimodal action distributions when necessary and achieve comparative or higher performance than Gaussian policies, particularly in under- conditioned settings. For overview of FPO's key insights and interactive results please see our anonymized supplemental website
advintegg-weiighted ratio computed from the con-
INTRODUCTION
Flow-based generative moodels-particulatty diffusion have emerged powerful tools for generative modeling across the domains of images (Saharia et al., 2022 Ho et al. 2022b).
videos (Brooks et al.) 2024 Polyak et al. |2024 Veo-Team al.) (024), speech (Liu et al. 2023
audio (Kong et al. 2021), robotics (Chi et al.] (022b)) and molecular ar dynamics (Raja et al. (2025). In
parallel, reinforcement learning (RL) has proven to be effective for optimizing neural networks with non-differentiable objectives, and is widely used as a post-training strategy for aligning foundation
models with task-specific goals (Chu et al. 2025 Liu et al. 2024).
In this work, we introduce Flow Policy Optimization (FPO), policy gradient algorithm for optimiz- ing flow-based generative models. FPO reframes policy optimization as maximizing an advantage- weighted ratio computed from the conditional flow matching (CFM) objective (Lipman et al. 2023). Intuitively, FPO shapes probability flow transform Gaussian noise into high-reward actions by re- inforcing its experience using flow matching. The method is simple to implement and can be readily integrated into standard techniques for stochastic policy optimization. We use PPO-inspired sur- rogate objective, enabling flow policies as a drop-in replacement for Gaussian policies in existing PPO frameworks.
FPO offers several key advantages. It sidesteps the complex likelihood calculations typically associ- ated with flow-based models, instead using the flow matching loss as a surrogate for -ikeeeiiddood in the policy gradient. The objective increases the evidence lower bound of high-reward actions. FPO treats the sampling procedure as a black box during rollouts, which allows for flexible inte- gration with any sampling approach opetheeh-whether whether deterministic stochastic, and with any number of steps during training and inference.
We theoretically analyze FPO's correctness and empirically validate its performance across diverse set of tasks. These include a GridWorld environment, 10 continuous control tasks from MuJoCo Playground (Zakka et al. 2025), and high-dimensional humanoid control- all trained from seratch.
"qqual contribution.

---

Published as a conference paper at ICLR 2026
FPO demonstrates robustness across tasks, enabling effective training of flow-based policies in high- dimensional domains. We probe flow policies learned in the toy GridWorld environment and find that on states with multiple possible optimal actions, it learns multimodal action distributions. On humanoid control tasks, we show that the expressivity of flow matching enables single-stage training of conditoondd control policies, where only root-level commands are provided. In contrast, Gaussian policies struggle to learn viable walking behaviors in such cases, suggesting the practical benefits of more flexible policy representations. We will open-source our code enable further research in this direction.
2 RELATED WORK
Policy Gradients. We study on-policy reinforcement learning, where a parameterized policy is op- timized to maximize cumulative reward in a provided environment. This is commonly solved with policy gradient techniques, which bypass the need for differentiable environment rewards by weight-
ing action log-probabilities with observed rewards or advantages (Sutton et al. 1999 Williams/
1992
Kakade, 2002; Peters & Schaal 2008 Schulman et al. 2015a 2017 Mnih et al.) 2016: Wang 2016 Shao et al. [024). Policy gradient methods are central in learning policies for general
ct al.
continuous control tasks (Duan et al. 2016; Huang et al.) 2024), robot locomotion (Rudin et al.,
2022 Schwarke et al., 2023 Mittal al. 2024 Allshire et al. 2025) and manipulation (AliiinnnAAAaaa (Akkaya ct al. 2019| et al. 2021 Qi et al. | 2025). They have also been adopted increasingly for
searching through and refining prior distributions in pretrained generative models. This has proven effective for alignment with human preferences (Ouyang et al., 2022; Christiano et al. 2023 and
improving reasoning using verifiable rewards (DeepSeek-A) Al et al.) 2025).
In this work, we propose a simple algorithm for training flow-based generative policies, such as diffusion models, under the policy gradient framework. By leveraging recent insights from flow matching (Lipman et al.) 2023), we train policies that can represent richer distributions than the diagonal Gaussians that are most frequently used for reinforcement learning for continuous con-
trol (Rudin et al. 2022 Schwarke et al. 2023 Mittal et al. 2024 Allshire et al.) 2025 Qi et al., 2025), while remaining compatible with standard actor-critic training techniques.
2023
Diffusion Models. Diffusion models are powerful tools for modeling complex continuous distribu- tions and have achieved remarkable success across a wide range of domains. These models have become the predominant approach for generating images (Ho et al. 2020 Song et al. 2022 Rom-
bach et al. 2022. Song & Ermmn 2020), videos (Ho et al.] 2022c Singer et al. 2022; Ho et al.
2022a Brooks al.] 2024), and more recently, robot actions (Chi et al.) 2024b) Black et al. 2024
NVIDIA at 2025). In these applications, diffusion models aim to sample from a data distribution
of interest, whether scraped from the internet or collected through human teleoperation.
Flow matching (Lipman et al., 2023) simplifies and generalizes the diffusion model framework. It learns a vector field that transports samples from tractable prior distribution the target data distribution. The conditional flow matching (CFM) objective trains the model to denoise data that has been perturbed with Gaussian noise. Given data r and noise e |O, 1). the CFM objective can be ex expressed as:
LerMO = Br.g(s). [100 u(x,, r 1)1
where x- = a. represents the partially noised sample at flow step T, an interpolation of noise and data with schedule defined by hyperparameters a, and σ,- (s(x,; 7) is the model's estimate of the velocity to the original data, and u(=r, r x:) is the conditional flow x c. The learned velocity field a continuous mapping that transports samples from simple, tractable distribution (e.g. Gaussian noise) to the training data distribution through ODE integration.
Optimizing likelihoods directly through flow models requires divergence estimation (Skreta et al., 2025) and is computationally prohibitive. Instead, flow matching optimizes variational lower bounds of the likelihood with the simple denoising loss above. In this work, we leverage flow matching directly within the policy gradient formulation.
Diffusion Policies. Diffusion- based policies have shown promising results in robotics and decision-
making applications (Chi et al., 2024a| Ajay et al.] 2023 B Black al.) 2024). Most existing ap- proaches train these models via behavior cloning (Janner et al. 2022] Chi et al. 2024b). where the
policy is supervised to imitate expert trajectories without using reward feedback.
2

---

Published as a conference paper at ICLR 2026
Recent work by Psenka al. (2023) explores off-policy training of diffusion policies via Q-score matching. While off-po remforcement learning continues to make progress (Seo et al. 2025 Fujimoto et al., 2018). on-policy methods dominate practical applications today. Methods like DDPO Black et al. (2023), DPPO Ren et al. (2024), and Flow-GRPO Liu et al. (2025) adopt on- policy policy gradient methods by treating initial noise values as observations from the environment, framing the denoising process as a Markov decision process, and training each step as a Gaussian policy using PPO. Our approach differs by directly integrating the conditional flow matching (CFM) objective into PPO-like framework Since FPO integrates flow matching as its fundamental primi- tive, agnostic to the choice of sampling method during both training and inference.
FLOW MATCHING POLICY GRADIENTS
3.1 POLICY GRADIENTS AND PPO
The goal of reinforcement learning is to learn policy wo that maximizes expected return in provided environment. At each iteration of online reinforcement learning, the policy is rolled out to collect batches of observation, action, and reward tuples (or, re) for each environment timestep These rollouts can used the policy gradient objective (Sutton et al. 1999) to increase likelihood of actions that result in higher rewards:
Bag-T((e log (a o.)
where A, is advantage estimated from the rollout's rewards ry and learned value function (Schul- man et al. (2015b).
The vanilla policy gradient is valid only locally around the current policy parameters. Large updates can lead to policy collapse or unstable learning. To address this, PPO (Schulman et al. 2017)
incorporates a trust region by clipping the likelihood ratio:
max Ea min (r(0)A,, clip(r(0), - (n)=))))))))
(3)
where edlip is tunable threshold and r(0) is the ratio between current and old action likelihoods:
To oz)
r(0)
Tataa o.))
PPO is performance. Like the standard policy gradient, however, it requires exact likelihoods for sampled actions. These quantities are tractable for simple Gaussian or categorical action spaces, but compu- tationally prohibitive estimate for flow matching and diffusion models.
popular choice for on-policy reinforcement learning because of its stability, simplicity, and
3.2 FLOW POLICY OPTIMIZATION
We introduce Flow Policy Optimization (FPO), online reinforcement learning algorithm for poli- cies represented as flow models 0g. There are two key differences in practice from Gaussian PPO. During rollouts, flow model transforms random noise into actions via sequence of learned trans- formations, enabling much more expressive policies than those used in standard PPO. Also, to up- date the policy, the Gaussian likelihoods are replaced with a transformed flow matching loss.
Instead of updating exact likelihoods, we propose proxy ŞPPO for the likelihood ratio. FPO's overall objective is the same as Equation 3
but with the ratio substituted:
max (a, min (0) FFO c 1 e
(5)
Intuitively, FPO's goal to steer the policy's 's probability flow toward high-return behavior. Instead of computing likelihoods, we construct simple ratio estimate using standard flow matching losses:
APPO (8) exp(CcrM, (a;; ot) Lerm oz)),
(6)
which, will discuss, can be derived from optimizing the evidence lower bound.
3

---

Published as a conference paper at ICLR 2026
For given action and observation pair, (ar is estimate of the per-sample conditional flow matching loss Larm,
Noc
(crM,o(ar = Nmc E (T)
Co(T): 6,) ( T(.0)) - (at <;)11
(8)
0 = (1- + TrEs:
(9)
where we denote flow timesteps with and environment timesteps with t. We include both timesteps
in of which represents an action at rollout time t with noise level т following Eq. | with OT schedule (Lipman et al. 2023). LCEM, and LCFM, share 6, (0, I) and T; E [0, 1] samples.
Properties. FPO's ratio estimate in Equation 6 serves as drop-in replacement for the PPO likelihood ratio. FPO therefore inherits compatibility with advantage estimation methods like GAE (Schulman et al. 20155b and GRPO (Shao al., 2024). Without loss of generality, is also compatible with flow and diffusion implementations based on estimating noise (Ho et al., 2020) or clean action a, (Ramesh et al. 2022), which can reweighted for mathematical equivalence to
La,crm (Krrras et al. 2022).
3.3 FPO SURROGATE OBJECTIVE
Exact likelihood computationally expensive even to estimate in flow-based models. Instead, is common to optimize the evidence lower bound (ELBO) proxy for log- log-likelihood::
(LBO,( oe) = log (a o.) - -DK
(10)
where DEL is the KL gap between the ELBO and true log-likelihood and xo is the distribution captured by sampling from the flow model. Flow matching and diffusion models use the conditional
flow matching loss, simple denoising reconstruction objective. Prior work (Kingma et al. (023)
shows that the CFM loss (Eq. []] corresponds to the negative ELBO. (Kingma et al. (023). Using this fact, the FPO ratio in Eq. corresponds to the ratio of ELBOs under current and old policies:
(0) exp(ELBO, (a, oz))
(11)
exp(ELBORR. (4: [o:))
Decomposing this ratio reveals a scaled variant of the true likelihood ratio (Equation
Te (a oc) exp (DKL.) 800)
(PO
(12)
To
(a oz)
cxp
(L)
Likelihood
Gop
Here, the ratio decomposes into the standard likelihood ratio and an inverse correction term involving the KI. gap. Maximizing this ratio therefore increases the modeled likelihood while reducing the KL gap gat-bott both of which are beneficial for policy optimization. The former encourages the policy to favor actions with positive advantage, while the latter tightens the approximation to the true log- likelihood.
3.4 ESTIMATING THE FPO RATIO WITH FLOW MATCHING
We estimate the FPO ratio using the flow matching objective directly, which follows from the lationship between the weighted denoising loss CY and the ELBO established by Kingma & Gao 2023). Ly is more general form of the flow matching and denoising diffusion loss that parame- terizes the model as predicting čo, an estimate of the true noise present the model input.
The weighted denoising loss C. for clean action a, takes the form:
C (a:) = 1.. ((+))
(-22)
dA
|1Eo (ao -
(13)
c
where w is a choice of weighting and X, represents the log-SNR at noise level T. We estimate this value with Monte Carlo draws of timestep and noise e:
f% (T, 6) ()r)))
1
(-
(- dA
(1oo(aT: -c13-
(14)
4

---

Published as a conference paper at ICLR 2026
Algorithm Flow Policy Optimization (FPO)
Require:
Policy parameters 0, value function parameters s, clip parameter MC samples Nmc 1: while not converged do
2: Collect trajectories using any flow model sampler and compute advantages Ag
3: For each action, store Nmc timestep-noise pairs {(T; E,) and compute lo(Ti, E.)
4: fola +
5:
for each optimization epoch do
6: Sample mini-batch from collected trajectories
7:
for each state-action pair (oz, az) and corresponding MC samples {(T,, Es)3 do Compute (T(T,, E,) using stored (T,, E))
8:
9:
f. c exp (-
G)) Le (T.
10:
(0) min(fagA, clip(fs. 1 + e).Ao) end for
11:
12: 0 <- Optimizzer(0,, V Σ (0))
13: end for
Update value function parameters & like standard PPO
15:
end while
The choice of weighting incorporates the conditional flow matching loss and standard diffusion loss as specific cases of a more general family L3 (ax).
We focus here the constant weight case (o(A) 1 (diffusion schedule), which yields the sim- plest theoretical connection. Similar results hold for many popular schedules, including optimal transport (Lipman al. 2023) and variance preserving schedules. Please the supplementary material for details.
For the diffusion schedule, Kingma & Gao 2023 proves that:
Cg (ax) -ELBO,(a:)) +
(15)
where e is a constant w.r.t 0. Geometrically, minimizing C points the flow more toward Minimizing Z` also maximizes the ELBO (Eq, and thus the likelihood of a,, so flowing toward specific action makes it more likely. This intuition aligns naturally with the policy gradient objective: we want to increase the probability of high-advantage actions. By redirecting flow toward such actions (i.e., minimizing their diffusion loss), we make them more likely under the learned policy.
Using this relationship, we express the FPO ratio (Eq, ITp in terms of the flow matching objective:
FPO exp (ELBOO)(((a)))))
exp(C (ax) CV. (at)),
(16)
exp(ELBO (arlo:))
where 4y. per Equation 7 can be estimated by averaging Nme draws of (r, c). We find the sample count Nme to be a useful hyperparameter for controlling learning efficiency. This estimator recovers the exact FPO ratio in the limit, although
only few draws in practice.
One possible concern with smaller Nme values is bias. A ratio estimated from only one (7, pair, (17)
F p (T, E) exp(C" (T, e) t/ (T,
is in expectation only an bound of the true ratio. This can be shown by Jensen's 's inequality:
[
FPO (T, c) 2 FPO
(18)
To understand the upward bias, we can use the der-vvvivetive trick to decompose the FPO gradient:
V Vorh FPO (T,c) = - PPo (T, e) Vo (T,
(19)
Since the gradient operator commutes with expectation, gradient term on the right side unbi- ased:
E.,[- ( = - = V (LB)
(20)
In other words, gradient estimates are directionally unbiased even with worst-case overestimation of ratios. Our experiments consistent with this result: while additional samples help, observe empirically in Section that FPO can be trained to outperform Gaussian PPO even with Nnc = 1. Algorithm detaalss FPO's practical implementation using this mathematical framework.
5

---

Published as a conference paper at ICLR 2026
Gridwarid with Goals
Learned Flow and Target Action Distribution at
Sampled Trajectories
Denoising sleps
Goal
Agent
stor
ewd
Figure 1: Grid World. (Left) 25x GridWorld with green goal cells. Each arrow shows denoised action sampled from the FPO-trained policy, conditioned on a different latent noise vector. (Center) At the saddle-point state (+) shown on the left, we visualize three denoising steps 7 the initial Gaussian gradually transforms into the target distribution through the learned flow, illustrated by the deformation of the coordinate grid. (Right) Sampled trajectories from the same starting states reach different goals, illustrating the multimodal behavior captured by FPO.
3.5 DENOISING MDP COMPARISON
Existing algorithms (Black et al. 2023 Ren et al. 2024 Liu et al. 2025) for on-policy reinforce- ment learning with diffusion models reformulate the denoising process itself Markov Decision Process (MDP). These approaches bypass flow model likelihoods by instead treating every step in the sampling chain as its own action, each parameterized Gaussian policy step. This has a few limitations that FPO addresses.
First, denoising MDPs multiply the horizon length by the number of denoising steps (typically 10-50), which increases the difficulty of credit assignment. Second, these MDPs do not consider the initial noise sample during likelihood computation. Instead. these noise values are treated as observations from the environment-tthe significantly increases the dimensionality of the learning problem. Finally, denoising MDP methods are limited stochastic sampling procedures by con- struction. Meanwhile, FPO inherits the flexibility of all flow/diffusion sampler choices, including fast deterministic and higher-order samplers. Perhaps most importantly, FPO is simpler to imple- ment because it does not require custom sampler the notion of extra environment steps.
EXPERIMENTS
We assess FPO's effectiveness by evaluating multiple domains. Our experiments include: (1) an illustrative GridWorld environment using Gymnasium (Brockman al.) 2016 Towers et al. 2024
(2) continuous control tasks with MuJoCo Playground (Zakka et al. 2025 Todorov et al. and (3) physics-based humanoid control in Isaac Gym (Makoviychuk al. 2021). These tasks vary in dimensionality, reward sparsity, horizon length, and simulation environments.
2012
4.1 GRIDWORLD
We first test FPO on a 25x25 GridWorld environment designed to probe the policy's ability to capture multimodal action distributions. As shown in Figure left, the environment consists of two high reward regions located as the top and bottom of the map (green cells). The reward is sparse: agents receive a single reward upon reaching goal a penalty, with no intermediate rewards. This setup creates saddle points where multiple distinct actions can lead to equally successful outcomes, offering natural opportunity to model diverse behaviors.
We train diffusion policy from scratch using FPO by modifying a standard implementation Yu 2020) of PPO. The policy parameterized as a two-layer MLP modeling p(a, s, az), where E IR is the action, E IR2 is the grid state, and of E I is the latent noise vector noise level т. initialized from (O0, I) at T = O. FPO consistently maximizes the return this environment. The anrows in Figure left shows denoised actions at each grid location, computed by conditioning on a random of (0, I) and running 10 steps of Euler integration. In Figure] center, we probe the learned policy by visualizing the flow over its denoising steps at the saddle point illustrated by

---

Published as a conference paper at ICLR 2026
BailinCup
CartpoleBalance
ChestahRun
FingerSpin
FingerTurnEayy
Eval Reward
600
800
600
600
450
400
300
400
200
200
200
150
200
0
15
30
45
60
15
30
45
60
0
15
30
45
60
0
15
30
45
60
15
30
45
60
FingerTumHard
FishSwim
PointMass
ReacherEasy
ReacherHard
Eval Reward
600
800
800
800
450
600
600
400
300
400
400
400
200
200
200
200
0
0
15
30
45
60
0
15
30
45
60
0
15
30
45
60
15
30
45
60
15
30
45
60
Gaussian PPO
Figure 2: Comparison between FPO and Gaussian PPO (Schulman et al. 2017 on DM Control Suite tasks. Results show evaluation reward mean and standard error (y-axis) over 60M environment steps (x-axis). We run 5 seeds for each task; the curve with the highest terminal evaluation reward bolded.
BallinCup
CartpoleBalance
CheetahRun
FingerSpin
FingerTurnEasy
Eval Eval Reward
800
A
800
600
800
600
600
450
600
400
400
300
400
200
200
150
HingerTurmHard
FishSwim
PointMass
ReacherEasy
ReacherHard
600
Eval Reward
600
800
600
600
600
400
300
400
400
200
150
200
200
FPO
DPPO
Figure 3: Comparison between FPO and DPPO (Ren et al., 2024 DM Control Suite tasks.
Results show evaluation reward mean and standard eтror (y-axis) over 60M environment steps (x- axis). We seeds for each task; the curve with the highest terminal evaluation reward is bolded.
the red star where either going up or down leads to the optimal reward. The initial Gaussian evolves into bimodal distribution, demonstrating that the policy captures the multi-modalityy of the solution at this location. Figure rright shows multiple trajectories sampled from the policy, initialized from various fixed starting positions. The agent exhibits multimodal behavior, with trajectories from the same starting state reaching different goals. Even when heading toward the same goal, the paths vary significantly, reflecting the policy's ability to model diverse action sequences.
We also train a Gaussian policy using PPO, which successfully reaches the goal regions. Compared FPO, it exhibits more deterministic behavior, consistently favoring the nearest goal with less variation in trajectory patterns. Please see the supplemental material for more details.
4.2 MuJoCo PLAYGROUND
Next, we evaluate FPO for continuous control using MuJoCo Playground Zakka et al. 2025). We compare three policy learning algorithms: (i) a Gaussian policy trained using PPO, (ii) a diffusion
policy trained using FPO. and (iii) diffusion policy trained using DPPO (Ren al. 2024). We eval- uate these algorithms on 5 seeds for each of 10 environments adapted from the DeepMind Control Suite (Tassa et al. 2018 T'unyasuvunakool al. 2020). Results reported in Figures and

---

Published as a conference paper at ICLR 2026
Methods
Goal conditioning
Success rate (t)
Alive duration (f)
MPJPE ())
Gaussian PPO
All joints
98.7%
200.46
31.62
FPO
All joints
96.4%
198.00
41.98
Gaussian PPO
Root + Hands
46.5%
142.50
97.65
FPO
Root + Hands
70.6%
171.32
62.91
Gaussian PPO
Root
29.8%
114.06
123.70
FPO
Root
54.3%
152.90
73.55
Table 2: Humanoid Control Quantitative Metrics. We compare FPO with Gaussian PPO with different conditioning goals, and report the success rate, alive duration, and MPJPE averaged over all motion sequences.
Policy implementations. For the Gaussian policy baseline, run the Brax (Freeman et al. 202 based implementation used by MuJoCo Playground (Zakka et al. 2025) PPO training scripts. We also Brax PPO a starting point for implementing both FPO and DPPO. Following Section (.2 only small changes are required for FPO: noisy action and timestep inputs are included as input to the policy network, Gaussian sampling is replaced with flow sampling, and the PPO loss's likelihood ratio is replaced with the FPO ratio approximation. For DPPO, we make the same policy network modification, but apply stochastic sampling (Liu al. 2025) during rollouts. We also augment each action in the experience buffer with the exact sampling path that was taken to reach it. Following the two-layer MDP formulation in DPPO (Ren et al. 2024), we then replace intractable action likelihoods with ciisicoondiioned sampling path likelihoods.
Hy Hyperparameters. We match hyperparameters Gaussian PPO, FPO, and DPPO training when- ever possible: following the provided configurations in Playground (Zakka et al. 2025), all exper- iments use ADAM (Kingmal 2014), 60M total environment steps, batch size 1024, and 16 updates per batch. For FPO and DPPO, we use 10 sampling steps, set learning rates to 3c-4, and swept clipping epsilon dlip E {0.01, 0.05, 0.1, 0.2, 0.3). For DPPO, we perturb each denoising step with Gaussian noise with standard deviation Øs. which we sweep eclip = 0.05 produces the best FPO results and gelip = 0.2, oe = 0.05 produced the best DPPO re- sults; use these values for all experiments. For fairnsss, we also tuned learning rates and clipping epsilons for Gaussian PPO. We provide more details about hyperparameters and baseline tuning in the appendix.
{0.01, 0.05, 0.1]. We found that
Results. We observe Figures 2 and that FPO-optimized policies outperform both Gaussian PPO and DPPO, achieving higher rewards on 8 of 10 Playground tasks.
Method
Reward
Gaussian PPO
667.81166.0
Gaussian PPO
577.2:174.4
Analysis. In Table we present average evaluation rewards for baselines, FPO, and several variations of FPO. We observe: (1) (T sampling important. Decreasing the number of sampled pairs generally decreases evaluation rewards. More samples can improve learning without requiring more expen- sive environment steps. (2) c-CFM is preferable over u-CFM in Playground. e-CFM refers to computing conditional flow matching losses by first converting velocity estimates to noise values; u-CFM refers to CFM directly on velocity estimates. In Playground, found that the former produces higher erage rewards. We hypothesize that this is because e scale is invariant to action scale, which results in better generalization for eliip choices. For fairness, we also performed learning rate and clipping ratio sweeps for the u-MSE ablation. (3) Clip- ping. Like Gaussian PPO, the choice of eelip in FPO signifi- cantly impacts performance.
DPPO
652.5+83.7
FPO
759.31145.3
FPO, 1 (T,c)
691,6++50.3
FPO, 4 (T,E)
731.2455.2
FPO, u-MSE
664.61148.5
FPO, -0.
623.3476.3
FPO, -02
526.4476.8
Table 1: FPO variant compar- ison. We report averages and standard enrors across MuJoCo tasks. USiig default hyperparam- eters from MuloCo Playground. IPOO results use 8 (r,c) pairs, e- MSE, gelip 0.05.
4.3 HUMANOID CONTROL
Physics-aware humanoid control is dimherodimensional than standard MuJoCo benchmarks, making a stringent test of FPO's capability. We therefore train a humanoid policy to track motion-capture (MoCap) trajectories in the PHC setting (Luo al. 2023a This experiment follows the goal-
8

---

Published as a conference paper at ICLR 2026
Gaussin
(a) Episode return along training.
(b) Root++hand conditioning.
(c) Rough terrain locomotion.
Figure 4: Physics-based Humanoid Control. (a) The curves show that FPO performance is close to that of Gaussian-PPO when conditioning on all joints and surpasses it when goals are reduced to the root or hands, indicating stronger robustness to sparse conditioning, (b) In the root haads goal setting, FPO (blue) tracks the reference motion (grey) while Gaussian-PPO (orange) falls, (c) Trained with terrain randomization, FPO walks stably across procedurally generated rough ground.
conditioned imitation- -eattiinn paradigm pioncered by DeepMimic (Peng et al. 2018). in which simulated characters learn to reproduce reference motions. Sparse goals are significantly more chal- lenging, requiring the policy to fill in the missing joint specification in physically plausible manner.
Implementation details. Our simulated agent is SMPL-based humanoid with 24 actuated joints, each with six degrees of freedom. The policy receives both proprioceptive observations and goal information computed from the motion-capture reference. A single policy is trained to track AMASS (Mahmood et al., 2019) motions following PHC (Luo et al. 2023a). We use the root height, joint positions, rotations, velocity, and angular velocity in local coordinate frame as the robot state. For goal conditioning, we compute the difference between the tracking joint informa- tion (positions, rotations, velocity, and angular velocity) and the current robot's joint information, as well as the tracking joint locations and rotations. We explore both full conditioning, i.e., condi- tioning on all joint targets, and under conditioning, i.e., conditioning only on the root or the root and hands targets. Note that the same imitation reward based on all joints is used for both conditioning experiments. We adopt the per-joint tracking reward from DeepMimic (Peng et al., 2018
Evaluation.
For evaluation, we compute the success rate, considering an imitation unsuccessful if the average distance between the body joints and the reference motion exceeds 0.5 meters at any point. We also report the average duration the agent stays alive till it completes the tracking or falls. Finally, we compute the global mean per-joint position error (MPJPE) on the conditioned goals.
Results. Figure 4a shows that successfully train FPO from scratch on this nigh-ddimensionll con- trol task. With full joint conditioning, FPO performance is close to Gaussian PPO. However, when the model is mnder-conditioned conditioned mnndttoned--cnddtionneddd only the root or the root and hands FPO outper-
forms Gaussian PPO, highlighting the advantage of flow-based policies. While prior methods have achieved sparse-goal control with Gaussian policies, they first train teacher policy that conditions on full joint reference and then distill the knowledge to sparse conditioned policies (Tessler et al., Li et al.
2024
2025) or train separate encoder observing sparse references (Luo et al. 2023b
2024
highlighting the challenging nature of this problem setup.
Figure 4b visualizes the behaviors in the + setting; FPO tracks the target closely, whereas the Gaussian policy drifts. Tableeiquantifies these trends, with FPO achieving higher success rates in the under-conditioned scenarios. Finally, as illustrated Fig. 4c FPO trained with terrain random- ization enables the humanoid to traverse rough terrain, showing potential for sim-to-real transfer. Please see the supplemental for training hyperparameter details.
DISCUSSION
We introduce Flow Policy Optimization (FPO), a simple and intuitive algorithm for training flow- based generative models using policy gradients. We demonstrate FPO across range of tasks where it shows promising results. Looking ahead, exciting future directions include investigating FPO on real robotic systems to test its sim-to-real capabilities. Another exciting direction is fine-tuning large pretrained diffusion policies (c.g., vision- Langunn-angggee-ee actiion models) with FPO, and exploring how to incorporate one-step flow methods such mean-flow Geng al. (2025) for improved efficiency.

---

Published as a conference paper at ICLR 2026
REFERENCES
Anurag Ajay. Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh Interna- tional Conference on Learning Representations, 2023.
1lge Akkaya, Marcin Andrychowicz. Macick Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's 's cube with a robot hand. arXiv preprint arXiv:1910.071 13, 2019.
Sina Alemohammad, Josue Casco-Rodrighe Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G Baraniuk. Self-consuming generative models go mad. International Conference on Learning Representations (ICLR). 2024.
Arthur Allshire, Hongsuk Choi, Junyi Zhang. David McAllister, Anthony Zhang, Chung Min Kim, Trevor Darrell, Pieter Abbeel, Jitendra Malik, and Angjoo Kanazawa. Visual imitation enables contextual humanoid control. arXiv preprint wrxiv2505.003729, 2025.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. /3301, 2023.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. Toi A vision- language-action flow model for general robot control, 2024. URL /tttps://axxi.. org/
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing. David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL
www. cow/research.
rideo-generation. meeeeerrriii--mmmeessees -aw---mmmmeesssssss--see-- simulators
Tao Chen, Jie Xu, and Pulkit Agrawal. A system for general in-hand object re-orientation. Confer- ence on Robot Learning, 2021.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, 2024a.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, 2024b.
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2023. URL https:://arxi..... wwtp:biiwrrg/a////oo
1706.03747
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training, arXiv preprint arXiv:2501.171611 2025.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai
10

---

Published as a conference paper at ICLR 2026
Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. I.. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei. Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang. W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng. Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang. Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang. Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo. Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li. Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang. Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepscek-rl: Incentivizing reasoning capability in llms via reinforce- ment learning, 2025.
/ttp:i///a.iiv.aaaaa org/ abs 2501. 12948
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference machine learning. pp. 1329-1338. PMLR, 2016.
C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2106. 13281, 2021.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods, 2018.
palttps:/aaxxt
09477
Zhengyang Geng, Mingyang Deng, Xingjian Bai, Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling. arXiv preprint 2ttiv:25005/344777 2025.
Jonathan Ho and Tim Salimans. Classifier- diffusion guidance, 2022. URL WtRhhttss///axiv
org/abs 2207. 12598
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models, 2022a. URL
2210. 02303
Jonathan Ho. William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:22
2022b.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models, 2022c. URL
03458
Shengyi Huang, Quentin Gallouédec, Florian Felten, Antonin Raffin, Rousslan Fernand Julien Dossa, Yanxiao Zhao, Ryan Sullivan, Viktor Makoviychuk, Denys Makoviichuk, Mohamad H Danesh, et al. Open rl benchmark: Comprehensive tracked experiments for reinforcement learn- ing. arXiv preprint arXiv:2402.03044 2024.
Michael Janner, Yilun Du Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.0990) 2022.

---

Published as a conference paper at ICLR 2026
Sham M. Kakade. A natural policy gradient. In Proceedings of the 14th International Conference on Neural Information Processing Systems (NeurIPS), pp. 1531-15333 2002.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 35:26565 -65777, 2022.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6990, 2014.
Diederik Kingma and Ruiqi Gao, Understanding diffusion objectives as the elbo with simple data augmentation, 2023. WR..htttss//arxiv.oooo www.or////a////// 2303///
Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models, 2023. URL https:://axxi...rrrrr www.oorg///a./....ooooo....e..aa 000pp//207...0..
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis, 2021. URL httpt: httttps://axxi..orgg/oo 2009,.000... 09761
Yixuan Li, Yutang Lin, Jieming Cui, Tengyu Liu, Wei Liang, Yixin Zhu, and Siyuan Huang. Clone: Closed-loop whole-body humanoid teleoperation for long-horizon tasks. arXiv preprini 2:Xiv:2506.0893/ 2025.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, 2023. URL. http:////axxiooooaab/////////
02747
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture- of-experts language model. arXiv preprint 1.XXv::400.04434. 2024.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D. Plumbley. Audioldm: aext-to-audio generation with latent diffusion models, 2023, URL ht Prttps://arxti org/ abs 2201//11222000
Jie Liu. Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint arXiv::505.. 250.25005005400, 2025.
Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, et al. Perpetual humanoid control for real- time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10895-10904, 2023a.
Zhengyi Luo, Jinkun Cao, Josh Merel, Alexander Winkler, Jing Huang, Kris Kitani, and Weipeng Xu. Universal humanoid motion representations for physics-based control. arXiv preprint NXXi:23100005882. 2023b.
Zhengyi Jinkun
Jinkun Cao, Sammy Christen, Alexander Winkler, Kris Kitani, and Weipeng Xu. Om- nigrasp: Grasping diverse objects with simulated humanoids. In Advances in Neural Information Processing Systems, volume 37, pp. 2161-2184, 2024.
Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael Black. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pp. 5442-5451, 2019.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo. Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gau-based physics simulation for robot learning. arXiv preprint arXiv:2108. 10470, 2021.
Mayank Mittal, Nikita Rudin, Victor Klemm, Arthur Allshire, and Marco Hutter. Symmetry con- siderations for learning task symmetric robot policies. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 7433-7439, 2024. doi: 10.11099CRA577447.0004 1061 1493.
12

---

Published as a conference paper at ICLR 2026
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Tim Harley, Timothy Lillierap. David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning (ICML), pp- 1928-1937, 2016.
NVIDIA, :, Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi "Jim" Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang. Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llontop. Loic Magne, Ajay Mandlekar, Avnish Narayan, Soroush Nasiriany, Scott Reed, You Liang Tan, Guanzhi Wang. Zu Wang, Jing Wang, Qi Wang. Jiannan Xiang, Yuqi Xie, Yinzhen Xu, Zhenjia Xu, Seonghyeon Ye, Zhiding Yu, Ao Zhang, Hao Zhang, Yizhou Zhao, Ruijie Zheng, and Yuke Zhu. Gr0Ot nl: An open foundation model for generalist humanoid robots, 2025. URL
pictps://ariise org/ abs 2503. 14734
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal. Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances neural information processing systems, 2022.
Xue Bin Peng. Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG), 1714):1-14 2018.
Jan Peters and Stefan Schaal. Natural actor critic. Neurocomputing,
2008
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih Yao Ma, Ching- Yao Chuang, et al. Movic gen: A cast of media founda- tion models. arXiv preprint arXiv:2410.13720 2024.
Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning a diffusion model policy from rewards via q-score matching. arXiv preprint arXiv:2312.11752 2023.
Haozhi Qi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik. General in-hand object rotation with vision and touch. In Conference on Robot Learning, pp. 2564. PMLR, 2023.
Haozhi Qi, Brent Yi, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik. From simple to complex skills: The case of in-hand object reorientation. arXiv preprint 2rXiv:2501.05439, 2025.
Sanjeev Raja, Martin Šipka. Michael Psenka, Tobias Kreiman, Michal Pavelka, and Aditi S Kr- ishnapriyan. Action-minimizatio meets generative modeling: Efficient transition path sampling with the onsager-machlup functional. arXiv preprint arXiv:2504. 18506, 2025.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06 2022.
Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majum- dar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimiza- tion. arXiv preprint arXiv:2409.00SS 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2022. URL httpp::/arxiv..rrt
abs/2112. 10752
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pp. 91-100. 10L.OR..MMRR 2022. html
URL htttps://proceeeinnggg mml. .wressp..ssssee.e. pprees/vi66e http:/111//rrdnni
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.005//2 2022.
13

---

Published as a conference paper at ICLR 2026
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1899.1897. PMLR, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: arXiv:1506.02433, 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Clemens Schwarke, Victor Klemm, Matthijs
der Boon, Marko Bjelonic, and Marco Hut- ter. -uriosity-drivee learning of joint locomotion and manipulation tasks. In Proceedings of The 7th Conference Robot Learning, volume 229 of Proceedings of Machine Learning Re- search, pp. 2594-2610. PMLR, 2023. URL htt.htttss//proeeeeeee http://ppooceenngg...... wng.mlr.......................mm press/v229/ pr.pesss//.g.......////22/11 schwarke html
Younggyo Seo, Carmelo Sferrazza, Haoran Geng, Michal Nauman, Zhao-Heng, Yin, and Pieter Abbeel. Fastid3: Simple, fast, and capable reinforcement learning for humanoid control, 2025.
URL httL:htps:// /ettpp::/arxiv.orrt
aww.aa/.........e. 22642
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Ander- son. The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305.17499; 2023.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. Ai models collapse when trained on recursively generated data. Nature, 631(8022)7555-55, 2024.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Texieoo-video generation without text-video data, 2022. URL https: http//aaxxvvvrg//e//// 1209.14799
Marta Skreta, Lazar Atanackovic, Avishek Joey Bose, Alexander Tong, and Kirill Neklyudov. The superposition of diffusion models using the itô density estimator, 2025. URL https:
/ //arxivvorr/abb//222....1
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022. URL
02250
Yang Song and Stefano Ermon. Generative by estimating gradients of the data distribution, 2020. URL htttpss:/aariivvor///aaaaa/ http:///abbb/9..555000 05600
Richard Sutton, David McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. In Proceedings of the 12th Interna- lional Conference on Neural Information Processing Systems (NeurIPS), pp. 1057-1063, 1999.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint 1801. 00690, 2018.
Chen Tessler, Yunrong Guo, Ofir Nabati, Gal Chechik, and Xue Bin Peng. Maskedmimic: Uni- fied physics-based character control through masked motion inpainting. ACM Transactions on Graphics (TOG), 43(6):1-21, 2024.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEF/RSJ international conference on intelligent robots and systems, 2012.
14

---

Published as a conference paper at ICLR 2026
Mark Towers, Ariel Kwiatkowski, Jordan Terr, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17332, 2024.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts. 6:100022, 2020.
Veo-Team, :, Agrim Gupta, Ali Razavi, Andeep Toor, Ankush Gupta, Dumitru Erhan, Eleni Shaw, Eric Lau, Frank Belletti, Gabe Barth-Maron, Gregory Shaw, Hakan Erdogan, Hakim Sidahmed, Henna Nandwani, Hernan Moraldo, Hyunjik Kim, Irina Blok, Jeff Donahue, José Lezama, Kory Mathewson, Kurtis David, Matthieu Kim Lorrain, Mare van Zee, Medhini Narasimhan, Miaosen Wang, Mohammad Babaeizadeh, Nelly Papalampidi, Nick Pezzotti, Nilpa Jha, Parker Barnes, Pieter-Jan Kindermans, Rachel Hornung, Ruben Villegas, Ryan Poplin, Salah Zaiem, Sander Dieleman, Sayna Ebrahimi, Scott Wisdom, Serena Zhang, Shlomi Fruchter, Signe Nørly, Weizhe Hua, Xinchen Yan, Yuging Du, and Yutian Chen. Veo 2024. URL https: htttt::///eppiinnn /deppmind. googlee/teechooooggss///seeeee veeeveoooovveev veo- 2/
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando de Freitas. Sample efficient actor-critic with experience replay. Proceedings of the 30th International Conference on Neural Information Processing Systems (NeurIPS), pp- 1061-1071, 2016.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 1992.
Eric Yang Yu. Ppo-for-beginners: A simple, well-styled ppo implementation in pytorch. https: //g.i/aw...o//errrcccyayyy///P--
2020. GitHub repository.
Kevin Zakka, Baruch Tabanpour, Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire, Erik Frey, Koushil Sreenath, Lueder A Kahrs, et al. Mujoco playground. arXiv preprint arXiv:2502 088444 2025.
15

---

Published as a conference paper at ICLR 2026
A.1 APPENDIX
In this supplementary material, we discuss the deferred proofs of technical results, elaborate on the details of our experiments, and present additional visual results for the grid world, humanoid control, and present a texi-to-image diffusion model finetuning experiment.
A.2 FPO DERIVATION
The mathematical details presented in this section provide expanded derivations and additional con- text for the theoretical results outlimed in Section 3 of the main text. Specifically, elaborate on the connection between the conditional flow matching objective and the evidence lower bound (ELBO) first mentioned Section 3.4, and provide complete derivations for the FPO ratio introduced in Section 3.3. These details are included for completeness and to situate our work within the theo- retical framework established by Kingma et al. Kingma Gao (2023), but are not necessary for understanding the core FPO algorithm or implementing it in practice.
First, we detail the different popular loss weightings used when training flow matching models laid out by Kingma et al. Kingma & Gao (2023). These weightings, denoted w(t)), determine how losses at different noise levels contribute to the overall objective and lead to different theoretical interpretations of Flow Policy Optimization.
Then, we show the more general result, which is that FPO optimizes the advantage-weighter ex- pected ELBO of the noise-pent cesst-eeturbed data. Specifically, for any monotonic weighting function (in- cluding Optimal Transport CFM schedules Lipman et al. (023 ), we can express the weighted loss as:
C7 (ax) (P) ELBO, (aī) + C1.
(21)
where Pw (T) is the distribution over timesteps induced by the weighting function, and ELBO, is the evidence lower bound at noise level r for the perturbed action a..
This means that FPO increases the likelihood of high-reward samples and the intermediate noisy samples of from the sample path. By weighting this objective with advantages Å,, we guide the policy to direct probability flow toward action neighborhoods that produce higher reward.
For diffusion schedules with uniform weighting w(Az) 1, we show somewhat stronger theoret- ical result. In this special case, the weighted loss directly corresponds to maximizing the ELBO of clean actions:
dd dr [[oo(aT:>+) <112 c
EL)O(a:) = -0000000010,,00000000 [
(22)
which is a more direct connection to maximum likelihood estimation.
A.2.1 Loss WEIGHTING CHOICES
Most popular instantiations of flow-based and diffusion models can be reparameterized in the weighted loss scheme proposed by Kingma et al. Kingma & Gao (2023). This unified framework expresses each version as an instance weighted denoising loss:
(x) (x) - B, -o..
1
dd
((0(0T;Ar) - <l1)),
(23)
where w(\,) is time-dependent function that determines the relative importance of different noise levels.
For those with a loss weight that varies monotonically with noise timestep т, the aforementioned relationship between the weighted loss and expected ELBO holds. Specifically, when w(), is monotonically increasing with r, Kingma et al. prove:
S (at) (T. [ELBO, + c1.
(24)
where ej is constant, and does vary with model parameters.
16

---

Published as a conference paper at ICLR 2026
These monotonic weightings include several popular schedules: (1) standard diffusion with uniform weighting ((,,) = Ho et al. (2020), (2) optimal transport linear interpolation schedule Lipman ct al. (2023). J which yiclds wi A, e A/2, and (3) velocity prediction (v-prediction) with cosine
schedule Salimans & Ho 2022). which also yields w(>,) = e A/2.
A.2.2 FLOW MATCHING AS EXPECTED ELBO OPTIMIZATION
To derive FPO the more general flow matching case, we begin with the standard policy gradient objective, but replace direct likelihood maximization with maximization of the ELBO for noise- perturbed data:
max E (aoalo)))))
E (L, A,]
(25)
where t is temporal rollout time and is diffusion//low noise timestep.
This formulation directly leverages the result from Kingma al. Kingma & Gao (2023) that for monotonic weightings, the weighted denoising loss equals the negative expected ELBO of noise-
perturbed data plus constant:
Cp (a.) (p).())) (ar) [ELBO, (aj)) c₁.
(26)
To apply this within trust region approach similar PPO, need define ratio between the current and old policies. Since we are working with expected ELBOs, the appropriate ratio becomes:
(0) exp(Emm ELBOO,
(a, (o))lo)
(27)
exp(Ep (T).aaa (a.) [ELBO, (a7))
This ratio represents the relative likelihood of actions and their noisy versions under the current policy compared to the old policy.
It is important to note that the constant C1 in the ELBO equivalence depends only the noise schedule endpoints Amin and Amaz. the data distribution, and the forward process, but not on the model parameter O. This is critical for our derivation. ensures that within single trust region data collection and training episode, this constant remains identical between the old policy Oat and the updated policy 0. Consequently, when forming the ratio (0), these constants cancel out:
)PP (0) (S) ELBO, (l)) + c1).
(T) (o))lo
ELBO, (aF) )99, +c) exp ELBO, old)
(28)
estimate this ratio through Monte Carlo sampling of timesteps t and noise e:
(F E) = exp(-Le(T, e) + (e,,
(29)
where (o(T, c) = 2 ( (o;: X+) - cll is the reparameterized conditional flow matching loss for single draw of random variables and 7.
As discussed in the main text, ,FPO overestimates the scale but unbiasedly estimates the direction of the gradient. We can reduce eliminate the scale bias by drawing more samples of and e.
A.2.3 FPO WITH DIFFUSION SCHEDULES
For the special case of standard diffusion schedules with uniform weighting w(x:) 1, we derive a stronger theoretical result connecting our optimization objective directly to the ELBO of clean (non-noised) data.
17

---

Published as a conference paper at ICLR 2026
Gridworld with Goals
Sampled Trajectories
Gaussian Policy
noise 0.0
noise 0.1
noisc 0.5
Gool
start
end
Figure A.1: Grid with Gaussian Policy. Left) 25 x 25 GridWorld with green goal cells. Each arrow shows an action predicted by the Gaussian policy. Right) Four rollouts under test-time noise perturbations (a = 0.0, 0.1, 0.5). While the Gaussian policy achieves the goal, its trajectories lack diversity and hit the same goal consistently when given the same initialization point.
As shown by Kingma et al. Kingma Gao 2023 when using uniform weighting, the weighted loss directly corresponds to the negative ELBO of the clean data plus a constant:
dd
EL)O(a,)) = ( -
[|āa(a]] A,) <112 +
(30)
dr
where 0g is different constant than C1 that also does not depend model parameter 0.
This means that minimizing the unweighted loss (wx(A,) = 1) is equivalent to maximizing the ELBO of the clean action ar, providing more direct connection traditional maximum likelihood estimation.
In the context of FPO, we therefore express our advantage-weighted objective
max max (a
(L)O,(at) A.]
(31)
In this case, the objective direct increases lower bound of the log- likeeikei of clean actions ay weighted by their advantages, rather than over woise-perturbed actions.
The FPO ratio in this case becomes:
exp(ELBO> exp exp(ELBOo,adl) (L))
PPO (0)
(32)
This specific case highlights the close relationship between FPO and traditional maximum likelihood methods common for PPO Schhlmmnn et al. (2017). FPO still retains the computational advantages of avoiding explicit likelihood computations.
As in the general case, our Monte Carlo estimator exhibits upward bias of gradient scale. We can use the same PPO clipping mechanism to control the magnitude of parameter changes.
A.2.4 ADVANTAGE-WEIGHIN FLOW MATCHING DISCUSSION
Advantage estimates are typically certecenteeed reduce variance estimating the policy gradient. Flow matching, however, learns probability flows which must be nonnegative by construction. Since advantages function as loss weights this context, they should remain positive for mathematical consistency. A constant shift does not affect policy gradient optimization, which follows from the same aeseiineinvarianee property that justifies using advantages in the first place. We find that both processed and unprocessed advantages work empirically.
A.3 GRIDWORLD
Figure shows
Isoos results from the Gaussian policy on the same Grid World trained using PPO. While the Gaussian policy can learn optimal behaviors, the trajectories resulting from it are not as diverse
18

---

Published as a conference paper at ICLR 2026
Learning Rate
Clipping Epsilon (s
0,3
0.2
0.1
0.05
0.03
0.01
0.0001
589.5
648.5
646.6
608.6
500.5
458.5
0.001
556.0
646.1
654.6
636.2
562.6
471.8
0.003
548.9
603.1
586.4
535.7
480.8
400.8
0.0003
567.0
631.8
667.8
650.9
570.4
492.0
0.0005
544.8
586.8
629.5
559.7
505.6
406.5
Table A.I: Hyperparameter sweep for Gaussian PPO on the subset of Playground tasks that evaluate on. All quantities are average rewards across 10 tasks, with 5 seeds per task. The default configuration in Playground Zakka et al. (2025) (before tuning) uses learning rate Ic-3 and clipping epsilon 0.3: the tuned variant we for results the main paper body sets learning rate to 3e-4 and clipping epsilon to 0.1.
BallinCup
CartpoleBalance
CheetahRun
FingerSpin
FingerTurnEay
1000
600
600
Eval Eval Reward
450
800
750
450
500
300
300
250
150
150
HinderTumHHar
FishSwim
PointMass
ReacherEasy
ReacherHard
Eval Reward
600
800
800
600
600
600
300
400
400
400
200
150
200
Gaussian PPO (tuned)
Gaussian PPO (before tuning)
Figure A.2: Gaussian PPO baseline results before and after tuning. We tune clipping epsilon and learning rate to maximize average performance across tasks. Results show evaluation reward mean and standard error (y-axis) over 60M environment steps (x-axis). We run seeds for each task; the curve with the highest terminal evaluation reward is bolded.
as those of the diffusion policy. We visualize 4 samples from the Gaussian policy with 0.0, 0.1, and 0.5 random noise perturbations at test time (Fig. A. (F. right). Note that despite being initialized at the midpoint of the environment, all shown positions lead single goal mode, never both.
A.4 MuJoCo PLAYGROUND
Table A.2 shows hyperparameters used for PPO training in the MuJoCo Playground environment. These are imported directly from the configurations provided by MuloCo Playground Zakka et al. 2025), but after sweeping hyperparameters to tune learning rate and clipping coefficients (Ta- ble A.I We visualize improvements from this sweep in Figure A.2 Our flow matching and diffusion- policies use the same hyperparameters, but adjust the clipping coefficient, turn off the entropy coefficient, and for DPPO Ren et al. (2024), introduce a stochastic sampling variance to account for the change in policy representation.
A.5 HUMANOID CONTROL.
In Table A.3 we report the detailed hyperparameters that we used for training both the Gaussian policy with PPO and the Diffusion policy with FPO in the humanoid control experiment. Note
19

---

Published as a conference paper at ICLR 2026
Hyperparameter
Value
Discount factor (y)
0.995 (most environments)
0.95
(BallInCup. FingerSpin)
GAE
0.95
Value loss coefficient
0.25
Entropy coefficient
0.01
Reward scaling
10.0
Normalize advantage
True
Normalize observations
True
Action repeat
1
Unroll length
30
Batch size
1024
Number of minibatches
32
Number of updates per batch
16
Number of environments
2048
Number of evaluations
10
Number of timesteps
60M
Policy network
MLP (4 hidden layers, 32 units)
Value network
MLP (5 hidden layers, 256 units)
Optimizer
Adam
Table A.2: PPO hyperparameters imported from MuJoCo playground Zakka al. (2025).
that we use the same of hyperparameters for both policies. In our project webpage, we also provide videos showing qualitative comparisons between the Gaussian policy and ours on tracking an under-conditioned reference, and visual results of FPO on different terrains.
Hyperparameter
Value
Hyperparameter
Value
Policy Settings
Hidden size Action perturbation std
512
Solver step size
0.1
0.05
Target KL divergence
None
Number of environments
4096
Normalize advantage
True
Training Settings
Batch size
131072
Minibatch size
32768
Learning rate
0.0001
LR annealing
False
LR decay rate
1.5e-4
LR decay floor
0.2
Update epochs
4
L2 regularization coef.
0.0
GAE lambda
0.2
Discount factor (?)
0.98
Clipping coefficient
0.01
Value function coefficient
1.2
Clip value loss
True
Value loss clip coefficient
0.2
Max gradient norm
10.0
Entropy coefficient
0.0
Discriminator coefficient
5.0
Bound coefficient
10.0
Table A.3: Policy training hyperparameters for humanoid control.
A.6 IMAGE REWARD FINE-TUNING
We explore fine-tuning a pre-trained image diffusion model on non-differentiable task using the JPEG image compression gym proposed in DDPO (Black et al. 2023). We report this experiment as negative result for FPO, due the difficulty of fine-tuning diffusion models their own output. Specifically, we find that repeatedly generating samples from a text-ta text-to-image tontat- diffusion model and training on them is highly unstable, even with manually-specified uniform advantages. We believe
20

---

Published as a conference paper at ICLR 2026
Figure A.3: Image Generation at Different Training Steps. We generate images using Stable Diffusion 1.5 finetuned with FPO as training progresses. We manually set advantages to I to eliminate the reward signal and investigate the dynamics of sampling from text-to-image diffusion model then training on the results in a loop. In the top row, we display images from training run using classifier free guidance (CFG) scale of 4. In the bottom row, display images from training run using a CFG scale of 2. Low CFG scales tend to encourage bluriness while high CFG scales encourage saturation and sharp geometric artifacts. Both diverge after few hundred epochs even with tuned hyperparameters.
that this is related to classifier-free guidance (CFG) Ho Salimans (2022). CFG is necessary to generate realistic images, however it is sensitive to hyperparameters, where too much or too little guidance introduces artifacts such as blur or oversaturation that do not reflect the original training data. Sometimes these artifacts not visible to human eyes. These artifacts further amplified over successive iterations of RL epochs, ultimately dominating the training signal.
This phenomenon aligns with challenges previously identified in the literature fine-tuning gener- ative models on their own outputs (Shumailov et al., 2024, 2023) Alemohammad et al., 2024). To illustrate this, fine-tune Stable Diffusion with all advantages set to to eliminate the reward sig- nal. This is equivalent to fine-tuning on self-generation data in an online manner. We explore CFG scales of and 4 in Figure A.3 We find that both CFG scales induce regression in quality over time. Specifically, the CFG scale of 2 makes the generation more blurry, while the scale of 2 causes the generated images to feature high saturation and geometry patterns. Both eventually diverge to abstract geometric patterns.
A.7 LARGE LANGUAGE MODEL USE
We used Large Language Models (LLMs) to aid in polishing the writing of this paper. We also used L.LM-based web agents help discover relevant related works. No LLMs were used to generate original scientific content.
21


Reviews:
[{"review_id": "jEPqoaFXU3", "reviewer": "Reviewer_HevE", "summary": "They propose Flow Policy Optimization (FPO): swap PPO’s likelihood ratio with an ELBO-ratio computed from conditional flow matching (CFM) losses. This lets you train flow/diffusion policies in a PPO-style loop without evaluating exact log-likelihoods. Empirically, they beat Gaussian-PPO and a diffusion-PPO baseline on most MuJoCo Playground tasks, and show robustness on a humanoid control benchmark.", "questions": "Questions:\n- Is there any ablations for choosing different weightings?\n- while you mentioned the method is agnostic to sampler choice, do you observe empirically any difference between SDE/ODE samplers? Does it affect the ratio variance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Pros:\n- Ratio-as-difference-of-CFM-losses is simple to implement and keeps GAE/GRPO compatibility\n-Clear ablations: effect of #MC samples, ω- vs u-parameterization, clipping sensitivity. Shows robustness under sparse goal conditioning in humanoid.", "weaknesses": "Cons:\n- ELBO is not exact likelihood. The ratio decomposes into true likelihood ratio times an inverse KL-gap factor. That second term is policy-dependent and unknown, so the proxy ratio is biased w.r.t. the true PPO ratio", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qi6UcjUDX0", "reviewer": "Reviewer_bNty", "summary": "This paper introduces Flow Policy Optimization (FPO), an on-policy reinforcement learning algorithm that enables training of flow-based generative models (including diffusion models) as policies within the policy gradient framework. The key innovation is replacing the intractable likelihood ratio in PPO with a surrogate ratio computed from conditional flow matching (CFM) losses.", "questions": "1, What is the main difference of your method compared to other reward-weighted methods, empirically speaking?\n\n2, Can the proposed method generalize to scopes other than robotics? Or what could be the domain-specific point?\n\nI am willing to raise my score if all my concerns are well solved.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1, The motivation is clear and significant. Training flow matching models directly from rewards can greatly popularize their usage to robotics.\n\n2, The evaluation is comprehensive to show the effectiveness of the proposed method on simple robotic tasks (with simulation).", "weaknesses": "1, The baseline is limited. There are existing methods which use direct rewards to weight the trajectory and are agnostic to sampling methods, although most of them are applied to text-to-image generation and other generation tasks, for example, [A]. The author should also implement some of these methods on robotics tasks and conduct simple evaluation, or at least include them as related works and describe the difference.\n\n[A] Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1Chg5TSRD5", "reviewer": "Reviewer_awZj", "summary": "The paper introduces flow matching policy gradients, a method for optimizing a policy using flow matching.\nThe benefit of this framework is in enabling a more general class of generative policies that do not require a pre-defined policy class (e.g., unimodal gaussian/beta distribution).", "questions": "Does FPO also work in a sparser setting?\nFor example in Tessler 2024 they go beyond root/hands conditioning and also show cases where the constraints are multiple frames into the future. This is a very underspecified and hard problem to solve and would be very impressive if FPO can still tackle it.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The benefit of this framework is in enabling a more general class of generative policies that do not require a pre-defined policy class (e.g., unimodal gaussian/beta distribution).\n\nI see the main benefit of this work shown in the humanoid control section.\nWe know that large NNs generally converge to global optima (\"there's always some decent direction so long as we have some random noise in the system\"). It isn't clear whether continuous control exhibits similar properties or not. \n\nThe mujoco playground experiments are low degrees of freedom and might indeed suffer from local optima, however experiments on the humanoid shows PPO outperform FPO -- this suggests that maybe in the case of very large action spaces we do not suffer from sub-optimal local minima.\n\nThe interesting result is then the ability of FPO to converge to multi-modal solutions when needed. This is shown in the under-specified humanoid problem where the policy receives rewards for the full-body pose but is only conditioned on a subset of these constraints.", "weaknesses": "I believe the work should focus more on the generative aspects of the method, as in the humanoid control effort, and less on toy problems.\nThe mujoco playground results are \"nice to have\" they show the method generally works. But the main strength of generative methods is in their ability to model a distribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6ydfuQnqit", "reviewer": "Reviewer_XkTY", "summary": "This paper introduces Flow Policy Optimization (FPO), a novel on-policy reinforcement learning algorithm for training flow-based generative models, particularly diffusion models, as policies. The core innovation is reformulating the policy gradient objective by replacing exact likelihood computations with a ratio derived from the conditional flow matching (CFM) loss. Specifically, FPO uses  a surrogate loss (See Sec. 3.3, 3.4) for the standard PPO likelihood ratio, enabling integration into the PPO-clip framework. The authors demonstrate that this ratio corresponds to optimizing an advantage-weighted evidence lower bound (ELBO), making FPO theoretically grounded while computationally tractable. Importantly, FPO is agnostic to the choice of sampling method during both training and inference, unlike prior denoising MDP approaches. Experiments across GridWorld, 10 MuJoCo Playground tasks, and high-dimensional humanoid control show that FPO successfully trains flow-based policies from scratch. The method demonstrates particular advantages in under-conditioned settings where multimodal action distributions are beneficial, outperforming Gaussian policies. The paper provides both theoretical analysis connecting the CFM objective to ELBO maximization and empirical validation showing FPO achieves competitive or superior performance compared to standard PPO and DPPO baselines.", "questions": "1. The paper claims sampling method agnosticism, but experiments only use 10-step Euler integration. Have you validated with DDIM, DPM-Solver, or higher-order methods? Does performance change significantly?\n\n2. In Eq. 12, you decompose $r_{FPO}$ into likelihood ratio × inverse KL gap. Can you provide empirical measurements of how each term evolves during training? Does the KL gap actually decrease?\n\n3. Why is the entropy coefficient set to 0 for FPO (Table A.2) but 0.01 for Gaussian PPO? Doesn't this disadvantage exploration for FPO?\n\n4. For the humanoid under-conditioned experiments, can you provide quantitative measures of policy multimodality (e.g., entropy, number of effective modes) rather than just success rates?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper presents a genuinely novel approach to combining flow matching with policy gradients. The key insight—using the CFM loss differential as a surrogate for log-likelihood ratios—is elegant and theoretically motivated. Unlike prior work (DDPO, DPPO) that treats the denoising process as an MDP, FPO directly integrates flow matching into the policy gradient framework, avoiding the artificial expansion of the horizon and observation space. The connection to ELBO optimization through existing framework is well-established, and the observation that gradient estimates remain unbiased despite upward bias in the ratio (Equation 18-20) is insightful.\n\n\nMoreover, the paper is well-written with clear motivation. Algorithm 1 provides a concise implementation overview, and the progression from standard PPO to FPO is logical. The GridWorld visualization (Figure 1) effectively demonstrates the multimodal behavior learned by FPO, showing the learned bimodal distribution at the saddle point. The humanoid control results clearly illustrate FPO's advantage in under-conditioned scenarios.", "weaknesses": "1. **Bias analysis incomplete**: While the paper shows gradient estimates are unbiased (Eq. 20), the impact of ratio overestimation on actual policy updates is not thoroughly analyzed. How does this bias interact with PPO clipping in practice? The claim that \"clipping mechanism controls magnitude\" needs more rigorous justification—does the clip threshold need adjustment to account for systematic overestimation?\n\n2. **Missing analysis of multimodality**: While GridWorld demonstrates multimodal learning, there's no quantitative analysis. How does the learned distribution compare to the true optimal distribution at saddle points? For humanoid control, are the policies actually multimodal or just higher variance? Entropy measurements or explicit distribution visualization would strengthen these claims.\n\n3. **Reproducibility concerns**: While code release is promised, many implementation details are missing. What network architecture is used for flow models? How is the timestep encoded?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)