PolicyFlow: Policy Optimization with Continuous Normalizing Flow in Reinforcement Learning
Authors: Shunpeng Yang
Ben Liu
Hua Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Continuous Normalizing Flow
Entropy Regularization
Proximal Policy Optimization
Multimodal Policy
Primary area: reinforcement learning
TL;DR: We present PolicyFlow, an on-policy reinforcement learning algorithm that unites continuous normalizing flows with PPO-style optimization and a novel Brownian entropy regularizer for expressive and stable multimodal policies.

Abstract: Among on-policy reinforcement learning algorithms, Proximal Policy Optimization (PPO) demonstrates is widely favored for its simplicity, numerical stability, and strong empirical performance. Standard PPO relies on surrogate objectives defined via importance ratios, which require evaluating policy likelihood that is typically straightforward when the policy is modeled as a Gaussian distribution. However, extending PPO to more expressive, high-capacity policy models such as continuous normalizing flows (CNFs), also known as flow-matching models, is challenging because likelihood evaluation along the full flow trajectory is computationally expensive and often numerically unstable. To resolve this issue, we propose PolicyFlow, a novel on-policy CNF-based reinforcement learning algorithm that integrates expressive CNF policies with PPO-style objectives without requiring likelihood evaluation along the full flow path. PolicyFlow approximates importance ratios using velocity field variations along a simple interpolation path, reducing computational overhead without compromising training stability. To further prevent mode collapse and further encourage diverse behaviors, we propose the Brownian Regularizer, an implicit policy entropy regularizer inspired by Brownian motion, which is conceptually elegant and computationally lightweight. Experiments on diverse tasks across vairous environments including MultiGoal, PointMaze, IsaacLab and MuJoCo Playground show that PolicyFlow achieves competitive or superior performance compared to PPO using Gaussian policies and flow-based baselines including FPO and DPPO. Notably, results on MultiGoal highlight PolicyFlow’s ability to capture richer multimodal action distributions.

Published as a conference paper at ICLR 2026
POLICYFLOW: POLICY OPTIMIZATION WITH CON- TINUOUS NORMALIZING FLOW IN REINFORCEMENT LEARNING
Shunpeng Yang'.4, Ben Liu" & Hua Chen
'Hon Kong University of Science and Technology. SSutherrn University of Science and Technology of Illinois Airana-Champaln Institute, "LimX Dynamics f Corresponding author, huachen@int1.. 2.....j..... edu.cn
*Zeejiang
ABSTRACT
Among on-policy reinforcement learning algorithms, Proximal Policy Optimiza- tion (PPO) demonstrates is widely favored for its simplicity, numerical stability, and strong empirical performance. Standard PPO relies on surrogate objectives defined via importance ratios, which require evaluating policy likelihood that is typically straightforward when the policy is modeled as a Gaussian distribution. However, extending PPO more expressive, high-capacity policy models such as continuous normalizing flows (CNFs), also known as flow- matchingg models, is challenging because likelihood evaluation along the full flow trajectory is com- putationally expensive and often numerically unstable. To resolve this issue, we propose PolicyFlow, novel on-policy CNF-based reinforcement learning algo- rithm that integrates expressive CNF policies with PPO-style objectives without requiring likelihood evaluation along the full flow path. PolicyFlow approximates importance ratios using velocity field variations along simple interpolation path, reducing computational overhead without compromising training stability. To further prevent mode collapse and further encourage diverse behaviors, pro- pose the Brownian Regularizer, an implicit policy entropy regularizer inspired by Brownian motion, which is conceptually elegant and computationally lightweight. Experiments on diverse tasks across various environments including MultiGoal, PointMaze, IsaacLab and MuJoCo Playground show that PolicyFlow achieves competitive or superior performance compared to PPO using Gaussian policies and bbsed baselines including FPO and DPPO. Notably, results MultiGial highlight PolicyFlow's 's ability to capture richer multimodal action distributions.
INTRODUCTION
Reinforcement learning (RL), particularly policy gradient (PG) methods, has achieved remarkable success in complex sequential decision-making tasks, ranging from robotic control (Andrychowicz et al., 2020; Rudin et al., 2022; Cheng et al., 2024; He et al., 2025) to aligning large language models with human preferences (Ouyang et al., 2022: Zhai al., 2024; Guo et al., 2025). Among PG methods, Proximal Policy Optimization (PPO) (Schulman et al., 2017) remains standard due to its simplicity and generally reliable performance, widely used in complex robotic control tasks (Lee et al., 2020; Chen et al., 2025), and recently for fine-tume fine-tuning generative policies (Black et al., 2023; Ren al., 2024). PPO optimizes policies via surrogate objectives based on importance ratios, which require nontrivial likelihood evaluation. For tractable computation, policies are typically modeled by Gaussian distribution. While convenient, Gaussian policies are limited representing complex, multimodal, or highly skewed action distributions, motivating the use of more expressive generative models.
In recent years, generative models, such as diffusion models (Ho et al., 2020; Song al., 2020) and continuous normalizing flows (Lipman et al., 2022; Tong et al., 2023), have emerged as a pow- erful class of models capable of capturing complex, multimodal distributions. These models have been successfully applied to imitation learning, where they model policy distributions directly from demonstration data (Chi et al., 2023; Ze al., 2024), effectively capturing trajectory diversity and

---

Published as a conference paper at ICLR 2026
complex behaviors. However, computing importance ratios likelihoods for these models typically requires iterative ODE/SDE simulations and path-wise backpropagation (Chen et al., 2018), which is computationally expensive and prone to exploding vanishing gradients. This makes direct ap- plication of such models in PPO-style updates slow, memory-intensive, and potentially unstable, limiting their practicality for efficient on-policy reinforcement learning.
Motivated by these challenges, we propose PolicyFlow, a novel on-policy RL algorithm that com- bines the expressiveness of continuous normalizing flows with PPO-style clipped objective, en- abling efficient and stable policy optimization. Our contributions are as follows:
Importance ratio approximation for CNF policies. PolicyFlow approximates importance ratios by evaluating the variations of CNF's velocity field along interpolation paths, avoiding the costly path-wise backpropagation.
Brownian eniropy regularization. We propose lightweight entropy regularizer inspired by Brow- nian motion (Einstein, 1905), which promotes monotonic entropy growth, mitigates mode col- lapse, and encourages diverse actions without explicitly computing the CNF policy's entropy.
These results demonstrate PolicyFlow's potential as a practical and expressive framework for on- policy RL. Code and project page are available at httt::::/poolccylloo2222222 hit.ii...iiitubi.iii...
2 RELATED WORK
2.1 FLOW/DIFFUSION- BASED REPRESENTATIONS OF RL POLICIES
Flow and diffusion models provide highly expressive, multi-modal distributions, making them at- tractive as policy parameterizations in reinforcement learning. Compared conventional categorical or Gaussian policies, these generative models allow richer action distributions and can potentially capture broader set of behaviors. In roboties, flow-based and diffusion-based models have been widely adopted as policy representations (Chi et al., 2023; Lei et al., 2025; Intelligence et al., 2025; Gao et al., 2025). However, progress has approached a bottleneck, as these models are typically trained solely through denoising score matching or flow matching on offline datasets, without incor- porating reinforcement learning. This limitation has motivated researchers to explore using RL to directly train generative policies.
In offline RI, diffusion-based policies have been widely adopted to model complex action pat- terns from static datasets, often guided by value functions or energy-based objectives (Wang et al., 2022: Lu et al., 2023; Psenka et al., 2023; Zhang et al., 2025). These approaches have achieved strong results D4RL benchmarks and inspired actor-critic variants that couple generative poli- cies with value estimation (Wang et al., 2024; Fang et al., 2024). To address the heavy sampling and computational demands of diffusion models, recent works have also explored more efficient formulations (Kang et al., 2023).
In online RL, the setting is more demanding because it requires efficient sampling, stable importance-ratio estimation, and tractable (or well-approximated likelihoods. Several works (Wang al., 2024; Chao et al., 2024; Ding et al., 2025) directly backpropagate policy gradients through the full diffusion/flow chain, which enables end-to-end optimization but risks exploding or vanishing gradients. Practical recipes for fine-tuning expressive diffusion policies with policy-gradient-svy updates have also been proposed in DPPO (Ren et al., 2024), which enables structured on- manifold exploration and stable long-horizar training. While effective in ane-tuning scenarios, its perfor- mance tends to degrade when training from scratch, as off -manifoldd exploration becomes necessary. FPO (McAllister et al., 2025) instead estimates policy importance ratios through an ELBO objective, which offers scalable approximation but introduces asymmetric estimation Dias-more reliable when the importance ratio increases than when decreases ppoenniallll amplifying variance and affecting stability. To mitigate this issue, FPO typically requires larger batch sizes during updates.
Overall, prior work illustrates both the promise and the limitations of expressive generative policies in online RL: while they expand the representable policy class, existing approaches may suffer from unstable optimization, high computational cost, or biased approximations. PolicyFlow is an on-policy algorithm that seeks to address these challenges without backpropagating through full
2

---

Published as a conference paper at ICLR 2026
generative chains, without treating diffusion internal Markov Decision Process in DPPO, and while avoiding the asymmetric bias in FPO.
2.2 POLICY ENTROPY REGULARIZATION
Entropy regularization has long been used to encourage exploration and prevent mode collapse in inforcement learning. Classical approaches show its effectiveness for categorical policies in discrete action spaces (Mnih et al., 2016) and Gaussian policies continuous control (Haarnoja al., 20. Extending this principle to flow-based policies, however, is challenging: action logliikels are generally intractable, making entropy estimation expensive.
In principle, closed-form dynamics of entropy under continuous normalizing flows can be derived via the divergence of the velocity field integrated along the flow path (Chen et al., 2018; Tian et al., 2024). While thcoretically sound, this requires costly divergence evaluation and path integration, which limits scalability. More heuristic solutions have also been explored: Wang et al. (2024) approximate entropy using Gaussian mixture models, adjusting the injected noise to diffusion output accordingly, while Ding et al. (2024) inject uniform noise into training samples to artificially inflate entropy. These strategies
effective specific cases, and can be good choices when additional computational cost is not a concern or when the range of action samples is known in advance.
Our approach introduces implicit entropy regularizer, inspired by Brownian motion, that directly shapes the velocity field toward entropy-increasing dynamics. This design avoids expensive log- likelihood computation and bypasses the need for hoc noise heuristics. Since entropy regulariza- tion was not explicitly addressed in methods such as FPO, our regularizer provides principled and lightweight alternative flow-based policy optimization.
BACKGROUND
We consider a Markov Decision Process (MDP) defined by the tuple (S, Asp,,,, 7), where the state space, A is the action space, p(s' s, a) denotes the transition dynamics with state S and action a E A, r(s, a) is the reward function, and y E [0,1] is the discount factor. The agent's objective is to learn policy (als) that maximizes the expected cumulative discounted return:
((T) (x(r(m)
Σ y r ax)
where r = (So, ag, S1, ......) denotes a trajectory sampled from the environment under policy n.
Among policy gradient algorithms, PPO has become of the most widely adopted due to its simplicity and empirical stability. PPO optimizes the policy by maximizing a clipped surrogate objective (Schulman et al., 2017):
(T(a))) Aa (s, a), clip ffaas)
T (als) aTals) - e. c. As AA(s,a)) (2)
(PP = Epa (x) min
where i(aas) a reference policy, (* is policy's state distribution (Schulman et al., 2015), (s a) is the corresponding advantage function, and the clipping range is small positive hy- perparameter (typically in the range [0.1, 0.3) that controls the maximum allowable deviation of the likelihood ratio from one. This formulation prevents the updated policy from deviating too far from the reference policy, thereby ensuring more stable learning in practice.
More recently, Frans al. (2025) showed that PPO and related algorithms can also be interpreted under a proxy objective of the form:
j(T) = Ep A a).
As long the divergence between the resulting policy n" arg max, (Tr) and the reference policy remains bounded optimizing this proxy objective guarantees monotonic improvement of the true objective, i.e., J(x") > J
3

---

Published as a conference paper at ICLR 2026
4 POLICY OPTIMIZATION WITH CONTINUOUS NORMALIZING FLOW
To overcome the limitations of Gaussian parameterizations, we propose to represent policies using continuous normalizing flows. Specifically, we define conditional flow yp : [0. R x R - R governed by the ordinary differential equation (ODE):
dt (I(zs) 24 (P)(())) Po(2;5)
where z E Rd is a latent variable, s E R" is the state and v [0, 1] x R x time-dependent velocity field which can be parameterized by neural network.
R" +
Rd
is
Similar to Wang et al. (2024), the policy generates actions by integrating the flow to its terminal time and adding Gaussian noise:
a = (2; + +,, ~ Pz()), n~~ n~NN(n;,,'))
(5)
Here, the injected noise n not only facilitates exploration but also ensures compatibility with the PPO-style surrogate objective, allowing us to naturally extend PPO's original formulation to con- tinuous normalizing flows. While various choices are possible for pz(z) in principle, we fol- low common practice in generative model and choose standard Gaussian distribution, namely (z(z) = (z; 0, 1).
This construction induces policy distribution
<(als) = π (alz, s)p==z) dz, =(alz,s) = (aaa; (Z;
(6)
This representation is strictly more expressive than a conventional Gaussian policy, as it can recover simple unimodal Gaussian distributions while also modeling arbitrarily complex, multimodal, or highly non-Gaussian action distributions.
Under this parameterization, the policy proxy objective in Eq. (3) can be rewritten as TT(aa z, s) #(a A+ As (s)saa)
j(z) = Reg. a) = (x)
(7)
which explicitly connects the flow-based policy representation with the standard objective used in
policy optimization.
In principle, the importance ratio be computed by simulating both flows and & through their
RL Advantage Guidance
ODEs during training. However, this is often com- putationally expensive and numerically unstable, as neural ODEs may suffer from exploding/vanishing gradients or high memory usage during training. Next, we describe an alternative objective that avoids directly simulating the ODEs to compute this impor- tance ratio during training.
Reference Velocity Og (X+;
New Velocity Uy (xx; s)
o 22;
X
Brownian Regularizer
Now let (: μ. ²²) denote the Gaussian density function with mean l and variance σ². A key observation is that the likelihood ratio between Gaussian distributions shift-invariant, which means
x <alz, s) Pn (a; 4p (2; a²) * *a),,s) s) Pn (a; @ (2; 62) = Pu (a Φ (2; & (2;s),
(8)
Pa (a (1 0, 62)
where 6,p (2; pi (z; s) $ (2) Directly computing dep (z; s) requires simulating the ODEs, which is computationally costly. To alleviate this, we approximate the terminal shift using the velocity variation along the following linear interpolation path:
X, (1 l)z + 4:i s),, LE [0, 1).
This approximation replaces the integral over the reference trajectory with an expectation over t along the interpolation path, yielding:
[ (a|z, s) मे aa 2 s)
Pr (a (z; (x;ss) o²)
Ep(()
(10)
Pa (a - S (2; 0²) where p(t) U(, 1] and the velocity field variation d., (xx;s) u4 (xx;8) (x(:))))
4

---

Published as conference paper at ICLR 2026
Remark (Approximation Error Bound) Theoretical analysis (see Appendix A for details) shows that this interpolation--as based approximation introduces only a first-orde error in the log under small update regimes, which can be naturally enforced by the clipping range in PPO.
Q (2: 8, [2) s);
Pn a 4 (2; 4.p (z:s). 0²)
Ep(t)
O()). (11)
Po.
Pn. (a 61 (2; 0²)
Importantly, this approach allows us to avoid simulating the full flow trajectory and propagating gradients along it during training, thereby maintaining compulational efficiency comparable to PPO with Gaussian policy.
Now, the velocity field v is parameterized by neural network with parameters 0. Finally, similar to PPO, we adopt a clipped surrogate-style objective to stabilize training:
FFlmn (0, o Epe(s).px (a)((())) p(t) [mnin (pA*(s, a), clip (p, 1 e, +e) As (12)
with approximate importance ratio
Pn (a 40 (z; s); (x s, E (x o²)
(13)
Pr. (z; 0, 0²
Thus, simulation of the ODE is only required during sampling (to compute Φ1 s)), while the train- ing objective can be efficiently estimated along the interpolation path with velocity filed variations, without simulating the ODE or backpropagating through the simulated flow trajectories in training.
4.1 POLICY ENTROPY REGULARIZATION
Policy entropy maximization is stan-itanding technique to encourage exploration and mitigate mode collapse in reinforcement learning. To tackle the difficulties of entropy regularization for flow- based policies, as discussed in Sec. 2.2, we propose novel entropy regularizer inspired by Brownian motion. Our method differs from entropy regularization: instead of explicitly computing policy entropy heuristically injecting noise, we directly regulate the velocity field to follow an entropy- increasing process. This perspective allows us to avoid costly leeeiilldood evaluation while still encouraging diverse exploration, shown as Fig. 1.
+
2
2
15
0
0
0
10
-2-
-2
0
2
-2
2
-2
2
(a)
(b)
(c)
(d)
Exploration Density Maps. (a) En- vironment overview: the agent is initialized at the green point for each episode, and the four red points indicate goal locations with equal rewards. (b) Exploration heatmap of PPO, showing limited coverage due to the simple Gaussian policy. (c) Exploration heatmap of PolicyFlow without the Brownian regularizer, which improves coverage but still leaves some regions under-explored. (d) Exploration heatmap of PolicyFlow with the Brownian regularizer, achieving near-complete cover- age of all feasible locations.
Figure 1: (PointMaze Mediim Mediim-DDverseeee
In Brownian dynamics, particles naturally spread toward a uniform distribution, and entropy mono- tonically increases during the process. Although Brownian motion is defined by a stochastic differ- ential equation, its probability path follows the classic heat equation (x) (x):(x) (Jor- dan et al., 1998). This equation connects directly to the continuity equation (x: (x))xx((x)) By choosing (x) - Vx log pt (x), the continuity equation recovers the heat equa- tion, showing that entropy growth can enforced via a carefully shaped velocity field aligned with the negative score.
Vx
In practice, to promote flow trajectories that expand like Brownian motion rather than collapse, we want learned velocity field to follow the negative score of reference flow. To obtain this score
5

---

Published as a conference paper at ICLR 2026
Algorithm PolicyFlow
Input: initial velocity field parameters 80, initial noise variance o initial value function pa-
rameters do
2: for iteration i 0, 1.2...... do
3: Set reference parameters <- 8, 0² c o.. The reference velocity field is i = V/
4: Collect a set of trajectories D. using the reference policy T8,
5: for each MDP step with state Sş do
6:
Sample latent variable Zą ~ (z) then Po Zk
7:
Compute şp: = Φ (Ze3 Sx) by simulating the ODE 4 6: = (r(Pri from = to at
Sample noise n; ~ (N o²) and form action aę Execute ag. observe next state 5k++1 and reward rı: Store transition (Sk, ag, Pk, S 2k: (pk) D; 11: end for
8:
4p + ny
9:
10:
12: For each step k, compute rewards-to-go R& and advantage estimates As using GAE for epoch l,...,, E do
for each mini-batch of transitions (5k ag. Ak. Zu. (Pk) from D; do
15:
Sample 4 ~ U|0,
Or sample 4 from the discrete time points used for numerical simulation of flow ODE Compute interpolation point Xt, (1 t 12 t24R
17:
18:
Compute approximate importance ratio
Ph Pn. (ax PR; Vt,, (xt,, 82, 0) - , (Xt, : 8& o //)//n /pn (ax: spk; 0, o²)
Compute the clipped surrogate objective for the mini-batch by
Flow Ex min(p:AA clip( - c)Ax)
20:
Compute Brownian regularizer vector
ne (Xt,; Sk. 0) = (1 t& )r, (xx, ; Sk. 0) - (x+, - tx Eia (xcr;s))
21:
Compute the regularization term for the mini-batch by
d
Reg
Es - wollITe (x 0)13 +
Wy 2
Σ 1=1
log 2TeO??
22:
Update policy parameters (01+1, 01+1) arg maxe, or ( (J Rog)
23:
Update value function parameters by minimizing the meen-square error:
@i+1 = arg min E& (Vo(st) R&))
end for
25:
end for
26:
end for
function from the reference velocity field, we can leverage the result of Liu et al. (2025), the velocity field and score function explicitly related:
Vx log (x;:s( 1-t (x x:).
(14)
Building this connection, purpose a practical entropy regularizer:
jRez (0, σr) = (s). - (xllm
Wg
log (2g(2))))))
(15)
2
where Wh, Wy 2 0 are tunable coefficients and (x(;;, (1 ()v,(xx;s (x; (x((x)))))
(16)
The first term (termed Brownian regularizer) in Eq. (15) encourages the learned velocity field to align with the negative score of the reference flow, promoting expansion trajectories and pre- venting collapse into narrow modes. Note that do not directly take the difference between the velocity field and the negative score, since this would involve factor of (1 - t) in the denominator, which becomes problematic as 1; d) is defined to safely enforce alignment while avoiding this

---

Published as a conference paper at ICLR 2026
singularity. The second term in Eq. (15) corresponds to the entropy of the Gaussian noise injected at the flow terminal, enhancing stochasticity and encouraging diverse exploration. Together, these two terms promote trajectory diversity and maintain the expressiveness of continuous normalizing flows in modeling complex and multi-modal action distributions (see Fig. 2).
Importantly, unlike previous entropy regularizers that require computing log- log-likelihooss, expen- sive divergence integration (Chen et al., 2018; Tian et al., 2024) or heuristic noise injection (Frans al., 2025; Wang et al., 2024), the Brownian regularizer provides principled yet computationally lightweight alternative.
The Brownian regularizer should not be regarded as a theoretically exact derivation. In particular, while our formulation leverages the relationship between the velocity field and score function under rectified flows, the velocity field in our policy not obtained via flow matching gradients, and thus does not strictly correspond to the rectified flow dynamics.
Remark
EXPERIMENTS
We benchmark PolicyFlow against FPO and DPPO because both methods extend PPO to generative policy classes that do not allow explicit likelihood evaluation. These algorithms currently represent the SOTA in applying on-policy RI. expressive, non-Gaussian policy parameterizations. There- fore, comparing to FPO and DPPO is essential for demonstrating the effectiveness of PolicyFlow as general and principled alternative for training generative policies.
We evaluate these algorithms across the benchmarks in MuJoCo Playground (Zakka et al., 2025) and LsaacLab (Mittal al., 2023). Using the MultiGoal environment, role in fully leveraging continuous normalizing flow to capture complex, multimodal distributions and avoid mode collapse.
the Brownian regularizer's
5.1 MULTIGOAL TEST
The MultiGoal environment, originally proposed by Haarnoja et al. (2017), is a two-dimensional square workspace with six fixed goal locations, which we use to evaluate how the Brownian reg- ularizer prevents mode collapse and how continuous normalizing flows enable more expressive, multi-modal policies. We modify the dynamics to a second-order system: the state includes posi- tion and velocity, and the action is acceleration. Full agent and environment details are provided in Appendix C.2.
PPO
DPPO
FPO
PolicyFlow (Inject Uniform Noise)
PolicyFlow (No Brownian Reg.)
PalicyFlow (With Brownian Reg.)
(a)
(b)
(c)
(d)
(c)
Figure 2: MultiGoal Test (Appendix C.2): sample 1000 trajectories starting at the same original
point. (a) PPO with Gaussian entropy regularization (w, 0.001) covers only a limited set of goals. (b,c) DPPO and FPO collapse to small number of modes, likely because neither method incorporates any form of entropy regularization. (d) PolicyFlow with uniform noise injection (Ding et al., 2024) (weight 0.05) still suffers from mode collapse, concentrating on only a few modes. (e) PolicyFlow with only Gaussian entropy regularization (w, 0.001) partially alleviates mode collapse. (f) PolicyFlow with the proposed Brownian regularizer (w,, = 0.25) and Gaussian entropy regularization (wy 0.001) achieves the most diverse and more balanced goal-reaching behaviors.
the agent starts at the workspace center, all six goals are equidistant and rewards are symmetric, so an optimal policy should reach each goal with roughly equal probability, reflecting the multi-modal nature of the task. As shown in Fig. 2, PPO, which employs a Gaussian policy, can only represent simple distributions and thus struggles produce trajectories that reach all six goal locations. While

---

Published as a conference paper at ICLR 2026
FPO and DPPO utilize generative models capable of expressing complex distributions, the lack of an effective entropy regularization mechanism prevents the agent from learning a sufficiently diverse set of trajectories. In contrast, PolicyFlow with the Brownian regularizer fully leverages the expressive power of continuous normalizing flows, resulting in more balanced, multi-modal action patterns and higher coverage of all goals.
5.2 MuJoCo PLAYGROUND AND ISAACLAB BENCHMARKS
MuJoCo Playground benchmarks. We evaluate PolicyFlow against current state-of-ther ofrtthe-rt flow- based methods on the MuloCo Playground benchmarks, including FPO (McAllister et al., 2025) and DPPO (Ren et al., 2024). All these methods are based on the PPO framework, so we also include PPO as a baseline. FPO represents the policy using continuous normalizing flows (CNFs), while DPPO uses diffusion models. The original implementations of FPO and DPPO do not include ex- plicit entropy regularization, which can limit the diversity of the learned policies. As previously shown in the MultiGoal test, PolicyFlow effectively preserves multi-modaa behaviors and diverse trajectories. Across the MuJoCo Playground tasks, PolicyFlow achieves performance comparable exceeding FPO in most environments, outperforming DPPO, and generally matching or sur- passing PPO. Careful examination of the training curves reveals that PolicyFlow often converges faster, indicating higher sample efficiency and effective exploration, complementing the observa- tions on policy diversity from the MultiGoal experiments.
The hyperparameter settings for PolicyFlow are provided Appendix C.4. To ensure a fair compar- ison, the hyperparameters for FPO and DPPO follow the tuned configurations from the FPO paper, and PPO uses the default settings recommended by the MuJoCo Playground repository.
BallInCup
CartpoleBalance
CheetahRun
FingerSpin
1.0
0.8
0.8
0.6
0.8
0.6
0.6
Episodic Reward (x10²)
0.3
0.5
0.4
0.0
0.2
0.2
0.2
0.0
0.0
0.0
16
16
24
16
24
8
16
FishSwim
ReacherEasy
ReacherHard
WalkerWalk
1.0
1.0
1.0
0.5
0.8
0.8
0.3
0.5
0.1
8
16
24
0
8
16
24
16
24
0
8
16
24
FPO
PPO
DPPO
PolicyFlow
Figure 3: Learning curves on MuJoCo Playground benchmarks. Plots show mean episodic reward with standard error (y-axis) over environment steps (X-axis, total 30M steps), averaged over ran- dom seeds.
IsaacLab benchmarks. We further evaluate PolicyFlow the IsaacLab benchmarks, suite of robotics environments spanning locomotion, manipulation, and navigation. IsaacLab a recently developed and rapidly growing framework maintained by NVIDIA, designed specifically for large- scale robot learning. Its high simulation fidelity, strong engineering support, and increasing popu- larity in the robotics community make it an ideal testbed for assessing the performance of RL al- gorithms. In this benchmark, we compare PolicyFlow only against PPO. Although FPO and DPPO are state-of-ther -hrt--rrt generative policy approaches, neither of them includes IsaacLab tasks in their original benchmark suites, and directly adapting their publicly released code to the IsaacLab en- vironment stack requires substantial engineering effort and nontrivial environment re-integration. Therefore, we use the PPO implementation from RSL-RL as our baseline, following the official IsaacLab hyperparameter configurations. PolicyFlow uses almost identical hyperparameters to PPO (see Appendix C.3), except for additions required by our entropy regularization mechanism. Full parameter details are provided in Appendix C.3. As shown in Table and Fig. 5 in Appendix C.3, PolicyFlow achieves asymptotic performance that consistently matches surpasses PPO across all tasks. Since PolicyFlow learns a dependent velocity field rather than a direct action mapping, the optimization problem is inherently more complex, which can lead to slower early-stage learning.
8

---

Published as a conference paper at ICLR 2026
Table 1: Terminal training episodic rewards across IsaacLab benchmarks.
Method
Lift-Cube
Navigation
Open-Drawer
Quadcopter
Anymal-D
G1
HI
Go2
PPO
153.1 + 3.0
3.5 + 0.3
99.8 + 1.7
141.8 + 0.5
21.5
25.4 1.2
29.3 + 0.9
27.9 + 0.3
PolicyHlow
154.6 + 0.6
4.2 + 0.1
99.1 111:00 0.7
141.0 + 0.09
24.6 +
0.2
30.0 + 1.1
27.3 = 0.2
27.4 0.9
vavaue
0.0027
0.41
0.00026
0.0069
0.33
Training time per iteration. We compare the per-iteration training time of PolicyFlow and PPO on IsaacLab environments, which reflects the computational cost of a single training step. As shown in Table 2, when the model parameters are roughly comparable to PPO, PolicyFlow increases per- iteration training time by less than 50% for the first six IsaacLab environments. Even when embed- ding dimensions are increased up to eightfold, the computational cost remains below twice that of PPO, demonstrating that PolicyFlow efficient in practice.
Table Per-iteration training time of PPO and PolicyFlow on IsaacLab benchmarks, averaged over 50 iterations on an RTX 5090 GPU.
Eavironment
Lüt-Cube
Navigation
Open-Drawer
Quadcopter
Anymal D
G1
II1
Go2
EmbeeSdig Dumensiots (Time / (bseevation)
64
64
64
64
64
256
512
512
PPO (ms)
45.0 a: 16.1
36.9 6.3
81.3 A. 14.7
37.8 a: 13.8
41.2 13.4
66.9 144
63.4
15.5
63.9 15.7
PolicyFlow (m8))
$7.7 + 20.8
$4.1 1.1.t.00 10.1
104 + 16.5
$5.6 15.3
57.1 = 15.7
90.6 t 12.3
1155 = 173
111.5 18.1
Remark We do not provide a direct comparison with FPO or DPPO because the implementations of these algorithms in the FPO open-source codebase are based on JAX, whereas PolicyFlow is implemented in PyTorch. Conducting direct comparison across different deep learning frameworks could lead to unreliable results, so we focus the analysis on PPO, which is implemented in the same framework and provides fair baseline.
5.3 SENSITIVITY TO CLIPPING RANGE PARAMETER
Our proposed approximation Eq. (13) of the importance ratio introduces an approximation error that is theoretically bounded by the clipping range shown in Appendix A. A smaller yields tighter bound and therefore reduces the approximation error; however, it also limits the effective update step size in policy optimization, which may slow down policy improvement. To empirically verify this trade-off, we conduct a sensitivity analysis in the IsaacLab ANYmal-D environment by evaluating four clipping ranges, e {0.1, 0.2, 0.3, 0.4], each with five random seeds. The results shown as Fig. 4a confirm our theoretical insight: smaller clipping ranges lead to lower approximation error but hinder learning progress due to overly conservative updates. Across all IsaacLab benchmarks, we adopt e = 0.2 as the default clipping range, which aligns with the official PPO configuration provided by Ls.
5.4 SENSITIVITY TO NETWORK INITIALIZATION AND TIME SAMPLING STRATEGY
We also investigate how different network initialization strategies affect performance. A com- mon choice for MLPs network is the Glorot initialization (Glorot & Bengio, 2010), which samples weights from a Gaussian distribution with appropriate variance. Alternatively, one may initialize all network parameters to zero. In our study, we compare three initialization schemes in the IsaacI lab ANYmal-D environment: (i) GI: standard Glorot initialization, (ii) GI+ZOL: Glorot initialization with the output layer additionally set to zero, and (iii) ZI: full zero initialization for all parameters. As shown in Fig. 4b, the Glorot initialization with zeroed output layer achieves the best empirical performance. Therefore, this scheme is adopted for initializing our models across all benchmark experiments.
In addition, time sampling is required to estimate the expectation over time in the objective func- tion Eq. (12). We evaluate several time-sampling strategies in the IsaacLab Navigation environment: (i) USC: Uniform sampling from the continuous interval [o. 1]: (ii) USD: Uniform sampling from the discrete ODE simulation time grid (0, 0.05, o.I, 0.95, 1.0]: and (iii) Multi-USD: sampling multiple time points é for each state-action sample, with each drawn uniformly from the same discrete grid. The results are shown in Fig. 4c. Overall, these three time sampe-ng strategies lead to only minor performance differences. Therefore, USD is used as the default choice in all benchmark experiments. Multi-USI is generally not recommended, as it introduces additional computational overhead without clear benefits.

---

Published as a conference paper at ICLR 2026
25
2.0.1
Epizodic Reward
25
2.0.2
Episodic Roward
20-
Episodic Reward
20
15-
-40
10
G
-60
USC
Gl
ZOL.
USD
0-
21
Multi
-80
0.0
0.5
1.0
1.5
2.0
2
10
10a
10a
Training Ieerations (x109
Training Iterations (x109
Training Iterations (log scale)
(a) Clipping Range Comparison
(b) Different Initialization
(c) Time Sampling Strategy
Figure 4: Ablation studies on key components of PolicyFlow.
5.5 DIFFERENT CHOICES OF INTERPOLATION PATH
In the preceding sections, we adopted the same
Table Terminal training episodic rewards using
interpolation path rectified flow. However, prior work on flow matching has explored alter- native interpolation strategies. Table 4 summa- rizes two additional choices beyond rectified- flow path. We evaluate these interpolation paths on one IsaacLab locomotion task and the Multi-
different interpolation paths.
Path / Env
ANYmal-D
MultiGoal
Rectified Flow Path
24.6 2: 0.2
8.79 + 0.02
Brochastic-Interpllne Pach
24.7 0.2
8.22 0.18
TrigFlow Path
21.5 21.5000 0.1
8.71 0.03
Goal test, keeping all agent settings identical except for the interpolation scheme and its correspond- ing Brownian regularizer. As shown in Table 3, all three paths achieve nearly identical converged episodic rewards on the ANYmal-D locomotion task, whereas TriFlow path and rectified-flow path yield better performance than -nnteppiinntr path the MultiGoal test. The slightly lower performance of the stochastic-nterppllan path may be due to our use of an approximate relation- ship between the score function and the velocity field along this interpolation, rather than exact equality. See Appendix B for the details of this approximation.
Table 4: Different interpolation paths for flow matching
Method / Interpolation Formula
Relationship between Score and Velocity Field
Rectified Flow (Liu et al., 2022)
t) log (x() tx((
(1 (1--) Xo + X1; E [0,
Stochastic Interpolant (Albergo et al., 2023) (1 xo + tx /22
(2(1 ) 2
Vx (x) su Lvx(x)
h, 10. h~ N(0,
TrigFlow (Lu & Song, 2024)
cos(1) log = sin(() vx(x) cus(()
X, cos(t) Xo + sin(t) X1, E [0,
6 CONCLUSION AND FUTURE WORKS
In this work, proposed PolicyFlow, on-policy reinforcement learning algorithm that integrates continuous normalizing flows with PPO-style optimization. By approximating importance ratios via velocity field variations along interpolation paths, PolicyFlow eliminates the need for costly path-wise backpropagation while maintaining stability and efficiency. In addition, our purposed Brownian regularizer provides principled yet lightweight way mitigate mode collapse and en- courage diverse exploration. Through extensive experiments on MultiGoal, IsaacLab, and MuJoCo Playground, PolicyFlow consistently matches or outperforms PPO and the SOTA methods FPO and DPPO. In particular, results on MultiGoal showcase PolicyFlow's 's ability to capture complex multi- modal action distributions. Looking forward, PolicyFlow offers versatile foundation for bridging generative modeling and reinforcement learning. While different interpolation paths show promise in practice, though their formal validation and theoretical implications remain to be explored. Other promising directions include fine-tuning flow-matching policies, extending PolicyFlow to offline RL and broader generative modeling tasks, and exploring its connection to diffusion models by incorpo- rating score-based objectives. Finally, developing more comprehensive theoretical understanding of PolicyFlow may further inspire algorithmic improvements and strengthen its applicability real- world decision-making.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We provide detailed algorithm descriptions in Sec. 4 and Algorithm 1, full hyperparameter settings in Appendix C, and MultiGoal environment configurations builds upon the public PPO implementations in RSL-RL. and SkRL. Most results are averaged over multiple random seeds, and our conclusions remain reliable under randomness.
Appendix C.2. Our implementation
REFERENCES
Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Eijndens Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08877, 2023.
OpenAI: Marcin Andrychowicz. Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301 2023.
Chen-Hao Chao, Chien Feng, Wei-Fang Sun, Cheng-Kuang Lee, Simon See, and Chun-Y Yi Lee. Maximum entropy reinforcement learning via energy-based normalizing flow. Advances Neural Information Processing Systems, 10556136-61666 2024.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Zixuan Chen, Mazeyu Ji, Xuxin Cheng, Xuanbin Peng, Xue Bin Peng, and Xiaolong Wang. Gmt: General motion tracking for humanoid whole-body control. arXiv preprint arXiv:2506. 14770, 2025.
Xuxin Cheng, Kexin Shi, Ananye Agarwal, and Deepak Pathak. Extreme parkour with legged robots. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 11443- 11450. IEEE, 2024.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, pp. 0278364924127366.. 2023.
Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi. Diffusion-based reinforcement learning via w-weighted variational policy optimization. Advances in Neural Information Processing Systems, 37:53945-53966 2024.
Shutong Ding, Ke Hu, Shan Zhong, Haoyang Luo, Weinan Zhang, Jingya Wang, Jun Wang, and Shi. Genpo: Generative diffusion models meet on on-policy reinforcement learning. arXiv preprint arXiv:2505.1877 2025.
Albert Einstein. Über die von der molckularkinetischen theorie der wärme geforderte bewegung von in ruhenden flüssigkeiten suspendierten teilchen. Annalen der physik, 4, 1905.
Linjiaji Fang, Ruoxue Liu, Jing Zhang, Wenjia Wang, and Bing- Yi Jing. Diffusion actor-critic: Formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning. arXiv preprint 2rX.v:2405.20555, 2024.
Kevin Frans, Seohong Park, Pieter Abbeel, and Sergey Levine. Diffusion guidance is controllable policy improvement operator. arXiv preprint arXiv:2505.3355. 2025.
Dechen Gao, Boqi Zhao, Andrew Lee, Ian Chuang, Hanchu Zhou, Hang Wang, Zhe Zhao, Jun- shan Zhang, and Iman Soltani. Vita: Vision-to-action flow matching policy. arXiv preprint 1.Xiv:2507.13331 2025.
11

---

Published as a conference paper at ICLR 2026
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl incentivizes reasoning Ilms through reinforce- ment learning. Nature, 645(808 (1088088)663333 (::33)663333 638, 2025.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference machine learning, pp. 1352-1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. Pmlr, 2018.
Junzhe He, Chong Zhang, Fabian Jenelten, Ruben Grandia, Moritz Bächer, and Marco Hutter. Attention-based map encoding for learning generalized legged locomotion. Science Robolics, 10(105):eadv360)) 2025.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. pio.s: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504. 1.t::2555.6.0544 16054, 2025.
Richard Jordan, David Kinderlehrer, and Felix Otto. The variational formulation of the fokker- planck equation. SIAM journal on mathematical analysis, 29(1):1-17, 1998.
Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36:67195- 67212, 2023.
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47)xeabc5986, 2020.
Kun Lei, Huanyu Li, Dongjie Yu. Zhenyu Wei, Lingxiao Guo, Zhennan Jiang, Ziyu Wang, Shiyu Liang, and Huazhe Xu. RI-100: Performant robotic manipulation with real-world reinforcement learning. arXiv preprint 20.xv.2500. /4830, 2025.
Yaron Lipman, Ricky TQ Chen, Heli for generative modeling. arXiv preprint arXiv:2210.027477 2022.
Maximilian Nickel, and Matt Le. Flow matching
Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint arXiv:2505.0544 05470, 2025.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint wrXiv:2209.03003 2022.
Cheng Lu and Yang Song. Simplifying, stabilizing and scaling continuous-time consistency models. arXiv preprint ar arXiv:2410. 11081, 2024.
Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy energy-guided guided diffusion sampling in offline reinforcement learning. In Inter- national Conference on Machine Learning, pp. 22825-22855 PMLR, 2023.
David McAllister, Songwei Ge, Brent Yi, Chung Min Kim, Ethan Weber, Hongsuk Choi, Haiwen Feng, and Angjoo Kanazawa. Flow matching policy gradients. arXiv preprint arXiv:2507.21053 2025.
12

---

Published as a conference paper at ICLR 2026
Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yumrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, and Animesh Garg. Orbit: A unified simulation framework for interactive robot learning environments. IEEE Robotics and Automation Letters, 86))3740-37747, 2023. doi:
10.1109/1RA.2223.277000
Volodymyr Maih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1937. PmLR, 2016.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning a diffusion model policy from rewards via q-score matching. arXiv preprint arXiv:2312.11752 2023.
Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majum- dar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimiza- tion. arXiv preprint arXiv:2409.008: 00588, 2024.
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference robot learning, pp. 91-100. PMLR, 2022.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.00347, 2017.
Clemens Schwarke, Mayank Mittal, Nikita Rudin, David Hoeller, and Marco Hutter. Rsl-rl: A learning library for robotics research. arXiv preprint arXiv:2509. 20771:550991771. 2025.
Antonio Serrano-Muñoz, Dimitrios Chrysostomou, Simon Bøgh, and Nestor Prana-Arexolalebaa
skrl: Modular and flexible library for reinforcement learning. Journal of Machine Learning Re- search, 24(254):1-9, 2023, URL http: jmlr..mrr..//.. org htpppspppps///// /v4 23 0112. html.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. basre generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455. 2020.
Yifeng Tian, Nishant Panda, and Yen Ting Lin. Liouville flow importance sampler. arXiv preprint arXiv:2405.06672. 2024.
Alexander Tong, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Kilian Fatras, Guy Wolf, and Yoshua Bengio. Conditional flow matching: Simulation-free dynamic optimal transport. arXiv preprint 300.002302.004822, 2(3), 2023.
Yinuo Wang, Likun Wang. Yuxuan Jiang, Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang, Liming Xiao, Jiang Wu, Jingliang Duan, et al. Diffusion actor-critic with entropy regulator. Advances in Neural Information Processing Systems, 37:54183-54206 2024.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.0619 2022.
Kevin Zakka, Baruch Tabanpour, Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire, Erik Frey, Koushil Sreenath, Lueder A. Kahrs, Carlo Sferrazza, Yuval Tassa, and Pieter Abbeel. Mujoco playground: An open-source framework for accelercee robot learn ing and sim-to-real transfer., 2025. URL https: : /ightb.t.bb/e////gg/gggg..hhh com/ hwop//oooeeeeee deepmind/ mu jocoo_playground.
13

---

Published as a conference paper at ICLR 2026
Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. arXiv preprint arXiv:2403.03354, 2024.
Simon Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Peter Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. Advances in neural information processing systems, 37:1 37:110935-11097,, 2024.
Shiyuan Zhang, Weitong Zhang, and Quanquan Gu. Energy- flow matching for offline reinforcement learning. arXiv preprint ar arXiv:2503.04977 2025.
USE OF LARGE LANGUAGE MODELS
Large Language Models (LLMs), specifically ChatGPT-5, were used for two purposes: (1) to aid with grammar checking, language polishing, and ensuring formatting consistency; and (2) for re- trieval and discovery tasks, such finding relevant related work. No content, ideas, scientific claims were generated or altered by LLMs.
A ERROR ANALYSIS OF THE POLICYFLOOW OBJECTIVE APPROXIMATION
We start from flow parameterization of the policy, derive variational expression for the terminal shift, and introduce interpolation path approxination. We then rigorously analyze the induced error, showing to first-order, and justify the approximation 's validity within the PPO framework.
A. VARIATIONAL FORMULA FOR THE TERMINAL SHIFT
Let the two flows, generated from random vectors, satisfy
(; (p)((z)))):
(17)
(2;2:)) = (x)
(18)
with the initial condition po(zzs) (z)zs) = z. Set the terminal shift as (z; = (z( (2; and the velocity field variation as 6,, (x: s) = u4 (x; s) (x
Linearizing around the reference trajectory &e yields the variational equation
بمة = J1d,, + 8e, (1iss)), with 6.0 = 0,
(19)
where the Jacobian is defined J, (x::sx:ss) x=²
Let (11, t) be the fundamental matrix of (T, = J, (T, with (tt,t) I. Then, the exact
terminal shift
(2; / (),1)).., (2)
(20)
A.2 INTERPOLATION PATH APPROXIMATION AND ERROR ANALYSIS
approximate the complex integral in Eq. (20) using a simpler linear interpolation path:
x, = (1 - t)z + t (z; t [0, 1].
(21)
The approximation for the terminal shift 8, := fo 8., s)tt The emror of this approximation is F dn J..- We decompose this error into two components:
/'' (fr,(P() (x:))
E
(11 1) (R.)dt
(22)
E
F2
Assume the following:
14

---

Published as a conference paper at ICLR 2026
(A1) The velocity field 8/ is C' in with uniformly bounded Jacobian, || I L.
(A2) The velocity variation o., is uniformly bounded, ||ó,, (x; S. e, and is Lipschitz in x with constant L (c) uniformly in (t.s). The assumption Lg justified as the variation itself stems from small policy update of size e.
The first error term, E1, arises from approximating fundamental matrix (11, () with the identity matrix I. Since (1,4) I|| = (11) and [|ō., O(e), the magnitude of this term is:
| < (1,t) I|| 116 (f.) = S.
O(c).
(23)
The second error term, E2, comes from replacing the true trajectory & with the linear path x,- The path deviation || x+] is generally ((1). Given the Lipschitz assumption on dz,
12ll L. Loller - xe|ldt = L. O()) o =(().
(24)
Since both error components are first-order, the total approximation error is also first-order. We thus correct the conclusion from the main text:
6,p (2:s) = .f 6c,
O(c).
(25)
A.3 FIRST-ORDER ERROR OF THE LIKELIHOOD RATIO
Let (2; 6,, (2; and ú fo ov, (xx;))dt. From Eq. (25), have u + Oee).
Taylor expansion of the Gaussian log-density shows that a first-order error the mean leads to first-order error the log- lieeliidl:oood:
+ +0()) =n (O(e)
((()r) = O(e). (26)
log Pm. (n; ú, o²) - log u, o²)= T
Using Jensen's 's inequality, we have
log u, 0²) 2 / logpa((; p. (n; on, (x:; o²) dl = Ep(t) Mlogaa(m on, (x:; (x,;8)),0)) (27)
Thus,
Ep()) [log Pm (n; dn, (x);s), 2 log Pm (n; u, O(e).
(28)
Consequently, the error in the log of the importance ratio is first-order:
Pr (n; 8,, (xx: o²) log
Pn n (nuu,r) x²)
Ep(e) log
= O(e).
(29)
Pm (m 0,62
Pr (n; 0, o2)
While this is a weak bound of the importance ratio eπror, it is sufficient to ensure the algorithm's 's practical effectiveness when using the PPO-style surrogate objective for policy improvement. first- order approximation error means that for small updates, the gradient computed with the approximate objective is a close match to the gradient from the exact objective. The PPO-style clipping mecha- nism inherently restricts the update size which minimizes the impact of this linear error term and preserves the stability of training. Therefore, the approximation remains a valid and computationally efficient method for optimizing continuous normalizing flow policies.
B RELATIONSHIP BETWEEN SCORE AND VELOCITY FIELD FOR STOCHASTIC INTERPOLANTS
Consider the stochastic interpolant used in flow matching:
X lx1 + (1 t)xg 7()2,
Xo ~ Po. X1 P1, NO T).
(30)
15

---

Published as a conference paper at ICLR 2026
Albergo al. (2023) derive the relationship between the function Vx (xg) along the interpolation path and the velocity field uy (xx) used in conditional flow matching (see Eq. (2.27) in their work):
ux (x+) = (x())) =()) (() Vx pxxe)(
(31)
where
(x,) = [x x [x] + (1 - 1)xo + x1)2== x1]
(32)
In general, obtaining a closed-form expression for üe (x+) is intractable, which prevents an exact analytic relationship between the score function and the velocity field under a stochastic interpolant. However, when 7() = O, the interpolant reduces to the deterministic path of rectified flows, for which the relationship becomes explicit. This motivates approximating Vy (x,) using the determinis-
tic identity:
By (x,) Rd (1- t) v (x, +x
(33)
Substituting this approximation into the expression for (x(x)) gives an approximate relationship between the stochastic velocity field and the score:
(1 - vx (x.) + Vx
My (x,, ~
(34)
Finally, choosing the commonly used stochasticity schedule
(tt) 2 t)t,
(35)
yields the expressions summarized in Table 4.
C EXPERIMENTAL DETAILS
Our algorithm builds upon the open-source frameworks SKRL (Serrano-Muñozz et al., 2023) and RSL-RL (Schwarke al., 2025). Specifically, inherit the implementation of the flexible replay buffer from SKRL and integrate it with the framework of the PPO implementation provided by RSL-RL.
C.1 MODEL ARCHITECTURE
The model used in PolicyFlow is based flow network that maps noise inputs to actions, conditioned on state or other context. The model comprises four main components:
va s)
MLP
Flow Network (MLP): multi-layer perceptron that predicts the velocity with noised actions and time and observation embedding as inputs.
Timestep Embedding (FourierEmbeddingg) Uses fixed set of frequencies to encode the scalar noise / time step into a high- dimensional representation. The embedding is computed
Fourier Embedding
Linear Embedding
1
1s
Lemb = MLP((cos(2xf1l)) (1)))())1(222
which allows the model to better capture temporal dependencies.
Observation Embedding (Lyne:rLayer): A linear layer that embeds observation vectors into fixed- dicd-dimensional space, which added the timestep embedding to modulate the flow network outputs.
Learnable Variance: It's the same as that in Gaussian policy parametrization PPO.
C.2 MULTIGOAL SETUPS
The MultiGoal environment is two-dimensional square workspace designed to evaluate the ability of agents to learn multimodal and balanced goal-reaching behaviors. Six fixed goals are placed evenly circle centered at the origin, with equal distance to the starting position. At the beginning
16

---

Published as a conference paper at ICLR 2026
of each episode, the agent is randomly initialized within the workspace and must learn reach the nearest goal as efficiently as possible.
The environment is modeled locity, and the action corresponds to a 2D acceleration vector. The reward is composed of two com- ponents: distance-based term encouraging the agent to approach the nearest goal, and an action penalty discouraging excessive control inputs. By combining these terms, the environment provides consistent evaluation of both goal-reaching accuracy and control efficiency.
second-order system: the state consists of both position and ve-
The Table 5 and Table 6 summarize the main environment configuration parameters and the defini- tions of state, observation, and reward functions.
The agent configurations for the MultiGoal task are summarized in Table 7 and Table Both PolicyFlow and PPO share common hyperparameters such learning rate, discount factor, GAE parameter, and clipping settings. Compared to PPO, PolicyFlow introduces an additional term, Brownian regularization loss, while PPO employs a standard Gaussian entropy regularizer.
Table 5: MultiGoal environment configuration
Parameter
Value
Maximum opisode length
50
Action clipping range
[-5.0,
Integration timestep (ft)
Chock wall collisions
True
Workspace
sprkspace limits
[-8.0, 8.0]
Number goals
Goal radius
5.0
Goal center
[0.0, 0.0)
Distance reward scale
1.0
penalty
Table 6: MultiGoal Environment: State/Observation and Reward Functions
Componcnt
Description Formula
Observation / State
State voctor
[0, My May Uy]
Actor / Critic observation
Action
Action
2D acceleration; [az, ay] (clipped to (-5, 5))
Reward Functions
Distance reward
exp( 00.3 d 1
where dai min, xill distance nearest goal
Action penalty
Pation
0.001 ||a]
reward
Pors Tieenno Fiction
17

---

Published as a conference paper at ICLR 2026
Table 7: PPO and PolicyFlow hyperparameter configurations for MultiGoal
Parameter
PolicyFlow
PPO
Desired KL divergence
0.01
0.01
Learning rate
2 10
2 10
Discount factor
GAE
0.95
0.95
Timo-limit bootstrap
True
True
buii-bbttches
Learning epochs
Gaussian eatropy
0.001
Brownian regularization loss scale
0.25
clip
Clip predicted values
True
Troe
Value clip
0.2
Value loss scale
1.0
Timesteps of second order Runge Kutta
12
integration
Actor hidden layers
[256, 128, 64)
[256, 128, 64]
hidden layers
[256, 64]
[256, 128, 64]
Actor/Critic activation
Mish
Mish
Number of parallel enviromments
1024
1024
Rollouts per environment
24
24
Table 8: Hyperparameter settings for Flow Matching Policy Optimization (FPO) on the MultiGoal environment.
Parameter
Value
Timesteps Fuler integration
Time embedding dimcasion Samples number per action
8
8
Average losses before exponentiation
Use discretized timestops in training
Tiue
Clipping epsilon for ratio update Training batch size
1024
Discount factor
0.99
Maximum
episode
Learning rate
1 x 10
Number parallel environments
4096
Mini- batc-bbtches per update
Total environment steps
1.2 10
leamnng epochs
GAE parameter X
0.95
Value loss scale
0.25
Actor hidden
[256, 128,
Critic hiddeen layers
[256, 128, 64]
ActorfCritic activation
18

---

Published as a conference paper at ICLR 2026
C.3 ISAACLAB
Isacc-Ltt-Cub--rraakk
Lseac-OpenDDrawaFFratt
lsaac-Quadcoper--DDa
100
120
-20
50
40
PPO
Episodic Reward
Pollcy/low
-80
0
0.0
4.0
6.0
0.0
4.0
0.0
2.0
4.0
0.0
4.0
lsaac-Velocity- Fla-nnymll-----
Isaac-elloctt-RouuhhCC
Jsaac-VelocityKooggh.
Leaag-Velocity-Rooe
30
20
20
20
20
10
0-
-10
0.0
4.0
4.0
Training Iteraioons (x107)
Figure 5: Learning curves (PolicyFlow v.S. PPO) on IsaacLab benchmarks. Plots show mean
episodic reward with standard error (y-axis) over training iterations (x-axis), averaged over ran- dom seeds.
Table Common hyperparameter settings for PPO and PolicyFlow used the all IsaacLab bench- marks
Parameter
PolicyFlow
Timesteps second-order Runge-Kutta
integration
Number of parallel enviromments
4096
4096
Learning rale aduptive
True
True
Desired KL. target
0.01
0.01
purameter
Time-limit bootstrap
Thue
True
Mini-batches per update
Learning epochs
s
Ratio clip
0.2
Clip predicted values
Time
True
Value clip
0.2
0.2
Value loss
Gradient norm clip
1.0
1.0
Actor/Critic activation
ELU
ELU
Table 10: Variable hyperparameter settings for PPO and PolicyFlow on three IsaacLab benchmarks.
Sit-Cube--ranna
Aavigation-ANYmalle
Quadcopter-Diret
Parameter
PolicyFlow
PPO
PolicyFlow
PPO
PolicyFlow
Acton/Crtic hidden layers
128.61))
[256.128.64]
[128,128]
[128.128]
[64,64]
[128,128]
Initial learning rale
Discount factor
0.98
0.98
0.99
0.99
0.99
0.99
Goussian entropy cocfficient Wa
0.006
0.004
0.005
0.0025
0.01
0.005
Brownian regularizer coefficient
0.002
0.0025
0.005
Time/Obserration emhedking dim
64
64
Rolleats per envirnment.
24
8
24
24
19

---

Published as a conference paper at ICLR 2026
Table 11: Variable hyperparameter settings for PPO and PolicyFlow on three IsaacLab benchmarks.
Flat-ANYmal-D
Rough-HI
Rough-Unitree-Go.
Parameter
PPO
PolicyFlow
PPO
PolicyFlow
PPO
PolicyFlow
Acton/Critic hiddln layers
[128.128.128]
1128,128.124
1512,256,128]
1512,256.128]
1512.256,128
$31,225.1281
Initial learning rale
1 x 10
x 10
x 10
1 X 10
x 10
1 x 10
Gaussian cotropy cocfficient
0.005
0.0025
0.01
0,005
0.01
0.008
Brownian regularizer coefficient wa
0.0025
0.005
0.002
Ztme//bservait erubedding dian
64
512
512
Rollouts per environment
24
24
24
Table 12: Variable hyperparameter settings for PPO and PolicyFlow on two IsaacLab benchmarks.
Open-Drawer-Frankke
Rough-G1
Parameter
PPO
PolicyFlow
PPO
PolicyFlov
Actor/Critic hidden layers
[256,128.64]
[256,128.64]
[512.256,125
[512,256,128]
Initial learning rate
x 10
3
1 10
Discount factor
0.99
0.99
0.99
Gaussian entropy coefficient My
0.001
0.0008
0.008
0.002
Brownian regularizer coefficient
0.0002
0.006
Hinee/bservattien embedding dim
64
256
Rollouts per cavironment
C.4 MuJoCo PLAYGROUND
Table 13: Common hyperparameters for PolicyFlow across MuJoCo Playground benchmarks.
Parameter
Value
GAE purumeter A
0.95
Time-limit bootstrap
True
Learning aduptive
False
Mini hutches per update
Learning epochs
Clip predicted values
False
Value
Timesteps of second order Runge Kutta integration
12
Actor hidden layers
[32, 32, 32, 32]
Criic hidden layers
[256. 256, 256, 256, 256, 256] Si1.U
Actor/Critic activation
Number parallel enviromments
1024
Rollouts per cnvironment
24
Table 14: Variable hyperparameters for PolicyFlow on MuJoCo Playground benchmarks (1/2).
Walker
FingerSpin
BallinCup
CartpoleBalance
FishSwim
Parameter
I Learning rate
x 10
5 10
x 10
x 10
3 10
Gaussian cotropy cocfficient wi
0.001
0.002
Brownian regulatizer coefticient
0.001
0.001
0.001
0.0001
0.002
Ratio clip
0.2
0.2
0.2
0.01
0.2
Gradient clip
1.0
10.0
1.0
0.5
5.0
20

---

Published as a conference paper at ICLR 2026
Table 15: Variable hyperparameters for PolicyFlow on MuJoCo Playground benchmarks (2/2).
FingerTUunlHard
FingerTuurnEany
CheetahRun
ReacherEasy
ReacherHard
Parameter
Learning rate
1 x 10-4
x 10-4
x 10-4
x 10-
x 10-4
Gaussian entropy coefficient Wi
0.006
0.002
0.002
Brownian regularizer coctficient
0.001
0.001
0.001
0.001
0.001
Ratio clip
0.2
0.05
0.05
0.05
0.05
Gradient clip
-1.0
10.0
10.0
5.0
5.0
21


Reviews:
[{"review_id": "h7Xix0PIro", "reviewer": "Reviewer_PmDD", "summary": "This paper introduces PolicyFlow, a novel on-policy reinforcement learning algorithm designed to leverage the expressive power of Continuous Normalizing Flows (CNFs) for policy representation within a PPO-like framework.  They show how to avoids the need for full ODE simulation and backpropagation during the policy update and how to implement the entropy regularization in a smart way in the generative policy situation by imitating the Brownian motion. They finally show improved performance compared to the traditional PPO and related flow-based PPO method.", "questions": "1, since the first-order approximation of the importance ratio may introduce bias into the gradient estimate, have you empirically assessed its impact? For example, did you observe any degradation in performance or training instability as the PPO clipping range ϵ increases, which would amplify the approximation error?\n\n2, could you clarify the sampling strategy for tk (Algorithm 1, lines 15–16)? In the final experiments, did you use uniform sampling U[0, 1], and if so, did it perform better than sampling from the discrete simulation time points?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors demonstrate how to bypass the computationally expensive full ODE simulation and backpropagation typically required when using Neural ODE-based policies with a PPO objective. Their key insight is to use an efficient approximation of the importance ratio, enabling stable on-policy training without the standard computational bottlenecks.\n2. the paper introduces a lightweight \"Brownian regularizer\" to enhance behavioral diversity and mitigate mode collapse.", "weaknesses": "1. How is the initial flow matching model for the method in this paper obtained? What is the impact of the initial model's performance on the overall method?\n\n2. The target distribution of the flow-based policy changes dynamically during training, yet the objective function samples only a single t from the path at each step. Could this, due to the varying sample weights (different values of A) for each t along the path, prevent the model from learning an effective distribution?\n\n3. A sensitivity analysis experiment should be conducted on the parameters used in the algorithm, e.g, Brownian regularizer weight wb.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dQUFDXVeUR", "reviewer": "Reviewer_UPw3", "summary": "The paper introduces PolicyFlow, an on-policy reinforcement learning algorithm that replaces the standard Gaussian policy used in PPO with a continuous normalizing flow parameterized by ordinary differential equations. This modification aims to enhance expressiveness for complex or multimodal action distributions. The approach also incorporates a Brownian motion-inspired implicit entropy regularizer to promote exploration without relying on explicit policy entropy terms. The paper compares PolicyFlow against PPO and other flow-based policy optimization methods, such as FPO, reporting similar or slightly improved empirical results on selected benchmarks.", "questions": "**Detailed Review:**\n\nThe paper proposes PolicyFlow, which replaces the Gaussian policy distribution in PPO with a continuous normalizing flow modeled through ordinary differential equations. The design aims to capture multimodal action distributions and improve policy expressiveness. The use of a Brownian motion-inspired implicit entropy regularizer seeks to enhance exploration without introducing explicit entropy terms in the loss.\n\nAlthough this approach is conceptually sound, the paper does not provide strong empirical or theoretical evidence that PolicyFlow meaningfully improves over PPO. The observed results in Figures and Tables suggest comparable performance rather than clear gains. The discussion of results should better address why these similarities occur and whether the proposed approach offers other advantages, such as stability or robustness.\n\nThe focus on comparison with FPO is noted, but it is unclear why this method receives particular attention when the central comparison should remain with PPO. It would be helpful to clarify the methodological differences that justify this choice. The analysis could also include visualization or theoretical examples that demonstrate how the normalizing flow structure affects action representation or optimization dynamics.\n\nWhile the Brownian entropy regularizer is an appealing idea, its effect on exploration and stability is not quantified. A comparison with standard entropy regularization would help assess whether it provides genuine benefit or simply replicates existing behavior. Similarly, the discussion on computational efficiency should more clearly address the trade-offs between the added parameterization and any observed gains in performance.\n\nThe runtime and parameter analyses are appreciated, as they help contextualize the computational implications of the proposed method. However, since the improvements are small and the training time increases, the practical benefit of adopting PolicyFlow remains uncertain. The lack of variance reporting in Tables 1 and 2 also limits confidence in the consistency of the results.\n\nIn summary, the idea of integrating normalizing flows into policy gradient methods is interesting and potentially valuable, but the paper does not yet establish sufficient justification, theoretical insight, or empirical advantage to support its adoption.\n\n**Questions:**\n\n1. What is the specific benefit of using normalizing flows over Gaussian policies in terms of policy expressiveness and action representation?\n2. How does the Brownian entropy regularizer contribute to exploration compared with standard entropy regularization?\n3. What is the motivation for emphasizing comparisons with FPO, and how do these comparisons support the main claims of the paper?\n4. How does PolicyFlow perform in terms of computational cost and sample efficiency relative to PPO?\n5. Could the authors include a simple theoretical or illustrative example showing how the flow-based policy captures multimodal actions more effectively?\n6. Are the experimental results averaged over multiple runs, and if so, could variance be reported to strengthen the reliability of the findings?\n7. What is the runtime difference between **sampling action** from the normalizing flow policy and from the Gaussian policy?\n8. Under what conditions does PolicyFlow provide a clear improvement over PPO, and how does this relate to its theoretical formulation within policy gradient optimization?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Addresses the expressiveness limitation of Gaussian policies by exploring normalizing flows for policy representation.\n- Introduces a Brownian motion-based entropy regularizer to encourage implicit exploration.\n- Presents a clear and structured implementation based on PPO.\n- Includes runtime and parameter analyses, offering transparency on computational cost.\n- Demonstrates engagement with related work, including comparisons to other flow-based methods.", "weaknesses": "- The reported empirical results are very close to PPO, providing limited evidence of improvement.\n- The additional model complexity and slower runtime are not justified by corresponding performance gains.\n- The theoretical connection between the flow-based representation and policy gradient optimization is underdeveloped.\n- The motivation for emphasizing FPO comparisons is not well justified relative to the paper’s main objective.\n- Benchmark evaluations and variance reporting are incomplete, limiting the strength of the experimental conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ydubmevv7M", "reviewer": "Reviewer_gsX4", "summary": "The paper introduces PolicyFlow, an on-policy RL method that uses continuous normalizing flows for more expressive policies. It avoids expensive backpropagation through flow trajectories by using an interpolation-based importance ratio. A Brownian regularizer is added to encourage exploration. Experiments on MuJoCo, IsaacLab, and MultiGoal show stable and competitive results compared to PPO and FPO.", "questions": "Can you clarify whether the interpolation-based importance ratio introduces any noticeable bias? It would help to know if you’ve compared it against an exact or less-approximated version.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper proposes an interesting and original idea — combining continuous normalizing flows with on-policy policy optimization in a practical way.\nThe paper is clearly written, with intuitive explanations and helpful figures that make the method easy to understand.\n\nExperiments cover multiple benchmarks (MuJoCo, IsaacLab, MultiGoal) and show consistent improvements over PPO and FPO.\n\nOverall, the work is technically solid and provides a promising direction for expressive yet stable flow-based policy learning.", "weaknesses": "Methodological Weaknesses:\n1. The proposed interpolation-based estimation of importance ratios is only heuristic; the paper does not quantify the bias introduced or establish convergence guarantees. Providing analytical error bounds or controlled experiments comparing with exact estimators would strengthen credibility.\n2. The Brownian regularizer, while novel, lacks clear motivation and comparison with existing entropy regularizers (e.g., Haarnoja et al., 2018; Chao et al., 2024). Its empirical benefit is only qualitatively shown; a quantitative ablation isolating its contribution is necessary.\n\n\nExperiments Weaknesses:\n1. The experimental evaluation is also fragmented—PolicyFlow is compared with FPO on MuJoCo and PPO on IsaacLab, preventing a unified cross-method assessment. \n2. Including all three under identical settings would clarify relative efficiency and expressivity. Furthermore, results are averaged over five seeds without significance testing\n3. The MuJoCo Playground evaluation covers only a moderate subset of tasks (e.g., BallInCup, CheetahRun, FingerTurnHard, ReacherHard) but omits the most challenging or high-dimensional environments such as Humanoid or Walker. This limits claims of scalability to complex continuous-control domains.\n\nMinor Issues:\n1. some notations is not well defined: Sec. 3. $s,a$ is not defined before using. $p_{\\pi}(s)$ is never defined. $\\pi^*$ is mentioned but not clearly defined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NKL4aX9c5X", "reviewer": "Reviewer_P1yA", "summary": "This paper proposes a novel on-policy algorithm named PolicyFlow. Its core contributions are:\n\n1) Efficient importance ratio approximation: Instead of computing the full flow path during training updates, PolicyFlow approximates importance ratios using velocity field variations along a simple \"interpolation path\". This significantly reduces computational overhead.\n2) Brownian entropy regularizer: To prevent potential mode collapse and encourage diverse behaviors , the authors introduce a lightweight entropy regularizer inspired by Brownian motion. It implicitly increases policy entropy by shaping the velocity field , avoiding the significant cost of explicit entropy or log-likelihood computation.\n\nThe authors conducts experiments in diverse environments, including MultiGoal, IsaacLab, and MuJoCo Playground. Particularly on the MultiGoal task, PolicyFlow demonstrates its ability to capture diverse multimodal action distributions.", "questions": "1) Table 5 shows that using different interpolation paths leads to drastically different performance on the MultiGoal task. This suggests that the choice of interpolation path is another critical design decision. Why does the paper default to the Rectified-Flow path? What advantages does it have, in theory or practice, compared to other paths?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The core contribution of this paper is a novel and efficient importance ratio approximation method. By evaluating velocity field differences along a simple interpolation path, it avoids the significant overhead of solving ODEs and backpropagation during PPO updates, making the application of CNF policies computationally feasible. Additionally, inspired by physical processes, the paper proposes a novel Brownian regularizer, which provides a lightweight means of preventing mode collapse by directly acting on the velocity field, thereby avoiding expensive entropy calculations.\n2) The method of this paper performs well on MultiGoal (Figure 1) and demonstrates high computational efficiency.", "weaknesses": "1) The baselines are limited, comparing only against PPO and FPO. Methods like GenPO and DPPO are missing. Furthermore, the method does not show an advantage in half of the tasks in the IsaacLab benchmarks.\n\n2) There are too many hyperparameters, making reproduction difficult. Beyond standard PPO hyperparameters, it also involves the sampling strategy for $t$, noise variance learning, time or observation embedding dimensions, integration steps, and $w_b$, $w_g$, among others. The paper provides limited discussion on the sensitivity to these hyperparameters and the default strategies used.\n\n3) No code available.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)