Flow-Based Policy for Online Reinforcement Learning
Authors: Lei Lv
Yunfei Li
Yu Luo
Fuchun Sun
Tao Kong
Jiafeng Xu
Xiao Ma
Venue: NeurIPS 2025 poster ()
Keywords: Online Reinforcemen Learning
Flow Matching
Primary area: reinforcement_learning

Abstract: We present $\textbf{FlowRL}$, a novel framework for online reinforcement learning that integrates flow-based policy representation with Wasserstein-2-regularized optimization. We argue that in addition to training signals, enhancing the expressiveness of the policy class is crucial for the performance gains in RL. Flow-based generative models offer such potential, excelling at capturing complex, multimodal action distributions. However, their direct application in online RL is challenging due to a fundamental objective mismatch: standard flow training optimizes for static data imitation, while RL requires value-based policy optimization through a dynamic buffer, leading to difficult optimization landscapes. FlowRL first models policies via a state-dependent velocity field, generating actions through deterministic ODE integration from noise. We derive a constrained policy search objective that jointly maximizes Q through the flow polciy while bounding the Wasserstein-2 distance to a behavior-optimal policy implicitly derived from the replay buffer.  This formulation effectively aligns the flow optimization with the RL objective, enabling efficient and value-aware policy learning despite the complexity of the policy class. Empirical evaluations on DMControl and Humanoidbench demonstrate that FlowRL achieves competitive performance in online reinforcement learning benchmarks.

Flow-Based Policy for Online Reinforcement Learning
Lei Ly Yunfei L Yu Luo", Fuchun Sun Tao Kong", Jiafeng Xu Xiao Ma? Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University
ByteDance Seed
Tsinghua University
23118309tongji. edu. cn; [1iyynfei. cloud, tackong, xjf, xiao bay.mm {luoyu19, fccuun)@singhuuee edu.
Abstract
We present FlowRL, novel framework for online reinforcement learning that integrates flow-based policy representation with Wasserstein- 2-rssttii---rgullariied opti- mization. We argue that in addition to training signals, enhancing the expressive- ness of the policy class is crucial for the performance gains in RL. Flow-based generative models offer such potential, excelling at capturing complex, multimodal action distributions. However, their direct application in online RL is challenging due to a fundamental objective mismatch: standard flow training optimizes for static data imitation, while RI. requires value-based policy optimization through a dynamic buffer, leading to difficult optimization landscapes. FlowRI. first models policies via state-dependent velocity field, generating actions through determinis- tic ODE integration from noise. We derive a constrained policy search objective that jointly maximizes Q through the flow policy while bounding the Wasserstein-2 distance to a behavior-optimal policy implicitly derived from the replay buffer. This formulation effectively aligns the flow optimization with the RL objective, enabling efficient and value-au aalue- policy learning despite the complexity of the pol- icy class. Empirical evaluations DMControl and Humanoidbench demonstrate that FlowRL RL achieves competitive performance in online reinforcement learning benchmarks. We have released code here
Introduction
Recent advances in iterative generative models, particularly Diffusion Models (DM) [16] 38] and
Treining Time (bours)
Normaltzad Score
Indorectoo CO 1201 (U6))
Flow Matching (FM) [23] 24 41. have demon-
strated remarkable success in capturing complex
multimodal distributions. These models excel tasks such high-resolution image synthesis 77) robotic imitation learning [). and protein struc- owing their expressivity and ability to model stochasticity. A promising yet underexplored application lies leveraging their multimodal generation capabilities to enhance re- inforcement learning (RL) policies, particularly in environments with highly stochastic or multi- modal dynamics.
Benchmarks
Time Metrics
ture prediction [18 3
Figure 1: (left) Normalized scores comparing FlowRL and DM-based RL (QVPO) on 12 chal- lenging DMC-hard and HumanoidBench tasks, and 3 DMC-easy middle tasks. (right) Compu- tational efficiency on the Dogrun task: 1M-step training time and single env step inference time.
"The work accomplished during the authors' internship ByteDance Seed.
1 Correspnnding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Traditional RL frameworks alternate between Q-function estimation and policy updates [39, often parameterizing policies Gaussian [3] or deterministic policies [2] maximize expected returns. However, directly employing diffusion or flow-based models as policies introduces fundamental challenge: the misalignment between RI. objectives, which aim to optimize value-aware distributions, and generative modeling, which imitates static data distributions. This discrepancy becomes exacerbated in online RL, where nonstationary data distributions and evolving Q-value estimates lead to unstable training [12)
While recent methods have pioneered the use of diffusion model (DM)-based policies in online reinforcement learning 45 8 43 these approaches still suffer from high computational cost and inefficient sample usage (see Section 2). By contrast, flow-based models (FMs), despite their ability to represent complex and multimodal policies, have yet to be effectively integrated into online RL frameworks.
Our method distinguishes itself by leveraging carefully selected replay buffer data as a reference distribution align flow-based policies with high-value behaviors while preserving multimodality. Inspired by prior works such as SIL (28) and OBAC [26). which utilised behaviour policies to guide policy optimization but limit policy expressivity to capture diverse behaviors, we propose unified framework that integrates flow-based action generation with (aaserrteiin22-rgulaaiii distribution matching. Specifically, our policy extraction objective simultancously maximizes Q- values through flow-based actor and minimizes distribution distance from high-reward trajectories identified in the replay buffer. By reformulating this dual objective as guided matchint loss, we enable the policy to adaptively imitate empirically optimal behaviors while exploring novel actions that maximize future returns. Besides, this approach retains the simplicity of standard actor-critic architectures, without requiring lengthy iterative sampling steps or auxiliary inference tricks [19 -Jyett fully exploits the multimddlltty of flow models to discover diverse, high-performing policies. We evaluate our approach challenging DMControl 440 and HumanoidBench [36), demonstrating competitive performance against state-of-the-as baselines. Notably, our framework achieves one-step policy inference, significantly reducing computational overhead and training instability caused by backpropagation through time (BPTT) 43 31. Experimental results highlight both the empirical effectiveness of our method and its practical advantages in scalability and efficiency, establishing robust pathway for integrating expressive generative models into online RL.
[TO]
2 Related Work
In this section, we provide a comprehensive survey of existing policy extraction paradigms based on iterative generative models based policy, with particular focus recent advances that leverage diffusion and flow-based models in offline or online reinforcement learning. We categorize these approaches according to their underlying policy optimization objective and highlight their respective advantages and limitations.
Generalized Behavior Cloning Generalized Behavior Cloning, often akin to weighted behavioral cloning or weighted regression [33 trains policies by imitating high-reward trajectories from replay buffer, weighted by advantage or value estimates, thereby avoiding BPTT. Previous methods like EDP [99 QGPO 255, QVPO [8], and QIPO [46] implemented these paradigms, enhancing computational efficiency by bypassing BPTT. However, as demonstrated in prior research, this approach has been empirically shown to be inefficient 30 31. and often leads to suboptimal performance.
Reverse process as policy parametrizations These methods use reparameterized policy gradients, computing gradients of the Q-function with respect to policy parameters directly through the genera- tive model's reverse sampling process, similar to the reparameterizzatio trick commonly employed in Gaussian-bartd based policies [13]. Previous methods, such as DQL 44), DiffCPS [15, Consistency- AC 19, and DACER 43), backpropagate gradients through the reverse diffusion process, which, while flexible, incurs significant computational costs due to iterative denoising and backpropagation through time (BPTT) (30] These factors limit the scalability of such algorithms to more complex environments. To address this, FQI. [I distills a one-step policy from a flow-matching policy, reducing computational cost, but requires careful hyperparameter tuning.

---

Other Approaches. Beyond above methods, alternative methods include action gradients 45 34] hybrid Markov Decision Processes (MDPs) 35 rejecctoon sampling or combinations of above strategies 27
The distinction between these methods underscores an inherent trade-off between computational simplicity and the efficiency of policy extraction. Generalized Behavior Cloning emphasizes case of implementation, often the expense of policy extraction efficiency. In contrast, reparameterized policy gradients facilitate direct policy updates but incur increased complexity. These observations highlight the necessity for further research achieve better balance between expressivity and scalability when applying iterative generative models to reinforcement learning.
Preliminaries
3.1 Reinforcement Learning
Consider the Markov Decision Process (MDPs) defined by a 5-tuple M (S,A,,, where E IR" and A E R represent the continuous state and action spaces, P(s'|s, a) : x A - A(S) denotes the dynamics distribution of the MDPs, r(s, a) x - (R) is reward function, E 10.1) gives the discounted factor for future rewards. The goal of RL is to find policy (als) A(A) that maximizes the cumulative discounted reward:
Jr ER.
Σ 2=0
2'r(st, az)
In this paper, we focus on the online off-policy RL setting, where the agent interacts with the en- vironment and collects new data into replay buffer 'D + 'D J {(s, a, r()r The replay buffer consequently maintains a distribution over trajectories induced by a mixture historical behav- ior policies πg. At the k-th iteration step, the online learning policy is denoted as Th, with its corresponding Q value function defined by:
QT* (s, ERRE.
Σ t=0
y'r(st, 5,00
(2)
and it can be derived by minimizing the TD emror [39]
arg min E(s.a,na'>> [ (8, -TM Q where TT*Q"* = = r(s, + 7 P( a' 12 (s', (s',a))
(3)
Similarly, distinguish the following key elements:
Optimal policy and Q-function: The optimal policy x² maximizes the expected cumulative
reward, and the associated Q-function Q* a) characterizes the highest achievable return.
Behavior policy and replay buffer: The behavior policy my responsible for generating the data stored in the replay buffer [22,26] Its Q-function, QT (s, a). reflects the expected return when following æz. Notably, D is closely tied to the distribution of ngs such that actions sampled from 'D are supported by those sampled from T. (i.e., a E D = a ~ TT).
Behavior-optimal policy: Among all behavior policies present in the buffer, we define kz* as the one that achieves the highest expected return, with Q-function Q a).
These definitions yield the following relationship, which holds for any state-action pair:
as) 2 T a (s, a) 2 (s
(4)
This relationship suggests that, although direct access to the optimal policy is typically infeasible, the value of the optimal behavior policy constitutes a theoretical lower bound 28 on the performance that can be achieved by policies derived from the replay buffer.

---

3.2 Flow Models
Continuous Normalizing Flows (CNF) model the varying probability paths by defining
a transformation between an initial distribution Po and target data distribution [23] This
transformation parameterized by a flow (x) governed by a learned time-dependent vector field
(x))) [5] following the ordinary differential equation (ODE):
dt vx(x) (x)),
(5)
and the continuity equation
42
dt Pr(x) v (=(()) Vz E E...
(6)
Flow Matching. Flow matching provides theoretically grounded framework for training continuous- time generative models through deterministic ordinary differential equations (ODEs). Unlike diffusion models that rely on stochastic dynamics governed by stochastic differential equa- tions (SDEs) [38] flow matching operates via a deterministic vector field, enabling simpler training objectives and more efficient sampling trajectories. The core objective is to learn neural velocity field Ug [0, 1] x Rd - Rd that approximates a predefined conditional target velocity field u(t, 2 x=//| ).
Given source distribution q(x") and target distribution p(a*), the training process involves mini- mizing the conditional flow matching objective
(23: LCRM (0) = E (-)((()))) [u,(t,z²)) x(t,x"(<²))
x >>>p,=~~g
where the linear interpolation path is defined as x' = lxt + (1 1)x2 with u(t, = x² -200
This formulation induces probability flow governed by the ODE:
dæ
v (t)
x² ~ g.
(8)
dt
which transports samples from q to p.
4
Method
In this section, we detail the design of our method. We first parameterize the policy as a flow model, where actions are generated by integrating a learned velocity field over time. For policy improvement, we model policy learning a constrained policy search that maximizes expected returns while bound- ing the distance to an optimal behavior policy. Practically, we circumvent intractable distribution matching and optimal behavior policy by aligning velocity fields with elite historical actions through regularization and implicit guidance, enabling efficient constraint enforcement.
4.1 Flow Model based Policy Representation.
We parameterize тe with (t s, a²), a sute-action-time dependent velocity field, as an actor for reinforcement learning. The policy ma can be derived by solving ODE
To as) a / vo s, a' s,
where a" ~ 12) The superscript t denotes the continuous time variable in the flow-based ODE process to distinguish it from discrete Markovian time steps reinforcement learning. (For brevity, the terminal condition at f is omitted in the notation.) The Flow Model derives deterministic velocity field to from an ordinary differential equation (ODE). However, when a is sampled from a random distribution, the model effectively functions stochastic actor, exhibiting diverse behaviors across sampling instances. This diversity in generated trajectories inherently promotes enhanced exploration in online reinforcement learning.
Recall the definition in Section 3.1 corresponding velocity fields as follows:
Following the notation of πy and πæ*, we can define the

---

Let Ug be the velocity field induced by the behavior policy ne. such that: ( a) = a.
where 5, ~ D, and a² ~ 12).
Similarly, let ug. denote the velocity field induced by the behavior-optimal policy ne. Ug- (s, a) = - a.
where a ngh, and a ~ NN r3).
4.2 Optimal-Beharee Constrained Policy Search with Flow Models
Building on the discussion in Section 3.1, where the optimal behavior policy is established as lower bound for the optimal policy, we proceed to optimize the following objective under constrained policy search setting:
0" arg max En 1Q (8, [s,a)];
(10)
s.t. D (T), SC..
Here, D(To, T8. denotes distance metric between the current policy and the optimal behavior policy distributions.
The objective is to maximize the expected reward Ee (Q" a)] while constraining the learned policy to remain within an e-neighborhood of the optimal behavior policy πæ- i.e., D(Te, TT- S This formulation utilizes the Q-function, widely used and effective approach for policy extraction, while ensuring fidelity to the optimal behavior policy.
Despite its theoretical appeal, this optimization paradigm exhibits two inherent limitations:
Challenges in computing distributional distances: For flow-based models, computing policy densities at arbitrary samples is computationally expensive, which limits the practicality of distance metries such the KL divergence for sample-based estimation and policy regularization.
Inaccessibility of the optimal behavior policy πz-: The replay buffer contains trajectories from a mixture of policies, making it difficult to directly sample from Tg- or to reliably estimate its associated velocity field, thereby complicating the computation of related quantities in practice.
4.3 A Tractable Surrogate Objective
To overcome the aforementioned challenges, we propose the following solutions:
Wasserstein Distance as Policy Constraints: We introduce a policy regularization method based the alignment of velocity fields. This approach bounds the Wasserstein distance between policies by characterizing their induced dynamic transport processes, thereby imposing direct empirical constraints the evolution of policies without requiring density estimation.
Implicit Guidance for Optimal Behaviors: Instead of explicitly constraining the policy to match the inaccessible TT* we leverage implicit guidance from past best pesso-perooming behaviors in the buffer, enabling efficient revisiting of arbitrary samples and encouraging the policy to remain within a high-quality region of the action space.
In particular, we adopt the squared Wasserstein-2 distance for its convexity with respect to the policy distribution and ease of implementation. This metric is also well-suited for measuring the velocity field between policies and enables efficient sample-based regularization within the flow-based modeling framework. In general,
define the Wasserstein-2 Distance 42 follows:
Definition 4.1 (Wasserstein-2 Distance) Given two probability measures and q on RR", the squared Wasserstein-2 distance between p and is defined as:
inf [ W >Ellip.))
J (x, u)|l= 1 yIl vllddddy,
(11)
where TTp, v) denotes the joint distributions of p and 4, 7 on R x R2 with marginals p and g. Specifically, we derive a tractable upper bound for the Wasserstein-2 distance (proof in A.1
5

---

Theorem 4.1 (W-2 Bound for Flow Matching) Let Uo and Ug- be two velocity fields inducing time-evolving distributions 7i (als) and x.. (als), respectively. Assume og is Lipschitz continuous with constant L. a' to (1 t)a². Then, the squared Wasserstein-2 distance between Te and Tga satisfies:
W (Ta, TT- s e2 ( Eawage [| uz- 112]
(12)
By explicitly constraining the Wasserstein-2 distance, the model enforces proximity between the current policy and the optimal policy stored in the buffer. This objective is inherently consistent with the generative modeling goal of minimizing distributional divergence. The regularization mechanism benefits from the representational expressiveness of flow-based models in capturing diverse,
action distributions while systematically restricting policy updates.
However, while the upper bound of Wasserstein-2 distance above is theoretically tractable, sampling directly from *g- or evaluating its velocity field remains computational barrier in practice. To circumvent this limitation, introduce an implicit guidance (I3) mechanism through the QT*", which is more readily estimable:
Bl'nTo -l((011) f (Q"H (s, cols, -a)))
(13)
a
max (QT Q
(14)
The constraint incorporates non-negative weighting function, as defined in Eq. (14). thereby establishing an adaptive regularization mechanism. A positive value of f signifies that the behavioral policy achieves superior performance relative to the current policy; under these cireumstances, the constraint adaptively regularizes the current policy towards the optimal behavioral policy.
The implicit form of the constraints in Eq. (2) enables efficient utilization of arbitrary samples from the replay buffer, thus improving sample efficiency. Moreover, by relaxing strict constraint on Wasserstein-2 distance, the modified objective enhances computational efficiency. Notwithstanding this relaxation, policy improvement guarantees remain valid, as demonstrated in the following theorem (proof in Appendix A.2
Behavior Po icy from Replay Buffer
With Guidance
Without Guldance
12-
Cenerated Semples Sourte Detriuuiion
Conerated Sumpless
Source Debricution
0.00
0.32
0
-12
0.36
-12
-5
12
-12 -5
-12
12
Figure 2: Illustration of Theorem 4.2 on a bandit toy example: (left) behavior data in the replay buffer; (middle) implicit value-guided flow matching steers the policy toward the high-performance behavior pT heatmap shows QT Qma", white lines indicate transport paths; (right) standard flow matching leads to dispersed sampling with high variance under limited flow steps.
Theorem 4.2 (Weighted CFM) Let TA (als) be the current policy induced by velocity field Ua.. and non-negative weighting function with x Q T Minimizing the objective [3] yields an improved policy distribution:
(a/s) f(s, u)s. (a|s)
(15)
2(s)
where Z(s) LAF -Tx(als) da is the normalization factor.

---

Figure 23 shows that, as guaranteed by Theorem 4.2, flow matching with guidance can steer the policy toward the TB* even without direct sampling from For details of the toy example settings, see Appendix B.4
4.4 A Practical Implementation
Building on the theoretical developments above, we now present practical implementation of FlowRL, as detailed in Algorithm
Policy Evaluation Recall the constraint in Eq. (13), which necessitates the evaluation of both the current policy value function Q and the optimal behavioral policy value function "=." The value function QT is estimated using standard Bellman residual minimization, as described in Eq. (3) For Q leveraging the definition of TT-, we similarly adopt the following objective:
arg min ((s.a.)))))) [10 (s) T Q (s, Q
(16)
= r(s, The r(s)) (7 EODD max Q (b',
T La'ND
(17)
To circumvent the difficulties of directly evaluating the max operator, we leverage techniques from offline reinforcement learning to estimate QT Among these approaches, we adopt expectile regression [20] due to its simplicity and compatibility with unmodified data pipelines. Specifically, the value function V* and the action-value function Q are estimated by solving the following optimization problems:
arg min (( [1 (Q (s, - (s))( 4
(18)
arg min E (s (Ir 7V=0' (s') Q (s. 0))² n
(19)
where = =1-) 1(x < 0) denotes the expectile regression loss and r is the expectile factor.
Policy Extraction Accordingly, the policy extraction problem for flow-based models be formulated as the following constrained optimization:
0º arg max a 10 (s, a)]
(20)
s.t. Ea.a-Da'' Da' a'
f (Q - -2)e (1o(s,,) (a Se.
(21)
Although closed-fome solution can be derived using the Lagrangian multiplier and KKT conditions, it is generally intractable to apply in practice due to the unknown partition function 32 Therefore, we adopt a Lagrangian form, leading to the following objective:
33
26
(())= = E. E3 a (3, ((QT=--"""
- (a- 22
J.
(22)
exploration
exploitation
Where À is the Lagrangian multiplier, which is often set as a constant in practice 21
Objective (22 can be interpreted comprising two key components: (1) maximization of the learned Q-function, which encourages the agent to explore unknown regions and facilitates policy improvement; and (2) a policy distribution regularization term, which enforces alignment with optimal behavior policies and thereby promotes the exploitation of high-quality actions.
Conceptual similarities exist between our method and both self-imitation learning 28 and tandem learning (2 [7]. self-imitation learning focuses on exploiting high-reward behaviors by encouraging the policy to revisit successful past experiences, typically requiring complete trajectories and modifi- cations to the data pipeline. In contrast, our method operates directly on individual samples from the buffer, enabling more flexible and efficient sample utilization. Tandem learning, by comparison, decomposes the learning process into active and passive agents to facilitate knowledge transfer, with primary emphasis on value learning, whereas our approach is centered on policy extraction.
7

---

Algorithm Flow RL
Require: Critic critcc critic QT value V flow model tg, replay buffer D weighting function
f
1: repeat
2:
for each environment step do
3:
a ~~ To (a)as)) r. a) DU a a',
4:
5: end for
6: for each gradient step do
7: Estimate value for no Update Q by 43
8: Estimate value for gg. Update Q by (19 update V by
18
9: Update Ug by 22
end for
11:
until reach the max environment steps
5 Experiments
To comprehensively evaluate the effectiveness and generality of FlowRL, we conduct experiments on diverse set of challenging tasks from DMControl [40] and HumanoidBench [36]. These
benchmarks encompass high-dimensione locomotion and human-like robot (Unitree H1) control tasks. Our evaluation aims answer the following key questions:
1. How does FlowRL compare to previous online RL algorithms and existing diffusion- based online algorithms?
2. Can the algorithm still demonstrate strong performance in the absence of any explicit exploration mechanism?
How does the constraint affect the performance?
We compare FlowRL against two categories of baselines to ensure comprehensive evaluation: (1) Model-free RL: We consider three representative policy parameterizations: deterministic policies (TD3 [2], Gaussian policies (SAC (13),, and diffusion-based policies (QVPO [8) the previous satte-of-the-art for diffusion- based online RL.. (2) Model-based RI: TD-MPC2 [4], a strong model-
based method these benchmarks, is included for reference only, as it is not directly comparable to model-free methods.
5.1 Results and Analysis
The main results are summarized in Figure 3. which shows the learning curves across tasks. FlowRL consistently outperforms or matches the model-free baselines the majority of tasks, demonstrating strong generalization and robustness, especially in challenging high-dimensional (e.g., the DMC dog domain, where E 222 and R:38) and complex control settings (e.g., Unitree H1). Compared to strong model-based baselines, FlowRL achieves comparable results but is much more efficient in terms of wall-clock time. Notably, both during the training and evaluation stage, we use flow steps N = 1, and do not employ any sampling-based action selection used in S [9]. Despite the absence of any explicit exploration mechanism, FlowRL demonstrates strong results, which can be attributed to both the inherent stochasticity and exploratory capacity of the flow-based actor and the effective exploitation of advantageous actions identified by the policy constraint. These findings indicate that, while exploration facilitates the discovery of high-reward actions, the exploitation of previously identified advantageous behaviors is equally essential.
5.2
A Alltion Studies
One of the central designs in FlowRL is the introduction of policy constraint mechanism. This design aims to guide the policy towards optimal behavior by adaptively weighting constraint based on the relative advantage of the optimal behavioral policy over the current policy. To rigorously assess the necessity and effectiveness of this component, we address Q3 by conducting ablation studies in which the policy constraint is omitted from FlowRL. Experimental results in Figure 4al indicate

---

GuadnupadWWlk
dagrun
dogirot
h1st hard
hfbalance, @ibalance_smpll
500
193
200
700km
Reaur
Reeur
Staps
steps
50095
Stops
walkerstand
DegWaik
HumannieWalk
hireach
crawi
1000
Retum
Ieeuum
Rubu
0.54 stoos
TM
2.5M
334
3.3M
24
03.04
slops
Figure 3: Main results. We provide performance comparisons for tasks (first column: DMC- easy/middle; second and third columns: DMC-hard: fourth and fifth columns: HumanoidBench), For comprehensive results, please refer to Appendix D All model-free algorithms (FlowRL, SAC, QVPO, TD3) are evaluated with 5 random seeds, while the model-based algorithm (TD-MPC2) uses 3 seeds. Note that direct comparison between model-free methods and the model-basee based TD-MPC2 is not strictly fair; TD-MPC2 is included just reference.
that the presence of the policy constraint leads to improvements in performance and, by constraining the current policy towards the optimal behavioral policy, enhances sample efficiency. These benefits are especially pronounced in environments with complex dynamics (e.g.. H1 control tasks from HumanoidBench)) highlighting the importance of adaptive policy regularization in challenging task settings.
dogrun
h1-walk-v0
dogrun
1000
1000
1000
Aln Comtril
Tee Hego
750
750
750
Return
Return
Return
800
500
500
250
0.5M steps
1M
1.5
1M
1.5M
2M
0.5M
1M
steps
steps
(a) Effect of the constraint: FlowRL with theconstraint achieves higher returns compared to the variant without the constraint.
(b) Sensitivity to flow steps: The number flow steps has limited effect on FlowRI. performance..
Figure 4: Ablation studies
We also investigate the sensitivity of the algorithm to different choices of the number of flow steps (N=1,5,10) 10). Experimental results in Figure 4bb demonstrate that varying the number of flow steps has only limited impact on the overall performance. Specifically, using a smaller number of flow steps does not substantially affect the final policy performance. On the other hand, increasing the number of flow steps results in longer backpropagation through time (BPTT) chains, which significantly increases computational complexity and training time. These findings suggest that FlowRL is robust to the choice of flow step and that single-step inference is generally sufficient for achieving stable and efficient learning in practice. We attribute this robustness to the learned Q-function acting as an implicit consistency signal (23): by assigning higher value estimates to desirable actions, the Q-function steers disparate sampling trajectories toward common high-value endpoints, so variation integration schedule has limited impact on realized behavior. From the flow- now-matching perspective, the objective in Eq. (23) can thus be interpreted as comprising two complementary terms: a weighted CFM term, and max-Q term that functions as an intrinsic, path-wise consistency constraint. Unlike the diffusionnflow-matchhe modeling- where few-step sampling typically relies on auxiliary explicit consistency constructs (e.g.. path-agreement constraints, distillation, or velocity averaging) max-Q objective in reinforcement
objectives used in generative

---

learning naturally provides form of self-consistency that aligns sampling paths toward high-return actions, obviating the need for additional consistency supervision.
(()) = -, a,a wno (s, -A
f(Q
Q10 - (a a 22)1 2 -
(23)
Consistency
Weighted CFM
Conclusion
We introduces FlowRL, practical framework that integrates flow-based generative models into online reinforcement learning through Wasserstein-2 distance constrained policy search. By parameterizing policies as state-dependent velocity fields, FlowRL leverages the expressivity of flow models to model action distributions. To align policy updates with value maximization, we propose an implicit guidance mechanism that regularizes the learned policy using high-performing actions from the replay buffer. This approach avoids explicit density estimation and reduces iterative sampling steps, achieving stable training and improved sample efficiency. Empirical results demonstrate that FlowRL achieves competitive performance.
Acknowledgments and Disclosure of Funding
This work was jointly supported by the Joint Funds of the National Key Research and Develop- ment Program of 200244N00002 1002), and National Natural Science Foundation of China (No.U22A2057) and Nanjing Major Science and Technology Special Project(No.2024050177,
References
[1] Richard Bellman. A markovian decision process. Journal of mathematics and mechanics, pages 679-684, 1957.
[2] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, al. A lsiionlanguageaction flow model for general robot control. arXiv preprint arXiv:2410.241666 2(3):5, 2024.
[3] Avishek Joey Bose, Tara Skhound-Sadeght Guillaume Huguet, Kilian Fatras. Jarrid Rector- Brooks, Cheng-Has Hao Liu, Andrei Cristian Nica, Maksym Korablyov, Michael Bronstein. and Alexander Tong. Se (3)-stochastic flow matching for protein backbone generation. arXiv preprint arXiv:2310.023391, 2023.
[4] Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209.14544 2022.
[5] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
[6] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 0278364924117222 2776666 2023.
[7] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[8] Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi. Diffusion- based reinforcement learning via q-weighted variational policy optimization. arXiv preprint arXiv:2405.16177 2024.
[9] Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning, arxiv preprint arXiv:2309.1698. 2023.
10

---

[10] Jiajun Fan, Shuaike Shen, Chaoran Cheng, Yuxin Chen, Chumeng Liang, and Ge Liu. Online weeghe- fine-tuning of flow matching with wasserstein regularization. arXiv preprint arXiv:2502.06001, 2025.
[1]] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. Advances in neural information processing systems, 34:20132 34-20132-20145, 2021.
[12] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[13] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on muchine learning, pages 1861-1870. Pmir, 2018.
[14] Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16828,, 2023.
[15] Longxiang He, Li Shen, Linrui Zhang, Junbo Tan, and Xueqian Wang. Diffeps: Diffusion model based constrained policy search for offline reinforcement learning. arXiv preprint arXiv:2310.05333, 2023.
[16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-666 -651, 2020.
[17] Tianying Ji, Yu Luo, Fuchun Sun, Xianyuan Zhan, Jianwei Zhang, and Huazhe Xu. Seizing serendipity: Exploiting the value of past success in off aff-policy actor-critie, 2024. URL https
//arxiv. brrg/bbs/in ab3//0/6...0....//// 02865
[18] Bowen Jing, Bonnie Berger, and Tommi Jaakkola. Alphafold meets flow matching for generating protein ensembles. arXiv preprint arXiv:2402.04884 2024.
[19] Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36: 77195-67212 2023.
[20] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning, arXiv preprint arXiv:2110.06166 2021.
[21] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
[22] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
[23] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Lc. Flow matching for generative modeling. arXiv preprint
2022.
[24] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03000 2000000000000000000 2022.
[25] Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning, pages 28825-22855. PMLR, 2023.
[26] Yu Luo, Tianying Ji, Fuchun Sun, Jianwei Zhang, Huazhe Xu, and Xianyuan Zhan. Offline- boosted actor-critic: Adaptively blending optimal historical behaviors in deep off-policy rl. arXiv preprint XrXiv:2405. 18520, 2024.
[27] Liyuan Mao, Haoran Xu, Xianyuan Zhan, Weinan Zhang, and Amy Zhang. Diffusion-dice: In- sample diffusion guidance for offline reinforcement learning. arXiv preprint 2rXiv:2407.20109, 2024.
11

---

[28] Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In Interna- tional conference on machine learning, pages 3878-3881 PMLR, 2018.
[29] Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. Advances in Neural Information Processing Systems, 34: 23283-23295, 2021.
[30] Scohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offlin rl? arXiv preprint arXiv:2406.09339 2024.
[31] Seohong Park, Qiyang Li, and Sergey Levine. arXiv:2502.02538 2025.
Flow q-learning.
arXiv preprint
[32] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. wevintage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 1911001910000077, 2019.
[33] Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pages 745-750, 2007.
[34] Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning diffusion model policy from rewards via q-score matching. arXiv preprint arXiv:2312.11752, 2023.
[35] Allen Z. Ren, Justin Lidard, Lars Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. arXiv preprint arXiv:2449.00550 2024.
[36] Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Picter Abbeel. Hu- manoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation. arXiv preprint arXiv:2403. 10506, 2024.
[37] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pages 387-395. Pmir, 2014.
[38] Yang Song, Jascha -ick-tic,,,,in, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020.
[39] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning. Journal of Cognitive Neuroscience, 11(1):126-13 134,,4,,, 1999.
[40] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 00090, 2018
[41] Alexander Tong, Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector- Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. arXiv preprint vrXiv:2302.0048) 2023.
[42] Cédric Villani al. Optimal transport: old and new, volume 338. Springer, 2008
[43] Yinuo Wang, Likun Wang, Yuxuan Jiang, Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang, Liming Xiao, Jiang Wu, Jingliang Duan, et al. Diffusion actor-critic with entropy regulator. Advances in Neural Information Processing Systems, 37:54183-5442 2024.
[44] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.0611 06/93, 2022.
[45] Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang. Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning. arXiv preprint rrXii:23305.33222 2023.
[46] Shiyuan Zhang, Weitong Zhang, and Quanquan Gu. Energy-weightt flow matching for offline reinforcement learning. arviv preprint arXiv:2503.04975 2025.
12

---

A Proofs in the Main Text
RL objective:
T argmaxQ
B
constraint policy search
Tg argmaxQ n0
8. t. W2 (TO, T
S
Upper bound for W-2 distance
Ty argmax
s.t. |lv0
U3 ||
S
Implicit guide
To = argmaxx
s.t. f Q - QT) U0o (a aa-aa)]1' al)l? Se
Lagrange form
E fa00 (s,a)) a)l El (fQQ" QTo) - (a - a3))
-
Figure 5: Theoretical sketch of FlowRI.
Here, present sketch of theoretical analyses in Figure 5 We model the policy learning constrained policy search that maximizes expected returns while bounding the distance to an optimal behavior policy. To avoid sampling from Tī. employ guided flow matching, which allows the constraint to utilize arbitrary data from the buffer. Finally, solve the problem using Lagrangian relaxation.
A.1
Proof for Theorem 4.1
Before the proof, we first introduce the following lemma 110]
Lemma 1 :let vif (xa) and (2o) be the two different flow maps induced by of and v starting from , and assume oy are Lipschitz continuous with constant Define their difference as A,(x")= = of (x:") -ig (2*)). (For notational consistency, we denote the time variable as superscript.) Then the difference satisfies the following inequality:
d Az)))) s (+f(())) 2s( (a5"(())))))(((((())"" LI dt
13

---

By rewriting equivalently, we have:
d -
di A,00)) vi((((())) -
+ u (1 (vf
6.(1))
(p(()
Since oi is Lipschitz continuous in a with constant L, have:
(1) - cv)il s L|x vl
By Lipschitz continuity,
16 (t)ll & (I)((=))
Then,
d A (t),(t) + -I4,(=*)
dt
This concludes the proof of the inequality satisfied by the difference of the two flow maps.
Let to and Ug. be two velocity fields that induce time-evolying distributions sg(als) and T (als). respectively((we omit the superscript
for policy distributions, i.e., mo(als) = (als)))) Assume ug- is Lipschitz continuous with constant L. Then, define f(t) = [[,,(z))l, by Lemma 1,we have:
di. ((1) = [10. LS(),
where (,( uo (s, Ug.. Then, we have,
d (e (--5(()) S -0-11116000)
dt
Then we can get (by simply intergrating from 0 to t both side and multiplying e Md):
- 5(0) S (
The initial policy distribution a p(aa) is shared between the two velocity fields, so f(0) Therefore,
f(t) S eto S' e -mmm e-bm)j0
At f 1,
f(1) se" //' - (0 (s, m) ug- ||dm.
By taking the expectation and using Jensen's inequality:
E [S(1))] S 2² S (1oos ---- d1
And use the definition of the Wasserstein-2 distance:
/ 3
W (a. TT-)
inf "EII(xo.000)
|| v112dy(x,)).
where [T m.-) denotes the set of all couplings between my and क.. Construct the following coupling 7 and define:
a (zo)
a33- wi.. (20).
14

---

By definition, the coupling is defined via the joint distribution of (a TT, TT-) induced by ag Po- So, for any coupling %
W TO. TT-
| 1r--5ll*e =11
With the constructed coupling substituted, we have
J
J 1 yll:dv(z.y) v) E,," [lv)((a)) - 2 (a") E. [f(1)9]
Recall that the flow-based policy models transport the initial distribution po to the final policy distributions To and T- at t 1. The squared Wasserstein-2 distance between To and IT can be bounded as
W2 xT-) s E. [1(1)]]
(24)
Thus,
(To, S e24 S: Bawny (lloll, us- (s,
(25)
A.2 Proof for Theorem 4.2
The weighted loss can be written as:
(w( [ SDD
p(s) [ S.aDD [(s, a) (x(a)s) ||v0(,, a', aa) ds
where p(s) is the state distribution in replay buffer, a ~ NNO, 13) ~ U(0, 1), at = +t +t+((1ttcc) Assuming the weighted policy distribution is: (a'ls) [(8, a) (als))
where 2(s) = [ Jaaan
f(s, a) (alss) da.
2(s)
Substituting above Th+ (a'ls) into the loss function, we have:
Lw (0) [ JaND p(s) (ss)
[ aODD T<+i(a'[s) a' (a aa) lda
The expectation form:
(w = EN
(m (()) [100ss a), (a (a]
The gradient of Lw(() is:
VoCw(O) EgnD amte)r [x() V [1 a, 1) (a a
Z(s) does depend on 0, that means, minimizing Lw (0) is equivalent to minimizing the expected loss under the new distribution T (als), provided that our assumption holds.
B Hyperparameters and Experiment Settings
In this section, we provide comprehensive details regarding the implementation of FlowRL, the baseline algorithms, the experimental environments. All experiments are conducted on a single NVIDIA H100 GPU and Intel(R) Platinum 8480C CPU, with two tasks running in parallel on the GPU.
B.1 Hyperparameters
The hyperparameters used in our experiments are summarized in Table 1 For the choice of the weighting function, we use f(x) 11xx) exp(x), where II(x)) the indicator function, i.e..
1, if
I[(x) = o, otherwise
For numerical stability, the Q function is normalized by subtracting its mean exclusively during the computation of the weighting function.
15

---

Table 1: Hyperparameters
Hyperparameter
Value
Optimizer
Adam
Critic learning rate
3 x 10-
Actor learning rate
x 10-1
Discount factor
0.99
Batchsize
256
Hyperparameters
Replay buffer size
10
Expectile factor т
0.9
Lagrangian multiplier X
0.1
Flow steps N
1
ODE Slover
Midpoint Euler
Network hidden dim
512
Value network
Network hidden layers
3
Network activation function
mish
Network hidden dim
512
Policy network
Network hidden layers
Network activation function
elu
B.2 Baselines
In experiments, have implemented SAC, TD3, QVPO and TD-MPC2 using their original code bases and slightly tuned them to match our evaluation protocol to ensure a fair and consistent comparison.
For SAC 13,, utilized the open-source PyTorch implementation, available at https://githubb wttpraant//ppooorro-o-tt----eeoe
TD3 [1] was integrated into experiments through its official codebase, accessible at https comtsuuu.omm//ffuu
QVPO 8 integrated into our experiments through its official codebase, accessible at https
TD-MPC2 14
employed with its official implementation from bicklashansenntdmppc and used their official results.
Figure 6: Task domain visualizations
16

---

B.3
Environment Details
We validate our algorithm on the DMControl 40 and HumanoidBench [36], including the most challenging high-dimensional and Unitree HI humanoid robot control tasks. On DMControl, tasks are categorized into DMC easy & middle (walker and quadruped domains), and DMC hard (dog and humanoid domains). On HumanoidBench, we focus on tasks that do not require dexterous hands.
Task
State dim
Action dim
Walker Run
24
Walker Stand
24
6
Quadruped Walk
78
12
Humanoid Run
67
24
Humanoid Walk
24
Dog Run
223
38
Dog Trot
223
Dog Stand
223
38
Dog Walk
223
Table 2: Task dimensions for DMControl.
Task
Observation dim
Action dim
HI Balance Hard
77
19
HI Balance Simple
64
H1 Crawl
51
19
H1 Maze
51
19
H1 Reach
57
19
H1 Sit Hard
64
Table 3: Task dimensions for HumanoidBench.
B.4 Toy Example Setup
We consider 2D toy example follows. The behavior policy is Gaussian mixture model with 10 components, each with mean
μ (10 cos(k22/)))) 10sin(2xk/10)) h: 0,1,....,
and covariance I. The initial distribution is Gaussian (NO, 0), I). Q QT* is defined
x (0, 8.66) 3,
600
and (xx I(x:) Flow steps N = 5.
C Limitation and Future Work
In this work, we propose a flow-based reinforcement learning framework that leverages the behavior- optimal policy constraint. Although competitive performance is achieved even without explicit exploration, investigating efficient adaptive exploration mechanisms remains a promising direction for future research.
D More Experimental Results
17

---

GuidrupedWalk
dogrun
humeroktrun
h1siL_hard "aard
Mibellancee simple
290
120
steps
3.54
1222
3.30 0.3M
1M
staps
kersland
2000
DogWWaik
1033
HumannaldWaik
hfreach
hicom
100300
0.12M
0.2M
1.90
90
150
staps
stapps
1020
We kenRun
dagsland
h1balencee hard
h-1 maze
Puatmm
Suoum
Reum
120
223
200
0.725
3299
steps
5V
01:4
1:04
Flowil
5AC
TD-V8C2
Figure 7: Experimental results are reported on 12 tasks drawn from HumanoidBench and DMC-hard, 3 tasks from DMC-easy & middle.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, the paper discusses its limitations in the appendix. Although compet- itive performance is achieved without explicit exploration mechanisms, the exploration regularization mechanism remains an important direction for future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors
18

---

should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Yes, the paper provides the full set of assumptions and complete proofs for all theorems in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, the paper fully discloses all information needed to reproduce the main experimental results. Pseudocode provided the main text, and all experimental settings, hyperparameters, and baseline details are included in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
19

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, the code will be made available after the open-source approval process is completed.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
swi.iiossiippcccc.oo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
20

---

Answer: [Yes]
Justification: Yes, all hyperparameters and experimental setup details necessary to under- stand the results are provided the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Yes, for model-free algorithms, five random seeds are used, and for model- based algorithm, three random seeds are used. All results are presented as mean + standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the eтror bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, the appendix provides detailed information about the specific computa- tional devices used for the experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
21

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification:
We make sure the code anonymous
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, the appendix discusses both potential positive societal impacts and limitations of the work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
22

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, all sources of data and code are properly credited in the appendix, with licenses and terms of use clearly indicated.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (c.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pataasess.paperswtt paperswi thcode. cow/coo////ooooo /atasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
ANs]
Justification:
23

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
24


Reviews:
[{"review_id": "VhyTjBY9aP", "reviewer": "Reviewer_Mh1b", "summary": "This paper introduces FlowRL, an RL algorithm that uses flow-based policies to generate actions via ODEs and regularizes them with a Wasserstein-2 constraint toward high-reward behaviors from the replay buffer. This avoids density estimation and aligns the policy with value-based objectives. The method is efficient, simple to implement, and shows strong results on DMControl and HumanoidBench.", "questions": "1. How much does FlowRL rely on getting $Q^{\\pi_{\\beta*}}$ right? If that estimate is noisy or off, does it hurt performance? Have you tried other ways to compute it?\n2. The method works without explicit exploration, but how does it do in sparse-reward tasks? Does the built-in stochasticity go far enough?\n3. Can FlowRL scale to things like vision inputs or language-conditioned tasks? Any challenges there?\n4. This feels close to offline RL—Since it relies on value-weighted behavior from the replay buffer,could it work directly in that setting?\n\nI’d consider raising my score with more insight into the above questions.", "limitations": "yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "#### **Strengths**:\n\n1. The paper is easy to read, and the method is pretty straightforward. No need for complex sampling or backprop through time, which keeps things practical.\n2. Compared to diffusion-based methods, FlowRL is much faster thanks to its one-step action generation. That also makes training more stable.\n3. Using Wasserstein-2 as a constraint is a nice touch—it lets the policy stay close to good behaviors without needing to compute densities.\n4. The experiments are solid. It works well on tough tasks like DMControl’s Dog and HumanoidBench, and fits cleanly into standard actor-critic pipelines.\n\n#### **Weaknesses**:\n\n1. While the paper presents a well-executed combination of known ideas, the overall novelty feels incremental. The use of flow-based policies and W2 regularization is thoughtfully integrated, but the core principle—guiding the policy using high-value replay buffer actions—follows a familiar pattern. I’d encourage the authors to better clarify what’s fundamentally new compared to prior works like SIL, AWR, and recent flow-based methods like FQL or DACER.\n\n2. The theoretical part (bounding W2, showing improvement via regularization) is correct but fairly standard. It mostly formalizes the intuition behind the method, rather than offering new theoretical insights.\n3. There’s no explicitly designed exploration strategy, although the stochastic nature of the flow-based policy does introduce some implicit exploration. This may still be a limitation in sparse-reward or exploration-heavy settings.", "was_revised": true, "final_justification": "The rebuttal clearly addressed my concerns about novelty, theory, exploration, applicability, and Q-function reliance. I don’t see any major issues left, and I recommend acceptance. I’d like to see the clarifications and new results included in the final paper.", "rebuttal": "Dear Reviewer,\n\nThank you for your thoughtful and detailed feedback on our paper. We are encouraged that you found our work to achieve solid performance on challenging tasks and that you found the method practical and easy to follow.\n\nGiven your reservations about our paper’s novelty and other questions, we would like to take this opportunity to address them in detail below.\n\n### **Our Core Contribution: A Principled Framework for Flow-based Policy Improvement(w1)**\n\nOur main contribution is **a stable and effective policy improvement framework for generative and expressive policies, based on constrained optimization**. We would like to respectfully clarify why we believe this contribution is foundational rather than incremental. Our framework is designed to **solve a fundamental conflict** in online reinforcement learning: the mismatch between a generative model's static, distribution-matching objective and the dynamic, return-maximization objective of RL. The Wasserstein-2 (W2) constraint creates a trust region around high-value behaviors, which stabilizes the expressive flow-based policy. This allows it to **safely maximize the Q-function to surpass the data, rather than merely imitating good actions**. This principled approach allows our method to achieve a combination that has been elusive for prior methods: solid performance on challenging tasks while maintaining efficient inference. The significance of this is best understood by a direct comparison to the works you mentioned:\n\n| Compared To | Core Distinction & FlowRL's Advantage |\n| :--- | :--- |\n| **AWR** | **Different theoretical foundations.** AWR is derived as the closed-form solution to a **forward-KL regularized objective** which **is a form of weighted behavioral cloning** with a simple Gaussian policy. This approach induces overly **pessimistic updates** in the online setting, as it fundamentally anchors the policy to the quality of existing data. In contrast, FlowRL's constrained optimization is explicitly designed to avoid this policy-confining pessimism with expressive policies and enable robust improvement. |\n| **SIL** | **Different conceptual, implementational, and architectural approach.** FlowRL is  motivated by resolving the core objective mismatch for expressive generative policies. To achieve this, we introduce a principled constrained optimization framework that not only stabilizes the expressive flow-based policy but operates at a much more sample-efficient **transition level**. SIL directly add an auxiliary imitation loss to A3C loss and need to **collecting complete, high-return trajectories** which is often impractical and sample-inefficient. | |\n| **DACER** | **Different role for the generative model and computational cost.** DACER uses the generative model as a simple reparameterization tool(as we discussed in related work), which leads to a reliance on computationally expensive and potentially unstable Backpropagation Through Time (BPTT). FlowRL leverages the full distribution-fitting power of flow models and  avoids BPTT. |\n| **FQL** | **Different learning paradigms and use cases.** FQL is a pure offline algorithm and need a separate policy distillation step. In contrast, FlowRL is an online method that natively achieves one-step inference by directly learning an expressive flow-based policy, completely bypassing the need for distillation. |\n\n### **On the Role of Our Theoretical Analysis(w2)**\n\nWe wish to clarify the purpose of our theoretical analysis. Its primary goal was to **provide a solid mathematical foundation** for our method's design. This foundation forges a principled connection between our framework and its implementation by proving that W2 regularization guides the policy towards high-reward distributions, which in turn informs our loss function design.\n\n### **On Exploration and Sparse-Reward Performance（w3/q2）**\n\nThank you for this valuable question, which gives us the opportunity to elaborate on our design choices regarding exploration.\nOur framework is **fully compatible** with existing exploration techniques (e.g., RND [2] and random action noise); we intentionally omitted them in the paper to provide a **clean and focused evaluation** of our core method's inherent capabilities.\n\nTo directly address your concern about exploration, **we have conducted two new sets of empirical results.**\n\n**1. Performance in a Sparse-Reward Setting:** To directly test performance in an exploration-critical setting, we modified the Metaworld \"coffee-button\" task (trajectory length=200) to provide  a sparse reward (+1 for success, 0 otherwise) and evaluated FlowRL against SAC, which is renowned for its strong exploration.\n\n| Algorithm | Success Rate(sparse) (Mean± std)|\n| :--- | :--- |\n| FlowRL  |  93%± 12% |\n| SAC |  73% ± 31% |\n\n**2. Quantitative Analysis of Policy Stochasticity:** We estimated the policy entropy on the highly exploration-heavy `dog-run` task using a KNN-based estimator [1] and compared it to the target entropy of a well-tuned SAC agent.\n\n| Algorithm | Average Policy Entropy |\n| :--- | :--- |\n| FlowRL (Ours) | ~ -31.27 |\n| SAC (Target Entropy) | ~ -38 |\n\nThe **higher entropy (less negative)** provides quantitative evidence that our policy is inherently stochastic and exploratory. \n\n---\n\n### **Question 1: How much does FlowRL rely on the Q-function estimate?**\n\nEstimation of $Q^{\\pi_\\beta*}$ is built upon the **principled and empirically validated technique of quantile regression**, as introduced in [3]. The stability and effectiveness of this methodology are well-established, having been corroborated by prior works [4, 5].\n\nWhile we start from this strong foundation, we also recognize that sensitivity to value estimation error is a common challenge for all off-policy algorithms that rely on bootstrapping, rather than a vulnerability unique to our method. To address this, our implementation is further fortified with the robust technique :Clipped Double Q-learning[6], which provides a strong safeguard against the well-documented problem of value oestimation.\n\nFurthermore, our framework is not rigidly dependent on a single estimator; substituting alternatives like Extreme Q-Learning [7] yields comparable performance. Therefore, the framework's reliance on the Q-function is built upon a foundation of proven, robust, and interchangeable components, ensuring its reliability is on par with state-of-the-art methods.\n\n### **Question 3: Can FlowRL scale to vision or language-conditioned tasks?**\n\nThis is an excellent question. Our framework is designed for broad applicability, as its **core algorithm is agnostic to the input modality**.\n*   **Orthogonal Contribution:** Our focus on state-based benchmarks was a deliberate methodological choice to isolate our core contribution, a common practice in foundational reinforcement learning research. We recognize that vision introduces distinct challenges, such as learning effective representations and data augmentation. Crucially, our contribution is orthogonal to these perception challenges.\n*   **Complementary Nature:** FlowRL can be seamlessly integrated with state-of-the-art techniques developed for these specific issues, acting as a complementary component.\n*   **Experimental Evidence:** To validate this, we ran a new experiment on a visual control task. The results below show FlowRL significantly improves a strong visual RL baseline (DrQ-v2), confirming its complementary value. To ensure a fair comparison and isolate the benefits of our policy improvement, we uniformly use a one-step TD update for both the baseline and our method.\n\n| Environment (Visual) | DrQ-v2 @1M frames | DrQ-v2 + FlowRL @1M frames |\n| :--- | :--- | :--- |\n| Quadruped-Walk | 472 ± 15.2 | **613 ± 17.3** |\n\n### **Question 4: Could FlowRL work in the offline RL setting?**\n\nThank you for this insightful question. We would like to clarify that FlowRL is designed as a purely **online** algorithm. The distinction lies in the primary research problem each addresses:\n*   Our work concentrates on **policy extraction** within the online loop.\n*   Offline RL need to address the **distributional shift** problem, typically by preventing Q-function overestimation for out-of-distribution (OOD) actions.\n\nWhile our core idea is adaptable, making it effective for the offline setting would require **integrating specific conservatism mechanisms** (e.g., from CQL [8]) to handle the OOD problem.\n\n---\nOnce again, we thank you for your valuable time and constructive feedback, which has pushed us to articulate our core contributions more clearly. We hope to have the opportunity to revise our manuscript based on your suggestions.\n\n### **References**\n[1] *Sample estimate of the entropy of a random vector* (Problems of Information Transmission, 1987) by Kozachenko & Leonenko.\n\n[2] *Exploration by Random Network Distillation* (ICLR, 2019) by Burda et al.\n\n[3] *Offline Reinforcement Learning with Implicit Q-Learning* (ICLR, 2021) by Kostrikov et al.\n\n[4] *Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic* (ICML, 2024) by Ji et al.\n\n[5] *Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL* (ICML, 2024) by Luo et al.\n\n[6] *Addressing Function Approximation Error in Actor-Critic Methods* (ICML, 2018) by Fujimoto et al.\n\n[7] *Extreme Q-Learning: MaxEnt RL without Entropy* (ICLR, 2023) by Garg et al.\n\n[8] *Conservative Q-Learning for Offline Reinforcement Learning* (NeurIPS, 2020) by Kumar et al."}, {"review_id": "A1ufYLy93W", "reviewer": "Reviewer_zBLY", "summary": "This paper introduces FlowRL, a novel method that integrates flow-based policy representations into online reinforcement learning. The approach uses the Wasserstein-2 distance between the current policy and a behavior-optimal policy as an implicit constraint objective. Experiments on DMC-easy/middle/hard and HumanoidBench demonstrate the method’s effectiveness and efficiency.", "questions": "1) How is Lipschitz continuity of $v_{\\beta}$ on $a$ ensured? Does the model include regularization techniques?\n2) The FlowRL optimization includes hyperparameters such as $\\epsilon$ and $\\lambda$. How were these values selected? Experimental justification is needed.\n3) The paper emphasizes reducing computational cost but lacks quantitative comparisons with other distance metrics such as KL divergence and Euclidean distance.\n4) The flow step sensitivity analysis is only conducted on the high-performing task Dogrun. It remains unclear whether adjusting flow steps impacts performance on other tasks like Dogtrot or Humanoidwalk.\n5) Typos: In the annotation of Figure 4(a), line 2, “theconstraint”", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1) This paper offers a valuable perspective on incorporating highly expressive policies into RL, which is both interesting and critical. It effectively employs the Wasserstein-2 distance as an implicit constraint, offering a more efficient alternative to the commonly used but computationally intensive KL divergence.\n\n2) Experimental results on normalized scores and training/inference time clearly demonstrate the advantages of FlowRL.\n\n3) The paper is well-structured and clearly written, making it easy to follow.\n\n4) The theoretical analysis of the Wasserstein-2 distance and flow matching enhances the clarity and soundness of the method.\n\n**Weakness**\n\nSome claims are insufficiently supported by the experiments. Please refer to the questions for details.", "was_revised": true, "final_justification": "My concerns are well addressed during the rebuttal. FlowRL provides a valuable approach to integrating highly expressive policies into RL using the Wasserstein-2 distance, and extensive experimental results demonstrate its effectiveness.", "rebuttal": "Dear Reviewer,\n\nWe sincerely thank you for your positive evaluation and for your insightful questions. You have identified key areas where our claims can be further substantiated, and we appreciate the opportunity to provide a more detailed justification for our methodological choices and to supplement our experimental results.\n\n### **Response to Q1: On the Lipschitz Continuity of the Velocity Field**\nThis is an excellent theoretical question. We are happy to provide a more formal justification. The Lipschitz continuity of the velocity field $v_{\\beta}$ is a well-grounded assumption that follows directly from its construction within our framework and the intrinsic properties of our network architecture.\n\nFirst, the velocity field $v_{\\beta}$ is derived by the historical behavior of the online policy; it is constructed from the past outputs of our network $v_{\\theta}$. Consequently, the functional properties of $v_{\\beta}$, such as continuity, are determined by the properties of the network $v_{\\theta}$ itself.\n\nSecond, the network $v_{\\theta}$ is intrinsically Lipschitz continuous due to its architecture. Our network is a Multi-Layer Perceptron (MLP) composed of fundamental operations (linear layers and common activations like ReLU/GeLU) that are known to be Lipschitz continuous. As the composition of Lipschitz functions is also Lipschitz continuous, the entire network is guaranteed to be a Lipschitz function over any compact domain[1]. This property is a cornerstone of modern deep learning theory, ensuring stable and well-behaved models, and is extensively relied upon in foundational literature[1,2,3].\nIn summary, $v_{\\beta}$ inherits its functional properties from the network architecture used to generate it. Since this architecture is intrinsically Lipschitz continuous, the assumption for $v_{\\beta}$ is well-founded，and we use none of regularization technique. We will add a brief note to the methodology section to make this justification explicit.\n\n### **Response to Q2: On Hyperparameter Sensitivity and Tuning**\nYou raise a crucial point regarding hyperparameter sensitivity. A robust algorithm should not be overly dependent on fine-tuning.\n1.  **On the Role of $\\epsilon$ (A Deliberate Design Choice):** We would like to clarify a key design choice in our optimization that enhances the method's practicality. While $\\epsilon$ serves as a constraint boundary in the Lagrangian formulation, our practical implementation fixes the dual variable $\\lambda$. In this setup, the policy gradient is **independent of the specific value of $\\epsilon$**. This was an intentional choice to **remove $\\epsilon$ as a tunable hyperparameter**, simplifying the method's application. We will revise our methods section to make this important point clearer.\n2.  **On the Sensitivity of $\\lambda$:** The parameter $\\lambda$ balances value maximization against W2 regularization. To quantitatively address your concern, we have conducted a new, comprehensive sensitivity analysis for $\\lambda$ on the challenging `Dog-run` task.\n\n| $\\lambda$ | Return (Mean ± Std) |\n| :--- | :--- |\n| 0.01 | 340.58 ± 29.14 |\n| 0.1 | **384.86 ± 13.76** |\n| 1 | 352.57 ± 15.01 |\n| 10 | 5.35 ± 0.47 |\n\nAs the results demonstrate, FlowRL exhibits robust performance across a wide range of $\\lambda$ (from 0.01 to 1, spanning two orders of magnitude). Performance only degrades at an extreme value ($\\lambda=10$), where excessive regularization  prevents the policy from improving beyond the initial data distribution. This analysis confirms that $\\lambda$ is not a critical hyperparameter and is easy to tune.\n\n\n### **Response to Q3: Quantitative Comparison of Distance Metrics**\nThis is a very practical question that directly addresses one of our method's core motivations. Our choice was a deliberate one, made for both principled and practical reasons, which becomes clear when contrasted with common alternatives.\n\n**Regarding KL Divergence:** While theoretically sound, explicitly minimizing the KL divergence in an online RL setting is computationally prohibitive. The primary bottlenecks are:\n  1. Target Distribution Cost: It requires explicitly computing or sampling from the target policy,$\\pi_\\beta^*$, at each update step.\n  2. Log-Probability Cost: Evaluating the exact `log_prob` of actions under a flow-based policy requires computing the log-determinant of the flow model's Jacobian. Backpropagating through this operation is extremely costly.\n  3. Sampling Overhead: Reliably estimating the KL divergence requires a significant number of sampling steps, further adding to the computational burden and introducing variance.\n\n\nTo provide quantitative evidence, we ran an ablation study comparing our method to a KL-constrained baseline where the KL divergence was estimated using a 5-step process.\nFirst, the computational overhead of the KL-based approach is significant, as shown by the wall-clock time for 1 million environment steps:\n\n| Distance Metric | Training Time (1M steps) |\n| --- | --- |\n| W2 Upper Bound (Ours) | ~8.1h |\n| Exact KL (5-step estimation) | ~11.4h |\n\nFurthermore, the learning process for the KL-constrained agent was highly erratic, with performance oscillating significantly.This is because the high variance from the KL divergence estimation introduces substantial noise into the learning signal, corrupting the policy update and leading to unstable training.\n\n| Env Steps | 0.5M | 0.7M | 1M | 1.5M | 2M |\n| --- | --- | --- | --- | --- | --- |\n| Return (KL Agent) | 98.8 ± 12.5 | 18.5 ± 9.2 | 91.5 ± 7.8 | 146.0 ± 29.7 | 179.5 ± 7.8 |\n\n(**We sincerely apologize that we cannot provide a more intuitive learning curve visualization, as conference guidelines prohibit adding new figures during the rebuttal period**).\n\n**Regarding Euclidean Distance:** We wish to clarify a key aspect of our method: computationally, the loss we optimize is in fact a standard Euclidean (L2) distance.\nThis is not an arbitrary choice. In our work, we derive a tractable upper bound on the true W2 distance, which simplifies to the Euclidean (L2) distance in the space of the policy's velocity fields. This approach provides: the computational simplicity of an L2 loss, combined with the geometric and theoretical grounding of optimal transport.\n\n### **Response to Q4: Flow Step Sensitivity Analysis**\nThank you for this valuable suggestion. To address your comment, we have expanded our sensitivity analysis for the number of flow steps to the **`Dog-trot`** and **`Humanoid-walk`** tasks. The results confirm that performance remains robust across a reasonable range of flow steps, reinforcing our claim of efficiency.\n\n| Steps | Dog Trot | HumanoidWalk|\n| ----: | ---------: | ----------: |\n|     1 |   764±33.78 |    797±22.85 |\n|     5 |   788± 41.36|    792±23.45 |\n|    10 |  774± 27.83 |   815±24.70 |\n\nFurthermore, we would like to highlight that this empirical finding is consistent with the underlying theory. The principle that a small number of flow steps is sufficient for effective policy generation is not task-specific and should apply broadly. For a more in-depth theoretical analysis on this point, We respectfully suggest that you refer to our discussion in response to Question 7 from Reviewer JNtZ, where we elaborate on this further.\n\nFinally, thank you for catching the typo (\"theconstraint\") in the caption for Figure 4(a). We will correct it. We are grateful for your detailed feedback.\n\nThank you again for your valuable review. **We are happy to engage in further discussion if anything remains unclear.**\n\n###  **References**\n[1] Lipschitz regularity of deep neural networks: analysis and efficient estimation (Neural Information Processing Systems, 2018) by Scaman, K., & Virmaux, A.\n\n[2] Regularisation of Neural Networks by Enforcing Lipschitz Continuity (arXiv, 2021) by Gouk et al.\n\n[3] Efficient and accurate estimation of the Lipschitz constant of deep neural networks (NeurIPS, 2019) by Fazlyab et al."}, {"review_id": "EWo1Sm0SNu", "reviewer": "Reviewer_JNtZ", "summary": "The authors propose a new method, **FlowRL**, for incorporating Flow Matching into the reinforcement learning (RL) framework using a guided expectile loss.", "questions": "1. Does this framework support the notion of multiple tasks within the same environment?\n2. How sensitive is the method to action and observation perturbations?\n3. Why is the number of steps truncated in *QuadrupedWalk*? Is this due to convergence?\n4. How large is the replay buffer? Does its size change during training? Are old examples replaced?\n5. What are the potential failure cases or environments where FlowRL may not perform well?\n6. In Figure 4, what exactly does the \"policy constraint change\" refer to? Is it referring to removing Equation 13 from Equation 22?\n7. Why does the method appear to be largely independent of the number of flow steps, even though flow matching typically improves with more steps? Can you hypothesize why?", "limitations": "The authors did not discuss limitations properly in the paper", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. Novel approach to incorporating Flow Matching into the RL framework.\n2. Strong empirical performance across tasks.\n3. Inference time is superior to previous methods.\n4. Demonstrates robustness with respect to the number of flow steps.\n\n---\n\n### Weaknesses\n1. No proper discussion of limitations. For example, do known issues with diffusion models also apply here?", "was_revised": true, "final_justification": "I think that the paper suggest a new strong method for online RL and would be beneficial for community.", "rebuttal": "We sincerely thank you for your thoughtful review and positive assessment of our work. Your questions are highly insightful and help us to clarify the contributions and nuances of our framework. We are happy to address each point in detail. \n\nThank you for raising a  point about limitations. As iterative generative models, both flow and diffusion models face similar challenges, while diffusion models introduce more complexity of their noise scheduling mechanisms.  We will add a discussion on these limitations to the manuscript.\n\n### **Response to Q1: Support for Multiple Tasks**\nThat is an excellent question regarding the generality of our framework. We would like to clarify the precise scope of our contribution.\nOur work **focuses on improving the fundamental reinforcement learning algorithm**. We address the core mechanism of how to represent a policy and extract value from experience data in the most efficient and stable way possible.\nThe challenges of multi-task learning—such as learning effective task embeddings, developing general representation learning schemes, and  scaling networks—are **orthogonal** to our contribution. FlowRL is designed to serve as the core policy learning component within such problems. Its clear separation of concerns allows our method to serve as a powerful and compatible building block for future multi-task research.\n\n###  **Response to Q2: Sensitivity to Perturbations**\nThank you for this question. It is important to first clarify the scope of our work to address a potential misunderstanding.\nThe problem you've described, concerning sensitivity to action and observation perturbations, falls squarely within the domain of **Non-Stationary Reinforcement Learning.** Our research, in line with standard practice for foundational RL algorithms, focuses on validating our core contributions on established, stationary benchmarks. This allows for a controlled and **fair comparison against prior methods that follow the same evaluation protocol**.\nHandling such perturbations typically requires a distinct set of specialized techniques that are complementary to, and orthogonal to our core algorithm. For instance, methods in that domain often employ recurrent architectures, system identification modules, or domain randomization.\n\n###  **Response to Q3: Truncation of Steps in Quadruped-Walk**\nThank you for this  observation. Your question highlights an important detail about our experimental protocol.\nFor the relatively simple tasks within the DMC-easy/middle suites, we adopted a uniform training length of 0.25M steps to ensure an efficient allocation of our limited computational resources. Our rationale was that within this timeframe, our method had already achieved performance comparable to the reported return of prior model-free algorithms[1] (around 800 on `quadruped-walk`).\n\nHowever, to provide a more definitive answer and explore the full potential of our method, we extended the experiment to 1M steps. We found that performance did not just convergence but  improved.\nThe result is as follows:\n| Environment | FlowRL @ 1M Steps |\n| --- | --- |\n| quadruped-walk |  978.08 |\n\n###  **Response to Q4: Replay Buffer Details**\nTo ensure a fair comparison and highlight the contributions of our algorithm itself, we followed the most standard and widely-used configuration for off-policy RL.\n- Size and Update Mechanism: The replay buffer has a fixed size of 1 million transitions. It operates as a First-In, First-Out (FIFO) queue. Throughout training, its size remains constant. Once the buffer is full, for each new transition added, the oldest one is discarded.\n\n###  **Response to Q5: Potential Failure Cases and Limitations**\nThis is an excellent and necessary question, as every algorithm has its application boundaries. \nThe primary limitations of FlowRL are intrinsically linked to the challenges inherent to off-policy learning itself. Therefore, FlowRL may not perform well in scenarios where off-policy methods are generally unsuitable. A prominent example is in large-scale Reinforcement Learning from Human Feedback (RLHF) for LLMs, where the industry has largely converged on on-policy algorithms like PPO and its variants.\n\n###  **Response to Q6: Clarification of Figure 4**\nYes, your understanding is perfectly correct. We will clarify this in the figure caption to make it unambiguous.\n\n### **Response to Q7: Independence from the Number of Flow Steps**\n\nThis is a very **deep and insightful question** that touches upon a core advantage of our framework. \n\nIn standard flow-based generative models, generation quality typically improves with more integration steps. Achieving high-quality, few-step sampling is a key research area, with established techniques including distillation (ReFlow), using an averaged velocity field (MeanFlow), or enforcing consistency (short cut models[2]).\n\nRather than enforcing consistency through auxiliary mechanisms, our framework **achieves consistency implicitly**.\nDiverging from pure generative modeling, where consistency is typically defined by the requirement that different sampling paths yield outputs of similar quality, we argue that in the context of reinforcement learning, this notion of consistency is naturally redefined by the agent's primary objective: maximizing cumulative return. Consequently, our requirement is that **different sampling paths should all produce actions that maximize the Q-value**.\nQ-function itself serves as this consistency condition, guiding the policy toward a desirable outcome regardless of the number of integration steps. From a generative modeling perspective, our objective function can be understood as explicitly encoding this idea:\n\n$$\n\\mathcal{L} = \\underbrace{Q(s,a',d)}\\_{\\text{consistency}} - \\lambda \\ \\underbrace{f \\cdot \\left\\Vert v - (a - a^0) \\right\\Vert^{2}}\\_{\\text{Weighted CFM}}\n$$\n\nIt is important to clarify the notation Q(s, a', d). The number of flow steps, `d`, is not an explicit input to the Q-network. However, the action `a'` is generated from the policy $\\pi_\\theta$ using `d`  integration steps, making `a'` functionally dependent on  `d`  . Consequently, the Q-function is implicitly conditioned on  `d`   through its dependence on the action `a'`, our notation serves to make this crucial dependency explicit in the formulation.\n\nTo further substantiate this viewpoint with empirical evidence, we adopt a training paradigm analogous to that of shortcut models[2], where the number of path integration steps is randomized during training. Specifically, during the training phase that involves action sampling (i.e., for both actor and critic updates), we randomly sampled the number of flow steps, `d`, from the range [1, 5] for each generated action. To assess the resulting policy's robustness, we then evaluated its performance under two distinct inference conditions during environment interaction: one using a fixed number of steps, and another where `d` was also randomly sampled. This experimental design allows us to decouple the training and inference conditions to verify that the policy has learned a consistent mapping that produces high-quality actions across different integration paths.\n\n| Training (actor/cirtic update) |  Env interaction |Return |\n| :--- | :--- | :--- |\n| Random `d` in {1～5} | Random `d` in {1～5} | 412.6 ± 20.1 |\n| Random `d` in {1～5} | Fixed (`d=1`) | 391.45 ± 15.8 |\n| Fixed (`d=1`) | Fixed (`d=1`) | 384.86 ± 13.76 |\n\nThe results strongly support our position. The policy exhibits high performance across all conditions, even in the mismatched `Random → Fixed` scenario. This demonstrates that the policy has learned a consistent mapping, capable of producing high-quality actions irrespective of the integration path. Our framework therefore provides a unified perspective: the RL objective of **maximizing the Q-value inherently serves as the consistency condition**, enabling robust and efficient few-step generation.\n\nThank you again for your valuable review.  We are happy to engage in further discussion if anything remains unclear.\n### **References**\n1] **Learning a Diffusion Model Policy from Rewards via Q-Score Matching** (ICML, 2024) by Psenka, Escontrela, Abbeel, & Ma.\n \n2] **One Step Diffusion via Shortcut Models** (arXiv, 2024) by Frans, Hafner, Levine, & Abbeel."}, {"review_id": "Ns3kw4kNCM", "reviewer": "Reviewer_NAN6", "summary": "This paper proposes an algorithm that can use flow-based policies for online constrained policy search. The key idea is to regularize the discrepancy between the online policy and the best behavior policy extracted from the replay buffer, which is derived from an upper bound the the Wasserstein-2 distance between the two policies. The authors demonstrate the efficacy of this algorithm with experiments on DMControl tasks and HumanoidBench tasks, and they also show the efficacy of their proposed regularization on two tasks.", "questions": "1. Since Eq (13) is the key idea of the proposed algorithm, how is the variant, without constraint, performs for all the tasks? I notice that for dogrun, the performance between \"with constraint\" and \"without constraint\" is quite small, and it seems that the variant \"without constraint\" achieves a return of around 250, which is still higher than all other methods. \n\n2. Why are the results of alternative methods not presented for h1-walk-v0?\n\n3. While you argue that the proposed methods takes less wall clock time when compared to diffusion-based methods, there is no quantitative results for that. Could you provide some?\n\n4. There is only one diffusion-based method in your selected alternative methods, so it is hard to tell if this method indeed enjoys performance benefit. Since you have mentioned several algorithms in your related work section, could you please provide results of more diffusion-based method?\n\n5. Since the motivation of using more expressive policy representations is to model multi-modal policies, could you please provide either qualitative or quantitative result for this aspect?\n\nI will re-consider my evaluation if these questions can be answered.", "limitations": "There is no negative social impact.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. This paper addresses an important topic in the recent RL literature: how to use more expressive policy classes for online RL.  \n2. The key idea looks reasonable, and the proposed algorithm is easy to implement. \n\n# Weakness\n1. The experiment results are not very convincing. See the question section below.\n2. The flow of this paper is hard to follow. In specific, the challenges of using flow models directly as policies are not discussed in detail, so readers cannot understand the reason behind using a constrained policy search setting.", "was_revised": true, "final_justification": "Although I understand that diffusion based policies can be difficult to train in online RL, the lack of comparison with alternative methods remains to be a weakness of this method. So I keep my original scoring.", "rebuttal": "We sincerely thank you for your valueable and constructive feedback. Your insightful questions have pushed us to significantly clarify the core contributions and experimental support of our work. We have taken your comments to heart and will incorporate substantial revisions into the manuscript to address every point you raised.\n\n###  **Clarifying the Core Motivation**\n\nThank you for this valuable point, which allows us to further clarify our core motivation. Our framework is designed to address a fundamental objective misalignment that arises when applying generative models to online reinforcement learning.\nThis misalignment stems from the discrepancy between a generative model's objective of **static distribution-matching** (i.e., replicating a fixed dataset of good actions) and the RL objective of **dynamic value-maximization** (i.e., discovering a policy that surpasses the best behavior seen so far).\n\nOur constrained optimization provides a principled solution that allows us to directly harness the powerful modeling capacity of the flow-based policy for the RL objective. The Wasserstein-2 (W2) constraint creates a trust region around high-value behaviors, which stabilizes the policy and guides its expressive power toward maximizing the Q-function. This enables the policy to safely surpass the data, rather than merely imitating the good actions.\n\nThis principled approach enables our method to deliver strong performance on challenging tasks  without sacrificing the efficiency inference, this is a capability that has been elusive for prior generative policy methods. \n\n---\n###  **Q1: On the Performance of the Unconstrained Variant**\n\nThis is a very valuable question, and your insightful observation that the unconstrained variant still outperforms baselines on `dog-run` is absolutely correct. Our response is twofold:\n\nFirst, as you noted, the performance gain from the constraint is more significant on `h1-walk` than on `dog-run`. This is expected in RL research;  **no single technique will dominate across all environments** due to vast differences in dynamics and reward landscapes. Our ablation studies were designed to show this:  we specifically showcase its effect on both a task where the benefit is modest and a complex environment like `h1-walk`, where its impact is critical.\n\nYour observation, however, goes even deeper. It touches upon the  challenge that the effectiveness of a policy representation often depends on the specific dynamics and reward landscape of the environment. Our  baseline results provide a clear illustration of this dependency. For instance, when comparing performance on the `dog` domain, we found that a deterministic policy (TD3) is  more effective than a  Gaussian policy (SAC). Conversely, on the DMC `humanoid` suite, the  Gaussian policy is much more better than deterministic policy \n\nWe believe your observation on `dog-run` is a clear example of this principle. We hypothesize that the flow model offers a uniquely powerful representation whose inductive biases are exceptionally well-suited to the `dog` domain's dynamics. This is supported by its strong performance across the entire  `dog` domain, where it even surpasses the powerful model-based baseline TD-MPC2.\n\nUnderstanding precisely which policy representations are best suited for which types of environments remains a valuable direction for future research.\n\nTherefore, your observation helps disentangle our contribution into two distinct parts:\n*   The flow model  provides a powerful  policy representation.\n*   The W2 constraint makes this powerful representation robust and stable, enabling it to work reliably across a wide range of tasks **with a single set of hyperparameters**. \n\nWe thank you for this insightful question, which has allowed us to articulate the dual nature of our contribution more clearly.\n\n### **Q2: Missing Baselines on h1-walk-v0**\n\nThank you for catching this. You are correct that these results were missing; this was an unintentional oversight on our part. We have now run all the corresponding baseline experiments on `h1-walk-v0` to ensure a complete and fair comparison. The updated results are presented in the table below.\n| Method          | Return（Mean±std） |\n| :---- | :----------- |\n| TD-MPC2(model-based)| 822.5± 27.4 |\n| QVPO      |  570.6 ±36.2|\n| SAC     | 195.1 ± 9.2 |\n| TD3     |  511.7±40.2|\n| FlowRL |  743±37.6|\n\n###  **Q3: Quantitative Time Comparison**\n\nWe have conducted a thorough analysis and present the wall-clock training and inference times in the table below. \n\n| Method | Training Time (1M Steps) | Inference Time (per action) |\n| :--- | :--- | :--- |\n| QVPO（Pytorch） |  ~11.6 hours | ~3200 µs |\n| DACER(JAX) |  ~4.9 hours | ~740 µs |\n| FlowRL (Pytorch)  | ~8.1hours | ~110 µs|\n| FlowRL (Ours, Optimized) | ~4.52 hours | ~110 µs|\n\n1.  **Training Efficiency:** Our baseline PyTorch implementation is  **~1.4x faster** than the pytorch implementation of QVPO.  Furthermore, by applying standard, off-the-shelf optimizations (`torch.compile` and AMP), we can reduce the training time to just **~4.5 hours**, making it competitive with even highly optimized JAX-based implentaion like DACER(JAX is 4~5x faster than pytorch).\n\n2.  **Inference Speed:**  FlowRL is nearly **30x faster**（Pytorch） per action than the multi-step sampling required by previous methods. This is a critical advantage for any real-world deployment scenario.\n\n\n### **Q4: Comparison with More Diffusion-Based Methods**\n\nThis is a good point, and we appreciate the suggestion to include a comparison with more diffusion-based methods.\n\nIn our early exploratory experiments, we did investigate both DACER (which is claimed to be SOTA in its publication) and QSM. We found they struggled to achieve meaningful performance and were often unstable on our challenging tasks, particularly on the HumanoidBench suite where they  failed to learn. We also noted these methods need **careful, task-specific hyperparameter tuning**, as mentioned in their original papers.\nDue to these  issues, we initially chose to proceed with QVPO, which proved to be a more robust baseline in our testing environment.\n\nHowever, to provide a more comprehensive comparison as requested, we have now conducted a dedicated set of experiments with DACER and QSM on the  DMC benchmark suite. The normalized results are summarized below.\n| Tasks | FlowRL (ours) | DACER | QSM |\n| --- | --- | --- | --- |\n| DMC-easy/medium | 83 | 64 | 72 |\n| DMC-hard | 64 | 7| 18 |\n\nThank you for suggesting adding more baselines. We will add results on the challenging HumanoidBench tasks to the final version and provide a discussion of the baseline performance in that context.\n\n\n### **Q5: Evidence for Multi-modal Policies**\nThis is an excellent question that directly probes our core motivation.\nWe wish to clarify that our research's primary focus is not on re-establishing that generative models are multi-modal—a property already explored in prior work. Instead, our contribution centers on the subsequent, critical challenge: **how to make these powerful representations stable and efficient for online policy optimization.**\n\nHowever, to dispel any doubt about the policy's expressive capability within our framework, we are happy to provide further evidence. Our manuscript  offers a qualitative illustration in **Figure 3 (middle)**, which shows the flow-based policy learning a multi-modal action distribution to solve the  task. \n\nTo validate this quantitatively, we performed a clustering analysis on the actions generated by the learned policy in `quadruped-walk`，specifically focusing on the first dimension of the action space. Our clustering analysis on the first action dimension in `quadruped-walk` was a two-step process: 1) We used DBSCAN to discover the number of action clusters. 2) We then located the center of each discovered cluster by computing its centroid .( **We sincerely apologize that we cannot include a t-SNE which is more intuitive, as conference guidelines prohibit adding new figures during the rebuttal period**) . The analysis, summarized below, confirmed the existence of two well-separated action clusters corresponding to the  distinct modes of behavior.\n\n| Analysis Method | Clusters Found | center1 | center2 |\n|-----------------|----------------|---------|--------|\n| DBSCAN          | 2             | -0.99     | 0.99    |\n\nWe are grateful for your detailed review, which is really important and valuable for us.  **We are happy to engage in further discussion if anything remains unclear.**"}]

Author rebuttal:

Meta-review: This paper proposes FlowRL, a novel framework for online reinforcement learning that integrates flow-based policy representation with Wasserstein-2-regularized optimization. The positive reviewer consensus, with two Accept scores and two Weak Accepts, supports the accept decision.
Decision: Accept (poster)