SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modeling
Authors: Yixian Zhang
Shu'ang Yu
Tonghe Zhang
Mo Guang
Haojia Hui
Kaiwen Long
Yu Wang
Chao Yu
Wenbo Ding
Venue: ICLR 2026 Poster (2026)
Keywords: Flow-based policy
Sample-Efficient Reinforcement Learning
Soft actor critic
Sequential Modeling
Primary area: reinforcement learning
TL;DR: We fix the unstable training of flow-based policies in off-policy RL by viewing them as RNNs, using GRU/Transformer designs to tame exploding gradients and achieve SOTA sample efficiency.

Abstract: Training expressive flow-based policies with off-policy reinforcement learning is notoriously unstable due to gradient pathologies in the multi-step action sampling process. We trace this instability to a fundamental connection: the flow rollout is algebraically equivalent to a residual recurrent computation, making it susceptible to the same vanishing and exploding gradients as RNNs. To address this, we reparameterize the velocity network using principles from modern sequential models, introducing two stable architectures: Flow-G, which incorporates a gated velocity, and Flow-T, which utilizes a decoded velocity. We then develop a practical SAC-based algorithm, enabled by a noise-augmented rollout, that facilitates direct end-to-end training of these policies. Our approach supports both from-scratch and offline-to-online learning and achieves state-of-the-art performance on continuous control and robotic manipulation benchmarks, eliminating the need for common workarounds like policy distillation or surrogate objectives. Anonymized code is available at \url{https://anonymous.4open.science/r/SAC-FLOW}

Published as a conference paper at ICLR 2026
SAC FLOW: SAMPLE-EFRICIENT REINFORCEMENT LEARNING OF FLOW-BASED POLICIES VIA VELOCITY- REPARAMETERIZED SEQUENTIAL MODELING
Yixian Zhang Shu' ang Yu Tonghe Zhang', Mo Guang", Haojia Hui Kaiwen Long', Yu Wang", Chao Yu"i, Wenbo Ding"
SIGS, Tsinghua University EEE, Tsinghua University CCamegie Mellon University 'Li Auto "Shanghai Al Laboratory
"Equal contribution *Corresponding Authors {yuchao, eeig
Off-Policy Loss Leads to Exploding Gradients
View Flow-Based Policy Sequential Models
Re-Parameterize
Velocity Network
Grad
Norm
View as RNN: Exploding Gradients
Stable Training
Off-Policy
Sampling Steps
Loss
Backpropagation Through Steps
GRU Cell
Re- Cc-Parammtriis
View as
Gaussian Noise
Hidden State
or
Gaussian Noise
Sampled Action
Pg
Transformer Decoder
Po-Paaramttrrze
Velocity Network
Velocity Network
Figure An Overview of SAC Flow. The multi-step sampling process of flow-based policies frequently causes exploding gradients during off-policy RL updates. Our key insight is to treat the flow-based policy sequential model, for which first demonstrate an algebraic equivalence an RNN. We then reparameterize the flow's 's velocity network using modern sequential architectures (e.g., GRU, Transformer). Our approach stabilizes off off-policy RL training and achieves state-of-the- art performance.
ABSTRACT
Training expressive flow-based policies with off-policy reinforcement learning is notoriously unstable due to gradient pathologies in the multi-step action sampling process. We trace this instability algebraically equivalent to residual recurrent computation, making it susceptible the same vanishing and exploding gradients as RNNs. To address this, we repa- rameterize the velocity network using principles from modern sequential models, introducing two stable architectures: Flow-G, which incorporates gated velocity, and Flow-T, which utilizes a decoded velocity. We then develop practical SAC- based algorithm, enabled by a noise-augmented rollout, that facilitates direct end- to-end training of these policies. Our approach supports both from-scratel and offline-to- coiinetto-nlling learning and achieves faute-of-the art performance on continuous control and robotic manipulation benchmarks, eliminating the need for common workarounds like policy distillation or surrogate objectives. Anonymized code is available https: /anonmoous 4open. wcwenceieeecc..cc wtt.eeene/////C/I///////
fundamental connection: the flow rollout is

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Flow-based policies have shown strong potential on challenging continuous-contry tasks, including robot manipulation. due to their ability to represent rich, multimodal action distributions (Black 2024 Lipman et al. 2022 Jiang et al. 2025). Early successes predominantly arose in imitation 202g]Jiang
et
learning, where a flow-based policy is trained to reproduce expert behavior from static datasets (Luo et al., 2025 Tarasov et al., 2025). However, pure behavior cloning is fundamentally limited: dataset coverage is often sparse and of mixed quality (Kim al. 2024 Garcia et al. 2025), and
the lack of environment interaction prevents exploration, making it difficult to exceed demonstrator
performance on hard tasks (Belkhale et al. 2023 Zare al., 2024
A natural next step is to train flow-based policies with reinforcement learning. On-policy variants of PPO adapted flows have demonstrated strong returns, yet they remain sample-inefficient (Schul- man et al. 2017 Zhang et al. 2025). Off-policy methods promise much higher data efficiency and early integrations with results (Todorov et al., 2012 Tunyasuvunakool et al. 2020 Lv et al. 2025 Park et al.) 2025). How- ever, these successes typically come with design compromises that leave a central issue unresolved. Either the update relies original flow, or the flow is distilled into a simpler one-step actor that can be optimized with standard off-policy losses. Both strategies reduce gradient stress but decouple optimization from the expres- sive generator and tend to blunt the benefits of multimodal flow-based policies (Park et al. 2025 Lv al. 2025).
policies on MuJoCo and DeepMind Control show encouraging
surrogate objectives that avoid differentiating through the rollout of the
We propose a different viewpoint: treat the flow-based policy as a sequential model. Concretely, show that the Euler integration used generate actions in the flow-based policy is algebraically identical to the recurrent computation of a residual RNN. This observation explains the instability observed with off-policy training: the same vanishing or exploding gradients known affect RNNs also afflict the flow rollout. Building on this link, we reparameterize the vanilla velocity network with the cell of modern sequential models that are designed to stabilize deep recurrent computa- tions. We introduce two such novel designs of the flow-based policy: Flow-G, which incorporates GRU-style gated velocity to regulate gradient flow across rollout steps, and Flow-T, which utilizes a Transformer-style decoded velocity to refine the action-time token via state-only cross-attention and a residual feed-forward network.
Our main contributions are summarized as follows:
A sequential model perspective for stable flow-based policies. We formalize the K-step flow rollout as residual RNN computation, providing a clear theoretical explanation for the gradient pathologies that cause instability in off-policy training. This insight allows us to reparameterize the velocity network with modern sequential architectures, leading to two novel, stable designs: Flow-G (GRU-gated) and Flow-T (Transformer-decoded)) Our approach resolves critical gradient pathologies, enabling direct and-to-end optimization and eliminating the need for surrogate objectives policy distillation.
A practical and sample-effecient SAC framework for flow policies. We develop SAC Flow, a robust off-policy algorithm built upon our stabilized architectures. By introducing noise-augmented rollout, we enable tractable likelihood computation for the SAC objective, a key technical hurdle. This approach yields two robust training procedures: (i) a stable fcom-scratch trainer for dewse-reward tasks and (ii) a unified affline-to-oning pipeline for rewrse-reward tasks.
Extensive experimental evaluation. We demonstrate the effectiveness of SAC Flow across multiple benchmarks. In from-scratch training on challenging MuJoCo tasks, our approach delivers performance gains of up to 130% over strong baselines. Furthermore, in complex offline-to-online manipulation tasks on OGBench, it achieves up to a 60% higher success rate. These results empirically validate the superior sample efficiency of our direct off policy training approach, with ablation studies further confirming the robustness of our designs.
2

---

Published as a conference paper at ICLR 2026
2
PRELIMINARIES
2.1 REINFORCEMENT LEARNING
We consider policy optimization in an infinite-horizon Markov decision process (S, p, r, A, with continuous state and action spaces. The transition function : S x A S + [0, oo) specifies the transition probability density, and rewards are rh = r(sh, ah) ) [rmins Tmax), where an sampled from the policy T(1(s)). The objective of reinforcement learning is to learn optimal policy л that maximizes the expected cumulative reward, n- = arg max, ET
| 2 ~
2.2 SOFT ACTOR-CRITIC ALGORITIIM
To encourage policies to maintain stochasticity and explore more effectively, the standard objective augmented with an entropy term, J(x) = E = (rt, + azH)], where H(T(( sn)) = - Raww(-legg) [log Ta s)] denotes the state-conditional policy entropy. In this setting, the Soft Actor-Critic algorithm (Haarnoja et al., 2018) is introduced to optimize this objective. The tar- get JTT) is typically approximated with the soft Q-function Qu(sn, an), which is updated through the TD loss:
T(Tb) = [Qw (8h: - (rn + 2Q (sh+1; #1+1) - ar log Ta (4x41 sx+1))]
where 04-1 ~ To (-1 81+1), (8h, an 8h+1; rh)) are sampled from the replay buffer, and & delayed copy of i through which gradients do not flow for stability. To maximize the soft Q- function (h,. an). the policy me is updated through
L(0) = a log To (a sn)) - Q (sh, ag)
a ~ To (k))
(2)
Here, ag highlights a reparameterized action sample that allows gradients to propagate from the policy the action, in contrast the TD update, where the action detached.
2.3 FLOW-BASED POLICY REINFORCEMENT LEARNING
Gaussian policies are the standard choice in continuous-contre RL Yang et al. 2021 Ziesche & Rozo 2024). yet single unimodal Gaussian cannot capture inherently multimodal action distri- butions. This limitation is especially harmful in long-horizon robotic control, as such tasks often benefit from policies that output temporally extended actions (i.e., "action chunking") (Li et al.) 2025). The distribution over these action sequences is often inherently multimodal, which uni- modal Gaussian fails to represent. Diffusion policies alleviate this by modeling arbitrary normal- izable distributions and have achieved state-of-the-art results on manipulation benchmarks (Bekris) ct al. 2025 Wang et al. 2022 Ren et al. 2024). but their iterative denoising makes both training and inference expensive. Recently, flow-based policies have emerged
simpler alternative: trained with flow-matching objectives, they offer easier training and faster inference while often matching
exceeding diffusion quality (Lipman et al. 2022 Park al. 2025 Zhang al. 2025).
flow-based policy transports a simple, state-conditioned base Po(: | s) over the action space A Idd to target policy (- | s) via time-indexed map 2 [0, 1] x AxS + A. with A, @:(AA 8) for / LE [0, 1]. where Ao poo- 8) and A p- s). The trajectory satisfies the ODE .l
di An x) = vtt, (Ao ss) s). where v is a learnable velocity field. We adopt Rectified Flow
(Liu et al. 2022 which uses the straight path A, (1 A) tA, and the standard Gaussian
base Po(- objective
8) NNO, In this case At, di A, A Ags yielding the flow-matching
i arg min E (V (A)=a,))< [|A Ao (t (1 A) +AA, 5) [1]]
(--Uni((01]]
where D denotes the dataset of state-action pairs. In inference, the learned field is integrated nu- merically with flow rollout to obtain:
A A A4, vo(li, S), 0 < < lK 1,
where At. = t+ - t. The resulting distribution over A, induced by Ao ~ ~N Ia) is denoted po(: 8) and serves the stochastic policy A no(: s).
3

---

Published as a conference paper at ICLR 2026
Flow-based policies can be trained offline from demonstrations using Equ. (3). and they can also be optimized with RL. On-policy methods (e.g., PPO-style training tailored to flows (Zhang et al.)
2025| Ren et al. 2024 Psenka et al. 2024)) attain strong performance challenging robotics tasks but remain sample pempll-inn ineefffficn.. Off-policy methods (e.g.. SAC, TD3) are highly sample-efficient (Mambelli et al. 2024), yet directly backpropagating through the K-step action sampling is often unstable, especially for large K (Park et al. 2025). To mitigate this, prior work either distills flow-based policy into a simpler actor trained with standard off-policy losses (Park al., (025) or
proposes surrogate pof-py objectives that train the velocity field without differentiating through the full flow rollout (Lv al.) 2025).
We take different route. We recast the flow rollout as a sequential model and redesign the velocity parameterization accordingly. We introduce Flow-G, which uses a GRU-style gated velocity, and Flow-T, which uses a Transformer-style decoded velocity. These parameterizations stabilize the K- step hackpropagating and allow direct off-policy training of the flow-based policy. We instantiate the framework with SAC, and the same formulation applies to other off-policy algorithms.
FROM FLOW ROLLOUT TO SEQUENTIAL MODELS
Grad Norm
RNN: unstable gradient
GRU: stable gradient
Transformer: stable gradient
Off-Policy Loss
AEL
Backpropagation Through Steps
Figure 2: An illustration of gradient norms during training. By conceptualizing a flow-based model as an RNN, the most basic sequential models, we observe that it still suffers from the exploding gradients during training. This motivates our work to model the flow-based model as advanced sequential architectures, such as GRU or Transformer. These models can be updated with stable gradients during the backpropagation process.
In this section, we reveal a key insight: flow-based policies are fundamentally sequential models. As conceptually illustrated in Fig. 2 standard flow rollouts exhibit gradient instabilities analogous to vanilla RNNs, while modern sequential architectures offer more stable gradient flow, motivating our velocity network designs.
Flow-hased policy RNN (Fig. 3a). Treat the intermediate action Ag the hidden state and (ti, s) as the input. Then Equ. is residual RNN step (Goel et al. 2017):
A Ar. + Au, s), with fe(-) = At, va(.),
(5)
where fo(-) denotes the RNN cell. Consequently, training a flow-based policy with off teflpooicy losses backpropagates through deep recurrent stack of K updates RNN, which is prone gradient explosion and vanishing (Bengio et al. 1994 Pascanu et al. 2013). This explains the instability observed when naively applying off off-policy reinforcement learning to standard flow- policies.
Flow-based policy as GRU (Flow-G, Fig. 3b). To improve gradient stability, we endow the velocity with a GRU-style update gate. Let gi Sin 20(1i, 8)) and let ün be a candidate network. Define
A A + Ad, (g. o (6g(t,, A 8) A.)))
where denotes elementwise multiplication and Sig(-) is the logistic sigmoid. Equ. 6 is exactly a flow sampling step with gated velocity vi = gi @ (0g - A..). ). which mirrors the structure of the update in GRU cell but expressed in the velocity parameterization used by the flow rollout. The gate network g, adaptively interpolates between keeping the current intermediate action and forming one.
4

---

Published as a conference paper at ICLR 2026
Flow-based policy Transformer (Flow-T, Fig. 3c). We parameterize the velocity function do using a Transformer architecture conditioned on the environment state 8. To maintain the Markov property of the flow, depart from traditional causal, autoregressive formulation. Instead, the model first computes independent embeddings for the current action-time token A,, and a single, global embedding for the state
A: = EA (db(())) Ar.), 4 Ess (
where EA and Es are linear projections. Within the Decoder layers, a diagonal mask is applied to the self-attention mechanism, effectively reducing it to position-wise transformation that processes each token A, independently, without mixing information across the time steps i. The crucial step for context integration is a dedicated cross-attention module, where each action token 4,, queries the shared state embedding Φ5. A stack of T. pre-norm residual blocks refines the action tokens:
y Φ (1-1) - + Cross/ [LN( (I-1)) context LN(())). T YY + FFF, (8)
for layers I = I...., L. where $ A. = AA Each block is completed by feed-forward network, and the final representation is projected to the velocity space:
A = Ar, + At, W. (LN(O))))))
where Wo is a linear projection and (f,, s), = Wo (IN( is the decoded velocity in Flow-T. Each velocity evaluation therefore executes L layers that refine the current action token A, based on the global state context from 4s. not on a causal history of other tokens. This state- conditioned refinement of the entire trajectory maintains the fundamental Markov property of flow- based policy while enabling stable integration with off- off-policy learning algorithms.
Takeaway for off-policy reinforcement learning. Equ. 5) establishes that standard flow rollout
a residual recurrent computation. Introducing a gate network leads to Flow-G in Equ. (6), which improves gradient stability. Replacing the velocity with the normalized residual block in Equ. (9) yields Flow-T. This architecture provides well-conditioned depth and, crucially, aggregates context with the well-established Transformer architectures.
The core technical motivation is to stabilize the recurrent computation in Equ. which suffers from the exploding/vanishing gradient problem due to unstable Jacobian products during backprop- agation. Our Flow-G and Flow-T designs directly mitigate this via stabilizing mechanisms. A detailed mathematical analysis of this instability and our solution is provided in Appendix B
These parameterizations serve as drop-in replacements for vo in Equ. 4 without altering the sur- rounding algorithm. As result, they enable direct and stable off-policy training with methods such as SAC, remove the need for auxiliary distillation actors and surrogate objectives, and keep flow rollout efficient at time.
4 TRAINING FLOW- POLICY VIA SAC
With gradient stability achieved through our sequential parameterizations (Flow-G and Flow-T), can now train flow- policies directly with off-policy reinforcement learning. The key technical challenge computing policy likelihoods for the K-step rollout in Equ. (4)- requirement for the reguoae-re objective in SAC. We solve this through a principled noise-augmented rollout that preserves the final action distribution while enabling tractable per-step likelihood computation.
Likelihood via a noise-augmented rollout. SAC requires explicit policy likelihoods for entropy regularization, but the deterministic K-step rollout in Equ. (4) yields intractable densities. We address this by making the rollout stochastic while preserving the marginal of the final action, which induces a product of per-step Gaussian transitions and tractable joint path density Pe( s) over intermediate actions A = (Argy... (A,g)...,AAA)) The construction details are deferred to Appendix A here the resulting log (AA s) drop-in entropy term.
5

---

Published as a conference paper at ICLR 2026
GRU Cell
Decoder
RNN Cell
Ati
AG
D
Vg
Layer Norm
0g
9
Ar
i
Sig
Decoder Layer
NN
A
t
NN
NN
Decoder Layer
5,
s, c2
Vg W, (LN (T)(
= O (Bg Aes)
Dg Dg
(a) Flow-based Policy RNN
(b) Flow-based Policy as GRU
Flow-based Policy as Transformer
Figure 3: Velocity network parameterizations for the flow-based policy, shown in the view of (a) RNN Cell: It represents the standard flow-based policy where the velocity os the direct output of neural network. This simple formulation is prone gradient instability. (b) GRU Cell: The velocity is computed using GRU-style gated mechanism. A gate adaptively controls the update strength from a candidate network Ü, which stabilizes gradient flow. (c) De- coder: The velocity is modeled using a Transformer decoder, where the action-time token A,, is refined through T. layers of tate-conditioned cross-attention to produce a decoded velocity.
quential models.
From-scratch training. With tractable likelihoods established, the SAC losses become straight- forward. Given critic Qú and flow-based policy лo (with Flow-G Flow-T as va), optimize:
Lactor (8) = a (A4 sh)) Qulsn, af),, A ~ To(- SN), an tanh(AA
(10)
Lecti (v)) (s an) (rn - Qu am+1) a log Pe (Ar+1 sh+1)) 2
(11)
where (sh, an, Th, S1+1) comes from the replay buffer, Ant1, ah+1 ~ no(: $1+1), and v is delayed copy.
Offline-to-onling training. For sparse-reward tasks where expert demonstrations are available, we modify the actor loss include proximity regularizer:
aece (0) = a log (A4 sn.) - Qu(sk, a + 3 Ila an113.
(sh, an.) ~ B. (12)
This approach begins with flow-matching pretraining on expert data via Equ. 3 then transitions to online learning while maintaining proximity to the replay buffer. The complete procedures are summarized in Algos,
and
Algorithm SAC Flow (from scratch)
1: Initialize critic @u. target Qzz flow-based policy me with Flow-G or Flow-T, replay buffer B.
2: for each update
3: Interact with the environment using no; push (St, az Tr, 8(+1) to B.
4: Sample ((sp, Oh, Th, $141)) ~ B.
5: Actor: draw af by K-step noisy rollout; minimize Equ. (10)
6: Critic: minimize Equ. (11) update target by exponential moving average.
7: end for
For clarity, we refer to our methods as SAC Flow-G and SAC Flow-T, corresponding to training with Flow-G and Flow-T via SAC, respectively. Both terms apply to both cor-schrach and offline- to-online training variants.
EXPERIMENT
We conduct extensive experiments locomotion and manipulation benchmarks to validate approach. The evaluation encompasses: (1) experimental setup and baseline comparisons for from-
scratch and offline-to-online training, (2) performance benchmarking of SAC Flow-G and SAC

---

Published as a conference paper at ICLR 2026
Algorithm 2 SAC Flow (offline-to-online)
1: Initialize Qu. Qú. set B +- D exppett.
2: for to Loff + Lon do
3: if ( Loff then
4: Interact with the environment using up; append to B.
5: end if
6: Sample {(5). ab. The Sh+1
Th=1 B.
7: Actor: minimize Equ. 112 with a from the noisy rollout.
8: Critic: minimize Equ.
update the target network.
9: if S Lon then
Flow-matching pretraining via Equ. (3)
11: end if
12: end for
3000
12500
10000
Retuon
2000
40000
Retim
7500
1000
2000
2500
0
0.0
0.2
0.4
0.6
0.3
0.0
0.2
0.4
10.6
0.5
1.0
0.0
0.2
01.
0.6
9.5
Steps
teo
Steps
1e6
Steps
160
(a) Hopper-v4
(b) Walker2D-v4
HalfCheetah-vd
12000
400000
6000
10000
300000
Rolann
Retum
2000
2000
100000
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.6
0.6
0.3
1.0
0.0
0.2
01.4
0.6
0.8
1.0
Steps
1c5
Steps
1d5
Steps
(d) Ant-v4
(e) Humanoid-va
(I) HumanoidStandup-v4
SAC Flow-T
SAC Flow-G
FlowRL
DIME
QSM
SAC
PPO
Figure 4: From-scratch training performance. Our SAC Flow-T and SAC Flow-G achieve compara- ble better performance accross all tasks except Humanoid (Fig. (a)-(f),, demonstrating significant sample efficiency and convergence stability.
Flow-T against recent methods, and (3) ablation studies analyzing the effectiveness of our design components. All results are averaged over 5 random seeds and use the 95% confidence interval.
5.1 SETTINGS
5.1.1
ENVIRONMENTS AND OFFLINE DATASETS
We evaluate our method on three benchmarks for locomotion and robotic manipulation: Mu- JoCo (Todorov al.] 2012) Brockman et al. (016). OGBench (Park et al.] (2024). and Robomimic [Mandlekar et al. 2021). MuJoCo tasks, which feature dense rewards, are used to evaluate from-scratch learning performance. Then we conduct offline-to-online experiments on OG- Bench and Robomimic, using their respective official offline datasets
!OGBench: http:://gtthh..cco//eeeeerrrrrrp
Robomimic:
https:
c. htt.

---

Published as a conference paper at ICLR 2026
1.0
0--
Sterres Rien
Stuzzen Bise
Soccess Rat
0.6
0.5
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.8
08
0.0
0.5
1.0
20
0.5
1.0
155
2.0
0.5
1.0 Stega
1.5
2.0
0.0
0.5
1.0 Stope
1.5
2.5
Stops
Ice
Steps
(a) Cube-Double-Te
(b) Cube-Triple-Taane
(c) Cube- Haabbe-Quarupll-Taaae
(d) Robomimic
SAC Flow-T
SAC Flow-G
FQL
QC-FQL
ReinFlow
Figure 5: Aggregated offline-to- to-line performance OGBench and Robomimic benchmarks.
Each curve shows the mean success rate averaged across multiple task instances within a domain. Specifically, the OGBench results for Cube-Double, Triple, and Quadruple (a-c) are each aggre- gated over five distinct single-task environments. The Robomimic result (d) is aggregated across the Lift, Can, and Square tasks.
5.1.2 BASELINES
For the rer-shhatch training, we compare SAC-Flow against five baselines. (1) Q-score matching (QSM) (Psenka et al. 2024) directly optimizes the diffusion policy's score function using the gra- dient of the Q-function. (2) DIME (Celik et al. 2025) a representative max-entropy RL method for diffusion policy, addressing the challenge of entropy calculation. (3) FlowRL (Lv et al. 2025) the state-of-the-art (SOTA) method, which trains a flow-based policy by directly maximizing the Q-value, regularized by Wasserstein-2 constraint. Finally we apply two classical RL algorithms: (4) SAC (Haarnoja et al. 2018) and (5) PPO (Schulman et al. 2017), with Gaussian policies as fundamental from-scratch baselines.
To evaluate the offline-to-onling capability, we select three baselines, including on-policy and off- policy methods. (1) ReinFlow (Zhang et al. (025)) solves the difficulty of calculating log probability through multi-step flow inference, enabling on-policy PPO update for flow-based policy. It should be noted that ReinFlow is only tested in Robomimic due lack of official implementation for its use in OGBench. (2) Flow Q-Learning (FQL) (Park et al. 2025) uses SAC-style update to achieve high data-efficient RL tuning. FQL uses a one-step policy estimate the flow model, avoiding the instability of backpropagation through time. And its successor, (3) Q-chunking FQL (QC-FQL) (Li et 2025). extends FQL handle action chunking by operating temporally extended action spaces.
Among all experiments, the sampling steps of flow-based policies are set to 4, and the denoising steps of diffusion policies are set to 16. More details of the experimental setting are described in Appendix Dnd Appendix F
5.2
MAIN RESULTS
Fig. 4 iillustrats the results for from-scratch training. Our methods, SAC Flow-G and SAC Flow-T, achieve superior or comparable performance across most MuJoCo tasks, with the exception of Hu- manoid. Although DIME and FlowRL generally converge faster than other baselines, our methods consistently surpass FlowRL, benefiting from direct optimization of the SAC objective. Further- more, SAC Flow outperforms DIME in Hopper (Fig I5a). Walker (Fig. [5b). and HumanoidStanduup (Fig, (f). while achieving comparable results in HalfCheetah (Fig. 15cp and Ant (Fig. [5d]. More- over, with the expressive parameterization of flow-based policy, our method achieves much higher final performance in challenging tasks, demonstrating up to a 130% improvement over the baseline (Fig. 4f). and remains convergence stability in simple tasks (Fig. 15a 15b and 15c). For reference,
we include the on-policy baseline, PPO, to highlight the superior sample efficiency of off off-policy algorithms. Finally, find that all from-scrate methods struggle in tasks with large exploration spaces and sparse rewards, such as Robomimic- Can and OGBench-cube (see Appendix F.1 Fig. underscoring the necessity of offline-to-onling training setting.
8

---

Published as a conference paper at ICLR 2026
Fig. 5 shows the offline-to-oin to-oliie training performance in sparse reward tasks. All methods trained on 1M offline updates followed by 1M online steps. In the challenging OGBench environ- ments, including cube-triple and cube-quadruple, our proposed methods, particularly SAC Flow-T, achieve rapid convergence and attain a state-of-the-art overall success rate. In the Robomimic envi- ronment, however, SAC Flow-T and SAC Flow-G only yield results comparable to QC-FQL. This is primarily because the training is strictly regularized with a large 3 value (Equ. (22). ). As a result, the learning capacity of the flow model severely limited, causing its performance to be similar
to that of the one-step policy in QC-FQL. We further compare the on-policy baseline, Reinflow, in Robomimic. Leveraging the high data efficiency of off off-policy learning, our SAC Flow-G and
SAC Flow-T outperform Reinflow under IM online steps. The additional results are available in Appendix F
5.3
ABLATION STUDY
Ona 2004
Sags
Sagt
Skge
Skuge
(a) Ant-v4
(b) Ant-Step 0
(c) Ant-Step 1
(d) Ant-Step 2
(e) Ant-Step 3
1000
2000
6.6
0.25
0.50
0.55
8.53
6.0
0200
0.5
300
0.5
6.3
6.5
(f) Walker2d-v4
Walker2d-Soe Step
Walker2d-Sol Step
(i) Walker2d-S5 Step 2
(i) Walker2d-Step 3
2:00
Skyge
Rays
Daps
(k) Hopper-v4
(1) Hopper-Step 0
(m) Hopper-Step
(n) Hopper-Step 2
(o) Hopper-Step
SAC Flow-T
SAC Flow-G
Naive SAC Flow
Novte Noter Towe 1ine- 1ine-1
Nars Floer SuperSSaadd Plandl Gradlen Noon Ratoo
Nave Floe Fiv.d Jenn
Se
0.59 Stape
1n0
105
Stape
(g) Ant-v4 Ratio
(r) Walker2d-v4 Ratio
(s) Hopper-v4 Ratio
Step 0
Step
Step
Step 3
Figure 6: Ablation study on velocity network parameterizations. Our SAC Flow-T and SAC Flow- G significantly reduce the gradient exploding and enable stable training. The first column (a, f, k) displays the episodic return for the environments. The subsequent four columns illustrate the gra- dient norms for steps 0 through 3 of the flow sampling process, respectively. The bottom row (q-s) visualizes the gradient norm ratio (Naive Flow-G), revealing that across all tasks, the Naive gra- dient norm explodes to approximately tenfold the stable magnitude around the time of performance collapse (10' steps).

---

Published as a conference paper at ICLR 2026
Ablation study on velocity network parameterizations. We begin by analyzing the gradient dy- namics of our proposed architectures, SAC Flow-G and SAC Flow-T. We benchmark these against a Naive SAC Flow baseline that utilizes standard MLP velocity parameterization without sequential modeling. As illustrated in the first three rows Fig. 6 the naive baseline exhibits severe gradient pathologies, characterized by erratic norm oscillations along the backpropagation path (specifically from sampling step & 3 back to k 0). In contrast, our methods maintain well-conditioned gradient norms across the entire rollout. This instability in the Naive SAC Flow directly precipitates performance degradation, as evidenced by its failure to learn in the Ant, Walker2d, and Hopper.
(Figs. 6a 6f 6k).
To provide a unified explanation despite the varying absolute gradient scales across tasks (c.g., gradients Ant are naturally larger than in Walker2d), we further analyze the relative stability by computing the gradient norm ratio (Naive SAC Flow / SAC Flow-G), shown in the bottom row of Fig. (g-S) This metric reveals a striking consistency: across all distinct tasks, the gradient norm of the Naive baseline escalates approximately tenfold that of the stable SAC Flow-G. Crucially, this 10-fold relative explosion typically peaks around 10a steps, which aligns perfectly with the inflection point where the Naive baseline's performance stagnates and begins to deteriorate. These empirical results conclusively validate that the standard flow rollout suffers from severe relative gradient instability, and our sequential reparameterizattons effectively mitigate this issue enable stable training.
Ablation study on flow sampling steps. Fig. shows the performance SAC FLow-T and SAC Flow-G Wow-GGone under sampling steps K = 4, 7, 10. A larger number of sampling steps can further challenge stability of gradient backpropagation. The experiments show that our approach, especially SAC Flow-T, is robust to the number of sampling steps.
1.0
6000
4000
Success Rate
Return
0.6
2000
0.4
0
-0000
0.0
0.2
0.4
0.6
0.8
1.0
0.0
1.0
1.5
Steps
1c6
Steps
le6
(a) Ant-v4
(b)
Flow-T (K=4)
Flow-T (K=7)
-.- Flow-T (K=10)
Flow-G (K-4)
Flow-G (K=7)
Flow-G (K=10)
Figure 7: Ablation study flow sampling steps. Our SAC Flow-G and SAC Flow-T are robust to the number of sampling steps.
CONCLUSION
In this paper, introduce SAC Flow, sample-efficient and high-performance off-policy RL al- gorithm for flow-based policies. SAC Flow addresses the issue of gradient instability in training flow-based policies by treating the flow-based model sequential model and reparameterizing its velocity network as a GRU or Transformer. We evaluate the performance of SAC Flow in both from-scratch and -o-iiee- training settings. SAC Flow demonstrates rapid convergence and achieves ofate-of-the-ar performance across multiple locomotion and manipulation tasks.
Looking forward, we will validate SAC Flow on real robots and explore lighter sequential param- eterizations with structure-awee aware updates, while studying sim-to-real robustness, tighter stability guarantees, and risk-aware objectives for reliable deployment.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We adhere to the ICLR Code of Ethics. Our research introduces foundational RL algorithms evalu- ated exclusively on public benchmarks and datasets. This work does not involve human subjects or sensitive data and presents no foreseeable direct ethical concerns.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, additional implementation details, hyperparameters, and experimental results are provided in the Appendix. All experiments leverage public benchmarks and datasets. The anonymous code is available at attps://anonymoore 4open. www.nce/t//AC--LL.....C
ACKNOWLEDGMENTS
We would like to thank Yuze Sheng for assistance with the real-robot experiments of SAC Flow. This work supported by the National Natural Science Foundation of China (No. 62406159, 62325405), Shenzhen Science and Technology Program (No. KJ/ZD20209903100000 10090500000 Zhong- guancun Academy (Grant No. C20250301), Beijing National Research Center for Information Sci- ence and Technology (BNRist), Beijing Innovation Center for Future Chips, and Ant Group.
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural informa- fion processing systems, 0199304-29320 2021.
Lars Ankile, Anthony Simeonov, Idan Shenfeld, Marcel Torn,, and Pulkit Agrawal. From imitation to resinel for precise assembly. 2025 IEEE International Conference on Robotics and Automation, pp. 01-08, 2025.
Kostas Bekris, Kris Hauser, Sylvia Herbert, Jingjin Yu, Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 44(10-11):1684-7700 2025.
Suneel Belkhale, Yuchen Cui, and Dorsa Sadigh. Data quality in imitation learning. Advances in Neural Information Processing Systems, 36:80375-80395, 2023.
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Leanning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-166, 1994.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, model for general robot contro. ArXiv Preprint, 2024.
T0: vision-langunge-cttion flow
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. ArXiv Preprini, 2016.
Onur Celik, Zechu Li, Denis Blessing, Ge Li, Daniel Palenicek, Jan Peters, Georgia Chalvatzaki, and Gerhard Neumann. Dime: Diffusion-based maximum entropy reinforcement learning, ArXiv Preprint, 2025.
Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi. Diffusion-based reinforcement learning via q- g-weighted variational policy optimization. Advances in Neural Information Processing Systems, 37:53945-53968. 2024.
Xiaoyi Dong. Jian Cheng, and Xi Sheryl Zhang, Maximum entropy reinforcement learning with diffusion policy. ArXiv Preprint, 2025.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.

---

Published as a conference paper at ICLR 2026
Ricardo Garcia, Shizhe Chen, and Cordelia Schmid. Towards generalizable vision- laiiuala robotic manipulation: A benchmark and I1m- Im-guided 3d policy. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pp. 8996-90022 2025.
Hardik Goel, Igor Melnyk, and Arindam Banerjee. R2n2: Residual recurrent neural networks for multivariate time series forecasting, ArXiv Preprint, 2017.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. ArXiv Preprint, 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Peter Holderrieth and Ezra Erives. An introduction to flow matching and diffusion models. ArXiv Preprint, 2025.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Ki- nal Mchta, and JoAGo GM AraAšjo. Cleanrl: High-quality iinle-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022.
Sunshine Jiang, Xiaolin Fang, Nicholas Roy, Tomás Lozano-Pérez, Leslie Pack Kaelbling, and Sid- dharth Ancha. Streaming flow policy: Simplifying diffusion flow-matching policies by treating action trajectories as flow trajectories. ArXiv Preprint, 2025.
Yeseung Kim, Dohyun Kim, Jieun Choi, Jisang Park, Nayoung Oh, and Dachyung Park. A survey on integration of large language models with intelligent robots. Intelligent Service Robotics, 17 (5):1091-1107, 2024.
Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking. ArXiv Preprint, 2025.
Steven Li, Rickmer Krohn, Tao Chen, Anurag Ajay, Pulkit Agrawal, and Georgia Chalvatzaki. Learning multimodal behaviors from scratch with diffusion policy gradient. Advances in Neu- ral Information Processing Systems, 37:38456-38479, 2024.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hame Maximilian Nickel, and Matt Le. Flow matching for generative modeling. ArXiv Preprint, 2022.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. ArXiv Preprint, 2022.
Hao Luo, Yicheng Feng, Wanpeng Zhang, Sipeng Zheng, Wang, Haoqi Yuan, Jiazheng Liu, Chaoyi Xu, Qin Jin, and Zongqing Lu. Being-h0, h0: ssaionlanguage-actto pretraining from large- scale human videos. ArXiv Preprint, 2025.
Lei Lv, Yunfei Li, Yu Luo, Fuchun Sun, Tao Kong, Jiafeng Xu, and Xiao Ma. Flow-based policy for online reinforcement learning. ArXiv Preprini, 2025.
Davide Mambelli, Stephan Bongers, Onno Zoeter, Matthijs TJ Spaan, and Frans A Oliehoek. When do oof-cc and on on-policy policy gradient methods align? ArXiv Preprint, 2024.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martín. What matters in learning from offline human demonstrations for robot manipulation. ArXiv Preprint, 2021.
Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, and Aviral Kumar. Policy agnostic rl: Offline rl and online rl fine-tuning of any class and backbone. ArXiv Preprint, 2024.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. ArXiv Preprint, 2024.
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. ArXiv Preprint, 2025.
12

---

Published as a conference paper at ICLR 2026
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, PP. 1310-1318. Pmlr, 2013.
Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning diffusion model policy from rewards via q-score matching. In International Conference on Machine Learning, pp. 41163-41188 182. PMLR, 2024.
Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majum- dar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimiza- tion. ArXiv Preprint, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv Preprint, 2017.
Denis Tarasov, Alexander Nikulin, Πlya Zisman, Albina Klepach, Nikita Lyubaykin, Andrei Pol- ubarov, Alexander Derevyagin, and Vladislav Kurenkov. Nina: Normalizing flows in action. training vla models with normalizing flows. ArXiv Preprint, 2025.
Emanuel Todorov, Tom Frez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEF/RSJ international conference on intelligent robots and systems, pp. 5026-5033, 2012.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Frez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm control: Software and tasks for continuous control. Software Impacts, 6: 6:100022, 2020.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. ArXiv Preprint, 2022.
Qisong Yang, Thiago D Simão, Simon H Tindemans, and Matthijs TJ Spaan. Wesac: Worst-case soft actor critic for safety-constrainee reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 10639-10646 2021.
Shu-Ang Yu, Feng Gao, Yi Wu, Chao Yu, and Yu Wang. D3p: Dynamic denoising diffusion policy via reinforcement learning. ArXiv Preprint, 2025.
Maryam Zare, Parham M Kebria, Abbas Khosravi, and Saeid Nahavandi. A survey of imitation learning: Algorithms, recent developments, and challenges. IEEE Transactions on Cybernetics, 2024.
Tonghe Zhang, Chao Yu, Sichang Su, and Yu Wang. Reinflow: Fine-tuning flow matching policy with online reinforcement learning. ArXiv Preprint, 2025.
Hanna Ziesche and Leonel Rozo. Wasserstein gradient flows for optimizing gaussian mixture poli- cies. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2026
A THE DERIVATION OF SAC LOSS IN THE FLOW-BASED POLICY
This appendix consolidates and expands our derivations for training SAC K-step flow roll- out, including the likelihood construction via noise-augmented rollout, the joint path density, the pathwise score expansion, gradients for actor/eritic, the temperature update, and practical notes for implementation.
A.1 NOISE-AUGMENTTHR ROLLOUT AND DRIFT CORRECTION
We start from the deterministic K s-step Euler rollout Equ.
Aut1 = Ar; + At, (t Ar, s), 0 = to < ....... <k 11
For likelihood-baser training, convert it into stochastic rollout that leaves the final marginal invariant by adding isotropic Gaussian noise with compensating drift (Holderrieth Erives 2025):
A A,, + be Ar, 8) At, + 08 Ati s ~
(13)
A convenient drift that matches rectified-flow families is
t A 2 1 - ti
t, 0 t,) 2() Al.
by (tz, Ac. 8)
volti, At,,
(14)
with by (0, to (0, -) Intuitively, the first factor inflates the learned velocity to counteract diffusion, and the second term contracts towards the straight path so that the terminal law remains unchanged. The detailed proof can be found pages 28 35 in (Holderrieth & Erives 2025).
Per-step transition.
Under Equ. 13 the conditional Au | is Gaussian:
go(Auus A 8; At.) = N + bo (tz, A 8) Ati, og At, Ia)
We denote Ato -N, Ia) the base. The final action is tanh(Atx)
A.2 JOINT PATH DENSITY AND SQUASHING JACOBIAN
Let A = =(Ato..... Aex)
The joint density factorizes as
K-1
Pc 8) = ((AA.)
II ne (Ar. ==0
Ar, s; Ati) ( det [(a)l="; a = tanh(Arg):
(15)
where ( is the standard Gaussian base density for Atg na()) is the per-step transition in Section A. and (a) is the Jacobian of the element-wise tanh squashing. The marginal policy density follows by integrating out the intermediate pre-activations:
no (a 8) = f PPo (Aro...., Aix- Aex = tanh ((a) s) dAi, ... dA
(16)
For element-wise tanh, || det [()]ll = -II a23)
A.3 PATHWISE EXPANSION OF THE MARGINAL SCORE
We derive the gradient of E, [log (a x)]. Using Equ. 15
Eg Elog Te(a s)] VoEA [log Te(a s)]
VoEA log ".../oelloo........ Aux-1 Alx tanh ((a) 8) dAi, ... ...dAx-.)) (17)
14

---

Published as a conference paper at ICLR 2026
Expanding the inner gradient yields
1
v f.... ((Aro)
[K-1
Va log (a 5) Ta(a 8)
Vo
II ne (Arss Ar, 5; At,) i=0
|| det ((a]l K-
K
1
f...] ((Aso)
II no (A
Ac, Ati)
Ta(a a 8)
k=0
k
Vo log no (Aem Ae, St.) || det I(a)]l d
(18)
Therefore,
K Σ
V Eag (logTo(o Ta(a [8]]= = EA
Vo log no (Aten | 8; At:))
(19)
where the Jacobian term does not contribute because it is independent of 0. Since He is Gaussian with mean m, A., by At, and covariance D, of St, each is closed form:
V log m1e oy (A - mm) TÔm; d doa + higher--rdeer terms if 0 depends 0.
00
σB 00
A.4 GRADIENTS OF THE SAC LOSSES UNDER THE JOINT PATH FACTORIZATION
Critic update. The target-matching loss
((bb) [Qu(sh, an) (rx + Q an+1) - a log (a(an sx))]²²
(20)
where 0/+11 ~ xp(- 11+1). Using the joint-path form,
VoL(v)= (s,, an.) (rx + (s am+1) - a log (A 8))) V Ve ax), (21)
where no gradients flow through Q Replacing the marginal log मe by Pe only changes a base- line and has negligible effect on learning behavior.
Actor update.
The actor loss is
L(O) log (a 8n) Qwn, (h),
(22)
with af tAng( Its gradient uses the pathwise form:
voL(O) V log T (Ah... Ale shi At.) VoQu
(23)
where the Q-term diffcrentiates through a.
A.5
PATH-REEGULARIZED SOFT CRITIC
This section explains how the joint and marginal densities relate and why the resulting critic can be naturally interpreted as path-regularized variant of maximum-entropy RI.
A.5.1 EXACT DECOMPOSITION OF THE ENTROPY TERM
For any (s. a) with tanh( (At ), the joint path-density admits the factorization
(c(A] s) Te(a (e s).).
(24)
where re(A | s) is the conditional distribution of the latent path given the final action:
(c(AA s)
f To(A la, s)d =
Te(A s) Ta s)*
(25)
15

---

Published as a conference paper at ICLR 2026
Taking logarithms and averaging under re(- (- s) yields the identity
(ro(A), (A s)] log To(a 8) h(ro(- (..
(26)
where
k(ro(. a, s)) -A a. log (AA a, s]]
(27)
the differential entropy of the conditional path distribution.
Equation equation 26 shows that the expected surrogate penalty log Pe s) differs from the true negative entropy logg (a 3) by exactly the path-entropy term h(re):
Erp log Pe ( s)] log To(a s) + h(ro(- a, s)).
(28)
Identity equation 28 reveals that using the surrogate - log Pe(AA s) in the critic corresponds to augmenting the original maximum-entropy objective with an additional path-entropy term:
Jours E
Er= ra + loggoo 8.) a h(ro(- (ee, st))
(29)
The extra entropy term encourages the conditional path distribution ro(- a. 8) to be diffuse rather than sharply concentrated. This induces a regularizing effect on the critic: actions whose flow rollouts exhibit high variability (large h(To)) receive an additional penalty through the surrogate. Empirically, this produces a more conservative soft Q-function and mitigates the well-known over- estimation issues encountered in off off-policy training.
A.5.2 WHY THIS IS ACCEPTABLE FOR OFF-POLICY FLOW TRAINING
important to emphasize that the purpose of our method not to exactly replicate the original SAC critic. Instead, our goal is to stabilize off-policy RL in the presence of multi-step flow roll- outs, whose training notoriously brittle due to compounding gradients and sensitivity to density- evaluation errors. The path-regularized critic trades small, well-understood bias for substantially improved numerical stability. This trade-off common and often desirable deep RL, where exact Bellman equations rarely satisfied under function approximation.
Moreover, the actor update in our method remains an exact policy gradient for the original maximum-entroppy objective, thanks the pathwise score expansion shown in Appendix A.3] The bias introduced by the critic therefore does not alter the policy objective being optimized; it only affects the value-based shaping signal used during training.
Finally, we note that the proposed flow-based policy construction (i.e., Flow-G and Flow-T) is not specific SAC. Methods such as TD3 rely solely on Q-function targets and do not require evaluat- ing log policy densities. In these settings, our flow policy can be used without any retropy-related complications. The critic design described above is only needed for maximum-entrops algorithms; other off- off-policy methods can directly adopt the same flow-based actor with no additional adjust- ments.
A.6 TEMPERATURE UPDATE (LEARNED a)
When learning a to match a target entropy H:
L(a) Eg Ex - sn) [-a To ( sx) + H)]
(30)
The gradient is Using the joint-path surrogate yields
V.L(a)= = EEsnag [log to |8))) + H7
(31)
K
V Val(a) = -E
log no (Ah.. I 14, At.) log det. J(a)))] + +
(32)
and set H unless otherwise noted.
16

---

Published as a conference paper at ICLR 2026
A.7 PRACTICAL NOTES AND IMPLEMENTATION DETAILS
Rollout length and noise. Use small K (e.g., 4) to control backprop depth and latency. Fix dg (e.g.. 0.10) learn lightweight state head; fixed schedules simplify tuning.
Squashing and Jacobian. Always squash Atx + = tanh(Atg) and include the exact Jacobian in Po of Equ. (15) to keep the entropy term correct.
Targets and normalization. Maintain a delayed target Qs with EMA. Pre-normalization in Flow-T and mild positive gate bias in Flow-G improve early stability.
Gradient flow. Flow-G gates the residual change to damp gradient amplification; Flow-T uses pre-norm residual blocks. Both act drop-in Up inside Equ.
toninee-o--nlnee In the regularized actor loss of the main text (Equ. (12 choose 3 large early to
stay on-replay, then anneal as online data grows, Flow-matching pretraining via Equ. (3) is optional but helpful for sparse rewards.
Efficiency. The entropy term scales linearly in K and action dimension d because it decomposes into per-step Gaussian factors.
Reproducibilityy We evaluate log Pe and its gradient with single noisy rollout per update; addi- tional variance reduction is possible but not required in our settings.
B DETAILED ANALYSIS OF GRADIENT STABILITY
This section provides a more formal mathematical justification for the gradient pathologies in stan- dard flow-based policies (when viewed as RNNs), as discussed in Section 3 and elaborates how our Flow-G and Flow-T architectures address these issues.
B.1 THE VANISHING/EXPPLOONNS GRADIENT PROBLEM IN STANDARD FLOW ROLLOUTS
As established in Equation (5). the standard K K-step flow rollout A,,, Ar. At, vo At, s) is algebraically equivalent to residual RNN, where A,, is the hidden state and fo(.) = At, (-(-) is the RNN cell.
In off-policy RL, the actor loss L(Ø) a function the final action ArR (e.g.. L(
-Q (sh. (anh(Agg)) )) from Equation (10)). To update the parameters of the velocity network 19, the gradient must be backpropagated Thrkk Through time (BPTT) from Atg back to Ato.
Let us analyze the gradient flow. The gradient of the loss T with respect an intermediate state A,, is:
K AA II
VA,, T, = (V
(V Aug L)
0A
044-1 is the Jacobian matrix of the transition: BAi,
where
A ② OA, oA (Ar, fe,t,, Ar,, =I
Ofolty. Ar,,
OA
Substituting this back, the gradient propagation over K steps becomes long product of these Jaco- bians:
K-1
Pfo(ts,Ar..9
V Aio I. -= (VAi D.
II 1=0
[+
(33)
OA
This is the core problem. As identified in the seminal work on LSTM (Hochreiter & Schmidhuber 1997), this long product of matrices is exponentially unstable. The error signal V AAg L is repeatedly multiplied by the Jacobians of the state transition.
Exploding Gradients: If the singular values of these Jacobians persistently greater than 1, the norm of the gradient will grow exponentially, leading to unstable training, as seen in our Naive baseline in Fig.
17

---

Published as a conference paper at ICLR 2026
Vanishing Gradients: Conversely, if the singular values persistently less than 1, the norm of the gradient will shrink exponentially, preventing the error signal from Atx from reaching the parameters that influenced At. At., etc. This makes learning long-term de- pendencies impossible.
This "gradient pathology" is the fundamental technical challenge that makes direct off-policy train- ing of standard flow-based policies notoriously unstable.
B.2 FLOW-G AND FLOW-T AS GRADIENT STABILIZERS
To address the instability of Equation 33 the objective is to design an architecture fo such that the product of Jacobians remains well-conditioned, with singular values centered around 1.0. Our Flow-G and Flow-T designs are explicitly motivated by architectures from the sequence-modeling literature (LSTMs, GRUs, and Transformers) that were invented to solve this exact problem.
The key innovation of LSTM (Hochreiter & Schmidhuber 1997) was the Constant Error Carousel (CEC). LSTM introduced the multiplicative gates (imput, forget, output), which are then trained to learn when to allow error signals into this stable "carousel" and when to use the information stored within it.
The LSTM Precedent:
Our Flow-G (GRU) and Flow-T (Transformer) architectures achieve similar outcome through re- lated, albeit more complex, mechanisms.
Flow-G (GRU:gateed: step becomes:
The Flow-G velocity (Fquation 6 is vo = Di ④ (No Ar,). The rollout
A Ar, + Ati (gs @ (fa (t, At,))
Rewriting this per-dimension (with Di (d) being the d-th dimension of the gate):
(d) = -1-Aot Al + At.g; (d) ol))
This is precisely the update form Gated Recurrent Unit (GRU). The Jacobian of this transition (ignoring terms from 0g:/AA, Aa and 00//0.0 for clarity) is approximately:
dA
diag( At, 9:)
aA
The crucial insight that the gate 9i [0, 1] is a learnable parameter. the network needs to preserve information (and its gradient) across many steps, it can learn to set 9: -> 0 for those steps. 84
When 9: - O, the Jacobian BA I. This mimics the CEC, allowing the gradient to flow unimpeded. Flow-G thus learns to dynamically regulate its own gradient stability, just GRU or LSTM does.
The Flow-T architecture (Equations 9 achieves stability not through explicit gating, but through its architectural design, which is standard in modern Transform- ers. The Jacobian a network, 8A,," well-behaved.
Flow-T (Transformer-decodeed::
aAi I+ At, DAi," Stability hinges on ensuring the Jacobian of the velocity
Flow-T accomplishes this via two key components:
1. Pre-Layer Normalization (Pre-LN): As shown in Equation 8). all inputs to the Cross- Attention and FFN sub-layers are passed through Layer Normalization (LN(-)). Pre-LN ensures the inputs to each layer are normalized, which has been shown to bound the mag- nitude of activations and their gradients, leading much more stable and "well-behaved" loss landscape.
2. Residual Connections: The outputs of the Cross-Attentias and FFN blocks are added to their inputs. This residual stream, ubiquitous in modern deep learning, provides clean identity path for gradients to flow backward, bypassing the complex computations of the sub-layers.
18

---

Published as a conference paper at ICLR 2026
This combination of Pre-LN and residual connections is comeersonne of modem Transformer chitectures precisely because it stabilizes gradients in very deep networks. It ensures the spectral norm of the velocity Jacobian, aA. remains controlled. This achieves the same goal as the
CBC: keeps the overall step Jacobian 0Au+1 close to the identity matrix, preventing the product -A,,
in Equation 33) from exploding or vanishing.
A Necessary Caveat: We must emphasize that unlike the original LSTM's Jocobian Der which
act-1" provides a provable guarantee of ac. = I (in its simplest form), a similar exact proof for Flow-G and Flow-T is intractable. The non-linear complexity of the candidate networks (0y), the gates (g: which also depend on As,). ) and the multi-layer, multi-head attention blocks (in Flow-T) makes a closed-form analysis of the full Jacobian product (Equation 33 infeasible.
_c,_1
However, our designs are not arbitrary. By importing these specific architectural motifs-which which explicitly engineered in the soquence-modeling literature solve the vanishing/exploding gra- dient problem-we create a strong inductive bias towards gradient stability, Our architectures effec- tively approximate constant-norn gradient path, which is what enables stable end-to-end off-policy optimization, our empirical results in Figure o and our ablation studies robustly confirm.
C EXTENDED RELATED WORK
We evaluate our approach against several state-of- thic-oof-hhe-tt art methods, categorized into two groups based on their training paradigm. From-scratch algorithms initialize randomly and learn entirely through environment interaction, while offline-to-online methods first pre-train expert demonstrations before transitioning to online reinforcement learning.
C.I FROM-SCRATCH TRAINING METHODS
The integration of generative models into reinforcement learning has emerged prominent search direction, with particular focus on training policies parameterized by diffusion and flow-based models. This line of work addresses the limitations of traditional unimodal policy representations by leveraging the expressive power of generative models to capture complex, multimodal action distributions.
Early efforts in this domain primarily concentrated on diffusion-bbsed policies. Q-Score Match- ing (QSM) (Psenka et al. 2024) pioneered this direction by establishing a theoretical connection between score functions and Q-value gradients, enabling direct policy optimization through score matching objectives. Building upon this foundation, several advanced methods have been pro- posed: QVPO (Ding al. (024) introduces Q-weighted variational policy optimization for im- proved sample efficiency; DDifPPG (Li et al.] 2024) extends policy gradient methods to diffusion models; MaxEntDP (Dong et al. 2025) incorporates maximum entropy principles; and DIME (Ce- lik et al. 2025) reformulates diffusion policy training through KL. divergence minimization between denoising chains and exponentiated critic targets.
More recently, attention has shifted toward flow-based policies, which offer computational advan- tages over diffusion models through deterministic ODE integration. FlowRL (Ly et al. (025) rep- resents the current state-of-the-arr in this category, proposing Wasserstein-2 regularized policy opti- mization that constrains the learned policy remain within proximity of optimal behaviors identi- fied in the replay buffer.
For our experimental evaluation, we select DIME and FlowRL as primary benchmarks for diffusion and flow-based approaches. respectively, based on their reported performance improvements over earlier methods such as QVPO and QSM. We additionally include QSM in our comparison as established many of the foundational concepts underlying subsequent developments in this field. Meanwhile, classical RI training methods for Gaussian policy, including PPO (Schulman et al.) 2017) and SAC (Haarnoja et al. 2018).
FlowRL (Lv et al. 2025). This approach directly optimizes flow-based policies using off-policy RL with Wasserstein regularization. The critic (s a) follows standard SAC updates, minimizing
19

---

Published as a conference paper at ICLR 2026
the temporal difference error:
(vb = "( (Qu - + d'))))
(34)
The key innovation lies in the actor update, which formulates policy optimization as constrained problem that maximizes Q-values while regularizing the velocity field ve using Wasserstein-2 distance constraint. practice, this is solved using Lagrangian relaxation:
-- E, B.,anDa'' (~0(0,1)
[f(2T (s, a) Qu olloo,, A., t) (a - co)l
(35)
where f(-) is non-negative weighting function, At (1-t)ao+to represents the flow interpolation path, and " denotes the optimal behavior policy derived from the replay buffer. The constraint
adaptively regularizes the policy toward perforeero behaviors when Q > Qu. effectively aligning the flow optimization with value-based policy improvement.
DIME (Celik et al., 2025). This method treats diffusion policies as exponential family distribu- tions and optimizes them via KL divergence minimization. The critic update remains standard:
Lo - [
(st, (stg (s)) a.))²
(36)
The actor update is more sophisticated, defining target marginal through the exponentiated critic Ro(als) = exp(Qu ( (s, a)) (s) and minimizing the KL divergence between the denoising chain and this target:
N
log To (an-ani [nn 8)
T(O)= = ET logmo(an1s) (8,
(37)
7(an An 8)
This approach leverages score matching to align the policy's 's score function with the action gradient of the Q-function, providing principled connec- tion between value-based and score-based learning. The critic follows double Q-Jearning update with target networks for stability:
QSM (Q-Score Matching) (Psenka al., 2024)
( -
(c) (12t1 + min (04(11) 2m1,2
)
(0) Esssaar (14
18811))-B
(38)
where Qo: denotes the target networks. The actor update represents the core innovation, training score function ( ae) to match the scaled action gradient of the Q-function:
(Ø) ae (st, a.) a [oQoo(ss,
(39)
where a controls the alignment strength. This formulation enables the policy to naturally follow the Q-function's 's action gradients, providing implicit policy improvement without explicit action sampling.
C.2 OPFLINE-TO-ONLINE TRAINING METHOOS
While fromtscratch training viable for many reinforcement learning tasks, it often struggles with sample efficiency in complex environments, particularly those with dense rewards. To address this limitation, the offline-to-online paradigm has become a prominent approach. This strategy involves two stages: first, pre-training policy an offline dataset of expert behaviors, and second, fine- tuning this policy through online interaction with the environment.
This paradigm was initially explored with diffusion- based policies, leading to the development of methods such as DPPO (Ren et al. 2024). D3P YYu et al. 2025), Resip (Ankile et al. (2025), and PA- RL Mark al. 2024). More recently, research has extended this approach flow-based policies, which are the focus of our work.
Within the flow-policy literature, methods be categorized by their online fine-tuning algorithm. For on- on-policy fine-tuning, ReinFlow (Zhang et al.) 2025) stands out by successfully adapting a pre- trained flow- policy using the PPO algorithm. For off-policy fine-tuning, FQL. (Park et al.
20

---

Published as a conference paper at ICLR 2026
state-of-the-art. However, crucial character- istic of these off-policy methods is their relancce on an auxiliary, distilled policy for online updates; they do not directly fine-tune the original flow model. Instead, they distill knowledge from the pre-trained flow-based policy into a simpler, unimodal policy that is more amenable to traditional off-policy RI..
2023) and its successor QC-FQL (Li et al. 2025)
For our experiments, we select ReinFlow, FQL, and QC-FQL as benchmarks. Our evaluation pri- marily concentrates on the off-policy methods demonstrate the effectiveness of our proposed direct fine-tuning approach for flow-based policies.
QC-FQL at. 2025) This approach employs three-network architecture: a critic Qo. a one- step noise-conditioned policy pg, and behavior flow-based policy fe. The method extends FQL to handle action chunking by operating in temporally extended action spaces. The critic processes action sequences and updated via:
LQ(0k)= = -Q00 at...., (so,......... a(th-1) ((h-1))
Σ K
1
Qb (sith: aeth....... ac+ ac+2h-1)
(40)
where rf represents the cumulative discounted reward over the action chunk horizon. The one-step policy is trained to maximize Q-values while maintaining proximity to the behavior policy outputs:
(+) Qo(se, (St. + [10 21)
[ap -
(41)
FQL (Park et al., 2025). This method represents a simplified version of QC-FQL with unit action chunks (h 1). The critic follows standard Bellman updates while the actor combines value max- imization with distillation regularization. The actor loss explicitly balances Q-value optimization against behavioral constraints:
L (w) Ea (oa a")] + D.lEs,D (s,= - no(s, 2)12
(42)
where pe represents a pre-trained behavioral clone used for regularization.
ReinFlow (Zhang et al., 2025 This approach augments flow-based policies with noise injection
networks to enable efficient likelihood computation during policy gradient updates. Following a warm-up phase for critic training, the method jointly optimizes the flow-based policy me and noise injection network dg through:
(0, |- Aoo (s, a) log Ro (ax4+41/4,, + R(a, 8; 0, Oata)
(43)
where Arw a) denotes advantage estimates and R((-) provides regularization to prevent excessive deviation from the previous policy.
Key Distinctions. Unlike these baseline approaches, our method enables direct training the flow- based policy via SAC (off-policy methods) without requiring auxiliary distillation actors, surrogate objectives, or complex multi-network architectures. The Flow-G and Flow-T parameterizations provide gradient stability while maintaining the expressive power the original flow-based policy throughout training.
D EXPERIMENTAL DOMAIN
To comprehensively evaluate our method, we conduct experiments across diverse suite of simu- lated environments. We utilize the classic MuJoCo benchmark (Todorov al. 2012) for standard from-scratch reinforcement learning. To assess performance in the more challenging offline-to- online setting, particularly with sparse rewards, employ complex manipulation tasks from OG- Bench (Park et al.) 2024) and numan-demonstreddd -baed tasks from Robomimic (Mandlekar et al. 2021). Visualizations of these environments are presented in Fig. 8
21

---

Published as a conference paper at ICLR 2026
(a) Ant
(b) Humanoid
(c) HalfCheotah
(d) cube-double
(c) cube-triple
(f) cube-quadruple
gg can
(h) square
Figure 8: Visualizations of the diverse simulation environments used for evaluation. Subfigures (a-c) show the MuJoCo locomotion tasks. Subfigures (d-f) depict the complex, sparse-reward ma- nipulation tasks from OGBench. Subfigures (g-h) illustrate the cemonstration-basee tasks from Robomimic. This selection provides a comprehensive testbed for evaluating both from-scratch learning and offline-to-online fine-tuning.
D.1 MuJoCo ENVIRONMENTS
We evaluate our method on six standard continuous control tasks from the MuJoCo physics sim- ulation benchmark (Todorov et al. 2012): Hopper-v4, Walker2d-v4, Hal fCheetah-v/ Ant-v4, Humanoid-v4, and aamaanoidttandu--v44 These environments feature simulated robots with varying degrees of complexity, where the primary objective to learn a locomotion policy that maximizes forward velocity without falling. They serve standard measure of perfor- mance for from-scratch RL algorithms.
D.2 OGBENCH ENVIRONMENTS
From OGBench Park et al. 2024 we select four challenging manipulation do- mains using their publicly available single-task versions. The selected domains include cube-dose sube-double// / UR-5 arm to place multiple objects in target locations, receiving a reward -Twron. where Twrong the number of incorrectly placed cubes. The eube- -hrile and cube-quadruple tasks are particularly difficult to solve from offline data alone, providing a rigorous testbed for the sample efficiency of offline-to-onlne algorithms. In the offline phase, we use the official 100M-size datase
tasks. In the cube tasks, an agent must control
D.3 ROBOMIMIC ENVIRONMENTS
We use three robotic manipulation tasks from the Robomimic benchmark (Mandlekar al. 2021 utilizing the multi-human datasets which contain 300 successful demonstration trajectories per task. The tasks are selected represent range difficulties: Li ft, a simple pick-and-place task volving cube; Can, an intermediate task requiring placing can into bin; and Squa the most challenging task, which requires the precise insertion of a square nut onto peg. We use the official Multi-Human (MH) dataset, containing 300 mixed trajectories per task, for offline pre-training.
22

---

Published as a conference paper at ICLR 2026
Table 1: Details of the experimental environments. The tasks span classic continuous control with dense rewards (MuloCo), complex manipulation with sparse rewards (OGBench), and challenging imitation-bases tasks also framed with sparse rewards (Robomimic). This selection provides com- prehensive benchmark with diverse state spaces, action dimensions, and reward structures. We use same dataset configuration
al. 2025).
Tasks
Reward Type Dataset Size Episode Length
Action Dimension
MuJoCo
Hopper-v4
Dense
1000
3
Walker2d-v4
Dense
1000
6
HalfCheetah-v4
Dense
1000
6
Ant-v4
Dense
1000
8
Humanoid-v4
Dense
1000
17
HumanoidStandup-v4
Dense
1000
17
OGBench
cube-double
Sparse
IM
500
5
cube-triple
Sparse
3M
1000
5
cube-quadrule-1100M
Sparse
100M
1000
5
Robomimic
lift
Sparse Sparse
31,127
500
7
can
62,756
500
7
square
Sparse
80,731
500
7
IMPLEMENTATION DETAILS FOR EXPERIMENTS
In this section, introduce the implementation details of the hyperparameter setting and network structures. We first begin with the from-scratch training:
E.1 FROM-SCRATCH TRAINING SETTING
In from-scratch training, we develop our algorithm based on CleanRL (Huang et al. 2022) which widely used benchmark codebase, where also the same implementation of PPO, SAC in it. The hyperparameters for SAC, PPO, and DIME are available in Tab. 2 and 4 For FlowRL, the official implementation except for unifying the parameter quantity. We run 5 seeds for all experiments and all plots use a 95% confidenceinterval.
Table 2: Common Hyperparameters for SAC Algorithms
Parameter
Value
Optimizer
Adam
(b1 0.5 for Flow-based approaches)
Batch size (M)
512
Replay buffer size
1 106
Discount factor (7)
0.99
Policy learning rate
3 x 10-4
Critic learning rate
10-3
Target network update rate (T)
0.005 for SAC
1.0 for Flow, Flow-G, Flow-T
Learning starts
50,000
Entropy coefficient (a)
0.2 (initial value)
Target entropy
-dim (A) for SAC, 0 for Flow, Flow-G, Flow-T True
Automatic entropy tuning
Number of online environment steps
1 10
23

---

Published as a conference paper at ICLR 2026
Table 3: Hyperparameters for PPO
Table 4: Hyperparameters for DIME
Parameter
Value
Parameter
Value
Optimizer
Adam
Discount factor (7)
0.99
Parallel envs
32
Target network update rate (r)
1.0
Discount factor (?)
0.99
Policy tau
1.0
GAE lambda
0.95
UTD
Learning rate
6 x 10-4
Policy delay
Num steps
1024
Batch size
512
Num minibatches
1
Critic v min
-1600
Update epochs
10
Critic
1600
Max grad norm
10.0
Actor Ir
3 x 10
Clip coefficient (c)
0.2
Critic Ir
x 10 10-4
Entropy coefficient
0.01
Entropy coefficient (a)
1.0 (init)
Total Timesteps
x100
Target entropy
6.0
Total Timesteps
x100
Architectures of the velocity network in flow-based policies (Figs. 9 11 We detail the network parameterizations for the velocity field used inside the flow rollout Equ. (4) Across all variants, the policy starts from a state-conditioned base Ar, (N, Ia), performs K Euler updates A Ar; + A/; (li, A s). and then applies tanh During training, we optionally pair Vi with squashing obtaín the final action Lanh (Atx) head to define the per-step Gaussian transition factors used by our noisy/iieeliooodfiiieee rollout.
Flow Pulicy (Classic Network)
Flow Flald
Enbegrastion
Output
Imprati
hord
Hine Syptt
XLX
Figure 9: The flow-based policy designed with classic networks. The velocity modeled with an arbitrary network; here, we use MLP as the representative. The whole flow rollout corresponds to the recurrent computation of the residual RNN.
Fig. 9 Classic (MLP) velocity network. The baseline flow-based policy instantiates vy with a feed-forwart network that is conditioned on the current intermediate action Ar,, the environ- ment state 8, and the normalized time index 4, Concretely, the input token the concatenation 8; AG; t J. followed by a shared MLP trunk and two small heads: (i) mean head ps(()) that produces the deterministic velocity
vo(4;, Ar,, 8) no (1,, 4r,;
and (ii) a log-standard- -evvittdd head log (e()) that parameterizes the per-step transition variance training. Plugging this vo into Equ. (4) yields the
when we use the noisy rollout for likelihood- standard residual update A At, + At, no (o[[s; A,;44) Algebraically, this is residual RNN step with residual function fo(.) = At, (-).). matching our sequence-men modell view in Equ. (5)
Fig, 10: Gated velocity (Flow-G). To stabilize gradients across the K sampling steps, replace the plain MLP velocity with GRU-style gated update. Let fo (gate network) and fr (candidate network) be two MLPs taking [s; it as input. Define the update gate and the candidate
9: = ([s Ag;t())) f1l)
to = ((s; tt))))))
where o-)) is the logistic sigmoid and Φ(-) is bounded activation (e.g., tanh). The gated velocity then
vo(li, Av, 8) 91 (6o Ar.);
24

---

Published as a conference paper at ICLR 2026
Flow Policy au GRU
Tepats
Flow Pickd
Integration
Ondyst
A.K-E
Baad Veletty
Ntrwch
Figure 10: The flow-based policy designed with Gated velocity. The velocity is modeled with both the gate network and the candidate network. The whole flow rollout corresponds to the recurrent computation of GRU.
which, when inserted into Equ. 4 yields the GRU-like residual step
A,, + A4, (y: @ (ūp As,)),
exactly as in Equ. (6) Intuitively, interpolates between "keeping" the current intermediate action (gi 0) and "rewriting" it by the candidate proposal (g: x 1). As in Fig. 9 also include log (-(-)
head for the per-step Gaussian factors used by the noisy rollout.
Truadesse (Dacadee Lapar
Homer
Flan Fild
Betogg stias
Oseyst
Figure 11: A schematic..
Fig. 11: Transformer--ecoderr velocity (Flow-T). Here implement vo with Transformer- style, pre-norm residual block that conditions on the state through cross-attention. We first form separate embeddings for the action-time token and the state:
A EA (o. A.,), T Es (s)
as in Equ. 7 where EA, E's are linear projections and o,, é. are positional/fcature encoders. We stack L (L in this figure) pre-norm decoder blocks. In each layer - I,... ... L, the action token is refined by state-only cross-attention and feed-forware network (no token-to-token mixing across time positions):
y(() A. + Cross/ (LN( context LN(
T Y,O) FFN(LN(Y
Equ. (8. Finally, the decoded token is projected to the velocity space
Up (a s), = W.(LN(1!))))
and the rollout step follows Equ.
A A, + At. W.(LN(
which matches Equ. 9 As in the other variants, parallel log (-(-) head provides per-step vari- ances for the Gaussian transition factors.
Takeaway: mapping to sequential models. Under our sequence-model perspective, the classic MLP velocity in Fig. realiizes residual RNN step in Equ. (5). the gated velocity in Fig. realizess a GRU-style residual update in Equ. (6). and the decoded velocity Fig. II realizes a Transformer Decoder refinement in Equ. 77) (9. All three drop-in parameterizations of Uy inside the same
flow rollout in Equ. differing only in how they regulate and condition information flow across rollout steps.
25

---

Published as a conference paper at ICLR 2026
Table Actor (velocity) architectures inside the Ktsepp flow rollout A.,.: A., + oo .t. All variants apply tanh squashing with Jacobian correction. Notation: da: [AA. Transformer d-64, heads =4, layers n2=2.
Aspect
Classic (MI.P)
Flow-G (GRUI-gate)
Flow-T (Transformerddecdde))
A: token + time cmb
Conditioning input
(et Aeçi te]
[s: Aışi time comb(t))
state as memory
MLP 256 -3:6--2200 256 RetC
Gate: 128 - du (awlar)
Decoder X =2
Backhone / blocks
Cand: 128 - du (swlan)
self-only SA, cross-attn( (+), FEN 4N
Hu Aqittl)
gc=o(fa), 6=50 tanh(fa) ea=gi @ (6-Ax,)
Velocity form
(6a E
tanh
tanh [-5, 2]
Log clamp
|-5, 2]
1- 2]
Action sampling steps K
4
Gate head init: W= 0, 6m5.0 hidden 128
d=61, ny= 22 32 يالاد
Notable inits/ dims
A A
step update
A AA,
Same as Classic
Same as Classic
E.2 OPFLINE-TO-ONLINE TRAINING SETTING
The network design in offline-to-nnline training is similar to the from-scratch training. Recall the actor loss:
(()) Po sn.) - Qu (f) ||ag - an a,1 (ah, 8h.) B.
observed that the setting of hyper-parameter 33 highly influences the training, where the regu- larization decides whether the optimized policy stays close to or not to the policy in the buffer. We basically adopt the same setting of 3 as (Li et al. 2025), where we detail in the Table
A comparison of the regularization parameter 3 across different environments and algo- rithms. The notation a/b specifies the value of the regularization parameter 3 for the offline learning phase (a) and the subsequent online learning phase (b). For instance, 10000/1000 1000 indicates that 10000 used for offline training and 3 1000 for online training.
Table 6:
Environments
FQL
QC-FQL
Flow-G
Flow-T
scene-sparse-
300
300
300
300
cube-double-*
300
300
300
300
cube-triple-*
300
100
100
100
cube-quadruple-1000
300
100
100
100
lift
10000
10000
10000/1000
10000/1000
10000
10000
10000/1000
10000/1000
square
10000
10000
10000/1000
10000/1000
Table 7 summarizes the actor-side architectures and hyperparameters for our offline-to-online vari- ants. We adopt action chunking (horizon If), which has been shown to be effective on complex tasks eL al. 2025). The parameter counts of Flow-G and Flow-T are less than or comparable to that of QC-FQL. We also use fewer denoising/sampling steps K than QC-FQL to improve efficiency with- out degrading training quality. For stability, we set the SAC target entropy to 0 and employ a fixed sampling noise level-contrary to from-scratch setting, where separate network adaptively tunes the noise schedule.
Sampling Steps Justification. We justify our choice of sampling steps (K = for our flow poli- cies. K = 16 for diffusion baselines) based two factors. First, this reflects the inherent efficiency of the models, as flow-based models (especially Rectified Flow) generally require significantly fewer integration steps than diffusion models require for denoising. Second, and most importantly,
26

---

Published as a conference paper at ICLR 2026
Table 7: Offlneeto-online settings and actor-specific hyperparameters.
Aspect
QC-FQL
Flow-G
Flow-T
(512x4)
(512x4) + gale
Decoder T. =2, d=128 Bg FPN
Actor
GELUU, I.
(h 250, swish)
0= 3 (50 tamh(() - a) decoder head
Velocity form
vols, o, f) by MLP
==0(f2)
(self+cross attn)
Action sampling steps
Action sampling sleps
Action sampling steps
Flo / steps
K=10 deterministic
K=4
==4
Sampling
0.10 autotune 0.2),
autotume (init 0.2).
SAC entropy (a)
NIA (no SAC)
01=38 10 12=0
=x=33
R =
tanb squasb
tanh Jacobian (for log-prob)
tanb Jacobian (for log-prob)
Action range
(deterministic)
gale head: W=0,, 6=5.0 hidden 256
Gate init / dims
Transformer
128, j 4, n
only in FTFN:
hidden
(512, 512, 512)
(512, 512, 512)
decuter per row ahove)
Action chunking
Trie
True
True
Opt LR/ WD
Adam, 3 10
Adam, 3 10
Adam, 3 10
Batch / /
256 / 0.99 0.005
256 00.9900.00
256 0.99//9.00... 0.000
adopted these values to ensure a fair and direct comparison with the key baseline papers. Our use of 4 sampling steps for SAC Flow-G and SAC Flow-T follows the established setting in the Rein- Flow (Zhang et al. 2025) baseline. Similarly, our use of 16 denoising steps for the diffusion- based baselines (e.g., DIME) matches the hyperparameter used in the DIME (Celik al. 2025) paper.
F MORE EXPERIMENT RESULTS
In this section, we show more tested experiments.
F.1 ADDITIONAL FROM-SCRATER RESULTS
PPO As shown in Tab. 3 we use stabler parameter (num minibatch=1), making PPO's data ef- ficiency little lower. We report PPO's training over larger number of steps in Fig. 12 The results show that the final performance of our PPO implementation is comparable to or exceeds other open sources results.
SAC To ensure fair and rigorous comparison in main results, we intentionally applied unified set of hyperparameters for each method across all tasks. Consequently, SAC to exhibit poor performance on Ant-v4 under this unified settings. In this section, we conduct additional experiments under task-specific hyper-parameters in Tab. 8 Fig. [3] shows that with task-specific hyperparameters, SAC baseline can indeed converge to 4700 return Ant-v4.
Results on Tasks with Sparse Rewards We finally test Cobomimic-Canl and OGBench-cube. Fig. [4 shows that all methods struggle on these two hard-exploration, sparse-rewasd tasks without pretrain, highlighting the necessity of offline-to-online training.
Additional evaluation Fig. 15 shows the interquartile mean (IQM) with 95% stratified bootstrap confidence interval as suggested by Agaawaa et al. (202)).
We also report the probability of improvement metric estimates the likelihood that our algorithm X outperforms a baseline Y on randomly selected task, formalized P(X > Y). Consistent with the method-
Agarwal et al. 2021 in Fig. 16 Specifically, the
27

---

Published as a conference paper at ICLR 2026
15000
3040
1000
12500
Rotum
Retur
7500
5000
1000
2500
Steps
Staps
Steps
(a) Hopper
(b) Walker2d
(c) HalfCheetah
6000
3000000
Rcturo
6000
Retur
2000000
2000
4000
100000
0.2
Steps
Steps
Steps
(d) Ant
(c) Humanoid
(f) HumanoidStandup
SAC Flow-T
SAC Flow-G
PPO
Figure Training curve PPO over larger number steps.
6000
Table 8: Task-cpecific Hyperparameters for SAC Ant-v4
4000
Parameter
Value
Return
Optimizer
Adam 1024
2000
Batch size (M)
Replay buffer size
106
Discount factor (7))
0.99
Policy learning rate
2 x 10
-2000
Critic learning rate
0.0
0.2
0.4
0.6
0.8
Target network update rate (7)
0.001
Steps
lc6
Learning starts
50,000
Entropy coefficient (ex)
0.2 (initial value)
SAC widkusifl Hiporpenciion
SAC Inbopatic Hepappasncion
Target entropy
-dim(A)
Automatic entropy tuning
True
Figure 13:
Training curves of SAC in Ant-v4, For the Ant task, SAC re- quires careful hyperparameter tuning to achieve stable convergence.
Number online environment steps
x 106
28

---

Published as a conference paper at ICLR 2026
-000
700
Relum
Relum
-900
1000
-1100
0.0
0.2
02.
0..6
0.8
1.0
0.0
01.
0.4
0.6
0.8
1.0
Steps
166
Steps
166
(a) Robomimic-Can
(b) Cube-Double-Task?
SAC Flow-T
SAC Flow-G
FlowRL
DIME
QSM
SAC
PPO
Figure 14: From-scratch training results on Rohomimic- and OGBench-cube. All methods struggle on the hard-exploration, sparse-reward tasks without pretrain, highlighting the necessity of oflline-to-onlne training.
3000
6000
12500
10000
Retura
2000
4000
Recmm
7500
2000
2500
0
0.0
0.2
0.4
0.6
0.3
0.0
0.2
0.4
18.6
0.5
1.0
0.0
0.2
0.4
0.6
9.5
Steps
leb
Steps
166
Steps
166
(a) Hopper-v4
Walker2D-v4
HalfCheetah-ve
400000
0000
300000
Metum
Retum
2000
4000
2000
1000000
2000
0.0
0.2
0.1
0.6
0.8
1.0
0.0
0.2
0.1
0.6
0.5
1.0
0.0
0.2
0.1
0.6
0.5
1.0
Steps
les
Steps
1eb
Steps
166
(d) Ant-v4
Humanoid-vá v4
HumanoidStandup.-e
SAC Flow-T
SAC Flow-G
FlowRL
DIME
QSM
SAC
PPO
Figure 15: From-scratch training results on Mujoco (IQM return).
ology of AAgarwal et al. (021)) we compute point estimates and 95% confidence intervals using stratified bootstrapping with 200 resamples across all tasks and seeds. This approach provides bust pairwise comparison that mitigates the skewing effects of outlier performances often observed aggregate metrics.
F.2 ADDITIONAL OFFLINE-TO-ONLINGG RESULTS
Fig. Robomimic.
abd Fig. 18 shows complete offline-to-online training performance in OGBench and
Additional evaluation Fig. 19 shows the interquartile mean (IQM) with 95% stratified boot-
strap confidence interval as suggested by /Aaawal et al. (202T). We also report the probability-of- improvement in Fig. 20
29

---

Published as a conference paper at ICLR 2026
Algoriihm Flow-T
Algoritim X: Flow-G
РРC
P90
Algaribon
Algarihm Y
QSM
FloRR
FlowRL
Fln-G
PlowTT
0.0
0.4
0.8
1.0
P(Flow-T
P(Flow-G
Flow-T against baselines
(b) Flow-G against baselines
Figure 16: -mmpoooiiey in Mujoco benchmark. Each row shows the probability of improvement, with 95% bootstrap Cls, that the algorithm X on the top outperforms algorithm Y the left, given that X was claimed to be better than Y.
F.3 ADDITIONAL ABLATION STUDY
We further analyze the sensitivity to the specifics the GRU and transformer. In main results, we set the default transformer parameter of Flow-Tae layer=2, head=4, d modell=96. Fig. shows SAC Flow-T is robust to these three specifics of transformer.
For Flow-G, we analyze its sensitivity gate network's width. We the default gate width to 512. Fig. [2 shows that SAC Flow-G maintains stable convergence with gate width = 256. However, insufficient capacity the gate network leads performance degradation (gate width=64).
F.4 FLOW-T/G CAN BE USED IN OTHER OFF-POLICY RL ALGORITHMS
SAC flow stabilizes gradients during BPTT to ensure stable off-policy RL training, functioning independently of specific algorithms. We further extended our evaluation to TD3 Fujimoto et al. (2018). Fig. 23 demonsttaases that our Flow-T/G architecture remains effective with TD3. contrast, directly fine-tuning the flow using TD3 results in failure converge sub-optimal performance.
F.5 PERFORMANCE ANALYSIS: DECOUPLING ARCHITECTURE FROM ALGORITIM
To further investigate the source of our performance gains and explicitly disentangle the contribution of our architectural design (Flow-T) from the algorithmic objective, we conducted an additional ablation study comparing our method against DIME (Celik et al. 2025) under identical architectural conditions.
DIME Flow-T Implementation. Standard DIME optimizes a variational lower bound involving a forward process prior and reverse path loss. To test whether these specific algorithmic components are the drivers of performance, we implemented DIME Flow-T. This variant retains the full DIME objective functions-includingg the injection of the prior gradient and the reverse path likelihood ratio-but replaces the standard backbone with our proposed Flow-T velocity parameterization.
Results and Discussion. We compared SAC Flow-T (our proposed method) against the new DIME Flow-T variant. The results illustrated in Figure 24 We observe key findings:
30

---

Published as a conference paper at ICLR 2026
4.0
(a) Cube- sube-Quadruple- (b) Cuhe-uuuuuuuuc Luabe--udrruple- (c) Cube-Quadruple- (d) Cube-Quadruple- (e) Cuhe-Quadruple Task5
Task1
Task2
Task3
Task4
00
62
16
(l)
Cube-Triple (g)
Cube-
Cube-Triple Triple- (h)
Cube-Tripp Triple- (i)
Cube-Triple
( Task5
Cube-Triple
Task
Task2
Task3
Task4
Says
(k)
Cube-Double- (l)
Cube-Double-
(m) Task3
Cube-Double-
(n)
Cube-Doublee
(o)
Task1
Task2
Task4
Task5
SAC Flow-T
SAC Flow-G
FQI
QC-FQL
training performance in OGBench. This figure illustrates the comprehensive training performance across tasks. All methods are trained on 1M offline updates followed by IM online interaction steps. Our methods, SAC Flow-T and SAC Flow-G, achieve competitive often superior performance across the evaluated benchmarks.
Figure 17: Complete
1.0
Success Rale
Success Rate
0.6
0.6
0.6
0.4
0.4
0.0
0.0
0.0
0.0
0.5
Steps
Steps
le6
Steps
le6
(a) Robomimic-
(b) Robomimic Can
(c) Robomimic-Sane
SAC Flow-T
SAC Flow-G
FQL
QC-FQL
ReinFlow
Figure 18: Complete offline-to-online training performance in Robomimic.
Architecture enhances DIME: DIME Flow-T significantly outperforms the original DIME baseline (as shown in main results). This confirms that Flow-T architecture provides substantial benefits in gradient stability and expressivity, regardless of the underlying RL objective.
SAC Simplicity prevails: Crucially, shown in Figure 24 SAC Flow-T consistently matches or outperforms DIME Flow-T (e.g., in HumanoldStanduup. This suggests that the additional algorithmic complexity of DIME does not yield marginal gains once the velocity network properly stabilized by Flow-T.
These results, combined with the successful application of Flow-T to TD3 (Section 5). demonstrate that the primary bottleneck in off-policy flow training is gradient instability in the rollout, which
31

---

Published as a conference paper at ICLR 2026
1.0
Sac0og Rato (K2M)
Stazren Bite
8.-
0.5
044
0.4
0.2
0.2
1x
0.2
0.0
0.0
08
0.0
1.0
0.5
1.3
2.0
0.5
1.5
2.0
0.0
0.5
2.5
Stops
Staps
Siega
Stope
(a) Cube-Double-Tie
(b) Cube-Triple-Tass
(c) Cube- Saae--uudduupeeaaaee
(d) Robomimic
SAC Flow-T
SAC Flow-G
FQL
QC-FQL
ReinFlow
Figure 19: Aggregated offlnne-o-onling performance on OGBench and Robomimic benchmarks (IQM).
Algocithm X: SAC Flow-G
Algorithm X: SAC Flow-T
FOL
SAC Flow-G
Algoriten Y
Algonthm Y
QC-FQL
SAC Flo-T
FQL.
P(SAC Flow-T
(a) SAC Flow-G
(b) SAC Flow-T
Figure 20: Probability-of-mproeeen for offline-to-online setting OGBench benchmarks.
Flow-T architecture effectively resolves. Once stabilized, the standard SAC objective is sufficient to achieve state-of-the-ar performance.
G LLM USAGE DISCLOSURE
We used large language model solely for writing polish. Its assistance was limited grammar and style edits, wording suggestions for titles/abstractcaatiions consistency of terminology, and minor LaTeX phrasing (e.g., figure/table captions and cross-reference text).
32

---

Published as a conference paper at ICLR 2026
6000
Return
4000
2000
0.0
0.2
0.4
0.6
0.8
1.0
Steps
le6
default
n_ head
layer
modell118
head-6
layer=3
model=64
Figure 21: Ablation study on specifics of SAC Flow-T, Our SAC Flow-T are robust the specifics of the transformers.
6000
Return
4000
2000
0
0.0
0.2
0.4
0.6
0.8
1.0
Steps
le6
default
gate width 256
gate width = 64
Figure 22: Ablation study on specifies of SAC Flow-G. SAC Flow-G maintains stable convergence with reduced gate widths. However, there exists degradation in final performance when the gate width is extremely small (64).
6000
5000
200000
Return
2000
Return
Roturn
3000
150000
1000
2000
1000
50000
0.2
Steps
Steps
Steps
(a) Hopper-v4
(b) Walker-v4
(c) HumanoidStandup-va
TD3 Flow-T
TD3 Flow-G
Naive TD3 flow
Figure 23: Evaluate our Flow-T/G architecture on TD3.
33

---

Published as a conference paper at ICLR 2026
300000
Return
2000
Return
Return
200000
1000
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.5
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Steps
leb
Steps
leb
Steps
166
(a) Hopper-v4
(b) Walker-v4
(c) HumanoidStandup-va
Flow-T
DIME Flow-T
DIME
Figure 24: Evaluate our Flow-T architecture on DIME loss.
34


Reviews:
[{"review_id": "pJ00whjSsA", "reviewer": "Reviewer_J2Zm", "summary": "The paper extends SAC to flow matching policies. In order to do so, the authors realize that flows can be expressed as residual RNNs, showing the same problems with gradient instabilities. To counter-act this effects, the authors introduce a GRU like flow architecture (FLOW-G) and a transformer like flow architecture (FLOW-T). In ordert to compute the entropy for SAC, the authors show that the flow can be transformed into a stochastic flow using Gaussian noise at each time step and we can take the gradient of the path derivative of the action trajectories to compute the gradient of the entropy. Both approaches are benchmarked on online RL and offline-2-online settings and show competetive performance.", "questions": "- The authors should investigate why DIME is not performing well for them. Maybe the v4 of the benchmark version requires different hyper-parameters\n- Please explain the algorithmic differences to DIME in much more detail (by adding stochasticity to the flow integration steps your process basically becomes a diffussion process, at least the generative process is the same (except for the parametrization)). \n- Please also elaborate on why the noise augmented rollout preserves the same marginal densities. This needs further justification or a reference where this has been shown.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The idea of using time-series architectures to improve the instabilities of flow-matching gradients in RL is novel and promising\n- the paper presents the first max-ent framework for flow matching\n- benchmarks seem to be competitive, yet I have some doubts regarding some evaluations (see weaknesses)", "weaknesses": "- The relation to existing methods is unclear: By introducing the noise augmented rollouts, the approach is very similar to diffusion based methods such as DIME. In DIME, they introduce a lower bound to the entropy which consists of the noise process and reverse noise process path. In this paper, only the reverse noise path is considered fro the gradient, this is the only difference that I can see (except for the architectures). However, while the derivation of using the gradient of the path measure for the entropy seems to be correct, it is unclear why the path measure should be used as additional reward in SAC. This is just done without any further justification. In contrast, the lower bound introduced by DIME provides a much more principled derivation also for the critic update (Showing that the critic update optimizes a clearly defined objective, which is not the case here) while resulting in almost the same algorithm. \n- The relation to  DIME needs to discussed in much more detail. As far as I can see it, the paper can be seen as an extension of DIME with more sophisticated architectures (GRU or transformer instead of residual RNN) and the missing forward noise process. However, it is unclear whether this modification benefits or harms the performance.  \n- I do not believe the reported results for some baselines are correct. DIME is performing much better then the reported results (even though it is evaluated on the benchmark versions v3 instead of v4, but I do not believe that this makes a substantial difference). DIME performs competitive to much stronger baselines than standard SAC such as BRO or SimbaV2, also on harder tasks than presented in this paper. While these baselines use Gaussian policies, they should be added to evaluate if the reported results are really SOTA performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B1kxpbBuyR", "reviewer": "Reviewer_rXyd", "summary": "The paper tackles instability when training flow-based policies with off-policy RL by viewing the K-step rollout as a deep residual RNN and proposing two stabilizing architectures that resemble a GRU-style gated cell and a Transformer decoder.", "questions": "- Line 132 “yet a single unimodal Gaussian cannot capture inherently multimodal action distributions, a limitation that is especially harmful in long-horizon robotic control” —  Actions being unimodal and the horizon of the tasks seem to be two rather independent things, can the authors elaborate more on this point?\n- Line 398 “Among all experiments, the sampling steps of flow-based policies are set to 4, and the denoising steps of diffusion policies are set to 16. More details of the experimental setting are described in Appendix C and Appendix D.” — How are the 4 steps and 16 steps determined respectively", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The problem is meaningful and of great importance to flow policies applied to RL\n- The proposed approach provides new perspectives to solve the problem, and the resulting solution is clean and simple\n- Empirical results demonstrate the strong performance of their approach, mostly matching or exceeding the state of the art", "weaknesses": "- Line 159 “Off-policy methods (e.g., SAC, TD3) are highly sample-efficient (Mambelli et al., 2024), yet directly backpropagating through the K-step action sampling is often unstable, especially for large K (Park et al., 2025).” — I think this core technical challenge deserves a more serious presentation with necessary math beyond verbal explanations. \n- The current presentation of Flow-G/T assumes that the architectures are natural choices inherent from modern improvement over RNNs. However, more mathematical details, e.g., showing the Jacobians after the proposed fixes, might strengthen the authors' argument\n- Figure 4: I am a bit confused about the results presented in Figure 4. For example, why is the reward for PPO so low, and looks like it just barely learns anything? The results do not match (are not even close to) common open source implementations, such as the ones reported in CleanRL (https://docs.cleanrl.dev/rl-algorithms/ppo/#experiment-results_2). Also, SAC does not seem to solve Ant-v4. That said, can the authors report how they came to the chosen hyperparameter configuration for all the algorithms?\n- Minor:\n    - “State-of-the-art sample efficiency and performance.” — This is not counted as a separate contribution, but more of the evaluation results of the first two contributions\n    - Color scheme chosen in a number of figures, e.g., Figs. 4 and 7, is visually indistinguishable", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MRlDqGiW1z", "reviewer": "Reviewer_bZhq", "summary": "This paper introduces **SAC Flow**, an off-policy reinforcement learning algorithm for **flow-based policies**.\nIt reinterprets the multi-step flow rollout as a **residual RNN**, linking instability in prior methods to recurrent gradient issues.\nTo stabilise training, the authors reparameterise the **velocity network** using sequential architectures: **Flow-G** (GRU-gated) and **Flow-T** (Transformer-decoded).\nA **noise-augmented rollout** allows tractable entropy computation in SAC.\nExperiments on **MuJoCo**, **OGBench**, and **Robomimic** report state-of-the-art sample efficiency over existing diffusion- and flow-based baselines, with ablations assessing gradient stability and robustness.", "questions": "see weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* **Novel conceptual framing:** The sequential-model reinterpretation of flow rollouts (residual RNN ≈ flow ODE) is insightful and could influence future flow-based RL research.\n* **Architectural innovation:** Flow-G and Flow-T represent principled design choices inspired by GRUs and Transformers, addressing stability through gating and pre-norm residuals.\n* **Algorithmic completeness:** The integration into SAC with a noise-augmented rollout is well-motivated and yields a self-contained training procedure.\n* **Empirical breadth:** Evaluations cover diverse benchmarks (MuJoCo, OGBench, Robomimic) and both from-scratch and offline-to-online regimes.\n* **Reproducibility:** Appendices provide detailed architectures and training algorithms.", "weaknesses": "**Summary:**\nThe paper contains a **promising conceptual contribution** — interpreting flow-based policies as sequential models and borrowing stability mechanisms from RNN/Transformer design — and achieves competitive empirical performance.\nHowever, the **central motivation (gradient instability in naïve SAC-Flow)** is **not convincingly evidenced**, and several key figures are inconsistent or uninformative.\nThe experimental section needs stronger causal and statistical support to meet ICLR’s bar for empirical soundness.\n\n### **Weaknesses**\n\n#### **1. Insufficient evidence for the claimed instability**\n\nThe paper’s central claim — that “Directly backpropagating through the K-step action sampling is often unstable” — lacks quantitative support.\nEven the ablation on velocity network parameterizations (Fig. 13a) shows gradient explosion only near the end of training, while Fig. 6(a) implies immediate collapse if I'm not mistaken.\nThese two plots contradict each other, and neither connects gradient norms to learning-curve failure.\nWithout aligned gradient-and-performance analysis, the claim of severe instability remains unproven.\nSo, I felt that aggregated plot of Fig. 6(a) is therefore potentially misleading and should be clarified or replaced with joint plots of gradient magnitude and return.\n\n#### **2. Ambiguous or uninformative results (Figs. 4g,h)**\n\nPanels (g) *Robomimic-Can* and (h) *Cube-Double-Task2* show all methods failing equally.\nThe authors acknowledge this only briefly as motivation for the offline-to-online setup (p. 7), but these subfigures add no evaluative value.\nThey should either be removed or replaced with quantitative evidence (e.g., normalized success rates, exploration metrics) illustrating why SAC Flow fails on sparse rewards.\n\n#### **3. Lack of aggregate evaluation across environments**\n\nPer-task curves make it difficult to judge overall performance.\nFollowing **Agarwal et al., NeurIPS 2021**, the authors should report **aggregate IQM returns** with **95 % bootstrap confidence intervals** and **probability-of-improvement** or **rank-based summaries** across all benchmarks.\nThis would clarify whether SAC Flow-T and -G consistently outperform baselines rather than winning selectively.\n* Agarwal et al., NeurIPS 2021 — “Deep Reinforcement Learning at the Edge of the Statistical Precipice.”\n\n#### **4. Missing causal evidence for performance gains**\n\nThe connection between gradient stabilization (Flow-G/T) and improved returns is asserted but not verified through controlled experiments isolating that factor.\nSo, it's related to my first point but without a quantitative baseline showing naïve SAC-Flow failure, it is unclear whether the gains stem from the proposed architectures or from implementation details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5LQXvtte5V", "reviewer": "Reviewer_dSSA", "summary": "This paper proposes to learn flow policies using RL. Unlike previous work in behaviour cloning that learns flow fields that match the demonstration trajectories, this paper allows the flow field to be optimised against an explicit objective function. Previous work in RL for flow fields has experienced problems with gradient instability --- this paper shows that using a GRU or transformer to stabilise the gradients, and show learning improvement relative to baselines on a series of benchmarks.", "questions": "How sensitive the performance is to the specifics of the GRU and transformers?\nAre there kinds of problems where this technique will not succeed?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Overall this paper proposes an interesting and novel modification to learning control flow fields. The paper is clearly written and the paper is careful to provide each step of SAC for flow fields, including the the proposed modifications. The authors show performance on both from-scratch problems and offline-to-online performance.\n\nThe experimental results show that the technique performs well on several baselines.", "weaknesses": "There are not significant weaknessess to the paper.\n\nHowever, the gradient stabilisation that the authors propose, whether it is Flow-G or Flow-T, is (I think) essentially a form of regularisation. In eliminating the high variance motion of the gradient, the gradient is biased, and this may harm the learning. And we see that while both Flow-T and Flow-G generally outperform the baselines (except for Robomimic-Can and Cube-Double-Task2, where no techniques succeed), neither one is a clear winner across all benchmarks. This absence of a clear winner shows that the specific bias of the GRU or the Transformer is in fact limiting performance in some cases. It would be helpful to have some insight into how to control this bias, and whether or not the bias could be controlled, for instance by slowly widening the reset gate of the GRU or the cross-attention of the transformer.\n\nI was also surprised that the GRU gradient norms are so much smaller for Flow-G and than for Flow-T (Figure 6). It would be good to understand how sensitive the performance is to the specifics of the GRU and transformer.\n\nGiven that this is a fairly modest step above flow-based RL, it would be good to have some understanding of the limitations of the technique. It is not clear if there are constraints on the reward function. Are there kinds of problems where this technique will not succeed?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)