Flow-Based Single-Step Completion for Efficient and Expressive Policy Learning
Authors: Prajwal Koirala
Cody Fleming
Venue: ICLR 2026 Poster (2026)
Keywords: Offline Reinforcement Learning
Generative Models
Flow Matching
Behavior Cloning
Goal-Conditioned Reinforcement Learning
Primary area: reinforcement learning
TL;DR: We introduce a generative model-based policy for efficient one-shot action generation, integrating flow-matching and shortcut-completion losses for stable, expressive offline reinforcement learning and behavior cloning.

Abstract: Generative models such as diffusion and flow-matching offer expressive policies for offline reinforcement learning (RL) by capturing rich, multimodal action distributions, but their iterative sampling introduces high inference costs and training instability due to gradient propagation across sampling steps. We propose the \textit{Single-Step Completion Policy} (SSCP), a generative policy trained with an augmented flow-matching objective to predict direct completion vectors from intermediate flow samples, enabling accurate, one-shot action generation. In an off-policy actor-critic framework, SSCP combines the expressiveness of generative models with the training and inference efficiency of unimodal policies, without requiring long backpropagation chains. Our method scales effectively to offline, offline-to-online, and online RL settings, offering substantial gains in speed and adaptability over diffusion-based baselines. We further extend SSCP to goal-conditioned RL (GCRL), enabling flat policies to exploit subgoal structures without explicit hierarchical inference. SSCP achieves strong results across standard offline RL and GCRL benchmarks, positioning it as a versatile, expressive, and efficient framework for deep RL and sequential decision-making.

Published as a conference paper at ICLR 2026
FLOW-BASED SINGLE-STEP COMPLETION FOR EFFICIENT AND EXPRESSIVE POLICY LEARNING
Prajwal Koirala & Cody Fleming lowa State University
Ames, lowa, USA
(prajwal, fleminge @iastate.edeee
ABSTRACT
Generative models such as diffusion and flow-matching offer expressive policies for offline reinforcement learning (RL) by capturing rich, multimodal action dis- tributions, but their iterative sampling introduces high inference costs and train- ing instability due to gradient propagation across sampling steps. We propose the Single-Step Completion Policy (SSCP), a generative policy trained with an augmented flow-matching objective predict direct completion vectors from in- termediate flow samples, enabling accurate, one-shot action generation. In an off-policy actor-critic framework, SSCP combines the expressiveness of genera- tive models with the training and inference efficiency of unimodal policies, with- out requiring long backpropagation chains. Our method scales effectively of- fline, offline-to-oniee to-online, and online RL settings, offering substantial gains in speed and adaptability over inffussio-based baselines. We further extend SSCP to goal- conditioned RL, enabling flat policies to exploit subgoal structures without ex- plicit hierarchical inference. SSCP achieves strong results across standard offline RL and GCRL benchmarks, positioning it versatile, expressive, and efficient framework for deep RL and sequential decision-making
INTRODUCTION
Learning effective policies from fixed datasets, without active environment interaction, is a central challenge in offline reinforcement learning (RI.) and related fields. these settings, the choice of policy parametrization plays pivotal role in determining the agent's ability to generalize and perform well in complex environments. While unimodal Gaussian policies have remained the de facto standard due to their simplicity and compatibility with gradient-bas based learning algorithms, their limited expressiveness often hampers performance multimodal or highly non-linear behavior
distributions (Fu et al. 2020 Lange al. 2012 Wang et al. 2022 Tarasov al. 2023).
Recent advances in generative modeling, particularly diffusion and flow-based methods, have intro- expressive alternatives traditional unimodal policy representations by enabling the modeling of complex, multimodal action distributions (Ho et al. 2020 Song et al., 2020 Wang al., 2022, Lipman et al.) 2022: Liu et al.) (022b). Diffusion-baseet policies, in par- licular. have demonstrated impressive performance in imitation learning (IL) settings, especially when learning from expert human demonstrations (Chi et al. 20244) Pearce et al., 2023). Despite these advantages, practical challenges like computational ineffiiency at inference time have lim- ited the broader adoption of such generative policy classes. For instance, diffusion models typically require tens to hundreds of denoising steps to generate single action, making them less practical for real-time control (Ding & Jin 2023] Park et al. 2025). Moreover, there is the difficulty of inte- grating iterative generative structures into conventional off-policy value-based learning frameworks. Their reliance on geneeggnerttion horizons renders backpropagation through time (BPTT) either intractable highly inefficient, impeding scalable off- off-policy gradient- madient--basd optimization in offline RL. Nonetheless, recent empirical studies suggest that behavior-comstrained policy gradient methods (such DDPG+BC and TD3+BC (Fujimoto & Gu 2021)) consistently outperform weighted be- havior cloning approaches like Advantage Weighted Regression (AWR), due to their ability to both
duced significantly
code available at wtt.t/hwwb.ooo..cooooemoe wow//raajwaaaa..oooooo..aaaaa.a Mal SS/P/SPP3330 B1 nngll- Steep-Come let lon-Palloo

---

Published as a conference paper at ICLR 2026
better exploit the value function and somewhat extrapolate beyond the coverage of dataset actions (Prr et al. 2024b).
While recent efforts have explored mitigation strategies, such reducing the number of generative steps or distilling policies into simpler networks, these approaches often introduce trade-off' in em- pirical performance, sensitivity to hyperparameters,
increased architectural complexity (Ding & Jin 2023 Janner et al. 2022 Park et al.) 2025). Motivated by these limitations, we propose flow- based single-step completion
powerful policy representation that combines the expressiveness of generative models with the efficiency and simplicity required for scalable learning across offline reinforcement learning. imitation learning, and broader robotic decision-makin anism is illustrated in Figure L where completion vectors provide a one-step generative shortcut the target action distribution, bypassing the need for iterative transport. By directly predicting completion vector toward the target action in a single generative step, our method Single-Step Completion Policy (SSCP) circumvents the inefficiencies of sequential generation, enables compat- ibility with standard value-based RL algorithms, and offers more practical and scalable solution for high-performance policy learning.
tasks. The mech-
Beyond one-shot action generation, we further explore the role of completion modeling in hierarchi- cal decision decision-making. Hierarchical methods subgoal structures by decomposing tasks into multiple decision layers. This naturally raises the question: just as SSCP compresses multi-step generation into a single-step policy without sacri- ficing performance, cision step while retaining their
goal-conditioned RL (GCRL) exploit
multi-level hierarchical policies also be compressed into single flat de-
strengths? Motivated by this, we extend short- cut/completion modeling GCRL, yielding GC-SSCP, flat inference policy that preserves subgoal benefits and significantly outperforms flat baselines, surpassing hierarchical state-of-the- methods such as HIQL (Park et al. 2023) on average in OGBench (Park al. 2024a tasks.
Our key contributions are as follows:
A novel policy method based on single-step completion using flow-matching genera- tive models. This policy class is expressive and tractable, enabling direct action sampling without iterative inference. (Section 3.2)
Compatibility with off off-policy policy gradient methods, such DDPG+BC without wor- rying about BPTT. Our approach yields strong empirical results on standard D4RL bench- marks and facilitates seamless offlineto-online finetuning. (Section 3.3
A framework for distilling hierarchical behavior into flat policies. We show how the single-step completion principle can be extended to exploit subgoal structures in long- horizon, sparse-reward, multi-goal conditioned environments. (Section 4.2)
T=
T=1
Velocity Vector 6(*,, r) = hols,, T,0)
Completion Vector h(*,,) = ho( 1-1-r)
Figure 1: Depiction of completion eeaedd flow matching: while velocity vectors propagate along the generative path, completion vectors enable shorteut one-step jumps to the target distribution. This forms the basis of our Single Step Completion Policy (SSCP) used in offline RL and related problems.
2 PRELIMINARIES
2.1 OFFLINE REINFORCEMENT LEARNING
Reinforcement learning (RL) seeks a policy v that maximizes expected return in Markov Decision Process (MDP) AS, R), where / (5o) I= so]- Offline RI. considers the
2

---

Published as a conference paper at ICLR 2026
setting where learning must occur solely from fixed dataset D {(s, a, r, s'}} collected by an unknown behavior policy ng, preeluding further environment interaction.
A central challenge in this setting is the distributional shift between the learned policy and ng. which can result in erroncous value estimates when я selects cuttof-distribution (OOD) actions (Fu-) jimoto et al. 2019 Kumar et al., 2020 Levine et al. 2020). To mitigate this, rehavior-constrainee approaches regularize π to remain close to Tg. often via a divergence constraint:
max (s), s.t. (T I 6.
(1)
A widely used formulation for enforcing such constraints combines off-policy policy gradient meth- ods (like Lillierap al. (2015); Fujimoto al. (2018)) with Behavior Cloning (BC) regularization, optimizing:
B( [Q(s, (+)) + a log <a 8)]
where balances exploitation and conservatism (Fujimoto & Gu 2021 Wang et al. 2022). This
serves as the foundation for our method discussed section 3.3
As an extension, in section we also consider offline goal-conditioned RL (Andrychowicz et al., 2017 Liu al. 2022a] Park et al. (2024), where policies and rewards are aditionally conditioned on goal state y E G. This setting often employs sparse rewards of the form r(s, a, g) = =s + g). encouraging the agent to reach the goal without requiring dense or hand-crafted signals.
2.2 DIFFUSION MODELS AND FLOW MATCHING.
Diffusion models learn to reverse stochastic forward process that gradually corrupts data with noise (Sohl- Dicksticcttein et al. 2015 Ho et al.] 2020 Song et al. 2020). The forward process is typically defined as a stochastic differential equation (SDE):
dx, f()xx dr 9(T) dw,.
where w, is standard Brownian motion, and f(T).g(T) govern the drift and diffusion schedules. The reverse-time dynamics, learned via denoising score matching, allow for approximate sampling by numerically solving reverse-time SDE its deterministic counterpart, the probability flow ODE. Most implementations train a neural network to predict the noise added at each timestep. minimizing weighted mean squared error between the predicted and true noise vectors.
Flow matching offers an alternative that bypasses stochastic dynamics by directly modeling deter- ministic transport between a base distribution and the data distribution (Lipman et al. 2022 Liu
et al. 2022b). Let (z) = -N(0. I) be a standard normal distribution, and Paata (x) the empirical data distribution. Flow matching defines a continuous interpolation path:
X, (1 7)z rx, [0. 1].
where z Po. Pdaa. and learns dime-dependent velocity field vo(x,, that maps points along this path. The optimal velocity field under this scheme is simply the displacement vector: v*(x,, T) x The flow matching objective minimizes the expected squared deviation from
this ideal field:
CPM = ETwp(()) ()), po xmp [x - (x 12)
(5)
Sampling from the learned model requires solving the ODE:
dx, T), X Po:
(6)
dr
using standard numerical solvers such Euler or Runge-Kurte Kutta methods. While diffusion and flow- based models excel at capturing multimodal distributions and generating high-fidelity samples, they require several iterative steps during inference, making them impractical for real-time control appli- cations.
2.3 BEHAVIOR CLONING, DIFFUSION POLICIES, AND FLOW-BASH POLICIES
Behavior Cloning (BC) formulates policy learning supervised learning from expert demonstra- tions. Given a dataset of state-action pairs (8, a) ~ DD the policy To(a (a 8) is trained to maximize the likelihood of expert actions:
Lac (, [- log (o(a |s]]-
3

---

Published as a conference paper at ICLR 2026
Recent works have explored the use of diffusion models expressive policy classes in BC and RL (Janner et al. 2022. Pearce et al. 2023 Ding & Jin 2023 Wang et al.) 2022 Chi et al. 2024). Instead of directly predicting actions. the model learns to denoise noisy actions through learned reverse process. Diffiusion Policy (Chi et al. 2024) adopts this framework by generating short- horizon action trajectories conditioned on past observations, effectively enabling receding-horizon planning. This formulation allows for modeling complex, temporally coherent, and multi-modal behaviors, making it well-suited for high-dimensional imitation learning from raw demonstrations.
Flow-based Behavior Cloning offers a deterministic alternative via flow matching. Instead of it- eratively denoising, the model learns time- deme-deenndent velocity field Ve that maps a noise sample z Po to an expert action a along a linear interpolation path a, = (1 r)z + ra. The model is trained using the flow matching objective:
BB E (s.a)=-D, ampo, r-p(r) [llvo(a,, 5, (a =)
(8)
At inference, action sample (a1) ) is generated by solving the following learned ODE: da,
volar, ag ~ Po-
(9)
dr
SINGLE STEP COMPLETION MODEL
In this section, we begin by introducing general framework for generative modeling using single- step completion models. Unlike conventional flow-based approaches in which the models learn to predict instantaneous velocity fields conditioned on intermediate states, our method incorporates an additional objective to learn completion shortcuts, normalized directions that complete the gen- erative process in a single step from any intermediate states in the flow trajectory. This yields an expressive and efficient policy class that bypasses costly iterative generation, making it particularly suitable for imitation and reinforcement learning settings introduced earlier. We conclude the sec- tion by developing offline RL algorithm that leverages this Single Step Completion Policy for continuous control. In the next section, we extend this completion modeling approach to derive flat inference policies that can effectively exploit subgoal structure in long-horizon coal-conditioned tasks.
3.1 LEARNING TO PREDICT SHORTCUTS
Unlike standard flow-matching models that predict only instantaneous velocities at т, requiring many small integration steps, shortcut models additionally condition on the desired step size d, enabling accurate long-range jumps that account for trajectory curvature and avoid discretization errors from large naive steps (Frans et al. 2024). Formally, shorteut model learns the nor- malized direction h(x,, T, d) that when sealed by step size d produces the correct next point: -++d I, h(x,, T, d)- d. Frans al. 2024 train the parametrized model ho(I,, d) us- ing the following additional self-consistency loss:
Lahoneut = Er, [lloo(z,, T,)
target (IT, T, d)112)
(10)
where 11 laaget stop gradient operation to block gradient flow during optimization. It is given by:
(x+ T, d) is a bootstrap target predicted by the model itself, but treated as fixed using
1
target hg
(a+ T, d) =
2 hh(E T, d/2) + ho(I d/2,
where 2' 4'T+d/2 = Ir + T,,ff,,,d/2)/)
3.2 LEARNING TO COMPLETE THE GENERATIVE PROCESS
While the self-consistency objective in Equation 10 enables training shortcut models without expen- sive ground-trum -trtt integration, it introduces a fundamental challenge: the targets used for supervision themselves generated by the model. Consequently, these bootstrap targets only reliable once the model has attained a certain level of accuracy, posing difficulties in early training with inaccurate predictions and in deep RL settings with evolving implicit target policies.
4

---

Published as a conference paper at ICLR 2026
Frans al. (2024) propose mixed-objective training scheme wherein a fraction of the training batch is supervised using standard flow matching loss with ground-truth velocity targets, while the remain- der is trained using the self-consistency loss. The proportion of flow-matching supervision is treated as a tunable hyperparameter, allowing the model to rely more heavily on stable targets during the initial training phases and gradually transition self-supervised learning performance improves. However, in domains such as robot learning, and in particular offline reinforcement learning, the reliance on internally generated pseudo-targets may be problematic. In these settings, generative regularization
models are often employed for behavior cloning or as part of to address distributional shift. If the shortcut model trained using inaccurate self-generated tar- gets, especially in early stages, it may lead to model drift or unsafe extrapolation. Therefore, while the self-consistency loss offers scalability and flexibility, its applicability in offline settings with distributional constraints warrants careful consideration.
Single-step completion vector. To integrate the shorteut learning process into a Q-learning frame- work and address the limitations of bootstrap targets in shortcut models, we propose alternative approach based on single-step completion prediction. Specifically, at each intermediate point along the flow or tramsport path, the model is additionally trained to predict completion vector- the normalized direction from the current sample to the final target sample (see Figure []] This auxil- iary prediction encourages the model's local dynamics to remain aligned with the overall trajectory, maintaining local consistency while enabling high-quality sample generation. In low-dimensional distribution learning problems such action spaces, this approach facilitates the learning of highly expressive multi-modal policies compared to unimodal alternatives. Unlike bootstrap-base methods that rely on self-generated targets, this formulation helps to directly predict the remaining trajectory toward the terminal point 21 via normalized direction vector. Formally, at any inter- mediate timestep 5 E [0. 1]. we define the single-step completion vector he (z+, T, d), with d set to (1 11-r), as the normalized direction from x, to the final state æ1. When scaled by the remaining time
shortcut
(1 (1-T). ). this vector yields the step required to transport x- to Z1:
2 Z, + ha 7, 1-r) (1-r).
(11)
This design allows for direct supervision from data, as 21 is drawn from the training distribution. The model is trained to minimize the squared error between the predicted final point ě1 and the true xi:
commplttion Er,,, mi [||e + ho(#,; T, 1-7) (11-7)-211333 11
(12)
where -, = (1-7)00 + T is a linear interpolation between To ~ Po and Φ ~ D. Alternatively, setting d 0 in (x,,, 7, d) corresponds the instantaneous flow velocity at (Z+,
From completion vectors to policy representation. By training against ground-truth final samples from the dataset, the completion vector prediction approach bypasses the need for bootstrap targets and mitigates the risk of model-induced distributional errors, making it particularly suitable for ap- plications in offline and safety-critical settings. While we present general method for generative modeling using the single-step completion models described above, this formulation can be natu- rally adapted to policy representation by an additional conditioning on the environment state. We term this Single-Step Completion Policy (SSCP), enabling supervised policy learning via behavior cloning or related techniques in learning-baa -based continuous control. This policy class offers an addi- tional advantage in the context of RL, where the same action samples generated via the single-step completion vector can also be employed in computing the deterministic policy gradient loss:
Lrg (0) E.~D i-Q nd (s, (T))))
(13)
where a (s) denotes the policy that uses the single-step completion model as function approxima- tor. The policy output is computed as:
To a, + ho(a,, 1)
7) (1 (1-r), 7 E [0.
(14)
where o, an intermediate action along the forward flow from ag to a1, conditioned on the state.
Notes on nomenclature: (1) The subscript r in a, denotes the interpolation timestep in the generative transport process, not the timestep along the reinforcement learning trajectory. Specifically, ao corresponds to noise, a is the behavior cloning target, and a, is the interpolated representation at intermediate step т. (2) While we refer to the loss above as a deterministic policy gradient loss, the policy induced by the single-step completion model is not deterministic in probabilistic sense, as it defines a transport between distributions. However, for a fixed input a,, the output (s) is deterministie, since both the model and the generative transport process are fully deterministic.
5

---

Published as a conference paper at ICLR 2026
3.3 OFFLINE RL WITH SINGLE STEP COMPLETION POLICY
We now instantiate our single-step completion model as an expressive and efficient policy for offline RL in continuous control, termed SSCQL (Single-Step Completion Q-Learning). The actor training objective in this method combines a flow-matching loss, a shortcut-based completion loss, and Q-learning objective. The flow loss aligns the model's velocity field with intermediate points along the generative path:
Crow E D, ampo, r-p(T) [1l 8, T, T) (a 2)
(15)
The completion loss directly supervises the model to regress to the terminal action in one step:
Leompletion () = Da, smpp, rmp((r) [lla, + 8,
(1-7) al
(16)
The total actor loss combines the low-matching objective (Eq. [5]. the completion loss (Eq. [6]
and the policy gradient loss (Eq. 13):
(0) 0n Caow (0) + Comoompitionn (0) + (0).
(17)
where on and 02 tune the strength of regularization that constrains the actor within the dataset (behavior policy) distribution. The critic is trained using the following Bellman backup loss:
( -r-+ min Qoy (oe,TT(sss)) 1=1,2
Co(0)= E(s,ana)
(18)
Compared to standard Flow-Matching (or Diffusion) -based policies,the shortcut completion mech- anism eliminates the need for iterative generation across three stages. First, training flow-matching actors with off- off-policy methods like DDPG typically requires generating actions via multi-step flow rollouts and backpropagating through the entire computation chain, which both memory- and compute-intensive Second, critic training in standard Q-learning involves sampling actions from the current policy at the next state, i.e., ( (s')), which again demands full flow tra- jectory rollout during training. By enabling single-step inference, our method resolves both is- sues efficiently. Third, test-time action generation similarly reduced to a single forward pass: Te(s) = 2 he(2, s, 0, 1), where Pu- The complete algorithm is presented in Appendix A.2
4 GOND-CONDD RL: DISTILLING HIERARCHY INTO FLAT POLICIES
4.1 OFFLINE GOAL-CONDITIONED REINFORCEMEN LEARNING
Goal-conditioned RL (GCRL) augments standard RL by conditioning the policy on a target goal g E G, enabling agents reach diverse goals without retraining. In offline settings, GCRL leverages static datasets and sparse goal-reaching rewards, typically r(s, a, g) = 11 * g). thereby avoid- ing the need for dense or engineered signals. The resulting goal-conditioned policy maa 8,9) must generalize across initial states and goals.Recen methods such as RIS (Chane-Sane et al. 2021), POR (Xu al. 2022), and HIQL (Park al. 2023) adopt a hierarchical structure that decom- poses the problem into high-level subgoal selection and low-level executable action selection, sig- nificantly outperforming flat policies. HIQL (Hierarchical IQL), which we build upon and compare against, learns both high-level policy x (St+ 8t, g) via
9) and low-level policy _é. (we. 81, 82+4)
regression (Peng et al., 2019 Kostrikov et al., (021):
L.D (9%) E(eereexx.a) [- cxp (3 A (st, Sttk: 9)) log ,b, (Se+k [ss
Ln, (0t) = (( |- exe (8 A at, s(+k)) log Tb, (at st, St+))]
4.2 LEARNING TO REACH GOALS VIA COMPLETION MODELING
While hierarchical decomposition, as in HIQL, can enhance generalization offline goal- conditioned reinforcement learning (GCRL), it remains open question whether explicit hierarchy is strictly necessary for subgoal discovery and exploitation. A promising and somewhat straight- forward alternative is to distill the benefits of subgoal guidance into a non-hierarchical architecture, thereby simplifying both training and inference while retaining the representational power of hierar- chical policies. Our approach draws inspiration from shorteut modeling and completion prediction

---

Published as a conference paper at ICLR 2026
frameworks introduced in earlier sections, leveraging them to learn flat goal-conditione policies from offline data. Specifically, we extend the shortcut-basee completion framework to a flat, goal- conditioned setting by leveraging single unified model capable of hierarchical reasoning through conditional inputs.
To exploit the subgoal structure within flat architecture without explicit use of hierarchi- cal learning, we propose a joint policy model mo that outputs both immediate action a, and an intermediate subgoal St+k, conditioned on the current state St, the final goal and an ab- stract hierarchy level indicator
g't k= ==0,d=1 k=1,= gg
a
k= 0,
(0, 1). The level variable k captures coarse (goal-level) Figure Hierarchy Distillation with Shortcuts or fine-grained (subgoal-level) reasoning. en- abling shared parameterization across the hierarchy: (ar, gg) ) Te(- St, 9, k), where g' denotes either subgoal 8t+* (for 0) or next state 8141 (for k 1). The model is trained using advantage- weighted regression losses derived from the hierarchical Q-function, with separate losses corresponding to each level of reasoning:
(o) = (s [- exp (3 A (St, Sc-k, g)) log no (at, 8itk 88, 88, 0)]
(0) (1 [- exp (B A ae sc+k)) log (a St+1 St, Sctk, 1)]
At inference, hierarchical reasoning is simulated through two-stage procedure: first, the model samples subgoal from the high-level distribution, (ă,, St+&) ~ TT(- Sts Ut k = 0): then, condi- tioned on the predicted subgoal, a low-level action is produced via (ar, St+1) 1 ~ TT St, St+k, k = 1). This is similar spirit flow-matching methods where same velocity function approximator used to predict velocity across all the points in the flow trajectory.
Shorteut-Based Flat Policy. To enable direct flat inference through multi-step completion, we augment the model imput with an additional variable d, which specifies the number of forward steps the hierarchy. This variable acts analogously to step size in the original shorteut framework and allows the model to simulate mulp reasoning using different levels of abstraction within the same architecture: (ar,g') g') ~ T $t, g. k, d).
Training in this setting uses analogous loss functions, policy now conditioned on both and d:
(o)
E (ss..er exp Asts SL+2. 9)) log te ma S14k st. 0, 1)]
(0) = (( eep (3 1 az )..+) log Ro (1,, 8641 8t, 864k, k = 1, = 11
To enable single-step prediction during test-time inference, incorporate self-supervised loss that aligns shortcut predictions with composed hierarchical rollouts:
(0) (((..g)~D, (..e))))D.(((.eet)))) (ooes)))
log Te (ar, Sr+1 St. g. = 0, 2)]
Specifically, for (k = 0, = 2). we train Ty to match targets T TT=| with gradient flow stopped through the target. We refer to this model as Goal-Conditioned SSCP (GC-SSCP), and use this shorthand in the results section. This training enables direct inference by sampling from no(- St, St. g, 0, 2) at test time, bypassing recursive hierarchical rollout.
EXPERIMENTS AND RESULTS
We empirically evaluate SSCP and its derived variants across diverse reinforcement learning set- tings to demonstrate their flexibility and effectiveness. While the main results are presented this section, additional results and details, including training curves, implementation specifics, hyper- parameters, and ablations, are provided in the appendix. Across the benchmark tasks, we compare against strong baselines and report both quantitative metrics and qualitative trends to highlight the efficiency, expressiveness, and generalizability tion of datasets/henchmarks and baselines is also provided in the appendix.
our approach. detailed discussion and justifica-

---

Published as a conference paper at ICLR 2026
5.1 OFFLINE RL
We evaluate SSCQL the D4RL benchmark (Fu et al,, 2020), focusing on continuous control tasks from the MuJoCo suite: HalfCheet ah, Hopper, and Walker2d. We consider three dataset types per environment, medi um, the method's performance under varying data distributions and qualities. Performance is measured using normalized scores, averaged over 10 evaluation episodes. In Table [ report the highest- performing method in bold, and highlight any method within 95% of the best with boldface as well.
and unddiuu-exper.. that assess
Table 1: Offline Reinforcement Learning Results.
Gym Tasks
BC
TD34BC
CQL
IQL
Diffuser
FQL
DQL
CAC
IDQL
SSCQL
Halfchectah-MI:
55.2
90.7
91.6
86.7
79.8
102.1
96.8
84.3
95.9
98.1==0.9
Hopper-ME
52.5
98.0
105.4
91.5
107.2
76.7
111.1
100,4
108.6
110.910.9
Walker2d- ME
107.5
110.1
108.8
109.6
108,4
102.6
110.1
110,4
112.7
111.1
Hallcheetah-M
42.6
48.3
44.0
47.4
44.2
55.6
51.1
69.1
51.0
52.310.5
Hopper-M
52.9
59.3
58.5
66.3
58.5
60.6
90.5
80.7
65.4
102.4 0.2
Walker2d-M
75.3
83.7
72.5
78.3
79.7
65.9
87.0
83.1
82.5
84,2:+0.9
Halfcheetah-Ne -MR
36,6
44.6
45.5
44.2
42.2
48.3
47.8
58.7
45,9
44,4.1..
Hopper-MR
18.1
60.9
95.0
94.7
96.8
50.7
101.3
99.7
92.1
101.4+0,4
Walker2d-MR
26.0
81.8
77.2
73.9
61.2
38.8
95.5
79.5
85.1
85,9113.4
Average
51.9
75.3
77.6
77.0
75.3
66.8
87.9
85.1
82.1
87.9
We compare SSCQL against both unimodal Gaussian policy baselines and state-of-the-art genera- tive policy approaches. Gaussian based methods include Behavior Cloning (BC), TD3+BC (Fuji- moto & Gu 2021), , IQL (Kostrikov et al. 2021), and CQL(Kumar al. 2020), covering both actor and critic regularization paradigms. Generative policy baselines include Diffuser (planning-basee) IDQL EHarsen-Estruch al. 2023) (rejection sampling), DQL (Wang al.
2022). and CAC (Ding) & Jin 2023) (both actor-regularizer diffusion methods). FQL (Park et af.
2025 shares a similar
motivation with our approach but applies flow matching only for behavior cloning and distills single-step policy into separate MLP; detailed comparison is provided in Appendix|A.) A.7
Table reports compute characteristics, comparing SSCQL against generative-policy baselines trained on identical GPU resources. SSCQL demonstrates a significant advantage in training and inference efficiency, being up to 64x faster in training than DQL, strong- performiiro diffusion actor baseline, and offering over an ofder-of-magnitude speedup in inference. Overall, SSCQL achieves consistently strong performance across all tasks while offering a dramatically streamlined training and deployment profile which highlights its practical advantages in offline reinforcement learning scenarios.
Table 2: Comparison of training and inference characteristics across generative modeling baselines.
Method Training Time
Inference Time (ms)
Training Steps
Denoising Steps
Training Batch
DQL
8 hours
1.27 + 0.48
IM
5
256
CAC
S hours
0.85 + 0.61
IM
2
256
IDQL
~ hour
2.01 1.50
3M
512
SSCQL
16 mins
0.27 0.36
500K
1024
5.2
OFFLINE TO ONLINE FINETUNING AND COMPLETE ONLINE TRAINING
We evaluate SSCQL's adaptability in both offline-to-online finetuning and fully online RI. Table reports finetuning results where prrtrniiret policies are further trained with online interaction. We compare against Cal-QL Nakamoto et al. (2023) SOTA method designed for the regime), DQL, and CAC (strong generative policy baselines). The numbers before and after "+' in the SSCQL headings denote the offline and online training steps respectively. The final scores with the best performance are bolded, and red arrow (+) indicates a drop that exceeds 10% of the offline score. SSCQL yields stable improvements, while DQL and CAC often degrade, underscoring the brittleness of diffusion based policies in online adaptation.
8

---

Published as a conference paper at ICLR 2026
Table Offline Online Finetuning Results
Gym Tasks
Ca-QL
DOL
CAC
SSCOL
ASCQL
Halfchoctah-ME
54-99
96.8-115599
84.3-499.6
7.03==006--97.32221
0.69HH082-:1006661
Hopper-ME
69-476
111.1
100.4 -65.4
111.2341.22 -111:77-111
Walker2d-Mi
96 -77
110.1
110.4+101.8
Halicheetah-M
52 193
-99.6
69.1 198.7
53.3210.32
:60.51110.26
Hopper-M
89 99-
90.5
61.9214.62
Walker2d-M
75-+103
87.0-+118.3
83.1-1108.9
Halfcheetab-Me
51 513393
47.8 47.8-196.3
58.7-580.7
51,24410.40 158.59-10.99
47,3010,47
+61.56=20119
Hopper-MR
76-+110
101.3-4684
99.7-474.6
91.75+ 19775-19667-999222411
Walker2d-MR
52-99
95.5-95.7
19.5-1102.0
92.89
9..8849.61-111138888888
We also examine fully online RL, with results summarized in Table 4 SSCQL. significantly outperforms DQL and CAC across all Gym lo- comotion tasks. Notably, both DQL and CAC experience reduced performance as their num- ber of generative steps increases (5 for DQL, 2
Table 4: Online Reinforcement Learning Results
Gym Tasks
DQL
CAC
SSCQL
Hallcheetah
43.5.14.1
56.1.16.9
9,447
Hopper
79.8-21.7
80,2:123.3
105.07
Walker2d
79.8431.23
74.9-36.33
102.90-5.26
Average
67.7
70.4
90.53
for CAC), suggesting that deeper backward computation graphs and longer denoising chains may hinder training stability. In contrast, SSCQL employs a single-step generative actor, avoiding back- propagation through time (BPTT), which likely contributes to more stable gradients and efficient learning dynamics. More details on offline-tooonlin finetuming as well as fully online training are available in Appendix A.3
5.3 BEHAVIOR CLONING
Table 5: Behavior Cloning Results
We evaluate SSCP on behavior cloning tasks using state-based, proficient human demon- stration datasets from RoboMimic (Mandlckar
Tasks
DP
L-G
IBC
BET
SSCP
Lift
1.00
0.96
0.41
0.96
1.00:10.00
Can
1.00
0.91
0.00
0.89
0.99110.01
cet al. 2021) and the contact-rich Push-T task (Florence et al., 2022). RoboMimic consists of five robotic manipulation tasks with high- quality human demonstrations, while Push-T requires precise manipulation of a T-shaped ob- ject under variable initial conditions and com- plex contact dynamics. In table we follow the evaluation protocol of Chi et (2024), and com-
Square
0.89
0.73
0.00
0.52
0.90-10.03
Transport
0.84
0.47
0.00
0.14
0.86
Toolhang
0.87
0.31
0.00
0.20
0.64:+0.09
Push-T
0.79
0.61
0.84
0.70
0.85:10.03
Average
0.88
0.68
0.21
0.57
0.87
pare against strong baselines including Diffusion Policy (DP), which uses 100 iterative denoising steps. Despite its single-step generation, SSCP achieves competitive or superior performance across all tasks, matching the performance of DP while offering significantly reduced computational over- head, Other baseline methods include LSTM- GMM et al. (202I) (L-G), Implicit Behav- ior Cloning FForence al. (2022) (IBC), and Behavior Transformer Shafiullah et al. (2022) (BET).
Full experimental details and training setups are provided Appendix
5.4 OFFLINE GOAL-CONDITIONED RL
We evaluate our proposed Goal-Conditioned Single-Step Completion Policy (GC-SSCP) in goal- conditioned offline RL using PointMaze and AntMaze tasks from OGBench, which challenge agents with long-horizon and hierarchical reasoning. The benchmark covers 16 environment-dataset combinations: fpointmaze, antmaze} x {mediumg um, large, giant, teleport] x (navigate, stitch). Following OGBench protocol, we report average success rates over 6000 evaluation episodes (8 training seeds x 50 test seeds x 5 goals x 3 last evaluation epochs), as shown in Table 6 For each dataset/environment,, we highlight bold both the overall best- performing method and the best-performing method among the flat-inference basclines.
We compare against goal-conditioned baselines including GCBC (Gosh et al., 2019 Lynch et al. 2020), GCIVL/GCIQL (Kostrikov et al. |2021; Park et al. 2023 2024a), QRL Wang et al. 2023) CRL Eysenbach et al., 2022), and HIQL (Park et al. 2023). Our method, GC-SSCP, consistently achieves strong performance across tasks, notably outperforming not only other flat baselines but also the hierarchical HIQL on average. Unlike HIQL, which only transfers top-level policy outputs to separate low-level controller, GC-SSCP employs shared policy network across abstraction

---

Published as a conference paper at ICLR 2026
Table 6: Offline Goal Conditioned Reinforcement Leamnngg Results.
Inference Type-
Flat
Hierarchical
Flat
Dataset 4
GCBC
GCTVI.
GCTQL
QRI.
CRL
HIQI.
GC-SSCP
nointmaze-medium -
916
63FF
5318
82E5
29:
79:15
92:13
pointmaze-larg parge-navigate
29:1
45:15
34:3
86199
39:17
58:15
95:1:3
pointmaze saatiiegggantt-aaaaaa navigate-vO
1+2
01:0
0::0
6811
27:110
46.19
73:17
wontmazze-cceeportnnavveee haallpppott-nviittee-eeeeee
25-3
45413
24
8-4
24
1841
501.5
nontmazze-mcciin-stiihte
23118
70114
21:99
80
021
74.16
8014
pointmazz-lrree-ssiich---
745
12:16
31:2
84-15
0=:0
13:6
14:4
nonttaaz--gium--siich---00
0::
0::0
0=:0
50
0=:0
04:0
0+:
pointmaaze-cllgpoottssiicc---
3119
44:12
25
9=55
4113
3414
35:55
antmazz-nadduunnnwvvgaa---
29-44
72+
71:14
88+3
95-1
96+11
94+1
antmaazz-laggennavvgaae--1
24++2
16+5
34++4
75+6
83-44
91+22
87+2
antmaaze-giant-nnvigaee---
0+0
0+0
14+3
16
65+t5
52+22
autmaaz-icllpor--nnvvgaaa--
26-23
3943
35:55
35455
53-2
42:-3
42
antmaze-men seeiuze--ddium-iiiee stitch--O
45:11
4416
29:16
5917
53:16
94.11
84.4
Hartmazz-large-sttee stitc---
1812
712
1812
11:2
67:15
40.1
antmazz-giamt-sticch---1
0+0
0+0
0+0
2+2
0+0
antmaze-eeeporr--ticch---
31+6
3943
17+2
24+5
31++4
36+2
33++4
Average
17.69
31.69
23.81
48.75
29.06
50.94
54.44
levels, enabling cross-level generalization without explicit hierarchical inference. We hypothesize this design promotes more coherent learning and sample reuse. Further implementation details and experimental insights are available Appendix B with extended ablation studies in B.6 We include the full pseudocode for the GC-SSCP algorithm (Algorithm(3) along with training curves across all evaluated environments. The appendix also presents ablation studies examining the effect of the AWR inverse temperature hyperparameter, the impact of using hierarchical versus flat inference, and the role of subgoal selection in policy performance.
CONCLUSION
We introduced a new policy class based on single-step completion models, enabling stable behav- ior cloning and actor-critic learning. Building on this, we proposed SSCQL, policy-constrained offline actor-critic framework that combines the expressiveness of generative policies with the ef- ficiency and stability of unimodal actor approximations. By leveraging flow-matching principles, SSCQL captures rich, multimodal action distributions while remaining compatible with off-policy optimization and single-step inference. We further extended this framework to goal-conditioned tasks through GC-SSCP, which employs shared architectures across reasoning levels to support flat inference and implicit cross- level generalization without explicit hierarchy. Empirical results across standard and goal-conditionel offline RL benchmarks and behavior cloning tasks validate SSCQL's practical benefits for robot learning, including strong performance, faster training and inference, and robust online adaptation.
Despite these promising results, several limitations remain. First, the expressiveness of single- step completion models, while effective in continuous control, may be limited in capturing highly multimodal high-dimensional behaviors compared multi-step generative policies. Similarly, in condiiondi RL, bypassing explicit hierarchical structure may hinder performance in very long-horizon tasks. Second, while typical DDPG+BC methods rely on single behavior regular- ization coefficient, SSCQL introduces two coefficients (01, (22) for flow matching and completion losses respectively. Although we provide guidance for tuning (Appendix AA44, performance can be sensitive to their choice, requiring task-specific tuning of these hyperparameters in the actor loss function. Third, SSCQL is currently designed for continuous action spaces; extending it to discrete or hybrid action spaces presents additional challenges in defining completion models. Incorporat- ing -ntrrpp formulations may also offer a path forward, potentially broadening SSCQL's applicability online RL and other decision decision-making domains well.
7 ACKNOWLEDGMENT
This work was partly supported by the National Science Foundation, USA, under grant NSF CNS- 2313104.
10

---

Published as a conference paper at ICLR 2026
8 REPRODUCIBILITY STATEMENT
We have taken care to provide the necessary resources and documentation to facilitate the repro- ducibility of our work. All experiments are conducted on publicly available datasets, with details, references, and acknowledgments provided in the appendix. The accompanying code includes clear usage and installation instructions, as well as scripts for reproducing results. Pseudocodes, hyper- parameters, design choices, and corresponding ablation studies are documented in the appendix, alongside additional results that further support our claims.
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abheel, and Wojeiech Zaremba. Hindsight experience play. Advances in neural information processing systems, 30, 2017.
James Bradbury, Roy Frostig. Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Mandderma--Milng et al. Jax: composable transformations of python+ numpy programs. 2018.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540. 2016.
Elliot Chane-Sanc Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In International conference on machine learning, pp. 1430-1440. PMLR, 2021.
Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209. 14548, 2022.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, 2024.
Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning. arXiv preprint arXiv:2309.16984 2023.
Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ Salakhutdinov. Contrastive learn- ing as goal-conditioned reinforcement learning. Advances in Neural Information Processing Sys- tems, 35:35603-35620 2022.
Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning, In Conference on robot learning. pp. 158-168. PMLR, 2022.
Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shorteut models. arXiv preprint arXiv:2410.12557, 2024.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2062. PMLR, 2019,
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. arXiv preprint urXiv:/9/1 06088, 2019.
11

---

Published as a conference paper at ICLR 2026
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learming. pp. 1861-1870. Pmlr, 2018.
Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint urXiv:2205.0999) 2022.
Prajwal Koirala and Cody Fleming. Solving offline reinforcement learning with decision tree re- gression. arXiv preprini arXiv:2401.11630, 2024.
Ilya Kostrikov. JAXRL: Implementations of Reinforcement Learning algorithms JAX, 10 2021.
URL httpps////ggttn.o.ccc///ttttttrr
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:2110.06169, 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33:1179-1191, 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforce- ment learning: State-of-the-art, pp. 45-73. Springer, 2012.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv 02971, 2015.
preprint
Yaron Lipman, Ricky TQ Chen, Heli for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Maximilian Nickel, and Matt Le. Flow matching
Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Prob- lems and solutions. arXiv preprint arXiv:2201.0888 088299, 2022a.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0300 03000, 2022b.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pp. 1113- 1132. Pmlr, 2020.
Yecheng Jason Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. How far i'll go: Offline goal- conditioned reinforcement learning via f -advantagge regression. arXiv preprint 2rXiv:2206.000023, 2022.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese. Yuke Zhu, and Roberto Martín- Martin. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv: 200.::2000003330 03298, 2021.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint arXiv:2006.09:355 2020.
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh. Max Sobol Mark. Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine Cal-ql: Calibrated offline rl pre-training for efficient online fine- tuning. Advances in Neural Information Processing Systems, 045622444622699 2023.
12

---

Published as a conference paper at ICLR 2026
Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Hiql: Offline goal-
conditioned rl with latent states as actions. Advances in Neural Information Processing Systems, 34891. 2023.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. arXiv preprint arXiv:2410.20092, 2024a.
Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline rl? arXiv preprint (iXxx:2406009399, 2024b.
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning, arXiv preprint arXiv:2502.02538, 2025.
Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Ser- gio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al. Imitating human behaviour with diffusion models. arXiv preprint arXiv:2301.1067 2023.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. ddvantage-weighhee regression: Simple and scalable off-policy reinforcement learning. arXiv preprint urXiv:1900.000 1910.00 2019.
Jan Peters and Stefan Schaal. Reinforcement learning by reward weight-w regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pp. 745- 750, 2007.
Nur Muhammad Shafiullah, Zichen Cui, Ariuntuya Arty Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning k modes with one stone. Advances in neural information processing sys- tems, 35:22955-2222 22968, 2022.
Harshit Sikchi, Rohan Chitnis, Ahmed Touati, Alborz Geramifard, Amy Zhang, and Scott Nickum. Smore: Score models for offline goal-conditioned reinforcement learning. arXiv preprint arXiv:2311.00013, 2023.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing, pp. 2256-2265, pmlr, 2015.
Yang Song. Jascha Sohl-Dicksted Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 1.Xiv:2011I13355: 2020.
Yang Song. Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In Proceedings of the 40th International Conference on Machine Learning, pp. 32211-32252. PMLR, 2023.
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: Research-oriented deep offline reinforcement learning library. Advances in Neural Infor- mation Processing Systems, 36:30997-31020 2023.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEF/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal goal-reaching reinforce- ment learning via quasimetric learning. In International Conference on Machine Learning, pp- 36411-36430 PMLR, 2023.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06199. 2022.
Haoran Xu, Li Jiang, Li Jianxiong, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning. Advances in neural information processing systems, 35:4085- 4098, 2022.
13

---

Published as a conference paper at ICLR 2026
A OFFLINE RL WITH SINGLE STEP COMPLETION Q-LEARNING
A.1 OFFLINE REINFORCEMENT LEARNING
The environment in sequential decision-makinn setting such as Reinforcement Learning (RL) is typically modeled as a Markov Decision Process (MDP), defined by the tuple (S, P, P,R). Here, S denotes the state space, A the action space, P(s'ls, a): 8 A X + [o. 1] the transition probability function, and R(ss, a, 8') : S x A x S - R the reward function. The objective in RI. is to learn a policy π that maximizes the expected cumulative reward when interacting with the environment. This expected return of a policy π from an initial state distribution po is given by:
T Σ En're 80 ~ po
V (80) = ET
(19)
where (s0; 0o; ro, $1,-..) denotes trajectory and y E [0, 1] is discount factor. The corre- sponding state-action value function and advantage function are defined as:
T Σ
a= = ET
Y'
5,
= ("..a)=(((s.)vVV
In offline RL, however, the agent does not have access to online interactions with the environ- ment (Lange et al. 2012 Levine et al. (2020). Instead, it must learn solely from a fixed dataset D = [(8 a, r, s')}. which is typically collected by an unknown behavior policy π This setting presents unique algorithmic challenges, as the learning process is confined to the empirical distribu- tion induced by kg, without the ability to actively explore or query new transitions.
Policy Regularization. A core difficulty in offline RL arises from the distributional shift between the learned policy æ and the behavior policy Ty used to collect the data. When π selects actions that are not well-represented in the dataset, the value estimates can become inaccurate, leading to
poor policy performance. This is often referred to as the problem of out-of-distribution (OOD) generalization extrapolation error (Fujimoto al. 2019 Kumar et al. 2020 Levine al.) 2020
To mitigate such issues, many offline RL methods restrict the learned policy to remain close to the behavior policy by enforcing explicit divergence constraints. general formulation is:
max (*()), subject to (TI((T((TT) < 6
(20)
where DKL denotes a statistical divergence measure (e.g., KL divergence), and tolerance parameter. While TH is unknown in practice, only finite sample from its state-action visitation distribution is observed through D.
This constrained objective motivates family of approaches that regularize policy updates to remain within the support of the dataset. These methods aim to directly optimize the expected return while ensuring that the learned policy does not diverge significantly from the behavior policy, strategy often referred to as behavior-constraiie is the use of behavior cloning (BC) regularizer in the policy update. Representative examples include weighted behavior cloning techniques (eg. Peters & Schaal (2007); |eng et al. (2019); Nair al. (2020); Koirala & Fleming (0024). well rejection sampling strategies guided by genera- tive modeling of the behavior policy (eg. Chen et al. (202)); Hansen-EEttuus et al. (2023). )) In this work, we consider an objective that combines off-policy policy gradient (like Lillicrap et 20115;;
policy optimization. A common instantiation of this idea
Fujimoto et al. (2018)) with behavior cloning loss. The objective function of such DDPG+BC
style policy extraction is (Fujimoto Gu. 2021
Wang et al. 2022):
max ((sa)) (Q(s, + a log ((a) s)],
(21)
The regularization coefficient a controls the trade-off between exploiting high-value actions and adhering to the support of the offline data. This formulation provides flexible framework for offline policy learning and serves as the foundation for our offline RL method that we discuss in detail in the following sections.
14

---

Published as a conference paper at ICLR 2026
Algorithm Single-Step Completion Policy Q-Learning (SSCQL)
Initialize: policy parameters 0, critic parameters &
Initialize target networks: 0' +- 0, Φ +-
for N gradient steps do
Sample batch {(x, a, r, 8'} D
Critic Update:
> see Eq 26
Compute target: y
m Qay (s', mo)(
Lo(01, 02) E, 1Q (s.a) yl?
6, C O 7 V Lo(0),02)
Actor Update:
Dr see Eq. 22 25
Sample noise xo (0, 1). set 1 f-
Sample T ~ P(T) and set 2G f (1 - a00 + T
Velocity Prediction Loss:
see Eq. 22
V 1- 11 20 1 ho 8, T. z//
fnw (0)
= z12
Completion Prediction Loss:
see Eq. 23
ho ho
T)
à Ie h(1 completion (0) = l|a - -al1
Loss:
see Eq. 24
LTO (0) - (Qøs (8, Qzz(
Total Actor Loss:
C (0) = o1 Lnow (0) + cr Ce2cemplltionn (0) + Lxa
0 + 0- 1/0 (0)
Target Updates: o. - Too (1 (1-Taa Te)
8¹ - Too + (1 - To)
end for
A.2 SSCQL: ALGORITHM AND OTHER DETAILS
As discussed earlier, a common formulation in offline reinforcement learning is optimize pol- icy и to maximize its expected return while constraining divergence from the behavior policy лg, which generated the dataset D. Since Fs is unknown and only accessible through samples, practi- cal approaches impose soft constraint by augmenting the objective with regularization term that encourages similarity to the dataset actions. This leads to methods that balance off-policy value max- imization with behavior cloning, where temperature parameter controls the trade-off between exploitation and adherence to the empirical data distribution. In our method, the policy Ty is implic- itly induced by completion model hg described in Section (.33 and decompose the behavior cloning regularization into two terms: a flow-matching loss and shortcut prediction loss. Specifi- cally, given initial noise (0, 7) and ground-truth action a, we define a, (1-7)² + as the interpolated sample at time 7 p(T).
Flow Matching Loss. the generatttten at (a,, T) and minimize the deviation from the true velocity a 2:
To enforce local flow consistency, we predict the instantaneous velocity of
Laow = Es,aD, (0.1), (<)(7) [[]h 5, T, 0) - (a - =) =)
(22)
To enable consistency in long-range predictions and efficient action generation, we train the model to predict normalized completion vector from a, and penalize deviation from the target:
Shortcut Completion Loss.
Lcompletion Eg (0,1) rmp(r) [lla, + ho(ar, 8, 1-7) (1-7) a 27
(23)
15

---

Published as a conference paper at ICLR 2026
Q-Guided Policy Loss. To steer the model toward high-value completions, add critic- critic-guided (off-policy policy gradient) loss based on the current estimate of the f-untotion:
Lag = EDDD (=()s,
(24)
Total Actor Objective.
The total loss for the actor model he combines these terms:
(0) Lnow + o2 Complettioo + Lrq
(25)
where a1 and a2 are scalar weights that balance the contribution of the flow and completion losses, respectively, and are treated as tunable hyperparameters.
Critic Update.
We learn the critic Qs using temporal-difference learning with target networks øl:
( (Bole.n)- (r+ (r+7: min 1=12
(o( Bls
Qøy (8', No mo(5'))
(26)
Inference. At test time, actions are sampled by drawing a noise vector zo ~ (0,0 1) and applying the one-step completion policy:
a
To(8) = In ho v0, =
(27)
as detailed in Algorithm 2
Algorithm Sampling from the Single-Step Completion Policy T
procedure (T(s)
Induced policy via completion prediction
Tp NN00.1 I) 7 0, d
-= ho (8, vo.
D see Eq. 27
return 20 + ū-
end procedure
A.3 OFFLINE ONLINE FINETUNING AND COMPLETE ONLINE TRAINING RESULTS
We evaluate SSCQL's adaptability in both offline-to-online finetuning and fully online RL. Online finetuning and online reinforcement learning using our proposed policy class follow straightfor- ward extension of the offline RL algorithm presented in algorithms and 2 The key difference in online finetuning lies in the transition from purely offline training to incorporating online inter- actions: after a fixed number of offline training steps, the replay buffer begins to accumulate new transitions from agent-environment interactions. In contrast, in the fully online RL setting, the re- play buffer is initialized as empty and is populated exclusively with data collected during training, without relying the offline D4RL dataset. For offline-to-online finetuning, we report performance across varying numbers of offline training and finetuning steps. In the online RL experiments, agents are trained for 2M gradient steps. Unless otherwise specified, the network architecture and imple- mentation details remain consistent with those used in the offline RI. setting. However, the flow matching loss coefficient and the completion loss coefficient were separately tuned for these exper- iments; their values reported in Table I0
Table reports finetuning results where offline-pretrained policies are further trained with online interaction. We compare against Cal-QL Nakamoto et al. (2023) (a SOTA method designed for the regime), DQL, and CAC (strong generative policy baselines). Final scores with the best performance bolded, and red arrow (-) indicates drop exceeding 10% from the offline score. SSCQL yields stable improvements, while DQL and CAC often degrade, underscoring the brittleness of diffusion-based policies in online adaptation.
16

---

Published as a conference paper at ICLR 2026
Table 7:
Offline to Online Finetuning Results
Gym Tasks
Cal-QL
DQL
CAC
SSCQL (00K))))KK)
SSCOL (250K=25KK)
Halfchoctah-ME
54-499
96.8-105.99
84.3-499.6
98.69=0.82
0.26.66:0.66
Hopper-ME
69-476
111.1-711.
-65.4
110.66=0.811-112..5:4.
111.23=:1.28
1
1111
Walker2d-ME
96-477
110.1->117.0
1104-+1018
109.81=0111-110:77:1111
Halfchoctah-M M
52-493
51.1-;99.6
69.1-99.77
$1881=0.59-58.5110
13.5220.522:000110002.
Hopper-M
89-
90.5-477.2
80.7-160.5
46.12-344
61.9214.62
Walker2d-M
75-+103
87,0->118.3
83.1-1108.
Halfcheetah-MRR
513
47.8 596.3
58.7-180.7
51.24=0..0
158.3910..99
47,30-20.47
161.56:00.99
Hopper-MR
76->110
101.3-68.4 -68.4
99.7-174.6
73.91
91.75:119677
9 195.32241982
Walker2d-MR
52-199
95.5-195.7
79.5->102.0
15.39-15500001022555 100122348.91
91.88-9.61 -113.88::0.08
We also examine fully online RL (2M train- ing steps, 3 random training seeds), with results summarized in Table X SSCQL significantly outperforms DQL and CAC across all Gym lo- comotion tasks. Notably, both DQL and CAC experience reduced performance as their num- ber of generative steps increases (5 for DQL., 2 for CAC), suggesting that deeper backward computa- tion graphs and longer denoising chains may hinder training stability. In contrast, SSCQL employs a single-step generative actor, avoiding backpropagation through time (BPTT), which likely con- tributes to more stable gradients and efficient learning dynamics.
Table 8: Online Reinforcement Learning Results
Gym Tasks
DQL
CAC
SSCQL
Halfcheetahh
43.514.1
56.1:66.9
63.63:19.47
Hopper
79.8:221.7
80.2423.3
105.07=4.007
Walker2d
79.8=31.23
74.9+36.33
5.2.90=5.26
Average
67.7
70.4
90.53
A.4 IMPLEMENTATION DETAILS AND HYPERPARAMETERS
The common hyperparameters used across all tasks are listed in Table 9 reducing the need for extensive hyperparameter tuning. In this setup, tuning was limited primarily to the coefficients on and (22, which control the trade-off between behavior cloning and Q-learning. Specifically, corresponds to the flow loss coefficient and 02 to the completion loss coefficient. In practice, fixing a1 = 1.0 and tuning only a2 was sufficient to obtain the results reported in Table The task-specific values of a2 used in our experiments are provided in Table I0
Table 9: Common Hyperparameters Across All Offline RL Experiments
Hyperparameter
Value
Discount Factor (7)
0.99
Batch Size
1024
Soft Update Rate for Q-Networks (Tx)
0.005
Soft Update Rate for Actors (To)
0.0005
Learning Rates for All Parameters
3 x 10
Time Dimension
128
Training Steps
500k
Actor Network Size
(512, 512, 512, 512)
Critic Network Size
(512, 512, 512, 512)
In the online reinforcement learning experiments (sec. A.3 table 8) agents are trained for million gradient steps using reduced batch sizes and smaller network architectures. Specifically, a batch size of 512 is employed, and both the actor and critic networks utilize hidden dimensions of (256, 256, 256). A smaller learning rate of 3 x 10-5 is used consistently across experiments. Furthermore, the flow-time embedding dimension t time_dim) halved to 64. Unlike in the offline RL exper- iments, gradually decreasing exploration probability is introduced during training, enabling the agent to sample random actions with higher frequency in early stages and reduce exploration over time.
Time and step size encoding. To provide temporal context in the flow process, we incor- porate learnable Fourier feature projections. Scalar time inputs and step sizes d are both separately encoded using a shared FourierFeatures module that maps each input & to [cos(2R=W)). (int:::W.siin(.//... sin(2r=W)),, where W E Ittimedim/2211 are learnable frequency weights. The re- sulting embeddings passed through separate two-layer MLPs with hidden size t me d dim, and
17

---

Published as a conference paper at ICLR 2026
Mish activation produce temporal features (t)), ((s) IR These features are combined additively and concatenated with the state and intermediate action pair (s, a,l as imput to the final vector field network.
Time distribution and sampling (p(T)). To emphasize early stages of the flow process during training, we employ non-uniform time sampling scheme. Rather than drawing time coordinates I ~~ (00,1) is common in standard flow matching, we apply the transformation = 12, resulting in samples from a (eta( 1) distribution. This follows from a
analysis:
1 X ~ (0, 1) and Y X², then py (v) = 2/8" which corresponds to Beta( 2. 1). This distribution
concentrates density near zero, with approximately 32% of samples falling in [0, 0.1], a threefold increase over uniform sampling. Such emphasis benefits training by allocating more capacity to early-time dynamics where transitions from noise to data are most critical. Moreover, this approach remains easy to implement via squaring and aligns well with inference conditions where we evaluate in our single step completion model.
Actor Loss Normalization. We implement adaptive scaling for Q-based policy updates to stabilize gradient magnitudes. For each batch, we define the loss as the negative mean Q-value, scaled by the inverse of its average absolute value:
Lo
E (s (o(s) (-Qp(s,To(s))
This normalization ensures that the policy gradient magnitudes remain invariant to the absolute scale of Q-values and helps minimize tuning efforts for behavior cloning coefficients on and (/2-
Implementation. Our implementation is based on the JAX/Flax framework and draws inspiration from the following public repositories:
fq] IDQI jaxr _m and jaxr Park et al. 2025
Bradbury al. 2018). Experiments were conducted on a machine with an Intel Core 19-1 3900KF CPU and an NVIDIA RTX 4090 GPU.
Mansen-Estruch et al., 2023] Kostrikov 202
Table 10: Hyperparameter 0r and ag used across different environments and dataset types.
Experiments
Offline
Offline-to-Onie
Online
Task / Dataset
Qq
02
Q1
Qr2
I.0
0.1
0.1
0.1
SalfCCheetah-Mediin
1.0
0.20
0.20
0.20
0.05
0.05
HalfCheetah- Meaihh-Me-uumm-eee Meepyy
1.0
0.05
0.05
0.05
Hopper-Medium Expert
1.0
0.75
1.0
0.75
Hopper-Medium
1.0
0.05
0.05
0,05
0.05
0.05
Hopper-Medium-RRppl
1.0
0.10
0.10
Walker22--Mciiun-Expett
1.0
0.1
0.1
0.1
Walker2d-Medium
1.0
0.5
0.5
0.5
0.05
0.05
Walker2d-MMcium-RRen
1.0
0.1
1.0
0.1
A.5 TRAINING CURVES
Figure shows offline RL performance using D4RL datasets in Gym MuJoCo locomotion tasks. Figure and Figure 5/aurer))eeeenttt offline-to-online fine-tuning curves with 100k and 250k offline/oninee steps, respectively. Finally, Figure shows online RL training performance in Gym MuJoCo lo- comotion tasks. Subplot (a) corresponds to the fully online setting with an empty replay buffer, while subplot (b) initializes the buffer with a medium-level dataset, similar offline-to-online fine- tuning with zero offline steps (i.c., offline data is used without offline policy improvement: OK+500K offline-ooonnline).
httas///fthhb..coo//tttt///////
18

---

Published as a conference paper at ICLR 2026
Normalized Retur
100
Normalized Retum
125
Normalized Retum
80
100
100
60
75
75
Med Jum-Expert
Medium-Expert
Medium-Replay
Medium-RRplay
25
25
Med lum
Medium
Med
0
0
0k
200k
400k
0k
200k
400k
0k
200k
400k
Training Steps
Training Steps
Training Steps
(a) HalfCheetah
(b) Hopper
(c) Walker2d
Figure 3: Training Curves for Offline RL with D4RL Datasets in Gym Mujoco Locomotion tasks
Normalized Return
80
Normelized Return
Normalzzdd Retum
100
80
60
40
Medium- Medium-Expert
40
40
20
Medium-Replay
20
Medium
200k
50k
200k
Training Steps
Training Steps
Training Steps
(a) HalfCheetah
(h) Hopper
(c) Walker2d
Figure 4: Training Curves for Offline to Online Finetuning with D4RL Datasets in Gym Mujoco
Locomotion tasks (100k offline and 100k online steps)
Normalized Return
100
Normalized Return
100
Normalizee Return
100
75
75
50
50
Med Meeperrtttertt
Med kum Replay
20
Medium
0 0K
200k
400K
0k
200k
400k
0K
200k
400k
Training Steps
Training Stops
Training Steps
(a) HalfCheetah
(b) Hopper
(c) Walker2d
Figure 5: Training Curves for Offline to Online Finetuning with D4RL Datasets Gym Mujoco Locomotion tasks (250k offline and 250k online steps)
A.6 ABLATION
Effect of the Coefficients o and 0g in Actor Loss. We perform an ablation study over the be- havior cloning regularization coefficients aj (flow-matching loss) and a2 (completion loss), with aj varied along the vertical axis and along the horizontal axis in Figure 7 The column correspond- ing to 02 reflects settings where only the now-matching loss is used, effectively removing the shortcut-completion objective. As evident across most environments, except omitting the completion loss significantly degrades one-shot action generation performance, high- lighting its importance. The best results are achieved when both ai and a2 are tuned to appropri- ately balance local flow consistency (via flow matching) and accurate flow trajectory completion (via completion loss). This confirms the complementary nature of the two losses: flow matching sures smooth interpolation along the flow, while the completion loss grounds the model in final-state consistency, enabling expressive and stable behavior in offline reinforcement learning.
19

---

Published as a conference paper at ICLR 2026
125
150
Normalized Return
Normalized Return
100
100
75
50
Walker2d
Walker2d
25
Hopper
Hopper
HalfCheetah
HalfCheetah
0
0
0.0M
0.5M
1.0M
1.5M
2.0M
0k
200k
400k
Training Steps
Training Steps
(a) Online Training
(b) Online Training with Non-Empty Replay Buffer
Figure 6: Training Curves for Online RL Gym Mujoco Locomotion tasks. (a. Replay buffer initialized empty. b. Replay buffer is initialized with medium-level dataset.)
80
100
2
59
51
46
45
60
5
0.61
83
79
81
2.8
92
51
57
60
75
61
52
46
-0.19
0.5
1.81
101
Returns
50
0.25
62
53
48
0.5
-0.19
81
83
82
0.05
0.99
70
74
52
50
20
25
58
53
49
47
-0.18
57
84
82
0.67
70
45
0
0.25
1
2
0.5
1
5
0
0.05
0.5
5
d2
Q2
02
(a) HalfChectah Medium
(b) Walker2d Medium
(c) Hopper Medium
Figure Ablation on 01 and az for Offline RL tasks
Effect of Bootstrap Targets Actor Loss. Figure evaluates the effect replacing the com- pletion loss with self-supervised shortcut loss that bootstraps targets from the actor itself. The completion loss in equation 12 uses sample (or action) from the dataset itself as a regression tar- get, but the bootsrap shorteut loss in equation I0 uses smaller step sizes to create self-consistency target. The purpose of these auxiliary losses in offline RL is to serve as proxy to behavior cloning- based policy regularization and restrict the actor toward the behavior policy Ta, thereby reducing extrapolation error by discouraging out-of-distribun while the completion loss uses ground- truth actions and encourages alignment with the dataset dis- tribution, the bootstrap shortcut loss does not provide such constraints, as its targets are generated by the model and may themselves be OOD.
(OOD) actions passed to the critic. However,
As hypothesized, increasing the coefficient of the bootstrap loss degrades performance, with larger values causing severe instability. This is especially evident HallCheetal-medium,, where Q2 0 already yields strong performance (as seen in earlier ablations), providing clean setting to isolate the impact the bootstrap loss. contrast, scaling the completion loss does not drastically harm performance (Figure 7) theoretically. it biases the policy toward behavior cloning but still remains stable. In the bootstrap case, however, higher ag places greater weight on poorly calibrated self- self-generated targets, destabilizing Q-learning as seen in the training log presented in Figure 8b
20

---

Published as a conference paper at ICLR 2026
Batch-average Q-value
Normalized Return
60
600
0²=0
400
40
02=0.05
02=0.5
200
20
02=1.0
02 0.55.0
0
0
Ok
200k
400k
0k
200k
400k
Training Steps
Training Steps
(a) Evaluation Log
(b) Training Log
Figure 8: Replacing Completion Loss with Bootstrap Shorteut Loss in the Actor Objective (HalfCheetah Medium; 01 = 1.0)
A.7 COMPARISON WITH FQL
Flow Q-Learning (FQL; Park al. (025) ) and method, SSCQL, both investigate single-step policy representations, but they adopt different design choices. In FQL, a flow-matching model is trained with behavior cloning (BC) objective, and its outputs used to regularize separate MI.P- based policy via behavior-regularizer policy optimization. Importantly, the flow model not used directly for action selection, neither during policy evaluation nor runtime inference. In contrast, SSCQL trains a single flow-based model that serves directly as the policy, The actor and behavior regularization components both depend solely on this model.
The key distinctions are presented in the following points.
SSCQL uses the same generative model for both training and inference. The single-step completion policy is distilled in the same flow model. However, FQL trains a separate MLP actor for policy distillation.
SSCQL uses dataset actions directly for behavior regularization, without relying on surro- gate targets from another model.
More importantly, there are no full-flow rollouts our training. While FQL avoids BPTT during policy optimization, it still requires full flow trajectory rollouts to generate BC tar- gets. SSCQL avoids this entirely and remains efficient throughout.
The final policy in SSCQL is generative model representing the completion vector field, enabling both efficient single-step inference and optional full-flow rollout when additional compute is available; results for this variant reported later in the section.
For empirical comparisons, Flow-QU -L does not report results on D4RL locomotion benchmarks. We ran the experiments ourselves and followed the suggested hyperparameter tuning, training with alpha values 10.1, 0.3, 1.0, 2.0). The results, along with the max performance (tuned hyperparameter) among the collected results, are reported below.
Table 11: Offline RL Empirical Result Comparison with Hyperparaameter-Tuned FQL
Gym Tasks
FQL(a 0.1)
FQL(a = 0.3)
FQL(a 1.0)
FQL(a = 3.0)
FQL(Max)
SSCQL
Halfcheetah-ME
102.11
96.19
91.47
89.29
102.11
98.1
Hupper-ME
49.61
60.24
76.70
61.93
76.70
110.9
Walker2d-ME
99.11
97.52
102.57
92.52
102.57
111.1
Halfcheetah-M
55.63
52.81
46.87
43.53
55.63
52.3
Hopper-M
60.62
54.06
48,91
60.62
102.4
Walker2d-M
31.55
45.55
64.24
65.91
Halfcheetah-MRR
48.33
44.56
39.43
39.27
48.33
44.4
Hopper-MR
27.00
30.54
50.70
44.95
50.70
101.4
Walker2I-MR 16.59
13.31
32,29
38.82
38.82
85.9
Unlike FQL, the final distilled policy in SSCQL remains a generative model that parameterizes a vector field. While our main focus is single-step inference, also implement variant for offline
21

---

Published as a conference paper at ICLR 2026
RL that performs multi-step integration along the learned completion vector field from timestep (r) O to 1, analogous to flow matching. Specifically, Euler's method with 10 discrete steps is used. And no additional hyperparameter tuning performed; i.c., the same completion vector field trained for single-step generation is directly reused for the Completion Flow results reported in Table T2
Table 12: Evaluation of Completion Vector Field: Multi-step (10-step) Rollout vs. Single-step Prediction. The model trained for SSCQL using single-step completion evaluated in a 10-step rollout setting, referred to as completion flow.
Environment
Completion Flow
Single Step Completion
Hallchectah-M
53.7
52.3
Halfcheetah- -MR
47.1
44.4
Hopper-M
103.1
102.44
Hopper-MR
94.8
101.4
Walker2d-M
83.9
84.2
Walker2d-MR
98.0
85.9
These results suggest that the learned completion vector encodes meaningful flow structure capable of guiding iterative action generation toward high-reward regions, even without explicit multi-step backpropagation during training. This highlights its dual capability: enabling efficient single-step inference by default, while also supporting Now-based rollout when additional compute is available.
B EXPLOITING SUBGOAL STRUCTURE IN GCRL WITH COMPLETION MODELING
B.1 OFFLINE GOAL- COADITONDD REINFORCEMENT LEARNING
coal-conditioned reinforcement learning (GCRL) is a paradigm for learning general-purposs poli- cies capable of reaching arbitrary target states without requiring policy retraining. Unlike standard reinforcement learning, where the policy is conditioned only on the current state, GCRL augments the Markov Decision Process (MDP) with goal space G, enabling goal-directed behavior. The goal space G often coincides with the state space or represent task-relevant subsets of state fea- tures (e.g.. spatial positions in navigation object locations in manipulation) (Liu et al. 2022a Andrychowicz et al. 2017).
In the offline setting, goal-conditioned RL seeks to learn from static datasets using sparse goal- matching rewards, eliminating the need for dense or eagineeggeereed reward functions. The objec- tive is to learn goal-conditioned policy (a s, S G + A that reliably reaches specified goal state g E G from arbitrary initial states. The reward function typically takes a sparse form r(s, a. g)
1 (s + g). penalizing the agent every step until the specified goal is achieved (Park)
ct al. 2024a Ma et al. 2022 Sikchi et al. 2023 Xu et al. (022).
Recent goal-conditioned RL (GCRL) methods such RIS (Chane-Sane et al. 2021). POR (Xu let al. 2002). and HIQL (Park et al., 2023) adopt a hierarchical formulation, decomposing control into high-level subgoal selection and low-level action execution. These methods have shown strong performance over flat policies. HIQL, which shares methodological similarities with POR, learns multi-goal policies from a shared goal-conditioned value function trained via implicit Q-learning. In this work, we start with and finally compare against this line of hierarchical policy design. Specif- ically, HIQL introduces two policies: high-level policy z that selects subgoals, and a low-level
policy Té, that generates actions conditioned to reach those subgoals. More specifically, at test time, the high-level policy samples intermediate subgoal $ T/) (. st. g), which is passed to the low-level policy. The low-level policy then selects an executable action conditioned on the currett state and subgoal: a, Té St. S2+4).
B.2 GOAL-CONDT IMPLICIT VALUE LEARNING (GCIVL).
Implicit Q-Learning (IQL) asymmetric loss function in SARSA-style policy evaluation framework to train a state-value network Vg. mitigating the impact of out-of -fdt----iiiibutiin actions during critic training (Kostrikov al. 202T.. The critic networks are trained via TD target and an
22

---

Published as a conference paper at ICLR 2026
expectile regression loss:
Lo(0q) = (( >Vo
Vov (5') - -o (s,
(28)
(v(vv) = E Lg Q2o (5, a) - Ver (s))
(29)
where the expectile loss is defined L = K5 1(u 0))u², with (0.5, 1.0).
We an adaptation of IQL to action-free variant called IVL in goal coal-conditioned setting to learn
a Goal- Cond-iiiininndd Implicit Value (GCIVL) (Xu et al.] 2022 Park et al. (023). Unlike standard
IQL,, GCIVI avoids Q-function training and instead constructs conservative value target directly
using a bootstrapped estimate from a target value network:
{(. g) r(s,g) g) + 'Vo (s'
(30)
where Vi, represents target network for the goal conditioned value prediction.
The GCIVI. objective then becomes:
Lv (ov) is) [L (4(s.g) - Vev (s,g))]
(31)
To further stabilize training and reduce overestimation bias, we employ two value networks and compute the final loss the sum of expectile losses across both.
B.3 HIERARCHICAL POLICY EXTRACTION, DISTILLATION AND EXECUTION
Recent works in goal-conditioned reinforcement learning (GCRL), such as RIS (Chane-Sane al., 2021] and HIQL (Par et al., 2023), propose hierarchical architectures that decouple the learning of complex tasks into high-level planning and low-level control. These methods employ high-level policy to generate subgoals and a low-level policy to produce primitive actions for reaching them. Such hierarchical decomposition facilitates more structured exploration and reduces overall policy approximation error, particularly when paired with goal-conditioned value function that serves as consistent learning signal for both levels.
HIQL builds upon Implicit Q-Learning (IQL) and its action-free variant to train a goal-conditioned value function, and extracts both high- and low-level policies using advantage-weighted regression (AWR) objectives. Notably, the hierarchical extraction procedure in HIQL is agnostic to the specific offline RL method used to obtain the value function, as also evidenced by RIS, which employs different critic training paradigm. The high-level policy =& proposes subgoals based on the cur- rent state and final goal, while the low-level policy Th_ generates actions achieve the proposed subgoals. Both policies are optimized via AWR with their respective advantage estimators:
LT (fh) = E [- exp (B- Ah S8+k: 9)) log nb) (St+k I st,9)]] exp (B A8(s, ae 86+k)) log T (a. 11, 8c,+()
(32)
C.e (0t) = (ler,
(33)
At deployment time, the hicrarchical agent executes as follows: given a state st and a goal g. the high-level policy samples intermediate subgoal 8t+* ~ nh, (. st.g), which is provided input to the low-level policy. The low-level policy then chooses an action conditioned on the current state and subgoal: or ( (- St, St+k). Our approach also follows this general philosophy:
train a goal-condittoned value function (e.g., using GCIVL) and use it to distill hierarchical properties
into flat policies in post hoc manner by exploiting subgoal structure. Loss functions for training
mhortcut-based flat policies to exploit this structure are:
Co( = Es a'))-~ exp (B AA (st, g)) log To(ar, g' st.g.k = 0, =1)]
(34)
(0) = E( [- exp (8- (8-A (s,,, a')gg)))))))))" log log st11 s k= 1, 1)]
(35)
(.) ((s..g) (oons)))rro.sssssssooo [- log ne (ar, 8141 8t, =)]
(36)
Here, g' denotes the subgoal located & timesteps ahead of s, in the trajectory, where i: = min ty T, t). This formulation ensures that g' remains a valid subgoal, with respect to the trajectory termination at the timestep T. the designated goal timestep lg. and the intended subgoal horizon k.
23

---

Published as a conference paper at ICLR 2026
encourages the policy to directly optimize over intermediate states (subgoals) in the trajectory using high-level advantage estimates. equation 35 L1 corresponds to standard short-horizon behavior cloning weighted by low-level advantages (AWR). Finally, in equation 36 we incorporate a shortcut loss L. that distills the hierarchical decision-making pro- cess into flat policy via supervised learning. This involves querying the hierarchical inference md=1 module (i.e., composition of Tk=U and T to generate behavior, which is then used as a target distribution for training the flat policy. The gradient is stopped through the target (ar, 9t-1) used in Eq. 36 This approach enables our flat policy to inherit hierarchical structure and long-horizon reasoning capabilities without explicitly learning multiple neural network controllers (since single shared architecture is employed) and without relying on multiple inference steps through the learned controller(s).
The first loss, Cu in equation 34
B.4 IMPLEMENTATION DETAILS
B.4.1 ADVANTAGE ESTIMATES AND GOAL REPRESENTATION
We adopt the simplified advantage estimates introduced in Park et al. (2023) to compute the advantage-weighue regression (AWR) losses Equations 34 and 33| leveraging a critic trained with the goal-conditioned implicit value learning (GCIVL) framework. Building the work of Xu jet al. (2022), Park et al. (2023) proposed an efficient approximation to the AWR advantage terms by replacing bootstrapped multi-step returns with value function differences. This simplification yields empirically strong results while avoiding additional critic complexity. We adopt these approxima- tions and compute the high-level and low-level advantage estimates as:
A" (st, g': 9) Ver (g's 9) Vør (St, 9); A9(s, Ot, g') Ver (s++1, g¹) Vør (sc: g¹)
(37)
(38)
where Ver (: denotes the GCIVe-trained value function conditioned on a goal, and the advantage estimates Å, computed as value differences, serve as practical proxies for true advantages in the hierarchical setting.
While our policy тe models the joint distribution over (a,,')), it does not predict the raw subgoal g' (e.g., 8t+*) directly. Instead, it outputs an encoded representation e(s, g') g of the subgoal, where e denotes a learned goal encoder. This encoder is jointly trained with the value network, such that the goal-conditioned value function is represented as Vov (s, ('))g))).
B.4.2 HIERARCHY CONDITION REPRESENTATION
Our goal conditioned policy Fe is additionally conditioned on discrete hierarchy indicators and d, which represent the current layer index and the step depth within the hicrarchy, respectively. These indicators modulate the behavior of the shared policy network to emulate distinet roles across the hierarchy. For example, Ti= represents the high-level policy, responsible for subgoal inference (although is also trained to optimize primitive actions via the high-level advantage A²), whereas T==1 corresponds to the low-level policy, primarily optimizing actions (but also trained to optimize next states 814-1 an immediate subgoal using A'). This shared parameterization enables structured policy composition, where d==1 d=1 denotes a standard two-layer hierarchical rollout, and = Tk=1
captures a shortcut inference from the top layer task goal to immediately executable action via shortcut policy distillation.
To encode the discrete hierarchy indicators (k, d). reuse the same temporal encoding chitecture developed for the flow- policy in SSCQL (Appendix A..4). Specifically, each discrete input is passed through learnable Fourier feature encoder that maps scalar to [cos(2xx:)), sin sin(2xxWW),. where W E Rdim/2x1 are learnable frequencies. These features are processed by separate two-layer MLPs with Mish activation and hidden size dim, producing em- beddings that are then concatenated with the state and goal embeddings to inform the policy net- work. This design enables the policy to condition flexibly on both temporal and structural context, supporting unified inference across hierarchical levels.
24

---

Published as a conference paper at ICLR 2026
Algorithm 3 Goal-Conditioned Single-Step Completion Policy (GC-SSCP)
1: Initialize: Policy parameters 0, critic parameters ov
2: Initialize target networks: 0' +- @v & Φv
3: for N gradient steps do
4: Sample batch {(5;, ae, r2, $141: ~D
5: Critic Update:
D. see Eq. 31
6: Compute target: ((5; g) Tt + Vo, (51+1.9)
7: Compute critic loss:
(vvvv) = Bs (1:))= [Le (Ā( g) Vav (sc.
8:
@v t - - or (vv
9:
Actor Update:
10:
Advantage Estimates;
D see Eq. 37 38
(s = Vor (6' Ver (5t
A's (1,),,,,'))) Ver (st+1.g') Vør (5t.g)
11:
AWR Loss:
see Eq.
34 35
= 0, 0. Lo(0) ED - exp (3. (st, 9)) log mo(an. 11,9, 10
(0) = ED - exp (3.A A' at. log To 8441 st. 1, d -dd 1)
Shorteut Completion Loss:
Distillaaiin; see Eq. 36
()) = (1o |- log Te 8t41 St,g, = 0, d = 2)]
13:
Total Actor Loss:
(0) = Co( ()) (0)
0 0 - 17 V (0)
15:
Target Network Updates:
16:
Hy + Ts Φv + (1 - TT) dy
8' To + (1-Tg)- 8'
18: end for
B.4.3 ALGORITIM AND HYPERPARAMETERS
The Coal-Conditioned Single-Stey Completion Policy (GC-SSCP) framework is outlined in Algo- rithm 3, We implement our method using the JAX machine learning library, utilizing the OGbench benchmark suite for standardized environment setups and baseline implementations. The common hyperparameters used across all goal-conditioned training environments are summarized in Table T3
Across tasts/datssets, we vary the discount factor (7) and the subgoal step interval for sampling the subgoals. The values are tabulated in Table [4] A discount factor of 0.99 performs well in medium- scale environments, while more complex or larger environments benefit from slightly higher value of 0.995. Similarly, medium tasks use a subgoal step size of 25, whereas larger environments employ 50 or 100 steps. We present detailed ablation study on subgoal step sensitivity in Section B.6
25

---

Published as a conference paper at ICLR 2026
Table 13: Common Hyperparameters for GC-SSCP Across All Offline GCRL Experiments
Hyperparameter
Value
Batch Size
1024
Soft Update Rate for Q-Networks (To)
0.005
Soft Update Rate for Actors (To)
0.001
Learning Rates for All Parameters
3 x 10
Hierarchy Level Dimension
128
Goal Representation Dimension
10
Training Steps
500k
Actor Network Size
(512, 512, 512)
Critic Network Size
(512, 512, 512)
IQL Expectile (5)
0.7
AWR Inv. Temperature (3)
5.0
Shortcut Loss Coefficient (A)
1.0
Table 14: Hyperparameters for GC-SSCP across OGBench Environments
Environment
Discount Factor (?)
Subgoal Steps
pointmaze-medium naviuate-v-i
0.99
pointazze-large-naiigtee
0.995
100
pointmare-giant-navigga---
0.995
50
pointmazeeelleppttnnvvvgga----
0.995
50
pointmase-mcdiun--siiih---
0.99
25
pointaaze-largesstich---
0.995
pointmaze-giian-ssiich---0
0.995
50
pointnasee-clepoorsstiic----
0.995
atthaaze-mmdiinn-nairgaa--v-
0.99
25
antmaaze-lage-navigatee-0
0.995
100
autmaazegianl--avviaee--0
0.995
100
autmaaze-elopoor-nnviigtt---
0.995
50
autmaze-mmcdum--siich-
0.99
autmaazelarge--ticch-000
0,995
50
antmaze-giante antmaze-giant-sticc--y
0.995
natmaze-teleporr-sticch.
0.995
50
B.5 TRAINING CURVES
Figure 9 presents training curves for offline goal-conditioned RL on OGBench environments across two task families: PointMaze and AntMaze, each with Navigate and Stitch variants. For each vari- ant, the results are shown for four environments: Medium, Large, Giant, and Teleport. For each of these environments, the evaluation is done across five different goals. The vertical axis indicates the mean success rate averaged over training seeds, with shaded regions representing the stan- dard deviation. The horizontal axis shows training steps from 0 to 500k. These curves highlight the performance and learning stability of our method across progressively more complex multi-goal
tasks.
B.6 ABLATION
AWR Inverse Temperatures. Although we use the same inverse temperature hyperparameter (3) for both AWR losses defined in Equations 34 and 35| it is not strictly necessary. By default, we set = 5.0 for both losses. To study the effect of this design choice, we perform coarse hyperparam- eter sweep over 31 (for the high-level controller) and Bg (for the low-level controller). using values (0, 1,5, 10).
Figure shows the results in the form of heatmap, with 31 along the vertical axis and Bg along the horizontal axis. The values represent the average success rate, computed over 50 evaluation seeds and different goals. We omit multiple training seeds here due computational constraints, this
26

---

Published as a conference paper at ICLR 2026
1.0
Normalized Return
Normalized Return
0.8
0.8
0.6
0.6
0.4
0.4
Medium
Large
0.2
Glant
0.2
Teleport
0.0
0.0
0k
200k Training Steps
400k
Ok
200k
400k
Training Steps
(a) PointMaze Navigate Tasks
(b) PointMaze Stitch Tasks
1.0
Normalized Return
Normalized Return
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0k
200k
400k
0k
200k
400k
Training Steps
Training Steps
(c) AntMaze Navigate Tasks
(d) AntMaze Stitch Tasks
Figure 9: Training Curves for Offline Goal Conditioned RL in OGBench Environments with Multi- goal Tasks
would require several repeated runs and evaluations per setting, While our default hyperparameter setting $1 82 performs well, certain other combinations yield slightly better results well.
Notably, when either 31 or 32 is set to 0, the corresponding AWR loss reduces to standard behavior cloning loss. We observe that setting 31 tends to degrade performance more significantly than setting 8n- = 0, suggesting that learning effective subgoals at the high level is more critical than fine-tuning low-level actions. Even when the low-level controller is trained with behavior cloning alone, the model can perform reasonably well if the high-level subgoal predictions are sufficiently informative.
Inference Type. Figure I1a presents an ablation on inference strategies. compare the proposed ==d1
flat inference md=2 against hierarchical inference T Tk=] PointMaze Teleport Navigate and AutMaze Large Navigate. The training curves show no significant performance gap. suggesting that hierarchy distillation effectively flattens the policy without sacrificing performance.
Subgoal Choice. Figure 11b studies the effect of the subgoal step interval, with values ranging from 5 to 500 on the x-axis and mean success rate on the y-axis. Both environments exhibit relatively stable performance across different subgoal intervals, with slight peak around 25-100 steps. Even at the extremes (5 or 500), the performance does not degrade substantially, indicating robustness to this hyperparameter.
27

---

Published as a conference paper at ICLR 2026
10
0.36
0.37
0.42
0.58
10
0.68
0.86
0.82
0.85
0.8
0.5
0.29
0.36
0.49
0.48
5
0.72
0.72
0.87
0.92
Score
B1
0.4
B1
0.6
0.32
0.51
0.56
0.4
0.82
0.84
0.84
0.85
0.3
0.4
0.21
0.28
0.44
0.25
0.54
0.34
0.29
0.31
5
10
5
10
B2
B2
(a) Pointmaze Teleport Navigate
(b) Antmaze Large Navigate
Figure 10: Ablation on 31 and B3 hyperparameters for Offline GCRL tasks
1.0
1.0
Mean Success Rate
0.8
Mean Success Rate
0.8
0.6
0.4
0.4
PMTN-Flat Flat
PMTN- Hierarchical
0.2
AMLN-Flat
PointMaze Teleport Navigate
AMLN-Htierarchical
AntMaze-Large-Naaigtte
0.0
0.0
100k
200k
300k
400k
500k
200
Training Steps
Subgoal Steps
(a) Ablation on Test-time Inference Type
(b) Ablation on Subgoal Steps Hyperparameter
Figure 11: Ablation study based on inference type and subgoal steps for Offline GCRL tasks
C BEHAVIOR CLONING
The experiments in this section are based on the Behavior Cloning (BC) benchmark, specifically the State Policy setting introduced in Diffusion Policy Chi et al. (2024). We closely follow the experimental setup and implementation provided the official Diffusion Policy GitHub reposi- tory Among the two baseline models Serided-CNN-biiee the Transformer-bass version due to its superior performance on the state-based robotic manipu- lation tasks considered. All implementations are in PyTorch. For fair comparison, we adapt our Single-Step Completion Policy (SSCP) to match the temporal context and action sequence condi- tioning used by Diffusion Policy. SSCP is trained to predict future action sequences based on history past states, aligning with the context modeling and planning regime of Diffusion Policy.
and Transformer baseerr-badwwweee adopt
We evaluate SSCP on behavior cloning tasks using state- state-based based, proficient human demonstration datasets from RoboMimic (Mandlekar et al. 202[) and the contact-rich Push-T task (Florence et al.) 2022). RoboMimic consists of five robotic manipulation tasks with high-quality human demon- strations, while Push-T requires precise manipulation of a T-shaped object under variable initial conditions and complex contact dynamics. In addition Diffusion Policy, other baseline methods include LSTM-GMM (L-G) Mandlekar et al. (202I), Implicit Behavior Cloning (IBC) Florence al. (2022). and Behavior Transformer (BET) Shafiullah al. 2022). The results reported in
Table 15
httpps://githuub.commeeet httt:://wttth....m...sssseeffffa
28

---

Published as a conference paper at ICLR 2026
While Diffusion Policy achieves strong results range of challenging manipulation tasks, it relies on iterative denoising with typically 100 reverse steps, resulting in significant inference-time latency. This poses critical limitation in real-time robotics applications, where responsiveness essential. In contrast, SSCP generates action predictions in single forward pass, completely eliminating iterative sampling inference. As demonstrated in experimental results, SSCP achieves competitive performance while offering inference efficiency. This positions SSCP as practical alternative for real-time deployment in robotics and controls.
1.0
1.0
Mean Success Rate
Mean Success Rate
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
SSCP 3 seeds
SSCP 3 seeds
0.0
0.0
0k
10k
20k
30k
0k
20k
40k
60k
80k
Training Steps
Training Steps
(a) Lift
(b) Can
1.0
1.0
Mean Success Rate
Mean Success Rate
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
SSCP 3 seeds
SSCP 3 seeds
0.0
0.0
0k
25k
50k
75k
100k
0k
100k
200k
300k
Training Steps
Training Steps
(c) Square
(d) Transport
1.0
1.0
Mean Success Rate
Mean Success Rate
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
SSCP 3 seeds
SSCP 3 seeds
0.0
0.0
100k
200k
300k
0k
10k
20k
30k
40k
Training Steps
Training Steps
(e) Toolhang
(f) Push-T
Figure 12: Training Curves for Behavior Learning Experiments (1000 epochs of training and evalu- ations with different seeds)
29

---

Published as a conference paper at ICLR 2026
The training curves of SSCP the imitation learning benchmark are presented in Figures f for the six robotic manipulation tasks: Lift, Can, Square, Transport, Tool-Hang, and Push-T. For each task, we train SSCP using three different random seeds for the model initialization and training, as well as distinct dataset seeds, ensuring diversity in data exposure and robustness to initialization. Each training run consists of 1000 epochs. To monitor learning progress and evaluate policy gen- cralization, we periodically evaluate the model every 50 epochs during training. Each evaluation involves running 50 independent rollouts using environment seeds not seen during training. This setup enables us to report a reliable estimate of the model's performance and assess the stability of the learned policy across different training runs.
Following the evaluation protocol from Chi et al. (2024). we report the average performance over the last few checkpoints (in our case, the last 5) each averaged across training seeds and evaluated over 50 different environment seeds (resulting in 3 x 50 150 rollouts per checkpoint, and a total of 150 = 750 rollouts per task). The results are summarized in Table [5] Note that while the baseline results taken from Chi et al. (2024) are the averages over the last 10 checkpoints, this choice is appropriate for their setting, as their models were trained for 4500 epochs. In our case, however, we only train for 1000 epochs, so averaging over the last 10 checkpoints would include models from the mid-tre mad-training phase. Hence, using the last 5 checkpoints provides a more accurate and fair reflection of final policy performance in setting.
Table 15: Behavior Cloning Final Results
Tasks
DP
L-G
IBC
BET
SSCP
Lift
1.00
0.96
0.41
0.96
1.00
Can
1.00
0.91
0.00
0.89
0.99-10.01
Square
0.89
0.73
0.00
0.52
0.90=0.03
Transport
0.84
0.47
0.00
0.14
0.86+10.88
Toolhang
0.87
0.31
0.00
0.20
0.64
Push-T
0.79
0.61
0.84
0.70
0.85
Average
0.88
0.68
0.21
0.57
0.87
In addition to reporting the average over the last checkpoints, Chi et al. (2024) also reports the max- imum performance achieved by each method during the entire 4500 epochs of training, evaluated every 50 epochs. Although we train SSCP for only 1000 epochs, we follow similar protocol and report its maximum performance in Table [6 Naturally, SSCP lags somewhat behind diffusion pol- icy in this setting, given its one-shot inference (compared to diffusion policy's 100-step iterative denoising) and shorter training duration. Nonetheless, SSCP remains competitive with diffusion policy and significantly outperforms all other baselines included in the comparison.
Table 16: Behavior Cloning Results (Max Performance)
Tasks
DP
L-G
IBC
BET
SSCP
Lift
1.00
1.00
0.79
1.00
1.00
Can
1.00
1.00
0.00
1.00
1.00
Square
1.00
0.95
0.00
0.76
0.93
Transport
1.00
0.76
0.00
0.38
0.90
Toolhang
1.00
0.67
0.00
0.58
0.74
Push-T
0.95
0.67
0.90
0.79
0.89
The key hyperparameters used in experiments summarized in Table 17 The architecture and hyperparameters used in our behavior cloning experiments are largely consistent with those of the original Diffusion Policy (Chi al. 2024). This similarity demonstrates that SSCP can be easily implemented on top of existing diffusion--aasee robot learning pipelines, achieving reliable performance with minimal architectural and hyperparametric modifications.
30

---

Published as a conference paper at ICLR 2026
Table 17: Hyperparameters used in Behavior Cloning Experiments
Hyperparameter
Value
Observation Horizon
2
Action Horizon
8
Action Prediction Horizon
10
Transformer Token Embedding Dimension
256
Transformer Attention Dropout Probability
0.3 (0.01 for Push-T)
Learning Rate
3e-4 (1c-3 for Toolhang)
Weight Decay
1e-10
Training Epochs
1000
Evaluation Frequency (in Epochs)
50
D BENCHMARK, TASKS AND BASELINES DETAILS
D.1 OFFLINE RL
D.1.1 D4RL DATASETS.
D4RI (Datasets for Deep Data-Driven Reinforcement Learning) a widely-used benchmark suite for offline reinforcement learning, where policies learned solely from fixed datasets without further interaction with the environment. These datasets are paired with standardized Gym envi- romments (Fu et al. 2020 Brockman al. 2016). We evaluate our method on continuous control tasks from the Gym- MuloCd locomotion suite (Todorov et al. 2012). including Ha wwweffheettto ah (2I) quadruped running), Hopper (2D monoped hopping), and Wa ker2d (2D biped walking). We consider three dataset types from D4RL: med í um, med i um cxmm-xxeet,... and med um-rep aaylay...... The medium um datasets are generated by policies with intermediate performance trained using Soft Actor Critic (Haarnoja et al. 2018). The medi um expert datasets combine data from both medium and expert-level policies. The medi um replay datasets consist of replay buffers col- lected throughout the training of medium-level policies.
(a) HalfCheetah
(b) Hopper
(c) Walker2d
Figure 13: Gym Mujoco 2D Locomotion Continuous Control Environments
D.1.2 BASELINES.
We compare SSCQL. against a broad spectrum baselines, encompassing both unimodal Gaussian policy methods and recent state-of-the-an generative approaches.
Implicit Q-Learning (IQL) (Kostrikov et al. 2021) addresses the offline RL setting by decoupling policy learning from out-of-distribution actions. IQL trains state-value function Vo using asym- metric loss that implicitly avoids extrapolation errors in the critic. It enables robust policy extrac- tion via advantage-weight
regression while maintaining off-policy stability. TD3+BC (Fujimoto & Gu 2021) combines the TD3 actor-critic framework (Fujimoto et al. 2018) with a behavioral
Foundation/D4RI
bttbss:/qymmiibarst https://github.r com/ dwwpmi /t
farama farama.org/
31

---

Published as a conference paper at ICLR 2026
cloning loss to regularize the policy towards the dataset. This simple hybrid effectively constrains policy updates to the support of the dataset, preventing the selection of out-of-distributee actions and overestimation by the Q-function. Conservative Q-Learning (CQL) (Kumar al. 2020) in- troduces a conservative penalty during Q-function training to lower the estimated values of unseen -uutrof-distributiion) actions. This encourages learned policies to remain close to the dataset's action distribution, reducing overestimation and improving reliability in offline RL.
Diffuser (Janner et al. 2022) formulates trajectory generation as a denoising diffusion process over full state-action sequences. During inference, the reverse diffusion is guided by the gradient of pre- dicted cumulative rewards, enabling model-based trajectory optimization toward high-return plans. Implicit Diffusion Q-Learning (IDQL) Mlaasen-EEstruce et al. (202)) uses a diffusion model to generate action samples, which are then filtered via rejection sampling using a Q-function trained with IQL. This approach retains the expressiveness of generative models while leveraging the Q- function for policy selection. Diffusion Q-L-eanning (DQL) (Wang et al. 2022) integrates diffusion- based generative models directly into policy optimization. It treats the policy as a stochastic denois- ing process and enables policy improvement by backpropagating Q-function gradients through the generative sampling chain. DQL exemplifies broader trend of incorporating iterative diffusion into reinforcement learning algorithms. Consistency Actor-Critic (CAC) (Ding & Jin (203)) leverages consistency models [Song et al. (2023) to accelerate diffusion policy execution. It trains few-step de- noising models that approximate the final output of the diffusion process, enabling faster inference. Although CAC improves training and inference efficiency relative to DQL, it still remains slower and less scalable than our method (SSCQL).
D.2 BEHAVIOR CLONING
D.2.1 SIMULATION ENVIRONMENTS AND DATASETS
We mainly evaluate our method on state-based tasks adopted by Chi et al. (024) from the Robomimic benchmark (Mandlekar et al., (02T), large-scale suite for imitation learning and of- fline reinforcement learning. We focus on all five manipulation tasks using the ph (proficient human) datasets, which contain high-quality teleoperated demonstrations. The tasks include: Li ft, where the robot must grasp and lift a small cube; Can, which involves placing a coke can from a bin into smaller target bin; Square, where the robot must insert square nut onto rod; Transport, dual-arm task requiring the coordinated retrieval, handoff, and placement of a hammer while remov- ing a piece of trash; and Tool Hang, where the robot assembles a hook-frame and hangs wrench by performing precise, rotation-heavy manipulations. We additionally evaluate on the Push-T T task, which involves pushing T-shaped block to a fixed target using circular end-effector. The task contact-rich and demands fine-grained control.
D.2.2 BASELINES
Diffusion Policy (Chi ct al., 2024b adopts a behavior cloning framework by generating short- horizon action trajectories conditioned past observations via conditional denoising diffusson process. This enables receding-horizon planning and allows modeling of complex, temporally co- herent, and multimodal behaviors, making it well-suited for high-dimensional imitation learning from raw demonstrations. The iterative denoising approach gracefully handles multimodal action distributions and offers stable training dynamics. BC-RNN (Mandlekar al., 102T) is behavior cloning variant that employs recurrent neural networks (RNNs) to capture temporal dependencies in sequential data. The policy is trained on sequences of states and actions, predicting actions autore- gressively. [hh et al. (2024) reinplemented this approach using LSTMs combined with Gaussian mixture models (LSTT-GMM) to better represent multimodal action distributions. Implicit Be- havior Cloning (IBC) (Florence et al.] 2022) trains implicit policies using energy-based models (EBMs), which represent complex, potentially multimodal action distributions without requiring explicit likelihoods. This contrasts with standard explicit behavior cloning models trained via mean squared error on dataset actions, offering greater flexibility at the cost of more challenging optimiza- tion. Behavior Transformer (BET) (Shafiullah et al.) 2022) was designed to model multimodal, unlabeled human behavior using transformer architectures. By autoregressively predicting action sequences with attention mechanisms, BET effectively captures long-range temporal dependencies.
32

---

Published as a conference paper at ICLR 2026
(a) Lift
(b) Can
(c) Square
(d) Transport
(c) Toolhang
(f) Push-T
Figure 14: Benchmark Tasks for Behavior Cloning Experiments
D.3
OFFLINE GoAL CONDITIONED RI.
D.3.1 OGBENCH
We evaluate on OGBench (Park et al. 2024a), recent benchmark for offline goal-conditioned reinforcement learning (GCRL) that tests multi-task policy learning across diverse navigation and control scenarios. We consider the PointMaze and AntMaze environments based on the MuJoCo simulator [Toorov al.) 2012), where agents must learn both high-level navigation and low-level locomotion from diverse, static datasets. PointMaze involves controlling a 2D point mass, while AntMaze requires quadrupedal control with an 8-DoF Ant robot.
consider 16 environment-dataset combinations: (pointmaze, antmaze] x {medi um, large, giant, teleport) x fnavigate, stitch). Maze sizes vary from medius um (smallest) to giant (twice the size of large). The teleport maze introduces stochasticity via black holes that transport the agent randomly selected white holes, of which is dead end, penalizing the 'lucky' optimism and requiring robust planning.
Dataset types include naviga te, collected via noisy goal-conditioned expert exploring the maze, and st i ch, composed of short goal-reaching trajectories that require policy stitching to reach dis- tant goals. All datasets are generated with low-level directional policy trained via SAC (Haarnojal et al. 2018 Following the OGBench evaluation protocol, we report average success rates over 6000 evaluation episodes training seeds 50 test seeds goals x 3 final evaluation epochs). Baseline results are taken directly from the OGBench benchmark paper, where each GCRL method trained for 1M steps and evaluated every 100k steps, with final performance averaged over the last three evaluations (800k, 900k, and 1M steps). For GC-SSCP, we train for 500k steps and evaluate every 50k steps, reporting averages over the final three evaluations (400k, 450k, and 500k steps).
D.3.2 BASELINES
Goal-Conditioned Behavioral Cloning (GCBC) (Lynch et al. 2020 Ghosh et al. 2019) performs behavioral cloning using future states in the same trajectory as goals. It lears conl-cond policy by treating the task as supervised learning over (s, 9, a) triplets, where g is a future state in the trajectory. Goal-Conditioned Implicit (V, Q)-Learning (GCIVL and GCIQL) are goal-
33

---

Published as a conference paper at ICLR 2026
(a) PointMaze
(b) AntMaze
(c) Navigate
(d) Stitch
Figure 15: OGBench Agent and Dataset Types
conditioned extensions of Implicit Q-Learning (IQL) (Kosrrkovv al.) 2021), an offline RL algo- rithm that fits conservative value functions via expectile regression. GCIQL learns both Q(s, a, .) and ((s.g), g). while GCIVL simpler V-only variant that omits learning Q. Quasimetric RL (QRL) (Wang et al. 2023) is value-learning algorithm that models the shortest-path distance between states as quasimetric, an asymmetric distance function. explicitly enforces the tri- angle inequality better capture the geometric structure of goal-reaching tasks. Contrastive RL (CRL) (Bysenbach et al. 2022) goal-conditioned RL method that performs one-step policy improvement using coul-conditioned value function trained via contrastive Icarning objective. Hierarchical Implicit Q-Learning (HIQL) (Park et al. 2023) extracts a two-level hierarchical pol- icy ccy-sonsis of high-level subgoal predictor and low-level action predictor-from single goal-conditioned value function. At test time, inference performed hierarchically: the high-level policy first predicts subgoal, which the low-level policy then uses to generate executable actions.
34


Reviews:
[{"review_id": "hoE2tqYanL", "reviewer": "Reviewer_a6ST", "summary": "This work considers generative modeling in RL, like diffusion models and flow-matching. Although these models have made great progress, they always rely on high inference costs and training instability caused by iterative sampling. To address this, this work proposes the Single-Step Completion Policy (SSCP), with an augmented flow-matching objective to predict direct completion vectors from intermediate flow samples, enabling accurate, one-shot action generation. This method can be extended into offline RL, offline-to-online RL, and online RL. This method is verified across various settings and different environments.", "questions": "See weaknesses above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This paper is well written and easy to follow.\n\n- Stable training and efficient sampling are core concerns in diffusion policies in RL.\n\n- The proposed method is flexible to different settings, like offline RL, online RL, and offline-to-online RL.", "weaknesses": "- The Q update function (4) utilizes the standard TD error. However, various works in offline RL propose that there is an overestimation error of the Q function caused by the distribution shift. Thus, several works will choose conservative Q learning techniques like IQL. What about the performance of using IQL in the offline setting?\n\n- What is the difference between online RL and offline RL when applying SSCP?\n\n- In offline-to-online experiments like Fig.4 and Fig.5, it seems that online fine-tuning in various environments can not improve the performance. Is there any explanation?\n\n- It is better to add offline-to-online and online RL experiments to the main text.\n\n- There are still some diffusion policies for RL that need to be discussed, including online fine-tuning [1-3] and offline diffusion planners [4-6].\n\nRef:\n\n[1] Policy agnostic RL: Offline RL and online RL fine-tuning of any class and backbone\n\n[2] Exploratory Diffusion Model for Unsupervised Reinforcement Learning\n\n[3] Efficient Online Reinforcement Learning for Diffusion Policy\n\n[4] What makes a good diffusion planner for decision making?\n\n[5] Simple hierarchical planning with diffusion\n\n[6] Latent diffusion planning for imitation learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6fddAWB3qJ", "reviewer": "Reviewer_o22J", "summary": "The authors propose single-step completion in flow matching, enabling transitions between arbitrary intermediate states rather than only predicting instantaneous velocity (the zero-jump case). While the standard self-consistent shortcut model relies on bootstrapped, potentially inaccurate targets—risking drift and unstable exploration—the authors address this by fixing the target to the final sample and learning one-step completions from any time step to the final one\n\nThis leads to the Single-Step Policy Completion (SSPC) objective, which learns complex, multimodal policies in a single step. The method achieves competitive performance with significantly faster training and inference. They further extend it to goal-conditioned RL, where a shortcut-based flat policy replaces hierarchical structures, improving efficiency while maintaining strong results.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 1, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Big efficiency gains while maintaining or exceeding baseline performance.\n- The paper is well-written and the method is well documented.\n- Figure 8 in the appendix shows the strength of SSCP against shortcut models and makes the case that relying on bootstrap targets induces instability in the training. **I think that this figure is important since it clearly motivates the use of SSCP and thus would like to see it in the main text.**", "weaknesses": "N/A", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qUWrlyj1Wj", "reviewer": "Reviewer_SYWJ", "summary": "This paper introduces Single-Step Completion Policy (SSCP), a flow-based generative policy framework for reinforcement learning that achieves single-step action generation while maintaining the expressiveness of multi-step generative models. The key innovation is training a completion model (instead of a shortcut model[1]) that predicts normalized completion vectors from intermediate flow points directly to target actions, bypassing iterative sampling. The authors demonstrate SSCP's effectiveness across three settings: (1) offline RL with behavior-constrained policy gradients (SSCQL), (2) goal-conditioned RL where hierarchical policies are distilled into flat inference (GC-SSCP), and (3) behavior cloning. The method achieves competitive or superior performance compared to diffusion-based baselines while offering substantial computational advantages.\n\n[1] Frans, Kevin, et al. \"One step diffusion via shortcut models.\" arXiv preprint arXiv:2410.12557 (2024).\n[2] Park, Seohong, Qiyang Li, and Sergey Levine. \"Flow q-learning.\" arXiv preprint arXiv:2502.02538 (2025).\n[3] Espinosa-Dice, Nicolas, et al. \"Scaling Offline RL via Efficient and Expressive Shortcut Models.\" arXiv preprint arXiv:2505.22866 (2025).\n[4] Sheng, Juyi, et al. \"MP1: MeanFlow Tames Policy Learning in 1-step for Robotic Manipulation.\" arXiv preprint arXiv:2507.10543 (2025).", "questions": "1. Why are multi-step actions worse in some cases, as shown in Table 9? Could you visualize the performance difference with x-axis = # of steps and y-axis = performance \n2. How does SSCP compare with other one-step policies proposed recently, like MP1?\n3. Can you explain why GC-SSCP fails as you scale up pointmaze-large-stitch to a larger setup?\n4. It has been claimed by many papers extending FQL that distillation of the policy is the bottleneck, while no one has verified this approach. Can you verify this by simply training a flow policy and performing BPTT to see if we can achieve stronger performance by just directly optimizing the Q-value of the flow policy? If that is the case, it will strongly support the necessity of having a stronger policy than a distilled one.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel and well-motivated approach: The completion vector formulation elegantly addresses a fundamental limitation of diffusion/flow policies—the need for iterative sampling—while maintaining expressiveness for multimodal action distributions. Unlike bootstrap-based shortcut methods [1], SSCP uses ground-truth targets from the dataset, avoiding early training instability.\n2. A significant practical advantage is that SSCP enables training generative policies without backpropagating through iterative generation chains, removing the requirement for distillation as shown in FQL\n3. The paper demonstrates consistent improvements across diverse benchmarks:\n4. The extension to goal-conditioned RL (GC-SSCP) is particularly innovative, showing that multi-level hierarchical reasoning can be compressed into a single flat policy without explicit hierarchical inference. This challenges the assumption that hierarchical structure is necessary for long-horizon tasks.\n5. The paper provides extensive ablations on key hyperparameters (α₁, α₂), bootstrap targets vs. completion loss, and demonstrates the learned completion field can support both single-step and multi-step rollouts (Table 9).\n6. Extensive comparisons with FQL in Appendix A.7", "weaknesses": "1. While the paper compares against FQL [Park et al., 2025], there are other recent few-step policy methods [3-4] that should be discussed and compared\n2. The paper doesn't provide clear guidance on when SSCP is expected to outperform alternatives\n3. While Table 9 shows multi-step rollout results, the analysis is limited\n4. In Figure 7, the performance change could be quite large depending on hyperparameters chosen", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)