Scaling Offline RL via Efficient and Expressive Shortcut Models
Authors: Nicolas Espinosa-Dice
Yiyi Zhang
Yiding Chen
Bradley Guo
Owen Oertell
Gokul Swamy
Kianté Brantley
Wen Sun
Venue: NeurIPS 2025 poster ()
Keywords: reinforcement learning
shortcut models
Primary area: reinforcement_learning
TL;DR: We introduce a novel offline RL algorithm that leverages shortcut models to scale both training and inference.

Abstract: Diffusion and flow models have emerged as powerful generative approaches capable of modeling diverse and multimodal behavior. However, applying these models to offline RL remains challenging due to the iterative nature of their noise sampling processes, making policy optimization difficult. In this paper, we introduce Scalable Offline Reinforcement Learning (SORL), a new offline RL algorithm that leverages shortcut models – a novel class of generative models – to scale both training and inference. SORL's policy can capture complex data distributions and can be trained simply and efficiently in a one-stage training procedure. At test time, SORL supports both sequential and parallel inference scaling by using the learned Q-function as a verifier. We demonstrate that SORL achieves strong performance across a range of offline RL tasks and exhibits positive scaling behavior with increased test-time compute.

Scaling Offline RL via
Efficient and Expressive Shortcut Models
Nicolas Espinosa-Dice Cornell University ne2290cornel1.cm edu
Yiyi Zhang Cornell University y223648cornel1.c edu
Yiding Chen Cornell University yc27730cornel1.ce edu
Bradley Guo Cornell University bzg48cornel1.com edu
Owen Oertell Cornell University ojo2@corne11 edu edu
Gokul Swamy Carnegie Mellon University
gawamy8andrev cmu edu
Kianté Brantley Harvard University
Wen Sun
Cornell University va455@corne1l.cm edu
kdbrantley@harvanr edu
Abstract
Diffusion and flow models have emerged as powerful generative approaches capa- ble of modeling diverse and multimodal behavior. However, applying these mod- els to offline reinforcement learning (RL) remains challenging due the iterative nature of their noise sampling processes, making policy optimization difficult. In this paper, we introduce Scalable Offline Reinforcement Learning (SORL), a new offline RL algorithm that leverages shortcut models-a a novel class of generative models-to scale both training and inference. SORL's ' policy can capture complex data distributions and can be trained simply and efficiently in one-stage training procedure. At test time, SORL introduces both sequential and parallel inference scaling by using the learned Q-function achieves strong performance across a range of offline RL tasks and exhibits pos- itive scaling behavior with increased test-time compute. We release the code nico- aico-espinosadico github... 10/ pcttsccs/sorll
verifier. We demonstrate that SORL
Introduction
Offline reinforcement learning (RL) |EEsst et all RNDS Lange et al. 2UF Levne et all 2020 is a paradigm for using fixed datasets interactions to train agents without online exploration. In this paper, tackle the challenge of scaling offline RL, for which there are two core components: training and inference.
In order to scale training, offline RL algorithms must be capable of handling larger, more diverse multi-modal datasets [ONeill et al 20244, Rataloy et all 2024 Park all 2 Hussing all
2023, Güriler et all 2023, Bu et all 2025]. While standard Gaussian based policy classes cannot model multi-modal data distributions, flow matching [Lipman et all, 2022 it al. 2022, Albergo all 2023] and diffusion models (Sohi-Dickstem et all 2015, Ho et all 202 Song et al., 202 have emerged as powerful, highly expressive model classes capable of modeling complex data dis- tributions. However, while generative models like diffusion and flow matching can model diverse offline data, applying them to offline RL challenging due to their iterative noise sampling pro- cess, which makes policy optimization difficult, often requiring backpropagation through time or distillation a larger model. Ultimately, we desire a training procedure that can efficiently train an expressive model class.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

The second core component in scaling offline RL is inference. During inference, we desire both ef- ficiency and precision: the agent must be able to make decisions rapidly (e.g. autonomous vehicles) but also take precise actions (e.g. surgical robots). Inspired by the recent work in test-time scaling of large language models ell 2023, DeepSeek-Al, 2025, Gui et all 2024, Browneetal 2024, Muennighoff et all 202 Madaan et all 20221 Qu et al, 2024. Qin et all 2025] we investigate how test-time scaling can be applied in offline RL with generative models. That is, we desire an approach the slow, many-step generation process of diffusion models-but can also leverage additional test-time compute when available.
that can perform inference efficiintly-avoieeee
Unfortunately. recent work in offline RL fails to achieve both of the desiderata necessary to scale offline RL. Distillation- based approaches offline RL with generative models [Ding and Jun, 802K) Chen ct all 2023 2024, Park et all. 2024] avoid extensive backpropagation through time during training. However, they may require more complex, two-stage training procedures (e.g. propagating through teacher/student networks), over which error compounds. They also have limited scaling of inference steps, thus losing expressivity when compared multi-step generative models [[rans approaches that leverage backpropagation through time [Wang et all 2022 He eta 2023. Zhang et al, 2022 eLa R022I] may learn policies with greater expressivity. However, diffusion models have slow inference, requiring a larger number of steps to generate high quality outputs [Frans et all 2024]].
aLal 2024]] Alternatively, diffusion-
At high-level, offline RL algorithms that employ generative models struggle with the following tradcoff. In order achieve training efficiency, we want to avoid performing many steps of the iterative noise sampling process during policy optimization, which often requires backpropagating many steps through time. However, modeling complex distributions in the offline data, such as di- verse or multi-modal data, may require larger number of discretization steps to allow for maximum expressivity. Finally, at inference-time, we desire the ability to both generate actions efficiently via few-step sampling procedure (e.g. in robot locomotion settings) and leverage additional test-time compute when available (e.g. in robot manipulation settings).
In this paper, we tackle the question of how to achieve efficient training. while maintaining expres- sivity, and scale with greater inference-time compute. We introduce Scalable Offfine Reinforcement Learning (SORL): a simple, efficient one-stage training procedure for expressive policies. Our key insight to leverage shortcut models [Frans all 20224), a novel class of generative models, in order to incorporate self-consistency into the training process, thus allowing the policy to generate high quality samples under any inference budget. SORL's 's self consistency property allows us to vary the number of denoising steps used for the three core components in offline RL: policy optimization (i.e. the number of backpropagation through time steps), regularization to offline data (i.e. the total number of discretization steps), and inference (i.e. the number of inference steps). We can use fewer steps for policy optimization, thus making training efficient, while performing inference under vary- ing inference budgets, depending on the desired inference-tine compute budget. We incorporate shortcut models into regularized actor-critic algorithm and make the following contributions:
1. We introduce SORL, an efficient, one-stage training procedure for expressive policies that can perform inference under any compute budget. including one-step inference. Despite its efficiency, SORL'S policy can capture complex, multi-modal data distributions.
2. Through a novel theoretical analysis of shorteut models, we prove that SORL'S training objective regularizes SORL's policy to the behavior of the offline data.
3. Empirically, SORL achieves the best performance against 10 baselines on range of diverse tasks in offline RL.
4. At test-time, SORL can scale with greater inference-time compute by increasing the number of inference steps (i.e. sequential scaling) and performing best-of-N sam- pling (i.e. parallel scaling). We show empirically that SORL can make up for smaller training-time compute budget with a greater inference-time compute budget. SORL can also generalize to more inference steps than the number of steps optimized during training.
2 Background
Markoy Decision Process. We consider an infinite-horizon Markov Decision Process (MDP) |iee WHLHHT [ M A, P. R.H.). X and A are the state space and action space, respectively.

---

:: P A + A(2) is the transition function, R X' x A - [0,1] is the reward function, and
Y is the discount factor. μ E A(X) is the starting state distribution. Let II = {5 : X + (AA) be the class of stationary policies. We define the state-action value function Q(x, a) X x A - IR. stationarypoolleeee
as Q" a) = E 1Σ R(z,, a:) 04 = z; O1 = a]. We define the state visitation distribution generated by policy π to be di == (1 -1--70 -) [ Pr, (x 20)] In the offline RL setting, we assume access to fixed dataset 'D = {(T,, r, Je....... such that the data was generated (x., 2²) ~ 2 by some behavior policy лB-
Flow Matching, We define flow matching [Lipman et al, 2022, Liu al 2022 as follows. Let p ((R)) be the target data distribution d-dimensional Euclidean space. Let & -V Z) and &1 ~ p" be two independent random variables. We define & to be the incerpiiterooltooonn between the Eg and Z1:
:= + (1 (1- 0 0
{ 1e is fully determined by the start point Eo and end point 21. We use (pehrejo.1) to denote the sequence of marginal distributions of & ,,'s, where μo) = N0 1) and pi =p"
The drift function ve(-) Rdd - Rd is defined to be the solution to the following least square regression:
min S. B.g-A( [|a 20- - f(tz + (1- t), t] dt. 1
(2)
We use fz to denote the solution to the optimization problem in Equation and define:
2 = fz (2.
E E [0. 1).
The drift function v-(-) induces an ordinary differential equation (ODE)
dt 21 02 (z)).
(3)
An appealing property of Equation is: if the initial 20 is drawn from Gaussian distribution (o, T). then the marginal distribution of ze, the solution the ODE, is exactly Pt, the marginal distribution of the linear-interpolation process [Lm all, 2022]. In practice, one can learn a drift function (-) by directly optimizing Equation E starting from z AO0 I), and solve the ODE: dæ. 6 (z))dl. The solution 21 is an approximate sample from the target distribution p".
Shorteut Models. Flow matching typically rely small, incremental steps, which can be compu-
lationally expensive at inference time. Shortcut models improve inference efficiency by learning to take larger steps along the flow trajectory [Erans all, MI2e]] The key insight of Frans etans-etaal all [202]]] is to condition the model not only on the timestep I, as in standard flow matching, but also on a step size h. The shortcut model s(z,, h) predicts the normalized direction from 2 towards the correct next point 21 such that
2t + s(2,, t, hi) 24th-
The objective is to learn a shortcut model sp sp(z,, t, h) for all combinations of 2,, t, h. The loss function is given by
- (0) = E =g =1 (.B.B(th))mmc...)
[|| (002,, - (z 20) 11? + 80 s0 1, 2h) Starget 1"]. (5)
Flow Matching
Bolf-Consistency
where Starget 59(2,, t, h)/2 so( t, h)22 and 2/44 2; + sg(2,, t, h)h.. h is sampled uni- formly from the reciprocals of powers of between 1 and M, the maximum number of discretization
logg M ) x Unif(O. 1). steps, and is sampled uniformly between 0 and 1, so p(h, t) Unif( (aaf((2-))) [[(2--]]___)) ]k=
The loss's 's first component is standard flow matching which ensures that when h = 1 1/M, the smallest step size, the model recovers the target direction 21 2p- The second component, self-consistency, encourages the model to produce consistent behavior across step sizes. Intuitively, it ensures that one large jump of size 2h is equivalent to two combined jumps of size h. Thus, the model learns to take larger jumps with more efficient, fewer-step inference procedures [rrans ala 2024j.
Note that the interval [0. 1] is discretized into M steps.
3

---

Algorithm 1: Scalable Offline Reinforcement Learning (SORL)
Data: Offline dataset D while not converged do
Sample (2r, a x', r) ~, a ~ 1), (h2 ~ p(h,t) at (1 - t)a ta!
Parallelize batch Noise action
Q Update
for all batch elements do U .....
MBIT) a TT(. xr
Choose number of inference stepa Sample action
Self- Self-Consistency for all batch elements do
81 f- 8q (q ((a',,,,. a) a+th t a' sch
First small step
Follow ODE
S1th + sa +h + h, h x)
# Second small step
Starget + (t 9 x)/2
Self target
Flow Matching for all batch elements do
+ 1/Mdis
Use smallest step size Flow-matching target
Starget < a a
Vollso(a', 2h x)
Starget 1| VoQ0(x,a")
Update actor
> Critic Update for all batch elements do MBBT) a ~ no(: (1(1mmmm))
Uniff1, In.f(
#
Choose number of inference steps
# Sample action
V (QolI, -r - -
# Update critic
3 Scaling Offline Reinforcement Learning
At a high-level, offline RL aims to optimize a policy subject to some regularization constraint, which can formulate
argmax
(
aR(T,
zell
Policy Optinisation
Regularization
where (T) the expected return over offline dataset D, JB the offline data generating policy, and R(T, TB) is regularization term (e.g. a divergence measure between π and TB).
The regularization constraint be implemented through behavioral cloning (BC)-style loss [WT et all 2019), such as in the behavior-regularized actor-critic formulation [Wi Wileaal 2019 Fujimoto
and Gui 2021. Tarasov et all, 202K5 Park et all 2025
argmax Ex,aND (a Q0(x, at)
+ a log (a(a x))
xell
Loaa
BC Loss
where the Q6 function is trained via minimizing the Bellman error. In order to incorporate generative models, the BC loss can be replaced with score matching or flow matching [Wang et all, 2022 Pars et atall 2024a). However, the core challenge of performing offline RL with generative models is the policy optimization component, due to the iterative nature of the noise sampling process.
3.1 Scalable Offline Reinforcement Learning (SORL)
Motivation. In order to scale offline RL, we desire an efficiently trained expressive policy that scalable under any inference budget. Our key insight is that, by incorporating self-consistency into training, SORL can vary the number of denoising steps used during policy optimization (i.c. the

---

number of backpropagation through time steps), regularization to offline data (i.e. the total number of discretization steps), and inference (i.e. the number of inference steps) thus enabling both efficient training of highly expressive policy classes and inference-time scaling. Unlike two-stage methods, which take existing diffusion models and later distill one-step capabilities into them, SORL is unified model in which varying-step inference is learned by a single network in one training run. P
Policy Class and Inference Procedure. We model our policy by the shortcut function sp. and sample actions via the Euler method with the shorteut function 8p- The full procedure is presented in Algorithm Q Slightly overloading notation, we condition on the number of inference steps m when generating actions from the policy (i.e. a ~ Tp(- 2;, m)). At test-time, we sample actions using M inference steps. Note that, since the inference process corresponds to solving a deterministic ODE, which is approximated using the Euler method, we can perform backpropagation through time on actions sampled via Algorithm @ during training.
Actor Loss. We present SORL's full training procedure in Algorithm II There are three compo- nents to SORL's training: the Q update, the regularization to offline data, and the self-consistency:
(0) (0)
(fM
(s)
Lose
Flow Matching Lose
Self Self-Connietency Loes
(1) Q Update. For the Q update, we first sample actions via the inference procedure in Algorithm 21 using maximum of M steps (i.e. backpropagating MBTT steps through time). The Q loss is computed with respect to this action:
(o) = B.-DBa - 1-Q
Since MBTT typically is small-we experiment with M = 1, 4, we can backpropagate the Q loss efficiently, without needing importance weighting or classifier gradients. In other words, even though sampling a" ~ Ty may involve multi-step generations, VaLqi. is still computable. Additionally, rather than use a fixed number of steps for sampling actions, we sample the number of steps uniformly from the set of powers of 2 between 1 and M/BT
(2) Offfine Data Regularization. We add a BC-style loss that serves as the regularization to offline data, which we implement via flow matching on the offline data, using M discretization steps. a represents a fully noised action (i.e. noise sampled from a Gaussian), and a! represents real action (i.e. actions sampled from the offline data D). The flow matching loss is thus:
(0) aaalaw (opp(h.))
[|| so(a", L, 1/1 x:)
(a' Velocity Target
(10)
Velocity Prediction
(3) Self-Come coffssooiiitency. We add a self-consistency loss to ensure that bigger jumps (e.g. the shorteut of an m-step procedure) are consistent with smaller jumps (c.g. the shortcut of a 2m-step procedure):
(S) = E
x.alaD 20 (
so r, 2h x))
Starget
(11)
Double-Step
Singla-Staps
where
Starget = ,0 1,/12 s sa 4hh + h+hh h xx//2
(12)
1st Single-Step
2nd Single-Step
and
+
h
sa h)
(13)
2nd Step's Actiion
Step's Action
Step Size Normalizad Direction
NNote that directly regularizing to the empirical offline data, as done via flow matching in SORL, is preferable regularizing with respect to learned behavior cloning (BC) policy, done by FQL, when the underlying distribution class unknown. This is because, in the nonparametrie setting, the empirical distribution is a statistically consistent estimator, and is minimax rate-optimal under common metrics like total variation and 2- Wasserstein distance.
In other words, sample Unif(2*) (log2 MI
5

---

Algorithm 2: SORL Action Sampling via Forward Euler Method
Input: State x, number of inference steps Output: Action a
(0, h 4---0 1/m
t
for n E (....., m 1 do
a a a
se (a, t x)
t +
return a
Critic Loss.
We train the critic via standard Bellman error minimization, such that
Lo(Ø) = (Qo(I, - 10 parget (x', o.
(14)
where ar ~ TT(- x() and Q
gaargtt is the target network [Mnili 2013, Park ala 2025).
3.2 SORL Inference-Time Scaling
The Benefit of Seff-Consistency. The loss function L. (0) includes both the flow matching error and the self-consistency error. Intuitively, training on the flow matching error alone yields a flow model, from which
sample actions by numerically solving the flow ODE in Equation B with the Euler method. Using more discretization steps M diss in the Euler method implies smaller step size (i.e. Amin 1/M is small), leading to a smaller discretization error. However, the Euler method requires (hmmin) number of iterations, so smaller step size Amin is more computationally expensive. By incorporating the self ceniisonsit error into the training objective, the shortcut model approximates two iteration steps of the Euler method into single step. By iteratively approximating with larger step sizes, the shortcut model achieves discretization error comparable to that of the Euler method with small step size, while only requiring constant computational cost.
Sequential Scaling, By training shortcut model with self-consistency. SORL'S 's policy can perform inference under varying inference budgets. In other words, SORL can sample actions Algorithm with varying number of inference steps Mfnf including one step. In order to implement sequential scaling, we simply run the sampling procedure in Algorithm for greater number of steps (i.c. larger mI up to the number of discretization steps M disc used during training.
Parallel Scaling, We also desire an approach to inference-tine -irennce--mmeeeee scaling that is independent of the number inference steps. We incorporate best-of-NN sampling [Fupmooo all 2019, Ghasemipour all 2021, Gui et all, R024, Nakamoto et all 2022), Lightman et al]. 2023, Brown et all, 2024]] following the simple procedure: sample actions independently and use verifier select the best sample. In SORL, we use the trained Q function as the verifier. We implement best-of-N sampling as follows: given state v, sample a1 U22 aN independently from the policy nh [x) and select the action with the largest Q value, such that
argmax (a
Q a)
(15)
Theoretical Analysis: Regularization To Behavior Policy
Offline RL aims to learn a policy that does not deviate too far from the offline data, order to avoid test-time distribution shift [Levine el all 2020]. In this section, we theoretically examine the question:
Will SORL learn a policy that is regularized the behavior of the offline data?
Through a novel analysis of shortcut models, we prove that the is yes.

---

Table 1:
SORL's Overall Performance. SORL achieves the best performance on 5 of the 8 envi- ronments evaluated, for total of 40 unique tasks. The performance averaged over seeds, with standard deviations reported. The baseline results are reported from Park all [2025]'s extensive tuning and evaluation of baselines on OGBench tasks. We present the full results Appendix
Gauzeean
Ditrunton
Flow
Shortout
Took Capegory
I2l.
ResRaC
IDQL
SRPU
CMC
FANAC
FURAC
IPQL.
P12.
SURL
0cBeach astaszo- wate--aaargeggggoooooooo tasks)
11
13
1.
21
11 24
33
6
60
25
79 13
02bench antaase--tt gtant-ooe oott--ongietan tacke)
0:23
1.
26
0:20
0.29
0.20
0.24
24
0cBench busasildraze aeweoiddaee--eeeee aedeeeeaaeeaaaeeeeea...
(S tasks)
2.s
ad a2
22 44
1-4
1-1
53
19 -1
38
04)
ass
58 =s
64
0c3ench hammsidraze- largl--aigggeeenn
(5 tasks)
1.a
2.
2
1:0
0.
0.40
0.00
2
11
4:12
0clecch untacceer pataaaaaannnneeaii
(6 basks)
1:3
8:07
0.
12
1mm
2=
12
16
33
60 =0
69
0KBench too--enngl-sinllltee (5 tacke)
5.11
63 23
91 12
95
80 05
85
81
70
79
96
97
UGllnnc cabe-double-Be eea--oobbeesingletaee (5 tanke)
3:l
7:23
12 21
16
2:21
0.22
5.29
15
14
20
26
06Beach acos-siagletask (5 tasks)
0.41
28 k1
41
46.42
20 61
40
30
45
30
56
57
In the training objective in Equation N. we include the BC-style flow matching loss LFM and self- consistency loss Lsc- The former ensures closeness to the offline data, while the latter allows for fast action generation. In this section, we demonstrate that this training objective can be interpreted as an instantiation of the constrained policy optimization Equation Unlike the kly-style regu- larization in Equation the CFM Esc term in the objective function is Wasserstein behavioral regularization, similar to that of et al [212]] We show that under proper conditions, the short- cut model will generate distribution close the target in 2-Wasserstein distance (W2) in Euclidean norm. This implies that as long as we minimize LrM + Lsc. we ensure the policies induced by the shorteut model will stay close to the behavior policy in W2 distance.
Following the setup of the shortcut model in Section D we first assume the shortcut model s(..-.) is trained properly (i.e. flow-matching and the self-consistency losses are minimized well). In other words, the shorteut model for the smallest step size s(','s 37 close to the ground truth drift function (c(-). and s(,,', (2)) is consistent with h:) on the evaluated time steps in inference.
Assumption 1 (Small Flow Matching and Self-Consistency Losses). There exist EFm <sc > 0, s.t.
and
for all 0,
2 M M
M. we have Excop []] t, +)) we ve(2))
S EPAM
for all h MF
2 2a and L h, 2h, li we huve
M
37
Expap |11o(1,, h)/2 2 5( 1,11/22 2 s(2,, 14, S <
where 2/44n = 2 + 8(2t, t,), hih.
Theorem (Regularization To Behavior Policy). Suppose the shortcut model s(2, h) is
L-Lipschitz in for all t and ht the drift function vz(2) is L.-Lipschitz in z for all t.
sup, E Ezoopi 2 2 M M M
[101 I Mo and L/M < 1. If Assumption W holds, then for all h
(2( .p') S e
eLu
M (M, +
+
CFM
Esc logg M
(16)
Discretiiation Error
Flov Matching Error
Solf-Consistency Error
where (1h) is the distribution of samples generated by the shortcut model with step size h and p* is the data distribution.
Discretization Error.
In the upper bound, the term eLv (M, + 1) corresponds to the Euler dis- M
cretization error-a well-known quantity numerical ODE solvers which vanishes as the num- ber of discretization steps M + c0. This term captures the inherent error from approximating continuous flows with finite-step shortcut models.
'For a cleaner presentation, we consider the unconditional setting and show a uniform upper bound on the Wasserstein distance for all step size h. We defer h-dependent upper bound to Appendix C
7

---

Large taska)
humezoidnaze- -aagge taake)
Perforaance (1)
Perfornance (1)
91
91
89
2
Inference Steps (M
Inference Stepe (Ad
antsoccer- arena
tasks)
cubo-double tacks)
scene-play (5 tasks)
Perfozmance (T)
Perforannc (1)
Port Performance (1)
60
51
58
17
21
Inference Stepa (M
Inference Stapa (Afitt
Inforence Stops (Mil))
Figure I: SORL's Sequential Scaling, For fixed training budget, SORL generally improves perfor- mance with greater test-time compute. We fix training budget of discretization steps and back- propagation steps through time (Miise = BBTTT 8) and vary the inference budget via the number of inference steps M The performance is averaged over seeds for each task, with 5 tasks per environment, and standard deviations reported.
Flow Matching and Self-Consistence Errors. The terms éтm and esc logz M account for the training approximation errors: 6pm measures the deviation between the shortcut models predicted velocity and the true drift under small step sizes, while esc log2 M captures the cumulative consis- tency етrоr over varying step sizes. Notably, when both LFM and Lsc are minimized effectively during training, these errors become negligible. Ignoring the self-consistency loss, our result is com- parable to the guarantees for flow models Roy et all [2022]]. However, our analysis incorporates self-consistency and validates it across all discretization levels, which is novel contribution.
Regularization To Behavior Policy. Theorem shows that when trained properly, the shortcut model generates a distribution close to the target distribution in 2-Wasserstein distance for all step sizes h. Thus, the BC-style flow matching loss LFM and self-consistency loss Lsc in the training ob- jective (Equation N enforce regularization to behavior policy in Wasserstein distance, Consequently, SORL not only learns a performant policy, but also guarantees that the learned policy remains close to the offline data distribution across varying step sizes.
Experiments
In this section, we evaluate the overall performance of SORL against 10 baselines across 40 tasks. We then investigate SORL's 's sequential scaling and parallel scaling trends. Furthermore, present additional results in Appendix D and ablation studies in Appendix D
5.1 Experimental Setup
Environments and Tasks. We evaluate SORL on locomotion and manipulation robotics tasks in the OGBench task suite [Park all M122a]] The experimental setup in this section follows the setup suggested by PPar all [2024x. 2025]. We document the complete implementation details in Appendix E Following Park et all [2025], we use OGBench's reward-based s singletask variants for all experiments, which are best suited for reward-maximizing RL.
Baselines.
We evaluate against three Gaussian Class offline RL algorithms (BC Pooerreaul 1988], IQL [Kostrikov et all, 2021]], ReBRAC [Tarasov et all, 2023al), three diffusion- algorithms (IDQL Estruch et all, 2023), SRPO
Chen et all, 2023],, CAC [Ding and Jin. 2023)), and four flow- based algorithms (FAWAC [Nair et all 2020, Park et all 2025), FBRAC [Zhang et al., 2025, Park et all 2025], IFQL [[ang et all, 2022 Park all 2025), FQL Park all 2025)). For the baselines we compare against in this paper, we report results from Park at [2025],, who performed an extensive tuning and evaluation of the aforementioned baselines on OGBench tasks. We provide a thorough discussion of the basclines in Appendix E

---

acene-play (5 taske)
Performance (1)
53
54
59
MErI
32
Ar
Boll
13
SORC"
2
Inference Stepa (A
Figure 2: SORL's Parallel Scaling, SORL generalizes to new inference steps at test-time, beyond what optimized through backpropagation during training. For each fixed training budget (i.e. fixed number of discretization steps Maise and backpropagation through time steps MBTT), we evaluate with varying inference steps Miní MBTT denotes the maximum number of steps used for backprop- agation through time in the Q update. The / hatch denotes best-of-N sampling, with N = 8, where the number of inference steps is greater than the number of backpropagation steps through time (i.e. Mini > M )BTT). SORL* denotes the best performance achieved by SORL in Table MI Results are averaged over 8 seeds for each of the tasks.
Evaluation. Following Park [2025]] we ensure fair comparison by using the same network size, number of gradient steps, and discount factor for all algorithms. Furthermore, we hyperparam- eter tune SORL with similar training budget to the baselines: we only tune one of SORL's training parameters on the default task in each environment. We average over 8 seeds per task and report standard deviations in tables. We bold values at 95% of the best performance in tables. For SORL, we use 8 discretization steps during training (2 fewer than the baselines), since SORL requires that discretization steps be powers of We detail all other parameters in Appendix El
5.2 Experimental Results
Q: Does SORL perform well on different environments?
Yes, SORL achieves the best performance on the majority of the diverse set of 40 tasks.
We present SORL's overall performance across range of environments in Table II Notably, SORL achieves the best performance on out of environments, including substantial improvements over the baselines on antmaze-large and antsoccer-arena The results suggest that while distillation approaches like FQL [Park et all [025] can achieve high performance, some tasks require greater expressiveness precision than can be achieved from onc-step policies.
Q: For fixed training budget, can SORL improve performance purely by test-time scaling?
Yes, SORL's 's performance is improved by increasing the number of inference steps at test-time.
We investigate SORL's sequential scaling by plotting the results of varying inference steps Minf given same, fixed training budget (i.e. holding the discretization steps Maise and the steps backpropa- gated through time MBIT constant), for Mil < MBBTI We plot the results in Figure IIl The results show noticeable improvement in performance as the number of inference steps increases, suggesting that SORL scales positively with greater test-time compute.
Q: Can SORL make up for less training- time compute with greater test-time compute?
Yes, given less training compute, SORL can match optimal performance with greater test-time com- pute.
We reduce training budget by reducing the steps of backpropagation through time from MBTT
8 for Table M and Figure M to Λ 1, 2, 4 for Figure Q However, through combination of inference sequential and parallel scaling, we recover the performance of the optimal policy from the best training budget (SORL*).

---

Q: At test-time, can SORL generalize to inference steps beyond what it was trained on?
Yes, through sequential and parallel scaling, SORL can use a greater number of inference steps at test-time than the number of backpropagation steps used during training.
Adise disc and backpropagation steps through time M MBTT), ) we evaluate on an increasing number inference steps Mi coupled with best-of-N sampling. The bar colors in Figure Q denote different training compute, dictated by the number of backpropagation through time steps MBTI, We apply best-of-N sampling on a greater number of inference steps than were used for backpropagation through time (i.e. Mina M thus testing SORL's ability to generalize to inference steps beyond what it was backpropagated on. From Figure A SORL can more inference steps than the number of backpropagation steps used during training (i.e. the number of steps used for backpropagation through time MBTT), up to a performance saturation point (approximately MBIT_ = 4).
Given fixed training budget (i.c. fixed number of discretization steps M
Although Figure E shows that best-of-N sampling can improve performance, the gain is nor theoret- ically guaranteed. Our verifier is the learned value estimator, not a ground-truth reward, so there is no statistical benefit to learning a verifier [Swamy et all RO2E] G Empirically, however, best-of-N effectively functions as an additional, inference-time policy-extraction step, searching over nearby actions and selecting the one with the highest Pas-similat to Park et all [20246]]s 's work. Thus, the post-hoe filtering may still be empirically beneficial in regaining performance lost to imperfect optimization.
Related Work
The goal of offline reinforcement learning (RL) [Levine et all 2020| is to learn policy solely through previously collected data, without further interaction with the environment. The core idea of much of the prior work is to maximize returns while minimizing discrepancy measure between the state-action distribution of the dataset and that of the learned policy. This goal has been pur- sued through various strategies: behavioral regularization [Nair et all 2020. Fujimoto and GU, 40741 Tarasov all 202 conservative value estimation [Kumar el all 2020], in-distribution maximiza- tion [Kostriko aL P11ZAL Xn all 20223, Garg et all 2023).. out-of-distribution detection [Yr et all 2020, Kidambi el all 2020, An et all. 20211 Nikulm el all
2023],, and representing policy [02L 20221 Park all [2025]]
models using generative modeling Chen all
[2021] lanner et all
Motivated by the recent success of iterative generative modeling techniques, such as denoising diffu-
sion DickseDe et all, 201S, Hll 2020, [Sog et al,, 2021I] and flow matching [Lipman et all,
2024, Esser et all 2024]. the use of generative models as a policy class for imitation learning and reinforcement learning has shown promise due to its expressiveness for multimodal action distribu- tions [Wang et al,, 2 Ren et all, 20245, ell 2024, Black et all 20244). However, its iterative noise sampling process is computationally inefficient [Ding and Jin 2025)) Some methods utilize two-stage approach: first training an iterative sampling model before then distilling it Frans et all [20221] eol [2020], Meng et all [2023], but this may lead to performance degradation and intro- duce complexity from the distillation models, along with error compounding across the two-stage procedure. Consistency models [Song and Dhariwal, 2023 are another type of unified model, but they rely extensive bootstrapping, such requiring specific learning schedule during training Frans et all 20244], making them difficult to train.
Discussion
We introduce SORL: simple, efficient one-stage training procedure for expressive policies in offline RI. SORL's key property is self-consistency, which enables expressive inference under any inference budget, including one-step. Theoretically, we prove that SORL regularizes to the behavior policy, us- ing a novel analysis of shortcut models. Empirically, SORL demonstrates the best performance across range of diverse tasks. Additionally, SORL be scaled at test-time, empirically demonstrating how greater compute at inference-tine can further improve performance. An avenue for future work to investigate how SORL can incorporate adaptive test-time scaling [Pan all 2025, Ma et all 2025], such as selecting the number of inference steps based on the gradient of the f-unctioo..
'I other words. if the actor were already acting greedily with respect to Qe (i.e. the case of perfect optimization), ranking additional samples by Qo cannot raise the expected value.
10

---

Acknowledgments
NED is supported by the NSF Graduate Research Fellowship under Grant No. DGE-2139899, GKS supported by an STTR grant. KB acknowledge: This work has been made possible part by gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence. WS acknowledges funding from NSF IIS-2530143, NSF IIS- 2154711, NSF CAREER 2339395, DARPA LANCER: LeArning Network CybERagents, Infosys Cornell Collaboration, and Sloan Research Fellowship.
References
Suzan Ece Ada, Erhan Oztop, and Emre Ugur. Diffusion policies for out-of-distribution generaliza- tion in offline reinforcement learning. IEEE Robotics and Automation Letters, 964):3116-1133, 2024.
Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Fijnden. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:23333.08797, 2023.
Gaon An. Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- based offline re- inforcement learning with diversified q-ensemble. Advances in neural information processing systems, 0177336-4447 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.064450 2016.
Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference Machine Learning, pages 1577-1594. PMLR, 2023.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian lchter, et al. To: A aision-language-actie flow model for general robot control. arXiv preprint 241.v::2410.211600 2024.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling, 2024. URL https: htt::///axxvvvvve wwwp:iii..rga/a///////e// 217877
Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng. Shenyuan Gao, Xindong He, Xu Huang, Shu Jiang, al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint arXiv:2503.06660) 2025.
Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv: 14548, 2022.
Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu. Score regularized policy optimiza- tion through diffusion behavior. arXiv preprint arXiv:2310.07297, 2023.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 55044-9999, 2021.
Tianyu Chen, Zhendong Wang, and Mingyuan Zhou. Diffusion policies creating a trust region for offline reinforcement learning. arXiv preprint 2rXiv:2405./9690 2024.
DeepSeek-AL. Deepscek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning, 2025. URL
12948.
Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning, arXiv preprint arXiv:2309.169944 2023.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 2005,

---

Patrick Esser, Sumith Kulal, Andreas Blatimann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning. 2024.
Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models. arXiv preprint arXivv2410. 2024.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 11220132-20143 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. International conference on machine learning, pages 2052-2062. PMLR, 2019.
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. arXiv preprint XXXx:2301.023225 2023.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq; Expected- max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pages (682-3691. PMLR, 2021.
Lin Gui, Cristina Gârbacea, and Victor Veitch. Bonbon alignment for large language models and the sweetness of best-of-n sampling. arXiv preprint arXiv:2406.0055 2024.
Nico Gürtler, Sebastian Blaes, Pavel Kolev, Felix Widmaier, Manuel Wüthrich, Stefan Bauer, Bem- hard Schölkopf, and Georg Martius. Benchmarking offline reinforcement learning on real-robot hardware. arXiv preprint arXiv:2307.15690 2023.
Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
Longxiang He, Li Shen, Linrui Zhang, Junbo Tan, and Xueqian Wang. Diffeps: Diffusion model based constrained policy search for offline reinforcement learning. arXiv preprint arXiv:2310.05333, 2023.
Dan Hendryeks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv: 160.004506. 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 68511, 2020.
Marcel Hussing, Jorge A Mendez, Anisha Singrodia, Cassandra Kent, and Eric Eaton. Robotic manipulation datasets for offline compositional reinforcement learning. arXiv preprint vrXiv:2307.07091, 2023.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273-1286, 2021.
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint trXXiv2205.0999)) 2022.
Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for 20081
offline reinforcement learning. 2023. URL
Rahul Kidambi, Aravind Rajeswaran. Pranceth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 21810-21823, 2020.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: arXiv:1412.00 1412.6982. 2014.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine Offline reinforcement learning with implicit q- learning, arXiv preprint arXiv:2110.06169, 2021.
12

---

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learming for offline reinforcement learning. Advances in neural information processing systems, 33:1179-119 2020.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney. Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, and Aleksan- dra Faust. Training language models to self-correct via reinforcement learning, 2024. URL https: haop::///axiv... wwt.s/.orggbbs//04303 12997777
Sascha Lange. Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforce- ment learning: State-of-the-ary pages 45 73. Springer, 2012.
Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Fung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, pages 6120-6130. PMLR, 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprini arXiv:2005.01643 2020.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint 2rXiv:2210.02747, 2022.
Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky TQ Chen. David Lopez-Paz, Heli Ben-Hamel and Itai Gat. Flow matching guide and code. arXiv preprint arXiv:2412.0626 2024.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0300 2022.
Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, and Saining Xie. Inference-time scaling for diffusion models beyond scaling denoising steps, 2025. URL https:://arrtt org/ abss/550..09//
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao. Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback, 2023. URL /wR.../Ep..///rtt..ooo//aaaaa
httpgg/ab//23000 17651.
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14297-14306, 2023.
V Mnih. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 1xXi::1312..602. 2013.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025. URL
http///awwiv..rr///////50.. 19393
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint arXiv:2006.09339, 2020.
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine- tuning. Advances in Neural Information Processing Systems, 06:62244-62269, 2023.
Mitsuhiko Nakamoto, Oier Mees, Aviral Kumar, and Sergey Levine. Steering your generalists: Improving robotic foundation models via value guidance. arXiv preprint arXiv:2410.13816, 2024.
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-exploration by random network distillation. International Conference on Machine Learning, pages 26228- 26244, PMLR, 2023.
13

---

OpenAl. Openai ol system card, 2024. URL /ttps://arxi...rgg
16720.
Abby ONeill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x-embodiment: Robotic learning datasets and models: Open x-embodiment collaboration O. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 0892-6903. IEEE, 2024.
Jiayi Pan, Xiuyu Li, Long Lian, Charlie Snell, Yifei Zhou, Adam Yala, Trevor Darrell, Kurt Keutzer, and Alane Suhr. Learning adaptive parallel reasoning with language models. arXiv preprint arXiv:2504. 15466, 2025.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. arXiv preprint 2111v:241..2009,, 2024a.
Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline rI? arXiv preprint arXiv:2406.093:: 2024b.
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. arXiv preprint arXiv:250..02538, 2025.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John
Wiley & Sons, 2014.
Tian Qin, David Alvarez-Melis, Samy Jelassi, and Eran Malach. To backtrack or not to backtrack:
When sequential search limits model reasoning. arXiv preprint arXiv:2504.07552, 2025.
Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive inccuusiieintrrospecciiiio Teaching lan- guage model agents how self-improve 2024. URL URL./Ettss//arrzvv.rr//////////0 18219.
Rafael Rafailov, Kyle Hatch, Anikait Singh, Laura Smith, Aviral Kumar, Ilya Kostrikov, Philippe Esstuch Victor Kolev, Philip Ball, Jiajun Wu, al. D5rl: Diverse datasets for data-driven deep reinforcement learning. arXiv preprint arXiv:2408.0844 2024.
Allen 7. Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. arXiv preprint arXiv:2409.0000 00588, 2024a.
Juntao Ren, Gokul Swamy, Zhiwei Steven Wu, J Andrew Bagnell, and Sanjiban Choudhury. Hybrid
inverse reinforcement learning. arXiv preprint arXiv:2402.08888 2024b.
Saptarshi Roy, Vansh Bansal, Purnamrita Sarkar, and Alessandro Rinaldo. 2-rectifications are enough for straight flows: A theoretical insight into wasserstein convergence. arXiv preprint arXiv:2410.149m 14949, 2024.
Harshit Sikchi, Qinqing Zheng, Amy Zhang, and Scott Niekum. Dual rl: Unification and new methods for reinforcement and imitation learning. arXiv preprint arXiv:2302.088560 2023.
Raghav Singhal, Zachary Horvitz, Ryan Techan, Mengye Ren, Zhou Yu, Kathleen McKeown, and Rajesh Ranganath. A general framework for inference-time scaling and steering of diffusion models, 2025. URL https::/arsiv.orrg/bb///0000
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265, pmlr, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint 2xXi:22000002002.. 2020.
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint arXiv:2310. 2023.
14

---

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In In- lernational Conference on Learning Representations, 2021. URL https://opent: //operrvee/eeeeeeew net7
POrum//AEEPIIGGTHRIIS
Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make rl efficient. arXiv preprint arXiv:2210.06718. 2022.
Gokul Swamy, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, and Andrew Bagnell. All roads lead to likelihood: The value of reinforcement learning in fine-tuning. arXiv preprint arXiv:2503.01065 2025.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the min- imalist approach offline reinforcement learning. Advances in Neural Information Processing Systems, 36:1 11992-116200 2023a.
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: Research-oriented deep offline reinforcement learning library. Advances in Neural Informa- fion Processing Systems, 36:30997-31020, 2023b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023. URL https://arxiae /rggabb//203....../oggggg// 11171.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou, Diffusion policies as an expressive policy class for offline reinforcement learning, arXiv preprint arXiv:2208.06199 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL
2200/ 11903.
Runzhe Wu, Yiding Chen, Gokul Swamy, Kianté Brantley, and Wen Sun. Diffusing states and matching scores: A new framework for imitation learning. arXiv preprint MXXv:2410.118855, 2024.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprini 19Xiv:/91..11366 2019.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xi- anyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. arXiv preprint ar3iv: /5810, 2023.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14142 2020.
Zishun Yu and Xinhua Zhang. Actor-critic aligmment for offline-to-online reinforcement learning. In International Conference on Machine Learning, pages (0452-40477) PMLR, 2023.
Ruoqi Zhang, Ziwei Luo, Jens Sjölund, Thomas Schön, and Per Mattsson. regropy- diffu- sion policy with q-ensembles for offline reinforcement learning. Advances in Neural Information Processing Systems, 37:98871-98899 2024.
Shiyuan Zhang, Weitong Zhang, and Quanquan Gu. Energy-weighted flow matching for offline reinforcement learning. arXiv preprint arXiv:2503. 04975, 2025.
15

---

NeurIPS Paper Checklist
The checklist designed to encourage best practices for responsible machine learning research, ad- dressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA]-
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (12 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to the checklist one of the factors in their evalu- ation. While "[Yes] generally preferable to "[No] is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No]" "[NA] in binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification elaborate, All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point the section(s) where related material for the question can be found.
grounds for rejection. While the questions are phrased
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Check- list",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The paper introduces SORL in offline RL, claims efficient and expressive pol- icy learning, and supports these with theoretical insights and empirical evaluations across 40 tasks in Section to 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
16

---

Answer: [Yes]
Justification: The paper discusses how SORL's best-of-N performance tied to the accu- racy of the learned Q-function in Section 5.4, and highlights the non-monotonic trends in some environments in Section 5. Furthermore, we present thorough discussion of limita- tions in Appendix [B]
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-ter system might not be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations their approach to ad- dress problems of privacy and faireess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The assumptions are stated in Assumption II The proof is in Appendix C
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided the core of the paper should comple- mented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
17

---

Answer: [Yes]
Justification: We thoroughly describe implementation and evaluation details Appendix
E Code will be released.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is either be a way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
model (e.g., a large language model), then there should
(d) We recognize that reproducibility may tricky some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code is included in the supplementary material. This paper uses the open- source dataset OGBench [Park all 21225)
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://mips.ce)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.co.. http:////wb////uuue////////aaaaa
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
18

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and testing details are discussed in Appendix E
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error metrics specified and reported for every empirical result. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative emror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computer resources are discussed in Appendix E.
19

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g.. preliminary failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The authors have reviewed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: To the best of the authors' knowledge, the research in this paper does not have directly negative societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (c.g., gated release of models, providing defenses addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
20

---

Answer: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
allow for controlled use of the model, for example by re-
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
This paper uses OGBench Park el cLa MIZER dataset.
which is open-sourced
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, saperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-pac re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
21

---

Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable). such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: For polish of writing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy /etees/rneerrpppcccc/oooferereee for what should or should not be described.
22

---

A Extended Related Work
Offline Reinforcement Learning. The goal of offline reinforcement learning (RL) [evinee et all 2020] is to learn policy solely through previously collected data, without further interaction with the environment. A considerable amount of prior work has been developed, with the core idea being to maximize returns while minimizing a discrepancy measure between the state-action distribution of the dataset and that of the learned policy. This goal has been pursued through various strategies: behavioral regularization [[aii et all, 2020, Fujimoto and Gu. 20221 aarasoy et 2023a. conser- vative value estimation [Kumar et all 2020], in-distribution maximization [Kostrikoy et all 20211
Xu el all 20033, Garg et al, 2023). out-of-distributii
detection [[Y ct all 20200 Kıdambi all
2020, An all 2021, Nukulm of all M023), dual formulations of RL [[ cf all 2021. Sikchi ef all
20 and representing policy models using generative modeling Chen Ch [M[2I]]. Janner et al
[20211 20221, Park et all [2025]] After training an offline RL policy, it can be further fine-tuned with additional online rollouts, which is referred to offline-to-online RL, for which several techniques
have been proposed [Lee et all 20211. Song et all 2022 Nakamoto et all 202E
St 2023, YIT
and Zhang, 2023,
Ren et all 20246].
Reinforcement Learning with Generative Models. Motivated by the recent success of iterative generative modeling techniques, such as denoising diffusion [Sohl-Dickstein et all 2015. Ho et all 2020, Song et all, 2021] and flow matching [Lipman et all 2024, Esser all 20241] the use of gener- ative models as policy class for imitation learning and reinforcement learning has shown promise due to its expressiveness for multimodal action distributions [Wang et all 2022 Ren 20247
eil 2002 Blakk etall 2024]. However, its iterative noise sampling process leads large time consumption and memory occupancy [[Ding and Jin, 2023). Some methods utilize a two-stage approach: first training an iterative sampling model before then distilling it Frans at [212e1]. Ho at [ Meng et al] [M023] but this may lead to performance degradation and introduce com- plexity from the distillation models, along with error compounding across the two-stage procedure. Consistency models [Song and Dhariwal, 20233 are another type of unified model, but they rely on extensive bootstrapping, such as requiring specific learning schedule during training [Frans et all 2024]] making them difficult to train.
Inference-Time Scaling. Recent advances in large language models (LLMs) OOeenAl 2024L DeepSeek-AT, 2025] have demonstrated the ability to increase performance at inference time with more compute through parallel scaling methods [Wang et all, 2023), Gui et all 2024, Brown et all 2024, Pan et all 2025)). sequential scaling methods that extend the depth of reasoning by increas- ing the chain of thought budget [Wer et all 20231 Muenmghoff et all RINE, Qin el all 2025). and self-correcting methods [Madaan Lll 2023 Ju et al. 2024), Kumar et all 2024
Generative models such as diffusion and flow models inherently support inference-time sequential scaling by varying the number of denoising steps (Sckt-Di all 2015, Lipman et al, 2022 and recent work extends their inference-time scaling capabilities through parallel scaling methods via reward-guided sampling [Ma et all, 2025, Singhal et all 2025]. However, prior reinforcement learning methods that use generative policies lose this inherent inference-time sequential scaling since they require the same number of denoising steps at both training and inference [Wang et all 2022. Kang all 2023), Zhang et al], 20244]. In contrast, SORL supports both sequential and parallel scaling at inference time, allowing for dynamic trade-off's between compute and performance, and improved action selection via the learned Q function.
Independent of generative models, prior work has proposed applying a form of best-of-N N sampling to actions from the behavior policy (i.e. behavioral candidates)
[ [Chen et all 2022, Fujimoto et all Ghasempour et al., 2021, Hansen-Estris et all 2023, Park et all 2024F, Nakamoto et all 20221] Park et all [2024]]] proposed two methods of test-time policy improvement, by using the gradient of the Q-function. One of Park et all [2024H]'s approaches relies on leveraging test-time states, which SORL's parallel scaling method does not require. The second approach proposed by
2019,
Park [20241]]]]] adjusts actions using the gradient of the learned Q-function, which is conceptually similar to our approach of best-of-N sampling with the Q-function verifier. However, their method requires an additional hyperparameter to tune the update magnitude in gradient space.
23

---

B Limitations
As noted in Section 5.2, the positive trend in parallel scaling may not occur across all environments. Our approach to parallel scaling uses the Q function as a verifier If the learned Q function is inaccurate or out of distribution, then additional optimization may not be beneficial [[Levine all 2020]. Additionally, while SORL is highly flexible, allowing for scaling of both training-time and inference-time compute budgets, SORL generally has a longer training runtime than FQL, one of the fastest flow-based baselines [Park all 2025], as evidenced by the runtime comparison (Figure B) in Appendix IX We believe that trading off greater training runtime for improved performance desirable in offline RL, since training does not require interaction with expert, environment, simulator. Finally, as noted in Park et al [2025], offline RL algorithms, including SORL, lack principled exploration strategy that may be necessary for attaining optimal performance in online RL.
24

---

C Proofs
We first present the upper bound on the Wasserstein distance with explicit dependency on h. Theorem 3 (Restatement of Theorem @ With Explicit Dependency on h)). Suppose the shortcut
model 8(2,1,1, 1,) I--Lipschitz in = for all and h, the drift function vx(2) is Ing-Lipschitz in for all t, sup, Exoppe [lealla I My and L/M
If Assumption D holds, then for all h
2 2 M. M
M..... M..... 1
S (1+ -1) exp { Im)
(
(ehe M
(M, +1 6PM + esc loga (Mh)
where 4hh) is the distribution of samples generated by the shortcut model with step size h and p* is the data distribution.
C.1 Proof of Theorem /
This theorem formalizes the key claim: minimizing the training objective keeps the learned pol- icy close the behavior policy. This closeness is measured using strong aeetic--Wsseeree distance-implyinn the model will not "drift" far from the offline data distribution during genera- tion. Under Lipschitz. assumptions and small www-mmtthhnn/seff-onnisssss the shortcut model generates samples whose distribution is close to the target data distribution in 2- D-Wasserstein distance (W2), uniformly over discretization step sizes
loss (Assumption M.,,
Proof Sketch. At high level, our goal is to show that, starting from the same ~ N(O,, I). running the sampling process of the shorteut model and rumning the ground truth flow ODE yield similar output measured by square error averaged over 2o (N0 7).
This argument constructs a coupling with marginal distributions 9oth) and p* and small transport cost. By definition, the Wasserstein distance between p(h) and p" is also small.
Our proof follows from three steps, described below.
(1) Small Step Error. We first show that the shortcut model provides good local approximation to the true dynamics when trained well at the smallest step size. In other words, under small flow- matching error and Lipschitz drift dynamies, the shortcut model has bounded error when running single inference step with the smallest stepsize:
Lemma (Single-Step Error with Minimum Step-Size). Let ho = M be the smallest stepsize. If
for all = 0, hu, 2ho hg, EzIMP [115(22; t, ho) 22(2)) I 2nm
Vz(x) is L-Lipschitz in both t and x; for all = 0, ho, 2ho...., ..... 1 ho, ExrP, [112((2)))133] < MM
then
Eg [|]z + s(ze, t, ho)ho Flz,,t, t, ho] S hi (Ly Lo (M, + 1) + +FMM)²
(17)
(2) Larger Step Error. We further demonstrate that the single inference step remains accurate even at coarser step sizes, thanks to the self-consistency constraint:
Lemma (Single-Step Error with Step Size h).. Let ho = M
1 be the smallest stepsize. If
for all h' and = 0, h/, 2h'
1-,', E F
t. (,+ F() (2,, +2/) <
22
for all t 0, ho. 2ho...., .....
ho. Ex P. [1 + s(2,, ho) - F(z,tt,
ho)ll 5
h
for all h' and 0, h., 2h' ...., 1 h'. s(., t,h')))) is L-Lipschitz,
25

---

thenfor h = 2" ho, and for all t = h, ,,,, h,
(, ho P(a,, t < exp V (1
1 2 ( (E + nesc).
(18)
(3) Composed Error Over Multiple Steps. Finally, even with repeated applications, the shorteut models errors remain controlled, showing stability over long horizons. In other words, if the single inference step error is small and L-Lipschitz, then the multi step trajectory error is bounded:
Lemma (Error of 1/h-Step Inference). If EE [[ + s(z,, t, h)h Fz t, + h2 for all t 0, h, 2h, ................................ 1, and s(, t, ht) is L-Lipschirz then
21 11 $ ( :.
(19)
Summary: Aggregation of Error Bounds. theorem by applying the lemmas order.
Given these insights, we finish the proof of the main
Proof of Theorem /
Let ho be the smallest step size. Suppose h ho-
By Lemma
Ez [llz s(z,, (o),,t,o))) ho) F ko) s hō Loo (Mo + 1) =ras) (20)
By Lemma E
V E (2,,, h) F(z,, t, 101 & hexp ( (Lue (Mo + 1) + FFM + nesc).
(21)
By Lemma E
S 1 (1+ LN)+ -1) exp 1 ( (Iyel (Me 1) FFM + nesc) | 2112
(22)
26

---

C.2 Proof of Lemma 8
Proof Sketch.
The proof's strategy is:
1. Compare the true dynamics with linear approximation & using ODE analysis.
2. Bound the deviation over time.
3. Add the flow-matching loss to account for model error.
Proof. Consider t E [t',t' + hol. Let
2: : F= t',t)
(23)
2 =y + u/ (Zx) - -²))
(24)
By definition,
dz
(x)
(25)
di
d
=00 =0r (zv)
(26)
dt
Then
d
12 2z =
2 - Eti J 2t
(27)
dt
dt dt
(2t
2,, ((1)) ve (2e))
(28)
< 21 (11))l(((2z) er (2e)
(29)
On the other hand, by chain rule:
æell2 d 1| 21112
dt 2 - le112 = 21|ze
(30)
By Equations Mn and 60
d
dt 11 - æell2 (1()((=)) ux (2r)122
(31)
dt
By triangle inequality:
(2))l22 + (Iv((E)) ((x)(2)) uy (2v lll (0())l
+ [[o((v)) (2(22)l12 (z0)l22
(32)
Because vx(x) is Lu-Lipschitz in 2, (lu((2)) v2(2))l22 L Zell2- Because 2, + ur (zr) (t (') and (x() is L.. -ipschii in
(33)
(1,) (x(2))) || + uy (t tt)) - (2x)ll2
(34)
s L
-t) t¹)
(35)
L I1 (ze )12b0-
(36)
Because (x(x)) is Tnp-Lipschitz in [1.((2r) (2r))l2 I Lt t²) s Loho-
(37)
Thus
d di |0 2.122 SL.12 E,l22 + L. (( (Zx))l2 + 1)ho
(38)
Multiplying e (u((-ee) on both side, get:
e -t, 1| Eell2 S e (t-(t-()) Lullee - E l22 + e (to L. (1 + 11ho (39)
dt
27

---

By rearranging:
di (c-bolt-t))ler - zell)
d
s bu(e-t) Lo (1ll (zv)lla + 1)
(40)
s L. (e)) 1ll + 1)ko.
(41)
Because Zp Ze, by integrating both side over t [t', t' + ho] have:
e Loho 120 - Zu+hool2 & Lo (2r )l2 + 1)hg-
(42)
Thus
b th ll2 SLo (2r) + 1) hgg
(43)
Taking square and expectation on both sides, we have:
E
tho E0+holl2
LLge hAE [[llow 1l2 1)²].
(44)
Thus
+1o - Ev+hio 1121
S Ln nā [((lvvr 1)31
(45)
S ho hg (VEllbe (1)) +1)
(46)
S Lue ho (Mo +1
(47)
By definition,
1 2r ll2 ||z0 + ve (2v) yho - (ev + s(zv, t', ho)ll)l
(48)
= [(vx((+)) s(zvi, tt, ho)h1) ho
(49)
Taking square and expectation on both sides, we have
E +ho
AiE [[v) - s(zv, ho]ll3] ( S S
(50)
Thus
V Eeg [1
2e S Es, [llzetho Bgr tho [1 + EE,, [1| - 2p
(51)
s Le bi (Md 1) + hUcpm
(52)
s ho ho (My + 1) + FFM)
(53)
28

---

C.3 Proof of Lemma
5
Proof Sketch. The proof's strategy is:
1. Define a recursive error relationship for step sizes hhh--
2. Use inductive bounding and logarithmic scaling of error growth.
3. Solve the recursion to show controlled error amplification.
Proof. We define Anr to be the maximum 1-step error induced by shortcut model with step size h/.
Formally, for all ht' > S.t. 1/h/ E E, we define,
V (10000 4, 10)1
max W EE (0,0 ....--.. h2
(54)
For all N and {0,2N
-2h'),
E
1, 21/0112]
EE
(1,,,, 2h') F(N (2,, 2h)
V [1,00 (z,, L, 22)
(55)
By assumption, the first term is bounded by 2h'esc. We now analyze the second term. Let b,tN
F(h) (ze, 1, h²), then
[[LFUNO t, 2h) F(z,, 21
(56)
[1IFNNO W,++ 2h') F(z + W,+ 2
(57)
[[ W, 2h') - (2 t N',++ 2
(58)
+ / ( 2h' - F (2 t + M,++ 21'
(59)
By triangle inequality,
|F(M (E 1t+ + h' t + 22) - FTh)) (244)) + N. + 2h')ll2
(60)
-1((4 + s(
h 2h')h') (244 s it H1, t 2h')
(61)
&11Ette 2 1| + h || ( + W, s(ztte, W, 11)ll
(62)
Because s(,,t,h) L-Lipschitz (Assumption
(2tthr [[F(()(E4+N.. t/, 2h') F(h) (2 thttthhttt 2h - (1 LA') 21
(63)
Thus
EE [[]P (2c4h/ + h 2h) - (2 t +
(64)
5(1 + Lh E -1leett- 11
(65)
=(1 + TM E.. [1] F 1013] S (1 + TN LN
(66)
By Equation 52
E [[ 1 + W1, 2h") - 11344,1-MM44 21 S AAr.
(67)
Combine everything together,
E [2,h 24²) t, + 21/)113]
(68)
<2h'ssc + (1 L + A 2h'csc + (2 + Lh Lh
(69)
29

---

Because t' is chosen arbitrarily, we have
AZw 52h' + (2+ + LB An
(70)
Let h' = ho and Ag = A2thor have:
Ac+1 S (2+ + Lhn2*) As + 2escha2
(71)
Solving this recursion, we have:
n-1
R-1
An Ao
II f=0
(2 + Lha2") 2escho
2h
II
(2 + Lho2')
(72)
j=k+1
A02"
n-1
II
1 + -
Lho2')
+ 2escho
2h2 k-
II
1+ (1+2 Lho22
(73)
2
j=0
k=0
j=k+1
n-1 II j=0
[1 1
1+ 1 [+2
= AA22
- Lin
+ escho
(74)
2
1=
-=++1
Because In(1 + r) S x for all x > 1, we have
An SA02" exp
n-1
Σ in
+ 2
1
[hu2*)
(75)
+ 2" Escho
n-1
Σ
exp
n-1
Σ In
+ Thn22) )
1
(76)
j=k+1
1
n-1
1
<AA2" exp
Lho
Σ
2
+ 2" escho
Σ
exp
Lio
2
(77)
2
j=o
k=
j-k+1
1 17
1
=Ao2" exp
2" escho
Σ
exp
(78)
2
k=0
<A02n exp
2
1
[1h222)
2d 2"Eshoo
n-1
Σ exp
2
(79)
K=U
=2" ho exp
(
1
[ (
(Au + nesc
(80)
1oo
cxe (1 2 Lh
( An
+ nesc
(81)
1o0
Because EgrP [ll=e + s(2,, t, ha)ha-
F(z,, t,t t + ho) s ng3e2 we have Ao S hoc. Thus
D Agrto An S hhepp (2(h) (E + nesc).
(82)
30

---

C.4 Proof of Lemma
Proof Sketch.
The proof's strategy is: 1. Define the recurrence over time steps for error accumulation,
2. Use Lipschitz continuity to control error propagation.
3. Solve the recurrence analytically.
Proof.
| 24+111
sv || + s(2t (h) L, hi (z+ s 1, 11
+ v [1l + s(2,, t, hihi 244t2l
(83)
< + Lh) / E [1I8(N) (egh)~zell -zel3 + he
(84)
Because B11=/()
=0113] = 0, by solving the recursion,
E [1 11 - (a+ Lh) -1) T.
(85)
31

---

D Full Results
Table 2: SORL's Overall Performance By Task. We present the full results on the OGBench
tasks. (*) indicates the default task in each environment. The results are averaged over 8 seeds with standard deviations reported. The baseline results are reported from Park cl al [ 's extensive tuning and evaluation of baselines on OGBench tasks.
Shortest
Tank
TQI.
TTg:.
SAPO
CMC
FANRC
PITAC
TPg
waagge--aag--nrrigaa www.rrigggeo--egeeaaee wtr---innggaat-taaa----- (+)
91 2:13
1
0.2.
42. pm
1:2
70
24 2.7
50
93
antwaze-large-nn large-narggggg www.nnaiggggaaaa gate--ingg sega--inggetattttaeee taa22-t
6.a
47
M8:
14:50 cm
2.24
0:2:
35 237
8
57. 2:0
71-22
29
72 47
61 4.x
26 cs
5-2
49 ac4
12 43
53 us3
-2 cur
03-
88 -10
antmere-l larg nort gate winglatesk twab6-90
23
10:21
511 27
antmaze large navigete kinglebask taaks
10
51.
23
2:30
1.21
35 cr
0:2
sataine-iiut-narie stnglatask taale vo
antaaze
O
0
14
25
Tar
19
49 uzz
0.11
0
0
10
10:1.
0
horanoideane radius nert grow sing atoot caak v0 (+)
15:a5
55 am
0:27
25
19:
67
horaso Minzee- toe wwr.gggo-sinnntaaaa
41:14 1.4
A1
47 1.0
40.11
76 u.3
04
80 1.
curancidenc redium mavigess sing- stast taaks vo
25 mm
2
74 s1
medium mavagase sing. horanoldeazee wooe-eedeiiiee wartu--iiigge-----ii 5aa50-9
1:
3.
boraso Maaze- worge-oroggte-sltaee wot.iiolooll-otttt----
1.41
horano deazze-largg
0
bozano ide
gatersae letask -10
T
8
3
48
11
horano montdeeee--agg -arge-la gate-eiig teggleskk-aaatee
1:2
1:2
0
0.24
2
her ideane- pargeeaee nwer gate waggleteek taxict--
0:3
2:2
0.24
0.20
0.24
0 an
1:36
ant rescoer-areta-arre orr.gggttggggeraeeee astapcoss "areea siate sisggessskkttusee
14
1.
122 a.
2.s
77 244
2.4
ustaccoss mas caseraaaaai seviggatesss was.e---ngeettaltttteee
G.
twe gate on anigg
cobe teihe-etgggeeee wy- wagggeaak----ett---
10
88.2
95
89
77
755
714
97
97
cubersing wig.geegepppeee 2e-play---aaa www.oeeeeaaeeee -12-22222200
(x)
92
96
82
80
58
711
07
90
nobe ettgle playy wing letook ta
M1
95
99:
96
98 21
87:20
82 a3
58
98
500
cobe astgle play wsng lebark taske vg
92
08
710 21
911 =2
70- mm
79 am
71
964 23
915
cabe-oiigle-piic wtw.pllsagaaeaaalllll
73.14
88T Lo
90 k.
212
80 1.4
76 uma
77 um
B31 2.
cabe-double-pe plaree--p---pllleee twlli11-
7.4m
sabe-double-piae pla---paa--llleeee weiggeraale-eee 10120000000--000
(+)
3.4-
T.
3.
22
play
17
cube-double- siaglesaat-ttttde-
cube-double wembbb---payy aigee
17
19
U5
100:00
100
87
96
98
1.00
99
scens play wsng lecask ta (+)
35 c/
2.
60 pm
15.04
40 m33
70 cm
80-
scane play sasgleeask-taskee
88
63 ssa
04.04
6:
49 sm
35- k3
78 e34
54 c4
96 m1
97
san--play-etngeeee wttgggtesos--ttkkeeo
813
0s:
3.
0.29
6:21
0.30
0:31
acene play-aing task6-y0
0.c
0
D.1 Per- Results
Q: What is SORL's overall performance each task?
SORL achieves the best performance on the majority the diverse set of tasks considered.
We present SORL's overall performance for each individual task in Table 2 Note that the only SORL training parameter that changes between environments is the Q-loss coefficient, which is common in offline RL [[aaaaovv all 202KE, Park all, Ri221E (025). The Q-loss coefficient was tuned the default task in each environment, and the same coefficient was used all 5 tasks in each environment. The experimental setup was kept consistent to ensure a fair comparison between SORL and the baselines reported Park all [2025].
32

---

cube-single-play (5 taska)
cube- single--nl (5 tasks)
7.32
Inference Time (ma)
Epoch Tine (as)
SORL
3.91
4.95
4.36
SDRL
4.28
4.62
4.55
FOL
MP91 1 M11
ME
EQL 32
M
Figure 3: Runtime Comparison. We vary SORL's training-time compute budger (i.e. the number of backpropagation steps through time MBTT) on the left and SORL's inference-tine compute budget (i.e. the number of inference steps M on the right. The performance is averaged over 5 seeds for each task, with 5 tasks per environment, and standard deviations reported.
D.2 Runtime Comparison
Q: How does SORL's runtime compare to the fastest flow-based baseline, FQL?
SORL'S scalability enables training runtime match that of FQL under low compute budgets, while exceeding it when larger budgets are allocated. During inference, SORL mainlains runtime compa- rable to FQL.
We present a runtime comparison between SORL and FQL. FQL is one of the fastest flow-based base- lines consider, for both training and inference [Park call 2025], FQL's policy is one-step distillation model. We select cube-single -play because SORL and FQL achieve similar perfor- mance on the environment (Table (J. The experiments were performed Nvidia RTX 3090 GPU, The epoch and inference times are averaged over the first three evaluations (i.e. the first training/e- valuation, training/evaluation at 100,000 gradient steps, and training/evaluattio steps). For the inference- ceterenncee-ieeeeeee plot, we use the maximum training-comp compute budget (i.e. M BfT 8).
200,000 gradient
Figure E demonstraaes SORL's scalability during both training and inference. traininng time, SORL's training runtime can be shortened by decreasing the number of backpropagation steps through time. At inference time, SORL maintains similar inference times compared to FQL under varying inference- time compute budgets.
33

---

E Ablation Studies
latmmzze-larg--mavigaat (5 taske)
Success
60
40
Min
20
M A
2
Backpropagation Stops Through Time (METT)
matsoccor-aren--nviiee (5 tasks)
scene- play (5 tasks)
Success
Success
20
Minl Mftt MUT
Minl Minl M/rr
0
2
8
2
8
Backpropagation Steps
Through Time (MBTI)
Backpropagation Steps
Through Time (MUTT)
Figure 4: Ablation Over Backpropagation Steps Through Time, MBTT We investigate the effect of varying the training-time time compute budget (i.e. the number of backpropagation steps through time MBTT). The performance averaged over 8 seeds for each task, with 5 tasks per environment, and standard deviations reported. We report results using one inference step (i.e. Mint = 1) and using the same number of inference steps as backpropagation steps through time (i.e. Minf ABBTT).
E.1 Backpropagation Steps Through Time, BTT
Q: How does increasing training-time compute affect performance?
SORL generally observes increasing performance with increased training-ting time compute (i.e increas- ing M ). up to a performance saturation point of BTT = 4.
One of SORL's 's unique strengths is its scalability at both training-time and inference-time. In this experiment, we consider the question of how much training-tiin -iime compute necessary. Based on the results in Figure g we see that, in general, performance improves with increased training-ting time compute (i.e. with an increasing number of backpropagation steps through time, ATTT).. However, performance seems to saturate around M8TT 4, suggesting that greater training-time compute may not be necessary for the environments considered.
The results suggest two key takeaways. First, backpropagation through time over more than one step generally necessary in SORL maximize performance. Insufficient training-time compute may lead to sub-optimal results, as evidenced in Figure E and Tables Mand[ where SORL with M BTT > 4 outperforms both the baselines and the versions of SORL with BBIT 4. Second, the results indi- cate that SORL does not require backpropagation through tens or hundreds of steps- as is common large diffusion models [HH all 2020. (Song et all 2020 procedure that could be computa- tionally prohibitive.
34

---

-aw.aaz--arge--aiigeeeeee (5 tasks)
80
60
Success
20
FQL
SORL
200
400
600
1000
Policy Network's s Hidden Layer Size
Figure 5: Ablation Over Policy Network Size. The performance is averaged over seeds for each task, with tasks per environment, and standard deviations reported. We use the same training-time and fifeeence--ime compute budgets for SORL Min! = 4). We train and evaluate FQL with the parameters used in the official implementation [Park
for Tables II and 2 (i.e. MBIT 8 and
2025. The only change SORL and FQL is varying the sizes of the policy network's 's hidden
layers.
E.2 Policy Network Size
Q: How does the performance of SORL vary with changing size of the policy network?
Increasing the capacity of the policy network increases the performance of SORL and FQL similarly.
We the same training-time and inference-time compute budgets for SORL as M and Q (i.e. .BTTT = 8 and M 4). We train and evaluate FQL with the parameters used in the official implementation [Park all 2025]. The only change SORL and FQL is varying the sizes of the policy network's hidden layers. We use an MLP with four hidden layers of the same size. Since FQL uses two actor networks of the same size in the official implementation [Park et all 2025]-a flow-matching network for offline data regularization and a one-step distillation network for the actor's policy- we scale both networks, keeping them the same size for this ablation study.
for Tables
As may be expected, increasing the capacity of the policy network increases the performance of both SORL and FQL, at similar rates, before plateauing larger sizes. Notably, SORL achieves slightly bet- ter gains across network sizes, which may stem from the shortcut model class being more expressive than FQL's one-step distillation policy.
35

---

Experimental and Implementation Details
In this section, we describe the setup, implementation details, and baselines used in the paper.
F.1 Experimental Setup
The experimental setup for our paper's main results (Table II and ( follows OGBench's official evaluation scheme et 20245 R0225).
Environments. We now describe the environments, tasks, and offline data used in our experiments. Our setup follows OGBench's official evaluation scheme [Park all 20225 and Park al [2025] and we restate the description of environments and tasks below.
We evaluate SORL on 8 robotics locomotion and manipulation roboties environments in the OG- Bench task suite (version 1, 1.) ePar 20245). a benchmark suite designed for offline RL. Specifically, use the following environments:
1. atthazzellngggaaaiigbbeeingggee
2. attaaee-rtaairrarrigaeeeinggeeer
3. auaaoodhhheeel-aggerrrrrgggiiigr
4. amaaaddchhzrgggga.c.eerr/g/-ioor
5. atteccarr.caaennrrr//tessaaaaaea
6. uuberssinggerllayyssigleeett---
7. cibe--oobbeepllayyysinggatta----
8. scene-play-singleetsx-1
The selected environments span range of challenging control problems, covering both locomo- tion and manipulation. The antmaze and humanoidmaze tasks involve navigating quadrupedal (8 degrees of freedom (DOF)) and humanoid (21 DOF) agents through complex mazes. antsoccer focuses on goal-directed ball manipulation using a quadrupedal agent. The cube and scene environ- ments center object manipulation with robot arm. Among these, scene tasks require sequencing multiple subtasks (up per episode), while puzzle emphasizes generalization to combinatorial object configurations. All environments (navigate for locomotion, play for manipulation), which are built from suboptimal, goal-agnostic trajectories, and therefore pose challenge for goal-directed policy learning. We evaluate agents us- ing binary task success rates (i.e., goal completion percentage), which is consistent with OGBenchs evaluation setup [Park [0223]
state- We follow the standard dataset protocols
Tasks. We use OGBench's reward-basee singletask variants for experiments Park et all [02ea] which are best suited for reward-maximizing RL. As described in the official implementa- tion [Park all 20224)) each OGBench environment offers five unique tasks, each associated with a specific evaluation goal, denoted by suffixes singletask-tase unique, fixed goals for the agent to accomplish. We utilize all five tasks for each environment. The datasets are annotated with semi-sparse reward, where the reward is determined by the number of remaining subtasks at a given step for manipulation tasks, or whether the agent reaches a terminal goal state for locomotion tasks [Park all 2022. 2025].
through -task5. These represent
Evaluation. We follow OGBench's official evaluation scheme |Pak et all [0248] We train al- gorithms for 1,000,000 gradient steps and evaluate 50 episodes every 100,000 gradient steps. We report the average success rates of the final three evaluations (i.c. the evaluation results at 800,000, 900,000, and 1,000,000 gradient steps). In tables, we average over 8 seeds per task and report standard deviations, bolding values within 95% of the best performance.
36

---

F.2
SORL Implementation Details
One of the strengths SORL is its implementation simplicity. In order to implement SDRL, we adapt Park et [2025]]s 's goen-source implementations of various offline RL algorithms (FQL, IFQL, IQL. ReBRAC), which are adapted from Park all [2024 's open-soure dataset and codebase. The major changes are implementing and training the shortcut model, and removing additional training complexity from FQL (e.g. the teacher/student networks). To ensure a fair comparison, we keep all major shared hyperparameters the same between the baselines and SORL (e.g. network size, number of gradient steps, and discount factor), unless otherwise noted.
Value Network. We train two Q functions and use the mean of the two in the actor update (Equa- tion 9). We use the mean the two Q functions for the critic update (Equation [E), except for antmaze-large and antmaze-giant tasks, where we follow Park et all [2025]] in using the mini- mum of the two values. The only change to the FQL baseline's value network is adding an input that encodes the number of inference steps used generate the actions.
Network Architecture and Optimizer. We use multi-layer perceptron with hidden layers of size 512 for both the value and policy networks. We apply layer normalization [Ba et all 2016] to value networks. We use the Adam optimizer [Kingma, 2014], which we add gradient clipping to.
Discretization Steps. For SORL, we use M 8 discretization steps during training (2 fewer than FQL), since Frans et all [2022]] suggests that discretization steps be powers of 2. By default for the results in Tables MI and E use M 8 steps of backpropagation through time and M 4 inference steps, except for humano ierrannoddazze-iee www.dm-meddddeee where we use Minf 2. other experiments, if the values of BTT and Minf changed, they explicitly noted.
Shorteut Model. Following the official implementation
shortcut models [Frans et all 20241),
for the self- self-consist loss we sample d ~ {2* log, M-1 and then sample uniformly on multiples of d between 0 and 1 (i.e. points where the model may be queried). The model makes a prediction for step size 2d, and its target is the concatenation of the two sequential steps of size d. For greater stability, we construct the targets via a target network. Unlike Erans et all [2022]] we do not require special processing of the training batch into empirical and self-consistency targets, nor do we require special weight decay. We simply use the entire batch for the critic and actor updates, including the Q loss, self-consistency loss, and flow-matching loss.
Tk=0
Hyperparameters. We largely use the same hyperparameters for SORL as those used in the FQL baseline (Table B) except for parameters that specific to vary with SORL (Table H). We hy- perparameter tune SORL with similar training budget to the baselines: we only tune one of SORL's training Parameters----oon ble M over the values {10, 50, 100, 500}. We then use the same QL coefficient on all the tasks in an environment. Varying the strength of regularization to the offline data is common in offline RL [larasov et all 20235, Park et all 2024F, 2025]. Following Park et at [0225] 's recom- mendation, normalize the Q loss.
(QL) coefficient-on the default task in each environment (Ta-
37

---

Table 3: Shared Hyperparameters Between FQL Baseline and SORL.
OPTIMIZER
ADAM KINOAA
GRADIENT STEPS
1,000,000
MINIBATCH SizE
256
MLP DIMENSIONS
[512, 512, 512. 512]
NONLINEARITY
GELU HIANN AND GIMPED MUSS
TARGET NETWORK SMOOTHING COFFFICIENT
0.005
DISCOUNT FACTOR
0.99 (DEPAULT), 0.995 (ANTMAZE-GIANT HUMAROIDNAZE, ANTSOCCER)
DISCRETIZATI STEPS
8
TIME SAMPLING DISTRIBUTION
UNIF((,,1)
CLIPPED Dot BLE LEAKNINGG
FALSE (DEFAULT), TRUE (ADROIT, ANTHAZE- (LARGE, GIANT) NAVIGATE)
Table 4: Hyperparameters for SORL.
HYPERPARAMETER
VALUE
LEARNING RATE GRADIENT CLIPPING NORM
1E-4
1
DISCRETIZATION STEPS
8
BC COFFFICIENT
10
SELF-CONSISTENCY COEFFICIENT
10
Q-Loss COEFFICIENT
TABLE
Table Q-Loss Coefficient for SORL.
ENVIRONMENT
Loss COEFFICIENT
Batuaze-larg-nnavigaeee (5 tasks)
500
antumaze-glant-nnavaate--v (5 tasks)
500
nuunaoo-dsszz--meeuuurrrrrgtt--p (5 tasks)
100
aunaaoozaazze--arggravvggggeg (5 taaks)
500
a (5 tasks)
500
cube-single-play--00 (5 tasks)
10
cube-double-playy-00 (5 tasks)
50
scene-play-vo (5 tasks)
100
38

---

F.3 Baselines
We evaluate against three Gaussian- bassd offline RL algorithms (BC [Poomeleau, 1988], IQL
[Kostrikoy et all 0021]], ReBRAC [Taaaaov et all 2023a ) three diffusion- algorithms (IDQL
Hansen-Estruch et all. 2023], SRPO [Chen et all 2023 CAC [Ding and Jin 2023). and four flow-
based algorithms (FAWAC [Nair all 2020]. FBRAC IZhang et all, 2025J. IFQL [Wang et all, 2022].
FQL [Park all 2025)). For the baselines we compare against in this paper, we report results from
Park et all [2025], who performed an extensive tuning and evaluation of the aforementioned base-
lines on OGBench tasks. We restate deseriptions of the baselines here.
Gaussian-Based Policies. To evaluate standard offline reinforcement learning (RL) methods that employ Gaussian policies, we consider three representative baselines: Behavior Cloning (Pomerleau, [1988], Implicit Q-Learning (IQL) [Kostrikoy et all, 2021], and ReBRAC [Tarasov Cal 20267] BC Poomerlca,, [|X|] serves as a simple imitation learning bascline that directly mim- ics the demonstration data without any explicit value optimization, while IQL [Kostrikoy et ata [021] represents a popular value-based approach that addresses the overestimation problem in offline RL through implicit learning of the maximum value function. ReBRAC [[aaaaoov et all, (023a)), more recent method that known to perform well on many D4RL tasks [[aaasoov et all 20233)] extends the BRAC framework with regularization techniques specifically designed to constrain the learned policy close to the behavior policy, thereby mitigating the distributional shift problem common in offline RL settings.
Diffusion-Basse Based Policies. To evaluate diffusion policy-based offline RL methods, we compare all, 2023]. SRPO [Chen all, [о23] and Consistency-AC (CAC) [Ding Jin 2023). IDQL [Hamsen-Esstme Estruch el at, 2023] builds on IQL by using a generalized critic and rejection sampling from a diffusion-based behavior policy. SRPO [Chen @eheneetal all 2023] replace the diffusion sampling with deterministic policy trained via score-regularized policy gradient to speed
to IDQL [Hansen-Estren
up sampling. CAC [Ding and Jmn, [023] introduces consistency-basee actor-critic framework to backpropagation through time with fewer steps.
Flow-Based Policies. To evaluate flow policy-based offline RL methods, we compare flow-based variants of prior methods, including new variants introduced by Park et all [2025]: FAWAC [Nai at [020] FBRAC [Wang et all 2022]. IFQL [[Hansen Estruch all 2023 and FQL [PPark et all 2025] as only a few previous works explicitly employ flow-based policies. FAWAC extends AWAC by adopting flow-based policy trained with the AWR objective and estimates Q" for the current policy using fully off-policy bootstrapping. FBRAC is the flow counterpart of Diffusion-QL. based on the naive Q-loss with backpropagation through time. IFQL is a flow-based variant of IDQL that relies on rejection sampling. FQL distills a one-step policy from an expressive flow-matte now-matching policy avoid costly iterative sampling. Unlike FQL, SORL uses a single-stage training procedure and does not require distilling model into one-step policy.
39


Reviews:
[{"review_id": "EQ4rozcBbo", "reviewer": "Reviewer_GJHL", "summary": "This paper presents Scalable Offline Reinforcement Learning (SORL), an efficient offline RL algorithm that leverages shortcut models to scale training and inference. SORL incorporates self-consistency into training, enabling it to optimize policies with fewer backpropagation steps while maintaining high expressivity. It achieves strong performance across diverse tasks and can dynamically adjust inference steps based on compute budget, demonstrating improved performance with increased test-time compute.", "questions": "1.\tHow does SORL perform on more complex or diverse tasks beyond the current environments tested? The current results might not reflect its performance on a broader range of real-world scenarios.\n2.\tWhat are the specific factors that contribute to SORL’s superior performance in some environments but not in others? Understanding these factors could provide deeper insights into the algorithm’s strengths and weaknesses.\n3.\tCan the authors provide more detailed analysis or additional experiments to further validate the robustness and generalizability of SORL’s performance? More comprehensive testing might be needed to fully assess its potential.\n4.\tWhile SORL aims to improve training efficiency, what is the trade-off in terms of computational resources required during inference? Does the potential for test-time scaling compensate for any increased computational demands?\n5.\tHow does SORL compare to other recent advances in offline RL, particularly those that also focus on scaling or improving inference efficiency? Are there any key works that should be included in the discussion to provide a more comprehensive comparison?\n6.\tWhat are the potential limitations of using shortcut models in SORL, especially in terms of capturing long-term dependencies or handling highly complex data distributions?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n1.\tThe paper tackles the significant challenge of scaling offline reinforcement learning, which is crucial for practical applications involving large datasets and complex environments.\n2.\tThe authors present their ideas and methods clearly, making the paper accessible and easy to understand, even for those new to the field.\n3.\tThe paper includes a thorough set of experiments across diverse tasks, demonstrating the robustness and effectiveness of the proposed method.\n\nWeaknesses:\n1.\tThe experiments are conducted on a relatively narrow set of environments (8 environments with 40 tasks in total). This might not fully capture the breadth of challenges in offline RL, limiting the generalizability of the results.\n2.\tWhile the paper compares SORL to 10 baselines, the results show that SORL only outperforms them in 5 out of 8 environments. This suggests that its superiority might not be consistent across all types of tasks.\n3.\tThe paper may not provide a comprehensive review of existing methods and their limitations, which is crucial for understanding how SORL fits into the broader context of offline RL research. A more thorough discussion could highlight the unique contributions of SORL more clearly.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their comments. We respond to their questions and concerns below. \n\n# 1. Environments and Tasks\n> The experiments are conducted on a relatively narrow set of environments (8 environments with 40 tasks in total). This might not fully capture the breadth of challenges in offline RL, limiting the generalizability of the results. \n\n> How does SORL perform on more complex or diverse tasks beyond the current environments tested? The current results might not reflect its performance on a broader range of real-world scenarios.\n- **We evaluate SORL against 10 baselines on 40 unique tasks, spanning 8 environments, on a recently released offline RL benchmark [Park et al., 2024a]**. \n- The 8 environments span locomotion and manipulation tasks, including long-horizon tasks (e.g. scene), tasks involving both high-level, goal-directed planning and low-level motor control (e.g. antmaze, humanoidmaze, antsoccer), and tasks involving sub-tasks (cube). \n- Our focus on robotics simulations is standard in the RL literature. \n\n> Can the authors provide more detailed analysis or additional experiments to further validate the robustness and generalizability of SORL’s performance? More comprehensive testing might be needed to fully assess its potential.\n- We emphasize that **our evaluation is performed on a diverse set of tasks and environments** that is either similar to [Park et al., 2025] or more diverse than prior work in offline RL with generative models [Hansen-Estruch et al., 2023, Chen et al., 2024, Ding et al., 2024]. \n\n# 2. Baselines\n> While the paper compares SORL to 10 baselines, the results show that SORL only outperforms them in 5 out of 8 environments. This suggests that its superiority might not be consistent across all types of tasks. \n- We emphasize that **SORL always achieved the second-best performance in _every_ environment evaluated**, and it achieved **the best performance on 5 environments (25 tasks)**. \n\n# 3. Review of Existing Methods\n> 3. The paper may not provide a comprehensive review of existing methods and their limitations, which is crucial for understanding how SORL fits into the broader context of offline RL research. A more thorough discussion could highlight the unique contributions of SORL more clearly.\n- In Appendix A, **we provided a discussion of related work**, and in Appendix E.3, **we provided a thorough discussion of all the baselines** we compare to in our paper. \n\n# 4. Factors Affecting SORL's Performance\n> What are the specific factors that contribute to SORL’s superior performance in some environments but not in others? Understanding these factors could provide deeper insights into the algorithm’s strengths and weaknesses.\n- **The key advantage of SORL over Gaussian-based methods (e.g. ReBRAC) is its more expressive policy class**. This policy class is better able to handle diverse, multi-modal data than Gaussian-based models, which cannot handle multi-modality. **The key advantage of SORL over distillation-based generative model approaches (e.g. SRPO, CAC, FQL) is SORL’s one-stage policy training procedure**, as compared to the two-stage distillation-based approaches over which error compounds. **SORL’s flexibility in selecting the number of discretization and inference steps allows for less discretization error** than a one-step approach. We believe this contributes to more precise actions at test-time.\n- The environments where SORL struggled (antmaze-giant and humanoidmaze-large) are very challenging tasks in offline RL, and it should be noted that none of the 10 baselines we considered solved these tasks well. **SORL always achieved _at least_ the second-best performance in _every_ environment and achieved the _best_ performance on 5 environments (25 tasks).** On the two environments where SORL does not achieve the best performance among the baselines, there is only one algorithm that outperforms it (ReBRAC on antmaze-giant and IFQL on humanoidmaze-large). Both ReBRAC and IFQL fail to achieve the best performance on any other environment, highlighting the difficulty of having reaching optimal performance on every environment.\n- At a higher-level, the antmaze-giant and humanoidmaze-large tasks are known to be challenging, long-horizon environments, and it is likely that these environments would benefit significantly from online exploration. As noted in the limitations discussed in Appendix B, SORL does not have a principled exploration strategy, which is standard for _offline_ algorithms. Depending on the coverage and quality of the offline dataset, purely offline learning may get stuck in local optima.\n- Relatedly, it is possible that policy extraction is not the primary bottleneck in these two environments under the given dataset size (1 million transitions). It may be the case that, for these two environments, **value learning is challenging**. If the learned value function (e.g. Q function) is poor, then it will be difficult to optimize a policy under that value function, irrespective of the policy extraction method, expressivity of the policy class, or discretization/integration error.\n\n# 5. Computational Resources During Inference\n> While SORL aims to improve training efficiency, what is the trade-off in terms of computational resources required during inference? Does the potential for test-time scaling compensate for any increased computational demands?\n- SORL offers flexibility during both training and inference. That is, SORL can take advantage of a larger compute budget during both training and/or inference. In Figure 2, we show that **SORL can compensate for less training-time computation with greater test-time computation**. Specifically, we consider a small training compute budget of $M^{\\text{BTT}}=1$, indicating that we only use one step of backpropagation through time during training. The baseline performance of SORL under such a small training budget is low ($\\text{SuccessRate} = 13$ with a single inference step, $M^{\\text{inf}}=1$). However, by applying inference-time sequential scaling (i.e. using inference steps $M^{\\text{inf}}=2$) and parallel scaling (i.e. Best-of-N with $N=8$), SORL’s performance improves to $\\text{SuccessRate=55}$, top performance among the 10 baselines.\n\n# 6. Comparison To Recent Advances\n> How does SORL compare to other recent advances in offline RL, particularly those that also focus on scaling or improving inference efficiency? Are there any key works that should be included in the discussion to provide a more comprehensive comparison?\n- **We will add the following paragraph to our related work for additional comparison**:\n     - Independent of generative models, prior work has proposed applying a form of best-of-N sampling to actions from the behavior policy (i.e. behavioral candidates) [Chen et al., 2022, Fujimoto et al., 2019, Ghasemipour et al., 2021, Hansen-Estruch et al., 2023, Park et al., 2024b]. Park et al. [2024b] proposed two methods of test-time policy improvement, by using the gradient of the Q-function. One of Park et al. [2024b]’s approaches relies on leveraging test-time states, which SORL’s parallel scaling method does not require. The second approach proposed by Park et al. [2024b] adjusts actions using the gradient of the learned Q-function, which is conceptually similar to our approach of best-of-N sampling with the Q-function verifier. However, their method requires an additional hyperparameter to tune the update magnitude in gradient space.\n\n# 7. Limitations\n> What are the potential limitations of using shortcut models in SORL, especially in terms of capturing long-term dependencies or handling highly complex data distributions?\n- **SORL utilizes a highly expressive policy class, shortcut models, so we believe that SORL is capable of handling highly complex data distributions as well as or better than other generative model-based offline RL methods.** However, the problem of capturing long-term dependencies relies on the Q function, for which we use the standard Bellman error minimization. Learning a robust value model is thought to be challenging in offline RL [Levine et al., 2020], though recent work has argued that policy extraction is the primary bottleneck in offline RL [Park et al., 2024a]. **Our paper’s approach does not improve the value learning component and instead focuses on improving policy learning and inference-time scaling.** \n- Additionally, **like all _offline_ RL algorithms, SORL does not have a principled exploration strategy**, as exploration is generally not necessary or desired in the offline setting. However, if SORL were to be applied in the _online_ setting, the lack of an exploration mechanism may limit SORL's performance on tasks requiring significant exploration."}, {"review_id": "iy4eYqlxuk", "reviewer": "Reviewer_VRv3", "summary": "The paper proposes a method to train a shortcut model for offline RL tasks. The training loss consists of three components: flow-matching loss, consistency loss, and Q loss. Unlike two-stage distillation frameworks such as FQL, the proposed method (SORL) is a single-stage approach that offers good scalability at both training and test time.", "questions": "- **Loss components and data usage (Alg. 1)**: Why does the algorithm use different data samples for computing different loss components? Is there any benefit to this? Would it cause issues if all components used the same batch?\n\n- **Critic update with sampled inference steps**: During critic updates, the algorithm samples $m$ inference steps. When $m$ is small, the resulting actions might be noisy. Could this hurt Q-estimation accuracy and affect Bellman equation convergence—especially if the shortcut model is not yet well trained?\n\n- **Equation 14 and min-Q trick**: I noticed from the appendix that the method uses the min-Q trick. Please consider integrating this directly into Equation 14 for better clarity.\n\n- **Figure 1 (AntMaze-large)**: Why does increasing inference steps lead to worse performance on AntMaze-large tasks?\n\n- **Table 2 (AntMaze-giant)**: SORL performs poorly on AntMaze-giant. It seems that generative RL methods struggle on this task. Any insights into why this might be?\n\n- **Figure 3 (Scalability benefits)**: Does Figure 3 suggest that training-time scalability is more beneficial than test-time scalability? What happens if you increase $M^{\\text{BTT}}$ to 16? Does performance continue to improve?", "limitations": "It would be promising to see the proposed method tested on more high-dimensional offline RL tasks to further validate its scalability.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n- The question of training one-step generative models directly for rl setting (rather than relying on two-stage frameworks) is timely and important.\n- The method is well designed. By incorporating Q-loss into shortcut model training, it offers scalability in both training and inference.\n- The theoretical analysis appears sound with no obvious flaws. The experiments are standard and show strong performance compared to recent baselines like FQL.\n\n**Weaknesses:**\n\n- Given that scalability is central to the paper, it's worth asking whether the method has been tested on higher-dimensional inputs (e.g., image-based observations). That would further support the scalability claims.", "was_revised": true, "final_justification": "I have no concerns remained and decide to keep my score as 4.", "rebuttal": "We thank the reviewer for their comment. We respond to their questions and concerns below.\n\n# 1. Higher-Dimensional Inputs\n> Given that scalability is central to the paper, it's worth asking whether the method has been tested on higher-dimensional inputs (e.g., image-based observations). That would further support the scalability claims.\n- Image-based observations are out of the scope of our paper. The **focus of our paper is on policy extraction and inference-time scaling**. Using image-based observations would require an encoder, the choice of which would significantly impact performance.\n\n# 2. Loss Components and Data Usage\n> Loss components and data usage (Alg. 1): Why does the algorithm use different data samples for computing different loss components? Is there any benefit to this? Would it cause issues if all components used the same batch?\n- In practice, **all components in our implementation use the same batch**. The separate data samples in Algorithm 1 was a mistake in our pseudocode. We will update the algorithm’s pseudocode to reflect the implementation.\n\n# 3. Critic Update\n> Critic update with sampled inference steps: During critic updates, the algorithm samples $m$ inference steps. When $m$ is small, the resulting actions might be noisy. Could this hurt Q-estimation accuracy and affect Bellman equation convergence—especially if the shortcut model is not yet well trained? \n- **By enforcing self-consistency during training, we ensure that sampling with smaller $m$, including the 1-step policy, achieves a discretization error comparable to the Euler method with larger $m$**. The reviewer’s question points to a common issue faced by actor-critic style methods: both the actor and critic updates depend on each other’s outputs, and they will have to handle \"noisy\" outputs at the start of training. We mitigate this issue by using target networks for both the actor and critic, which ensures a “slower update” for both the actor and critic. As discussed in Appendix E.2, SORL does not require significant hyperparameter tuning beyond the standard regularization parameter in offline RL. \n\n# 3. Min-Q Trick\n> Equation 14 and min-Q trick: I noticed from the appendix that the method uses the min-Q trick. Please consider integrating this directly into Equation 14 for better clarity.\n- Thank you for the comment, and we will update Equation 14 accordingly.\n\n# 4. Figure 1 (AntMaze-Large)\n> Figure 1 (AntMaze-large): Why does increasing inference steps lead to worse performance on AntMaze-large tasks?\n- **The reason for the sequential scaling trend in antmaze-large remains unclear**, but notably, **SORL achieves the best performance of any method under every number of inference steps**, and they are all a substantial improvements over the 10 baselines. The difference in SORL’s performance under the varying inference steps is smaller compared to other environments. \n\n# 5. Table 2 (AntMaze-Giant)\n> Table 2 (AntMaze-giant): SORL performs poorly on AntMaze-giant. It seems that generative RL methods struggle on this task. Any insights into why this might be?\n- Antmaze-Giant is a very challenging task in offline RL, and it should be noted that **SORL achieves the second-best performance on this environment**. Additionally ReBRAC fails to achieve the best performance on any other environment, highlighting the difficulty of having reaching optimal performance on every environment.\n- More generally, the key advantage of SORL and other generative model-based approaches over Gaussian-based methods (e.g. ReBRAC) is its more expressive policy class. This policy class is better able to handle diverse, multi-modal data than Gaussian-based models, which cannot handle multi-modality. The key advantage of SORL over distillation-based generative model approaches (e.g. SRPO, CAC, FQL) is SORL’s one-stage policy training procedure, as compared to the two-stage distillation-based approaches over which error compounds. SORL’s flexibility in selecting the number of discretization and inference steps allows for less discretization error than a one-step approach. We believe this contributes to more precise actions at test-time. \n- **It is possible that policy extraction is not the primary bottleneck in these two environments** under the given dataset size (1 million transitions). It may be the case that, for these two environments, value learning is challenging. If the learned value function (e.g. Q function) is poor, then it will be difficult to optimize a policy under that value function, irrespective of the policy extraction method, expressivity of the policy class, or discretization/integration error. \n\n# 6. Figure 3 (Scalability Benefits)\n> Figure 3 (Scalability benefits): Does Figure 3 suggest that training-time scalability is more beneficial than test-time scalability? What happens if you increase\n- **The question of whether training-time or test-time scalability is more beneficial is difficult to answer because it relies on a tradeoff between compute and performance**. For some settings, test-time compute may be low because the robot must perform inference quickly (e.g. autonomous vehicles), while in other tasks, precision may be valued over speed (e.g. surgical robotics)."}, {"review_id": "l16sJFAswg", "reviewer": "Reviewer_rsZE", "summary": "The authors introduce Scalable Offline Reinforcement Learning (SORL), a new offline RL algorithm that leverages shortcut models to scale both training and inference. SORL's policy can be trained efficiently in a one-stage training procedure. At test time, SORL supports both sequential and parallel inference scaling by using the learned Q-function as a verifier. Experiments are conducted to validate the effectiveness of the proposed method.", "questions": "Apart from other weaknesses, I list my confusion about the equations in weakness 3 here.\n\n1. In Eq. 7, the minus sign should be plus sign?\n\n2. In Eq. 2, is the trivial solution $f=z_1-z_0$? This makes the objective 0, achieving the smallest value (since $||\\cdot||_2>0$).", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n1. The scaling of the (offline) RL method is an important and interesting problem. \n2. The paper is well written. \n\n**Weaknesses:**\n1. Straightforward Theory. It seems like the conclusion that Theorem 2 tries to prove is trivial: due to the flow matching loss in Eq.2, the policy is learned with regularization, and hence, SORL can learn a policy that is regularized to the behavior of the offline data. It seems like a standard offline RL approach.\n\n2. Lacking Training Scalibility Experiments. The authors claim the proposed method scales both training and inference in the paper. Yet, the training scalability experiments are missing. \n\n3. Some Equations Flaws. See the question part.\n\n4. Novelty concern. The paper employs an existing type of generative model, the shortcut model (proposed around 2023), to implement the policy in offline RL. Is this enough to construct a novel idea or just a model implementation trick?", "was_revised": true, "final_justification": "I have read the author's response. For the novelty, inference-time scaling is also an existing method. However, the combination of the short-cut model and the inference-time scaling seems ok. I will increase my score to 4.", "rebuttal": "# 1. Theory\n> Straightforward Theory. It seems like the conclusion that Theorem 2 tries to prove is trivial: due to the flow matching loss in Eq.2, the policy is learned with regularization, and hence, SORL can learn a policy that is regularized to the behavior of the offline data. It seems like a standard offline RL approach.\n- We agree with the reviewer that the ultimate aim of Theorem 2 involves a standard one for offline RL and generative model theory: proving that minimizing the training objective keeps the learned policy “close” to the behavior policy, where closeness is measured by Wasserstein distance. \n- However, we emphasize two points: **1) our analysis is an important theoretical grounding for our algorithm**, and **2) our analysis of shortcut models is novel**. While the overall aim of our theoretical results may be standard, we are using a novel class of generative model that has not been used before in offline RL. Moreover, the paper that introduced shortcut models [Frans et al., 2024] lacked any theoretical results, **making our theoretical analysis of shortcut models the first to our knowledge**.\n- More specifically, Theorems 2 and 3 involve the convergence of the shortcut model. The proof of the shortcut model’s convergence is non-trivial because of distributional shift: the flow-matching loss and the self-consistency loss are trained on the _ground-truth_ marginal distribution, not the marginal distribution _during training_. This distributional shift happens at the level of the shortcut model’s inference step, which is separate from the standard offline RL-notion of distributional shift, which occurs at the the MDP-level. The distributional shift we analyze is further complicated because of the self-consistency step in shortcut models. In summary, we argue that the our theoretical analysis of shortcut models is a novel and important contribution to the theoretical grounding of our algorithm.\n\n# 2. Training Scalability\n> Lacking Training Scalability Experiments. The authors claim the proposed method scales both training and inference in the paper. Yet, the training scalability experiments are missing.\n- As discussed in the paper, there are two principal ways of scaling training. First, in order to scale training, offline RL algorithms must handle large, diverse, multi-modal datasets. **To demonstrate SORL’s training scalability, we test it on a variety of sub-optimal datasets and offline RL tasks**. We show that, for a fixed dataset, SORL can generally use the data to achieve a higher performance than the other 10 baseline methods considered. The second avenue to scale training is focused on computation (i.e. how much computation is required to train an algorithm.) We follow prior work in training SORL (and all baselines) for 1M gradient steps. **We then provided a runtime comparison to the fastest baseline in Appendix C.2**.  We show that SORL is flexible: it can be optimized for fast training, as measured by training runtime, by decreasing the number of backpropagation steps through time, $M^{\\text{BTT}}$. And, SORL can be optimized to leverage greater computation during training to further increase performance, by increasing the number of backpropagation steps through time, $M^{\\text{BTT}}$.\n\n# 3. Novelty\n> Novelty concern. The paper employs an existing type of generative model, the shortcut model (proposed around 2023), to implement the policy in offline RL. Is this enough to construct a novel idea or just a model implementation trick?\n- Prior work [Hansen-Estruch et al., 2023, Chen et al., 2023, Ding and Jin, 2023, Wang et al., 2022, Park et al., 2025] focus on scaling training in offline RL by leveraging expressive policy classes that can model complex data distributions. As the reviewer points out, part of our paper follows a similar approach, leveraging shortcut models as the new type of generative model for offline RL training. However, **our paper extends beyond a simple model implementation trick by tackling the question of inference-time scaling in offline RL with generative models**. Motivated by the recent work in language model test-time scaling, SORL introduces methods for sequential and parallel scaling which enable flexible inference: SORL can perform both fast inference or high precision inference, depending on the test-time compute budget, without having to be retrained. We argue that introducing avenues for test-time scaling in offline RL presents a novel contribution unaddressed by prior work. \n\n# 4. Equations\n> In Eq. 7, the minus sign should be plus sign?\n- Yes, the minus sign should be plus. We will fix this mistake.\n\n> In Eq. 2, is the trivial solution $f=z_1 - z_0$? This makes the objective 0, achieving the smallest value (since $\\|\\cdot \\|_2 > 0$).\n- To clarify, the function $f$ in Eq. 2 takes $(x_t, t)$ as inputs, not $(x_0, x_1)$. The setting we consider here is standard see Liu et al. [2022]."}, {"review_id": "yslAhvG6o1", "reviewer": "Reviewer_3Dwq", "summary": "This paper proposes SORL, a new offline RL algorithm that leverages expressive generative models as a policy class and allows for direct optimisation of the policy with the gradients of the critic using shortcut models. This avoids the instability of back propagation through time (BPTT) that exists when optimizing vanilla flow/diffusion models with RL due to the iterative denoising process. Using shortcut models allows for using different denoising steps for the policy optimisation and the behaviour cloning terms, achieving the best of both paradigms. It also means that practitioners can scale inference time compute when available, using more steps when precision is important, or fewer steps in compute constrained or latency sensitive environments (like edge deployments). They conduct extensive evaluations to show how SORL achieves better performance compared to existing baselines and also validate their inference time scaling hypothesis. That is, allowing for more denoising steps at inference generally leads to better performance. This is in line with the recent literature studying test time scaling in LLMs and is very useful for practical deployments of offline RL methods.", "questions": "- I am unsure of the range of $h$ that is used to define the self consistency loss in Eq 5. It seems to be used to both define the number of steps ($\\in [1, M]$)  as well as the dt time interval ($\\in [0, 1]$) that the shortcut model is conditioned on. The text mentions (in L110) that $h \\in [1, M]$ and in L111 $h \\in ( 2^k )_{k=0}^{log_2M}$, where $M$ is the number of discretisation steps. But then shouldn't the $h$ in the flow matching part of $\\mathcal{L}^S$ be 1 and not $1/M$? Since the flow matching loss is for one step. But L112 defined the flow loss as when $h=1/M$ meaning h refers to the dt?\n- In Algorithm 1 is there a reason that you compute the component losses of the actor on separate sections of the batch? That is the Q maximisation loss is only on the first third, the self consistency only on the second third and so on?\n- Also in Algorithm 1, isn't the sign of the grad of the Q term inverted, and it is missing the flow matching term too I think?\n- It might be easier to understand Fig 2 if the x axis was grouped by $M^{BTT}$. That way the gains from increasing $M^{inf}}$ for a fixed training budget (fixed $M^{BTT}$ would be easier to see. Maybe the figure could be split into two, one which tests for scaling inference steps for lower training $M^{BTT}$ and another which tests generalization to inference steps beyond what was used in training.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Comprehensive evaluation against relevant baselines. The paper presents results on a variety of control tasks and show that SORL beats strong baselines in challenging tasks.\n- The paper validates the claim of inference time scaling, with experiments showing better performance as $M^{inf}$ is increased.\n- Shortcut models preserve the expressivity of the network while also being suitable for RL optimisation without needing a separate distillation step as in FQL.\n- The method is well motivated and easy to implement. \n\nWeaknesses\n- There are some notations and figures which are unclear. See questions below.\n\nOverall I have no major concerns with the paper.", "was_revised": true, "final_justification": "The paper is well motivated and presents strong results. Integrating RL with expressive generative models is of relevance to the RL and robotics community. For these reasons, I believe the paper should be accepted.", "rebuttal": "We thank the reviewer for their comments and respond to their questions below.\n\n# 1. Questions\n> I am unsure of the range of $h$ that is used to define the self consistency loss in Eq 5. It seems to be used to both define the number of steps ($\\in [0, 1]$) as well as the dt time interval ($\\in [0, 1]$) that the shortcut model is conditioned on. The text mentions (in L110) that $h \\in [1, M]$ and in L111 $h \\in \\left ( 2^k \\right )^{\\log_2 M}_{k=0}$, where $M$ is the number of discretisation steps. But then shouldn't the $h$ in the flow matching part of $\\mathcal{L}^S$ be 1 and not $1/M$? Since the flow matching loss is for one step. But L112 defined the flow loss as when $h=1/M$ meaning h refers to the dt?\n- Thank you for the comment. We will correct the definition of shortcut models to avoid overloading notation and avoid the confusion. The last parameter passed into the shortcut model s_theta should be the _step size_. \n\n> In Algorithm 1 is there a reason that you compute the component losses of the actor on separate sections of the batch? That is the Q maximisation loss is only on the first third, the self consistency only on the second third and so on?\n- In practice, all components in our implementation use the same batch. The separate data samples in Algorithm 1 was a mistake in our pseudocode. We will update the algorithm’s pseudocode to reflect the implementation.\n\n> Also in Algorithm 1, isn't the sign of the grad of the Q term inverted, and it is missing the flow matching term too I think?\n- Yes, we will update the expression accordingly. \n\n> It might be easier to understand Fig 2 if the x axis was grouped by $M^{\\text{BTT}}$. That way the gains from increasing $M^{\\text{inf}}$ for a fixed training budget (fixed $M^{\\text{BTT}}$ would be easier to see. Maybe the figure could be split into two, one which tests for scaling inference steps for lower training and another which tests generalization to inference steps beyond what was used in training.\n- Thank you for the suggestion. We will update the plots accordingly."}, {"review_id": "TxYBINrAGy", "reviewer": "Reviewer_Ewda", "summary": "This paper studies the problem of applying diffusion and flow models to the offline RL problem. The authors introduce a method called Scalable Offline Reinforcement Learning (SORL) which leverages *shortcut models* (a novel class of generative models) for scalability in training time and inference time. The authors establish theoretical results that the shortcut models output a distribution that is close (in Wasserstein) to the data distribution (and thus, that the offline RL policy is regularized towards the behavior policy). Experiments are conducted.", "questions": "1. What is the point of the $Q$ loss $L_{QL}(\\theta)$ in Eq. 9?\n2. Can you provide more analysis of why certain environments didn't benefit from this approach?\n3. What are the actual benefits of this method in terms of total wall-clock time?", "limitations": "See above", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "At a high level, this paper amounts to \"importing\" shortcut models into RL by using them as a policy class in an otherwise standard RL pipeline. Despite the lack of algorithmic novelty, the approach is well-motivated and the authors are convincing about the benefits of this approach, namely that it can simultaneously give expressive multi-modal policies, provide efficient training, and achieve inference-time training. The experimental results are mixed but mostly strong and the theoretical analysis of shortcut models is novel (from what I can tell) and may be of independent interest. The shortcut model analysis itself is not a complete analysis of the SORL method (it only analyses the regularization term by itself). For the experimental results, it would have been more convincing if total wall-clock time was measured, since the authors claim that faster training and inference is possible with this method. \n \nTypos: \n1. Algorithm 2: $n$ should be $t$\n2. Critic loss is missing an expectation over data\n3. Line 192: \"we prove that the is *yes*.\"", "was_revised": false, "final_justification": "", "rebuttal": "# 1. Typos\n> Algorithm 2: n should be t \n\n> Critic loss is missing an expectation over data\n\n> Line 192: \"we prove that the is yes.\"\n\n- Thank you for the comments. We will update the paper accordingly.\n\n# 2. Q Update in Eq. 9\n> What is the point of the loss in Eq. 9?\n- The **purpose of the Q loss is to update the policy in a way that selects actions with a higher Q value** (that is, a higher expected return). This is the “RL optimization” component of our loss. The flow matching loss serves as our “regularizer” to the offline data.\n\n# 3. Analysis of Performance\n> Can you provide more analysis of why certain environments didn't benefit from this approach?\n- **The key advantage of SORL over Gaussian-based methods (e.g. ReBRAC) is its more expressive policy class.** This policy class is better able to handle diverse, multi-modal data than Gaussian-based models, which cannot handle multi-modality. The key advantage of SORL over distillation-based generative model approaches (e.g. SRPO, CAC, FQL) is SORL’s one-stage policy training procedure, as compared to the two-stage distillation-based approaches over which error compounds. **SORL’s flexibility in selecting the number of discretization and inference steps allows for less discretization error than a one-step approach**. We believe this contributes to more precise actions at test-time. \n- The environments where SORL struggled (antmaze-giant and humanoidmaze-large) are very challenging tasks in offline RL, and it should be noted that none of the 10 baselines we considered performed well on these environments. SORL always achieved at least the second best performance among the 10 baselines considered. On the two environments where SORL does not achieve the best performance among the baselines, there is only one algorithm that outperforms it (ReBRAC on antmaze-giant and IFQL on humanoidmaze-large). Both ReBRAC and IFQL fail to achieve the best performance on any other environment, highlighting the difficulty of having reaching optimal performance on every environment.\n- At a higher-level, the antmaze-giant and humanoidmaze-large tasks are known to be challenging, long-horizon environments, and **it is possible that these environments would benefit significantly from online exploration**. As noted in the limitations discussed in Appendix B, SORL does not have a principled exploration strategy, which is standard for _offline_ algorithms. Depending on the coverage and quality of the offline dataset, purely offline learning may get stuck in local optima. \n- Relatedly, **it is possible that policy extraction is not the primary bottleneck in these two environments** under the given dataset size (1 million transitions). It may be the case that, for these two environments, value learning is challenging. If the learned value function (e.g. Q function) is poor, then it will be difficult to optimize a policy under that value function, irrespective of the policy extraction method, expressivity of the policy class, or discretization/integration error. \n\n# 4. Runtime\n> What are the actual benefits of this method in terms of total wall-clock time?\n- In Appendix C.2, **we provided a runtime comparison against the fastest baseline** FQL [Park et al., 2025], which **we decompose into training and inference**. We show that SORL offers flexibility under varying compute budgets: SORL can take advantage of greater computation during both training and inference, or opt for a faster training/inference procedure."}]

Author rebuttal:

Meta-review: This paper proposes SORL, an offline RL algorithm that integrates shortcut models with flow-matching, Q-learning, and inference-time scaling. The reviewers generally agree that the paper is clearly written, well-motivated, and supported by a broad set of experiments. They find the approach timely and relevant, with strong empirical results and a compelling evaluation.

That said, the AC finds the novelty to be limited. The use of generative models (e.g., diffusion and flow matching) to define more expressive policy classes than Gaussian-based policies was pioneered by earlier work such as Diffusion QL. Likewise, diffusion distillation techniques are not new, and one-step training (as opposed to pretraining followed by distillation) has already been studied in the vision domain. The combination of Q-learning loss, diffusion/flow-matching objectives, and shortcut models therefore does not constitute a fundamentally new direction, though applying this combination to offline RL is a useful and relevant contribution.

The AC also has concerns regarding the authors’ claim that “The key advantage of SORL over distillation-based generative model approaches (e.g., SRPO, CAC, FQL) is SORL’s one-stage policy training procedure, as compared to the two-stage distillation-based approaches over which error compounds.” In the vision domain, state-of-the-art one-step generation performance (for unconditional, label-conditional, and text-to-image diffusion models) remains firmly achieved by score-distillation methods, not by training one-step models from scratch. If the authors’ explanation were fully correct, this would contradict established trends in vision research. The AC therefore encourages the authors to revisit this claim and provide a more careful justification for why SORL shows stronger empirical performance than existing pretraining-plus-distillation approaches.

In summary, while the paper does not offer substantial theoretical or algorithmic novelty, it tackles an important and timely problem and demonstrates strong empirical results. On balance, the AC considers this a relevant contribution to the offline RL community, but advises the authors to moderate claims about the source of performance gains in the final version.
Decision: Accept (poster)