ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulation
Authors: Songyuan Zhang
Oswin So
H M Sabbir Ahmad
Eric Yang Yu
Matthew Cleaveland
Mitchell Black
Chuchu Fan
Venue: ICLR 2026 Poster (2026)
Keywords: Offline reinforcement learning
support constraint
flow model
Primary area: reinforcement learning
TL;DR: Considering offline RL, we propose ReFORM, a two-stage flow policy that realizes the support constraint by construction and avoids the OOD issue without constraining the policy improvement.

Abstract: Offline reinforcement learning (RL) aims to learn the optimal policy from a fixed
dataset generated by behavior policies without additional environment interactions. One common challenge that arises in this setting is the out-of-distribution
(OOD) error, which occurs when the policy leaves the training distribution. Prior
methods penalize a statistical distance term to keep the policy close to the behavior policy, but this constrains policy improvement and may not completely
prevent OOD actions. Another challenge is that the optimal policy distribution
can be multimodal and difficult to represent. Recent works apply diffusion or
flow policies to address this problem, but it is unclear how to avoid OOD errors
while retaining policy expressiveness. We propose ReFORM, an offline RL method
based on flow policies that enforces the less restrictive support constraint by construction. ReFORM learns a behavior cloning (BC) flow policy with a bounded
source distribution to capture the support of the action distribution, then optimizes
a reflected flow that generates bounded noise for the BC flow while keeping the
support, to maximize the performance. Across 40 challenging tasks from the OGBench benchmark with datasets of varying quality and using a constant set of
hyperparameters for all tasks, ReFORM dominates all baselines with hand-tuned
hyperparameters on the performance profile curves.

Published as a conference paper at ICLR 2026
REFORM: REFLECTED FLOWS FOR ON-SUPPORT OFFLINE RL VIA NOISE MANIPULATION
Songyuan Zhang! Oswin So
H. M. Sabbir Ahmad
Eric Yang Yu!
Matthew Cleaveland" Mitchell Black
Chuchu Fan
MIT
Boston University "MIT Lincoln Laboratory
(szhang21, oswinso, eyyu, chuchu @mi t. edu $sabbir920bu.ce (matthew. cleave land, mitche 11 black) all.mit.edu
ABSTRACT
Offline reinforcement learning (RL) aims learn the optimal policy from a fixed dataset generated by behavior policies without additional environment interac- tions. One common challenge that arises this setting is the out-of-distribttion (OOD) error, which occurs when the policy leaves the training distribution. Prior methods penalize a statistical distance term to keep the policy close to the be- havior policy, but this constrains policy improvement and may not completely prevent OOD actions. Another challenge is that the optimal policy distribution be multimodal and difficult represent. Recent works apply diffusion flow policies to address this problem, but it is unclear how to avoid OOD errors while retaining policy expressiveness. We propose FRM, an offline RL method based on flow policies that enforces the less restrictive support constraint by con- struction. ReFORM learns a behavior cloning (BC) flow policy with a bounded source distribution to capture the support of the action distribution, then optimizes a reflected flow that generates bounded noise for the BC flow while keeping the support, to maximize the performance. Across 40 challenging tasks from the OG- Bench benchmark with datasets of varying quality and using a constant set hyperparameters for all tasks, ReFORM dominates all basclines with hand-tuned hyperparameters on the performance profile curves.
INTRODUCTION
Offline reinforcement learning (RL) trains an optimal policy from a previously collected dataset without interacting with the environment (Levine al. 2020). This technique is especially useful in domains where large datasets are already available and environment interactions are expensive and potentially unsafe (Fu et al.) (022)). However, there are two major challenges. First, the lack of online exploration makes the distribution shift especially dangerous. That is, for out-of-distribution (OOD) actions not represented the dataset, the learned Q-function can produce overly optimistic estimates that lead the policy astray (Levine al., 2020). Second, traditional policy classes are typically represented using a unimodal distribution such as a Gaussian (Kumar et al. 2020 Tarasov et al., 2023). whereas more complex offline datasets and tasks require multimodal action distributions.
Prior works attempt to address the OOD issue by keeping the Icarned policy close to the behavior policy by regularizing statistical distance the behavior policy (Wang al., 2018 Peng al., 2019 Mao et al., 2023a Kumar et al.) 2019 Wu et al.) 2019). However, selecting a distance mea- surement along with an appropriate regularization weight can be difficult depending on the task and dataset. Perhaps the most common type of statistical distance used is the Kullback-Leibler (KL)
divergence (Wang et 2018 Peng al. 2019 Wu al. 2019 Jaques et al. 2019 Siegel al. 2020 Nair et al. 2020; Wang et al.) 2020 Kostrikov et al. 2022; Park et al. 2025b), which can avoid the OOD issue but can also be too restrictive and produce an overly conservative policy. For example, if the dataset has low density on the optimal behavior, the KI. divergence regularization
will encourage the learned policy to be suboptimal. Similar works (Wu et al.) 2019 Kumar et al.)
Prooject website: https: h //mtt realm. gi github. 1o/reform/ DISTRIBUTION STATEMENT Approved public release. Distribution is unlimited.

---

Published as a conference paper at ICLR 2026
2019) have considered alternative statistical distances such as the Wasserstein and MMD distances that have been shown to improve performance on certain tasks. However, these methods do not completely prevent OOD actions, and the need to choose regularization weight remains problem.
To tackle the challenge of multimodal action distributions, recent works have proposed using diffu- sion policies (Hansen- et al. 2023a) and flow policies (Park et al., (025b) model complex action distributions in the dataset. However, it remains unclear how to address the OOD issue with these highly expressive function classes without hurting their expressivity.
In this work, we propose Reflected Flows for On-support offline RL via noise Manipulation (ReFORM), an offiine RL method that aims to address both above issues by constraining a flow policy using the less restrictive support constraint. Rather than regularizing the learned policy via a statistical distance, we only require the actions produced to stay within the support of the action distribution of the behavior policy. ReFORM learns behavior cloning (BC) flow policy from the dataset, and additionally learns reflected flow (Xie et al.] (024) noise generator that manipulates the source distribution of the BC policy within its support. This approach enables us to realize the support constraint by construction without regularization, therefore avoiding the need to specify any regularization weights. In other words, our method bypasses the hyperparameter sensitivity issue by having constant hyperparameters. To summarize our contributions:
We propose ReF ORMO a two-stage flow policy that realizes the support constraint by construc- tion and avoids the OOD issue without constraining the policy improvement.
We propose applying reflected flow to generate constrained multimodal noise for the BC flow policy, thereby mitigating OOD errors while maintaining the multimodal policy.
Extensive experiments on 40 challenging tasks with datasets of different qualities demonstrate that, with a constant set of hyperparameters, ReF ORM dominates all baselines using flow policy structures with the best hand-tuned hyperparameters on the performance profile curve.
RELATED WORK
Distributional shift mitigation in offline RL. A fundamental challenge of dynamic programming methods in offline RL is the OOD problem, where the learned policy tries to exploit erroneous Q- values from extrapolation error (Levine et al.) 2020) and generates OOD actions. Consequently, many offline RL methods have proposed to constrain or penalize the statistical distance between the learned policy and the behavior policy, either with an additional loss term or by regressing to the estimated optimal policy, to mitigate this distribution shift issue. Examples include using the maximum mean discrepancy (MMD) distance (Kumar et al. 2019), Wasserstein distance (Wu al.
2019) and KL divergence (Wang et al. 2018 Peng et al. 2019: Wu et al., 2019; Jaques et al.) 2019
Siegel et al.) 2000 Nair et al.]]2020) Wang et al., 2020 Kostrikov et al. 2022 Park al. (0256). One key challenge with these methods is that the amount of regularization is hyperparameter that needs to be tuned for each task and dataset (Park et al. 2025a b) and can significantly affect the method's performance. Moreover, argued by Kumar et al. (2019). constraining the divergence can be too restrictive in cases where optimal actions happen with very low probability under the behavior policy. Another family of methods uses the support of the behavior policy, either by regularizing the policy (Kumar et al. 2019] Wu et af. 2022, Mao et al.]
2023a Zhang et al.) (203),, or via
regularizing the Q-function outside the support (Kumar et al. 2020 Lyu al., 2022 Mao et al.)
2023b. Cen et al. 2024)). Our work falls in the category of enforcing support constraints on the
learned policy. However, instead of approximating the support constraint by a suitably designed regularization term, our method enforces the support constraint by construction by optimizing in the BC policy's 's (bounded) latent space.
Fine-tuning flow-based models for offline RL. BC methods using diffusion models (Sohl-
Dickstein et al., 2013 Ho et al. 2020 Song et al. 2020) or flow matching (Lipman et al. 2023
Liu et al. [0022] Albergo & Canden-Hijnden 2022) have seen increasing use in the control and
robotics communities (Chi et al. 2023 Reuss et al. 2023 Pearce et al. 2023 Wang et al.) 2023) However, since BC aims to mimic the dataset, its performance is tied to the performance of the behavior policy. To fix this, one can consider tneettuing the learned flw-based model to maximize user-supplied reward function. Following the success of fine-tuning blow- models for image generation (Uchara et al. 2024, Black et al., 2024 Bammigg-EEnrice et al. 2024), fine-tuning has
also been applied the offline RI. setting Clansen-Fsttuud et al.] 2023a Chen et al. 2024 Park
2

---

Published as a conference paper at ICLR 2026
et al., 2025b Ding & Jin 2024 Zhang et al. 2025). However, almost all fine-tuning methods for offline RL tackle the problem of distribution shift with an additional loss term penalizing statistical distance from the behavior policy, with the weight of this term being sensitive hyperparameter that needs to be tuned for each task and dataset (Park et al., 2025b).
Latent space optimization in generative modeling. Instead of fine-tuning the flow model directly, another line of work considers optimizing the distribution in the latent space, i.e., initial noise, of the generative model. In the context of image generation, methods that optimize the initial noise using
either regression (Li et al., 2025; Guo et al. [024] Zhou et al. 2024] Ahn et al. [024]] Eyring et al., 2025 or RL (Miao al., 2025) have found success in improving the quality of generated images. In RL, Singh et al. (2021) explored using normalizing flows (Dinh et al. (016) to improve exploration in online RI., Since offline RI. was not the focus of their work, they not restrict the output of their learned latent space policy. Consequently, the policy can output unbounded and potentially OOD samples in the latent space, which is harmful in the offline RL. setting. Recently, [Zhou et al.) (222]) and Wagenmaker et al. (2025) have applied this idea to offline RL for Conditional Variational Autoencoders (Sohn et al.J (2015) and diffusion policies, respectively, but they additionally restrict the latent space policy to fixed action magnitude. Here, the action magnitude roughly controls how likely the latent action is under the behavior policy, playing a similar role to the statistical distance regularization coefficient in existing offiin RL works. As we will show in Section [5] the final performance is quite sensitive to this hyperparameter, which varies on different tasks and different datasets. In contrast, our proposed method does not have any such hyperparameters that play similar role, removing the need for adapting them each time the environment or dataset changes.
3 PRELIMINARIES
Offinee RL. Let A(Y) be the set of probability distributions over space X, and denote place- holder variables with gray. A Markov Decision Process (MDP) is defined by a tuple M (S, A. Po. p,)), where S is the state space, A C Rdd is the d-dimensional action space, r(s,a):
S - ER is the reward function, Pa E (S) the initial state distribution, P(s'ls,)) S x A A(S) is the transition dynamics, and y E [0. 1] is the discount factor. Given dataset of N trajec-
tories D = = T2,... TN) generated by some behavior policy <=((]s) : - A(A), where т; = (8o: 0o, 81: 01 ................................ SH. (H.) the goal of offline RL is to find policy To (a((ss) S - ((A) parame- terized by that maximizes the expected discounted return R(TT) = EExppe (r) 1h (s an)], where p**(r) (r) po (sa) NTO (ag) so) ((s (so 00) ................................ wo (a (sH) Note that in the offline RL setting,
sampling in the environment with policy my is not allowed.
OOD actions are key challenge in offline RL (Levine et al. 2020). Many actor-critic methods learn the policy-condee conditiood state-action value function (i.e., Q-function) Q(s, a) S X A + R. For a policy Te, this is defined as
H
QTh (s. E
(r (shh so s, ag a. an (Q())). Vh 2
corresponding to the expected discounted return obtained by applying action from state s then following policy тy. For parameters o, Qa is commonly learned with fitted Q evaluation using the TD error
((Ø)= Eegga, D D'aa''' (175,))++777)66 Q3) -f
(2)
where Q5e is target network (e.g., with soft parameters updated by polyak averaging (Polyak & Juditsky 1992)). However, if the policy To samples OOD actions a', the target Q5e can produce an erroneous OOD value and cause the learned policy to incorrectly optimize for the OOD value (Levine et al.) 2020). To address this issue, many offline RL methods regularize the statistical distance between the learned policy and the behavior policy (c.g., with the KL divergence (Peng (Hansen-Estruu et al. 2023b) or Wasserstein distance (Wu et al. Park et al. 2025b)), resulting in the following objective for policy improvement:
al. 2019 Fujimoto & Gu |
2019
L())= = E -..DD
2 (8, + aDTT [3)
(3)
3

---

Published as a conference paper at ICLR 2026
where (--ll-) is some statistical distance, e.g.. DKL for KL divergence or Dwz for the Wasserstein distance. However, this regularized objective introduces an additional hyperparameter a that needs to be hand-tuned for each experiment (Park al. b).
Flow matching. Flow matching (Lipman et al. 2023 Liu et al. 2022, Albergo & Vanden-Lijnden
2022 has recently become an increasingly popular way of training flow-based generative models.
Given a target distribution p(x) E (R flow matching learns a time-dependent velocity field
a(f that transforms a simple source distribution g(2) (e.g. standard Gaussian A/ Id) at t
to the target distribution p(æ) at t = 1. The resulting flow 2)), [0, 1] x R - F mapping
samples from the source a q to the target (11,)) p. then the solution to the ordinary
differential equation (ODE)
म. ((,z)) =((,,)=((((,,))) (0,2) r.
Flow matching is a simple yet powerful technique alternative to denoising diffusion (Ho et al. 2020) capable of generating complex multimodal target distributions.
METHOD
To solve the problem of OOD actions, any given state s, the chosen action should be con- strained to lie within the support supp(Tx(-Is)) = {a Tz(a|s) > 0} of the behavior policy тş- However, constraining common statistical distances, such the KI. divergence or the Wasserstein distance, theoretically leads to problems from the perspective of support constraints We provide the following theoretical results with all proofs provided in Appendix A
First, constraining the KL divergence is a sufficient but not necessary condition to enforce support constraints (Kumar et al. 2019. Mao et al. 2023a). Formally, we have the following result:
Proposition 1. Given state 8 E S, for any such that 0 I < CC, P || <r(+(s)) S implies supp(ra(-(s) S supp( (-1s)). On the other hand, for any M > O, there exist distributions Ty and hT such that supp( (-(s) (s)) C supp(TT(-Is)) but DKL (Ta( || TT(-ss)) > M.
Proposition T tells us that the KL divergence constraint more restrictive than the support constraint. This additional restriction has been found to impede the performance improvement of To over xy (Mao et al. 2023a). While this issue can be alleviated with small a in (3), in practice, this can result in OOD problems due to estimation errors (Levine al. 2020).
Wasserstein distance is another statistical distance used by previous works. However, constraining the Wasserstein distance cannot enforce support constraints despite its strong empirical performance in offline RL (Park et al. 20256). Formally, we have the following result:
Proposition Given a state s E S, suppose that supp(r=(-1s)) * A. Then, for any 6 > O, there exists a policy Th such that supp (T(-(s) g supp(TT((-ss) but Dw2((T(-1s)) TTp-(s)) S
Motivated by the above thcoretical challenges of the KL divergence and Wasserstein distance in addressing the issue of OOD actions, instead consider the following support-constrained policy optimization problem to tackle this issue directly.
H Σ
7 r and
max R(TO) ET
(5a)
s.t. supp(Tr(-1s)) S (1pp(
E.
(5b)
Unfortunately, enforcing the support constraint 5b)) is a challenging problem since (i) accurately es- timating the (upp(Taa(1s)) (Grover et al.) 2018). and (ii) enforcing supp(TT(-|s)) C supp(Tx(((s)) given an estimate of supp(=r((1s)) (Zhang et al. 2023). are both nontrivial to solve for.
To tackle these problems, we propose learning a BC flow policy vhe (t, 2; that transforms source distribution QBC into a state-conditioned target distribution (Bc(a])) ) RJ TT((ass) In particular, we
bby interpreting the constant as Lagrange multiplier, regularization with fixed coefficient as in he interpreted equivalently enforcing constraint (Teevine et al. 2020)
4

---

Published as a conference paper at ICLR 2026
BC flaw original source 9mx
BC flow original target Poc
BC
Semples -D
Rei0
BC flow policy Ve,
Dietill
BC flow manipulated source Quc
BC Row manipulated target
Q-value
Noise generator flow Ve
One-step policy μa,
Maximize
Figure 1: ReFORM algorithm. The process with gray arrows indicates the BC flow policy, learned transform simple source distribution (BC ((BBg) a target distribution PBC that matches the dataset DD. The blue arrows indicate the ReFORM process, where we learn flow noise generator to generate a manipulated source distribution ÿaC for the BC policy so that the manipulated target ýBC maximizes the Q value while staying inside the support (denoted in red) of the BC policy.
use a QBC with bounded support such that Supp( can be approximated by the image of supp(gBc)
under the BC flow. One benefit of this approach that this enables learning a policy that satisfies the support constraints by construction by taking advantage of the property that for any sample e supp(gec) within the (bounded) source distribution's support, Ve, (1, s) supp(pnc(((s)) R supp(Taa( Hence, we propose to construct the policy Ta as the composition of some noise generator with the BC flow Vor the generated noise distribution ¹BC has the same support as QBC. i...
supp(ānc) S. supp(?ne)
(6)
then the pushforward of ÂBC under the, naturally satisfies the support constraints (5b). With the support constraint (5b) satisfied by construction, solving the support-constraing policy optimization problem (5) reduces to performing unconstrained optimization of the objective (5a).
Remark 1. This idea of outputting noise not new. Prior works have proposed similar "noise manipalatioonstterrnng techniques for fine-tuning diffusion models and flow models (Li et al.) [025 Guo et al. 2024 Miao et al.) [202 Wagenmaker al. 2025). One key difference that we choose the source distribution of the flow model to be a distribution with bounded support, which enables better approximation of the support of πz. Moreover, we propose a different form of the noise generator Qnc than prior works that maintains the high expressivity of flow-based policies.
We call our method ReFORM, which we summarize in Figure 1 In the following subsections, elaborate on each of these components in detail.
4.1 BASED BEHAVIOR POLICY LEARNING
ReFORM begins by learning BC flow policy that transforms the source distribution QBC PBC(-Is), which approximates (-(sss). We choose чBC = (BB)),, the uniform distribution over the d-
dimensional hypersphere with radius so that
supp(1pc) = Bdd = (2 E Rd
|||||| 1).
We discuss the choice of in Appendix
To learn the BC flow policy the we learn its corre- B S - d parameterized by 0 such that solving
sponding velocity field ve, ()) [0.
the ODE (4h gives actions the (1, 2; for z ~ QBC. We apply a simple linear flow for learning
the velocity field following Park et al. (2025b) with loss
LBC((1)
[t, 21;8) (a -
(8)
where -1 t)) + ta is the linear conditional probability path.

---

Published as a conference paper at ICLR 2026
4.2 REFLECTED FLOW-BASED NOISE MANIPULATION
A key component in enforcing the support constraints as proposed above is the use of a noise gener- ator with the same support as the BC flow-policy's source distribution gBC- Prior works that apply similar "noise manipulation" or "noise steering" techniques implement the generated noise `BC as a truncated Gaussian (e.g.. by clipping or squashing with tanh). However, the use of a unimodal ÂBC severely limits the expressiveness of ÔBC and thus also that of the resulting learned policy TT.
One way to improve the expressiveness by replacing the Gaussian distribution with a flow-based generative model, as has been done with the actions. We propose to do the same, but to the noise instead. Specifically, we choose to use a flow noise generator Vo, (t, wx; s) [0. 1] B S - B and denote its associated velocity field as Vo, (t, wgs)) s) [0. 1] By x S + R However, the support of a flow-hased generative model is generally unconstrained, which violates our requirement on the support of ÂBC (6). To resolve this, we propose to use a reflected flow (Xie et al., [004)) which
can be used to guarantee that samples from Ve, are contained within supp(gBc) by considering the following reflected ODE (Xie et al. (004) instead of (4):
Vo, (0, w; = w, where the reflection term dL,, compensates the outward velocity at a supр((gBC) by pushing the motion back to supp()sc) (Xie et al. 2024).
dr/e, (t, w; 8) ve, (t, Ve, (t, w; s); s) + ddr,
(9)
For convenience, let Mol (2; voo (1, s) and peg (w;s) vo, (1, w; s), and let po(w;;) s) le, (He, s) denote their composition. We optimize the noise generator the, to maximize the expected Q-value of the learned policy μs with the following loss
(f2) = E (.)-)(()))) Qt9 pe,
pe, (Ho, (w; s)]]
(10)
noting that the parameters of the BC policy 01 stay fixed when optimizing 02.
We have yet to specify the reflection term dT., in (9), as many choices of dL, constrain the ODE to remain within supp(gBc). In particular, we wish for the reflection term dL, to be robust to
numerical integration. Fortunately, supp(pBc) = B being a hypersphere ( simplifies this design.
Consider solving the normal ODE using the popular Euler method:
2k+11 2 + V0_ (kAt, wi St, E IN.... - 1}, vee (1, + EN,
(11)
where N is the number of integration steps, At= N. and 2g = w. For the reflected case (9. we propose modifying the Euler method (TT) by performing a projection back into the hypersphere after every Euler step. This gives us the following reflected Euler method
1k+1 1( E BB] 2441 + (1 1(2x+1 E )( (ax+1 (vo, (k&St, w; s) ne+ 1
(12) Es and (:, is the where êx+1 = 2/ ug, (k&t, wi s) At follows the original Euler step, n& |
inner product. We then propose to choose dL, that defined implicitly by the above procedure. Note that (12) has the same complexity as (11) because (12) only contains one step projection.
For this to be a valid reflected flow, samples 2 from the proposed reflected Euler method [22 should satisfy the desired support constraints E supp(qnc) = B which we formally state below.
Theorem 1. The target distribution of the noise generator stays within the support of the original source distribution of the BC policy, i.e., supp(ğиc) S supp(qsc).
Combining Theorem Twith the ideas from above then allows to formally prove that the resulting action distribution stays within the support supp(pBC) and hence does not result in OOD actions: The manipulated target distribution ÿBc of the BC flow policy remains within the support of the original BC policy, ie., supp(fnc) S supp(pBc).
Theorem 2.
Theorem 2 guuarantees that the learned policy provably avoids OOD actions without any regulariza- tion terms. This avoids the need for costly hyperparameter tuning for each environment and dataset, and also does not impede the potential improvement of the learned policy.
4.3 POLICY DISTILLATION
One drawback of our proposed method is that computing the gradient of the actor loss VoLNG 10 requires computing the gradient V200,, which involves a long backpropagation through time

---

Published as a conference paper at ICLR 2026
(BPTT) chain since Há, is evaluated with Euler integration. To reduce the computational burden,
follow Park et al. (2025b) and distill (Salimans & Ho 2022 Geng et al. 2023 2025) the learned BC
flow policy by learning a one-step policy Pa. (2; B x A parameterized by 0 that directly
maps the latent variable & to the action a with the following distillation loss:
L (01) = E ES |IAA,, (z: - no, (2; s)[
(13)
5 EXPERIMENTS
In this section, we conduct experiments to answer the following research questions. Additional details for our implementation, environments, and algorithm hyperparameters, and full results with more ublations are provided in Appendix C
(Q1): How does ReF ORM perform compared to other offline RL algorithms with flow policies?
(Q2): Does ReFORM avoid the OOD issue without limiting the performance improvement?
(Q3): Is it necessary for the BC policy's s source distribution to have bounded support?
(Q4): Is the reflected flow necessary for generating the targeted noise?
(Q5): How is our design of the reflection term?
(Q6): Is the distillation of the BC flow policy necessary?
5.1 SETUP
Environments. We evaluate ReFORM and the baselines 40 tasks from the OGBench offline RL benchmark (Park et al., (025a) designed in 4 environments, including locomotion tasks and manipulation tasks. We use two kinds of datasets, CLEAN and NOISY. The CLEAN dataset consists of random environment trajectories generated by an expert policy. The NOISY dataset consists of random trajectories generated by highly suboptimal and noisy policy.
Baselines. We compare ReFORM with the state-of-the-art offline RL. algorithms with flow policies.
including Flow Q-Learning (FQL) (Park et al., 2025b). Implicit Flow Q-Learning (IFQL) (Park
et al. 2025b), and Diffusion Steering via RL (DSRL) Wagenmaker et al.) 2025). Since FQL s performance highly depends on the a hyperparameter (Eq. (3)) we consider three variants of FQL:
FOL (M) uses the a" that is hand-tuned for each environment using the CLEAN dataset by Park jet al. (2025b), (S) uses & = a" a10 and FQL (L) uses a = 10 a². IFQL is the flow version of IDQL (Haasen-Estruch et al., 2023b) implemented in Park et al. (2025b). For DSRL, we use the hand-tuned noise bound by Wagenmaker et al. (2025). Note that ReF ORM uses the same
hyperparameters across all tasks.
Evaluation Metrics. We run each algorithm with 3 different seeds for each task and evaluate each converged model on 32 different initial conditions. We define the normalized score for each task as the return normalized by the minimum and maximum returns across all algorithms.
5.2
MAIN RESULTS
(Q1): ReFORM achieves the best overall performance with a constant set of hyperparame- ters. As recommended by Agarwal al. (02T)) we plot the performance profile over all tasks with different datasets in Figure [ It is clear that ReFORM achieves the best performance for both the CLEAN and NOISY datasets. For the CLEAN dataset, DSRL and FOL (M) achieve the second and third best respective performance because their hyperparameters are specifically hand-tuned for these environments. However, for the NOISY dataset, the performance of both DSRL and FQL (M) drops significantly, whereas FQL (S) becomes the second-best method behind ReFORM. This high- lights the hyperparameter sensitivity of the baseline methods. Moreover, we observe that when the behavior policy performs poorly (i.e., on NOISY), a stronger density-based regularization impedes the ability of the learned policy to improve (see FQL (L)).
Importantly, ReFORM achieves the highest fraction normalized scores close 1, indicating that ReFORM does not limit the improvement of the learned policy as discussed in Section 4 The use of support constraint allows the learned policy to apply any action with the support, including ones
7

---

Published as a conference paper at ICLR 2026
Dataset:
CLEAN
Dataset: No1av
Fraction tasks with score
1.00
BaFORM
Al 0.75
DBAL
0.50
FQ1.00
0.25
(QL(L)
0.00
IFQL
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Normalized Score (7)
Normal ized Score (T)
CLEAN and NOISY datasets. For given normalized score (X-axis), the performance profile shows the probability that a given method achieves a score 2 (see Agarwal al. (202T) for details). On the CLEAN dataset, ReFORM achieves greater scores with higher probabilities than all other baselines. The same is true on the NOISY dataset except for small set of normalized scores around 0.9 where ReFORM and FOL (S) have similar probabilities within the statistical margins.
Figure 2: Performance profile
(a) BC.
(b) DSRL.
(c) IFQL.
(d) FOL (S).
(c) FQL (M)
(f) FOL (L).
(g) ReF ReFOR
Figure 3: Learned policy distributions with the toy example. The Q-value reaches the maximum at the lower left and upper right corners (See the Q-value plot in Figure I.. The red boundaries denote the estimated supp(#BC))
that have low density under the behavior policy To Therefore, the learned policy does not suffer from a performance upper bound related to the behavior policy.
(Q2): ReFORM maximizes the performance while avoiding OOD. We design a toy example to better visualize and compare the learned policies. The toy example has a 2-dimensional action space with a Q-value that grows when approaching the lower left and the upper right corners (see the Q-value plot in Figure D. We plot the policy distributions of BC and all algorithms in Figure 3 ReFORM maximizes performance by reaching both corners while staying within the support of the BC policy. DSRL collapses to single mode in the upper right corner and remains far from the boundaries of the support because the generated noise of DSRL is unimodal and squashed. We compare the generated noise in more detail in Appendix C.4 Figure 16. IFQL remains similar to BC policy because importance sampling less efficient for finding the maximum. FQL faces OOD error due to its use of Wasserstein distance regularization (as discussed in Proposition
5.3 ABLATION STUDIES
To study the functionality of each component ReFORM, we conduct the following experiments in toy environment and the cube-single environment with the NOISY dataset answer Q3-Q5. All details can be found in Appendix C.3.3
(Q3): Having bounded support for the BC flow policy's source distribution is crucial. We investigate the effect of satisfying support constraints (and hence the necessity of using a source distribution with bounded support) by using Gaussian (0, Id) with unbounded support as the source distribution for both the BC flow policy and the flow noise generator following Wagenmaker
al. 20025 (ReFORM (U)) Figure 4 shows that ReFORM (U) suffers from severe OOD problems
"This plotted support slightly differs because = (x(;)) for ReFORM, but QBC = (0, 1²) for others. *The support estimation has numerical errors, few samples of BC/IFQI. outside.
8

---

Published as a conference paper at ICLR 2026
ReFORM
ReFORM(U)
(o
120
Caussian
(oussian(O.5)
-150
Gaussian 7)
-200
Caussian
0.0
0.5
1.0
8e508M
3eFDBMCCube) BuFORK
(EFOREEDODLsssi11 Bu (Ccooiiia)
Normalized Score
Bos DIEDBRC.OR cank)
BoF (EFORMKPP))
Figure 4: Ablations. Left: normalized scores ReFORM and its variants with different source distributions. Right: training curves of ReFORM and its variants by changing its components.
and does not learn anything. This confirms that the ability of ReFORM to satisfy support constraints using a source distribution with bounded support is crucial to good performance.
We next change the source distribution of the flow noise generator of ReFORM (U) back to ((B))) while keeping QBC = (O, (d) for the BC flow policy. We also add the reflection term back to the noise generator. We change so that B the ξ-confidence level of (BC. We vary & within {0.1, 0.3, 0.5, 0.7, 0.9} (Gaussiaa (E)). These baselines are highly sensitive to the choice of &, whereas ReFORM both avoids this additional hand-tuned hyperparameter & and achieves better per- formance than the best performing Gaussian (E) (Figure 4).
(Q4): The reflected flow improves the quality the generated noise. We consider replacing the reflected flow with three different generative models that also generate noise within the hypersphere By, MLP noise generator (ReFORM (MLP) ), a "squashed flow" that applies a tanh at the end (ReFORM (tanh) and squashed Gaussian (ReFORM (Gaussian) ) similar to DSRL. All base- lines perform worse than ReFORM (Figure 4: the MLP and the Gaussian fail to capture multimodal distributions, while tanh squashing suffers from gradient vanishing.
(Q5): Our design of the reflection term works the best within our considered choices. We con- sider two other options for the reflection term. First, ReFORM (Cube) replaces hypersphere- shaped domain By with a hypercube-shaped domain, while still applying the reflection term as introduced Xie et al. (2024). Second, ReFORM (Sphere) shares our hypersphere-shapeee main, but instead of compensating the outbound velocity, it reflects the outbound velocity back inbound once the sample hits aB Figure 4 shows that these two variants cannot perform similarly to ReFORM. We hypothesize that compensating for the outbound velocities makes the training pro- cess more stable than reflecting the outbound velocities. We leave finding theoretical explanations of this phenomenon to future work.
do-
(Q6): Removing the BC flow policy distillation slightly degrades the performance of ReFORM. We compare ReFORM with its variant ReFORM (NoDist by removing the distillation of the BC flow policy. Figure 4 shows that ReFORM (NoDistill) performance decreases slightly compared with ReF ORM. This suggests that longer backpropagation chain can be harmful, which matches the observation in Park et al. (2025b).
6 CONCLUSION
We propose ReFORM for realizing the support constraint with flow policies in offline RL. ReFORM simultancously learns a BC flow policy that transforms bounded uniform distribution in hyper- sphere to the complex action distribution that matches the behavior policy, and flow noise gener- ator that transforms bounded uniform distribution to a complex noise distribution being fed into the BC policy. With reflected flow on the noise generator, the noise generator is capable of gener- ating complex multimodal noise while staying within the domain of the prior distribution of the BC policy. Therefore, ReFORM avoids the OOD issues by construction, putting no further constraints limiting the performance of the learned policy, and leams complex multimodal policy. Our tensive experiments on 40 challenging tasks with the OGBench offline RL benchmark suggest that ReFORM achieves the best performance with only single set of hyperparameters, eliminating the

---

Published as a conference paper at ICLR 2026
costly fine-tuning process of most offline RL methods. The reflected flow noise generator can also be potentially combined with other generative- modell-abdddele policies, including diffusion policies.
Limitations. We identify several promising avenues for future work. Although our distillation step avoids BPTT through the BC flow, training the noise generator still relies on BPTT, which can be computationally intensive for deep models. This process can be potentially improved with shortcut models (Espinosa-Die) Dice et al. (025). or by applying pre- pre-trained BC model and latent space RL (Wagenmaker al., 20.5). Furthermore, our method ensures that the policy my remains within the support of the BC policy, meaning that it inherits any potential OOD errors made by the BC model itself. Integrating behavior cloning methods with stricter support constraints, diagnosing when the BC model generates OOD errors, applying pre-trained BC model could mitigate this dependence. Moreover, the design of the reflection term is a nascent area, and exploring more adaptive even learned reflection terms presents exciting direction for developing more powerful policy improvement methods. In addition, ReF FRRM applies the simplest value function learning method and actor-critic structure similar to Park et al. (2025b), which be potentially improved by other methods (Mao et al. 2023b; Garg et al.) 2023; Liu et al., 2024) Agrawalla et al.) 2025). Finally, ReFORM learns slower than algorithms imposing statistical distance regularization when the dataset contains expert policies due to the lack of any explicit regularization to keep the learned policy close to the expert policy.
REPRODUCIBILITY STATEMENT
For better reproducibility, we provide all the proofs of theoretical results in Appendix A and im- plementation details, including all hyperparameters in each environment of all algorithms in Ap- pendix C.3 The benchmark use open-source and published in Park et al. (2025a). We also included the source code of our algorithm in the supplementary materials.
ACKNOWLEDGEMENTS
This material is based upon work supported by the Under Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702-15-D-0001 or FF88702-25-D-BOM In addition, Zhang, So, and Fan are supported by the National Science Foundation (NSF) CAREER Award #CCF-2238030. This work was also supported part by a grant from Amazon. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Under Secretary of Defense for Research and Engineering.
2025 Massachusetts Institute of Technology.
Delivered to the U.S. Government with Unlimited Rights, as defined in DFARS Part 252.227-7013 or 7014 (Feb 2014). Notwithstanding any copyright notice, U.S. Government rights in this work are defined by DEARS 252.227-7013 or DEARS 252.227-7014 as detailed above. Use of this work other than specifically authorized by the U.S. Government may violate any copyrights that exist in this work.
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in neural informa- tion processing systems, 34:29304-29320, 2021.
Bhavya Agrawalla. Michal Nauman, Khush Agrawal, and Aviral Kumar. floq: Training critics via flow-matching for scaling compute in value-based rl. arXiv preprint (tti:2500900866..3 2025.
Donghoon Ahn, Jíwon Kang, Sanghyun Lee, Jaewon Min, Minjae Kim, Wooseok Jang, Hyoungwon Cho, Sayak Paul, SconHwa Kim, Eunju Cha, et al. A noise is worth diffusion guidance. arXiv preprint arXiv:2412.03885, 2024.
Michael S Albergo and Eric Vanden Eijuden. Building normalizing flows with stochastic inter- polants, arXiv preprint vrXiv:2209.15577 2022.
10

---

Published as a conference paper at ICLR 2026
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
Philip Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learn- ing with offline data. In International Conference on Machine Learning, pp. 1577-1594, PMLR, 2023.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In The Twelfth International Conference on Learning Rep- resentations, 2024. URL /oppa//veeeeeeeeeeeeeee net www.mrirriioooooo BAcdc--CWjja (CWjDGGJFD
Zhepeng Cen, Zuxin Liu, Zitong Wang, Yihang Yao, Henry Lam, and Ding Zhao. Learning from sparse offline datasets via conservative density estimation. In The Twelfth International Confer- ence on Learning Representations, 2024.
Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu. Score regularized policy optimiza- tion through diffusion behavior. In The Twelfth International Conference on Learning Represen- tutions, 2024. URL
net/ www
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, pp. 02783649241273668. 2023.
Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. URL
wttp://ee//o//mmeeooooeo net. forum? diaaamdddivvdddddne
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint 1605v 08803, 2016.
Carles Domingo-Enrich Michal Drozdzal, Brian Karrer, and Ricky TQ Chen. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. arXiv preprint arXiv:2409.068: 008.099..88..000 2024.
Nicolas Espinosa-Dice Dice, Yiyi Zhang, Yiding Chen, Bradley Guo, Owen Oertell, Gokul Swamy, Kianté Brantley, and Wen Sun. Scaling offline RL via efficient and expressive shorteut mod- els. In The Thitt Annual Conference on Neural Information Processing Systems, 2025.
URL https://openreview net. hott/oooummddduaarpppp
Luca Eyring. Shyamgopal Karthik, Alexey Dosovitskiy, Nataniel Ruiz, and Zeynep Akata. Noise hypernetworks: Amortizing test-time compute in diffusion models. arXiv preprint arXiv:2508. 099668,,08.09968, 09968, 2025.
Justin Fu. Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Gu. A minimalist approach to offline reinforcement learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural In- formation Processing Systems, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme leaamnnng: Maxent rl without entropy. In International Conference on Learning Representations, 2023.
Zhengyang Geng, Ashwini Pokle, and Zico Kolter. One-step diffusion distillation via deep equi- librium models. Advances in Neural Information Processing Systems, 36:41914-41911 2023.
Zhengyang Geng, Mingyang Deng, Xingjian Bai, Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling. arXiv preprint arXiv:2505. /347,, 2025.
Aditya Grover, Manik Dhar, and Stefano Ermon. Flow-gan: Combining maximum likelihood and adversarial learning in generative models. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
11

---

Published as a conference paper at ICLR 2026
Xiefan Guo, Jinlin Liu, Miaomiao Cui, Jiankai Li, Hongyu Yang, and Di Huang. Initno: Boosting -ott-to-image diffusion models via initial noise optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9380-9389, 2024.
Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idgl: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023a.
Philippe Haansn--sstucee Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies, 2023b.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
2XXi:1606.084155 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 23:6840-6851, 2020.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off- off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.004460 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:
7/ //arxiv.oro otg. abs 1412. 6980
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In International Conference on Learning Representations, 2022.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learnin via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33:1179-1191, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives open problems. arXiv preprint arXiv:2005.01643 2020.
Zeming Li, Xiangyue Liu, Xiangyu Zhang, Ping Tan, and Heung--Yeugg Shum. Noisear: Autore- gressing initial noise prior for diffusion models. arXiv preprint arXiv:2506.01337, 2025.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Repre- sentations, 2023.
Tenglong Liu, Yang Li, Yixing Lan, Hao Gao, Wei Pan, and Xin Xu. Adaptive advantage-guided policy regularization for offline reinforcement learning. In International Conference on Machine Learning, pp.
PMLR, 2024.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0300 200000000 2022.
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported trust region opti- mization for offline reinforcement learning. In International Conference on Machine Learning. pp. 23829-2333 1. PMLR, 2023a.
Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported value regulariza- tion for offline reinforcement learning. Advances in Neural Information Processing Systems, 36: 40587-40609, 2023b.
12

---

Published as a conference paper at ICLR 2026
Yanting Miao, William Loh, Pacal Poupart, and Suraj Kothawade. A minimalist method for fine- tuning text-to-image diffusion models. arXiv preprint arXiv:2506. 12036, 2025.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.
Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski. Piotr Mitoš, and Marek Cygan. Big- ger, regularized, optimistic: scaling for compute and sample efficient continuous control. Ad- vances in neural information processing systems, 37:1 13038-113000 13071, 2024.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. In International Conference on Learning Representations (ICLR), 2025a.
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. In International Conference on Machine Learning (ICML., 2025b.
Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Ser- gio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al. Imitating human behaviour with diffusion models. arXiv preprint 2rX1v:2301.10677,222 2023.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off- off-policy reinforcement learning. arXiv preprint 19Xiv:1910000177, 2019.
Boris Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging.
SIAM journal on control and optimization, 30(4):838 855, 1992.
Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal conditioned imitation learning using score-based diffusion policies. Robotics: Science and Systems, 2023.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022. URL URLLittps://ppereeviee
peet/Eorum22 TIdIXIpzhoI
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp. Abbas Abdolmaleki, Michael Ne- unert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08336, 2020.
Avi Singh, Huíhan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. In International Confer- ence on Learning Representations, 2021. URL
net/ forum?id
Ysuv WOFeKR
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-2265, pmlr, 2015.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances neural information processing systems, 28, 2015.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. 2020.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach offline reinforcement learning. In Seirth- Conference Neural Information Processing Systems, 2023.
Masatoshi Uehara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali. Gabriele Scalia, Nathaniel Lee Diamant, Alex M Tseng, Tommaso Biancalani, and Sergey Levine. Fine-tuning of continuous- time diffusion models as entropy-rreularizzed control. arXiv preprint arXiv:2402, /5/94, 2024.
13

---

Published as a conference paper at ICLR 2026
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park. Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning. arXiv preprint 2rXiv:2506.1577 2025.
Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imitation learning for batched historical data. Advances in Neural Information Processing Systems, 31, 2018.
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. SRRLhttps: https::/ http://wwwww.eee//oruu.... how.rr/eeeeee.eet.eoooueeeeaaaaa
Ziyu Wang. Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 14.7768-7773 2020.
Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy opti- mization for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:31278-2 32291, 2022.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: /911: /./ 1361, 2019.
Tianyu Xie, Yu Zhu, Longlin Yu, Tong Yang, Ziheng Cheng, Shiyue Zhang, Xiangyu Zhang, and Cheng Zhang. Reflected flow matching. In Proceedings of the 41st International Conference on Machine Learning, pp. 54614-54634, 2024.
Jing Zhang, Chi Zhang, Wenjia Wang, and Bingyi Jing. Constrained policy optimization with ex- plicit behavior density for offline reinforcement learning. Advances in Neural Information Pro- cessing Systems, 16:5616-5630, 2023.
Shiyuan Zhang, Weitong Zhang, and Quanquan Gu. Energy-weighted flow matching for offline rein- forcement learning. In Proceedings of the International Conference on Learning Representations (ICLR 2025), 2025.
Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforce- ment learning. In Conference on Robot Learning, pp. 1719-1735. PMLR, 2021.
Zikai Zhou, Shitong Shao, Lichen Bai, Shufei Zhang, Zhiqiang Xu, Bo Han, and Zeke Xic. Golden noise for diffusion models: A learning framework. arXiv preprint arXiv:2411. 1.09502, 2024.
14

---

Published as a conference paper at ICLR 2026
A PROOFS
A.1 PROOF OF PROPOSITION
Proof. We first prove the first statement. We prove this by contradiction. Suppose supp(TT(-(s)) g supp(TTa-Iss). Then, there exists a region {a E A To(als) > O, (T((ass) = 0} with non-zero measure. By the definition of the KL divergence, we have
/
TT(aas) log To(a|s) da + To (
[ (a(a))) log
To Ta(a)s)
(To(TT(-1s) ll xp(-ss))
da.
JEBB
JEEAYB
(14)
where the first term is oo and the second term is finite. Therefore, we have DKL ( TT{-s))) = co, which contradicts the condition that (To((T((1ss) TT((ss)) S 00.
We then prove the second statement. Consider Ta (-1s) = N(,, 1) and k = NN 1). We have supp(ma(((s)) S supp(Tz(((s)), The KL. divergence between them is
DKL (To((s) (T(-ss)) 4
(15)
Therefore, for any M > 0, we can choose
V2M so that DKL(To(-Is) ll TT(-Is)) > M.
A.2 PROOF OF PROPOSITION
2
Proof. For simplicity, consider a given state s S. We define ps(.) = TT(-1s) and po(-) = To(-s)) We prove by construction. We consider the optimal transport problem. First, we define source region within the support of pa. Consider a small ball B1 E supp(ps) centered at a1- The probability mass in the ball is 6 = Js. (a)dd. Second, we define target region. Consider another small ball B2 &f supp(p3) centered at a2 with the same radius B₁. Let the distance between the two balls be d = |ai - azll. We define the new probability such that
(p((a.
if Bi and & B2, if B1. if B2.
po(a)
(16)
po(a 02 a1),
Then, we have supp(ps) & supp(ps). We make d v % by choosing the source region B1 close to the boundary of supp(ps) and the target region B₂ close to Bi. Then, we have
Dwe(ps po) < {
Idll's(a)dda = d2 / ps(a)da = d'6 e
(17)
JEB
Therefore, we have Dw p1 Se
A.3 PROOF OF THEOREM
Proof. Remember that the source distribution the BC flow policy qBC = L(BF). We prove the theorem by showing that 2 (B,d) for all k E {0,1,... N 1}. which implies that B for all z ÂBC. We prove this by induction.
First, we have =0 w By because ((B)). Next, assume that z E ((B)). Then, have the following two cases:
Case 1: | || < l. Following Eq. 12 we have 2k+1 = Ek41 B
Case 2: | > L. Following Eq.
12 we have
21+1 (Va, (ttt, w: At, n 1)
(18)
(1 (1 (vo, (kAt, w; 8) At, ma+1)) ne+1-
In addition, we have (vo, (kA1, w: 8) At, mE41) (Ex+1 =k, mx+1) || (2k, ne+1).
(19)
15

---

Published as a conference paper at ICLR 2026
Plugging this into the previous equation, get,
2k+1 (#k, 7 n241-
(20)
Hence, we get,
- 1| = 1 mk+iil S || S
(21)
Thus our reflection ensures 24+11 E B Vk. Therefore, we have 2 = 2N E By, for all ~ QBC- As result, supp(}nc) C supp(pnc).
A.4 PROOF OF THROREM 2
Proof. Let E QBC be sample from ¹BC- We have supp(ānc). Following Theorem I have supp(/Bc) S supp(ysc). Therefore, we have E supp(yBc). Now consider the original target distribution PBC- Its support is the set of all points generated by applying the flow do, to all points in the support of qBC, i.e.
supp(pac) = {vh, (1, supp(mmc)).
(22)
Since we have & E supp(gвc). then by definition, we have Va, (1, 2; E supp(pec). This is true for all 2 ~ [BC. Therefore, by the definition of the support of pBc, i.e.
supp(ōBc) {the, (1, | supp(ūBc))
(23)
we have supp(fnc) C supp(rnc).
B ALGORITHM DETAILS
We provide the step-by-step explanation of ReFORM in Algorithm 1 where RF 8, w, N) means solving the reflected ODE (9) following the projected Fuler step (12) with the velocity field v, state sample from the source distribution w, and number of Euler steps N.
C EXPERIMENTS
C.I COMPUTATION RESOURCES
The experiments 13th Gen Intel(R) Core(TM) 17-13700KF CPU with 64GB RAM and an NVIDIA GeForce RTX 4090 GPU. The training time is around 80 minutes for 10 steps for ReFORM.
C.2 ENVIRONMENTS
We conduct experiments the recently published OGBench benchmark (Park et al., 2025a). We use environments (1 locomotion environment and 3 manipulation environments), 5 tasks in each environment, with different datasets, for total 40 tasks. Since OGBench originally designed for offline goal-conditioned RL, we use the single-task variants ("-singletask") for OGBench tasks to benchmark standard reward-maximizing offline RL. The reward functions in OGBench are semi-sparse. For the locomotion task, the reward functions are always - 1 for not reaching the goal and 0 for reaching the goal. Manipulation tasks usually contain several subtasks, and the rewards are bounded by Deask and 0, where Peask is the number of subtasks. All episodes end when the agent achieves the goal.
In our experiments, we consider the following tasks with the CLEAN dataset, where the demonstra-
tions are randomly generated by an expert policy:
16

---

Published as a conference paper at ICLR 2026
Algorithm ReFORM Algorithm
1: Input: Offline dataset D; total Buler number of steps N, radius /
2: Networks: Critic (s ); BC flow field vo, (4, 2j 8); noise flow field ua, (1, wz s); one-step BC
flow policy Má, (2)
3: while not converged do
4: Sample batch a, a, s,) s'} ~
5:
6:
Critic update u(BB)) 2 + RF 8', N)
7:
8:
9:
d' + Pg, (2)
10:
Update & to minimize E (r+ -
11:
Train vector field vo in the BC flow policy μoo
13:
(( I1 f1f0 a
14:
15:
[~
16:
It + (1-t)=+ta)
17:
Update 0, to minimize E [llon (1, 21; - (x1 - =²|
18:
19:
Train one-step policy Po,
20:
(B al + Mo, (2:
21:
Update 81 minimize E |||am ne, (z; s)
23:
24:
Train vector field Ga2 in the flow noise generator 1402 (BB))
25:
f- (F w, N)
27:
a + Me, (2;
28:
Update 0g to minimize E [[-Qo(s, a**2)
We also consider the NOISY dataset, where the demonstrations are randomly generated by highly suboptimal and noisy policy:
More details about the environment and videos of the demonstrations can be found in the OGBench
paper (Park et al. 2025a).
C.3 IMPLEMENTATION DETAILS AND HYPERPARAMETERS
C.3.1 DETAILS OF REFORM
Flow policies. We parameterize the velocity fields of the BC flow policy no, and the flow noise generator Uø, with MLPs. We use the Euler method to solve ODE (4) for the BC flow policy, and the projected Fuler step (12) to solve the reflected ODE (9) for the flow noise generator. 10 Fuler steps are used for both Fuler integration for all environments.
Q-functions. Following the standard implementation of Q-functions RL, we train two Q func- tions to improve stability. Two aggregation methods used to aggregate the two Q-values for different environments following Park et al. (2025b). For most environments, we take the mean of the Q-values for aggregation (Ball et al. 2023 Nauman al. 2024). except for the
17

---

Published as a conference paper at ICLR 2026
Table 1: Training steps for all algorithms for each task.
Task
Dataset
Training step
CLEAN
X 10"
NOISY
8 106
CLEAN
2 106
NOISY
3 X 106
CLEAN
2 106
NOISY
1 106
CLEAN
2 X 106
CLEAN
x 106
NOISY
1 106
NOISY
2 x 10
Table 2: Common hyperparameters for all algorithms.
Hyperparameter
Value
Learning rate Optimizer
0.0003
Adam (Kingma & Ba 2015)
Maximum gradient norm
10
Target network smoothing coefficient
0.005
Discount factor
0.995
MLP dimensions
5112,12,550 512,
Nonlinearity
GELU (Hendrycks & Gimpel 2016)
Flow steps
10
Flow time sampling distribution
4(0, 1]
Minibatch size
256
Clipped double Q-learnig
False (default), True (antmaze-large)
aatmaze-large environment, where we take the minimum the two Q-values (Van Hasselt et al, 2016 Fujimoto al. 2018).
Selection of the radius of the hypersphere By. the action space for physical systems is ways compact, we select the hypersphere By to be the smallest hypersphere that contains the action space, i.e., = miny (I' Rd A C By).. Note that, as the action space hyperbox, in most cases, we can compute the solution easily, or, otherwise, use overapproxima- tion of By Therefore, this choice does not impose any limitation on our approach. We also present experimental results of the sensitivity of ReFORM w.r.t. / in Appendix C.4
known and is usually
Neural Network architectures. For all neural networks our experiments, we use ML.Ps with hidden layers and 512 neurons each layer. We apply layer normalization (Ba et al. 2016) to the Q-function networks to stabilize training.
Training and evaluation. The difficulty of tasks in OGBench can be very different. Therefore, we use different training steps for different tasks (Table [T. For each task, we train each algorithm with 3 different seeds and evaluate the model saved the last epoch for 32 episodes.
C.3.2 DETAILS OF BASELINES IN MAIN RESULTS
We choose the state-of-the-art offline RL methods with flow policies our baselines, including FQL (Park et al. 2025b). IPQL Glansen-Estruce et al.) 2023b Park et al. 2025b). and DSRL (Wagen- maker al. 2025 We implement the baselines FQL and IFQL following the original implementa- tion provided in Park et al. (2025b), and DSRL also following the original implementation provided
in Wagenmaker al. 22025
18

---

Published as a conference paper at ICLR 2026
Table 3: Environment-speciti hyperparameters for FQL and DSRL.
Environment
FQL (S) a
FOL (M)
FQL (L) a
Noise bound for DSRL
antme
10
100
1.25, 1.25)
cube-single
30
300
3000
0.5, 0.5
cube-double
30
300
3000
[ 1.5, 1.5
scene
30
300
3000
0.75 0.75
antmaze large
cube-E sube-eingge
cube-double
BePOTM
BePORM
RaFORM
RaFORM
D6R0.
6882.
DSRL
D8R2
FQL(8)
FQL(33)
(Q)(3)
PQL(S)
PQL00
EQL(K)
FQL(O)
FQL(M)
PQ)(C1)
PQL.(1)
PQ).(1)
PQL.(1)
IFQ1.
IFQL.
IPQL
IFQL
0.0
1.0
Figure Normalized scores with the CLEAN dataset.
C.3.3 DETAILS OF BASELINES IN ABLATION STUDIES
ReFORM (U). ReFORM (J) modifies ReFORM by changing the source distribution of both the BC policy and the noise generator from (Bft) to (N,0 In other words, we have QNG qBC NNO0, for ReFORM
Gaussian (E). Gaussian (E) modifies ReFORM by changing the source distribution of the BC
policy from (( to (0, Id), then choose so that B the É-confidence level of N Id), i.e. PPF x (E), where PPF is the percent point function of d-dimensional x² distribution.
ReFORM (MLP). ReFORM (MLP) modifies ReFORM by changing the reflected flow noise gener- ator to an MLP noise generator f(s) S B which maps the state to a point within supp(gBc).
ReFORM (Lanh). ReFORM (Lanh) modifies ReFORM by removing the reflection term in the flection ODE (9) i.e., using (TT) instead of (12) when integrating the noise generator flow. Then, after the Fuler integration and getting following (TT), we tanh squash the norm of that it stays within By In other words, 2 = [ tanh -l
ReFORM (cube). ReFORM (cube) modifies ReFORM by changing the domain of gNG and @BC to [-,, 1]d Then, the reflected ODE is solved by first using the Euler integration (11) to get 2, and then applying - (2 + 1)mod4 2| following Xie et al. (2024).
ReFORM (sphere). ReFORM (sphere) modifies ReFORM by changing the reflection term from compensating the outbound velocity "bouncing back", like billiards.
ReFORM (NoDistill). ReFORM (NoDistill removes the distillation part of ReF ORM, i.c., the actor loss (1O) backpropagated through the BC flow policy instead of the one-step policy.
C.3.4 HYPERPARAMETERS
The choice of hyperparameters largely follows Park et al. (2025b). We provide the common hyper- parameters shared for all algorithms in Table 2 and the environment-specifee hyperparameters for FOL and DSRL in Table 3 Note that all environment-speen specific hyperparameters for FQL (M) and DSRL are the same provided in their original papers (with the CLEAN dataset), which are hand- tuned for each environment. As the basclines were not tested on the NOISY dataset their original papers, use the same hyperparameters for them in the same environment with the CLEAN dataset.
19

---

Published as a conference paper at ICLR 2026
an Lmaze large
sabble
cube-double
acene
ReFORM
RoFORM
ReFORM
DSRL.
DenL
DSRL
DeRL
(QL(S)
(QL(3)
PQL(3)
FQL(S)
FQ1.00 (K)
FQL(00)
(Q)
FQL((M)
(QL(L) (t)
(QL(L)
FQL(L)
FQL(L)
IFQL.
TFQ1.
0.0
0.5
1.0
0.0
0.5
0.5
1.0
0.5
Figure 6: Normalized scores with the NOISY dataset.
-000
-200
1000
000
0.0
1-2
0.30
0.20
05
30
0.75
100
100
000
XBI
Figure 7: Training curves for antmaze- large environment with the CLEAN dataset.
Taok
Tank
Tank
Tack
Taok &
300
-301
Testning Staps
x
Textning Staps
Teatning Stape
Tratntng Doage
KI2
Tratning Stage x11
(LL
Figure 8: Training curves for cube-single environment with the CLEAN dataset.
Tack
Taok
Taok
Tack
930
700
20
2.
1.
64
60
15
00
1a
Figure 9: Training curves for cube-double environment with the CLEAN dataset.
CEC
AX0
1000
Trateing Stape
xa0
Trataiin Strpe
Tvireiig Strpe
Tea wing Strpe sni
Tratwiig Stepe Xto
200L
HQL000
POULL
Figure 10: Training curves for scene environment with the CLEAN dataset.
C.4 ADDITIONAL RESULTS
Normalized scores for each environment and each dataset. We present bar plots of the in- terquantile means (IQM) (see Agarwal et al. (202T) for details) of the normalized scores for each algorithm in each environment with the CLEAN dataset (Figure|5) and the NOISY dataset (Figure 6 We can observe that ReFORM consistently achieves the best or comparable results in all environ- ments with both datasets, with constant set of hyperparameters. DSRL and FOL (M) generally perform the second and third best in environments with the CLEAN dataset. However, their perfor- mance drops when the NOISY dataset used.
Full results. We present the full per-task results of all 40 tasks in Table The results are averaged over 3 seeds and 32 runs per seed. The results are bolded if the algorithm achieves at or above 95% of the best performance following Park et al. (2025a).
20

---

Published as a conference paper at ICLR 2026
Table Full results. We present full results (normalized score) on 40 OGBench tasks. The results are averaged over 3 seeds and 32 runs seed. The results are bolded if the algorithm achieves at or above 95% of the best performance following Park et (2025a). To save space, the "singletask lags are omitted from task names.
TFOT.
FOT.
FOT.
FOT.
antmaze
CLEAN
040
5112
9613
141
85:19
65:1
antnaze- attaazee_arge-arigttttt--aa atteazze-irgg-araggtt-----x----- aitereee_largrrra//i66111----333
CLEAN
5644
62:-44
7245
CLEAN
0+60
91#6
90+7
8+11
77+9
61+2
CLEAN
04:
67::4
0::0
68=12
CLEAN
212
6144
76126
6444
8619
90:3
404:30
8117
91:6
attaazee_1rgg--gggoo-----22222
NOISY
0++0
0+:0
0+:0
41+13
0+00
91+6
atteazze--rrg--ooopo------2222-
04:0
04:0
9248
NOISY
36215
01:0
04:
56:137
04:0
628
NOISY
01:0
0::0
01:0
161:22
0::0
88::44
cube-ssiggl--playyttakk-----
CLEAN
40+-13
474-5
8642
57+12
60+-43
97-
cube-=singg---laayttaa222--
20-17
T3+10
5745
oube-singl--pllyy-taa2----
CLEAN
1425
18:1
7746
44135
68421
9911
CLEAN
30:6
19:17
7349
7743
59:119
89:11
cbbe-singgl-ppaayyttaa----
CLEAN
43112
25:19
85214
7343
61128
97
abbo-sinngle--ooaaytttaa2----
abba--inngla-0011y-t222222-
NDISY
53:15
24:2
7143
97
45110
100=0
NOIY
68:16
515
54:3
98
98:2
4042
2+1
6345
9421
31:5
100=1
cube-sihggle-0031---aaa5-5-
NOISY
37::4
312
7217
9611
61:3
99
aaba-donalle-llay-ttaa2----
CLEAN
4226
7
37#7
32#2
68
74
aube-doubble-2layytta2222-1
CLEAN
22+10
44-1
3043
243
474-3
90--12
cube-dombbe--llayttta333--
CLEAN
1742
1+1
1746
444
42430
90#7
auba-donille-play-ttaaaex-
CLEAN
30:15
11:11
2526
411
30:123
9017
cube-toubll-pplyytta88---
12:15
211
211:10
2617
82::11
NOISY
6245
644
12:14
68411
8649
94:66
cubo-doobble--ooiyyytaa222--
NOISY
543
21:1
2+1
37+18
764-25
56-220
uube-doutlle-noda---aaak---
NOIS
54:3
342
716
15:4
75:330
52:22
uuberdoubll--odiayttaa------
10:33
612
71:
201:12
33:47
cube-doubble-ooiyy-taa255
NOISY
811
6:::4
21.0
9.1
90:110
67 :7:223
scene-play-taskk-v0
CLEAN
99:11
6946
98
23+17
94:22
95
scane-play-tacl2--00
CLEAN
41:7
846
7343
27412
8741
99:1
scome-play-task3-v0
5622
5214
80:1
201:15
8522
9911
scene-laaytasek4-00
CLEAN
21:11
15:111
65:126
16:22
97:12
25:110
scene-play-tack5--0
65-45
7646
22+-23
99+1
=cane-noisy-taskkle0
NOIRY
87:29
27:11
87#1
100:E0
39--433
99:1
acane-noisy-taall2-00
40:2
11:1
2315
951-
1522
61:14
aceme-noisy-task3-1
NOISY
66:13
31:
66:14
9613
45:1:5
6913
scene-ooiey-taskk--
57:84
5344
scome-no1ey-taskk5-00
NOISY
57+:21
59+11
70:22
98:1
24:117
98#
Task
Task
Tusl
Tusk
Tual
Traseing Stepe
X10
Trassing Stape
Tracning Staps
Trassing
X10
Trasniig Stepa x107
Figure I1: Training curves for antmaze -araaazzeelrreeeee large environment with the NOISY dataset.
Awwand
DAR
Tralarng u Btoge
XI1
Trisning Stape
Training Goape
XII
Traveiing Staps
Trant Pa Stape XT0
DELL
POLORD
Figure 12: Training curves for cube-single environment with the NOISY dataset.
21

---

Published as a conference paper at ICLR 2026
Tasi
Tacx 2
Tadl
Taok
Tank
-200
-cre
850
0.0
600
Treasing Ztepo
xup
Trassing Stage
Trauning Slaaps
XT2
Toouning Blugs
x32"
Tewining Bloga
xad
DRA)
ERQ)
PQLCC
Figure 13: Training curves for cube-double environment with the NOISY dataset.
Tank
Task
1100
2000
000
0.00
bJ
4.
15
Figure 14: Training curves for scene environment with the NOISY dataset.
Revard
Revard
RoFORM ReFOOM
ReFORM
Gaups in (0.5)
Galsas an (0.1)
Gauas an (0.7)
Gauseian (0.3)
Gourei (0. .9)
0.0
0.2
044
0.5
0.8
0.6
0.8
1.0
Steps
x106
Stepe
x10
(a) Comparing ReFORM and ReFORM (U).
(b) Comparing ReFORM and Gaussian (E)
Figure 15: More training curves for ablation studies, ReFORM (U) changes both gNG and qBC
from (B to the standard Gaussian distribution and removes the reflection term. Gauss a (E) keeps QNG (B(B but changes ęBC the standard Gaussian distribution. Then, the radius of the hypersphere ( is chosen such that the standard Gaussian distribution has probability mass & in By
Training curves. We present the training curves for all tasks in all environments with both the CLEAN dataset (Figure Figure [0) and the NOISY dataset (Figure Figure 14.. In addition, we present training curves corresponding to Figure (left) in the main pages in Figure 15
D4RL results. We further conduct experiments in D4RL (Fu et al. 2020) antmaze and adoit envi- ronments test the performance of ReFORM across different benchmarks. Although the benchmark is different, we maintain the hyperparameters of ReFORM as those in OGBench. The results (D4RL normalized return) are shown in Table and ReF ORM still consistently achieves the best or compa-
rable results. DSRL is omitted in these results because its paper (Wagenmaker al. 2025) does not report the best hyperparameters of DSRL in these environments.
Visual manipulation results. We also conduct experiments in OGBench (Park et al. 2025a) vi- sual manipulation environments to test the performance of ReFORM with higher-dimensional image- based inputs. Similarly, we maintain the hyperparameters of ReF ReFORM. The results (return) are shown in Table 6 and ReFORM performs the best. DSRL is omitted because its best hyperparame- ters in these environments are not reported in Wagenmaker et al. (2025).
Visualization of the generated noise in the toy example. For the toy example presented in Sec- tion 5.3, it is possible to visualize the generated noises directly for ReFORM and DSRL. We visual- ize the noises Figure [6] We observe that the generated noise with reflected flow (ReFORM) is more concentrated while retaining two modes, while with a Gaussian distribution squashed by tanh (DSRL), the generated noise unimodal and spreads out a lot.
A Ablaioon on the radius of the hypersphere By One hyperparameter introduced in ReFORM is the radius I of the hypersphere By As discussed in Appendix C.3 select the smallest 1 such
22

---

Published as a conference paper at ICLR 2026
Table 5: D4RL results. We present the following results environments in the D4RL Fu et al. (020)) benchmark. The results are averaged over 3 seeds and 32 runs per seed. The results are bolded the algorithm achieves at above 95% of the best performance following Park et al. 2025a).
Environment
IFQL
FQL (L)
FQL (M)
FQL (8)
ReFORM
antmaze-unaze-12
9117
8524
9911
88213
97:0
antmaaze-umazz-rdivrrs---
55228
57.10
88s5
61
8313
aitmaaze--neddum-paaayy22
3+4
1446
92+1
52+18
85+4
anturaze-recduunddieeree--
24+34
944
81
24
80 +:4
attmazee-larg--playy22
17#21
43+10
61
3=4
71+4
antmaze-larre-ddiverse--
28227
55+4
85#8
8+12
69+9
pen-human-vi
65+1
48
59+4
31#4
6447
pen-cloned-vi
81+s
61+7
66
57#6
70+6
pen-expert-vi
1201a
105=7
128
107=10
129
door-human-v1
321
2+1
O4o
0zo
411
door-cloned-v1
-0#0
0+e
3+2
0=0
1+1
door-expert-vi
8945
104+1
105
102#0
104#4
Table 6: Visual manipulation results. We present the following results on visual manipulation environments in OGBench Park et al.] (2025a). The results are averaged over 3 seeds and 32 runs seed. The results are bolded if the algorithm achieves at or above 95% of the best performance following Park al. (025a).. To save space, the -singletask tags are omitted from task names.
Task
Datasct
IFQL
FQL(I)
FOL (M)
EQL(())
RePORM
CLEAN
117=7
1650216
-11029
-938219
-108212
NOISY
95+2
1176
-00322
-37+3
-52+7
(a) Generated noise of ReFORM. (b) Generated noise of DSRL.
Figure 16: Visualization of the generated noises in the 2-dimensional toy example.
23

---

Published as a conference paper at ICLR 2026
-30
Recard
100
150
022
0.4
0.6
1.0
Steps
XII
ReFURMI vib
ReFORM oubby
RoFORK
4v
RI
(5
RePORK 31 2070
Figure 17: Sensitivity analysis of ReFORM w.r.t. I.
Table 7: Approximate training time of ReFORM and the baselines.
Algorithm
ReFORM
FQL
IFQL
DSRL
Training time (minutes, 10 steps)
80
40
35
55
that A By in implementation. In OGBench environments, the action space is [-1, so we choose I vd. To study the sensitivity of ReFORM w.r.t. 1, we conduct experiments in the cube-single environment with the NOISY dataset, and vary in f0.25v 0.5vd, 2d, The results are shown in Figure [7] which shows significant difference among these choices. Therefore, ReFORM robust w.r.t the choice and empirically I can be chosen as any number close to the scale of the action space.
Training time. We report the training time of ReFORM and all baselines in Table 7 The table shows that ReFORM indeed doubles the training time compared FQL due the 2-stage flow. However, as shown in our experiments, FQL is sensitive to hyperparameters, and searching for optimal hyperparameters requires significantly more runs. On the contrary, ReFORM can be used without any hyperparameter searching.
C.5
CODE
We provide code for ReFORM in our supplementary materials.
D THE USE OF LARGE LANGUAGE MODELS
This paper uses Large Language Models to correct spelling and grammar issues.
24


Reviews:
[{"review_id": "47MsZDRvJN", "reviewer": "Reviewer_6RET", "summary": "This manuscript proposes the ReForm Framework, to cope with the OOD problem in offline RL as well as the multimodal issue of optimal policy distribution. The method first proposes the method to map the source distribution to the supported action distribution, and then utilizes the generated bounded noise via Reflected Flow to optimize the Q value. Experiments on more than 40 challenging tasks demonstrate the superoirty of the proposed framework.", "questions": "See the weakness above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The OOD issue, as well as the distribution of optimal action policy, are classic topics in offline RL, it is appreciated that the authors consider these issues from the new perspectives. \n\n* The use of bounded source distribution and reflect flow is quite novel and appealing. It fundamentally avoids OOD actions being explored. \n\n* The reflected flow noise generator can produce complex multimodal noise, which is helpful for some scenarios where real actions distribution are quite complex.", "weaknesses": "1. Some related references are missing, and it is suggested to consider the related work in the manuscript. \n\n* https://arxiv.org/abs/2202.06239\n\n* https://arxiv.org/abs/1705.08868\n\n* https://arxiv.org/abs/2301.12130\n\n2. The model design is appealing, however, the performance of the model also relies on the quality of behavior cloning model. How is the model performance if the BC model is not well estimated. Does the author consider about the robustness of the proposed method? \n\n3. Although the authors suggest distillation model, the training cost still seems high. Does the author consider about this? Or it is suggested to consider some ablation study on this issue.\n\n4. From the results in Figure 4, why ReForm(NoDistill) is slightly worse than ReForm, the reviewer is confused about this. For other variants, it seems their performances are much inferior to ReForm and ReForm(NoDistill). So is there a light and efficient version of ReForm?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z7o7oRrV96", "reviewer": "Reviewer_gwFF", "summary": "This paper introduces an approach to offline RL that constrains the learned policy within the support of the data generating policy while still leveraging expressive policy classes.", "questions": "Did the authors consider using a Gaussian-based policy for noise generation (similar to DSRL) instead of the flow-based model? See other questions above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents strong empirical results and thorough ablations justifying the design choices. In particular, ReFORM achieves strong performance across a variety of environments and tasks while using the same set hyperparameters, which is uncommon for offline RL algorithms.", "weaknesses": "The paper does not analyze potential reasons for why ReFORM outperforms the baselines in certain environments and datasets (clean vs noisy) but not others.\n\nThe paper does not investigate how this approach may scale to higher-dimensional state-action spaces. Does ReFORM’s approach of constraining to the data generating policy’s support work in a higher-dimensional space such as image-based inputs?\n\nThe paper does not compare to state-of-the-art algorithms on OGBench such as SORL [1] and floq [2]. (floq may constitute concurrent work and thus may not need to be compared against.)\n\nThe paper argues that ReFORM maintains high expressivity of the policy. However, the policy that is optimized via the Q-function is a one-step distillation policy, similar to FQL. One-step distillation policies have been shown to be less expressive than the multi-step policies employed by DSRL [3] and SORL [1]. \nThe authors justify this design choice via an ablation (Figure 4), though these results seem to contradict [1]’s findings. Are there possible explanations for this?\n\n- [1] Espinosa-Dice et al., “Scaling Offline RL via Efficient and Expressive Shortcut Models”\n- [2] Agrawalla et al., “floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL”\n- [3] Wagenmaker et al., “Steering Your Diffusion Policy with Latent Space Reinforcement Learning”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gy0lT2lBCW", "reviewer": "Reviewer_Q1Am", "summary": "This paper presents ReFORM, a new method for on-support offline reinforcement learning.\nThe authors observe that existing approaches—such as CQL, IQL, and FQL—rely on explicit distance-based regularization (e.g., KL, Wasserstein, MMD) between the learned policy and the behavior policy to avoid out-of-distribution (OOD) actions.\nHowever, these constraints depend heavily on tuning hyperparameters and can lead to excessive conservatism.\n\nReFORM instead enforces the support constraint by construction.\nIt introduces a two-stage reflected flow policy:\n\nBehavior-Cloned Flow — learns a bounded latent-to-action mapping with uniform latent noise U(Bₗᵈ), guaranteeing that all generated actions lie within the dataset support.\n\nReflected Noise Flow — optimizes the latent noise inside the same bounded domain via reflection dynamics, maximizing Q values while preserving support inclusion.\n\nThe paper provides theoretical proofs that the reflected flow maintains support containment for both latent and action distributions (Theorems 1 & 2), eliminating OOD actions.\nEmpirically, on OGBench (40 tasks with clean and noisy datasets), ReFORM achieves the best average performance among FQL, IFQL, and DSRL—without hyperparameter tuning.", "questions": "How does ReFORM perform compared with A2PR (Adaptive Advantage-Guided Policy Regularization) or XQL on D4RL benchmarks?\nIncluding these baselines would help assess competitiveness against the latest SOTA.\n\nWhat is the computational overhead of the reflection ODE and distillation step relative to FQL or IQL?\n\nCould the reflection principle be combined with diffusion-based policies or used as a projection operator for general offline RL methods?\n\nHow sensitive is ReFORM to the latent domain size? Would a learned bound improve performance?\n\nCan the authors release the full code to confirm reproducibility of OGBench results?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality:\nReFORM introduces a conceptually fresh idea—noise reflection for on-support control—that differs from prior regularization-based approaches.\nThe theoretical analysis clarifies when KL or Wasserstein constraints fail to guarantee support inclusion, which is insightful.\n\nTechnical Quality:\nThe paper provides rigorous derivations and clear theorems connecting reflection dynamics to support preservation.\nThe algorithmic design (bounded latent + reflected flow + distillation) is internally consistent and implementable.\n\nClarity :\nWriting, figures, and notation are excellent.\nThe relationship between theory and algorithm is clearly explained, and the overall flow of the paper is easy to follow.\n\nSignificance:\nThe idea of enforcing support “by construction” could inspire future work in flow- or diffusion-based offline RL.\nThe “hyperparameter-free” advantage is appealing for practical use, though broader validation is needed.", "weaknesses": "Limited Empirical Scope: Experiments are confined to OGBench; no evaluation on D4RL, Adroit, or visual RL tasks.\nThe claim of “state-of-the-art performance” is weakened by the absence of comparison with recent strong baselines such as A2PR (arXiv 2405.19909), XQL, and EDQL.\n\nLacking computational cost analysis—reflection dynamics and distillation likely add overhead.\n\nModerate Empirical Depth: Ablation results (λ, number of clusters, reflection strength) are only in the appendix; some should appear in the main text. \n\nIt is unclear how the reflected flow interacts with other policy constraints (e.g., entropy regularization or behavior cloning loss).\n\nWhile “support-by-construction” is elegant, it may restrict exploration in mixed-quality datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EikSiBU45r", "reviewer": "Reviewer_NkRQ", "summary": "The paper proposes ReFORM, an offline RL method that builds a flow-policy constrained to the support of the behavior policy by construction. It first learns a behavior-cloned flow policy whose source distribution is a bounded hypersphere, then manipulates the latent noise via a reflected flow so that the pushforward actions remain on-support while still allowing expressive, multimodal policies. The authors prove that the reflected flow preserves support and thus avoids OOD actions without statistical-distance regularization, and they report strong results on 40 OGBench tasks using a single hyperparameter setting, with performance profiles outperforming flow-based baselines.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method guarantees that the manipulated latent distribution and the resulting action distribution stay within the BC support (Theorems 1–2), directly addressing OOD actions without tuning a divergence weight.\n\nAdapting reflected ODEs to latent noise yields an expressive, multimodal policy class while rigorously maintaining the bounded domain.\n\nBy dropping statistical-distance regularization, ReFORM uses constant hyperparameters across tasks, yet performs competitively or better on performance profiles.\n\nResults across 40 tasks (clean/noisy datasets) show consistent gains against FQL/IFQL/DSRL, with informative ablations (source distribution, reflection design, distillation).", "weaknesses": "The work does not report results on widely used D4RL benchmarks (e.g., Locomotion/AntMaze v2), which would strengthen comparability with prior offline RL papers beyond OGBench. (Authors evaluate OGBench tasks instead.)\n\nDependence on BC support quality. Because actions are restricted to the BC support, any mis-estimated or overly conservative BC support could cap achievable performance; the paper could analyze this failure mode more explicitly (e.g., diagnostics when the BC model under-covers optimal modes)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)