Value Flows
Authors: Perry Dong
Chongyi Zheng
Chelsea Finn
Dorsa Sadigh
Benjamin Eysenbach
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Distributional Reinforcement Learning
Flow Matching
Primary area: reinforcement learning

Abstract: While most reinforcement learning methods today flatten the distribution of future returns to a single scalar value, distributional RL methods exploit the return distribution to provide stronger learning signals and to enable applications in exploration and safe RL.  While the predominant method for estimating the return distribution is by modeling it as a categorical distribution over discrete bins or estimating a finite number of quantiles, such approaches leave unanswered questions about the fine-grained structure of the return distribution and about how to distinguish states with high return uncertainty for decision-making. The key idea in this paper is to use modern, flexible flow-based models to estimate the full future return distributions and identify those states with high return variance. We do so by formulating a new flow-matching objective that generates probability density paths satisfying the distributional Bellman equation. Building upon the learned flow models, we estimate the return uncertainty of distinct states using a new flow derivative ODE. We additionally use this uncertainty information to prioritize learning a more accurate return estimation on certain transitions. We compare our method (Value Flows) with prior methods in the offline and online-to-online settings. Experiments on $37$ state-based and $25$ image-based benchmark tasks demonstrate that Value Flows achieves a $1.3\times$ improvement on average in success rates.

Published as a conference paper at ICLR 2026
VALUE FLOWS
Perry Dong
Chongyi Zheng
Chelsea Finn!
Dorsa Sadigh' Benjamin Eysenbach²
Stanford University
Prinneeoo University
perrydestanford.edd
chongyis@@prineeton.eddd
ABSTRACT
While most reinforcement learning methods today flatten the distribution of future returns to single scalar value, distributional RL methods exploit the return distri- bution to provide stronger learning signals and enable applications exploration and safe RI. While the predominant method for estimating the return distribution is by modeling as a categorical distribution over discrete bins or estimating a finite number of quantiles, such approaches leave unanswered questions about the fine-grained structure the return distribution and about how to distinguish states with high return uncertainty for decision-making The key idea in this paper is use modern, flexible flow-base wawdd models to estimate the full future return distribu- tions and identify those states with high return variance. We do by formulating new flow-matching objective that generates probability density paths satisfying the distributional Bellman equation. Building upon the learned flow models, esti- mate the return uncertainty of distinct states using new flow derivative ODE. We additionally use this uncertainty information to prioritize learning more accurate return estimation on certain transitions. We compare our method (Value Flows) with prior methods in the offline and online-to-online settings. Experiments on 37 state-based and 25 image-based benchmark tasks demonstrate that Value Flows achieves 1.3x improvement on average in success rates.
Website:
wtw.huuu.iioaaauellllllel. fllwslows
Code: https: wttp://l..coooccoiiiiiooiit
http://oooonngg/vaaiiiiaaae watpiiiiizegg///aaaaa ffows com/ chongy
INTRODUCTION
While many of the recent successes in reinforce- ment learning (RL) have focused estimat- ing future returns as a single scalar, modeling the entire future return distribution can provide stronger learning signals and indicate bits about uncertainty in decision-making. Distributional RL promises to capture statistics of future re- turns, achieving both strong convergence guaran- tees (Bellemare et al., 2017; Wang et al., 2023a; 2024) and good performance on benchmarks such Atari and D4RL (Bellemare et al., 2017; Dabney et al., 2018; Ma et al., 2021). This pa- per aims to understand the benefits of using more flexible representation of the return distri- bution, both critic in actor-critic RL and for estimating the variance in the future returns.
returns
"tiee
Figure I: Value Flows models the return distribution at
Modern generative models, such flow match- ing (Lipman et al., 2023; 2024; Liu et al., 2023; Albergo Vajden-,ijnden, 2023) and diffusion models
each time using a flow-matching model that is opti- mized obey the Bellman equation at each transition.
et al., 2015; Ho et al., 2020; Song et al., 2021), have shown great success in representing complex, multi-modal distributions continuous spaces. Building upon their successes,
"Equal contribution. Author order determined by coin flip.

---

Published as a conference paper at ICLR 2026
will use expressive flow-based models to fit the return distributions and estimate the return uncertainty of different states. These bits of uncertainty information, in turn, allow us to prioritize learning the return estimation on certain transitions.
The main contribution of our work is Value Flows, a framework for modeling the return distribution using wow-mathhing methods (Fig. 1). The key idea is to formulate distributional flow-matching objective that generates probability density paths satisfying the distributional Bellman equation automatically. Using the learned flow-based models, we estimate the return variance of distinct states using a new flow derivative ODE and use it to reweight the flow-matching objective. Value Flows also enables efficient estimation of the return expectation (Q-value), bridging a variety of policy extraction methods. Experiments on 37 state-based and 25 image-based benchmark tasks show Value Flows outperforms alternative offline and Aifline-to-online methods by 1.3x improvement on average.
2 RELATED WORK
Distributional RL views the RL problem through the lens of the future return distribution instead of a scalar Q-value (Engel et al., 2005; Muller et al., 2024; Morimura et al., 2010). Prior work has shown both strong convergence guarantees (Wang et al., 2023a; 2024) and good empirical performance (Farebrother et al., 2024; Bellemare et al., 2017; Ma et al., 2025) of this family of methods, enabling applications in exploration (Mavrin et al., 2019) and safe RL (Ma et al., 2021; 2025). However, those methods typically parameterize the return distribution as discrete bins and minimize the KL divergence to the distributional Bellman target (Bellemare et al., 2017) use finite number of quantiles to approximate the full return distribution (Dabney et al., 2018). In contrast, Value Flows models the full return distribution directly using state-of-the-art generative models.
Perhaps the most popular modern generative models are autoencoders (Kingma & Welling, 2013), autoregressive models (Vaswani et al., 2017), denoising diffusion models (Sohl-Dickstein al., 2015; Ho et al., 2020; Song et al., 202 and flow-matching models (Lipman et al., 2023; 2024; Liu et al., 2023; Albergo & Vajden-Eijnden, 2023). In the context of RL, these generative models have succeeded in modeling the trajectories (Chen et al., 2021; Ajay et al., 2023), the transitions (Alonso et al., 2024; Janner et al., 2021; Farebrother et al., 2025b), the skills (Ajay et al., 2021; Pertsch et al., 2021; Frans et al., 2024; Zheng et al., 2025), and the policies (Wang et al., 2023b; -astunnEsstruch et al., 2023b; Dong et al., 2025c; Park et al., 2025c; Dong et al., 2025b). We employ the state-of-the- flow-matchi objective (Lipman al., 2023) to estimate the future return distribution.
This paper solves offline RL and offline-to-online RL problems. The goal of offline RL is to learn a policy from previously collected datasets, often through conservative behavioral regularization (Peng et al., 2019; Fujimoto & Gu, 2021; Kostrikov et al., 2021a; Hansen-Estrus et al., 2023b; Park et al., 2025c) value constraints (Kumar et al., 2020b; Kostrikov et al., 2021b). After learning the offline policy, offline-to-online RL uses interactions with the environment to fine-tune an online policy, often requiring balancing exploration (Yang et al., 2023; Mark et al., 2023), calibrating values (Nakamoto et al., 2024), or maintaining offline data (Nair et al., 2021; Ball et al., 2023; Dong et al., 2025a;b). Our approach focuses on estimating the full return distribution, resulting in more accurate Q-value estimations that benefit both settings.
Prior work has used confidence weight from Q estimates to reweight the Bellman error (Kumar et al., 2020a; 2019; Lee et al., 2021; Schaul et al., 2015), aiming to mitigate the issue of instability in Q-learning (Tsitsiklis & Roy, 1997: Baird, 1995; van Hasselt et al., 2015). These methods construct the uncertainty weight using bootstrapped errors from Q estimations (Kumar et al., 2020a; Schaul et al., 2015) or the epistemic uncertainty from an ensemble of Q networks (Lee et al., 2021). Our work builds on these prior works by using variance to reweight the critic objective (a flow-matching objective in our case), and is most similar to prior methods that do weighting based on alcatoric uncertainty (Kumar et al., 2020a; Schaul et al., 2015).
3 PRELIMINARIES
We consider a Markov decision process (MDP) (Sutton et al., 1998; Puterman, 2014) defined by a state space S, an action space A C Rdd, initial state distribution p A(S), bounded reward
2

---

Published as a conference paper at ICLR 2026
function : S x A - [Tmin: m'ma]] a discount factor Y [0, 1), and transition distribution p: S x A A(S). where A(X²) denotes the set of all possible probability distributions over a space uppercase X to denote random variable,
X. We will use h to denote the time step in an MDP, use lowercase x to denote the value of X, use Fx to denote the cumulative distribution function (CDF) of X, and use px to denote the probability density function (PDF) of X. In Appendix A.I, we include a brief discussion about the expected discounted return and the Q function in RL.
This work considers the offline RL problems (Lange al., 2012) (Sec. 4.4), which aim to maximize the return using a fixed dataset. We will use D = {(s, a, r, 8') to denote the dataset with transitions collected by some behavioral policies. We also extend our discussions to affliee-to-oning RL problems (Sec. 4.4), where algorithms first pre-train a policy from the offline dataset D and then fine-tune the policy with online interactions the exact same environment. Those online interactions will also be stored in D, enabling off-policy learning.
Distributional RL. Instead of focusing on the expected discounted return (scalar), distributional
RL (Morimura et al., 2010; Bellemare et al., 2017 7) models the entire return distribution. Given a policy π, we denote the (discounted) return random variable as 2" Σx= 70 An) E Tim
Amax
I'm I-7*
Zmin
and denote the conditional return random variable as z (s(aa) = I-1
r(s, a) + +h (S A,).2 We note that the expectation of the conditional return random variable 2" (" a) equivalent to the Q function: (T a) = (S( S1 [25 (s)
Prior work (Bellemare et al., 2017) defines the distributional Bellman operator under policy π for a conditional random variable Z(s, a) as
T (=, r(s,a) + 77'(S' A'),
where S" and A' are random variables following the joint density p(s' a))))(aa ( and d denotes identity in distribution. This definition implies two important relationships for the CDF and the PDF of the random variable T Z(s, a) (see Lemma 1 and Lemma 2 in Appendix A.2). We will use these two implications to derive our method in Sec. 4.1 and Sec. 4.2. Prior work (Bellemare et al., 2017) has already shown that the distributional Bellman operator T is a y-contraction under the p- o--Wasserstein distance, indicating that 7 has a unique fixed point. We include a justification for the fixed point z (s, a) in Lemma 3, and will use the contraction property of T estimate the return distribution with theoretical guarantees (Sec. 4.1 & 4.2).
Flow matching. Flow matching (Lipman al., 2023; 2024; Liu al., 2023; Albergo & Vanden- Eijnden, 2023) refers to family of generative models based on ordinary differential equations (ODEs). The goal of flow matching methods is to transform a simple noise distribution into target distribution px over some space X' C Rdd We will use / to denote the flow time step and sample the noise from a standard Gaussian distribution A(0, T) throughout our discussions. Flow matching uses time-dependent vector field v : [0, 1] x E + Rd to construct time-dependent diffeomorphic flow o [0. 1] x iR + gdd (Lipman et al., 2023; 2024) that realizes the transformation. The resulting probability density path p: [0, 1] x Rd + (RR)) of the vector field u satisfies the continuity equation (Lipman et al., 2023). Appendix A.3 includes the flow ODE (Eq. 11) and the continuity equation (Sec. 12) connecting Φ, v and p. In Sec. 4, we will estimate the return distribution using flow-based models, utilizing the continuity equation (Sec. 4.2) and flow ODE (Sec. 4.3).
Prior work has proposed various formulations for learning the vector field (Lipman et al., 2023; Liu et al., 2023; Albergo & Vanden-Fijnded 2023). We adopt the simplest flow matching objectives called conditional flow matching (CFM) (Lipman et al., 2023) to construct loss functions for optimizing the return vector field (Sec. 4.2). Appendix A.3 includes detailed discussions of flow matching methods.
4 VALUE FLOWS
In this section, we introduce our method for efficiently estimating the return distribution using an expressive Now-matching model, resulting in an algorithm called Value Flows. We first formalize the
While we consider deterministic rewards, our discussions generalize to stochastic rewards used in prior work (Bellemare et al., 2017; Dabney ct al., 2018; Ma et al., 2021).
By definition, we also write 2n 7" (So, Ao).
3

---

Published as a conference paper at ICLR 2026
problem setting. providing motivations and desiderata for the flow-based return estimation. We then introduce the update rules for the return vector field, deriving a distributional flow matching loss to fit the discounted return distribution. Our practical algorithm will build upon this distributional flow matching loss and incorporate uncertainty in the return distribution.
4.1 MOTIVATIONS AND DESIDERATA
While prior actor-critic methods (Fujimoto et al., 2018; Haarnoja et al., 2018) typically involve esti- mating the scalar Q function of a policy, we instead consider estimating the entire return distribution using state-of-the-art generative models. Our desiderata are threefold: we want our model capable of (1) estimating the full distribution over returns without coarse- gaasse-raiiing (2) learning probability densi- ties satisfying the distributional Bellman backup (Eq. 9), while preserving convergence guarantees, and (3) prioritizing learning more accurate return estimation at transitions with high variance. Prior distributional RL methods either discretize the return distribution (Bellemare et al., 2017) or use finite number of quantiles to represent the return distribution (Dabney et al., 2018), violating these desiderata.
To achieve these goals, we use an expressive flow-based model to represent the conditional return random variable 2 (s, a) (desiderata (1)). We will learn a time-dependent vector field R x [0 x S x A + IR to model the return distribution. The desired vector field vt will produce a time- dependent probability density path p" R x [0, 1] x S x A + A A(R) that satisfies the distributional Bellman equation (Eq. 10) for any flow time step t E [0. 1].
(2' (1) = T' T'p t, a. = E (p( pp (2 (5' (='-r(8,0) t, S. (1,,
7 [ t..s.ad)-
(2)
Thus, (=² 11,8,) converges the discounted return distribution pz- (a s, (desiderata (2)). To optimize the vector field toward v*, Sec. 4.2 will construct flow matching loss that resembles temporal difference learning (Sutton, 1988). We will show that both the expected return (Q function) and variance of the return are easy to compute using the initial vector field u(E 0, a) and the derivative of the vector field 0v/Oz E R respectively (desiderata (3); See. 4.3). Our practical algorithm (Sec. 4.4) will weight our flow matching loss using the variance estimate.
4.2 ESTIMATING THE RETURN DISTRIBUTION USING FLOW-MATCHING
We start by constructing the update rules for the vector field and the probability density path, and then derive the preliminary flow matching losses fit the return distribution. We will use these preliminary losses to construct the practical loss used in our algorithm (Sec. 4.4).
The vector field update rule. Similar to the standard Bellman operator (Agarwal et al., 2019; Puterman, 2014) 4), the distributional Bellman operator preserves convergence guarantees regardless of initialization (Bellemare et al., 2017). This property allows us to first construct update rules for vector field 0(2* t, 8, a) and a probability density path p(2" t, s, a) separately and then relate them via the continuity equation (Eq. 12).
Specifically, given policy т and the transition p(s' 8, a), we start from a randomly initialized vector field (2(2" 1,8,)) (iteration & = 0) that generates the probability density path (=("" 148, a). We construct a new vector field and a new probability density path using 2k and px:
1 (=+1 118, a) A (=p(2" 11,,800 - Bpteeleea) Pke
2" =2-r((s,u) r(s,a) a)
1,
Ukt1(2" (t,,,) e Eptorlea)) Pk (=-r((aa)) 1, 1,8,, d') ux [z'-n(aaa)
L, L aa'
Pk+1 (zt 5,
Importantly, these definitions only instantiate the functional form of the new vector field u4+ and the new probability density path Ps+1, missing a connection between them. Establishing an explicit relationship between the new vector field uk+1 and the new probability density path P*+11 requires using the continuity equation (Eq. 12) between the vector field eg and probability density path px. which we show in the following proposition.
4

---

Published as a conference paper at ICLR 2026
Proposition 1 (Informal). Given the vector field U2 that generates the probability density path Pk- the new vector field Uk+1 generates the new probability density path px+1-
See Appendix B.1 for the detailed statement and proof. There are two implications of this proposition. First, since the new vector fireld Up+1 generates the new probability density path Ph++1, applying the distributional Bellman operator to the probability density path pt: is equivalent to learning a vector field 1k+1 that satisfies Eq. 3. This implication allows us to convert the problem of estimating the full return distribution into the problem of learning vector field, which naturally fits into the wow-matching framework. Second, since the distributional Bellman operator 7 is a c-notracction, repeatly applying T to the probability density path pk guarantees convergence to p (=* f, 8, a) (Eq. 2). Therefore, we can construct a flow matching loss to learn the vector field 2k+1 with a guarantee to converge toward v².
The distributional flow matching losses. We now derive the preliminary flow matching losses for estimating the return distribution only using transition samples from the dataset D. Given the vector field vi. one simple approach to learn a new vector field v satisfying the update rule in Eq. 3 is to minimize the mean squared error (MSE) between and Uk+1- We call this loss the distributional flow matching (DFM) loss:
CDPM (v, Un. E, ((s..r)--,,
(0(2² (t,,,)) U t, t.s.
(t.,,a)
It is easy to verify that arg min, LDFM (1 02) Uk+1 (see Lemma 4 in Appendix B). Similar the standard flow matching loss (Lipman et al., 2023), this loss function is not practical because of (1) the a). and (2) the intractable integral (the expectation)
unknown transition probability density p(s' in the vector field uk+1- To tackle the issue of the intractable vector field Uj+1, we optimize the alternative distributional conditional flow matching (DCFM) loss:
[( (u(e) (t,,,a) 2k
x0 -T ? t, 1,5', t,st,a))
(DCMM(U, Ux) = E D..aai"")-D.. 2 (-'mr (
a' d'>a(d')'").
We can interpret the transformed returns (2* 155--1 / as convolving the probability density path pk and use a change of variable to simplify this DCFM loss (see Lemma 5 in Appendix B). Unlike the DFM loss LDEM, the DCFM loss LDCFM can be easily estimated via samples from the dataset D. Like the connection between the flow matching loss and the conditional flow matching loss in Lipman et al. (2023), the DCFM loss has the same gradient as the DFM loss, indicating that they have the same minimizer.
Proposition (Informal). The gradient V CDEM(U, Ug)) is the same as the gradient V V,,CDEM (v, ux).
See Appendix B.1 for the detailed statement and proof. Importantly, optimizing LOCEM does not require access to the full transition distribution and prevents taking the intractable integral. LDCFM is a TD loss because it corresponds to applying the distributional Bellman operator 7 to the probability density path Рą- Furthermore, our flow-based model produces estimations of the return expectation (Q function: Sec. 4.4) and the return variance (aleatoric uncertainty; Sec. 4.3) easily, both of which will be incorporated into our algorithm.
We note that simply optimizing (D 1x) might produce a degenerated vector field, e.g., 0. In particular, our initial experiments suggest that naively optimizing (DCFM(") produced a trivial performance on some tasks. To stabilize learning, we use the return predictions at the next state-action pair (s', a') and the flow time = to construct a bootstrapped target return similar to the TD target in Q-learning (Watkins & Dayan, 1992). The resulting loss function, called bootstrapped conditional flow matching (BCFM) loss LBCFM- M. resembles the standard fitted Q-learning (Riedmiller, 2005). In addition, use target vector field 5 to replace the historical vector field vk. Prior work has shown that using a target network prevents the model from collapsing (Grill et al., 2020; Tian et al., 2021). See Appendix C.1 for further discussions.
4.3 HARNESSING UNCERTAINTY IN THE RETURN ESTIMATION
One benefit of using the flowe models to estimate the full return distribution is that we can easily compute the return variance every state-action pair (s, a). These return variances measure the
5

---

Published as a conference paper at ICLR 2026
aleatoric uncertainty in the MDP, indicating the noise that stems from the environmental transitions. We incorporate the return uncertainty information into our method and use it to prioritize learning more accurate return distribution transitions with higher return variance. We first present an estimation of return expectation using the initial vector field and an approximation of return variance using the Taylor expansion. We then introduce new ODE that relates the derivative of the learned diffeomorphic flow, allowing us to estimate the return variance in practice. Using this variance estimation, we define the confidence weight for reweighting the distributional flow matching losses.
Prior work (Frans et al., 2025) has shown that the learned vector field u (from Sec. 4.2) at the flow time 0 points toward the dataset mean. We adopt the same idea estimate the return expectation (s, a)] i.e., the Q value) using Gaussian noise é ~ (0, 1). Additionally, we estimate the return Z*
variance Var z (s, a)) using a first-order Taylor approximation on the corresponding (diffeomorphic)
flow Φ : R x [0, 1] x S x A - I. of the vector field v. Specifically, the flow o(= i, a) transforms another noise 6g ~ (O, 1) into a return prediction @(co 1, 8, a). We linearize the return prediction around noise using Taylor expansion and then compute the variance of this lincarization.
Proposition 3 (Informal). The initial vector field v(e O, 8, a) produces an estimate for the return expectation, while the first-order Taylor approximation of the flow 6(<o 1, 1,8,0 a) around e produces an estimate for the return variance,
(e (c
È [Z" (s, Ee-oro. (0lpleleeee,ee,eeeeeeeess (,,s Varl (s, = BEwN((o.)
(6)
See Appendix B.2 for the detailed statement and proof. In practice, computing the flow derivative 0o/Oc θe requires backpropagating the gradients through the ODE solver, which is unstable and computationally expensive (Park al., 2025c). One key observation is that the flow derivative 0Φ/ de follows another flow derivative ODE, drawing a connection with the vector field derivative dv 0v/Oz. We will use this flow derivative ODE to compute the flow derivative 06/0c efficiently.
Proposition 4 (Informal). The flow derivative 0b/O dc and the vector field derivative 0u/Oz satisfy flow derivative ODE.
See Appendix B.2 for the detailed statement and a proof. This flow derivative ODE, together with the flow ODE (Eq. 11), enables computing both the return prediction and the return variance estimation (Eq, 6) using a numerical solver (Alg. 2) and an efficient vector-Jacobian product (VJP) implementation (JAX Developers, 2025).
We will use the estimated variance to reweight our flow matching losses (Appendix C.I). Since the distributional RL algorithms typically model the alcatoric uncertainty, a high return variance indicates high stochasticity in the environment, requiring fine-grained predictions. Therefore, the goal of our confidence weight is to prioritize optimizing the vector field at state-action pairs with high return uncertainty. Adapting the confidence weight in prior work (Lee et al., 2021), we define the confidence weight for a state-action pair (s, a) and noise as
w(s a)) (-r/ Orb dc (cl (c(1 1, 0.5,
where a-)) denotes the sigmoid function, >> O is a temperature, and w(s, e) E [0.5, 1]. For simplicity, we can use one Gaussian noise as a Monte Carlo estimator for both the return expectation and the return variance (Eq. 6). We include visualization of the confidence weight for different tem- peratures in Appendix D.2. Our confidence weight is an increasing function with respect to the return variance estimate, indicating that a higher uncertainty results in higher weight. In Appendix C.I, we discuss the complete flow matching losses for fitting the return distribution.
4.4 THE COMPLETE ALGORITHM
We now discuss the policy extraction strategies based on the flow-based return models and summarize the complete algorithm of Value Flows. We consider two different behavioral-regullrizee policy extraction strategies for offline RL and offline-to-onligg RL. First, for offline RL, following prior work (Li et al., 2025; Chen et al., 2022), we rejection sampling maximize Q estimates (Eq. 6) while implicitly imposing a KL constraint (Hilton, 2023) toward a fixed behavioral cloning (BC) policy. Second, for online fine-tuning in offline-to-online RI., following prior work (Park al.,

---

Published as a conference paper at ICLR 2026
Algorithm Value Flows is an RL algorithm using flow matching model the return distribution.
1: Input The return vector field the target return vector field uig the BC flow policy Two the one-step flow policy Tay and the dataset D.
2: for each iteration do
3: Sample a batch transitions {(s, a, s)rr)) ~ D and a batch noises {e} ~ (0,
4: Compute the confidence weight w(s. a, e) using the Euler method and VJP Sec. 4.3
5: Train the return vector field by by minimizing L Flow (0)
b Appendix C.1
V Offline RL
6: Train the BC flow policy Tu by minimizing LBC Flow (w) Online fine-tuning offline-to-online RL
Appendix C.2
7: Train the one-step flow policy Fn by minimizing Lome-step Flo (1)
Appendix C.2
8: Update the target return vector field 2a using Polyak averages
9: Return U0, Tws and Ty.
CS1
CODAC
Flows
1- Wasseersein distance
press button
6174
close window
9223
doss drawer
3.108
Ground
with confidence wegle
Figure 2: Visualizing the return distribution. (Column 1) The policy completes the task of closing the window and closing the drawer using the buttons lock and unlock them. (Column 2) C51 predicts noisy multi-modal distribution and (Column CODAC collapses infers smooth return histogram resembling the ground-trun return distribution. (Column 5) Quantitatively. Value Flows achieves 3x lower 1- I-Wasserstein distance than alternative methods. See See. 5.1 for details.
single
mode. (Column 4) Value Flows
2025c), we learn stochastic one-step policy to maximize the Q estimates while distilling it toward the fixed BC flow policy. See Appendix C.2 for detailed discussions.
Our algorithm, Value Flows, consists of three main components: (1) the vector field estimating the return distribution, (2) the confidence weight prioritizing learning the return distribution uncertain transition, and (3) the flow policy selecting actions. Using neural networks to parameterize the return vector field ug. the BC flow policy nw. and the one-step flow policy Zy, optimize them using stochastic gradient descent. Alg. summarizes the pseudocode of Value Flows, and our open-source implementation is available online.?
7

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTS
The goal of our experiments is to answer the following key questions:
(Q1) Does Value Flows predict returns that align with the underlying return distribution?
(Q2) Can Value Flows effectively learn a policy from a static offline dataset?
(Q3) How sample efficient is Value Flows in offlne--oo-onliee learning compared to prior methods?
(Q4) What are the key components of Value Flows?
Experiment Setup. The most relevant prior work is distributional RL methods, which model the return distribution using a categorical distribution (C51 (Bellemare et al., 2017)) or a finite number of quantiles (IQN (Dabney al., 2018) and CODAC (Ma et al., 2021)). We will compare Value Flows against these prior methods by visualizing the return predictions and measuring the success rates in offline and offline-to-online settings, We also compare Value Flows with prior RL methods that learn the scalar Q values with Gaussian policy (BC, IQL (Kostrikov et al., 2021a), and ReBRAC (Tarasov et al., 2023)) or flow policy (FBRAC, IFQL, FQL (Park et al., 2025c)) for completeness. Our experiments will use standard benchmarks introduced by prior work offline RL. We choose set of 36 state-based tasks from OGBench (Park et al., 2025a) and D4RL (Fu et al., 2021) and choose set of 25 mage--asse tasks from OGBench to evaluate our algorithm in the offline setting. Following prior work (Park et al., 2025c), we report means and standard deviations over 8 random seeds for state-based tasks (4 seeds for image- amage-based tasks). We defer the detailed discussions about environment and datasets to Appendix E. and baselines to Appendix E.2. We present the hyperparameters in Appendix E.3.
5.1 VISUALIZING RETURN DISTRIBUTIONS OF VALUE FLOWS
One of the motivations of designing Value Flows is to flow-based models estimate the entire return distribution (Sec. 4.1). We hypothesize that the proposed method fits the under- lying return distributions with lower distributional discrepancy. To investigate this hypothesis, we study the return predictions from Value Flows and compare them against two alternative distributional RL methods: C51 (Bellemare al., 2017) and CODAC (Ma et al., 2021). On from OGBench, we use learned FQL agent to collect
scene-play- wat.gglttaaktttss2--aa 5000 optimal and suboptimal trajectories as our evaluation dataset to visualize the return distribution for different methods. These trajectories are nut-of-distributio because (1) they are collected by learned agent instead of the behavioral policy, and (2) they yield some combinatorial generalization behaviors (see Appendix E.1 for explanations). We visualize the ID return histograms inferred by different algorithms, including the ground -oout return distribution for reference. For fair comparison, use 5000 return samples and 60 bins for each histogram. For quantitative evaluations, measure the 1- I-Wasserstein distances (Panaretos & Zemel, 2019) between the histogram of each method and the ground-truth histogram.
Fig. shows the resulting histograms, superimposing the ground-truth return distribution, along with the 1-Wasserstein distances for different methods. Observe that Value Flows predicts smooth return histogram resembling the ground-truth return distribution, while baselines either infer a noisy single return mode (CODAC). Numerically, our method achieves a 1 I-Wasserstein distance 3 lower than the best- pest-performing baseline. These results suggest that Value Flows can effectively estimate the multimodal return distribution.
multi-modal distribution (C51) or collapse
5.2 COMPARING TO PRIOR OFFLINE AND OFFLINE-TO-ONLING METHODS
Offfine RL. Our next experiments compare Value Flows prior offline RL methods, including those estimating the return distribution using alternative mechanisms. We compare our method against baselines on both state-based tasks and image-based tasks. Results in Table I aggregate over 5 tasks in each domain of OGBench and 12 tasks from D4RL, and we defer the full results to Appendix Table 3. These results show that Value Flows matches or surpasses all baselines on 9 out of 11 domains. On state OGBench tasks, most baselines performed similarly on the
httpp:://gi thub. cot. chongyi zheng/va uue flows
8

---

Published as a conference paper at ICLR 2026
Table Offline evaluation on OGBench D4RL benchmarks. Value Flows achieves the best or near-best performance on 9 out of 11 domains. Following prior work (Park et al., 2025c), we average results over 8 seeds (4 seeds for image- based tasks) and bold values within 95% of the best performance for each domain. See Table 3 for full results.
Guussian Policies
Policies
Domain
BC
IQI.
RoBRAC
FBRAC
IFQI.
FQI.
CSI
IQN
CODAC
Value Flows
cube-docb_ cube-drubee-plly 15 tasks)
211
6:2
12:.3
15.18
14:25
20
2100
82.1.8
61 61:16
cabe-czip-e-lla aly 15 taska)
040
1=1
1=0
0::
4=2
040
64:0
2+:1
14=33
guzzle-Sk3-pla (5 tasks:
241
22 22
144.5
19 10
00
140
154
20
87 13
putsle 4ad play (5 Lasts;
0+0
7=2
14-3
15+
26
2
17-5
0+0
27
20+11
27 -44
comne-play 15 tasks)
5.12
28:1.
41.1
45.1
30 10:14
36
411
$0.1
35.11
39
(3 tasks)
84.15
8..14
30
87
38 18
74.14
15
( taske)
6:1.4
1111
6.5
10
5-2
4.1.2
13..4
-isual-cube-doli doubb--dubbe-play (5 taakel
11:66
1=:1
2m
222
1
13=2
vianal -punxlee e-eell-----e Se3-playy 65 tasks)
2+
20
21
30
19++
25 2
viausl play 16 beace)
26
28
11+11
21
41 +-4
41 m
45 =
D4R1, adrolt (12 tasksi
68
50
52
-
52
48
48
50++3
52.+1
50.+22
cube double
puzzle 3x3
cube-doute
puzzle-3x3
100
100
80
80
Succcess Rate
80-
80
60
60
20
20-
20
200
400
800
1000
200
400
600
800
1000
Steps (x1000)
Steps (x1000)
(x1000)
Figure Regularizing the flow-matching loss is important. The regularization coefficient À needs to be tuned for better performance.
Figure 5: Rewcighting flow-matching objective boosts success rates. Choosing the correct confidence weight boosts the performance of Value Flows.
easier domains able to obtain 40% mean improvement. those more challenging state-based tasks, Value Flows achieves 1.6x higher success rates than the best performing baseline. In addition, Value Flows is able to outperform the best baseline by 1.24x using RGB images as input directly (visual tasks).
scene-play, and D4RL adroit), while Value Flows
Average Performance Across Environments
Offline-to-Online RL. We next study whether the proposed method can be used for online fine-tuning. We hypothesize that Value Flows continues outperforming prior state-of-the-art RL and distributional RI. methods using on- line interactions, without any modifications to the distributional fow-matching losses. Results
100
Average Success Rate
80
60
in Fig. 3 suggest that Value Flows achieves strong fine-tune performance with high sample efficiency in the online fine-tuning phase. On puzzle-4x4 4x4-p1aayyy Value Flows achieves 15% higher performance than all of the prior offline-to-online RL algorithms. See Appendix Table for the full results and Fig. 6 for the complete learning curves. Taken together, Value Flows can be widely applied in both offline and offline-to-online settings.
0
500
1000
1500
2000
Steps (x1000)
Value Flows
IFOL
ION
FQL
RLPD
Figure 3: Offtine-to-olline evaluation. Using the same distributional flow-matching objective, Value Flows achieves higher average success rates. See Fig. for the full results.
5.3 THE KEY COMPONENTS OF VALUE FLOWS
Our final set experiments studies the key components of Value Flows: the regularization loss (Eq. 17), and the confidence weight (Eq. 7). To study the effects of these two components, conduct ablation experiments on two representative ttate-based OGBench tasks, varying the regularization coefficient À for the regularization loss and the temperature for the confidence weight.

---

Published as a conference paper at ICLR 2026
First, results in Fig. 4 suggest that adding the BCFM regularization loss into the distributional flow matching objective is important for making progress on the puzzle-3x3 task, while choosing the correct regularization coefficient À boosts the performance of Value Flows by 2.6x. We also observe that the best À value increases the sample efficiency of our algorithm by 2x compared to the unregularized variant. We conjecture that the BCFM regularization serves learning the full return distribution. Second, Fig. 5 demonstrates that using the confidence weight reweight the distributional flow matching loss increases the success rate, where an appropriate temperature т results in 60% higher success rate on average. In Appendix D.I, we include additional experiments studying the robustness of Value Flows against the number of flow steps (Fig. 7) in the Euler method and the number of candidates in rejection sampling (Fig. 8) for action selections.
efficient anchor for
Additional experiments. In Appendix D.3, we investigate the computational time and GPU memory consumption of Value Flows. Appendix D.4 demonstrates the capability of Value Flows in an MDP requiring exploration and risk- control. Appendix D.5, we include an ablation study showing that using Gaussian policies improves the performance of Value Flows on D4RL adroit t tasks. The ablations in Appendix D.6 study the role of the LRCFM loss, which mainly serves as regularization.
6 CONCLUSION
We present Value Flows, an RL algorithm that uses modern, flexible flow-based models to estimate the full future return distributions. Theoretically, we show that our objective generates probability path satisfying the distributional Bellman equation. Our experiments demonstrate that Value Flows outperforms control tasks.
offline RL and oflline-to-olliee RL methods in complex continuous
Limitations. Although focus on learning the full return distribution using flow-matching and reweighting our flow-matching objective using uncertainty estimation, remains unclear how to disentangle the epistemic uncertainty from the aleatoric uncertainty with the current method. In addition, our discussion is orthogonal to the policy extraction mechanisms. Finding the appropriate policy extraction method within the distributional RI. framework might reveal the true benefits of estimating the full return distribution.
REPRODUCIBILITY STATMENT
We include additional implementation details for hyperparameters, datasets, and evaluation pro- tocols in Appendix E. Our open-source implementations, including baselines, can be found https:: / ////itttt hitthuub...
htt///hhngiizzzenngge wwwpng/aaaeeeeeeeeeeee flow.... We run experiments for state- based tasks on A6000 GPUs for up to 6 hours, and run experiments for image-based tasks on the same type of GPUs for up to 16 hours.
ACKNOWLEDGEMENTS
We thank Catherine Ji, Bhavya Agrawalla, and Seohong Park for their helpful discussions. We thank Seohong Park for sharing the data for some baselines in our experiments. This work used the Della computational cluster provided by Princeton Research Computing, as well as the Ionic and Neuronic computing clusters maintained by the Department of Computer Science at Princeton University. This work was in part supported by an NSF CAREER award, AFOSR YIP, ONR grant N00014-22-1-2222 This material based upon work supported by the National Science Foundation under Award Numbers 2441665, 2237693, and 1941722. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
REFERENCES
Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 32:96, 2019.
10

---

Published as a conference paper at ICLR 2026
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. OPAL: Offline primitive discovery for accelerating offline reinforcement learning. In International Confer- ence on Learning Representations, 2021. URL https: https://ppenreieew oorperneeeeeeeeet/frre1eeee forum?id= V69L.GWJD1 TN.
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? The Eleventh International Conference Learning Representations, 2023. URL https::/ net wtw.orruneettttttttteeeoea forum?
Michael Samuel Albergo and Eric Vanden-Eijnden Building normalizing flows with stochastic interpolants. In The Eleventh International Conference on Learning Representations, 2023. URL
https: wwwpeeeeeeeveeeeeeeeeeeeeeeeeeee reveewevew eww net/ forum?
Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari, 2024. URL
attps://arxtee org/ wwt./oogg/////oooooooooo ab/p//bb//2455 12399.
L. Baird. Residual algorithms: Reinforcement learning with function approximation. Proceedings of the Twelfth International Conference, pp. 30-37, 1995.
Philip Ball, Laura Smith, lla Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data, 2023. URL httpt:://aaxtiiiii
www./oog///bboo/ooo htt.://s///aa.oooooo//////////// 02948.
Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International conference on machine learning, pp. 449-458. PMLR, 2017.
Mare G Bellemare, Will Dabney, and Mark Rowland. Distributional reinforcement learning. MIT
Press, 2023.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclau- rin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, al. Jax: compos- able transformations of python+ numpy programs. 2018.
Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209. 14548, 2022.
Huayu Chen, Cheng Lu, Chengyang Ying. Hang Su, and Jun Zhu. Offline reinforcement learning via
high-fidelity generative behavior modeling. In The Eleventh International Conference on Learning Representations, 2023. URL https: httpss://ppeeeee http///ppppeeeeeeeeeee/ee/eeeeee net / forum 2iw=42zs3aaa22ee
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 14:15084-15097. 2021.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, pp. 1096- 1105. PMLR, 2018
Perry Dong, Alec M. Lessing, Annie S. Chen, and Chelsea Finn. Reinforcement learning via implicit imitation guidance, 2025a. URL https: arwiv.iiivv.///.v wwtp:/wwwwrr//////55556// 07505.
Perry Dong, Qiyang Li, Dorsa Sadigh, and Chelsea Finn. Expo: Stable reinforcement learning with expressive policies, 2025b. URL htl.tttts://// htt.t:::////xxx.ooo/o////a/ http//abb/2550.. /tw.a////527777766..6.
Perry Dong, Suvir Mirchandani, Dorsa Sadigh, and Chelsea Finn. What matters for batch online
reinforcement learning in robotics?, 2025c. URL https:/ http///axxvvv.////// ottp//iioorrreeeooe wwtpdiiv.rrraa///////a/ee/e 2255 08078.
Yaakov Engel, Shie Mannor, and Ron Meir. Reinforcement learning with gaussian processes. In Proceedings of the 22nd international conference on Machine learning, pp. 201-208, 2005.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning.
Journal of Machine Learning Research, 6, 2005.
11

---

Published as a conference paper at ICLR 2026
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor- actor-learner architectures. In International conference machine learning, pp. 1407-1416. PMLR, 2018.
Jesse Farebrother, Jordi Orbay. Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, and Rishabh Agarwal. Stop regressing: Training value functions via classification for scalable deep rl, 2024. URL
pettps//arxDar owroor///aaoo.. http//bs/24000 03950.
Jesse Farebrother, Matteo Pirotta, Andrea Tirinzoni, Rémi Munos, Alessandro Lazaric, and Ahmed Touati. Temporal difference flows. arXiv preprint (5t:::2503009817,, 2025a.
Jesse Farebrother, Matteo Pirotta, Andrea Tirinzoni, Rémi Munos, Alessandro Lazaric, and Ahmed Touati. Temporal difference flows, 2025b. URL http://ppp.//aaaa...o.a//////...
Kevin Frans, Seohong Park, Pieter Abbeel, and Sergey Levine. Unsupervised zero-shot reinforcement learning via functional reward encodings. In International Conference on Machine Learning, pp. 13927-13942. PMLR, 2024.
Kevin Frans, Danijar Hafner, Sergey Levine, and Picter Abbeel. One step diffusion via shortcut models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: wwwpe/eeeeeeeeeeeeeeeeeee .eetteeeeieeeeeeeeeeeeeee wwt.:/eneeeetteteeeeeeeeeeeo
Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep q-learning algorithms. In International Conference on Machine Learning, pp. 2021-2032 PMLR, 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2021. URL https: arxivv org/ wwt.///g/////ooooooe/...//...ese 07219 9.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning, 2021. URL https: www.aaavvvooooo org abaaabaaal00
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods, 2018. URL https: arxiv.iiiivvv//ii org/ abs 111110002997777 09477.
Scott Fujimoto, David Meger, Doina Precup, Ofir Nachum, and Shixiang Shane Gu. Why should i trust you, bellman? the bellman error is poor replacement for value error. In International Conference on Machine Learning, pp. 6918 6943. PMLR, 2022.
Izrail Moiseevitch Gelfand, Richard A Silverman, et al. Calculus of variations. Courier Corporation, 2000.
-aan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21253 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning. pp. 1861-1870. Pmlr, 2018.
Philippe Hansen-Estruch Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit learnnin as actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573. 2023a.
Philippe Hansen Estrrcch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit Ieaarnnn as an actor-critic method with diffusion policies, 2023b. URL https: // rxxiv org abs/ /2304 10573.
Dan Hendrycks and Kevin Gimpel. arXiv: Tal::666..8444 08415, 2016.
Gaussian error
linear units (gelus).
arXiv preprint
12

---

Published as a conference paper at ICLR 2026
Jacob Hilton. KI divergence of max-of-n. https://wwe http:////wwwwwcccc....o...../// k1 pdÉ, 2023. URL https:://// /tpp:://wwit /www://wwww///. jacobh. wtt.kkk/oookk....
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in
neural information processing systems, 33:6840-683 68510, 2020.
Michael Janner, Igor Mordatch, and Sergey Levine. Generative temporal difference learning for http://aaxxxx....aa//////aa. a201/000..2020... 14496.
infinite-horizon prediction, 2021. URL
JAX Developers. The autodiff cookbook. https://docs:
batebooks/autoooie http://aauuuoiiiiocooooooooooooo cookbook htmi, 2025. URL http://dcc...iidddde//////tatees aatett/nntebooks/attcciff www.book..ooooooooooooooooooo. html.
Vadim Kaplunovsky, Gradient, divergence, curl and related formulae. https://web2.
utexas. attp:as.eeda/raaaan//aaasse////s /wtp///ffoppo//o///////o../toftt pdf, 2016. URL https://web2.
Ramtin Keramati, Christoph Dann, Alex Tamkin, and Emma Brunskill. Being optimistic to be conservative: Quickly learning evar policy. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 4436 4443, 2020.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.690 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114 2013.
arXiv preprint
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning, 2021a. URL https: : aariiviv httpxi..rrg/bb//11111 06169.
lya Kostrikov, Jonathan Tompson, Rob Fergus, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization, 2021b. URL https::/aaxii..com
/2103.
08050.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off off-policy q-learming via bootstrapping error reduction, 2019. URL https: /arxivvvvvv.////////aa///aaaaava
wwwwwwwwwwwwwaa..aaaaaooeeeaeeae wwg//bs///aaaooooooooooooee /190 00949.
Aviral Kumar, Abhishek Gupta, and Sergey Levine. Discor: Corrective feedback in reinforcement learning via distribution correction. Advances in neural information processing systems, 33: 18560-18572, 2020a.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning, 2020b. URL httph::tt///.
httttt///aaxii http/aaavvvv/vaa///a//eee/...ava org/ aws///s0ss 04779.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning: State-of-the--ae pp. 45-73. Springer, 2012.
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International conference machine learning, pp. 6131-6141. PMLR, 2021.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic g-ensemble. In Conference Robot Learning, pp. 1702-1712. PMLR, 2022.
Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking. arXiv preprint ar arXiv:2507.07969, 2025.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. The Eleventh International Conference on Learning Repre- sentations, 2023. URL https: www.//eeeeeeeeeeeeeeeeeeeeeeeeee hwt.wwwwww.eeeeee........... forum? (orrmm?d==PqMRDDD PVMPVVMDCC
Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky TQ Chen, David Lopez-Paz, Paz, Heli Ben-Hamul and Itai Gat. Flow matching guide and code, arXiv preprint arXiv:2412.00 062644 2024.
13

---

Published as a conference paper at ICLR 2026
Xingchao Liu, Chengyue Gong, and qiang liu. Flow straight and fast: Learning generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2023. URL https: wwtpp//ppwe....eepeeeppe.eee...p net xett/oorummddxx//]11im
Xiaoteng Ma, Junyao Chen, Li Xia, Jun Yang, Qianchuan Zhao, and Zhengyuan Zhou. Dsac: Distributional soft actor-critic for risk-sensitive reinforcement learning, 2025. URL https:
www.xii..rggaaaa.. 2004. 14547.
Yecheng Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative offline distributional reinforcement learning. Advances in neural information processing systems, 14:19235-19247, 2021.
Max Sobol Mark, Archit Sharma, Fahim Tajwar, Rafael Rafailov, Sergey Levine, and Chelsea Finn. Offline retraining for online rl: Decoupled policy learning to mitigate exploration bias. arXiv preprint (rXxi:2310008588 2023.
Borislav Mavrin, Shangtong Zhang, Hengshuai Yao, Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional reinforcement learning for efficient exploration, 2019. URL https::////// //axxiv..vvvv
arg/abs/1900 06125.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare. Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 5187540)):52-- 2015.
Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Nonparametric return distribution approximation for reinforcement learning. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 799-806, 2010.
Timothy H Muller, James L Butler, Sebastijan Veselic, Bruno Miranda, Joni D Wallis, Peter Dayan, Timothy EJ Behrens, Zeb Kurth-Nelson, and Steven W Kennerley. Distributional reinforcement learning in prefrontal cortex. Nature Neuroscience, 27()):003-40) 2024.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets, 2021. URL https::/arxiv..ort orq.abs//00/....ooo 09359.
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh. Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-qi: Calibrated offline rl pre-training for efficient online fine-tuning. Advances in Neural Information Processing Systems, 36:62244 62269, 2023.
Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning, 2024. URL https: /ttp:s://aaiiiia org/ abs 22003. 05479.
Victor M Panaretos and Yoav Zemel. Statistical aspects of wasserstein distances. Annual review of statistics and its application, 6(1):405-433 2019.
Seohong Park, Tobias Kreiman, and Sergey Levine Foundation policies with hilbert representations. In International Conference on Machine Learning, pp. 39737-39761. PMLR, 2024.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking 200
offline goal-conditioned rl, 2025a. URL httpt::///axii.crrtrreee//
Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine. Horizon reduction makes rl scalable. arXiv preprini 04X1122500604168, 2025b.
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. arXiv preprint arXiv:2502.02538, 2025c.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression Simple and scalable off-policy reinforcement learning, 2019. URL htRLtts///aaxi..ooo abs/1911 00177.
Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pp. 188 204, PMLR, 2021.
14

---

Published as a conference paper at ICLR 2026
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Martin Riedmiller. Neural fitted q iteration-first experiences with a data efficient neural reinforcement learning method. In European conference on machine learning, pp. 317-328. Springer, 2005,
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:151 1.0595,, 2015.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 7266-2265, pmlr, 2015.
Yang Song. Jascha
Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. URL https: : oppnrevieee www. forum? id=PxTIG12RRHS,
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, (1):944,, 1988.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning, 2023. URL httpss//aaxxiv.org abs/2305 09836.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning, pp. 10268-10278 PMLR, 2021.
J. N. Tsitsiklis and B. Van Roy. An analysis of temporal-diiference learning with function approxi- mation. In JEEE Transactions on Automatic Controle, pp. 674-690, 1997.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning, 2015. URL https: / //wpivvvvvv///////////.../a/iwi org/ wttg/ab////te /109 06461.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Kaiwen Wang, Kevin Zhou, Runzhe Wu, Nathan Kallus, and Wen Sun. The benefits of being distributional: Small-loss bounds for reinforeement learning, 2023a. URL https://arxiv 2305. 15703.
Kaiwen Wang, Owen Oertell, Alekh Agarwal, Nathan Kallus, and Wen Sun. More benefits of being distributional: Second-order bounds for reinforcement learning, 2024. URL https:
// //axxiv.org/abs////...... 224022/2./b/22222...// 07198.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning, 2023b. URL
061 06193.
Christopher JCH Watkins and Peter Dayan. Q-learning Machine learning, ((3):279-292. 1992.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning, 2019. URL https:///// arxiv. org/ /as////11..1111/
Hanlin Yang, Chao Yu, Siji Chen, et al. Hybrid policy optimization from imperfect demonstrations. Advances in Neural Information Processing Systems, 36:4653-4663, 2023.
Chongyi Zheng, Seohong Park, Sergey Levine, and Benjamin Eysenbach. Intention-conditioned flow occupancy models. arXiv preprint arXiv:2506. 08900, 2025.
15

---

Published as a conference paper at ICLR 2026
Zhiyuan Zhou, Andy Peng, Qiyang Li, Sergey Levine, and Aviral Kumar. Efficient online rein- forcement learning fine-tuning need not retain offline data, arXiv preprint arX2:22412.07762 2024.
16

---

Published as a conference paper at ICLR 2026
Table 2: Summary notations and their meanings.
Notation
Meaning
S
the state space
A
the action space the initial state distribution
P
A(X)
set possible probability distributions over a space the reward function
((8,0)
the discount factor
p(s' (,,))
the env invommeenal transition distribution
h
RL timestep
PX
CDF and PDF of the random variable X
D (1,,
the offine dataset
2", (s,
return random variable and conditional return random variable of policy the function policy
Q
T*
the distributional Bellman operator
<m C~, I)
noise sampled from the standard Gaussian distribution
ole t), ofe a,
time-depen dee-deeendent diffeomorphic flow
v(x' f) u(2" a,))
time- dependetnn vector field
p(a' f), 2f f,s.a
time-dependent probability density path return vector field and return probability density path at iteration the distributional flow matching loss
(,,2),, Pela (,),
CDeM
DDCP
the distributional conditional flow matching loss expectation and variance estimation of random variable X the confidence weight
B[x]. Var(X)
0(),0,6)
the confidence weight temperature
P(Y)
probability event Y divergence operator
div(-)
LerM
the conditional flow matching loss Jacobians of vector field v and flow &
Jo
(ov(X)
covariance estimation random variable X polynomial higher order terms the bootstrapped conditional flow matching loss the weighted DCFM loss, the weighted BCFM loss, and the Value Flow loss the regularization coefficient of the weighted BCFM loss
0(-)
LBCFM
L.DCFM LuBCFM, Lissuue Flow
n
the flow BC policy and flow one-step policy Q estimation state-action pair (s,a)
(s.a) a)
Cine-step Flom
the flow one-step policy loss the flow BC distillation coefficient in the flow one-step policy loss
A PRELIMINARIES
A.1 THE RETURN AND THE Q FUNCTION
The goal of RL is to learn policy S + A(A) that maximizes the expected discounted re- turn AL
D / ((S,, An)] over trajectories (So, Ag. S1, A1, ..... Starting from a state-action pair (s, a). we denote the conditional trajectories as (S, = s, Au = a, S1, A1.
This notation allows to define the Q function of the policy
Q=
(5 .....) [Em= ~r(SS, Aw)]. Prior actor-critic methods (Fujimoto et al., 2018; Haarnoja et al., 2018) typically estimate the Q function by minimizing the temporal difference (TD) error (Ernst et al., 2005; Fu et al., 2019; Fujimoto et al., 2022). We will construct TD errors to learn the entire return distribution Sec. 4.2.
A.2 DISTRIBUTIONAL REINFORCEMEN LEARNING
Lemma 1 (Proposition Morimura et al. (2010)). For a policy n, return value E [Amine Emarly state S, action a A, the cumulative distribution function of the random variable T 7(, a).
17

---

Published as a conference paper at ICLR 2026
L.e, Frzz(: 8, s, satisfies
2 - r(s, 8 8,
(z-n(3,a)
Fr=z(= &,, = (1t0))
Fz
(8)
For completeness, include full proof.
Proof. By the definition of identity in distribution, we have
F (2 8, s, F (((, (y s,a).
Expanding the definition the CDF (((5,)))2((5'11)) gives us
2((5. (1) 8, a) P(r(s, + A') S z s,
=p) [P (r(s, a) + +Z S s a')l
Z(s', d)) =-r(8, r(s, a)
P
s s',a'
Z(s' ((sssa)) = =-r rs)(,, ?
P
s',a'
r(s, a) 8 d
Fz
in (a) we use the law of total probability and (b) we use the definition the CDF Fz. Thus, we conclude that 2f 8, a) [l)))"ll..)) Fz s-r(s.0)
The connection between the CDFs of 7 and Z also allows to derive an identity between their PDFs, suggesting an alternative definition the distributional Bellman operator:
Lemma (Chapter 4 of Bellemare al. (2023)). For policy n, return value E [=mins =mar] stale E S, and action a E A, the probability density function of the random variable T Z(s, a), i.e. pr=z(- z( (-1 8, a, satisfies
2 (=-r(3,a) s', s., (-=="1300000
PT= -
pz
7
?
Alternativly, the distributional Bellman operator can operate on the density function directly,
T"pz( s, _ pT z(2 s,
For completeness, we include a full proof.
Proof. Since the PDF continuous random variable can be obtained by taking the derivative of its CDF, we can easily show the desired identity by taking the derivative on both sides of Eq. 8.
d FT= s,
r(s, a)
Fz
s'
d=
dz
7
pr=2(3 s,
(a) E Bp(r
pz
r(s (s,
s',
?
where we use the chain rule of derivatives in (a).
The stochasticity of the distributional Bellman operator TT mainly comes from (7) the environmental transition p(s'' 8, a) and (2) the stochastic policy T(a s). We next justify the unique fixed point z" of the distributional Bellman operator T
Lemma 3. The distributional Bellman operator T admits the unique fixed point Z Specifically, we have
T= a) d 2= a); T pz 8,a) pz- (= 8, a).
(10)
18

---

Published as a conference paper at ICLR 2026
Proof. By definition of (" a) and T", we have
Z (s, = r(S. = s, s,A0 a) + Σ
20~2r(5 An)
r(s, a) +
r(8,0) + 7
_ (s,a),
with S' and A' being random variables of the next state-action pair, in (a) we use the stationary property of MDP, and in (b) we plug in the definition of z= Thus, we conclude that T 2 ( a) ("(s,a), a). By Lemma 1, the CDF of 2d (s, satisfies
2 2- r(s, (2-r(s,a)
Fz- (2 8, = FT z- (2 8, a) = Epter (a))))
Fzz
8 8²,
Taking derivatives with respect to z on both sides, the PDF of 7" (s, satisfies
Pz- (z 8, a) = - -B(ollaa)) pz"
(z-r(s,a) ( a)
2 22-
s' ,,
We conclude that 7 pz= (z 8, a, - pz-(z a) using the definition of the distributional Bellman operator T on PDFs (Lemma 2).
A.3 FLOW MATCHING
Flow matching (Lipman et al., 2023; 2024; Liu
Algorithm 2 Euler method for solving the flow ODE (Eq. 11).
al., 2023; Albergo & Vanden-Eijnden, 2023) refers to family of generative models based on ordinary differential equations (ODEs), which are close cousins of denoising diffusion mod- els (Sohl-Dickstein et al., 2015; Song et al., 2021; Ho et al., 2020) based on stochastic dif- ferential equations (SDEs). The deterministic
1: Input The vector field v, the noise the initial flow time fit the final flow time timall and the number of steps T.
2: Initialize /nnit Al larat larat bunt, x² 6.
3:
for each step = limit lini + At, linal do
nature of ODEs equips flow-matching methods with simpler learning objectives and faster infer- ence speed than denoising diffusion models (Lip- man et al., 2023; 2024). The goal of flow match-
4: I'+At < x² + v(x" [t) At.
5: Return fadd.
ing methods is to transform simple noise distribution into target distribution some space X' C RR We will use to denote the flow time step and sample the noise from a standard Gaussian distribution A La) throughout our discussions. We will use X to denote the d-dimensional random variable that generates the target data samples using the distribution px-
Specifically, flow matching uses a time-dependent vector field v: [0, 1] Rdd Rd to construct time- demendennnd diffeomorphic flow Φ [O, 1] x Rd + Rd (Lipman et al., 2023; 2024) that realizes the transformation from a single noise e generative sample & by following the ODE
d
a(c () = v(((c (t) It), dt
0( =c, o(e 1 = f.
(11)
The vector field v generates time- -emeedeenndent probability density path p [0. 1] Rd A(R") if satisfies the continuity equation (Lipman et al., 2023)
pppla (t) + div(p(a* t))(=a [t) = 0,
(12)
where div(-) denotes the divergence operator. Prior work has proposed various formulations for learning the vector field (Lipman et al., 2023; Liu et al., 2023; Albergo & Vanden-Eijnden, 2023). We adopt the simplest flow-mater mawching objective, based on optimal transport (Liu et al., 2023), called conditional flow matching (CFM) (Lipman al., 2023). Using UNIF([O, 1)) denote the uniform
19

---

Published as a conference paper at ICLR 2026
distribution the unit interval and x' fo + (1 t)e to denote linear interpolation between the ground truth sample x and the Gaussian noise e, the CFM loss can be written as
LerM = E (-U)
[||=(2* [t) - (xc c)
(13)
Practically, we can generate sample from the vector field v by numerically solving the ODE (Eq. 11). We will use the Euler method (Alg. 2) as our ODE solver following prior practice (Liu et al., 2023; Park al., 2025c).
B THEORETICAL ANALYSIS
B.1 ESTIMATING THE RETURN DISTRIBUTION USING FLOW-MATCHING
Proposition 1. For a policy n, an iteration E N, a return value 2t [amin: Zmaz]; a flow time t E 1), state s E S, action E A, and a vector field ox(2" t, a)) that generates the probability density path ps(2* i, 8, a), the new vector field 6x41(2* 1, a) generates the new probability density path (=41 t,8,))
Proof. By definition, a vector field generates a probability density path, meaning that they both satisfy the continuity equation (Eq. 12) (Lipman et al., 2023). We will check the continuity equation for u (=' 11,, a) and p (2' 8,8 a). On one hand, for P1+11 we have
01" Pk+t 1 (2* (,,,a)
(
1
2t r(s, a)
Øt -pp( B
Pk
t, a'
(
- r(s, a)
1, 1,8%00 al
0h Ph
(c) pp(s ((a))
div
(
(2'-rr8s (5-r((,,0) (3,
t, (')
( 2 r(8,)(
( )
t,
in (a), we plug in the definition of Ph++1 (2* a). in (b), we swap the partial differentiation with the expectation because the integral does not depend on time t, and in (c), we apply the continuity equation for pk- On the other hand, by the definition of uk+1 we have
(2' | [t,,s, a,,, a,)
1
2 (95-r(s,00
r(s,
t,s,a 0k ( 2t -
(=--r(s,a) r(s,a)
- Pk
t, (.
Since the divergence div(-) is linear operator (Kaplunovsky, 2016) and the expectation does not depend on the return value 2, swap the divergence with the expectation, resulting in
i 0t Pk+ (2' at, div (2*+1(2" a) 4) It, s,),
(14)
which means U and 1 follow the continuity equation exactly.
Lemma 4. Given a vector field uk. the distributional flow matching loss (Eq. 4) has the minimizer argmin, CDrM(U, vk) Uk+1-
Proof. Since LDFM is an MSE loss, intuitively, the minimizer of it will be 66+1- Formally, we compute the minimizer of LOrM by using the calculus of variations (Gelfand et al., 2000) and deriving the functional derivative of CDEM with respect to v, ie. (L 08)/
L (v, vk) 2D(s, a, r) 1 a) (o(=* (1,,,)) - ex+i(=" ()),
8v
Setting this functional derivative zero, have arg min, LDEM(V, Uk+1-
20

---

Published as a conference paper at ICLR 2026
Lemma 5. Given vector field Uy. the distributional conditional flow matching (Eq. 5) can be rewritten as
[[o(r + 4 (1,8,)) - us. (2¹) 8', a
(D, ub) = E a' a'mon('ll''), ".aaall""))""pppp pps (="(e,o'aa')
Proof. With slight abuse of notations, use ž to denote the samples from the convolved probability density path pa (( r t, s, a,// Setting (3* - rr/ = 2", we have
[ (0(5" [t,,,)) -
(7-
2 -2-
t, t,s',a' , 1)
(OCEM(U, ub) = E
at "" d'on7(a' "="),,"">"""" pa
?
E (s.a,,,))) D,errrr a a'mur(''l'"),= t's''a'''
[v(( + (x
(15)
Proposition 2. For a policy n. a vector field Us that generates the probability density path Pa ar iteration k E N, and candidate vector field have LDEM(U, v&) LDCFM(U, Uk) + const,, where the constant is independent of Therefore, the gradient V VoLDEM(,, vi) is the same as the gradient V CC,DFM(U,
Proof. We first expand the quadratic terms LDEM.
oft, (s,a) uk t, 2º -,-- d.dd) 2
(1,-- 7
= 2(, s,a)² 20(t, 2¹ s, a,))
t, 2¹ (,,---1
s', + Ub
(t, 2' 2'-- (.2--r/ s'
2
Since only the first two terms depend on the vector field v, we next examine the expectation of them respectively.
[u(2*
[,,,,a] (a)
[u(=" [1,5',a')
2 (2"),1("")1,s,)))
((,.a.)- (=
[u(2"
(t,,,))0441( 1 s,,
7 (
1E
ph x''rr t, t,s', a. Uk
2--rr t, aass
(b) B(e.ar)) (=')(("
u(2" (,,
1,
(c) (s.0,r,5 (D
u(2' (t,,,aee a'"mta'ls")='">3pp (ss)"
x -
t, s, ).)
in (a), we plug in the definition of PE++1 and use the next state s' in the transition from the dataset as the sample from the transition probability, in (b), we plug in the definition of U441, and, in (c), we use the linearity of expectation. Thus, we conclude that (DEM vw) = LDCFM(U, vx) const., where the constant is independent of v.
B.2 HARNESSING UNCERTAINTY IN THE RETURN ESTIMATION
With slight abuse of notations, prove the following Lemmas a generalization of Proposition and Proposition 4 to the standard flow-matching problem. We will use the notations introduced in Appendix A.3.
21

---

Published as a conference paper at ICLR 2026
Lemma 6. (Formal statement of conclusions from Frans al. (2025)) For vector field learned by the conditional flow matching loss LCFM (Eq. 13) for fitring data from the random variable X, noises sampled from (0, Ia)- have the vector field at flow time 0 produces estimate for the expectation I [
[[ = -E--N((..a) [u(e 0))
Proof. We note that LCFM is also MSE loss, and will use the idea in Lemma 4 to find its minimizer. Specifically, we use the calculus of variations and derive the functional derivative of CCFM with respect to v, i.e., &CcrM(o)/Ou:
8LCAM(U) 2E
[o(x* 1t) (x - c)]
x-px (10)
t~UNIF((0.1]) (0
20(x" t) (m [x-
Setting this functional derivative zero, have
0x (x" t) arg min LCFM (v) E p(c..ul))) (r-
x²
At flow time 0, we have in 20 (1 1) 1)- sampled independently, the conditional distribution p(æ, x², reduces to p(z, x. 0) px(x).
c Since x ~ (x(x) and NO, Ia) are
Plugging this specific conditional distribution into the minimizer v*, we have
(c () [x-px(x) - c) (x) [x]
Taking expectation over e N,, Ia) gives us
Ee~A((o,La) [e" 0)] = (x) B[X].
Thus, we conclude that, for a learned vector field v, Ba-a(o.aa) [u(e 0)] is an estimate for the expectation [E[
We next discuss the variance estimation using a learned flow. We will use J, E Rd to denote the Jacobian of vector field v(x' f) with respect to the input x' and Jg Rdxd to denote the Jacobian of the corresponding diffeomorphic flow o(c t) with respect to the input c:
de B
Se
Sd de
0z
de
Jo
J =
Bug Dor
Bdg
Sct
0c
Lemma 7. For a vector field v fating data from the random variable X with the corresponding diffeomorphic flow o, two independent noises eg and sampled from N Ta), the first-order Taylor approximation of the flow ò(Eo 1) around e is
d(co 1) = o(c 1) 1o(c) - c).
This first-order Taylor approximation produces an estimate for the covariance Cov(X):
Cov(X) -EENN(O.La) [Jole [1) 1)
Proof. The Taylor expansion of ((Ea 1) around &(c 1) is
(loo 1) = Ø(e 1) + Jo(e 1)(en t) + 0 - =
where O 2(l denotes terms with order higher than (co-ce (c) samples from the random variable using the first-order Taylor expansion of Φ(eo 1) around ő(e 1),
T (co -c). Thus, approximate
-= p(c 1) Φ(c 1) + Jo 1) c).
22

---

Published as a conference paper at ICLR 2026
By the property (affine transformation) of covariance, we have the covariance estimate for X at
Cov(X |e) = Cov(o(eo 1))
Jo(e [)) 11)
JØ(e 1) 1)T.
Taking expectation (O, Ia) both side, conclude that
Cov(X)
[ 1s] [Joee 1)Ja(e []]]]
= Ee-N((0,L
We are now ready to prove Proposition 3 for the return random variable 7 (s.
Proposition 3. For a policy state S. action E A, two independent noises and sampled from (0, 1), the learned return vector field v fitting the conditional return distribution 7 (s, a), the first-ondrr Taylor approximation of the corresponding diffeomorphic flow Φ(co 1, a) around e
(< 1,8, a) R dfe 1,,,a) + (,,,a(o0
We have the vector field at flow time t produces an estimate for the return expectation E [ (s, a)}, while the first-order Taylor upproximation of the flow produces estimate for the return variance Varl 2 (8,a)):
( @ dc (
2 1dTH a)] = E.-ro.nlle (, 0,),))), Var(Z"(s,a)) -E--N(O.1)
Proof. Applying Lemma 6 and Lemma 7 to the 1-dimensional conditional return random variable 7 (8, a), we get the desired estimates.
We next discuss a lemma that relates the Jacobian of the flow Jg to the Jacobian of the vector field Jg using a flow Jacobian ODE.
Lemma 8. For learned vector field fitting data from the random variable X with the corresponding diffeomorphic flow o, noise sampled from (0, 1a), the flow Jacobian Js and the vector field Jacobian Ju satisfy the following flow Jacobian ODE,
d = (t)o(ee)t) Jo(e lai
(16)
where = z follows the flow ODE (Eq. 11).
Proof. By definition, the vector field and the diffeomorphic flow & satisfy the flow ODE (Eq. 11) for any flow time / [0, 1].
= (()(c(t)(() x² ='= (
Taking Jacobians with respect to the d-dimensional noise on both sides gives us
[=] J. [t)Jo(c
Since the covariance of the noise is the identity matrix Ig. we set Jo(e 0) Ia to conclude the proof.
Similarly, the learned return vector field with its diffeomorphic flow á satisfies the following flow derivative ODE for the conditional return random variable 7" (s,
23

---

Published as a conference paper at ICLR 2026
Proposition 4. For state S, an action E A, noise sampled from N 1), and a learned return vector field e with the corresponding diffeomorphic flow o. the flow derivative 0 de and the field derivative 0v/Oz satisfy the following flow derivative ODE.
d Odb
δΦ (c (t,,,,,),,
0Φ de
dt de (c 0) = 0z
(2" lt, a))
(c 0, 0, =1
de
where 2' = d t, s, a) follows the flow ODE (Eq. 11).
Applying Lemma 8 to the 1-dimensional conditional return random variable Z² a), we get the desired flow derivative ODE.
Proof.
L
C COMPONENTS OF THE PRACTICAL ALGORITHM
C.I PRACTICAL FLOW MATCHING LOSSES
Our practical loss for fitting the return distribution involves the DCFM loss and bootstrapped regularization based on TD learning.
We will use a target vector field ö (Farebrother al., 2025b), which aggregates all the historical vector fields {vx)). J. to replace the single historical vector field in LDCPM (Eq. 15). The reasons for introducing the target vector field ê are twofold. First, using a target network a widely used technique in RL. This technique has been proven to smooth learning and enable standard TD learning (Mnih et al., 2015; Farebrother et al., 2025a). Second, a vector field minimizing the preliminary loss CDCPM might collapse to constant, e.g., O, because the vector field is fitting the output of itself. However, the same problem has already existed in a family of self-supervised representation learning methods, e.g., BYOL (Grill al., 2020), where they learn representations Instead of using the model from the previous iteration, they use an exponential moving average of the historical models (target model) predict target outputs. In the experiments of Grill al. (2020), this target network helps prevent model collapse, and this observation motivates us to use the target vector field in Value Flows. Additionally, Tian al. (2021) provides a theoretical characterization of how BOO avoids this failure mode. Using 2f ~ p(= it, s's a') to denote the sampling procedure of (I) first sampling a noise c A((0, 1) and (2) then invoking the Euler method (Alg. 2), we have
predict output from the model itself.
(U) EE (sans"))=D=UUuu((()) [cor | a
(,,,,) (0))))))))))
(,,,,,))))
(17)
In initial experiments, naively optimizing (DCE((U) produced trivial performance some tasks. To stabilize learning, we use the return predictions at the next state-action pair (s', a') and the flow time = 1 to construct bootstrapped target return and invoke the standard conditional flow matching loss (Eq. 13). The resulting loss function, called hootstrapped conditional flow matching (BCFM) loss, resembles the standard Bellman error. Specifically, using 2to r(s,a) -ye+ to denote the target return, and using 2ID = tzip + (1 1t-t) t)e to denote a linear interpolation between =1 and e, the BCFM loss can be written
(B) = E (= [[v(2tD a, - (ztD -e)) a'>r((a))s)) "pp= 1,)),a')
Therefore, using À to denote a balancing coefficient, the regularized flow matching loss function is LDCFM()) + \LRCFM(V). Our complete loss function also incorporates the confidence weight (Sec. 4.3) into the DCFM loss and the BCFM regularization:
(UDCEM(T) E. ".a.n.e)>DD UN),
(w(s,,,)) (u(r + 72" (,8,a) vn (=* 1,, a²))
a' d'oxt(a' (e), pppt"
L. C. (v) (K.a.r.)) ) t~ UN))((0.1) d'ax(a'' (or(alle))a -pp (1,5',a))
[u(ssa,c) (olc,,c)(((((fff)))) st,,,)) - c))
We Lvatue Flow (v) (WD + ALWBCEM(U) denote the practical loss for fitting the return distribution.
*The same noise is used sample z! and construct stp and =1p e.
24

---

Published as a conference paper at ICLR 2026
cube-triple
humanoidmaze-meddum
cube-double
100
100
100
Success Rate
80
80
80
20
20
20
0
500
1000
1500
2000
500
1000
1500
2000
500
1000
1500
2000
Steps (x1000)
Steps (x1000)
Steps (x1000) scene
antmaze-large
puzzle-4x4
100
Success Rate
60
60
60
40
40
40
20
20
20
0
500
1500
2000
1000
500
2000
Steps (x1000)
Steps (x1000)
Steps (x1000)
Value Flows
FQL
IFQL
IQL
IQN
RLPD
Figure 6: Value Flows continues outperforming prior methods with online interactions. Value Flows can be used directly for online fine-tuning and achieve strong performance without modifications to the distributional flow-matching objective. We aggregate results over seeds.
C.2 POLICY EXTRACTION STRATEGIES
We consider two different behavioral-regulariz policy extraction strategies for offline RL and offline-to-online RL. First, for offline RL, following prior work (Li et al., 2025; Chen et al., 2022), we rejection sampling maximize estimates while implicitly imposing KL constraint (Hilton, 2023) toward a fixed behavioral cloning (BC) policy. Practically, for a state s, we learn a stochastic BC flow policy 7 S Rd A that transforms d-dimensional noise ea NO, Ia) into action r (s, ea) E A using the standard conditional flow matching loss LBC Flow (TF) (Park et al., 2025b). Rejection sampling first uses the learned BC flow policy to sample
of actions {ai,--. aan).
and then selects the best action that maximizes the Q estimates (Eq. 6):
((s, = Eo- 0, s, a)],
a"
arg max
((s
{ar aN ax a.
Second, for online fine-tuning in Rd + A to minimize DDPG-style loss with behavioral learn flow one-step policy regularization (Fujimoto & Gu, 2021). This loss function guides the policy to select actions that maximize the Q estimates while distilling toward the fixed BC flow policy:
RL, following prior work (Park et al., 2025c),
LOne-stop Flow (TO (TT = ....D.ee [- [-Q(s, (8, ea) + (s ea) (s, =
(18)
where controls the distillation strength. The benefit of learning parametric policy is introducing flexibility for adjusting the degree of behavioral regularization during online interactions, mitigating the issue of over-pessimism (Lee al., 2022; Zhou al., 2024; Nakamoto al., 2023).
D ADDITIONAL EXPERIMENTS
D.1 ANALYSIS ON ABLATIONS
In this section, we conduct full ablation experiments to study the effects of (1) the regularization loss coefficient, (2) the confidence weight, (3) the number of flow steps in the Fuler method, and (4) the
25

---

Published as a conference paper at ICLR 2026
Table Full offline evaluation on OGBench and D4RL benchmarks. We present the full evaluation results on 49 OGBench tasks and 12 D4RL tasks. Following prior work (Park et al., 2025c), we use an asterisk (*) to indicate the task for hyperparameter tuning in each domain. We aggregate the results over 8 seeds (4 seeds for image-based tasks), and bold values within 95% of the best performance for each task.
BC
RQL
ReBRAC
FBRAC
IFQL
FdL
CSI
IQN
COENC
Vilue Plona
cooe-dssblerp
D-6-0
1:4:1
7:5-3
23.01 10
0--
35 66:
6:00
24:1-
02 =
76-27
B:k-d
0-5-6
3-3
8-3
0:000
15:4-6
06
73
cuse-dbubi htt.g//sogggttaaa-cac-----
sube-dsuble-ppe htg.lleeee.----taaaaaa--e-eeee
4.2.1
222
17.4
00
82
14:06
31.133
96
100 11
1
1
aoune play singleiase Leaks
02:07
15.15
14:5
06
04
04.52
acune play Mingleiase Leaks
242
3
0:00
2
8=1
(200
72.17
puzzle-2e ha3--paa--inn atask-task
5-6-3
33.6-
10
91-6-3
90 =-
71-13
09-44
Suszie-2sh-pleg singgetask-taaak2--
1.3
2
puazin-300 hol--ply--ann--eee
185
2
0.00
(mnle-345-play-e0 otask-rask
144
5-8-3
7-3
0-000
16-83
0-000
0--00
49
41:22
puzzle-tx--paj---
D.1.g
T.L.A
12..1
6.0.0
12.14
90.1 10
0.1.6
9-1.3
15..5
15.17
33
puscie
Lusks
0:6.0
4.5
7
2
2
7
0
16:06
cupe Leippe-pppp-e p.ay www.:aicatttaa-ee. sescca
0.0
283
21:20
0.4-0
D-5-0
cin-ttrppee-- sy-singl ara0
D..
0.6-0
0-0
0.4-0
1
1--00
C-4-0
T--3
cie--triplepp aa-ppaa---geeeeeeee btr.tgggetaab-aaoo
B4000
play
pen-hiran-11
71
76
110
11.L 22
53
28:1-1
07-
145
coor-huran-91
0:2-0
2-000
0=00
cear-elenad-o:
co3r-eapart-vi
2
8:11
1:11
3::1
159
jamer-cloted
2-11
11:99
0::0
6:00
1
Pawner xxperr-xpe:t-11
0:30
0.2.9
0.1
USD
relocate-sionedet
1
1:31
0.50
0.2-1
02:0
relocate-aspart-v
306
100
105 20
104.5.3
107.41
108 =
103.00
100 2
105:1
Yixual antraaee aediun nerigasa singlutast beakt vo
$4.5 16
81.53
82
visual anteasee sudiut nevigase singiulask Leak2 v0
00:2.3
90 =1
42:24
00:22
25.22
15:0
e-ring lataaaaaaaadd-- vi -antrazn--ndiinnnnee get wet.r---iiggaa-t-aaaeeeo
HD:
10.11
11.12
97 1
25
43.12
28 2
T1 36
vinual antraas-- wawppott paappttttaaissteee.. wiggstte-siieaa---aaaae tastt22222-
19:22
1023
9.55
13.233
visual cantrate-ttttteeeee soor--ptattt gat.-eiiggeeneeeeo sigglettat--aan
747
10
-antrane-ee -aae-gte--nnn www.--nnnggee--eeneeeee
(+)
34 X1
4114
simua cas dodble singlulesk Leai3
2-22
2
sisual col dodbe p.4y zingiutask LLase v0
9=1
0:0
0:00
01000
sibaa cul dodbe p.ay singiuteuk Laakb
0:0
vi sttaal--oon--gla---111 wtt.:oglyy-trggeetttaaaee
16:11
96--10
21--16 16
35-415
0-0
0-00
visual acane 21a--111a--22 aingleaaz -a3keeaaaat---3--00
0:20
0:0
0.00
1
120
10
visuel acace 24ay aisgle:ast LeakS-90
0.00
0:2.
0.0
020
0.0
D20
0.0
1
7=2
vioual siowal-puadee puscle-
0.5
12
050
0:00
0:00
622
185
viesal viesal-puxzle-D0 hii--la--inng-ee sta.----nn-tttk---aaaa...
111
0.00
253
6.1
0.C
0.0
120
250
Table 4: Offline-to-online evaluations on OGBench tasks. We present full results on 6 OGBench tasks in the filline-to-onling setting. Value Flows achieves strong fine-tuning performance compared to baselines. The results are averaged over seeds, and we bold values within 95% of the best performance for each task.
IQN
IFQL
FQL
RIPD
IQU
Value Floes
attumaae Large navigele
6
11 3 2=
87
90
0=0 3+2
hunanoidia ienaoiidaaae www.eeeeedddnneemm neviggat
23
14
12
7
1227
020
2
cube-daubl ep
29
12+
21
40=11 1111+922
02
0+0-
0+0-0+0
65
nube-cirt pll--ppass
55
14+6
27+5
+
0+0-0+0
5+1
70
puzsie-4x4--130 scene-play
2212 1111
2512-10
10112
8-3-
152
010
100 10
511-010
14.1--51 1
010-0110
010--00
11111-10111
01000001
10
1514-100
13 1033+000..1 1.0
number of candidates in rejection sampling. Again, we compute means and standard deviations over 8 random seeds.
To better understand the role of the BCFM regularization loss, we ablate over different val- ues of the regularization coefficient À and compare the performance of Value Flows. We present the results in Fig. 4. We observe that removing the BCFM regularization loss (A 0) results in poor performance on both cube-double- dobbbe--auullepl----a-a wwwggiiinnnttttaaaaeeeeass puzzle-3xe -eeelle------llllsnnep- wtwpppylllsslllllsllaap-l---eee-
and
while Value Flows's performance starts to in-
26

---

Published as a conference paper at ICLR 2026
cube-douue double
puzzle 3x3
cubre double
puzzle 3x3
100
FS-1
100
F5-1
100
NS-16
100
ME-16
P5
F5m2
80
F5-2
Surcess Paate
80-
M5-12
N5-8
80-
NC-12
N5-00
40
33
40-
40
20
20-
20
1500
250
400
200
200
1090
250
420
000
1500
500
500
1000
Steps (x] (x100.))
Steps (x1000)
Steps (x1000)
Steps (x1000)
Figure 7: Value Flows is robust against the number of flow steps in the Euler method the return vector field after a certain threshold (10).
Figure 8: The number of candidates used in rejection sampling has minor effects on the success rates of Value Flows.
crease after adding some regularizations. Choosing the correct regularization coefficient À boosts the mean performance of Value Flows by 2.6x and increases the sample efficiency of our algorithm by 2x. This suggests the BCFM regularization is an important component of Value Flows.
A key design choice of Value Flows is the confidence weight. We next discuss the ef- fect of using different temperatures in the confidence weight. As shown Fig. 5, we observe that, on the puzzle-32t 1oslee-333ppl- wttee--pyyssnnneaaaa--eeee stngletaskkttsske-- confidence weight (r = 0) can result in slower convergence of the success rate. On the bube-double-play-ee sag.yssssnnlttakkttaaa...
task, using constant
task, using constant confidence weight re- sults in slightly worse performance. We see that an appropriate temperature T boosts the success rate by 60% on average. Comparing against different magnitudes of the confidence weighting, find that our choice in Table is optimal.
Our final experiments study the effects of the number of flow steps in the Fuler method and the number of candidates in rejection sampling. We ablate the number of flow steps within {1, 5, 10, 20, 30) and ablate the number of rejection sampling candidates within {8, 16, 32, 64), presenting the results in Fig. 7. Note that we keep the number of flow steps for the flow policy fixed, and only vary the number of flow steps when sampling from the learned return model. We observe that, on the w---duube-pllyyy innssinnllee stask- steps (1 or 2) does not significantly decrease the performance of Value Flows. In contrast, using small number of flow steps the puzzle-3x3 -aap---pplylss----ee wingy-innnllnno wnglltase etask-tasst -takk 4 4-v0 task can drastically reduce the success rate. Specifically, using flow steps resulted in a 20% decrease in success rate compared to the results with 10 flow steps, while using or 2 flow steps will result in near-zero success rates. We also observe that the performance saturates after increasing the number of flow steps to 10. We conjecture that the number of flow steps affects the expressiveness and accuracy of the return distribution. Empirically, we found that 10 flow steps worked well and kept it fixed throughout our experiments.
task, even using a very small number of flow
We also study the effect of the number of candidates in rejection sampling for policy extrac- tion. Results in Fig. 8 show that Value Flows is robust the number of candidates in rejec- tion samples, except that using 32 candicates results in a 30% decrease in success rate on the puzzle- 1xiee-xx3-ppp -aallee-x-pply------ singgltaase
task4-v0 task. In our experiments, we found that 16 candidates are sufficient across a variety of tasks.
D.2 VISUALIZING THE CONFIDENCE WEIGHT
We visualize the confidence weight (log scale) with different temperature 7 in Fig. 9, varying the input & within [0, 1). These results indicate that the range of our confidence weight [0.5, 1] and the confidence weight is monotonic increasing function with respect to the input æ. We observe that smaller temperature 7 results in a drastically increasing value in confidence weight. Given the same input æ, a larger temperature 7 results in lower confidence weight.
D.3 COMPUTATIONAL TIME AND RESOURCE CONSUMPTION
We analyzed our experiment logs to investigate further the computational time and GPU memory consumption of different methods. Specifically, we analyze experiment logs on one state-based OG- Bench task (puzz le 4 -al-ppllysssa wtw.siinnnngttetttaeeeeaake (4ttaaskk)) and image-hased OGBench
27

---

Published as a conference paper at ICLR 2026
(GB)
60
04
Veuseccteedrnub
puzdeebol
IgL
FORAC
-283 1.100
ReBRAC
Figure 9: The confidence weights with different tem- peratures. smaller temperature results drastically increasing weight confidences for larger variance in returns.
Figure 10: Computational time and resource usage of different algorithms. Training Value Flows requires intermediate wall time and GPU memory compared with prior methods.
coptmaa policy
replace
not replace
terminal state
-2
$10
CVaRa.1
NV0100,8800
Q-optimal policy
Q-learning
[--130, 20)
C51
((-130, 20)
(M1100 20)
IQN
Value Flows
80
81
8g
-60
1000
2000
3000
4000
5000
((0, 10
[O 10
gradient steps
Figure 11: The machine replacement MDP. (Left) The machine replacement MDP consists of two actions replace and not replace with the reward distributed as Gaussians. This MDP is challenging because it requires exploration and risk-sensitive control. (Right) Value Flows converges to the optimal CVaRo.1 with a number of gradient steps similar to IQN, while C51 failed to converge within 5000 gradient steps. The Q-learning immediately converges the optimal expected return and able to learn a risk-sensitive policy.
comparing Value Flows to task baselines: IQL, ReBRAC, FBRAC, IFQL, FQL,, and IQN. We report wall-time and maxi- selection
mum GPU memory usage aggregated across 8 seeds (4 seeds for the image-based task) for different methods. Results in Fig. 10 suggest that training Value Flows requires computational time similar to training FQL, and the mean wall time of Value Flows is 50% of the mean wall time of IQN. To maximize GPU usage, Value Flows used substantially less GPU memory than FBRAC and IQN because they directly use the Euler method in the actor loss (FBRAC) or employ additional neural networks to encode quantiles (IQN). Taken together, training Value Flows requires intermediate wall time and GPU memory compared with prior methods.
D.4 SOLVING A ASK-SENSITIVE MDP
We now use a domain that requires exploration and risk-sensitive control demonstrate the capability of Value Flows. Following prior work (Keramati et al., 2020), we choose variant of the Machine Replacement MDP (with only 11 states) to conduct our experiments (Fig. 11 (Left)), measuring the CVaRa.1 performance instead of the expected return. This MDP is especially challenging because of the chain structure, and also requires risk senkseesi control due to the high variance in the reward distributions at the end of the chain. Therefore, it serves algorithms. Note that the optimal policy for expected return (Q-optimal policy) always chooses not replace, taking the risk of high variance reward 8g. In contrast, the optimal policy for maximizing
good evaluation task for distributional RL
28

---

Published as a conference paper at ICLR 2026
100
RABRKC
150
Vakte Ticks
ell BCTH
Volee R0mM
onl DCFV
Value Flows
nomatred
300
50
docheepert
aube-irppl
Figure 12: Using Gaussian BC policy instead of
Figure 13: Training Value Flows only using the BCFM loss yields near-zero success rates. In contrast, minimiz- ing the DCFM loss provides the most learning signal for Value Flows
flow BC policy for rejection sampling improves the per- formance of Value Flows on D4RL adroit tasks by 10%, matching the sfate-of-the-ane performance achieved by ReBRAC.
CV (-VaRo.--optimal policy) will choose replace at state $9 avoid the high variance reward.
Since the machine replacement MDP has a discrete action space, we remove the policy extraction procedure in Value Flows and use the learned vector fields to choose greedy actions directly, resem- bling the original C51 (Bellemare et al.. 2017) and IQN (Dabney et al., 2018) implementations. We use random policy and uniformly initialize the start state to collect a dataset with 10 transitions for offline training. We compare Value Flows against C51 and IQN and use CVaRp.1 instead of the Q-value (CVaRo.s) to optimize all methods. We compute the means and standard deviations of CV over 4 random seeds. Results in Fig. 11 (Right) show that Value Flows converges to the optimal CVaRg,1 with the number of gradient steps similar to IQN (600), while C51 failed to converge within 5000 gradient steps. We also observe that Q-learning immediately converges to the optimal expected return and is not able learn risk-sensitive policy.
D.5 USING GAUSSIAN POLICIES ON D4RL ADROIT TASKS
We hypothesize that flow- policies generally performed worse than Gaussian policies on D4RL adroit tasks (see Table and Table 3). This hypothesis is consistent with the experiment results in prior work, e.g., Table 2 of Park et al. (2025c). The D4RL adroit tasks consist of a very narrow dataset distribution: small amounts of human demonstrations, large amounts of expert data from fine-tuned RL policy, and a mixture of human demonstrations and behavioral cloning trajectories (Fu al., 2021). We conjecture that this narrow dataset distribution might impose difficulty for the expressive flow-matching policy to learn a multi-modal action distribution, resulting in sampling diitt--distribution actions during policy extraction.
To test this hypothesis, we ran additional ablations to study the effects of using Gaussian policies in Value Flows on D4RL adroit tasks. Instead of learning a flow BC policy in Appendix C.2, we learn Gaussian BC policy and then apply rejection sampling. We choose to conduct experi- ments on the four expert variants D4RL adroit tasks: pen-expert-vll dcor-expert-vi. harmer expert-vl, and relocate expert-vi aggregating results over 8 seeds. Results Fig. 12 suggest that using Gaussian BC policies improves the mean performance of Value Flows by 10%, matching the state-of-ther of-the-art performance achieved by ReBRAC.
D.6 ROLE OF THE BOOTSTRAPPED CONDITIONAL FLOW MATCHING LOSS
In the complete Value Flows flow matching objective, we include BCFM regularization loss LWBCTM to stabilize learning. This section studies whether the DCFM loss LaDCFM the BCFM loss LWBCFM provides the most learning signal for Value Flows. We hypothesize that simply using the BCFM loss train the vector field in Value Flows does not provide sufficient learning signal to model the return distribution.
To test this hypothesis, conduct ablation experiments comparing variant of Value Flows trained only using the BCFM loss against the original Value Flows. We evaluate the success rates OGBench tasks cube-triple-play-eeeeee sttaaaattaaaiiiliataataataaaa and
29

---

Published as a conference paper at ICLR 2026
computing means and standard deviations of success rates after training over seeds. Results Fig. 13 suggest that training Value Flows only using the BCFM loss yields near-zero success rates both tasks. In contrast, minimizing the DCFM loss provides the most learning signal for Value Flows. Thus, the BCFM loss mainly serves as regularization.
E EXPERIMENT DETAILS
We implement Value Flows and all baselines using JAX (Bradbury et al., 2018), adapting the FQL (Park et al., 2025c) codebase. Our open-source implementations can be found at https: http::///ithub../ con/ http://oooooy-ehheaaau---------
E. ENVIRONMENTS
We use OGBench (Park et al., 2025a) and D4RL (Fu et al., 2021) benchmarks perform experimental evaluations. OGBench is a benchmark for offline goal-conditioned reinforcement learning, with single-task variants for standard reward-maximizing RL algorithms. We choose the single-task variants for all domains in Table 3. Following prior work (Park et al., 2024), we use the default task from each domain tune hyperparameters. For locomotion tasks, the reward either - or O, where 0 indicates task completion. For manipulation tasks. the reward varies between -fask and depending on how many subtasks are complete. The datasets were collected by scripted non-Markovian policies that randomly interact with different objects in the environment. However, the successful completion of each task requires the agent to sequentially arrange those objects in specific order (sequential reasoning), which is unseen in the dataset. Hence, the agent must be able to generalize across trajectories the dataset and plan and sequentially combine learned manipulation skills (combinatorial generalization) (Park al., 2025a). We use the following datasets from OGBench for each domain.
cube-double-playy-v0
cube-trippe-playy-00
scene-play-v0
puzzle-33x-pllay-v0
puzzlee-444-paay--0
visual-scene--llayyww
yieuall-puzzle-33333paay---
visual--cube--bbbblee-payyyvvv
In our offlinetooonliinexxxrieeeeeee we chose to conduct experiments on the following task.
cuteetttiihleepplyyygnnnngeeessr
cube-triplln
senn--playyssnnnleeaaa--tt222222
D4RL is popular benchmark for studying offline RL. We measure the performance by the normalized returns following Fu et al. (2021). We use the following tasks from Adroit, which require dexterous manipulation within a 24-DoF action space.
pen-human-vi
30

---

Published as a conference paper at ICLR 2026
pen-cloned-vi
pen-expert-v1
door-human-vl
doer-cloned-vl
expe-ex
hammer-human-ve
hammer-cloned-vl
hammer-expezttvi
re relocate-human-vl
re relocate-cloned-i
relocate-
The subbe-double and sube-triple tasks involve complex pick-and-place of colored cube blocks. The scene tasks involve long-horizon reasoning and interaction with various objects in the scene. The puzzle-3x3 and puzzle-4x4 tasks require solving the "Lights Out" puzzles using the robot arm and further test combinatorial generalization. The Adroit benchmarks involve controlling 28-DoF hand to perform dexterous skills: spinning pen, opening door, relocating ball, and using a hammer to knock a button. The antmaze and humano dma ze tasks are designed navigate challenging maze by controlling
8-DoF ant and 21-DoF humanoid, respectively. The visual variants of these navigation tasks are challenging, as there is no low-dimensional state information, and the algorithm must directly learn from 64 64 x 3 RGB images. We include challenging state- state-based and image- baage- variants for manipulation tasks as they require tackling multimodal returns and long-hou morizon reasoning. We also include some image- based navigation tasks to evaluate the algorithms.
E.2 METHODS FOR COMPARISON
We compare Value Flows against 9 baselines, measuring the performance using success rates (OGBench) and discounted cumulative returns (D4RL... Specifically, we first compare to BC, IQL (Kostrikov et al., 2021a), and ReBRAC (Tarasov et al., 2023) as representative methods that learn scalar Q values with Gaussian policy. The second set of baselines is state-of-the-art methods that learn scalar Q values with a flow policy (FBRAC, IFQL, FQL) (Park et al., 2025c). We also include comparisons against prior distributional RL methods that model the return distribution categorical distribution (C51 (Bellemare et al., 2017)) or finite number quantiles (IQN (Dabney flow policy for all these distributional RL
al., 2018), 8), and CODAC (Ma al., 2021)). We baselines. Following prior work (Park et al., 2025c), we report means and standard deviations over random seeds for state-based tasks (4 seeds for image-based tasks). We describe the details of each method below.
BC. Behavior cloning maximizes the log likelihood of the action at given state. We train Gaussian policy to maximize log likelihood with (512, 512, 512, 512) MLP as the backbone.
IQL (Kostrikov et al., 2021a). Implicit Q-Learning is an offline RL method that uses expectile regression learn the best action within the support of the behavioral dataset. Following prior work (Park et al., 2025c), we learn Gaussian policy via AWR and perform hyperparameter search over the AWR inverse temperatures a in (0.3, 1,3, 10} for each environment (Table 6).
ReBRAC (Tarasov et al., 2023). ReBRAC is an offline actor-critic method based on TD3+BC (Fu- jimoto & Gu, 2021). This algorithm implements several design choices, such as layer normalization and critic decoupling for better performance. We perform sweeps over the actor and critic BC coefficients. For the actor BC coefficient dı, search values in (0.003, 0.01, 0.03. 0.1. 0.3. 1}. For the critic BC coefficient, we search over values in (0, 0.001, 0.01, 0.13. Results for different tasks can be found in Table 6.
31

---

Published as a conference paper at ICLR 2026
IFQL (Park et al., 2025c). IFQL is variant of Implicit Diffusion Q-Learning (IDQL; (Hansen- Estruch et al., 2023a)) with flow policy. IFQL uses the flow policy to propose candidates and uses rejection sampling to select actions in Value Flows. We select the number of candidates in rejection sampling from (32, 64, 128} (Table 6).
FBRAC (Wu al., 2019). Following prior work (Park et al., 2025c), we implement FBRAC as a variant of behavior- behavior-regulariized actor-critic (BRAC) with flow policies. This method requires backpropagating the gradients through the ODE solver. We select the BC coefficient a from (1.3, 10, 30, 300) (Table 6).
FQL (Park et al., 2025c). Flow Q-Learning uses a one-step flow policy to maximize the Q estima- tions learned by the standard TD error. It also incorporates a behavioral regularization term towards BC flow policy (Eq. 18). We consider the BC distillation coefficient a in {3, 10, 30, 100, 300, 1000} (Table 6).
C51 (Bellemare et al., 2017). C51 is distributional RL method that discretizes the return distribu- tion into a fixed number of bins and uses cross-entropy loss update these categorical distributions. We tune the number of atoms N in (51, 101) (Table 6). Because the original C51 algorithm aims to solve the Atari game with discrete action space, greedily selects actions that maximize Q-value estimated by the distributional returns (no explicit policy). However, since our evaluation benchmarks (Appendix E. have continuous action spaces, we choose to learn an explicit flow BC policy and use rejection sampling to select actions (Appendix C.2), resembling the greedy action selection in a discrete action space. Using policy extraction method of Value Flows, but also prevents sampling aat-of-distributio actions (Chen et al., 2023).
rejection sampling for C51 1 not only matches the
IQN (Dabney et al., 2018). IQN is a distributional RL method that approximates the return distribution by predicting quantile values at randomly sampled quantile fractions. We select the temperature и: the quantile regression loss from {0.7, 0.8, 0.9, 0.95) (Table 6). Similar to the adaptation in C51, we also learn an explicit flow BC policy and use rejection sampling (Appendix C.2) to select actions that maximize the Q estimated by IQN.
CODAC (Ma et al., 2021). CODAC combines the IQN critic loss with conservative constraints. Similar to Park et al. (2025c), we use a one-step flow policy to perform DDPG-style policy extraction (Appendix C.2). We fixed the conservative penalty coefficient to 0.1. We sweep the temperature in the quantile regression loss over {0.7, 0.8, 0.9, 0.95] and sweep the BC coefficient 0 over {100, 300, 1000, 3000, 10000, 30000) (Table 6).
Value Flows. Our method, Value Flows, uses flexible flow-based models to estimate the full distribution of future returns and to identify epistemic uncertainty across different states using the learned generative flow models. We describe the components of the practical Value Flows algorithm in Appendix C.I. In our experiments, we sweep the BCFM regularization coefficient A f0.3, 0.5, 1, 3, 10} and the confidence weight temperature E {0.01, 0.03, 0.1, 0.3. 1, In addition, the flow-based distillation strength ar is the key hyperparameter for our offline-to-online experiments. We select the distillation strength a from (10,30, 100, 300}. See Table for the specific values of the hyperparameter for each domain or task.
E.3 HYPERPARAMETERS
We include common hyperparameters for Value Flows and baselines in Table 5 and domain-specine hyperparameters for different methods in Table 6.
'The hyperparameters for - and humano' dmaze-medi um-navigate used for offline-t-online experiments.
32

---

Published as a conference paper at ICLR 2026
Table 5: Common hyperparametters for Value Flows and baselines.
Hyperparameter
Value
optimizer
Adam (Kingma, 2014)
batch size
256
learning rate
x 10-4
MLP hidden layer sizes
(512, 1122,512,00 512, 512,512)
MLP activation function use actor layer normalization use value layer normalization number of flow steps in the Euler method
GELU (Hendrycks Gimpel, 2016)
Yes
Yes
10
number of candidates in rejection sampling
16
target network update coefficient
x 10-3
number of Q ensembles image encoder
2
small IMPALA encoder (Espeholt et al., 2018; Park et al., 2025c) random cropping
image augmentation method
image frame stack
Table 6: Domain specific hyperparameters for Value Flows and baselines. The complete descriptions of each hyperparameter can be found in Appendix E.2. Following prior work Park et al. (2025c). we tune these hyperparameters for each domain from OGBench benchmarks on the default ("*") task in Table 3. indicates the values do exist."
Discount
IQU
ReBRAC
HBRAC
IQQL
FQL
CS1
IQN
CODAC
Value Hlos
Og
antneeee" ragennavi 1at--naeigggeeeeeeee
0150
10
322
10
0.7
10
0.01
30
0.995
32
30
100
aube-doubie-play
aube-triple-play
0.995
0.03
100
300
51
01.95
1000
0.03
300
puasle-3x3-p-ay
0.99
10
0.3
0.001
100
32
1000
101
0.8
0.05
100
0.5
0.3
puzsle-4x4-piet
scene-play
0.1
0.001
100
300
51
0.95
100
300
viausl-anttaae--edd
0.50
0.01
0.003
100
32
100
0.9
0.3
0.03
1.sssl-cuube-bbuee--p...
0.995
0.1
100
100
:i2uxl-puzzle->x3-p0
0.99
10
0.3
0.01
100
32
300
0.5
0.3
0.3
-lsual-scene-play
0.99
0.1
0.003
100
100
0.95
pen-human
0.59
300000
10000
0.8
10000
pen-cloned
0.99
10000
32
10000
51
0.5
0.8
10000
3
pen-expert
30000
3000
51
0.8
10000
door-human
300000
30000
0.9
10000
door-cloned
0.99
10000
128
30000
51
0.9
0.9
30000
10
0.3
0199
30000
32
30000
51
0.9
10000
10
0.3
hammez-hunan
hamer-oionn
0.09
100000
32
10000
51
0.7
0.8
10000
0.3
hammer-expp
(1.00
30000
32
30000
51
0.8
10000
10
oost hoannnn
celocate-dl bocste-clonel
3000
30000
51
0.9
30000
0.01
relocete-ewe eeeppet
0.99
30000
32
30000
101
0.9
0.9
10000
3
0.1
33


Reviews:
[{"review_id": "VxeEld8lV0", "reviewer": "Reviewer_n5xq", "summary": "The paper addresses the inability of prior works in distributional RL to model return distributions due to the approximations made by discretizing the distribution. The paper leverages the recent success of flow models to model the full return distribution and better characterize the uncertainty of the return distribution for better flow matching. The authors ensure that at every step, the return distribution follows the distributional Bellman equation. Doing so, they only need to learn the vector field $v$ which they do using a variant of the distributional flow matching loss (which has the same gradient). They perform experiments on a number of state-based and image based domains in both offline RL and offline-to-online RL.", "questions": "(1) Why do you think that aleatoric uncertainty would be better than epistemic?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The authors build on the recent success of flow models to model full return distributions for distributional RL which can be a major move forward from the discrete assumptions in prior works.\n\n(2) The authors derive a way in which they only need to model the vector field and the flow would follow from it. Seems a clean and nice way to avoid complexities still staying within the theoretical bounds. \n\n(3) The authors propose a flow matching loss that avoids transition dynamics and other complexities. The proposed loss has the same gradient as that of the true loss.", "weaknesses": "(1) While the authors mention that distributional RL has advantages in exploration and safety, the domain chosen by them does not seem to be using these properties of distributional RL. To study the effects of modeling the full distribution and uncertainty, a better domain can be chosen. \n\n(2) The preliminaries for flow matching should be better. Currently there is no explanation on how $\\phi$ is computed (which is relevant in the later sections of the paper). Similarly, since Lemma 2 is crucial for understanding of the method, it should be in the main paper and not in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bvGBEbHUQv", "reviewer": "Reviewer_pDXE", "summary": "The paper proposes a method to estimate the return distribution using flow matching. To estimate this distribution, the authors leverage a loss based on the distributional Bellman equation and demonstrate how to integrate it into the flow matching loss. This estimate can then be used to train policies either offline or in an offline-to-online setting. The method is validated on state- and image-based tasks from OGBench and D4RL.", "questions": "My main concern is about the DCFM loss. The loss seems ill-posed; in particular, $v=0$ is a solution of Equation (5). This loss is not the one used in the experiments, and the reader needs to refer to Appendix C.2 for a clearer understanding. Indeed, in Appendix C.2, the authors mention that the DCFM loss produced divergent vector fields. To address this issue, the authors proposed adding a regularization term. However, I believe the problem is intrinsic to the loss introduced in the main paper. The \"regularization\" wBCFM seems more sound than the main loss itself.\nI think the paper requires at least a major rewrite to properly highlight this \"regularization\" in the main paper. Similarly, the policy learning aspect should be explained in more detail in the main paper.\n\nQuestions:\n- Could you comment on the fact that $v=0$ is a solution of Equation (5)? Why should this loss work despite that?\n- You tested different regularization coefficients $\\lambda$. Could you also train using only the wBCFM? Does it significantly change the performance?\n\nLooking forward to the rebuttal.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- As far as I know, the approach is novel.\n- I think the topic is interesting and relevant to the community.\n\nHowever, I believe the weaknesses outweigh the positive aspects by a large margin.", "weaknesses": "- The contributions are not clear. The paper presents a loss in the main text and defers the details to the appendix. However, I believe the details are more important than what is stated in the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qkySMDXtVM", "reviewer": "Reviewer_V9wD", "summary": "This paper proposes Value Flows, a reinforcement learning (RL) framework that models the entire return distribution using flow-matching generative models instead of discretized or quantile-based approximations. The method introduces a distributional flow-matching (DFM) objective that enforces the learned density paths to satisfy the distributional Bellman equation. The authors further derive a flow derivative ODE to estimate the variance of the return distribution, which is used as a confidence weight to reweight the flow-matching loss. Value Flows is evaluated on 37 state-based and 25 image-based benchmark tasks, showing a reported 1.3× average improvement in success rates over prior offline and offline-to-online RL baselines (Table 1, Fig. 3).", "questions": "- C51 is an online distributional RL algorithm. When applying it to the offline setting, was any adaptation made—such as introducing conservative measures—or was it run directly under its original configuration?\n- How does the algorithm perform on the classic D4RL MuJoCo benchmarks? Does it also show a clear performance advantage there?\n- Is the return distribution visualization in Figure 2 based on samples from the in-distribution dataset, or on out-of-distribution samples collected separately in the environment? If it is in-distribution, could you additionally provide experiments with out-of-distribution samples? I believe that it would be more meaningful if the distribution learned by value flows demonstrates generalization.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Using the flow-matching approach to model the return distribution for each state-action pair is an interesting and intuitively reasonable idea. Flow matching is a more flexible distribution modeling technique that can handle complex distributions, making it more suitable for modeling return distributions in challenging tasks compared to previous distributional RL methods.\n- This paper provides a thorough theoretical analysis and derivation of the Value Flows method, and I believe it is a highly effective and reliable approach.\n- The experimental section provides substantial evidence supporting the authors’ claims. The return distributions learned by Value Flows are much closer to the ground truth compared to previous distributional RL methods. Moreover, on benchmarks such as OGBench and D4RL, Value Flows demonstrates clear performance advantages over the baselines.\n- The writing in this paper is well-organized, and I can easily grasp the information the authors intend to convey.", "weaknesses": "- The paper lacks an analysis of the algorithm’s time and GPU memory consumption. It remains unclear whether flow matching requires more computational time and resources compared to previous methods.\n- Some parts of the paper’s demonstrations are not entirely rigorous. For example, in Table 1, categorizing C51 under flow policies seems inappropriate to me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "427NKav6Hc", "reviewer": "Reviewer_uJ4U", "summary": "This paper proposes Value Flows, which utilizes expressive flow-based model to represent the conditional return random variable. Specifically, Value Flows learns a parameterized vector field model to estimate for the return expectation and variance for full future return distributions. Experiments on 37 state-based and 25 image-based benchmark tasks demonstrate the effectiveness of Value Flows when compared with various baselines in offline and offline-to-online settings.", "questions": "1.\tCould you provide more explanations of vector field, such as its illustration in flow matching and its specific role in Value Flow?\n2.\tWhy does Value Flows perform worse than ReBRAC and even IQL in D4RL adroit tasks?\n3.\tCould the authors compare the compute efficacy of Value Flows with other methods?\n4.\tIf the flow step is very small such as 1 or 2, how could it affect the performance of Value Flows?\n5.\tWhy is the target return vector field introduced in the algorithm of Value Flows?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe motivation of this paper of using flow matching to model complex, multi-modal distributions in the domain of RL is clear. Value Flows deeply integrates flow matching into distributional RL setting to estimate return distribution.\n2.\tBoth the theoretical and experimental parts are strong. For example, Value Flows achieves the best or near-best performance on 9 out of 11 domains.\n3.\tIn Figure 2, they authors provide the example to show Value Flows has better 1-Wasserstein distance when modeling return distribution than both C51 and CODAC.", "weaknesses": "1.\tThe method section is hard to follow as it is full of symbols and the logic is not explicit. The authors could provide more illustrations to explain the how to connect flow matching to distributional RL.\n2.\tThere are some typos. In Line 065, the format of citations is not correct.\n3.\tSome details, such as the complete loss function, are omitted in the main paper and provide in the Appendix, making the paper not self-contained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)