Expressive Value Learning for Scalable Offline Reinforcement Learning

## Abstract

Abstract Reinforcement learning (RL) is a powerful paradigm for learning to make sequences of decisions. However, RL has yet to be fully leveraged in robotics, principally due to its lack of scalability. Offline RL offers a promising avenue by training agents on large, diverse datasets, avoiding the costly real-world interactions of online RL.
Scaling offline RL to increasingly complex datasets requires expressive generative models such as diffusion and flow matching. However, existing methods typically depend on either backpropagation through time (BPTT), which is computationally prohibitive, or policy distillation, which introduces compounding errors and limits scalability to larger base policies.
In this paper, we consider the question of how to develop a scalable offline RL approach without relying on distillation or backpropagation through time. We introduce Expressive Value Learning for Offline Reinforcement Learning ( EVOR ): a scalable offline RL approach that integrates both expressive policies and expressive value functions. EVOR learns an optimal, regularized Q Q -function via flow matching during training. At inference-time, EVOR performs inference-time policy extraction via rejection sampling against the expressive value function, enabling efficient optimization, regularization, and compute-scalable search without retraining .
Empirically, we show that EVOR outperforms baselines on a diverse set of offline RL tasks, demonstrating the benefit of integrating expressive value learning into offline RL.

## 1 Introduction

Reinforcement learning (RL) is a powerful paradigm for learning to make sequences of decisions, having been successfully applied to the fine-tuning of pretrained large language models (LLMs). However, the success of RL in the language domain has yet to be matched in robotics. In contrast to the language setting, robot interactions occur in the real world, which can be costly, time-consuming, and may pose safety concerns. The constraints of real-world RL naturally motivate the offline RL setting, where an agent attempts to learn from a diverse, often sub-optimal dataset without further interaction with the environment.

Offline RL scalability spans three primary axes: (1) scaling data, (2) scaling models, and (3) scaling compute. To scale data, offline RL algorithms must learn from larger, more diverse datasets that are often sub-optimal and multi-modal (e.g., generated by multiple policies of varying quality). The need to model such complex data distributions necessitates more powerful models. A promising direction for scaling offline RL is to leverage powerful, expressive generative models such as diffusion (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021) and flow matching (Lipman et al., 2024; Esser et al., 2024).

Existing offline RL methods using generative models typically treat the generative model as the policy, enabling richer action distributions than standard Gaussian-based policies in continuous control (Hansen-Estruch et al., 2023; Chen et al., 2023; Ding & Jin, 2023; Wang et al., 2022; Espinosa-Dice et al., 2025; Park et al., 2025b; Zhang et al., 2025).
At a high level, diffusion and flow-based RL policies generate actions through an iterative denoising process, which requires backpropagating through the sampling steps (i.e., backpropagation through time).
Backpropagation through time is computationally expensive, memory-intensive, and can degrade the general knowledge of the underlying base policy (e.g., a vision-language model (VLM) in the vision-language-action (VLA) setting) (Ding & Jin, 2023; Zhou et al., 2025b; c).

As an alternative to backpropagation through time, distillation-based methods compress the multi-step policy (e.g., a standard diffusion or flow model) into a one-step model, which can be more efficiently optimized through standard policy gradient techniques (Ding & Jin, 2023; Chen et al., 2023; Park et al., 2025b). However, distillation-based methods face a fundamental limitation: while expressive models can be used for the base policy (e.g., to model the offline data distribution), the policy actually optimized and executed is a less expressive, one-step model.
While a one-step model may suffice for simple simulation-based tasks, it is difficult to scale to larger base policies (e.g., VLAs) or more complex real-world settings, partly due to compounding errors between the teacher (the base policy) and the student (the distilled policy).

In this paper, we tackle the question:

Can we develop a scalable offline RL approach
without relying on policy distillation or backpropagation through time?

A natural alternative to policy gradients is rejection sampling: sample multiple action candidates from the base policy and choose the one with the highest value according to a learned value function (e.g., $Q$-function).
However, existing rejection sampling methods suffer from two key limitations: the learned value function is not regularized, and value functions are typically limited to multilayer perceptron (MLP) networks.
Standard approaches learn the value function from the offline dataset, resulting in $Q^{\pi_{\textnormal{ref}}}$, the $Q$-function under the data-generating policy ${\pi_{\textnormal{ref}}}$, which is not a provably optimal solution to the standard KL-regularized offline RL objective (Zhou et al., 2025a).
Additionally, the $Q$-functions used in continuous state-action spaces are typically standard MLP networks.
The primary method for improving the value function’s expressivity is simply increasing the size of the value network, which requires more backpropagation through the additional layers in the neural network.
Motivated by the recent success of generative models for computer vision and RL policies, we investigate how value learning can be improved by integrating flow matching.

Finally, we consider the third axis of scale—compute—and, in particular, how to leverage additional inference-time compute.
Existing approaches to inference-time scaling typically use dynamics or world models for additional planning during inference, such as model predictive path integral control (Williams et al., 2017), model-based offline planning (Hafner et al., 2019; Argenson & Dulac-Arnold, 2020), planning with world models (Hafner et al., 2023), and Monte Carlo tree search (Chen et al., 2024).
While promising, these methods either do not leverage expressive models, or they require learning and maintaining an auxiliary model of the environment, which can introduce additional sources of approximation error and scaling challenges.

The preceding limitations highlight a key gap in the scalability of existing offline RL approaches: although expressive generative models have been integrated into policies, the same level of expressivity has yet to be brought to value functions. In this paper, we bridge this gap through Expressive Value Learning for Offline Reinforcement Learning (EVOR): an approach for learning an optimal solution to the KL-regularized offline RL objective with both expressive policies and expressive value functions. EVOR achieves the following desiderata for scalable offline RL:

- 1.
EVOR avoids policy distillation and backpropagation through time during policy optimization. EVOR avoids learning a new policy and instead optimizes the base policy through inference-time policy extraction. Unlike standard rejection sampling approaches, EVOR uses an optimal, regularized $Q$-function.
- 2.
EVOR learns an expressive, optimal $Q$-function via flow matching. EVOR employs flow-based temporal difference (TD) learning to learn an optimal, regularized solution to the offline RL objective.
- 3.
EVOR enables inference-time scaling and regularization. EVOR provides a natural mechanism for inference-time scaling: performing additional search, guided by the optimal value function, without additional training.

## 2 Related Work

##### Offline Reinforcement Learning.

Offline RL tackles the problem of learning a policy from a fixed dataset without additional environment interactions (Levine et al., 2020). In addition to the standard reward maximization goal of online RL, the key problem of offline RL is avoiding distribution shift between train-time (i.e., the offline dataset) and test-time (i.e., the learned policy’s rollout). Numerous strategies have been proposed for the offline RL setting. A common approach is to employ behavior regularization, which forces the learned policy to stay close to the dataset via behavioral cloning or divergence penalties (Nair et al., 2020; Fujimoto & Gu, 2021; Tarasov et al., 2023).
Other approaches include in-distribution maximization (Kostrikov et al., 2021; Xu et al., 2023; Garg et al., 2023), dual formulations of RL (Lee et al., 2021; Sikchi et al., 2023), out-of-distribution detection (Yu et al., 2020; Kidambi et al., 2020; An et al., 2021; Nikulin et al., 2023), and conservative value estimation (Kumar et al., 2020). Farebrother et al. (2024); Nauman et al. (2025) propose training value functions via classification-based objectives, instead of the standard regression-based objectives. Rybkin et al. (2025) propose scaling laws for value-based reinforcement learning.
Policies trained via offline RL can subsequently be used for sample-efficient online RL in a procedure known as offline-to-online RL (Lee et al., 2021; Song et al., 2022; Nakamoto et al., 2023; Ball et al., 2023; Yu & Zhang, 2023; Ren et al., 2024b; Park et al., 2025b; Li et al., 2025).

##### Offline Reinforcement Learning with Generative Models.

Standard offline RL approaches rely on Gaussian-based models in continuous state-action spaces. However, recent work has focused on representing policies via powerful sequence or generative models Chen et al. (2021); Janner et al. (2021; 2022); Wang et al. (2022); Ren et al. (2024a); Wu et al. (2024); Black et al. (2024); Park et al. (2025b); Espinosa-Dice et al. (2025), taking advantage of more powerful generative models like diffusion (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021) and flow matching (Lipman et al., 2022; Liu et al., 2022; Lipman et al., 2024). These generative models are known to be more expressive than Gaussian-based models, enabling them to capture more complex, multi-modal distributions. Modeling complex distributions is particularly relevant to the offline RL setting, where the offline dataset may be composed of multiple data-generating policies of varying qualities. However, diffusion and flow models rely on an iterative sampling process that can be computationally expensive (Ding & Jin, 2023). To address this problem, some methods utilize a two-stage procedure to first train an expressive generative model on the offline dataset, and then distill it into a one-step model that is then used for policy optimization (Ding & Jin, 2023; Chen et al., 2023; Meng et al., 2023; Park et al., 2025b). Espinosa-Dice et al. (2025) propose an approach for avoiding both distillation and extensive backpropagation through time by leveraging shortcut models for flexible inference, but rely on a standard, Gaussian-based value function. Additionally, generative models have been used for plan generation in offline RL (Zheng et al., 2023) and energy-guided flow and diffusion models, incorporating reward feedback in the flow and diffusion training (Zhang et al., 2025).
Farebrother et al. (2025) propose integrating flow matching with Bellman-style updates for successor representation learning.

##### Inference-Time Scaling in Offline Reinforcement Learning.

Inference-time scaling in reinforcement learning often takes the form of leveraging dynamics or world models for additional planning during inference. Approaches include model predictive control (Richalet et al., 1978; Hansen et al., 2022), model predictive path integral control (Williams et al., 2015; 2017; Gandhi et al., 2021), model-based offline planning (Hafner et al., 2019; Argenson & Dulac-Arnold, 2020), sequence modeling (Janner et al., 2021; 2022; Kong et al., 2024), planning with world models (Hafner et al., 2023), and Monte Carlo tree search (Chen et al., 2024). Additional approaches include applying rejection sampling to the learned value function at inference-time (Chen et al., 2022; Fujimoto et al., 2019; Ghasemipour et al., 2021; Hansen-Estruch et al., 2023; Park et al., 2024b) or using the gradient of the learned value function to adjust actions at inference-time (Park et al., 2024b). Generative models like flow matching and diffusion models naturally support a form of sequential scaling by increasing the number of steps in the iterative sampling process (Ho et al., 2020; Song et al., 2020; Liu et al., 2022; Lipman et al., 2022). Espinosa-Dice et al. (2025) takes advantage of flexibility in the number of denoising steps used when sampling actions from the policy. However, existing approaches do not leverage generative models for value learning like EVOR. By leveraging more expressive models for value learning, EVOR can better take advantage of larger, more complex offline datasets.

## 3 Background

##### Markov Decision Process.

We consider a finite-horizon Markov decision process (MDP) $(\mathcal{X},\,\mathcal{A},\,P,\,r,\,H)$, where $\mathcal{X}$ is the state space, $\mathcal{A}$ is the action space, $P$ is the transition function, $r:\mathcal{X}\times\mathcal{A}\rightarrow[0,1]$ is the reward function, and $H$ is the MDP’s horizon (Puterman, 2014).
An offline dataset $\mathcal{D}=\{(x_{h},a_{h},r_{h},x_{h+1})\}$ is collected under some unknown reference policy ${\pi_{\textnormal{ref}}}$, which could be multi-modal and sub-optimal.
In the offline RL setting, we do not assume access to environment interactions.

##### Offline Reinforcement Learning.

The offline RL objective is generally expressed as a combination of a policy optimization term and a regularization term, such that

$$ $\operatorname*{argmax}_{\pi\in\Pi}\underbrace{J_{\mathcal{D}}(\pi)}_{\texttt{Policy Optimization\vphantom{Regularization}}}-\underbrace{\eta\textnormal{Reg}(\pi,{\pi_{\textnormal{ref}}})}_{\texttt{Regularization\vphantom{Policy Optimization}}}$ (1) $$

where $J_{\mathcal{D}}(\pi)$ is the expected return over offline dataset $\mathcal{D}$, ${\pi_{\textnormal{ref}}}$ is the unknown data-generating policy, and $\textnormal{Reg}(\pi,{\pi_{\textnormal{ref}}})$ is a regularization term (Espinosa-Dice et al., 2025). The regularization term generally takes the form of a divergence measure between $\pi$ and ${\pi_{\textnormal{ref}}}$, with KL divergence being most common. The offline RL objective can be expressed as the soft value of a policy subject to KL regularization:

$$ $V^{\pi,\eta}=\mathbb{E}_{\pi}\left[\sum_{h=1}^{H}r(x_{h},a_{h})-\eta\text{KL}\left(\pi(x_{h})\|{\pi_{\textnormal{ref}}}(x_{h})\right)\right],$ (2) $$

where the expectation is over a random trajectory $(x_{1},a_{1},\dots,x_{H},a_{H})$ sampled according to $\pi$ and the KL divergence is
$\operatorname{KL}(p\|q)=\mathbb{E}_{z\sim p}\left[\log\left(p(z)/q(z)\right)\right]$ (Zhou et al., 2025a).
The objective is to learn the optimal, regularized policy $\pi^{\star}=\operatorname*{argmax}_{\pi\in\Pi}V^{\pi,\eta}$. Ziebart et al. (2008) showed that

$$ $\displaystyle V_{H+1}^{\star,\eta}(x)$ $\displaystyle=0,$ (3) $\displaystyle Q_{h}^{\star,\eta}(x,a)$ $\displaystyle=r(x,a)+\mathbb{E}_{x^{\prime}\sim P_{h}(x,a)}\left[V_{h+1}^{\star,\eta}(x^{\prime})\right],$ (4) $\displaystyle\pi^{\star,\eta}(a|x)$ $\displaystyle\propto{\pi_{\textnormal{ref}}}(a|x)\exp\left(\eta^{-1}Q_{h}^{\star,\eta}(x,a)\right),$ (5) $\displaystyle V_{h}^{\star,\eta}(x)$ $\displaystyle=\eta\ln\mathbb{E}_{a\sim{\pi_{\textnormal{ref}}}(\cdot\mid x)}(x)\left[\exp\left(\eta^{-1}Q_{h}^{\star,\eta}(x,a)\right)\right].$ (6) $$

For convenience, we drop the $\eta$ superscript when clear from context.

##### Reward-To-Go.

We define the reward-to-go under the unknown data-generating policy ${\pi_{\textnormal{ref}}}$, starting at state $x$ and taking action $a$, as

$$ $Z(x,a):=\sum_{h=0}^{H}r(x_{h},a_{h}),\quad x_{0}=x,\;a_{0}=a,\;x_{h+1}\sim P_{h}(\cdot\mid x_{h},a_{h}),\;a_{h+1}\sim{\pi_{\textnormal{ref}}}(\cdot\mid x_{h+1}),$ (7) $$

We define $R(\cdot\mid x,a)$ as the law of the random variable $Z(x,a)$, so $R(\cdot\mid x,a)\overset{D}{=}Z(x,a)$. In other words, $R(\cdot\mid x,a)$ is the distribution of rewards-to-go under ${\pi_{\textnormal{ref}}}$, starting at state $x$ and taking action $a$.
We can thus define

$$ $\pi^{Z,\eta}(a|x)\propto{\pi_{\textnormal{ref}}}(a|x)\mathbb{E}_{z\sim Z(x,a)}\left[\exp\left(z/\eta\right)\right].$ (8) $$

We can also define $R^{\pi}(\cdot\mid x,a)$ as the distribution of rewards-to-go under a policy $\pi$.

##### Flow Matching.

We define flow matching (Lipman et al., 2022; Liu et al., 2022; Lipman et al., 2024) as follows.
Let $p(x)\in\Delta(\mathbb{R}^{d})$ be a data distribution. Given a vector field $v_{t}$, we construct its corresponding flow, $\phi:[0,1]\times\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$, by the ordinary differential equation (ODE)

$$ $\displaystyle\frac{d}{dt}\phi_{t}(x)$ $\displaystyle=v_{t}(\phi_{t}(x))$ (9) $\displaystyle\phi_{0}(x)$ $\displaystyle=x$ (10) $$

We employ Lipman et al. (2024)’s flow matching, which is based on linear paths and uniform time sampling, such that the training objective is

$$ $\min_{\theta}\mathbb{E}_{\begin{subarray}{c}x^{0}\sim\mathcal{N}(0,I_{d})\\ x^{1}\sim p(x)\\ t\sim U[0,1]\end{subarray}}\left[\|v_{\theta}(t,x^{t})-(x^{1}-x^{0})\|_{2}^{2}\right]$ (11) $$

where $x^{t}=(1-t)x^{0}+tx^{1}$ is the linear interpolation between $x^{0}$ and $x^{1}$.

Figure: Algorithm 1 EVOR Training via Flow-Based TD Learning

## 4 Expressive Value Learning for Offline Reinforcement Learning

In this section, we tackle the question:

How do we learn an optimal, expressive value function for the KL-regularized offline RL objective?

Our key insight is to integrate the power of flow matching in TD learning through a distributional approach, which we refer to as Expressive Value Learning for Offline Reinforcement Learning (EVOR).

### 4.1 Learning A Base Policy Through Flow Matching

In offline reinforcement learning, we aim to learn or fine-tune a policy from a dataset collected under an unknown data-generating policy ${\pi_{\textnormal{ref}}}$. Depending on the setting, we either have access to a base policy ${\pi_{\textnormal{base}}}$ (e.g., a pre-trained generalist model) or we must learn the policy from scratch. Both settings are compatible with our approach, and we first show how a base policy can be learned in the setting when no initial base policy is provided.

##### Base Policy Objective.

In the setting where a starting base policy is not known, we train a policy ${\pi_{\textnormal{base}}}$ that predicts actions via behavioral cloning (Pomerleau, 1988) on the offline dataset’s state-action pairs. By formulating the objective as a supervised learning problem rather than a more complex RL procedure, we can employ any generative model to learn the base policy. We use flow matching here, such that the loss is given by:

$$ $\mathcal{L}_{\text{BC}}(\phi)=\mathbb{E}_{\begin{subarray}{c}(x,a^{1})\sim\mathcal{D},\,a^{0}\sim\mathcal{N}\\ t\sim\textnormal{Unif}(0,1)\end{subarray}}\Big[\Big\|\underbrace{v_{\phi}(a^{t},t\mid x)}_{\mathclap{\texttt{Velocity Prediction}}}\hskip 10.00002pt-\hskip 10.00002pt\underbrace{(a^{1}-a^{0})}_{\mathclap{\texttt{Velocity Target}}}\Big\|^{2}\Big]$ (12) $$

where $a^{0}$ represents a fully noised action (i.e., noise sampled from a Gaussian) and $a^{1}$ represents a real action (i.e., action sampled from the offline data $\mathcal{D}$). Leveraging an expressive model like flow matching enables ${\pi_{\textnormal{base}}}$ to model complex action distributions and multi-modal offline data.
Through [Equation 12](https://arxiv.org/html/2510.08218v1#S4.E12), we learn a base policy ${\pi_{\textnormal{base}}}\approx{\pi_{\textnormal{ref}}}$, subject to finite sample and optimization errors.

### 4.2 Expressive Value Learning via Flow-Based TD Learning

Next, we turn to the problem of learning an expressive value function.
Rather than use standard value learning methods, our key insight is to integrate flow matching into distributional TD learning, thus enabling us to leverage the expressivity of flow matching with the variance reduction and improved credit assignment of TD learning.

Intuitively, we can think of flow matching as a method of transporting samples from a prior distribution (e.g., samples from a Gaussian) to a target distribution (i.e., the data distribution).
For EVOR, we set the target distribution to the distribution of rewards-to-go under ${\pi_{\textnormal{ref}}}$, denoted by $R(\cdot\mid x,a)$, which is a distribution we have samples from in the offline dataset. In the remainder of the subsection, we explain the flow-based temporal difference (TD) learning objective and high-level intuition behind it. Its formal derivation can be found in [Appendix A](https://arxiv.org/html/2510.08218v1#A1).

##### Distributional Bellman.

Recall that TD learning uses the Bellman equation to learn a value function by constructing a bootstrap target (i.e., the right-hand side (RHS) of the Bellman equation) (Bellman, 1966; Sutton & Barto, 1998), such that

$$ $Q(x,a)=r(x,a)+\mathbb{E}_{P,\pi}Q(X^{\prime},A^{\prime}).$ (13) $$

Notably, the Bellman equation also holds under distributions (Jaquette, 1973; Sobel, 1982; White, 1988; Bellemare et al., 2017), such that

$$ $\underbrace{Z(x,a)}_{\mathclap{\texttt{LHS of Distributional Bellman}}}\overset{D}{=}\overbrace{r(x,a)+Z(X^{\prime},A^{\prime})}^{\mathclap{\texttt{RHS of Distributional Bellman}}}$ (14) $$

where $Z(X^{\prime},A^{\prime})$ denotes the random return.

Figure: Algorithm 2 EVOR Inference via $Q^{\star}_{\theta}$ Reweighting

##### Flow-Based TD Objective.

Flow matching learns to transport a prior distribution into a target data distribution. The distributional Bellman equation provides two distributions on either side of the equation. To construct a flow-based TD objective, we set the RHS of the distributional Bellman equation as the target distribution, and match the velocities between the LHS and RHS distributions.

More specifically, we learn a conditional flow model $s_{\theta}(\cdot\mid x,a,t)$ that transports base noise sampled from a Gaussian, $z^{0}_{x,a}\sim\mathcal{N}(0,I_{d})$, to a terminal variable $z^{1}_{x,a}\sim R_{\theta}(\cdot\mid x,a)$, such that the learned distribution $R_{\theta}(\cdot\mid x,a)\approx R^{{\pi_{\textnormal{base}}}}(\cdot\mid x,a)$, where $R^{{\pi_{\textnormal{base}}}}(\cdot\mid x,a)$ is the distribution of rewards-to-go of ${\pi_{\textnormal{base}}}$. The bootstrap target is given by

$$ $s_{\text{target}}:=r(x,a)+\gamma\mathbb{E}_{a^{\prime}\sim{\pi_{\textnormal{base}}}(\cdot\mid x^{\prime})}\bar{s}_{\theta}(z^{t}\mid x^{\prime},a^{\prime},t),$ (15) $$

and the loss is given by

$$ $\mathcal{L}_{\text{FlowTD}}(\theta)=\underbrace{\mathbb{E}_{(x,a,r,x^{\prime})\sim\mathcal{D}}}_{\mathclap{\texttt{Dataset's State-Action-Reward}}}\overbrace{\mathbb{E}_{z^{1}\sim\bar{R}_{\theta}(\cdot\mid x,a)}}^{\mathclap{\texttt{Sample Reward-To-Go}}}\underbrace{\mathbb{E}_{t\sim\textnormal{Unif}(0,1)}}_{\mathclap{\texttt{Sample Time}}}\Big\|\overbrace{s_{\theta}(z^{t}\mid x,a,t)}^{\mathclap{\texttt{Velocity Prediction of LHS}}}-\underbrace{\text{stopgrad}(s_{\text{target}})}_{\mathclap{\texttt{Velocity Target of RHS}}}\Big\|_{2}^{2}.$ (16) $$

We sample a state-action-reward-next-state tuple from the offline data $(x,a,r,x^{\prime})\sim\mathcal{D}$, a time $t\sim\textnormal{Unif}(0,1)$, and the next action from the base policy $a^{\prime}\sim{\pi_{\textnormal{base}}}(\cdot\mid x^{\prime})$. We construct a linear interpolant $z^{t}=(1-t)z^{0}+tz^{1}$ by sampling a reward-to-go $z^{1}\sim R(\cdot\mid x,a)$ and a noise sample $z^{0}\sim\mathcal{N}(0,I_{d})$. The reward-to-go sample $z^{1}$ can be sampled from the dataset $\mathcal{D}$ or a target version of the learned reward model $\bar{R}_{\theta}(\cdot\mid x,a)$.
We sample a reward-to-go from $R_{\theta}(\cdot\mid x,a)$ using the standard forward Euler method applied on the learned flow model $s_{\theta}(\cdot\mid x,a,t)$.

##### Flow-Based TD Target.

The velocity target in [Equation 15](https://arxiv.org/html/2510.08218v1#S4.E15) represents a flow field that generates samples from the RHS of the distributional Bellman ([Equation 14](https://arxiv.org/html/2510.08218v1#S4.E14)), $r(x,a)+Z(X^{\prime},A^{\prime})$. The RHS of the distributional Bellman (i.e., the target distribution in flow matching) corresponds to the reward-to-go distribution translated by the one-step reward $r(x,a)$. Under flow matching, such a translation shifts every particle in the flow trajectory by a constant amount. The vector field specifies the instantaneous rate of change of particle positions, so adding a constant shift to all trajectories increases the velocity everywhere by the same constant.
Consequently, the target
$s_{\mathrm{target}}=r(x,a)+\gamma\mathbb{E}_{a^{\prime}\sim{\pi_{\textnormal{base}}}(\cdot\mid x^{\prime})}\bar{s}_{\theta}(z^{t}\mid x^{\prime},a^{\prime},t)$
is the one-step reward shift and the expected next-state flow under ${\pi_{\textnormal{base}}}$.

Next, we consider the question:

How do we obtain an optimal value function from the reward model?

Building on results from Ziebart et al. (2008) and Zhou et al. (2025a), we show that the optimal, regularized $Q$-function can be obtained using the learned reward model.

###### Theorem 1 (Optimal Regularized Value Functions (Zhou et al., 2025a ) ) .

Under deterministic transitions, the optimal value and $Q$-functions are given by

$$ $\displaystyle V_{h}^{\star,\pi}(x_{h})$ $\displaystyle=\eta\ln\mathbb{E}_{{\pi_{\textnormal{ref}}}}\left[\exp\left(\eta^{-1}\sum_{t\geq h}^{H}r(x_{t},a_{t})\right)\,\middle|\,x_{h}\right],$ (17) $\displaystyle Q_{h}^{\star,\pi}(x_{h},a_{h})$ $\displaystyle=\eta\ln\mathbb{E}_{{\pi_{\textnormal{ref}}}}\left[\exp\left(\eta^{-1}\sum_{t\geq h}^{H}r(x_{t},a_{t})\right)\,\middle|\,x_{h},a_{h}\right].$ (18) $$

Using [Theorem 1](https://arxiv.org/html/2510.08218v1#Thmtheorem1), we can express the optimal, regularized $Q$-function as a function of ${\pi_{\textnormal{ref}}}$’s reward-to-go distribution $R(\cdot\mid x,a)$, such that

$$ $Q_{h}^{\star}(x_{h},a_{h})=\eta\ln\mathbb{E}_{z\sim R_{h}(\cdot\mid x_{h},a_{h})}\exp\left(\eta^{-1}z\right)$ (19) $$

Through the flow-based TD learning objective ([Equation 16](https://arxiv.org/html/2510.08218v1#S4.E16)), we learn a reward model $R_{\theta}(\cdot\mid x,a)\approx R^{\pi_{\textnormal{base}}}(\cdot\mid x,a)$, where ${\pi_{\textnormal{base}}}\approx{\pi_{\textnormal{ref}}}$ if the base policy is learned well.
In practice, we can approximate the expectation via sample averaging, and the full training procedure is shown in [Algorithm 1](https://arxiv.org/html/2510.08218v1#algorithm1). While the assumption of deterministic dynamics can be strong in certain settings, it is frequently imposed in the analysis of offline RL algorithms (Edwards et al., 2020; Ma et al., 2022; Schweighofer et al., 2022; Park et al., 2023; Ghosh et al., 2023; Wang et al., 2023; Karabag & Topcu, 2023; Park et al., 2024a; c), and we only use it here to derive the optimal expression for the $Q$-function. We then empirically validate our results on recent offline RL benchmarks in [Section 5](https://arxiv.org/html/2510.08218v1#S5).

**Table 1: EVOR’s Overall Performance. EVOR outperforms the baselines on all five environments, for a total of 25 unique tasks in the OGBench task suite (Park et al., 2024a). Results are averaged over five seeds per task, with standard deviations reported. The full results are reported in [Appendix B](https://arxiv.org/html/2510.08218v1#A2).**
| Task Category | QC-1 | QC-5 | EVOR |
| --- | --- | --- | --- |
| OGBench antmaze-large-navigate-singletask ($5$ tasks) | $9$ $\pm 6$ | $7$ $\pm 2$ | 50 $\pm 4$ |
| OGBench antmaze-large-stitch-singletask ($5$ tasks) | $9$ $\pm 5$ | $8$ $\pm 4$ | 15 $\pm 1$ |
| OGBench cube-double-play-100M-singletask ($5$ tasks) | $55$ $\pm 5$ | $57$ $\pm 19$ | 81 $\pm 2$ |
| OGBench pointmaze-medium-naviate-singletask ($5$ tasks) | $91$ $\pm 10$ | 99 $\pm 0$ | 99 $\pm 0$ |
| OGBench scene-play-sparse-singletask ($5$ tasks) | $47$ $\pm 4$ | 83 $\pm 4$ | 87 $\pm 6$ |

###### Theorem 1 (Optimal Regularized Value Functions (Zhou et al., 2025a ) ) .

### 4.3 Inference-Time Policy Extraction, Regularization, and Scaling

EVOR’s training procedure focuses on learning an expressive value function, and it trains the base policy via flow matching on the offline dataset, leading to the natural question:

How does EVOR optimize the base policy beyond the offline dataset
without distillation or backpropagation through time?

##### Inference-Time Policy Extraction.

Instead of learning a new policy during training, EVOR performs inference-time policy extraction using the learned distributional reward model. A common approach to inference-time policy extraction is to perform rejection sampling with the learned value or $Q$-function (Fujimoto et al., 2019; Ghasemipour et al., 2021; Gui et al., 2024). Given a state $x$, sample actions independently from the base policy $a_{1},a_{2},\ldots,a_{N}\sim{\pi_{\textnormal{base}}}(\cdot\mid x)$, and select the action with the largest $Q$ value, such that

$$ $\operatorname*{argmax}_{a\in\{a_{1},a_{2},\ldots,a_{N}\}}Q(x,a)$ (20) $$

However, using the $Q$-function trained on the offline dataset $\mathcal{D}$, which is generated by ${\pi_{\textnormal{ref}}}$, is not an optimal solution to the KL-regularized offline RL objective, and optimizing it may lead to distribution shift and poor performance at test-time (Zhou et al., 2025a). Instead, we utilize our expression for the optimal $Q$-function,

$$ $Q^{\star}(x,a)=\eta\ln\mathbb{E}_{r\sim R(\cdot\mid x,a)}\exp(r/\eta),$ (21) $$

where $R$ is the conditional distribution of rewards-to-go under ${\pi_{\textnormal{ref}}}$. In practice, we approximate the expectation via sample averaging, and we can construct a softmax over the $Q^{\star}$ values, as shown in [Algorithm 2](https://arxiv.org/html/2510.08218v1#algorithm2).

##### Inference-Time Regularization and Scaling.

EVOR’s formulation provides a natural mechanism for inference-time regularization and scaling. Since actions are sampled from the base policy, running EVOR with varying temperatures $\tau_{R}$ and $\tau_{Q}$ controls the strength of regularization and policy optimization. Increasing ${N_{\pi}}$ corresponds to performing additional test-time search, while decreasing it enables faster inference under smaller compute budgets. Crucially, these parameters can be varied at test-time without retraining, allowing for both inference-time scaling and regularization.

## 5 Experimental Results

Figure: Figure 1: EVOR’s Inference-Time Scaling. EVOR can perform inference-time scaling by increasing the number of action candidates ${N_{\pi}}$, performing greater search at inference time with the expressive value function. Leveraging greater inference-time compute results in better performance, up to a saturation point. Results are averaged over three seeds per task, with standard deviations reported.
Refer to caption: x1.png

In this section, we investigate the performance of EVOR, focusing on the following question:

What is the benefit of expressive value learning?

### 5.1 Experimental Setup

##### Environments and Tasks.

We follow the experimental setup of prior works leveraging the OGBench task suite (Park et al., 2024a; 2025b; Espinosa-Dice et al., 2025; Li et al., 2025), specifically evaluating EVOR on locomotion and manipulation robotics tasks. We describe the full implementation details in [Appendix D](https://arxiv.org/html/2510.08218v1#A4).

##### Baselines.

Rather than comparing to all of the existing offline RL algorithms benchmarked on OGBench, we aim to isolate the effect of expressive value learning in order to demonstrate its benefit specifically. Thus, we compare to $Q$-chunking (QC, Li et al. (2025)), a recent offline RL algorithm that is closest to EVOR. Like EVOR, QC learns a base policy via flow matching and extracts an optimized policy via rejection sampling. The key difference between QC and EVOR is how the value function is learned—the exact difference we aim to isolate. QC can employ action chunking in both its policy and value function, and we compare EVOR to both QC with (QC-5) action chunking and without it (QC-1). We select the action chunk length (5) based on Li et al. (2025)’s recommendation.

##### Evaluation.

For a fair comparison, we use the same network size, number of gradient steps, and discount factor across all algorithms, following Park et al. (2025b); Espinosa-Dice et al. (2025). Moreover, we use the official QC implementation and its parameters. We bold values at 95% of the best performance in tables.

### 5.2 Experimental Results

Q: What is EVOR’s overall performance?

Across five environments and 25 distinct tasks, EVOR achieves the best overall performance compared to the baselines. Environment-level aggregation results are shown in [Table 1](https://arxiv.org/html/2510.08218v1#S4.T1), with full task-level results in [Appendix B](https://arxiv.org/html/2510.08218v1#A2).

Q: Does using expressive models for value learning improve performance?

Yes. As shown in [Table 1](https://arxiv.org/html/2510.08218v1#S4.T1), EVOR ’s expressive value learning approach consistently outperforms standard value learning, including those employing action chunking (QC-5). The results suggest that expressive value learning provides performance benefits in the settings considered.

Q: How can EVOR take advantage of greater inference-time compute?

As shown in [Figure 1](https://arxiv.org/html/2510.08218v1#S5.F1), with greater inference-time compute, EVOR can evaluate more action candidates (${N_{\pi}}$), leading to improved performance (up to a saturation point). We present the full results for inference-time scaling in [Appendix C](https://arxiv.org/html/2510.08218v1#A3).

Q: How can EVOR perform inference-time regularization?

As shown in [Figure 2](https://arxiv.org/html/2510.08218v1#S5.F2), by varying the temperature parameters $\tau_{R}$ and $\tau_{Q}$, EVOR can vary the level of regularization to the base policy compared to the level of policy optimization. As $\tau_{Q}$ decreases, the action selection becomes more greedy; as $\tau_{Q}$ increases, the action selection becomes more regularized to the base policy ${\pi_{\textnormal{base}}}$ (i.e., the performance of EVOR with ${N_{\pi}}=1$).

Q: What training parameters must EVOR tune per environment?

A key advantage of EVOR is that it uses the same training and evaluation parameters for all environments in [Table 1](https://arxiv.org/html/2510.08218v1#S4.T1), despite the environments spanning distinct locomotion and manipulation tasks. In contrast, policy gradient-based offline RL algorithms generally tune parameters per environment (Park et al., 2024a; 2025b; Espinosa-Dice et al., 2025). We present an ablation study of evaluation parameters in [Appendix C](https://arxiv.org/html/2510.08218v1#A3).

Q: Does rejection sampling-based policy extraction outperform reparameterized policy gradients?

We do not consider that question in this work. The purpose of this paper is to investigate scalable methods for expressive value learning in offline RL. In our empirical results, we isolate the effect of expressive value learning by using a consistent policy extraction method (rejection sampling).
We acknowledge that rejection sampling may not be the most effective policy extraction scheme, as argued by Park et al. (2024b). However, unlike policy gradient methods, rejection sampling does not require backpropagation through time or distillation—both key bottlenecks in scaling offline RL that this paper seeks to avoid.

Figure: Figure 2: Ablation Over EVOR’s Evaluation Parameters. EVOR uses the same training parameters for all environments in this paper. However, we investigate the effect of varying the temperature parameters $\tau_{R}$ and $\tau_{Q}$ at inference-time on the performance of EVOR. As $\tau_{Q}$ decreases, the action selection becomes more greedy, while as $\tau_{Q}$ increases, the action selection becomes more regularized. Set to a high value, EVOR becomes equivalent to the base policy (i.e., the performance with ${N_{\pi}}=1$). Results are averaged over three seeds per task, with standard deviations reported.
Refer to caption: x2.png

## 6 Discussion

In summary, EVOR is a scalable approach to offline reinforcement learning that integrates both expressive policies and expressive value learning. EVOR learns an optimal solution to the KL-regularized offline RL objective, which enables inference-time policy extraction without model distillation or backpropagation through time. Furthermore, EVOR can perform inference-time scaling by performing greater search, guided by the expressive value function, and it can adjust the level of regularization to the base policy without retraining.
In this paper, we focus on scalable offline RL by avoiding distillation and backpropagation through time, leading us to rejection sampling against an expressive value function. However, as noted by Park et al. (2024b), reparameterized policy gradients are an effective policy extraction technique. Future work may explore how EVOR’s expressive value learning could be integrated with policy gradient techniques and action chunking in a scalable manner.

#### Acknowledgments

NED is supported by the NSF Graduate Research Fellowship under Grant No. DGE-2139899.
KB acknowledge: This work has been made possible in part by a gift from the Chan Zuckerberg
Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial
Intelligence.
WS acknowledges fundings from NSF IIS-2154711, NSF CAREER 2339395, DARPA LANCER: LeArning Network
CybERagents, Infosys Cornell Collaboration, and Sloan Research Fellowship.

## References

- An et al. (2021)
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song.
Uncertainty-based offline reinforcement learning with diversified q-ensemble.
*Advances in neural information processing systems*, 34:7436–7447, 2021.
- Argenson & Dulac-Arnold (2020)
Arthur Argenson and Gabriel Dulac-Arnold.
Model-based offline planning.
*arXiv preprint arXiv:2008.05556*, 2020.
- Ba et al. (2016)
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.
Layer normalization.
*arXiv preprint arXiv:1607.06450*, 2016.
- Ball et al. (2023)
Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine.
Efficient online reinforcement learning with offline data.
In *International Conference on Machine Learning*, pp. 1577–1594. PMLR, 2023.
- Bellemare et al. (2017)
Marc G Bellemare, Will Dabney, and Rémi Munos.
A distributional perspective on reinforcement learning.
In *International conference on machine learning*, pp. 449–458. PMLR, 2017.
- Bellman (1966)
Richard Bellman.
Dynamic programming.
*science*, 153(3731):34–37, 1966.
- Black et al. (2024)
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al.
$\pi_{0}$: A vision-language-action flow model for general robot control.
*arXiv preprint arXiv:2410.24164*, 2024.
- Chen et al. (2022)
Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu.
Offline reinforcement learning via high-fidelity generative behavior modeling.
*arXiv preprint arXiv:2209.14548*, 2022.
- Chen et al. (2023)
Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu.
Score regularized policy optimization through diffusion behavior.
*arXiv preprint arXiv:2310.07297*, 2023.
- Chen et al. (2024)
Jiayu Chen, Wentse Chen, and Jeff Schneider.
Bayes adaptive monte carlo tree search for offline model-based reinforcement learning.
*arXiv preprint arXiv:2410.11234*, 2024.
- Chen et al. (2021)
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.
Decision transformer: Reinforcement learning via sequence modeling.
*Advances in neural information processing systems*, 34:15084–15097, 2021.
- Ding & Jin (2023)
Zihan Ding and Chi Jin.
Consistency models as a rich and efficient policy class for reinforcement learning.
*arXiv preprint arXiv:2309.16984*, 2023.
- Edwards et al. (2020)
Ashley Edwards, Himanshu Sahni, Rosanne Liu, Jane Hung, Ankit Jain, Rui Wang, Adrien Ecoffet, Thomas Miconi, Charles Isbell, and Jason Yosinski.
Estimating q (s, s’) with deep deterministic dynamics gradients.
In *International Conference on Machine Learning*, pp. 2825–2835. PMLR, 2020.
- Espinosa-Dice et al. (2025)
Nicolas Espinosa-Dice, Yiyi Zhang, Yiding Chen, Bradley Guo, Owen Oertell, Gokul Swamy, Kiante Brantley, and Wen Sun.
Scaling offline rl via efficient and expressive shortcut models.
*arXiv preprint arXiv:2505.22866*, 2025.
- Esser et al. (2024)
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al.
Scaling rectified flow transformers for high-resolution image synthesis.
In *Forty-first international conference on machine learning*, 2024.
- Farebrother et al. (2024)
Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taïga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, et al.
Stop regressing: Training value functions via classification for scalable deep rl.
*arXiv preprint arXiv:2403.03950*, 2024.
- Farebrother et al. (2025)
Jesse Farebrother, Matteo Pirotta, Andrea Tirinzoni, Rémi Munos, Alessandro Lazaric, and Ahmed Touati.
Temporal difference flows.
*arXiv preprint arXiv:2503.09817*, 2025.
- Fujimoto & Gu (2021)
Scott Fujimoto and Shixiang Shane Gu.
A minimalist approach to offline reinforcement learning.
*Advances in neural information processing systems*, 34:20132–20145, 2021.
- Fujimoto et al. (2019)
Scott Fujimoto, David Meger, and Doina Precup.
Off-policy deep reinforcement learning without exploration.
In *International conference on machine learning*, pp. 2052–2062. PMLR, 2019.
- Gandhi et al. (2021)
Manan S Gandhi, Bogdan Vlahov, Jason Gibson, Grady Williams, and Evangelos A Theodorou.
Robust model predictive path integral control: Analysis and performance guarantees.
*IEEE Robotics and Automation Letters*, 6(2):1423–1430, 2021.
- Garg et al. (2023)
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon.
Extreme q-learning: Maxent rl without entropy.
*arXiv preprint arXiv:2301.02328*, 2023.
- Ghasemipour et al. (2021)
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu.
Emaq: Expected-max q-learning operator for simple yet effective offline and online rl.
In *International Conference on Machine Learning*, pp. 3682–3691. PMLR, 2021.
- Ghosh et al. (2023)
Dibya Ghosh, Chethan Anand Bhateja, and Sergey Levine.
Reinforcement learning from passive data via latent intentions.
In *International Conference on Machine Learning*, pp. 11321–11339. PMLR, 2023.
- Gui et al. (2024)
Lin Gui, Cristina Gârbacea, and Victor Veitch.
Bonbon alignment for large language models and the sweetness of best-of-n sampling.
*arXiv preprint arXiv:2406.00832*, 2024.
- Hafner et al. (2019)
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson.
Learning latent dynamics for planning from pixels.
In *International conference on machine learning*, pp. 2555–2565. PMLR, 2019.
- Hafner et al. (2023)
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap.
Mastering diverse domains through world models.
*arXiv preprint arXiv:2301.04104*, 2023.
- Hansen et al. (2022)
Nicklas Hansen, Xiaolong Wang, and Hao Su.
Temporal difference learning for model predictive control.
*arXiv preprint arXiv:2203.04955*, 2022.
- Hansen-Estruch et al. (2023)
Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine.
Idql: Implicit q-learning as an actor-critic method with diffusion policies.
*arXiv preprint arXiv:2304.10573*, 2023.
- Ho et al. (2020)
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
*Advances in neural information processing systems*, 33:6840–6851, 2020.
- Janner et al. (2021)
Michael Janner, Qiyang Li, and Sergey Levine.
Offline reinforcement learning as one big sequence modeling problem.
*Advances in neural information processing systems*, 34:1273–1286, 2021.
- Janner et al. (2022)
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine.
Planning with diffusion for flexible behavior synthesis.
*arXiv preprint arXiv:2205.09991*, 2022.
- Jaquette (1973)
Stratton C Jaquette.
Markov decision processes with a new optimality criterion: Discrete time.
*The Annals of Statistics*, 1(3):496–505, 1973.
- Karabag & Topcu (2023)
Mustafa O Karabag and Ufuk Topcu.
On the sample complexity of vanilla model-based offline reinforcement learning with dependent samples.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 37, pp. 8195–8202, 2023.
- Kidambi et al. (2020)
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims.
Morel: Model-based offline reinforcement learning.
*Advances in neural information processing systems*, 33:21810–21823, 2020.
- Kingma (2014)
Diederik P Kingma.
Adam: A method for stochastic optimization.
*arXiv preprint arXiv:1412.6980*, 2014.
- Kong et al. (2024)
Deqian Kong, Dehong Xu, Minglu Zhao, Bo Pang, Jianwen Xie, Andrew Lizarraga, Yuhao Huang, Sirui Xie, and Ying Nian Wu.
Latent plan transformer for trajectory abstraction: Planning as latent space inference.
*Advances in Neural Information Processing Systems*, 37:123379–123401, 2024.
- Kostrikov et al. (2021)
Ilya Kostrikov, Ashvin Nair, and Sergey Levine.
Offline reinforcement learning with implicit q-learning.
*arXiv preprint arXiv:2110.06169*, 2021.
- Kumar et al. (2020)
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine.
Conservative q-learning for offline reinforcement learning.
*Advances in neural information processing systems*, 33:1179–1191, 2020.
- Lee et al. (2021)
Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim.
Optidice: Offline policy optimization via stationary distribution correction estimation.
In *International Conference on Machine Learning*, pp. 6120–6130. PMLR, 2021.
- Levine et al. (2020)
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.
Offline reinforcement learning: Tutorial, review, and perspectives on open problems.
*arXiv preprint arXiv:2005.01643*, 2020.
- Li et al. (2025)
Qiyang Li, Zhiyuan Zhou, and Sergey Levine.
Reinforcement learning with action chunking.
*arXiv preprint arXiv:2507.07969*, 2025.
- Lipman et al. (2022)
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le.
Flow matching for generative modeling.
*arXiv preprint arXiv:2210.02747*, 2022.
- Lipman et al. (2024)
Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky TQ Chen, David Lopez-Paz, Heli Ben-Hamu, and Itai Gat.
Flow matching guide and code.
*arXiv preprint arXiv:2412.06264*, 2024.
- Liu et al. (2022)
Xingchao Liu, Chengyue Gong, and Qiang Liu.
Flow straight and fast: Learning to generate and transfer data with rectified flow.
*arXiv preprint arXiv:2209.03003*, 2022.
- Ma et al. (2022)
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang.
Vip: Towards universal visual reward and representation via value-implicit pre-training.
*arXiv preprint arXiv:2210.00030*, 2022.
- Meng et al. (2023)
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans.
On distillation of guided diffusion models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 14297–14306, 2023.
- Nair et al. (2020)
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine.
Awac: Accelerating online reinforcement learning with offline datasets.
*arXiv preprint arXiv:2006.09359*, 2020.
- Nakamoto et al. (2023)
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine.
Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning.
*Advances in Neural Information Processing Systems*, 36:62244–62269, 2023.
- Nauman et al. (2025)
Michal Nauman, Marek Cygan, Carmelo Sferrazza, Aviral Kumar, and Pieter Abbeel.
Bigger, regularized, categorical: High-capacity value functions are efficient multi-task learners.
*arXiv preprint arXiv:2505.23150*, 2025.
- Nikulin et al. (2023)
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov.
Anti-exploration by random network distillation.
In *International Conference on Machine Learning*, pp. 26228–26244. PMLR, 2023.
- Park et al. (2023)
Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine.
Hiql: Offline goal-conditioned rl with latent states as actions.
*Advances in Neural Information Processing Systems*, 36:34866–34891, 2023.
- Park et al. (2024a)
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine.
Ogbench: Benchmarking offline goal-conditioned rl.
*arXiv preprint arXiv:2410.20092*, 2024a.
- Park et al. (2024b)
Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar.
Is value learning really the main bottleneck in offline rl?
*arXiv preprint arXiv:2406.09329*, 2024b.
- Park et al. (2024c)
Seohong Park, Tobias Kreiman, and Sergey Levine.
Foundation policies with hilbert representations.
*arXiv preprint arXiv:2402.15567*, 2024c.
- Park et al. (2025a)
Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine.
Horizon reduction makes rl scalable.
*arXiv preprint arXiv:2506.04168*, 2025a.
- Park et al. (2025b)
Seohong Park, Qiyang Li, and Sergey Levine.
Flow q-learning.
*arXiv preprint arXiv:2502.02538*, 2025b.
- Pomerleau (1988)
Dean A Pomerleau.
Alvinn: An autonomous land vehicle in a neural network.
*Advances in neural information processing systems*, 1, 1988.
- Puterman (2014)
Martin L Puterman.
*Markov decision processes: discrete stochastic dynamic programming*.
John Wiley & Sons, 2014.
- Ren et al. (2024a)
Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz.
Diffusion policy policy optimization.
*arXiv preprint arXiv:2409.00588*, 2024a.
- Ren et al. (2024b)
Juntao Ren, Gokul Swamy, Zhiwei Steven Wu, J Andrew Bagnell, and Sanjiban Choudhury.
Hybrid inverse reinforcement learning.
*arXiv preprint arXiv:2402.08848*, 2024b.
- Richalet et al. (1978)
Jacques Richalet, André Rault, JL Testud, and J Papon.
Model predictive heuristic control.
*Automatica (journal of IFAC)*, 14(5):413–428, 1978.
- Rybkin et al. (2025)
Oleh Rybkin, Michal Nauman, Preston Fu, Charlie Snell, Pieter Abbeel, Sergey Levine, and Aviral Kumar.
Value-based deep rl scales predictably.
*arXiv preprint arXiv:2502.04327*, 2025.
- Schweighofer et al. (2022)
Kajetan Schweighofer, Marius-constantin Dinu, Andreas Radler, Markus Hofmarcher, Vihang Prakash Patil, Angela Bitto-Nemling, Hamid Eghbal-Zadeh, and Sepp Hochreiter.
A dataset perspective on offline reinforcement learning.
In *Conference on Lifelong Learning Agents*, pp. 470–517. PMLR, 2022.
- Sikchi et al. (2023)
Harshit Sikchi, Qinqing Zheng, Amy Zhang, and Scott Niekum.
Dual rl: Unification and new methods for reinforcement and imitation learning.
*arXiv preprint arXiv:2302.08560*, 2023.
- Sobel (1982)
Matthew J Sobel.
The variance of discounted markov decision processes.
*Journal of Applied Probability*, 19(4):794–802, 1982.
- Sohl-Dickstein et al. (2015)
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.
Deep unsupervised learning using nonequilibrium thermodynamics.
In *International conference on machine learning*, pp. 2256–2265. pmlr, 2015.
- Song et al. (2020)
Jiaming Song, Chenlin Meng, and Stefano Ermon.
Denoising diffusion implicit models.
*arXiv preprint arXiv:2010.02502*, 2020.
- Song et al. (2021)
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.
Score-based generative modeling through stochastic differential equations.
In *International Conference on Learning Representations*, 2021.
URL [https://openreview.net/forum?id=PxTIG12RRHS](https://openreview.net/forum?id=PxTIG12RRHS).
- Song et al. (2022)
Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun.
Hybrid rl: Using both offline and online data can make rl efficient.
*arXiv preprint arXiv:2210.06718*, 2022.
- Sutton & Barto (1998)
Richard S. Sutton and Andrew G. Barto.
*Reinforcement Learning: An Introduction*.
The MIT Press, Cambridge, MA, 1998.
- Tarasov et al. (2023)
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov.
Revisiting the minimalist approach to offline reinforcement learning.
*Advances in Neural Information Processing Systems*, 36:11592–11620, 2023.
- Wang et al. (2023)
Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang.
Optimal goal-reaching reinforcement learning via quasimetric learning.
In *International Conference on Machine Learning*, pp. 36411–36430. PMLR, 2023.
- Wang et al. (2022)
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou.
Diffusion policies as an expressive policy class for offline reinforcement learning.
*arXiv preprint arXiv:2208.06193*, 2022.
- White (1988)
Douglas J White.
Mean, variance, and probabilistic criteria in finite markov decision processes: A review.
*Journal of Optimization Theory and Applications*, 56(1):1–29, 1988.
- Williams et al. (2015)
Grady Williams, Andrew Aldrich, and Evangelos Theodorou.
Model predictive path integral control using covariance variable importance sampling.
*arXiv preprint arXiv:1509.01149*, 2015.
- Williams et al. (2017)
Grady Williams, Andrew Aldrich, and Evangelos A Theodorou.
Model predictive path integral control: From theory to parallel computation.
*Journal of Guidance, Control, and Dynamics*, 40(2):344–357, 2017.
- Wu et al. (2024)
Runzhe Wu, Yiding Chen, Gokul Swamy, Kianté Brantley, and Wen Sun.
Diffusing states and matching scores: A new framework for imitation learning.
*arXiv preprint arXiv:2410.13855*, 2024.
- Xu et al. (2023)
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan.
Offline rl with no ood actions: In-sample learning via implicit value regularization.
*arXiv preprint arXiv:2303.15810*, 2023.
- Yu et al. (2020)
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma.
Mopo: Model-based offline policy optimization.
*Advances in Neural Information Processing Systems*, 33:14129–14142, 2020.
- Yu & Zhang (2023)
Zishun Yu and Xinhua Zhang.
Actor-critic alignment for offline-to-online reinforcement learning.
In *International Conference on Machine Learning*, pp. 40452–40474. PMLR, 2023.
- Zhang et al. (2025)
Shiyuan Zhang, Weitong Zhang, and Quanquan Gu.
Energy-weighted flow matching for offline reinforcement learning.
*arXiv preprint arXiv:2503.04975*, 2025.
- Zheng et al. (2023)
Qinqing Zheng, Matt Le, Neta Shaul, Yaron Lipman, Aditya Grover, and Ricky TQ Chen.
Guided flows for generative modeling and decision making.
*arXiv preprint arXiv:2311.13443*, 2023.
- Zhou et al. (2025a)
Jin Peng Zhou, Kaiwen Wang, Jonathan D Chang, Zhaolin Gao, Nathan Kallus, Kilian Q Weinberger, Kianté Brantley, and Wen Sun.
q: Provably optimal distributional rl for llm post-training.
*CoRR*, 2025a.
- Zhou et al. (2025b)
Zhongyi Zhou, Yichen Zhu, Junjie Wen, Chaomin Shen, and Yi Xu.
Vision-language-action model with open-world embodied reasoning from pretrained knowledge.
*arXiv preprint arXiv:2505.21906*, 2025b.
- Zhou et al. (2025c)
Zhongyi Zhou, Yichen Zhu, Minjie Zhu, Junjie Wen, Ning Liu, Zhiyuan Xu, Weibin Meng, Ran Cheng, Yaxin Peng, Chaomin Shen, et al.
Chatvla: Unified multimodal understanding and robot control with vision-language-action model.
*arXiv preprint arXiv:2502.14420*, 2025c.
- Ziebart et al. (2008)
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al.
Maximum entropy inverse reinforcement learning.
In *Aaai*, volume 8, pp. 1433–1438. Chicago, IL, USA, 2008.

## Appendix A Flow-Based Temporal Difference Learning

We restate the flow-based TD objective and describe its derivation.

##### Distributional Bellman.

TD learning uses the Bellman equation to learn a value function by constructing a bootstrap target (i.e., the right-hand side (RHS) of the Bellman equation) (Bellman, 1966; Sutton & Barto, 1998), such that

$$ $Q(x,a)=r(x,a)+\gamma\mathbb{E}_{P,\pi}Q(X^{\prime},A^{\prime}).$ (22) $$

The Bellman equation also -holds under distributions (Jaquette, 1973; Sobel, 1982; White, 1988; Bellemare et al., 2017), such that

$$ $\underbrace{Z(x,a)}_{\mathclap{\texttt{LHS of Distributional Bellman}}}\overset{D}{=}\overbrace{r(x,a)+\gamma Z(X^{\prime},A^{\prime})}^{\mathclap{\texttt{RHS of Distributional Bellman}}}$ (23) $$

where $Z(X^{\prime},A^{\prime})$ denotes the random return.

##### Goal.

At a high level, flow matching learns to transport a known prior distribution into a target data distribution. To construct a flow-based TD objective, we set the RHS of the distributional Bellman equation as the target distribution, and match the velocities between the LHS and RHS distributions. We learn a conditional flow model $s_{\theta}(\cdot\mid x,a,t)$ that transports base noise $Y_{x,a}(0)\sim\mathcal{N}(0,I_{d})$ to a terminal variable $Y_{x,a}(1)\sim R_{\theta}(\cdot\mid x,a)$, such that the distribution $R_{\theta}(\cdot\mid x,a)\approx R(\cdot\mid x,a)$.

##### Conditional Flow Model.

We learn a conditional velocity field $s_{\theta}(y\mid x,a,t)$ that defines the ODE

$$ $\frac{d}{dt}Y_{x,a}(t)=s_{\theta}(Y_{x,a}(t)\mid x,a,t),\qquad Y_{x,a}(0)\sim p_{0}.$ (24) $$

Solving (i.e., “running”) this ODE from $t=0$ to $t=1$ is done by integration, giving the terminal random variable

$$ $Y_{x,a}(1)=Y_{x,a}(0)+\int_{0}^{1}s_{\theta}(Y_{x,a}(\tau)\mid x,a,\tau)d\tau.$ (25) $$

Let $R_{\theta}(\cdot\mid x,a)$ denote the induced terminal distribution. Our goal is to learn $R_{\theta}(\cdot\mid x,a)\approx R(\cdot\mid x,a)$.

##### Distributional Bellman.

By the definition of discounted reward-to-go,

$$ $Z(x,a)\overset{D}{=}r(x,a)+\gamma Z(X^{\prime},A^{\prime}),$ (26) $$

where $X^{\prime}\sim P(\cdot\mid x,a)$, $A^{\prime}\sim{\pi_{\textnormal{base}}}(\cdot\mid X^{\prime})$, and $Z(X^{\prime},A^{\prime})\sim R(\cdot\mid X^{\prime},A^{\prime})$.
Equivalently, we can say

$$ $R(\cdot\mid x,a)=\mathcal{L}\left(r(x,a)+Z^{\prime}\right),\quad Z^{\prime}\sim R(\cdot\mid X^{\prime},A^{\prime}),$ (27) $$

where $\mathcal{L}$ is the law of the random variable.
Taking expectation of Equation [26](https://arxiv.org/html/2510.08218v1#A1.E26) yields

$$ $\mathbb{E}_{Z\sim R(\cdot\mid x,a)}[Z]=r(x,a)+\gamma\mathbb{E}_{X^{\prime}\sim P(\cdot\mid x,a)}\mathbb{E}_{A^{\prime}\sim{\pi_{\textnormal{base}}}(\cdot\mid X^{\prime})}\mathbb{E}_{Z^{\prime}\sim R(\cdot\mid X^{\prime},A^{\prime})}[Z^{\prime}].$ (28) $$

##### Flow Integral and Expectation.

Going back to the ODE solution, we have

$$ $Y_{x,a}(1)=Y_{x,a}(0)+\int_{0}^{1}s_{\theta}(Y_{x,a}(\tau)\mid x,a,\tau)d\tau.$ (29) $$

Taking expectation first and then applying Fubini’s theorem, we have

$$ $\displaystyle\mathbb{E}\left[Y_{x,a}(1)\mid x,a\right]$ $\displaystyle=\mathbb{E}\left[Y_{x,a}(0)\right]+\mathbb{E}\left[\int_{0}^{1}\left[s_{\theta}(Y_{x,a}(\tau)\mid x,a,\tau)\right]d\tau\mid x,a\right]$ (30) $\displaystyle=\mathbb{E}\left[Y_{x,a}(0)\right]+\int_{0}^{1}\mathbb{E}\left[s_{\theta}(Y_{x,a}(\tau)\mid x,a,\tau)\mid x,a\right]d\tau.$ (31) $$

By definition of $p_{0}$ being zero mean, $\mathbb{E}[Y_{x,a}(0)]=0$, leaving us with:

$$ $\mathbb{E}\left[Y_{x,a}(1)\mid x,a\right]=\int_{0}^{1}\mathbb{E}\left[s_{\theta}(Y_{x,a}(\tau)\mid x,a,\tau)\mid x,a\right]d\tau.$ (32) $$

If we perform flow matching well, such that $R_{\theta}(\cdot\mid x,a)\approx R(\cdot\mid x,a)$ (subject to finite sample and optimization errors), then

$$ $\int_{0}^{1}\mathbb{E}\left[s_{\theta}(Y_{x,a}(\tau)\mid x,a,\tau)\mid x,a\right]d\tau=r(x,a)+\gamma\mathbb{E}_{X^{\prime},A^{\prime}}\left[\int_{0}^{1}\mathbb{E}\left[s_{\theta}(Y_{X^{\prime},A^{\prime}}(\tau)\mid X^{\prime},A^{\prime},\tau)\mid X^{\prime},A^{\prime}\right]d\tau\right].$ (33) $$

Finally, we have the following condition:

$$ $\mathbb{E}\left[s_{\theta}(Y_{x,a}(t)\mid x,a,t)\mid x,a\right]=r(x,a)+\gamma\mathbb{E}_{X^{\prime},A^{\prime}}\mathbb{E}\left[s_{\theta}(Y_{X^{\prime},A^{\prime}}(t)\mid X^{\prime},A^{\prime},t)\mid X^{\prime},A^{\prime}\right],\quad\forall t\in[0,1].$ (34) $$

##### Flow-Based TD Objective.

Putting this all together, we have the flow-based TD loss:

$$ $\mathcal{L}_{\text{FlowTD}}(\theta)=\underbrace{\mathbb{E}_{(x,a,r,x^{\prime})\sim\mathcal{D}}}_{\mathclap{\texttt{Dataset's State-Action-Reward}}}\overbrace{\mathbb{E}_{z^{1}\sim R_{\bar{\theta}}(\cdot\mid x,a)}}^{\mathclap{\texttt{Sample Reward-To-Go}}}\underbrace{\mathbb{E}_{t\sim\textnormal{Unif}(0,1)}}_{\mathclap{\texttt{Sample Time}}}\Big\|\overbrace{s_{\theta}(z^{t}\mid x,a,t)}^{\mathclap{\texttt{Velocity Prediction of LHS}}}-\underbrace{\text{target}(x,a,z^{t},t)}_{\mathclap{\texttt{Velocity Target of RHS}}}\Big\|_{2}^{2}],$ (35) $$

where

$$ $\text{target}(x,a,z^{t},t):=r(x,a)+\gamma\mathbb{E}_{a^{\prime}\sim{\pi_{\textnormal{base}}}(\cdot\mid x^{\prime})}s_{\bar{\theta}}(z^{t}\mid x^{\prime},a^{\prime},t).$ (36) $$

We sample a state-action-reward-next-state tuple $(x,a,r,x^{\prime})\sim\mathcal{D}$ from the offline data, a time $t\sim\textnormal{Unif}(0,1)$, and the next action from the base policy $a^{\prime}\sim{\pi_{\textnormal{base}}}(\cdot\mid x^{\prime})$. We construct an interpolant $z^{t}=(1-t)z^{0}+tz^{1}$, which adds noise the ground-truth sample, by sampling a reward-to-go $z^{1}\sim R(\cdot\mid x,a)$ and a noise sample $z^{0}\sim\mathcal{N}(0,I_{d})$. The reward-to-go sample $z^{1}$ can be sampled from the dataset or a target version of the learned reward model $R_{\bar{\theta}}(\cdot\mid x,a)$.
We sample a reward-to-go from $R_{\theta}(\cdot\mid x,a)$ using the standard forward Euler method applied on the learned flow model $s_{\theta}(\cdot\mid x,a,t)$.

##### Flow-Based TD Target.

The velocity target in [Equation 15](https://arxiv.org/html/2510.08218v1#S4.E15) represents a flow field that generates samples from the RHS of the distributional Bellman ([Equation 14](https://arxiv.org/html/2510.08218v1#S4.E14)), $r(x,a)+Z(X^{\prime},A^{\prime})$. The RHS of the distributional Bellman (i.e., the target distribution in flow matching) corresponds to the reward-to-go distribution translated by the one-step reward $r(x,a)$. Under flow matching, such a translation shifts every particle in the flow trajectory by a constant amount. The vector field specifies the instantaneous rate of change of particle positions, so adding a constant shift to all trajectories increases the velocity everywhere by the same constant.
Consequently, the target
$s_{\mathrm{target}}=r(x,a)+\gamma\mathbb{E}_{a^{\prime}\sim{\pi_{\textnormal{base}}}(\cdot\mid x^{\prime})}\bar{s}_{\theta}(z^{t}\mid x^{\prime},a^{\prime},t)$
is the one-step reward shift and the expected next-state flow under ${\pi_{\textnormal{base}}}$.

## Appendix B Full Results

**Table 2: EVOR’s Overall Performance By Task. We present the full results on each OGBench task. (*) indicates the default task in each environment. The results are averaged over five seeds with standard deviations reported.**
| Task | QC-1 | QC-5 | EVOR |
| --- | --- | --- | --- |
| antmaze-large-navigate-singletask-task1-v0 (*) | $3$ $\pm 3$ | $2$ $\pm 1$ | $\mathbf{22}$ $\pm 11$ |
| antmaze-large-navigate-singletask-task2-v0 | 0 $\pm 0$ | 0 $\pm 0$ | $\mathbf{62}$ $\pm 5$ |
| antmaze-large-navigate-singletask-task3-v0 | 43 $\pm 26$ | $28$ $\pm 11$ | $32$ $\pm 4$ |
| antmaze-large-navigate-singletask-task4-v0 | 0 $\pm 0$ | 0 $\pm 0$ | $\mathbf{65}$ $\pm 9$ |
| antmaze-large-navigate-singletask-task5-v0 | 0 $\pm 0$ | $4$ $\pm 3$ | $\mathbf{69}$ $\pm 5$ |
| antmaze-large-stitch-singletask-task1-v0 (*) | 3 $\pm 2$ | 3 $\pm 2$ | 0 $\pm 0$ |
| antmaze-large-stitch-singletask-task2-v0 | 0 $\pm 0$ | 0 $\pm 0$ | 1 $\pm 0$ |
| antmaze-large-stitch-singletask-task3-v0 | $28$ $\pm 31$ | $21$ $\pm 16$ | 67 $\pm 5$ |
| antmaze-large-stitch-singletask-task4-v0 | 0 $\pm 0$ | 0 $\pm 0$ | 5 $\pm 1$ |
| antmaze-large-stitch-singletask-task5-v0 | 15 $\pm 14$ | 16 $\pm 5$ | $4$ $\pm 3$ |
| cube-double-play-100M-singletask-task1-v0 (*) | $88$ $\pm 6$ | $78$ $\pm 11$ | $\mathbf{96}$ $\pm 2$ |
| cube-double-play-100M-singletask-task2-v0 | $55$ $\pm 4$ | $54$ $\pm 24$ | $\mathbf{93}$ $\pm 4$ |
| cube-double-play-100M-singletask-task3-v0 | $49$ $\pm 11$ | $56$ $\pm 26$ | $\mathbf{94}$ $\pm 6$ |
| cube-double-play-100M-singletask-task4-v0 | $23$ $\pm 4$ | 38 $\pm 17$ | $32$ $\pm 8$ |
| cube-double-play-100M-singletask-task5-v0 | $60$ $\pm 7$ | $59$ $\pm 26$ | $\mathbf{90}$ $\pm 6$ |
| pointmaze-medium-navigate-singletask-task1-v0 (*) | 97 $\pm 3$ | $\mathbf{99}$ $\pm 1$ | $\mathbf{99}$ $\pm 1$ |
| pointmaze-medium-navigate-singletask-task2-v0 | $85$ $\pm 15$ | $\mathbf{100}$ $\pm 1$ | $\mathbf{99}$ $\pm 2$ |
| pointmaze-medium-navigate-singletask-task3-v0 | 98 $\pm 3$ | $\mathbf{99}$ $\pm 2$ | $\mathbf{99}$ $\pm 2$ |
| pointmaze-medium-navigate-singletask-task4-v0 | $76$ $\pm 39$ | $\mathbf{100}$ $\pm 0$ | $\mathbf{100}$ $\pm 0$ |
| pointmaze-medium-navigate-singletask-task5-v0 | 100 $\pm 0$ | $\mathbf{100}$ $\pm 0$ | $\mathbf{100}$ $\pm 0$ |
| scene-play-sparse-singletask-task1-v0 | $93$ $\pm 2$ | $\mathbf{100}$ $\pm 0$ | $\mathbf{100}$ $\pm 0$ |
| scene-play-sparse-singletask-task2-v0 (*) | $85$ $\pm 5$ | $\mathbf{99}$ $\pm 1$ | $\mathbf{98}$ $\pm 1$ |
| scene-play-sparse-singletask-task3-v0 | $52$ $\pm 15$ | $\mathbf{93}$ $\pm 3$ | $\mathbf{89}$ $\pm 5$ |
| scene-play-sparse-singletask-task4-v0 | $4$ $\pm 4$ | $\mathbf{91}$ $\pm 3$ | $84$ $\pm 24$ |
| scene-play-sparse-singletask-task5-v0 | 0 $\pm 0$ | $33$ $\pm 16$ | $\mathbf{64}$ $\pm 17$ |

Q: What is EVOR’s task-level performance?

Across five environments and 25 unique tasks, EVOR achieves the best performance compared to the baselines. EVOR’s expressive value learning method outperforms standard value learning methods. From the results in [Table 2](https://arxiv.org/html/2510.08218v1#A2.T2), we observe that EVOR outperforms or matches standard value function learning methods (QC), even compared to a method that employs action chunking (QC-5), suggesting that expressive value learning can improve performance over standard value function learning.

## Appendix C Ablation Studies

Figure: Figure 3: Ablation Over Number of Action Candidates ${N_{\pi}}$. Results are averaged over three seeds per task, with standard deviations reported.
Refer to caption: x4.png

Q: [Task-Level] How can EVOR take advantage of greater inference-time compute?

As shown in [Figure 3](https://arxiv.org/html/2510.08218v1#A3.F3), when given access to greater inference-time compute, EVOR can increase the number of action candidates ${N_{\pi}}$, resulting in better performance (up to a saturation point).

Figure: Figure 4: Ablation Over Reward-To-Go Temperature Parameter $\tau_{R}$. Results are averaged over three seeds per task, with standard deviations reported.
Refer to caption: x9.png

Figure: Figure 5: Ablation Over $Q^{\star}$ Temperature Parameter $\tau_{Q}$. Results are averaged over three seeds per task, with standard deviations reported.
Refer to caption: x14.png

Q: [Task-Level] How can EVOR perform inference-time regularization?

As shown in [Figure 4](https://arxiv.org/html/2510.08218v1#A3.F4) and [Figure 5](https://arxiv.org/html/2510.08218v1#A3.F5), by increasing varying the temperature parameters $\tau_{R}$ and $\tau_{Q}$, EVOR can vary the level of regularization to the base policy compared to the level of policy optimization. As $\tau_{Q}$ decreases, the action selection becomes more greedy, while as $\tau_{Q}$ increases, the action selection becomes more regularized. Set to a high value, EVOR becomes equivalent to the base policy (i.e., the performance with ${N_{\pi}}=1$).

## Appendix D Experimental and Implementation Details

In this section, we describe the setup, implementation details, and baselines used in the paper. Our code is available at
[nico-espinosadice.github.io/projects/evor](https://nico-espinosadice.github.io/projects/evor).

### D.1 Experimental Setup

We follow OGBench’s official evaluation scheme (Park et al., 2024a), with the reward-maximizing offline setup of Park et al. (2025b); Espinosa-Dice et al. (2025). We restate the experimental setup here. Following Park et al. (2025b); Espinosa-Dice et al. (2025), we use OGBench’s singletask variants for all experiments, corresponding to reward-based tasks that are suitable for our reward-maximizing offline RL setting.

##### Environments and Tasks.

EVOR is evaluated on manipulation and locomotion robotics tasks in version 1.1.0 of OGBench (Park et al., 2024a), including

- 1.
antmaze-large-navigate-singletask-v0
- 2.
antmaze-large-stitch-singletask-v0
- 3.
cube-double-play-singletask-v0
- 4.
pointmaze-medium-navigate-singletask-v0
- 5.
scene-play-singletask-v0

We use the three unique tasks (e.g., antmaze-large-navigate-singletask-task{1,2,3,4,5}-v0) for each environment listed above, where each task provides a unique evaluation goal. Each environment’s dataset is labeled with a semi-sparse reward (Park et al., 2024a), and we use the sparse reward for scene-play, following Li et al. (2025). For the cube-double-play environment, we use the 100M size dataset provided by Park et al. (2025a).

The selected environments consist of locomotion and manipulation control problems. The antmaze tasks consist of navigating a quadrupedal agent (8 degrees of freedom) through complex mazes. The cube and scene environments manipulated objects with a robotic arm. The goal of scene tasks is to sequence multiple subtasks. The environments are state-based. We test both navigate and stitch datasets for locomotion and play for manipulation. These datasets are built from suboptimal, goal-agnostic trajectories, which poses a challenge for goal-directed policy learning (Park et al., 2024a). Following Park et al. (2025b); Espinosa-Dice et al. (2025), we evaluate agents using binary task success rates (i.e., goal completion percentage), which is consistent with OGBench’s evaluation setup (Park et al., 2024a).

##### Evaluation.

We follow OGBench’s official evaluation scheme (Park et al., 2024a). Algorithms are trained for 1,000,000 gradient steps and evaluated on 50 episodes every 100,000 gradient steps. The average success rates of the final three evaluations (i.e., the evaluation results at 800,000, 900,000, and 1,000,000 gradient steps) are reported. Tables average over 3 seeds per task and report standard deviations, bolding values within 95% of the best performance.

### D.2 EVOR Implementation Details

##### Flow Matching.

EVOR is implemented on top of Li et al. (2025)’s open-source implementation of QC, which is adapted from Park et al. (2024a)’s open-source codebase. We implement flow matching using the same, standard velocity field as QC.

##### Network Architecture and Optimizer.

Following Park et al. (2025b); Espinosa-Dice et al. (2025); Li et al. (2025), we use a multi-layer perceptron with 4 hidden layers of size 512 for both the value and policy networks. We apply layer normalization (Ba et al., 2016) to value networks and use the Adam optimizer (Kingma, 2014). All of these parameters are shared between EVOR and the baselines.

##### Hyperparameters.

We use the same hyperparameters for both EVOR and QC. Unlike many offline RL algorithms (Park et al., 2025b; Espinosa-Dice et al., 2025), EVOR does not change training parameters between environments in this paper. EVOR uses ${N}=1$ during training (instead of the ${N}>1$ used during evaluation) for better efficiency.

##### Inference Procedure.

EVOR’s inference procedure is shown in [Algorithm 2](https://arxiv.org/html/2510.08218v1#algorithm2). Actions are sampled from the base policy ${\pi_{\textnormal{base}}}$ via the forward Euler method, shown in Algorithm [3](https://arxiv.org/html/2510.08218v1#algorithm3).

Figure: Algorithm 3 ${\pi_{\textnormal{base}}}$ Action Sampling via Forward Euler Method

Recall that the optimal $Q$-function is given by:

$$ $Q^{\star}(x,a)=\eta\ln\mathbb{E}_{r\sim R(\cdot\mid x,a)}\exp(r/\eta),$ (37) $$

where $R$ is the conditional distribution of rewards-to-go under ${\pi_{\textnormal{ref}}}$. We learn an estimate of $R$ via the flow-based TD objective, such that $R_{\theta}(\cdot\mid x,a)\approx R(\cdot\mid x,a)$. We approximate the expectation via sample averaging, as shown in [Algorithm 2](https://arxiv.org/html/2510.08218v1#algorithm2), such that

$$ $\operatorname{LogSumExp}(z^{(j)})=\tau^{\star}\,\log\frac{1}{{N}}\sum_{j=1}^{N}\exp\left(\tfrac{z^{(j)}}{\tau^{\star}}\right)$ (38) $$

We then construct a weighted softmax via the $Q^{\star}$ approximation, such that

$$ $\operatorname{softmax}(Q^{\star}(x,a^{(j)}))=\frac{\exp(Q^{\star}(x,a^{(j)})/\tau)}{\sum_{j=1}^{N_{\pi}}\exp(Q^{\star}(x,a^{(j)})/\tau)}$ (39) $$

**Table 3: Shared Hyperparameters Between QC Baselines and EVOR.**
| Parameter | Value |
| --- | --- |
| Learning rate | 3e-4 |
| Optimizer | Adam (Kingma, 2014) |
| Gradient Steps | 1e6 |
| Minibatch Size | 256 |
| MLP Dimensions | [512, 512, 512, 512] |
| Target Network Smoothing Coefficient | 5e-3 |
| Discount Factor $\gamma$ | 0.99 |
| Discretization Steps | 10 |
| Time Sampling Distribution | Unif([0,1]) |
| Number of Action Candidates ${N_{\pi}}$ | 32 |

**Table 4: Hyperparameters for EVOR.**
| Hyperparameter | Value |
| --- | --- |
| Beta $\beta$ | 1e-3 |
| $Q^{\star}$ Beta $\beta^{\star}$ | 1 |
| Number of RTG Samples $N^{\text{RTG}}$ | 1 (Train), 50 (Eval) |

**Table 5: Hyperparameters for QC.**
| Hyperparameter | Value |
| --- | --- |
| Action Chunk Length | 1 (QC-1), 5 (QC-5) |
| Critic Ensemble Size | 2 |

### D.3 Baselines

Rather than compare to all of the existing offline RL algorithms benchmarked on OGBench, we instead aim to isolate the effect of expressive value learning in order to demonstrate its benefit specifically. Thus, we compare to $Q$-chunking (QC, Li et al. (2025)), a recent offline RL algorithm that is closest to EVOR. Like EVOR, QC learns a base policy via flow matching and extracts an optimized policy via rejection sampling. The key difference between QC and EVOR is in how the value function is learned, which is the exact difference we aim to isolate. QC can employ action chunking in both its policy and value function, and we compare EVOR to both QC with (QC-5) action chunking and without it (QC-1). We select the action chunk length (5) based on Li et al. (2025)’s recommendation.

Given an action chunk length of $k$, represented as $\bm{a}_{t:t+k}=(a_{t},a_{t+1},\ldots,a_{t+k})$, the $Q$-function is updated via

$$ $Q(x_{t},\bm{a}_{t:t+k})\leftarrow\sum_{t^{\prime}=1}^{t+k-1}\left[r_{t^{\prime}}\right]+Q(x_{t+k},\bm{a}_{t+k:t+2k})$ (40) $$

and actions are sampled via

$$ $\bm{a}\leftarrow\operatorname*{argmax}_{\bm{a}\in\{\bm{a}_{1},\bm{a}_{2},\ldots,\bm{a}_{N}\}}Q(x,\bm{a}),$ (41) $$

where $a_{1},a_{2},\ldots,a_{N}\sim{\pi_{\textnormal{base}}}(\cdot\mid x)$. This yields the following loss function for learning the $Q$-function:

$$ $L(\theta)=\mathbb{E}_{\begin{subarray}{c}x_{t},\bm{a}_{t}\sim\mathcal{D}\\ \{a^{i}_{t+k}\}^{N}_{i=1}\sim{\pi_{\textnormal{base}}}(\cdot\mid x_{t+k})\end{subarray}}\left[\left(Q_{\theta}(x_{t},\bm{a}_{t})-\sum_{t^{\prime}=1}^{k}r_{t+t^{\prime}}-Q_{\bar{\theta}}(x_{t+k},\bm{a}_{t+k})\right)^{2}\right],$ (42) $$

where $\bm{a}_{t+k}=\operatorname*{argmax}_{a\in\{\bm{a}^{i}_{t+k}\}}Q(s,\bm{a})$.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: Since the paper emphasizes Scalable Offline Reinforcement Learning, could the authors further validate the effectiveness of Expressive Value Learning on the D4RL MuJoCo datasets? Compared with OGBench, the behaviors in D4RL datasets are typically more suboptimal and diverse, making them a strong testbed for scalability and robustness. Would Expressive Value Learning help improve performance under such more challenging, low-quality offline data distributions? This seems like a very meaningful way to verify the scalability claims of the proposed method.
soundness: 2
strengths: - Clear motivation and problem formulation. The authors clearly identify the scalability bottlenecks in prior generative-policy offline RL methods—specifically the reliance on BPTT or distillation. They motivate EVOR as a natural step forward: integrating expressive flow-based modeling into the value function, not just the policy (Sec. 1–3).
- Elegant inference-time optimization mechanism. EVOR’s rejection-sampling-based policy extraction (Algorithm 2, Eq. 22) allows scalable test-time optimization without retraining. The mechanism directly leverages the expressive Q function and supports compute scaling via N_\pi and temperature parameters (τ_R, τ_Q), which are empirically validated. EVOR’s inference-time scaling shows performance increasing monotonically with compute, aligning well with scaling-law intuition in modern generative RL systems. This feature enhances real-world deployability in compute-flexible settings.
- Empirical results show benefit. Table 1 and Table 2 demonstrate consistent improvement on all five OGBench environments, with particularly strong gains on difficult manipulation and navigation tasks.
summary: This paper introduces EVOR (Expressive Value Learning for Offline Reinforcement Learning), a new scalable offline RL framework that integrates expressive generative value functions with expressive policies. Existing methods that use diffusion or flow-matching policies in offline RL often rely on backpropagation through time (BPTT) or policy distillation, both of which limit scalability.

EVOR avoids both by:
1. Learning a regularized, expressive Q-function using flow matching, resulting in an optimal solution to the KL-regularized offline RL objective.
2. Performing inference-time policy extraction using rejection sampling guided by the expressive value function.
3. Supporting inference-time scaling and regularization by adjusting the number of sampled actions and softmax temperatures.

Empirical results on OGBench (25 tasks across 5 environments) show consistent improvements over Q-Chunking (QC-1 and QC-5) baselines.
weaknesses: - Limited novelty compared to prior works. The construction of the value function in EVOR essentially follows the same principle as "Q♯: Provably Optimal Distributional RL for LLM Post-training", which estimates Q^*(x,a) by sampling reward-to-go values and applying a softmax/log-sum-exp aggregation. The only difference is that EVOR learns the reward-to-go distribution via flow matching, but this idea is already very similar to TD-Flow (Farebrother et al., 2025) and Intention-Conditioned Flow Occupancy Models (ICFOM), which also learn value functions from offline data using flow-based velocity fields. Hence, the claimed contribution is incremental rather than conceptually novel.
- Restricted to KL-regularized RL formulation. EVOR is derived specifically under the KL-regularized RL objective, which is known to suffer from over-conservatism due to the KL divergence (penalizing exploration beyond the dataset distribution). This limitation may constrain the method’s ability to generalize to other regularizers (e.g., support-based ones) or to handle cases where mild extrapolation beyond the offline data is beneficial.
- Dependence on rejection sampling for policy extraction. The learned Q(s,a) is only utilized through rejection-sampling-based policy extraction, which makes the method heavily reliant on the quality of the offline dataset. When the offline data are suboptimal or sparse in high-value regions, rejection sampling cannot meaningfully exceed the dataset performance, since all sampled actions originate from the base policy. This limits EVOR’s applicability in low-quality or diverse-data regimes, where gradient-based policy improvement or explicit policy learning may be preferable.
- Comparative scope is narrow. Experiments are restricted to QC baselines and only on OGBench datasets. Other recent scalable offline RL approaches (e.g., Flow Q-Learning) are missing. Since EVOR’s main claim is “scalability,” benchmarking against state-of-the-art methods beyond QC would strengthen the empirical case.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: - Could you provide additional results with popular baselines in OGBench? As the authors denote in the paper, comparing with baselines trained with standard offline RL objectives may escape the scope of this paper. However, comparing with diffusion-based (or flow-based) policy learning [1, 2] (representatives for BPTT) or policy distillation methods [3] (representatives for distillation) would provide an in-depth view of scalable offline RL with generative models.
- Could you visualize how EVOR learns complex behaviors from the dataset using flow matching TD objectives and policy extractions? For instance, Diffusion Policy [4] demonstrates strong multi-modal capability by visualizing modes in trajectory levels.

# Minor problems (do not affect the score)
- There are several typos:
- - In L86, does not learn require $\rightarrow$ does not require
- - In L91, the Q-function learned $\rightarrow$ the learned Q-function
- - In 199, samples from in $\rightarrow$ samples from
- The Related Work section is duplicated in the Appendix. I appreciate the extended related work section in the Appendix, but offline RL with generative models and inference-time scaling in offline RL are exactly the same as the section in the main paper.

[1] Park S, Li Q, Levine S. Flow q-learning. arXiv 2025.

[2] Hansen-Estruch P, Kostrikov I, Janner M, Kuba JG, Levine S. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv 2023.

[3] Ding Z, Jin C. Consistency models as a rich and efficient policy class for reinforcement learning. arXiv 2023.

[4] Chi C, Xu Z, Feng S, Cousineau E, Du Y, Burchfiel B, Tedrake R, Song S. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research 2025.
soundness: 3
strengths: - The paper is well structured and sounds coherent.
- EVOR demonstrates strong performance on diverse tasks in OGBench.
- The authors provide empirical observations on EVOR's inference-time scaling property by carefully designing ablation studies.
summary: This paper proposes EVOR, a novel flow-matching-based offline reinforcement learning algorithm that scales to data, model, and inference-time computation without enforcing the backpropagation through time or policy distillation. EVOR introduces expressive policy and value learning via flow-based temporal difference learning with the distributional Bellman operator and the reward-to-go approach. Furthermore, EVOR enables inference-time-scalable offline RL by leveraging rejection sampling with the learned optimal value function and searching the test-time hyperparameters (e.g., $\tau$ temperature) without further training. Experiments highlight the contribution of expressive value learning against the baseline with flow matching policy learning in OGBench.
weaknesses: - The novelty of EVOR is marginal. The theoretical analysis of expressive flow matching policy learning strictly depends on Theorem 2.2 in Zhou et al. While it is noticeable that EVOR suggests a novel expressive value learning by introducing a flow-based TD objective with reward-to-go, experimental results on verifying the performance of EVOR on diverse offline RL tasks are insufficient to claim the practicality of EVOR. I appreciate the main results on comparing QC with EVOR and the additional ablation studies on inference-time scaling, while I believe further experiments on differentiating EVOR with state-of-the-art offline RL methods (specifically, offline RL with generative models) would support the novelty of this paper.
- It is unclear how EVOR scales to different axes of scalability: the data and model in offline RL. The authors introduce three heterogeneous axes for scaling offline RL in the introduction: the data, model, and computing. However, the primary results in the paper mainly consider the inference-time scaling property. Offline trajectories often involve multi-modal, sub-optimal behaviors from unknown policies, which makes it hard for the model to learn such diverse behaviors with limited scalability. While OGBench provides diverse tasks with thoroughly collected sub-optimal behaviors, comparing only the performance of EVOR in single tasks may overshadow the true scalability of the data and model.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 2
questions: 1.	It would be better if the authors add ablations on flow-based TD learning, compare to Gaussian critics or classification-based Q-updates (Farebrother et al., 2024) to show explicit benefit.
2.	It would be better if the authors include wall-clock efficiency metrics to show training/inference time versus QC and policy distillation methods.
3.	It would be better if the authors discuss potential integration with policy gradients.  The discussion section (p. 9) hints at combining EVOR with reparameterized gradients; elaborating on this could position EVOR as a foundation for hybrid methods.
soundness: 3
strengths: The paper addresses a timely and underexplored issue: how to scale offline RL with expressive generative models without incurring the computational and stability issues of BPTT or distillation. While diffusion and flow models have been extensively used for policy modeling, they have not been effectively integrated into value learning. EVOR’s insight, that flow-based value functions can provide a scalable and expressive mechanism for both regularization and inference-time optimization, is conceptually elegant and fills an important gap.
The theoretical formulation is solid and well-grounded, and this clarity and generality make EVOR theoretically appealing and reproducible.
Unlike prior work that focuses on expressive policies, EVOR emphasizes expressive value functions, a neglected component in scaling offline RL. This focus on value expressivity represents a genuine conceptual advance and could influence future RL architectures.
EVOR achieves consistent improvements across many tasks, outperforming QC-1 and QC-5 baselines. Importantly, EVOR uses the same training hyperparameters across all environments.
summary: This paper proposes EVOR (Expressive Value Learning for Offline Reinforcement Learning), a scalable offline RL framework that integrates expressive generative models (flow matching) into both the policy and the value function.
The authors identify a key bottleneck in recent diffusion- and flow-based offline RL: Existing methods either rely on backpropagation through time (BPTT), which is computationally expensive, or on policy distillation, which introduces compounding errors and limits scalability to larger base policies (e.g., vision-language-action models).
EVOR addresses this by: 1. Learning expressive value functions via flow-based temporal difference (TD) learning, avoiding Gaussian approximations 2. Performing inference-time policy extraction through rejection sampling against the expressive learned value function, enabling scalable optimization and regularization without retraining. 3. Leveraging inference-time compute scaling  and regularization. EVOR thus provides a unified framework for scalable, expressive, and compute-efficient offline RL, avoiding BPTT and distillation entirely. Empirically, EVOR significantly outperforms the baseline Q-Chinking (QC) across 25 tasks in the OGBench suite (antmaze, cube manipulation, scene play, and pointmaze). It also demonstrates strong scaling properties with increased inference compute and robustness to hyperparameters.
weaknesses: Although the authors justify focusing on QC to isolate “value learning expressivity,” it would be more compelling to include additional baselines such as IQL and CQL. 
All results are on simulation-based OGBench tasks. Given the paper’s claim of “scalability,” evaluation on higher-dimensional or real-robot tasks (e.g., Pi0, RT-1 datasets) would strengthen the impact. Even a discussion on computational resource scaling (training time, memory vs. QC) would help quantify efficiency.
While the ablations on inference-time parameters are thorough, there is no study isolating the contribution of flow-based TD learning itself versus standard TD learning with a Gaussian critic. A simple comparison of “EVOR without flow-based TD” would help clarify whether expressivity, or the training dynamics, drive the observed improvements.
The derivation of the optimal regularized Q-function (Eq. 16) assumes deterministic transitions, which rarely hold in realistic robotics. Although the authors later extend to stochastic settings (Section 3.2), a discussion of approximation error or theoretical relaxation would improve rigor.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: Have you compared the training time and the number of parameters with non-generative offline methods such as CQL [1]?

Reference:

[1] Kumar et al. "Conservative Q-Learning for Ofﬂine Reinforcement Learning", NeurIPS, 2020.
soundness: 2
strengths: - EVOR enables inference-time policy extraction without relying on model distillation or backpropagation through time.
- The proposed approach outperforms baselines in various benchmarks.
- This paper provides both empirical and theoretical analyses.
summary: This work presents a generative offline RL algorithm, which avoids policy distillation and backpropagation through time during policy optimization. Experiments on OGBench validate the effectiveness of the proposed method.
weaknesses: - The authors argue that one-step models are difficult to scale to larger base policies (e.g., VLAs) or real-world tasks. However, no corresponding experiments on VLA or real-world settings are provided to support this claim.
- The paper includes only a single baseline (Q-chunking) and lacks comparisons with other competitive methods, such as FQL [1].
- The evaluation is limited to OGBench. It would be convincing to include results on other benchmarks, such as D4RL [2].

References:

[1] Park et al. "Flow Q-Learning", ICML, 2025.

[2] Fu et al. "D4RL: Datasets for Deep Data-Driven Reinforcement Learning", arXiv preprint arXiv:2004.07219, 2020.


Decision: Reject