VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learning

## Abstract

Abstract Offline reinforcement learning (RL) learns effective policies from pre-collected datasets, offering a practical solution for applications where online interactions are risky or costly. Model-based approaches are particularly advantageous for offline RL, owing to their data efficiency and generalizability. However, due to inherent model errors, model-based methods often artificially introduce conservatism guided by heuristic uncertainty estimation, which can be unreliable. In this paper, we introduce VIPO, a novel model-based offline RL algorithm that incorporates self-supervised feedback from value estimation to enhance model training. Specifically, the model is learned by additionally minimizing the inconsistency between the value learned directly from the offline data and the one estimated from the model. We perform comprehensive evaluations from multiple perspectives to show that VIPO can learn a highly accurate model efficiently and consistently outperform existing methods. In particular, it achieves state-of-the-art performance on almost all tasks in both D4RL and NeoRL benchmarks. Overall, VIPO offers a general framework that can be readily integrated into existing model-based offline RL algorithms to systematically enhance model accuracy. Our code is available at https://github.com/NUS-CORE/vipo .

## 1 Introduction

Offline reinforcement learning (RL) (Lange et al., 2012; Levine et al., 2020) aims to learn effective policies exclusively from a pre-collected behavior dataset, eliminating the need for online interaction with the environment. This approach offers a compelling solution for tasks where online interactions entail significant risks or exorbitant costs. Due to its potential to transform static datasets into powerful decision-making systems, offline RL has gained popularity in recent years (Kalashnikov et al., 2018; Prudencio et al., 2023). While off-policy RL algorithms can in principle be directly applied to the offline datasets, recent studies show that they can perform poorly when applied offline (Fujimoto et al., 2019; Kumar et al., 2019). This is primarily attributed to the distribution shift between the learned and the behavior policies, as learning policies beyond the behavior policy requires querying the actions not observed in the dataset. Similarly, value function evaluation on out-of-distribution (OOD) actions is usually inaccurate. In turn, maximizing an inaccurate value function during policy improvement often overestimates the value of OOD actions, which is a core challenge in offline RL.

To mitigate overestimation, a common paradigm in offline RL is to incorporate conservatism into algorithm design. Model-free offline RL algorithms (Fujimoto et al., 2018; Wu et al., 2019; Kumar et al., 2020; Wang et al., 2022) achieve this by learning a pessimistic value function (value-based approach) to discourage choosing OOD actions or constraining the learned policy (policy-based approach) to avoid visiting such actions. However, these algorithms often suffer from overly conservatism as many of them solely learn from the dataset (Wang et al., 2018; Chen et al., 2020; Kostrikov et al., 2021). Meanwhile, it is crucial to balance conservatism and generalization since being overly conservative hinders finding a better policy. In contrast, model-based offline RL algorithms (Kidambi et al., 2020; Yu et al., 2020; Sun et al., 2023) ensure conservative by learning a pessimistic dynamics model which penalizes the value of OOD actions. Additionally, model-based algorithms have the potential for broader generalization as they can incorporate dynamics models to generate synthetic data that are not present in the dataset (Yu et al., 2020). This advantage makes model-based methods particularly well-suited for offline RL, motivating their further investigation in this paper.

Figure: Figure 1: Comparison of VIPO with previous model-based approaches for learning a pessimistic dynamics model. (A) Previous model-based methods make a single use of data to learn an ensemble of models and then use its uncertainty to apply *ad-hoc*, pessimistic adjustments to the model predictions. (B) VIPO leverages the data in two ways: (1) it learns a value function $V_{d}^{\mu}(s)$ directly by minimizing the mean-square Bellman error (MSBE) loss; (2) it first learns the dynamics model by minimizing the negative log-likelihood (NLL) loss and then estimate the ensemble value functions $V_{m}^{\mu}(s)$. VIPO utilizes the discrepancy between these two types of value functions as an additional self-supervised loss to improve the model learning performance.
Refer to caption: fig5.png

Most prior model-based algorithms rely on uncertainty estimation to incorporate conservatism. They commonly train an ensemble of $N$ models through maximum likelihood on the dataset and quantify uncertainty based on the standard deviation among the predictions of these $N$ models. Specifically, (Yu et al., 2020) employs the max-aleatoric uncertainty quantifier, (Kidambi et al., 2020) adopts the max-pairwise-diff uncertainty quantifier, and (Lu et al., 2021) utilizes the ensemble-standard-deviation uncertainty quantifier. Nevertheless, it was shown that the uncertainties of their learned models using these methods are inaccurate and unreliable when dealing with complex datasets in practice (Ovadia et al., 2019). This aligns with our findings: as shown empirically in [Section˜4.2](https://arxiv.org/html/2504.11944v2#S4.SS2), even in Gym tasks the uncertainties estimated by MOPO-learned models fail to reflect the expected increase as the data-drop ratio grows, whereas those from VIPO-learned models track this trend well. This indicates that VIPO captures epistemic uncertainty due to data loss more effectively than MOPO and thus is likely to learn a more accurate model.

In this paper, we introduce Value Function Inconsistency Penalized Offline Reinforcement Learning (VIPO), a model-based offline RL algorithm that aims to learn a highly accurate model by integrating a value function inconsistency loss into model training. VIPO differs fundamentally from prior methods in two key aspects. First, our approach makes a dual-usage of the data in a self-supervised fashion to improve the model training accuracy. As illustrated in [Figure˜1](https://arxiv.org/html/2504.11944v2#S1.F1),
previous model-based RL methods make only a single-purpose use of the data: they learn an ensemble of models and obtain a pessimistic model by applying ad-hoc adjustments to the model outputs based on the uncertainty estimated from the ensemble. In contrast, VIPO leverages the data in two complementary ways. The first use of the data is to directly learn an approximated value function of the behavior policy by minimizing the empirical Mean Square Bellman Error (MSBE). The second use of the data is to learn a model by minimizing the negative log-likelihood loss (NLL), followed by value evaluation to obtain another approximated value function of the behavior policy. If the learned model is accurate, the two value functions should coincide owing to the uniqueness of the value function, and their alignment (or lack thereof) provides a direct quantifier of model accuracy. Since offline RL must rely entirely on the fixed dataset, effectively exploiting it is critical. Compared with prior approaches that use the data for a single purpose, our method leverages it in two complementary ways.

Second, unlike previous works which do not mitigate the ensemble-model uncertainty in the training stage, we proactively incorporate the value function inconsistency into the model training loss in a self-supervised manner to improve model learning. By establishing the analytical expression for the gradient of the augmented loss in [Theorem˜3.3](https://arxiv.org/html/2504.11944v2#S3.Thmtheorem3), we enable the model to be trained via gradient descent, ensuring a balance between data fidelity and alignment in value functions. To the best of our knowledge, no existing model-based offline RL method has incorporated either uncertainty or inconsistency into the training stage. Our approach is the first to proactively mitigate inconsistency during training, instead of ad-hoc adjustments of the model predictions.

Our empirical results demonstrate from multiple perspectives that incorporating value function inconsistency into model error enhances model accuracy, highlighting the effectiveness of the aforementioned two novel designs. VIPO offers a general framework that can be readily integrated into existing model-based offline RL algorithms to systematically enhance model accuracy. We achieve this without relying on expensive architectures such as diffusion or transformer, rather than by comprehensive exploitation of data.
Moreover, VIPO outperforms prior offline RL algorithms and achieves state-of-the-art (SOTA) performance on most tasks of both D4RL and NeoRL benchmarks.

## 2 Preliminaries

### 2.1 Offline Reinforcement Learning

RL problems are commonly formulated within the framework of a Markov Decision Process (MDP), defined by the tuple $\mathcal{M}=(\mathcal{S},\mathcal{A},r,\rho_{0},P,\gamma)$. Here, $\mathcal{S}$ denotes the state space, $\mathcal{A}$ represents the action space, $r(s,a):\mathcal{S}\times\mathcal{A}\to[-r_{\max},r_{\max}]$ is a bounded reward function, $\rho_{0}(s)$ specifies the initial state distribution, $P(s^{\prime}\,|\,s,a):\mathcal{S}\times\mathcal{A}\times\mathcal{S}\to\mathbb{R}_{+}$ defines the transition kernel, and $\gamma\in[0,1)$ is the discount factor. Given a policy $\pi(\cdot\,|\,s)$, its performance can be evaluated by the expected cumulative long-term reward, defined as

$$ $\displaystyle J(\pi)=\mathbb{E}_{s_{0}\sim\rho_{0},a_{t}\sim\pi,s_{t+1}\sim P}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})\right].$ (1) $$

The goal of RL is to learn a policy $\pi(\cdot\,|\,s)$ that maximizes the expected cumulative long-term reward, i.e., $\pi^{\ast}=\mathop{\arg\max}_{\pi}J(\pi)$.

As shown in Eq. ([1](https://arxiv.org/html/2504.11944v2#S2.E1)), the classical RL framework requires online interactions with the environment $P$ during training. In contrast, the offline RL learns only from a fixed dataset $\mathcal{D}=\{(s,a,r,s^{\prime})\}$ collected by the behavior policy $\mu(\cdot\mid s)$, where $s$, $a$, $r$, and $s^{\prime}$ denote the state, action, reward, and next state, respectively. That is, it aims to find the best possible policy solely from $\mathcal{D}$ without additional interactions with the environment.

### 2.2 Model-based Offline RL Algorithms

Model-based offline RL algorithms aim to derive the optimal policy by utilizing a learned dynamics model. These methods define a pessimistic MDP $\widehat{\mathcal{M}}=\langle\mathcal{S},\mathcal{A},\hat{r},\rho_{0},\hat{P},\gamma\rangle$, which shares the same state and action spaces as the original MDP but employs the learned transition dynamics $\hat{P}(s^{\prime}\,|\,s,a)$ and reward function $\hat{r}(s,a)$. $\hat{P}$ and $\hat{r}$ are typically estimated through supervised regression on the dataset $\mathcal{D}$, with additional incorporation of uncertainty penalties to discourage exploration of out-of-distribution (OOD) actions. These pessimistic models then serve as proxies for the real environment, enabling the simulation of transitions and subsequent use for planning.

The success of model-based RL algorithms hinges on the ability to learn an accurate model. Once the model is learned, various approaches, such as model predictive control (MPC) (Williams et al., 2017), dynamic programming (Munos & Szepesvári, 2008), or model-based policy optimization (Janner et al., 2019), can be employed to recover the optimal policy by solving the pessimistic MDP $\widehat{\mathcal{M}}$.

## 3 Our Method

### 3.1 Value Function Learning

We first learn an approximated value function of the behavior policy directly from the dataset. In the MDP $\mathcal{M}=(\mathcal{S},\mathcal{A},r,\rho_{0},P,\gamma)$, under the behavior policy $\mu(\cdot\,|\,s)$, the Bellman backup for obtaining the corresponding value function $V$ is defined as

$$ $\displaystyle\mathcal{T}^{\mu}V(s):=\mathbb{E}_{a\sim\mu(\cdot\,|\,s),s^{\prime}\sim P(\cdot\,|\,s,a)}\big[r(s,a)+\gamma V(s^{\prime})\big].$ $$

Therefore, from the offline dataset $\mathcal{D}$, we can define the following empirical Bellman operator:

$$ $\displaystyle\widehat{\mathcal{T}}^{\mu}_{d}V(s):=\begin{cases}\dfrac{1}{|\mathcal{D}_{s}|}\displaystyle\sum_{(s,a,r,s^{\prime})\in\mathcal{D}_{s}}\big[r+\gamma V(s^{\prime})\big],\ \text{if }|\mathcal{D}_{s}|>0,\\ 0,\qquad\qquad\qquad\qquad\qquad\qquad\ \text{otherwise},\end{cases}$ (2) $$

where $\mathcal{D}_{s}\subseteq\mathcal{D}$ is the set of transitions in the dataset $\mathcal{D}$ that begin in state $s$, $|\mathcal{D}_{s}|$ is the number of such transitions, and $\gamma$ is the discount factor.

We then show that the empirical Bellman operator has a unique fixed point.

###### Proposition 3.1 .

The empirical Bellman operator $\widehat{\mathcal{T}}^{\mu}_{d}$ has a unique fixed point $V_{d}^{\mu}(s)$ such that

$$ $\displaystyle\widehat{\mathcal{T}}_{d}^{\mu}V_{d}^{\mu}(s)=V_{d}^{\mu}(s).$ (3) $$

From [Proposition˜3.1](https://arxiv.org/html/2504.11944v2#S3.Thmtheorem1), we can learn a value function $V_{d}^{\mu}(s)$ from the offline dataset $\mathcal{D}$. Intuitively, when $\mathcal{D}$ is densely sampled from rollouts of the behavior policy $\mu$, the learned $V_{d}^{\mu}(s)$ provides an accurate approximation of the true value function $V^{\mu}(s)$, such that $V_{d}^{\mu}(s)\approx V^{\mu}(s)$.

We then train another approximated value function corresponding to the behavior policy with the help of a learned model. We learn an approximate dynamics model $P_{\theta}(s^{\prime},r\,|\,s,a)$ through maximum likelihood estimation and define the Bellman operator associated with $P_{\theta}$ as follows:

$$ $\displaystyle\widehat{\mathcal{T}}^{\mu}_{m}V(s):=\mathbb{E}_{a\sim\mu(\cdot\,|\,s),(r,s^{\prime})\sim P_{\theta}}\big[r+\gamma V(s^{\prime})\big].$ (4) $$

Note that if the learned model $P_{\theta}$ accurately approximates the true model ($P$ and $r$), it follows that $\widehat{\mathcal{T}}_{m}^{\mu}$ becomes equivalent to $\mathcal{T}^{\mu}$.

###### Proposition 3.2 .

The Bellman operator induced by $P_{\theta}$, denoted $\widehat{\mathcal{T}}^{\mu}_{m}$, has a unique fixed point $V_{m}^{\mu}(s)$ such that

$$ $\displaystyle\widehat{\mathcal{T}}_{m}^{\mu}V_{m}^{\mu}(s)=V_{m}^{\mu}(s).$ (5) $$

From [Proposition˜3.2](https://arxiv.org/html/2504.11944v2#S3.Thmtheorem2), we can iteratively apply $\widehat{\mathcal{T}}_{m}^{\mu}$ to obtain the unique value function $V_{m}^{\mu}$,
representing the value function learned from the model. This value function serves as another approximation of $V^{\mu}(s)$, such that $V_{m}^{\mu}(s)\approx V^{\mu}(s)$.

The core idea is that if the learned model $P_{\theta}$ is accurate, the value function $V_{m}^{\mu}(s)$—obtained from the model—should align with $V_{d}^{\mu}(s)$—derived from the dataset—as both aim to approximate the true value function $V^{\mu}(s)$. Therefore, we define the value function inconsistency loss as follows:

$$ $\mathcal{L}_{vic}(\theta)=\mathbb{E}_{s\sim\rho_{0}}\left[\big(V^{\mu}_{d}(s)-V^{\mu}_{m}(s)\big)^{2}\right],$ (6) $$

where $\rho_{0}$ dentoes the initial distribution, and $\mathcal{L}_{vic}$ depends on $\theta$ because $V_{m}^{\mu}(s)$ is derived from the learned model $P_{\theta}$.

To learn the model, we begin with minimizing the negative log-likelihood, which we refer to as the original model (learning) loss,

$$ $\displaystyle\mathcal{L}_{\text{ori}}(\theta)=-\mathbb{E}_{\mathcal{D}}\big[\log P_{\theta}(s^{\prime},r\mid s,a)\big],$ (7) $$

where $P_{\theta}$ is parameterized by a neural network that models the next state and reward as a Gaussian distribution conditioned on the current state and action:

$$ $\displaystyle P_{\theta}(s_{t+1},r_{t}\mid s_{t},a_{t})=\mathcal{N}(\mu_{\theta}(s_{t},a_{t}),\Sigma_{\theta}(s_{t},a_{t})).$ $$

We incorporate the value function inconsistency loss into the original model loss to construct the following augmented model loss:

$$ $\mathcal{L}_{aug}(\theta)=\mathcal{L}_{ori}(\theta)+\lambda\mathcal{L}_{vic}(\theta),$ (8) $$

where $\lambda>0$ is a user-chosen hyperparameter that balances the contributions of the two loss components. Thus, $\mathcal{L}_{aug}$ serves as the overall loss for training the dynamics model which aims to optimize both the original model loss and the value function inconsistency loss. Note that similar to (Yu et al., 2020), to improve robustness, we learn an ensemble of $N$ dynamics models, each with different initializations.

###### Proposition 3.1 .

###### Proposition 3.2 .

### 3.2 Model Gradient Theorem

To optimize $\mathcal{L}_{aug}$, it is essential to compute its gradient. However, in deriving the gradient of the augmented loss function, a significant challenge emerges from the implicit dependency of the model-learned value function $V_{m}^{\mu}(s)$ on the dynamics model parameters $\theta$. Unlike the original loss term $\mathcal{L}_{ori}$, whose gradient can be readily computed through automatic differentiation, the value function $V_{m}^{\mu}(s)$ is obtained via a recursive Bellman backup defined in Eq. ([4](https://arxiv.org/html/2504.11944v2#S3.E4)) that involves multiple steps of state transitions and discounted future rewards. This recursive structure embeds a hidden dependence on $\theta$, thereby precluding the direct application of conventional differentiation techniques.

In the following, we establish the analytical expression for the gradient of the augmented loss function, which forms the foundation for implementing VIPO. A key component of our approach is the recursive unrolling of gradients, allowing us to track the impact of model parameters across multiple transitions.

Let $\rho_{\theta}^{\mu}(s\rightarrow s^{\prime},t)$ denote the state density at $s^{\prime}$ after transitioning for $t$ time steps from state $s$ under the policy $\mu$ and the learned model $P_{\theta}$. Denote the (improperly) discounted state transition probability from $s$ to $s^{\prime}$ by $d_{\theta}^{\mu}(s,s^{\prime})$, defined as $d_{\theta}^{\mu}(s,s^{\prime})=\sum_{t=0}^{\infty}\gamma^{t}\rho_{\theta}^{\mu}(s\rightarrow s^{\prime},t)$.

We have the following Model Gradient Theorem.

###### Theorem 3.3 (Model Gradient Theorem) .

Let $\theta$ represent the parameters of the dynamics model $P_{\theta}$. Denote $V^{\mu}_{d}(\cdot)$ as the value function learned from the offline dataset $\mathcal{D}$, satisfying Eq. ([3](https://arxiv.org/html/2504.11944v2#S3.E3)), and $V^{\mu}_{m}(\cdot)$ as the value function learned from the dynamics model $P_{\theta}$, satisfying Eq. ([5](https://arxiv.org/html/2504.11944v2#S3.E5)). Then:

$$ $\displaystyle\nabla_{\theta}\mathcal{L}_{aug}(\theta)=\nabla_{\theta}\mathcal{L}_{ori}(\theta)$ $\displaystyle-2\lambda\mathbb{E}_{s\sim\rho_{0},s^{\prime}\sim d_{\theta}^{\mu},a^{\prime}\sim\mu,(r^{\prime},s^{\prime\prime})\sim P_{\theta}}\big[\big(V_{d}^{\mu}(s)$ $\displaystyle\ -V_{m}^{\mu}(s)\big)(r^{\prime}+\gamma V_{m}^{\mu}(s^{\prime\prime}))\nabla_{\theta}\log P_{\theta}(s^{\prime\prime},r^{\prime}|s^{\prime},a^{\prime})\big]$ $$

[Theorem˜3.3](https://arxiv.org/html/2504.11944v2#S3.Thmtheorem3) provides the gradient of the augmented model loss, which underlies the practical implementation of VIPO. The proof of Theorem [3.3](https://arxiv.org/html/2504.11944v2#S3.Thmtheorem3) is given in Appendix [B](https://arxiv.org/html/2504.11944v2#A2).

Figure: Algorithm 1 VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learning

###### Theorem 3.3 (Model Gradient Theorem) .

### 3.3 Practical Algorithm

We now introduce our practical algorithm, VIPO.

In VIPO, neural networks are employed to approximate the value function. To approximate $V^{\mu}_{d}(s)$, we learn $V_{d}(s)$ by minimizing the following empirical mean squared Bellman error (MSBE):

$$ $\displaystyle\mathcal{L}_{V_{d}}(\varphi_{d})=\mathbb{E}_{(s,a,r,s^{\prime})\sim\mathcal{D}}\left[\left(r+\gamma\bar{V}_{d}(s^{\prime})-V_{d}(s)\right)^{2}\right],$ (9) $$

where $\varphi_{d}$ is the parameter of primary state value network $V_{d}$ and $\bar{V}_{d}$ is the target state value network. $\bar{V}_{d}$ is obtained using an exponentially moving average of parameters of the state value network (soft update): $\bar{\varphi}_{d}\leftarrow\tau\varphi_{d}+(1-\tau)\bar{\varphi}_{d}$, where $\tau\in[0,1]$. Eq. ([9](https://arxiv.org/html/2504.11944v2#S3.E9)) serves as the surrogate objective for learning the Bellman backup defined in Eq. ([2](https://arxiv.org/html/2504.11944v2#S3.E2)).

We then learn $V_{m}(s)$ to approximate $V_{m}^{\mu}(s)$ by minimizing the following loss function:

$$ $\displaystyle\mathcal{L}_{V_{m}}(\varphi_{m})=\mathbb{E}_{(s,a)\sim\mathcal{D},{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}(r,s^{\prime})\sim P_{\theta}}}\left[\left(r+\gamma\bar{V}_{m}(s^{\prime})-V_{m}(s)\right)^{2}\right],$ (10) $$

where $\varphi_{m}$ is the parameter of the primary network $V_{m}$, and $\bar{V}_{m}$ is the target network obtained by the exponentially moving average of parameters of $V_{m}$. Note that $r$ and $s^{\prime}$ are sampled from the learned dynamic model $P_{\theta}$ instead of from the offline dataset. Eq. ([10](https://arxiv.org/html/2504.11944v2#S3.E10)) serves as the surrogate for learning the Bellman backup defined in Eq. ([4](https://arxiv.org/html/2504.11944v2#S3.E4)).

From [Theorem˜3.3](https://arxiv.org/html/2504.11944v2#S3.Thmtheorem3), computing the model gradient requires samples of the form $(s,a,r,s^{\prime},a^{\prime},r^{\prime},s^{\prime\prime})$. However, existing datasets provide only randomly sampled tuples $(s,a,r,s^{\prime})$. Nevertheless, all benchmark tasks (and typically most challenging real-world applications) are continuous-control problems with very small sampling intervals (typically 0.008s–0.015s in standard Gym environments). Consequently, the state change over a single step is numerically insignificant. In practice, therefore, we leverage the available samples $(s,a,r,s^{\prime})$ to closely approximate $(s^{\prime},a^{\prime},r^{\prime},s^{\prime\prime})$, and this approach proves highly effective. Then the gradient can be estimated as follows: $\nabla_{\theta}\log P_{\theta}$ can be obtained through automatic differentiation, $V^{\mu}_{d}$ is approximated by $V_{d}$, and $V^{\mu}_{m}$ is approximated by $V_{m}$. To enhance stability during the learning process, we replace $V_{d}$ and $V_{m}$ with the target value functions $\bar{V}_{d}$ and $\bar{V}_{m}$, resulting in the following surrogate gradient:

$$ $\displaystyle\nabla_{\theta}\mathcal{L}^{surr}_{aug}(\theta)=\nabla_{\theta}\mathcal{L}_{ori}(\theta)$ $\displaystyle-2\lambda\mathbb{E}_{(s,a)\sim\mathcal{D},(s^{\prime},r)\sim P_{\theta}}\big[(\bar{V}_{d}(s)$ (11) $\displaystyle-\bar{V}_{m}(s))(r+\gamma\bar{V}_{m}(s^{\prime}))\nabla_{\theta}\log P_{\theta}(s^{\prime},r|s,a)\big].$ $$

Remark. We next provide intuition for why the proposed surrogate gradient is effective by drawing an analogy to the policy gradient theorem (Sutton et al., 1999): $\nabla_{\theta}J(\theta)=\mathbb{E}_{s\sim\rho^{\pi_{\theta}},a\sim\pi_{\theta}}[Q^{\pi_{\theta}}(s,a)\nabla_{\theta}\log\pi_{\theta}(a|s)]$. In our surrogate gradient, the term $r+\gamma\bar{V}_{m}(s^{\prime})$ serves as an approximation of the Q-value corresponding to the learned model. Analogous to the policy gradient case, where it updates the policy $\pi_{\theta}$ to maximize $J(\theta)$, in our model gradient case, $(r+\gamma\bar{V}_{m}(s^{\prime}))\nabla_{\theta}\log P_{\theta}(s^{\prime},r|s,a)$ updates $P_{\theta}$ to improve the value function, where $\log P_{\theta}(s^{\prime},r|s,a)$ plays a role similar to $\log\pi_{\theta}(a|s)$ in the policy gradient theorem. Specifically, $(r+\gamma\bar{V}_{m}(s^{\prime}))\nabla_{\theta}\log P_{\theta}(s^{\prime},r|s,a)$ is the direction that increases $V_{m}$. Multiplying this term by $-2\lambda(\bar{V}_{d}(s)-\bar{V}_{m}(s))$ ensures that when $\bar{V}_{d}(s)>\bar{V}_{m}(s)$, the gradient descent update in our algorithm updates the model to increase $\bar{V}_{m}$ towards $\bar{V}_{d}$, and vice versa. Consequently, this surrogate gradient promotes consistency between value functions.

We now present the complete algorithm flow in Algorithm [1](https://arxiv.org/html/2504.11944v2#alg1), which outlines the dynamics model training process of VIPO. The process begins by computing the original model loss using Eq. ([7](https://arxiv.org/html/2504.11944v2#S3.E7)). Next, $V_{d}(s)$ and $V_{m}(s)$ are learned by minimizing Eq. ([9](https://arxiv.org/html/2504.11944v2#S3.E9)) and Eq. ([10](https://arxiv.org/html/2504.11944v2#S3.E10)), respectively. With these components in place, the model parameters $\theta$ are updated using Eq. ([11](https://arxiv.org/html/2504.11944v2#S3.E11)).

The key part of [Algorithm˜1](https://arxiv.org/html/2504.11944v2#alg1) is the training of the model, since
the success of model-based RL algorithms hinges on the ability to learn an accurate model. Once the model is learned, various approaches, such as model predictive control (MPC) (Williams et al., 2017), dynamic programming (Munos & Szepesvári, 2008), or model-based policy optimization (MBPO) (Janner et al., 2019), can be employed as the planner to recover the optimal policy by solving the surrogate MDP $\widehat{\mathcal{M}}$.

## 4 Experiments

In this section, we present three distinct experiments to empirically demonstrate, from three perspectives, that VIPO consistently learns a highly accurate model compared to previous methods.

Benchmark Results. In Subsection [4.1](https://arxiv.org/html/2504.11944v2#S4.SS1), we evaluate VIPO’s performance on the D4RL and NeoRL benchmarks. Specifically, we replace the models used in previous methods (MOPO and MOBILE) with the model trained by VIPO, while retaining their policy extraction components (i.e., the planner). This substitution results in significant performance improvements, enabling VIPO to achieve state-of-the-art performance across all tasks. We emphasize that the classic method MOPO and the pre-SOTA method MOBILE are selected merely as representative examples. In principle, VIPO can be integrated with any existing model-based offline RL algorithm, and such integration is expected to yield performance improvements

Revisiting Uncertainty. In Subsection [4.2](https://arxiv.org/html/2504.11944v2#S4.SS2), we design an experiment to reevaluate the uncertainty defined in MOPO (Yu et al., 2020). Specifically, we remove a portion of the Walker2d-medium-replay dataset and train models on the reduced dataset using both MOPO and VIPO. Intuitively, a higher drop ratio, indicating less available data about the environment, should result in higher model uncertainty. However, models trained with MOPO fail to capture this trend while models trained with VIPO exhibit a clear positive correlation.

Predictive Capability. In Subsection [4.3](https://arxiv.org/html/2504.11944v2#S4.SS3), we evaluate the predictive capability of the model trained with VIPO compared to a model trained using the original model loss, which does not account for value function inconsistency—the latter has been adopted by MOPO, MOReL, and MOBILE. The results on the D4RL benchmark indicate that VIPO achieves lower predictive error.

**Table 1: Normalized average returns on D4RL Gym tasks. The experiments are run on MuJoCo-“v2" datasets over 4 random seeds. r = random, m = medium, m-r = medium-replay, m-e = medium-expert. MOPO* indicates the results of MOPO retrained in “v2” datasets as reported by (Sun et al., 2023). The returns labeled with * in random tasks indicate values obtained from our training, as they were not reported in the original paper. We bold the highest mean.**
| Task Name | TD3+BC | CQL | IQL | DQL | MOPO* | MOReL | COMBO | RAMBO | MOBILE | VIPO-MOPO | VIPO(Ours) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| halfcheetah-r | 10.2 | 31.3 | 16.1* | 22.1* | 38.5 | 25.6 | 38.8 | 39.5 | 39.3 | 38.9 | $\bm{42.5\pm 0.2}$ |
| hopper-r | 11.0 | 5.3 | 10.8* | 17.5* | 31.7 | 53.6 | 17.9 | 25.4 | 31.9 | 30.2 | $\bm{33.4\pm 1.9}$ |
| walker2d-r | 1.4 | 5.4 | 7.7* | 14.1* | 7.4 | 37.3 | 7.0 | 0.0 | 17.9 | 9.5 | $\bm{20.0\pm 0.1}$ |
| halfcheetah-m | 42.8 | 46.9 | 47.4 | 51.1 | 73.0 | 42.1 | 54.2 | 77.9 | 74.6 | 76.3 | $\bm{80.0\pm 0.4}$ |
| hopper-m | 99.5 | 61.9 | 66.3 | 90.5 | 62.8 | 95.4 | 97.2 | 87.0 | 106.6 | 103.7 | $\bm{107.7\pm 1.0}$ |
| walker2d-m | 79.7 | 79.5 | 78.3 | 87.0 | 84.1 | 77.8 | 81.9 | 84.9 | 87.7 | 81.2 | $\bm{93.1\pm 1.8}$ |
| halfcheetah-m-r | 43.4 | 45.3 | 44.2 | 47.8 | 72.1 | 40.2 | 55.1 | 68.7 | 71.7 | 71.9 | $\bm{77.2\pm 0.4}$ |
| hopper-m-r | 31.4 | 86.3 | 94.7 | 101.3 | 103.5 | 93.6 | 89.5 | 99.5 | 103.9 | 103.8 | $\bm{109.6\pm 0.9}$ |
| walker2d-m-r | 25.2 | 76.8 | 73.9 | 95.5 | 85.6 | 49.8 | 56.0 | 89.2 | 89.9 | 80.5 | $\bm{98.4\pm 0.25}$ |
| halfcheetah-m-e | 97.9 | 95.0 | 86.7 | 96.8 | 90.8 | 53.3 | 90.0 | 95.4 | 108.2 | 103.4 | $\bm{110.0\pm 0.4}$ |
| hopper-m-e | 112.2 | 96.9 | 91.5 | 111.1 | 81.6 | 108.7 | 111.1 | 88.2 | 112.6 | 117.7 | $\bm{113.2\pm 0.1}$ |
| walker2d-m-e | 105.7 | 109.1 | 109.6 | 110.1 | 112.9 | 95.6 | 103.3 | 56.7 | 115.2 | 100.9 | $\bm{117.7\pm 1.0}$ |
| Average | 54.6 | 61.6 | 60.6 | 70.4 | 70.3 | 64.4 | 66.8 | 67.7 | 80.0 | 76.5 | 83.6 |

### 4.1 Benchmark Results

In this subsection, we evaluate the performance of VIPO on the D4RL and NeoRL benchmarks. Specifically, we demonstrate its effectiveness by replacing the model in previous methods with the one learned by VIPO while keeping the policy extraction component (i.e., the planner) unchanged. For example, in the case of MOPO, we substitute its model with the one learned by VIPO and use the same planner, resulting in the VIPO-MOPO algorithm listed in [Table˜1](https://arxiv.org/html/2504.11944v2#S4.T1). Compared to the retrained MOPO* results reported by (Sun et al., 2023) on the “v2” datasets (original results in (Yu et al., 2020) are trained on “v0” datasets and achieve an average score of 36.7), VIPO-MOPO achieves superior performance across all tasks.

For the VIPO algorithm results in [Table˜1](https://arxiv.org/html/2504.11944v2#S4.T1), we employ [Algorithm˜1](https://arxiv.org/html/2504.11944v2#alg1) with a planner same as MOBILE, which uses the soft actor-critic (SAC) (Haarnoja et al., 2018) framework to recover $\pi_{\rm out}$. Details of the planner are provided in [Algorithm˜2](https://arxiv.org/html/2504.11944v2#alg2) in the Appendix. Our findings indicate that VIPO outperforms previous results across all tasks, establishing a new SOTA. Therefore, the notable improvements observed with VIPO-MOPO over MOPO and VIPO over MOBILE provide empirical evidence that VIPO learns a more accurate model.

Additionally, VIPO’s performance on the NeoRL benchmark ([Table˜2](https://arxiv.org/html/2504.11944v2#S4.T2)) achieves SOTA on most tasks. These results underscore VIPO’s ability to learn a highly accurate model compared to prior methods.

To evaluate the performance of VIPO, we utilize several standard D4RL offline reinforcement learning benchmark tasks (Fu et al., 2020) from OpenAI Gym (Brockman, 2016) and the near-real-world NeoRL benchmark (Qin et al., 2022), both simulated using MuJoCo (Todorov et al., 2012). We defer the introduction of D4RL, NeoRL, and the compared baseline methods to [Section˜E.1](https://arxiv.org/html/2504.11944v2#A5.SS1).

#### 4.1.1 D4RL experiment results

The D4RL experimental results are shown in Table [1](https://arxiv.org/html/2504.11944v2#S4.T1). We report the normalized score for each task and the average performance over all tasks. These results are obtained during the final online evaluation conducted after training. Our results indicate that VIPO consistently outperforms all baseline methods across all tasks, achieving the highest average score compared to the other approaches. The MOPO results shown in Table [1](https://arxiv.org/html/2504.11944v2#S4.T1) are labeled as MOPO*, as reported by (Sun et al., 2023), based on the "v2" datasets, whereas the original paper reported results based on "v0" datasets. For additional information, please refer to Appendix C.3 in (Sun et al., 2023).

The VIPO-MOPO implementation utilizes the same planner as MOPO, replacing their model with the one learned by VIPO. Similarly, the VIPO implementation adopts the same planner as MOBILE, substituting their model with ours. Therefore, the notable improvements observed with VIPO-MOPO over MOPO and VIPO over MOBILE provide empirical evidence that the Value Inconsistency approach results in more effective models compared to those trained with the original model loss function. For a more detailed comparative analysis of the VIPO’s model learning performance, please refer to Section [4.3](https://arxiv.org/html/2504.11944v2#S4.SS3).

Table: Table 2: Normalized average returns on NeoRL tasks. The experiments are run on MuJoCo-“v3” datasets over 4 random seeds. L = low, M = medium, H = High. We bold the highest mean.

#### 4.1.2 NeoRL experiment results

In our evaluation, we selected six offline RL algorithms as baseline methods: BC, CQL, TD3+BC, EDAC, MOPO, and MOBILE (refer to Section [E.1](https://arxiv.org/html/2504.11944v2#A5.SS1)). The experimental results are summarized in Table [2](https://arxiv.org/html/2504.11944v2#S4.T2), where (L, M, H) denote three dataset types—low, medium, and high quality, respectively. Our proposed VIPO method demonstrates superior performance on most tasks. Notably, compared to the previous state-of-the-art method MOBILE, VIPO demonstrates significant performance improvements on specific tasks, including Walker2d-L, Hopper-L, and Hopper-H. This outstanding performance on the challenging NeoRL benchmark highlights the potential of our algorithm for real-world applications.

### 4.2 Revisiting Uncertainty

In this experiment, we demonstrate that the model uncertainty learned by MOPO is unreliable due to the inaccuracies in its model. In contrast, VIPO can successfully capture the expected level of uncertainties.

#### 4.2.1 Uncertainty Definition

In MOPO, the max-aleatoric uncertainty quantifier is utilized, with the model dynamics represented as $\hat{P}_{k}(s^{\prime}|s,a)=\mathcal{N}(\mu_{k}(s,a),\Sigma_{k}(s,a))$, where $\mu_{k}$ and $\Sigma_{k}$ denote the mean and covariance matrix of the multivariate Gaussian modeling the $k$-th transition dynamics in the ensemble. The uncertainty is defined as

$$ $\displaystyle U(s,a):=\max\limits_{k}\|\Sigma_{k}(s,a)\|_{\rm F},$ (12) $$

where $\|\cdot\|_{\rm F}$ denotes the Frobenius norm. As justified in MOPO, this uncertainty estimator captures both epistemic and aleatoric uncertainty of the true dynamics. To ensure a fair and consistent comparison, we adopt the same estimator to quantify the uncertainty of the models learned by both MOPO and VIPO.

#### 4.2.2 Experiment Setup

The experiment is conducted on the D4RL Walker2d-medium-replay dataset, which contains 301,698 transitions $(s,a,r,s^{\prime})$. From this dataset, 1,000 transitions are randomly selected. For each selected $(s,a)$ pair, the dataset is searched to identify all other $(s^{\prime},a^{\prime})$ pairs that fall within the range $(s\pm 0.8,a\pm 0.8)$. These additional points, together with the original 1,000 points, form a candidate dataset, resulting in a total of 49,952 points. This approach allows us to capture data points with similar (s, a) values, which likely share similar contexts in the MDP.

To evaluate the robustness of uncertainty measures, dropout ratios ranging from 0% to 100% are applied to the candidate dataset in 10% increments. For each dropout ratio, a subset of points from the candidate dataset is randomly removed, and the remaining points are merged back into the original dataset to create modified datasets. An ensemble of transition models is trained on each modified dataset by MOPO and VIPO, respectively, and the uncertainty in the regions corresponding to the perturbed $(s,a)$ pairs is evaluated separately. The uncertainty is calculated using Eq. ([12](https://arxiv.org/html/2504.11944v2#S4.E12)) and averaged across the initially selected 1000 state-action pairs.

Figure: Figure 2: Uncertainty of models trained by MOPO and VIPO, averaged over 4 random seeds.
Refer to caption: x1.png

#### 4.2.3 Experiment Results

Our experiment is based on the premise that decreasing the amount of data should lead to higher uncertainty in a well-learned model, because limited information about the environment naturally entails greater uncertainty. To test this, we progressively drop portions of the candidate dataset and train models with MOPO and VIPO under identical settings. We use Eq. ([12](https://arxiv.org/html/2504.11944v2#S4.E12)) to evaluate the uncertainties. As illustrated in [Figure˜2](https://arxiv.org/html/2504.11944v2#S4.F2), VIPO’s uncertainty grows consistently with the drop ratio, aligning with the expected trend, whereas MOPO’s uncertainty remains largely insensitive to data reduction. This result highlights VIPO’s potential to capture uncertainty more faithfully and to learn a more accurate model.

### 4.3 Predictive Capability

**Table 3: Model predictive error comparison on D4RL Gym tasks, averaged over 6 random seeds.**
| Task Name | OL Model Error | VIPO Model Error |
| --- | --- | --- |
| halfcheetah-r | 0.079 $\pm 0.008$ | 0.073 $\pm$ 0.002 |
| hopper-r | 0.0003 $\pm$ 1e-4 | 0.0003 $\pm$ 1e-4 |
| walker2d-r | 0.295 $\pm 0.024$ | 0.247 $\pm 0.013$ |
| halfcheetah-m | 0.550 $\pm 0.060$ | 0.193 $\pm$ 0.035 |
| hopper-m | 0.009 $\pm 0.002$ | 0.007 $\pm$ 3e-4 |
| walker2d-m | 0.287 $\pm 0.122$ | 0.270$\pm 0.085$ |
| halfcheetah-m-r | 0.396 $\pm 0.045$ | 0.289 $\pm$ 0.023 |
| hopper-m-r | 0.017 $\pm 0.006$ | 0.014 $\pm$ 0.002 |
| walker2d-m-r | 0.285 $\pm 0.020$ | 0.219 $\pm 0.005$ |
| halfcheetah-m-e | 0.081 $\pm 0.024$ | 0.070 $\pm$ 0.010 |
| hopper-m-e | 0.002 $\pm$ 6e-4 | 0.002 $\pm$ 4e-4 |
| walker2d-m-e | 0.077 $\pm 0.002$ | 0.070 $\pm 0.0026$ |
| Average | 0.173$\pm$ 0.026 | 0.121 $\pm$ 0.015 |

In this section, we train a model using only the original loss (referred to as OL Model in [Table˜3](https://arxiv.org/html/2504.11944v2#S4.T3)) defined in Eq. ([7](https://arxiv.org/html/2504.11944v2#S3.E7)), which aligns with the approach adopted by previous methods such as MOPO, MOReL, and MOBILE. Subsequently, we train a model with VIPO. Finally, we assess the predictive accuracy of the OL Model and VIPO Model by comparing the predicted reward and next state $(\hat{r},\hat{s}^{\prime})$ with the true values $(r,s^{\prime})$ for a given state-action pair $(s,a)$.

For all tasks, the dataset is split into training and validation sets in a 9:1 ratio. The model predictive error is measured as the mean squared error between the predicted and true values on the validation set. To ensure consistency, the same validation set is used for evaluating model performance across all tasks.

As summarized in Table [3](https://arxiv.org/html/2504.11944v2#S4.T3), our results demonstrate that the VIPO achieves lower predictive error than the OL Model on the majority of tasks. On average, VIPO reduces the model error from 0.173$\pm$ 0.026 (OL Model) to 0.121$\pm$ 0.015, reflecting a relative reduction of approximately 30%. This validates the effectiveness of incorporating the value function inconsistency loss into model training. These findings further confirm that VIPO can effectively learn a highly accurate model.

## 5 Conclusion

This paper proposes VIPO, a model-based offline RL algorithm that incorporates value function inconsistency into model training. Through extensive evaluations from multiple perspectives, we show that VIPO learns accurate models efficiently and consistently outperforms existing methods. As a general and readily applicable framework, VIPO can be integrated into model-based offline RL algorithms to improve model accuracy. Our work may encourage researchers to exploit data more comprehensively, rather than relying solely on costly architectures such as diffusion or transformers.

## References

- An et al. (2021)
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song.
Uncertainty-based offline reinforcement learning with diversified q-ensemble.
*Advances in neural information processing systems*, 34:7436–7447, 2021.
- Bai et al. (2022)
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang.
Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning.
*arXiv preprint arXiv:2202.11566*, 2022.
- Brockman (2016)
G Brockman.
Openai gym.
*arXiv preprint arXiv:1606.01540*, 2016.
- Chen et al. (2020)
Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross.
Bail: Best-action imitation learning for batch deep reinforcement learning.
*Advances in Neural Information Processing Systems*, 33:18353–18363, 2020.
- Chua et al. (2018)
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine.
Deep reinforcement learning in a handful of trials using probabilistic dynamics models.
*Advances in neural information processing systems*, 31, 2018.
- Fu et al. (2020)
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine.
D4rl: Datasets for deep data-driven reinforcement learning.
*arXiv preprint arXiv:2004.07219*, 2020.
- Fujimoto & Gu (2021)
Scott Fujimoto and Shixiang Shane Gu.
A minimalist approach to offline reinforcement learning.
*Advances in neural information processing systems*, 34:20132–20145, 2021.
- Fujimoto et al. (2018)
Scott Fujimoto, Herke Hoof, and David Meger.
Addressing function approximation error in actor-critic methods.
In *International conference on machine learning*, pp. 1587–1596. PMLR, 2018.
- Fujimoto et al. (2019)
Scott Fujimoto, David Meger, and Doina Precup.
Off-policy deep reinforcement learning without exploration.
In *International conference on machine learning*, pp. 2052–2062. PMLR, 2019.
- Haarnoja et al. (2018)
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.
Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.
In *International conference on machine learning*, pp. 1861–1870. PMLR, 2018.
- Janner et al. (2019)
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine.
When to trust your model: Model-based policy optimization.
*Advances in neural information processing systems*, 32, 2019.
- Kalashnikov et al. (2018)
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al.
Scalable deep reinforcement learning for vision-based robotic manipulation.
In *Conference on robot learning*, pp. 651–673. PMLR, 2018.
- Kidambi et al. (2020)
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims.
Morel: Model-based offline reinforcement learning.
*Advances in neural information processing systems*, 33:21810–21823, 2020.
- Kostrikov et al. (2021)
Ilya Kostrikov, Ashvin Nair, and Sergey Levine.
Offline reinforcement learning with implicit q-learning.
*arXiv preprint arXiv:2110.06169*, 2021.
- Kumar et al. (2019)
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine.
Stabilizing off-policy q-learning via bootstrapping error reduction.
*Advances in neural information processing systems*, 32, 2019.
- Kumar et al. (2020)
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine.
Conservative q-learning for offline reinforcement learning.
*Advances in Neural Information Processing Systems*, 33:1179–1191, 2020.
- Lange et al. (2012)
Sascha Lange, Thomas Gabel, and Martin Riedmiller.
Batch reinforcement learning.
In *Reinforcement learning: State-of-the-art*, pp. 45–73. Springer, 2012.
- Levine et al. (2020)
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.
Offline reinforcement learning: Tutorial, review, and perspectives on open problems.
*arXiv preprint arXiv:2005.01643*, 2020.
- Li et al. (2010)
Lihong Li, Wei Chu, John Langford, and Robert E Schapire.
A contextual-bandit approach to personalized news article recommendation.
In *Proceedings of the 19th international conference on World wide web*, pp. 661–670, 2010.
- Lu et al. (2021)
Cong Lu, Philip J Ball, Jack Parker-Holder, Michael A Osborne, and Stephen J Roberts.
Revisiting design choices in offline model-based reinforcement learning.
*arXiv preprint arXiv:2110.04135*, 2021.
- Munos & Szepesvári (2008)
Rémi Munos and Csaba Szepesvári.
Finite-time bounds for fitted value iteration.
*Journal of Machine Learning Research*, 9(5), 2008.
- Ovadia et al. (2019)
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek.
Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.
*Advances in neural information processing systems*, 32, 2019.
- Prudencio et al. (2023)
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini.
A survey on offline reinforcement learning: Taxonomy, review, and open problems.
*IEEE Transactions on Neural Networks and Learning Systems*, 2023.
- Qin et al. (2022)
Rong-Jun Qin, Xingyuan Zhang, Songyi Gao, Xiong-Hui Chen, Zewen Li, Weinan Zhang, and Yang Yu.
Neorl: A near real-world benchmark for offline reinforcement learning.
*Advances in Neural Information Processing Systems*, 35:24753–24765, 2022.
- Rafailov et al. (2021)
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn.
Offline reinforcement learning from images with latent space models.
In *Learning for dynamics and control*, pp. 1154–1168. PMLR, 2021.
- Rigter et al. (2022)
Marc Rigter, Bruno Lacerda, and Nick Hawes.
Rambo-rl: Robust adversarial model-based offline reinforcement learning.
*Advances in neural information processing systems*, 35:16082–16097, 2022.
- Singh et al. (2020)
Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar, and Sergey Levine.
Cog: Connecting new skills to past experience with offline reinforcement learning.
*arXiv preprint arXiv:2010.14500*, 2020.
- Sun et al. (2023)
Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu.
Model-bellman inconsistency for model-based offline reinforcement learning.
In *International Conference on Machine Learning*, pp. 33177–33194. PMLR, 2023.
- Sutton et al. (1999)
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour.
Policy gradient methods for reinforcement learning with function approximation.
*Advances in neural information processing systems*, 12, 1999.
- Todorov et al. (2012)
Emanuel Todorov, Tom Erez, and Yuval Tassa.
Mujoco: A physics engine for model-based control.
In *2012 IEEE/RSJ international conference on intelligent robots and systems*, pp. 5026–5033. IEEE, 2012.
- Wang et al. (2018)
Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al.
Exponentially weighted imitation learning for batched historical data.
*Advances in Neural Information Processing Systems*, 31, 2018.
- Wang et al. (2022)
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou.
Diffusion policies as an expressive policy class for offline reinforcement learning.
*arXiv preprint arXiv:2208.06193*, 2022.
- Williams et al. (2017)
Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M Rehg, Byron Boots, and Evangelos A Theodorou.
Information theoretic mpc for model-based reinforcement learning.
In *2017 IEEE international conference on robotics and automation (ICRA)*, pp. 1714–1721. IEEE, 2017.
- Wu et al. (2019)
Yifan Wu, George Tucker, and Ofir Nachum.
Behavior regularized offline reinforcement learning.
*arXiv preprint arXiv:1911.11361*, 2019.
- Yu et al. (2018)
Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, Trevor Darrell, et al.
Bdd100k: A diverse driving video database with scalable annotation tooling.
*arXiv preprint arXiv:1805.04687*, 2(5):6, 2018.
- Yu et al. (2020)
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma.
Mopo: Model-based offline policy optimization.
*Advances in Neural Information Processing Systems*, 33:14129–14142, 2020.
- Yu et al. (2021)
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn.
Combo: Conservative offline model-based policy optimization.
*Advances in neural information processing systems*, 34:28954–28967, 2021.

## Appendix A Related Work

Offline RL focuses on learning effective policies solely from a pre-collected behavior dataset and has demonstrated significant success in practical applications (Rafailov et al., 2021; Singh et al., 2020; Li et al., 2010; Yu et al., 2018). Similar to online RL, offline RL has been explored using both model-free and model-based algorithms, distinguished by whether or not they involve learning a dynamics model.

Model-free offline RL. Existing model-free methods on offline RL can be roughly categorized into the following two taxonomies: pessimistic value-based methods and regularized policy-based methods. Pessimistic value-based approaches achieve conservatism by incorporating penalty terms into the value optimization objective, discouraging the value function from being overly optimistic on out-of-distribution (OOD) actions. Specifically, CQL (Kumar et al., 2020) applies equal penalization to Q-values for all OOD samples, whereas EDAC (An et al., 2021) and PBRL (Bai et al., 2022) adjust the penalization based on the uncertainty level of the Q-value, measured using a neural network ensemble. In comparison, regularized policy-based approaches constrain the learned policy to stay close to the behavior policy, thereby avoiding OOD actions. For instance, BEAR (Kumar et al., 2019) constrains the optimized policy by minimizing the MMD distance to the behavior policy. BCQ (Fujimoto et al., 2019) restricts the action space to those present in the dataset by utilizing a learned Conditional-VAE (CVAE) behavior-cloning model. Alternatively, TD3+BC (Fujimoto & Gu, 2021) simply adds a behavioral cloning regularization term to the policy optimization objective and achieves excellent performance across various tasks. IQL (Kostrikov et al., 2021) adopts an advantage-weighted behavior cloning approach, learning Q-value functions directly from the dataset. Meanwhile, DQL (Wang et al., 2022) leverages diffusion policies as an expressive policy class to enhance behavior-cloning.

Model-based offline RL. We focus on Dyna-style model-based RL (Janner et al., 2019), which learns a dynamics model from the dataset and uses it to augment the dataset with synthetic samples. However, due to inevitable model errors, conservatism remains crucial to prevent the policy from overgeneralizing to regions where the dynamics model predictions are unreliable. For example, COMBO (Yu et al., 2021) extends CQL to a model-based setting by enforcing small Q-values for OOD samples generated by the dynamics model. RAMBO (Rigter et al., 2022) incorporates conservatism by adversarially training the dynamics model to minimize the value function while maintaining accurate transition predictions. Most model-based methods achieve conservatism through uncertainty quantification, penalizing rewards in regions with high uncertainty. Specifically, MOPO (Yu et al., 2020) uses the max-aleatoric uncertainty quantifier, MOReL (Kidambi et al., 2020) employs the max-pairwise-diff uncertainty quantifier, and MOBILE (Sun et al., 2023) leverages the Model-Bellman inconsistency uncertainty quantifier. In this work, we achieve conservatism by incorporating the value function inconsistency loss, enabling the training of a more reliable model.

## Appendix B Proof of the Model Gradient Theorem

###### Proof.

The gradient of the original loss can be obtained through automatic differentiation. Now we focus on the gradient derivation of the value consistency loss as defined in Eq. ([6](https://arxiv.org/html/2504.11944v2#S3.E6)).

$$ $\displaystyle\nabla_{\theta}\mathbb{E}_{s\sim\rho_{0}}\left[\big(V_{d}^{\mu}(s)-V_{m}^{\mu}(s)\big)^{2}\right]=-2\mathbb{E}_{s\sim\rho_{0}}\left[\big(V_{d}^{\mu}(s)-V_{m}^{\mu}(s)\big)\nabla_{\theta}V_{m}^{\mu}(s)\right].$ (13) $$

Then, we need to calculate the gradient of $V^{\mu}_{m}(s)$ in terms of $\theta$. To achieve this goal, we start by decomposing the value function using the Bellman equation, which is inspired by the previous work (Rigter et al., 2022). For the given state $s$:

$$ $\displaystyle V_{m}^{\mu}(s)$ $\displaystyle=\sum_{a}\mu(a\,|\,s)Q_{m}(s,a)$ (14) $\displaystyle=\sum_{a}\mu(a\,|\,s)\sum_{s^{\prime},r\sim P_{\theta}}\big(r+\gamma V_{m}^{\mu}(s^{\prime})\big)\cdot P_{\theta}(s^{\prime},r\,|\,s,a),$ $$

where $Q_{m}(s,a)$ is the state-action value function under dynamics model $P_{\theta}$ and the behavior policy $\mu$. Applying the product rule:

$$ $\displaystyle\nabla_{\theta}V_{m}^{\mu}(s)=$ $\displaystyle\sum_{a}\mu(a\,|\,s)\sum_{s^{\prime},r}\Big[\big(r+\gamma V_{m}^{\mu}(s^{\prime})\big)\cdot\nabla_{\theta}P_{\theta}(s^{\prime},r\,|\,s,a)$ (15) $\displaystyle+P_{\theta}(s^{\prime},r\,|\,s,a)\cdot\nabla_{\theta}\big(r+\gamma V_{m}^{\mu}(s^{\prime})\big)\Big]$ $\displaystyle=$ $\displaystyle\sum_{a}\mu(a\,|\,s)\sum_{s^{\prime},r}\big(r+\gamma V_{m}^{\mu}(s^{\prime})\big)\cdot\nabla_{\theta}P_{\theta}(s^{\prime},r\,|\,s,a)$ $\displaystyle+\gamma\sum_{a}\mu(a\,|\,s)\sum_{s^{\prime}}P_{\theta}(s^{\prime}\,|\,s,a)\cdot\nabla_{\theta}V_{m}^{\mu}(s^{\prime}).$ $$

Define $\psi(s)=\sum_{a}\mu(a|s)\sum_{s^{\prime},r}(r+\gamma V_{m}^{\mu}(s^{\prime}))\cdot\nabla_{\theta}P_{\theta}(s^{\prime},r|s,a)$, and additionally define $\rho_{\theta}^{\mu}(s\rightarrow x,n)$ as the transition probability under the policy $\mu$ from state $s$ to $x$ after $n$ steps in the learned MDP model $P_{\theta}$. Then, we can rewrite the above equation as:

$$ $\displaystyle\nabla_{\theta}V_{m}^{\mu}(s)$ $\displaystyle=\psi(s)+\gamma\sum_{s^{\prime}}\rho_{\theta}^{\mu}(s\rightarrow s^{\prime},1)\nabla_{\theta}V_{m}^{\mu}(s^{\prime})$ (16) $\displaystyle=\psi(s)+\gamma\sum_{s^{\prime}}\rho_{\theta}^{\mu}(s\rightarrow s^{\prime},1)\big[\psi(s^{\prime})+\gamma\sum_{s^{\prime\prime}}\rho_{\theta}^{\mu}(s^{\prime}\rightarrow s^{\prime\prime},1)\nabla_{\theta}V_{m}^{\mu}(s^{\prime\prime})\big]$ $\displaystyle=\psi(s)+\gamma\sum_{s^{\prime}}\rho_{\theta}^{\mu}(s\rightarrow s^{\prime},1)\psi(s^{\prime})+\gamma^{2}\sum_{s^{\prime\prime}}\rho_{\theta}^{\mu}(s\rightarrow s^{\prime\prime},2)\nabla_{\theta}V_{m}^{\mu}(s^{\prime\prime})$ $\displaystyle=\sum_{s^{\prime}}\sum_{t=0}^{\infty}\gamma^{t}\rho_{\theta}^{\mu}(s\rightarrow s^{\prime},t)\psi(s^{\prime}),$ $$

where the last equation of Eq. ([16](https://arxiv.org/html/2504.11944v2#A2.E16)) is obtained by continuing to unroll $\nabla_{\theta}V(\cdot)$.

Substituting Eq. ([16](https://arxiv.org/html/2504.11944v2#A2.E16)) into Eq. ([13](https://arxiv.org/html/2504.11944v2#A2.E13)), we have

$$ $\displaystyle\nabla_{\theta}\mathbb{E}_{s\sim\rho_{0}}$ $\displaystyle\left[\big(V_{d}^{\mu}(s)-V_{m}^{\mu}(s)\big)^{2}\right]$ (17) $\displaystyle=$ $\displaystyle-2\mathbb{E}_{s\sim\rho_{0}}\left[\big(V_{d}^{\mu}(s)-V_{m}^{\mu}(s)\big)\nabla_{\theta}V_{m}^{\mu}(s)\right]$ $\displaystyle=$ $\displaystyle-2\mathbb{E}_{s\sim\rho_{0}}\left[\big(V_{d}^{\mu}(s)-V_{m}^{\mu}(s)\big)\sum_{s^{\prime}}\sum_{t=0}^{\infty}\gamma^{t}\rho_{\theta}^{\mu}(s\rightarrow s^{\prime},t)\psi(s^{\prime})\right]$ $\displaystyle=$ $\displaystyle-2\sum_{s\in\mathcal{S}}\rho_{0}(s)\big(V_{d}^{\mu}(s)-V_{m}^{\mu}(s)\big)\sum_{s^{\prime}}\sum_{t=0}^{\infty}\gamma^{t}\rho_{\theta}^{\mu}(s\rightarrow s^{\prime},t)\psi(s^{\prime})$ $\displaystyle=$ $\displaystyle-2\sum_{s\in\mathcal{S}}\sum_{s^{\prime}\in\mathcal{S}}\rho_{0}(s)\big(V_{d}^{\mu}(s)-V_{m}^{\mu}(s)\big)d^{\mu}_{\theta}(s,s^{\prime})\psi(s^{\prime}),$ $$

where $d^{\mu}_{\theta}$ is the (improper) discounted state transition probability from $s$ to $s^{\prime}$. Note that from the definition of $\psi(s)$, we have

$$ $\displaystyle\psi(s)=$ $\displaystyle\ \sum_{a}\mu(a|s)\sum_{s^{\prime},r}(r+\gamma V_{m}^{\mu}(s^{\prime}))\cdot\nabla_{\theta}P_{\theta}(s^{\prime},r|s,a)$ $\displaystyle=$ $\displaystyle\ \sum_{a}\mu(a|s)\sum_{s^{\prime},r}(r+\gamma V_{m}^{\mu}(s^{\prime}))P_{\theta}(s^{\prime},r|s,a)\frac{\nabla_{\theta}P_{\theta}(s^{\prime},r|s,a)}{P_{\theta}(s^{\prime},r|s,a)}$ $\displaystyle=$ $\displaystyle\ \mathbb{E}_{a\sim\mu,(r,s^{\prime})\sim P_{\theta}}\big[(r+\gamma V_{m}^{\mu}(s^{\prime}))\nabla_{\theta}\log P_{\theta}(s^{\prime},r|s,a)\big].$ $$

Therefore, we have

$$ $\displaystyle\psi(s^{\prime})=\mathbb{E}_{a^{\prime}\sim\mu,(r^{\prime},s^{\prime\prime})\sim P_{\theta}}\big[(r^{\prime}+\gamma V_{m}^{\mu}(s^{\prime\prime}))\nabla_{\theta}\log P_{\theta}(s^{\prime\prime},r^{\prime}|s^{\prime},a^{\prime})\big]$ $$

Plugging $\psi(s^{\prime})$ into Eq. ([17](https://arxiv.org/html/2504.11944v2#A2.E17)), we obtain

$$ $\displaystyle\nabla_{\theta}\mathbb{E}_{s\sim\rho_{0}}\left[\big(V_{d}^{\mu}(s)-V_{m}^{\mu}(s)\big)^{2}\right]=$ $\displaystyle\ -2\sum_{s\in\mathcal{S}}\sum_{s^{\prime}\in\mathcal{S}}\rho_{0}(s)\big(V_{d}^{\mu}(s)-V_{m}^{\mu}(s)\big)d^{\mu}_{\theta}(s,s^{\prime})\psi(s^{\prime})$ $\displaystyle=$ $\displaystyle\ -2\mathbb{E}_{s\sim\rho_{0},s^{\prime}\sim d_{\theta}^{\mu},a^{\prime}\sim\mu,(r^{\prime},s^{\prime\prime})\sim P_{\theta}}\bigg[\big(V_{d}^{\mu}(s)-V_{m}^{\mu}(s)\big)\cdot$ $\displaystyle\ (r^{\prime}+\gamma V_{m}^{\mu}(s^{\prime\prime}))\nabla_{\theta}\log P_{\theta}(s^{\prime\prime},r^{\prime}|s^{\prime},a^{\prime})\bigg].$ $$

Hence, we finish the proof.
∎

###### Proof.

## Appendix C Proof of Propositions

Proof of [Proposition˜3.1](https://arxiv.org/html/2504.11944v2#S3.Thmtheorem1)

###### Proof.

We want to show there is exactly one function $V^{\ast}$ satisfying

$$ $\displaystyle V^{\ast}(s)\;=\;\widehat{T}^{\mu}_{d}V^{\ast}(s)$ $$

for any state $s$, which means $V^{\ast}$ is the unique fixed point of $\widehat{T}^{\mu}_{d}$.

In the following, We prove that $\widehat{T}^{\mu}_{d}$ is a $\gamma$-contraction with respect to the supremum norm. Let $\|V\|_{\infty}=\sup_{s}|V(s)|$ denote the sup norm of a value function $V$. We will show

$$ $\displaystyle\|\widehat{T}^{\mu}_{d}V-\widehat{T}^{\mu}_{d}W\|_{\infty}\;\leq\;\gamma\;\|V-W\|_{\infty}\quad\text{for all }V,W.$ $$

Fix any state $s$, we consider the difference $\widehat{T}^{\mu}_{d}V(s)-\widehat{T}^{\mu}_{d}W(s)$.

If $|\mathcal{D}_{s}|>0$, then

$$ $\displaystyle\widehat{T}^{\mu}_{d}V(s)\;=$ $\displaystyle\;\frac{1}{|\mathcal{D}_{s}|}\,\sum_{(s,a,r,s^{\prime})\in\mathcal{D}_{s}}\bigl[r+\gamma\,V(s^{\prime})\bigr],$ $\displaystyle\widehat{T}^{\mu}_{d}W(s)\;=$ $\displaystyle\;\frac{1}{|\mathcal{D}_{s}|}\,\sum_{(s,a,r,s^{\prime})\in\mathcal{D}_{s}}\bigl[r+\gamma\,W(s^{\prime})\bigr].$ $$

It follows that

$$ $\displaystyle\widehat{T}^{\mu}_{d}V(s)-\widehat{T}^{\mu}_{d}W(s)\;=\;\frac{\gamma}{|\mathcal{D}_{s}|}\sum_{(s,a,r,s^{\prime})\in\mathcal{D}_{s}}\bigl[V(s^{\prime})-W(s^{\prime})\bigr].$ $$

Therefore, we have

$$ $\displaystyle\bigl|\widehat{T}^{\mu}_{d}V(s)-\widehat{T}^{\mu}_{d}W(s)\bigr|\;$ $\displaystyle=\;\frac{\gamma}{|\mathcal{D}_{s}|}\;\Bigl|\sum_{(s,a,r,s^{\prime})\in\mathcal{D}_{s}}\bigl[V(s^{\prime})-W(s^{\prime})\bigr]\Bigr|\;$ $\displaystyle\leq\;\frac{\gamma}{|\mathcal{D}_{s}|}\sum_{(s,a,r,s^{\prime})\in\mathcal{D}_{s}}\bigl|V(s^{\prime})-W(s^{\prime})\bigr|.$ $$

Since $\bigl|V(s^{\prime})-W(s^{\prime})\bigr|\;\leq\;\|V-W\|_{\infty}$ for all $s^{\prime}$, it holds that

$$ $\displaystyle\bigl|\widehat{T}^{\mu}_{d}V(s)-\widehat{T}^{\mu}_{d}W(s)\bigr|\;\leq\;\frac{\gamma}{|\mathcal{D}_{s}|}\sum_{(s,a,r,s^{\prime})\in\mathcal{D}_{s}}\|V-W\|_{\infty}\;=\;\gamma\;\|V-W\|_{\infty}.$ $$

If $|\mathcal{D}_{s}|=0$, by definition, we have

$$ $\displaystyle\bigl|\widehat{T}^{\mu}_{d}V(s)-\widehat{T}^{\mu}_{d}W(s)\bigr|\;=\;0\;\leq\;\gamma\,\|V-W\|_{\infty}.$ $$

Overall, for any state $s$, we have

$$ $\displaystyle\bigl|\widehat{T}^{\mu}_{d}V(s)-\widehat{T}^{\mu}_{d}W(s)\bigr|\;\leq\;\gamma\,\|V-W\|_{\infty}.$ $$

Taking the supremum over $s$ yields

$$ $\displaystyle\|\widehat{T}^{\mu}_{d}V-\widehat{T}^{\mu}_{d}W\|_{\infty}\;=\;\sup_{s}\bigl|\widehat{T}^{\mu}_{d}V(s)-\widehat{T}^{\mu}_{d}W(s)\bigr|\;\leq\;\gamma\,\|V-W\|_{\infty}.$ $$

Hence $\widehat{T}^{\mu}_{d}$ is indeed a $\gamma$-contraction under the sup norm.

By the Banach Fixed Point Theorem, any $\gamma$-contraction ($0\leq\gamma<1$) on a complete normed vector space has a unique fixed point. Therefore, there exists a unique $V^{\ast}$ such that

$$ $\displaystyle V^{\ast}(s)\;=\;\widehat{T}^{\mu}_{d}V^{\ast}(s),\quad\forall s.$ $$

We denote $V^{\ast}(s)$ by $V_{d}^{\mu}(s)$ which completes our proof.
∎

Proof of [Proposition˜3.2](https://arxiv.org/html/2504.11944v2#S3.Thmtheorem2)

###### Proof.

This follows the same argument as the standard proof of the Bellman operator’s fixed-point uniqueness, substituting the true model $P$ with the learned model $P_{\theta}$.
∎

###### Proof.

###### Proof.

## Appendix D Planner details

Figure: Algorithm 2 Planner

To implement Algorithm [1](https://arxiv.org/html/2504.11944v2#alg1), we adopt the objective function defined in (Janner et al., 2019; Chua et al., 2018) as the original model loss. The specific form of the original model loss is expressed as:

$$ $\mathcal{L}_{ori}(\theta)=-\mathbb{E}_{\mathcal{D}}\left[\log P_{\theta}(s^{\prime},r|s,a)\right]$ (18) $$

For instance, we might define our model predictive model $P_{\theta}$ to produce a Gaussian distribution with diagonal covariances, parameterized by $\theta$ and conditioned on $s$ and $a$, i.e.: $P_{\theta}(s^{\prime},r|s,a)=\mathcal{N}(\bm{\mu}_{\theta}(s,a),\bm{\Sigma}(s,a))$. Then the original model loss has the following form:

$$ $\displaystyle\mathcal{L}^{G}_{ori}(\theta)=\sum_{n=1}^{N}[\bm{\mu}_{\theta}(s_{n},a_{n})-s_{n+1}]\bm{\Sigma}_{\theta}^{-1}(s_{n},a_{n})[\bm{\mu}_{\theta}(s_{n},a_{n})-s_{n+1}]+\log\det\bm{\Sigma}_{\theta}(s_{n},a_{n}).$ (19) $$

Table: Table 4: Hyperparameters of Policy Optimization in VIPO.

**Table 5: Fine-tuned hyperparameters of VIPO.**
| Domain Name | Task Name | $\lambda$ | $h$ | $\eta$ | $\beta$ |
| --- | --- | --- | --- | --- | --- |
| Gym | halfcheetah-random | 0.35 | 5 | 0.05 | 0.5 |
| hopper-random | 0.55 | 5 | 0.05 | 5.0 |  |
| walker2d-random | 0.3 | 5 | 0.5 | 0.5 |  |
| halfcheetah-medium | 0.35 | 5 | 0.05 | 0.5 |  |
| hopper-medium | 0.45 | 5 | 0.05 | 1.5 |  |
| walker2d-medium | 0.3 | 2 | 0.5 | 0.5 |  |
| halfcheetah-medium-replay | 0.35 | 5 | 0.05 | 0.1 |  |
| hopper-medium-replay | 0.8 | 5 | 0.05 | 0.1 |  |
| walker2d-medium-replay | 0.3 | 1 | 0.5 | 0.5 |  |
| halfcheetah-medium-expert | 0.35 | 5 | 0.05 | 1.0 |  |
| hopper-medium-expert | 0.55 | 5 | 0.05 | 5.0 |  |
| walker2d-medium-expert | 0.3 | 1 | 0.5 | 2.0 |  |
| NeoRL | HalfCheatah-L | 0.4 | 5 | 0.05 | 0.5 |
| Hopper-L | 0.03 | 5 | 0.5 | 2.5 |  |
| Walker2d-L | 0.35 | 1 | 0.5 | 2.5 |  |
| HalfCheatah-M | 0.4 | 5 | 0.05 | 0.5 |  |
| Hopper-M | 0.01 | 5 | 0.5 | 1.5 |  |
| Walker2d-M | 0.35 | 1 | 0.5 | 2.5 |  |
| HalfCheatah-H | 0.4 | 5 | 0.05 | 1.5 |  |
| Hopper-H | 0.03 | 5 | 0.5 | 2.5 |  |
| Walker2d-H | 0.35 | 1 | 0.5 | 2.5 |  |

We use Algorithm [2](https://arxiv.org/html/2504.11944v2#alg2) as a concrete implementation of the PLANNER in Algorithm [1](https://arxiv.org/html/2504.11944v2#alg1). [Algorithm˜2](https://arxiv.org/html/2504.11944v2#alg2) is adapted from Algorithm 1 in (Sun et al., 2023). Given a pre-trained environment model $P_{\theta}$ generated by [Algorithm˜1](https://arxiv.org/html/2504.11944v2#alg1), the agent simulates $h$-step rollouts starting from the state in $\mathcal{D}$ in the learned model $P_{\theta}$ and then stores these synthetic transitions to the replay buffer $\mathcal{D}_{m}$. For policy training, we incorporate the uncertainty quantification $\mathcal{U}(s,a)$ with the soft actor-critic (SAC) algorithm (Haarnoja et al., 2018). Specifically, we use the uncertainty quantification based on Bellman Inconsistency proposed in (Sun et al., 2023):

$$ $\displaystyle\mathcal{U}(s,a)$ $\displaystyle=\operatorname{Std}\left(P_{\theta}^{i}Q_{\psi}(s,a)\right)$ (20) $\displaystyle=\operatorname{Std}\left(\gamma\mathbb{E}_{\begin{subarray}{c}\{s_{j}^{\prime}\}\sim P_{\theta}^{i}\\ \{a_{j}^{\prime}\}\sim\pi\end{subarray}}\left[\min_{k=1,2}\bar{Q}_{\psi_{k}}(s_{j}^{\prime},a_{j}^{\prime})\right]\right),$ $$

where $\bar{Q}_{\psi_{k}}$ is the target state-action value network obtained by an exponentially moving average of parameters of the state-action value network $Q_{\psi_{k}}$. The objective function for critics is defined as:

$$ $\mathcal{L}_{critic}=\mathbb{E}_{(s,a,r,s^{\prime})\sim\mathcal{D}\cup\mathcal{D}_{m}}\left[\left(Q_{\psi_{k}}-y\right)^{2}\right],$ (21) $$

where the target value for $(s,a,r,s^{\prime})\in\mathcal{D}$ is

$$ $y=r+\gamma\left[\min_{k=1,2}\bar{Q}_{\psi_{k}}(s^{\prime},a^{\prime})-\alpha\log\pi_{\phi}(a^{\prime}|s^{\prime})\right],$ (22) $$

and the target for $(s,a,r,s^{\prime})\in\mathcal{D}_{m}$ is

$$ $y=r+\gamma\left[\min_{k=1,2}\bar{Q}_{\psi_{k}}(s^{\prime},a^{\prime})-\alpha\log\pi_{\phi}(a^{\prime}|s^{\prime})\right]-\beta\mathcal{U}(s,a).$ (23) $$

The policy is optimized by solving the following optimization problem:

$$ $\pi_{\phi}=\max_{\phi}\mathbb{E}_{\begin{subarray}{s}\sim\mathcal{D}\cup\mathcal{D}_{m}\\ a\sim\pi_{\phi}\end{subarray}}\left[\min_{k=1,2}Q_{\psi_{k}}(s,a)-\alpha\log\pi_{\phi}(a|s)\right]$ (24) $$

## Appendix E Experimental Details

### E.1 Experimental setups

D4RL.
The D4RL Mujoco tasks serve as a benchmark for evaluating offline RL algorithms in continuous control environments, such as Hopper, Walker2d, and HalfCheetah. These tasks employ datasets of varying quality, including random, expert, and mixed trajectories, to evaluate agents’ ability to derive effective policies from offline data. This setup provides a standardized framework for assessing offline RL performance in continuous control domains.

NeoRL.
NeoRL’s MuJoCo tasks provide offline RL benchmarks in environments such as HalfCheetah-v3, Walker2d-v3, and Hopper-v3. These tasks are based on conservative datasets generated from suboptimal policies, mimicking real-world scenarios characterized by limited and narrowly distributed data. This setup poses a challenge for algorithms to learn effective policies from constrained offline datasets, promoting the development of methods applicable to practical settings. Notably, NeoRL offers varying numbers of trajectories for training data across tasks: 100, 1,000, and 10,000. For our experiments, we uniformly selected 1,000 trajectories for each task.

Baselines.
In D4RL tasks, we evaluate VIPO against various offline RL algorithms, including model-free methods such as TD3+BC (Fujimoto & Gu, 2021), which trains a policy subject to a behavior cloning constraint. We also compare VIPO to CQL (Kumar et al., 2020), which calculates conservative Q-values for out-of-distribution (OOD) samples, and IQL (Kostrikov et al., 2021), which learns optimal policies offline through expectile regression on the dataset to address extrapolation errors. Additionally, we consider DQL (Wang et al., 2022), which utilizes diffusion models to effectively generate optimal actions from offline datasets. In the model-based category, we include MOPO (Yu et al., 2020), which incorporates uncertainty-aware dynamics models to penalize OOD state-action pairs; MOReL (Kidambi et al., 2020), which constructs uncertainty-aware, penalized MDPs to restrict policy learning to in-distribution states; RAMBO (Rigter et al., 2022), which uses adversarial models to handle distributional shifts in offline data; and MOBILE (Sun et al., 2023), which applies Bellman-inconsistency uncertainty quantification for robust offline policy learning.

For the NeoRL tasks, we compare VIPO against BC, CQL, TD3+BC, EDAC, MOPO, and MOBILE. BC (Behavioral Cloning) learns by directly imitating the data-generating policy through supervised learning. EDAC (An et al., 2021) addresses value overestimation by decoupling the target policy from the behavior policy.

### E.2 Benchmarks

We perform experiments on Gym tasks (v2 version) included in the D4RL (Fu et al., 2020) benchmark. In addition, we leverage the NeoRL benchmark, which offers a more challenging evaluation setting that closely resembles real-world scenarios, to provide a more comprehensive assessment of offline RL algorithms. NeoRL tasks are constructed using conservative datasets generated from suboptimal policies, reflecting real-world conditions characterized by limited and narrowly distributed data. This design poses substantial challenges for algorithms to learn effective policies from constrained offline datasets, thereby driving the development of approaches better suited for practical applications.

The following sections outline the sources of the reported performance on these benchmarks.

D4RL. (1) For MOPO^∗ (Yu et al., 2020), as the original paper reported results on "v0" datasets, we reference the experimental results provided in (Sun et al., 2023), which are based on the "v2" datasets. For CQL (Kumar et al., 2020), we report the scores obtained on the "v2" datasets using the codebase available at [github.com/yihaosun1124/OfflineRL-Kit](github.com/yihaosun1124/OfflineRL-Kit). (2) For TD3+BC (Fujimoto & Gu, 2021), MOReL (Kidambi et al., 2020), COMBO (Yu et al., 2021), RAMBO (Rigter et al., 2022), and MOBILE (Sun et al., 2023), we directly cite the performance results reported in their respective original papers, as these studies evaluated Gym tasks using the "v2" datasets (refer to Table [1](https://arxiv.org/html/2504.11944v2#S4.T1)). (3) For IQL (Kostrikov et al., 2021) and DQL (Wang et al., 2022), we conducted experiments on the "v2" random datasets using the codebase provided by the authors of the respective papers. For other "v2" datasets, the results are taken directly from their original publications.

NeoRL. The performance results for BC, CQL, and MOPO are sourced from the original NeoRL paper. For TD3+BC and EDAC, we report the scores retrained by the authors of (Sun et al., 2023).

Figure: Figure 3: The model prediction error on four walker2d datasets of the D4RL task: (a) walker2d-random-v2; (b) walker2d-medium-v2; (c) walker2d-medium-replay-v2; (d) walker2d-medium-expert-v2.
Refer to caption: walker2d-random-v2.png

### E.3 Hyperparameters

The hyperparameters selected for each task are detailed in Table [4](https://arxiv.org/html/2504.11944v2#A4.T4). We list the hyperparameters that we finetuned as follows. All experiments were conducted using NVIDIA RTX 4090 GPUs.

Balance Coefficient $\lambda$: For Gym tasks, the balance coefficient $\lambda$ is optimized by searching within the range $\{0.1,0.2,0.3,0.35,0.4,0.45,0.5,0.55,0.6,0.7,0.8,0.9\}$. For NeoRL tasks, the search range is restricted to $\{0.001,0.01,0.03,0.1,0.2,0.3,0.35,0.4,0.45,0.5,0.55,0.6,0.65\}$. Notably, in the hopper task, a smaller value of $\lambda$ has been found to result in better performance.

Rollout Length $h$: VIPO employs short-horizon rollouts, consistent with methods like MOPO and MOBILE. The rollout length $h$ is fine-tuned within the range $\{1,2,5,7\}$ for both Gym and NeoRL tasks.

Real Ratio $\eta$: The real ratio $\eta$ defines the proportion of data sampled from the offline dataset in each batch. Specifically, $\eta$ is tuned within the range $\{0.05,0.5\}$.

Penalty Coefficient $\beta$: For Gym tasks, $\beta$ is fine-tuned within the range $\{0.1,0.5,1.0,1.5,2.0,2.5\}$. For NeoRL tasks, the tuning range for $\beta$ is $\{0.5,1.0,1.5,2.0,2.5\}$.

The fine-tuned hyperparameters for each task are summarized in Table [5](https://arxiv.org/html/2504.11944v2#A4.T5).

## Appendix F Ablation Study

Figure: Figure 4: The policy training process on four walker2d datasets of the D4RLtask: (a) walker2d-random-v2; (b) walker2d-medium-v2; (c) walker2d-medium-replay-v2; (d) walker2d-medium-expert-v2.
Refer to caption: walker2d-random-v2-policy.png

In this section, we conduct ablation experiments to analyze the effect of the hyperparameter $\lambda$ in Eq. ([8](https://arxiv.org/html/2504.11944v2#S3.E8)) on four Walker2D datasets from the D4RL benchmark. Note that when training a model, the algorithm may stop at different stages, depending on the specific algorithm convergence judgment conditions. Specifically, in addition to limiting the maximum number of gradient updates, we also limit the number of invalid gradient updates, i.e., if the model prediction error is reduced by less than 1%, the gradient update is considered invalid. The experimental results, presented in Fig. [3](https://arxiv.org/html/2504.11944v2#A5.F3), indicate that $\lambda$ significantly influences the model dynamics learning process. Selecting an appropriate value of $\lambda$ leads to a model with reduced prediction error. Figure [4](https://arxiv.org/html/2504.11944v2#A6.F4) illustrates the policy training process using models trained with different values of $\lambda$. The results suggest that VIPO facilitates learning a policy that achieves a higher score.

## Appendix G More Experiments on Adroit Tasks

Table: Table 6: Normalized average returns on D4RL Adroit tasks, averaged over 4 random seeds.

The Adroit benchmark presents a set of high-dimensional manipulation tasks using a 24-DoF simulated robotic hand. The tasks include writing with a pen, hammering a nail, and opening a door, each requiring fine-grained control and coordination. We consider two types of datasets: “human”, which consists of 25 expert demonstrations collected from real human teleoperation, and “cloned”, a 50-50 mixture of these demonstrations and data generated by a cloned policy trained on them.

In Table [6](https://arxiv.org/html/2504.11944v2#A7.T6), we report the evaluation results of VIPO on the Adroit benchmark from D4RL. The inherent complexity of the Adroit benchmark arises from its high-dimensional observation and action spaces combined with relatively sparse demonstration data. VIPO addresses these challenges by incorporating a value-informed loss into the dynamics model training, encouraging consistency between the predicted values under the learned dynamics and the empirical returns from the dataset. This promotes more value-aligned dynamics without explicitly optimizing the policy.

The results illustrate that VIPO consistently outperforms several established methods, including model-free algorithms (BC, CQL, TD3+BC) and prominent model-based approaches (MOPO, MOBILE) across multiple benchmark tasks (see Table [6](https://arxiv.org/html/2504.11944v2#A7.T6)). Specifically, notable performance gains are observed in tasks such as pen-human (52.6 ± 7.7), pen-cloned (71.1 ± 9.5), and hammer-cloned (2.1 ± 0.2), underscoring VIPO’s capacity to manage the intricacies associated with high-dimensional manipulation scenarios.

## Appendix H Declaration

I declare that Large Language Models (LLMs) were used solely for language polishing in this paper. No other usage of LLMs was involved.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: Please refer to the "Weaknesses" section, I will raise my score if my concerns are addressed.
soundness: 3
strengths: 1. The idea of value function inconsistency as a self-supervised loss to enhance model accuracy in model-based offline RL is novel. The methodology and experimental setup are clearly presented, with an appendix and code provided for additional details.
2. The paper presents a well-defined theoretical framework for the gradient of the augmented loss function, contributing to a deeper understanding of the algorithm’s mechanics.
summary: The paper introduces VIPO, a novel model-based offline reinforcement learning (RL) algorithm designed to address the inherent challenges of offline RL, such as overestimation of values and model uncertainty. By incorporating value function inconsistency into the model training process, VIPO improves the model's accuracy and its ability to generalize from limited data. The paper presents empirical results across multiple benchmarks (D4RL, NeoRL), showing that VIPO consistently outperforms previous methods, demonstrating its efficacy in learning accurate models from offline datasets.
weaknesses: 1. My main concern lies in the insufficient experimental evaluation. The paper primarily evaluates the algorithm on simpler locomotion tasks, such as Walker and Hopper, which may not fully demonstrate its generalization capabilities. Test results on tasks that rely on visual input(V-D4RL), navigation tasks(Antmaze) or more complex manipulation tasks, such as Fetch environment, would significantly improve the robustness and generalization of the reported findings. 

2. Additionally, the paper lacks ablation studies, such as evaluating the impact of the number of ensemble models (N) or removing the value inconsistency penalty, which are essential for understanding the contribution of each component to the overall performance of the algorithm.

3. While the paper highlights the improved performance of VIPO, it lacks a detailed analysis of the computational cost and training time required by VIPO compared to existing algorithms. Including this information would provide a more balanced perspective on the algorithm’s practical utility, particularly for real-world applications.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 2
questions: Since the main benefit is improved model accuracy over previous conservatism strategies, how does the predictive capability of the VIPO model (Table 3) compare directly against models trained using other prominent model-based conservatism strategies, such as the maximum pairwise difference or the adversarial approach? 


The core VIPO method uses the MOBILE planner (Algorithm 2). Could the authors include an ablation study that swaps the planner used in Algorithm 2 for a simpler, less aggressive one (e.g., a standard SAC planner without the uncertainty penalty $\beta\mathcal{U}(s,a)$ term) to isolate how much of the performance gain is attributable solely to the $\mathcal{L}_{vic}$-trained model versus the aggressive mobile-like policy optimization loop?
soundness: 3
strengths: 1. The central idea is the innovative dual-usage of offline data to generate a self-supervised signal for model training. This contrasts fundamentally with previous methods that used the data only once to learn the model ensemble.

2. The derivation of the Model Gradient Theorem (Theorem 3.3) is a significant technical achievement, providing the analytical expression needed to compute the gradient of the complex augmented loss.
summary: This paper introduces VIPO (Value Function Inconsistency Penalized Offline Reinforcement Learning), a novel model-based offline RL algorithm designed to significantly improve the accuracy and reliability of the learned dynamics model. The core motivation is that previous model-based methods rely on heuristic uncertainty estimation to enforce conservatism, which is often unreliable in practice.
weaknesses: 1. The calculation of the surrogate gradient (Eq. 11) relies on the practical assumption that the short sampling interval in continuous-control problems makes the state change over a single step numerically insignificant. This allows approximating $(s', a', r', s'')$ using the available single-step samples $(s, a, r, s')$. This is a heuristic approximation that introduces an unquantified error, and its effectiveness may degrade in environments with high-frequency dynamics or more complex state transitions.

2. The benchmark results show that VIPO outperforms COMBO and RAMBO, but the critical experiments demonstrating uncertainty reliability (Figure 2) and predictive capability (Table 3) only compare VIPO to the Original Loss (OL) Model (which is MOPO/MOREL/MOBILE's model objective). Including a direct comparison of the predictive capability against models trained using other uncertainty-based model training methods would have provided a more comprehensive validation.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. In Table 3, measuring model error only for a rollout length of 1 is somewhat limited. Since model rollouts are typically performed for horizons of 5 or more, comparisons over longer horizons would provide a more comprehensive evaluation.

2. How would the performance change if a different type of model uncertainty, other than the one used in MOPO, were employed? It would be interesting to see whether VIPO remains effective under alternative uncertainty estimation schemes.
soundness: 3
strengths: 1. VIPO introduces a novel perspective on value function inconsistency, which has been largely overlooked in previous model-based offline RL works.

2. The paper is clearly structured, allowing readers to follow the overall flow and reasoning easily.

3. When comparing several algorithms, VIPO achieves SOTA performance.
summary: This work proposes VIPO, a model-based offline reinforcement learning algorithm that incorporates value function inconsistency into the training loss of the dynamics model. The key idea is to update the model by minimizing the discrepancy between the value function learned from offline data and the value estimated from the learned dynamics. The authors further derive how to compute the model gradients with respect to the parameters of the proposed loss function. In the evaluation, VIPO outperforms several SOTA model-based offline RL algorithms on the D4RL MuJoCo and NeoRL benchmarks.
weaknesses: 1. As I understand it, MOBILE quantifies uncertainty through the inconsistency of Bellman estimations under an ensemble of learned dynamics models. VIPO leverages value function inconsistency as a self-supervised learning signal that directly guides the model training. Considering that these two algorithms are based on different conceptual perspectives, it may not be appropriate to present the combined performance of VIPO and MOBILE as the representative result of VIPO in the evaluation.

2. In fact, when comparing the performance between MOPO and VIPO-MOPO, there are several offline datasets (e.g., hopper-r, walker2d-m, halfcheetah-m-r, walker2d-m-r, etc.) where MOPO outperforms VIPO-MOPO. This raises uncertainty about whether the performance improvement of VIPO is truly significant or consistent across tasks.

3. In Figure 2, the uncertainty scales for MOPO and VIPO are presented on separate axes, making direct comparison between the two methods difficult.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: - Could you conduct additional experiments in more complex domains to verify that the empirical gains do not solely stem from the practical designs? Current tasks often contain narrow bounds in state or action spaces, which aligns with heuristics that numerical differences are trivial. For instance, the tabletop robotic manipulation benchmark [1] provides tasks with a wide-ranging state space (e.g., distance in Cartesian space from a hand to an object).
- Are there alternative ways to approximate the augmented model learning loss instead of replicating the same tuple for the next time-step tuple, which can be further extended to domains outside of locomotion tasks? I believe suggesting more comprehensive directions for implementing the model gradient theorem would significantly improve the novelty of the paper.
- What is the limitation of VIPO? Discussions on the potential drawbacks of the proposed method should be addressed.

[1] Mandlekar, Ajay, et al. "What matters in learning from offline human demonstrations for robot manipulation." arXiv 2021.

Minor problems
- L122: Maybe a typo? $P(s'|s,a) : \mathcal{S} \times \mathcal{A} \rightarrow \mathcal{S}$
- L170: What is the meaning of "densely sampled"? Does this sentence stand for a nearly full-coverage offline dataset?
- Table 1: Bold faces do not denote the best scores. (MOReL is the best in *hopper-r* and *walker2d-r*, VIPO-MOPO is the best in *hopper-m-e*)
soundness: 3
strengths: - The authors provide comprehensive explanations on connecting theoretical foundations and practical implementations.
- The authors rigorously provide supplementary proofs for theorems on value learning and the model gradient theorem.
- Experiments on D4RL show improved performance across model-free and model-based baselines.
- Experiments on NeoRL demonstrate an interesting perspective of VIPO when the given data is limited.
summary: This paper presents a novel offline model-based reinforcement learning method called VIPO that incorporates value inconsistency loss in the offline model learning, where two distinct values are learned from the same offline dataset by introducing dual usage of the offline data. Authors contend that the uncertainty estimation of the conventional offline model-based methods is hard to rely on when complex offline data is given to the agent. They further suggest a two-way value learning approach: learning a value solely learned from the offline data and another value function learned with the empirical transition dynamics model, supported by theoretical analysis and empirical observation. They conduct an empirical study with model-free and model-based offline RL baselines on the D4RL and NeoRL benchmarks. VIPO achieves superior performance across benchmarks, highlighting the lower model prediction error and higher empirical performance.
weaknesses: - The novelty of VIPO is limited. The main contribution relies on reducing uncertainty estimations to learn an accurate model by introducing an auxiliary value inconsistency loss into the original model learning. However, there is no guarantee that the learned model predicts reliable synthetic rollouts under out-of-support data in the current methodology. Besides, the value inconsistency loss largely depends on the assumption that the learned values precisely estimate ($V^\mu_d (s)$ and $V^\mu_m (s)$) approximate the true value(L171, L188), whereas the learned value function fails to predict accurate values when the given $(s,a)$ pair lies outside of the offline data in general [1]. Based on my conjecture, employing a surrogate objective with heuristics in Section 3.3 contributes to the empirical performance gains to some extent, while those observations benefit the task-specific problem formulation- MuJoCo locomotion tasks often exhibit tightly bounded states and actions.
- Experimental results present conflicting views from the perspective of the authors. In Table 1, MOPO outperforms VIPO-MOPO in five out of twelve cases when comparing the effect of planners, which contrasts with the authors' argument that VIPO-MOPO outperforms MOPO across all tasks in L353. Additionally, the y-axis ticks of Figure 2 are misaligned across baselines. While it is notable that VIPO demonstrates increasing uncertainty when the drop ratio increases, the absolute difference is marginal when the y-axis is scaled with each other.

[1] Levine, Sergey, et al. "Offline reinforcement learning: Tutorial, review, and perspectives on open problems." arXiv 2020.


Decision: Reject