The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning
Authors: Anya Sims
Cong Lu
Jakob Nicolaus Foerster
Yee Whye Teh
Venue: NeurIPS 2024 poster (2024)
Keywords: Offline Reinforcement Learning
Model-Based Reinforcement Learning
Primary area: reinforcement_learning
TL;DR: Existing offline model-based methods fail under the true dynamics. This reveals the previously-unnoticed “edge-of-reach” problem, and leads us to propose RAVL - a new method that continues to work even in the absence of model uncertainty.

Abstract: Offline reinforcement learning (RL) aims to train agents from pre-collected datasets. However, this comes with the added challenge of estimating the value of behaviors not covered in the dataset. Model-based methods offer a potential solution by training an approximate dynamics model, which then allows collection of additional synthetic data via rollouts in this model. The prevailing theory treats this approach as online RL in an approximate dynamics model, and any remaining performance gap is therefore understood as being due to dynamics model errors. In this paper, we analyze this assumption and investigate how popular algorithms perform as the learned dynamics model is improved. In contrast to both intuition and theory, if the learned dynamics model is replaced by the true error-free dynamics, existing model-based methods completely fail. This reveals a key oversight: The theoretical foundations assume sampling of full horizon rollouts in the learned dynamics model; however, in practice, the number of model-rollout steps is aggressively reduced to prevent accumulating errors. We show that this truncation of rollouts results in a set of edge-of-reach states at which we are effectively "bootstrapping from the void." This triggers pathological value overestimation and complete performance collapse. We term this the edge-of-reach problem. Based on this new insight, we fill important gaps in existing theory, and reveal how prior model-based methods are primarily addressing the edge-of-reach problem, rather than model-inaccuracy as claimed. Finally, we propose Reach-Aware Value Learning (RAVL), a simple and robust method that directly addresses the edge-of-reach problem and hence - unlike existing methods - does not fail as the dynamics model is improved. Since world models will inevitably improve, we believe this is a key step towards future-proofing offline RL.

The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning
Anya Sims University of Oxford anya. mnya.simssstass. ox. ac. uk
Cong Lu University of Oxford
Jakob N. Foerster FLAIR, University of Oxford
Yee Whye Teh University of Oxford
Abstract
Offline reinforcement learning (RL) aims to train agents from pre-collected datasets. However, this comes with the added challenge of estimating the value of behaviors not covered in the dataset. Model-based methods offer a potential solution by training an approximate dynamies model, which then allows collection of additional synthetic data via rollouts in this model. The prevailing theory treats this approach online RL in an approximate dynamics model, and any remaining performance gap is therefore understood as being due to dynamics model errors. In this paper, we analyze this assumption and investigate how popular algorithms perform the learned dynamics model is improved. In contrast to both intuition and theory, if the learned dynamics model replaced by the true error-free dynamics, existing baadel-based methods completely fail. This reveals key oversight: The theoretical foundations assume sampling of full horizon rollouts in the learned dynamics model; however, in practice, the number of model-rollout steps is aggressively reduced prevent accumulating errors. We show that this truncation of rollouts results in set of edge-of ofgechf-reach states at which we are effectively "bootstrapping from the void." " This triggers pathological value overestimation and complete performance collapse. We term this the edge-of-reach problem. Based on this new insight, we important gaps existing theory, and reveal how prior model-based methods are primarily addressing the edge-of-reach problem, rather than model- inaccuracy as claimed. Finally, we propose Reach-Aware Value Learning (RAVL), simple and robust method that directly addresses the edge-of-reach problem and hence- unlike existing methods does not fail as the dynamics model improved. Since world models will inevitably improve, we believe this is key step towards future-proofing offline RL
Introduction
Standard online reinforcement learning (RL) requires collecting large amounts of on on-policy data. This can be both costly and unsafe, and hence represents a significant barrier against applying RL robotics [2] 4 20], and also against scaling RL to more complex problems. Offline RI. 6 23 aims to remove this need for online data collection by enabling agents to be trained on pre-collected datasets. One hope (21) is that it may facilitate advances in RL similar to those driven by the use of large pre-existing datasets in supervised learning [3]
in domains such as healthcare [28] 32
OOr code is open-sourced
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Accuracy of dynamics model: Interpolating from random to learned to perfect
-02
BG
0.2
04
0.6
0.8
1.0
DECREASING ACCURACY
LEARNED DYNAMICS MODEL
INCRIASSIGG DYNAMICS
TRUE
MODBK ACEURACCY
DYNAMICS
DARL normalized score
20-
Caperi per termarca
KEVL (ours)
EXISTING METHOSS BALLA!
8
Dynamics pe penally (MCPP)
(Our)
Figure I: Existing offline model-based RL methods fail if the accuracy of the dynamics model is increased (with all else kept the same). Results shown are for MOPO [36]. but note that this failure indicates the failure of all existing uncertainty-basse 'uncertainty' is zero. By contrast, method is much more robust to changes in dynamics model. The axis shows linearly interpolating next states and rewards the learned model with the true model (center-tright) and random model (center->lef)) with results on the D4RI. W2d-medexp benchmark (min/max over seeds). The full set of results and experimental setup are provided in Table [and Appendix C.22 respectively.
methods since each of their specific penalty terms disappear under the true dynamics
The central challenge in offline RI. is estimating the value of actions not present in the dataset, known as the out-of-sample action problem [17] P A naive approach results in extreme value overestimation due to bootstrapping using inaccurate values at out-of-sample state-actions [18]. There have been many proposals to resolve this, with methods largely falling into one of two categories: model-
free 8 10
19 or model-based [14 24 29 36).
Model-free methods typically address the out-of-sample action problem by applying form of conservatism or constraint to avoid using out-of sut-le--ample actions in the Bellman update. In contrast, the solution proposed by model-based methods is to allow the collection of additional data at any previously out-of-sample actions. This is done by first training an approximate dynamics model on the offline dataset [13 30), and then allowing the agent collect additional synthetic data in this model via k-step rollouts (see Algorithm[] [ The prevailing understanding is that this can be viewed as online RL in an approximate dynamics model, with the instruction being then simply "run any RL algorithm on M until convergence" (36), where M the learned dynamics model with some form of dynamics uncertainty penalty. Existing methods propose various forms of dynamics penalties (see Table 7). based on the assumption that the remaining performance gap compared to online RL is solely due to inaccuracies in the learned dynamics model.
This understanding naturally implies that improving the dynamics model should also improve performance. Surprisingly, we find that existing offline model-based methods completely fail if the learned dynamics model is replaced with the true, error-free dynamics model, while keeping everything else the same (see Figure D. Under the true dynamics, the only difference to online RL is that in online RL, data is sampled as full-length episodes, while offline model-based RL, data is instead sampled as k-step rollouts, starting from a state in the original offline dataset, with rollout length & limited to avoid accumulating model errors. Failure under the true model therefore highlights consequences.
that truncating rollouts has critical and previously-overloon
We find that this rollout truncation leads to a set of states which, under any policy, can only be reached in the final rollout step (see red in Figure 2). The existence of these edge-of-reach states is problematic as it means Bellman updates (see Equation (Th) use target values that are never themselves trained. This is illustrated in Figure 2 and described in detail in Section 3 This effective "bootstrapping from the void" triggers a catastrophic breakdown in Q-learning Concisely, this issue can be viewed as all actions from edge-of-reach states remaining out-of-reach over training. Hence, contrary to common understanding. the out-of-sample action problem central to model- free methods is not fully resolved by model- model-based approach. In fact, in Section 3.4 we provide detailed analysis suggesting that this is the predominant source of issues on the standard D4RL benchmark. In Section 66.5 we consequently reexamine how existing methods work and find they are indirectly and unintentionally addressing this issue (rather than model emrors as claimed).
This is also referred the out-of -iittr-distiiittion action
18 action distribution shift problem 19

---

Algorithm 1: Pseudocode for the base procedure used in offiine model-based methods. In sunmarry (1) Train dynamics model M Datow, then (2) Train an agent (often SAC) with k-step rillouts the learned model starting from & E Pajou. Existing methods consider issues to be due to errors M and hence introduce dynamics uncertainty penalties (see Table 7. We find the predominant source of issues to be the edge-of-reach problem, and hence instead propose using value pessimism (RAVL) (see Section
Algorithm 1 Base model-based algorithm (MBPO) Additions in existing methods and RAVI (ours)
1: Require: Offline dataset Domtice
2: Require: Dynamics model M = (T. R) trained on Domine Augment M with an uncertainty penalty*
3: Specify: Rollout length & 2 I, real data ratio E [0,1]
4: Initialize: Replay buffer Pmitiou 0. policy ne. value function Qu (both from random)
5: for epochs I.... do
6: (Colleet data) Starting from states in Dotline. collect k-step rollouts in M with we. Store data Drelious
7: (Train agent) Train The and Qo on Drolious U Dottine (mixed with ratio r) Add Q-value pessimism
8: end for
"This uncertainty penalty collapses zero with the true error-free dynomics (M- M) (see experiments Table 2).
We have the following problem: Indirectly addressing the edge-of saae-of-rrccc problem means existing model-based methods have a fragile and unforeseen dependence on model quality and fail catastroph- ically as dynamics models improve. Since model improvements are inevitable, and since tuning of hyperparameters is particularly impractical in offlin RI., this is a substantial practical barrier.
In light of this, we propose Reach-Aware Value Learning (RAVL), a simple and robust method that directly addresses the edge-of-reach problem. RAVL achieves strong performance on standard benchmarks, and moreover continues to perform even with error-free, uncertainty-free dynamics models (Section 6). Our model-based method bears close connections to model-free approaches, and hence in Appendix
we present unified perspective of these two previously disjoint subfields.
2 Background
Reinforcement Learning. We consider the standard RL. framework 31. in which the environment is formulated as Markov Decision Process, M = (S, T, R, no: 7): where S and A denote the state and action spaces, T(s'|s |s a) and R(s, a) denote the transition and reward dynamics, Ho the initial state distribution, and (0, 1) is the discount factor. The goal in reinforcement learning is to learn a policy π (a(s)) that maximizes the expected discounted return =T 1- 7 R(st, dt)].
Actor- Critic Algorithms. The broad class of algorithms we consider are actor-critic [5] methods which jointly optimize policy π and state-action value function (Q-function). Given a dataset D of (state, action, reward, nextstate) transitions, actor-critic algorithms iterate between two steps:
1. (Policy evaluation) Update Qs to fit the current policy using the following Bellman update:
Key expression:
Q5(5,0) a)
+ + (o)s))
(s,a, D
(1)
Update value (s.
Using value (6', (e)))
2. (Policy improvement) Update Ta to increase the current Q-value predictions: Qo(s, TT(s)
Offline RL and the Out-of-Sample Action Problem. In offline RI., training must rely only fixed dataset of transitions Dumtnee {(s, a, r, s')( (1)),. collected by some policy T The central problem in this offline setting is the out-of-sample action problem: The values of Q& being updated are at any state-actions that appear as (s, a) in Dottine- However, the targets of these updates rely on values (s', a²) where a' ~ no (see Equation (1). Since a' ~ mo (on-policy) whereas ~ n² (off- policy), (s', a') may not appear in Domlne. and hence may itself have never been updated. This means that updates can involve bootstrapping from 'out-of- 'out-of-sample" and hence arbitrarily misestimated values and thus lead to misestimation being propagated over the entire state-action space. The max operation (implicit in the policy improvement step) further acts to exploit any misestimation, converting misestimation into extreme pathological overestimation. As result Q-values often tend to increase unboundedly over training, while performance collapses entirely [8] 23
Thiis step contains an implicit max operation which we refer to Section 3.2 'We use the terms sfa-lee- and distri--isttibuttio' interchangeably, as is done in the literature.
3

---

Model-Based Offline RL. Model-based methods [30 aim to solve the oft-o--sample issue by allowing the agent to collect additional synthetic data in a learned dynamics model. They generally share the same base procedure as descrihed in Algorithm This involves first training approximate dynamics model M (S, T, R. Mo; 7) on Dottine- Here, [ss'ls, a) and R(s, a) denote the learned transition and reward functions, commonly realized as a deep ensemble [5 122. Following this an agent is trained using an online RL algorithm (typically SAC (II). for which data is sampled as k-step trajectories (termed rollouts) under the current policy, starting from states in the offline dataset Dottine- The base procedure of training a SAC agent with model rollouts does not work out of the box. Existing methods attribute this as due to dynamics model errors. Consequently, a broad class of methods propose augmenting the learned dynamics model M with some form of dynamics uncertainty penalty, often based on variance over the ensemble dynamics model [2 14] 24 29 36]. We present some explicit examples in Table 7 Crucially, all of these methods assume that, under the true error-free model, no intervention should be needed, and hence all the uncertainty penalties collapse to zero. In the following section, show that this assumption leads catastrophic failure.
Edge-of-reach states:
Dataset of transitions:
and ir umitied for diarity)
k k-step rollouts, (k+ Rollout 1 Rollost 2
States sampled as 5p
Edgo sagg--f-rraach reach
D Prollouts
hence campled only final rollout step
States reachable
steps
Rollout
States reachable steps
Dottine
Rtllisst
-F-RACC STATES arre used for Largets but are never updated !!!
),
a......
Statos being updated,
States for targets,
EDGE-OF-REACH STATES:
Triggers complete collapse!
Reachable in & steps but reachable steps
Rallouts
trancated
+ See Equation (1)
Figure 2: The previously unnoticed edge-of-reach problem. Left illustrates the base procedure used in offline model-based RL, whereby synthetic data is sampled k-step trajectories "rollouts" starting from state in the original offline dataset. Edge-of-reach states are those that can be reached k-steps, but which cannot (under any policy) be reached in less than k-steps. We depict the data collected with two rollouts, one ending in 8k = D. and the other with 8 = C. Right then shows this data arranged into a dataset of transitions as used in Q-updates. State D is edge-of-reach and hence appears in the dataset as s' but never
Bellman updates therefore bootstrap from D, but never update the value D (see Equation ([). ). (For comparison consider state C: also sampled at 8s. but unlike D it is not edge-of-reach, and hence also sampled at s:çk meaning it updated and hence does not cause issues.)
3 The Edge-of-Reach Problem
In the following section, we formally introduce the edge-of-reach problem. We begin by showing the empirical failure of SOTA offline model-based methods on the true environment dynamics. Next, we present our edge-of-reach hypothesis for this, including intuition, empirical evidence the main D4RL benchmark 77. and theoretical proof of its effect on offline model-based training.
3.1 Surprising Failure with the True Dynamics
As described in Sections and 2 prior works view offline model-based RL online RL in an approximate dynamics model. Based on this understanding they propose various forms of dynamics uncertainty penalties to address model errors (see Table 7. This approach is described simply as: "two steps: (a) learning a pessimistic MDP (P-MDP) using the offline dataset; (b) learning a near-optimal policy in this P-MDP" [14. This shared base procedure is shown in Algorithm [ where *P-MDP" refers to the learned dynamics model with a penalty added to address model errors.
This assumption of issues being due to dynamics model errors naturally leads to the belief that the ideal case would be to have perfect error-free dynamics model. However, in Figure and Table we demonstrate that, if the learned dynamics model is replaced with the true dynamics.

---

all Denamics-penalizer methods completely fail on most environments Note that under the true dynamics, all existing dynamics penalty-based methods assume no intervention is needed since there are no model errors. As a result, their penalties all become zero (see T and they all collapse to exactly the same procedure. Therefore the results shown in Table indicaae iindiatt the failure of all existing methods. In the following section, we investigate why having perfectly accurate dynamics leads to failure.
3.2 The Edge-Of-Reach Hypothesis (Mlustrated in Figure 2
Failure under the error-free dynamics reveals that "model errors" cannot explain all the problems in offline model-based RI. Instead, it highlights that there must second issue. On investigation we find this to be the "edge-of-reach problem." We hegin with the main intuition: in offline model-based RL, synthetic data rooloutss is generated as short k-step rollouts starting from states in the original dataset Doffine (see Algorithm|] D.. Crucially, Figure 2 (left) illustrates how, under this procedure, there can exist some states which can be reached in the final rollout step, but which cannot under any policy- be reached earlier. These edge-of-reach states triggers a breakdown in learning, since even with the ability to collect unlimited data, the agent is never able to reach these states 'in time' to try actions from them, and hence is free to 'believe that these odge-of-reach states are great.'
More concretely: 2 illustrates how being sampled only at the final rollout step means edge-of-reach states will appear in the resulting dataset Diollous nexistates s', but will never appear in the dataset as states s. Crucially, in Equation ( we see that values at states 8 are updated, while values at s' are used for the targets of these updates. Edge-of-reach states are therefore used for targets, but are never themselves updated, meaning that their values can be arbitrarily misestimated. Updates consequently propagate misestimation over the entire state-action space. Furthermore, the max operation in the policy improvement step (see Section [ exploits any misestimation by picking out the most heavily overestimated values (33 The mixestimatiin is therefore turned into overestimation, resulting in the value explosion seen in Figure [3] Thus. contrary to common understanding. the -ut-le-sample action problem key in model-free offline RL can be seen to persist in model-based RL.
Edge-of-Re Reach Hypothesis: Limited horizon rollouts from fixed dataset leads to 'edge-of- reach' states: states which are used as targets for Bellman- based updates, but which are never themselves updated. The resulting maximization over misestimated values in Bellman updates causes pathological value overestimation and a breakdown in Q-learning.
When is this an issue in practice? Failure via this mode requires (a) the existence of edge-of-reach states, and (b) such edge-of-reach. states to be sampled. The typical combination of < H along with a limited pool of starting states (s Doftine) means the rollout distribution is unlikely to sufficiently cover the full state space S, thus making (a) likely. Moreover, we observe pathological 'edge-of-reed -eecch seeking' behavior in which the agent appears to 'seek out' edge-of-reach states due to this being the source of overestimation, thus making (b) likely. This behaviour is discussed further in Section 4
In Appendix A we give thorough and unified view of model-free and model- offline RL, dividing the problem into independent conditions for states and actions and examining when we can expect the edge-of-reach problem to be significant.
3.3 Definitions and Formalization
Definition 1 (Edge-off
oe-ce states). Consider a deterministic transition model T 8 x A S, rollout length k. and some distribution over starting states Vu. For some policy : S - A, rollouts are then generated according to so ~ vo(.), 0, T(-(st) and 8t41 ~T(-|s ot)
O....,k 1.
giving (so, on, ..,..., 8.)) 8.). Let us use (s) to denote the marginal distributions over
We define state s S edge-of-reach with respect (T, k, vo) if: for L = k, 3 n S.I. Pt. (8) > 0. but, for t = 1,...,k - and V n, Pt. (s) = 0. In our case, vo is the distribution of states in Doftine.
In Appendix B
we include an extension to stochastic transition models, proof of how errors can consequently propagate to all states, and a discussion of the practical implications.
'For clarity, this change (Approximate-<Tee is described in Algorithm I as pseudocode, and also in Appendix H terms of how it relates to other experiments presented throughout the paper.
5

---

3.4 Empirical evidence on the D4RL benchmark
There are two potential issues in offline model-based RL: (I) dynamics model errors and subsequent model exploitation, and (2) the odge-of-reach problem and subsequent pathological value overestima- tion. We ask: Which is the true source of the issues observed in practice? While (1) is stated as the sole issue and hence the motivation in prior methods, the "failure" results presented in Figure 1 and Tableearr are strongly at odds with this explanation. Furthermore, in Table we examine the model rewards sampled by the agent over training. For (/) to explain the Q-values seen in Figure 3 the sampled rewards in Table would need to be on the order of 10 Instead, however, they remain less than 10, and are not larger than the true rewards, meaning (1) does not explain the observed value explosion (see Appendix (D2)). By contrast, the edge-of-read saage-ofrracc-innuuee pathological overestimation mechanism of (2) exactly predicts this value explosion, with the observations very closely resembling those of the analogous model-free out-of-sample problem [8]. Furthermore, (2) is consistent with the "failure" observations in Table II Finally, we again highlight the discussion in Section 3.2 where we explain why the edge-of-reach problem can be expected to occur in practice.
Figure 3: The base procedure results in poor per- formance (left) with exponential increase in Q- w0 values (right) on the D4RL benchmark. Approx Q' indicates the Q-value for a normalized score of 100 (with 7 0.99). Results are shown for Walker2d-medexp (6 seeds), but we note similar trends across other D4RL. datasets.
(a) D4RL Normalized Score
10
hh)
O-values
20t0
10f
10f
10f
10
10f
Epoch
Epoch
Dose
RAVL
Apprix
4 Analysis with a Simple Environment
In the previous section, we presented the edge-of-reach hypothesis as an explanation of why existing methods fail under the true dynamics. In this section, we construct simple environment to empirically confirm this hypothesis. We first reproduce the observation seen in Figures and S and Table T of the base offline model-based procedure resulting in exploding Q-values and failure to learn despite using the true dynamics model. Next, we verify that ofge-of-reach states are the source of this problem by showing that correcting value estimates only at these states is sufficient to resolve the issues.
4.1 Sctup
We isolate failure observed Section 3.1 with the following setup: Reward defined as in Figure 4.. the transitions function is simply s' - + a E R2, and initial states (analogous to E Domine) are sampled from M4 = U[-2,, 2,² (the area shown in the navy blue box Figure 4).
In applying the offline model- -ased procedure (see Algorithm I to this (true) environment we have precisely the same setup in Section [31]] where again the only difference compared online RL is the use of truncated k-step rollouts with k = 10 (compared to full horizon H = 30). This small change results in the existence of edge-of-reach states (those between the red and orange boxes).
4.2 Observing Pathological Value Overestimation
Exactly as with the benchmark experiments in Section 3.1 the base model-based procedure fails despite using the true dynamics (see blue Figure 4 Base), and the Q-values grow unboundedly over training (compare Figure 4 and Figure 3b).
Looking the rollouts sampled over training (see Figure we see the following behavior:
(Before 25 epochs) Performance initially increases. (Between 25 and 160 epochs) Value misestimation takes over, and the policy begins to aim toward unobserved state-actions (since their values can be misestimated and hence overestimated). (After 160 epochs) This 'edge-of-reade seeking' behavior compounds with each epoch, leading the agent to eventually reach edge-of-ree states. From this point onwards, the agent samples edge-of-reach states at which never receives any corrective feedback. The consequent pathological value overestimation results in complete collapse in performance. In Figure D we visualize the final policy and see that it completely ignores the reward function, aiming instead towards an arbitrary edge-of-reach state.

---

(a) Reward function
Baae-Orachhatcc
(4) RAVL (Ours)
23
199
23
25
30
31B
125
330
175
230
25 75
2:30
125
130
375
200
(4) Returna training
(0 Q-vatues over training
Figure 4: Experiments on the simple environment, illustrating the edge-of-reach problem and potential solutions. (a) Reward function, (b) final (failed) policy with naive application of the base procedure (see Algorithm [ (c) final (successful) policy with patching in oracle Q-values for edge-of-reach states, (d) final (successful) policy with RAVL, (c) returns evaluated training, (f) mean Q-values evaluated training.
4.3 Verifying the Hypothesis Using Value Patching
Our hypothesis is that the source of this failure is value misestimation edge-of-reach states. Our Sase-OrracePaachh experiments (see yellow Figure verify this by showing that patching in the correct values solely edge-of-reach states is sufficient completely solve the problem. This is particularly compelling as in practice we only corrected values at 0.4% of states over training. In Section 5 we introduce our practical method RAVL, which Figures 4 green and 7 show has an extremely similar effect to that of the ideal but practically impossible Base- OraclePatch intervention.
RAVL: Reach-Aware Value Learning
As verified in Section 4.3 issues stem from value overestimation edge-of-reach states. To resolve this, we therefore need to (A) detect and (B) prevent overestimation at edge-of-reach states.
(A) Detecting edge-of-react states: As illustrated in Figure 2 (right), edge-of-reach states states at which the Q-values are never updated, i... those that are out-of-distribution (OOD) with respect to the training distribution of the Q-function,
DDullouts A natural solution for OOD detection is measuring high variance over a deep ensemble [22]. We can therefore detect edge-of-reach states using ensemble of Q-functions. We demonstrate that this is effective in Figure
(B) Preventing overestimation at these states: Once we have detected edge-of-reach states, we may simply apply value pessimism methods from the offline model-free literature. Our choice of an ensemble for part (A) conveniently allows us to minimize over an ensemble of f-functions, which effectively adds value pessimism based ensemble variance.
Our resulting proposal is Reach-Awee Value Learning (RAVL). Concretely, we take the standard offline model-based RI procedure (see Algorithm|] [ and simply exchange the dynamics pessimism penalty for value pessimism using minimization over an ensemble of N Q-functions:
(s) + min Vo(o', (s()))
for n I,.., N
We include EDAC's 1 ensemble diversity regularizer, and in Section 7 we discuss how the impact of value pessimism differs significantly in the model-based (RAVL.) vs model-free (EDAC) settings.
Empirical Evaluation
In this section, we begin by analyzing RAVL on the simple environment from Section Next, we look at the standard D4RI. benchmark, first confirming that RAVI. solves the failure seen with the true dynamics, before then demonstrating that RAVL achieves strong performance with the
7

---

learned dynamics. In Appendix F we include additional results on the challenging mixel-asee V-D4RL benchmark on which RAVL now represents new state-of-the-arr, Finally Section 66.3
we reexamine prior model-based methods and explain why they may work despite not explicitly
addressing the edge-of-reach problem. We provide full hyperparameters in Appendix C.2 ablations showing that RALL is stable over hyperparameter choice in Appendix G.2
and
6.1 Simple Environment
Frequency density
Testing on the simple environment from Section (see green Figure[4... we observe that RAVL behaves the same as the theoretically optimal but practically impossible Base-O Prse-OOrcleeatch method. Moreover, in Figure we see that the Q-value variance over the ensemble is significantly higher for edge-of-reach states, meaning RAVL is detecting and penalizing edge-of-reach states exactly intended.
Effective RAVL penalty
within-raach
edge reach
Figure 5: RAVL'S effeettvv penalty of Qeensemble variance on the environment in Section showing that as intended edge-of-reacee of-rechccccc states have sig- nificantly higher penalty than within- reach states.
6.2 D4RL with the True Dynamics
Next, we demonstrate that RAVL works without dynamies uncertainty and solves the 'failure' ob- served in Section 3.1 Table shows results on the standard offline benchmark D4RL 7 MuloCo [34] v2 datasets with the true (zero error, zero uncertainty) dynamics. We see that RAVL. learns the near-optimal policies, while existing methods using the base model-based procedure (Algorithm completely fail. Section 6.3 we examine how this is because existing methods overlook the critical edge-of-rean reach problem and instead only accidentally address it using dynamics uncertainty metrics. In the absence of model uncertainty, these methods have no correction for edge-of ofrrchh states and hence fail dramatically.
Tsows
Table True dynamics (zero error, zero uncertainty) Existing model-basee methods are presented as different approaches for dealing with dynamics model errors. Surprisingly. however, all existing methods fail in the absence of dynamics errors (when the learned approximate model is replaced by the true model). This reveals that existing methods are unintentionally using their dynamics uncertainty estimates address the previously unnoticed edge-of-reach problem. By contrast, RAVL directly addresses the edge-of-reach problem and hence does not fail when dynamics uncertainty is zero. Experiments are on the D4RL MuloCo v2 datasets. Statistical significance highlighted (6 seeds). **Note that while labeled as 'MOBILE', the results with the true dynamics will be identical for any other dynamics penalty- based method since penalties under the model all zero (see Table 7. Hence these results indicate the failure of all existing dynamics uncertainty- boaee methods.
MOBILE (SOTA)**
RAVL
Dynamics model
Approximate
True
Approximate
True
medium
1..6.11.2
72.2.4.11 (47%)
78.712.0
76.4.13.4
mixed
71.7
72.2+4.11 (11%)
74,9+2.0
77.3+2.2
Halfcheetah
medexp
108.2+2.55
84.9=17.6 (122%)
102.1+8.0
109.3+1.3
medium
106.6.10.6
17.2- (184%)
90.6.1.1.9
101.6.10.9
mixed
103.9+1.0
71.5=32.2 ((31%)
103.1
101.1
Hopper
medexp
112.610.2
6.11:688 (196%)
110.1
107.2+:0.9
medium
87.711.1
7.2.1:.6 (192%)
86.3+11.6
101.242.3
mixed
89.9
7.91:1.9 (191%)
83.04:2.5
86.4+3.0
Walker2d
medexp
115.2+007
7.712.1 (193%)
115.5
103.5+11.
6.3 D4RL with Learned Dynamics
Next, we show that RAVL also performs well with a learned dynamics model. Figure shows RAVL successfully stabilizes the Q-value explosion of the base procedure, and Table 2 shows RAVL largely matches SOTA, while having significantly lower runtime (see Section 6.4). RAVI gives much higher performance on the Halfcheetah mixed and medium datasets than its model-free counterpart EDAC, and in Section 6.5) we discuss why we would expect the effect of value pessimism in the model-based setting (RAVL) to inherently offer much more flexibility than in the model-free setting (EDAC).
8

---

Table 2: comprehensive evaluation of RAVL. the standard DARL. MuloCo benchmark. We show the standard deviation of the final performance averaged over 6 seeds. Our simple approach largely matches the statc-of-the-an without any explicit dynamies penalization and hence works even in the absence of model uncertainty (where dynamics uncertainty based methods fail) (see Table T.
Model-Free
Model-Baser
Environment
BC
CQL
EDAC
MOPO
COMBO
RAMBO MOBILE
RAVL (Ours)
random
2.2
31.3
28.4
38.5
38.8
39.5
39.3
34,4:12.0
medium
43.2
46.9
65.9
73.0
54.2
77.9
74.6
78.7=22.0
Halfcheetah
mixed
37.6
45.3
61.3
72.1
55.1
68.7
71.7
74.9112000
medexp
44.0
95.0
106.3
90.8
90.0
95.4
108.2
102.1:8..0
random
3.7
5.3
25.3
31.7
17.9
25.4
31.9
31,410.1
medium
54.1
61.9
101.6
62.8
97.2
87.0
106.6
90.6
Hopper
mixed
86.3
101.0
103.5
99.5
103,9
103.1::1.1
medexp
53.9
96.9
110.7
81.6
111.1
88.2
112.6
110.1+1.8
random
1.3
5.4
16.6
7.4
7.0
0.0
17.9
17.117.9
medium
70.9
79.5
92.5
84.1
81.9
84.9
87.7
16.
Walker2d
mixed
20.3
76.8
87.1
85.6
56.0
89.2
89.9
83.0
medexp 90.1
109.1
114.7
112.9
103.3
56.7
115.2
115.5+22.4
We additionally include results for the challenging pixel-based V-D4RL benchmark for which latent- space models are used (in Appendix F and accompanying ablation experiments (in Appendix G.2). In this setting, RAVL represents a new SOTA, giving a performance boost of more than 20% for some environments. These results are particularly notable as the pixel-based setting means the base algorithm (DreamerV2) uses model rollouts in an imagined latent space (rather than the original state-action space MBPO). The results therefore give promising evidence that RAVL is able generalize well to different representation spaces.
6.4 Runtime Discussion
Vectorized ensembles can be scaled with extremely minimal effect on the runtime (see Table T0). This means that, per epoch, RAVL is approximately 13% faster than the SOTA (MOBILE, due to MOBILE needing multiple extra forward passes to compute its uncertainty penalty). Furthermore, in total, find that RAVL reliably requires 3x fewer epochs to converge all but the medexp datasets, meaning the total runtime is approximately 70% faster than SOTA (see Appendix G.
6.5 How can prior methods work despite overlooking the edge-of-reach problem?'
While ostensibly to address model errors, we find that existing dynamics penalties accidentally address the edge-of-reach problem: In Figure 6 we a positive correlation between the penalties used in dynamics uncertainty methods and R RAVL's effective penalty of value ensemble variance. This may be expected, dynamics uncertainty will naturally be higher further away from Dottine which is also where edge-of-reach states are more likely to lie. In Section 3.4 we present evidence suggesting that the edge-of-rest ofcrechh-rccc problem likely the dominant source of issues on the main D4RL benchmark, thus indicating that the dynamics uncertainty penalties of existing methods are likely indirectly addressing the edge-of-reach problem. In general. dynamics errors are a second orthogonal problem and RAVL can be casily combined with appropriate dynamics uncertainty penalization [24] 29 for environments where this a significant issue
Related Work
Model-Based Methods. Existing offline model-based methods present dynamics model errors and consequent model exploitation the sole source issues. broad class of methods therefore
propose reward penalties based on the estimated level of model uncertainty [2]
14
24 29 36).
typically using variance over dynamies ensemble. Rigter by setting up an adversarial two-player game between the policy and model. Finally, most related to our method, COMBO 377, penalizes value estimates for state-actions outside model rollouts. However, similarly to Yu et al. [36], COMBO is theoretically motivated by the assumption of infinite horizon model rollouts, which we show overlooks serious implications. Critically, in contrast to our approach, none of these methods address the edge-of-reach problem and thus they fail environment
[27] aim to avoid model exploitation

---

models become more accurate. A related phenomenon of overestimation stemming from hallucinated states has been observed in online model-based RL
Offline Model-Free Methods. Model-free methods can broadly be divided into two approaches to solving the out-of-sample action problem central to offline model-free RI. (see Section
constraint methods and value pessimism-basee methods. Action constraint methods [ 17 2 action avoid using -ul-of-sample actions in the Bellman update by ensuring selected actions are close the dataset behavior policy n². By contrast, value pessimism-based methods aim to directly regularize the value function to produce low-value estimates for out-of-sample acte-actions [ 16 19]. The edge-of- of-reach problem is the model- model-based equivalent of the out-of out-of-sample action problem, and this unified understanding allows us to transfer ideas directly from model-fie mooel--free literature. RAVL is based on EDAC's ss [ use of minimization over Q-ensemble 9 35], and applies this to model-base offline RL.
aim to
What is the effect of value pessimism in the model-based vs model-free settings?
EDAC [ can be seen RAVL's model-free counterpart, however, the impact of value pessimism in the model-free vs the model-based settings is notably different. Recall that, with the ensemble Q-function trained on (s, r, s') ~ Du. the state-actions that are penalized (due to being outside the training distribution) are any (s', a') that are out-of- -ff-tri-istitttiion with respect to the (s. a)'s in the dataset Dr- Recall also that updates use values at (s',a') where d' ~ Ty (see Equation (T).
In the model-free case (EDAC): The dataset is Dn = faiffines sampled from the dataset behavior policy (off-policy), whereas the actions a' are sampled on-policy. This means that EDAC penalizes any (s', a') where a' differs significantly from the behavior policy.
fline, meaning the actions a are effectively
In the model-based case (RAVL): The dataset is Du -rolloutss meaning now both and a' are sampled on-policy. As result, the only a') which will now be out-of-distribution with respect to (s, a) (and hence penalized) are those where the state s' is out-of-distribution with respect to the states in Drollouts. This happens when s' reachable only in the final step of rollouts, i.c. exactly when s' is "edge-of-reach" (as illustrated in Figure Compared to EDAC, RAVL can therefore be viewed as "relaxing" the penalty and giving the freedom to learn a policy that differs significantly from the dataset behavior policy. This distinction is covered in detail in Appendix A
8 Conclusion
This paper investigates how offline model-based methods perform as dynamics models become more accurate. As interesting hypothetical extreme, test existing methods with the true error-free dynamics. Surprisingly, we find that all existing methods fail. This reveals that using truncated rollout horizons (as per the shared base procedure) has critical and previously overlooked consequences stemming from the consequent existence of 'edge-of-reach" states. We show that existing methods are indirectly and accidentally addressing this edge-of-reach problem (rather than addressing model errors as stated), and hence explain why they fail catastrophically with the true dynamics.
This problem reveals close connections between model and model-free approaches and leads us to present a unified perspective for offline RL. Based on this, we propose RAVL. a simple and robust method that achieves strong performance across both proprioceptive and pixel-based benchmarks. Moreover, RAVL directly addresses the edge-of-reach problem, meaning that unlike existing methods- RAVL does not fail under the true environment model, and has the practical benefit of not requiring dynamics uncertainty estimates. Since improvements to dynamics models are inevitable, we believe that resolving the brittle and unanticipated failure of existing methods under dynamics model improvements is an important step towards 'future-pros piature-proofig' offline RL.
Limitations
Since dynamics models for the main offline RL benchmarks are highly accurate, the rege-hf-reacch effects dominate, and RAVL is sufficient to stabilize model-based training effectively without any explicit dynamics uncertainty penalty. In general, however, edge-of-reach issues could be mixed with dynamics error, and understanding how to balance these two concerns would be useful future work. Further, we believe that studying the impact of the edge-of-reach effect in a wider setting could be an exciting direction, for example investigating its effecct an implicit exploration bias in online RL.
10

---

Acknowledgments and Disclosure of Funding
Anya Sims and Cong Lu are funded by the Engineering and Physical Sciences Research Council (EPSRC). The authors would like to thank the anonymous Agent Learning in Open-Endednesss Workshop NeurIPS 2023 and ICLR 2024 reviewers for positive and constructive feedback which helped to improve the paper. We would also like to thank Philip J. Ball and Shimon Whiteson for helpful feedback and discussions on earlier drafts of this work.
References
[1] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-basee offline reinforcement learning with diversified q-ensemble. M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors. Advances in Neural Information Processing Systems, volume 34, 2021. URL htpptttt::/ppooceddnngg 0r122668885aaa111111600002266000
sourips.com cc/ httppppccccpppppp///////// patppr/p22///fle///////
pdf
16 20
[2] Philip Ball, Cong Lu, Jack Parker-Holder, and Stephen Roberts. Augmented world models facilitate zero-shot dynamics generalization from single offline environment. In Marina
Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 619-629, PMLR, 18-24 Jul 2021. URL httpps:/pproceeddnggg.
mlr grres/vv39// htt99//aa1laa..... html
[3] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Merbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffree Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. []
[4] Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint arXiv:2/04.0070
2021. |
[5] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, 2018 URL httppss://prccceeinngssss neurips. cc/ paper cww.pippppcpppprrrrrr.ppe
file
16
[6] Damien Emsst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research (JMLR), 6(18):503-556, 2005. [
[7] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
22
[8] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. 2 10
[9] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research. PMLR, 2018. URL
mlr press/v80/ wwwt////wimooooo.a..oo html
10
[10] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforeement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning. volume 97 of Proceedings of Machine Learning Research, pages 2062-2062. PMLR, 09-15 Jun 2019. Utt.https://prrceediin mlr. wwwpps/sse...... http:///fffoooooof/o/// html
[11] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Muchine Learning Rexearch, pages 1861-1870. PMLR, 10-15 Jul 2018. URL httphttp///// http:::::///rroedinngs mlr press/ hae////0//haaaaa
html 4 16

---

[12]
Taher Jafferjee, Ehsan Imani, Erin Talvitie, Martha White, and Micheal Bowling, Hallucinating value: A pitfall of dyna-style planning with imperfect environment models. arXiv preprint 2ttt:20000.043366. 2020. TO
[13] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances Neural Information Processing Systems, 2019.
16
19
[14] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model-based offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, 2020. URL https::/ neur ips cc/
Paper. pdf
18
[15] Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 1999. 3
[16] Iyya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In Proceedings the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 2021.
URL httpps://proceedinggs mlr. prw.s/rrres///333 http////////////trr/eee ioostrikov2aa...mm html
10
[17] Iya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022. URL https
/openreview hww.:/eee/ffoooo
T0
[18] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- off-policy q-learning via bootstrapping error reduction. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing
Systems, volume 32, 2019. 2 6
15
16
[19) Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Sys- tems, volume 33, 2020. URL htttttt://///osoo pdf aa2b2061826a5df3222111 2201:222111100000000022222220
neurips.co tttpcc//ppppaa shappe/20200ff1a0.
[20] Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. A workflow for offline model-free robotic reinforcement learning. arXiv preprint arXiv:2109, 108/3, 2021. [
[21] Aviral Kumar, Anikait Singh, Frederik Ebert, Mitsuhiko Nakamoto, Yanlai Yang, Chelsea Finn, and Sergey Levine. Pre-training for robots: Offline rl enables learning new tasks from handful of trials, 2023. [
[22] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings the 31st International Conference on Neural Information Processing Systems, NIPS' 17, 2017.
[23] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020. URL /tapp://arxxvi org/ / 2005. 01643
[24] Cong Lu, Philip Ball, Jack Parker-Holder, Michael Oshorn,, and Stephen J. Roberts. Revisiting design choices in offline model based reinforcement learning. In International Conference on Learning Representations, 2022. URL
2
9 [6 19
[25] Cong Lu, Philip J. Ball, Tim G. J. Rudner, Jack Parker-Holder, Michael A Osbome,, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL
17 20
[26) Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Proceedings of the 3rd Conference on Learning for Dynamics and Control, volume I 144 of Proceedings of Machine Learning Research,
pages 1154-1166 1168. PMLR, 2021. 20
12

---

[27] Mare Rigter, Bruno Lacerda, and Nick Hawes. RAMBO-RI RL: Robust adversarial model-based offline reinforcement learning. Advances Neural Information Processing Systems, 2022.
URI http://pppeeeeeeeee..eeerrrr/raa
9
[28] Chamani Shiranthika, Kuo-Wei Chen, Chung-Yih Wang, Chan- Yang, BH Sudantha, and Wei-Fu Li. Supervised optimal chemotherapy regimen based on offline reinforcement learning.
IEEE Journal of Biomedical and Health Informatics, 26(9):4763-477 2022.
[29] Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu. Model- Bellman inconsistency for model-based offline reinforcement learning. In Proceedings of the 40th International Conference
Machine Learning, volume 202 of Proceedings of PMLR R. 23-29 Jul 2023. URL https
Machine Learning Research, pages 331 33177-33194
//proceedings.nt mlr. www.mmmr...r.....m wttppppwwseeeepppppee wttp://v22/ssssee w220//u222222 sun23q. html
9 T6 18
[30] Richard S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting.
SIGART Bull., 2(4): 163, jul 1991. ISSN 5719. doi: 10.1 10.1145/122344112377 URL
https://doi: http:://ddii..rg////// 10. 111111121//144441 122377
[31] Richard Sutton and Andrew G. Barto. Reinforcement Learning: An Introductionn The MIT Press, second edition, 2018. URL http:///wwmmeeeeedaaaaaaeeeeo///
2nd
html 3
[32] Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical considerations for healthcare settings. In Machine Learning for Healthcare Conference, pages
2-35. PMLR, 2021. 1
[33] Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL: Research-oriented deep offline reinforcement learning library. In 3rd Offline RL Workshop: Offline RL as "Launchpad", 2022. URL UtLbttps::///c htttps://openreiieet wwtprrvvieewnee/frrrrreeee net/forum?
SyASSBBbb
17
[34] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physics engine for model-based control. In 2012 JEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012. doi: 00.1009/
[35] Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-leaming, 2015. TO
[36] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: -asee- offline policy optimization. In Advances in Neural Information Processing Systems, 2020. URL https://prcceeddanne neurips. wttppppcccccpc///cccppp/p
2000f htt.:///000ee77bbbbe00000------p pdf
18
[37] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimiza- tion. In Advances in Neural Information Processing Systems, volume 34, pages
28954-28967, 2021. URL
pdf
13

---

Supplementary Material
Table of Contents
A Unified Perspective of Madel-Based and Model-Free RL
15
A Definitions)
15
A.2
Q-Leaming Conditions
15
A.3
Comparison Between Model-Free FFee and Mlodel-Basee Methods
15
Error Propagation Result
16
Proof
16
B.2
Extension to Stochastic Environments and Practical Implementation
16
Implementation Details
16
Algorithm
16
Hyperparameters
17
C.3 Pixel-Based Hyperparameters
17
Additional Tables
18
D.I Existing Methods Penalties All Collapse to Zero Under the True Dynamics
18
D.2
Model Errors Cannot Explain the Q-value Overestimation
18
Additional Visualizations
19
E. Comparison to Prior Approaches
19
Behaviour of Rollouts Over Training
19
Evaluation on the Pixel-Based V-D4RL Benchmark
20
Runtime and Hyperparameter Sensitivity Ablations
20
Runtime
20
G.2
Hyperparameter Sensitivity Ablations
21
Summary of Setups Used in Comparisons
14

---

A A Unified Perspective of Model-Based and Model-Free RL
We supplement the discussion in Section 3.2 with a more thorough comparison of the -am-ol--ammle and edge-of-reach problems, including how they relate to model-free and model-based approaches.
A.1 Definitions
Consider dataset of transition tuples D {ss a2 F2, Sy, dim.d.......... collected according some Og,
dataset policy x 7 (-Is). Compared Section 2 we include the addition of a done indicator d, where d; = indicates episode termination (and d; 0 otherwise). Transition tuples thus consist of state, action, reward, nextstate, done. Consider the marginal distribution over state-actions p3,a (c; :), over states p (.), and conditional action distribution pa). (-1s). Note that Pale (-1s) = (-16). We abbreviate is in distribution with respect to p as x Edist p.
A.2 Q-Learning Conditions
As described Section given some policy æ, we can attempt to learn the corresponding Q-function with the following iterative process:
+++l + arg min
on (Q((( -
|r + (1 d))* (s', d') d'))a)
(3)
inpul
Bellmman turget
Q-learning relies on bootstrapping, hence to be successful we need to be able to learn accurate estimates of the Bellman targets for all (s, a) imputs. Bootstrapped estimates of Q('s,, a') are used in the targets whenever + 1. Therefore, for all a') we require:
Combined state-action condition: (s',a') E dist P2,a or d 1.
In the main paper, this combined state-action perspective for simplicity, however, we can equivalently divide this state-action condition into independent requirements on the state and action follows:
State condition: s' Ediitt p.. or d 1,
Action condition: a' Edist (s)) (given the above condition is met and d + 1).
Informally, the state condition may be violated if DD consists of partial or truncated trajectories, and the action condition may be violated if there is significant distribution shift between π D and π.
A.3 Comparison Between Model-Free and Model-Based Methods
In offline model-free RL, D D Dolline. with x² = 8 For the settings we consider, Dolline consists of full trajectories and therefore will not violate the state condition. However, this may happen in a more general setting with Domine containing truncated trajectories. By contrast, the mismatch between и (used to sample a' in Q-learning) and T (used sample in the dataset Dontime often does lead to significant violation of the action condition. This exacerbates the overestimation bias in Qllearning (see Section 7 and can result in pathological training dynamics and Q-value explosion over training 18)
On the other hand, in offline model-based RL, the dataset D Drollouts is collected on-policy according to the current (or recent) policy such that 77" xs n. This minimal mismatch between π and π means the action condition is not violated and can be considered to be resolved due to the collection of additional data. However, the procedure of generating the data D = Drottouss can be significantly exacerbate the state condition problem, trajectories means the resulting dataset Drollouss is likely to violate the state condition. Due to lack of exploration, certain states may temporarily violate the state condition. Our paper then considers the pathological case of edge-of-reach states, which will always violate the state condition.
the use of short truncated-horizon
This comparison between model-based and model-free is summarized in Table 3
15

---

Table 3: A summary of the comparison between model-free and model-based offliee RI. in relation to the conditions Q-learning descrihed Appendix
Action condition violation
State condition violation
Main source of issues
Model-free
Yes, common practice
Yes. but uncommon in practice
Action condition violation
Model-based
No
Yes, common in practice
State condition violation
B Error Propagation Result
Proposition 1 (Error propagation from edge-of-reach states). Consider rollout of length k, (80, ao; 81, .1,...,88). Sk). Suppose that the state 8ą edge-of-reach and the upproximate value func- tion (sk, (sx)) has error e. Then, standard value iteration will compound ervor - to the estimates of Qu+ (st, at) for tort- l,...,k 1. (Proof in Appendix B
B.1 Proof
In this section, we provide proof of Proposition 1. Our proof follows analogous logic to the error propagation result of Kumar et
Proof. Let denote Q" as the optimal value function, & (s. as) (s) (s, a)l the error at iteration of Q-Learning, and 6 (8, a) (8( TQ the current Bellman error. Then first considering the k case,
(s,,, a+) = =Q2 a.) (s,, az)l
(4)
1Q,(st, at) -TQ; ( ar) (Q)-) (st, az)
(5)
S 1Q,(se, a.) -TQ---Q---- - a:) 1 + 18t, Q" (st, a+)
(6)
85(se; ai) + 0(+1)
(7)
(5(8), +
(8)
Thus the errors at edge-of-reach states are discounted and then compounded with new errors at Q³ (sk-1, For f < 1, the result follows from repeated application of Equation 7 along
the rollout.
B.2 Extension to Stochastic Environments and Practical Implementation
With stochastic transition models (e.g. Gaussian models), we may have the case that no state will truly have zero density, in which case we relax the definition of edge-of-reach states slightly (see Section 3.3 from Pt,= (8) = 0 to Pt. (s) e some small
During optimization, in practice, model rollouts are sampled in minibatches and thus the above error propagation effect will occur on average throughout training. An analogous model-free statement has been given in Kumar et al. [18); however, its significance in the context of model-based methods was previously not considered.
C Implementation Details
In this section, we provide full implementation details for RAVL.
C.1 Algorithm
We use the base model- based procedure as given in Algorithm I and shared across model-based offline RL methods [4
24
29 36 This involves using short MBPO-style [13] model rollouts to
train an agent based SAC [I]. We modify the SAC agent with the value pessimism losses of
EDAC [I. Our dynamics model follows the standard setup in model-based offline algorithms, being realized as deep ensemble
[5] and trained via maximum likelihood estimation.
16

---

C.2 Hyperparameters
For the D4RL 7 MuJoCo results presented in Table 2 sweep over the following hyperparameters and list the choices used in Table "Base" refers to the shared base procedure in model-based offline RL shown in Algorithm!]
(EDAC) Number Q-ensemble elements Nemue the range 500,50)
(EDAC) Ensemble diversity weight 7, in the range {1, 10, 100)
(Base) Model rollout length k, in the range (1,5)
(Base) Real-to-synthetic data ratio r, in the range {0.05, 0.5}
The remaining model-based and agent hyperparameters are given Table Almost all environments use small Noritie 10, with only the Hopper-medexp dataset needing Neritie = 30.
For the uncertainty- dynamics model experiments Table we take the same hyperparameters as for the learned dynamics model for N, k, and and try different settings for E {1, 10, 100, 200}. The choices used are shown Table Note that for existing methods based on dynamics penalties, the analogous hyperparameter (penalty weighting) has no effect under the true dynamics model the penalties will always be zero (since dynamics uncertainty is zero). For the experiments Figure with intermediate dynamics model accuracies, we tune the ensemble diversity coefficient n) over ? E {1, 10, 100, 200) for RAVL, and analogously we tune the uncertainty penalty coefficient over À E {1, 10, 100, 200} for MOPO.
Our implementation based the Clean Offline Reinforcement Learning (CORL, Tarasov al. [ repository, released at https: / github. com/tinkoff. www.nooooaiiiooo ai/CORL under an Apache-2.0 license. Our algorithm takes on average hours run using a V100 GPU for the full number of epochs.
Table 4: Variable hyperparameters for RAVL used in D4RL MuJoCo locomotion tasks with the learned dynamics only n) with the settings used shown in
(see Table With the uncertainty free dynamics (see Table I brackets.
Environment
N Naritic
k
random
10
10 (1)
medium
10
1 (
5
0.05
HalfCheetah
mixed
10
100
0.05
medexp
10
1
5
0.5
random
10
100
0.05
medium
10
(10)
0.5
Hopper
mixed
10
100)
0.5
medexp
30
(10)
0.5
random
10
(
5
0.05
medium
(100)
0.5
Walker2d
mixed
10
10 (100)
0.5
medexp
10
(200)
0.5
Table 5: Fixed hyperparameters for RAVL used in D4RL MuJoCo locomotion tasks.
Parameter
Value
epochs
3,000 for medexp; 1,000 for rest
gamma
0.99
learning rate
3 x 10
batch size
256
buffer retain epochs
number rollouts
50,000
C.3 Pixel-Based Hyperparameters
For the V-D4RI. 25 DeepMind Control Suite datasets presented in Table we use the default hyperparameters for the Offline DV2 algorithm, which are given in Table 6 We found keeping the
17

---

uncertainty weight À = 10 improved performance over À 0 which shows RAVL can be combined with dynamics penalty- -assed methods.
Table 6: Hyperparameters for RAVI used V-D4RL. DeepMind Control Suite tasks.
Parameter
Value
ensemble member count (K)
imagination horizon (H)
5
batch size
64
sequence length (L)
50
action repeat
2
observation size
[64, 64]
discount (?)
0.99
optimizer
Adam
learning rate
[model = x 10-4, a0*o,.ccr-ciiic = 10 10-5)
model training epochs
800
agent training epochs
2,400
uncertainty penalty uncertainty weight (X) number of critics (Neritie)
mean disagreement
10
5
We used hyperparameter sweep over 5, 5, 20} for Nertic but found single value of 5 worked well for all environments we consider.
D Additional Tables
D.1 Existing Methods' Penalties All Collapse Zero Under the True Dynamies
For completeness, Table we we include the expressions for the various dynamics penalties proposed by the most popular model-based approaches. These penalties are all based on estimates of dynamies model uncertainty. Since there is no uncertainty under the true model, all these penalties collapse to zero with the true dynamics.
Table show how existing dynamics penalized offline MBRL algorithms reduce the same base procedure when estimated (epistemic) model uncertainty is zero (as it is with the true dynamics).
Offline MBRL Algorithm
Penalty
Penalty with the true dynamics
MOPO
30
maxi=l..... N [X (s,aa))l a)lle
MORcL
14
waxig,.gnllludss.. a as)
0
MOBILE
29
(f
0
D.2 Model Errors Cannot Explain the Q-value Overestimation
In Figure 3 we observe that the base offline model-based procedure results in Q-values exploding unboundedly to beyond 10.0 over training. Existing methods attribute this to dynamics model errors and the consequent exploitation of state-action space areas where the model incorrectly predicts high reward.
In Table we examine the rewards collected by an agent in the learned dynamics model. Under the "model emrors' explanation, we would expect to see rewards sampled by the agent in the learned model to significantly higher than under the true environment. Furthermore, to explain the Q-values being on the order of 1010, with 7 = 0.99 the agent would need to sample rewards on the order of 10 However, the per-step rewards sampled by the agent are on the order of 10" (and are not higher than with the true environment).
Model errors and model exploitation therefore cannot explain the explosion in Q-values seen over training. By contrast, the edge-of-reach problem exactly predicts this extreme value overestimation behavior.
18

---

Table 8: Per-step rewards with the base offline model-based procedure (see Algorithm 1b. Rewards in model rollouts are close those the true dynamics, showing that model exploitation could not explain subsequent value overestimation. Further, it indicates (as in Janner et al. 13. Lu et al. 24 that the model is largely accurate for short rollouts and hence is unlikely to be vulnerable to model exploitation.
Environment
Model reward
True reward
Hc-med
4.89+1.22
5.01
Hop-mixed
2.37
2.44
W2d-medexp
3.96.11.41
3.96
E
Additional Visualizations
E.1
Comparison to Prior Approaches
we plot the dynamics uncertainty-base penalty used in MOPO 36 against the effective penalty in RALL of variance of the value ensemble. The positive correlation between MOPO's penalty and the edge-of-reach states targeting penalty of RAVL indicates why prior methods may work despite not considering the crucial edge-of-reach problem.
Figure 6
hopper- hopper-mediium medium replay-v2
walkkrrd mediuut expert-v2
balfchoctah mediutt--2mm
107
201
Vake entzmblle vartance
201
107
107
307
1.7
10
10a
10n4
10a
13a7
10t1
uncertsinty penally
Dynamics uncertainty penalty
Dynamics uncertainty penalry
Figure 6: We find that the dynamics uncertainty-base penalty used in MOPO 33 is positively correlated with the variance of the value ensemble of RAVI,, suggesting prior methods may unintentionally address the edge-of- reach problem. Pearson correlation coefficients are 0.49, 0.43, and 0.27 for Hopper-mixed, Walker2d-medexp. and Halfcheetab-medium respectively.
E.2 Behaviour of Rollouts Over Training
In Figure 7 we provide a visualization of the rollouts sampled over training in the simple environment for each of the algorithms analyzed in Figure 4 (see Section 4.. This accompanies the discussion in Section 4.2 of the behavior over training.
19

---

( Base-OraciePatch
Sampled as intial states,
as for
Observed
Sample
Rescheble as 1..
Reachable
RAVL (Ours)
Figure 7: A visualization of the rollouts sampled over training on the simple environment in Section 4 We note the pathological behavior of the baseline, and then the success of the ideal intervention Base- Base-Oracle OateOrrclePaah, and practically realizable method RAVL.
F Evaluation on the Pixel-Based V-D4RL Benchmark
The insights from RAVL also lead improvements in performance the challenging pixel-based benchmark V-D4RL [25]. The base procedure in this pixel-based setting uses the Offline DreamerV2 learned latent space.
algorithm of training on trajectories
We observe in Table that RAVL gives strong boost in performance on the medium and medexp level datasets, while helping less in the more diverse random and mixed datasets. This observation fits with our intuition the edge-of-reach problem: the medium and medexp level datasets are likely to have less coverage of the state space, and thus would expect them to suffer more from edge-of-reach issues and the 'edge-of-reach seeking behavior' demonstrated in Section [42 The performance improvements over Offline DreamerV2 and other model-based baselines including LOMPO (6 therefore suggests that the edge-of-reach problem general and widespread.
We also note that these results are without the ensemble diversity regularizer from EDAC (used on the D4RL benchmark), which we anticipate may further increase performance.
Table 9: show that RAVL extends the challenging pixel-based V-D4RL benchmark, suggesting the out-of-reach problem is also present in the latent-space setting used by the base procedure for pixel-based algorithms. Mean and standard deviation given over 6 seeds.
V-D4RL Environment
Offline DV2
LOMPO
RAVL (Ours)
random
28.7=13.0
21.9+8.1
29.7+2244
mixed
56.5
34.74119.7
49.3
Walker-walk
medium
39.1
43.4111.1
4..
medexp
43.9-:34.4
39.2+11.5
47.6
random
31.7+12.7
11.415.1
36.9
mixed
61.6
36.34:1366
17.6
Cheetah-run
medium
17.2+-3.5
16.448.3
18.3+44.3
medexp
10.4::3.5
11.911.9
12.6144
Overall
35.5+12.0
26.9+F10.9
37.6
G
Runtime and Hyperparameter Sensitivity Ablations
G.1 Runtime
We compare RAVL to SOTA method MOBILE. The additional requirement for RAVL compared to MOBILE that RAVL requires increasing the Q-ensemble size: from N in MOBILE to 10 in RAVL (for all environments except one where we use N 30). The additional requirement for MOBILE compared to RAVI. that MOBILE requires multiple additional forwards passes through the model for each update step (for computing their dynamics uncertainty penalty).
20

---

In Table [0we we show that ensemble size can be scaled with very minimal effect on the runtime. Even increasing the ensemble size .V- 100 (far beyond the maximum ensemble size used by RAVL) only increases the runtime by at most 1 %. Table 7 of MOBILE reports that MOBILE's requirement of additional forward passes increases runtime by around 14%
Table 10: Timing experiments showing that ensemble size for standard vectorized ensembles can be scaled with extremely minimal effect on runtime. Runtime increases by just 1% with Q Qennemble size increased from N = 2 (as used in the base procedure SAC agent) to N = 100. RAVL uses N 10 (for all environments except one where we N 30), meaning the runtime increase compared the base model-based procedure almost negligible. Other methods more computationally expensive changes to the base procedure, hence making RAVL significantly faster than SOTA (see Appendix G.1
Ensemble size (N)
2
10
50
100
Time relative N
(baseline)
x x1.00 + 0.03
x1.01 + 0.04
x1.01 0.02
G.2 Hyperparameter Sensitivity Ablations
In Table [ we include results with different settings of RAVL's ensmble diversity regularizer hyperparameter y). We note minimal change in performance with sweep across two orders of magnitude. This is an indication that RAVI. may be applied to new settings without much tuning.
For RAVL's second hyperparameter of Qeensemble size N, note that in the main benchmarking of RAVL against other offline methods (see Table 2 we use the same value N = 10 across all except environment. This again indicates that RAVL is robust and should transfer to new settings with minimal hyperparameter tuning.
Table 11: Ablations results over RAVL'S 's's diversity regularizer hyperparameter 1).
Hyperparameter
Halfcheetah medium
Halfcheetah mixed
1.0
78.7 + 2.0
61.7 + 5.6
10.0
74.8 1.5
73.8 1.9
100.0
72.1 + 2.2
74.9 + 2.0
H
Summary of Setups Used in Comparisons
Throughout the paper, we compare several different setups in order to identify the true underlying issues in model-based offline RL. We provide summary of them in Table 12 More comprehensive descriptions of each are given the relevant positions in the main text and table and figure captions.
Table 12: We summarize the various setups used for comparisons throughout the paper. denotes application to the simple environment (see Section 4) All methods use k-step rollouts from the offline dataset (or from fixed starting state distribution the case of the simple environment). Figures and Tables
Name Used
Dynamics Model
Agent
Type
Penalty
N Neritic
Table
Figure
Approximate"
Ensemble
(Best Table
2
Table
"True" 'aase
True
n/a
2
Tables 8
Figure
Ensemble
X
2
Tables 8
Figure
RRVVL (Ours)'
Ensemble
X
>
2
0
Figures Figures
'Base
True
n/a
2
0
RRVLL (Ours)*s
True
n/a
>
2
0
"Note that the times reported for EDAC in Table 7 of MOBILE are not representative of practice, since these are with vectoriizd Q Qeenemblle (confirmed from their open-source code). which would require just simple change to 5 lines of code.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly define the scope of the paper and its contributions. For our main result (of fixing the failure under the true dynamics model) we provide full results over the standard D4RL. benchmark (7 with 6 seeds (see Table For our central hypothesis, we include extensive analysis, both on the standard D4RL. benchmark (Section 3.4) and through the construction of simple environment (Section 4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We have a dedicated section for this (Section 9 where we discuss the limi- tations of our analysis and how our work may be extended, particular how would be interesting to examine the interactions of the edge-of- reach problem in settings with less accurate dynamies models.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
22

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We have proposition in Appendix B For this we clearly state all the
assumptions, provide full proof, and also include discussion of the practical implications and limitations.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide complete details of hyperparameters and training settings in Ap- pendix C For all the analysis experiments in particular, we aim for the reader to understand exactly what we have done such that they can appreciate the results fully. As such we have described them in detail (for example Section 4), including further clarification of the terms we have used in Table
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
23

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have open-sourced our code at htttps://aoonymoor -ooee science/ of 8096, and have provided full set of hyperparameters and training settings in Appendix The code is clean and has all the defaults set to the exact training settings used. Furthermore, the README contains simple setup instructions and the exact bash command needed to reproduce the main results. All the datasets used are standard open-sourced benchmarks.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
cublic/guidess http://ooouusissionnoooo Peiiinnoolcyy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
/ //nips. //nips.ccppip wttpcccpppbbc...c wttp::cccpppp...cccceoooeeoc/ce/ http:/guiiss/e/eeee http://dddd//eeuuuiiiinaoaeeeaee
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: For the benchmark experiments we have provided full detail of the hyper- parameters and training settings used in Appendix [ Note also, that we have made sure match all the default settings exactly in line with the base procedure of other papers in this field to allow for meaningful comparison. For each of the analysis experiments, we have aimed for the reader to completely understand the exact setup such that they fully appreciate the results. As such we have described the setup of each in detail.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
24

---

7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All the results are over at least 4 seeds, and we use seeds for the main benchmarking results. We have made sure to provide error bars all relevant tables and figures, and in each case have stated what they show in the accompanying caption. In tables, we have highlighted statistical significance appropriately where relevant.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative етгог rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided average runtime statistics in Appendix C.2 for the main benchmarking results (including the type of machine), and have also provided detailed comparative timing results and discussion in Appendix G.T
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have read and adhere the code of ethics.
Guidelines:
25

---

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper discusses the general potential impact of offline RL in Section | terms of societal impact, the paper on simulated control and we do not believe has relevance in the way meant by this question.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risk, we do not release any models datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
26

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have provided the appropriate reference to the repository that our code is based on in Appendix C Cincluding the accompanying license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssii thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification: We release our code (currently anonymized) the link provided. It is well documented, with clear straightforward instructions on how to reproduce the experiments.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
27

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "SvJLI47X5C", "reviewer": "Reviewer_biRH", "summary": "The paper investigates a phenomena in model-based offline RL that most existing algorithms fail when they are given the perfect model of the environment. Through multiple experiments, the paper argues the failure is due to the wrong estimated values at the states in the end of model rollouts. These states are called edge of reach and are used for targets of Q values but are never trained themselves. The paper continues to propose a fix in the form of pessimistic target value estimate using ensemble Q networks.", "questions": "See Q1, and weakness 3 above.", "limitations": "I find the limitation discussions sufficient.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper does an admirable job with empirical investigations of the root cause of the observed phenomenon.  I find the perfect model experiments, the simple environment example, and the patch oracle fix to be a very convincing set of evidences for the paper's claim. The paper sets a good example for this kind of empirical research.\n- The introduced fix is practical and simple, and I believe might improve other existing model-based offline RL algorithms as well\n- The empirical results obtained by the \n- I find the paper well written and free of typos.", "weaknesses": "**1)** The paper could benefit from more investigations on the necessity of the _detection_ of an edge-of-reach state. The current proposed value learning only uses the pessimistic Q target when it decides that the state is an edge-of-reach state. \n\n**Q1)** I wonder whether if this distinction is necessary and if we could always use the pessimistic value and get the similar results.\n\nI believe this could be doable because in states with low variance, the pessimistic value will not defer from the normal one anyway. Potentially, this removes a hyperparameter which sets the threshold for the choice of value target. It also reduces the number of ensemble networks needed for the algorithm.\n\n**2)** The paper could better position itself in the literature. The pessimistic value target is a ubiquitous trick in RL to overcome value estimation. See for example TD3 [1] paper for actor critic methods. The issue where models query Q networks in untrained states has also been pointed out in MBRL research. See for example [2]. \n\n**3)** It probably is out of the scope of the paper, but I wonder how much of this issue is shared with online MBRL algorithms. They also usually only create rollouts of a limited length. A discussion on this matter will be welcome.\n\n**4)** It could be a personal matter, but the paper has too many future referencing. For example lines 160, 220, 231, 253, 260, 261. I find these confusing as a reader. Jumping to read the future section while reading is not ideal or even possible. Also when the future section is being read, the connection made in the past sections to the current one is probably forgotten. I recommend reducing these references and only make such connections when necessary. A general comment that more discussions about a topic will appear in the future could be enough for the reader without asking them to jump. \n\n\n[1] Fujimoto, Scott, Herke Hoof, and David Meger. \"Addressing function approximation error in actor-critic methods.\" International conference on machine learning. PMLR, 2018.\n[2] Jafferjee, Taher, et al. \"Hallucinating value: A pitfall of dyna-style planning with imperfect environment models.\" arXiv preprint arXiv:2006.04363 (2020).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their valuable feedback on our submission.\n\nWe are particularly encouraged by their appreciation of the care we took with the empirical investigations, in particular with the comments that we provide “a very convincing set of evidences for the paper's claim,” and “set a good example for this kind of empirical research”.\n\n---\n\n**W1/Q1 - “necessity of the *detection* of an edge-of-reach state”**\n\nThe suggestion of removing the explicit “detection” step is in fact exactly how RAVL is implemented (see Equation 2). We just chose to describe it as (A) detecting and (B) penalizing edge-of-reach states to provide intuition of what RAVL is effectively doing. Thank you for this question. We will make it clearer in the paper that this two-phase description is just for intuition purposes.\n\n**W2 - Position in literature and additional references [1] and [2]**\n\nThank you for highlighting this. We have referenced [1] briefly in the related work section already, but we will use the camera-ready extra page allowance to more thoroughly discuss value pessimism in online RL, including the references [1] and [2] you suggested.\n\n**W3 - Effect of edge-of-reach states in *online* MBRL**\n\nThank you for this question. Yes, we are also interested in extending this work to online MBRL. One key distinction here is the pool of initial rollout states. In *offline* MBRL this is fixed over the whole of training ($s_0 \\sim \\mathcal{D}_\\text{offline}$), while in *online* MBRL this changes as the agent collects additional trajectories online. The fixed initial state distribution is an important aspect of edge-of-reach states (see Definition 1). There may, however, be a somewhat analogous notion of edge-of-reach states in online MBRL. As briefly mentioned in the limitations section, we would be interested to investigate whether this could be used (or perhaps already plays a role) as an implicit exploration bias in online MBRL, and believe this could be an exciting direction for future work.\n\n**W4 - Future referencing**\n\nThank you for bringing this to our attention. We agree that the current future referencing may be confusing. Your pointer of instead notifying the reader that “more detail *will* appear in the future” rather than asking them to jump, is extremely helpful and we will definitely make this change. We plan to also use the camera-ready extra page allowance to remove future referencing where possible.\n\n---\n\nWe thank the reviewer again for their valuble feedback. The pointers (in particular about style of future referencing) are really helpful. We hope we have addressed all of your questions. Please let us know if any further discussion would be useful as we would be more than happy to answer additional questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RMsRypcTIJ", "reviewer": "Reviewer_PtsT", "summary": "This paper identifies and investigates a previously neglected issue in offline model-based RL called the \"edge-of-reach problem\", which is due to the truncated rollouts used in model-based RL to mitigate the compounding errors of the dynamics model. The authors proposed Reach-Aware Value Learning (RAVL) to address the edge-of-reach problem using value pessimism. They show RAVL's strong performance on the D4RL and V-D4RL benchmarks.", "questions": "1. Why does RAVL not benefit from reduced dynamics errors according to table 1?\n2. It would be interesting to show how RAVL's performance change by varying the accuracy of the dynamics model, similar to figure 1.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The edge-of-reach problem is interesting and previously overlooked in offline model-based RL. This paper is the first to formally investigate the problem.\n2. Comprehensive, well-designed experiments to support the claims.", "weaknesses": "1. The proposed method doesn't outperform existing baselines according to the reported results on D4RL. Although it's claimed that \"RAVL can be easily combined with appropriate dynamics uncertainty penalization\", it's not supported by any empirical evidences. Hence, it's unclear how well RAVL work in general when combined with other model-based methods with design choices orthogonal to value pessimism.\n2. I found the paper a bit hard to read due to the formatting, e.g. too many long italics phrases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful review of our submission.\n\nWe are encouraged that the reviewer found the problem to be “interesting,” and the experiments “comprehensive, well-designed”, and are grateful they appreciated that the work is “the first to formally investigate the problem”. We have addressed their questions and comments below.\n\n---\n\n**W1 - Combining RAVL with dynamics uncertainty penalization**\n\nThe reviewer asked for support of the claim that \"RAVL can be easily combined with appropriate dynamics uncertainty penalization.”\n\nIn terms of implementation: this is extremely simple since dynamics penalization is typically a change to the dynamics model’s reward, while RAVL’s value pessimism is a change to the agent’s value function update.\n\nIn terms of empirical evidence: we refer the reviewer to the V-D4RL experiments. As stated in Appendix C.3, for the RAVL results we add RAVL’s value pessimism in addition to the dynamics pessimism already used in the base procedure. As such, the V-D4RL results are an example of combining RAVL’s value pessimism with dynamics uncertainty penalization, exactly as requested. This provides evidence that RAVL’s value pessimism is effective even when the model is less accurate and dynamics uncertainty is necessary. We will make the details in Appendix C.3 more prominent in the camera-ready submission.\n\n**W2 - Formatting e.g. italics affecting readability**\n\nHaving looked over the paper again we completely agree with this. We will make sure to refine our use of italics and bold text etc. ahead of the camera-ready submission. Thank you for bringing this to our attention.\n\n**Q1 - Why does RAVL not benefit from reduced dynamics errors according to Table 1?**\n\nThank you for this question.\n\nSection 3.4 claims and gives evidence that dynamics model errors are already negligible for the main D4RL benchmark (and that issues are instead caused by the edge-of-reach problem).\n\nThe trend referred to (namely that reducing dynamics errors does not affect performance) is therefore exactly what we would expect, and the results in Table 1 can hence be seen as further evidence that (contrary to common understanding) dynamics errors contribute negligibly to the issues seen in offline model-based RL.\n\n**Q2 - Results of RAVL's performance with different dynamics model accuracies**\n\n[Please see the rebuttal PDF.]\n\nWe have run additional experiments (4 seeds), and have added results for RAVL to the original Figure 1 as requested (please see the new Figure 1 in the rebuttal PDF). As expected, RAVL’s performance remains strong as dynamics model accuracy increases. Furthermore, RAVL appears to be significantly more robust than dynamics uncertainty-based methods as dynamics model accuracy is reduced.\n\nWe sincerely thank you for asking for this. These results significantly strengthen our paper and we are excited to add them to the camera-ready submission.\n\n---\n\nWe thank the reviewer again for their invaluable feedback. We hope we have addressed all of your questions and concerns. Please let us know if there is anything remaining as we would be more than happy to discuss further. If we have been able to address your concerns, we humbly ask if you would consider raising your score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RHQjHBO3iy", "reviewer": "Reviewer_GKpt", "summary": "The paper presents a novel analysis of the challenges faced by offline model-based reinforcement learning (RL) when the dynamics model becomes increasingly accurate. The central thesis is that existing methods fail under true, error-free dynamics due to a previously unaddressed issue termed the \"edge-of-reach\" problem. The authors provide a thorough investigation, theoretical analysis, and empirical evidence to support their claims. They propose Reach-Aware Value Learning (RAVL), a method that directly addresses this problem, demonstrating robust performance across benchmarks.", "questions": "1. I wonder how existing uncertainty-penalty-based offline model-based RL methods perform if I gradually increase the length of rollouts in the estimated dynamics model along with the improved accuracy of the model (a very intuitive trick).\n\n2. Does the \"edge-of-reach\" hypothesis exist in many online RL scenarios? For example, researchers usually early stop the trajectory according to a pre-defined maximal length of the trajectory in the Gym Mujoco benchmark.", "limitations": "The authors have discussed the limitations of their work properly.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized, with a clear abstract, introduction, and conclusion that effectively summarize the contributions and findings.\n\n2. The authors have provided open-source code and detailed hyperparameter settings, facilitating the reproduction of the experiments.\n\n3. Rational experiments on standard benchmarks and a simple environment validate the \"edge-of-reach\" hypothesis and demonstrate the effectiveness of RAVL.", "weaknesses": "The methodology contribution of this paper is limited. While I assume that the \"edge-of-reach\" problem is significant to the current offline RL literature, the proposed method by the authors fails to demonstrate obvious superiority compared with the SOTA offline model-free or model-based methods (see Table 2 of this paper). One reason, I guess, is that the D4RL benchmark is too simple to underscore the superiority of addressing the \"edge-of-reach\" problem. On the other hand, I expect that authors can validate their method on more complex and challenging benchmarks and provide more convincing results. I am willing to reconsider my score after seeing this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their careful feedback on our submission.\n\nWe are grateful that they appreciated our “thorough investigation, theoretical analysis, and empirical evidence to support [our] claims” and “robust performance across benchmarks.” \n\nWe found their questions particularly insightful (see below).\n\n---\n\n**W1.1 - Performance on D4RL**\n\nThe reviewer highlights that, on the D4RL benchmark, we only match SOTA rather than surpassing it. Below we discuss why this is expected and why we believe this does not affect the key contribution of our paper.\n\nIn Section 6.5 we describe why existing methods can sometimes be made to work despite misunderstanding the underlying issues in offline RL. (tl;dr: If there is a lucky correlation between dynamics uncertainty and EOR states, then the dynamics penalties of existing methods can indirectly address the true EOR problem.) For the common D4RL setup (on which many existing methods have been developed) this lucky correlation holds (see Figure 6), meaning existing methods can be made to work well, and we would not necessarily expect RAVL to beat them. (Further, the majority of SOTA scores are close to, or already exceed, expert level $>100$.)\n\nIn general, however, this lucky correlation is not guaranteed, and the indirect approach of existing methods means we may expect them to be very brittle to dynamics model changes. Indeed, in Figure 1 of the rebuttal PDF, we show that existing methods completely break down if the dynamics model is changed slightly. RAVL, by contrast, is much more robust due to directly addressing the true underlying EOR problem (see Figure 1 in the rebuttal PDF).\n\nWe believe the correction of a significant misunderstanding, and RAVL’s consequent much-improved robustness are highly valuable contributions to the community.\n\n**W1.2 - Performance on more complex and challenging benchmarks**\n\nWe thank the reviewer for asking about more complex and challenging benchmarks. We would like to highlight our additional results on the pixel-based V-D4RL benchmark in Appendix F (moved to the Appendix due to space limitations).\n\nHere RAVL represents a new SOTA, giving a performance boost of more than 20% for some environments. These results are particularly notable as the pixel-based setting means the base algorithm (DreamerV2) uses model rollouts in an imagined latent space (rather than the original state-action space as in MBPO). The results therefore give promising evidence that RAVL is able to generalize well to different representation spaces.\n\nWe will use the camera-ready extra page to feature these results in the main paper. We hope this answers the reviewer’s concern with a much higher-dimensional and more challenging benchmark.\n\n**Q1 - What happens if we increase the rollout length as dynamics accuracy improves?**\n\nWe thank the reviewer for this interesting and insightful question. Tuning the rollout length as suggested is indeed what we tried first.\n\nIn the limit of perfect dynamics accuracy this of course works, since rollout length can be increased to full horizon and the procedure becomes online RL. For intermediate dynamics model accuracies, however, we experimented with this extensively but found we were unable to get existing methods to work.\n\nOur intuition for why this is is as follows:\n\nConsider a 2D environment like in Section 4, and imagine we have a dynamics model for which error and uncertainty is significant to the left, and negligible to the right. The model being imperfect means rollouts will need to be truncated. (All existing methods heavily truncate rollouts e.g. $k=1,5$ to avoid compounding errors, even with dynamics uncertainty penalties). However, rollout truncation means we now have EOR states.\n\nThe question is: *Is it possible to tune dynamics uncertainty-based methods to work in this setting?*\n\nUncertainty penalties can successfully penalize the EOR states to the left. However, it is impossible to get uncertainty penalties to target the EOR states to the right (since model uncertainty to the right is negligible).\n\nThis illustrates that we can have dynamics models for which it is theoretically impossible to get uncertainty-based methods to work, even allowing for tuning the rollout length. (RAVL, by contrast, would be expected to work in this setting since it directly ensures all EOR are targeted.)\n\nThe much stronger reliance on the dynamics model for uncertainty-based methods compared to RAVL (as highlighted by this setting) fits with our empirical observations of existing methods being much more brittle to changes in the dynamics model. See for example our new results (Figure 1 of the rebuttal PDF), in which we find that RAVL is significantly more robust, both when model accuracy is increased and decreased.\n\nWe thank the reviewer for asking this. We found it to be a very interesting and insightful question and would be more than happy to discuss further.\n\n**Q2 - Does the \"edge-of-reach\" hypothesis exist in many online RL scenarios?**\n\nThank you for another insightful question!\n\nThe trajectory cutoff, for example $H=1000$ for MujoCo, can indeed be viewed as a truncated horizon. This means the EOR problem could in theory be present in online RL. We anticipate, however, that $H=1000$ is sufficient for the agent to be able to reach all possible states within this horizon, meaning that the set of EOR states will be empty and the EOR problem will not be present.\n\nAs briefly mentioned in the limitations section, however, we are interested in investigating whether truncated horizons could be used in some way in online RL to harness the EOR effect and induce an implicit exploration bias.\n\n---\n\nWe thank the reviewer again for their valuable feedback and insightful questions. We hope we have addressed all of your concerns. Please let us know if you have any remaining questions. If we have been able to address your concerns we humbly ask if you would recommend acceptance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L9Cj1bsNDM", "reviewer": "Reviewer_8ru8", "summary": "This paper identifies an overlooked problem in offline model-based reinforcement learning (RL).\nOffline model-based RL approaches commonly perform online RL inside a learned dynamics model and\ndue to model errors, they generally use truncated $k$ step rollouts, instead of full trajectories (episodes).\nIn this paper, the authors replace the learned dynamics model with the true simulator and show that offline model-based\nRL approaches fail even when they have access to the true dynamics.\nThis is due to states which can only be reached at the end of the $k$ step rollouts (which the authors coin \"edge-of-reach\" states) and thus, only appear in the target of the Q update. As such, the Q-function is never updated at these \"edge-of-reach\" states so any overestimation is gradually propagated over the entire state-action space.\nThe authors provide a nice diagram which provides intuition and shows the effect in a simple environment.\nThey then propose an algorithm which overcomes the so-called \"edge-of-reach\" problem by using an ensemble of Q functions.\nThey demonstrate good performance in D4RL and V-D4RL.", "questions": "- What is on the x-axis of Figure 5? What environment is this from? This is not clear from the text or caption.", "limitations": "The limitations are adequately discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think this is a good paper that should be accepted.\nIt identifies an overlooked problem in offline model-based RL which seems like it will be an important contribution to the community.\nThe authors have also done a nice job providing intuition for the problem via Figure 2 and the example in Section 4.\n\nBased on the identified problem, the authors also propose a novel method which is simple yet effective.\nThey have also demonstrated the effectiveness of their approach on data sets with both proprioceptive and image-based observations.", "weaknesses": "I have no major weaknesses. Nevertheless, I do have some comments which I provide to help improve the paper.\nThe main weakness of this paper is that it looks messy and overcrowded. There are multiple reasons for this which I will now detail.\n\nThe use of italics throughout the text makes the paper look very messy.\nSometimes the italics refer to questions whilst other times they just seem to be random words.\nI would advise the authors to remove all italics.\n\nThere are too many subsections.\nFor example, in Section 6 there are 5 subsections and each subsection is a single paragraph.\nI would advise the authors to replace \\subsection with \\paragraph.\nSimilarly, on Lines 283/293/303 the authors use unnumbered subsections. This looks very messy.\nI suggest the authors replace this with \\paragraph so that the text starts immediately after the bold text and does not have a line break before.\n\nThere are a lot of incorrect uses of colons, which makes the writing quality feel poor. Please remove the colons on lines 12/90/114/122/137/144/154/176/195/273/305.\n\nThe authors do not mention how the Q-function is initialized. I think it is very important to include all of this information. \nFor example, if we initialize the Q-function to be zero at all inputs wouldn't that help prevent overestimation bias from \"edge-of-reach\" states?\n\n## Minor comments and typos\n- Line 89 - $\\\\}\\_{i=1,\\ldots,N}$ should be $\\\\}\\_{i=1}^{N}$\n- Line 122 - incorrect use of semicolon. This should just be \"and\".\n- Line 137 - the full stop should be outside the quote.\n- Line 167 - $0\\ldots k-1$ should be $0,\\ldots,k-1$\n- Line 170 - Why is the last sentence of the definition not italicised?\n- Line 249 - the comma should be outside the quote.\n- Line 306/308/309/31 - the dataset subscript looks wrong.\n- Sections shouldn't lead straight into subsections, e.g. 6 into 6.1. \n    - Add a few sentences explaining what you are introducing in this section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their obvious extreme care in providing us with feedback. We are encouraged by the reviewer’s comments that the paper “should be accepted,” \"will be an important contribution to the community,” and are grateful for their detailed comments which will help us improve our paper.\n\n---\n\n**Weaknesses - Formatting, punctuation, use of subsections, etc.**\n\nWe thank the reviewer for their detailed feedback on the organization and formatting of the paper. Having revisited the paper, we completely agree with the reviewer’s comments. We will use the extra page allowance to make the paper feel less crowded, and will make all of the suggested changes (use of italics, commas and colons, subsections, etc.) ahead of the camera-ready submission. We are grateful for the care taken in providing a detailed list of suggested edits, and would like to thank the reviewer again for their time in helping us improve the paper.\n\n**Q1 - Implementation details of Q-function initialization**\n\nWe use the default initialization scheme (Kaiming uniform). We thank the reviewer for asking this and will add this implementation detail to the paper.\n\n**Q2 - Effect of initializing the Q-function at zero**\n\n[Please see the Rebuttal PDF.]\n\nThe reviewer asked whether the edge-of-reach problem could be solved by simply initializing the Q-function to be zero for all inputs. This is a good question and gets to the heart of the pathological issues involved in the edge-of-reach problem. Initializing a network so that its output is zero given any input is non-trivial. (The obvious way to do so would be to initialize all weights to zero, but the network would then not train.) Thus, we instead we consider what would happen if we initialize so that all Q-value outputs are very close to zero.\n\nAs the reviewer commented, in this case we might expect the edge-of-reach problem to be avoided since edge-of-reach states cannot then be a source of overestimation if their Q-values are significantly less than the optimal Q-values. However, as shown in the histograms in the rebuttal PDF, the standard NN initialization that we use already in all our experiments already results in all Q-values being very close to zero at initialization ($<1e-2$). Moreover, all these values are much lower than the optimal Q-values ($\\sim 5e+3$) and much lower that the Q-values reached over training ($>1e+10$) (see Figure 3). Yet we still observe the edge-of-reach problem. This means the above line of reasoning does not hold.\n\nInstead what happens is that, when values at within-reach states are increased, the use of NNs as function approximators means that the Q-values at nearby edge-of-reach states may also increase. Q-values at edge-of-reach states are therefore not guaranteed to remain small over training, and hence can become a source of overestimation. Moreover, overestimated Q-values lead to other nearby Q-values also being raised, which can then lead to the original Q-values being raised further, thus setting up a pathological positive feedback cycle whereby Q-values increase unboundedly over training (as seen in Figure 3). This is investigated and explained more thoroughly in [1] and [2] in the context of the analogous model-free offline problem (where function approximation means out-of-sample state-actions become a source of pathological overestimation).\n\nWe thank the reviewer for this important question. We will use the camera-ready extra page allowance to include some more of the detail from [1] on the role function approximation and this pathological positive feedback loop.\n\n[1] - Aviral Kumar, et al. Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction, 2019.\n\n[2] - Sergey Levine, et al. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020.\n\n**Q3 - Clarification on the $x$-axis of Figure 5**\n\nThe $x$-axis of Figure 5 shows variance over RAVL’s Q-ensemble. Since RAVL takes the minimum over this Q-ensemble (see Equation 2), the penalty being added is effectively proportional to the Q-ensemble variance, hence why we labeled the $x$-axis as “RAVL’s effective penalty.” We described this briefly in the caption, but we will make this clearer in the camera-ready version.\n\n---\n\nWe thank the reviewer again for their careful review of our submission. The question about initializing the Q-function at zero was particularly insightful. We are also extremely grateful for the feedback about formatting and the detailed list of suggested edits. We agree with these and will make the suggested changes. Thank you for your help in improving our paper. Please let us know if you have any more questions or comments as we would be more than happy to discuss further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers. It is clear they have each taken time and care, and their feedback has greatly helped us improve and refine our work.

We are gratified by the reviewers' overall positive assessment of our work, including comments that the paper “should be accepted,” "will be an important contribution to the community" (Reviewer 8ru8), and the appreciation that “this paper is the first to formally investigate the problem” (Reviewer PtsT).

We are also pleased that the reviewers appreciated RAVL’s “robust performance across benchmarks” (Reviewer GKpt), found the set of experiments to be “a very convincing set of evidences for the paper’s claim,” and found the paper to “set a good example for this kind of empirical research” (Reviewer biRH).

While reviewers commented that the paper was “well-organized” (Reviewer GKpt), and “well written” (Reviewer biRH), the key shared feedback seemed to be that editing the formatting (punctuation, italics, forward referencing, etc.) would make the paper easier to read. We thank the reviewers for this feedback and will be sure to make these changes ahead of the camera-ready submission, especially with the extra page allowance.

**New results showing RAVL’s robustness with changing dynamics accuracy**

[Please see the Rebuttal PDF.]

We are excited to present additional results (as requested by Reviewer PtsT) where extend Figure 1 of the paper to include RAVL. The original version of Figure 1 showed how dynamics uncertainty-based methods fail as dynamics accuracy is increased (in red). Please see Figure 1 of the Rebuttal PDF where we have now added RAVL (in purple).

The new results show that, while existing methods fail, RAVL maintains performance as dynamics accuracy is increased. Additionally, in the other direction, we also find that RAVL is more robust to decreasing dynamics model accuracy.

This observation of RAVL’s much stronger robustness to dynamics model changes fits with our understanding of the edge-of-reach problem, since existing methods address the edge-of-reach problem indirectly via relying on the uncertainty of the dynamics model, while RAVL directly addresses the problem without relying on the dynamics model’s uncertainty. We are excited to share these new results as they further highlight the strength of our approach.

Meta-review: This paper explores the so-called "edge-of-reach" problem in offline model-based RL, which arises when using short, truncated rollouts to generate synthetic data. Contrary to prevailing theories, the paper finds that improving the learned dynamics model does not necessarily enhance performance; in fact, using an error-free dynamics model can lead to the complete failure of existing methods. This failure is due to the existence of "edge-of-reach" states, where value estimates become unreliable because these states are only encountered at the end of rollouts and not during intermediate steps. To address this issue, the paper proposes Reach-Aware Value Learning (RAVL), a method that directly targets the edge-of-reach problem, thereby preventing fatal value overestimation resulting in the collapse of performance. RAVL demonstrates robust performance even as dynamics models improve, offering a more future-proof approach to offline RL.

The reviews are overall positive, and thus recommend to accept the paper. Authors are encouraged to incorporate additional results provided during the dicussion phase.
Decision: Accept (poster)