The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning

Authors: anya  sims
cong  lu
jakob n foerster
yee  whye teh
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 18:48:02

Abstract: Offline reinforcement learning (RL) aims to train agents from pre-collected datasets. However, this comes with the added challenge of estimating the value of behaviors not covered in the dataset. Model-based methods offer a potential solution by training an approximate dynamics model, which then allows collection of additional synthetic data via rollouts in this model. The prevailing theory treats this approach as online RL in an approximate dynamics model, and any remaining performance gap is therefore understood as being due to dynamics model errors. In this paper, we analyze this assumption and investigate how popular algorithms perform as the learned dynamics model is improved. In contrast to both intuition and theory, if the learned dynamics model is replaced by the true error-free dynamics, existing model-based methods completely fail. This reveals a key oversight: The theoretical foundations assume sampling of full horizon rollouts in the learned dynamics model; however, in practice, the number of model-rollout steps is aggressively reduced to prevent accumulating errors. We show that this truncation of rollouts results in a set of edge-of-reach states at which we are effectively "bootstrapping from the void." This triggers pathological value overestimation and complete performance collapse. We term this the edge-of-reach problem. Based on this new insight, we fill important gaps in existing theory, and reveal how prior model-based methods are primarily addressing the edge-of-reach problem, rather than model-inaccuracy as claimed. Finally, we propose Reach-Aware Value Learning (RAVL), a simple and robust method that directly addresses the edge-of-reach problem and hence - unlike existing methods - does not fail as the dynamics model is improved. Since world models will inevitably improve, we believe this is a key step towards future-proofing offline RL.

Main paper: 
 introduction: 
 Standard online reinforcement learning (RL) requires collecting large amounts of on-policy data. This can be both costly and unsafe, and hence represents a significant barrier against applying RL in domains such as healthcare [28, 32] or robotics [2, 4, 20] , and also against scaling RL to more complex problems. Offline RL [6, 23] aims to remove this need for online data collection by enabling agents to be trained on pre-collected datasets. One hope [21] is that it may facilitate advances in RL similar to those driven by the use of large pre-existing datasets in supervised learning [3] . Results shown are for MOPO [36] , but note that this failure indicates the failure of all existing uncertainty-based methods since each of their specific penalty terms disappear under the true dynamics as 'uncertainty' is zero. By contrast, our method is much more robust to changes in dynamics model. The x-axis shows linearly interpolating next states and rewards of the learned model with the true model (center→right) and random model (center→left), with results on the D4RL W2d-medexp benchmark (min/max over 4 seeds). The full set of results and experimental setup are provided in Table 1 and Appendix C.2 respectively.The central challenge in offline RL is estimating the value of actions not present in the dataset, known as the out-of-sample action problem [17] . 2 A naïve approach results in extreme value overestimation due to bootstrapping using inaccurate values at out-of-sample state-actions [18] . There have been many proposals to resolve this, with methods largely falling into one of two categories: modelfree [1, 8, 10, [17] [18] [19] or model-based [14, 24, 29, 36] .Model-free methods typically address the out-of-sample action problem by applying a form of conservatism or constraint to avoid using out-of-sample actions in the Bellman update. In contrast, the solution proposed by model-based methods is to allow the collection of additional data at any previously out-of-sample actions. This is done by first training an approximate dynamics model on the offline dataset [13, 30] , and then allowing the agent to collect additional synthetic data in this model via k-step rollouts (see Algorithm 1) . The prevailing understanding is that this can be viewed as online RL in an approximate dynamics model, with the instruction being to then simply "run any RL algorithm on M until convergence" [36] , where M is the learned dynamics model with some form of dynamics uncertainty penalty. Existing methods propose various forms of dynamics penalties (see Table 7 ), based on the assumption that the remaining performance gap compared to online RL is solely due to inaccuracies in the learned dynamics model. This understanding naturally implies that improving the dynamics model should also improve performance. Surprisingly, we find that existing offline model-based methods completely fail if the learned dynamics model is replaced with the true, error-free dynamics model, while keeping everything else the same (see Figure 1 ). Under the true dynamics, the only difference to online RL is that in online RL, data is sampled as full-length episodes, while in offline model-based RL, data is instead sampled as k-step rollouts, starting from a state in the original offline dataset, with rollout length k limited to avoid accumulating model errors. Failure under the true model therefore highlights that truncating rollouts has critical and previously-overlooked consequences.We find that this rollout truncation leads to a set of states which, under any policy, can only be reached in the final rollout step (see red in Figure 2 ). The existence of these edge-of-reach states is problematic as it means Bellman updates (see Equation (1) ) use target values that are never themselves trained. This is illustrated in Figure 2 , and described in detail in Section 3. This effective "bootstrapping from the void" triggers a catastrophic breakdown in Q-learning. Concisely, this issue can be viewed as all actions from edge-of-reach states remaining out-of-reach over training. Hence, contrary to common understanding, the out-of-sample action problem central to model-free methods is not fully resolved by a model-based approach. In fact, in Section 3.4 we provide detailed analysis suggesting that this is the predominant source of issues on the standard D4RL benchmark. In Section 6.5 we consequently reexamine how existing methods work and find they are indirectly and unintentionally addressing this issue (rather than model errors as claimed).Algorithm 1: Pseudocode for the base procedure used in offline model-based methods. In summary: (1) Train a dynamics model M on Doffline, then (2) Train an agent (often SAC) with k-step rollouts in the learned model starting from s ∈ Doffline. Existing methods consider issues to be due to errors in M and hence introduce dynamics uncertainty penalties (see Table 7) . We find the predominant source of issues to be the edge-of-reach problem, and hence instead propose using value pessimism (RAVL) (see Section 5) . (Collect data) Starting from states in Doffline, collect k-step rollouts in M with π θ . Store data in Drollouts 7:(Train agent) Train π θ and Q ϕ on Drollouts ∪ Doffline (mixed with ratio r) Add Q-value pessimism 8: end for *This uncertainty penalty collapses to zero with the true error-free dynamics ( M = M ) (see experiments Table 2 ).We have the following problem: Indirectly addressing the edge-of-reach problem means existing model-based methods have a fragile and unforeseen dependence on model quality and fail catastrophically as dynamics models improve. Since model improvements are inevitable, and since tuning of hyperparameters is particularly impractical in offline RL, this is a substantial practical barrier.In light of this, we propose Reach-Aware Value Learning (RAVL), a simple and robust method that directly addresses the edge-of-reach problem. RAVL achieves strong performance on standard benchmarks, and moreover continues to perform even with error-free, uncertainty-free dynamics models (Section 6). Our model-based method bears close connections to model-free approaches, and hence in Appendix A.2 we present a unified perspective of these two previously disjoint subfields. 
background: 
 Reinforcement Learning. We consider the standard RL framework [31] , in which the environment is formulated as a Markov Decision Process, M = (S, A, T, R, µ 0 , γ), where S and A denote the state and action spaces, T (s ′ |s, a) and R(s, a) denote the transition and reward dynamics, µ 0 the initial state distribution, and γ ∈ (0, 1) is the discount factor. The goal in reinforcement learning is to learn a policy π(a|s) that maximizes the expected discounted returnE µ0,π,T [ ∞ t=0 γ t R(s t , a t )].Actor-Critic Algorithms. The broad class of algorithms we consider are actor-critic [15] methods which jointly optimize a policy π and state-action value function (Q-function). Given a dataset D of (state, action, reward, nextstate) transitions, actor-critic algorithms iterate between two steps:1. (Policy evaluation) Update Q ϕ to fit the current policy using the following Bellman update:Key expression: Q ϕ (s, a) Update value at (s, a) ← r + γQ ϕ (s ′ , π θ (s ′ )) Using value at (s ′ , π θ (s ′ )) (s, a, r, s ′ ) ∼ D (1) 2. (Policy improvement) Update π θ to increase the current Q-value predictions: Q ϕ (s, π θ (s)) 3 .Offline RL and the Out-of-Sample Action Problem. In offline RL, training must rely on only a fixed dataset of transitions D offline = {(s, a, r, s ′ ) (i) } i=1,...,N collected by some policy π β . The central problem in this offline setting is the out-of-sample4 action problem: The values of Q ϕ being updated are at any state-actions that appear as (s, a) in D offline . However, the targets of these updates rely on values at (s ′ , a ′ ) where a ′ ∼ π θ (see Equation ( 1)). Since a ′ ∼ π θ (on-policy) whereas a ∼ π β (offpolicy), (s ′ , a ′ ) may not appear in D offline , and hence may itself have never been updated. This means that updates can involve bootstrapping from 'out-of-sample' and hence arbitrarily misestimated values and thus lead to misestimation being propagated over the entire state-action space. The max operation (implicit in the policy improvement step) further acts to exploit any misestimation, converting misestimation into extreme pathological overestimation. As a result Q-values often tend to increase unboundedly over training, while performance collapses entirely [18, 23] .Model-Based Offline RL. Model-based methods [30] aim to solve the out-of-sample issue by allowing the agent to collect additional synthetic data in a learned dynamics model. They generally share the same base procedure as described in Algorithm 1. This involves first training an approximate dynamics model M = (S, A, T , R, µ 0 , γ) on D offline . Here, T (s ′ |s, a) and R(s, a) denote the learned transition and reward functions, commonly realized as a deep ensemble [5, 22] . Following this an agent is trained using an online RL algorithm (typically SAC [11] ), for which data is sampled as k-step trajectories (termed rollouts) under the current policy, starting from states in the offline dataset D offline . The base procedure of training a SAC agent with model rollouts does not work out of the box. Existing methods attribute this as due to dynamics model errors. Consequently, a broad class of methods propose augmenting the learned dynamics model M with some form of dynamics uncertainty penalty, often based on variance over the ensemble dynamics model [2, 14, 24, 29, 36] .We present some explicit examples in Table 7 . Crucially, all of these methods assume that, under the true error-free model, no intervention should be needed, and hence all the uncertainty penalties collapse to zero. In the following section, we show that this assumption leads to catastrophic failure. 1)). (For comparison consider state C: C is also sampled at s k , but unlike D it is not edge-of-reach, and hence is also sampled at s i<k meaning it is updated and hence does not cause issues.)3 The Edge-of-Reach ProblemIn the following section, we formally introduce the edge-of-reach problem. We begin by showing the empirical failure of SOTA offline model-based methods on the true environment dynamics. Next, we present our edge-of-reach hypothesis for this, including intuition, empirical evidence on the main D4RL benchmark [7] , and theoretical proof of its effect on offline model-based training.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We have a dedicated section for this (Section 9) where we discuss the limitations of our analysis and how our work may be extended, in particular how it would be interesting to examine the interactions of the edge-of-reach problem in settings with less accurate dynamics models. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
surprising failure with the true dynamics: 
 As described in Sections 1 and 2, prior works view offline model-based RL as online RL in an approximate dynamics model. Based on this understanding they propose various forms of dynamics uncertainty penalties to address model errors (see Table 7 ). This approach is described simply as: "two steps: (a) learning a pessimistic MDP (P-MDP) using the offline dataset; (b) learning a near-optimal policy in this P-MDP" [14] . This shared base procedure is shown in Algorithm 1, where "P-MDP" refers to the learned dynamics model with a penalty added to address model errors.This assumption of issues being due to dynamics model errors naturally leads to the belief that the ideal case would be to have a perfect error-free dynamics model. However, in Figure 1 and Table 1 we demonstrate that, if the learned dynamics model is replaced with the true dynamics, all dynamics-penalized methods completely fail on most environments. 5 Note that under the true dynamics, all existing dynamics penalty-based methods assume no intervention is needed since there are no model errors. As a result, their penalties all become zero (see Table 7 ) and they all collapse to exactly the same procedure. Therefore the results shown in Table 1 indicate the failure of all existing dynamics-penalized methods. In the following section, we investigate why having perfectly accurate dynamics leads to failure.The Edge-Of-Reach Hypothesis (Illustrated in Figure 2 )Failure under the error-free dynamics reveals that "model errors" cannot explain all the problems in offline model-based RL. Instead, it highlights that there must be a second issue. On investigation we find this to be the "edge-of-reach problem." We begin with the main intuition: in offline model-based RL, synthetic data D rollouts is generated as short k-step rollouts starting from states in the original dataset D offline (see Algorithm 1). Crucially, Figure 2 (left) illustrates how, under this procedure, there can exist some states which can be reached in the final rollout step, but which cannot -under any policy -be reached earlier. These edge-of-reach states triggers a breakdown in learning, since even with the ability to collect unlimited data, the agent is never able to reach these states 'in time' to try actions from them, and hence is free to 'believe that these edge-of-reach states are great.' More concretely: Figure 2 (right) illustrates how being sampled only at the final rollout step means edge-of-reach states will appear in the resulting dataset D rollouts as nextstates s ′ , but will never appear in the dataset as states s. Crucially, in Equation ( 1), we see that values at states s are updated, while values at s ′ are used for the targets of these updates. Edge-of-reach states are therefore used for targets, but are never themselves updated, meaning that their values can be arbitrarily misestimated. Updates consequently propagate misestimation over the entire state-action space. Furthermore, the max operation in the policy improvement step (see Section 2) exploits any misestimation by picking out the most heavily overestimated values [23] . The misestimation is therefore turned into overestimation, resulting in the value explosion seen in Figure 3 . Thus, contrary to common understanding, the out-of-sample action problem key in model-free offline RL can be seen to persist in model-based RL.Edge-of-Reach Hypothesis: Limited horizon rollouts from a fixed dataset leads to 'edge-ofreach' states: states which are used as targets for Bellman-based updates, but which are never themselves updated. The resulting maximization over misestimated values in Bellman updates causes pathological value overestimation and a breakdown in Q-learning.When is this an issue in practice? Failure via this mode requires (a) the existence of edge-of-reach states, and (b) such edge-of-reach states to be sampled. The typical combination of k ≪ H along with a limited pool of starting states (s ∈ D offline ) means the rollout distribution is unlikely to sufficiently cover the full state space S, thus making (a) likely. Moreover, we observe pathological 'edge-of-reach seeking' behavior in which the agent appears to 'seek out' edge-of-reach states due to this being the source of overestimation, thus making (b) likely. This behaviour is discussed further in Section 4.In Appendix A, we give a thorough and unified view of model-free and model-based offline RL, dividing the problem into independent conditions for states and actions and examining when we can expect the edge-of-reach problem to be significant.Definition 1 (Edge-of-reach states). Consider a deterministic transition model T : S × A → S, rollout length k, and some distribution over starting states ν 0 . For some policy π : S → A, rollouts are then generated according to s 0 ∼ ν 0 (•), a t ∼ π(•|s t ) and s t+1 ∼ T (•|s t , a t ) for t = 0, . . . , k -1, giving (s 0 , a 0 , s 1 , . . . , s k ). Let us use ρ t,π (s) to denote the marginal distributions over s t .We define a state s ∈ S edge-of-reach with respect to (T , k, ν 0 ) if: for t = k, ∃ π s.t. ρ t,π (s) > 0, but, for t = 1, . . . , k -1 and ∀ π, ρ t,π (s) = 0. In our case, ν 0 is the distribution of states in D offline .In Appendix B, we include an extension to stochastic transition models, proof of how errors can consequently propagate to all states, and a discussion of the practical implications.There are two potential issues in offline model-based RL: (1) dynamics model errors and subsequent model exploitation, and (2) the edge-of-reach problem and subsequent pathological value overestimation. We ask: Which is the true source of the issues observed in practice? While (1) is stated as the sole issue and hence the motivation in prior methods, the "failure" results presented in Figure 1 and Table 1 are strongly at odds with this explanation. Furthermore, in Table 8 , we examine the model rewards sampled by the agent over training. For (1) to explain the Q-values seen in Figure 3 , the sampled rewards in Table 8 would need to be on the order of 10 8 . Instead, however, they remain less than 10, and are not larger than the true rewards, meaning (1) does not explain the observed value explosion (see Appendix D.2). By contrast, the edge-of-reach-induced pathological overestimation mechanism of (2) exactly predicts this value explosion, with the observations very closely resembling those of the analogous model-free out-of-sample problem [18] . Furthermore, (2) is consistent with the "failure" observations in Table 1 . Finally, we again highlight the discussion in Section 3.2 where we explain why the edge-of-reach problem can be expected to occur in practice. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: We have one proposition in Appendix B. For this we clearly state all the assumptions, provide a full proof, and also include a discussion of the practical implications and limitations. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
analysis with a simple environment: 
 In the previous section, we presented the edge-of-reach hypothesis as an explanation of why existing methods fail under the true dynamics. In this section, we construct a simple environment to empirically confirm this hypothesis. We first reproduce the observation seen in Figures 1 and 3 and Table 1 of the base offline model-based procedure resulting in exploding Q-values and failure to learn despite using the true dynamics model. Next, we verify that edge-of-reach states are the source of this problem by showing that correcting value estimates only at these states is sufficient to resolve the issues.We isolate failure observed in Section 3.1 with the following setup: Reward is defined as in Figure 4a , the transitions function is simply s' = s + a ∈ R 2 , and initial states (analogous to s ∈ D offline ) are sampled from µ 0 = U ([-2, 2] 2 ) (the area shown in the navy blue box Figure 4 ).In applying the offline model-based procedure (see Algorithm 1) to this (true) environment we have precisely the same setup as in Section 3.1, where again the only difference compared to online RL is the use of truncated k-step rollouts with k = 10 (compared to full horizon H = 30). This small change results in the existence of edge-of-reach states (those between the red and orange boxes).Exactly as with the benchmark experiments in Section 3.1, the base model-based procedure fails despite using the true dynamics (see blue Figure 4 , Base), and the Q-values grow unboundedly over training (compare Figure 4f and Figure 3b ).Looking at the rollouts sampled over training (see Figure 7 ) we see the following behavior:(Before 25 epochs) Performance initially increases. (Between 25 and 160 epochs) Value misestimation takes over, and the policy begins to aim toward unobserved state-actions (since their values can be misestimated and hence overestimated). (After 160 epochs) This 'edge-of-reach seeking' behavior compounds with each epoch, leading the agent to eventually reach edge-of-reach states. From this point onwards, the agent samples edge-of-reach states at which it never receives any corrective feedback. The consequent pathological value overestimation results in a complete collapse in performance. In Figure 4b we visualize the final policy and see that it completely ignores the reward function, aiming instead towards an arbitrary edge-of-reach state. Our hypothesis is that the source of this failure is value misestimation at edge-of-reach states. Our Base-OraclePatch experiments (see yellow Figure 4 ) verify this by showing that patching in the correct values solely at edge-of-reach states is sufficient to completely solve the problem. This is particularly compelling as in practice we only corrected values at 0.4% of states over training. In Section 5 we introduce our practical method RAVL, which Figures 4 green and 7 show has an extremely similar effect to that of the ideal but practically impossible Base-OraclePatch intervention.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We provide complete details of hyperparameters and training settings in Appendix C. For all the analysis experiments in particular, we aim for the reader to understand exactly what we have done such that they can appreciate the results fully. As such we have described them in detail (for example Section 4), including further clarification of the terms we have used in Table 12 .Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
ravl: reach-aware value learning: 
 As verified in Section 4.3, issues stem from value overestimation at edge-of-reach states. To resolve this, we therefore need to (A) detect and (B) prevent overestimation at edge-of-reach states.(A) Detecting edge-of-reach states: As illustrated in Figure 2 (right), edge-of-reach states are states at which the Q-values are never updated, i.e., those that are out-of-distribution (OOD) with respect to the training distribution of the Q-function, s ∈ D rollouts . A natural solution for OOD detection is measuring high variance over a deep ensemble [22] . We can therefore detect edge-of-reach states using an ensemble of Q-functions. We demonstrate that this is effective in Figure 5 .(B) Preventing overestimation at these states: Once we have detected edge-of-reach states, we may simply apply value pessimism methods from the offline model-free literature. Our choice of an ensemble for part (A) conveniently allows us to minimize over an ensemble of Q-functions, which effectively adds value pessimism based on ensemble variance.Our resulting proposal is Reach-Aware Value Learning (RAVL). Concretely, we take the standard offline model-based RL procedure (see Algorithm 1), and simply exchange the dynamics pessimism penalty for value pessimism using minimization over an ensemble of N Q-functions:EQUATIONWe include EDAC's [1] ensemble diversity regularizer, and in Section 7 we discuss how the impact of value pessimism differs significantly in the model-based (RAVL) vs model-free (EDAC) settings. 
empirical evaluation: 
 In this section, we begin by analyzing RAVL on the simple environment from Section 4. Next, we look at the standard D4RL benchmark, first confirming that RAVL solves the failure seen with the true dynamics, before then demonstrating that RAVL achieves strong performance with the learned dynamics. In Appendix F, we include additional results on the challenging pixel-based V-D4RL benchmark on which RAVL now represents a new state-of-the-art. Finally in Section 6.5 we reexamine prior model-based methods and explain why they may work despite not explicitly addressing the edge-of-reach problem. We provide full hyperparameters in Appendix C.2 and ablations showing that RAVL is stable over hyperparameter choice in Appendix G.2. Testing on the simple environment from Section 4 (see green Figure 4 ), we observe that RAVL behaves the same as the theoretically optimal but practically impossible Base-OraclePatch method. Moreover, in Figure 5 , we see that the Q-value variance over the ensemble is significantly higher for edge-of-reach states, meaning RAVL is detecting and penalizing edge-of-reach states exactly as intended.Next, we demonstrate that RAVL works without dynamics uncertainty and solves the 'failure' observed in Section 3.1. Table 1 shows results on the standard offline benchmark D4RL [7] MuJoCo [34] v2 datasets with the true (zero error, zero uncertainty) dynamics. We see that RAVL learns the near-optimal policies, while existing methods using the base model-based procedure (Algorithm 1) completely fail. In Section 6.5 we examine how this is because existing methods overlook the critical edge-of-reach problem and instead only accidentally address it using dynamics uncertainty metrics.In the absence of model uncertainty, these methods have no correction for edge-of-reach states and hence fail dramatically. Next, we show that RAVL also performs well with a learned dynamics model. Figure 3 shows RAVL successfully stabilizes the Q-value explosion of the base procedure, and Table 2 shows RAVL largely matches SOTA, while having significantly lower runtime (see Section 6.4). RAVL gives much higher performance on the Halfcheetah mixed and medium datasets than its model-free counterpart EDAC, and in Section 6.5, we discuss why we would expect the effect of value pessimism in the model-based setting (RAVL) to inherently offer much more flexibility than in the model-free setting (EDAC). We additionally include results for the challenging pixel-based V-D4RL benchmark for which latentspace models are used (in Appendix F), and accompanying ablation experiments (in Appendix G.2). In this setting, RAVL represents a new SOTA, giving a performance boost of more than 20% for some environments. These results are particularly notable as the pixel-based setting means the base algorithm (DreamerV2) uses model rollouts in an imagined latent space (rather than the original state-action space as in MBPO). The results therefore give promising evidence that RAVL is able to generalize well to different representation spaces.Vectorized ensembles can be scaled with extremely minimal effect on the runtime (see Table 10 ). This means that, per epoch, RAVL is approximately 13% faster than the SOTA (MOBILE, due to MOBILE needing multiple extra forward passes to compute its uncertainty penalty). Furthermore, in total, we find that RAVL reliably requires 3× fewer epochs to converge on all but the medexp datasets, meaning the total runtime is approximately 70% faster than SOTA (see Appendix G.1).While ostensibly to address model errors, we find that existing dynamics penalties accidentally address the edge-of-reach problem: In Figure 6 we see a positive correlation between the penalties used in dynamics uncertainty methods and RAVL's effective penalty of value ensemble variance. This may be expected, as dynamics uncertainty will naturally be higher further away from D offline , which is also where edge-of-reach states are more likely to lie. In Section 3.4 we present evidence suggesting that the edge-of-reach problem is likely the dominant source of issues on the main D4RL benchmark, thus indicating that the dynamics uncertainty penalties of existing methods are likely indirectly addressing the edge-of-reach problem. In general, dynamics errors are a second orthogonal problem and RAVL can be easily combined with appropriate dynamics uncertainty penalization [24, 29, 36] for environments where this is a significant issue.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: For the benchmark experiments we have provided full detail of the hyperparameters and training settings used in Appendix C. Note also, that we have made sure to match all the default settings exactly in line with the base procedure of other papers in this field to allow for meaningful comparison. For each of the analysis experiments, we have aimed for the reader to completely understand the exact setup such that they can fully appreciate the results. As such we have described the setup of each in detail.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. We have read and adhere to the code of ethics. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: The paper discusses the general potential impact of offline RL in Section 1.In terms of societal impact, the paper is on simulated control and we do not believe it has relevance in the way meant by this question. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
related work: 
 Model-Based Methods. Existing offline model-based methods present dynamics model errors and consequent model exploitation as the sole source of issues. A broad class of methods therefore propose reward penalties based on the estimated level of model uncertainty [2, 14, 24, 29, 36] , typically using variance over a dynamics ensemble. Rigter et al. [27] aim to avoid model exploitation by setting up an adversarial two-player game between the policy and model. Finally, most related to our method, COMBO [37] , penalizes value estimates for state-actions outside model rollouts. However, similarly to Yu et al. [36] , COMBO is theoretically motivated by the assumption of infinite horizon model rollouts, which we show overlooks serious implications. Critically, in contrast to our approach, none of these methods address the edge-of-reach problem and thus they fail as environment models become more accurate. A related phenomenon of overestimation stemming from hallucinated states has been observed in online model-based RL [12] .Offline Model-Free Methods. Model-free methods can broadly be divided into two approaches to solving the out-of-sample action problem central to offline model-free RL (see Section 2): action constraint methods and value pessimism-based methods. Action constraint methods [8, 17, 18] aim to avoid using out-of-sample actions in the Bellman update by ensuring selected actions are close to the dataset behavior policy π β . By contrast, value pessimism-based methods aim to directly regularize the value function to produce low-value estimates for out-of-sample state-actions [1, 16, 19] . The edge-of-reach problem is the model-based equivalent of the out-of-sample action problem, and this unified understanding allows us to transfer ideas directly from model-free literature. RAVL is based on EDAC's [1] use of minimization over a Q-ensemble [9, 35] , and applies this to model-based offline RL.What is the effect of value pessimism in the model-based vs model-free settings?EDAC [1] can be seen as RAVL's model-free counterpart, however, the impact of value pessimism in the model-free vs the model-based settings is notably different. Recall that, with the ensemble Q-function trained on (s, a, r, s ′ ) ∼ D □ , the state-actions that are penalized (due to being outside the training distribution) are any (s ′ , a ′ ) that are out-of-distribution with respect to the (s, a)'s in the dataset D □ . Recall also that updates use values at (s ′ , a ′ ) where a ′ ∼ π θ (see Equation ( 1)).In the model-free case (EDAC): The dataset is D □ = D of f line , meaning the actions a are effectively sampled from the dataset behavior policy (off-policy), whereas the actions a ′ are sampled on-policy. This means that EDAC penalizes any (s ′ , a ′ ) where a ′ differs significantly from the behavior policy.In the model-based case (RAVL): The dataset is D □ = D rollouts , meaning now both a and a ′ are sampled on-policy. As a result, the only (s ′ , a ′ ) which will now be out-of-distribution with respect to (s, a) (and hence penalized) are those where the state s ′ is out-of-distribution with respect to the states s in D rollouts . This happens when s ′ is reachable only in the final step of rollouts, i.e. exactly when s ′ is "edge-of-reach" (as illustrated in Figure 2 ). Compared to EDAC, RAVL can therefore be viewed as "relaxing" the penalty and giving the agent freedom to learn a policy that differs significantly from the dataset behavior policy. This distinction is covered in detail in Appendix A. 
conclusion: 
 This paper investigates how offline model-based methods perform as dynamics models become more accurate. As an interesting hypothetical extreme, we test existing methods with the true error-free dynamics. Surprisingly, we find that all existing methods fail. This reveals that using truncated rollout horizons (as per the shared base procedure) has critical and previously overlooked consequences stemming from the consequent existence of 'edge-of-reach' states. We show that existing methods are indirectly and accidentally addressing this edge-of-reach problem (rather than addressing model errors as stated), and hence explain why they fail catastrophically with the true dynamics.This problem reveals close connections between model-based and model-free approaches and leads us to present a unified perspective for offline RL. Based on this, we propose RAVL, a simple and robust method that achieves strong performance across both proprioceptive and pixel-based benchmarks. Moreover, RAVL directly addresses the edge-of-reach problem, meaning that -unlike existing methods -RAVL does not fail under the true environment model, and has the practical benefit of not requiring dynamics uncertainty estimates. Since improvements to dynamics models are inevitable, we believe that resolving the brittle and unanticipated failure of existing methods under dynamics model improvements is an important step towards 'future-proofing' offline RL. 
limitations: 
 Since dynamics models for the main offline RL benchmarks are highly accurate, the edge-of-reach effects dominate, and RAVL is sufficient to stabilize model-based training effectively without any explicit dynamics uncertainty penalty. In general, however, edge-of-reach issues could be mixed with dynamics error, and understanding how to balance these two concerns would be useful future work. Further, we believe that studying the impact of the edge-of-reach effect in a wider setting could be an exciting direction, for example investigating its effect as an implicit exploration bias in online RL.A A Unified Perspective of Model-Based and Model-Free RLWe supplement the discussion in Section 3.2 with a more thorough comparison of the out-of-sample and edge-of-reach problems, including how they relate to model-free and model-based approaches. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risk, we do not release any models or datasets. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Consider a dataset of transition tuples D = {(s i , a i , r i , s ′ i , d i )} i=1,...,N collected according to some dataset policy π D (•|s). Compared to Section 2, we include the addition of a done indicator d i , where d i = 1 indicates episode termination (and d i = 0 otherwise). Transition tuples thus consist of state, action, reward, nextstate, done. Consider the marginal distribution over state-actions ρ D s,a (•, •), over states ρ D s (•), and conditional action distribution ρ D a|s (•|s). Note that ρ D a|s (•|s) = π D (•|s). We abbreviate x is in distribution with respect to ρ as x ∈ dist ρ.As described in Section 2, given some policy π, we can attempt to learn the corresponding Q-function with the following iterative process:EQUATIONQ-learning relies on bootstrapping, hence to be successful we need to be able to learn accurate estimates of the Bellman targets for all (s, a) inputs. Bootstrapped estimates of Q(s ′ , a ′ ) are used in the targets whenever d ̸ = 1. Therefore, for all (s ′ , a ′ ), we require:Combined state-action condition: (s ′ , a ′ ) ∈ dist ρ D s,a or d = 1.In the main paper, we use this combined state-action perspective for simplicity, however, we can equivalently divide this state-action condition into independent requirements on the state and action as follows:State condition: s ′ ∈ dist ρ D s or d = 1, Action condition: a ′ ∈ dist ρ Da|s (s ′ ) (given the above condition is met and d ̸ = 1).Informally, the state condition may be violated if D consists of partial or truncated trajectories, and the action condition may be violated if there is a significant distribution shift between π D and π.In offline model-free RL, D = D offline , with π D = π β . For the settings we consider, D offline consists of full trajectories and therefore will not violate the state condition. However, this may happen in a more general setting with D offline containing truncated trajectories. By contrast, the mismatch between π (used to sample a ′ in Q-learning) and π β (used to sample a in the dataset D offline ) often does lead to significant violation of the action condition. This exacerbates the overestimation bias in Q-learning (see Section 7) , and can result in pathological training dynamics and Q-value explosion over training [18] .On the other hand, in offline model-based RL, the dataset D = D rollouts is collected on-policy according to the current (or recent) policy such that π D ≈ π. This minimal mismatch between π D and π means the action condition is not violated and can be considered to be resolved due to the collection of additional data. However, the procedure of generating the data D = D rollouts can be seen to significantly exacerbate the state condition problem, as the use of short truncated-horizon trajectories means the resulting dataset D rollouts is likely to violate the state condition. Due to lack of exploration, certain states may temporarily violate the state condition. Our paper then considers the pathological case of edge-of-reach states, which will always violate the state condition. Proposition 1 (Error propagation from edge-of-reach states). Consider a rollout of length k, (s 0 , a 0 , s 1 , . . . , s k ). Suppose that the state s k is edge-of-reach and the approximate value function Q j (s k , π(s k )) has error ϵ. Then, standard value iteration will compound error γ k-t ϵ to the estimates of Q j+1 (s t , a t ) for t = 1, . . . , k -1. (Proof in Appendix B.)In this section, we provide a proof of Proposition 1. Our proof follows analogous logic to the error propagation result of Kumar et al. [18] .Proof. Let us denote Q * as the optimal value function, ζ j (s, a) = |Q j (s, a) -Q * (s, a)| the error at iteration j of Q-Learning, and δ j (s, a) = |Q j (s, a) -T Q j-1 (s, a)| the current Bellman error. Then first considering the t = k -1 case,EQUATIONThus the errors at edge-of-reach states are discounted and then compounded with new errors at Q j (s k-1 , a k-1 ). For t < k -1, the result follows from repeated application of Equation ( 7) along the rollout.With stochastic transition models (e.g. Gaussian models), we may have the case that no state will truly have zero density, in which case we relax the definition of edge-of-reach states slightly (see Section 3.3) from ρ t,π (s) = 0 to ρ t,π (s) < ϵ for some small ϵ.During optimization, in practice, model rollouts are sampled in minibatches and thus the above error propagation effect will occur on average throughout training. An analogous model-free statement has been given in Kumar et al. [18] ; however, its significance in the context of model-based methods was previously not considered.In this section, we provide full implementation details for RAVL.We use the base model-based procedure as given in Algorithm 1 and shared across model-based offline RL methods [14, 24, 29, 36] . This involves using short MBPO-style [13] model rollouts to train an agent based on SAC [11] . We modify the SAC agent with the value pessimism losses of EDAC [1] . Our dynamics model follows the standard setup in model-based offline algorithms, being realized as a deep ensemble [5] and trained via maximum likelihood estimation.For the D4RL [7] MuJoCo results presented in Table 2 , we sweep over the following hyperparameters and list the choices used in Table 4 . "Base" refers to the shared base procedure in model-based offline RL shown in Algorithm 1.• The remaining model-based and agent hyperparameters are given in Table 5 . Almost all environments use a small N critic = 10, with only the Hopper-medexp dataset needing N critic = 30.For the uncertainty-free dynamics model experiments in Table 1 we take the same hyperparameters as for the learned dynamics model for N , k, and r, and try different settings for η ∈ {1, 10, 100, 200}.The choices used are shown in Table 4 . Note that for existing methods based on dynamics penalties, the analogous hyperparameter (penalty weighting) has no effect under the true dynamics model as the penalties will always be zero (since dynamics uncertainty is zero). For the experiments in Figure 1 with intermediate dynamics model accuracies, we tune the ensemble diversity coefficient η over η ∈ {1, 10, 100, 200} for RAVL, and analogously we tune the uncertainty penalty coefficient λ over λ ∈ {1, 5, 10, 100, 200} for MOPO.Our implementation is based on the Clean Offline Reinforcement Learning (CORL, Tarasov et al. [33] ) repository, released at https://github.com/tinkoff-ai/CORL under an Apache-2.0 license. Our algorithm takes on average 6 hours to run using a V100 GPU for the full number of epochs. For the V-D4RL [25] DeepMind Control Suite datasets presented in Table 9 , we use the default hyperparameters for the Offline DV2 algorithm, which are given in Table 6 . We found keeping the uncertainty weight λ = 10 improved performance over λ = 0 which shows RAVL can be combined with dynamics penalty-based methods. We used a hyperparameter sweep over {2, 5, 20} for N critic but found a single value of 5 worked well for all environments we consider.For completeness, in Table 7 we include the expressions for the various dynamics penalties proposed by the most popular model-based approaches. These penalties are all based on estimates of dynamics model uncertainty. Since there is no uncertainty under the true model, all these penalties collapse to zero with the true dynamics. [14] max 1≤i,j≤N ||µ i ϕ (s, a) -µ j ϕ (s, a)||2 0 MOBILE [29] Stdi=1,...,N { T i (s, a)} 0In Figure 3 we observe that the base offline model-based procedure results in Q-values exploding unboundedly to beyond 10 10 over training. Existing methods attribute this to dynamics model errors and the consequent exploitation of state-action space areas where the model incorrectly predicts high reward.In Table 8 we examine the rewards collected by an agent in the learned dynamics model. Under the 'model errors' explanation, we would expect to see rewards sampled by the agent in the learned model to be significantly higher than under the true environment. Furthermore, to explain the Q-values being on the order of 10 10 , with γ = 0.99 the agent would need to sample rewards on the order of 10 8 . However, the per-step rewards sampled by the agent are on the order of 10 0 (and are not higher than with the true environment).Model errors and model exploitation therefore cannot explain the explosion in Q-values seen over training. By contrast, the edge-of-reach problem exactly predicts this extreme value overestimation behavior. In Figure 6 , we plot the dynamics uncertainty-based penalty used in MOPO [36] against the effective penalty in RAVL of variance of the value ensemble. The positive correlation between MOPO's penalty and the edge-of-reach states targeting penalty of RAVL indicates why prior methods may work despite not considering the crucial edge-of-reach problem. In Figure 7 , we provide a visualization of the rollouts sampled over training in the simple environment for each of the algorithms analyzed in Figure 4 (see Section 4). This accompanies the discussion in Section 4.2 of the behavior over training. F Evaluation on the Pixel-Based V-D4RL BenchmarkThe insights from RAVL also lead to improvements in performance on the challenging pixel-based benchmark V-D4RL [25] . The base procedure in this pixel-based setting uses the Offline DreamerV2 algorithm of training on trajectories in a learned latent space.We observe in Table 9 that RAVL gives a strong boost in performance on the medium and medexp level datasets, while helping less in the more diverse random and mixed datasets. This observation fits with our intuition of the edge-of-reach problem: the medium and medexp level datasets are likely to have less coverage of the state space, and thus we would expect them to suffer more from edge-of-reach issues and the 'edge-of-reach seeking behavior' demonstrated in Section 4.2. The performance improvements over Offline DreamerV2 and other model-based baselines including LOMPO [26] therefore suggests that the edge-of-reach problem is general and widespread.We also note that these results are without the ensemble diversity regularizer from EDAC [1] (used on the D4RL benchmark), which we anticipate may further increase performance. We compare RAVL to SOTA method MOBILE. The additional requirement for RAVL compared to MOBILE is that RAVL requires increasing the Q-ensemble size: from N = 2 in MOBILE to N = 10 in RAVL (for all environments except one where we use N = 30). The additional requirement for MOBILE compared to RAVL is that MOBILE requires multiple additional forwards passes through the model for each update step (for computing their dynamics uncertainty penalty).In Table 10 we show that ensemble size can be scaled with very minimal effect on the runtime. Even increasing the ensemble size to N = 100 (far beyond the maximum ensemble size used by RAVL) only increases the runtime by at most 1%. In Table 11 we include results with different settings of RAVL's ensmble diversity regularizer hyperparameter η. We note a minimal change in performance with a sweep across two orders of magnitude. This is an indication that RAVL may be applied to new settings without much tuning.For RAVL's second hyperparameter of Q-ensemble size N , we note that in the main benchmarking of RAVL against other offline methods (see Table 2 ) we use the same value of N = 10 across all except one environment. This again indicates that RAVL is robust and should transfer to new settings with minimal hyperparameter tuning. Throughout the paper, we compare several different setups in order to identify the true underlying issues in model-based offline RL. We provide a summary of them in Table 12 . More comprehensive descriptions of each are given in the relevant positions in the main text and table and figure captions. Justification: The abstract and introduction clearly define the scope of the paper and its contributions. For our main result (of fixing the failure under the true dynamics model) we provide full results over the standard D4RL benchmark [7] with 6 seeds (see Table 1 ). For our central hypothesis, we include extensive analysis, both on the standard D4RL benchmark (Section 3.4) and through the construction of a simple environment (Section 4). Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Our code is open-sourced at: github.com/anyasims/edge-of-reach.This is also referred to as the out-of-distribution action[18] or action distribution shift problem[19].This step contains an implicit max operation which we refer to in Section 3.2.We use the terms 'out-of-sample' and 'out-of-distribution' interchangeably, as is done in the literature.For clarity, this change (Approximate→True) is described in Algorithm 1 as pseudocode, and also in Appendix H in terms of how it relates to other experiments presented throughout the paper.Note that the times reported for EDAC in Tableof MOBILE are not representative of practice, since these are with a non-vectorized Q-ensemble (confirmed from their open-source code), which would require just a simple change to 5 lines of code. 


Decision: This paper explores the so-called "edge-of-reach" problem in offline model-based RL, which arises when using short, truncated rollouts to generate synthetic data. Contrary to prevailing theories, the paper finds that improving the learned dynamics model does not necessarily enhance performance; in fact, using an error-free dynamics model can lead to the complete failure of existing methods. This failure is due to the existence of "edge-of-reach" states, where value estimates become unreliable because these states are only encountered at the end of rollouts and not during intermediate steps. To address this issue, the paper proposes Reach-Aware Value Learning (RAVL), a method that directly targets the edge-of-reach problem, thereby preventing fatal value overestimation resulting in the collapse of performance. RAVL demonstrates robust performance even as dynamics models improve, offering a more future-proof approach to offline RL.

The reviews are overall positive, and thus recommend to accept the paper. Authors are encouraged to incorporate additional results provided during the dicussion phase.

Reviews: review: paper_summary: paper_summary: The paper investigates a phenomena in model-based offline RL that most existing algorithms fail when they are given the perfect model of the environment. Through multiple experiments, the paper argues the failure is due to the wrong estimated values at the states in the end of model rollouts. These states are called edge of reach and are used for targets of Q values but are never trained themselves. The paper continues to propose a fix in the form of pessimistic target value estimate using ensemble Q networks.
strength_weakness: strength_weakness: **1)** The paper could benefit from more investigations on the necessity of the _detection_ of an edge-of-reach state. The current proposed value learning only uses the pessimistic Q target when it decides that the state is an edge-of-reach state. 

**Q1)** I wonder whether if this distinction is necessary and if we could always use the pessimistic value and get the similar results.

I believe this could be doable because in states with low variance, the pessimistic value will not defer from the normal one anyway. Potentially, this removes a hyperparameter which sets the threshold for the choice of value target. It also reduces the number of ensemble networks needed for the algorithm.

**2)** The paper could better position itself in the literature. The pessimistic value target is a ubiquitous trick in RL to overcome value estimation. See for example TD3 [1] paper for actor critic methods. The issue where models query Q networks in untrained states has also been pointed out in MBRL research. See for example [2]. 

**3)** It probably is out of the scope of the paper, but I wonder how much of this issue is shared with online MBRL algorithms. They also usually only create rollouts of a limited length. A discussion on this matter will be welcome.

**4)** It could be a personal matter, but the paper has too many future referencing. For example lines 160, 220, 231, 253, 260, 261. I find these confusing as a reader. Jumping to read the future section while reading is not ideal or even possible. Also when the future section is being read, the connection made in the past sections to the current one is probably forgotten. I recommend reducing these references and only make such connections when necessary. A general comment that more discussions about a topic will appear in the future could be enough for the reader without asking them to jump. 


[1] Fujimoto, Scott, Herke Hoof, and David Meger. "Addressing function approximation error in actor-critic methods." International conference on machine learning. PMLR, 2018.
[2] Jafferjee, Taher, et al. "Hallucinating value: A pitfall of dyna-style planning with imperfect environment models." arXiv preprint arXiv:2006.04363 (2020).
questions: questions: See Q1, and weakness 3 above.
limitations: limitations: I find the limitation discussions sufficient.
paper_summary: paper_summary: This paper identifies and investigates a previously neglected issue in offline model-based RL called the "edge-of-reach problem", which is due to the truncated rollouts used in model-based RL to mitigate the compounding errors of the dynamics model. The authors proposed Reach-Aware Value Learning (RAVL) to address the edge-of-reach problem using value pessimism. They show RAVL's strong performance on the D4RL and V-D4RL benchmarks.
strength_weakness: strength_weakness: 1. The proposed method doesn't outperform existing baselines according to the reported results on D4RL. Although it's claimed that "RAVL can be easily combined with appropriate dynamics uncertainty penalization", it's not supported by any empirical evidences. Hence, it's unclear how well RAVL work in general when combined with other model-based methods with design choices orthogonal to value pessimism.
2. I found the paper a bit hard to read due to the formatting, e.g. too many long italics phrases.
questions: questions: 1. Why does RAVL not benefit from reduced dynamics errors according to table 1?
2. It would be interesting to show how RAVL's performance change by varying the accuracy of the dynamics model, similar to figure 1.
limitations: limitations: Yes
paper_summary: paper_summary: The paper presents a novel analysis of the challenges faced by offline model-based reinforcement learning (RL) when the dynamics model becomes increasingly accurate. The central thesis is that existing methods fail under true, error-free dynamics due to a previously unaddressed issue termed the "edge-of-reach" problem. The authors provide a thorough investigation, theoretical analysis, and empirical evidence to support their claims. They propose Reach-Aware Value Learning (RAVL), a method that directly addresses this problem, demonstrating robust performance across benchmarks.
strength_weakness: strength_weakness: The methodology contribution of this paper is limited. While I assume that the "edge-of-reach" problem is significant to the current offline RL literature, the proposed method by the authors fails to demonstrate obvious superiority compared with the SOTA offline model-free or model-based methods (see Table 2 of this paper). One reason, I guess, is that the D4RL benchmark is too simple to underscore the superiority of addressing the "edge-of-reach" problem. On the other hand, I expect that authors can validate their method on more complex and challenging benchmarks and provide more convincing results. I am willing to reconsider my score after seeing this.
questions: questions: 1. I wonder how existing uncertainty-penalty-based offline model-based RL methods perform if I gradually increase the length of rollouts in the estimated dynamics model along with the improved accuracy of the model (a very intuitive trick).

2. Does the "edge-of-reach" hypothesis exist in many online RL scenarios? For example, researchers usually early stop the trajectory according to a pre-defined maximal length of the trajectory in the Gym Mujoco benchmark.
limitations: limitations: The authors have discussed the limitations of their work properly.
paper_summary: paper_summary: This paper identifies an overlooked problem in offline model-based reinforcement learning (RL).
Offline model-based RL approaches commonly perform online RL inside a learned dynamics model and
due to model errors, they generally use truncated $k$ step rollouts, instead of full trajectories (episodes).
In this paper, the authors replace the learned dynamics model with the true simulator and show that offline model-based
RL approaches fail even when they have access to the true dynamics.
This is due to states which can only be reached at the end of the $k$ step rollouts (which the authors coin "edge-of-reach" states) and thus, only appear in the target of the Q update. As such, the Q-function is never updated at these "edge-of-reach" states so any overestimation is gradually propagated over the entire state-action space.
The authors provide a nice diagram which provides intuition and shows the effect in a simple environment.
They then propose an algorithm which overcomes the so-called "edge-of-reach" problem by using an ensemble of Q functions.
They demonstrate good performance in D4RL and V-D4RL.
strength_weakness: strength_weakness: I have no major weaknesses. Nevertheless, I do have some comments which I provide to help improve the paper.
The main weakness of this paper is that it looks messy and overcrowded. There are multiple reasons for this which I will now detail.

The use of italics throughout the text makes the paper look very messy.
Sometimes the italics refer to questions whilst other times they just seem to be random words.
I would advise the authors to remove all italics.

There are too many subsections.
For example, in Section 6 there are 5 subsections and each subsection is a single paragraph.
I would advise the authors to replace \subsection with \paragraph.
Similarly, on Lines 283/293/303 the authors use unnumbered subsections. This looks very messy.
I suggest the authors replace this with \paragraph so that the text starts immediately after the bold text and does not have a line break before.

There are a lot of incorrect uses of colons, which makes the writing quality feel poor. Please remove the colons on lines 12/90/114/122/137/144/154/176/195/273/305.

The authors do not mention how the Q-function is initialized. I think it is very important to include all of this information. 
For example, if we initialize the Q-function to be zero at all inputs wouldn't that help prevent overestimation bias from "edge-of-reach" states?

## Minor comments and typos
- Line 89 - $\\}\_{i=1,\ldots,N}$ should be $\\}\_{i=1}^{N}$
- Line 122 - incorrect use of semicolon. This should just be "and".
- Line 137 - the full stop should be outside the quote.
- Line 167 - $0\ldots k-1$ should be $0,\ldots,k-1$
- Line 170 - Why is the last sentence of the definition not italicised?
- Line 249 - the comma should be outside the quote.
- Line 306/308/309/31 - the dataset subscript looks wrong.
- Sections shouldn't lead straight into subsections, e.g. 6 into 6.1. 
    - Add a few sentences explaining what you are introducing in this section.
questions: questions: - What is on the x-axis of Figure 5? What environment is this from? This is not clear from the text or caption.
limitations: limitations: The limitations are adequately discussed.

Comments: title: Rebuttal Acknowledgement
Please respond to author response if you haven't
comment: Thank you for your reply and for rasing your score. We really appreciate your support of our paper.
Thank you for your reply. We are deeply grateful for your continued support of our paper.
Thank you for the response and the additional experiments. I have increased my score.
I thank the authors for the rebuttal. I have read all the comments and I am maintaining my score.
Thank you for your reply, and again for your helpful feedback that has helped us improve our paper. We deeply appreciate you raising your score. Please let us know if you have any remaining concerns and if there is anything we can provide in the discussion period that would enable you to raise your score further.
Thank you for your response -- I appreciate all the explanations and additional results. I'm raising my score to 5.
Dear Reviewers,

Before the author-reviewer discussion period ends, please make sure that you have read the author responses, acknowledge that your review reflects them, and engage with the authors if necessary.

Thank you!
Thank you for the speedy response and running this extra experiment. I didn’t actually mean for you to run it, I just wanted to make you aware of the possible initialisation. But nevertheless, interesting to see that this also doesn’t help.

Hopefully the other reviewers engage soon and we can discuss things further.
Thank you for your reply. We had not considered this initialization. As you said, it would still not be expected to solve the problem as the overestimation mechanism described in our original answer still holds.

We have run some experiments to confirm this. The table below shows the final performance and $Q$-values for 3 seeds of Walker2D-medium with MBPO with and without this initialization method. As expected, this does not solve the problem and we see the same Q-value explosion and no significant difference between these initialization methods. (For reference, expert performance is $\sim 100$, and correct Q-values of the optimal policy are $\sim 5e+3$.)

|    *(3 seeds)*  |  Final Score |  Final $Q$-values  |
|:-:|:-:|:-:|
|Default initialization       | (-0.2, 2.3, -0.1) | (4e11, 2e12, 5e12) |
|Suggested zero initialization| (4.1, -0.3, -0.7) | (6e11, 3e13, 1e11) | 


We thank you again for your response and for your recommendation of acceptance for our paper!
I would like to thank the authors for considering my comments and providing a detailed rebuttal. I have also taken the time to read the other reviews and comments. I believe this paper should be accepted so I stand with my initial score of 7.


One minor comment regarding Q-function initialisation. Could you not just initialise the Q-function by misusing it from itself and freezing the extra network, like,
$$Q(s,a) = Q_{\theta}(s,a) - Q_{\theta_0}^{\text{frozen}}(s,a).$$
I appreciate this would probably not solve the issue, but this is what I meant when I suggested initialising the Q-function to be zero everywhere.