Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
Authors: Jihwan Jeong
Xiaoyu Wang
Michael Gimelfarb
Hyunwoo Kim
Baher abdulhai
Scott Sanner
Venue: ICLR 2023 poster (2023)
Keywords: Offline reinforcement learning
model-based reinforcement learning
model-based value expansion
Bayesian inference

Abstract: Offline reinforcement learning (RL) addresses the problem of learning a performant policy from a fixed batch of data collected by following some behavior policy. Model-based approaches are particularly appealing in the offline setting since they can extract more learning signals from the logged dataset by learning a model of the environment. However, the performance of existing model-based approaches falls short of model-free counterparts, due to the compounding of estimation errors in the learned model. Driven by this observation, we argue that it is critical for a model-based method to understand when to trust the model and when to rely on model-free estimates, and how to act conservatively w.r.t. both. To this end, we derive an elegant and simple methodology called conservative Bayesian model-based value expansion for offline policy optimization (CBOP), that trades off model-free and model-based estimates during the policy evaluation step according to their epistemic uncertainties, and facilitates conservatism by taking a lower bound on the Bayesian posterior value estimate. On the standard D4RL continuous control tasks, we find that our method significantly outperforms previous model-based approaches: e.g., MOPO by $116.4$%, MOReL by $23.2$% and COMBO by $23.7$%. Further, CBOP achieves state-of-the-art performance on $11$ out of $18$ benchmark datasets while doing on par on the remaining datasets.

Published as a conference paper at ICLR 2023
CONSERVATIVE BAYESIAN MODEL-BASED VALUE EXPANSION FOR OFFLINE POLICY OPTIMIZATION
Jihwan Jeong" 1:3 Xiaoyu W Gimelfarb 13, Hyunwoo Kim Baher Abdulhai! & Scott Sanner
'University of Toronto, 'LG Al Research, 'Vector Institute
{jihwan jeong, enxiaoyu. wang, mike. mwk...imella.aa................a waw.egaaaeaaaeeaiaaaea.........a wtt.rrarooaa.oooooooooooooooooo. wtw.onto..cooooo ca,
hwkimě grenelarreehhhhh ai, baher baber.abdulhai www.hhhhaaettooottooooom ssannerêmie utoronto.co
ABSTRACT
Offline reinforcement learning (RL) addresses the problem of learning a perfor- mant policy from fixed batch of data collected by following some behavior policy. Model-based approaches are particularly appealing in the offline setting since they can extract more learning signals from the logged dataset by learning a model of the environment. However, the performance of existing model-basee approaches falls short of model-free counterparts, due to the compounding of esti- mation errors in the learned model. Driven by this observation, we argue that it is critical for model-based method to understand when to trust the model and when to rely on model-free estimates, and how to act conservatively w.r.t. both. To this end, derive elegant and simple methodology called conservative Bayesian model-based value expansion for offline policy optimization (CBOP), that trades off model-free and model-based estimates during the policy evaluation step ac- cording to their epistemic uncertainties, and facilitates conservatism by taking a lower bound on the Bayesian posterior value estimate. On the standard D4RL continuous control tasks, we find that our method significantly outperforms pre- vious model-based approaches: e.g., MOPO by 116.4%, MOReL by 23.2% and COMBO by 23.7%. Further, CBOP achieves state-of-the-an performance on 11 out of 18 benchmark datasets while doing on par on the remaining datasets.
INTRODUCTION
Fueled by recent advances in supervised and unsupervised learning, there has been a great surge of interest in data-driven approaches reinforcement learning (RL), known as offline RL (Levine al., 2020). In offline RL, RL agent must learn good policy entirely from logged dataset of past interactions, without access to the real environment. This paradigm of learning is particularly useful in applications where it is prohibited or too costly to conduct online trial-and-errow explorations (e.g., due to safety concerns), such as autonomous driving (Yu et al., 2018), robotics (Kalashnikov et al., 2018), and operations research (Boute et al., 2022).
However, because the absence of online interactions with the environment that give correcting signals the learner, direct applications of online off-police algorithms have been shown fail in the offline setting (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Kumar et al., 2020). This is mainly ascribed to the distribution shift between the learned policy and the behavior policy (data-logging policy) during training. For example, in Q-learning based algorithms, the distribu- tion shift in the policy can incur uncontrolled overestimation bias in the learned value function. Specifically, positive biases in the Q function for out-of-distribuu up during policy maximization, which leads to further deviation of the learned policy from the be- havior policy, resulting in vicious cycle of value overestimation. Hence, the design of offline RL algorithms revolves around how to counter the adverse impacts of the distribution shift while achieving improvements over the data-logging policy.
(OOD) actions can be picked
"Equal contribution Corresponning authors

---

Published as a conference paper at ICLR 2023
target mean
2:50
10"
Expected herizen
2
10.-
1.00
0:75
100
Training epochs
Training epochs
Figure 1: Prevention of value overestimation & adaptive reliance on model-based value predictions. (Leff) We leverage the full posterior over the target values to prevent value overestimation during offline policy learning (blue). Without conservatism incorporated, the target value diverges (orange). (Right) We can automatically adjust the level of reliance on the model-based and bootstrapped model model-free value predictions based on their respective uncertainty during model-based value expansion. The "expected horizon' (E[h) = Ln Wi, h, En 1) shows effective model-based rollout horizon during policy optimization. E[h] is large at the beginning, but it gradually decreases the model-free value estimates improve over time. The figures were generated using the hopper-random dataset from D4RL (Fu al., 2020).
In this work, we consider model-based (MB) approaches since they allow better use of given dataset and can provide better generalization capability (Yu al., 2020; Kidambi et al., 2020; Yu al., 2021; Argenson & Dulac-Amold, 2021). Typically, MB algorithms e.g.. MOPO (Yu al., 2020), MOReL (Kidambi et al., 2020), and COMBO (Yu al., 2021) adopt the Dyna-style policy optimization approach developed in online RL (Janner et al., 2019; Sutton, 1990). That is, they the learned dynamics model to generate rollouts, which are then combined with the real dataset for policy optimization.
We hypothesize that we can make better use the learned model by employing it for target value estimation during the policy evaluation step of the actor-critic method. Specifically, we can compute h-step TD targets through dynamics model rollouts and bootstrapped terminal Q function values. In online RL, this MB value expansion (MVE) has been shown to provide a better value estimation of a given state (Feinberg et al., 2018). However, the naïve application of MVE does not work in the offline setting due to model bias that can be exploited during policy learning-
Therefore, it is critical to trust the model only when it can reliably predict the future, which can be captured by the epistemic uncertainty surrounding the model predictions. To this end, we propose CBOP (Conservative Bayesian MVE for Offline Policy Optimization) to control the reliance on model-based and model-free value estimates according to their respective uncertainties, while mitigating the overestimation errors in the learned values. Unlike existing MVE approaches (e.g., Buckman et al. (2018)), CBOP estimates the full posterior distribution over a target value from the h-step TD targets for h O, ..... H sampled from ensembles of the state dynamics and the Q function. The novelty of CBOP lies its ability to fully leverage this uncertainty in two related different h-step targets informed by the posterior uncertainty; and (2) by using this weighting to derive conservative lower confidence bounds (LCB) on the target values that mitigates value overestimation. Ultimately, this allows CBOP reap the benefits of MVE while significantly reducing value overestimation in the offline setting (Figure l).
ways: (1) by deriving
adaptive weighting
We evaluate CBOP on the D4RL benchmark of continuous control tasks (Fu et al., 2020). The ex- periments show that using the conservative target value estimate significantly outperforms previous model-based approaches: e.g., MOPO by 116.4%, MOReL by 23.2% and COMBO by 23.7% 7%. Fur- ther, CBOP achieves state-of-the-an performance on 11 out of 18 benchmark datasets while doing on par on the remaining datasets.
BACKGROUND
We study RL in the framework of Markov decision processes (MDPs) that are characterized by tuple (S,A,T, do, 7); here, S is the state space, A is the action space, T (s' (s'[ss,a a) is the tran- sition function, r(s, a) is the immediate reward function, dg the initial state distribution, and [0, 1] is the discount factor. Specifically, we call the transition and reward functions the model of the environment, which we denote as f (T,r). A policy æ is a mapping from S
2

---

Published as a conference paper at ICLR 2023
to A. and the goal of RL is to find optimal policy x² which maximizes the expected cumu- lative discounted reward, as Do rr,(s, at)], where So ~ do, Sy ~ T(-ss-11, at-1), and a x²" (-)s)) Often, we summarize the quality of policy by the state-action value function (s, == Bas ?'rs, ]s) = S, ag = a), where a, ~ T(-|s)) ()) vt 0.
Off- Off-policy actor-critic methods, such SAC (Haarnoja et al., 2018) and TD3 (Fujimoto et al., 2018), have enjoyed great successes in complex continuous control tasks in deep RL, where parameterized neural networks for the policy Th (known as actor) and the action value function Qo (known as critic) are maintained. Following the framework of the generalized policy iteration (GPI) (Sutton & Barto, 2018), we understand the actor-critic algorithm as iterating between (i) policy evaluation and (ii) policy improvement. Here, policy evaluation typically refers to the calculation of Qa(s, Ta (s)) for the policy no, while the improvement step is often as simple as maximizing the currently evaluated Qø: i.e., maxo B (Q, To [s))] (Fujimoto et al., 2018).
Policy Evaluation At each iteration of policy learning, we evaluate the current policy ma by min- imizing the meun squared Bellman error (MSBE) with the dataset D of previous state transitions:
L( D) = MSBE (.s.a. (v(s, a 8') - ())
y(s, a, s') r(s, a) + 70r (s',aa), a' ~ no (-1s')
(2)
where y(s, a's') is the TD target at each (s. a), towards which Qs is regressed. A separate target network Qø is used in computing y to stabilize learning (Mnih et al., 2015). Off-policy algorithms typically use some variations of (2), e.g., by introducing the clipped double-Q trick (Fujimoto et al., 2018), in which minj=1.2 Qat (s', a') ) is used instead of Q (s', a') )) prevent value overestimation.
Model-based Offline RL In the offline setting, we are given fixed set of transitions, D, collected by some behavior policy Tg, and the aim is to learn policy π that is better than TT. In particular, offline model-based (MB) approaches learn the model f = (f, f) of the environment using DD to facilitate the learning of a good policy. Typically, f is trained to maximize the log- ligelikeldoood of its predictions. Though MB algorithms often considered capable of better generalization than their model-free (MF) counterparts by leveraging the learned model, it is risky to trust the model for OOD samples. Hence, MOPO (Yu al., 2020) and MOReL. (Kidambi construct and learn from a pessimistic MDP where the model uncertainty in the next state prediction penalized in the reward. Criticizing the difficulty of accurately computing well-calibrated model uncertainty, COMBO (Yu et al., 2021) extends CQL (Kumar et al., 2020) to the model based regime by regularizing the value function on OOD samples generated via model rollouts. These methods follow the Dyna-style policy learning where model rollouts are used to augment the offline dataset (Sutton, 1990; Janner et al., 2019).
al., 2020)
Model-based Value Expansion (MVE) for Policy Optimization An alternative to the aforemen- tioned Dyna-style approaches MVE (Feinberg et al., 2018), which is arguably better suited to seamlessly integrating the power of both MF and MB worlds. In nutshell, MVE attempts to more accurately estimate the TD target in (2) by leveraging a model of the environment, which can lead to more efficient policy iteration. Specifically, we can use the h-step MVE target Rr (s, a, s') for yss, a. s'):
as, = =(s,a,)')sin((s (s, Σ (Be, â.) + Qor an+1); 1=0
(3)
(So, âo, fo, 81) as,,,,,s'), S f-1(s- â.-), à, ~ To (-(6:)), 14sh+1.
where (s, s') is obtained by the h-step MB return plus the terminal value at + (h = 0 reduces back to MF). In reality, errors in the learned model f compound if rolled out for a large h. Thus, is standard to set a small number.
CONSERVATIVE BAYESIAN MVE FOR OFFLINE POLICY OPTIMIZATION
The major limitations of MVE when applied to offline RL are as follows:
3

---

Published as a conference paper at ICLR 2023
1. The model predictions š, and ft in (3) become increasingly less accurate as t increases because model errors can compound, leading to largely biased target values. This issue is exacerbated in the offline setup because we cannot obtain additional experiences to reduce the model error.
2. The most common sidestep to avoid the issue above is use short-horizon rollouts only. However. rolling out the model for only a short horizon even when the model can be trusted could severely restrict the benefit of being model-based.
3. Finally, when the model rollouts go outside the support of D. R2, in (3) can have a large overestimation bias, which will eventually be propagated into the learned Qo function.
Ideally, we want control the reliance on the model f and the bootstrapped Qs according to their respective epistemic uncertainty, while also preventing Qs from accumulating large overestimation errors. That is, when can trust f. we can safely roll out the model for more steps to get a better value estimation. On the contrary, the model is uncertain about the future it predicts, we should shorten the rollout horizon and bootstrap from Q early on. Indeed, Figure I (right) exemplifies that CBOP relies much more on the MB rollouts at the beginning of training because the value function is just initialized. As Q becomes more accurate over time, CBOP automatically reduces the weights assigned to longer MB rollouts.
Below, we present CBOP, Bayesian take on achieving the aforementioned the MF and MB value estimates based on their uncertainty while obtaining a conservative estimation of the target (s, a, s'). To this end, we first let (Se, ar) denote the value of the policy π at (Sy, ar) in the learned MDP defined by its dynamics f; that is,
goals: trading off
¹ (S a²) = J yhhh (((5443)444)) (6), = (8,, a(), â ~ (-(884+))
(4)
Note that in the offline MBRL setting, typically cannot learn Q due having only an approxi- mation f of the model, and thus we focus instead on learning Q
Although there exists a unique (T, a) at each as given a fixed model f, we can- not directly observe the value unless we in- finitely roll out the model from (s, a) until termination, which is computationally in- view each R, vh de-
Algorithm 1 Conservative Bayesian MVE
Input:
(St, ar, I',, S, Qo
1. Sample Rd vh S H using f and Q as in (3)
2. Estimate μh, Fn according to (8), (9)
3. Compute the posterior N ((,, o) using (7)
feasible. Instead, fined in (3) as a conditionally independent (biased) noisy observation of the true underlying parameter Q From this assumption, we can con- struct the Bayesian posterior over Q* given the observations RA. wh. With the closed-form posterior distribution hand, we can take various conservative estimates from the distribution; we use the lower confidence bound (LCB) in this work. Algorithm 1 summarizes the procedure at a high-level. Please see Algorithm in Appendix B.1 for the full description of CBOP.
return conservative value target (e.g., LCB μ (o)
3.1 CONSERVATIVE VALUE ESTIMATION VIA BAYESIAN INFERENCE
In this part, we formally discuss the conservative value estimation of CBOP based on Bayesian pos- terior inference. Specifically, the parameter of interest is Q and we seek its posterior estimation:
H
P P(OT Ro. RR)) x P Ro..... Rio...,,Rr Ru ②-) PP p(Q")= - P(O))
II P Rí
I
(5)
A=0
where we assume that Rr (h- O,... ..... H! are conditionally independent given QT (see Appendix A where we discuss in detail about the assumptions present in CBOP).
In this work, we model the likelihood of observations P P( as normally distributed with the mean μ4 and the standard deviation oh:
Rn Q NN oi),
(6)
We will omit (s, a,s') henceforth if it clear from the context.
4

---

Published as a conference paper at ICLR 2023
since it leads to closed-form posterior update. Furthermore, since RA, can be seen sum of future immediate rewards, when the MDP is ergodic and y is close to 1, the Gaussian assumption (approximately) holds according to the central limit theorem (Dearden et al., 1998). Also, note that our Bayesian framework is not restricted to the Gaussian assumptions, and other surrogate probability distributions such the Student-t distribution could be used instead.
For the prior, we use the improper (or uninformative) prior, P(²") = 1, since is natural to assume that we lack generally applicable prior information over the target value across different environ- ments and tasks (Christensen et al., 2011). The use of the improper prior well justified in the Bayesian literature (Wasserman, 2010; Berger, 1985), and the particular prior we use in CBOP cor- responds to the Jeffreys prior, which has the invariant property under change of coordinates. The Gaussian likelihood and the improper prior lead to a 'proper' Gaussian posterior density that inte- grates to 1, from which we can make various probabilistic inferences (Wasserman, 2010).
The posterior (5) is a Gaussian with mean I and variance o², defined as follows:
#2
2 A ( Re)
BACD
Σ
En Ph
EHH
Ph
ph. (7)
N
where p 1/a² and Ph = 1/a7, are the pre- cisions of the posterior and the likelihood of Rr. respectively. The posterior mean μ corresponds to the MAP estimation of Q Note that has the form of a weighted sum, Σn /h, with w/, >\/E/ Pa. E (0, 1) being the weight allocated to Rr. If the variance of RG for some h is relatively
Figure 2: The graphical model representation of CBOP
large, we give a smaller weight to that observation. If, on the other hand, fú all have the same PH). we recover the usual H-step return estimate. Recall that the quality of R2 is determined by that of the model rollout return and the bootstrapped terminal value. Thus intuitively speaking, the adaptive weight w/ given by the Bayesian posterior allows the trade-off between the epistemic uncertainty the model with that of the Q function.
variance (e.g. Po
Figure 2 illustrates the overall posterior estimation procedure. Given transition tuple (s, a, r, r's),, we start the model rollout from 81 s'. At each rollout horizon h. the cumulative discounted reward o 7'
f is sampled by the dynamics model and the terminal value Qn is sampled by the Q function (the sampling procedure is described in Section 3.2). We then get R, by adding the h-step MB return samples and the terminal values 1 which deem as sampled from the distribution TP( ("") parameterized by μ4g of (we use the sample mean and variance). These individual h-step observations are then combined through the Bayesian inference to give us the posterior distribution over Q²..
It is worth noting that the MAP estimator can also be derived from the perspective of variance opti- mization (Buckman et al., 2018) over the target values. However, we have provided much evidence in Section 4 and Appendix D.3 that the point estimate does not work in the offline setting due to value overestimation. Henee, it is imperative that we should have the full posterior distribution over the target value, such that we can make conservative estimation rather than the MAP estimation.
To further understand the impact of using the MAP estimator for the value estimation, consider an estimator Q of Q and its squared loss: L(Q",, Q) = -(¹= -Q==~Q²² Q)2. It is known that the posterior mean of Q² minimizes the Baves risk w.r.t. I(Q"T,Q, QQ (Wasserman. 2010), meaning that the pos- terior risk (.("", called the (generalized) Bayes estimator of Q", which is an admissible estimator (Robert, 2007). Despite seemingly advantageous, this result has a negative implication in offline RL. That is, the MAP estimator minimizes the squared loss from (s, a) over the entire support of the posterior, weighted by the posterior distribution. Now, the distribution shift of π from n. can lead to signif-
Ro....,,RR )dQ= is minimized at Q = μ.. In this context, l is also
icantly biased Q" compared to the true Q". In this case, the quality of the MAP estimator when
5

---

Published as a conference paper at ICLR 2023
evaluated in the real MDP would be poor. Especially, the overestimation bias in the MAP estimation can quickly propagate to the Qú function and thereby exacerbate the distribution shift.
3.2 ENSEMBLES OF DYNAMICS AND Q FUNCTIONS FOR SAMPLING I1-STEP MVE TARGETS
In this section, we discuss how we estimate the parameters phy ot of IP (Ra from the ensemble of dynamics models and that of Q functions.
Assume we have bootstrapped dynamics ensemble model f consisting of K different models (f,,...,fkk fKk trained with different sequences of mini-batches of 'D (Chua et al., 2018; Janner al., 2019). Similarly, we assume a Q ensemble of size M. Given a state & and an action à, we can construct the probability over the next state $e+1 and reward ft by the ensemble as follows:
P(S Pelse. a.)
Σ PP (ft)
PP A ar, fs.
where Pf((x) is the probability of selecting the kth model from the ensemble, which is 1 1/K when all models are weighted equally. Now, the sampling method that exactly follows the probabilistic graphical model shown in Figure would first sample a model from the ensemble at each time step, followed by sampling the next state transition (and reward) from the model, which should then be repeated K times per state generate a single sample. Then, we evaluate the resulting state $441 and action å Ta ($++1) with the Q ensemble to obtain M samples. To obtain N trajectories from single initial state estimate f4, and of for h 1, , H, the overall procedure requires O( (NKH) computation, which can quickly become infeasible for moderately large K and N values.
To reduce the computational complexity, we follow Chua al. (2018) where each particle prop- agated by a single model of the ensemble for H steps. With this, we can obtain N trajectories of length H from one state with H( instead of ((NKH (below we use N = K, i.e., we gen- (so, To, ro, S1), we create K numbers of particles by replicating 81 K times, denoted as 6(4) Vk. The kth particle is propa- gated by fixed model fo and the policy To for H steps, where (k) , fa(s a(m)) and
erate one particle per model). Concretely, given a single transition
a (R) To (s( At each imagined timestep t E H + 1], M number of terminal values are sampled by the Qø ensemble at (s!+) a, =((x))
Despite the computational benefit, an implication of this sampling method is that it no longer directly follows the graphical model representation Figure 2. However, we can still correctly estimate μt and of by turning to the law of total expectation and the law of total variance. That is,
Exs
Ru 7] = fz Em,
Ri
(8)
where the outer expectation is w.r.t. the dynamics ensemble sampling probability P(fx) = 1/k. Hence, given a fixed dynamics model ft. we sample RG by following and compute the average of the h-step return, which is then averaged across different ensemble models. In fact, the resulting μi is the mean of all aggregated M x K samples of Ra-
The h-step return variance Var,, ( (Rx(T) decomposes via the law of total variance as following: of VarT [RNT]]
R Var
R IT, [i]] Var E
Ru
r. fic
Here, A is related to the epistemic uncertainty of the Q ensemble; while B is associated with the epistemic uncertainty of the dynamics ensemble. The total variance Varz, (ft, (r) captures both uncertainties. This way, even though we use different sampling scheme than presented in the graphical model of Figure 2, we can compute the unbiased estimators of the Gaussian parameters.
Once we obtain (4, and of we plug them into (7) to compute the posterior mean and the variance. A conservative value estimation can be made by ÍLCB = μ tá with some coefficient & > 0 (Jin et al., 2021; Rashidinejad et al., 2021). Under the Gaussian assumption, this corresponds to the worst-case return estimate in a Bayesian credible interval for Q We summarize CBOP in Al Algorithm 2 in Appendix B.

---

Published as a conference paper at ICLR 2023
Table 1: Normalized scores D4RL MuloCo Gym environments. Experiments with 5 seeds.
MGPO
MORdl
COMBO
CQI.
TD3
EDAC
IQL.
CROP
balichectuh
35.4 2.5
25.6
38.8
35.4
10.2 # 1.3
28.4 1.0
32.8 0.4
Fand
hopper
11.77::0.4
53.6
17.9
10.8
11.0
31.3 1.0.
31.4 4 0.0
walker2d
18.6 2.6
37.3
7.0
21.7
0.4
me molhun
halichoetihh
42.3 12.3.211 1.6
42.1
54.2
44.4
42.8
0.3
67.5 1.2
47.4
74.8 + 0.2
happer
12.4
95.4
99.5
0.6
0.1
walker2d
17.8 = 19.3
77.8
76.5
79.7 1.8
92.5 - 0.8
78.3
95.5 : 0.4
modium reploy
ballcheetaah
53.1
40.2
63.9
hopper
67.3 24.7
93.6
73.1
48.0 6
31.4
3.0
101.8 0.5
94.7
104.3 1: 0.4
Walker2d
39..0 1..0+....0
49.8
66.0
26.7
25.2 5.1
87.1 2.3
73.8
92.7 + U.9
modhum oxpent
halichoctih
63.3 38.0
53.3
90.0
62.4
97.9 4.4
107. 1 + 2.0
86.7
105.4 1.6
hopger
23.7
108.7
113.1
98.7
112.2
110.7 0.1
91.5
111.6 0.2
walkerZd
95.6
expat
balfcheetah hopper
+ 3.4
100.4
walkerr2d
1U6.7 2.7
115.1 1.9
122.7 # 0.8
toll replay
halfchoctuh hagger
85.5 +
106.4 = 0.7
108.1 0.3
walker2d
90.8 0.7
107.8 0.2
4 EXPERIMENTS
We have designed the experiments answer the following research questions: (RQ1) Is CBOP able to adaptively determine the weights assigned to different h-step returns according to the relative uncertainty of the learned model and that of the function? (RQ2) How does CBOP perform in the offline RL benchmark? (RQ3) Does CBOP with LCB provide conservative target Q estimation? (RQ4) How does having the full posterior over the target values compare against using the MAP estimation in performance? (RQ5) How much better is it to adaptively control the weights to /i-step returns during training opposed using a fixed set weights throughout training?
We evaluate these RQs on the standard D4RL offline RL benchmark (Fu et al., 2020). In particular, use the D4RL MuJoCo Gym dataset that contains three environments: halfcheetah, hopper, and walker2d. For each environment, there are six different behavior policy configurations: random (r), medium (m), medium-replay (mr), medium-experi (me), expert (e), and full-replay (fr). We release our code at https::t///.m. / www...tuu..cc...............eect com www.wwiwwwogg////eeeeeeeeee
4.1 CBOP CAN AUTOMATICALLY ADJUST RELIANCE ON THE LEARNED MODEL
To investigate RQ1, we use the notion of the ex- pected rollout horizon, which we define E[h]
Expected rallout horizon sth]
164
-0002
E Wh. h. Here, wh is the weight given to the
1.4
00.00
Valldate aalidation 055
mean of R2 as defined in (7), which sums to 1. A
12
larger E[h] indicates that more weights are assigned
12-
to longer-horizon model-based rollouts.
11
0.014
Figure/ 1 already shows that [[[]] decreases as the Q function becomes better over time. On the other hand, Figure shows how the quality of the learned model affects E[h). Specifically, namics model on halfcheetah-m for different num- bers of epochs (10.... 100); then, trained the policy with CBOP for 150 epochs.
26
10
30 60
80
90
100
Epochs (dynamics training)
trained the dy-
Figure 3: [[ during CBOP training with the dynamics model trained for different numbers of epochs. CBOP can place larger weights longer- horizon rollouts as the dynamics model becomes more accurate.
4.2 PERFORMANCE COMPARISON
To investigate RQ2, we select baselines covering both model-based and model-free approaches: (model-free) CQL (Kumar et al., 2020). IQL (Kostrikov et al., 2022). TD3+BC (Fujimoto & Gu, 2021), EDAC (An al., 2021); (model-based) MOPO (Yu al., 2020), MOReL (Kidambi al., 2020), and COMBO (Yu et al., 2021). Details of experiments are provided in Appendix C.I.

---

Published as a conference paper at ICLR 2023
Table I shows the experimental results. Comparing across all baselines, CBOP presents new state- of-the-art performance in 11 tasks out of 18 while performing similar in the remaining configura- tions. Notably, CBOP outperforms prior works in medium, mediun-replay, and full-replay config- urations with large margins. We maintain that these are the datasets of greater interest than, e.g.. random expert datasets because the learned policy needs to be much different than the behavior policy in order to perform well. Furthermore, the improvement compared to previous model-based arts is substantial: CBOP outperforms MOPO, MOReL, and COMBO by 116.4%, 23.2% and 23. 7% (respectively) on average across four behavior policy configurations.
4.3 CBOP LEARNS CONSERVATIVE VALUES
To answer RQ3, we have selected configurations (m, me, and mr) from the hopper environment and evaluated the value function at the states randomly sampled from the datasets, i.e., Ex=D E [s)] (nb. similar analysis is given in CQL). Then, we com- pared these estimates with the Monte Carlo estima-
Table 2: Difference between the values predicted by the learned Q functions and the true discounted returns from the environment.
CQL
CBOP
Task name
Mean
Max
Mean
Max
tions from the true environment by rolling out the learned policy until termination.
hopper
61.84
-3.00
-5..83
-16.21
hopper-mr
-142899
-28.73
.172.45
hoppor-me -79.67
-5.16
-114.39
-11.24
Table reports how large are the value predictions compared to the true returns. Notice that not only the mean predictions are negative but also the maximum values are, which affirms that CBOP indeed has learned conservative value functions. Despite the predictions by CBOP being smaller than those of CQL in hopper-mr and me, we can see that CBOP significantly outperforms CQL in these settings. See Appendix D..1 for more details.
4.4 ABLATION STUDIES
LCB MAP in the offline setting To answer RQ4, we compare CBOP with STEVE (Buckman et al., 2018) which is equivalent to using the MAP estimation for target Q predictions. Figure (left) shows the case where the value function learned by STEVE blows up (orange). Further, include the performance of STEVE in all configurations in Appendix D.3. To summarize the results, STEVE fails learn useful policies for 11 out of 18 tasks. Especially, except for the fr datasets, using the MAP estimation has led to considerable drops in the performances in the hopper and walker2d environments, which reaffirms that it is critical to have the full posterior distribution over the target values such that we can make conservative target predictions.
Adaptive weighting For RQ5, also considered alternative way of combining Rd vh by explicitly assigning a fixed set of weights: uniform or geometric. We call the latter X-weighting, in reference the idea of TD(A) (Sutton, 1988). We evaluated the performance of the fixed weighting scheme with various À E (0, 1) values, and report the full results in Appendix D.3. In summary, there are some À values that work well in specific task. However, it is hard to pick single that works across all environments, and thus À should be tuned as hyperparameter. In contrast, CBOP can avoid this problem by automatically adapting the rollout horizon.
Benefits of full posterior estimation To ablate the benefits of using the full posterior distribu- tion in conservative policy optimization, we have compared CBOP to a quantile-based approach that calculates the conservative estimate through the a-quantile of the sampled returns ((s, a, s') (3) from the ensemble. The experimental details and results are reported in Appendix D.3. In sum- mary, have found that CBOP consistently outperformed this baseline on all tasks considered, and CBÔP was more stable during training, showing the effectiveness of the Bayesian formulation.
5 RELATED WORK
In the pure offline RL setting, is known that the direct application of off-policy algorithms fails due value overestimation and the resulting policy distribution shift (Kumar et al., 2019; 2020; Fujimoto Gu, 2021; Yu et al., 2021). Hence, it critical to strike the balance between conser- vatism and generalization such that mitigate the extent of policy distribution shift while ensuring
8

---

Published as a conference paper at ICLR 2023
that the learned policy Te performs better than behavior policy Tg. Below, we discuss how existing model-free and model-basee methods address these problems in practice.
Model-free offline RL Policy constraint methods directly constrain the deviation of the learned policy from the behavior policy. For example, BRAC (Wu et al., 2019) and BEAR (Kumar et al., 2019) regularize the policy by minimizing some divergence measure between these policies (e.g., MMD or KL divergence). Alternatively. BCQ (Fujimoto et al., 2019) learns a generative model of the behavior policy and uses to sample perturbed actions during policy optimization. On the other hand, value regularization methods such as CQL (Kumar et al., 2020) add regularization terms to the value loss in order to implicitly regulate the distribution shift (Kostrikov et al.., 2021; Wang et al., 2020). Recently, some simple yet effective methods have been proposed. For example, TD3+BC (Fujimoto & Gu, 2021) adds a behavioral cloning regularization term to the policy objective of an off- off-policy algorithm (TD3) (Fujimoto et al., 2018) and achieves SOTA performances across variety of tasks. Also, by extending Clipped Double Q-learning (Fujimoto et al., 2018) to an ensemble of N Q functions, EDAC (An al., 2021) achieves good benchmark performances.
Model-based offline RL Arguably, the learning paradigm of offline RL strongly advocates the use of a dynamics model, trained in a supervised way with a fixed offline dataset. Although a learned model can help generalize to unseen states or new tasks, model bias poses a significant challenge. Hence, it is critical to know when to trust the model and when not to. MOPO (Yu et al., 2020) and MOReL (Kidambi et al., 2020) address this issue by constructing and learning from a pessimistic MDP whose reward is penalized by the uncertainty of the state prediction. On the other hand, COMBO (Y (Yu et al., 2021) extends CQL within the model-based regime by regularizing the value function on OOD samples generated via model rollouts. Rigter et al. (2022) also takes an adversarial approach by optimizing the policy with respect to a worst-case dynamics model. In contrast to these, CBOP estimates a full Bayesian posterior over values by using ensembles of models and value functions during policy evaluation of an actor-critic algorithm. In principle, having the full distribution that CBOP provides could also facilitate the use of other risk-informed statistics and epistemic risk measures to address value overestimation (see, e.g., Eriksson & Dimitrakakis (2020)).
Model-based value expansion Unlike Dyna-style methods that augment the dataset with model- generated rollouts (Sutton, 1990; Janner et al., 2019), MVE (Feinberg et al., 2018) uses them for better estimating TD targets during policy evaluation. While equally weighted h-step model returns were used in MVE, STEVE (Buckman et al., 2018) introduced an adaptive weighting scheme from the optimization perspective by approximately minimizing the variance of the MSBE loss, while ig- noring the bias. Interestingly, the Bayesian posterior mean (i.e., the MAP estimator) we derive in (7) matches the weighting scheme proposed in STEVE. However as we show in Figure and 10, using the MAP estimator as value prediction in the offline setting often results in largely overestimated Q values, which immensely hampers policy learning. See Section 3.1 for the related discussion.
6 CONCLUSION
In this paper, we present CBOP: conservative Bayesian model-based value expansion (MVE) for offline policy optimization. CBOP is model-based offline RL algorithm that trades off model-free and model-based value estimates according to their respective epistemic uncertainty during policy evaluation while facilitating conservatism by taking a lower bound on the Bayesian posterior value estimate. Viewing each h-step MVE target as conditionally independent noisy observation of the true target value under the learned MDP, we derive the Bayesian posterior distribution over the target value. For practical implementation of CBOP, Q function to sample MVE targets to estimate the Gaussian parameters, which in turn are used to compute the posterior distribution. Through empirical and analytical analysis, we find that the MAP estimator of the posterior distribution could easily lead to value overestimation when the learned MDP is not accurate under the current policy. In contrast, CBOP constructs the LCB from the Bayesian posterior as a conservative estimation of the target value to successfully mitigate the issue while achieving state-of thete-rr performance on several benchmark datasets.
the ensemble of dynamics and that of

---

Published as a conference paper at ICLR 2023
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G Bellemare. Deep reinforcement learning the edge the statistical precipice. Advances in Neural Informa- tion Processing Systems, 2021.
Gaon An, Seungyong Moon, Jang- Hyun Kim, and Hyun Oh Song. Uncertainty-basel offline rein- Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL htttps://ppenrewee wttpr/ewwwee//eumtiiiieiee worummiiiii 20 2UvaSo122h3.
forcement learning with diversified q-ensemble.
net
Arthur Argenson and Gabriel Dulac-Aroold. Model-based offline planning. In International Confer- Learning Representations, 2021. URL https: http://pppneeeeeeeeeeeee net.eeiwwwweeeeeeeee.eee.....e.e www.:///wwwweeeeeeoooommmmm forem/ff
OMNB165xzd4.
James O Berger. Statistical decision theory and Bayesian analysis; 2nd ed. Springer series in statistics, Springer, New York, 1985. doi: 10.1007/978-114757--26--- URL httpt:// //eds.
cern. wtt..c//rrcoord332799999
Robert N. Boute, Joren Gijsbrechts, Willem van Jaarsveld, and Nathalie Vanvuchelen. Deep
reinforcement leamning for inventory control: roadmap. European Journal of Opera- tional Research, 298(2):401-412, 2022. ISSN 0377-2217. doi: http://ddi..rg110.000110
2021.07.016. URL https: http://wwwww..m www.:wwwcicc.e..ccccoco rect cccm///cencce/rtticle/epp. 50377221721006111
Leo Breiman. Bagging predictors. Machine learning, 24(2):123-140, 1996.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. In Proceedings of the 32nd International Conference Neural Information Processing Systems, NIPS' 18, pp-
8234 8244, 2018
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 50844-5544,,7, 2021.
R. Christensen, W. Johnson, A. Branscum, and T.E. Hanson. Bayesian Ideas and Data Analysis: An Introduction for Scientists and Statisticians. Chapman & Hall/CRC Texts in Statistical Sci- ence. Taylor & Francis, 2011. ISBN 9781439803554, URL //ttpks///boks google ca/ books?id=qPERhcbenn
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning handful of trials using probabilistic dynamics models. In Advances in Neural In- formation Processing Systems, volume 31, 2018. URL https:://proceedinggg neurips. .dd.
ee/ htpppaapr///////// / www.:/eeeeeeeeeeeeeeeeeeeeeeeeee 185977 0111155919111111111111115-----11
Richard Dearden, Nir Friedman, and Stuart Russell. Bayesian q-learning. Aaaifiaai, 1998:761-768, 1998.
Bradley Efron. The jackknife, the bootstrap and other resampling plans. SIAM, 1982.
Hannes Eriksson and Christos Dimitrakakis. Epistemic risk-sensitive reinforcement learning. In ESANN, 2020.
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I. Jordan, Joseph Gonzalez, and Sergey
Levine. Model-based value estimation for efficient model-free reinforcement learning, 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR, abs/2004.07219, 2020.
Scott Fujimoto and Shixiang Gu. A minimalist approach to offline reinforcement learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https:/ http::///ppeeevieee.. net ffffmmirrid Q32U7dzWXpc.
10

---

Published as a conference paper at ICLR 2023
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation enror in actor-critic methods. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th Inter- national Conference
Machine Learning, ICML 2018, Stockholmsmudssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1582-1591. PMLR, 2018. URL http:: http::/ httphht:tpppoceeeinieeee
ml www.mraarr.rr......
prwsspssss/.... / f wotoiimmollli 8a html.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th International Conference on Machine Learning, vol- ume 97 Proceedings of Machine Learning Research, pp. 2052-2062. PMLR, 09-15 Jun 2019. URL htt.h::tt///........m /tte::/prrrocee /tpoo/poooeeeeee ngs. ml prwss / 97 fu o 9a hhmll.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research. pp. 1861-1870. PMLR, 10-15 Jul 2018. URL https: httttt//pprcccediiii htt:://pccceeeengceeeeee mmlr.
press/v80
htt////00/aaaaa haarno fa188. html.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When trust your model: Model- based policy optimization. In Advances in Neural Information Processing Systems, 2019.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline r1? In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 5084-5096, PMLR, 18 24 Jul 2021. URL https: : /ttp//peeoceceneeeeeeeeeeee mlr ..l.rppzee press /vcss//v135 v139 /es/s/innii jin2le..... html.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Scal- able deep reinforcement learning for vision-based robotic manipulation. In Proceedings The 2nd Conference Robot Learning, volume 87 of Proceedings of Machine Learning Research, pp. 651-673. PMLR, 29-31 Oct 2018. URL https: / pttppeeiecceeaeeeeeeeeeepee...ae Kalashnikoon novl Ba html.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel Model- based offline reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learn- ing with fisher divergence critic regularization. In Marina Meila and Tong Zhang (eds.), Pro- ceedings of the 38th International Conference on Machine Learning, volume 139 of Proceed- ings of Machine Learning Research, pp. 5774-5783. PMIR, 18-24 Jul 2021. URL https: html.
/ //proceedings... ml www..press/vve /velsvl11999 httss///11///kktrkk/ooos htt///kstrikkkoo
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In International Conference on Learning Representations, 2022. URL https:// openreview net/ forum?ideeeoooo.o.. 58n2s 2J
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, volume 32, 2019. URL https: /tattttp::/prrceeeeeee /ttp///eooooeeeeeeeee/eepeeeooce neurips...... cc/ cc/pappr/2019/
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Inter- national Conference on Machine Learning, pp. 3703-3712. PMLR, 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives open problems. CoRR, abs/2005.01645 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness. Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran,

---

Published as a conference paper at ICLR 2023
Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529- 533, February 2015. ISSN 00280836. URL
attp://dke wtt.xxdoi.oorrgo. www.do.......o.o.. mtt.///1enau.r.........
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.0905: 2020.
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging of- fline reinforcement learning and imitation learning: A tale of pessimism. In M. Ran- zato, A. Beygelzimer, Dauphin, P.S. Liang and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 11702-11716, Curran Asso- ciates, Inc., 2021. URL 60ce36723e 7bbac:044 Ffw.eeeeefccc4444aee
pdf.
Mare Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. arXiv preprint arXiv:2204.12581/ 2022.
C. Robert. The Bayesian Choice:
From Decision-Theoretic Foundations to Computational Imple- mentation. Springer Texts in Statisties. Springer New York, 2007. ISBN 9780387715988, URL
https:/ /ttp////ooook///// google. ca ca/books/ ?id=6044 $PPp:p44.
Richard S. Sutton. Learning predict by the methods of temporal differences. Machine Learn- ing. 3(1):9-44, August 1988. URL http: wwww cs sutton-88 88, pdf.
Richard S. Sutton. Integrated architectures for learning, planning, and reacting based on approxi- mating dynamic programming. In Bruce Porter and Raymond Mooney (eds.), Machine Learning Proceedings 1990, pp. 216 224. Morgan Kaufmann, San Francisco (CA), 1990.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
Jianhao Wang, Wenzhe Li, Haozhe liang, Guangxiang Zhu, Siyuan Li, and Chongjie Zhang. Offline reinforcement learning with reverse model-based imagination. Advances in Neural Information Processing Systems, 34:29420-29432, 2021.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression. In Advances in Neural Information Processing Systems, vol- ume 33, pp. 7768-7778, 2020. URL htttps://proceedingg wtw.://rrreeddii............... wttpcper///////////// 2020/file haaeee//8cbbb555 bddbee (wwel/bbbb66778888 677788799884442222222 Peedddaaer.ppdee
Larry Wasserman. All of Statistics: A Concise Course in Statistical Inference. Springer Publishing
Company, Incorporated, 2010. ISBN 1441923225.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. http://xaiv..oog/a//////////// 11361.
CoRR, abs/1911. 2019. URL
Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell. BDD1008 abs/1805.0445 805.04677, 2018. URL http: http:://arxiii //tpxiv.ivv//////////eae//aaaava www.vv.or/////..ooooooo.o org/orrggao.... haagg///////////////// 1000. 04687.
diverse driving video database with scalable annotation tooling. CoRR,
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In Advances in Neural Infor- mation Processing Systems, 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: conservative offline model-based policy optimization. CoRR, abx/2102.08363, 2021.
12


Reviews:
[{"review_id": "dUI0_ATWzx1", "reviewer": "Reviewer_fgeG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents CBOP, a new approach for model-based offline reinforcement learning, where the RL agent is trained to optimize a lower bound on the Bayesian posterior estimate of values. In this approach, each h-step return from a particular state in the offline dataset is treated as a sample from the Bayesian posterior estimate of the value in that state, from which the posterior distribution over the target values is derived. The authors further balance model-free and model-based estimates of the values during policy evaluation by incorporating a lower bound on the Bayesian posterior value estimates for conservativity. \n \n", "summary_of_the_review": "The paper appears to be solid, well-written and clear to understand. It proposes a novel method for model-based offline RL, which is straightforward and easy to follow. CBOP shows strong promise in the experiments compared to other SOTA methods. Although there are some comments that still need to be addressed to improve the paper, I vote for accepting the paper.", "strength_and_weaknesses": "Strengths:\n+ The paper is well written and clear to understand. I honestly enjoyed reading the paper.\n+ The experiments show strong and promising results for CBOP compared to state-of-the-art.\n+ The proposed method is novel (to some extent), straightforward, and easy to follow.\n\nWeaknesses:\n- The proposed method has not been theoretically analyzed. In my opinion, any theoretical analysis and guarantees would add great value to the current work.\n- There are some parts in the paper that deserve to be discussed in more detail. For further elaboration, please refer to the questions below.\n- Along with an ensemble of dynamics models, CBOP also utilizes an ensemble of value functions, which makes this method computationally more demanding than other state-of-the-art methods.\n- Similar to MOPO, CBOP heavily relies on the diversity among the ensemble networks. As a result, if the networks within the ensemble turn out to be correlated to one another (which is not uncommon in practice), this method may potentially fail to work properly.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "X9Ym4ZIJrX", "reviewer": "Reviewer_swCh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- Paper studies the offline RL setting, and specifically model-based approaches to offline RL.\n- Paper hypothesizes that model-based value expansion (MVE) can help improve the quality of model-based offline RL algorithms. However, a naive application is found to be unsuccessful due to model bias which is particularly prominent in the offline RL setting. \n- To overcome above challenge, the paper devices a conservative estimation procedure based on uncertainty quantification, similar in spirit to prior work like MOPO and MOReL.\n- The paper empirically evaluates the proposed method (CBOP) on D4RL benchmark tasks and find significant improvements over prior methods.", "summary_of_the_review": "Paper extends MVE to offline RL setting. While this is conceptually an incremental advancement, the proposed algorithm empirically displays improvements over prior work. Overall, I recommend a neutral to weak accept rating.", "strength_and_weaknesses": "- The paper can be interpreted as incorporating MVE to the offline RL setting. While this is incremental in nature, it nevertheless is a useful contribution and can be of interest to the offline RL community.\n- Having a concrete algorithm box to describe the method can be useful to better understand the nuances of practical application.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PELuyQJpzTr", "reviewer": "Reviewer_aFWb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies model-based RL in the offline setting. Offline RL typically requires regularization/penalty to avoid distributional shift and enable reasonably accurate value estimation. In contrast to previous model-based algorithms, which penalize only according to model uncertainty, this paper proposes a Bayesian-inspired conservative value estimation mechanism which incorporates both model and Q function uncertainty.\n\nThe authors describe how to approximate the posterior mean and variance of the Q values using ensemble methods. With these in hand, the algorithm can compute a conservative estimate of the Q value via the lower confidence bound. The proposed algorithm, CBOP, demonstrates strong performance on the D4RL benchmark and is (empirically) shown to provide conservative estimates of the actual returns.", "summary_of_the_review": "I think the paper is a useful contribution overall. It presents a novel algorithm that appears effective and is well-backed with empirical support. However, the paper could be further strengthened with additional experiments, as detailed above.", "strength_and_weaknesses": "Strengths:\n* Jointly balancing the uncertainty in the model and Q function seems a good idea to me, and this paper presents a reasonably principled way to do that.\n* Experimental results are good. CBOP closes the performance gap between model-based and model-free algorithms in the offline setting, and is demonstrably conservative w.r.t. the true returns.\n* In the appendix, the authors provide evidence supporting their assumption of Gaussianity.\n\nWeaknesses\n* It is not obvious to me that the Bayesian formulation is necessary; the gains in performance could simply be from being conservative over the model and Q function jointly. A possible baseline algorithm to test this question would compute the target via a conservative quantile of all the values $\\{\\hat{R}_h^{k,m}\\}$. This would be a more appropriate baseline than STEVE (which is not conservative because it was not designed for the offline setting), and if CBOP outperforms such an algorithm, the case for the Bayesian approach would be stronger.\n* The comparison of CBOP-estimated values to actual returns is only performed for 3 datasets, all the same task. Including results for all (or at least more) tasks in the appendix would strengthen the reader’s confidence in the algorithm.\n* The hyperparameter tuning sounds a bit dubious. The authors claim “The only hyperparameter that we have tuned is the LCB coefficient $\\psi$“, but then they also state environment-dependent values for $M$ and $\\eta$.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5CnwDfrSfa", "reviewer": "Reviewer_yFv4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for model-based offline RL which leverages a learned model to perform model-based value expansion (MVE) in the policy evaluation phase of an actor critic style algorithm. Motivated by the risks of compounding errors and instability when using a model learned only from limited data from a behavior policy in offline RL, the authors propose a Bayesian approach which aims to automatically determine the degree to leverage the learned model, while also leveraging the posterior to apply conservatism to mitigate overestimation bias in the learned Q function. They demonstrate the approach outperforms model-based approaches on standard benchmarks, and achieves state-of-the-art performance on several domains. ", "summary_of_the_review": "Overall, I think the approach presents an interesting idea for combining the benefits of model-free and model-based methods for offline RL, and demonstrates that this idea can lead to impressive performance, which supports it's acceptance. However, the paper could be significantly improved with a clearer discussion of the probabilistic model and the assumptions made to translate the predictive distributions obtained through the DNN ensembles to the assumed measurement model on the true Q function.", "strength_and_weaknesses": "Strengths:\n- The paper was well written, and explained concepts clearly. The background section provided the right level of information to the reader, explaining the key steps of generalized policy iteration, and how learned models can be used in the policy expansion phase.\n- Leveraging the Bayesian posterior over Q values to construct a lower confidence bound for conservative exploration is an elegant way to incorporate conservatism and prevent overestimation bias as the state/action distribution deviates from that of the behavior policy.\n- The experimental evaluation was thorough and demonstrated a significant performance improvement relative to baselines. I appreciated the comparisons to alternative weighting techniques, highlighting the benefit of the Bayesian inference based approach over simpler algorithmic choices like $\\lambda$-weighting\n\nWeaknesses:\n- The discussion of the probabilistic model and assumptions was imprecise:\n    - Section 3.1 seeks to define a posterior belief over $\\hat{Q}^\\pi$, the Q function for acting under $\\hat{f}$ with policy $\\pi$ and accruing rewards according to $\\hat{r}$. However, then in section 3.2, this posterior is interpreted as a credible interval for $Q^\\pi$, the Q function for the true dynamics, and then used to construct the targets for Q function regression. As the true Q function is true quantity of interest, it would make more sense to frame section 3.1 as estimating p(Q^pi \\mid \\hat{R}_0, …, \\hat{R}_H), and likewise to treat $\\hat{R}_h | Q^\\pi$ as normally distributed, as opposed to $\\hat{R}_h | \\hat{Q}^\\pi$. \n    - The discussion motivating using the conservative estimate as opposed to the MAP estimator of the posterior belief in section 2 was hard to follow. Specifically, the last paragraph on page 5 felt unnecessary, as pointing out that $\\mu$ is an admissible estimator of $\\hat{Q}^\\pi$ does not provide any evidence that a lower-confidence bound estimate would lead to a less biased estimate of the true Q function than the MAP estimate under the posterior.\n    - The state-transition model using the ensemble dynamics model in section 3.2 conflates aleatoric and epistemic uncertainty by proposing that at each timestep, we first sample a dynamics model, and then sample a transition from the chosen model. This could lead to sample trajectories that are not actually dynamically feasible under any one of the individual models. The authors mention that in practice, they hold the model fixed over the course of the rollout when sampling for the purposes of reducing computational complexity. While this does reduce the sampling requirements, it also alters the probabilistic model to one that correctly separates uncertainty over a stationary dynamics model from stochasticity in individual transitions, as discussed in [Chua et al, 2018]. However, as the authors acknowledge in the Appendix, this breaks the conditional independence assumption they leverage to compute the Bayesian posterior.\n    - It is unclear why the mean and variance computed according to (9) and (10) should yield the mean and variance of the observation model p( R_h | \\hat{Q} ). Ensembles of neural networks can be viewed as performing approximate Bayesian inference, i.e. modeling p( f | D), where the prior p(f) depends on the initialization of the ensemble weights []. The authors should clarify why the predictive distribution when marginalizing over the two ensembles reflects the conditional distribution p( R_h | \\hat{Q} ). \n- Experiments were limited to deterministic environments:\n    - While the derivation of the algorithm allowed for environments with stochastic transitions, the experiments focused on domains with deterministic transition and reward functions. I would be interested to see how stochasticity inherent to the environment would impact the performance of CBOP.\n- Hyperparameters were tuned based on final online evaluation performance.\n    - The reported results relied on different LCB coefficients for different environments, which as the authors report in the appendix C.2., was chosen based on final online evaluation performance. Given that this sort of hyperparameter optimization would not be possible in true safety-critical settings which motivate offline RL, it would have been useful to see the impact of this hyperparameter on performance. Specifically, I would imagine that as $\\psi$ is increased from 0, we would see an trend that interpolates between the MAP based performance of STEVE  to the reported results in CBOP. \n    - The initialization of the Q ensemble seemed to play a large part in CBOP's performance -- indeed, as the experiments demonstrated, using a randomly initialized Q ensemble led to poor results as the predictive variance was not calibrated. Furthermore, the authors leveraged gradient diversification to improve the performance of the algorithm. I would have liked to have seen a greater discussion about the importance of a calibrating variance estimates form the Q ensemble, i.e. ensuring the ensemble variance is an accurate estimate of $Var(R_0 | \\hat{Q}^\\pi))$ -- especially in the body of the text. \n\nMinor comments:\n- There are some notation mistakes / incomplete sentences, especially in the appendix. For example, on page 16, the definition of $Q_\\phi^k = \\arg\\min_\\phi …$ is not a valid equation, and the first paragraph of C.2 ends with a fragment.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster