Entropy-Regularized Model-Based Offline Reinforcement Learning
Authors: Soroush Ghandi
Maryam Tavakol
Venue: Submitted to ICLR 2023 (2023)
Keywords: Reinforcement learning
model-based
offline RL
entropy regularization

Abstract: Model-based approaches to offline Reinforcement Learning (RL) aim to remedy the problem of sample complexity in offline learning via first estimating a pessimistic Markov Decision Process (MDP) from offline data, followed by freely exploring in the learned model for policy optimization. Recent advances in model-based RL techniques mainly rely on an ensemble of models to quantify the uncertainty of the empirical MDP which is leveraged to penalize out-of-distribution state-action pairs during the policy learning. However, the performance of ensembles for uncertainty quantification highly depends on  how they are implemented in practice, which can be a limiting factor. In this paper, we propose a systematic way to measure the epistemic uncertainty and present \abbrv, an Entropy-regularized Model-based Offline RL approach, to provide a smooth error estimation when leaving the support of data toward uncertain areas. Subsequently, we optimize a single neural architecture that maximizes the likelihood of offline data distribution while regularizing the transitions that are outside of the data support.  Empirical results demonstrate that our framework achieves competitive performance compared to state-of-the-art offline RL methods on D4RL benchmark datasets.

Under review as a conference paper at ICLR 2023
ENTROPY-REGULARIZED MODEL-BASED OFFLINE REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Model-based approaches to offline Reinforcement Learning (RL) aim to remedy the problem of sample complexity in offline learning via first estimating pes- simistic Markov Decision Process (MDP) from offline data, followed by freely exploring in the learned MDP for policy optimization. Recent advances in model- based RL techniques mainly rely on ensemble of models to quantify the un- certainty of the empirical MDP which is leveraged to penalize out-of-distribution late-action pairs during the policy learning. However, the performance of ensem- bles for uncertainty quantification highly depends on how they are implemented practice, which can be limiting factor. In this paper, we propose a systematic way to measure the epistemic uncertainty and present EMO, an Entropy-regulane Model-based Offline RL approach, to provide a smooth enror estimation when leaving the support of data toward uncertain areas. Subsequently, we optimize a single neural architecture that maximizes the likelihood of offline data distribution while regularizing the transitions that are outside of the data support. Empirical results demonstrate that our framework achieves competitive performance com- pared to state-of-the-ar offline RL methods on D4RL benchmark datasets.
INTRODUCTION
Following the major success of deep Reinforcement Learning (RL) in numerous applications (Mnih al.) [2013: 2015; Silver et al.) 2018), offline RL has emerged to cope with the problems where simulation or online interaction is impractical, costly, and/or dangerous, thus, allowing to automate a wide range of decision-making problems from healthcare and education to finance and robotics (Levine et al.] 2020). The primary challenge in these scenarios is however that learning new policies from data stored with a different (possibly sup-optimal) policy, aka behavior policy, suffers from distributional shifts resulting in extrapolation error, which is infeasible to improve due to lack of additional exploration (Fujimoto et al.] 2019 Kumar et al. 2019). This is why standard (online) RL methods perform poorly in offline settings YYu et al. (2220). Consequently, several model-free offline RL algorithms are introduced to regularize the learned policies to stay close to the behavior policy, by constraining mut-of-distribution trajectories (Fujimoto et al.) 2019: Kumar al. 2019
Wu et al. 2019; Kumar et al. 2020 Agarwal et al. 2020).
In model-free methods, policy optimization is limited to already observed states which most likely do not provide sufficient coverage of the entire state space. Alternatively, model-based methods first learn the corresponding empirical Markov Decision Process (MDP) using the offline dataset and then freely explore in the learned environment for policy optimization, which can attain excellent sample efficiency compared to model-fit free methods (Chua et al. 2018 Janner al., 2019). Most recently, model-based algorithms are specifically designed for offline settings to address distributional shifts in the learned model and have been proved effective in certain problems compared to their model-
free counterparts Yu et al. 2020 2021 Kidambi et al. 2020
Zhan et al. 2021
Swazinna et al.
2021 Chen et al. 2021 Rigter et al. 2022)
However, prominent model-based methods, i.e., MOPO YYu et al. 2020 and MOReL (Kidambi
et al. 2020), mainly leverage an ensemble of models for uncertainty quantification. Ensemble un- certainty quantification is special case of uncertainty quantification in Bayesian neural networks with latent variables using nearest-neightt methods, introduced by Depeweg et al. (2018). where each model in the ensemble corresponds to sample from the posterior distribution. In these methods,

---

Under review as a conference paper at ICLR 2023
measure of ensemble discrepancy determines the estimation error. This can be particularly restric- tive when theoretical assumptions on the ensemble do not hold in practical scenarios. In practice, an ensemble usually consists of a small number of models, where each model is a different initialization of the same neural architecture, trained on the same data. Hence, the models in the ensemble are likely to correlate to one another after training, which might make their variation poor indicator of uncertainty. Yu et al. (022)) study this behavior and demonstrate that the uncertainty estimated via maximum variance over the learned ensemble (as in MOPO) struggles accurately predict the model's 's error, , and could lead to poor performance (see Fig. 2 in Yu et al. (202T)).
Accordingly, there have been efforts to eliminate the need for bootstrap ensembles for uncertainty estimation in model-based offline RL methods. Yu et al.] (2021) propose utilizing model rollouts to conservatively learn the Q-function by penalizing the values over out-of-distribution areas, while Rigter et al. (2022) introduce an adversarial framework for training the policy and the model at the same time, such that each step, the policy trained to maximize the return, while the model is tuned to minimize it. In addition, [Tenenholtz et al. (2021) propose to quantify uncertainty using a k-nearest neighbors approach, where the distance measure is defined as an approximate metric on the learned (Riemannian) manifold in latent space encoded by a VAE. Although RAMBO- RL (Rigter et al., (022) and COMBO (Yu et al., (221)) have shown promising empirical results on standard benchmark datasets, they both forgo the modularity aspect of methods such as MOPO and MOReI., and GELATO (Tennenholtz et al.] (02T) is computationally expensive. Instead, we aim get the best of both worlds and present a general-purpose task-agnostie, computationally efficient framework to learn pessimistic model of the environment that can be coupled with any RL algorithm to learn optimal policies, without ensemble learning.
In this paper, we address this problem by proposing a novel method that eliminates the need for ensemble uncertainty quantification, while still being modular in the sense that the trained model can be combined with arbitrary RL algorithms to learn arbitrary tasks. Therefore, we present EMO, an Entropy-regularte
Model-based Offline RI. approach, that learns a pessimistic MDP using only a single model which can provide accurate estimates of the dynamics in the support of offline data while softly quantifies an upper bound for the uncertainty of model predictions when leaving the data support. To this end, we devise regularized loss function to minimize the negative likeliool of the model w.r.t. the offline data distribution, and simultaneously, maximize the entropy of pre- dictions outside of the data support in single model. Furthermore, we propose warm-start the learning procedure by only optimizing the unconstrained objective function, where the initial learned model in this step is used to generate rollouts for optimizing the uncertainty estimation.
Our extensive empirical study illustrates that our approach achieves better or on par performance compared state-of-the-art (SOTA) offline RL techniques, both model-free and model- on D4RL benchmark datasets for MuJoCo environments.
RELATED WORK
Offline reinforcement learning (Lange et al. 2012), which allows for optimizing policies from static offline datasets, has received lot of attention throughout the recent years, as the practical issues of applying online RL to many real-world scenarios became more apparent. Model-free offline RL approaches optimize a policy solely based on the visited states from the static offline data, without utilizing a learned model of the environment. Constraining the policy to be close to the behavior policy (Kumar et al. 2019; Fujimoto et al. 2019; Wu al. 2019. Fujimoto
Gu (02T), conservative estimation of value functions (Kumar et al. 2020 Kostrikov et al. 2021), incorporating the uncertainty of predictions to stabilize Q-functions (Agarwal al. [020 Wu al. 202I), and
adversarial training of actor and critic (Cheng et al. (022) are among active lines of work in model- free offline RI., However, due to their limited generalization, the performance of model-free methods is highly reliant on the optimality of the offline data.
On the other hand, model-based approaches incorporate a model of the environment to improve generalization and sample efficiency, which is used as a surrogate for the actual MDP to optimize a policy, combined with the original offline data. MOPO (Yu et al.] 2020) and MOReL (Kidambi et al. 2020) incorporate ensemble uncertainty estimation to penalize highly uncertain transitions. COMBO ( Yu et al. 202T) combines the idea of conservative estimation of value functions in CQL
(Kumar al. 2020 with model-based learning framework. RAMBO-RL (Rigter al. 2022)
2

---

Under review as a conference paper at ICLR 2023
Oclinn Dataset
Pessimistic Model
Policy
Model Optimiration
Medel Optimisution
up
Regularization
# (a|s)
Minit
Figure 1: General scheme of EMO.
proposes an adversarial framework for training the policy and the model the same time, such that at each step, the policy is trained to maximize the return, while the model is tuned to minimize it.
As discussed in Yu et al. 2021), algorithms like MOPO and MOReL that rely on ensemble uncer- tainty estimation, are prone to erroneous estimations of uncertainty. On the other hand, COMBO and RAMBO-RL are not modular frameworks; COMBO applies the idea of conservative learning of value function as a part of its proposed RL algorithm, and RAMBO-RL tunes the model to be pes- simistic with respect to the current policy of the agent; therefore, the resulting model from learning particular task with either of these methods cannot be utilized to learn new task in the environment. EMO, however, tries to eliminate the need for ensemble uncertainty quantification, while remaining modular framework that can be extended to various tasks and RL algorithms.
EMO: ENTROPY
MODEL- -ASED OFFLINE RL
3.1 PRELIMINARIES
The Reinforcement Learning (RL) problem is characterized via a Markov Decision Process (MDP) Sutton & Barto 2018), represented by . = (S, space and action space, respectively, T(s' is, a) is the transition distribution, r(s, a) is the reward function, 7 E (0, 1) is the discount factor, and po is the distribution of the initial state. A policy (als) is defined as a mapping from states distribution over actions, S x A + [0, 1]. and the goal is to learn a policy I²" which maximizes the expected discounted return 77s (x) when followed
Pu), where and denote the state
arg max 7he (TT) where TMe (T) E.
However, in offline settings, we are not able to accurately evaluate the return values under dif- ferent policies as there no further interaction with the environment. Instead, a static dataset D {(s), a,, rı, s))) !ー which is collected under arbitrary policy in the environment is pro- vided. Therefore, the objective turns into finding the best policy that can be solely optimized on the available offline data D. Note that in this context, best policy might be different from the optimal policy, as the performance of the resulting policy, regardless of the learning algorithm, is affected and capped by factors such distribution and optimality of the static data.
In this paper, we focus on model- based offline reinforcement learning, where an empirical model of the environment is estimated and leveraged to enhance sample efficiency over model-free ap- proaches. In this framework, the offline dataset is used to train pessimistic MDP AM which is then employed as a surrogate for the actual model of the environment. Subsequently, the RL agent interacts with this model and optimizes its policy based on both the acquired information and the original transitions from the offline data. Ideally, we aim to find a policy A* with min- imum opttmmiiyy with respect to the optimal policy, i.e., *" arg min The (TT" - The (T).
** arg min πy (T*) 7/ (T))
The general workflow of EMO is depicted in Figure I In this figure, the offline dataset is first utilized to train a pessimistic model of the environment in a two-phase training regime. After that, the model is considered as surrogate for the actual environment to train policy # (als).
3

---

Under review as a conference paper at ICLR 2023
3.2 MODEL OPTIMIZATION
In this section, we present a novel approach for model optimization, as the main part of EMO, that learns a pessimistic MDP using only single neural network with two aims. First, the estimated model should accurately capture the dynamics of the environment (which can only be reliable in the support of offline data.) and second, it should relatively quantify the uncertainty associated with model predictions in the form an error estimator. Consequently, we address these goals in a regularized optimization setting, where both the likelihood of the data as well as the error estimation are jointly optimized in a single model.
The error estimator, denoted by u, allows the framework to relatively differentiate between the reliable and unreliable predictions of the model, which accordingly, can be utilized to penalize unreliable predictions based on their estimated uncertainty. Let r(s,a) a) and u(8, a) be the asso- ciated reward function and estimated error for particular state-action pair (s,a), respectively. A pessimistic model can be determined by defining an alternative reward value in the form of F(s,a) a) r(s, a) Au(s, a), where À is a constant value control the amount of penalty associ- ated with the state-action pair. As a result, the policies will be prevented from exploiting unreliable predictions, which can translate conservative exploration in the actual environment.
Inspired by Yu et al. ( Kidambi et al. (2020), we characterize the one-step model of the en- vironment with a Gaussian distribution over the next state and reward, conditioned the current state and action, i.e., P(s', r|s, a) = (Ja a), Ex a)), where 0 and & are the weights of the corresponding neural networks. Both the mean vector (o(s,, a) and covariance matrix 2 (s, a) are of the size d + 1, where d is the dimensionality of the state space and the covariance matrix is sumed to be diagonal. In this work, the main reason for explicitly estimating the covariance matrix lies in the fact that the entropy of a Gaussian distribution, as quantitative measure for uncertainty, is formulated as function of the covariance matrix
+ 1) (1 + log(2x)),
H = log (et (Eo(s,
2
2
and assuming that the covariance matrix is diagonal, we have:
Σ log (s,)()) (d 1) (1 log(2
1+1
H 2
Consequently, (s, a) can solely act as the uncertainty/error estimator, while pa(s, a) is trained to model the transition dynamics. In this way, both terms can be optimized in a single model to learn a pessimistic MDP that allows for utilizing the offline data in more efficient and reliable way.
Furthermore, we propose a two-step algorithmic framework for learning the pessimistic model of the environment. During the first phase, indicated by the warm-up phase, the model is trained via maximum likelihood following the prior work ( (Yu et al.) 2020 Kidambi et al., 2020). while in the second phase, which we call the regularization phase, the model is leveraged to generate synthetic rollouts, that are then used to maximize the entropy of model predictions over the data points. Note that the warm-up phase is essential to the next phase, since the additionally gener- ated data from the model needs to be as close to the actual dynamics of the environment as possible, particularly, in the support of offline data and possibly its generalizable neighborhood.
3.2.1 WARM-UP PHASE
In this phase, employ the Gaussian negative likeliioldoood (NLL) loss to train initial model Mini of the environment the offline dataset. Let BD be batch sampled from the offline data D, the NLL objective function denoted by L1 can be written as in Equation
o; (B))
1
Σ
Σ
d+1
( log (og(Eo.(ss,a) + (pa.(s, a) -
1s..v1?)) (s,(s,
(2)
BDl
(s.a,,,)))
1=1
At the end of the warm-up phase, we expect (s, a) to provide accurate and reliable predictions for (s', r) conditioned on (s, a) in the support of the offline data. Additionally, depending on the generalizability of the model, the performance can be extended to neighborhood around the offline
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Generating Rollouts
Require: D, plo Eạ. Tex batch size b, rollout horizon h. penalty coefficient À
1: Set Bx, + 0
2: for 1, 2, 1.2,.... b (in parallel) do
3: Set <-
4: Sample state 8 from D for the initialization of the rollout.
5:
for j= 1,2, 2,., do
6:
Sample an action 0j ~ Te(s)) Obtain sj+1, ())
7:
8:
(Optional) Compute ř, = A (t)))s))),
9: Add (sj, 83+11 p end for
11: Add p to B,,,
12: end for
13:
return B..
data distribution. Similarly, Eg(s, a) can capture the uncertainty in the areas that we have the support of offline data. However, no devoted argument can be made about jeg and Eg over cuttoo-distributioo (OOD) data points, and thus, they might predict arbitrary values in those regions.
3.2.2 REGULARIZATION PHASE
If we directly employ the trained model from the warm-up phase surrogate for the actual envi- ronment and optimize policy, the learned policy will most likely perform poorly in the real MDP (Levine al. 2020). This happens because while training the policy, the RL algorithm will query the model in OOD data points as well, meaning that will rely on potentially inaccurate predictions of the model, which might lead to inferior policies due over- under-estimation of the values. Nevertheless, this is not an inherent issue in standard model-based RL as the model can be improved over time by collecting more transitions, whereas in offline settings, the problem remains due to lack of additional interactions with the real environment. Therefore, we aim to find an efficient way to ei- ther prevent the policies from exploiting unreliable predictions, or prevent the unreliable predictions from affecting the values of state- action pairs.
To address this problem, we propose using regularized objective function and a second phase for training the model. Once we have preliminary empirical model, we expand the loss function to include second objective to maximize the entropy model predictions over the OOD domains. Since the model is characterized by Gaussian distribution, entropy is a monotonically increasing function of det (Eo(s, a)) Σ, a) (Equation D.. Hence, higher entropy, which translates to higher uncertainty, results in a higher value for det (E)(s,a)) and vice versa. As a result, the maximized entropy, which can be attained by maximizing det (E a)), provides an upper-bound estimation on the uncertainty of model predictions.
Let po(s,a) a) and (s a) be the mean and covariance of the Gaussian distribution over the next state and reward, i.e., P(s', r|s, a) = - (Ho(s, a), Eg (s, (s))), and BB_, be sample batch of rollouts generated from the trained dynamics ( a) using an exploration policy Te(a]s), i.e., (s', r))
ps (s, a ~ (als))), the second loss L2 is thus defined as
d+1
Cz(0:
Σ
Σ
1
log (Eo,(s,
(3)
[
2
(3.,,,,"")
which together with C introduced in Equation yield the hybrid loss
C(O, d: Bp, B..) (1, d: Bp) (B,
(4)
and is fixed constant to control the effect of regularization the NLL loss.
Algorithm D summarizes the procedure to generate a sample batch of rollouts of length h given an offline dataset D, trained model in the form of / (Po(s, a), (s a)), and exploration policy
5

---

Under review as a conference paper at ICLR 2023
Algorithm 2 EMO
Require: Offline data D, exploration policy Te. batch size b, rollout horizon h, penalty coefficient
X, regularization coefficient a
1: Initialize 0 and Φ
2: for K₁ iterations do
Warm-up phase
3: Sample batch of transitions Bp from the offline dataset D.
4: Compute (0, o; B-)) (Equation 2)
5: Compute gradients and update and Φ.
6: end for
7: for K2 iterations do
Regularization phase
8:
Sample a batch of transitions Bp from the offline dataset D.
9:
Compute (9, d; Bp)) (Equation 2
10:
Generate a batch of transitions B using model rollouts (Algorithm
Compute (2(:; ) (Equation 3
12:
Compute L( d; Bp, IB, L1 Li d; Rp) (L
Compute gradients and update and Φ.
14:
end for
Te(als). In this algorithm, (s(ss,) models the transition dynamies and Eg(8,a) a) is used as the uncertainty estimator. We leverage an exploratory policy as well as the dynamics model, which is initially trained the warm-up phase, to generate rollouts additional to samples from offline data to be employed in the regularized optimization problem. Note that the dynamics model will be gradually updated as the training progresses. Accordingly, Algorithm plots the overall learning procedure of EMO, in which the batches of data from both the offline dataset as well as the generated rollouts from the model are utilized to minimize the hybrid loss in Equation
In the regularization phase, the model is trained to maximize the likelihood of the offline data, collected by (possibly unknown) behavior policy, while maximizing the entropy of predictions over the distribution induced by exploratory policy. Hence, Li ensures that μe and Eg maintain their accuracy in the support of offline data, while L2 aims to increase 2g as we leave the support of offline dataset (over the distribution induced by the exploration policy TT.)) As result, Eg a) can used as an upper bound indicating how reliable/accurate the trained model is for a certain pair of and a, as long as the distribution of rollouts generated by the exploration policy covers this particular state-action pair. In other words, as long as the distribution of rollouts generated from Te covers the potential distributions of other exploratory policies, (which will be used later during policy optimization in Section 3.3 (s, a) can be leveraged as the error estimator to relatively penalize unreliable predictions.
Furthermore, by considering a small value for we ensure that the effect of C2 will be negligible compared L1 where the distribution of offline data D overlaps with the distribution of rollouts under the exploratory policy T.. In this way, the performance of the model is practically unaffected the support of offline data and possibly other generalizable neighborhoods. Note that L2 will still be effective in the OOD areas, since Li is non-existent in those regions.
Algorithm 3 General Framework for Model-based Offline RL
Require: Offline dataset D = {(s;, a;, ri, > penalty coefficient A.
1: Train the dynamics model μıg and admissible uncertainty estimator Es using D. (Algorithm
2: Define u(s, a) / det (8,
3: Define empirical MDP AM with dynamics no and reward F(s, a) = r(s, a) Xu(s, a).
4: Run any RL algorithm on M until convergence to obtain A* arg max n (T).
x
3.3 POLICY LEARNING
Once the model is trained using EMO, we will utilize its components to define a pessimistic MDP, which be coupled with any policy optimization technique to obtain the output policy #*. Sub-

---

Under review as a conference paper at ICLR 2023
sequently, the overall learning framework, which also adheres to prior work (Yu al. 2020: Ki-
dambi et al. 2020). is summarized in Algorithm 3 In this algorithm, the offline dataset D is first used to train dynamics model μlg along with admissible error/unceraainty estimator Σg. Next, a new pessimistic MDP is defined as M A, 7, Pu), with TN = {uo,(s, (s. a)} and F(s, a) = r(s, a) Au(s, a). where w(s, a) = vir(2 (s, a)) (see section 3.2 Lastly, the pes- simistic MDP M is leveraged as surrogate model to train an RL algorithm and obtain #*.
3.4 THEORETICAL GROUNDS OF EMO
We argue that EMO is an extension to MOPO YYu al. 2020), while addresses the limitations of MOPO regarding learning an ensemble of models for uncertainty estimation. Accordingly, we ex- pand the theoretical grounds of MOPO EMO, where we guarantee conservative policy evaluation and safe policy improvement of EMO, regardless of the stochasticity of the environment. For a detailed discussion on the theoretical analysis, refer to Appendix A.1
4 EMPIRICAL STUDY
In this section, we evaluate the performance of our approach on D4RL benchmark datasets for Mu- JoCo environments (Fu et al. 2020). We include data from three different environments: halfchee- tah, hopper, and walker2d, and four different types from each, i.e., random, medium, medium-replay. and medium-expent which results in twelve different datasets. Throughout the experiments, we aim to investigate the following questions: (1) How does EMO perform compared to SOTA methods on standard offline RL benchmark? (2) What impact do entropy regularization and reward penalty have on the performance of the trained policies? (3) What is the effect of exploration policy Te the generalization ability of the trained models and the performance of the resulting policies? (4) What the effect of regularization coefficient a on the performance of EMO?
4.1 EXPERIMENTAL SETUPS
Following the setup in Yu et al. (2020), we characterize the model as a 4-layer feed-forward neural network across all domains, with 200 hidden units in each layer. Subsequently, the output of the last hidden layer is fed into a two-head network architecture to generate ( a) and (s a), where Me and Eg are two outputs of single neural network, i.c., they share the same network, except for their output layers. Instead of directly estimating the reward function, the model predicts the center of mass velocity, and the reward is calculated afterwards based on its formulation in each domain.
For all the experiments, a soft actor-critic (SAC) agen (Harnoja et al. 2018) is used as the re- inforcement learning agent for policy optimization on the trained pessimistic MDP M. At each time step, batch of k-step rollouts are generated and added to the replay memory of the SAC agent, where the actions for generating the rollouts are taken based on the current policy of the agent, while transitions and rewards are produced by M. Next, the agent optimizes its policy based on samples from both its replay memory and the offline data D. Note that in our implementation, the agent will only utilize its own replay memory for policy optimization, meaning that samples from offline data are not directly utilized for policy optimization (see Algorithm 5 in Appendix A..).. Consequently, the resulting policies from SAC are evaluated in the real MuJoCo environment for testing.
4.2 OVERALL PERFORMANCE
We compare the performance of EMO SOTA model-based offline algorithms, i.e., MOPO Yu et al. 2020). MOReL (Kidambi et al. 2020), COMBO (Yu et al. (021)). RAMBO-RL (Rigter et al. 2022). and GELATO (Tennenholtz et al. 2021), well as model-free counterparts, i.e., UWAC Wu et al. 202T), CQL (Kumar et al., 2020), and ATAC (Cheng et al. 2022). The performance results of both model-based and model- free techniques are summarized Table T All the presented scores are normalized according to the procedure proposed in Fu et al. (2020). For EMO, the results are the performance of policy at the last iteration of training, averaged over random seeds +
7

---

Under review as a conference paper at ICLR 2023
Table 1: Performance results of offline RL algorithms on D4RL benchmark datasets.
Dataset
Environment
EMO (Ours)
RAMBO-RL
COMBO
MOPO
MOReL
UWAC
COL
ATAC
GELATO
randoen
random
bopper
31.6
0.1
25,4 1:
0.6
11.7
22.4
10.8
17.5
21.2
randeen
walker2d
5.3
2.9
0.0
= 0.1
7.0 1,5
13.6
37.3
13.5
7.0
6.8
9.0
medium
balfcheetah
0.9
77.9 1:
0.6
42.3
46.5
52.3
hopper
medium
walkerld
84.1 2.5
84.9 +
$1.9
17.8
77.
31.5
79.
89.6
27.5
medium replay
ballcheetsh
56.7 5.9
68.7 1: 1.7
55.1 $5.14:0.4
53.1
40.2
46.8
46.2
43.0
medium--eeply
77.4 16.9
99.5 1:
39.5 1.0.
67.5
39.4
102.5
mediumm-replay
89.2
medium expert
ballcheetah
80.2
3.9
95.4
90.0
2.3
63.3
53.3
127.4
62.4
91.8
65.8
medium-este
hopper
92.1
-: 2.8
88.2 1: 6.5
111.1
23.7
108.7
134.7
111
111.9
17.7
medium-ese expert
walker2d
110.8
3,0
12.3
103.3 2.5
standard error. The results for CQL are taken from D4RL benchmark white-paper (Fu et al. 2020). As for the values other methods, the result are taken from their respective papers.
The outlined results in Table demonstrate that EMO outperforms MOPO and GELATO in almost all cases, which shows that EMO, as an extension to MOPO, clearly improves upon its predecessor by replacing ensemble uncertainty quantification with entropy regularization. Furthermore, EMO achieves competitive results compared to COMBO and MOReI, outperforming both on out of 12 datasets, which places EMO among the piehest-performing model-based algorithms. Although EMO can achieve comparable results to RAMBO-RL in certain scenarios, its performance falls short some cases, and only outperforms RAMBO-PE RL. on one dataset. This can be attributed to the fact that RAMBO-RI is task-specinf method, which tunes the model to be pessimistic with respect the current policy of the agent, while EMO utilizes a general purpose, task-agnostic model of the environment for policy optimization. In addition, bear in mind that EMO achieves this level of performance using only a single model the environment, while an ensemble of models utilized in other model-based methods. Morcover, Table illlustraess that EMO outperforms CQL and UWAC techniques in 8 out of 12 datasets, and performs competitively against ATAC, while outperforming ATAC 4 out of 12 datasets, which highlights the effectiveness of simple, efficient method keeping up with the SOTA baselines.
Table 2: Performance results policy optimization on different model contigurations
Dataset
Environment
EMO
NLL.+AC (A #
NLL+SAC (A
medium
halfcheetah
68.5 0.9
35.1 8.0
0.9
medium
walker2d
84.1 = 2.5
13.0 + 13.0
5.6
medium-replay
halfcheetah
56.7 = 5.9
A.0
10.8 - 1.6
medium-replay
walker2d
83.6 1.3
18.7 1 15.8
6.4 1 1.2
4.3 ABLATION STUDIES
In order address question (2), conduct an experiment compare the performance of EMO with two simplified versions of the algorithm: (i) training the policy on a model that is only trained via the NLL loss (i.e.. the resulting model from the warm-up phase) without utilizing the reward penalty (A = 0), denoted by NLL+SAC (A 0), and (ii) the same model with the reward penalty (A & O) leveraging the covariance matrix from the warm-start phase, indicated by NLL+SAC (A 0). The outcomes of the experiment are outlined in Table 2 on 4 different datasets. The results confirm that although penalizing the reward values in OOD areas improves the performance of the model, it is still insufficient to achieve comparable results to state-of-the-art The reason lies in the fact that the penalties depend on the arbitrary predictions of the covariance matrix in the OOD domains. However, by incorporating the entropy regularization step, ensure that reward penaltics will be proportional to the associated uncertainty of the predictions, which leads to a considerable gap between the performance of the resulting policies. This finding empirically validates the impact of the entropy regularization coupled with penalizing the reward values in uncertain regions.
In the second ablation study, investigate the impact of using informed exploration policy vs. random policy to address question (3). One could argue that for a specific task, utilizing an informed policy (i.e., the current policy of the agent) instead of random policy can be beneficial. In this case,
8

---

Under review as a conference paper at ICLR 2023
Table 3: Performance results of policy optimization for random vs. informed exploration policy
Dataset
Environment
EMO
Modified EMO
RAMBO-RL
medium-replay
halfcheetah
56.7
61.1 = 0.4
68.7 1.7
medium-replay
walker2d
83.6 + 1.3
85.5 3.5
89.2 1 2.1
the entropy regularization phase can be guided more efficiently toward the areas that are more likely to be explored by the learning agent. Subsequently, the coverage of the rollouts generated under informed policy is expected to be a small subset of the rollouts generated from a random policy.
Consequently, we propose modified version of EMO, summarized in Algorithm 4in Appendix AA.5 in which the (SAC) agent and the model (only the regularization phase) are trained simultaneously. rather than modular. In this variant, the model utilizes the current policy of the agent as the explo- ration policy for entropy regularization in order to benefit from an informed exploration scheme. We further examine this modification on two offiin datasets, namely walker2d-medium-repp medium-replay, compared to RAMBO-RL. The summarized results in Table [ indicate that informed exploration can indeed improve the performance of trained policies. Such improve- ments in the performance expected since the modified version of EMO has a more specialized, task-specific approach model optimization compared to the original version. Furthermore, though EMO can achieve comparable results compared RAMBO- RL and even outperform it one set of data (see Table [T. the modified version can tighten the gap and make the results more competitive RAMBO-RT. However, we still prioritize the original EMO, as it has general pur- pose, task-agnostic pessimistic model of the environment, which can be employed in any modular framework with any RL algorithm for training policies. Whereas this cannot be achieved using the modified version of EMO or methods such as RAMBO-RL.
and
walker2d
To answer question (4), demonstrate the per- formance of EMO and modified EMO for different configurations of the regularization coefficient a on walker2d-mecium-rqph repllyy dataset in figure 2 The figure illustrates how a affects the performance of both methods, also compared to RAMBO-RL. As we increase a, the regularization term in Equation 4 becomes more dominant, leading to more con- servative algorithms, and vice versa. As a result, smaller values of preferred to achieve best per- formance when offline data has limited coverage of the state-action space contains less informative data. Conversely, when the offline data provides a better coverage, higher values for a are preferred. In addition, since the datastes from the medium-replay category are considerably smaller than other D4RL benchmark data (aka limited coverage), both EMO and modified EMO are expected to perform better when is set to smaller numbers, shown in fig- ure However, if a is too small, then the algorithms will not regularize very well, and that can lead decrease in performance as well. The same can be said when the algorithm becomes too conservative, which is the case when a is set too large.
RASIBO EL BSNO Mislined IMV
130
110
Normaliaed
70
50
10
0.0001
0.01
Figure 2:
Performance of EMO variants in
terms of a on malker2d-medium-rrpla
5 CONCLUSIONS
In this paper, we presented EMO, an entropy-regularized optimization algorithm to learn a pes- simistic MDP for model-based offline RL problems. In this framework, we devised hybrid loss function to minimize the NLL the model on the distribution of offline data while maximizing the entropy OOD domains. We thus optimized both objectives in single model rather than ensemble of models as in SOTA model-based approaches. Morcover, our empirical study on D4RL benchmark data showed that our approach competes with SOTA offline RL techniques.

---

Under review as a conference paper at ICLR 2023
6 REPRODUCIBILITY STATEMENT
To ensure the reproducibility of the results, the codes are provided in the supplementary materials. Configurations and the choice of hyperparameters are also included in the supplementary materi- als in readme.. txt file. To facilitate the understanding of theoretical formulation and practical implementation, additional algorithms and theories are also included in the appendix.
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference Machine Learning, pp. 104-114. PMLR, 2020.
Xiong-Hui Chen, Yang Yu, Qingyang Li, Fan-Ming Luo, Zhiwei Qin, Wenjie Shang, and Jieping Ye. Offline model-based adaptable policy learning. Advances in Neural Information Processing Systems, 34, 2021.
Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. arXiv preprint arXiv:2202.02444 2022.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018.
Stefan Depeweg, Jose-Miguel Hermandez-Lobato, Finale Doshi- and Steffen Udluft. Decom- position of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. In International Conference on Machine Learning, pp. 1184-1193, PMLR, 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2062. PMLR, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Jennifer Dy and An- dreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul 2018. URL PRLbttps:: http::s://ppocceedinno
mlr www.sppsssssepp..e... http://0//hhaaaaa fal8b............ html
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine When to trust your model: Model- based policy optimization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché A'Aché-Buc, E. Fox, and R. Gamnett (eds.). Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL httpp:::pp...........p... http:///oooeeee/ige//eee h http55fffS6eefff30.cccca22266363 neg.rrrrssss.................... http:ipssccppppp////1990
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Bal- can, and H. Lin (eds.), Advances in Neural Information Processing Syxtems, volume 33, pp. 21810-21823. Curran Associates, Inc., 2020. URL httpt::h/pooceeeeeeeeeeee ngs new.rirrissssnneenneeeee..... 9b88d4 3524442233345 hc/paper/222D//LLe//eeeeeee Sat.E//EEEE...000000222222211000 Papprr
llya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5783. PMLR, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- fff-policy q- learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
10

---

Under review as a conference paper at ICLR 2023
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179- 1911199-111,, 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning, In Reinforce- ment learning, pp. 45-73. Springer, 2012.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tu- torial, review, and perspectives on open problems, 2020. URL WKLLittps://axxiv.rrg abs/ 2005.01640
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Toannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature, 51 (177750):529-538 2015.
Mare Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning, 2022. URL
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanetot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play, Science, 362(6419): 1144, 2018
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Phillip Swazinna, Steffen Udluft, and Thomas Runkler. Overcoming model bias for robust offline deep reinforcement learning. Engineering Applications of Artificial Intelligence, 104:104366, 2021.
Guy Tennenholtz, Nir Baram, and Shie Mannor. Gelato: Geometrically enriched latent model for offline reinforcement learning, arXiv preprint 133: 2021.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1rX1v:1911.11361, 2019.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.088 202.00008444 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33. pp. 14129-14142. Curran Asso- ciates. Inc., 2020. URL http::tttpppccceeeeeeeee neuripss.......................e cc/paaper/222/E11e .df
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in Neural Information Processing Systems, 34, 2021.
Xianyuan Zhan, Xiangyu Zhu, and Haoran Xu. Model-based offline planning with trajectory prun- ing, arXiv preprint arXiv:2105.07331,

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 THEORETICAL GROUNDS OF EMO
As discussed Section 3.4 EMO can be considered as an extension to MOPO Yu et al. 2020 with the aim to address the limitations of ensemble uncertainty estimation. Accordingly, in this section, we expand upon Lemma 4.1 from MOPO in order to establish the theoretical grounds for EMO.
A.1.1 PRELIMINARIES
We specify Markov decision process (MDP) by the tuple M (S,A, Ho, 7), where S and A denote the state space and action space, respectively, T(s'|s, [s, a) is the transition dynamics, r(8, a) is the reward function, 7 E (0, 1) is the discount factor, and μig is the distribution of the initial state, A policy π [aass) is defined as a mapping from states to a distribution over actions : S x A + [0, 1]. If we define P(s, s|mg; n, T') as the probability of being in state at time step when following policy x from an initial state sampled from Po. in an environment with transition dynamics T, then the discounted state-action visitation distribution of policy under dynamics can be defined as (7, a) (1 (als) Er=o 7 7 = s|/00 ,, T). The goal is to learn policy that maximizes the expected discounted return 7h (T) when followed: max, [as (TT) == 1
I-7 (1.0)))p3( [r(s, a)]-
The value function, (s) Ex,1 T 22 at)lso s). regarded as the value of particular state s under the policy which is defined as the expected discounted return under т when starting from 8.
In offline RL framework, we only have access to a static dataset of transition tuples D {(8, a, r, s')}, which collected by running a behavior policy n/ the real environment. In offline RL setting, the goal is to find the best possible policy using the static offline dataset D.
A.1.2 THEORETICAL FORMULATION
We start by expanding the theoretical formulation of EMO and proving that EMO guarantee conservative policy evaluation and safe policy improvement. First, we quantify the relationship between the performance of a policy iT under two arbitrary MDPs. From the theoretical formulation of MOPO (Yu et al. 2220),, we have:
Lemma A.1.1 (Telescoping lemma). Let M and M be two MDPs with the same reward func- tion but different dynamics T and T respectively. For any arbitrary policy let G M
(s,a) ==
E B [M Ey (VT Then,
M (T) - (TT) = 2 E (o (7 (s [G (s, a)]
(5)
1-7
Next, we expand on Lemma A.1.1 to include MDPs with different rewards as well.
Lemma A.1.2. Let M and M be two MDPs with reward functions r and f, and transition dy- namics T and T respectively. For any arbitrary policy let GT. M VV and f(s,a) r(s,a) ar. Then,
(s,a) = E Ee LVI (s'))
Tx (T) - (TT = 1-> (sa + (s,a)
(6)
Proof. Considering f(s,a) r(s, + òr(s, we have
1 -ag
Tai (T)
E E1sa)) I-7 -
The first term in the RHS of Equation corresponds to the return of the policy under dynamics π and reward According Lemma A.I.I we have
12

---

Under review as a conference paper at ICLR 2023
1 y-y E 0 = 7aa (T) + 1-7 (a
7
(8)
Now, by substituting Equation 8 into Equation
we get
γ ( (s (0. G (3, + 1-7 (isaa)-pp(sa((((sss.)))
1
1 1- (x)(()) - -Ta((T)=
(9)
B ( + 7
Lemma A.1.2 quantifies the difference between the return of a policy under two arbitrary MDPs, as long as they share the same (S, A, Po, (o.7). For the specific case of EMO, the actual MDP, and M the learned MDP the offline data, based on EMO algorithm, before
consider M as
applying the reward penalty. For now, we model the penalized reward function by f(8, a) = f(s, a)
Aår(s, a), where AAr(s, a) is the penalty apply the learned reward, such that À 2 0. Let M (S, T, No: 7) denote the MDP with transition dynamics I and reward function F(s,a). We have
Mi (T)=
E (s (6
1 -
(10)
X
E Bisa))
Peaaropptaalarr Arr(s)
E (G (2.))
7
The first term in the RHS of Equation 10 corresponds to the return of the policy under dynamics f and reward f, namely (T)). Using Lemma A.1.2 we can rewrite this term as
( (T)
E Booa)oot( (T) 1 ((sa)=of(aa))0 (or(ss +70 (s, a)
E
1
(11)
By substituting 11 in 10 we have
A 1 1-
(T)(T) T/M (MT)
= 1
E (n)))) + (s
B (,, (p
Ar a)]
(12)
In order to achieve conservative policy evaluation, we need to ensure that the performance any policy not overestimated under the penalized MDP M. Thus, we need to ensure that 7 (x) Mas (T)) for all n.
Proposition A.1. The penalized MDP M preserves conservative policy evaluation if.
(Ar(s XBE (0 [Ar(s, a)) 2 Boo.))ogttsa)lr(ss... 7GT (s, VT EI
(13)
Proposition A.I establishes the condition on the adjustment AAr(s, a) to guarantee conservative policy evaluation. Once again, note that in order preserve conservative policy evaluation in the penalized MDP M, the condition stated in 13 should hold for any arbitrary policy A direct implication of Proposition A. that training any policy in M is equal optimizing lower bound on the return under the real MDP M.
Practical Implication. If ensure [(s.a)=of (rr a)] 0, Vn E TI, can satisfy the condition on conservative policy evaluation by choosing large enough, assuming that ör(s, a) and G (8, a) are bounded.
M
13

---

Under review as a conference paper at ICLR 2023
In addition to conservative policy evaluation, want to guarantee safe policy improvement over the behavior policy T well. Let i be the optimal policy trained on the penalized MDP M. We first quantify the performance difference between n and i under the actual MDP M:
7a (T) TTM
7 (T)
( (2 (n,)) + G (s,0))
E ((a))
1 1-
1-7
1
x
-1 +
E
(c.))rop (0,a) fr (r((s a) G (s, a))
E (0,)) (4.) Ar
1
= [ - -7h5 (TT
B ((aa)) [ a) ( (s,
+
7 E (e.a) (c. (r((s a) G M (s,
X
+
(B (B(a))) |rr(s a) E (sa)) (s.a)
(sa)))(rssss... (Ar(s,)))
Note that since न is the optimal policy under the penalized MDP M, then we have Maa (R)
Vai (TD) C2 BT 0.
Proposition A.2. Let i(als) be the optimal policy trained on the penalized MDP M. Then, क is safe policy improvement over x i.e. nas ( 2 TM TM () if
A(B A(B(ssa)=0 (sa) (Ar(s, a)] B (sa))p3" (s.a) (na))A((s)))) [A((s,)))
(e. (ōr(s, + (8,a))
(14)
E (na)mp3 (s,a) (or(r + + (s, - (1 - ()
which satisfies
(#) - "nM (T=")
-Cy (m B)
R (( (na)) G
(15)
1 (00)~03 (ör(s a) + G (s,
(B. Ar(s, a)] E (aa) (n.a) Ar(s,
Proposition A.2 establishes the condition on the penalty AAr(s, a) to guarantee safe policy improve- ment over π mn the form of Inequality 14 and quantifies the improvement over π the form of
fo
Equation 15
Practical Implication. we can ensure (ss.a))pg((sa) (A((s) a)) 2 W (s.a))0;" (e.o) (e.))Arr( [Ar(s, a)}.
we can satisfy the condition for safe policy improvement by choosing À large enough, assuming that ör(s, a) and GH M (8, a) bounded. Another similar yet more practical approach could be to ensure that (s..))=p, (s.a) [Ar(s,)) a)] E (sa)))77 (s.a) [Ar(s, a)] for all æ E II, which subsumes the
original condition.
Conservative policy evaluation of EMO. In EMO, we define the penalty term positive, increas- ing function of the entropy Ar(s, a) u(s. a) / det (s, Thus, we have Ar(s, a) > for all (s, a) E S A, and it obvious that (s.aa) [Ars, a)] for all E II; as a result, according to the practical implications Proposition A.I conservative policy optimization can be achieved by choosing À large enough.
14

---

Under review as a conference paper at ICLR 2023
Safe policy improvement of EMO. Although we cannot theoretically guarantee safe policy im- provement for Ar(s,a) /dee(Eaa (E, (s. a)) (which is also the case for MOPO Yu et al. 2020
MOReL Kidambi et al. (2020), and COMBO Yu et al. (202I)) it is reasonable to assume that in practice, regardless of the stochasticity of the environment, Ar(s, a) is smaller over Piz (8, a) than (7 a) for any arbitrary я. Note that throughout the regularization phase of EMO, we specifically try to maximize the entropy over the OOD domains, and as result, we expect lower entropy over the support of data, which we can assume has distribution very close to P7 (8, a). By defin- ing the penalty term as a positive, increasing function of the entropy, that (c..a) [Ar(s, 2 E (ss.))=p;"" " (s.a) [Ar(s, 4)] for all π E II, and guarantee safe policy improvement according to practical implications of Proposition A.2 by choosing À large enough.
can practically assume
Thus, EMO in its original formulation, can be applied to any environment, whether deterministic or stochastic, and guarantee conservative policy evaluation and safe policy improvement.
A.2 DIFFERENTIATION BETWEEN ALEATORIC AND EPISTEMIC UNCERTAINTY AND APPLICABILITY TO STOCHASTIC ENVIRONMENTS
Our model does not explicitly differentiate between aleatoric and epistemic uncertainties over OOD domains, and penalizes the rewards based on an upper bound over total uncertainty, instead of only epistemic uncertainty. However, we argue that this is a good approach in practice, even in the case of stochastic environments. In general, uncertainty, be it aleartoric or epistemic, is source of error in RL algorithms. If we have a reliable and accurate estimation of either of these uncertainties, we can calculate their effect in our evaluations and algorithms. Otherwise, we should find ways to indirectly account for these sources of error, in order to prevent our methods from exploiting potential errors caused by these sources (e.g. by forming performance lower bounds with reward penalties etc.).
Same as any other measure, we argue that aleatoric uncertainty cannot be reliably quantified over OOD domains either, especially when we only have single model; even if we have ensemble of models, where we can estimate the aleatoric uncertainty by averaging over the variances of each model (Depeweg et al., 2018), the estimation would be inaccurate, as in practice, the sample size from the posterior distribution of parameters (the number of models in the ensemble) is small and samples are potentially correlated for the reasons discussed in the paper. Thus, alcatoric uncertainty over OOD domains becomes an unmeasurable source of error itself. As a result, it is reasonable to penalize a measure of total uncertainty over OOD domains, rather than only penalizing epistemic uncertainty, as we cannot confidently rely on the estimated aleatoric uncertainty over such domains. As for the in-distribution alcatoric uncertainty, we will show that our method preserves the reliable estimation of in-distribution aleatoric uncertainty. For this, we will discuss the characteristics of model trained based on EMO over the support of data as well as OOD domains. We first restate the formulation of the proposed hybrid loss in Equation 4
C(O, d; BD, Br.) = (0, o: Br) + (or
where L1 corresponds to NLL loss, L2 corresponds to entropy regularization term, and a is the regularization coefficient. Please note that as discussed in Section 3.2.2 of the paper, we assume a has a small value.
(i Since we assume that a is small, hybrid loss will be dominated by the NLL loss (L1) over the support of offline data. As result, (s, a) will actually correspond to the aleatoric uncertainty over those regions (which, in the case of deterministic environments, is close to zero). Note that this estimation is reliable since it is learned over the support of offline data.
(ii Over OOD domains, NLL loss (C1) does not exist (as we do not have any supervised data over OOD domains), and hybrid loss will be dominated by entropy regularization term. Thus. Eg(s, a) will correspond to an upper bound of total uncertainty/error over OOD domains.
As result, the estimated aleatoric uncertainty over the in-distribution data, is preserved by Eg(s, a), as discussed in (i). We also present in Table 4 the average value of /det(E (s. a)) over the samples drawn from the offline dataset for: (1) a model trained based on EMO: and (2) an NLL model, which goes to show that the uncertainty quantification is practically unaffected over in-distribution data,
15

---

Under review as a conference paper at ICLR 2023
meaning that the reliable aleatoric uncertainty is preserved in model trained by EMO, which can be accounted for with minimal modifications to the original EMO algorithm, e.g. by excluding from the penalty term.
Environment
Dataset
EMO
NLL
walker2d
medium
1.340
1,414
walker2d
medium-replay
2.694
2.792
walker2d
medium-expert
1.273
1.378
Table 4: the average value of /det(E a)) over the samples drawn from the offline dataset D for: (1) model trained based on EMO; and (2) an NLL model.
Still, we would like to mention that EMO in its original form proposed in this paper, theoretically guarantees conservative policy evaluation and safe policy improvement, regardless of the stochas- ticity of the environment (please refer to our theoretical analysis in Appendix A.I].
A.3 PRACTICAL EFFECTIVENESS OF ENTROPY MAXIMIZATION
We present in Table 5 the average error (penalty) predicted by EMO trained on different datasets over: (1) Samples drawn from offline dataset D; (2) Generated rollouts of horizon H = 2 using random exploration policy; and (3) Generated rollouts horizon = 5 using a random exploration policy. Please note that the predicted penalty for each transition directly corresponds to the upper bound of total uncertainty attributed to the transition by EMO. The difference between the average predicted errors goes show that our method of entropy maximization is indeed effective and trust- worthy, as there is a distinguishable difference between uncertainty estimation over in-distribution data (samples from D), and datasets that dominated by OOD samples (generated rollouts). The difference between average predicted errors over rollouts of different horizons, however, depends on the value of hyperparameter a. When large, the algorithm more conservative, thus it is expected to see small margin between the average eтror for rollouts of different horizons, as is the and malkerdddmedium--xppp
case for expect to see a larger margin between the average error over rollouts of varying horizons, as is the case of walker2d- walker2d-meddiu--eenl medium-replay.
but when comparatively smaller, we
The difference between the relative scale of predicted errors over different datasets comes from the upper bound on the predicted entropy. We view the optimal relative scale of OOD uncertainty against in-distribution uncertainty optimality). As a result, in our practical implementation
function of environment and dataset charachteristics (e.g. coverage,
EMO, we set an upper bound on the uncertainty prediction of the model, i.c. Eg(s, a) < Σmaz, V(s. a) S A, which indirectly con- trols the scale of OOD uncertainty against the in in-distribution uncertainty. For each environment and dataset configuration,
treat Emaz hyperparameter, and optimize it along other hyperparame- ters of EMO. Although our observations are not conclusive and generalizable, we have observed that environment-datien configurations which allow models to have better generalization (c.g. stable en- vironments such as halfcheetah, and datasets that provide broad coverage such as medium datasets), will have smaller optimal values for Ymaz. In addition, we have also observed that optimality of the dataset can affect the optimal value of Emaz. Datasets with more optimal transitions tend to have larger optimal values for Σmaz.
On top of that, we conduct another experiment to compare the average of predicted penalties of EMO and MOPO associated OOD samples. this experiment, we train the models of EMO and MOPO on walker2d- aalker2d-medium-ree medium-replay dataset, and calculate the average of predicted penalties of each model on generated rollouts under the learned models using random exploration policy. For rollout horizon H 2, the average of predicted penalties for EMO was 7.198 compared to 2.818 of MOPO; and, for rollout horizon H = 5. the average of predicted penalties for EMO was 8.544 against 3.011 of MOPO, which shows that EMO takes more conservative approach by penalizing an upper bound of uncertainty rather than penalizing a (potentially inaccurate) estimation of the uncertainty.
16

---

Under review as a conference paper at ICLR 2023
Environment
Dataset
D
H
walker2d
medium
1.340
11.386
11.465
walker2d
medium-replay
2.694
7.198
8.544
walker2d
medium-expert
1.273
30.962
31.177
Table 5: Average error predicted by EMO for 3 different datasets over: (1) Samples drawn from offline dataset D; (2) Generated rollouts of horizon H 2 using a random exploration policy: and (3) Generated rollouts of horizon H = 5 using a random exploration policy.
A.4 COMPUTATION AND MEMORY EFFICIENCY
Although cannot directly compare EMO against previous ensemble-based methods in terms of computation resource (as their implementations are different, which can heavily affect such mea- sures), we still provide some indirect indications of the computation and memory efficiency EMO. EMO manages to achieve this level of performance with about 0.34M parameters, while ensemble methods (MOPO, COMBO, RAMBO-RL) operate with around 1.IM parameters, which an indi- rect indication that EMO improves upon SOTA in terms of memory and computational efficiency. In addition, we present in Figure the performance progress of EMO against MOPO on walker2d- medium-expert dataset, averaged over 3 random seeds, where can attribute the faster convergence rate EMO as an indirect indicator of its computational efficiency compared to MOPO.
Hill
300
1003
200
3000
Saration
Without moving average.
(b) Moving average of window size 10 for both EMO and MOPO.
MONO
500
Iecrtion
(c) Moving average of window 10 for EMO and of size 100 MOPO.
Figure 3: Performance progress of EMO against MOPO walker2d-medium- aged over random seeds. The raw data is depicted in Figure 3a order to make the the raw figure more interpretable, apply two configurations of moving averages over the results of EMO and MOPO.
dataset, aver-
17

---

Under review as a conference paper at ICLR 2023
A.5 REFERENCED ALGORIHHMS
Algorithm Modified Version EMO
Require: Offline data D, batch size b, rollout horizon h, penalty coefficient A, regularization coef-
ficient
1: Initialize 0 and ø
2: Initialize policy
3: for K iterations do
Warm-up phase
4: Sample batch of transitions Bp from the offine dataset D.
5: Compute L (0, o; Bp) (Equation 2)
6: Compute gradients and update and Φ.
7: end for
8: for K2 iterations do
9:
for K3 iterations do
Sample batch transitions BB from the offline dataset DD
11:
Compute C1 (0,00 Φ; p (Equation 2)
12:
Generate a batch transitions B. using model rollouts (Algorithm) !
13:
Compute (d; (B,)) (Equation
14:
Compute L(0, o; B. B. L1 (8, BD) + aL2(0; Bx.).
15:
Compute gradients and update and ø.
16:
end for
Define empirical MDP M with dynamies μm and reward F(8, a) ((8,a) -u((s a), where
u(ss (ss a) = /det (s, for K4 iterations do
18:
Update on M with any arbitrary RL algorithm.
20: end for
end for
22:
return
Algorithm Policy Optimization Method for Experiments
Require: pessimistic MDP M AS, 7/ Po), rollout batch size b, rollout horizon h, offline dataset D
1: Initialize policy æ and empty replay buffer 2 Poodel
2: for K iterations do
for 1.2...., (in paralle!) do 4: Sample the initial state $1 of the rollout by sampling from offline data D.
3:
5: for j 1, 2..,, / do
6:
Sample action based the current policy of the agent aj (83)).
7:
Compute the next state in the pessimistic MDP s' (T,,,))))))
8:
Compute the reward in the pessimistic MDP F3 = F(sj,aj).
9:
Add transition (sj, ajy Fj, 8/) to Dodett
10:
end for
end for
12: Draw samples from D U D modelt use SAC to update
end for
18


Reviews:
[{"review_id": "eSax2KhmXl", "reviewer": "Reviewer_8Cwi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents Entropy-regularized Model-based Offline RL (EMO), a model-based offline RL algorithm that learns a pessimistic MDP, where the uncertainty quantification is performed without using ensemble models. The dynamics model is trained in two phases. In the first warm-up phase, the model is learned by maximizing the log-likelihood. In the second regularization phase, an additional dataset is generated by an exploration policy on the learned model, and the model is optimized by maximizing the log-likelihood for the offline dataset while maximizing the entropy for the generated dataset by the exploration policy. In the experiments, EMO performs competitively with the baseline algorithms.\n", "summary_of_the_review": "Although the proposed method is appealing in that it learns only a single model for uncertainty estimation, the experimental results are not convincing enough by mixed results. Also, I have a concern that the method would be difficult to deal with stochastic environments since aleatoric uncertainty and epistemic uncertainty cannot be distinguished in the proposed framework.", "strength_and_weaknesses": "[Strengths]\n1. This paper presents a method that provides uncertainty quantification using a single model. This is favorable in terms of computational efficiency, compared with the existing model-based offline RL methods that use ensembles.\n\n\n[Weaknesses]\n1. It seems that the method cannot be applied to stochastic environments. In the proposed framework, it is unclear how to differentiate between aleatoric uncertainty and epistemic uncertainty when using a learned covariance matrix for uncertainty quantification. I think we may need to penalize epistemic uncertainty but not penalize aleatoric uncertainty.\n2. Mixed experimental results: EMO outperforms baselines in some domains but underperforms in other domains.\n3. Modified EMO shares a similar spirit with RAMBO-RL, in terms of adversarial model update, but RAMBO-RL still outperforms Modified EMO in Table 3.\n4. In Figure 2, it seems the performance of EMO is highly sensitive to the hyperparameter $\\alpha$.\n\n[Questions]\n1. Do $\\mu_\\theta$ and $\\Sigma_\\phi$ share the lower-layer network parameters? or are they completely separate networks?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aDOJumrALfZ", "reviewer": "Reviewer_hpPE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a new method for model-based offline RL.\\\nIt uses both a simple uncertainty estimate based on the modeling error instead of an ensemble-based uncertainty estimate of the model, and maximizes the entropy of the model outside the data distribution. \\\nThe method is tested on multiple datasets from two different deterministic MuJoCo benchmarks against various model-based and model-free offline RL methods and achieves competitive results.", "summary_of_the_review": "A solid paper presenting another approach to avoid misestimation of the return in OOD situations in model-based offline RL.\\\nThe proposed method is simple and effective according to the experiments. \\\nA few small improvements should be made. \\\nFor the future, it would be important to investigate whether the approach is also effective in stochastic environments.\n", "strength_and_weaknesses": "**Strengths**\n- The paper addresses an important topic.\n- The method delivers good results on the benchmarks studied.\n\n**Weaknesses**\n\nIn my opinion, there is no major weakness that would justify rejection.\n\n**Suggestions for improvement that should be implemented**\n- The text still needs minor corrections. For example, lowercase is used several times where uppercase is necessary (e.g. \"algorithm 5\", \"appendix A.2\", \"table 3\", \"equation 4\"), while uppercase is used in at least one place where lowercase is required (\"In this Algorithm,\"). There are still broken sentences, like \"that can lead a decrease in performance as well\". In the bibliography, some words are wrongly written in lower case, like \"fisher\" or \"\"q-learning\".\n\n- When discussing uncertainty estimation without ensemble for offline RL, the older paper [1] should be discussed.\n\n- Three repititions of the experiments are very few. The results could be significantly different in the long term mean. A standard deviation calculated using only three values is very unreliable.\n\n\n**Suggestions for improvement in the future**\n- The method should be tested on a stochastic benchmark. \nWhen estimating reward uncertainty based on the modelling error, as in this paper, it is not ensured that only epistemic uncertainty is used and not aleatoric uncertainty. In case of deterministic environments, as used in this paper, this is not a problem because the aleatoric uncertainty is zero. In stochastic environments, however, it could be that the approach is sub-optimal because it also penalizes areatoric uncertainty, which is unnecessary and detrimental to the expected value of the performance.\n\n**Further remark**\n\nA +/- should always be followed by the uncertainty of the measured value. This is usually the standard error rather than the standard deviation in the case where the measured value is a mean. Ultimately, however, it is up to the experimenter (i.e., the authors in this case) to estimate the uncertainty appropriately. Thus, if the authors choose to report the standard deviation (which here is sqrt(3-1) times larger than the standard error) as the uncertainty, then the authors are choosing to make a conservative estimate of the uncertainty, which is perfectly fine, especially given the very small number of only three repititions.\n\n\n\n\n[1] Depeweg et al., Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning, 2018", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "T4wGZzR4ru3", "reviewer": "Reviewer_Y21Y", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work analyses model-based approaches to offline RL. Instead of considering model ensemble, it suggests using entropy regularization as a way to explore beyond the data support.  ", "summary_of_the_review": "The lack of technical novelty and my concerns regarding experiments lead me to a weak accept. \n\n\n", "strength_and_weaknesses": "The paper is clearly written and easy to follow. However, although I am not familiar with the offline RL literature, I found the proposed method quite standard, ie, regularizing the loss for better generalization. \nAlso, I am a bit skeptical regarding the experimental results. I explain why below. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8ZaPEkQ5hu", "reviewer": "Reviewer_bC5e", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents EPO, a novel model-based offline RL algorithm. The highlight of EPO is that it utilizes only a single model to estimate uncertainty, in contrast with previous works that require an ensemble of models. \n\nThe algorithm has multiple phases. In phase one, the algorithm fits a Gaussian model for the transition and reward. In phase two the algorithm maximizes the differential entropy of the fitted Gaussian model over an exploratory dataset, which is collected by some exploratory policy on the fitted environment from phase one. Finally, EPO optimizes the model over the fitted environment with an uncertainty penalization (based on the entropy of the fitted model) over the reward.\n\nThe authors conducted experiments on the MuJoCo environment and showed that EPO is competitive with some offline RL SOTAs that utilize an ensemble of models for uncertainty quantification.", "summary_of_the_review": "The authors present an interesting attempt at uncertainty quantification with a single model. The approach appears to be promising at least in the test environments. Nevertheless, the underlying model assumptions with the Gaussian distribution could be strong, which may limit the generality of EPO as a general workflow.", "strength_and_weaknesses": "$\\textbf{Strength:}$\n\nThis work attempt to estimate uncertainty in RL with a single model, which frees practitioners from deploying ensembles in uncertainty quantification. Estimating uncertainty with a single model is an important research direction in both offline RL and online exploration. The paper is well-written and easy to follow. The proposed EPO algorithm has better modularity than previous works, in the sense that it can incorporate various existing model-free RL algorithms (on top of its pessimistic environment estimation).\n\n\n$\\textbf{Weaknesses:}$\n\nI have a few questions after reading the paper. Having some of those questions unresolved could be the potential weakness of the paper. Meanwhile, having some of those questions resolved could increase the potential impact of this paper.\n\n$\\textbf{Questions:}$\n\nThe claimed general approach seems to be limited by various model assumptions. In particular,\n\n$\\textbf{Q1:}$\nHow does EMO differentiate between aleatoric uncertainty and epistemic uncertainty? Gaussian modeling of next-step transition should also incorporate aleatoric uncertainty of $s'$ and $r$ unless the transition and reward are assumed to be deterministic (which is indeed the case for the MuJoCo environment in experiments though). Will such an assumption hinder the generality of the proposed EMO framework?\n\n\n$\\textbf{Q2:}$\nThe adopted (diagonal) Gaussian model requires continuous states, which may further hinder the generality of EMO. \n\n$\\textbf{Q3:}$\n(Minor) In addition, the Gaussian model has only a single peak, which may limit the modeling of uncertainty in the next-step transition (there could only be one high-probability outcome).\n\n\nIn addition, I have a few question regarding uncertainty modeling in EPO. \n\n$\\textbf{Q4:}$\nHow does the explorative policy affect the performance of EPO? Is it trustworthy to consider data sampled from the explorative policy to be OOD data (the trajectory may overlap with the in-distribution dataset)? Why not using the modified EPO described in section 4.3, which seems to penalize more relevant OOD data? As a side remark, previous work (e.g., [1-4]) also sample OOD actions of training policy to enforce more relevant OOD penalization.\n\n$\\textbf{Q5:}$\nThe uncertainty estimate is maximized over OOD data in the training stage (the regularization phase). How does such maximization of OOD uncertainty affect the uncertainty estimate, in particular, the relative scale of OOD uncertainty against the in-distribution uncertainty? Is the estimated uncertainty (after maximization on OOD data) trustworthy?\n\nMinor clarification question:\n\n\n$\\textbf{Q6:}$\nHow does EPO compare against previous ensemble-based methods in terms of the computation resource? I imagine that this could be one of the major advantages of utilizing a single model for uncertainty estimation.\n\n\n[1] A. Kumar et al., Conservative Q-Learning for Offline Reinforcement Learning. (2020)\n\n[2] I. Kostrikov et al., Offline Reinforcement Learning with Fisher Divergence Critic Regularization. (2021)\n\n[3] C. Bai et al., Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning. (2022)\n\n[4] Lyu et al., Mildly Conservative Q-Learning for Offline Reinforcement Learning. (2022)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6OUt41egjU", "reviewer": "Reviewer_EzoK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies pessimistic model-based offline reinforcement learning. Motivated by the fact that ensemble methods do not provide accurate uncertainty quantifiers used for constructing penalties subtracted from rewards, the authors present an entropy-regularized algorithm that learns a pessimistic model and provides uncertainty estimation outside the support of offline data. Empirical evaluations show comparable performance to SoTA offline RL algorithms on the D4RL benchmark.", "summary_of_the_review": "It is unclear whether the approach presented in this paper is sound and useful. Theoretical and empirical justification is not presented for this choice of uncertainty quantifier. The paper misses empirical comparison with important related work. The approach has limited novelty and empirical performance is not very strong. ", "strength_and_weaknesses": "**Strengths**\n- The paper studies an important topic which is uncertainty quantification for pessimistic offline RL. The paper is well-motivated as the ensemble methods do not offer reliable uncertainty estimation. \n- The proposed approach has the benefit of decoupling pessimistic model-learning and policy-learning steps, similar to model-based methods MOReL and MOPO and unlike SoTA model-based offline RL methods COMBO and RAMBO-RL.\n- The proposed algorithm is simple.\n- In addition to evaluations on the D4RL benchmark, the authors also conduct ablation studies investigating the effect of exploration policy as well as regularization and regularization strength on learned policies.\n\n**Weaknesses**\n- The algorithm and technical contributions have limited novelty. The proposed algorithm is an instantiation of pessimistic lower confidence bound methods such as MOPO that subtract a penalty from rewards to account for partial coverage in offline data. \n- It is unclear whether the algorithmic idea of building penalty terms based on determinants of covariance matrices is sound. It appears that the covariance is originally meant to capture the aleatoric uncertainty of the transitions (based on the definition on page 4) and later is used for the penalty terms, which instead require capturing epistemic uncertainty. No theoretical justification for this choice (even in a simple example) is presented.\n- I feel the focus of the paper should be to justify why this particular uncertainty quantifier is sound and useful. So it's important to conduct some theoretical and/or empirical investigations showing that the approach actually gives better uncertainty quantification than other methods such as ensembles or the method in [1] instead of solely relying on the end-to-end result on offline RL benchmarks.\n- The paper does not compare with model-based methods that go beyond using ensembles for uncertainty estimation such as the GELATO algorithm [1]. The paper also does not empirically compare with the ATAC algorithm [2]. Generally, the literature review is very incomplete.\n- The empirical performance compared to existing methods is not very strong.\n- The setting considered is limited. The algorithm focuses on the special case that transitions are Gaussian and uses learned covariance as an uncertainty quantification measure.\n- Discussion in paragraph 2 in the Introduction regarding model-based vs. model-free methods in offline RL is not entirely accurate and the claims appear to be strong and not fully justified. For example, it is unclear whether model-free methods are *inherently* overly conservative or model-based methods are better at generalization. This claim is not supported in theoretical studies of offline RL and is also not entirely supported by empirical observations (for example, CQL (model-free) outperforms MOPO (model-based)).\n\n**References**\n\n[1] Guy Tennenholtz, Nir Baram, and Shie Mannor. Latent geodesics of model dynamics for offline reinforcement learning. In Deep RL Workshop NeurIPS 2021, 2021.\n\n[2] Ching-An Cheng, Tengyang Xie, Nan Jiang, Alekh Agarwal Proceedings of the 39th International Conference on Machine Learning, PMLR 162:3852-3878, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject