Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
Authors: Kaiyang Guo
yunfeng shao
Yanhui Geng
Venue: NeurIPS 2022 Accept (2022)
Keywords: Offline reinforcement learning
model-based reinforcement learning
Bayesian learning

Abstract: Model-based offline reinforcement learning (RL) aims to find highly rewarding policy, by leveraging a previously collected static dataset and a dynamics model. While the dynamics model learned through reuse of the static dataset, its generalization ability hopefully promotes policy learning if properly utilized. To that end, several works propose to quantify the uncertainty of predicted dynamics, and explicitly apply it to penalize reward. However, as the dynamics and the reward are  intrinsically different factors in context of MDP, characterizing the impact of dynamics uncertainty through reward penalty may incur unexpected tradeoff between model utilization and risk avoidance. In this work, we instead maintain a belief distribution over dynamics, and evaluate/optimize policy through biased sampling from the belief. The sampling procedure, biased towards pessimism, is derived based on an alternating Markov game formulation of offline RL. We formally show that the biased sampling naturally induces an updated dynamics belief with policy-dependent reweighting factor, termed Pessimism-Modulated Dynamics Belief. To improve policy, we devise an iterative regularized policy optimization algorithm for the game, with guarantee of monotonous improvement under certain condition. To make practical, we further devise an offline RL algorithm to approximately find the solution. Empirical results show that the proposed approach achieves state-of-the-art performance on a wide range of benchmark tasks.

Model-Based Offline Reinforcement Learning with Pessimism-Mddulated Dynamics Belief
Kaiyang Guo"
Yunfeng Shao Huawei Noah's ' Ark Lab
Yanhui Geng
Abstract
Model-based offline reinforcement learning (RL) aims to find highly rewarding policy, by leveraging a previously collected static dataset and dynamics model. While the dynamics model learned through reuse of the static dataset, its gener- alization ability hopefully promotes policy learning if properly utilized. To that end, several works propose to quantify the uncertainty predicted dynamics, and explicitly apply it penalize reward. However, the dynamies and the reward are intrinsically different factors context of MDP, characterizing the impact of dynamics uncertainty through reward penalty may incur unexpected tradeoff between model utilization and risk avoidance. this work, we instead maintain belief distribution over dynamics, and evaluate/optimize policy through biased sampling from the belief. The sampling procedure, biased towards pessimism, is derived based mally show that the biased sampling naturally induces an updated dynamics belief with poliey-dependent reweighting factor, termed PessimisnnMModulatee Dynamics Belief. To improve policy, we devise an iterative regularized policy optimization algorithm for the game, with guarantee of monotonous improvement under cer- tain condition. To make practical, we further devise an offline RL algorithm to approximately find the solution. Empirical results show that the proposed approach achieves state-of-the performance on wide range of benchmark tasks.
alternating Markov game formulation of offline RL. We for-
Introduction
In typical paradigm of RL, the agent actively interacts with environment and receives feedback to promote policy improvement. The essential trial-and-error procedure can be costly, unsafe or even prohibitory in practice (e.g. robotics [1], autonomous driving (2), and healthcare (3), thus constituting a major impediment to actual deployment RL. Meanwhile, for a number of applications, historical data records are available to refleet the system feedback under predefined policy. This raises the opportunity to learn policy in purely offline setting.
In offline setting, no further interaction with environment is permitted, the dataset provides a limited coverage in state-action space. Then, the policy that induces distiiitibittionn (OOD) state-action pairs can not be well evaluated in offline learning phase, and deploying online potentially attains terrible performance. Recent studies have reported that applying vanilla RL algorithms to offline dataset exacerbates such distributional shift [4-6), making them unsuitable for offline setting.
To tackle the distributional shift issue, number of offline RL approaches have been developed. Specifically, one category of them propose to directly constrain the policy close to the one collecting data g or penalize Q-value towards conservatism
OOD state-action pairs II While they achieve remarkable performance gains, the policy regularizer and the Q-value penalty tightly restricts the produced policy within the data manifold. Instead, more recent works consider to
"Corresponding to: guokaaiyangdhuavei.ccom com
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

quantify the uncertainty of Q-value with neural network ensembles [12. where the consistent Q-value estimates indicate high confidence and can be plausibly used during learning process, even for OOD However, the uncertainty quantification over OOD region highly relies on how neural network generalizes [15]. As the prior knowledge of Q-function hard to acquire and insert into the neural network, the generalization is unlikely reliable to facilitate meaningful uncertainty quantification [16]. Notably, all these works are model-free.
state-action pairs 13 14
Model-based offline RL optimizes policy based on a constructed dynamics model. Compared to the model-free approaches. one prominent advantage that the prior knowledge of dynamics is easier to access. First, the generic prior like smoothness widely exists in various domains [17]. Second, the sufficiently learned dynamics models for relevant tasks can act as a data-driven prior for the concerned task [18/20), With richer prior knowledge, the uncertainty quantification for dynamics is more trustworthy. Similar to the model-free approach, the dynamics uncertainty can be incorporated to find reliable policy beyond data coverage. However, an additional challenge is how to characterize the accumulative impact of dynamics uncertainty on the long-term reward, as the system dynamics is with entirely different meaning compared to the reward or Q-value.
Although existing model-based offline RL literature theoretically bounds the impact of dynamics uncertainty on final performance, their practical variants characterize the impact through reward penalty 6 21 22 Concretely, the reward function is penalized by the dynamies uncertainty for each state-action pair [21]. or the agent is enforced to a low-reward absorbing state when the dynamics uncertainty exceeds a certain level (6). While optimizing policy in these constructed MDPs stimulates anti-uncertainty behavior, the final policy tends to be wver-conservative. For example, even the transition dynamics for a state-action pair is ambiguous among several possible candidates, these candidates may generate the states from which the system evolves similarly Then, such state-action pair should not be treated specially.
Motivated by the above intuition, we propose pessinism--modulated dynamics belief for model-based offline RL. In contrast with the previous approaches, the dynamies uncertainty is not explicitly quantified. To characterize its impact, we maintain a belief distribution over system dynamics, and the policy is evaluated/optimizze through biased sampling from it. The sampling procedure, biased towards pessimism, is derived based on an alternating Markov game (AMG) formulation of offline RI.. We formally show that the biased sampling naturally induces an updated dynamics belief with policy-depent deiiy-deeendent reweighting factor, termed degree of pessimism is monotonously determined by the hyperparameters in sampling procedure.
Dynamics Belief. Besides, the
The considered AMG formulation can be regarded as a generalization of robust MDP, which is proposed as a surrogate optimize the percentile performance in face of dynamics uncertainty However, robust MDP suffers from two significant shortcomings: 1) The percentile criterion
23 24
over-conservative since it fixates on single pessimistic dynamics instance 25
26 : 2) Robust
MDP is constructed based on an uncertainty set, and the improper choice of uncertainty set would further aggravate the degree of conservatism [27 28). The AMG formulation kept from these shortcomings. To solve the AMG, we devise an iterative regularized policy optimization algorithm, with guarantee of monotonous improvement under certain condition. To make practical, we further derive an offline RL algorithm to approximately find the solution, and empirically evaluate it on the offline RL benchmark D4RL. The results show that the proposed approach obviously outperforms previous -fthe-of-the-art (SoTA) in 9 out of 18 environment-datass configurations and performs competitively in the rest, without tuning hyperparameters for each task. The proof of theorems in this paper are presented in Appendix B
2 Preliminaries
Markov Decision Process (MDP) A MDP is depicted by the tuple (S,A, T,r, Po. 7o)7), where S, A are state and action spaces, T(s'ls. a) is the transition probability, r(s. a) is the reward function, Po(s) is the initial state distribution, and y is the discount factor. The goal of RL is to find the policy s + ((A) that maximizes the cumulative discounted reward:
J(T, T) = Ep..T..
Σ (rr((s, crrlssnaa) t=
(1)
"rr from these states, the system evolves differently but generates similar rewards.

---

where A() denotes the probability simplex. In typical RL paradigm, this is done via actively interacting with environment.
Offline RL In offline setting, the environment is unaccessible, and only static dataset D {(s, a, r, r)) is provided, containing the previously logged data samples under an unknown behavior policy. Offline RL aims to optimize the policy by solely leveraging the offline dataset.
To simplify the presentation, we assume the reward function r and initial state distribution Po are known. Then, the system dynamics is unknown only in terms of the transition probability T. Note that the considered formulation and the proposed approach can be casily extend to the general case without additional technical modification.
Robust MDP With the offline dataset, a straightforward strategy is first learning a dynamics model T(s'|s, a) and then optimizing policy via simulation. However, due to the limitedness of available data, the learned model is inevitably imprecise. Robust MDP (23 is surrogate to optimize policy with consideration of the ambiguity of dynamics. Concretely, robust MDP is constructed by introducing an uncertainty set T = {r} to contain plausible transition probabilities. If the uncertainty set includes the true transition with probability of (1 5), the performance of any policy x true MDP can be lower bounded by miureT (TI, with probability of at least (1 6). Thus, the percentile performance for the true MDP can be optimized by finding a solution to
max min ( T). π
TET
Despite its popularity, Robust MDP suffers from two major shortcomings: First, the percentile criterion overly fixates on a single pessimistic transition instance, especially when there are multiple optimal policies for this transition but they lead to dramatically different performance for other transitions 25 26 This behavior results in unnecessarily conservative policy.
Second, the level of conservatism can be further aggravated when the uncertainty set is inappropriately
constructed [27]. For a given policy и, the ideal situation that T contains the (1 - 5) proportion of transitions with which the policy achieves higher performance than with the other 8 proportion. Then, min-e7 [TT, 7) is exactly the d-quantile performance. This requires the uncertainty set to be policy- dependent, and during policy optimization the uncertainty set should change accordingly. Otherwise, i is predetermined and fixed, it is possible to have 7' & T with non conzzero probability and satisfying ((T,.T') min,eT J(x", T). where x² is the optimal policy for (2). Then, adding r' into T does not affect the optimal solution of the problem 2). This indicates that we are essentially optimizing a 8' -uqanttile performance, where 8' can be much smaller than 5. In literature, the uncertainty sets mostly predetermined hefore policy optimization 23 31
3 Pessimism-Modulate Dynamics Belief
In short, robust MDP is over-conservative due to the fixation on a single pessimistic transition instance and the predetermination of uncertainty set. In this work, we strive to take the entire spectrum of plausible transitions into account, and let the algorithm by itself determine which part deserves more attention. To this end, we consider an alternating Markov game formulation of offline RL, based on which the proposed offline RL approach is derived.
3.1 Formulation
Alternating Markov game (AMG) The AMG a specialization of two-player zero-sum game, depicted by (S, A. G, r, Y). The game starts from state sampled from pu, then two players alternatively choose actions a E A and a E under states s E and 3 E S, along with the game transition defined by G($|s, a) and G(s)s, a). At each round, the primary player receives reward r(s, a) and the secondary player receives its negative counterpart -r(s,a).
Offline RL as AMG We formulate the offline RL problem as an AMG, where the primary player optimizes a reliable policy for our concerned MDP in face of stochastic disturbance from the secondary player. The AMG constructed by augmenting the original MDP. both have the transition probability, we game transition and system transition to differenniate them.
3

---

For the primary player, its state space S, action space A and reward function r(s, a) are same with those in the original MDP. After the primary player acts, the game emits a NNssize set of system transition candidates T which later acts as the state of secondary player. Formally, T is generated according to
G - TT5=
II To0
Pa PT (T*0),
where (-)(.) re-denotes the plausible system transition 7(-|s, a) for short, and P7 is given belief distribution over 730, According to (3). the elements in Tsa independent and identically dis- tributed samples following P7. The major difference to uncertainty set in robust MDP is that the set introduced here unfixed and stochastic for each step. To distinguish with uncertainty set, we call it candidate set. The belief distribution P7 can be chosen arbitrarily to incorporate knowledge of system transition. Particularly, when the prior distribution of system transition is accessible, Pfe can be obtained as the posterior by integrating the prior and the evidence 'D through Bayes' rule.
The secondary player receives the candidate set T state. Thus, its state space be denoted by S = A~ (S), i.e. the n-fold Cartesian product of probability simplex over S. Note that the state T also takes the role of action space, i.e., A T meaning that the action of secondary player is to choose a system transition from the candidate set. Given the chosen 7 E Tea, the game evolves by sampling 750 i.e.
G ( T ā " (s'),
and the primary player receives s' to continue the game. In the following, we use P. (Te0) to compactly denote the game transition G (5 = se [s, a) in (3) and omit the superscript sa in T and ! when it is clear from the context.
For the above AMG, we consider a specific policy (explained below) for the secondary player, such that the cumulative discounted reward of the primary player with policy π can be written as:
I1
J(T) = E [min mo
k To
E To.n,.y
|min
k EET
IE
Σ
x an)
(5)
where the subscripts of T and T denote time step, the expectation is over 8q ~ Pa- 8:20 ~ Te- 12-1(-18--1,44 ag (T(ss)) and T P.N, and the operator min] k xex denotes finding kth minimum of f(x) over E X. The policy of secondary player is implicitly defined by the operator / (x). When changing & E {1.2.... ... NN) the secondary player exhibits various degree of adversarial or aggressive disturbance to the future reward. From the view of original MDP, this behavior raises flexible tendency ranging from pessimism to optimism when evaluating policy m.
The distinctions between the introduced AMG and the robust MDP are twofold: 1) With belief distribution over transitions, robust MDP will select only part of its supports into uncertainty set, and the set elements are treated indiscriminatingl, It indicates that both the possibility of transitions out of the uncertainty set and the relative likelihood of transitions within the uncertainty set are discarded. However, in the AMG, the candidate set simply contains samples drawn from the belief distribution, implying no information drop in an average sense. Intuitively, by keeping richer knowledge of the system, the performance evaluation is more exact and away from excessive conservatism; 2) In robust MDP, the level of conservatism is expected to be controlled by its hyperparameter However, as illustrated in Section 2 smaller 5 does not necessarily corresponds to more conservative performance evaluation, due to the extra impact from uncertainty set construction. In contrast, for the AMG, the degree of conservatism is adjusted by the size of candidate size N and the order of minimum k. When changing values of k or N, the impact to performance evaluation is ascertained, as formalized in Theorem 3
To evaluate (TT), we define the following Bellman backup operator:
8% ( a) = r(s, a) + +Bpgy miinJ [Q(s', dd']]
(6)
As the operator depends N, k and we emphasize pessimism in offline RL, we call it (N, k)- pessimistic Bellman backup operator. Compared to the standard Bellman backup operator in Q- learning, T additionally includes the expectation over T ~ IP, and the k-minimum operator over is still contraction mapping, based which ((x) T. Despite these differences, we prove that & k can be easily evaluated.

---

Theorem 1 (Policy Evaluation). The (N, k) pesssimiisic Bellman backup operator By k is contrac- tion mapping. By starting from any function Q : S x A + R and repeatedly applying B the sequence converges to IN with which we have ( Ex [QN,x(00 ao)].
3.2 Pessimism-Modullteed Dynamies Belief
With the converged Q-value, we are ready to establish a more direct connection between the AMG and the original MDP. The connection appears as the answer natural question: the calculation of (6) encompasses biased samples from the dynamics belief distribution, can we treat these samples as the unbiased ones sampling from another belief distribution? We give positive answer in the following theorem.
Theorem 2 (Equivalent MDP with Moodulasm-Mdullteed Dynamics Belief). The alternating Markov game (5) is equivalent to the MDP with tuple (S,A, T,r, Po; 7), where the transition probability ('8'18, a) Baps [T (s") is defined with the reweighted belief distribution Pap:
Pa (T")) w R [ (5', a'); :k, N (T"))
w(2;k, [F(=)] [[x= N-K
(8)
and F() is cumulative density function. Furthermore, the value of w(w;; k. N) first increases and then decreases with x, and its maximum is obtained at the k-1 N-1 quantile, i..,, x² = F-1
In right-hand side of 7. Toa itself is random following the belief distribution, thus EyT", x [W,, (s', a'), as a functional of 7* is also a random variable, whose cumulative density function is determined by the belief distribution Pgp. Intuitively, as pessimism indicator for transition 7*0, with larger value indicating less pessimism.
treat [,.",=((Q% [QN (6', a')]
From Theorem the maximum of w is obtained at 7": F(rr. [QTr (o'a(d',))l) t i.e. the
transition with quantile pessimism indicator. Besides, when E,-., [Q7 (s', ad)] departs the N=1 quantile, the reweighting coefficient for its 7a decreases. Considering the effect of to Psa and the equivalence between the AMG and the refined MDP, percentile performance. Compared to the standard percentile criteria, J(x) is derived by reshaping belief distribution towards concentrating around certain percentile, rather than fixating on single percentile point. Due to this feature, we term Prr Bessinism-Modullee Dynamics (PMDB).
say that J(rr) is soft
Lastly, recall that all the above derivations are with hyperparameters l and N, we present the monotonicity of Q%, over them in Theorem Furthermore, by combining Theorem with Theorem 3 we conclude that J (T) decreases with N and increases with k.
Theorem (Monotonicity). The converged Q-function Qw,x are with the following properties:
Given any k. the Q-function QT,, element-wisely decreases with N E [K, k + 1, 1,...
Given any N, the Q-function QT. element-wised increases with & E {1, ... , .N..
The Q-fanction NN element-wisely increases with N.
Remark 1 (Special Cases). For N = == we have
Pop Pap. Then, the performance is evaluated through sam- pling the initial belief distribution. This resembles the com- mon methodology in model-based RL (MBRL), with dynam- ics belief defined by the uniform distribution over dynamics model ensembles. For l = 8(N - -1)+1 +1 and N - Do, Tpsp asymptotically collapses be a delta function. Then, J(T) degrades to fixate on a single transition instance. It is equiv- alent to the robust MDP with the uncertainty set constructed as {750 : (7 > 0, O, [Q% aa)] 2 F-) In this sense, the AMG is successive interpolation between MBRL and robust MDP.
Robut
MBRL
5
Figure 1: Monotonicity of Q-values. The arrows indicate the directions along which Q-values increase.
5

---

4 Policy Optimization with Pesimisn-Mottlateed Dynamics Belief
In this section, we optimize policy by maximizing (TT). The major consideration is that the methodology should adapt well for both discrete and continuous action spaces. In continuous setting of MDP, the policy can be updated by following stochastic/determinnstie policy gradient [32 33 However, for the AMG, evaluating J(T) itself involves an inner dynamic programming procedure as in Theorem to maximize J (T) via gradient-basse method. In this section, we consider series of sub-problems with Kullback-Leibler (KL) regularization. Solving each sub-problem makes prominent update to the policy, and the sequence of solutions for sub-problems monotonously improve regarding J(T). Based on this idea, we further derive offline RL algorithm to approximately find the solution.
1 As each evaluation of ((T) only produce one exact gradient, it is inefficient
4.1 Iterative Regularized Policy Optimization
Define the KL-regularized return for the AMG by
J(T; = E
[inn
ToETo
E To
[mnn]
MET
E Tous
a(()))
1=0
-a DDLL( || pl u(-lss)))
(9)
where a 2 is the strength of regularization, and μ is a reference policy to keep close with.
KL-regularized MDP is considered in previous works to enhance exploration, improve robustness to noise or insert expert knowledge [34 38.. Here, the idea is to constrain the optimized policy in neighbour of a reference policy so that the inner problem is adequately evaluated for such small policy region.
To optimize JT,; we introduce the soft (N, pesssiiiisi Bellman backup operator:
a log Ey exp (L )
B (s.. a) = r(s.a) a) + E
min |min| k E
E,
(10)
Theorem 4 (Regularized Policy Optimization). The soft (N, -e-pssiimistt Bellman backup operator Bix is a contraction mapping. By starting from any function Q x A + and repeatedly applying By the sequence converges to Qiv with which the optimal policy for [(x; p1) is obtained as #* (als) x p(als) exp (8.).
Apparently, the solved policy #* depends on the reference policy μ. and setting μ arbitrarily aforehand can result in suboptimal policy for (mr. In fact, we can construct sequence of sub- sub-problems, with I chosen as an improved policy from last sub-problem. By successively solving them, the impact from the initial reference policy is gradually eliminated.
Theorem 5 (Iterative Regularized Policy Optimization). By starting from any stochastic policy To + ((A) and repeatedly finding Ti41 J(T;-1;;))) >> -[[T n.). the sequence of { monotonically improves regarding [x),, ie., J( 2 ((T)). Especially, when To(als) > 0, Vs,a
and { are obtained via regularized policy optimization in Theorem we have mi(als) - 30 for
mi(al|s)
any s, a, a' such that lim; ( a) lim; (s,
Ideally, by combining Theorems and 5 the policy for J(x) can be continuously improved by infinitely applying soft pessimistic Bellman backup operator for each of the sequential sub-problems.
with Structured Variational Posterior). According to Theorem 2 PMDB can be recovered with the converged Q- function QK From an end-lo-end view, we have an initial dynumics belief Peg, then via the calculation based on the belief samples and the reward function, we obtain the updated dynamics belief Pap. It is likely that we are doing some form of posterior inference, where the evidence comes from the reward function. In fact, the iterative regularized policy optimization can be formally recast as Cotiuutioon-aiizzzatiinn algorithm for offline policy optimization, where the Expectation step correponds to structured variational inference procedure for dynamics. We elaborate Appendix
Remark 2 (Iterative Regularized Policy Optimization as

---

4.2 Offline Reinforcement Learning with Pessimism-Modulace Dynamics Belief
While solving each sub-problen makes prominent update to policy compared with the policy gradient method, we may need to construct several sub-problems before convergence, then exactly solving each of them incurs unnecessary computation. For practical consideration, next introduce smooth-evolving reference policy, with which the explicit boundary between sub-problems is blurred. Based on this reference policy, and by further adopting function approximator, devise offline RL algorithm to approximately maximize J(T).
The idea of smooth- reference policy is inspired by the soft-updated target network in deep RL literature [39 40]. That is setting the reference policy as slowly tracked copy of the policy being optimized. Formally, consider parameterized policy Th with the parameter Φb The reference policy is set as f = TØr, where o' the moving average of & : o' +- w + (1 - wi )o¹ With small enough w1, the Q-value of the state-action pairs induced by T (or its slight variant) can be sufficiently evaluated, before being used to update the policy. Next, we detail the loss functions to learn Q-value and policy with neural network approximators.
Denote the parameterized Q-function by Qo with the parameter O. is trained by minimizing the Bellman residual of both the AMG and the empirical MDP:
Lo(0) = -WD (Do(s..a) -ano(s..)) 21
+ + ( ( (1))
with
Qaaals a) = r(s, (1))+7(mmne >LminJ#eeT B, a log Bng exp (Hen((s
(12)
QUDr(+, a) = rss a) + . - aloo Bug exp (2
(13)
where Qa represent the target Q-value softly updated for stability 44), i.c., O' + w2 + (1 (1- (1-wz) )0' and D' is the on-policy data buffer for the AMG. Since the game transition is known, the game can be executed with multiple counterparts in parallel, and the buffer only collects the latest sample for each of them. To promote direct learning from D, we also include the Bellman residual of the empirical MDP in
As with policy update, Theorem states that the optimal policy for [T;; 14) is propotional to u(als) exp (1Qiva(s,)) Then, we update T& by supervisedly learning this policy, with μ and Qive replaced by the smooth-evolving reference policy and the learned Q-value:
Tor (-1s) exp (2Qo(s,))
Ta((1s))
Lp(Ø)= = ESNDUD
DKL
Eng [exp (s)
= AA B ama
exp
(o(s,a) log (als)) + B,
(14)
where A and B constant terms. In general, (14) can be replaced by any tractable function that measures the similarity of distributions. For example, when no is Gaussian, we can apply the recent proposed 3-NLL [4] in which each data point's contribution to the negative log- lig-liielihood loss is weighted by the 3-exponentiated variance to improve learning heteroscedastie behavior.
To summarize, the algorithm alternates between collecting on-policy data samples in AMG and updating the function approximators. In detail, the latter procedure includes updating the Q-value with (I1), updating the optimized policy with (14), and updating the target Q-vulue as well as the reference policy with the moving-average rule. The complete algorithm is listed in Appendix D
5 Experiments
Through the experiments, we aim to answer the following questions: 1) How does the proposed approach compared to the previous SoTA offline RL algorithms on standard benchmark? 2) How does the learning process in the AMG connect with the performance change in the original MDP? 3) 3 the monotonicity of J(z) over N and for any specified म, and it easy to verify that this statement also holds when considering optimal policy for each setting of (N, k). However,
7

---

with neural network approximator, our proposed offline RL algorithm approximately solves the AMG.
Then, how is the monotonicity satisfied in this case?
to answer these questions. As PMDB relies on the initial dynamics belief, inserting additional knowledge into the initial dynamics belief will result in unfair comparison. To avoid that, we consider an uniform distribution over dynamics model ensembles as the initial belief. The dynamics model ensembles are trained in supervised manner with the offline dataset. This is similar to previous model-based works [ 21.. where the dynamics model ensembles are considered for dynamies uncertainty quantification. Since hyperparameter tuning for offline RL algorithms requires extra online test for each task, we purposely keep the same hyperparameters for all tasks, except when answering the last question. Especially, the hyperparameters in sampling procedure are N = 10 and k = 2. The more detailed setup for experiments and hyperparameters can be found in Appendix G The code available onling
We consider the Gym domains in the D4RL. benchmark
5.1 Performance Comparison
We compare the proposed offline RL algorithm with the baselines including: BEAR S and BRAC 77. the model-free approaches based on policy constraint, CQL (9). the model-free approach by penalizing Q-value, EDAC [13] the previous SoTA on the D4RL benchmark, MOReL [6], the model-based approach which terminates the trajectory if the dynamies uncertainty exceeds a certain degree, and BC, the behavior cloning method. These approaches are evaluated on a total of eighteen domains involving three environments (hopper, walker2d, halfcheetah) and six dataset types (random, medium, expert, medium-expert medium-replay, full-replay) per environment. We use the v2 version of each dataset.
The results are summarized in Table [ Our approach PMDB obviously improves over the previous SoTA on 9 tasks and performs competitively in the rest. Although EDAČ achieves better performance in walker2d with several dataset types, its hyperparameters are tuned individually for each task. The later experiments
the impact of hyperparameters indicate that larger k or smaller N could generate better results for walker2d and halfcheetah. We also find that PMDB significantly outperforms MOReL, another model-based approach. It is encouraging that our model-based approach achieves competitive or better performance compared with the SoTA model-free approach, as model based approach naturally has better support for multi-task learning and transfer learning, where the offline data from relevant tasks can be further leveraged.
Task Name
BC
BEAR
BRAC
CQL.
MORel.
EDAC
PMDB
hopper-random
3.7400.6
3,6
8.1
5.3=0.6
38.1=10.1
25.3++:0.44
32.7=0.11
hopper-medium
54.1+388
55.34:3.2
77.816.1
61.916.4
84.0=17.0
101.6:0666
106.8:10.2
hopper expert
107.719.7
39.4:120.5
78.11:526
106.519.1
80.411.4.9
110.1:0.11
111.7:10.3
hogper-medium-erye
53.914.77
66.218.55
81.318.0
96.9115.1
105.618.2
110.7.10.1
111.8:10.6
hopper- medi-me-eeeeeee replay
16.614488
$7.7:1.6.5
62.71:30.4
86.317.33
$1.81170
101 0.5.0...
106.2:10.6
hopper full replay
19.9 12.9
54.0.1:2..0
107.4.0.5
101.9.0.66
94.4
105.4....7
109.1
walker2d-ramden
1.3:0.1
4.3::1122
1.31:14
1.11177
16.0:7..7
16.61:7000
21.8:10.1
walker2d-msdium
70.9=11.0
59.8:2-4.00
59.7
79.5=3322
12.8++1.9
92.5++0.88
94.2:11.1 1.1
vulker2d-experr
108.710.22
110.1::0.6
55,2+162.2
109.3-0.1
62.6:1:29.9
115.1:::.9
115.9
walker2d medium expert
90.1
107.0.1..9
9.3118899
109.1.0.22
107.5....6
1.4.7.1..9
111.9.10.2
walker2d wallkerdd-medium-eee replay
20.34.98
12.244.77
40.11447.9
16.8
40.8+200.4
87.1-22.3
79.9-10.2
walker2d-full-rppay
68.8=17.7
79.6=
969+222
94.2=11.9
84,8+13.1
99.8=0077
95,4=0077
batfeheetah-randeee randdom
2:2::0.0
12.6:110
24.3+0077
31.3:335
38.911888
28.41:10
+: 0.2
hulfcheetah medium
43.2:1:0.6
42.810.11
51.10.3
46.910.4
60.714.44
65.9
75.6 1.3
halfcheetah-eee expert
91.8:1..5
92.6-0066
39.0-11..8
97.3-1111
8.4-11.8
106.8
105.7:: 1.0
halfeheetah-mediin dedium---tt
14.0+=1.6
45.74-4.2
$2.3+0.11
95.0==14
80.4+11.7
106.3+1.9
108.5-10.5
hallchhetta-nnedun--ooo medium-roplay
37.6+2.11
39.4:0.8
48.620.4
45.3-003
44.5+5.66
61.3+199
71.7=11.11
ballcheetah replay
62.910888
60.1
78.010077
7..9:00.9
70.11:5.1
8,
90.0:10.8
Average
52.4
54.0
65.1
85.2
Table 1: Results for D4RL datasets. Each result is the normalized score computed as (score random policy score) (expert policy score random policy score), + standard deviation. The score of our proposed approach is averaged over 4 random seeds, and the results of the baselines are taken from 13
5.2 Learning in Alternating Markov Game
Figure presents the learning curves in the AMG, as well as the received return when deploying the policy being learned in true MDP. The performance in the AMG closely tracks the true perfor-
http:////000rrree//mmst/////// and https:// htteeo//w/.iiiipeaoooa////eeeeer
Coode is released at

---

mance from the lower side, implying that it can act reasonable surrogate to evaluate/optimize performance for the true MDP. Besides, the performance in the AMG improves nearly monotonously, verifying the effectiveness of the proposed algorithm to approximately solve the game.
hopper-medium
walker2d-medium
tel
hallshaetah-meddum
1.0
3
0.8
Rotum
0.4
0.2
MDP
AMG
0.50
0.75
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Training Steps
Training Steps
1es
Training Sleps
Figure 2: Learning and test curves for medium datasets.
Recall that PMDB does not explicitly quantify dynamics uncertainty to penalize return, Figure checks how the dynamics uncertainty and the Q-value of visited state-action pairs change during the learning process. The uncertainty is measured by the logarithm of standard deviation of the predicted means from the N dynamics samples, i.e., log (std [="] T)). The policy being learned is periodically tested in the AMG for ten trials, and we collect the whole ten trajectories of state-action pairs. The solid curves in Figure denote the mean uncertainty and Q-value over the collected pairs, and shaded regions denote the standard deviation. From the results, the dynamics uncertainty first sharply decreases and then keeps slowly increasing trend. Besides, in the long-term view, the Q-value is correlated with the degree of uncertainty negatively in the first phase and positively in the second phase. This indicates that the policy first moves to the in-distribution region and then tries to get away by resorting to the generalization of dynamics model.
1.00
hopper-medium
2.0
walker26-medium
68
1.20
halfcheetah-medium
108
Logatt pppeeceedd
-0.56
45
0.6-
51
0.46
81
Auoroge Ovaku
Uncertainty Q-vollas
2.10
-0.8
36
-0.30-
04
3.65
-22
-5.20
0.00
0.50
-1.80
2.0
Trening Steps
TaB
Trainng Steps
1wB
Training
Figure 3: Change on the dynamics uncertainty and Q-value of the encountered state-action pairs during learning process. The dynamics uncertainty of state-action pair (s, a) is measured by log (std (EE(()s, [s"]: ]i E T)).
In Figure we also notice that the large dip of Q-value is accompanied with the sudden raise of dynamics uncertainty. We suspect this is due to the optimized policy being far from the dataset. We try verify by checking the maximal return covered by the offline dataset. shows that the maximal normalized returns provided by offline datasets are 99.6, 92.0 and 45.0 respectively for hopper, walker2d and halfcheetah, while the proposed approach achieves 106.8, 94.2 and 75.6. The policy optimization is more significant for halfcheetah (where we observe the large dip), indicating the policy should move further from the dataset.
The above finding also explains why the AMG performance in Figure runs into large dip only for halfcheetah: with the larger dynamics uncertainty, the secondary player can choose the more pessimistic transition. However, we want to highlight that it is normal behavior of the proposed algo- rithm and does not mean instability, we handling the alternating Markov game, specialization of zero-sum game, Besides, we can see that even when the AMG performance goes down the MDP performance still stable.
5.3 Practical Impact of Hyperparameters in Sampling Procedure
Table |lists the impact of k. each setting, we evaluate the learned policy in both the true MDP and the AMG. The performance in the AMGs improve when increasing k. This is consistent with the theoretical result, even that we approximately solve the game. Regarding the performance in true MDPs, we notice that A corresponds to the best performance for hopper, but for the others k = 3 better. This indicates that tuning hyperparameter online can further improve the performance. The impact of N is presented in Appendix H suggesting the opposite monotonicity.

---

hopper-medium
walker2d-medium
halfcheetah-mmedimm
MDP
AMG
MDP
AMG
MDP
AMG
106.2.10.2
91.6412.2
82.6.10.5
33.3=2.66
70.7:10.8
63.1
2
106.8:10.2
106.2:1..
94.2411.1
77.2=3.7
75.6:11.3
67.3
3
10.8
106.6:2.1
105.1+00.
82.5=0.5
77.3+0.5
70.1
Table 2: Impact of k. with N = 10.
6 Related Works
Inadequate data coverage is the root of challenge in offline RL. Existing works differ in their methodology to reacting in face of limited system knowledge.
Model-free offline RL The prevalent idea is to find policy within the data manifold through model-free learning. Analogous to online RL, both policy-based and value-based approaches are devised to this end. Policy-based based approaches directly constrain the optimized policy close to the behavior policy that collects data, via various measurements such as KL divergence 77 MMD [S] and action deviation 4 ( Value-based approaches instead reflect the policy regularization through value function. For example, CQL enforces small Q-value for OOD state-action pairs (9). AICE penalizes return with the f-divergence between optimized and offline state-action distributions ( and Fisher-BRC proposes a novel parameterization of the Q-value to encourage the generated policy close the data [T]]. Our proposed approach is more relevant to the value-based scope, and the key difference to existing works is that our Q-value is penalized through an adversarial choice of transition from plausible candidates.
Learning within the data manifold limits the degree to which the policy improves, and recent works attempt relieve the restriction. Along the model-free line, EDAC [3] and PBRL [14] quantify uncertainty of Q-value via neural network ensemble, and assign penalty to Q-value depending on the uncertainty degree. In this way, the OOD state-action pairs touchable if they pose low uncertainty on Q-value. However, the uncertainty quantification over OOD region highly relies on how neural network generalizes [15]. As the prior knowledge of Q-function is hard to acquire and insert into the neural network, the generalization is unlikely reliable to facilitate meaningful uncertainty quantification [16].
Model-based offline RL Model-based approach is widely recognized due to its superior data efficiency. However, directly optimizing policy based on an offline learned model is vulnerable to model exploitation [22] 433. A line of works improve the dynamics learning for seek of robustmess 45 to distributional shift. In terms of policy learning, several works extend the
44 or adaptation
idea from model-free approaches, and constrain the optimized policy close to the behavior policy
when applying the dynamics model for planing [46] or policy optimization [47 48 There are also
recent works incorporating uncertainty quantification of dynamics model to learn policy beyond data coverage. Especially, MOPO 21 and MOReL [6] perform policy improvement in states that may not directly occur in the static offline dataset, but can be predicted by leveraging the power of generalization. Compared to them, our approach does not explicitly characterize the dynamics uncertainty reward penalty. There are also relevant works dealing with model ambiguity in light of Bayesian decision theory, which are discussed in Appendix
7 Discussion
We proposed model-based offline RL with Pessinism-Modultte Dynamies Belief (PMDB), framework to reliably learn policy from offline dataset, with the ability of leveraging dynamics prior knowledge. Empirically, the proposed approach outperforms the previous SoTA in a wide range of D4RL tasks. Compared to the previous model-based approaches, we characterize the impact of dynamics uncertainty through biased sampling from the dynamics belief, which implicitly induces PMDB. As PMDB is with the form of reweighting an initial dynamics belief, it provides a principled way to insert prior knowledge via the belief to boost policy learning. However, posing a valuable dynamics belief for arbitrary task is challenging, as the expert knowledge is not always available. Besides, an over-aggressive belief may still incur high-risk behavior in reality. Encouragingly, recent works have done active research to learn data-driven prior from relevant tasks. We believe that integrating them as well as developing safe criterion to design/learn dynamics belief would further promote practical deployment of offline RL.
10

---

References
[1] Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off- policy updates. In IEEE ICRA, 2017.
[2] Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A. Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE TITS, 2021.
[3] Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: survey, ACM Computing Surveys, 2021.
[4] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In ICML, 2019.
[5] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- off-policy Q-learming via bootstrapping error reduction. In NeurIPS, 2019.
[6] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model-based offline reinforcement learning. In NeurlPS, 2020.
[7] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning, arXiv preprint 1rXiv:1911.1 2019.
[8] Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. EMaQ: Expected-max Q-learning operator for simple yet effective offline and online RL. In ICML, 2021.
[9] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. In NeurlPS, 2020.
[10] llya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. ICML, 2021.
[11] Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Al- gacDICE: Policy gradient from arbitrary experience. arXiv preprint 1rX.v.111..02074, 2019.
[12] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurIPS, 2017.
[13] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- bbased based offline reinforcement learning with diversified Q-ensemble. In NeurIPS, 2021.
[14] Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. arXiv preprini arXiv:2202.1 1566, 2022.
[15] Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. In NeurIPS, 2020.
[16] Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. A simple baseline for Bayesian uncertainty in deep learning. In NewrIPS, 2019.
[17] Andy Zeng, Shuran Song, Johnny Lee, Alberto Rodriguez, and Thomas Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. IEEE T-RO, 2020.
[18] Amy Zhang, Clare Lyle, Shagun Sodhani, Angelos Filos, Marta Kwiatkowska, Joelle Pineau, Yarin Gal, and Doina Precup. Invariant causal prediction for block MDPs. In ICML, 2020.
[19] Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In ICLR, 2021.
[20] Philip Ball, Cong Lu, Jack Parker-Holder, and Stephen Roberts. Augmented world models facilitate zero-shot dynamics generalization from single offline environment. In ICML, 2021.

---

[21] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. In NeurIPS, 2020.
[22] Cong Lu, Philip J. Ball, Jack Parker-Holder, Michael A. Osborne, and Stephen J. Roberts. Revisiting design choices in model-based based offline reinforcement learning. In ICLR, 2022.
[23] Arnab Nilim and Laurent El Ghaoui. Robust control of Markov decision processes with uncertain transition matrices. Operations Research, 2005.
[24] Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust Markov decision processes. Mathematics of Operations Research, 2013.
[25] Elita A. Lobo, Mohammad Ghavamzadeh, and Marek Petrik. Soft-robust algorithms for batch reinforcement learning. arXiv preprint arXiv:2011.14495, 2020.
[26] Esther Derman, Daniel J. Mankowitz, Timothy A. Mann, and Shie Mannor. Soft-robust actor- critic policy-gradient. In UAI, 2018.
[27] Marek Petrik and Reazul Hasan Russel. Beyond confidence regions: Tight Bayesian ambiguity sets for robust MDPs. In NeurlPS, 2019.
[28] Bahram Behzadian, Reazul Hasan Russel, Marek Petrik, and Chin Pang Ho. Optimizing percentile criterion using robust MDPs. In AISTATS, 2021.
[29] Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. In ICML, 2019.
[30] Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High-confidence off-policy evaluation. In AAAI, 2015.
[31] Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Fast Bellman updates for robust MDPs. In ICML, 2018
[32] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NeurlPS, 1999.
[33] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In ICMI., 2014.
[34] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In ICML, 2017.
[35] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, 2018
[36] Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized Markov decision processes. In ICML, 2019.
[37] Alexandre Galashov, Siddhant M. Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojeiech M. Czarnecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess. Information asymmetry in KL-rregulariedd RI. In ICLR, 2019.
[38] Noah Y. Siegel, Jost Tobias Springenberg, Felix Berkenkamp. Abbas Abdolmaleki. Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin A. Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. In ICLR, 2020.
[39] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 2015.
[40] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.
12

---

[41] Maximilian Seitzer, Arash Tavakoli, Dimitrije Antic, and Georg Martius. On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks. arXiv preprint arXiv:2203.09168 2022.
[42] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219. 2020.
[43] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In NeurIPS, 2019.
[44] Byung-Jun Lee, Jongmin Lee, and Kim Kee-Eung. Representation balancing offline model- based reinforcement learning. In ICLR, 2020.
[45] Toru Hishinuma and Kei Senda. Weighted model estimation for offline model-based reinforce- ment learning. NeurIPS, 2021.
[46] Arthur Argenson and Gabriel Dulac-Arold. Model-based offline planning. In ICLR, 2021.
[47] Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment-efffcient reinforcement learning via model-based offline optimization. In ICLR, 2021.
[48] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: Conservative offline model-based policy optimization. In NeurlPS, 2021.
[49] RT Rockafellar and Stanislav Uryasev. Optimization of conditional value-at-risk. Journal of Risk, 2000.
[50] Aviv Tamar, Huan Xu, and Shie Mannor. Scaling up robust MDPs by reinforcement learning. arXiv preprint 1306.6188 2013.
[51] Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. In ICML, 2019.
[52] Daniel J. Mankowitz, Nir Levine, Rae Jeong, Abbas Abdolmaleki, Jost Tobias Springenberg, Timothy A. Mann, Todd Hester, and Martin A. Riedmiller. Robust reinforcement learning for continuous control with model misspecification. In ICLR, 2020.
[53] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In ICML, 2017.
[54] Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In ICML, 2019.
[55] Elena Smirnova, Elvis Dohmatob, and Jérémie Mary. Distributionally robust reinforcement learning. In ICML Workshop, 2019.
[56] Mare Rigter, Paul Duckworth, Bruno Lacerda, and Nick Hawes. Planning for risk-aversion and expected value in MDPs. In ICAPS, 2022.
[57] Esther Derman, Daniel Mankowitz, Timothy Mann, and Shie Mannor. A Bayesian approach to robust reinforcement learning. In Uncertainty Artificial Intelligence Conference, pages 148-658,11 2020.
[58] Mare Rigter. Bruno Lacerda, and Nick Hawes. Risk-averse Bayes-adaptive reinforcement learning. In NeurlPS, 2021.
[59] H.A. David and H.N. Nagaraja. Order Statistics. Wiley, 2004.
[60] Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint 1XX5:1800000090 2018.
[61] Mare Rigter, Bruno Lacerda, and Nick Hawes. RAMBO-RL: Robust adversarial model-based offline reinforcement learning. arXiv preprint arXiv:2204. 2022.
13


Reviews:
[{"review_id": "bzEaOVOtqHL", "reviewer": "Reviewer_LbFj", "summary": "This paper points out that characterizing the impact of dynamics uncertainty through reward penalty may incur unexpected over-conservative behaviors. To overcome this problem in existing model-based offline RL works, the paper proposes to maintain a belief distribution over dynamics belief and optimize policy via biased sampling from the distribution. The sampling procedure is derived based on an AMG formulation of offline RL. For practical consideration, an offline RL approach named PMDB is further designed. Results on the D4RL benchmark show the state-of-the-art performance of PMDB. ", "questions": "1. I’m not sure if I actually understand the updating of game transitions/dynamics beliefs. Could you please explain it to me in detail?\n2. PMDB relies on the initial dynamics belief and the paper uses a uniform distribution over dynamics as the initial belief. Do you mean inserting additional knowledge into the initial dynamics belief is to pre-define an initial dynamic belief? If so, could you give some examples of the scenarios in which we obtain priors on dynamic belief? If involved the expert knowledge, how much will the performance of the algorithm \n3. I’m concerned about whether the training of PMDB is stable considering the design of this framework?\n\n", "limitations": "The paper discussed the limitation that expert knowledge is not always available thus it’s hard to manually pre-define the dynamics belief. Besides, I think another limitation is the lack of ablations on several key components. ", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n* Motivation: The paper points out an important while has been ignored problem in model-based offline RL, that it’s unsuitable to directly characterize the dynamic uncertainty by reward penalty. \n* Theoretical Contribution: To solve the problem, the paper formulates the offline RL problem as an AMG and discusses the relationship between AMG and robust MDP. Throughout the formulation, several interesting theoretical results are provided. The illustration that AMG is a successive interpolation between model-based RL and robust MDP provides an intuitive explanation of the theoretical results.\n* Empirical Contribution: The paper verifies the proposed PMDB method on D4RL and shows the SOTA performance without the need to tune hyperparameters for each task. \n\nWeaknesses:\n* It may be due to the complexity of the overall framework, the paper is somewhat hard to follow. It took me a very long time to understand the symbols and formulations defined in Section 3.1. \n* Although the framework is very complicated, it seems the core parts that work are: (1) dynamically adjusting the game transitions in each training iteration; (2) Updating the Q-function based on the K-percentile dynamics instead of the worst one. While with the theoretical proof, it would be better to do ablations on the above two parts to further illustrate the effectiveness of the designs.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Vid6s74DSYt", "reviewer": "Reviewer_998y", "summary": "Existing studies on model-based offline RL often rely on the quantification of the  uncertainty in the learned dynamics and utilize it to regularize the reward.  One drawback of this approach  is that it may result in unreliable evaluation on the impact of the uncertainty and incur unexpected tradeoff between model utilization and risk avoidance. To resolve this issue, the authors introduced a novel model-based offline RL algorithm with pessimism-modulated dynamics belief, where the uncertainty is not explicitly quantified but  a belief distribution over system dynamics is used for sampling instead, when updating the policy evaluation and policy update.  The sampling procedure is cleverly modeled as an alternating Markov game and the degree of pessimism can be determined by the hyper parameters in the sampling procedure. The proposed offline RL algorithm solves the AMG by iterative regularized policy optimization with monotonous improvement guarantee. The empirical results on D4RL shows the efficiency and performance improvement of the proposed algorithm. ", "questions": "How to construct a sufficient belief without expert knowledge  for an arbitrary task? How to tune the hyper parameter $k$ and $N$ when processing a new task? \n\nHow to determine the weights in Eqn. (11)? The optimization over the Bellman residual of both the AMF and the empirical MDP ia applied with the same weight (both are 1).", "limitations": "One limitation of this work is that is is unclear a prior how to obtain a sufficient belief without expert knowledge. It will be of interest to shed light on how to construct a valuable (initial) dynamic belief for arbitrary tasks.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n+ The proposed methodology is explained in detail and the paper is easy to follow. The empirical results and ablation studies shwocase the efficiency of the proposed approach on the standard benchmarks. \n\n+ The formulation of offline RL as AMG is interesting and leads to the development of pessimism-modulated dynamics belief derivation.\n\n+ The theoretical results on the pessimism-modulated dynamics belief is promising and serves good guidance on the empirical offline RL algorithm design.  \n\nWeakness:\n- The experimental results can be made more comprehensive. In Figures 2 and 3, what is the possible cause of the large dip in half-cheetah-medium large dip in halfcheetah-medium task?\n\n- How to tune the hyper parameter $k$ and $N$ when processing a new task? \n\n- How to determine the weights in Eqn. (11)? The optimization over the Bellman residual of both the AMF and the empirical MDP ia applied with the same weight (both are 1).\n\n- Line 194,   the reference to Eqn. (18) is undefined.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Uuwsmb6R8fj", "reviewer": "Reviewer_y5WY", "summary": "The paper proposes an Alternating Markov Game formulation of offline RL which induces a MDP with pessimistic dynamics. The authors derive an approximate solution to this game which requires no uncertainty penalization and show strong results on the D4RL MuJoCo benchmark.", "questions": "- How does the slow-evolving policy update relate to TRPO/PPO style KL-constrained policy optimisation?\n- It would be interesting to understand more the motivation behind the design of the optimization procedure in Section 4. How inefficient is solving the original problem, what is the overall gain in training speed by using the reference policy?\n", "limitations": "Efficiency of the algorithm should be detailed in the main paper, as well as discussion of related work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Strong theoretical background including derivation of connection between AMG and the equivalent MDP, well-related to standard offline MBRL in the $N=k=1$ case.\n- Strong empirical results on the D4RL MuJoCo benchmark. The offline evaluation is thorough uses a single hyperparameter setup across all environments, thus avoiding using online samples to tune performance for future deployment.\n\nWeaknesses:\n- Presentation is sometimes dense, related work should be discussed in the main paper.\n\nMinor:\n- Typo on line 22, 345: trial instead of trail.\n- The version of D4RL dataset used in the main evaluation should be highlighted, there are performance differences between v0-v2.\n- Section link in line 498 broken.\n- [1] considers a related formulation which also induces pessimistic dynamics. It would be valuable to compare these works, but I appreciate this work appeared after the ICML deadline.\n\n[1] Rambo-rl: Robust adversarial model-based offline reinforcement learning. M Rigter, B Lacerda, N Hawes.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HMvhWqJy8rL", "reviewer": "Reviewer_Bm2w", "summary": "The paper proposes a mechanism for offline RL based on a two player game with a Bayesian environment, in which an RL actor picks a policy and a environment actor picks environment transitions from a likely set pessimistically. The authors show general convergence and existence results in their framework and empirical results based on the established D4RL framework.", "questions": "- 295: if you assume online-access here, do you break offline RL assumptions? In that case, the core assumptions of the scenario should be clarified here. This would make the comparison between this algorithm and fully offline methods slightly less convincing, if you assume i.e. a few-shot scenario.\n- Theorem 2: is that MDP stationary? It depends on pi, so that might make things more complicated? If the policy has to be learned in a changing MDP, how does that affect learning? Theorem 1 states that the operator is a contractive mapping, but \n- (14) why not KL as difference of distributions, or Wasserstein? Is there a specific reason for this concrete formulation?", "limitations": "A brief discussion is given, without mentioning any major drawbacks in depth.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "== Strengths ==\n- The core theory is well laid out and supported with theoretical results as well as empirical investigation.\n- The structure is overall well laid out and easy to follow. Some minor points of confusion and suggestions are discussed below, but in general, the paper is presented in a clear legible fashion.\n\n== Weaknesses ==\n- The paper is missing a clear related work section. This makes it harder to position it in the relevant literature and makes it difficult to assess whether concurrent similar strands of the literature were reviewed. Since I am not an expert in the concrete field the authors are tackling, some of the following concerns might be wrong or missing relevant details, but I will list them for discussion:\n  - My biggest concern is that no other works on Bayesian Robust RL were cited. The formulation the authors present seems novel, but to the best of my knowledge several other works have tackled Bayesian Robust RL and a clear comparison would greatly improve the paper. I.e. how does the proposed method relate to https://arxiv.org/abs/1905.08188 and similar works.\n  - The other strand of research I am missing comparison to is CVaR style algorithms, which seem similar in its setup in targeting a quantile of returns. For example, this paper from NeurIPS 2021 seems very related https://papers.nips.cc/paper/2021/hash/08f90c1a417155361a5c4b8d297e0d78-Abstract.html\n- The set generating mechanism and definitions in (3) could be explained better. Big $\\tau$ seems to refer to a set of transitions, but $\\bar{s}$ is a state (or a single transition)? This seems to be explained in the text under the equations, I think it might be clearer for readers if the state and action spaces of the 2nd player was fully explained in the introduction of the formalism.\n\nUnclear lines/grammatical errors:\n- Line 51: with entirely different implication compared to the reward or Q-value?\n- Line 140: To differentiate with uncertainty set we call it candidate set, but to avoid redundancy we reuse the notation of T?\n- Line 24: in vast of applications?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept