Offline Model-based Adaptable Policy Learning
Authors: Xiong-Hui Chen
Yang Yu
Qingyang Li
Fan-Ming Luo
Zhiwei Tony Qin
Shang Wenjie
Jieping Ye
Venue: NeurIPS 2021 Poster (2021)
Keywords: Reinforcement Learning

Abstract: In reinforcement learning, a promising direction to avoid online trial-and-error costs is learning from an offline dataset. Current offline reinforcement learning methods commonly learn in the policy space constrained to in-support regions by the offline dataset, in order to ensure the robustness of the outcome policies. Such constraints, however, also limit the potential of the outcome policies. In this paper, to release the potential of offline policy learning, we investigate the decision-making problems in out-of-support regions directly and propose offline Model-based Adaptable Policy LEarning (MAPLE). By this approach, instead of learning in in-support regions, we learn an adaptable policy that can adapt its behavior in out-of-support regions when deployed. We conduct experiments on MuJoCo controlling tasks with offline datasets. The results show that the proposed method can make robust decisions in out-of-support regions and achieve better performance than SOTA algorithms.

Offline Model-based Adaptable Policy Learning
Xiong-Hui Chen', Yang Yu 1,,, Qingyang Li?, Fan-Ming Luo', Zhiwei Qin",
Wenjie Shang", Jieping Ye²
National Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China 2 Al Labs, Didi Chuxing
3 Polixir.ai
chenxh@lamda nju. edu. en, yuyenju. edu. en, gingyanggl@diidgglobbll. luofm@lamda. nj..nuu..uuu.....
edu.
edu. en
{qinzhiwei, shangwen hegpanggeeeie...ggaaaigggllllaa yejjeeping/@didiggolbll..oom
Abstract
In reinforcement learning, promising direction to avoid online trial-and-error costs learning from an offline dataset. Current offline reinforcement learning methods commonly learn in the policy space constrained to in-support regions by the offline dataset, in order to ensure the robustness of the outcome policies. Such constraints, however, also limit the potential of the outcome policies. In this paper, to release the potential of offline policy learning, we investigate the decision-makee making problems in out-of-support regions directly and propose offline Model-based Adaptable Policy LEarning (MAPLE). By this approach, instead of learning in in-support regions, we learn adaptable policy that adapt its behavior in out-of-support regions when deployed. We conduct experiments on MuloCo controlling tasks with offline datasets. The results show that the proposed method can make robust decisions in out-of-support regions and achieve better performance than SOTA algorithms.
Introduction
Recent studies have shown that reinforcement learning (RL) is promising approach for real-world
applications, c.g., sequential recommendation systems [1, 2, 3, 4] and robotic locomotion skill learning [5, 6]. However, the trial-and-error of RL in the real world [7] obstructs further applications in cost-sensitive scenarios [8].
Offline (batch) RL learns policy within static dataset collected by behavior policy without additional interactions with the environment [8, 9, 10, 11]. Since it avoids costly trial-and-erroo in real-world environments, offline RL is promising way handle the challenge in cost-sensitive applications. A significant challenge of offline RI. in answering counterfactual queries, which asks about how the performance (e.g., Q value) would have been if the agent unseen action sequence, then learning to make optimal decisions based on the performance [8]. Fujimoto al. [10] have shown that the distributional shift of states and actions, which comes from the discrepancy between evaluated policies and behavior policies, often leads to large extrapolation error in value function estimation. In traditional model-free algorithms, the extrapolation error in value function estimation hurts the generalization performance of the learned policies. Since the additional samples, which can correct value estimation errors, are unavailable in the offline setting, the performance of learned policies based on value function is unstable [10].
execute
On the other hand, model-based RI. techniques, which learn dynamics models from collected datasets and learn the value function and policies based on the dynamics models, do not need to estimate
CCoresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

the value functions rely on the collected datasets. However, similar challenges occur in dynamics model approximation. The dynamics model might overfit the limited dataset and suffer extrapolation errors in regions that behavior policies have not visited, which causes instability of the learned policy when deployment [12]. Here we call out-of-support regions. Moreover, in model inference, the compounding error, that is. the accumulated prediction errors between simulation trajectories and reality, would be large even if the one-step prediction error is small [13, 14]. Recent studies in offline model-based RL [12, 15] have made significant progress in MuJoCo tasks [16]. These methods constrain policy sampling in dynamics models for robust policy learning. By using large penalty [12] or trajectory truncation [15, 12] in the regions with large prediction uncertainty (uncertainty is designed metric to evaluate the confidence of prediction correctness) or compounding error, policy exploration is constrained in the regions of dynamics models where the predictions corrected with high confidence, so as to avoid exploiting regions with risks of large extrapolation error. However, the constraints dynamics models lead to conservative policy learning process, which limits the potential of leveraging dynamics models: The visits on states and actions in out-of-support regions more likely be inhibited by the constraints, making the learned policy restrict the agent to be in similar regions as the behavior policy.
From the perspectives of counterfactual queries, we consider that model-based RL is promising to handle offline RL - ideal reconstructed dynamics models can simulate the transition dataset without the distributional-shii problem given any policy, and the value function can be estimated via the "simulated" transition dataset directly. The bottleneck of offline model-based RL comes from the policy learning in the approximated dynamics model with extrapolation error. In this paper, instead of learning by tightly constraining policy exploration in in-support regions, we investigate decision-making in out-of-support regions directly. Finally, we propose new offline policy learning framework, offline Model-based Adaptable Policy LEarning (MAPLE), to address the aforementioned issues. Ideally, MAPLE tries model all possible transition dynamics in the out-of-support regions. Then an Adaptable policy is learned to be aware of each case to adapt its behavior to reach optimal performance. In the practical version of MAPLE, we use an ensemble technique construct ensemble dynamics models. To be aware of each case of the transition dynamics and learn an adaptable policy, we use a meta- lettallarnnng technique that introduces an extra wwtroomenntcootteee extractor structure to represent dynamics patterns, and the policy adjusts itself according to the environment contexts.
We conduct experiments on the MuloCo tasks. The results show that the sampling regions for robust offline policy learning can be extended via constructing transition patterns in out-of-support regions to cover the real case. The output adaptable policy yields better performance than SOTA algorithms when deployed. MAPLE gives a new direction to handle the offline policy learning problem in the dynamics models: Besides constraining on sampling and training dynamics models with better generalization, we can also model out-of-distribution regions by constructing all possible transition patterns.
2 Related Work
Reinforcement learning (RL) has shown to be promising approach to complex real-world decision- making problems [1, 2, 3, 4]. However, unconstrained online trial-and-error in the training of RL agents prevents further applications of RL in safety- ceriticl scenarios since it might result in large economic losses [8, 17, 18. 19]. Many studies propose overcome the problem by offline (batch) RL algorithms [20]. Prior works on offline RL are based on model-free algorithms. To overcome the extrapolation error, which is introduced by the discrepancy between the offline dataset and true state-action distribution of learned target policies [10], these methods are designed to constrain target policies be close to the behavior policies [10, 211, to apply ensemble methods for robust value function estimation [22], or to re-wetght samples in datasets with importance sampling [23]. Most recent studies have shown that policy learning with an approximated dynamics model has good potential to take robust actions outside the action distribution of behavior policies [15, 12]. The challenge comes from the extrapolation error of the dynamics models in oft-of-support regions. To address the issues, these methods learn policies from dynamics models with uncertainty constraints. Uncertainty a measure of prediction confidence on next states. The uncertainty often computed by the inconsistency in the ensemble dynamics model predictions for each state-action pair. Kidambi et al. [15] construct terminating states based on a hard threshold on uncertainty, while Yu et al. [12] use soft reward penalty to incorporate uncertainty. The penalty constrains policy exploration and

---

In-support region Learned policies
(a) learn to adapt (MAPLE)
(b) learn by constraining
Figure I: Illustration of MAPLE compared with learning by constraining. The pointed lines represent the optimal trajectories of the learned policies. There are several policies in MAPLE since the method learns adapt to multiple dynamics models. The gray oval represents the in-support region.
optimization to the regions with high consistency for better worst-case performance in the deployment environment [15, 12].
The difference between the aforementioned model-based methods and MAPLE shown in Figure 1. Compared with previous methods [15, 12] learned by constraining (in Figure 1(b)), we learn adapt to all possible dynamics transitions the states in out-of-support regions (in Figure ((a)..
Background and Notation
In the standard RL framework, an agent interacts with an environment governed by Markov Decision Process (MDP) [24]. The agent learns a policy π (ar(s:), which chooses an action a₂ E A given a particular state E S, at each time-step denote the state and action spaces, respectively. The reward function r; = r(Be, 0r) E evaluates immediate performance of the action a, given the state st. The goal of RL is to find an optimal policy n which maximizes the multi-step cumulative discounted reward (i.e., long-term performance).
{0, 1, ..., ..,,, where T is the trajectory length. S and A
The objective of RL is max, (T) Er-p(ri=,p) Ex=o>""> where 7 is the discount factor, and p(r|mT p) is the probability of generating trajectory
[so, Gn .... -1- sT under the policy and a dynamics model p(8++1 [8t, a.). In particular, p(r т) == do (8)) - p(8++1 8t, at (a, (st), where do ds() is the initial state distribution. A common way to find an optimal policy T" is to optimize the policy with gradient ascent along (., [24, 25].
In the offline RL setting, we are given only a static dataset D = {s;,, ai Tú, 11 collected by some unknown policy. The goal is to obtain policy that maximizes J, by only using the static dataset.
4 Method
We argue that in current offline model-based methods, the constraints of sampling to in-support regions of the dynamics model lead conservative policy learning process, limiting the potential of leveraging dynamics models. In this paper, to relax the constraints on the dynamics model, we investigate decision-male making out-of-support regions directly.
In this section, we first give a motivating example to show our ideal solution to out-of-support region docision-making (in Section 4.1). Then, introduce practical algorithm of the proposed solution for complex tasks, based on meta-learning techniques (in Section 4.2 and Section 4.3).
4.1 Decision-Making in Out-of-Support Regions
By rethinking the scheme of offline model-basee RL, without loss of generality, we first formulate the problem as decision-making with a partially known dynamics model (Pak-DM) in surrogate objective. In this problem, we have two dynamics models: a target dynamics model p and partially known dynamics model p', where p is the deployment environment in the offline RI setting, and p' is used approximate p. Due to the bias of data sampling in the offline setting and the
3

---

limitation on the capacity of the function approximator, only in part of state-action space, we have s, a), while the transitions other parts of space are uncertain. We call the satisfied space "accessible space" (a.k.a., in-support regions) and its complement "inaccessible space" (a.k.a. out-of-support regions). In this problem, we assume the two subspaces have been predefined in some ways as in the offline setting (e.g., we can define the space in which an uncertainty quantification is larger than a threshold as the inaccessible space) and then discuss the decision-makinn problem in this setting. Formally, given an accessible space X, and an inaccessible space X, the partially known dynamics model is defined as:
al p(s"|s,
(p(s'|s,a) Unknoww
[s, a] E X [s, a] X;
(5'18, =
where denotes the state-action concatenated space for brevity and [s, a) denote a vector concate- nating and a. Our objective is to find a policy n² to maximize Jp by only querying the partially known dynamics model p'. If X; = 0, that is p (ss"'ss, a) = p(s'|8, a), Vs E ESaa E A, the problem is reduced to a vanilla model-based policy learning problem with an oracle dynamies model. For simplification, we assume the oracle reward function is given, but it can also be formulated as
partially known reward function in a similar way. Now we give an example of a Pak-DM in Figure 2.
r(Ci) = 100
a
C2
r(A) =1
D
C2
((2)) 20
Figure 2: An example of a Pak-DM. Each node denotes a state. Here we consider an MDP with finite state space, including A, B, C, C2 and D. The directed edges denote the transition process. Here we consider a one-action transition in all states except for state B. On state B, we have action a and 8, which are denoted as square nodes. By taking a state B, the state will change to A. However, the transition after taking 3 in is unknown. We use dashed directed edges to denote the possible transitions. In our formulation, edges to any node are valid. But we just consider the edges from B to C1 and C2 and omit the edges to A, B, and D for better readability. The reward function r(s) will give reward when the agent reaches A, C2 or C2-
In this setting, the model-based policy learning algorithms with constraints ([12, 15)) can be summa- rized as: finding the optimal policy without reaching inaccessible space. It might output a conservative policy since it avoids making decisions that might lead agents to out-of-support regions. Taking Figure 2 as an example, the output policy would be run in the loop of A - B + a - A. will avoid the penalty
avoid taking B on state B. If the real transition is p B, 3) C2. the policy -20 since C2 will not be reached. On the other hand, while p(B, B) = C1. the policy would miss the large bonus 100 in C1.
Our question is, how should we make good decisions in out-of-support regions directly so that we can make better use of the approximated p' for better performance? We give new paradigm to solve the problem which is the ideal implementation version of MAPLE:
1. (Training) Construct a dynamics model set { via modeling all possible transitions in X, (It is impractical to do that with infinite state space. We will give practical solution in Section 4.3);
2. (Training) Learn the optimal policy from each model P. to form the optimal policy set [m] J;
3. (Deploymentt Initialize a state 8o from the deployment environment p;
4. (Deployment) Probe the environment p by selecting an action a such that [s, a] E X, After getting the next state s' = p(s')s (ss a), store the tuple (s, a, s') in memory (c.g., replay

---

buffer) D. If there is no action a that allows [s, a] E X, randomly select a policy from the policy set take an action.
5. (Deployment) Reduce the policy set by only keeping the policies whose corresponding
transition model Pr can explain the experiences in the memory: {o:} < {o p(
8', V(s, n, &) D, E {ir]} and (T + (Tāle E (A:)):
6. (Deployment) Repeat Step 4 and 5 until the policy set is reduced a single policy.
In this paradigm, we solve the decision-making problem in out-of- regions by probing the uncertainty part of the deployment environment and adapting the policy for the environment. In Figure 2, the ideal MAPLE solution would construct two dynamics models: P1 where ý1 (B, Ct. and PO where (2 3) = Cz. Then we learn two optimal policies {T Tpp for each dynamics model. At deployment, we first randomly select policy from the policy set to make decisions. Upon reaching B for the first time, where the transition on action | is uncertain, we take action B and get the next state. If the next state is C1, the policy will reduce to x otherwise to T2.- Therefore, if pBB, () C2. the policy would initially run A - B + 3 C2 D A and then run in the loop of A + B - a + A because the latter yields higher rewards. If (B, C₁, the policy would always run in the loop of A + B 3 + C1 - D A.
We then dive into the performance difference of the two paradigms for decision- Formally, give Theorem to describe it. The full proof can be found in Appendix A.
Given a target dynamics model p. a policy Ta learned by adapting, a policy Te learned by constraints, and the maximum step Nm taken by na to probe and reduce the policy set to a single policy, the performance gap between Ta and Te satisfies:
Theorem
Jp(Ta) - - Jpa
where De denotes the performance gap of the optimal policy n² and To while Dp denotes the performance degradation of MAPLE compared with n¹ because of the phase of probing. Jos (T*) denotes the performance degradation of x² on the dynamics model p caused by different initial state distribution: Jpa. (T²) = Rag (1(5)) [V* (s)] t1)) (V)(V (8)], where (1)) and & (s) denote the state distribution induced by x² and n. at the Nn. + 1 step and V (s) denotes the expected long-term rewards of n²" al state s.
We can see that the performance gain of n. is that it can automatically converge to the optimal policy after the loop of probing and reducing (i.e., Ac), while the cost of ma comes from additional probing on inaccessible space, including less reward getting when probing (i.e., Ap) and worse initial state distribution after probing (i.e., Jpa (T")).
Based on Theorem 1, we give the principles for choosing between the paradigms: Firstly, with a larger performance gap of Ac. Ta can reach better performance than Te- On the other hand, the tasks with large penalties on undesired behavior might make A, larger, which reduces the overall performance of a.. Besides, the tasks where sub-optimal behaviors easily lead agents to states with low value, e.g., unsafe states which are prone to terminate the trajectory, might make Jps (T*) large, which also reduces the overall performance of T..
4.2 Efficient Decision-Making in Out-of-Support Regions with Meta-learning Techniques
It is computationally inefficient to learn optimal policies independently for each dynamics model since the policies' behaviors would be similar in in-support regions. For better efficiency, we introduce a context-aware adaptable policy, inspired by meta-learning techniques, to represent the set of learned policies. Here, first introduce a new notation: the environment-conten vector z E 2, where Z denotes the space of the context vectors. Given a set of dynamics models T := { each P. can be represented by a vector 2. Formally, there is a mapping context extractor. The context-aware policy T [a]ss, 22 takes actions based on the current state s and the vector of environment-conee optimal wewiiromment-conter extractor o² to satisfy: 3 E II, v E T, Ja (T ) = max; JT(T), where no := I ( (6( pp)). st) is an adaptable policy and denotes the policy class. We discuss the input of & later. In addition, we define the optimal adaptable policy x to be one that satisfies
T + Z. We call ø an environment-
z of a given environment = o(p). where X T. We define the
vp E T, ())(TT)-) max, JT(T)). With the optimal o" and T and given any in T, the adaptable
5

---

policy can make the best decisions with the output of environment-come context 2. To achieve this, given a dynamics model set T, we optimize @ and Tg by the following objective function:
arg max Ep- [J)(T))). 0.5g
where denotes a sample strategy to draw dynamics models ş from the dynamics model set 7 s.t. P(A) > 0, Vp E TT We take uniform sampling strategy in the following analysis.
To learn the context by d(=]p), the main question is: What are suitable inputs to e for context learning? In the robotics domain, similar environment contexts have been proposed recently [26, 27,
28]. The policy incorporates an online system identification module o(2z (s, To:e), which utilizes the history of past states and actions To: = | do,...., ...,, 8t-1, ( to predict the parameters of the dynamics in simulators. For example, T could be a trajectory of robot interactions with varying friction coefficients, and is the value of the coefficient. In practice, a recurrent neural network (RNN) used to embed the sequential information into environment-conte In MAPLE, we follow the same structure to model the extractor but the trajectories are rollout in the constructed dynamies models. If the reward function is also partially known, rt-1 should be considered, that is 2 Φ(st, 11-1. 2t-)) 4-1).
8t]
vectors 2, = ( 4 2t-1), 1).
With an RNN-based environment-conter extractor ei optimized with Equation (1), the context-aware policy π can automatically probe environments and reduce the policy set. Considering a given i-step partial trajectory To: and a subset of deterministic dynamics models T T where the dynamics model ò ' is consistent with Tox, the objective from i + 1 to T on given To: can be rewritten kmi++ rrl] Since the dynamics models in T' are indistinguishable at step i + 1, the optimal policy at this step would converge to a stochastic policy if the optimal actions are different among the dynamics models. If is sampled uniformly from T' and the optimal cumulative rewards _k=it1 7*rī are the same for each dynamics model, the optimal policy at + 1 is to mniformly-samplee actions from the optimal actions of each dynamics model. If the optimal cumulative rewards are different, the action probabilities are weighted by the cumulative rewards of each dynamics model. On the other hand, partial trajectories from different dynamics models might predict the same z. If the optimal actions in the same state are conflicting, increase the performance of objective Equation (I), the policy gradient has to backpropagate from и to ø. If the partial trajectories To. are different, the contexts in these partial trajectories would be distinctive. Finally, the output action distribution of the context-aware policy would be optimized in subset of the dynamics models in which the partial trajectories Tox are consistent.
as (p [Br=p(n,
12 T
4.3 Practical Implementation of Offline Model-based Adaptable Policy Learning
From the decision-making problem with Pak-DM to the real offline setting, the additional thing we should consider is the recognition of the inaccessible space and the construction of the dynamics model set. Especially in tasks with infinite state-action space, it is impractical to find the exact inaccessible space and recover all possible transitions in it. As a practical implementation, we use the ensemble technique to learn the dynamics model set, which would predict similar transitions in the accessible space and tend to predict different transitions in inaccessible space. With large ensemble models with different initialized weights, we can construct large number of transition cases in the inaccessible space. If the real transition pattern falls into the variant of the ensemble dynamics model set, relying on the interpolation ability of the environment-contex extractor Φ. the adaptable policy π can take appropriate actions. However, only relying on the randomness of the initialization, to cover real cases in all out-of-support region need sufficiently enough dynamics model set, which would be highly expensive. In order to trade off the cost of model construction and better adaptivity, the practical implementation, we use several tricks to constrain the policy exploration in the ensemble dynamics model set: 1) We add some mild constraints to the exploration region. To mitigate the compounding error of the model, we constrain the maximum rollout length fixed number. Besides, at each step, a penalty is calculated according to the model uncertainty U(s, (t), which measures the prediction uncertainty of the learned transition models (st, az). As a result, a reward provided to agents consists of two parts: reward given by reward function and penalty calculated by U (st, a.). The constraints are the same as MOPO [12], but the coefficients are more relaxed; 2) As we increase the rollout length, the large compounding error might lead the agent to reach entirely unreal regions in which the states would never appear the deployment environment. These samples are useless for adaptable policy learning and might make the training process unstable.

---

Algorithm Offline model-based adaptable policy learning
Input:
O, as an environment-contxx extractor, parameterized by yo; Adaptable policy network Te
parameterized by 0: Offline dataset Doffi Trajectory termination rule f; Rollout horizon HI; Process:
Generate an ensemble of m dynamics models } via supervised learning
Initialize an empty buffer Dollouti Add hidden states a to each tuple Drollat and initialize with
for 2, 3, do
Randomly select the dynamies model Pi {pt} and sample 81, up, 20 from Doff for (=1, 2, H do
Sample 24 from @z(2/5e, at-1, 2-1) and then sample az from To ( zt) Rollout one step 8(+1 (s(sstt, 0e) and r1+1 r(5,, as)
Compute the terminal state d = =((+++)
Compute the reward penalty: r2+1 <- T+41 (U(s
Add ($t41, Tr+1, d++1, St. dg. 2²) to Drottou
Break the rollout if de+1 is True end for
Update the stored hidden states in Dair with di. Use SAC [29] to update yp and via Equation (1) with Droilout and Dar
end for
Given task, construct some simple rules discriminate the entirely unreal regions and terminate the trajectory rollout (i.e., set the "done" flag to True) when reaching these regions. In implementation, terminate trajectories when the predicted next states out of range of (-8min, $max), where 8min and 8max are two hyper-parameters to define a reasonable range of state space.
Based on the above techniques, we propose the practical implementation of offline model-based adaptable policy learning in Algorithm 1. More details can be found in the Appendix D.
5
Experiments
evaluate MAPLE on multiple offline MuloCo tasks [16]. All the details MAPLE's training and evaluation given in Appendix E and Appendix F. We release our code at Github
5.1 Comparative Evaluation on Benchmark Tasks
Table 1: Results on MuJoCo tasks. Each number is the normalized score proposed by Fu et al. [30] of the policy at the last iteration of training, standard deviation. Among the offline RL methods, we hold the highest mean for each task.
Environment
Dataset
MAPLE
MOPO
MOPO-loose
SAC
BEAR
BC
BRAC-v
CQL
Walker2d
random
21.7 + 0.3
13.6 2.6
5.4
4.1
6.7
9.8
0.5
7.0
Walker2d
medium
56.3 + 10.6
11.8 19.3
32.6 18.0
0.9
33.2
6.6
81.3
79.2
Walker2d
mixed
76.7 + 3.8
39.0
35.7
3.5
25.3
11.3
0.4
26.7
Walker2d
med-expert
73.8 + 8.0
44.6+12.9
66.7 14.8
-0.1
26.0
6.4
66.6
111.0
HalfCheetah
random
38.4 1.3
+.5
3.1
30.5
25.5
2.1
28.1
35.4
HalfCheetah
medium
50.4 1.9
4
1.6
-4.3
38.6
36.1
45.5
44.4
HalfCheetah
mixed
59.0 0.6
5..1
36.9 15.0
-2.4
36.2
38.4
45.9
46.2
HalfCheetah
med-expert
63.5 6.5
63.3++3.00
1.00
1.8
51.7
35.8
45.3
62.4
Hopper
random
10.6
11.7+
10.6
9.55
1.6
12,0
10.8
Hopper
medium
21.1 1.2
28,0 12.4
1..9
0.8
58.0
Hopper
mixed
87.5 10.8
67.5 24.7
83.1 6.5
0.9
Hopper
med-expert
42.5
23.7
25.1
4.0
111.9
We test MAPLE standard offline RL tasks with D4RL datasets [30]. The ensemble dynamics model set is trained via supervised learning. We repeat each task with three random seeds. In the

---

model learning stage, we train 20 models for each task and select 14 of them as the ensemble model for policy learning. The horizon H set to 10 in these tasks. The policy trained for 1000 iterations in the policy learning stage.
We compare MAPLE with: (1) MOPO: Learn ensemble model via supervised learning and learn policy in the ensemble models with uncertainty penalty [12]: (2) MOPO-1oose: MOPO algorithm with the same hyperparameter as MAPLE for constraints, which is looser than MOPO; (3) BEAR:
Learn policy via off-policy RL while constraining the maximum mean discrepancy of the current policy to the behavior policy [1 (4) BC: Imitate the behavior policy via supervised learning; (5)
SAC: Perform typical SAC updates with the static dataset [29]; (6) BRAC-v: A behavior-regularize actor-critic proposed by Wu al. [21]; (7) CQL: Learn an action-valle function with regularization to obtain a conservative policy [3 11. Results of (3-7) and (I) are taken from the work of [30] and [12].
Table I has shown the performance of 12 tasks. In summary, the performance of MAPLE on 7 tasks is better than other SOTA algorithms. Besides, MAPLE reaches the best performance among the SOTA model-based conservative policy learning algorithms in 10 out of the 12 tasks. These results demonstrate the superior generalization ability of MAPLE.
We can also find that the performance of MAPLE and MOPO are higher than other model-free baseline algorithms in most of the tasks, which reveals that model-based methods can find better policies by taking actions outside of the action distribution of the behavior policies. However, in Hopper experiments, model -ree methods BC, CQL, and BRAC-v often reach better performance. We consider that it is because the environment is unstable, more diverse collected data is needed for robust dynamics model learning. Even in MAPLE, a finite number of ensemble dynamics models might not cover the real case that make robust adaptation. Therefore, only in the "Hopper-mixed" task, which has diverse collected data, MOPO and MAPLE can improve the deployment performance.
We also conduct MOPO-loose for each task, which shares the same prpra-pprratters with MAPLE, including the ensemble model size, rollout length, and penalty coefficient. The results show that for some cases (e.g., Walker2d-mer mexeeeeperr and Hopper-mixeds MOPO- can also enhance the performance. We consider the improvement coming from the constraints in original implementations to be too tight. However, in most of the tasks, the improvement is not significant MAPLE. This phenomenon indicates that the improvement of MAPLE does not come from parameter tuning but self-adaptation mechanism.
The implementation of MAPLE shares the cross-domain idea with meta-RL, particularly domain randomization and sim2real. In the current implementation, the online system identification (OSI) method meta RL is employed [26, 27, 28J. Other techniques, e.g., VariBAD [32], can also be integrated into MAPLE. We construct another variant of MAPLE, i.e., VariBAD-MAPLE, as another implementation. SariBAD-MAPE Compared with the original implementation of MAPLE, BariBAD-MAPLE can do better than MAPLE in Walker2d melkkerdd-medimmmeem and HalfChectah-randdee but worse than MAPLE in Walker2d-mixo and HalfCheetah-mixx The detailed results can be found in Appendix F.6.
can also reach a significantly better performance than MOPO.
5.2
Analysis of MAPLE
0.8
cormalpedie retur
6.6.
0.8
nommllpad
8.5
Mu
So H--20
m-7 1-200
0.4
m-7 huud0
100 Ii-10
0.3
NOTO
parg
1--10
02
11-20
203
400
600
200
2003
0.0
200
402
630
502
1002
200
400
600
500
1000
epachs
epochs
epochs
(a)
H=10
(b) H=20
(c) H=40
Figure 3: The learning curves of MAPLE with different parameerssssers m and H. The solid curves are the mean of normalized return and the shadow is the standard error.
8

---

The ultimate target of MAPLE is handling the decision-mand making challenge out-of-uupport regions. By constructing all possible transitions in out-of-support regions, the probing-reducing loop can find the optimal policy finally. However, it is impractical construct dynamics model set that covers all possible transitions in out-of-support regions. In practice, we construct an ensemble dynamics model set and use loose constraints on policy sam- pling to expand the exploration boundary for better asymptotic performance. Therefore, with larger size of the model set, the dynamics models can cover more real transitions in out-of-support regions, then MAPLE is expected to have better performances by this section, to verify the argument,
3.0
Jurted anpole
Geploycc norm malaaaddd rew
2.0
trained
2.0
0.0
25
32
35
horizon
Figure 4: A Comparison of trained and deployed rewards at the 1000th epoch based 50 and among different H.
relaxing the constraints. analyze the relationship among constraint degree, the size of ensemble models, and the asymptotic performance.
In MAPLE, the constraint degree can be evaluated by the rollout length (H) or the reward penalty coefficient (A); The size of ensemble models is m. The asymptotic performance is evaluated by the cumulative rewards at the 1000th iteration. We select Walker2d- medium to verify the argument. With the fixed X, we compare the asymptotic performance of different H and
We give the results in Figure 3. Firstly, for all H, by increasing the size of m, the performance of the converged policies is improved, On the other hand, without enough ensemble models, too loose constraints will result in worse performance. In particular, for the setting of m = 50, when H 10, the final normalized return is near 0.7. However, as H increases, the asymptotic performance drops gradually. When = 40, the asymptotic performance is even worse than MOPO. As can be seen in Figure 4, the trained one-step reward is similar among different horizons. It means that the performance of policies in dynamics models is similar. The worse performance indicates that the adaptable policy overfits the finite dynamics models and ø fails to infer correct environment-contex to the adaptable policy when deployed. The issue can be remedied by constructing more dynamics models. As depicted in Figure 3(c), by increasing the model size 200, the asymptotic performance recovers around 80%.
We also conducted additional experiments in Walker2d-mixed and HalfCheetah- to visualize the z. The result reveals that different dynamics models are distinguished by z and are approximated divided into several groups. Besides, sample trajectories for 1000 time-step in the deployment environment. we found that the value of oscillated within a region. The detailed results can be found in Appendix F.5.
5.3 MAPLE with large dynamies model set
Based on the above analysis. we can get an empirical conclusion that in- creasing the model size is signifi- cantly helpful to find a better and robust adaptable policy via expand- ing the exploration boundary. There- fore, we conduct another variant of the MAPLE algorithm, MAPLE-200, which uses 200 ensemble dynamics
Table 2: Results MuloCo tasks with MAPLE-200.
Environment
Dataset
MAPLE-200 200
MAPLE
Walker2d
random
22.1 + 0.1
0.3
Walker2d
medium
81.3 0.1
Walker2d
mixed
0.9
7.8
Walker2d
med-expert
107.0
8.0
HalfCheetah
random
3..6
3.3
models for policy learning and ex-
HalfCheetah
medium
1.4
1.9
pands the rollout horizon to 20.
HalfCheetah
mixed
69,5
5..0
The results of MAPLE-200 can be found in Table 2. We can see that the empirical conclusions not only suit the falker2d-mediim but also other tasks. In of the tasks, MAPLEE200 reaches at least similar performance to MAPLE, In the tasks like Walker2d-med-exper,, HalfCheetah-mixxd, Hopper-medium, and Hopper- med-expert, the performance improvement of MAPLE-200 is significant. Besides, in the tasks of
HalfCheetah
med-expert
55.4
63.5 6.5
Hopper
random
10.7 0.2
10.6
Hopper
medium
44.1 2.6
21.1 1.1 1.2
Hopper
mixed
85.0 1.0
87.5 10.8
Hopper
med-expert
95.3 7.3
42.5 4.1

---

Hopper-medium and Hopper-mede medeexp where MAPLE and MOPO fail to reach comparable performance to model-free offline methods, MAPLE-200 can reach similar or much better results.
MAPLE-200 demonstrates a powerful adaptation ability. However, we point out that, by increasing the 10x size of ensemble dynamics models, the time overhead for MAPLE-200 training is also larger. For example, by using NVIDIA Tesla P40 and Xeon(R) E5-2630 to train the algorithms, the time overhead of MAPLE-200 is 10 times longer than MAPLE. Besides, obtain dynamics models that covered more real cases in out-of-support regions than MAPLE-200, the size of ensemble models becomes much larger, which is one of the limitations for current MAPLE implementation.
6 Discussion and Future Work
Prior work has demonstrated the feasibility of model-based policy learning in the offline setting by using the conservative policy learning paradigm [12, 15]. is an important breakthrough from zero to one in the offline setting, but it is far from the end of offline model- model-based policy learning. In this work, we investigate the deccision-making first formulate the problem as decision-making in Pak-DM and propose MAPLE, paradigm to solve the problem. We also give a theorem to describe the pros and cons of the paradigms to give principles for the paradigm selection. We verified MAPLE in the MuJoCo tasks, and get our empirical conclusion: by increasing the size of the model set, we can expand the exploration boundary in the approximated dynamics models by using adaptable policy to make better and robust decisions in deployment environments.
problems in out-of-support regions directly. We
MAPLE gives another direction to handle the offlin model-based learning problem: Besides con- straining on sampling and training dynamics models with better generalization, we can model out-of-distribution regions by constructing all possible transition patterns. The current limitation lies (I) The extractor's generalization ability depends on the neural network itself, which is uncontrollable to some degree. Adding some auxiliary tasks might handle this issue; (2) Ensemble is the direct way to cover real transitions in out-of-support regions. However, as the size of the model set becomes large, it is hard to generate new different dynamics models to cover the real cases only by increasing the size of the model. More efficient ways to increase the cover real transitions of the dynamics model set should be further explored.
the implementation:
Acknowledgements
This work is supported by the National Key Research and Development Program of China (2000AAA0107200) and the NSFC (61876077). We thank Zheng-Mat Zhu, Shengyi Jiang, Rong-Jun Qin, Yu-Ren Liu, and the anonymous reviewers for their useful suggestions on the papers.
References
[1] Xiting Wang, Yiru Chen, Jie Yang, Le Wu, Zhengtao Wu, and Xing Xie. reinforcement learning framework for explainable recommendation. In 2018 IEEE International Conference on Data Mining, pages 587-596. IEEE, 2018
[2] Xiangyu Zhao, Liang Zhang, Zhuoye Ding, Long Xia, Jiliang Tang, and Dawei Yin. Recom- mendations with negative feedback via pairwise deep reinforeement learning. In Proceedings of the 24th, ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1040-1048 London, UK, 2018
[3] Xiangyu Zhao, Changsheng Gu, Haoshenglun Zhang, Xiaobing Liu, Xiwang Yang, and Jiliang Tang. Deep reinforcement learning for online advertising in recommender systems. CoRR, abs/1909.03602. 2019.
[4] Han Cai, Kan Ren, Weinan Zhang, Kleanthis Malialis, Jun Wang, Yong Yu, and Defeng Guo. Real-time bidding by reinforcement learning in display advertising. In Proceedings of the 10th. ACM International Conference on Web Search and Data Mining, Cambridge, United Kingdom, 2017.
10

---

[5] Xue Bin Peng, Erwin Coumans, Tingnan Zhang. Tsang- Wei Edward Lee, Jie Tan, and Sergey Levine. Learning agile robotic locomotion skills by imitating animals. CoRR, (etp/2044007888 2020.
[6] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. CoRR, abs/1812.05905, 2018.
[7] Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research, 37(4-5))421-455 2018.
[8] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CoRR, abs/2005.01643, 2020.
[9] Noah Y. Siegel, Jost Tobias Springenherg. Felix Berkenkamp. Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin A. Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In Proceeding of the 8th. International Conference on Learning Representations, Addis Ababa, Ethiopia, 2020.
[10] Scott Fujimoto, David Meger, and Doina Precup. Off- off-policy deep reinforcement learning without exploration. In Proceedings of the 36th. International Conference on Machine Learning, Long Beach, CA, 2019.
[11] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- off-policy q-learning via bootstrapping error reduction. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, Vancouver, Canada, 2019.
[12] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: model- offline policy optimization. CoRR, abs/2005.13239, 2020.
[13] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, 8- 14 December 2019, Vancouver, BC, Canada, pages
2019.
[14] Stéphane Ross and Drew Bagnell. Agnostic system identification for model-based reinforcement learning. In Proceedings of the 29th International Conference Machine Learning, Edinburgh, UK, 2012.
[15] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model-based offline reinforcement learning. CoRR, 0bs/2005505951 2020.
[16) Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In Proceedings of the 24th. IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, Vilamoura, Portugal, 2012.
[17) Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham, and Simon Dollé. Offline A/B testing for recommender systems. In Yi Chang, Chengxiang Zhai Yan Liu, and Yoelle Maarek, editors, Proceedings of the Eleventh ACM International Conference Web Search and Data Mining, pages 198-206. ACM, 2018
[18] Georgios Theocharous, Philip S. Thomas, and Mohammad Ghavamzadeh. Personalized ad recommendation systems for life-time value optimization with guarantees. In Proceedings of the 25th. International Joint Conference on Artificial Intelligence, pages 1806-1812, Buenos Aires, Argentina, 2015. AAAI Press.

---

[19] Philip S. Thomas. Georgios Theocharous, Mohammad Ghavamzadeh, Ishan Durugkar, and Emma Brunskill. Predictive off- policy evaluation for nonstationary decision problems, with applications to digital marketing. In Satinder P. Singh and Shaul Markovitch, editors, Proceedings of the 3st. AAAI Conference on Artificial Intelligence, pages 4740 4745, CA, USA, 2017. AAAI Press.
[20] Sascha Lange, Thomas Gabel, and Martin A. Riedmiller. Batch reinforcement learning. In Marco A. Wiering and Martijn van Otterlo, editors, Reinforcement Learning, volume 12 of Adaptation, Learning, and Optimization, pages 45-73. Springer, 2012.
[21] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. CoRR, abs/1911.11366 2019.
[22] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. International Conference on Machine Learning, 2020.
[23] Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradient with state distribution correction. CoRR, abs/1904.08477 2019.
[24] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
[25] Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems 12, pages 1057-1063, Denver, Colorado, 1999.
[26] Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-Ren transfer of robotic control with dynamics randomization. In Proceedings of the 35th. IEEE International Conference Robotics and Automation, pages I 1-8, Brisbane, Australia, 2018.
[27] Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej. Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojeiech Zaremba, and Lei Zhang. Solving rubik's cube with a robot hand. CoRR, abs/ 0bbs//910.0711.. 2019.
[28] Wenhao Yu, Jie Tan, C. Karen Liu, and Greg Turk. Preparing for the unknown: Learning a univer- sal policy with online system identification. In Nancy M. Amato, Siddhartha S. Srinivasa, Nora Ayanian, and Scott Kuindersma, editors, Robotics: Science and Systems XIII, Massachusetls Institute of Technology, July 12-16, 2017, Cambridge, MA, 2017.
[29] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off- off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th. International Conference on Machine Learning. pages 1556-1865, Stockholmsmässan. Sweden, 2018.
[30] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR, abs/2004.07219, 2020.
[31] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, 2020.
[32] Luisa M. Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep RL via meta-learning In 8th International Conference on Learning Representations, Addis Ababa, Ethiopia, 2020.
12


Reviews:
[{"review_id": "HF3R-dHLiln", "reviewer": "Reviewer_3man", "summary": "This paper proposes to learn adaptable policies in an offline RL setting. One of the limitations of offline RL methods is decision-making in the out-of-support region. This paper tries to leverage existing meta-learning algorithms to learn a latent code z from trajectories as a context variable. Then, a context-conditioned policy would take in both states and context variables for generating actions. They experimentally show this method outperform previous methods on standard offline RL benchmarks. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper considers a novel setting wherein the deployment time, the learned policy might need to adapt to have the best performance. They borrow the techniques from meta-learning to learn context-aware policies that can adapt.  The paper is well written for clearly explaining the motivation, intuition, and practical considerations. The paper would contribute toward realistic offline RL methods.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Mujoco locomotion tasks are standard for offline RL, but in D4RL, there are many more meaningful data that can be used to test the proposed method. It usually remain unclear what happened in the environment while using locomotion tasks as a testbed. For example, except for seeing improved scores, it is unclear to me what has been improved what has been not. Is it really out-of-support? Is context-variable z informative? Can you visualize z? \n\nMAPLE-200 only helps in 2/3 of the tasks. This seems a bit contradictory to the theory part. \n\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "49Uv07VrAqp", "reviewer": "Reviewer_g3xy", "summary": "This paper presents Model-based Adaptable Policy LEearning (MAPLE), a model-based offline RL algorithm that learns a policy that can quickly adapt to an unknown environment during execution. Existing offline RL algorithms constrain the policy to visit only in-support regions, but MAPLE aims to model all possible dynamics models for out-of-support regions and try to learn a dynamic-conditioned (context-conditioned) policy. Policies for various uncertain dynamics models are prepared first, and gradually filter out policies that do not correspond well to transition observations when deployed. However, solving it exactly is intractable, and a practical approximation is presented, which uses an ensemble dynamics model set and an RNN-based environment-context extractor. Experimental results show that MAPLE outperforms baselines in D4RL benchmarks, showing better generalization ability.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well written and easy to follow. In contrast to existing offline RL algorithms whose computed policy is fixed when deployed, MAPLE can exploit new experiences obtained during test time and can exhibit fast adaptation to the true environment. I found this research direction interesting and novel in the context of offline RL.\n- Still, I have some concerns when compared with meta RL, sim2real, domain randomization techniques. It would be great to provide some explanations of what makes MAPLE distinct from those approaches (e.g. applying meta RL whose task distribution was simply replaced with the uniform distribution over the ensemble of the learned dynamics model).\n- In the experiments, MAPLE is the only algorithm that uses RNN-based policy and exploits new samples obtained during deployment. Thus, it might be unsurprising that MAPLE outperforms the baseline algorithms. Then, for example, can VariBAD [1] (where the task distribution is defined by the ensemble of learned dynamics models) be comparable with MAPLE? If so, it would be great to see the comparison results.\n- Why the reward penalty $-\\lambda U(s,a)$ is required for MAPLE? What will be the result without such uncertainty penalty?\n- In Figure 2, if the transition after taking $\\beta$ is totally unknown, we need to consider infinitely many 'stochastic' transition models, rather than considering deterministic transitions only.\n- line 101: $p(\\tau | \\pi)$ -> $p(\\tau | \\pi, \\rho)$\n\n[1] Zintgraf et al., VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning, 2019\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations have been discussed in the paper.", "review_text": "", "extra_scores": {"ethical_concerns": "-", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "3lpS8FqymbG", "reviewer": "Reviewer_yAV4", "summary": "The authors propose a new offline model-based RL method MAPLE. The method relies on training an ensemble of dynamics with the offline data in addition to an environment-context extractor model that maps from dynamics to a latent embedding, which is part of the input to the policy. The authors claim such an environment-context extractor could both eliminate inconsistent dynamics and thus the remaining policy, after the probing stage, is the optimal one w.r.t. the ground truth dynamics during deployment. The experiments show that MAPLE achieves competitive results compared to state-of-the-art offline RL algorithms. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The development of the paper is well-established. Sec. 4.1 provides a good warm-start in the ideal situation, and the theory result gives good intuition on the tradeoffs and algorithm designs. Though there are still gaps between the transitions that could be fixed (please see below).\n\n2. The algorithm is clearly presented, along with the network architecture provided in the appendix. Seems easy to reproduce the results. \n\n3. The empirical results seem promising. The algorithm achieves competitive results compared to the most state-of-the-art offline reinforcement learning algorithms. \n\nWeakness:\n\n1. The definition of the optimal environment-context extractor $\\phi^{\\ast}$ (in line 194) seems quite strong. The current definition imposes a strong assumption that there exists a single policy that is optimal for all dynamics models. The assumption could be more  reasonable if it is $\\exists \\phi^{\\ast}, \\forall \\hat{\\rho} \\in \\mathcal{T}, \\exists \\pi_{\\phi^{\\ast}} \\in \\Pi, J_{\\hat{\\rho}}(\\pi_{\\phi^{\\ast}}) = max_{\\pi}  J_{\\hat{\\rho}}(\\pi)$. This is also because the paper seems to treat the policy and the environment-context extractor as separate classes, and it seems more realistic to assume a global optimal $\\phi$ but not policy $\\pi$ that is optimal for every dynamics model (as Eq.1 suggests, the best we can do is to seek for a policy with maximum expected returns on the distribution of the dynamics models).\n\n2. The introduction to the set of dynamics $\\mathcal{T}$ seems abrupt. At the first glance, it could be hard to follow why one suddenly needs to optimize the policy over a set of dynamics since the results in the previous section assume a convergence to a single policy/dynamics at the end. It could be better if an explanation is provided here for the dynamics model set (e.g., introduce stochasticity while all models are deterministic, or model ensemble, etc.).\n\n3. The reviewer is not fully convinced of the \"probing\" ability of the environment-context extractor. Across the algorithm there is actually no dynamics-related signal while training $\\phi$. As described in the paper, \"If the optimal actions in the same state are conflicting, to increase the performance of objective, Equation (1), the policy gradient has to backpropagate from $\\pi$ to $\\phi$\" (line 223-224). Since there is no guarantee that the learned dynamics could extrapolate accurately in out-of-support regime, training $\\phi$ purely based on the returns could also make $\\phi$ exploit the errors of the models. For example, even if the dynamics of the deployment environment is included in the dynamics set. If all dynamics are consistent up to timestep $i$ but there exists a dynamics that induces a larger expected return than deployment dynamics then the policy, together with $\\phi$, will more likely select the optimal action of that particular environment, other than the optimal action of the deployment environment. \n\nOriginality: \n\nThis work is an incremental work that improves on current model-based offline reinforcement learning algorithms that combine ensemble model learning and pessimism. \n\nClarity/Quality: \n\nMost of the paper is well written. Some of the technical details could be better explained (as suggested in the weakness section).\n\nSignificance: \n\nThe major contribution of the work is the empirical competitiveness of the proposed algorithm. The reviewer is not fully convinced by the technical contribution of the work (namely, the technical soundness of the environment-context extractor model).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This work does not seem to have a negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "9"}, "extra_text": {}}, {"review_id": "2JcJb6Hsq9", "reviewer": "Reviewer_yJkf", "summary": "Conservatism, i.e. avoiding out-of-distribution actions (and states), is a major theme in offline RL because of the need to avoid erroneous extrapolation beyond the data. This paper takes an alternative approach and instead trains a policy that adapts when deployed beyond the data distribution. To do this, a context encoder RNN is trained to produce latent codes given the episode history, and the encoder and policy are jointly optimized to maximize average performance across a large ensemble of pretained dynamics models. The algorithm, offline Model-based Adaptable Policy LEarning (MAPLE), is shown to achieve strong results on the MuJoCo tasks from the D4RL benchmark. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The algorithm is largely a combination of the ideas of MOPO and PEARL, but the combination is novel to my knowledge and well-motivated in the setting discussed. It is different than PEARL in that the meta-learning is happening in a model-based fashion across different dynamics models. However in that sense it is conceptually related to MB-MPO (https://arxiv.org/pdf/1809.05214.pdf), an algorithm which meta-learns a policy via MAML to quickly adapt to any model in the ensemble. I think MB-MPO should be cited and discussed in related work.\n\nThe theorem appears correct, and perhaps useful for high-level intuition regarding when/why the algorithm succeeds, but the result is not very informative in my opinion because all the heavy lifting is being done by the unknown coefficient $N_m$, for which no further analysis is presented.\n\nThe results on the D4RL MuJoCo tasks look good! One suggestion you may consider to further improve the paper is to experiment with tasks that require leaving the behavior distribution in order to succeed. This was one of the motivations of MOPO, and the authors developed two tasks specifically for this purpose (cheetah-jump and ant-angle).\n\nThe writing of the paper is clear and I have no significant issues with it.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "They discuss the limitations thoroughly (which is appreciated), but not the societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Most reviewers agreed that the paper has an interesting new way of performing offline rl (i.e., how to do well in out-of-support regions), and the empirical results on standard benchmark are very promising. 
Decision: Accept (Poster)