DROP: Conservative Model-based Optimization for Offline Reinforcement Learning
Authors: Jinxin Liu
Hongyin Zhang
Zifeng Zhuang
Yachen Kang
Donglin Wang
Bin Wang
Jianye HAO
Venue: Submitted to ICLR 2023 (2023)

Abstract: In this work, we decouple the iterative (bi-level) offline RL optimization from the offline training phase, forming a non-iterative bi-level learning paradigm that avoids the iterative error propagation over two levels. Specifically, this non-iterative paradigm allows us to conduct inner-level optimization in training (ie, employing policy/value regularization), while performing outer-level optimization in testing (ie, conducting policy inference). Naturally, such paradigm raises three core questions (that are not fully answered by prior non-iterative methods): (Q1) What information should we transfer from inner-level to outer-level? (Q2) What should we pay attention to when using the transferred information in outer-level optimization? (Q3) What are the benefits of concurrently conducting outer-level optimization during testing? Motivated by model-based optimization, we proposed DROP, which fully answered the above three questions. Particularly, in inner-level, DROP decomposes offline data into multiple subsets, and learns a score model (Q1). To keep safe exploitation to score model in outer-level, we explicitly learn a behavior embedding and introduce a conservative regularization (Q2). During testing, we show that DROP permits deployment adaptation, enabling an adaptive inference across states (Q3). Empirically, we evaluate DROP on various benchmarks, showing that DROP gains comparable or better performance compared to prior offline RL methods.

Under review as a conference paper at ICLR 2023
DROP: CONSERVATIVE MODEL-BASED OPTIMIZA- TION FOR OFFLINE REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, we decouple the iterative (bi-level) offline RL from the offline train- ing phase, forming non-iterative bi-level paradigm that avoids the iterative error propagation over two levels. Specifically, this non-iterative paradigm allows us conduct inner-level optimization in training (i.e., employing policy/value reg- ularization), while performing outer-level optimization in testing (i.e., conduct- ing policy inference). Naturally, such paradigm raises three core questions (that are not fully answered by prior non-iterative offline RL counterparts like reward- conditioned policy); (Q1) What information should we transfer from the inner- level to the outer-level? (Q2) What should we pay attention to when exploiting the transferred information for the outer-level optimization? (Q3) What are the bene- fits of concurrently conducting outer-level optimization during testing? Motivated by model-basee optimization (MBO), we proposed DROP, which fully answers the above three questions. Particularly, in the inner-level, DROP decomposes of- fline data into multiple subsets, and learns a MBO score model (A1). To keep safe exploitation the score model in the outer-level, we explicitly learn a behavior embedding and introduce a conservative regularization (A2). During testing, we show that DROP permits deployment adaptation, enabling an adaptive inference across states (A3). Empirically, we evaluate DROP on various tasks, showing that DROP gains comparable or better performance compared to prior methods.
INTRODUCTION
Offline reinforcement learning (RL) (Lagge et al. 2012 Levine et al. 2020) describes task of learning policy from previously collected static data. Due to the overestimation of values at out-of- distribution (OOD) state-actions, recent iterative offline RI. methods introduce various policy/value regularization to avoid deviating from the offline data distribution (or support) in the training phase. Then, these methods directly deploy the Iearned policy in online environment to test the perfor- mance. To unfold our following analysis, we term this kind of learning procedure as iterative bi-level offline RL [left,. wherein the inner-level optimization refers to trying to eliminate the OOD issue by constraining the policy/value function, the outer-level optimization refers to trying to learn better policy that will be employed at testing. Here, we the "iterative" term to emphasize that the inner-level and outer-level are iteratively optimized in the training phase. However, without enough inner-level optimization (OOD regularization), there is still a distribution shift between the behavior policy and the policy to be evaluated. Further, due to the iterative error exploitation and propagation (Brandfonbrener et al., 2021) over the two levels, performing such an iterative bi-level optimization completely in training often struggles to learn a stable policy/value function.
In this work, we thus advocate for non-iterative bi-level optimization (Figure D right) that decou- ples the bi-level optimization from the training phase, namely, performing inner-level optimization (eliminating OOD) in training and performing outer-level optimization (updating policy) in testing- Intuitively, incorporating the outer-level optimization into the testing phase can eliminate the itera- tive error propagation over the two levels. Then, three core questions! are: (Q1) What information ("E)") should we transfer from the inner-level to the outer-level? (Q2) What should we pay spe- cial attention to when we exploit "I"" for outer-level optimization? (Q3) Notice that the outer-level optimization and the online rollout test form a new loop (*)3") what new benefit does this give us?
Next will use A1, A2, and A3 to denote our answers the raised questions (Q1, Q2, and Q3) respectively.

---

Under review as a conference paper at ICLR 2023
Iterative bi-level offline RL optimization
Mon-iterativv bi-level offline RL optimization
Training phase
Testing phase
Training phase
Testing phase
Lenerr-levelee
Outer-level
?
St
(Online rollout
Inner-lever
' (a, (Online rollout)
13
Outer-level
Figure 1: A framework for bi-level offline RL optimization, where the inner-level optimization refers to regularizing the policy/value function (for OOD issues) and the outer-level refers to updating the policy (for reward maximizing). Non-iterative offline RI. decouples the joint optimization (of two levels) from the training phase, where "[77" transferred from the inner-level to the outer-level depends on the specific choice of algorithm used. In Table 1
we will summarize different choices for '[?'".
Intriguingly, prior works under such a non-iterative framework have proposed to transfer (as "(""" in Q1) filtered trajectories (Chen et al,, 202T), a reward-conditioned policy (Emmons et al. 2021 Kumar et al.] (019b), and the Q-value estimation of the behavior policy (Brandfonbrener et al., [022|Guucchre et al. 202I). all of which, however, partially address the aforementioned questions (we will elaborate these works in Table M. In this work, we propose new alternative method that transfers an embeddiing-conditooned (Q-value) score model and we will show that this method sufficiently answers the above questions and benefits most from the non-iterative framework.
Before introducing our method, we introduce a conceptually similar task (to the non-iterative bi- level optimization) offine model-based optimization (MBO, Trabucco al. 2021 which aims to discover, from static input-score pairs, a new design input that will lead to the highest score. Typically, offline MBO first learns a score model that maps the input to its score via supervised regression (corresponding to inner-level optimization), and then performs inference with the learned score model (as "(2r"), for instance, by optimizing the input against the learned score model via gra- dient ascent (corresponding to the outer-level). To enable this MBO implementation in offline RL, we are required to decompose an offline RL task into multiple sub-tasks, each of which thus corre- sponds to behavior policy-return (parameters-retun) pair. However, there are practical optimiza- tion difficulties when learning the score model (inner-level) and performing inference (outer-level) on high-diment -igh-dimensional policy's parameter space (as input for the score model). At inference, directly extrapolating the learned score model (<12²) also tends to drive the high-dimensional candidate pol- icy (parameters) towards out-of-distribution. invalid, and low-scoring parameters (Kumar & Levine, 2020). as these are falsely and over- opttiiitiitt scored by the learned score model.
To tackle these problems, suggest (A1) learning low-dimensional embeddings for these sub-taske tub-tasks decomposed in the MBO implementation, over which we estimate an embedding conditioned Q- value the MBO score model ('TP" in Q1), and (A2) introduce conservative regularization, which pushes down the predicted scores on OOD embeddings. so as to avoid over-optimistic exploitation and protect against producing unconfident embeddings when conducting outer-level optimization (policy/embedding inference). Mcanwhile, (A3) learning embedding permits deployment adapta- tion, which means we can dynamically adjust inferred embeddings across different states in testing (aka test-time adaptation). We name our method DROP (Design fROm Policies). Compared with standard offline MBO for parameter design (Trabucco et al. 2021), deployment adaptation in DROP leverages the MDP structure of RI. tasks, rather than simply conducting inference at the beginning of test rollout. Empirically, we demonstrate that DROP can effectively extrapolate a better policy that benefits from the non-iterative framework by answering the above three questions, and can achieve comparable or better performance compared to many prior offline RL algorithms.
2 PRELIMINARIES
2.1 REINFORCEMENT LEARNING AND OFFLINE REINFORCEMENT LEARNING
We model the interaction between agent and environment as Markov Decision Pro- cess (MDP) (Sutton& & Barto 2018), denoted by the tuple (S, A, R. P,) where S is the state space,
2 Pleasee note that this MBO is different from the regular model-based RI. (MBRI. for short), where the model MBO denotes a score model while that in MBRL deontes the transition dynamics (or reward) model.
2

---

Under review as a conference paper at ICLR 2023
is the transition kernel, R: A is the reward function, and pu : S + [0, 1] is the initial state distribution. Let क E == { : x A + [0, 1]] denotes a policy. In RL, aim to find a stationary policy that maximizes the expected discounted
A is the action space, P: S A [0,
return [x) == R 1Ei=0 7 R(ss, ar)] in the environment, where T = (So: an, Yo, $1, a1....), rt R(S,, az), is sample trajectory and y (0. is the discount factor. We also define the state-action value function Q" (s. a) == Er>x -t= ? A(s,, az][so S, ag = a]. which describes the expected discounted return starting from state s and action a and following π afterwards, and the state value function V*(s) (s) (als( [Q= as) To maximize JTr,, actor-critic algorithm alter- nates between policy evaluation and improvement. Specifically, given initial Q" and x², it iterates
gh+ (s, a) f- arg min B(E...))
(R(s, a) + +-Bar (ark)) 19 (s', a()] - ((s.a))) (I)
n * (als) arg max EgDDI amai(ass) (s.
where the value function (critic) Q(s, a) is updated by minimizing the mean squared Bellman error with experience replay dataset D and, following the deterministic policy gradient theorem (Sil- ver et al. 2014). the policy (actor) I ralss) is updated to maximize the estimated Q*+* (alss))
(s.
In offline RL (Levine et al. 2020), the agent is provided with static data D {r} which consists of trajectories collected by running some data-generating policies. Note that here we denote static offline data D. distinguishing from the experience replay D in online setting. Unlike the online RL problem, where the experience 'D+ in Equation | can be dynamically updated, the agent, in offline RL. is not allowed interact with the environment collect new experience data. As result, naively performing policy improvement as in Equation 2 may evaluate the estimated (*'s", a') on actions that lie far outside of the static offline data D, resulting in pathological values 1 a) that incur large error. Further, iterating policy evaluation and improvement will cause the learned policy k++ s) to be biased towards nat-of-distribution actions with erroneously overestimated values.
2.2 OFFLINE MODEL-BASED OPTIMIZATION
Model-based optimization (MBO) Trabucco et al. 2022) aims to find an optimal design input x* with a given score function f X y R. i.e., x' arg max (x)). Typically, we can repeatedly query the oracle score model f for new candidate design, until it produces the best design. However, we often do not have the oracle score function f*, but are provided with static offline dataset {(x, y}} of labeled input-score pairs. To track such offline MBO question, we can fit parametrie model ʃ to the oracle score function f via the empirical risk minimization (ERM), + arg min, Ex.y [J(x) - v)² Then, starting from the best point in the dataset, we can perform gradient ascent on the design input and set the learned optimal design x* = x := GradAscent, /oannnt((x) (x3. K) (for simplicity, next we will omit subscript in GradAscenty), ) where
x X +
for h 0, K-1.
Since the aim is to find a better design input beyond all the designs in the dataset and while directly optimizing score model f with ERM can not ensure new candidates nut-of-distribution design in- puts) receive correct scores, thus one crucial requirement is to conduct confident extrapolation.
3 DROP: DESIGN FROM POLICIES
We present our framework in Figure 2 In Sections 3.1 and 3.2 we will answer questions Q1 and Q2, setting a learned MBO score model "Mi' (A1) and introducing conservative regularization over the score model (A2). In Section 3.3 we will answer Q3, where we show that we can conduct outer-level optimization during testing, enabling an adaptive embedding inference across states (A3).
3.1
TASK DECOMPOSITION
Our core idea is to explore MBO in the non-iterative bi-level offline RL framework (Figure riihtt, while capturing the structural characteristics of RL tasks and answering the raised questions (Q1, Q2, and Q3) in the introduction section. To begin with, we first decompose the offline data D into N offline subsets D|v] == (Di,...; Dx... In other words, decompose offline task, learning
3

---

Under review as a conference paper at ICLR 2023
training phase (inner-level (
testing phase
D
21
Zz
outer- level
(<2|n)
23
Z4
(((2)
B(a
a(),
B(ass,2)
offline data (task)
sub-tasks
task embedding & conservative MBO
deployment
adaptation
Figure 2: Overview of DROP. Given static offline dataset D, we decompose the data into N (= in diagram) subsets {D, In ...., N... over which we learn task embedding o(z (n) and con- duct MBO by learning multiple behavior policies (modeled by contextual policy) ([a|s, z) and score model (s, a, z). During testing, at state S, we can adapt the optimal policy (contextual variable/embeedding) with x= (a(ass) 3 (als, 2" (s)), where z²(s (s) arg max, [s, ((ass, 2), z).
with D, into multiple offline sub-tasks, learning with D, E DiN] respectively. Then, for sub-task n E [1, Nj. we can perform behavior cloning (BC) to parametric behavior policy Bn. S x A [0, 1] to model the corresponding subset D,;
B. + arg max ((s.a)),, log (a(ss))
N.
8.
Such a decomposition also comes with an additional benefit that it provides an avenue to exploit the hybrid modes in offline data D, because that D is often collected using hybrid data-generating behavior policies (Fu et al.] 2020). which suggests that fitting a single behavior policy may not be optimal model the multiple modes of the offline data distribution (see Appendix C. for the empirical evidence). Thus, to encourage the emergence of diverse sub-tasks which capture distinct behavior modes in D (this is not our focus in this work"). we simply perform task decomposition according to the returns of trajectories in D, heuristically ensuring that trajectories in the same sub- task share similar returns and trajectories from different sub-tasks have distinct returns.
3.2 TASK EMBEDDING AND CONSERVATIVE MODEL BASED OPTIMIZATTION
Naive model-based optimization (MBO) over behavior policies. Benefiting from the above of- fline task decomposition, we can conduct MBO over a set of input-score (x, y) pairs, where we model the (parameters of) behavior policies 8n as the design inputs x and the corresponding pected returns at initial state J(33)) as scores Note that, ideally, evaluating behavior policy 3n. i.e. calculating J(,,) = Es [ (so)]. with subset D, will never trigger the overestimation of values in the inner-level optimization. By introducing score model II + R (as the transferred information " in Q1), we can then perform luter-level policy inference with
π. (a|s) < arg max S (T), where f arg min E., [(((8.) ((Bh))
(5)
x
However, directly performing optimization-based inference (outer-level optimization), max, (TT), will quickly find an invalid input for which the learned score model f outputs erroncously large values (Q2). Furthermore, parameters of policies, accounting for the fact that the parameters of input behavior policies lie on narrow manifold in a high-dimensional parametric space (Kumar & Levine! 2020).
is particularly severe we perform the inference directly over the
Task embedding. To enable feasible policy inference, we propose decouple the MBO techniques from the high-dimensional space of policy parameters. We achieve this by learning a latent embed- ding space Z with information bottleneck (dim(Z) < min(N, dim (II))) conditioned on the sub-task n, from which the high-dimensional parameters of behavior policies can be inferred.
We compare three different decomposition rules in Appendix C.2 Further, in Appendix C.4 adopt
CVAE conduct automatic task decomposition (treating each trajectory individual task).
4

---

Under review as a conference paper at ICLR 2023
We can thus the embedding 2 to represent sub-tasks (or the corresponding behavior poli- cies). Formally, we learn a task embedding Φ I x 2 + [0, 1] and a contextual behavior policy B 2 A - [0, 11. which replaces N separate behavior policies in Equation 4
B(a|s, z), (z)n) + arg max (Pn llo ((als, ((z/n))]
(6)
3.0
Conservative model-based optimization. In principle, by substituting the learned task embed- ding (( (2) and the contextual behavior policy ((ass, x) into the original objective in Equa- tion we can then conduct MBO over the embedding space: learning f 2 - IR with J(3,)) and setting the optimal embedding z" = arg max, f() and the corresponding policy == =aaas)) = f(als, where x" can be inferred with gradient ascent as in Equation 3 However, we must deliberate a new distribution shift in the Z-space, stemming from the original distribution shift in the parametric space when directly optimizing Equation
miny E d(tii))
Motivated by the energy model (LeCun et al., 2006) and the conservative regularization (CQL, Ku- mar et al. (2020)), we introduce the conservative score model learning, additionally regularizing the scores of out-of- distrribbbttoon embeddings (z))::
f c arg min (x()())))) (f(z) J s.t. Ex(x)) [f(z)] - En,0(x/n) [f(z) < n.
(7)
/
Intuitively, as long as the scores of out-of-distrrbution embeddings Bu(()) [f(z] is lower than that of in-distribution embeddings En,o(s)a) [f(z]] (up to a threshold 77), conducting embedding infer- ence with x² = arg max, ((z) would produce the best and confident solution, avoiding towards embeddings that are far away from the training set {Ø(z)n), n .V..
Now that we have reframed the non-iterative bi-level offline RL problem as one of offline MBO: in the inner-level optimization (Q1), we set the practical choice for "("" as the learned score model if (A1); in the outer-level optimization (Q2), we introduce task embedding and conservative regular- ization to avoid over-optimistic exploitation when exploiting f for policy/embedding inference (A2). In next section, we will show how to slightly change the form of the score model ʃ. so as to leverage the (MDP) structural characteristic (loop "127") of RL. tasks and answer the left Q3.
3.3 DEPLOYMENT ADAPTATION
Recalling that we update fz) to regress the value at initial state Es, [V in Equation 7 we then conduet outer-level inference with z" arg max, f(z) and rollout the 2"-conditioned policy T" Taasss) == B(ass, z²) until the end of rollout episode at deployment (testing). In essence, this inference produces an extrapolation over the distribution of the behavior policies (corresponding to embeddings). Going beyond the (outer-level) inference only at the initial state, we propose that a implementation can benefit by performing inference at any rollout state in testing (A3).
To enable deployment adaptation, we model the score model with f: S x Ax Z + R. taking a state- action as extra input. Then, we encourage the score model regress the values of behavior policies
(f(s, a, z) Q (s, =. For simplicity, instead of learning additional value function Qo, for each behavior policy, learn the score model directly with the TD-error used for learning the value function Q (s, as together with the conservative regularization in Equation 7
over all state-action pairs in each sub-task, min, Enn.ofann
in Equation [
f f- arg min ED, -DM E (
(R(ss,a) a) + + a', ((2lm) - [ a, (l ((ll)))) (8)
s.t. En, Ea-D, Ea ((jss2)))) (s,) - En,oee/n) BB (f a, 2)] S 7
where f denotes a target network and we update the target f with soft updates: f - (1 (1-v)) v)f uf.
In testing, we thus can dynamically adapt the outer-level optimization, setting policy inference with (a|s)= = [(ass x² (s)), where 2" z*(s) = arg max, [s, ([a)s, z), z). Specifically, at any state in the deployment phase, we perform gradient ascent to find the optimal behavior embedding 2 (s) z (s) == Grad 2q; K), where 2i is the starting point and (s) t (s) 2f (s. ((a z), z=zz; for k = 0, -.1.
(
WWe feed the one-hot encoding of the sub-task specification (n 1,.... N) into the emhedding network Φ.
5

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison of five non -on-iterative bi-level offline algorithms, where R(-) denotes the return of sampling 7 or starting from (s, a), the checkmark A2 indicates whether the exploitation (outer- level) to "0" is regularized and that in A3 indicates whether deployment adaptation is supported.
Inner-level
Outer-level
"I"" in A1
A3
F-BC
filter r with high R(T)
behavior cloning
filtered {r} a(a)s,))
RvS-R
min, -I [log (als, R(T))]
handcraft Riarget
x
x
Onestep
ming L(Q(s, a), R(s, a))
arg max a (s. B(ass))
@g(s.
COMs
miny [[f(3,), R(T))
arg max, f(3)
f(3) f(s, a,
x
DROP
miny [(f(s, R(s, a, 2))
arg max, (s, (alz))
3.4
CONNECTION PRIOR NON-ITERATIVE OFFLINE COUNTERPARTS
In Table we summarize the comparison with prior representative non-iterative offline RL methods. Intuitively, our DROP (leveraging returns to decompose D) is similar in spirit to F-BC and RvS-R (Chen al. 2021 Emmons et al., 2021). J. both of which use return R(T) to guide the inner-level optimization. However, both F-BC and RvS-R leave Q2 unanswered. In outer-level, F-BC can not enable policy extrapolation, which heavily relies on the data quality offline tasks, and RvS-R needs to handeraft target return (as the contextual variable for (a which also probably triggers the potential distribution shift between the hand-crafted contextual variable and that used for learning the contextual policy (see examples in Figure 6 of Emmons et al. (201))
Diving deeper into the bi-level optimization, we can also find DROP combines the advantages of Onestep (Brandfonbrener al., (202]) and COMs (Trabucco et al. (02I), where Onestep per- forms outer-level optimization in action space (arg max,), ). COMs performs that in parameter space (arg maxx). while our DROP performs that in embedding space (arg max,). )) As result, the choice of (s, a, z) in DROP allows us conduct safe exploitation over "12"" in outer-level (Q2) and lever- age the structural characteristic of RL task (the loop "0"""" in Q3), rather than simply conducting outer-level optimization at initial states as in COMs (corresponding to the objective in Equation (5).
3.5 PRACTICAL IMPLEMENTATION
Algorithm DROP (Training)
Algorithm DROP (Testing / Deployment)
Require: Dataset of trajectories, D = }T}.
Require: Env, Balss, 2). and f(s, 2)
1:
Initialize d(z/n), ([ass, 2). and [(s a, 2).
1: EEE..Reeert
Decompose into N sub-sets D[N]-
2: while not done do
while not converged do
3: Inference (deployment adaptation):
4: Sample a sub-task: Dn ~ DiN]-
z (s:) = arg max f(s,, B(ae|se: 2). z).
5: Learn é, B, and f with a
4: Sample action: a, ~ 8[a,)(())))))) z" (s:)).
6: end while
Step Env: 81+1 P(s St a.).
Return: ((a|s, z) and f(s, a, 2).
6: end while
We now summarize the DROP algorithm (see Algorithm for the training phase and Algorithm 2 for the testing phase). During training (inner-level optimization), we alternate between updating d(z(n), B(a)s, z). and f(s, a, z), wherein we update & with both maximum likelihood loss and TD-error loss in Equations 6 and g During testing (outer-level optimization), for each state s, we use the gradient ascent in Equation 9 to choose the optimal embedding 2 2s)). Instead of simply sampling single starting point zi. we choose N starting points corresponding all the embeddings ..... N) of sub-tasks, and then choose the optimal z (s) from those updated embeddings for which the learned outputs the highest score: x (*(s) = arg max, f(s, ((ass,)) 2) s.t. a E { [GradAscent(s,, 2n, K)ln 1,...,N] N). Then, sample action from x² (als) [[aas, z (s)). For more optimization (trainng//essinng) details, we refer the reader to Appendix
(z., 1, n
4 RELATED WORK
Offline RL. In offline RL, learning with static offline data is prone to exploiting out-of-distribuu (OOD) state-action pairs and producing over-estimation of values, which makes vanilla iterative

---

Under review as a conference paper at ICLR 2023
policy learning and value optimization challenging (Rashidinejad et al. 2021 To eliminate the
problem, a number of methods have been explored, in essence, by either introducing a policy'value regularization in the iterative loop or trying to eliminate the iterative procedure itself.
Iterative methods: Sticking with the normal iterative updates in RL, offline policy regularization
methods aim to keep the learning policy be close to the behavior policy under probabilistic
distance (Cang et al. [021 Fujimoto& & Gu [021 Kostrikov et al.) 2021a; Kumar et al. 2019a]
Liu et al. 2022; Nair et al. 2020a Peng et al. 2019 Siegel et al. [020]] Wu al.] 2019 Zhang
et al. (021). Some works also conduct implicit policy regularization with variants of importance
sampling Lee et al. (202T); Liu et al. (2019); Nachum et al. (2019). Besides regularizing policy,
is also feasible to constrain the substitute value function in the iterative loop. Methods constraining
the value function aim at mitigating the over-estimation, which typically introduces pessimism to
the prediction of the Q-values (Chebotar et al., [021] Jin et al. [02T] Kumar et al.. 2020; Li et al.,
2022 Ma et al. 2021a b) or penalizes the value with an uncertainty quantification (An et al. 2021
Bai et al.) 2022 Rezacifar et al. 2021 Wu et al. 2021), making the value for
state-actions more conservative, Similarly, another branch of model-based methods (Kidambi et al. 2020 Yu et al. 2020 2021b Rigter et al., 2022) also perform iterative bi-level updates, alternating
between regularized evaluation and improvement. Different from these works, DROP only evaluates values of behavior policies in the inner-level, avoiding error propagation between two levels.
Non-iterative methods: Another complementary line of work studies how to eliminate the itera- tive updates, which simply casts RL as a weighted conditional imitation learning problem (Q1)- Derived from the behavior-regullrizztin RL (Geist et al. 2019 Vieillard et al.
2020), the former conducts weighted behavior cloning: first learn a value function for the behavior policy, then weigh the state-action pairs with the learned values or advantages (Abdolmaleki et al. 2018 Chen et al.) 2019 Wang et al., 2020; Peng et al., (2019). Besides, some works also propose implicitly behav- ior policy regularization that also avoids estimating the value of new candidate policies, initializing the learning policy with a behavior policy |Matsushima et al. (2020) or performing only a "one- step" update (policy improvement) over the behavior policy (Fujimoto et al. 2019 Gulcehre et al.) [222]. For the latter, this branch method typically builds upon the hindsight information match- ing (Andrychowicz et al. 2017]] Eysenbach et al. 2020 Pong et al. 2018 Wan al. (021), assum- ing that the future trajectory information can be useful to infer the middle decision that leads to the future and thus relabeling the trajectory with the reached states returns. Due to the simplicity and stability, RvS byS-based methods thus advocate for learning a goal-conditioned or reward- swward-conditioned
policy (Chen al. 2021; Ding et al./ 2019 Emmons al. [2021 Furuta et al., 2021 Janner al.,
2021 Lin et al.] 2022] Srivastava et al. 2019 Yang et al. (002)) with supervised learning. However,
these works do not fully exploit the non-iterative bi-level framework and fail to answer the proposed
questions, which either does not regularize the inner-level optimization before exploiting "["" in the
outer-level (Q2), or does not support the deployment adaptation in testing (Q3).
Offline model-based optimization (MBO). Similar to offline RL, the main challenge of MBO is
to reason about uncertainty and OOD values (Brookes et al.) 2019 Fannjiang & Listgarten 2020).
since direct gradient-ascent against the learned score model can easily produce invalid inputs that are falsely and highly scored. To counteract the effect of model exploitation, prior works introduce
various techniques, including normalized maximum likelihood estimation (Fu & Levine 2021),
model inversion networks (Kumar & Levine 2020). local smoothness prior (Yu et al. 202 and conservative objective models (COMs) (Trabucco et al. 2021). Compared to COMs, DROP shares similarity with the conservative model, but instantiates on the embedding space instead of the pa- rameter space. Such difference is nontrivial, not only because DROP allows OOD sampling (aimed at pessimism) directly in embedding space, avoiding an adversarial training as in COMs, but also because DROP allows deployment adaptation, enabling dynamical inference across states in testing.
5 EXPERIMENTS
In this section, we present our empirical results. We first give examples to illustrate the deployment adaptation. Then we evaluate DROP against prior offline RL algorithms on D4RL benchmark. Finally, we compare DROP with prior (offline) latent-based baselines. For more ablation studies wrt to the decomposition rules and the conservative regularization, we refer readers to the appendix.
Illustration of deployment adaptation. To better understand the deployment adaptation of DROP, include four comparisons that exhibit different embedding inference rules at testing:
7

---

Under review as a conference paper at ICLR 2023
Grad. Ascent
20N1
(5a)
z" (5c)
(s))
z 2s())
Normalieed retum
100
0.75
0.75
0.75
0.75
5-
0.50
0.50
0.50
0.50
1-0
1-50
50
0.25
0.25
100
0.25
0.25
0.00
0.00
0.00
17
0.00
0.0
(a)
0.8
0.0
(b)
0.8
0.0
(c1)
0.8
0.0
(c2)
0.8
(d)
Figure 3: Visualization of the embedding inference (a, b, cl, c2) and performance comparison (d). i|n] denotes embeddings of all behavior policies; zn (so). z (Sa), (5()) and x (s:) denote the sele- cted embeddings in DROP-Best, DROP-Grad, DROP- Best-Ada and DROP- DROP-Grad-AA Grad-Ada respectively.
(1) DROP-Best: At initial state So. we choose the best embedding from those embeddings of behav- ior policies, 25 arg max; [So; (( (So z), z) s.t. s E Z[N] {-1,.... ..... Z, ), and keep this embedding fixed for the entire episode, i.e., setting π (a. |s:) = B a (s, (so)).
(2) DROP-Grad: At initial state So, we conduct inference (gradient ascent on starting point z (8o) with x (So) arg max [So ((au|so. 2), z), and keep this embedding fixed throughout the rollout.
(3) PRoP-Best-Ada: We adapt the contextual policy by setting π (ay (sr) = ( |s,, z (sx)), where choose the best embedding (St) directly from those embeddings of behavior policies for which the score model outputs the highest score, i.e., (5²) = arg max, f fss, ((a, [st. z). 2) s.t. z E [-)-
(4) DROP-Grad-Ade Grad-Ada (gradient- brradeeet-based adaptation as described in Section 3.5
We set x (a,(ss, 3( (Se, zz (s:)) and choose the best embedding from those updated embeddlnggs of behavior poli- cies, i.e. 2" (St) = arg max, [S,,, 3(a (St, 2), z) s.t. 2 E [GradAscent/ Zn, K)|n = I,..., N).
In Figure 3 we visualize the four different inference rules and report the corresponding performance in the seatheetah-meddung medium-expert task {Fu et al. 2020). In Figure 3 (a), we set the starting point as the best embedding z (5o) in ZiN], and perform gradient ascent to find the optimal z (so) for DROP-Grad. In Figure (b), we can find that at different time steps, DROP-Best-Ade chooses different embeddings (as contextual variables for [s, -)) At high level, performing such dynamical inference enables us to combine different embeddings, switching behavior policies at dif- ferent states. Further, in Figure 3 (c1, c2), we find that the additional inference (with gradient ascent) in DROP-Grra-AAe allows to extrapolate beyond the embeddings of behavior policies, and thus re- sults in sequential composition of new embeddings (policies) across different states. For practical impacts of these different inference rules, we provide the performance comparison in Figure B (d), where we can find that performing gradient-bant optimization (* (*-Grad-9 outperforms the natural selection among these embeddings of behavior policies in sub-tasks (*- M--Best-)), and rollout with adaptive embedding inference (DROP-*AAda outperforms that with fixed embeddings (DROP-*).
Empirical performance on benchmark tasks. We evaluate DROP on a number of tasks from the D4RL dataset and make comparison with both prior iterative and non-iterative offline algorithms [
Considering that DROP follows the non- non-iterative offline RL paradigm, we compare DROP with prior non-iterative offline baselines (BC, F-BC, DT (Chen et al. 2021). RyS-R. Onestep, and COMs) in the main paper. Compared with our DROP, these basclines do not fully answer the raised questions (see Table D. which either does not regularize the inner-level optimization before exploiting "12"' in outer-level (Q2), or does not support the deployment adaptation in testing (Q3). Moreover, we also provide the comparison with CQL, which inspires us to design the conservation in Equation In Table 2 we show the evaluation results for AntMaze- and Gym-* Gymm-*medduum-- * tasks in D4RL vv2 where we can find DROP (-Grad-Ada) achieves better performance than these non-iterative offline RL baselines overall. Compared with CQL, DROP shows superior performance in AntMaze-large- and Syym---medium-exper (m.-xp..) tasks, while leads inferior performance in AntMaze-medium and Gyym--mediuu-reehh mentation (see motivation in next paragraph and details in Appendix C.4 which further improves DROP's performance and retains superior/comparable performance in all tasks.
(m.-rep.) tasks. As an extension, we also design DROP+CVAE imple-
Comparison with latent policy methods. Note that one additional merit of DROP is that it nat- urally accounts for hybrid modes in D by conducting task decomposition in inner-level, we thus
'Due page limit, we mainly provide the comparison with prior iterative baselines in Appendix
8

---

Under review as a conference paper at ICLR 2023
Table Comparison with non-iterative methods on D4RL (*-v2). For all results of our method, we
average the normalized returns across 5 seeds; for each seed, we run 10 evaluation episodes. For comparison, we use and denote DROP- Grad-Ada achieves better performance compared with Onestep and COMs (most related baselines in Table I respectively, (BA: Best-Ada, GA: Grad-Ada)
DROP
DVOP&CVAE
Tasks
BC
F-BC
DT
RvS-R
Onestep
COMs
CQL
BA
GA
GA
unnaze
54.6
60
65.6
64.4
64.3
63.3
74
78
80
90.5 +: 2.4
umaze-divvrse
46.5
70.1
46.7
84
62
666
85.1 +
mediurn-play
0
42.1
4.5
0.3
40
61.2
34
30
68.2 16.5
medium-diverss
0
37.2
0.6
7.7
0
26.7
53.7
24
30
75.4 + 9.4
lurge-play
0
28
0
3.5
0
33.3
15.8
36
42
50.1 = 13.6
large-diverse
34.3
0.2
3.7
10
14.9
20
26A
52.2
12.0
I. IL-ex.. m.-rpp
walker2d
62.5
66.6
60.6
66.4
33.9
86.1
60.9
61.9
83.5
hopper
18.1
75.9
82.7
73.5
77.3
49.7
97.8
83,4
87.4
96.3 2.5
halfcheetahh
36.6
40.6
36.6
38
38.4
41.4
47.3
40.4
4
50.9 - 1.6
walker2d
107.5
109
108.1
106
111.8
93.5
109.5
106.8
106.9
109.3 = 0.4
hopper
52.5
110.9
107.6
101.7
81.4
109
102
102.5
105.9A
107.2 = 1.7
halfcheetah
55.2
92.9
86.8
92.2
77
76
85.8
88.5
88.9
102.2 1.5
total
396.1
739.9
607.0
625.9
577.6
623.9
832.
736.5
765.3
970.9
compare DROP to latent policy methods (PLAS (Zhou et al. 2020) and LAPO (Chen et al. 2022 that use con- ditional variational autoencoder (CVAE) model offline data and also account for multi-modes in offline data. Es- sentially, both our DROP and baselines (PLAS and LAPO)
Table 3: Comparison on D4RL *-v2.
PLAS
DROP
DROP
umaze
70.7
umaze-diverse
45.3
animaze
medium-play
16
34
30
learn a latent policy in the inner-level optimization, ex- cept that we adopt the non-iterative bi-level learning while baselines are instantiated under the iterative paradigm. By answering Q3, DROP permits deployment adaptation, enabling to dynamically switch/stitch "skills" (latent- policy/behaviors as shown in Figure 3 and encouraging high-level abstract exploration in testing. However, the
meidum-diverse
0.7
24
30
large-play
0.7
36
42
large-diverse
0.3
LAPO
DROP
DROP+
medium
walker2d
80.8
79.1
82.1
hopper
51.6
59.5
61.5
halfcheetah
46
43.1
52.4
aim of introducing the latent policy in PLAS and LAPO
random
walker2d
3
5.2
hopper
23.5
5.5
20.8
to regularize the inner-level optimization, which fairly answers Q2 in the iterative offline counterpart but can not
halfchectah 30.6
2.3
32.0
provide the potential benefit (deployment adaptation) by answering Q3 in the non-iterative paradigm.
We provide the comparison results in Table We can observe that DROP (-Best-Ada) and DROP (-Grad-Ada) consistently achieves better performance than PLAS on AntMaze- tasks. On the Gym *-medium domain, DROP (-Grad-Ada) also performs better than LAPO. However, there is big performance gap between DROP and LAPO on the *-random domain. We speculate that is mainly caused by the decomposition rule. In our DROP implementation, we heuristically use return to conduct task decomposition (motivated by RyS-R (Emmons et al. (02I)), while LAPO and PLAS conduct decomposition (learning latent policy) automatically. Similarly, to bridge the gap. also adopt CVAE to model the offline data and afterwards take the learned latent embedding in CVAE as the embedding of behaviors, instead of conducting return-guided task decomposition. We provide implementation details (DROP+CVAE) in Appendix/C.4faam new results in Tables@aand where we can see such CVAe-based DROP implementation can bring substantial performance improvement. Further, in Table 6 (Appendix C.4), we compare DROP+CVAE to IQL (Kostrikov) et al. 2021b). consistently demonstrating the competitive empirical performance DROP approach against state-of-art offline iterative/non-iter -iteratiiv baselines.
CONCLUSION
In this work, we introduce non-iterative bi-level offline RL and based on this paradigm, raise three questions (Q1, Q2, and Q3). To answer that, we reframe the offline RL problem as one of MBO and learn score model (A1), introduce embedding learning and conservative regularization (A2), and propose deployment adaptation in testing (A3). We evaluate DROP on various tasks, showing that DROP gains comparable or better performance compared to prior methods.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Abhas Abdolmaleki, Jost Tobias Springenherg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum posteriori policy optimisation. arXiv preprint arXiv:1806.06%% 2018.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline re- inforcement learning with diversified q-ensemble. Advances Neural Information Processing Systems, 34, 2021.
Marcin Andrychowicz, Filip Wolski, Alex Ray. Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience re- play. Advances in neural information processing systems, 30, 2017.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-drivee offline reinforcement learning. arXiv preprint arXiv:2202.11566 2022.
David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off- policy evaluation. Advances in Neural Information Processing Systems, 34:4933 4946, 2021.
David Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. In International conference machine learning, pp. 773-782. PMLR, 2019.
Catherine Cang, Aravind Rajeswaran, Pieter Abbeel, and Michael Laskin. Behavioral priors and dynamics models: Improving performance and domain transfer offline rl. arXiv preprint arXiv:2106.091 19, 2021.
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, et al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint arXiv:2104.07744 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34, 2021.
Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Yuan Gao, Jianhao Wang, Wenzhe Li, Bin Liang, Chelsea Finn, and Chongjie Zhang. varialle advantage-weighted policy optimization for offline rl. arXiv preprint arXiv:2203.088 089499, 2022.
Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best-action imitation learning for batch deep reinforcement learning. arXiv preprint arXiv:1910.121799 2019.
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. Advances in neural information processing systems, 32, 2019.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline via supervised learning? arXiv preprint arXiv:2112..10551 2021.
Ben Eysenbach. Xinyang Geng, Sergey Levine, and Russ R Salakhutdinov. Rewriting history with inverse rl: Hindsight inference for policy improvement. Advances in neural information process- ing systems, 33:1478: 13:14483-14799 2020.
Clara Fannjiang and Jennifer Listgarten. Autofocused oracles for model-based design. Advances in Neural Information Processing Systems, 13:12945-12256, 2020.
Justin Fu and Sergey Levine. Offline model- optimization via normalized maximum likelihood estimation. arXiv preprint arXiv:2102.07970,, 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep
data-driven reinforcement leaming. CoRR, abs/2004.07219, 2020. URL https://arxiv.
brg/abs/ 2000 07219
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
10

---

Under review as a conference paper at ICLR 2023
Scott Fujimoto, David Meger, and Doina Precup. Off Off--oliiy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-20622 PMLR, 2019.
Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for offline hindsight information matching. arXiv preprint arXi::21 2021.
Matthieu Geist, Bruno Scherrer, and Olivier Pictquin. A theory of regularized markov decision processes. Kamalika Chaudhuri and Ruslan Salakhutdinov (cds.), Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Cali- fornia, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2160-2169. PMLR, 2019.
Caglar Gulcehre, Sergio Gómez Colmenarejo, Ziyu Wang, Jakub Sygnowski, Thomas Paine, Konrad Zolna, Yutian Chen, Matthew Hoffman, Razvan Pascanu, and Nando de Freitas. Regularized behavior value estimation. arXiv preprint ar Xiv::103300957. 2021.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as big sequence modeling problem. Advances in neural information processing systems, 34, 2021.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline ri? In International Conference Machine Learning, pp. 5084-5096. PMLR, 2021.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 21810-21823, 2020.
lya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofrr Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5783. PMLR, 2021a.
Nya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint 10.0::6 2021b.
Aviral Kumar and Sergey Levine. Model inversion networks for model-based optimization. Ad- vances in Neural Information Processing Systems,
2020.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- off-policy q- learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019a.
Aviral Kumar, Xue Bin Peng, and Sergey Levine. Reward-conditioned policies. arXiv preprint 191i: 134655, 2019b.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-1191, 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforce- ment learning, pp. 45-73. Springer, 2012.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial energy-based learning. Predicting structured data, (0), 2006.
Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, pp. 6120-6130. PMIR, 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643. 2020.
Jinning Li, Chen Tang, Masayoshi Tomizuka, and Wei Zhan. Dealing with the unknown: Pessimistic offline reinforcement learning. In Conference on Robot Learning, pp. 1455-1464. PMLR, 2022.
Qinjie Lin, Han Liu, and Biswa Sengupta. Switch trajectory transformer with distributional value approximation for multi-task reinforcement leaming. arXiv preprint arXiv:2203.074 03311,, 2022.

---

Under review as a conference paper at ICLR 2023
Jinxin Liu, Hongyin Zhang, and Donglin Wang. Dara: Dynamics-awarc reward augmentation in offline reinforcement learning. arXiv preprint arXiv:2203.0666 2022.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off- Off-policy policy gradient with state distribution correction. arXiv preprint S3dv::900408473. 2019.
Xiaoteng Ma, Yiqin Yang. Hao Hu, Qihan Liu, Jun Yang, Chongjie Zhang, Qianchuan Zhao, and Bin Liang. Offline reinforcement learning with value-based episodic memory. arXiv preprint arXiv:21 10.09790. 2021a.
Yecheng Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative offline distributional reinforce- ment learning. Advances in Neural Information Processing Systems, 34, 2021b.
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. arXiv preprint (rXXi::006.03647, 2020.
Ofir Nachum, Bo Dai, Πya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv: arXiv:1912.088: 1912.0207, 2019.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint 200352000600335, 2020a.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint urXiv:2006.0935) 2020b.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00117,,0 2019.
Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine Temporal difference models: Model- free deep rl for model-based control. arXiv preprint arXIv:18802.0908) 2018.
Dushyant Rao, Fereshtch Sadeghi, Leonard Hasenclever, Markus Wulfmeier, Martina Zambelli, Giulia Vezzani, Dhruva Tirumala, Yusuf Aytar, Josh Merel, Nicolas Heess, et al. Learning trans- ferable motor skills with hierarchical latent mixture policies. arXiv preprint 12.00221 2021.
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline rein- forcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 34, 2021.
Shideh Rezaeifar, Robert Dadashi, Nino Vieillard, Léonard Hussenot, Olivier Bachem, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning as anti-exploration. arXiv preprint arXiv:2/06.0643./ 2021.
Mare Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. arXiv preprint arXiv:2204. 12052, 2022.
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp. Abbas Abdolmaleki, Michael Ne- unert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08336, 2020.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pp- 387-395. PMLR, 2014.
Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojeiech Jaskowski, and Jürgen Schmidhu- ber. Training agents using upside-down reinforcement learning. arXiv preprint arXiv: arXiv:19/2.0289 1912.02877, 2019.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
12

---

Under review as a conference paper at ICLR 2023
Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine. Conservative objective mod- els for effective offline model-based optimization. In International Conference on Machine Learn- ing. pp. 10358-100000 10368. PMLR, 2021.
Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Design-bench: Benchmarks for data-driven offline model-based optimization. arXiv preprint arXiv:2202.08455 2022.
Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, and Matthieu Geist. Leverage the average: an analysis of KL regularization in reinforcement learning. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-FFlrina Balcan, and Hsuan-Tien Lin (eds.), Advances Neural Information Processing Systems 33: Annual Conference on Neu- ral Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Michael Wan, Jian Peng, and Tanmay Gangwani. Hindsight foresight relabeling for meta- reinforcement learning. arXiv preprint arXiv:2109 09031, 2021.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, et Critic regularized regression. arXiv preprint arXiv:2006. 15134 2020.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1ww.I.v:911.... 911111111,,1,,111111111111111111 2019.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08 2021.
Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking goal-conditioned supervised learning and its connection to offline rl. arXiv preprint arXiv:2202.04473 2022.
Sihyun Yu, Sungsoo Ahn, Le Song, and Jinwoo Shin. Roma: Robust model adaptation for offline model-based optimization. Advances in Neural Information Processing Systems, 34, 2021a.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offiin policy optimization. Advances in Neural Information Processing Systems, 33:14129-144 14142,, 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954-28966 2021b.
Chi Zhang, Sanmukh Kuppannagari, and Prasanna Viktor. Brac+: Improved behavior regularized actor critic for offline reinforcement learning. In Asian Conference on Machine Learning, pp- 204-219. PMLR, 2021.
Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforce- ment learning. arXiv preprint mrXiv:2011.072/33 2020.
13

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
Here we provide the content of the appendix:
Discussion and future work
More experiments
C.1
Single behavior policy vs. multiple behavior policies
Decomposition rules
C
Online fine-tuning
C44
DROP CVAE
c
Ablation study
Implementation details
Additional results
B DISCUSSION AND FUTURE WORK
Limitations. DROP also has several limitations. First, the offline data decomposition dominates the following bi-level optimization, and thus choosing suitable decomposition rule is crucial require- ment for policy inference (see experimental analysis in Appendix C..2) An exciting direction for fu- ture work is to study generalized task decomposition rules (Rao et al. 202T). In Appendix C.4 also exhibit a potential of such generalized task decomposition by introducing CVAE into DROP's implementation, and find such combination (DROP CVAE) can bring practical performance im- provement. Second, we find that when the number of sub-tasks is too large, the inference is unstable, where adjacent checkpoint models exhibit larger variance in performance (such instability also ex- ists in prior offline RL methods, discovered by Fujimoto & Gu (2021)). One natural approach to this instability is conducting online fine-tuning (see Appendix C.3 for our empirical studies).
Going forward, we believe our work suggests a feasible alternative for generalizable offline robotic learning: by decomposing a single robotic dataset into multiple subsets, offline policy inference can benefit from performing model-based optimization (MBO) and the joint deployment adaptation procedure.
Social impact: Beyond general offline RL improvement, the authors do not foresee negative social impacts.
C MORE EXPERIMENTS
C.1 SINGLE BEHAVIOR POLICY VS. MULTIPLE BEHAVIOR POLICIES
ditmaze-medium-play
mitmize-mediumddivrreee
antmaze-large- large---laggppaay
antmaze-larrge large-dverre
Trahing loss
Traning loss
Training loss
0.2
0.2
0.1
0.1
0.1
100
100
50
100
50
100
Training steps
Training steps
Training steps
Trining steps
5
10
50
100
200
500
1000
Figure 4: Learning curves of behavior cloning AntMaze suites (*-v2) in D4RL,, where the x-axis denotes the training steps, and the y-axis denotes the training loss. The number N in the legend denotes the number of sub-tasks. If N we learn single behavior policy for the whole of- fline dataset.
In Figure 4 provide empirical evidence that learning a single behavior policy (using BC) is not sufficient to characterize the whole offline dataset, and multiple behavior policies (conducting task decomposition) deliver better resilience to characterize the offline data than single behavior policy.
14

---

Under review as a conference paper at ICLR 2023
helfchectal--raddoo
balfchectab-mmddim
ha hulfchoctab-modinn-rphle
2.4
100
37
Normalied
2,3
50-
33
2.2
Rand
Quan
Rank
Quat.
Rand
Quan
Quan
hopper-random
hopper mendu
Huppeer-nediumm-coe expert
Mopper-medium-replan
Normalzoe
50
Rand
Quan
Rank
Rand
Quan
Rank
Rand
Quan
Rank
Rand
Quan
Rank
Nomalize rotums
walker2d-random
walker26-medium
mellk-r2d-mddii--xxpet
walker22d-mdiuu--rplaay
100
so-
20
10
Quan
Rond
Quian
Quann
Ronk
Quann
Normaltadd reum
antmaaze-umssn
antmazee antmssze-umaze-dee umaze-diverse
antmazze mucii
antmazze meaiizzee-ediun--die diverrs
75
75
50
40
50
50
20
0+
Rand
Quan
Bank
Rand
Quan
Rank
Rand
Quan
Rank
Rand
Quan
Rank
Normalzzdd rebran
animase-large-play
antmasz-llrre-ddee large-diverse
50-
DROP Grod
Rank DROP
Rand DRUP Bee Ada
Bank DROP iBlss
25
Rond DRUOP-CCdd-MM
Rank DROP-Gesé-Ada
Quan DROP-Ceal
DROP--lest-dds
Rand
Quain
Rank
Raidd
Quan
Kank
Quan DROP-Geul-Ada
Figure 5: Comparison of three different decomposition rules on D4RL MuJoCo-Gym suite (+-v0) and AntMaze suite (*-v2), where "Rand", "Quan" and "Rank" denote the Random, Quantization, and Rank decomposition rules respectively. We can find across 18 tasks (AntMaze and MuloCo- and DROP-Grad Ada), Rank is more stable and yields better performance compared with the other two decomposition rules.
Gym suites) and 3 embedding inference methods (DROP-Grad,
C.2 DECOMPOSITION RULES
In DROP algorithm, we explicitly decompose an offline task into multiple sub-tasks, over which we then reframe the offline policy learning problem one of offline model- based optimization. In this section, we discuss three different designs for the task decomposition rule.
Random( N, M): We decomposition offline dataset D == {r) into N subsets, each of which con- tains most M trajectories that are randomly sampled from the offline dataset.
Quantization N, M): Leveraging the returns of trajectories in offline data, we first quantize offline trajectories into N bins, and then randomly sample at most trajectories (as sub-task) from each bin. Specifically, in the i-th bin, the quantized trajectories (7:} satisfy Rmin + A + < Return(T.) S
Rmin + A 1), where and Rmin denote the maximum and minimum trajectory returns in the offline dataset respectively.
(Ra--Ra.. Return(7.) denotes the return trajectory Tú, and R N:
Rank(N. M): We first rank the offline trajectories descendingly based their returns, and then se- quentially sample M trajectories for each subset. (We adopt this decomposition rude in main paper.)
In Figure 5 we provide the comparison of the above three decomposition rules (see the selected number of sub-tasks and the number of trajectories in each sub-task in Table 9 We can find that across a verity of tasks, decomposition rule has fundamental impact on the subsequent model- based optimization. Across different tasks and different embedding inference rules, Random and Quantization decomposition rules tend to exhibit large performance fluctuations, which reveals the importance of choosing suitable task decomposition rule. In paper. adopt the Rank decom- position rule, as it demonstrates a more robust performance shown in Figure 5 In Appendix [C.4] adopt the conditional variational auto-encoder (CVAE) conduct automatic task decomposi-
15

---

Under review as a conference paper at ICLR 2023
Table 4: Comparison between DROP (using the Rank decomposition rule) and filtered behavior cloning (F-BC) on D4RL AntMaze and MuJoCo suites (*-v2). We take the baseline results of BC and F-BC from Emmons al. 2021), where F-BC is trained over the top 10% trajectories, ordered by the returns. Our DROP resulss are computed over 5 seeds and 10 episodes for each seed.
Tasks
BC
F-BC
DROP-Grad
DROP-Best-Ada DROP-Grad-Ae
crimaze-umazz
54.6
60
72 + 17.2
78 11.7
80 + 12.6
cetmazz-umaze-diing
45.6
46.5
48 + 22.3
62 16
66 + 12
42.1
24 1 10.2
34 1 12
21
antmaze-medium diverse
37.2
20 +
24
16.7
antmaze-lagge-play-
28
24 1418
36 1 17.4
42 17.2
antmazz-large-divvrre
34.3
14
20 + 14.1
26 + 13.6
antmaze total
100.2
248.1
254
274
halfcheetah
random
2.3
2
2.3
+ 0
2.3 + 0
2.3 + 0
hopper
random
4.8
4.1
5.1
1 0.8
5.4 1 0.7
5.5 1 0.6
walker2d
random
1.7
1.7
2.8
1.7
3 + 1.6
+ 1.8
halfcheetah
medium
42.6
42.5
42.4 0.7
42.9 0.4
43.1 +
hopper
medium
52.9
56,9
57.5
60.3 6.1
59.5 5.1
walker2d
medium
75.3
75
76.5
75.8
79.1 1.4
halfcheetah
mepium-replay
36.6
40.6
3
40.4 0.8
40.3 1.2
hopper
medium-replay
18.1
75.9
48 + 17.7
83.4 6.5
87.4 2.1
walker2d
medium-replay
26
62.5
37.4 1 13.5
60.9 7.4
61.9 2.3
halfcheetah
medium-expert
55.2
92.9
86.6 + 3.9
88.5 + 1.2
88.9 +
hopper
medium-expert
52.5
110.9
103.5 1 6.3
102.5 1 6.2
105.9 1 4.9
walker2d
medium-cxpert
107.5
109
2
106.8 + 3.9
106.9 + 3.6
mujoco-gym total
475.5
674
609.1
672.2
683.8
tion (treating each trajectory in offline dataset as individual task) and find such implementa- tion (DROP+CVAE) can further improve DROP's performance. future work, we also encourage better decomposition rules decompose offline tasks enable more effective model-base optimization for offline RL tasks.
Comparison with filtered behavior cloning. We also note that the Rank decomposition rule leverages more high-quality trajectories than the other two decomposition rules (Random and Quan- tization). Thus, a natural question ask is, is the performance of Rank better than that of Random and Quantization due to the presence of more high-quality trajectories the decomposed sub-tasks? That whether DROP (using the Rank decomposition rule) only conducts behavioral cloning over those high-quality trajectories, thus leading to better performance.
To the above question, compare DROP (using the Rank decomposition rule) with the filtered behavior cloning (F-BC), where the latter (F-BC) performs behavior cloning after filtering for trajectories with highest returns. We provide the comparison results in Table We can find that in AntMaze tasks, the overall performance of DROP is higher than that of F-BC. For the MuJoCo- Gym suite, DROP-based methods outperforms F-BC on these offline tasks that contain a plenty of sub-optimal trajectories, including the random, medium, and medium-replay domains. This result indicates that DROP can leverage the sort of embedding inference (extrapolation) to find a better policy beyond all the behavior policies in sub-tasks, which is more effective than simply performing imitation learning on a subset of the dataset.
C.3 ONLINE FINE-TUNING
Online fine-tuning (checkpoint-leve)), Figure 6 we show the learning curves of DROP-Bes four DR4L. tasks. We can find that DROP exhibits a high-variance (in performance) across
16

---

Under review as a conference paper at ICLR 2023
halfchectah-mma macitttu-mmmmmmepaaa
logper-modium-relay
antmaze- antmaze-large-pla
return
4000-
50000
Egitode return
3000
returm return
3000
return
0.75
5000
2000
Egisode
2000
0.50
2000
seed-1
1000-
1000
0.25
0.00
50
100
50
100
50
100
50
100
Training steps
Training steps
Traning steps
Training steps
Figure 6: Learning curves of DROP, where the x-axis denotes the training steps (k), y-axis denotes the evaluation return (using DROP-Best embedding inference rule). We only show two seeds for leg- ibility.
training step 5 which means the performance of the agent may be dependent on the specific stopping point chosen for evaluation (such instability also exists in prior offline RL methods (Fujimoto & Gu 202T
To choose suitable stopping checkpoint over which perform the DROP inference (DROP- Gradd and DROP- Grad-Ada), we propose to conduct checkpoint-level online fine-tuning (see Algorithm in Section for more details): we evaluate each of the latest I" checkpoint models and choose the best one that leads to the highest episode return.
In Figure 7 we show the total normalized returns across all the tasks in each suite (including Maze2d, AntMaze, and MuJoCo-Gym), We can find that in most tasks, fine-tuning (FT) can guar- antee a performance improvement. However, we also find such fine-tuning causes negative im- pacts in performance in AntMaze(*-vO) suite. The main reason is that, in this checkpoint-level fine-tuning, we choose the "suitable" checkpoint model using the DROP-Best embedding inference rule, while we adopt the other three embedding inference rules (DROP-Grase DROP-Grad- -rad-Grad-Ada at the test time. Such finding also implies that the success of DROP's deployment adaptation is not entirely dependent on the best embedding across sub-tasks (i.e., the best embed- ding z (5)) in DROP-Best Bestt,, but requires switching between some "suboptimal" embeddings (using CROP-Best-Ada extrapolating new embeddings (using DROP-Grad-Ada).
and
Total nomallidd rotum
Maze2d
AntMazo(-v0)
MuloCo-Gym(-vd)
AntMaze(-v2)
MuJoCo-Gym(-v2)
200
500
50
200
500
100
250
100
250
DROP-Grad
DROP-Gra FT
DROP Best Ada
DROP-Rest TPP-Rett.MM. Ado FT
DROP Grad Ada
DROP- Gradd Ada FT
DROP- Grad
DROP Best Ado
DROP Bett Ada PT
DROP Grad DROP-Gmu- And Ada FT
DROP- Grad
DROP PROP-Best Best-Ada
DROP Grad Grad-Ada Ada
DROP- Gnd
DROP- Grad FT
DROP- Best Adl
DROP Best- Ads FT
DROP Grad-Ads
DROP-Gnd--MM Add FT
DROP Crad
DROP-GrA Grad FT
DROP DROP-Best/ Best Ada
DROP- Dest Ada FT
DROP-Gru GooP--Gd-AA Ado
DROP-Gmi--Aln FT
DROP-Gra
PRod-Grad
DROP Be--Bett--dd
DROP-Grad Gry-Gndd-dd
Ada
Figure 7: Total normalized returns across all the tasks in Maze2d, AntMaze, and MuJoCo-
Gym suites.
Online fine-tuning (embedding-level), Beyond the above checkpoint-leved fine-tuning procedure, we can also conduct embedding-leve -level online fine-tuning: we aim to choose a suitable gradient up- date step for the gradient- based embedding inference rules (including DROP- and DROP- Gradd Ada). Similar to the checkpoint-leve fine-tuning, we first conduct the deployment adaptation pro-
'In view of such instability, we evaluate methods over multiple checkpoints for each seed, instead of choosing the final checkpoint models during the training loop (see the detailed evaluation protocol Ap- pendix D).
Conversely, if the performance of DROP depends on the best embedding across sub-tasks (ie., (so) in DROP-Best), then checkpoint model we choose by fine-tuning with DROP-Best should enable consis- tent performance improvement for rules that perform embedding inference with Grad-Ada. However, we find performance drop in AntMaze(*-v0) suite, which means these is no ex- plicit dependency hetween the best embedding 2 (5o) and the inferred embedding using the adaptive inference nules (DR)
and DROP.
17

---

Under review as a conference paper at ICLR 2023
cedure (DROP-GEE Grad and DROP-Grad-Ade Grad-Ada) over set of gradient update steps, and then choose the best step that leads to the highest episode return (see Algorithm in Section D for more details).
In Table 5] compare DROP (DROP-Grai and ods (AWAC (Nair et al, 2020b), CQL (Kumar et al.) 2020) and IQL (Kostrikov et al., ) reporting the initial performance and the performance after online fine-tuning. We can find that the embedding-leve tiee-tuning (0.3M) enables a significant improvement in performance. The fine- tuned PROP-Grad-AAe (0.3M) outperforms the AWAC and CQL counterparts in most tasks, even though we take less rollout steps to conduct the online fine-tuning (baselines take 1M online rollout steps, while DROP-based fine-tuning takes 0.3M steps). However, there is still big gap between the fine-tuned IQL and the embedding-level fine-tuned DROP (0.3M). Considering that there remains 0.7M online steps in the comparison, we further conduct "parametric-levar fine-tuning (updating on medium-* and large- tasks, we
to three offline RL meth-
the parameters of the policy network) for our can find which achieves competitive fine-tuning performance even compared with IQL.
Table 5: Online fine-tuning results (initial performance + performance after online fine-tuning). The baseline results of AWAC, CQL, and IQL are taken from Kostrikov et al. (202 where they run 1M online steps to fine-tune the learned policy. For our DROP method (DROP-Gres and DROP-Grad-Ada) we run 0.3M (= Gcheckpoint Km.. x 1t00steps per episode) online steps to fine-tune (embedding-leww) the policy, i.e., aiming to find the optimal gradient ascent step that is used to infer the contextual embedding z" (so) z (S:) for x² (s)
{(a |s,, (see Algorithm for the details). Moreover, for medium-* and large-* tasks, we conduct additional parametric-leve fine- tuning, with 0.7M online steps update the policy's parameters. Our DROP results computed
over 5 seeds and 10 episodes for each seed.
Task (*-v0)
AWAC
CQL
IQL
DROP Grad
DROP-Grad-ae
umaze
56,7 59
70.1
>> 99.4
86.7
> 96
70 > 96 1.2
76 > 98 1.
umaze-diverse
49.3 49
31.1
99.4
75
84
54 88 +
66 - 94 4.9
medium-play
-
23
+
72
+ 95
20-566
30 - 50 6.3
+ 94 + 2.9
medium-diverse
0.7 -+ 0.3
23
>> 32.3
68.3
-> 92
12 - 22-344149
22 + 1499
96
1 0.8
large-play
->
25.5
46
> 38 1.9
+ 163
1.3
large-diverse
+
->
42.6 + 60.7
20 - 40
13.6
22 - 46 1 10.2
58
1 4.5
0.3M
C.4 DROP + CVAE
CVAE-based embedding learning. Similar LAPO (Chen et al. 2022) and PLAS (Zhou al., 2020). we adopt the conditional variational auto-encoder (CVAE) to model offline data. Specifically, we learn the contextual policy and behavior embedding:
Baass, z), (zzls) arg max EIs.)) [og (als, 2)] (L(o(z)))ll(z)).
(10)
8,6
Then, we learn the score model ff with the TD-error and the conservative regulaazzio:
f arg min Eamiiinssaaa (s,
(R(s,a) + 7 F(('s al, o(zls)) f(s,
(11)
s.t. (..DDx=n(x))a001 where / denotes target network and µzz) denotes the uniform distribution over the Z-space.
(f(s,,,z) - D.a.D,xx--ollssaa---laaa.a
[f z)] - n,
In testing, we also dynamically adapt the outer-level optimization, setting policy inference with n (als) (a.ss 2 (s)). where 2" *ss arg max, (s, ((a z),
In Table 6 we compare DROP+CVAE (-Grad-Ada) with LAPO Chen et al. 2022). PLAS Zhou CQL
al. 2020 CQL Kumar et al. 2020)), IQL Kostrikov et al. (b021b) and the naive implementation of DROP( P-GrPP-----iiaaee Ada) (conducting return return-guided task decomposition and afterward learning behavior embedding as in Equation 6). We highlight that even there is big performance gap between DROP and baselines (LAPO and PLAS) in Gym-MuJoCo *-random tasks, our CVAE-basee implementa- tion (DROP+CVAE) can bridge such performance gap. Further, in *-medium tasks, DROP+CVAR
18

---

Under review as a conference paper at ICLR 2023
Table 6: Comparison (on D4RL benchmark) between DROP (including the implementation of eeturn-guided task decomposition and the implementation CVAE-based embedding learning), latent policy baselines (LAPO and PLAS) and other two representative baselines (CQL and IQL).
LAPO
PLAS
CQL
IQL
DROP
DROP+CVAE
umaze
70.7
74.0
87.5
90.5 + 2.4
umaze-diverse
91.3
45.3
84.0
62.2
85.1 + 7.8
antmaze
medium-play
16
61.2
71.2
30
68.2 + 16.5
medium-diverse
85.7
0.7
53.7
70.0
30
75.4 = 9.4
large-play
0.7
15.8
39.6
42
50.1 = 13.6
large-diverse
61.7
0.3
14.9
47.5
26
52.2 + 12.0
walker2d-random
1.3
3.1
-0.23
5.4
3
5.2 +
medium random
hopper-random
23.5
10.5
8.3
7.9
5.5
20.8 = 0.3
halfcheetah--random
30.6
25.8
22.2
13.1
2.3
32.0
- 2.5
walker2d-medium
80.8
44.6
82.1
77.9
79.1
82.1
+ 5.2
hopper-medium
51.6
32.9
71.6
65.8
59.5
61.5
+ 3.7
halfchhetahhmmedin
46
39.3
49.8
47.8
43.1
52.4
+ 2.2
can further improve the performance (of DROP), surpassing both LAPO and PLAS that similarly adopt the CVAE model the offline data. Besides, we compare DROP+CVAE to baselines CQL and IQL, where we see DROP+CVAE can also achieve a competitive performance.
C.5 ABLATION STUDY
Note that our embedding inference depends on the learned score model f. Without proper regularization, such inference will lead to out-of-distributio embeddings that are erro- neously high scored (Q2). Here conduct an ablation study to examine the impact of the conservative regularization used for learning the score model.
100
100
75
75
50-
Grad
Grad-Ada
Grad w/o Reg
25
Grad-Ada w/o Reg
60
120
150
90 120 150
In Figure 8 compare DROP-Grad and DROP-Grad-Ar to their naive implementa- tion (w.o. Reg) that ablates the regularization on
Figure 8: Ablation on the conservative regular- ization. The y-axis represents the normalize re- turn, and the x-axis represents the number of gradient-ascent steps used for embedding inference at deployment. We plot each random seed as a transparent line; the solid line corresponds to the average across seeds.
We can find that removing the conservative regularization leads to unstable performance when changing the update steps of gradient-based optimiza- tion. However, we empirically find that in some tasks such naive implementation (w/ Reg) does not necessarily bring unstable inference (Ap- pendix B Although improper gradient update step leads to faraway embeddings, to some extent, mubeeding-condiitie behavior policy can correct such deviation.
D IMPLEMENTATION DETAILS
For the practical implementation DROP, we parameterize the task embedding function Φ(z[n), the contextual behavior policy B(als, z) and the score model f(s, a, z) with neural networks (see Appendix D for specific architectures). For Equation g we construct Lagrangian and solve the optimization through primal-dual gradient descent. For the choice of μız),, simply set μzz) to be the uniform distribution over the Z-space and empirically find that such uniform sampling can effectively avoid the -ut-i--distributtin extrapolation at inference.
19

---

Under review as a conference paper at ICLR 2023
Lagrangian Relaxation. To optimize the constrained objective in Equation 8 in the main paper, we construct a Lagrangian and solve the optimization through primal-dual gradient descent,
(D,,-Diy (s (R(s.a) a) 7f(s', a', (() f(s a, d(zln))) }]
min max f A=0
[f a. z)] - d(x Dr, (((alss.) [f a. z)] n)
This unconstrained objective implies that if the expected difference in scores of out-of-distribution embeddings and in-distribution embeddings is less than threshold À is going to be adjusted to O, on the contrary. À is likely to take a larger value, used to punish the over-estimated value func- tion. This objective encourages that out-of-distribution embeddings score lower than in-distribution embeddings, thus performing embedding inference will not lead to these out-of-distribution embed- dings that are falsely and over-optimistically scored by the learned score model.
In our experiments, we tried five different values for the Lagrange threshold ry (1.0, 2.0, 3.0, 4.0 and 5.0). We did not observe a significant difference in performance across these values. Therefore, simply set = 2.0.
2
a Enc
Enc
Enc 0
1
Enc
Enc
n
2
S
s,a
Figure 9: Architectures of the task embedding network d(z)s), the contextual behavior policy ((ass, 2), and the score function f(s, a, z) (from left right).
Hyper-parameters. In Figure ((((s), the contextual behavior policy (aass, z), and the score function [ss a, 2), where the corre- sponding hyper-parameters provided in Table 7 For the gradient ascent update steps (used for embedding inference), we set K 100 for all the embedding inference rules experiments.
provide the network architecture of the task embedding
Table 7: Hlyper-parametert
Enc 0
Enc 1
Enc 2
Enc 3
Enc 4
Optimizer Hidden layer
Adam
Adam
Adam
Adam
Adam
2
2
3
2
Hidden dim
512
512
512
512
512
Activation function
ReL.U
ReLU
ReL.U
ReLU
ReL.U
Learning rate Mini-batch size
1.00E-03
1.00E-03
1.00E-03
1.00E-03
1.00E-03
1024
1024
1024
1024
1024
In Table we provide the number of sub-tasks, the number trajectories in each sub-task, and the dimension of the embedding for each sub-task (behavior policy). The selection of hyperparameter is based on two evaluation metrics: (1) the fitting loss of the decomposed behavioral policies to the offline data, and (2) the testing performance of DROP. Specifically.
(Stepl) Over a hyperparameter (the number sub-tasks) set, we conduct the hyperparam- eter search using the fitting loss of behavior policies, then we choose/filter the four best hyperparameters;
(Step2) We follow the normal practice of hyperparameter selection and tune the four hy- permeters selected in Step1 by interacting with the simulator to estimate the performance of DROP under each hyperparameter setting.
20

---

Under review as a conference paper at ICLR 2023
Algorithm 3 DROP: Online fine-tuning (checkpoint-levee)
Require: Env, last T' checkpoint models: (a z) and (t=1,
1: RMAX- -..
2: Prest None.
3: Joest < None.
4: while 1,... do do
5: = Env.Reset().
6: 2z < Conduct embedding inference with DROP-Best.
7: Return + Evaluate 3t and ft on Env, setting x (a(ss) = B(als, (s)))
8:
if RMAA < Return then
9: Update the best checkpoint models: Phest Be, soest f S2. 10: Update the optimal return: RMAX <- Return.
11: end if
12:
end while
Return: Bbeat and feees.
We provide the hyperparameter sets in Table In Step2, we tune the (fil- tered) hyperparameters using 1 seed, then evaluate the best hyperparameter by training on an additional seeds and finally report the results on the 5 total seeds (see next "evaluation pro- tocol"). In Antmaze domain, a single fixed N works well for many tasks; while in Gym-mujoco and Adroit domains, we did not find a fixed N that provides good results for all tasks in the corresponding domain in D4RL, thus we use the above hyperparameter selection rules (Stepl and Step2) to choose the number N.
Table 8: Hyperparameter (the number of sub-tasks) set.
tasks
the number of sub-tasks
Antmaze
500 (v0), 150 (v2)
Gym-mujoco
10, 20, 50, 100, 200, 500, 800, 1000 10, 20, 50, 100, 200, 500, 800, 1000
Adroit
Environment details. For the comparison of our method to prior iterative offline RL methods, we consider the vO versions of the datasets in D4R1 3 We take the baseline results of BEAR, BCQ, CQL, and BRAC-p from the D4RL paper (Fu et al. 2020), and take the results of TD3+BC from their origin paper (Fujimoto & Gul 2021). For the comparison of our method to prior non-iterative offline RL method, we use the v2 versions of the dataset in D4RL. All the baseline results of be- havior cloning (BC), Decision Transform (DT), RvS-R, and Onestep are taken from Emmons al.) (202I). In our implementation of COMs, we take the parameters (neural network weights) of be- havior policies as the design input for the score model; and during testing, we conduct parameters inference (outer-level optimization) with 200 steps gradient ascent over the learned score function, then the rollout policy initialized with the inferred parameters. For the specific architecture, we instance the policy network with dim(S) input units, two layers with 64 hidden units, and a final output layer with (A)..A).
Evaluation protocol. We evaluate our results over seeds. For each seed, instead of taking the final checkpoint model produced by a training loop, we take the last T (T ( 6 in our experiments) checkpoint models, and evaluate them over 10 episodes for each checkpoint, That to say, we report the average of the evaluation scores over 5geed X 6checkpoint X 10ppisode rollouts.
Online fine-tuning (checkpoint-level) Instead of re-training the learned (final) policy with online rollouts, we fine-tune our policy with enumerated trail-and-erros over the last T checkpoint models (Algorithm 3) Specifically, for each seed, we run the last T checkpoint models in environment one episode for each checkpoint. The checkpoint model which achieves the maximum episode return is returned. In essence, this fine-tuning procedure imitates the online RL evaluation protocol: if the current policy is unsatisfactory, we can use checkpoints of previous iterations of the policy.
Online fine-tuning (embedding-level); The emhedding-leeel fine-tuning aims to find a suitable gradi- ascent step that is used to conduct the embedding inference in DROP- or DROP-Grad-Ace
We noticed that Maze2D-v0 in the D4RI. dataset (https://rail.com http:/rail.eccs.berkeeeee barkeele.edd/datttee is not available, used v1 version instead in our experiment. For simplicity, we still v0 in the paper exposition.
21

---

Under review as a conference paper at ICLR 2023
Algorithm 4 DROP: Online fine-tuning (embedding-level)
Require: Env, last checkpoint models: (a z) and fi(s, (t ..,,.,,,,
1: RMAX- -00.
2: Prest None.
3: Joest <- None.
4: Abest f-0
5: while = 1, .T' do
6: while k 1. mux do
7:
Env.Reset)).
# Conduct embedding inference with DROP-Grad or DROP-Grad-Ada
8:
Return + Evaluate 3r and fe on Env, setting x (als) [[a x (Sn)) or B(als, 2" 28)),
where we conduct gradient ascent steps to obtain z (5o) or z (s).
9:
if RMAX < Return then
10:
Update the best checkpoint models: Bbest +- Br, fbest f- ft.
11:
Update the best gradient update step: libest +- k.
12:
Update the optimal return: RMAX <- Return.
end if
14:
end while
end while
Return: Boest fbest and khees-
Thus, we enumerate list gradient update steps and pick the best update step (according to the episode returns).
Codebase. Our code is based on httpes//gttha..cco////eeeeu///// We provide our source code in the supplementary material.
Computational resources. The experiments were run computational cluster with 22x GeForce RTX 2080 Ti, and 4x NVIDIA Tesla V100 32GB for 20 days.
22

---

Under review as a conference paper at ICLR 2023
Table 9: The number (N) of sub-tasks, the number (M) of trajectories in each sub-task, and the dimension (dim(z)) of the embedding for each sub-task.
Parameters (*-v0)
Parameters (*-v2)
Domain
Task Name
M dim(z)
M dim(z)
umaze
500
5
Maze 2D
medium
150
50
large
100
15
umaze
500
50
150
50
umaze-diverse
500
50
150
50
Medium-plat
500
50
150
50
Antmaze
Medium-diverse
500
50
150
50
Large-play
500
50
150
50
Large-diverse
500
50
150
50
random
1000
1
1000
1
medium
100
100
halfcheetah
medium-expert
1000
1
1000
medium-replay
50
50
random
100
2
100
2
medium
100
5
100
5
hopper
medium-expert
100
2
100
2
medium-replay
50
5
10
30
random
500
2
500
medium
50
5
50
5
walker2d
medium-expert
50
5
50
5
medium-replay
1000
5
10
50
cloned
1000
2
door
expert
500
5
human
50
3
cloned
1000
hammer expert
500
5
human
20
3
cloned
500
5
pen
expert
500
5
human
50
5
cloned
500
5
relocate
expert
500
5
human
50
4
E ADDITIONAL RESULTS
Comparison with iterative offline RL baselines. Here. we compare the performance of DROP (Grad, Best-Ada, and Grad-Ada ) to iterative offline RL baselines (BEAR (Kumar et al.) 2019a)
BCQ (Fujimoto al. 2019) CQL (Kumar al.) 2020 BRAC-p (wu et al. 2019). and
TD3+BC (Fujimoto & Gul 2021) that perform iterative bi-level offline RL paradigm with (explicit
or implicit) value/policy regularization
inner-level. In Table 10 we present the results the
Maze2D, AntMaze, Gym-MuJoCo, and Adroit suites in standard D4RL benchmark (*-v0), where we can find that DROP-Grad- Grad-Ada performs comparably or surpasses prior iterative bi-level works on most tasks: outperforming (or comparing) these policy regularized methods (BRAC-p and TD3+BC) on 25 out of 33 tasks and outperforming (or comparing) these value regularized algorithms (BEAR, BCQ, and CQL) on 19 out of 33 tasks.
23

---

Under review as a conference paper at ICLR 2023
Comparison with RvS baselines. As complement the results shown in Table in the main paper, we provide the performance comparison for more tasks (Gym-MuJoCo and AntMaze 4-v2 suites in D4RL) in Figure IT. We find similar results presented in our main paper: DROP consistently outperforms baselines in AntMaze tasks (the last three rows of sub-figures in Figure [1] and reaches comparable results most tasks in Gym-MuJoCo suite.
Across different environments, we also find DROP exhibits more robust performance. Although baseline Onestep shows impressive performance in Gym-MuJoCo tasks, can see that Onestep fails to make progress in AntMaze-medium- and AntMaze-large- tasks. However, we find that DROP-based methods exhibit significant performance improvement in this AntMaze suite. We attribute the success of DROP outperforming Onestep (conducting only behavior policy improve- ment) to three advantages: (1) DROP learns multiple behavior policies; (2) DROP conducts policy improvement (corresponding to the embedding inference procedure) over multiple behavior policies; (3) DROP permits deployment adaptation, enabling the agent "switch" behavior policies.
halfchectah medium
hopper-medium
walker2d-
walker2d medium-exppe exppett
Normalied rotumis
40
24
80
30-
22
20
60
30
150
90
150
30
DROP-Grd-Add
DROP-Grad-Ada wio Reg
Figure 10: The performance comparison of DROP- PRROP-Gra--Aee Grad-Ada and GroP--rad-Ada w/o Reg, where we ablate the conservative regularization for the w/o Reg implementation. The y-axis denotes the normalized return, the x-axis denotes the number gradient-ascent steps used for embedding infer- ence at deployment.
we provide more results for the ablation of the conservative reg- ularization term in Equation in the main paper. We can find that for the halfchectah-mmeding and hopper-medium tasks, the performance of DROP-Grad-Ace w/o Reg depends on the choice of the gradient update steps, showing that too small or too large number of gradient update step deterio- rates the performance. Such result is also consistent with COMs (Trabucco et al., 2021), which also observes the sensitivity of naive gradient update (i.e., w/o Reg) number of update steps used for design input inference. By comparison, conservative score model learned with DROP-Grad-Adi exhibits more stable and robust the gradient update steps.
Ablation studies. In Figure 10
Further, we also find that in walker2d-mediie medium and walker2d-medium--oee medium-expert tasks, the naive gra- dient update (w/o Reg) does not affeet performance significantly across wide range of gradient update steps. The main reason is that although the excessive gradient updates lead to faraway em- beddings, conditioned the inferred embeddings, the learned contextual behavior policy can safe- guard against the embeddings distribution shift. Compared to prior model-based optimization that conducts direct gradient optimization (inference) over the design input itself, such "self-safeguard" is special merit in the offline RL domain as long as we reframe the offline RL problem as one of model-based optimization and conduct inference over the embedding space. Thus, we encourage the research community to pursue further study to this model-based optimization view for the offline RL problem.
DROP results. the paper.
In Table
11 we provide our complete results (including the variance) all tasks
24

---

Under review as a conference paper at ICLR 2023
Table 10: Comparison of our method to prior offline methods that perform iterative (regularized) paradigm on D4RL.. We take the baseline results BEAR, BCQ, CQL and BRAC-p from Fu et al. (020), and the results of TD3-BC from Fujimoto & Gu (02T). For all results of our method (DROP), we average the normalized returns across seeds; for each seed, we run evaluation episodes. For proper comparison, or better performance compared with value and policy regularized offline RL methods respectively.
and to denote DROP (*-Ada) achieves comparable
Value Reg.
Policy Reg.
DROP.
Task Name
BEAR
BCQ
CQL
BRAC-p
TD3+BC
Grad
Best-Ada
Grad-Ada
maze2d
umaze
3.4
12.8
5.7
4.7
18.6
18.7
21.3
medium
29.0
32.4
17.0
24.3
large
4.6
6.2
12.5
10.4
26.7
14.7
28.8
73.0
78.9
74.0
50.0
72.0
78.0
80.0
umaze-diverse
61.0
55.0
84.0
40.0
48.0
62.0
66.0
antmaze
medium- play
0.0
24.0
medium-diverse
8.0
0.0
53.7
0.0
20.0
24.0 A
30,0
large-play
0.0
6.7
15.8
0.0
24.0
3
42.0
large-diverse
0.0
2.2
14.9
0.0
14.0
20.0
26.0
halfcheetah
random
25.1
35.4
24.1
2.3
2.3
2.3
medium
41.7
40.7
44.4
43.8
42.8
42.4
42.9
43.1
medium-expert
53.4
64.7
62.4
44.2
97.9
86.6
88.5
88.9 A
medium-replay
38.6
38.2
46.2
45.4
43.3
39.5
40.4
40.3
random
11.4
10.6
10.8
11.0
11.0
5.1
5.4
5.5
hopper
medium
52.1
54.5
58.0
32.7
99.5
57.5
60.3
59.5
medium-expert
96.3
110.9
98.7
1.9
112.2
103.5
102.5
105.9
medium-replay
33.7
33.1
48.6
0.6
31.4
48.0
83.4
87.4
walker2d
random
7.3
1.4
2.8
3.0
medium
59.1
53.1
79.2
77.5
79.7
76.5
75.8
79.1
medium-expert
40.1
57.5
111.0
76.9
101.1
107.5
106.8
106.9
medium-replay
19.2
15.0
26.7
-0.3
25.2
37.4
60.9
61.9
cloned
-0.1
0.0
0.4
-0.1
0.5
2.5
2.7
door
expert
103.4
99.0
101.5
-0.3
98.6
102.2
102.6
human
-0.3
0.0
9.9
-0.3
3.3
1.9
3.0
hammer
cloned
2.1
0.3
0.3
0.3
0.3
expert
127.3
107.2
86.7
0.3
65.7
73.3
77.7
human
0.3
0.5
4.4
0.3
1.1
0.3
2.1
cloned
26.5
44.0
39.2
1.6
76.7
77.1
82.4
pen
expert
105.9
114.9
107.0
113.1
118.6
116.7
human
-1.0
68.9
37.5
8.1
71.1
85.2
81,5
relocate
cloned
-0.3
-0.3
-0.1
-0.3
0.1
0.5
0.2
expert
98.6
41.6
95.0
2.5
6.2
5.4
human
-0.3
-0.1
0.2
-0.3
0.0
0.0
0.0
25

---

Under review as a conference paper at ICLR 2023
ha-random
ha-medium
ha-medium-experr
ha Aol-medium--te mediuu--eppay
BC
DI
RyS-R
Onestop
DROP Grad
DROP Best-Adaa Ada
PKOP-GrrddAAde
0.0
5.0
20
20
40
ho-random
ho-medium
ho-medium expert
no-mmedium-expert
ho-medium-replly
RvS
Onestep
DROP-Gmd
DROP-Best-Ada
DROP-Gmd-Ada
100
wa-random
wa-medium
wa-medium-expert
wva-medium-replee
RyS-R
Onestop
DROP-Grs Grad
PROP-Best-Adl
DROP-Cird-Ada
5.0
100
antmazzeumaaze
antmaze- -iaeaz---umaze-ivereeee
BC
DT
RyS-R
Onestep
DRoP-Grad
DROP-Best-Ade
GROP-Grad-Ada
50
antmaze-mediume
antmaze antmaze-mediumg medium-diverse
BC
DT
RvS-R
Onestep
DROP-Grad
Best--Bst-Ada
DROP-Grad-Ada
20
40
20
40
mentmaze-large-pae
an tanztmzz--ate lawauz---rrgediirreee
BC
DT
RvS-R
Onestep
DROP-Grad
DROP Best Ada
DROP-Grad.Ae Ada
25
50
20
40
Figure 11: Comparison with non-iterative methods on D4RL (*-v2), where ha = halfcheetah, ho hopper, and walker2d. Each bar denotes the average of normalized returns. The baseline results of behavior cloning (BC), Decision Transform (DT), RvS-R, and Onestep are taken from Emmons al. 2021). Our DROP results are computed over seeds and 10 episodes for each seed.
26

---

Under review as a conference paper at ICLR 2023
DROP- DROP-Grad-He CaPP-Grad--ia Ada
80.0 + 12.6
66.0 = 12.0
30.0 21.0
30.0 = 16.7
42.0 + 17.2
26.0 +: 13.6
2.3 2.3 0.0
43.1 1 0.4
88.9 2.0
40.3 1::1111 1.2
5.5 0.6
59.5 5.1
105.9 4.9
87.4 + 2.1
3.0 1.8
79.1 1.4
106.9 1 3.6
61.9 61.9 2.3
Table 11: Normalized return DROP-Grree Gradd DROP-Bess Best-Ada and DROP Grad-Grddaaaaa Ada accross Maze2d, AntMaze, MuJoCo- -ymm and Adroit suites.
DROP Ber--Best-dd
78.0 + 11.7
16.0
34.0 + 12.0
12.0
36.0 + 17.4
20.0 + 14.1
2.3 4.3 0.0
42.9 0.4
88.5 + 1.2
40.4 4: 0.8.
+.7
60.3 = 6.1
102.5 10..5:6.2 6.2
83.4 4 6.5
3.0 + 1.6
75.8 3.0
106.8 3.9
60.9 7.4
*-y2
62.0
24.0
DROP-Grad PROP-Best-Are DROP-Grad-Ada DROP-Grad
72.0 + 17.2
48.0 + 22.3
24.0 - 10.2
20.0 = 19.0
- 8.0
14,0 + 8.0
2.3 0.0
42.4 = 0.7
3.9
39.5 1: 1.0
5.1 0.8
57.5 - 6.4
+.3
17.7
2.8 1.7
76.5 1: 2.4
1 2.0
37.4 13.5
4
24.0
86.6
103.5
48.0
107.5
37.4
21.3 1 1.8
24.3 5.4
28.8 + 18.8
76.0 + 19.6
66.0 + 10.2
30.0 6.3
22.0 + 14.7
16.0 + 13.6
22.0 +: 14.7
2.3 0.0
37.1 0.4
96.5 7.4
33.6 6.8
11.1 0.4
46.5 - 12.8
112.5 0.8
32.0 4.0
4.1 0.8
24.7 9.7
91.3 5.7
16.9 16.9 7.0
2.7 3.2
102.6 1.6
3.0 2.6
0.3 0.0
77.7 + 25.0
2.1 +33
82.4 + 36.6
116.7 1. 23.5
81.5 :: 16.2
5.4 3
0.0 0.0
0.2
*-v0 (-vl for Maze2d)
18.7 3.6
18.1 = 8.9
14.7 + 20.6
72.0 +: 18.3
68.0 = 7.5
26.0 + 18.5
20.0 1 11.0
20.0 + 21.0
22.0 +: 24,8
2.3 0.0
36.8 = 0.1
89.6 +: 11.5
36.7 1: 3.3
11.1 0.4
52.2 +: 23.1
112.0 = 0.2
32.6 = 3.8
4.1 0.7
26.3 1 8.2
91.7 10.2
13.9 + 4.3
2.5 2.5
102.2 + 2.4
1.9 + 2.2
0.3 0.0
73.3 + 15.3
0.3 = 0.1
77.1 = 6.0
118.6 1: 13.3
8 4: 13.5
0.5 0.5
6.2 4,9
0.0 = 0.0
18.6 18 5.5
17.0 + 10.6
26.6 + 15.1
12.77
70.0 + 12.7
54.0 + 20.6
20.0 2 6.3
12.0 + 16.0
18.55
16.0 + 18.5
20.0 = 8.9
2.3 - 0.0
36.8 = 0.6
85.6 1 15.1
32.6 1: 8.4
11.1 + 0.4
45,8 + 14.3
111.2 + 1.8 1.8
31.2 1 6.0
3.9 0.9
19.2 + 13.4
84.4 + 23.2
13.1 4.8
0.5 4: 0.7
98.6 + 5.5
3.3 4.5
0.3 0.0
65.7 26.6
1.1 + 1.6
76.7 1 15.0
113.1 1 10.2
71.1 34.1
0.1 = 0.0
2.5 2.5 3.3
0.0 = 0.0
halfcheetah- weahheetah-medium-- medium-expert
halfcheetah medium-teppla
antmaze-min meedum----------------------- diverse
walker2d- walker2d-medhumte -ed-im--ddmepaaaa
antmaze-une weee--umazz-eeeeee
antmaze-ment meee--eedummmmm play
antmaze- antmaze-large-ee large- play
antmaze- wereee-aare--ee lare--arre-diieesee diveevee
Sitchectah-random -random
halfcheetah medium
hopper medium-exxpert
hopper medium-rooee medium---aay replay
walker2d-mede (edd--eddum-eepee
walker2d medium
maze2d-umaze
maze2d-mediin md-medium
maze2d larg
milmaze-umaze
halfehectah-
hopper-randon random
hopper medium
walker2d- walker2d-randons random
door-cloned
door-expert
door-human
hammer-cloned
hammer-expeer expert
hammer-hi hammer-human human
pen-cloned
pen-expert
pen-human
relocate-clone
relocate- Procate-expert expert
relocate human
MuJoCo-Gym
Mazeld
AniMaze
Adroit
27


Reviews:
[{"review_id": "y2Pf68v9Fm", "reviewer": "Reviewer_f5fq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tries to summarize current offline RL algorithms into a bi-level optimization problem and offers a new algorithm in the non-iterative camp (that is sufficiently different from previous non-iterative work such as RvS). ", "summary_of_the_review": "I'm currently giving this paper a 6 because I think it's well-written and offers interesting insights. I also have a series of questions regarding the technical and conceptual parts of the paper. Based on the answer, I will re-assess my score for this paper.", "strength_and_weaknesses": "This is a very novel paper that provides a fascinating new insight into Offline RL based on the COMs [1] framework.\nThe extension on top of COMs is simple, but the observations made in the paper (about how we learn $Q$ and how we use $Q$ to find policy) are nontrivial and fascinating. \n\nTechnical Questions:\n1. For deployment-time adaptation (Figure 2 Right side, Algorithm 2), what exactly is adapting? It doesn't seem like the authors are performing parameter updates in $\\beta(a|s, z^*)$ (contextual policy) or score function $f(s, a, z)$. So what is the adaptation here? When I first read it, I thought the authors were doing online learning (aka online fine-tuning), but it doesn't seem to be the case.\n2. At test time, performing gradient ascent to get $z^*$ seems computationally expensive. Can authors comment on how computationally efficient this algorithm is (compared to other algorithms like CQL)? \n3. How does the introduction of context $z$ help solve the OOD penalization problem? In another way, why searching $\\arg\\max_a Q(s, a)$ for policy is bad, while searching $z^* = \\arg\\max_z f(s, a, z)$ then decode as $\\beta(a|s, z^*)$ is better? Because the process seems similar, the authors also trained $f$ to output a lower score when $z$ is OOD. I don't know why this is better than the pessimistic approach on the $(s, a)$ space.\n4. For Table 1, would all the iterative Offline RL Algorithms (i.e., BCQ, CQL, COMBO, AWAC) satisfy all Q1, Q2, Q3? So DROP essentially is just an improvement over supervised learning for RL methods? Although, I should point out that DROP is doing TD-error learning in Equation (8), so I'm not sure why the authors think it's comparable to RvS, Decision Transformer, etc. \n\nStrengths:\n1. The experiments are very comprehensive.\n2. The authors demonstrated expert-level knowledge, summarizing and abstracting prior work into a new framework that could potentially provide insight into the problem domain.\n\n[1]: Conservative Objective Models for Effective Offline Model-Based Optimization", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZRGrhy8Z84", "reviewer": "Reviewer_rqxp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a non-iterative actor-critic approach for offline RL, which tackles the erroneous overestimation problem of the critic by combining the ideas of the non-iterative training procedure, test-time policy adaptation (both from Brandfonbrener et al., 2021), and conservative objective optimization (from Trabucco et al., 2021). \n\nThe key insight of this paper is to fit the assumed multiple modes of offline data distribution by learning a mode-specific embedding as the condition of the actor and the critic. To this end, specific training methods include: \n1. Divide the offline training datasets into multiple sub-tasks based on certain decomposition rules.\n2. In the offline training phase, the policy network and the embedding network are learned jointly with behavior cloning, and the score model is learned with TD-errors. \n3. In the online testing phase, slightly different from optimizing the entire policy network (Trabucco et al., 2021), the authors perform gradient ascent in the embedding space to find the optimal z* that can maximize the estimated objectives of the learned score model.\n\nCompared with previous work, the technical contributions of this paper are:\n1. The idea of task decomposition;\n2. The efficient policy adaptation approach at test time by performing optimization in the compact embedding space.", "summary_of_the_review": "In general, I think this paper proposes an interesting model to tackle the out-of-distribution learning problem in offline RL tasks, and contains rich ablation studies of different task decomposition rules and embedding inference methods. I was inclined to accept this paper given its novelty, however, \n- The paper lacks sufficient results to demonstrate the performance advantage over previous iterative value regularization offline RL methods, especially considering the model incorporates the CQL regularization when learning the score model. These experiments seem unconvincing to me. I strongly suggest further validation of the model by comparing its effectiveness with CQL in the revised paper. \n- The overall writing should be improved. Core algorithms and important results should not be placed in the appendix part. Please amend them accordingly.\n\n\n", "strength_and_weaknesses": "Strength:\n1. It is a reasonable idea to split the offline datasets into N sub-tasks and condition both the score model and the policy model on a task embedding z. The proposed method has two advantages: \n- By explicitly considering the multi-modal distribution of the offline datasets and inferring the task embedding at test time, it can improves the domain adaptation ability of the policy. \n- By optimizing z (which is compact), it can ease the high-dimensional optimization difficulty and thus extend mode-based optimization (Trabucco et al., 2021) to test-time policy adaptation. \n2. Good ablation study on the embedding inference approaches and the decomposition rules (better to move into the main body of the paper).\n\nWeaknesses:\n1. The writing of the paper can be largely improved. \n- The structure of this paper is not very well organized. Frankly speaking, I spent a lot of time reading this paper so that I could understand Q1-Q3 and find out how they relate to each contribution of the proposed method. It might be better to describe the overall framework and the technical novelty before going to each detailed component in the method section. When I read the method section, I felt a little bit lost until I finished Sec 3.3 which introduced the last test-time adaptation module. I suggest describing the general pipeline of the model in the first paragraph of Sec 3 for a better reading experience.\n- Fig 1 is less informative, which only shows the differences between iterative and non-iterative actor-critic offline RL methods, rather than illustrating the unique properties of the proposed method. Since the work of OneStep (Brandfonbrener et al., 2021) also follows the non-iterative framework, this figure does not shed a light on the novel contribution of DROP.\n- The main body of the paper is not self-contained. I have to frequently flip between the main text and the appendix. In my view, at least the core algorithms and some important quantitative experimental results (such as the full comparisons with existing offline RL methods) should be placed in the main text. \n- Misleading notations. In Sec 2.1, \"mu\" stands for the initial state distribution, while in Eq (8), mu is the distribution of the latent sub-task embedding \"z\". This might raise confusion. Suggest using two different notations here.\n- Use ``xxx'' for the quotation marks in LaTeX, instead of ''xxx''.\n2. For the experiments,\n- In Table 2, the proposed model is only compared with OneStep and COMs in the AntMaze environments, which is severely inadequate to support the advantages of the proposed method.\n- The model was claimed to outperform previous value regularized methods including BEAR/BCQ/CQL on 19/33 tasks (Page 8), however, in Table 10, CQL achieves a better performance on 16/33 tasks than the proposed model. These results undoubtedly weaken the effectiveness of the proposed method, especially considering that it also borrows the regularization term from CQL.\n- As a further ablation study, it would be good to show the experimental results without deployment adaptation, which can better verify the effectiveness of performing outer-level optimization in the embedding space rather than the action space from the original COMs.\n\n\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "C6kFDRpcMN", "reviewer": "Reviewer_X8xm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents DROP, a non-iterative framework for policy optimization in offline RL, which separates the model training on offline data from deploying the model for testing/adaptation. The approach first splits the data into N subsets to learn distinct behavior policies, which are further used to learn a contextual behavior policy and task embeddings for a conservative model optimization to be used for policy inference. Experiments on D4RL benchmark data show that the proposed work performs better or on par compared to several competitors.", "summary_of_the_review": "The paper is well-written and introduces a novel approach for offline RL that separates policy inference from the offline training of a conservative model. However, the position of the contributions in the domain of model-based offline RL is not clear and the paper should be improved in terms of clarity. I hence vote for a reject.", "strength_and_weaknesses": "Strengths:\n+ The paper is well-written and visually appealing.\n+ The idea of decoupling conservative behavioral cloning from policy inference is interesting and somewhat novel.\n+ The experiments are extensive and reproducible, and the results are promising on the D4RL benchmark data. \n\n\n\nWeaknesses:\n- The clarity of the paper in terms of contributions, approach, and argumentation should be improved\n- The literature is not sufficiently reviewed and the position of the paper in the domain is not clear.\n- The significance of the contributions on the model-based offline RL domain is not completely clear (see comments below)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8A7FszgE0Xx", "reviewer": "Reviewer_8eDD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper deals with offline RL. It takes the position that iterated policy evaluation can be given so little trust that behavior cloning is preferable.\nIt makes the assumption that it is useful to decompose the data set into episodes and (assuming that different policies were active in different episodes) to create individual behavior clonig policies for individual episodes or appropriate groups of episodes.\nThe algorithm is evaluated on the D4RL benchmark, where exactly this assumption is realized.", "summary_of_the_review": "The paper is very hard to read.\nThe paper proposes an algorithm that exploits a property of the offline data in the D4RL benchmarks. It does not discuss whether this property can be expected in practice.  ", "strength_and_weaknesses": "**Strengths**\n* The algorithm may be useful in special cases that may arise in practice.\n\n**Weaknesses**\n* The presentation needs to be improved significantly wrt structure and writing.\n* The paper proposes an algorithm that exploits a property of the offline data in the D4RL benchmarks. It does not discuss whether this property can be expected in practice.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject