Know Your Boundaries: The Advantage of Explicit Behavior Cloning in Offline RL
Authors: Wonjoon Goo
Scott Niekum
Venue: Submitted to ICLR 2023 (2023)

Abstract: We introduce an offline reinforcement learning (RL) algorithm that explicitly clones a behavior policy to constrain value learning. In offline RL, it is often important to prevent a policy from selecting unobserved actions, since the consequence of these actions cannot be presumed without additional information about the environment. One straightforward way to implement such a constraint is to explicitly model a given data distribution via behavior cloning and directly force a policy not to select uncertain actions. However, many offline RL methods instantiate the constraint indirectly---for example, pessimistic value estimation---due to a concern about errors when modeling a potentially complex behavior policy. In this work, we argue that it is not only viable but beneficial to explicitly model the behavior policy for offline RL because the constraint can be realized in a stable way with the trained model. We first suggest a theoretical framework that allows us to incorporate behavior-cloned models into value-based offline RL methods, enjoying the strength of both explicit behavior cloning and value learning. Then, we propose a practical method utilizing a score-based generative model for behavior cloning. With the proposed method, we show state-of-the-art performance on several datasets within the D4RL and Robomimic benchmarks and achieve competitive performance across all datasets tested.

Under review as a conference paper at ICLR 2023
KNOW YOUR BOUNDARIES: THE ADVANTAGE OF EXPLICIT BEHAVIORAL CLONING IN OFFLINE RL
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce an offline reinforcement learning (RL) algorithm that explicitly clones behavior policy to constrain value learning. offline RL, is often important prevent a policy from selecting unobserved actions, since the consequence of these actions cannot be presumed without additional information about the environment. One straightforward way to implement such constraint is to explicitly model a given data distribution via behavior cloning and directly force policy not to select uncertain actions. However, many offline RL methods instantiate the constraint indirectly- for example, pessimistic value estimation errors when modeling a potentially complex behavior policy. In this work, we argue that it is not only viable but beneficial to explicitly model the behavior policy for offline RL because the constraint can be realized in stable way with the explicitly cloned model. We first suggest a theoretical framework that allows us to incorporate behavior-cloned models into value-based offline RL methods, enjoying the strength of both explicit behavior cloning and value learning. Then, we propose a practical method utilizing a score-based generative model for behavior cloning to better handle the complicated behaviors that an offline RL dataset might contain. The proposed method shows state-of-the-ar performance on several datasets within the D4RL and Robomimic benchmarks and achieves competitive performance across datasets tested.
to a concern about
INTRODUCTION
The goal of offline reinforcement learning (RI) is to learn policy purely from pre-generated data. This data-driven RL paradigm is promising since it opens up possibility for RL to be widely applied to many realistic scenarios where large-scale data is available.
Two primary targets need to be considered in designing offline RL algorithms: maximizing reward and staying close to the provided data. Finding a policy that maximizes the accumulated sum of rewards is the main objective in RL, and this can be achieved via learning an optimal Q-value function. However, in the offline setup, it is often infeasible to infer a precise optimal Q- Q-value function due to limited data coverage (Levine et al. 2020 Liu al., 2020); for example, the value of states not shown in the dataset cannot be estimated without additional assumptions about the environment. This implies that value learning can typically be performed accurately only for the subset of the state (or state-action) space covered by a dataset. Because of this limitation, offline RL algorithms should implement some form of imitation learning objectives that can force policy to stay close to the given data. Because of this limitation, some ferm of imitation learning objectives that-can force policy stay close to the given data warrants warrants-consideeee consideration in offline-Ri RL.
Recently, many offline RL algorithms have been proposed that instantiate an imitation learning objective without explicitly modeling the data distribution of the provided dataset. For instance, one approach applies the pessimism under uncertainty principle in value learning (Buckman et al., 2020 Kumar et al. 2020; Kostrikov et al. 202 order to prevent out-of-distribution actions from being selected. While these methods show promising practical results for certain domains, it has also been reported that such methods fall short compared to simple behavior cloning methods (Mandlekar et al. Florence et al. 2021) which only model the data distribution without exploiting any reward information. We hypothesize that this deficiency occurs because the imitation learning objective in these methods is only indirectly realized without explicitly modeling the data distribution (e.g. by

---

Under review as a conference paper at ICLR 2023
pessimistic value prediction). Such indirect realization could be much more complicated than simple behavior cloning for some data distributions since it is often entangled with unstable training dynamics caused by bootstrapping and function approximation. Hence, implicit methods are prone to over-regularivation (Kumar et al. 2021) or failure, and they may require delicate hyperparameter tuning to prevent this deficiency (Emmons et al. (022). Yet, the same time, it is ebvious that behaviereeeeerrreeee cannol eannot a from data distrrbutittittoiddooe composs of subeptimal policies.
To this end, we ask the following question in this paper: Can offline RL benefit from explicitly modeling the data distribution via behavior cloning no matter what kind of data distribution is given? Previously, there have been attempts to use an explicitly trained behavior cloning model in offline RI. (u et al. 2019; Kumar al. 2019; Fujimoto et al., 2019 Liu al. 2020), but we argue that two important elements are missing from existing algorithms. Hirst, high fideiity behavior behavior- elening has not been-achieved, despite need in for preeiiieeetiiiaaiieeeeeeee behavier peliey (Levine et al.) 2020). First, high-fidelity generative models have not been integrated with offline RL algorithms even though inaccurate estimation of behavior policy could limit the final performance of the algorithm (Levine al. 2020). FFlorence et al. (202T) have tried an energy- based generative model, but the proposed method is an imitation learning that does not incorporate a value function. Second, wained eloning
chaing eenpelllliiiiiilllllllliee models have only been utilized with heuristies proxy formulations that (Wu et al.] (2019) Kumar et al., (2019). Second, the trained behavior cloning models have only been utilized with certain limited forms, such as KL (Wu al. 2019) or MMD (Kumar et al. (2019) divergence between the cloned policy and an actor policy. Therefore, we tackle these two problems by: first, incorporating state-of-the-arr score-based generative model (Song & Ermon 2019 2020 Song et al. (02T) to fulfill the high-fidelity required for offline RL, and second, by proposing theoretical framework, direct Q-penalization (DQP), that provides a mechanism to integrate the trained behavior model into value learning. Furthermore, DQP can provide an integrated view of different offline RL algorithms, helping to analyze the possible failures of these algorithms.
We evaluate our algorithm on various benchmark datasets that differ in quality and complexity, namely D4RL and Robomimic. Our method shows not only competitive performance across different types of datasets but also state-of-the- thett results on complex contact-rich tasks, such as the transport and tool-hang tasks in Robomimic. The results demonstrate the effectiveness of the proposed algorithm as well as the practical advantage of explicit behavior cloning, which was previously considered a bottleneck that would limit the final offline RL performance (Levine et al. 2020 unnecessary oF infeasible.
To summarize, our contributions are: (1) We provide theoretical framework for offline RI, DQP, which provides a unified view of previously disparate offline RL algorithms; (2) Using DQP, we suggest a principled offline RL formulation that incorporates an explicitly trained behavior cloning model; (3) We propose practical algorithm that instantiates the above formulation, leveraging a score-based generative model; and (4) we achieve competitive and state-of-the--e art performance across variety of offline RL datasets.
RELATED WORKS
The end goal of offline RL is to extract the best possible policy from a given dataset, regardless of the quality of the trajectories that compose the dataset (Ernst al. 2005 Riedmiller 2005 Lange
et al. 2012 Levine et al. 2020). One of the simplest approaches to tackle this problem is imitation learning (IL) (Schaal) [999]] et al. 202T) hoping to recover the performance of the behavior
policy which generated the dataset. However, simple imitation would fail to achieve the end goal of offline RL since one cannot outperform the behavior policy by just imitating it.
This problem is commonly addressed with value learning, trying to resolve the distribution shift problem that arises in the offline setup. Since distribution shift commonly results in overestimation of values, offline RL algorithms try to estimate values pessimistically for out-of-distribution inputs (Kumar et al. 2020; Goo & Niekum 2021), sometimes by explicitly quantifying the certainty with trained transition dynamics model YYu et al. [020 Kidambi et al.] 2020), a generative model (Rezacifar et al.) 202T), or a pseudometric (Dadashi et al.) (002)). The distribution shift is also commonly addressed by constraining policy to be close to the behavior one. Specifically, based on
2

---

Under review as a conference paper at ICLR 2023
how the constraint is instantiated, policy- colliy--onsttaint methods can be further categorized into implicit
methods, which constrain a policy via weighted behavior cloning or linear gradient combination
(Peng et al. 2019 Wang et al. [2200] Kostrikov et al.] 2021b; Branddonbrener et al. [022]] FFjjimoto
& Gu 2021 Wang et al. 2022). and explicit methods (Fujimoto et al. [019)] Kumar et al. 2019)
Wu et al.] 2019: regularization.
Liu et al. 22220), which constrain the policy learning via value penalty or policy
We generally follow the learning structure of the explicit policy-constraint method. Especially, our offline RL algorithm is closely related to the works of Fujimoto et al. (2019): WW et al.] (2019); Liu et al. (2020) in which a behavior policy 3 is explicitly cloned first, and the cloned policy is directly used to instantiate the eelicy-constrain. However, we try to enhance its performance by first, suggesting a principled policy-constraining method that utilizes the cloned policy, and second, using powerful generative model for explicit behavior cloning that can greatly reduce the BC error and thereby allow precise policy regularization.
Concurrent with our work, Wang et al. (2022) propose offline RL algorithm that leverages diffusion-based generative model. Their method based on the works of Fujimoto & Gul (2021) in which the ordinary actor loss is linearly combined with behavior cloning loss. The proposed algorithm is simple and minimalistic, but the theoretical justification of the approach has not been fully addressed.
3 PRELIMINARIES
We use Markov Decision Process (MDP) foundation for our mathematical framework. An MDP defined by a tuple M (S, A. T, do, r, 7): set of states E S, a set of actions a E A, transition dynamics T = p(s'|s, a). an initial state distribution do(so). reward function r(s, a), and a discount factor 7. In this setup, the goal of reinforcement learning is to find an optimal policy rr(aa (als) that maximizes the expected sum of discounted reward (return) (Tr):
H
π²" arg max J(T) = arg max Eycup x
('r(s), ov)
(1)
1=0
where 7 sequence of states and actions (sp. ao.... SH. 0H) SH of length H, and Pa is trajectory
distribution of a policy म, which can be represented as pa (T) = (s II,"
(s,, az).
We can directly optimize the return when we can compute the gradients of J(TØ)) with respect to the policy parameters & (Williams, 1992, Schulman et al., 2017] 2015). but this approach is not straightferwardly extend to offline setting since on-policy data is typically required to compute the gradient. Instead, it is more common for offline RI. methods to extend dynamic programming approaches which are formed around the action value function QF a) which is formally defined
Em=n"" "ase"rr(se,,a) av) The Q function of certain policy mk always implies a greedy policy 1 which is better than or equal to the evaluation target policy ma- Therefore, optimal policy can be found by iteratively evaluating Q function for new greedy policy t until convergence:
as: (T(ss,, az) Ey-ep. (s..a)
QTe lim (BTA) ) 'QQu-- (policy evaluation),
(2)
(als) = arg max, Q** (policy improvement),
where 1 the Bellman operator, which has the ground truth Q" as a unique fixed point (Lagoudakis) & Parr (003). This algorithm is called policy iteration. Although the convergence of the algorithm is
restricted to the scenario where the unique fixed point is reachable (Sutton & Barto 2018), policy iteration has been widely used as backbone for most offline RL algorithms due to its extensibility to off off-policy data; policy evaluation can be done with off off-policy data using bootstrapping. When the value function and the policy represented with parameters 0 and respectively, the policy iteration algorithm with bootstrapping has the following form:
0k+1 + arg ming E, an..n''' d(Qo(s, a),r (eo
(3)
0x+1 t arg max s Eew (s+1()
3

---

Under review as a conference paper at ICLR 2023
where k is update step, d is distance metric such as squared l2 or Huber loss, and D is provided (offline) dataset that contains transition tuples D = (s, a, r, s'). For the brevity of notation, we denote Qb. Qu and Th == k--
4
METHOD
In this paper, we consider offline RL algorithms that utilize the policy iteration scheme shown in Eq. which covers several offline RL methods (Kumar et al.) 2020 2019 Wu et al.] (019). In this family of methods, the correctness of the value function becomes the major concern since the policy evaluation could diverge due to the data restriction imposed by the offline setup. Specifically, divergence can happen because of the bootstrapping and the function approximation; when some estimates are erroncously high due to poor generalization, the over-estimated values are likely to be picked up the policy improvement step and feed back to policy evaluation via bootstrapping, completing a vicious cycle that causes training to diverge. Therefore, offline RL methods focus on solving the overestimation problem with different regularization methods, such as policy constraints 2019 Wu et al. 2019) or pessimism (Kumar et al. 2020 Goo & Niekum 2021).
(Kumar et al.
solution the over-estimation problem is directly penalizing estimation (Reza- eifar et al. 2021 Dadashi et al. 2021) with penalty function p(s,a): Qs(s,a) Qo(s,a) p(s, a). This can be a solution because this penalty function, if chosen carefully, can reduce erroneously high values and prevent them from being propagated via bootstrapping. We refer this family of algorithms as Direct Q penalization (DQP).
One ttraighffoweer
In DQP, can easily observe that the penalty function that describes the oracle Q estimation error (i.e. p(s, a) = Qe - Q=*) is the best solution. Therefore, we want to design a penalty function that resembles the oracle error. Since the estimation error is likely to occur more often for out-of- distribution state-action pairs (s,a') & D. few ad-hoe methods have been proposed for the purpose of measuring the eestimation error or epistemic uncertainty of Qo: the aleatoric uncertainty of the transition dynamics model is suggested as a proxy for the epistemic uncertainty of Qa (Yu et al.) 2020), and generative models (Rezaeifar et al. 202T) or pseudometrics (Dadashi et al.] 2021) are proposed with the purpose of distinguishing whether particular (s, a) is in-distribution or not.
However, it not not been thoroughly investigated how pew-peenlty functions affect the polieypleeeeee iteration process, nor what penalty functions are best for offline Hilline-RL, Without answers te these questions, DQP methods can only be understood as ad-hoe methods in which heuristically designed penalty functions are used te prevent the everestimation. DQP provides a unified way represent different offline RL algorithms in terms of penalty functions. However, in order for a unified perspective to provide a better understanding of offline RL algorithms and thereby be useful for developing better offline RL algorithms, it is necessary to investigate the effect of the penalty function under the policy iteration scheme. To this end, we address the following questions: (1) What is the effect of direct Q- p-enaaizzation in the context of the policy iteration framework?? (2) How can we design an appropriate penalty function based on this analysis?: (3) How can we instantiate the penalty function and achieve strong performance across different offline RI. datasets?
4.1 THEORETIC BACKGROUND ON DIRECT Q-PENALIZATION
We describe a theorem that answers the first question: soft policy iteration (Haarnoja et al. 2018 with penalized value function Q is equivalent to policy iteration regularized by DKt. (s)) m (s)) where (a]ss))) == softmax (-axxx-(pp0 p(s, a)). This theorem is generalized version of the theorem shown in (Rezaeifar et al. (021). which does not require unnecessary assumptions on the penalty function.
Theorem 1 (Equivalence between KL-policy regularization and DQP). ). The The following two algorithms are equivalent.
Policy iteration w/ KL-policy regularization:
0x+1 t- arg min, EDDD (Q((s,)), (s)(s') >DKL (=x((=((=))1TT((()))
Da41 t- magx E.DD (To, 1x+))(s) (
4

---

Under review as a conference paper at ICLR 2023
Soft policy iteration (Haarnoja et al. 2018 w/ penalty:
Ok+1 f- arg ming Es E, d + -D)(s)) Z(s') H (Ta(
@R+1 t- arg max B.-D (To, Qx-1 -p)(s + H (TO((S)
where d a distance metric, (u1, uz) = Ea uI(; a) u))z() a), and (() = In E. exp - p(s.
Proof. The common term in KL policy regularization be rearranged follows:
(T,, (s) DEL (x()))((((p)))
(z, (s) (म, In In (s) + 1.0++lmpp)> (T. (s)
(z, -r, 7(+) + + (=(s))
Then, we can get the equivalence when we replace the term with the rearranged term. Note that the normalization term (s)) can be dropped in the policy update step since (s)) is not function of ø. Also, we can safely ignore (s)) in the policy evaluation step when [2(s Zss)) < e for any pair
of s') E S x S, because it does not affect the policy improvement step.
The theorem straightforward since describes a special case of regularized policy iteration Geist et al. (2019), namely KL-control (Peters et al.] 2010 [Shhulan et al. 20115). in which the policy is regularized through KL-divergence with respect to another policy. Yet, the theorem shows way to interpret any penalty function from the point of view of KL-control and vice versa. Therefore, by using the theorem, we can have a unified view of previously disparate offline RL algorithms. In Table [] compare different offline RL algorithms in terms of the penalty function that each algorithm uses.
4.2 WHAT MAKES A GOOD PENALTY FUNCTION?
Theorem [ shows the connection between a penalty function and its effect as a policy regularizer, and can help to guide the construction of effective, principled penalty function. Specifically,
propose penalty function that can instantiate the support set constraint (Kumar et al. 2019 Liu
al. 2020), which restricts the action space of a trained policy be in the support set of the behavior policy p s). The support set constraint is an effective way to solve the offline RL problem in that the suboptimality caused by the constraint is bounded (Kumar et al., 2019 Liu al. 2020). While previous works express the constraint in terms of the distribution-constrann
Bellman operator, we represent the constraint via a penalty function since allows us to compare different offline RL algorithms under the same viewpoint, helping to analyze the possible failures of these algorithms.
The following penalty function instantiates the support set constraint:
0 for ((saa)]8(als) 2 c)
p(s. a)
oo otherwise
where e is a threshold hyperparameter to decide whether (s, a) is considered out-of-support or not, The penalty function carries the same effect the filtration operator in (Liu et al. 2020) under the policy iteration scheme. This is because the function prevents out- out-of-support actions from being chosen by the policy while it imposes no preference over in- n-support actions; a rare action that has not occurred often in a dataset can be selected as long as it provides a high Q value. This indifference is a desirable property when good trajectories compose only a small portion of a dataset, since good actions could be drowned out by more frequent actions if the penalty function is designed to prefer more frequent actions.
We can also confirm the characteristics of the penalty function by observing the flip side: the penalty- induced policy Tp and the KL-constraint DKL ( [T| [Tp)). Since the reverse KL term makes π seek mode of Tp which is the uniform distribution for the in-support actions. the policy is guided to select one of the actions in the support set while there is no preference over actions in the set. Therefore, the penalty function instantiates the support constraint.
5

---

Under review as a conference paper at ICLR 2023
Given the proposed penalty function, the similarity between different offline RL methods can be observed. For instance, we can see that BRAC-KL and CQL penalize the out-of-suppor actions infinitely: when {{a)s) is zero, the penalty becomes infinite. However, some discrepancies can also be noted, and this provides some hints on how and why other methods could fail due to excessive or insufficient pessimism.
First, BRAC-KL. could fail because it prefers actions that are more frequently executed by the behavior policy. This can be easily seen from the KL-policy regularization perspective. Since Ty(s) - B in BRAC-KL, the penalty would make a policy to seek the mode of {aas)) when the KL regularization term dominates the policy update, Therefore, the algorithm could work like behavior cloning that disregards rare but good actions in the provided dataset. CQL could also exhibit a similar problem since the penalty function is defined with J3. Like BRAC-KI,, when the value function implied policy 1 selects an action that is infrequent in [3, it will be harshly penalized. Especially when CQL is tuned to strongly penalize the -usppor action (i.e., Og is large), it could force the policy to mimic the dataset (Kumar al. 2021 Similarly, TD3+BC and Diffusion RL (Wang et al. 2022 could suffer from the same problem when the penalty strength a is not properly tuned.
Another common problem that arises in other penalty functions is their use of proxies and their formu- lation that replace the () for example, a conditional variational autoencoder (CVAE) (CVAE)(Rezaiifa) Reeaaifaar et al. 202T), a pseudometric (Dadashi al. 202T, transition dynamics model (Yu al. 2020) are estimated instead of the behavior policy 3, and penalty functions are designed heuristically with the proxy estimates. While such formulations could show some positive correlation to the suggested penalty function, there is no clear connection that allows us to interpret the penalty in terms of B or the support set.
BEAR, on the other hand, is designed to implement the support set constraint as ours, so it could avoid the problem of BRAC-KL or CQL that prefers more frequent actions. However, BEAR could be inaccurate since they instantiate the constraint without explicitly modeling the behavior policy 3. Especially, they resort maximum mean discrepancy (MMD) distance since it can be computed only using the samples from a dataset, but the use of the distance metric is only empirically justified (Kumar et al. (2019). In contrast, we directly instantiate the support set constraint by explicitly modeling the behavior policy.
4.3 PRACTICAL IMPLEMENTATION
We now propose practical algorithm that instantiates the penalty function designed above. Essen- tially, the designed penalty function serves to determine whether an action a at a certain state is likely to be executed by the behavior policy 33. Therefore, we can implement the penalty function simply by cloning the behavior policy explicitly and checking the likelihood (als)) with the cloned model.
There have been other research works that have tried to model a behavior policy using generative models, such variational auto-encoders (VAEs) (Fujimoto et al., 2019 Rezaeifar al. 2021). However, the performance of these approaches is limited compared to methods that do not explicitly clone the behavior policy 3. We presume that the reason for this failure is the limited expressivity of the generative model; since the behavior policy 3 can be complex. discontinuous, and multi-modal, only a very expressive model can successfully model the policy. To this end, we chose to use
Table 1: The penalty functions of different offline RL algorithms.
p(s,a) a)
Remark
BRAC-KL
log Balss)
BRAC-MMD2
MMD (TR. MMD3 3)
TD3+BC
a(a ((s)
Wang al. 2022 is also similar.
Anti-Exploration
ala Dec Denc(E((sss
Ene and Dec conditional VAE.
PLOFF
a m1Qu(s, (())((())xx((((e(ee))))))))))))) a))
D is pseudometric.
MOPO
a (s)
Σ is the std. of the trained T.
CQL
a R4 1]
Pa is the soft-policy given Q+-

---

Under review as a conference paper at ICLR 2023
score-based generative model (Song & Ermon 2019 2020 Song et al. 2021 which has recently shown great success in generating high-qualite images. Furthermore, the score-based generative model allows exact likelihood computation which is essential in instantiating the penalty function. We briefly examine the ability of the score-based generative model using four discontinuous multi- modal distributions, and the results are shown in Figure | In all four cases, the inferred probability distribution is very sharp, and its log probability resembles the penalty function we proposed.
In the score-based generative model, target distribution p(z) is indirectly expressed and trained in the form of the gradient of a log probability density function V. log p(æ:), often referred to as the (Stein) score function (Liu al., 2016). When we approximate the score function of behavior policy ((ass) accurately via score- motehing algorithm (i.e. s (alss z Va)(als)) (Song & Ermon 2019), we can instantiate the penalty function with a hyperparameter e which decides whether a certain action a given is considered to be sufficiently in-support or not. While this formulation allows direct instantiation of the penalty function that can be plugged into the DQP framework, computationally prohibitive. This is because, first, we need run an iterative algorithm to compute the log-likelihood from the score function, and second, it could hurt the generalization performance of the value function since Q has to output extremely wide range of values including negative infinity. Therefore, we propose a practical approximation of the policy iteration algorithm that utilizes the proposed penalty function.
The key observation is that the policy trained on top of the penalized value function will never select out-of-support actions due to penalization. This allows two modifications the original policy iteration algorithm: First, we can perform policy evaluation only considering in-support state-action pairs; i.e., we can bootstrap from one of the samples from {(a(s), and we can skip the penalty computation since the penalty is zero for in-support data-points under the suggested penalty function. While sampling using the score function also requires expensive iterative computation, we can greatly reduce the computation by prepopulating samples for states that exist in the dataset and repeatedly using it in the policy evaluation step. Specifically, the policy evaluation is done with the following loss function:
L Leollcc-evall Eggo d Co Qo(o,a),++ K-th at (esipp(((s)))
(Qols'
(5)
where Qğ is slowly updated target network, supp(3) is a set of samples approximating supp(3), and K-th is an operator that selects the K-th item among candidates. When K 1, it becomes max operator. Both Qi and K-th operators are adapted to stabilize the learning.
Second, we can skip the policy improvement step since policy evaluation done with pre- me-geneeatedd samples, not depending on any parameterized policy, Instead, we can define implicit policy using the last Qo and s
exp (aQo(s, a))
exp (aA(s, a))
a (als)) Ed esupp(s) exp (aQo(s,a))
(6)
Σ Esuppo)) exp (aA(s,a'))
where A(s,a) a) = Qo(o,a) (mpp(E)) En Esupp(e)) Qa(s, a') is an advantage function, and a is a temperature parameter that controls the policy softness with regard to the Q or A; when a is zero or infinity, the resulting policy becomes ((ass) or greedy with regard to Qo. respectively. To sample an action from the policy π, we can sample one action from an empirical action distribution consisting of samples generated from 8 on the fly. Alternatively, we can also train parameterized policy T s) using divantage-weighted regression (AWR) (Neumann & Peters 2008 Peng et al. (019) with the advantage function A(s,a).
The resulting algorithm can be regarded as one special type of Q-learning in which we restrict the domain of the maximum operator to in-support actions. We refer to this algorithm as Action- Restricted Q-learning (ARQ). The pseudocode of ARQ is shown in Algorithm [ Also, ARQ can be seen as an extension of MBS-QI QI (Liu et al., 2020 in that makes the existing algorithm applicable to MDP with continuous action space. Note that ARQ is one way to instantiate the penalty function, mainly due to the expensive computational cost of the score-based generative model. We discuss about other possible instantiations in the discussion section.
7

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Action-Restricnn Q-learning (ARQ)
Input:
Iaput: Dataset D = {(s, a, r, s")}. Hyperparameter N, c, K, a Initialize S6 (alss. Qo a), and (a(a))) (if needed)
Train &6 with a score matching algorithm (Song et al. 2021
Sample N in-support actions for E while until convergence do Update 0 with V VeLpolicy-ovall (Eq. 5
D (i.e., (p(a c)
while until convergence do
Update @ with V D.DD (eas(e,e) log To (alx)]] AWR return sg. Qo, and T
EXPERIMENTS
Our empirical goal is to design an algorithm that enjoys the strength of both explicit behavior cloning and value learning. Therefore, the main goal of the experiments is to check whether the proposed algorithm ARQ achieves competitive performance on different types of datasets, ranging from a dataset that consists of near-optimal data in which explicitly cloning behavior is adequate, to dataset containing various suboptimal trajectories in which learning a value function is necessary.
The implementation of ARQ consists of four steps: score-based generative model sy learning, sampling. Q-learning Qe. and optional explicit policy Te training. As for the hyperparameters, we tune the hyperparameters K and for each group of datasets using random search while we N = 30 and e e (i.e., 30 samples are generated and dropped if the likelihood is lower than = 9) all across the datasets tested. For the detailed implementation details and hyperparameters, please refer Appendix A.2 or the provided code
The proposed method is evaluated on various simulated benchmark datasets from simple low- dimensional locomotion tasks to complex contact-rich manipulation tasks. Specifically, we use locomotion (Brockman al. 2016). Adroit (Rajeswaran et al. 2018). Kitchen (Gupta et al.) 2019). and Antmaze tasks in D4RL (Fu et al. 2020), and six manipulation tasks in Robomimic (Mandlekar al., 202I). We use medium-replay, medium, expert, and medium expert datasets of the locomotion task. We use machine- geneerne-eeeertedd (mg.), proficient-humar of Robomimic, each of which consists replay buffer of an SAC training run, trajectories of a proficient human demonstrator, and trajectories of multiple human demonstrators with different levels of proficiency.
(ph.), and multi-human (mh.) datasets
We compare the proposed method to behavior cloning baselines, specifically ordinary BC and implicit behavior cloning (Florence et al.) 2021). and prior state-of-thh-an offline RL. methods. Namely, we compare the performance of our method with TD3+BC (Fujimoto & Gu (021), Decision Transformer (DT) (Chen et al., 2021), One-step RL. (Brandfonbrener et al.) 2021)) CQL (Kumar et al.) 2222). and IQL Kostrikov al. 2021b). The aggregated results are displayed in Table
The proposed algorithm ARQ shows competitive performance on all ranges of datasets, from near- optimal ones in which simple behavior cloning is sufficient, suboptimal datasets in which value learning is necessary. Also, ARQ exhibits state-of- performance on complex and contact-rich tasks, such as adroit, kitchen, and Robomimic datasets. The results indicate the practical effectiveness of the proposed algorithm, as well as the advantage of performing behavior cloning explicitly with high-fil highffidelity models.
To examine the importance of each component in ARQ, we run two ablations; we evaluate an implicit policy defined only with ss without any value function, and an implicit policy incorporating Q" instead of ARQ. The results of the ablations affirm our hypotheses. First, when the dataset is near- optimal (e.g., adroit-human or proficient-hume hummn datasets), explicitly modeling the behavior policy can address the problem, and similar performance is obtained when we use ARQ. Next, we confirm the necessity of value learning and the ability of ARQ in leveraging the explicitly cloned behavior model in learning value function. Especially in the tasks where trajectory stitching is required (e.g.. kitchen and antmaze datasets), we can see performance improvement from Sy and Q(3) + s6 to ARQ+ Sp. and we achieve state-of-the-ard performance with the help of the explicit models.
Please the attached supplementary files. The code will he disclosed to public upon publication.
8

---

Under review as a conference paper at ICLR 2023
Table Aggregated performance of prior methods, ours, and ablations (sy and Q(3) + sw) on D4RL (Fu et al. 2020) and Robomimic (Mandlekar et al. 2021) datasets. Each number represents the mean relative performance over 100 episodes. 0 and 100 represent the performance of random and expert policy, respectively. Unless noted as (ours) or (repro.), all the numbers are borrowed from Kostrikov et al. (2021b); Fujimoto & Gu (2021); Mandlekar al. (2021). The numbers generated by us are averaged over 3 different random seeds. We run IQL on Robomimic by ourselves using the
author- author-provideee provided implementation.
Without reward
With Qo
With reward value function
Q(3)
ARQ
ARQ
Impl. BC
One-
TD3 +BC
BC
(ours)
step
(ours)
+8g
DT
CQI.
IQI.
(ours)
+1g
(ours)
+5p
locomtion-v2 (total)
739
521
639
1007
1000
947
adroit-v0 (total)
105
116
160
94
118
95
161
human-vO (total)
67
99
87
89
52
77
45
90
cloned-vO (total)
37
29
62
71
42
41
50
71
kitchen-v0 (total)
155
160
160
204
complete
65
85
74
75
44
63
37
77
partial
38
38
45
59
50
46
50
70
mixed
52
38
51
51
51
39
S7
antmaze-v0 (total)
100
121
125
215
112
164
304
378
416
327
umaze
55
59
88
umaze-div.
46
61
53
62
58
med.-play
71
69
med.-div.
0
70
65
large-play
0
40
18
large-div.
0
48
23
D4RL (total)
1,099
1,046
1,472
1,538
1,663
1,637
1,639
Q(13)
ARQ
ARQ
BC
(ours)
8g
+8g
BCQ
CQL*
(repro.)
IQL
+
+5p
(ours)
(ours)
(ours)
robomimic (total)
701
644
749
592
281
342
659
750
mg.-lift
65
29
86
91
64
79
79
82
mg.-can
65
19
62
60
ph.-lift
100
99
100
93
58
100
98
ph.-can
95
95
26
95
ph.-square
79
24
69
ph.-transport
17
1
30
ph.-toolhang
3
71
mh.-lift
100
51
95
mh.-can
86
25
86
mh.-square
53
44
12
51
mh.-tranpoort
D4RL + robomimic
1,799
1,690
2,221
1,818
2,005
2,296
2,389
represents that the best performance during training iterations picked posterioni
It is also noteworthy that the ablated method with Q3 shows competitive results large number of benchmarks. Echoing prior research (Goo Niekum 2021 BBandfonbrener et al. (02T), these results indicate that the vast majority of offline RL benchmarks
be resolved without iterative value learning, while most offline RL algorithms tackle problems that arise from Therefore, in order fairly evaluate the offline RL algorithms and thereby foster the advance of the offline RL field, it essential to focus on environments that require value learning (e.g., antmaze) or develop new benchmarks.
DISCUSSION
We investigate an offline RL algorithm that combines explicit behavior cloning and value learning. We provide theoretical framework. DQP, which enables various offline RL algorithms to be expressed in terms of different penalty functions, and we derive a principled penalty function that can leverage behavior cloning model. Then, we provide a practical algorithm, ARQ, which realizes the derived

---

Under review as a conference paper at ICLR 2023
penalty function. We implement the algorithm with powerful generative model maximize the full potential of ARQ. As a result, the proposed algorithm shows competitive results on most of the D4RL and Robomimic benchmarks and yields state-of-the-arr results in several tasks. This indicates the hetp:aaeeeccceeeeeeeeeeeeeeeeeee thatieinnn-------e it weteeeeeaeeeryeeeee or i--infaasible to-stiimttttttteeeeeeeeeee a hehaaiierpellee--te behavier
effline-RI likely incorret= These results indicate that explicitly cloning a behavior policy can be actually advantageous, which has been avoided because of the performance limitations that can arise from inaccurate modeling of the policy.
The major drawback of the proposed algorithm is the computationally expensive sampling procedure. While it needs to be computed only once before the value learning step, it can take several hours to generate samples (90K samples are generated per hour on our in in-house workstation with an Nvidia GTX 1080 Tí). Therefore, future research may examine how to reduce computational burden, for instance, by using different generative models for behavior cloning. Or, it may be possible to devise a method that directly utilizes the score function under the actor-critic framework; since the actor update step with the penalized Q function ( - Q p) only requires a gradient of p, not an exact penalty value, the computational bottleneck may be avoided if the gradient of the penalty function can be computed directly from the score function.
REFERENCES
David Brandfonbrener, William F Whitney, Rajesh Ranganath, and Joan Bruna. Offline RL without off-policy evaluation. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL fattpa://openreviee.
sett/forummid-LU687EE
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/ abs/1606.01540, 2016.
Jacob Buckman, Carles Gelada, and Mare G Bellemare. The importance of pessimism in fixed-dataset policy optimization. arXiv preprint ar arXiv:2009.06799 2020.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34, 2021.
Robert Dadashi, Shideh Rezaeifar, Nino Vieillard, Léonard Hussenot, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning with pseudometric learning. In ICML, pp. 2307-2318, 2021,
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline RL via supervised learning? In International Conference on Learning Representations, 2022. htt.attss//ppeeevi..ee
net/ wttp://orommmm.--///........
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 2005.
Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In 5th Annual Conference on Robot Learning, 2021. URL TtRLhhtps:://oot /ttttp:://operrvviee wwt. forum?
id=ri F3a5fx365
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Thirty-Fifh Conference Neural Information Processing Systems, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062, 2019.
Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. In International Conference Machine Learning, pp. 2160-2169, PMLR, 2019.
10

---

Under review as a conference paper at ICLR 2023
Wonjoon Goo and Scott Nickum. You only evaluate once: simple baseline algorithm for offline RL. In 5th Annual Conference on Robot Learning, 2021. URUhttips://ppeneeiiee net/
Eorum?1d=GlgsuNIIE
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long horizon tasks via imitation and reinforcement learning. Conference on Robot Learning (CoRL), 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pp. 1861-1870, 2018. URL http:://proceddnggs mlr /
w800haarrojallB.ff
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
Aapo Hyvärinen and Peter Dayan, Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims, Morel: Model- based offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Bal- can, and H. Lin (cds.), Advances in Neural Information Processing Systems, volume 33, pp. 21810-21823. Curran Associates, Inc., 2020. URL BRLLhtttps::/prrceeding neur iwprrpp..coo cc/
Diederik P Kingma and Max Welling. 1rXiv:1312.6114, 2013.
Auto-encoding variational bayes.
arXiv preprint
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference Machine Learning, pp. 5774-5783. PMLR, 2021a.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021b.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems 32, pp. 11784-11794, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative g-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0477 2020.
Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. A workflow for offline model-free robotic reinforcement learning. In 5th Annual Conference on Robot Learning, 2021. .nt/ wtw.://www..ne/ee.riiii.e... id E
URL httpt:::/
Michail G Lagoudakis and Ronald Parr. Least-squares policy iteration. The Journal of Machine Learning Research, 4:1 107-1 1449, 2003.
Sascha Lange Thomas Gabel, and Martin Riedmiller. Batch reinforcement learninn In Reinforcement learning, pp. 45-73. Springer, 2012.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement Icarninng Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0166 2020.
Qiang Liu, Jason Lee, and Michael Jordan. A kernelized stein discrepancy for goodness-of-ff tests. In International conference machine learning, pp. 286-284. PMLR, 2016.

---

Under review as a conference paper at ICLR 2023
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch reinforce- ment learning without great exploration. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang. Rohun Kulkarni, Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin What matters in learning from offline human demonstrations for robot manipulation. In 5th Annual Conference on Robot Learning, 2021. URL htt.:httss////t httptttsss//ppnnreeeeee few fortt/form?
Gerhard Neumann and Jan Peters. Fitted q-iteration by advantage weighted re- gression. In Advances in Neural Information Processing Systems, volume 21, URL E $7992lbbae40a57792228 oat.aeeeeees71110ooooeeeeeeeeeee Paper.
2008.
neeriips wttpp/ape/////////LLLLL hrtpoeeddigs..purrrsss..///rrrrp
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighttd regression: Simple and scalable off off-policy reinforcement learning, arXiv preprint arXiv:1910.00117 2019.
Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In Twenty-Fourts
AAAI Conference on Artificial Intelligence, 2010.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018.
Shideh Rezacifar, Robert Dadashi, Nino Vieillard, Léonard Hussenot, Olivier Bachem, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning as anti-exploration, 2021.
Martin Riedmiller. Neural fitted q iteration-first experiences with a data efficient neural reinforcement learning method. In European conference on machine learning, pp. 317-328. Springer, 2005.
Stefan Schaal, Is imitation learning the route to humanoid robots? Trends in cognitive sciences, 3(6): 233-242, 1999.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 1195-11107, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference Learning Representations, 2021. httptttsss:/ppeeeeeeeeeeee
net www.:eerttffrtt
1dd=xXIGG2RRRH
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cam- bridge, MA, USA, 2018 ISBN 078-0262039246.
net/book/the-bon theookk/theeee book 2nd html
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computa-
fion, 2367):1611-1674, 2011.
Zhendong Wang. Jonathan Hunt, and Mingyuan Zhou. Diffussio policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06/99 06/33, 2022.
12

---

Under review as a conference paper at ICLR 2023
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processsig Systems, volume 33, pp. 7768-7778. Curran As- sociates, Inc., 2020. URL htttts:: ptcppppss/ www.:es.ennrrrs..cccccccc pat.pcccpppppppa///////1/1e 3B
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-22 222,, 1992.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning.
arXiv preprint arwiv:1911.11... 2019.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. arXiv preprint 20X. 2020.
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A. SCORE-BASED GENERATIVE MODEL. (SONG& & ERMON 2019: 2020: SONG ET AL.. 2021
In the score-based generative model, target distribution p(z) is indirectly expressed and trained in the form of the gradient of a log probability density function V, log p(z), often referred to as the (Stein) score function (Liu et al. 2016). This method circumvents several problems of other generative models. The main advantages are, first, we can circumvent the problem of the inferring normalizing constant that arises likelihood-baat -base methods (Kingma & Welling 2013 LeCun et al. 2006), and second, we can train the score function without worrying about training instability that arises in adversarial training Goodfellow et al. (0144) via score- score-matching algorithms Hyvärinen & Dayan (2055); Vincent 2011); the score-matching algorithm minimizes the gap between ground- truth score function and the estimates: Tywrr matching = pxp)) [| V log p(æ) Sw [1)) Since the objective is essentially regression with l2 loss, the loss function does not require any assumptions on the parameterized function sv, unlike the energy-based model LeCun et al. (2006) in which strong regularization is often required for stable training. These advantages make it possible model complex behavior policy / with the high fidelity that offline RL requires. We briefly examine the ability of the score- based generative model using four discontinuous multi-modal distributions, and the results are shown in Figure A. In all four cases, the inferred probability distribution is very sharp, and its log probability resembles the penalty function we proposed.
A.2 IMPLEMENTATION DETAILS
The implementation of ARQ consists of four steps: score-based generative model 8y learning, sampling. Q-learning, and optional explicit policy T training. For the score-based generative model, sampling, and likelihood computation, we generally follow the implementation of (Song et al. 2021) which trains time-dependent neural network that approximates the reverse-time stochastic differential equations (SDE); neural network learns to reverse the progressive diffusion process that turns data point into random noise. Specifically, we use value- preserving SDE (VPSDE) with a neural network having residual connections (He al. 2015) with the embedding size of 256, and we stack 3 residual blocks. We use swish for nonlinearity. Since our target distribution is conditional (i.e., conditioned on a state) unlike the original formulation, we extend a time-dependent neural network to be function of state, action, and time. We train the network with Adam with a learning rate of lc-4 and batch size of 512, and we apply an exponential moving average with an average coefficient of 0.999. Since the number of training data samples varies across datasets, tested the different number of training iterations and ensembles to prevent overfitting. For training iterations, we tested 150,000, 300,000, and million steps, and for ensemble, we tried single and ensemble models. For ensemble training, we train each model with different random samples from the same data pool. The speccfc- hyperparameters are shown in Table A.I
The second step of the ARQ algorithm is prepopulating samples for value learning. We use predictor-corrector algorithm (Song et al. (2021) to generate samples by solving a reverse SDE. Again, we followed the implementation of Song et al. (202T). which uses the Fuler-Maruyama method as a predictor and Langevin dynamics as corrector. We discretize the time domain [1e-3,1] of
step
piecewise
split_cirzle
spiral
12-
10-
02
0.8
08-
15
06-
00-
92
07-
0.4
-05
0.8
4.0
07
1..0
0.0
0.5
100
Figure A.I: The probability estimated by the trained score- generative model for complex, discontinuous, and multi-modal conditional distributions ((als). The state and action spaces one-dimensional, and the conditional probability estimated using the model sg. is represented with a color map. Red x marks represent the training samples.
14

---

Under review as a conference paper at ICLR 2023
Table A.1: -ppesct- hyperparameters used in training a score function Sy..
D4RL
Robomimic
Locomotion
Kitchen
Adroit
AntMaze
MG
PH. MH
Training Iterations # Ensembles
1,000,000
300,000 1
300,000 3
1,000,000
1,000,000
150,000 3
1
3
So into 500 steps, and we execute single corrector step for every predictor step. For Langevin dynamics, we dynamically adjust the noise scale using the norm of the score; we use a score-to-noise ratio of 0.16 used in Song al. (202T). Since the suggested penalty function is defined based on its probability {(a[s), we need to compute the likelihood of generated samples using Sg. For the likelihood computation, we use an instantaneous change-of-variabe formula on top of the ordinary differential equation induced from the SDE. To solve the inverse problem of ODE, we use the RK45 algorithm of scipy. For the detailed formulation and algorithmic detail regarding SDE, we refer the original paper (Song & Ermon 2019: 2020, Song et al. 2021 our code. We use = across all experiments, and we drop any samples that show a lower probability than the given threshold e. We generate 30 samples for both ARQ and the implicit policy using S6-
For ARQ training, we use an MLP with 2 layers and 256 activation nodes to parameterize the value function Qg. and apply ReIU nonlinear activation. Also, shape the reward function of datasets following (Kostrikov et al. 2021b); for the locomotion tasks, the reward is normalized by multiplying the ratio of returns between the worst and the best trajectories in the dataset, and for the antmaze tasks, the reward is set to 1 except the goal state. Similarly, we densify the reward function of Robomimic using the same technique used for the antmaze. For stability, we train two functions and slowly moving target network with a polyak coefficient of 0.995. We train I million timesteps using Adam optimizer and batch size of 512. We perform a rough random search with the following range of values for the following ayperparameters: K E [3, 6.9). 9). a learning rate E [3e-4.le-4]. The chosen hyperparameters are shown in Table A.2 For the Q used the ablation study, we use the same parameterization of the value function, but we train a single Q function with a slowly moving target network. The value function is trained million time steps using Adam with a learning rate of le-4 and batch size of 512.
Table A.2: Dptaset-specific hyperparameters used or in ARQ.
D4RL.
Robomimic
Locomotion
Kitchen
Adroit
AntMaze
MG
PH, MH
Learning rate
3e-4
le44
le-4
3e-4
le-4
le-4
K-th Reward
9
9
9
3
9
9
Normalized
Original
Original
-1 except goal except goal
For the implicit policy that is based on the samples of sg. we first generate 30 samples using 8y, then resample action from the categorical distribution that treats advantages logits. Similarly, for the explicit policy, we train a policy using weighted behavior cloning (Peng et al. (019)) where the weight computed using the advantage. We use state-independent stochastic policy used in (Kostrikov et al., (0215) for the locomotion, kitchen, and adroit tasks of D4RL datasets, which predicts the mean u(aass s) and the indeeeneddnnn standard deviation (a) of a Gaussian distribution. We use a 2-layer MLP having 256 hidden units with ReLU activations. For the antmaze tasks, we use the deterministic policy that omits the standard deviation prediction. Similarly, a deterministic policy is used for the Robomimic datasets, but we use dense Resnet blocks for the parameterization. We stack ResNet blocks, each of which has 2048 embedding dimensions. We tried Gaussian Mixture Network (GMM) as suggested in (Mandlekar et al.) 2021), but we could not replicate the reported performance in the BC setting. For D4RL and Robomimic, we train a policy for million and 300,000 steps respectively, using the Adam optimizer with a learning rate of 3e-4. The key hyperparameter for the policy is the temperature term a. We tested the following range of values: [0.1, 1.0, 10.0, 30.0], and we display the chosen values in Table A.3 along with other hyperparameters.
15

---

Under review as a conference paper at ICLR 2023
Table A.3: Dataset-specific hyperparameters used in the implicit policy and the explicit policy Te.
D4RL
Robomimic
Locomotion
Kitchen
Adroit
AntMaze
MG
PH, MH
for Q* + 5v a for ARQ sp
10
10
0.1 0.1
10
10
1
State-independent stochastic
T Training Iterations a for ARQ TΦ
Det.
Det.-ResNet
1,000,000
300,000
10
10
10
0.1
A.3 FULL EXPERIMENT RESULTS
We display the full experiment results in Table A.4
Table A.4:
Performance of prior methods and ours on D4RL et al. 2020) and Robomini(Mandlekar al. (202T) datasets. Each number represents the performance relative to random policy as 0 and expert policy as 100. Unless noted as (ours) or (repro.), all the numbers are borrowed from Kostrikov et al. (022bb), Fujimoto & Gu (202I), and Mandlekar et al. (202T). The numbers generated by us averaged over 3 difterent random seeds. The standard deviations of multiple runs are also displayed.
Without reward
Q"
With roward valuue function
BC
Impl. BC
Ag (onas))
One-
step
(ours)
Q(3)
DT
AWAC
TDJ -BC.
CQL.
1QU.
(surs)
ARQ
(curs)
ARQ
expert-12
hopper
112
110
39
(5.1)
99
(2.1)
85
108
111
110
111
(0.2)
98
(0.51
walker
56
82
106
(0.6)
107
(0:4)
57
110
104
110
109
(0:4)
108
(0.1)
halfchiah
105
81
86
(1.1)
79
95
(0.2)
85
(0.5)
medium-v22
hopper
38
57
66
58
(0.31
78
81
med med--exp-v2
hopper
45
92
(0:3)
(2.5)
walker
74
(2.8)
107
(0.2)
75
110
(0.5)
(0.4)
haffehnah
55
35
46
(0:4)
77
(1.83
87
43
91
92
87
91
(U.7)
81
00.31
mod--mp-22 -2
hmpper
11
(0.3)
98
62
(0.9)
81
37
61
95
RI
(24.2)
78
(3.21
walker
26
10
20
(0.3)
50
69
(2.0)
67
27
82
77
74
66
(7.0)
71
(1.2)
halfchnah
25
26
(0.8)
38
41
(0.3)
41
45
44
(0.5)
41
00.31
human-00
84
76
73
(1.9)
72
(5.2)
74
(1.3)
04.90
(1.0)
closed-vII
pen
(7.1)
(2.7)
dour
(0:4)
11
(2.0)
(0:4)
(4.4)
relocate
(0.0)
0
(0.0)
(0.0)
00.00
hammer
(0.2)
(0.3)
(0.3)
(2.6)
kitchen v0
complete partial
65
85
74
(3.7)
75
(1.2)
44
63
37
(14.2)
72
(1.8)
38
38
45
(2.7)
59
(4.9)
46
50
(5.0)
70
(2.7)
mixed
51
52
51
57
(2.9)
unmmaze-v
55
58
(2.13
81
(4.5)
59
57
88
(0.8)
94
(1.7)
49
58
(7.0)
med--ply
(15.6)
large-play
(0.5)
(17.7)
(1:7)
lango-div
(0.00
(0.5)
48
(6.2)
23
(5.0)
BC
(onas)
Ag
(ouns)
Q65)
BCQ'
CQL
(regeo.)
IQL
(ours)
6%%
(ours)
machine gon.
lift
65
29
(2.4)
86
(0.5)
79
(1.2)
82
(0.5)
can
65
19
(2.4)
55
(6:5)
62
(0.5)
60
(1.2)
pro. humas
lift
100
99
(0.8)
100
(0.5)
100
93
58
100
(0.0)
98
(0.0)
95
95
(0.8)
93
(2:4)
38
26
(2.2)
95
(0.5)
79
66
(2.90
72
(4.0)
24
(2.2)
69
(3.4)
27
(5.4)
Vool-Jung
(5.9)
96
51
(2.5)
(1.7)
(4.8)
tmanaport
15
(3.1)
17
(2.6)
(4.9)
(1.7)
represents that the bes perfontance during training Serations picked postasior.
16


Reviews:
[{"review_id": "s6I3mVNH2nM", "reviewer": "Reviewer_CL7N", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies if explicitly modeling the behavior policy is beneficial in offline RL. Focusing on offline RL methods based on value function penalizations, the authors argue that multiple previous methods all design the penalty in a way that reduces the chance of overestimating the value of out-of-support state-action pairs. Based on this observation, the authors hypothesize that a good penalty term should prohibit out-of-support state-action pairs. Based on this penalization scheme, the authors train a score-based generative model to guide an approximated policy iteration algorithm. On standard offline RL benchmark datasets, the authors show that the proposed method could outperform previous methods that do not explicitly model the behavior policy. ", "summary_of_the_review": "I'm inclined to accept this paper because the paper is clearly presented and the proposed method is well-justified. ", "strength_and_weaknesses": "### Strength\nThe paper is clearly presented and the proposed method achieve a good performance. \n\n### Weaknesses\nI didn't notice any big weakness. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pRyP5GhLdzV", "reviewer": "Reviewer_8jQf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new offline RL algorithm that directly models the behavior policy with SOTA score-based generative models and use the learned behavior policy as an explicit constraint in offline RL. The authors argue that explicitly modeling behavior policy can avoid failure cases where previous methods with implicit behavior policy constraints seek the mode of the behavior policy and ignore rare but useful actions. Empirically, the methods outperform prior methods in D4RL and Robomimic datasets.", "summary_of_the_review": "Based on the comments above, I think I would to vote for a weak accept and it would be great if the authors address some of the concerns in the sections above.", "strength_and_weaknesses": "Strength:\n\n1. The idea of using a powerful generative model for learning explicit behavior policy to mitigate the issue of simply seeking the mode of the behavior actions is reasonable and intuitive.\n2. The empirical results are strong and show that the method can outperform prior BC and offline RL methods on D4RL (especially antmaze and kitchen tasks that require stitching) and Robomimic (where the data is mostly demos) tasks.\n3. The paper is clearly written and easy to follow.\n\nWeaknesses:\n\n1. I think the main point of using an explicit behavior policy is to avoid seeking the mode, overly mimicking the dataset, and not relying a proxy. However, these points are mostly based on intuition without clear empirical or theoretical evidence/justification. Theorem 1 simply shows the equivalence between soft policy iteration with penalty and policy iteration with KL regularization, which is less relevant to the main point of the paper. Therefore, I think the real justification of why this approach should be preferred is somewhat missing.\n2. The empirical results are nice, but I think the authors should compare to [1]. It is unclear if we just need a more expressive policy or we really need to model the behavior policy.\n\n[1] Wang, Zhendong, Jonathan J. Hunt, and Mingyuan Zhou. \"Diffusion policies as an expressive policy class for offline reinforcement learning.\" arXiv preprint arXiv:2208.06193 (2022).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VcjDO1pp1qH", "reviewer": "Reviewer_8gEu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an offline RL method that combines a BCQ-style critic update (sample $k$ actions from $\\beta(a \\mid s)$ and take the max) and an AWR-style actor update. One important detail is that the behavioral policy $\\beta(a \\mid s)$ is fit using a much more powerful model class (a score-based generative model) than prior work. Empirically, the proposed method outperforms IQL and other baselines on the majority of tasks from D4RL and robomimic.", "summary_of_the_review": "Overall, I think this paper is studying an interesting problem (how powerful explicit behavioral models can be used for offline RL). While the current experiments and writing have a number of limitations (see above),  I think that a revised version of the paper would make for a strong submission to a future conference.", "strength_and_weaknesses": "Strengths\n* Empirical results -- The empirical results are quite strong. If it is the case that these benefits are coming solely from the more powerful generative model (rather than the new offline RL algorithm), I think that would be a pretty interesting result. One reason it's surprising is that the policies that collect these datasets are usually pretty simple, so it's counterintuitive that we'd need really powerful density models to learn the behavioral policy.\n* It's great that the paper released code! \n\nWeaknesses\n* Relationship with prior work -- The paper does not sufficiently discuss the similarities with prior methods. As noted below, I think many of the claims about prior work are not true, and the proposed method is very similar to prior work. The strong empirical results suggest that this is simply a matter of writing -- situating the method relative to prior works, noting that the method is actually very similar to prior work (perhaps don't claim that as a contribution), and arguing that the score-based generative model is the important contribution. One way this could be done is to *not* propose any new method, but show that using a more powerful model for estimating $\\beta(a \\mid s)$ can be used to improve *existing* offline RL methods. In fact, the $s_\\psi$ and $Q(\\beta) + s_\\psi$ columns in Table 2 already do this.\n* Correctness -- I'm not sure about some of the arguments in the paper (see details below). Again, I think this is purely a problem with writing, but I suspect that many parts of the paper will have to be rewritten.\n* The proposed method seems to require per-domain hyperparameter tuning.\n\n\nThere are a number of claims that I'm unsure about.\n* \"behavioral cloning cannot extract a good policy\" -- This seems to contradict the earlier claim in the introduction that certain off-policy methods \"fall short compared to simple behavioral cloning.\"\n* \"high-fidelity behavioral cloning has not been achieved\" -- Cite. Note that IBC and [1] might be counterexamples.\n* \"despite the need in offline RL for precist estimation of behavior policy (Levine)\" -- Where does (Levine,) argue this?\n* \"have only been utilized with heuristics or proxy formulations\" -- I don't think this is true. Many prior works (including those cited) do provide theoretical justification for the proposed methods.\n* \"which was previously considered unnecessary or infeasible (...) \" -- I don't think these citations say this. Also, this countradicts the claim in the previous paragraph that (Levine 2020) argues for high-fidelity behavioral cloning.\n* Related work -- I think some of the categorization here isn't accurate. For example, some of the methods in the first paragraph do \"value learning,\" and arguably all of the methods in the first paragraph \"try to resolve the distribution shift problem\"\n* Related work, method -- I'd recommend acknowledging the large body of offline RL work that came before 2019. Much of it is referred to as \"batch RL\" rather than \"offline RL\".\n* \"$p(s, a) = Q_\\theta - Q^{\\pi_\\phi}$\" -- This is a neat idea, but seems different from what the rest of the paragraph discusses. Prior methods that estimate uncertainty are estimating the _magnitude_ of this difference, but not the _sign_ of the difference.\n* \"it has not been thoroughly investigated\" -- I don't think this is true. There is a fair bit of work in this area. For example, BRAC compares a bunch of different divergence measures.\n* \"can only be understood as ad-hoc methods\" -- I don't think this is true. Many offline RL papers (including those cited) include theoretical motivations for the proposed method. \n* \"where $\\pi_p(a \\mid s) = \\text{softmax}(-p(s, a))$ -- Is the $p(s, a)$ here different from the one defined in paragraph 3 of the method section? \n* Theorem 1 -- I think this is trivial, as KL control is a special case of regularized policy iteration (see, e.g., [2]).\n* \"K-th is an operator\" -- Is this only well defined for discrete actions?\n* Eq. 6 -- This doesn't integrate to one. I think the numerator is missing a term $\\delta(a \\in \\text{supp}(\\beta))$.\n\nMinor writing comments\n* Abstract -- it's a bit unclear how the proposed method is different from prior methods that add explicit behavioral cloning (e.g., BCQ, TD3+BC).\n* \"warrants consideration\" -- the passive voice is to be avoided.\n* \"attempts to use an explicitly trained behavioral cloning model in offline RL\" -- I think there are many more citations for this.\n* \"Offline RL is about exploitation\" -- I'm not sure this is true.\n* \"are preferred over\" -- Who prefers them? Add a citation\n* \"utilizes the half side of the information\" -- I didn't understand this.\n* \"support the idea\" -- What idea?\n* \"stabilizing the learning\" -- Does this refer to the learning of value-based methods?\n* \"value-based methods, which can be categorized as\" -- I think \"which\" is referring to the wrong noun\n* \"instantiating it can\" -- What does \"it\" refer to?\n* \"straightforwardly\" -- Cut.\n* $s_\\psi(s, a)$ -- It'd be good to formally state what this is defined to be (the gradient of ...)\n* AWR -- Cite Neumann 2008 [3], which proposed a similar method much earlier.\n* Table 2 -- I'd recommend positioning this Table one page earlier. Clarify what $s_\\phi$ is.\n* Figure A.1 -- It's hard to see the predicted density because it's occluded by the red Xs\n\n[1] https://arxiv.org/abs/2208.06193\n\n[2] https://proceedings.mlr.press/v97/geist19a/geist19a.pdf\n\n[3] https://proceedings.neurips.cc/paper/2008/file/f79921bbae40a577928b76d2fc3edc2a-Paper.pdf\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ArBmZUqZQYG", "reviewer": "Reviewer_piV5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new offline RL algorithm--ARQ--that explicitly models the behavior policy and uses it in a support-constraint in the Bellman backup of the algorithm. The authors make the following contributions:\n\n(1) Relate many existing offline RL algorithms as using different penalty functions in the Bellman backup.\n\n(2) Propose ARQ, that instantiates the penalty function as a support constraint.\n\n(3) Evaluate ARQ and ablations against existing BC and offline RL algorithms.", "summary_of_the_review": "Though I believe ARQ itself is quite similar to existing ones, and I am not sure why those differences may be advantageous, the algorithm achieves surprisingly good empirical performance. I believe that the paper demonstrates that modeling the behavior policy can be done effectively, and improve upon offline RL algorithms that explicitly avoid this. This, to me, is a surprising and novel finding. Because of this, I recommend that the paper be accepted.", "strength_and_weaknesses": "Strengths:\n\n(1) The theoretical unification of existing offline RL algorithms is useful.\n\n(2) The algorithm achieves strong empirical performance on many different tasks, suggesting that modeling the behavior policy is not as difficult as previously anticipated. This opens the door for many new possible offline RL algorithms.\n\nWeaknesses:\n\n(1) The algorithm requires sampling many actions to compute the Bellman backup, which makes it much slower than existing offline RL algorithms. The authors do point this out as a limitation, however.\n\n(2) The algorithm is similar to many existing offline RL algorithms that use a support-constraint. Namely, MBS-PI in [1] looks extremely similar to me. The one difference I noticed is that the former performs policy-iteration, whereas ARQ uses AWR to extract a policy.\n\n[1] https://arxiv.org/pdf/2007.08202.pdf", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject