State-Action Joint Regularized Implicit Policy for Offline Reinforcement Learning
Authors: Shentao Yang
Zhendong Wang
Huangjie Zheng
Mingyuan Zhou
Venue: ICLR 2022 Submitted (2022)
Keywords: Implicit Policy
State-action Visitation
Distribution Matching
Generative Adversarial Networks

Abstract: Offline reinforcement learning enables learning from a fixed dataset, without further interactions with the environment. The lack of environmental interactions makes the policy training vulnerable to state-action pairs far from the training dataset and prone to missing rewarding actions. For training more effective agents, we propose a framework that supports learning a flexible and well-regularized policy, which consists of a fully implicit policy and a regularization through the state-action visitation frequency induced by the current policy and that induced by the data-collecting behavior policy. We theoretically show the equivalence between policy-matching and state-action-visitation matching, and thus the compatibility of many prior work with our framework. An effective instantiation of our framework through the GAN structure is provided, together with some techniques to explicitly smooth the state-action mapping for robust generalization beyond the static dataset. Extensive experiments and ablation study on the D4RL dataset validate our framework and the effectiveness of our algorithmic designs.

Under review as a conference paper at ICLR 2022
STATE-ACTION JOINT REGULARIZED IMPLICIT POL- ICY FOR OFFLINE REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Offline reinforcement learning enables learning from a fixed dataset, without fur- ther interactions with the environment. The lack of environmental interactions makes the policy training vulnerable to state-action pairs far from the training dataset and prone to missing rewarding actions. For training more effective agents, we propose a framework that supports learning a flexible and well-regularized pol- icy, which consists of a fully implicit policy and regularization through the state- action visitation frequency induced by the current policy and that induced by the data-collecting behavior policy. We theoretically show the equivalence between molicy-matching and tite-aatioo-viiiaation matching, and thus the compatibility of many prior work with our framework. An effective instantiation of our frame- work through the GAN structure provided, together with some techniques to explicitly smooth the state-action mapping for robust generalization beyond the static dataset. Extensive experiments and ablation study on the D4RL, dataset val- idate our framework and the effectiveness of our algorithmic designs.
INTRODUCTION
Offline reinforcement learning (RL), also known batch RL, aims training agents from previously-collecte fixed datasets that are typically large and heterogeneous, with a special empha- ct al.
sis on no interactions with the environment during the learning process (Ernst et al.
2005
Lange
2012 Fujimoto et al. [2019] Kumar et al.) 2019) Wu et al. 2019 Agarwal et al.,
2020
Siegel
ct al. 2020 Wang al.j 2020p. This paradigm extends the applicability of RL. where the environ-
mental interactions are costly or even potentially dangerous, such as healthcare (Tenng et al.
2017
Gottesman al., 2018, Nie al. 2019), autonomous driving (Yurtsever et al.) 2020). and recom- mendation systems (Swaminathan et al. 2017 Gilotte al. 2018). While (online) off-policy RL algorithms, such as DDPG (Lillierap et al.) 2016). TD3 (Fujimoto et al. 2018), and SAC (Haarnoja et al. 2018 8a) could be directly adopted in an offline setting, their application be unsuccessful (Fujimoto et al.) 2019 Kumar et al. 2019). especially in high-dimensional continuous control tasks, inevitable and data samples non-exhaustive. Such failures may be attributed to the shift between the state-action visitation frequency induced by the current policy and that by the data-collecting behavior policy, where unseen state-action pairs are presented to the action-value estimator, resulting in possibly uncontrollable extrapolation errors (Fujimoto) al. 2019 Kumar al. 2019). In this regard, one approach offline RL is control the differ- ence between the observed and policy-induced induced state-action action visitation frequencies, so that the current policy mostly generates state-at aatten pairs that have reliable action-value estimate.
where function approximations
Previous work in this line of research typically (1) regularizes the current policy to be close to the behavior policy during the training process, i.e., policy or state-conditional action distribution matching: (2) uses Gaussian policy class with a learnable mean and diagonal covariance matrix (Kumar et al. 2019: Wu et al. 2019). See Appendix [ for a detailed review. However, at any given state s, the underlying action-value function over the action space may possess multiple local maxima. A deterministic or uni-mode stochastic policy may only capture one of the local optima and neglect lots of rewarding actions. An even worse situation occurs when such policies exhibit a strong mode-covering behavior, artificially inflating the density around the average of multiple rewarding actions that itself may be inferior.

---

Under review as a conference paper at ICLR 2022
Previous work under the poliey-matching theme mainly takes two approaches. One approach di- rectly estimates the divergence between the conditional distributions over actions (Wu et al. 2019). However, on tasks with continuous state space, with probability one, no state can appear in the dataset more than once. In other words, for each observed state 8;. the offline dataset has only one corresponding action a, from the behavior policy. Thus, one is only able use single point to assess whether the current policy is close to the data-collecting behavior policy at any particular state, which may not well reflect the true divergence between the two conditional distributions. The other approach, e.g., Kumar et al. (2019), resorts to two-step strategy: First, fit generative model T (a s) to clone the behavior policy; Second, estimate the distance between the fitted behavior pol- icy Taa 8) and the current policy, and minimize that distance as a way to regularize. While this approach, which can acquire multiple samples from the cloned behavior policy, is able to accurately estimate the distance between the current policy and cloned behavior, its success relies heavily on how well the inferred behavior-cloning generative model mimics the true behavior policy. On tasks with large or continuous state space, however, the same problem arises that each state-conditional action distribution is fitted on only one data point. Moreover, some prior work use conditional VAE (CVAE, (Shhn et al. (2015)) as the generative model to clone the possibly-multmodda behavior pol- icy, which further suffers to the problem that CVAE may exhibit a strong mode-covering behavior that allocates large probability density to low data-density regions. Inasmuch as these weaknesses, one may naturally question on how good the samples from such cloned policy resemble the truth, and further, on the quality of the constraint imposed by sample-basee calculation of the distance between such cloned behavior policy and the current policy.
To address these concerns, we are motivated to develop a new framework that not only supports an expressive policy, which can be as flexible as needed, but also well regularizes this flexible policy towards the data-collecting behavior policy. Specifically, (1) instead of using the classical deter- ministic or uni-modal Gaussian policy, we train fully implicit policy for its flexibility to capture multiple modes in the action-value function; (2) to avoid the aforementioned potential problems in the gellic-matching regularization, tion frequency induced by the current policy and that induced by the behavior policy, as an auxiliary training target. Hence, our approach does not need to build generative model to clone the behavior policy. On the theoretical side, we prove in Section 3 that the approach of matching the behavior and current policies is equivalent to matching their corresponding state-action visitation frequencies, which reveals the compatibility of many prior work with our framework. Similar notion of matching the state-action visitations in offline RL taken by the DICE family (Nachum al. 2019 Lee al., 2021a), but they either use Gaussian policy or a mixture of Gaussian policy with a per- per-dataset tuned number of mixtures. Besides, these algorithms have high computational complexity. which, together with inflexible policies and intensive hyperparameter tuning. limit their practical applicability. We instantiate our framework with a GAN structure that approximately minimizes the Jenson Aanson-Shanoon divergence between the visitation frequencies. Furthermore, we design techniques to explicitly en- courage robust behavior of our policy at states not covered in the static dataset. We conduct ablation study on several components of our algorithm and analyze their contributions. With all these consid- erations, our full algorithm achieves state-of of-the-art performance on various tasks from the D4RL dataset (Fu et al. 202I), validating the effectiveness of our framework and implementation.
directly control the distance between the state-action visita-
BACKGROUND AND MOTIVATION
We first present some background information and then introduce toy example to illustrate the motivations of the proposed framework for offline RL.
Offline RL. Following the classic RL setting (Sutton & Barto, 2018). the interaction between the agent and environment modeled as a Markov decision process (MDP), specified by the tuple M = (S, A. P, 7). where & denotes the state space, A the action space, 7 E (0. 1] the discount factor, P(s' 8, a) : S x A + [0. 1] the environmental dynamics, and r(s, a) : S x A + [Rminn Rmax] the reward function. The goal of RL is to learn a policy Ta(a, 8t), parametrized by d. that maxi- mizes the expected cumulative discounted reward R. A R (st, a.) = E 2 7 *r++*+1]
In offline RL. (Fujimoto et al. 2019 Kumar et al. 2019 2020 Levine et al. (020). the agent only has access to a fixed dataset D {(s, a, r. s') 3'), consisting of transition tuples from rollouts by some behavior policies ne [t(a| s). We denote the sttteacction visitation frequency induced by the behavior
2

---

Under review as a conference paper at ICLR 2022
policy Te ds(s, a) and its marginaa the state visitation frequency, ds(s). Similarly, do (s, a) and (s(s) are the counterparts for the current policy Ts. Here, dy(s, a) = ( and we assume ID ds(s,a) a). The visitation frequencies in the dataset are denoted as (s, a) and (s),, which are discrete approximations to da (8,, a) and do (s). respectively.
Actor-Critic Algorithm. Denote (* a) E 7'r( st, Ote 8o 8,00 al as the action-value function. In the actor-critic scheme (Sutton & Barto 2018), the critic Q" (s. a) is often approximated by neural network Qo (s, a), parametrized by 0 and trained by applying the Bellman operator (Lillicrap et al. 2016
Haarnoja et al.
2018a Fujimoto el al. 2019) as
arg ming (Qo(s,a) (r(s, a) pl ( a d'-attll (+)) [Qo(s', a')))]
The actor T aims at maximizing the expected value of Qo, with the training objective expressed as arg max {J (TT) (ordg(s), a a) [Qo (8, a)l}
(1)
where (ø(s) (s) is the state visitation frequency under policy TΦ. In offline RL, sampling from d& (s) is infeasible as no interactions with the environment are allowed. A common and practically effective
approximation (Fu et al. 2019 Levine et al. 2020) to Equation I
J ( ~ Ex (*), a- x) [Qe a)],
(2)
where sampling from ds(s) can be implemented easily as sampling from the offline dataset ID.
Generative Adversarial Nets. GAN (Goodfellow al. 2014) provides a framework train deep generative models, with two neural networks jointly tramed in an adversarial manner: generator Gø. parametrized by Φb that fits the data distribution and discriminator Dw. parametrized by w, that outputs the probability of a sample coming from the training data rather than Ga. Samples x's from the generator's ' distribution do (x) are drawn via z ~ p=(x), x = Gø (z), where P= is some noise distribution. Both G and Dw are trained via two-player min-max game
ming max {V (D GΦ) Ey--d(() [log Dwo (3) + (z-p,(=) [log (1 D Dw (Gp())))) (3)
where du (-) is the data distribution. Given the optimal discriminator Dia Go. the training ob-
jective of G& is determined by the log 4 2- JSD (d]llds)) with the global minimum achieved if and only if de = dn- Therefore, one may view GAN as a distributional matching framework that approximately minimizes the Lunsen-Shannow divergence between the generator distribution and data distribution.
divergence (JSD) (Lin 1991) between du and
de as V (Di; Ga)
Motivations. To illustrate our motivations of training an expressive policy under an appropriate regularization, we conduct a toy experiment of behavior cloning, as shown in Figure I where we use the x- and y-axis values to represent the state and action, respectively. Figure la illustrates the state-action joint distribution of the data-collecting behavior policy that we try to mimic. For Figures 1b Tel we use the same test-time state marginal distribution, which consists of an equal mixture the behavior policy's state distribution and a uniform state distribution between -1.5 and 1.5. If the inferred policy well approaches the behavior policy, we expect (1) clear concentration on the eight centers and (2) smooth interpolation between centers, which implics a good and smooth fit to the behavior policy. We start the toy experiment with fitting CVAE model, a representative behavior-cloning method, to the dataset. As shown in Figure 1b] CVAE exhibits a mode-covering behavior that covers the data density modes at the expense of overestimating unwanted low data density regions. Hence, the regularization ability is questionable of using CVAE as a proxy for the behavior policy in some prior work. Replacing CVAE with the conditional GAN (CGAN, MMiza & Osindero (2014)), i.e., replacing the KL loss with JSD loss, but adopting the Gaussian policy popular in prior offline RL work partially alleviates the issues but drops necessary modes, as shown in Figure Ic This shows the inflexibility of Gaussian policies. Replacing the Gaussian policy in CGAN with an implicit policy, while still training CGAN via sampled-based moticy-matching, improves the capability of capturing multiple modes, as shown in Figure [d] Nevertheless, it is still prone to mode collapse and interpolates less smoothly between the seen states. Finally, training the implicit-policy CGAN via direct state-action-visieeee
joint matching leads to the best performance. As shown in Figure Ie it concentrates clearly on the eight centers and interpolates smoothly between the seen states. Thus, constraining the state-action visitations can be effective way to regularize implicit policies in offline RL. These observations motivate us to train implicit policies in offline RI., with sample-based regularization on the state-action visitations.
3

---

Under review as a conference paper at ICLR 2022
(a) Truth
(b) CVAE
(c) G-CGAN
(d) CGAN
(e) GAN
Figure 1: Performance of approximating the behavior policy on the eight-Gaussian dataset. A conditional VAE ("CVAE"), conditional GAN ("CGAN"), and Gaussian generator conditional GAN ("G-CGAN") are fitted using the conditional- -iitiiinaaldstttiuttoonnn (policy) matching approach. A conditional GAN ("GAN") is fitted using the
Gaussian-generator
basic state-action joint- visitation matching strategy (Section (.11 More details are provided in Appendix D.I
THEORETICAL ANALYSIS
As discussed in Section ! and detailed in Appendix [A] one common theme in prior work in offline RL is controlling the distance between the behavior policy and current policy during the training pro- cess. In this section, we prove that this approach, in essence, controls the corresponding state-action visitations. This analysis theoretically links line of prior offline-RI RRL work with proposed frame- work, manifesting the generality of our algorithmic idea. Note that (s a) = dø (s)) a (a s) and similarly for (s a.. Hence, it is sufficient to show the closeness between dæ and ds(s) when ( (a 8) is close to Th 8). Below we give our analysis for the matrix (finite state space) case. Continuous state-ppace cases may be analyzed similarly and are left for future work. The Proofs are deferred to Appendix C
Notation. Denote A-i- matrix A with its i-th row removed; <(A) as the 2-norm condition number of A: 1 as a row vector of all ones and I as an identity matrix, both with an appropriate di- mension. Assume that the state space S is finite with cardinality N, N, i.e.e.. S (s' ........ s s²). The tran- sition probabilities associated with policy के over S is then N N matrix Tr. whose (i, entry (t.)) = & (S++1 s SS, 8*) JA P (S++1 s³ S; = 8 A, = ar) T (ar 8*) da. and similarly for Tú, the transition matrix associated with mh. Note that in this case, ds(s), ds(s) are vectors and we denote dø a ds(s), do & ds(s) RN and dø d, + Ad.
Theorem 1. Denote
max; 2,.. N+. (1-1T)
Emaz
-io
If
maxi I.....N Tbb (.18") T (.18")ll, Se
and 1.j. Th. > 0, (1,2...., N). then
2TV do) ||dd]] ||d dolli S Nimaz
as 6 0.
T-e Kmux
Remark. (1) We note that Kmax is a constant for fixed To and can be calculated by iteratively removing columns of Ti, and computing the SVD of the referred matrix. (2) The assumption that Th, 0, V.j E {1,...,N] N} can be satisfied by substituting the zero entries in the original transition matrix with a small number and ro-nozalleeed each row of the resulting matrix, as in the PageRank algorithm (Page et al. 1998 Langville & Meyer 2004).
In practice, the offline dataset often consists samples collected by several policies. Equiv- alently, the behavior policy Ta [s) is mixture of single policies. Assume that there are K such policies (TTb (+|s)] with mixture probabilities {ux) K i.e. To(- ()
K
IK wh Tb, (- 8), Ex= Ws = 1. Since we collect ID by running each Tb, (+18) a proportion of ww of total time, we may decompose ID D = UK D2 where DA consists of wy proportion of data in ID. Thus, ds(s) = Ek wydn, (s) and the approximation dz xe (s(s) has population version ds)s) x Ex= w dr, (s) e ds(s). As before, denote d,, (s) E INN as the limiting state- occupancy measure induced by Te, on M: TT, as the transition matrix induced by Rbs over S; and ds do, Adg. We extend Theorem T into this mixture of policies case follows.
4

---

Under review as a conference paper at ICLR 2022
Theorem 2. Denote
Kmaz.k maxi-2 N+1 ( ToT -ia) )
Kmaz max;- I...,K Kmmaz,k
If
maxx=11... maxx I....,N 1oe (-18¹) TT (-18")l1 SE
Komar
and T Thp > 0,Vi,j {1. (1....,N} N2. 1 [T,...,K K then
(dV, ds) dd - dollı S Ex=z Wk I-c Keman
<.Mmmxx
c Cimax
as c - 0. 0
I-
In particular, if wy = 1/K, Ak k= {I..... ..... K}. then
2TV do) = dd doll 1 S E
CRman.
as 0.
I-c I-C Kemax
Remark. We note that Kmar, is constant for fixed To, and Kmar constant for fixed {mb, k
k=11
We notice that similar analysis has been given the prior work of bounding DKL. (d)(s)] by O (c/(1 - ?)) (Schulman al. 2015 Levine al. 2020). However, this prior work deals with (unnormalized) discounted visitation frequencies while our bound is devoted to undiscounted visitation frequencies, since neither the data collection (i.e., policy rollout) nor the state-action- visitation regularization involve the discount factor. In short, the definitions of dø(s) and ds(s) in our work are different from this prior work. Note that this prior work depends on - on the denominator and hence cannot be applied to the undiscounted case (discount factor y 1). Furthermore, instead of the KL. divergence, we bound the total variation distance between the state visitation frequencies, which a well-defined metric.
STATE-ACTION JOINT REGULARIZEI IMPLICIT POLICY
In this section discuss an instance of framework that will be used in our empirical study in Section 55] Concretely, we train a fully implicit policy via a GAN structure to approximately minimizes the JSD between the state-action visitation frequency induced by the current policy and that induced by the behavior policy. Our basic algorithm is discussed in Section 4.1 followed by three enhancing components presented in Section 4.2 to build up our full algorithm.
This instantiation manifests three facets we consider important in offline RL: (1) the flexibility of the policy class, (2) an effective sample-bass based regularization without explicitly modelling the behavior policy, and (3) smoothness of the learned policy.
4.1 BASIC ALGORITHM
Motivated by the standard actor-critic and GAN frameworks described Section our basic al- gorithm consists of critic Qe. an actor Tg. and discriminator Dw- For training stability, adopt the double Q-learning (Hasselt 2010) and target network formulation to train a pair of critics Qo Qo and the target networks Qo; Qo. Tty.
To penalize uncertainty in the action-valus estimates at future states, while controlling conservatism, follow Fujimoto et al. (2019) and Kumar al. (2019) to use the following critic-training target:
2 (s, 2 r(s, a) + B. (a (-le))
min Qe; (s', a') + +11 max Qe; (s', a') f=1,2
(4)
j=1,2
[o, 1]. where one a' is sampled at each s' the mini-batch. The training objective for both critic networks is minimizing the suare--uurrd-erroor between their re- spective action-value estimates Qo, (s, a) and Q (s. a) over state-action pairs in the mini-batch.
with some hyperparameter
Our actor consists three components: an implicit policy, state- aaw.aaiiiiiiiaattionnn joint regulariza- tion, and conservative training target.
Implicit Policy. As discussed in Sections D and 2 a deterministic or Gaussian policy may miss important rewarding actions, even concentrate inferior average actions. For online off-policy
5

---

Under review as a conference paper at ICLR 2022
RL, Yue et al. (2020) shows the benefit of introducing implicit distribution mixed Gaussian policy. Generalizing that idea to offline RL, we train a fully implicit policy, which transforms a given noise distribution into the coneiiooniiional action distribution via a neural network, in reminiscent of the generator in CGAN. Specifically, given 8,
T& (+1 2),,
(z
(5)
where Th is deterministic function and (2) is some noise distribution. As shown in Figures ld and Ic] an implicit policy has stronger capability of learning multi-modal policy, if needed.
State-action Joint Regularization. As discussed in Section 2 directly sampling from (s(s) is infeasible in offline RL. Motivated by Equation 2 we approximate dæ (8) 3) by (8) in our imple- mentation, which allows an easy sampling scheme that simply samples from D. without needing
an importance sampling correction that may possess high variance (Liu et al. 2018). Such an ap- proximation is classical in the off-policy and offline RL literature (Degris et al. 2012 Levine et al.) 2020), with demonstrated empirical effectiveness in offline RL (Fu et al. 2019), besides its usage off-policy RI. (Silver et al. 2014, [Schulman et al. 2015 Lillierap et al. 2016). With this sim- plification we can efficiently maximize the similarity between dy a) and (s, a) with respect to sample-based estimate of some statistical divergence, such as the Using notations in GAN, the generator sample and the data sample y are defined
divergence.
x 88, a), 8 ~ D2 ā ~ [- (1;;;; (8, a) ~ D, independent of 8
(6)
We then constraint this statistical divergence value, named generator loss (g((), in the training of actor. In our instantiation of approximately minimizing JSD via GAN, b.g E.z [log (1 Dw (=)(()))
Our choice of directly matching state-action joint visitations mitigates the issue of uncontrollable extrapolation errors in the vetion-value function estimate, since proximity of visitation frequencies leads to reduced chances of estimating the action-values of state-action pairs far from the offline dataset. Furthermore, the problem the policy-matching approach fitting each state-conditional action distributions
only one data point be circumvented, since state-action pairs (s;, a,) in the offline dataset are all viewed as samples from the joint visitation frequency, instead of each pair being separately viewed as one sample from the state-conditional distribution, i.e., a, π (s))- joint matching approach implicitly encourages the smoothness
Besides, the ftte-actionn-viiiatiii of the state-action mapping, namely, similar states should have similar actions. This is because, for example, the discriminator in GAN can easily discriminate as "fake" a generator sample x should it has state similar data sample but action very different from. This smoothness feature helps ensure reliable generalization of our policy to unseen states.
Actor- -oati-Tr Target. To prevent the accumulation of accidental errors in the training process and in the approximation of action- value function, we adopt the strategy in Kumar et al. 2019) to train the policy with respect to a conservative estimate of the action-values. Specifically, we exploit the double-Q structure and use the minimum of the two action-value estimates for policy improvement. For the ease of optimization, we use the Lagrange form of the constraint optimization problem and penalize the generator loss (Ø) while improving the policy. Our policy-training target is
arg ming - EEL-DE
[ming-112 Qo, (8, a)] - a L,(())
where a is fixed Lagrange multiplier. At the test time. we follow Fujimoto et al. (211)) and Kumar al. (2019) to first sample 10 actions from अΦ and then execute the action that maximizes Qo.
The discriminator is trained to better distinguish samples from dg(s, a) and from ds(s, a). It aids the matching of the ttae-actionn-visitation frequencies through outputting (g()). ).). As an example, for approximately minimizing JSD via GAN, the discriminator outputs the probability that the input, either the x or y in Equation 6 comes from de (&, a). In this case, the discriminator is trained to minimize the error in assigning as "fake" and y as "true," the inner maximization of Equation 3
4.2 ENIIANCING COMPONENTS
In this section we present three enhancing components for further improving our basic algorithm.
Multiple Action-samples at Bellman Backup. The expectation part of the critic learning target in Equation 4 can be better estimated, in terms of a smaller sample variance, by averaging over N, actions each s' in the mini-batch, rather than just one a in Section 4.1

---

Under review as a conference paper at ICLR 2022
State-smoothing at Bellman Backup. Due the stochastic nature of environmental transition, multiple next states s' are possible after taking action a at state 8, while the offline dataset D only contains one such s'. Since the agent is unable to interact with environment to collect more data in offline RI., local exploration (Sinha et al. 2021) in the state-space appears as an effective strategy regularize the Bellman backup by taking consideration of states close to the records in the offline dataset. We assume that: (1) small transformation to a state results in states physically plausible in the underlying environment (as in Sinha et al. (2021) ): (2) when the state space is continuous, the transition kernel P (- s, a) is locally continuous and centers at the recorded s' in the dataset.
With these assumptions, we propose to fit Qo a) on the value of small region around the recorded next state s'. Specifically, with a pre-specified standard deviation dB, we sample around s' as & s' + 6, 6 (0,03 and modify Equation 4 as
(s, - r(s, a) T.aaa (-18)
À min Qo; (8, -=1.2
max Qo; (8, aa) (8) j=1,2
where Ng á are sampled to estimate the expectation. This strategy is equivalent to using a Gaussian distribution centered at s' to approximate the otherwise non-smooth 6 transition kernel manifested in the offline dataset. Similar technique is also considered as the target policy smoothing regulariza- tion (Fujimoto et al. (2018), though smoothing therein is applied on the target action.
State-smoothing at Joint-matching. As described in Section 4.1 we approximate (s(s) by dp in the sample-based estimate of the chosen statistical distance. However, do( is in essence discrete and the idea of smoothing the discrete state-distribution can be applied again to provide better coverage of the state space. This design explicitly encourages a predictable and smooth behavior at states unseen in the offline dataset. Specifically, with some pre-specified o3. we modify the sampling scheme of š in Equation as
Our strategy is akin to sampling from a kernel density approximation with data points s D and with radial basis kernel of bandwidth Ø..
(Wasserman 2006 of ds(s)
Algorithm shows the main steps of our algorithm, instantiated by approximately minimizing JSD via GAN, and a detailed version is in Appendix Implementation details are in Appendix [.2]
Algorithm 1 State-Action Joint Regularized Implicit Policy (GAN-JSD, Main Steps)
Initialize policy network Té, critic network Qo, and Qo discriminator network Dw- for each iteration do
Sample a mini-batch of transitions B ((s,a,r, ~ D.
From Equation 8 train the critics by arg mine, (Qo, (s.a) ((s, a))² over (8. B, for j Calculate generator loss L.g using Du and the æ. y Equation [6] with state-smoothing Section 4.2 Optimize policy network Te by Equation]] 7
1,
Optimize discriminator Du to maximize (gg))(() [log D.. (y)) + R. [log (1 - (==(=)))-
end for
EXPERIMENTS
As discussed in Section T we consider it important the flexibility of the policy class, the well- regularization and the smoothness of the learned policy. To this end, we develop our state-action joint regularized implicit policy, which is a framework that supports training an expressive implicit policy via an effective sample-bac based regularization without an explicit modelling of the behavior policy, and with techniques encouraging smoothness. In this section we will test an instantiation of our framework on the continuous-contro RL tasks. Specifically, implicit policy, state-action-vviiitaionn matching, and our full algorithm (Section 5.1). We then show in ablation study (Section 5.2) the contributions of several building blocks.
will first show the effectiveness of
Implementation. We use GAN to approximately control the Jemsen-Shannon divergence between the state-action visitation frequencies. Our source code builds the official BCQ repository and largely follows its network architectures. Our implementation of the GAN structure and the hyper-
parameter choices follow the literature (Goodfellow al. 2014 Mirza & Osindero 2014
Radford
7

---

Under review as a conference paper at ICLR 2022
et al. 2016:
Salimans et al. 2016 White 2016 Goodfellow 2017). To mimic hyperparameter-
agnostic setting, we minimize hyperparameter tuning across datasets. Implementation details and hyperparameter setting of our full algorithm with GAN foint-matching is in Appendix D.2.1
5.1 MAIN RESULTS
To validate the effectiveness of our framework, we test three implementations of the GAN instan-
tiation: (1) basic algorithm (Section 4.1 regularized by the classical policy-matching" ("GAN-
Cond:,asic"), ) (2) basic algorithm regularized by the state-action-viste visiiatiisitaiion joint matching ("GAN-
Joint:Bascc",, (3) full algorithm, which adds state-smoothing techniques onto our basic algorithm ("GAN-Joint"), We compare them with the state-of-the-art (SOTA) offline-RL. algorithms: BEAR (Kumar et al. 2019). BRAC (Wu et al. 2019) (BRAC-v: value penalty), BCQ (Fujimoto et al, 2019). and CQL (Kumar al. 2018ap. We re-run CQL using the official source code (details in Appendix [D.2.2] Results for other algorithms are from Fu et al.
2020 together with offline soft actor-critic (SAC) (Haarnoja al.
(202T). Table [ validates the effectiveness of our full algorithm, to- gether with the efficacy of the implicit policy, state-action-visitaaiie
matching, and state-smoothing.
Our full algorithm average outperforms the baseline algorithms, and its performance is relatively stable across tasks and datasets that possess diverse nature. Our full algorithm especially shows performance on the high-dimensional Adroit tasks and the Maze2D
robust and comparatively-gont tasks that are collected by non-Mark Mon-Markovian policies, which may traditionally be considered as hard in offline RL. On the Gym-MuJoCo domain, our full algorithm shows its ability to learn from datasets collected by mixture of behavior policies, and from medium-quality examples, which are likely the datasets to encounter in real-world application. These results support our design of implicit policy, matching the state-action visitation frequencies, and explicit state-smoothing.
Comparing (GAN-Cond:Bassi with the baseline algorithms, especially BEAR and CQL that use Gaussian policies, we see that an implicit policy does in general help the performance. This aligns with our intuition in Sections Tand of the incapability of the uni-modal Gaussian policy in captur- ing multiple action-modes.
[f
To verify the gain of our joint- wwt.-iiiitiioomacchiiiii approach over the classical policy matching, apart from the comparison between our full algorithm with BEAR and BRAC, the SOTA policy-matching algorithms, we further compare Jasit with Basic.. We see that on 11 out of 16 datasets, "GAN-Joint:Basii wins against LAAN-Cond:Baatt are close. This empirical result may be attributed to the advantage of itate-action-viiiaatiin match- ing, e.g, good regularization and smoothness in the state-action mapping (Section 4.1p.
while results other datasets
Comparing "GAN-Joint" with "GAN-Joint.Basic"" we see that our state-smoothing techniques do in general help the performance. This performance gain may be related to smoother action-choice at states not covered by the offline dataset, and a more regularized Bellman backup. Note that different datasets may require different smoothing strength, and thus this comparison can potentially be more significant should one is allowed to per-dataset tune the smoothing hyperparameter.
5.2 ABLATION STUDY
The ablation study serves to answer the following questions: (a): Is implicit policy better than Gaussian policy in our framework? (b): Does state-smoothing at joint-matching help? (c): Does
state-smoothing at Bellman backup matter? (d): How important is the standard deviation of the Gaussian noise injected in statesmmothiig?? Unless stated otherwise, hyperparameters for all algo- rithmic variants in all datasets are in Table
(a): We compare the performance of basic algorithm with its variant where the implicit policy therein is replaced by a Gaussian policy. To make a fair comparison, the critics, discriminator, and hyperparameter setting remain the same. Technical details are in Appendix (D22.3)
Table presents the results. On 11 out of 16 datasets, our basic GANN-joint-matching algorithm has higher average return than the Gaussian policy variant. This empirical result coincides with our intuition in Section 4. and results in Section J. that a Gaussian policy is less flexible to capture all
Same is used in generator and data sample, i.e., ä s. Implementation follows
Wu al. 2019
8

---

Under review as a conference paper at ICLR 2022
Table 1: Normalized returns experiments from the Maze2D, Gym-Mojoco, and Adroit domains. High average scores and low average ranks are desirable.
D4RL suite of tasks, We perform experiments on tasks
Task Name
SAC-off
BEAR
BRAC-v
BCQ
CQL
GAN-Cond.Basic
GAN-Joint: Basic
GAN-Joint
maze2d-umaze maze2d-mas 2d-medium
3.4
-16.0
12.8
50.5
52.3
19.6
57.6 11.0
40.1 16.9
26.
29.0
33.8
8.3
30.7
42.6 I8.2
39.4 10.3
69.6 25.6
maze2d-large
4.6
40.6
6.2
43.7
36.9 17.
$2.1
26.
halfchectal--medine
-4.3
41.7
46.3
40.7
39.0
438 = 0.2
43.7 0.5
44.1 0.3
walker vallernddmedium
59.1
81.1
53.1
60.2
65.5
9.0
69.3
Aopper-medium
0.8
52.1
31.1
54.5
34,5
67.5 21.
66.3 15.4
60.1
halfcheetah medium replay
47.7
32.3
wa walker medium-replay
1.9
19.2
0.9
15.0
16.4
6,9 2.3
9.9
10.2 2.4
hopper medi replay
33.7
halfcheetah- metht--edimmm-e
medium-expert expert
1.8
53.4
41.9
64.7
34.5
76.0+99.2
74.2+6.1
75.8 10.1
walker2d waller22dmedione medmmiiom--xpertt
-0.1
40.1
81.6
57.5
79.8
73.3
76.5
71.2 22.0
hopper www.im-edddeeeeeee
1.6
96.3
0.8
110.9
103.5
68.0 180++113 21.3
70.9 26.8
99.9 + 29.0
pen-human
-1.0
68.9
2.1
52.9 15.9
61.0 13.7
45.5
pen-cloned
23.5
26.5
-2.5
44.0
1,5
1944
14.5
31.3
18.0 14.4
pen-expert door-expert
6.1
105.9
-3.0
114.9
95.9
126.1 = 17.7
129.1
14.4
141.1 14.8
7,5
103.4
-0.3
99.0
87.9
100.8 3.8
104.1 3.6
103,4 3.7
Averago Score
10.0
44.1
24.1
51.4
47.1
55.6
59.6
61.4
Average Rank
4.8
5.5
4.3
4.6
3.9
2.9
3.2
the rewarding actions, of which an implicit policy is likely to be capable. Appendix visualizes this comparison and shows in plots that Gaussian policies do leave out action modes in offline RL.
(b): We compare our full algorithm with its variant of no state-smoothing in the matching of state- action visitations. The results are shown in Table 5 where our full algorithm overall performs better than the state- smootsmmo variant. This performance gain may be attributed to better coverage of the state-space by the smoothed state distribbtti (Section 4.2) which is related to a more predictable and smoother action choice at unseen states. As stated previously, this comparison can potentially be more significant should one is allowed to per-dataset tune the a, parameter.
(c): We compare full algorithm with its variant of state-smoothing in the Bellman backup. Ta- shows the results. Again, overall our full algorithm performs better than the no state-smoothing version, showing the benefit of smoothing the empirical transition kernel òg (Section 4.2). e.g. tak- ing the stochasticity of state-transitie into account. As before, we use the same smoothing strength across all datasets, while a per-dataset tuning of the ов parameter may improve the distinction.
(d): To simplify hyperparameter tuning, in actual implementation fix оB =J ف a (see Ap- pendix D.2.1). To test the robustness to the hyperparameter, we run our full algorithm under E {1 10 - 3.3 x 10-3, 3 10 x 10-4, 10-4 0}. Table 7 shows the normalized re- turns. We see that our algorithm is relatively insensitive to the setting of a. especially in the range 10 10-3) where the performance varies little with a. A too-small cannot pro- vide enough smoothing to the state distributions. On the contrary, too-large may highly distort the information contained in the offline dataset, such as the state-transition kernel. both cases, a degradation in the overall performance is expected.
E [1 10
CONCLUSION AND FUTURE WORK
In this paper, develop framework that supports learning flexible while well-regularized pol- icy in offline RL. Specifically, we train fully implicit policy via regularization on the difference between the state-action visitation frequency induced by the current policy and that induced by the data-collecting behavior policy. An effective instantiation of our framework through the GAN struc- ture provided for approximately minimizing the JSD between the visitation frequencies. Other divergence metrics, such as the MMD, may also be applied and are left for future work. We further augment our algorithm with explicit state-smoothing techniques to enhance generalizability on states beyond the offline dataset. On the theoretical side, we show the equivalence between policy- matching and state-action-visstatioo matching, and hence the compatibility of many prior algorithms with our framework. We note that our implementation-wiss. simplification of approximating current policy's 's state visitation frequency by the behavior's can be improved by a more tactful approxima- tion of current policy's 's state occupancy measure, which is left for future work. Nevertheless, the effectiveness of our framework and implementations is validated through extensive experiment and ablation study on the D4RL dataset.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
To help reproduce our empirical work, we provide detail algorithmic description in Appendix B and implementation details, including the hyperparameter choice and model architecture, in Appendix For reproducing our theoretical results, a step-by-step proof is provided Appendix
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An Optimistic Perspective on Of- fline Reinforcement Learning, 2020.
Martín Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. ArXiv, abs/ 701 701.07775
2017.
Peter Baxendale. E. Harris's Contributions to Recurrent Markov Processes and Stochastic Flows. The Annals of Probability, 39(2):417 422848, 2011. ISSN 00911790
Marc G. Bellemare, Ivo Danihelka, Will Dabney, S. Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and R. Munos. The Cramer Distance Solution Biased Wasserstein Gradients. ArXiv, abs/1705.10745 2017.
Mikolaj Binkowski, Danica J. Sutherland, Michael Arbel, and A. Gretton. Demystifying MMD GANs. ArXiv, abs/ 000/1000,,,141,, 2018.
Catherine Cang, Aravind Rajeswaran, P. Abbeel, and M. Laskin. Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL. ArXiv, abs/2106.09119, 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, M. Laskin, P. Abbeel, A. Srinivas, and Igor Mordatch. Decision Transformer: Reinforcement Learning via Sequence Modeling. ArXiv, abs/2106.01344 2021.
Marco Cuturi. Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances. arXiv: Machine Learning, 2013.
T. Degris, Martha White, and Sutton. Off-Policy Actor-Critic. ArXiv, 0ess1205.48835 2012.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-Based Batch Mode Reinforcement Learn- ing. J. Mach. Learn. Res., 6:503-556, 2005.
Jean Feydy, Thibault Séjourné, François-Xavier Vialard, Shun-ichi Amari, Alain Trouve, and Gabriel Peyré. Interpolating between Optimal Transport and MMD using Sinkhorn Divergences. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2681-2690, 2019.
Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing Bottlenecks in Deep Q- learning Algorithms. In ICML, 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for
Deep Data-Driven Reinforcement Learning, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing Function Approximation Error in Actor-Critic Methods. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1587-1596. PMLR, 10-15 Jul 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-Policy Deep Reinforcement Learning without Exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2052-2062. PMLR, 09-15 Jun 2019.
Alexandre Gilotte. Clément Calauzènes. Thomas Nedelec, Alexandre Abraham, and Simon Dollé. Offline A/B Testing for Recommender Systems. Proceedings of the Eleventh ACM International Conference Web Search and Data Mining, 2018.
10

---

Under review as a conference paper at ICLR 2022
I. Goodfellow. NIPS 2016 Tutorial: Generative Adversarial Networks. ArXiv, abs/ Abs/1700.00160, 2017.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Z. Ghahramani, Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances Neural Infor- mation Processing Systems, volume 27. Curran Associates, Inc., 2014.
Omer Gottesman, Fredrik D. Johansson, J. Meier, Jack Dent, Donghun Lee, Srivatsan Srinivasan, Linying Zhang, Ding, David Wihl, Xuefeng Peng, Jiayu Yao, Isaac Lage, C. Mosch, Li wei H. Lehman, M. Komorowski, A. Faisal, L. Celi, D. Sontag, and Finale Doshi-Velez. Evaluating Reinforcement Learning Algorithms Observational Health Settings. ArXiv, abs/1805. 12298, 2018.
A. Gretton, K. Borgwardt, M. Rasch, B. Schölkopf, and Alex Smola. A Kernel Two-Sample Test. Mach. Learn. Res., 13:723-773, 2012.
Caglar Gulcehre, Sergio Gómez Colmenarejo, ziyu wang, Jakub Sygnowski, Thomas Paine, Konrad Zolna, Yutian Chen, Matthew Hoffman, Razvan Pascanu, and Nando de Freitas. Addressing Extrapolation Error in Deep Offiin Reinforcement Learning. 2021.
Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Im- proved Training of Wasserstein GANs. In NIPS, 2017.
Tuomas Haarnoja, Haoran Tang, P. Abbeel, and Sergey Levine. Reinforcement Learning with Deep Energy-Based Policies. In ICML,, 2017.
Tuomas Haamoja, Aurick Zhou, Pieter Abbeel, and Sengey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, G. Tucker, Sehoon Ha, Jie Tan, Vikash Ku- mar, Henry Zhu, Abhishek Gupta, P. Abbeel, and Sergey Levine. Soft Actor-Critis Algorithms and Applications. ArXiv, abs/18122 1s200185.. 2018b.
H. V. Hasselt. Double Q-learning. In NIPS, 2010.
Jonathan Ho and Stefano Ermon. Generative Adversarial Imitation Learning. In D. Lee, Sugiyama, U. Luxburg, I. Guyon, and Garnett (eds.), Advances Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, A. Lapedriza, Noah J. Jones, S. Gu, and Rosalind W. Picard. Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog. ArXiv, abs/1907.00456, 2019.
Nathan Kallus and Angela Zhou Confounding-Robee Robust Policy Evaluation in Lnfinite-Horizo Rein- forcement Learning, 2020.
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, 2014.
Diederik Kingma and Max Welling. Auto-Encoding Variational Bayes, 2013.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline Reinforcement Learning with Implicit Q-Learning, ArXiv, abs/21 10.06169, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-Learning for Of- fline Reinforcement Learning. In H. Larochelle, M. Ranzato. R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1199.1191. Curran Associates, Inc., 2020.
11

---

Under review as a conference paper at ICLR 2022
Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and D. Vetrov. Controlling Overes- timation Bias with Truncated Mixture of Continuous Distributional Quantile Critics. ArXiv, abs/2005.04269, 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch Reinforcement Leurning, pp. 45-73. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 378-3-642-27645-- doi: 10.1007/ 978-3-642-27745-3 2.
Amy N. Langville and Carl D. Meyer. Deeper inside PageRank. Internet Mathemalics, 1(3):335- 380, 2004. ISSN 1542-7951.
Romain Laroche and P. Trichelair. Safe Policy Improvement with Baseline Bootstrapping, In ICML, 2019.
Jongmin Lee, Wonseok Jeon, Byung-Jun Lee, J. Pineau, and Kee-Eung Kim. OptIDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation. ArXiv, abs/2106.10783, 2021a.
Kimin Lee, M. Laskin, A. Srinivas, and P. Abbeel. SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning. In 2021b.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offlin Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems, 2020.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and B. Póczos. MMD GAN: Towards Deeper Understanding of Moment Matching Network. In NIPS, 2017.
T. Lillicrap, Jonathan J. Hunt, A. Pritzel, N. Heess, T. Erez, Yuval Tassa, D. Silver, and Daan Wierstra. Continuous Control with Deep Reinforcement Learning, CoRR, abs//509.005 10991, 2016.
Jianhua Lin. Divergence Measures Based on the Shannon Entropy. IEEE Transactions Informa- tion theory, 37:145-151, 1991.
Long-Ji Ji Lin. Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and Teaching. Machine Learning, ((3-4):293333 321, 1992.
Q. Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the Curse of Horizon: Infinite- Horizon Off-Policy Estimation. In NeurIPS, 2018.
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Shane Gu. Deploymenn-Efficieet Reinforcement Learning via Model-Based Offline Optimization. ICLR, abs/2006.03647, 2021.
Carl D. Meyer. Matrix Analysis and Applied Linear Algebra. Society for Industrial and Applied Mathematics, USA, 2000. ISBN 1450.
Mchdi Mirza and Simon Osindero. Conditional Generative Adversarial Nets. ArXiv, abs/1411.1784, 2014.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral Normalization for Generative Adversarial Networks. ArXiv, abs/1802.05957,, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing Atari with Deep Reinforcement Learning. 2013.
A. Mousavi, Lihong Li, Qiang Liu, and Denny Zhou Black-box Off-policy Estimation for Infinite- Horizon Reinforcement Learning. ArXiv, abs/2003.11126. 2020.
A. Müller. Integral Probability Metrics and Their Generating Classes of Functions. Advances in Applied Probability, 29:429-443. 1997.
Ofir Nachum, Bo Dai. Ilya Kostrikov, Yinlam Chow, Lihong Li, and D. Schuurmans. AlgaeDICE: Policy Gradient from Arbitrary Experience. ArXiv, abs/19 abs/1912.00207, 2019.
12

---

Under review as a conference paper at ICLR 2022
Xinkun Nie, Emma Brunskill, and Stefan Wager. Learning When-to-Treat Policies. Journal of the American Statistical Association, 116:392 409, 2019.
L. Page, S. Brin, R. Motwani, and T. Winograd. The PageRank Citation Ranking: Bringing order to the Web. In Proceedings of the 7th International World Wide Web Conference, pp. 161-172, Brisbane, Australia, 1998.
G. Peyré and Marco Cuturi. Computational Optimal Transport. Found. Trends Mach. Learn., 11: 355-607, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. CoRR, abs/151 1.064444 2016.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, J. Schulman, E. Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstra- tions. ArXiv, hts//709.10066 2018.
Tim Salimans, I. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Im- proved Techniques for Training GANs. NIPS, 2016.
Erwin Schrödinger. Sur théorie relativiste de l'électron et l'interprétation de mécanique quan- tique. Annales de l'institut Henri Poincaré, 2:269-310, 1932.
J. Schulman, Sergey Levine, P. Abbeel, Michael I. Jordan, and Philipp Moritz. Trust Region Policy Optimization. ArXiv, 502.05502. 2015.
Dino Sejdinovic, Bharath Sriperumbudur, Arthur Gretton, and Kenji Fukumizu. Equivalence of Distance-based and RKHS-based Statistics in Hypothesis Testing. The Annals of Statistics, 41 (5):2263-229) 2013. ISSN 00905364, 21688966.
Noah Y. Siegel, Jost Tobias Springenberg. Felix Berkenkamp, Abbas Abdolmalcki, Michael Ne- unert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning. 2020.
D. Silver, Guy Lever, N. Heess, T. Degris, Daan Wierstra, and Martin A. Riedmiller. Deterministic Policy Gradient Algorithms. In ICML, 2014.
Samarth Sinha, Ajay Mandlekar, and Animesh Garg. S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning, 2021.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning Structured Output Representation using Deep Conditional Generative Models. In NIPS. 2015.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
Adith Swaminathan, A. Krishnamurthy, Alekh Agarwal, Miroslav Dudik, J. Langford, D. Jose, and Zitouni. Off-policy Evaluation for Slate Recommendation. NIPS, 2017.
H. Tseng, Yi Luo, Sunan Cui, Jen- -zung Chien, R. Ten Haken, and Naqa. Deep Reinforcement Learning for Automated Radiation Adaptation in Lung Cancer. Medical Physics, 44:669026705, 2017.
Núria Armengol Urpí, S. Curi, and A. Krause, Risk-Averss Offline Reinforcement Learning. ArXiv, abs/2102.05371, 2021.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. Critic Regularized Regression. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.). Advances in Neural Information Processing Systems, volume 33, pp. Curran Associates, Inc., 2020.
Larry Wasserman. All of Nonparametric Statistics. Springer, 2006.
Tom White. Sampling Generative Networks. arXiv: Neural and Evolutionary Computing, 2016.
13

---

Under review as a conference paper at ICLR 2022
Yifan Wu, George Tucker, and Ofir Nachum. Behavior Regularized Offline Reinforcement Learning, 2019.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, J. Susskind, Jian Zhang, R. Salakhutdinov, and Hanlin Goh. Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning. In ICML, 2021.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: Conservative Offline Model-Based Policy Optimization. ArXiv, abs/2102.08363, 2021.
Yuguang Yue, Zhendong Wang, and Mingyuan Zhou. Implicit Distributional Reinforcement Learn- ing. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Floring Balcan, and Hsuan- Tien Lin (eds.). Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and K. Takeda. Survey of Autonomous Driving: Common Practices and Emerging Technologies. IEEE Access, 8:58443-58469, 2020.
Ruiyi Zhang, Bo Dai, Lihong Li, and D. Schuurmans. GenDICE: Generalized Offline Estimation of Stationary Values. ArXiv, abs/2002.09072 2020.
Huangjie Zheng and Mingyuan Zhou Exploiting Chain Rule and Bayes' Theorem to Compare Probability Distributions, 2021.
14

---

Under review as a conference paper at ICLR 2022
A RELATED WORK
Offline Reinforcement Learning. Three major themes currently exist in offline-RR RL research. The first focuses on more robustly estimate the action-value function (Agarwal et al.) 2020 Gulcehre 2021) or provide a conservative estimate of the Q-values (Kumar et al., 2020 Yu et al. 2021
iet al.
Sinha et al. 2021), which may better guide the policy optimization process. The second research
theme aims at designing a tactful behavior-cloning scheme so as to learn only from "good" actions in
the offline dataset (Wang et al. 2020] Chen al. 202I). Most similar to our approach, the third line of research tries to constraint the current policy to be close to the behavior policy during the training process, under the notion that Q-value estimates at unfamiliar state-action pairs can be pathologically
worse due to a lack of supervised training. Specifically, Kumar et al. (2019) and Wu et al. (2021) use conditional variational autoencoder (CVAE) (Kingma Welling 2013 [Shhn et al. 2015) to train a behavior cloning policy to sample multiple actions at each state for calculating the MMD constraint. Wu et al. (2019), Siegel et al. (2020), and Cang et al. (202T) fit a (Gaussian) behavioral prior to the offline dataset trained by (weighted) maximum likelihood objective. Jaques et al. (2019) consider pre-trained generative prior of human dialog data before applying KL-control to the current policy. Note that these work essentially constraint the distance between the current policy and the cloned behavior policy. Lanoche & Trichelair (2019) assume a known stochastic data-collecting behavior policy. Fujimoto al. (2019) and Urpi et al. (202T) implicitly control the state-conditional action distribution by decomposing action into a behavior cloning component, trained by fitting a CVAE onto the offline data, and perturbation component, trained optimize the (risk-averse) returns. Besides, some work, such as Wu et al. (2019), directly estimates and regularizes the divergence between the conditionl action distributions. In the paper, we instead take on the perspective of matching the state-action joint visitation frequencies, which avoids using single point to estimate the divergence between distribution and removes the need for a good approximator to the behavior policy, while implicitly smooths the mapping from state to actions (Section [p.. Note that if we use the same states in constructing the generator samples and the data samples, i.e., & = s in Equation our implementation of state-action- vittttioniistaiioon matching is fundamentally equivalent to matching the state-conditional action distributions, as in some prior work.
Online Off-policy Reinforcement Learning. A large class of modern online off-policy deep rein- forcement learning algorithms trains the policy using experience replay buffer (Lin) 1992), which
is storage of the rollouts of past policies encountered in the training process (Mni et al. 2013
Lillicrap et al. [0166] Haarnoja et al. [207]] Fujimoto et al., 2018 8 Haarnoja et al.) 2018b Kuznetsov
ct al. 2020) Yue et al. [2020] Lee et al. 12021b. This approach essentially use the state- fre-
quency of past policies to approximate that of the current policy (Equation 2). We adopt this notion in both the policy improvement step and in the implementation of aaate-action-visiaation joint match- ing, with additional smoothing techniques to encourage the smoothness of the empirical transition kernel and the approximated state- visitatiiit frequency of current policy (Section 4.2).
Computational Distribution Matching. Many computationally efficient algorithms exist to match two probability distributions with respective to some statistical divergence. GAN (Goodfellow et al.) 2014) approximately minimizes the lensen-Shannor divergence between the the model's 's distribu- tion and the data- data-generating distribution. Similar adversarial training strategy is further applicd to estimate a class of statistical divergence, termed the integral probability metrics (Müller [1997), in a sample-based manner. For example, (rrjosl./Arjossky /Arjovsky et al. 2011 ) Gulrajani et al. (017); Miyato et al. (2018) estimate the Wasserstein-1 1 distance by enforcing the Lipschitz norm of the witness function to be bounded by 1. Li et al. 7): Binkkwski et al. (2018) consider Maximum Mean Discrepancy
(MMD) (Gretton et al. 2012) with learnable kernels. Bellemare et al. (2017) studies the energy
distance, an instance of the MMD (Sejdinovic et al.; 2013). Another important class of statisti- cal divergence, the optimal transport distance, is also studied under computational lens. Peyré &
Cuturi (2019) summarizes recent developments in computational optimal transport. Cuturi (2013); Feydy et al. (2019) study sinkhorn-based divergence, which solve an entropic regularized form of the optimal transport problem (Schrödinger) 1932) and may result in a physically more feasible distance measure in some real-world application. Zheng & Zhou| (202I) contributes to this line of research by developing conditional transport algorithm, which possesses better control over the and mode-seeking behaviors in distributional matching. In this paper, we consider the classical GAN structure approximately control the JSD between the state-action visitation frequencies, since the GAN structure is simple, effective and sell-studied, Other divergence metrics may also be applicable in our framework and are left for future work.
15

---

Under review as a conference paper at ICLR 2022
B FULL ALGORITHM
Algorithm 2 State-Action Joint Regularized Implicit Policy (GAN-JSD, Detailed Version)
Learning rate Ne. Nwii larget smoothing factor 3: noise distribution P=(x): policy network with parameter Φ; critic network Qo, and Qo, with parameters 0g 01,0;; discriminator network Dw with parameter we; generator loss Lg; standard deviation σB, oj; number of smoothed states Na; number of actions a' at a' Na; number of epochs for warm start Nwarmi policy frequency k. Output: Learned neural network parameters db 0,,02. w. (// Initialization) Initialize Φ, 0,,02, w. Initialize ø¹ +- db Oi c- 01. 022 c- Oz. Load dataset D. for each epoch do for each iteration within current epoch do
Input:
Sample a mini-batch of transitions B ((s, r. s') ~ {// Policy Evaluation)
For each s' E B sample NB with noise standard deviation for state-smoothing via.
8 s' + t,, - N
Sample N, corresponding actions à Far( 8) for each 8.
Calculate ((s.a) a)
Q (s. a) a() + NNNNN Cea,) Amin,=1.2 Qo ($, + (1 A) max Qoy (4, a)
Minimize the critic loss with respect to Oz. 1,2, over (s, a) E B, with learning rate noo
arg mine, 1 [8] Eianer (Qo, (8, a) a(s.a))
{// Policy Improvement)
Add state-smoothing to with noise standard deviation a, using ~ -V (0. 5 4- t.
Resample [B] new states
independent
Form the generator sample æ and data sample y using æ (s.a), ā a~ T y f (s.a) a) E B.
Calculate the generator loss (Φ) D Ex [log (1 Dw (x))) using æ and discriminator Dw.. if iteration count % k = 0 then if epoch count < Warm then Optimize policy with respect to (d) only, with learning rate ne. i.e.,
arg ming a (Ø)).
else Optimize policy with Icarning rate n for the target
arg ming [&] Ea- [min,=1.2 Q0 (s, a Lo(D).
end if
clse
Skip the policy improving step.
end
{// Training the Discriminator)
Optimize the discriminator D., maximize [8] E, log (w)( 18) Σ> [log (1 Dwc with
respect to w with learning rate nw-
{// Soft Update the Target Networks)
o -BΦ end for end for
B)6'; 0' & 50, (1 (133)) for j 1,2.
C PROOFS
We follow the offline RL literature (Liu et al. 2018 Nachum et al. 2019 Kallus & Zhou 2020
Mousavi et al. 2020 Zhang et al. 2020) assume the following regularity condition the MDP
16

---

Under review as a conference paper at ICLR 2022
structure, which ensures ergodicity and that the limiting state occupancy measure exists and equals to the stationary distribution of the chain.
Assumption 1 (Ergodicity of MDP). The MDP M is ergodic, i.e., the Markov chains associated with any To and any á under consideration is positive Harris recurrent (Baxendale 2011).
Lemma 3. Let A E RNXN be nonsingular and let + R x A- E IR Let AA RNXN be arbitrary perturbation on A. Assume that the norm N satisfies || ||Aa|]] S || for all and æ.
|AA|| |||]
(A + AA) (x + Ax) = b and
k(A)
then
|
((A)
|AII
A|l
K(A)
((A)
[AA
A
-((A)
TAIl
AA
Proof. Let = x + Ax, (xc + Ar)
AA
AA + AAT = =
Ax
A AA R. Then,
|AA
|| Ax
- ||A ||| |A|l (||=]] + 1| = k(A)
(1|=)) + -|Ax=))
|
||
|AA|
1 1-)((A) (A)
||||| |
S ((AA)
||>|||
|A|
||a||
x(A)
|A
||
x(A)
|
1 - x(A)
|AAI
|AAl
=(A)
TAA
since x(A) || ||||| < 1 by assumption.
Proof of Theorem By ergodicity, (s).). ds(s) R uniquely exists. For d (dg, d,} T (T Th). stationarity implies that d is an eigenvector of T associated with eigenvalue and furthermore
dd = - d= = I-TT T
(10)
STER(N
Since T is positive matrix, by the Cerron-Frobente theorem (Meyer 2000), is an eigenvalue of T with algebraic multiplicity, and hence geometry multiplicity, 1. The eigen-equation T it has unique solution e up to a constant multiplier. Since T d d. the eigenspace of T associated with eigenvalue 1 is span ((d)). Hence dim ker (I T 1 = rank [ -TT) N
rank
T
N. The reason is that if 3v + 0 s.t. T 0, then
To- -I-T T
1 1
-
v = c d for scalar c and 1 = c=0,
and hence v = cd which contradicts to v +
Since rank T N, dim (fv N+1 T xTT==O)
1. For such a + 3i {2,... . N
1} s.t. 22 * 0. WLOG, assume UN+1 * 0. Let A E R N be the first N rows of f, then
rank(A A) N. The reason is that if rank (A) < N, 3 # 0 w A = 0, then
(w)
T w T A 0
and UN+1 40 (w (w..0) .0 is not constant multiple of v = dim ker (TT) 2 2, which
contradicts to the fact that rank invertible.
T = N. Thus, we conclude that rank (A) N = A is
17

---

Under review as a conference paper at ICLR 2022
Let e(2) = .......... 0)² RR then Equation 10 implies A e(t), Plug in d,p, do, TTp Tb and define As, As similarly, we have As dø e!!), As db = For Tb and Tø. we notice that by Jensen's 's inequality,
N
A, = az) ITb (arls') - - (ar (0) da, / P (S Ss ='
Σ
(T To)
S Σ
/
P (S = = s' = =a.)
I (a, s*) T (ar | da
Tbb (ar 8²) - - (a=18") da
Tb (-1s)) - T (-1s¹)l 112
Therefore, by assumption,
N Σ j=1
Th - To = max
(Th
To)
$
max 1= _=.....N
T| (-1 To (-1s)) Se.
im I....NN
N
Let As As + AA. For AA
11. we notice that
1
1 T >>1
AAAl1 |A
Aoll1 <
T
I
0
(Tb TT)
"|,
[TB Tallax S
T
T
Notice that matrix 1-norm satisfies | <
M
||0||] for all matrix M and vector u, that
Abll 2 and that |db]] = Lemma
3 implies that
Adll
|| Ad|]
l| dolli
|dbl1
(((A))
[AA <((A))
(11)
KV (As))
C cR(Ab)
k(Ab)
1 -e((AA)
Kmma
0 - 0.
&emar
Proof of Theorem 2 By ergodicity, (s(s), do, (s) E RN uniquely exists, Vh. For E {ds, de....... do, and T E {T Thh T'....., T ). we follow the steps and notations in the proof
of Theorem to conclude that rank (A) = N = A is invertible and that Ad el)) E RV Plug in dø. do, To. Th, and define As, Ad, similarly, we have Agds = (2) As, dh, = e11) For the transition matrix Tū induced by the mixture of policies The we have
= To,( = Pns (S+1= = s'1S= = = =/ (S = s I' 8' A a:) To (ael 8') da,
Σ Wk / P (S = s Ss = 8¹ A, = aa) Tbe (at (8)) da; =
K
K
Σ k=1
wk T Ton.
=
k=1
and therefore T3 Ek Tog
For To, and To. as in the proof of Theorem by Jensen's inequality,
(Tor To);s s |T (-1s²) = (-1
18

---

Under review as a conference paper at ICLR 2022
Therefore, by assumption,
N (Thy sp) 5
|To - Tol = max i=
max iI......
The (-1 - T (.18*)]1 Se.
g=1
Let As = Ab. + AAbb. For
| |l we have
AAAb 11l = || - -Abr 111 S 1|,b, T 11 = Tb Toll S c.
Note that V Vk,l
Ab 111 21+1 Σ -i=1 The = 1, ||db, |1 1. Lemma
implies that
Ad [ = || = 1| doalli S - c- (Ab²)
<k (Ab))
Kmman.k
[bb 1
Nemax,
Thus for the relative distance between dæ and de, we have
K Σ k=1
|dbll
Σ do (s*)
Σ
wgdbo (8')
Σ
Wk
dbg
(s*)
Wk
1,
1=1
k=1
|d
doll2
K
dd
dolli =
Σ
(wx ds - wadbb))
|cb1l1
k=1
Σ
wk l|ds
deu
111
Σ
wk
C.mmz.k k
C Kmmaz
0.
Kmaaz,
Kmaz
km
A=1
Plug wi = 1/K, (1,..., ......,K) K} into the second to last equation, we get
|| doll2 S
e Kmaaz,
-> 0, as 6 +
c KKmaz,k
k=1
Remark. In general, dj To = -k= w dJ + +E waw, dj, T + d.. One sufficient condition is d, Th d, Vi,j = do, = db, Vi,j = Tb, To, Vi,j. similar to Ho & Ermon 2016 In such case, Ty reduces to a single policy, not mixture, and Theorem aapplies.
D TECHNICAL DETAILS
D. TOY EXPERIMENT
Denote the total sample size Nootll we follow the convention to construct the eight-Gaussian dataset as in Algorithm 3 In our experiment we use Niotal = 2000.
Algorithm 3 Constructing the Hight-Gaussian Dataset
Input: Total sample size Nrotal-
Output:
Generated dataset Daasssian
while Dataset size < Ntotal do
Draw random center e uniformly
c~ (- (1 -1) [,01,00000020000000000---4444440
Sample datapoint (x, ~ (xc,))x00 L2). Store æ in the dataset. end while
We are interested in the 2-D eight-Gaussian dataset because (a) the conditional distribution of p(y 1) is multi-modal in many x; and (b) interpolation is needed to fill-in the blanks between Gaussian-centers, where smooth-interpolltionn into circle is naturally expected.
19

---

Under review as a conference paper at ICLR 2022
To rephrase this dataset into offline reinforcement learning setting, define state and the corresponding as action. Note that in the behavior cloning algorithm, the information reward, next state, and the episodic termination is not required. Hence, the generated dataset Caussian serve as an offlin RI dataset readily applicable to train behavior cloning policies.
In order compare the ability approximate the behavior policy by the KL loss and the JSD loss, the Gaussian policy and the implicit policy, the conditionalldistriuutiin matching and the joint- visitation matching, fit a conditional VAE ("CVAE"), Gaussian generator conditional GAN ("G-CGAN") and conditional GAN ("CGAN") using the conditional-diittibttiin matching ap- proach similar to Kumar al. (2019). We fit, using basic joint-visitation matching strategy, a con- ditional GAN ("GAN"). As discussed in Appendix A the major distinction between "CGAN" and "GAN" that the former uses the same states constructing the generator samples and the data samples while the later resamples states.
The network architecture of our conditional VAE as follows.
Conditional Variational Auto-encooe auto-encoder (CVAE) in Toy Experiment
Encoder
Decoder
Linear (atttttsttteeemaaaaae
н)
Linear (state ceerraat__dmmllttet___ee H) BatchNormid (H)
BatchNormle d (H)
ReLU
ReLU
Linear (H, H/ /2)
Linear (H, H//2)
BatchNormid (H//2)
BatchNormid (2/
ReLU
ReLU Linear (H//2 action_ding
mean = Linear 12/2 latent_ dim)
log_std Linear (H/ /2, latent_dim)
with hidden dimension H = 100 and latent dimension latent.din = 50. CVAE is trained for 1200 epochs with mini-batch size of 100 and random seed 0, using the meea-sqquaed-errrrr as the reconstruction loss, and the Gaussian-case closed-form formula in Kingma & Welling 2013 for the KL term.
The network architecture of our conditional GAN, used in "CGAN" and "GAN," is as follows.
Conditional Generative Adversarial Nets (CGAN) in Toy Experiment
Generator
Discriminator
Linear
H)
Linear ceatteedddmaaacttinoo H) LeakyReIU (0.1) 1)
BatchNormid (H)
ReLU
Linear (H, H//2)
Linear (H, H//2)
LeakyReIU (0.1)
BatchNormi d (H//2)
Linear (H//2,
ReLU
Linear (H//2, action dim)
where the structure BatchNormid, LeakyReLU follows Radford et al. (016)) Here we again use H = 100, z_dim 50. Conditional GAN is trained for 2000 epochs with mini-baich size of 100 and random seed 0. We follow Radford et al. (2016) to train CGAN using Adam optimizer with 31 0.5.
The network architecture of our Gaussian-generator version conditional GAN, used the exper- iments of Appendix F, is follows.
Generator
Linear sistateedimm H)
BatchNormid (H)
ReLU
Linear (H, H/ /2)
BatchNormid (H/ (2/22)
ReLU
mean Linear (H//2, action_dim), log_std Linear HH//2, action_dim)
20

---

Under review as a conference paper at ICLR 2022
with Discriminator and other technical details the same CGAN. This
version
of CGAN is again trained for 2000 epochs with a mini-batch size of 100, random seed 0, and 31 = 0.5 in the Adam optimizer.
Our test set is formed by a random sample of 2000 new states (r) from [-1.5, 1.5] together with the states in the training set. The performance on the test set thus shows both the concentration on the eight centers and the smooth interpolation between centers, which translates into a good and smooth fit to the behavior policy. Figure I shows the training set ("Truth") and the kernel-density-estimaae plot of each methods.
D.2 REINFORCEMENT LEARNING EXPERIMENTS
In practice, the rollouts contained in the offline dataset have finite horizon, and thus special treatment is needed per appearance of the terminal states in calculating the Bellman update target. We follow the standard treatment (Mnih et al. 2013
Sutton & Barto 2018 to define the update target y as
r (s. + V (s',a') (s)
if s' is non-terminal state if s' is terminal state
@ as
where @' (s', a') refers to the expectation term in Equation 4 for basic algorithm (Section 4.1 I or the expectation term in Equation 8 for the enhanced versions with state-smoothing the Bellman Backup (Section 4.2).
For simplicity, we follow White 2016) choose the noise distribution p= (z) as the multivari- ate standard normal distribution, where the dimension of z is conveniently chosen as dim (z) = min(10, Sttateddim/ee /2). To sample from the implicit policy, for each state s, we first sample in- dependently z ~ .N (0, I). We then concatenate 8 with x and feed the resulting [8, x] into the deterministic policy network to generate stochastic actions. To sample from a small region around the next state s' (Section 4.2p, we keep the original s' and repeat it additionally Ng times. For each of the N1 replications, we add an independent Gaussian noise e ~ (N I) The original s' and its NB noisy replications are then fed into the implicit policy to sample the corresponding action.
For fair comparison, we use the same network architecture as the official implementation of the BCQ algorithm, which will be stated in detail in Sections DD2.1] Due to limited computational resources, we leave a fine-tuning of the noise distribution P= (z), the network architectures, and the optimization hyperparameters for future work, which also leaves room for further improving our results.
For more stable training of the policy, we adopt the warm start strategy (Kumar et al. 2020 Yue ctt al. 2020). Specifically, in the first Nwarm epochs, the policy is trained to minimize Cg only, with the same learning rate N7 the rest epochs that also try maximize the expected Q-values.
Datasets. We use the continuous control tasks provided by the D4RL dataset (Fu et al., 2021
to conduct algorithmic evaluations. Due limited computational resources, select therein the "medium-exper "medium-replay""" and "medium" datasets for the Hopper, HalfCheetah, Walker2d tasks in the Gym-MuJoCo domain, which are commonly used benchmarks in prior work (Fujimoto (o
et al. 2019 Kumar et al., 2019) Wu et al., 2019) Kumar et al.) 2020). We follow the literature (Cang al. 2021 Chen et al.
2021; Kostrikov al.) [2021] to not test on the "random" and "expert" datasets as they are less practical (Matsushima et [202 1) and can be respectively solved by directly using standard off-policy RI. algorithms (Agarwal et al. (222)) and the behavior cloning algorithms. We note that a comprehensive benchmarking of prior offline-RL RL. algorithms on the "expert" datasets is currently unavailable in the literature, which is out of the scope of this paper. Apart from the Cym-MuloCo domain, we also consider the Maze2D domair 3 of tasks for the non-Markovian data- collecting policy and the Adroit tasks (Rajeswaran et al. 20T8) for their sparse reward-signal and high dimensionality.
Evaluation Protocol. In all the experiments, we follow Fu et al. (022 to use the "vO" version of the datasets in the Gym-MuJoCo and Adroit domains. In our preliminary study, we find that the rollout results of some existing algorithms can be unstable across epochs in some datasets, even
We use the tasks "maazeddumazze""" "maze2d-medium," and "maze2d-large." We use the tasks humnn"" " "pen-cloned," "pen-expert," and "door-expent."
21

---

Under review as a conference paper at ICLR 2022
towards the end of training. To reduce the randomness in evaluation, we report, for our algorithm, the mean and standard deviation the last five rollouts across three random seeds {0. 1, We train all agents for 1000 epochs, where each epoch consists of 1000 mini-batch stochastic gradient descent steps. We rollout each agent for 10 episodes after each epoch of training.
D.2.1 GAN JOINT MATCHING
In approximately matching the Aansen-Shannon divergence between the state-action visitation fre- quencies via Generative Adversarial Nets, crucial step is to stably and effectively train the GAN structure. To this end, we adopt the following tricks from the literature.
To provide stronger gradients early in training, rather than training the policy कΦ to minimize
E [log (1 D (x))]
we follow (Goodfellow et al. 2014) to train T to maximize
Ex [log (x))
Motivated by (Radford et al. 2016), we LeakyReLU activation in both the generator and discriminator, with default negat veees 01.
To stabilize the training, we follow (Radford et al. 2016) to use a reduced momentum term 31 =
0.4 in the Adam optimizer (Kingma & 2014).
We follow (Radford al. 2016) to use actor and discriminator learning rate 7/ = 1/o 2 10-1
To avoid overfitting of the discriminator, we are motivated by Salimans et al. (2016) and Goodfel- Iow (2017) to use one-sided label smoothing with soft and noisy labels. Specifically, the labels for the data sample y is replaced with a random number between 0.8 and 1.0, instead of the original No label smoothing is applied for the generator sample æ, therefore their labels are all O.
The loss function for training the discriminator in GAN is the Binary Cross Entropy between the labels and the outputs from the discriminator.
Furthermore, motivated by TD3 (Fujimoto et al. 2018 and GAN (Section 2}. we update Tt ()) once per & updates of the critics and discriminator.
Table 2 shows the hyperparameters for our GAN joint-matching framework. Note that several sim- plifications are made to minimize hyperparameter tuning, such as fixing 7 = 1/o as (Rafford et al.
2016 and d a.j- We also fix N, to ease computation.
We comment that many of these hyperparameters can be set based on literature, for example, we use
Vob
1/00 x 10-1 as in Radford et al. (2016), Hee = 3 x 10-4 and Nwarm = 40 as in Kumar al. (2020), = 0.75 as in Fujimoto et al. (2019) and policy frequency A = 2 as Fujimoto et al. 20118. Unless specified, the same hyperparameter is used across all datasets.
Below we state the network architectures of the actor, critic, and the discriminator in GAN. Note that use pair of critic networks with the same architecture perform clipped double Q-learning.
Actor
Critic
Linear (state_dimmttee destate_dimtnoisee_eee
400)
Linear
dim, 400)
LeakyReLU
LeakyReLU
Linear (400, 300)
Linear (400, 300)
LeakyReLU
LeakyReLU Linear (300, 1)
Linear (300, action_di dim)
max _xc_action tanh
Discriminator in GAN
Linear Sittaatt_dimmacttin...o 400) Note that the network architectures follow LeakyReLU Linear (400, 300) LeakyReLU Linear (300, 1) Sigmoid
from Fujimoto et al. (2019) and that all the LeakyReLU activation uses the default negative.slope=0.00
22

---

Under review as a conference paper at ICLR 2022
Table Default Hyperparameters GAN joint matching.
Hyperparameter
Value
Optimizer
Adam Kingma & Ba (2014)
Learning rate fl
3 x 10
Learning rate T/. Thw
2 x 10 10-4
Log Lagrange multiplier log for non-Adroit datasets
4.0
Log Lagrange multiplier log for Adroit datasets
8.0
Evaluation frequency
10s
Training iterations
106
Batch size
512
Discount factor
0.99
Target network update rate 3
0.005
Weighting for clipped double Q-learning À
0.75
Noise distribution (z)
N (0,
Standard deviations
state smoothing on: DJ
3 x 10-4
Number of smoothed states in Bellman backup NB
50
Number of actions à at each & Na
1
Number of epochs for warm start warm
40
Policy frequency k Random seeds
2
{0,
D.2.2 RESULTS OF CQL
note that the official CQL GitHub repository does not provide hyperparameter settings for Maze2D and Adroit domain of tasks. For datasets in these two domains, we train an CQL agent us- ing five hyperparameter settings: four recommended Gym-MuJoCo settings and one recommended Ant-Maze setting. We then calculate the average (normalized) return of the last five rollouts over random seeds {0, 1, 1,2] and report the per-dataset best results across those five hyperparameter set- tings. We comment that this approach is conventonnn,,,ll,, but rather compensation for the missing of recommended hyperparameter settings, and may give CQL some advantage on the Maze2D and Adroit domains. For the Gym-MuJoCo domain, follow Kumar al. (2020) the recom- mended hyperparameter setting across all datasets.
D.2.3 ABLATION STUDY ON GAUSSIAN POLICY
The network architecture of the Gaussian policy we used in the ablation study (Section 5.2 follows common practice ((Harmoja al. 2018a Kumar al. (2200).
Gaussian Policy
Linear (state_dim, 400)
LeakyReLU
Linear (400, 300)
LeakyReLU
mean Linear (300, action dim)
log_std = Linear (300, action_dim)
Critics and discriminator are the same in the implicit policy case (Appendix D.2.1
For action selection from the Gaussian policy, a given state & is first mapped to the mean p(s)
and standard deviation vector (s). raw action is sampled Crew N' (H((s), diag (o²(s)))
Finally, araw is mapped into the action space max action tanh (raw).
For fair comparison, other technical details, including the training procedure and hyperparameter setting, exactly the the implicit policy case (Appendix D.2.1
23

---

Under review as a conference paper at ICLR 2022
E ADDITIONAL TABLES
E.I RAw SCORES OF THE MAIN TABLE
Table we report the un-normalized results corresponding to Table 1 for reference.
Table 3: Raw returns for experiments on the D4RL suite of tasks. We performs experiments on tasks from the Gym-Mojoco, Maze2D, and Adroit domains. For our algorithm, we report in this table the mean and standard deviation of the raw returns of the last five rollouts across three random seeds {0, 1, 1.2). We run CQL ourselves and report the average raw return of the last five rollouts over random seeds (0, 1, 2}. Results for other algorithms are from Fu al. 202T.
Task Name
SAC-oft
BEAR
BRAC-V
BCQ
CQL
GAN-Cond Basic
GAN-Joint: Basic
GAN-Joint
maze2ddumaze
145.6
28.6
1.7
41.5
93.5
96.0
103.3 15.2
79.2 23.3
maze mazziddmmdiumm
82.0
89.8
102.4
35.0
94.3
125.8 + 48.0
117.3 + 27.3
07.1++67.7
maze2d-large
123.5
105.4
197.3
halfeheerah-medimm
808.6
4897.0
5473.8
4767.9
4566.2
5155.3 + 30.8
5149.0 61.6
5197.4 31.6
walkerrdd medium
44.2
2717.0
3725.8
2441.0
2763.2
3009.3 1: 312.3
3212.9 1 416.3
3181.2
396.9
hogper mediumn
5.7
1674.5
990.4
1752.4
1103.6
2175.0 1: 692.5
2142.5 1: 499.6
1935,2 887.4
halfcheetta medium rediam-repply
581.3
4517.9
5640.6
4463.9
5105.5
3726.5
+ 295.0
3599.9 + 342.3
3832.3 281.1
wallker2d medium replay
688.7
hopper-mediune medium--replay
93.3
1076.8
-0.8
1057.8
941.3
812.1 1: 51.8
1166.9 = 168.8
938.9 1 82.3
halfehoctah- sestah-mediu---pper
-55.7
6349.6
7750.8
9150,7 1142.8
8936. 760.8
1259.3
walkker28 2d medium-expert
-5.1
1842.7
3747.5
2640.3
3666.6
3367.5 1: 631.3
3513.0 1: 691.8
3269.7 : 1008.6
32,9
3113.5
5.1
3588.5
3347.8
2193,1 + 693.5
2286.4 = 870.9
3232.2 + 944.3
pen-human
66.3
114.7
2149.0
159.8
1672.2 +
1915.7 407.
pen-cloned
797.6
885.4
22.2
1407.8
139.8
675.1 - 431.2
1028. 4 = 639.7
633.3 + 430.5
pen-expert
277.4
3254.1
3521.3
2954.5
3855.7 528.6
3943.6 429.0
4302.1 441,
door-expert
163.8
2980.1
.66.6
2850.7
2525.8
2904.6 + 110.3
3001. + 106.1
2979.9 108.9
E.2 TABLES FOR THE ABLATION STUDY
Table 4
correspond to the results for our ablation study in Section 5.2
Table 4: Normalized returns for comparing the implicit policy with the Gaussian policy on the basic algorithm (Section 4.1 on the D4RL suite of tasks. The reported number are the means and standard deviations of the normalized returns of the last five rollouts across three random seeds (0, 1,2).
Name
GAN-Joim-Matching: Basic GAN-Joint-Matching Basic, Gaussian Policy
maze2d-umaze umaze maze2d-medium
57.6 = 11.0
23,9 8.1
39.4 + 10.3
0.9 7.7
maze2d-large
52.1 20.8
3.3 + 0.7
halfcheetah-mmedinn
1..7:00
43.8 0.4
malker2d-medium
7..0
51.6 10.9
hopper-medium
79.8 + 14.1
halfcheetah- medium- replay
31.3
33.0
9.9
9.1
mepper-mediu-rreee replay
36.5
26.3 1 2.2
halfcheetah- -ed--mm-cddeeneee
74.2 6.1
76.8 66.866.7
lokkerrd-mediumn expert
76.5 15.1
99.62219
hopper-medium-exxt exppett
70,9 + 26.8
84.6 10.6
pen-human
61.0 - 13.7
$5.7 21.11
pen-cloned
31.3 = 21.5
36.6 14.6
pen-expert
129.1 + 14.4
118.8 15.1
door-expert
104.1 + 3.6
39.5 + 24.6
Average Score
59.6
46.5
24

---

Under review as a conference paper at ICLR 2022
Table Normalized returns comparing our algorithm its counterpart no state smoothing in joint matching of the action-visitation The reported number are the means and standard deviations of the normalized returns of the last five rollouts across three random seeds {0, 1,.
Task Name
GAN-Join-Matching Full Algorithm
GAN-Join-Matcching: No Stale-smootting in Joint Matching
maze 2d-mmaze
40.1 16.9
47.6
maze ad-medium
69.6 25.6
35.4 5.5
maze large
71.3 26.0
69.9 1 26.5
hulfchectah medium
44.1 + 0.3
44.0 0.4
walker2d-medimg
69.3 = 8.6
62.7 = 8.6
hopper-ma medium
27.3
halfeheetal Ieheetah medium- replay
33.1 2.3
31.8 2.7
walker2d-mee ced-eddiun--rppll
10.2 = 2.4
9.9
hopper- seee---eiii-----eee repayy
29.5 : 2.5
29.5 + 2.4
hallchettah medium expert
75.8 10.1
69.1 8.3
71.2 22.0
73.8 18.4
hoper-medium-expett
99.9 29.0
67.8 18.9
pen- hun-human
pen cloned
18.0 1 14.4
22.5 1 17.4
pen-expert door-expert
103.4
102.5 = 3.2
Average Score
61.4
59.1
Table 6: Normalized returns for comparing our full algorithm with its counterpart state-smoothing in the Bellman backup. reported number are means standard deviations normalized returns of last five rollouts across three random seeds (0, 1, 2].
Task Name
GAN-Joint Moich-Ma Full Algarithm GAN-Joint GAN-ocintMaiccingg No State-smouttiig Bellman Backup
maze2d-umaze
60.7 18.3
maze2d medium
69.6 25.6
$1.2 10.3
maze2d-large
halfcheetah medium
44.1 1 0.3
44.1 1 0.4
walker22 medium
60.7
hopper-medium
60.1 27.3
66.9 + 22,9
halfcheetah- salat--mddum--pppae
33.1 2.3
33.4 2.7
walker 2d medium replay
10.2 2.4
11.6 1 1.7
seppe-mediummeees
29.5
30.8 = 2.9
hallchettal medium expert
75.8 1 10.1
71.9 1 9.8
walker2d 2d medium-esppeee
71.2 22.0
83.3 31.0
axpper-medium-exper pen human
99.9 + 29.0
72.1 + 20.6
45.5 24.5
40.7 29.5
pen-cloned pen-expert dour-expre expeet
18.0 14.4
27.7 15.1
141.1 = 14.8
130.8 = 11.3
Average Score
61.4
58.8
Table 7: Normalized returns under several values of a σB a. (Appendix D in the full algorithm of GAN- maationnn-ma The reported number are the means and standard deviations of the normalized returns of the last five rollouts across three random seeds (0. 1,2).
Task Name
1
10
10-3
0=3x 10
10
maze2d-uman
48.4 +: 21.4
48.3 19.7
41.7 11.5
40.1
16.9
54.9 : 10.4
50.8 4 24.0
mave2d-med mediimm
58.7
33.6
48.7
64.0
239
696 25.6
46.9 1.9++1555 15.5
26.4 5.7
maze2d- large
87.1
21.
62.
713=26.0
61.0
62.3
hulfcheetah medium
43.0 0..0+:00 0.4
43.7
43.9 0.4
4
43.8 + 0.3
44.0 + 0.4
walker2d-medium
6..4779
10.3
69.3486
64.6 13.8
63.8
hopper medium
23.5
66.7 20.8
63.3 21.0
60.1 27.3
74.5 19.3
89.6 27.9
saffcheetah-meeiimm-ee
32.1424
31.5
3.,3
33,1 2.
31.
31.5 11.5
walker 2d medium replay
10.2
hopper-meet medium- replay
28.7 3.8
20.1+27
10.5+299
29.5+
31.3 1.9
29.2 1.5
halfcheetah- medium- expert
19.9
74,2 13.0
76.8 13.4
75. 10.1
71.3
70.7
walker2d medium expert
67.4+1600
63.4 22.2
69.7 17.3
71.2 22.0
63.4 23.1
77.2
18.4
hopper-medium--e medium-expent pen human
20.5 6.8
56.7.12.55
99.9 29.0
66.7 19.6
62.0
19.5
pen-cloned
4.5 1.9
19.6 11.8
23.3
13,2
18.0 14.4
36.6 18.4
40.0
20,8
pen expert dnor-expert
74.2
26.6
132.8
11.1
132.8 17.9
141.1 14.8
136.6 10.8
132.0
19.4
29.1 9.7
104.1 1.6
104.2
103.4 3.7
102.9 +.9
102.3 4.8
Average Score
41.3
57.4
59.4
61.4
60.3
59.5
25

---

Figure 2: Performance of approximating the behavior policy the eight-Gaussian dataset by conditional GAN with default (implicit) generator and Gaussian generator. conditional GAN ("CGAN") and Gaussian- generator conditional GAN ("G-CGAN") are fitted using the conditional-distriattin matching approach. A conditional GAN ("GAN") and a Gaussian-genas generator conditional GAN ("G-GAN") are fitted using the basic joint-visitation matching strategy (Section 4.1 Performance is judged by (1) clear concentration on the eight behavior Under review as a conference paper at ICLR 2022
F FURTHER COMPARISON BETWEEN IMPLICIT AND GAUSSIAN POLICY
In Section 5.2 we note that uni-modal stochastic policy, such as the Gaussian policy, is less flexible capture all the rewarding actions, on which an implicit policy may well. In this section we visualize such difference.
Figure compares the fitting of the eight-Gaussian toy dataset by implicit policy and Gaussian policy. Specifically, Figure 2a plots the dataset; Figure 26 plots CGAN with the default implicit generator (implicit policy) fitted by the conditional-disee disttibbuinni matching approach; Figure plots CGAN with Gaussian generator (Gaussian policy) fitted by the conditional-distrion -istribbutio matching approach; Figure 2d plots CGAN with implicit policy fitted by the basic joint-visitation matching strategy (Section 4. ; Figure pelptts CGAN with Gaussian policy fitted by the basic joint-visitation matching strategy. Experimental details are on Appendix D.I
(a) Truth
(b) CGAN
(c) G-CGAN
(d) GAN
(e) G-GAN
centers, (2) smooth interpolation between centers, which implies a good smooth policy, Details of this toy experiment are presented on Appendix D.I
We that whatever training strategies, Gaussian policies fail to learn multi-modal state-conditional action distributions, even if needed. Even though the Gaussian policy version of CGAN may still correctly capture some modes in the action distributions, an improvement CVAE, they miss other modes. Besides, these Gaussian policy versions interpolate less-smoothly between the centers. In offline RL, these weaknesses is related the missing of some rewarding actions and less-predictable action-choices at unseen states.
the mode-covering
To visualize the differences between the implicit and the Gaussian policy in the offline RL setting, we plot the kernel density estimates of the distributions of the first two difoonddimessonss in the "maze2d-umaze-vi" dataset, where performance difference is shown in Table Specifically, Figure plots the distribution of actions in the offline dataset. Figure 3b and 3c respectively plot action distributions produced by the final Gaussian policy and the final implicit policy generating Table
(a) Truth
(b) Gaussian Policy
(c) Implicit Policy
Figure 3: KDE plots of the first two dimensions actions in "maze2d-umaze-vi"" dataset. From to
right: (a) Action distribution in the offline dataset; (b) Action distribution produced by the final Gaussian policy in Table z (c) Action distribution by the final implicit policy (Section 4.] in Table
26

---

Under review as a conference paper at ICLR 2022
We see from Figure 3a and Figure 3b that Gaussian policy leaves out two action modes, namely, modes on the upper-left and upper-right corners. Figure shows that our implicit policy does cap- ture all the modes shown in Figure 3a Note that "maze2d-umaze" is a navigation task requiring agents to reach a goal location (Fu et al. 2021). Gaussian policy thus may miss out some direc- tions in the offline dataset pertaining to short paths the goal state, which may explain its inferior performance on this dataset in Table
27


Reviews:
[{"review_id": "Y3T4J-S0pIJ", "reviewer": "Reviewer_aVY7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "$\\textbf{Strengths:}$\n&nbsp;\n\nMethod\n- This direct matching state-action joint access alleviates the problem of uncontrollable extrapolation errors in the estimation of the Q-value function, because the proximity of the access frequency leads to a decrease in the chance of estimating the Q-value of the state-action pair far away from the offline data set.\n- Since the state-action pairs in the offline data set are all regarded as samples from joint visits, the problem in the policy matching method that fits the distribution of each state-conditional action on only one data point can be avoided.\n- This method implicitly encourages the smoothness of state-action mapping, that is, similar states should have similar actions. This smoothing feature helps to ensure that the policy is reliable when promoting to unseen states. \n\nExperiments\n- The authors show the effectiveness of implicit policy, state-action-visitation matching, and their full algorithm in the experimental results. I believe the most valuable experiments are the toy experiments to illustrate their motivations as the findings in the toy experiments may help the policy constraint methods that need to recover the behavior policies to improve their methods further.\n\nWriting\n- The paper is well structured and clearly written. I enjoy reading it. \n\n&nbsp;\n\n$\\textbf{Weaknesses:}$\n&nbsp;\n\nExperiments\n- The experimental results seem not very prominent. One reason may be because the experimental setting does not highlight the proposed method’s advantages as there are too few modalities in the used D4RL offline data. I suggest the authors to verify their method on datasets with more modalities to further discover the method’s full potential.\n\n", "summary_of_the_paper": "One of the existing Offline RL algorithms is to constrain the learned policy, such as constraining the learned policy to be consistent with the behavior policy itself or the action distribution based on state conditions, or adopting a Gaussian policy. However, in any given state s, the potential action value function in the action space may have multiple local maxima. Therefore, only one point can be used to evaluate whether the current policy is close to the behavior policy in any specific state, which may not well reflect the true difference between the two conditional distributions, and the stochastic policy that leads to determinism or unimodality may only capture one of the local optima leads to ignoring many other high-value actions. Especially when such policies such as CVAE and other models exhibit strong model-recovering behavior, large probability density may be assigned to low data density areas, resulting in exaggerating the density of high-value actions.\n\nIn response to the above problems, the method proposed in this article:\n- Uses an implicit policy to capture the multi-modality of the action value function\n- Controls the <s,a> access frequency in the data set to be as consistent as possible with the <s,a> access frequency of the learned policy as an additional training target, without the need to explicitly construct a behavior policy\n- provides the theory that proves, the method of matching behavior policy and current policy is equivalent to matching their corresponding <s,a> access frequency\n", "summary_of_the_review": "This paper developed a framework that supports learning a flexible while well-regularized policy in offline RL. The technical novelty is sufficient and significant. The empirical results show the method reach satisfied performance on D4RL while I'm looking forward to seeing potentially better results on datasets with more modalities. Overall, I recommend to accept this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Us32crnavUB", "reviewer": "Reviewer_H1Xc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The idea of learning policy with latent/implicit structure can be promising in offline RL as the latent structure might suffer less from the distribution shift. The paper borrows some method from GAN training which is novel to offline RL and show they are effective.\n2. The paper proposed a number of practical enhancement that can be generalized to a large range of offline policy learning algorithms in section 4.2. These methods are well motivated and tested in the ablation study. I think this study is helpful for many offline RL work.\n\nWeakness:\n1. The theory in section 3 is weak and closely related to many previous results. In Kumar et al. (2019), Theorem 4.2 also showed a bound on the visitation ratio when the policy distance is small enough. The results is very similar to this in some sense. However both of them are not really enough to support the claim that it is sufficient to require the policy distance to be small to bound the visitation distance. The key part is in k_max. Assuming policy distance smaller than 1/k_max which is probably requiring some exponentially (in horizon) small number, and cannot hold in any meaningful scenarios. In the end, any constant policy difference can still propagate exponentially to the state visitation distance.\n2. The key algorithmic contribution, $L_g(\\phi)$, seems vague in some sense. Since the state distribution is fixed, and the state samples in x and y are two i.i.d. samples, the loss can be decoupled into two parts: Construct another example $x' = (\\tilde{s}, a')$ where $\\tilde{s}$ is the same as in $x$ and $a' \\sim \\pi_b$. The the distance between $x'$ and $y$ is a constant, then the regularization loss is actually optimizing the conditional action distribution between $\\pi_\\phi$ and $\\pi_b$, which seems not so different from previous work, and it is not so clear why the form in paper is more promising.", "summary_of_the_paper": "This paper proposed a regularized policy learning algorithm for offline reinforcement learning. The implicit policy is trained by a GAN-like framework, and the regularization loss constrains the distance between learned policy and behavior policy. Experiments and ablation study on the D4RL dataset validate the proposed framework and algorithmic designs.", "summary_of_the_review": "Overall, this paper proposed a novel algorithmic idea in offline RL. I think the algorithmic contribution in this paper can be helpful to the community. Though some theoretical claim/results in the paper needs to be better explained.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HtDr84hoR_9", "reviewer": "Reviewer_Ht1b", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1.\tThe paper is well written and easy to follow. It provides good motivation for using implicit policies and regularizing the state-action visitation distribution. The visualization in Figure 1 is also helpful. \n\nWeaknesses\n1.\tThe theoretical analysis assumes $T_{i,j} > 0$. Although the paper claims that the assumption can be satisfied by substituting the zero entries with small numbers, doesn’t that change $||d_\\phi – d_b||_1$? Does that bound still hold (how much error would be introduced if we substitute all zero entries)? \n2.\tI don’t understand the purpose of the theoretical analysis. The paper mentions that the goal is to show that “this line of prior work is compatible in theory with our proposed framework”, but it seems like the theory just says that regularizing policy implies regularizing the state-action visitation. Is that what the paper trying to show or do I miss anything here? Moreover,  I think similar analysis has been done in the existing works (e.g., Appendix A and B in [1] or Section 4.2 of [2]). Can you explain how the analysis in the paper differs from the existing works? What is novelty in the theoretical analysis in this paper?  \n3.\tIn the empirical section, the paper mentions that “we do not test on the random and expert datasets as they are less practical”. However, I think the point of the empirical study is to understand how the proposed algorithm performs compared to baselines, not how practical the algorithm or the dataset is. Therefore, I think the empirical results would be clearer with all dataset reported. \n4.\tThe results in Table 1 do not show confidence interval for baselines and the average scores. I find it hard to see a clear conclusion from the table. For example, for many tasks, it is not clear whether the proposed algorithm outperforms baselines significantly or not. Moreover, showing the proposed algorithms have higher average scores seems pointless since the scores are not normalized (please correct me if I am wrong). It is possible that one algorithm wins on one task with a big margin and loses on other tasks with a small margin to have a higher average score. Is this algorithm better?\n5.\tOne of the main points of the paper is about regularizing the state-action visitation distribution.  However, in Section 4.1, the paper uses the state distribution from the dataset to approximate $d_{\\phi}(s)$. Doesn’t the algorithm basically reduce to policy regularizing algorithm by doing this approximation? Approximating the state-action visitation of a target policy is still an ongoing topic in the OPE community, so I don’t think it is okay to just say that we can use the dataset to approximate the state distribution. This is my biggest concern of the paper. \n\n[1] John  Schulman, et al. “Trust region policy optimization.”  \n[2] Sergey Levine, et al. “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems”\n", "summary_of_the_paper": "The paper proposes offline RL algorithms which consist of several key components: (1) use implicit policies instead of Gaussian policies (2) regularize the state-action visitation distribution instead of the policy. The paper provides a theoretical analysis to show the equivalence between regularizing policy and regularizing state-action visitation, and provides an empirical study on several deep RL environments. ", "summary_of_the_review": "I am leaning towards a recommendation to reject this paper since the proposed algorithm does not properly regularize the state-action visitation, and the theoretical and empirical results do not support the main points of this paper as mentioned in the Main Review section. \n\n---\nAfter the author response, I think the paper provides a good empirical study on state-action joint regularization methods, so I increased my score for the empirical novelty and significance. However, my main concern remains so I am keeping my overall score.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose to use implicit policies (similar to a conditional GAN) with a GAN-inspired regularizer. Theoretically, they show an equivalence between policy-matching and state-action-visitation matching. Finally, they evaluate their approach on D4RL and showed improved performance as well as ablations.

Reviewers did not find the theoretical contribution to be significant.  While the exact form may be novel, the general result has been shown in previous work and they only use the general result as a loose motivation for their approach. All reviewers acknowledge their empirical improvements as the primary strength of the paper. While a central component of their story is joint state-action regularization, Reviewer Ht1b identified that their proposed approach does not appear to directly regularize the joint state-action distribution, but rather behaves more similarly to existing policy constraint methods. I agree with Reviewer Ht1b and after much back-and-forth discussion (both Reviewer Ht1b and myself) with the authors, I have not been persuaded otherwise.

The paper has a lot of potential - strong empirical results, but the justification and explanation of the method needs to be rewritten in light of the policy constraint regularization or a stronger argument needs to be put forth in support of joint state-action regularization. I don't think this diminishes the results though, but without this substantial revision, I cannot accept the paper at this time.
Decision: Reject