Offline Reinforcement Learning with Mixture of Deterministic Policies
Authors: Takayuki Osa
Akinobu Hayashi
Pranav Deo
Naoki Morihira
Takahide Yoshiike
Venue: Accepted by TMLR ()

Abstract: Offline reinforcement learning (RL) has recently attracted considerable attention as an approach for utilizing past experiences to learn a policy.  Recent studies have reported the challenges of offline RL, such as estimating the values of actions that are outside the data distribution. To mitigate offline RL issues, we propose an algorithm that leverages a mixture of deterministic policies. When the data distribution is multimodal, fitting a policy modeled with a unimodal distribution, such as Gaussian distribution, may lead to interpolation between separate modes, thereby resulting in the value estimation of actions that are outside the data distribution. In our framework, the state-action space is divided by learning discrete latent variables, and the sub-policies corresponding to each region are trained. The proposed algorithm was derived by considering the variational lower bound of the offline RL objective function. We show empirically that the use of the proposed mixture policy can reduce the accumulation of the critic loss in offline RL, which was reported in previous studies. Experimental results also indicate that using a mixture of deterministic policies in offline RL improves the performance with the D4RL benchmarking datasets.

Published in Transactions on Machine Learning Research (09/2023)
Offline Reinforcement Learning with Mixture of Deterministic Policies
Takayuki Osa
os0@mit.autokgo..ac.p
The University of Tokyo, RIKEN
Akinobu Hayashi Honda RED Co., Ltd.
akinobu. akinobuu hayashi@@j..oom honda
Pranav Deo Honda RED Co., Ltd.
pranad deo@jp.hondd honda
Naoki Morihira Honda RED Co., Ltd.
naoki_ moooi__mordiirr@@phhooddd
Takahide Yoshiike Honda RED Co., Ltd.
takahide takahide-_oshiikeeee gostedee_ehhhieej..ooooo
Reviewed on OpenReview:
https://openreviii http:pawwewaaa.eeeeec/mmraadddad
Abstract
Offline reinforcement learning (RL) has recently attracted considerable attention as an ap- proach for utilizing past experiences to learn policy. Recent studies have reported the challenges of offline RL, such as estimating the values of actions that are outside the data distribution. To mitigate offline RL issues, we propose an algorithm that leverages a mixture of deterministic policies. When the data distribution is multimodal, fitting a policy mod- eled with a unimodal distribution, such as Gaussian distribution, may lead to interpolation between separate modes, thereby resulting in the value cstimation of actions that are out- side the data distribution. In our framework, the stale-action space is divided by learning discrete latent variables, and the sub-policies corresponding to each region trained. The proposed algorithm was derived by considering the variational lower bound of the offline RL objective function. We show empirically that the use of the proposed mixture policy can reduce the accumulation of the critic loss offline RI., which was reported in previous studies. Experimental results also indicate that using a mixture of deterministic policies in offline RI. improves the performance with the D4RL benchmarking datasets.
1 Introduction
Reinforcement learning (RL) (utton & Barto 2018) has achieved remarkable success in various applica- tions. Many of its successes have been achieved in online learning settings where the RL agent interacts with the environment during the learning process. However, such interactions often time-consuming and computationally expensive. The aim of reducing the number of interactions in RL has spurred active interest in offline RL (Levine et al. (2020), also known as batch RL (Lange el al.) 2012). In offline RL, the goal is to learn the optimal policy from a prepared dataset generated through arbitrary and unknown processes. Prior work on offline RL has focused on how to avoid estimating the Q-values of actions that are outside the data distribution (Fujimoto al.) 2019 Fujimoto & Gu (021]. In this study, we propose addressing it from the perspective of the policy structure. Our hypothesis that, if the data distribution in given

---

Published in Transactions on Machine Learning Research (09/2023)
Samples with high Q-values
Unimodal distribution
fitted samples
Fit deterministcc policy for each region
Region there no datapoint
(a) Samples in state-action space,
(b) Result of fitting unimodal
(c) Proposed approach.
distribution.
Figure 1: Illustration of the proposed approach. (a) In offline RL, the distribution of samples is often multimodal; (b) Fitting unimodal distribution to such samples can lead to generating the action out of the data distribution; (c) In the proposed approach, first the latent discrete variable of the stale-action space is learned, and then deterministic policy is learned for each region.
dataset is multimodal, the evaluation of the out-of-distribution actions can be reduced by leveraging a policy conditioned on discrete latent variables, which can be interpreted as dividing the state-action space and learning sub-policics for each region. When the data distribution is multimodal, as shown in Figure [a), fitting a policy modeled with a unimodal distribution, such as Gaussian distribution, may lead to interpola- tion between separate modes, thereby resulting in the value estimation of actions that are outside the data distribution (Figure (b)). To avoid this, we employ mixture of deterministic policies (Figure [c)). We divide the state-action space by learning discrete latent variables and learn the sub-policies for each region. Ideally, this approach can help avoid interpolating separate modes of the data distribution.
The main contributions of this study are as follows: 1) it provides a practical algorithm for training a mixture of deterministic policies in offline RL and 2) investigates the effect of policy structure in offline RL. Although it expected that a mixture of deterministic policies has advantages over a monolithic policy, it is not trivial to train a mixture of deterministic policies. We derived the proposed algorithm by considering the variational lower bound of the oflline RL objective function. We refer to the proposed algorithm deterministic mixture policy optimization (DMPO). Additionally, we proposed a regularization technique for mixture policy based on mutual information. We empirically demonstrate that the proposed regularization technique improves the performance of the proposed algorithm. A previous study (Brandfonbrener el al., 2021) reported the accumulation of critic loss values during the training phase, which was attributed to generating out-of-distribution actions. In our experiments, we investigated the effect of the policy structures in offline RL through comparison with methods that use a monolithic deterministic policy, Gaussian policy, and Gaussian mixture policy. We empirically show that the use of mixture of deterministic policies can reduce the accumulation of the approximation error in offline RL. Although a mixture of Gaussian policies has been used in the online RL literature, show that the use of a Gaussian mixture policy does not significantly improve the performance of an offline RL algorithm. Through experiments with benchmark tasks in D4RL (Fu et al. 2020), we demonstrate that the proposed algorithms are competitive with prevalent offline RL methods.
2 Related Work
Recent studies have shown that regularization is a crucial component of offline RL (Fujimoto al.] 2019:
Kumar et al. 2020 Levine al. 2020 Kostrikov al.] 2021). For example, Kostrikov et al. proposed a regularization based on Fisher divergence Kostrikov et al. 22221) and Fujimoto et al. showed that simply adding a behavior cloning term to the objective function in TD3 can achieve state-of-the-art performance D4RL benchmark tasks Fujimoto & Gu (2021). Other studics have investigated the structure of the critic, proposing the use of an ensemble of critics (An et al.) 2021) or offering a one-slep oflline RL approach (Gul- cehre et al. 2020 Brandfonbrener et al.] 2021 Goo & Niekum 2021). Previous studies (Fujimooo et al. 2019 Fujimoto & Gu 2021) have indicated that the source of the value approximation error is "extrapolation error" that occurs when the value of state-action pairs that are not contained in given dataset estimated. We hypothesize that such an "extrapolation error" can be mitigated by dividing the state-action space, which
2

---

Published in Transactions on Machine Learning Research (09/2023)
can be potentially achieved by learning discrete latent variables. We investigate the effect of incorporating policy structure as an inductive bias in offline RI, which has not been thoroughly investigated.
Learning the discrete latent variable in the context of RL is closely related to a mixture policy, where a policy is represented as a combination of a finite number of sub-policies. In a mixture policy, one of the sub-policies is activated for a given state, and the module that determines which sub-policy is to be used is often called the gating policy (Daniel et al. 2016). Because of its two-layered structure, mixture policy is also called hierorchical policy (Daniel et al. 2016). Although note that well-known hierarchical RL framework with temporal abstraction is the option critic (Bacon ct al. 2017). Because we consider policics without temporal abstraction, we use the term "mixture policy," following the terminology in Wulfmeeer et al. (2021). Previous studies have demonstrated the advantages
did not consider temporal abstraction in this study,
of mixture policies for online RL (Osa et al.) 2019 Zhang & Whiteson, 2019 Wulfmeier et al. 2020: 2021
Akrour et al.) 202T). In these existing methods, sub-policies are often trained to cover separate modes of the Q-function, which is similar to our approach. Although existing methods have leveraged the latent variable in offline RL (Zhou et al. 2020; Chen et al. 2021b 2022), the latent variable is continuous in these methods. For example, Chen et al. recently proposed algorithm called latent-van vatiibevariable wedghtagge-eighhtedd policy optimization (LAPO), which leverages continuous latent space for policy learning Chen et al. (2022). LAPO incorporates an importance weight based on the advantage function and learns the continuous latent variable. Although LAPO can achieve state-of-the-art performance on well-known benchmark tasks, we empirically show in this study that LAPO suffers from a surge of the critic loss during training.
3 Problem Formulation
Reinforcement Learning Consider a reinforcement learning problem under a Markov decision pro- cess (MDP) defined by a tuple AS,,, 7,d), where S is the state space, A is the action space, P(*++1|81, a+) is the transition probability density, r(s, a) is the reward function, 7 is the discount fac- tor, and d(so) is the probability density of the initial state, A policy (aass) S x A + R is defined as the conditional probability density over the actions given the states. The goal of RL is to identify policy that maximizes the expected return E[Ro [r), where the relurn is the sum of the discounted rewards over time given by RG = > > starting from state s and taking action a, then following policy a under given MDP (Sutton & Barto, 2018).
ax). ax). The Q-function, (T a), is defined as the expected return when
In offline RL, it is assuned that the learning agent is provided with a fixed dataset, D {(s, a, r comprising states, actions, and rewards collected by an unknown behavior policy. The goal of offline RL. is to obtain policy that maximizes the expected return using D without online interactions with the environment during the learning procoss.
Objective function We formulate the offline RL problem as follows: given dataset D = {(s,, az H obtained through the interactions between behavior policy B(a)s) and the environment, our goal is to obtain policy æ that maximizes the expected return. In the process of training a policy in oflline RL, the expected return is evaluated with respect to the states stored in the given dataset. Thus, the objective function is given by:
[TT) = EanD, [s- (s,
where f" is a function that quantifies the performance of policy n. There are several choices for f as indicated in |Schulman et al. (2016). TD3 employed the value function, f= a) = Q= a). and (s. A a) (Mnih et al. 2016). Other previous studies
A2C employed the advantage-functin employed shaping with an exponential function, such as f ("((sa)) exp Q (s.a) (Peters & Schaal (207))
or Id (s, a) = exp (AT a)) (Neumann & Peters. 2008 Wang ct al. 2018). Without loss of generality, we assume that the objective function is given by Equation
We derive the proposed algorithm by considering
the lower bound of the objective function of offline RL in Equation 1
3

---

Published in Transactions on Machine Learning Research (09/2023)
Mixture policy In this study, consider a mixture of policies given by
<(als)
Σ ZEZ
Tgate Tgate(z)s)) (abb)aa
(2)
where z is a discrete latent variable, (2))) is the gating policy that determines the value of the latent variable, and Tsub x) is the sub-policy that determines the action for a given & and z. We assume that sub-policy suub Faub(a)s,2, x) is deterministic; the sub-policy determines the action for given & and x in a deterministic manner as a po(s. 2), where po 2) parameterized by vector Additionally, we assume that the gating policy Tgatt 8) determines the latent variable
= arg max Qu (8, z²),,
(3)
x'
where (s, a) is the estimated Q-function parameterized by vector w. This gating policy is applicable to objective functions such as f" a) = exp (s" (= (" a), and f" a) = exp (s" Please refer Appendix A for details.
4 Training a mixture of deterministic policies by maximizing the variational lower bound
We consider a training procedure based on policy iteration (Sutton & Barto 2018 in which the critic and policy are iteratively improved. In this section, we describe the policy update procedure of the proposed method.
4.1 Variational lower bound for offline RL
To derive the update rule for policy parameter O, we first consider the lower bound of objective function ((r) in Equation []] We assume that f* a) in Equation is approximated with for (s, a), which is parameterized with vector w. In manner similar to Dayan & Hinton (1977; Kober & Peters (2011).
when for (s, a) > 0 for any and a, we can determine the lower bound of log J(T) using Jensen's incquality follows:
log J(T) = log / d (s)) a ddddda
(4)
Ta(a
log ({)))(aa)s
fo (s, a a)dsda
(5)
3(ass)
/ d (s)))(())))))))))))))))))))))))) log
ro(a|s)
B(ass) fs, a)dsda
(6)
(,, log (o a) E(DD)D B(a)B)) ((0
(7)
where B(als) is the behavior policy used for collecting the given dataset, and (s) is the stationary dis- tribution over the state induced by executing behavior policy B(als). The second term in Equation 7 is independent of policy parameter 0. Thus, we can maximize the lower bound of J(x) by maximizing Enn log (a (s) a.). When employ (" a) exp (A" a)), and the policy is Gaussian, the resulting algorithm is equivalent to AWAC (Nair et al. 2020). To employ mixture policy with discrete latent variable, we further analyze the objective function in Equation 7 As in Kingma & Welling (2014):
Sohn et al. (2015), we obtain a variant of the variational lower bound of the conditional log-like likllikkihhoo:
log (og 2 - DKI. (gs (=)ss(=))ss) (esp(()e) (log To (aaaasss z)]
Levae a;; Φ)
(8)
where (0(2|s, a) the approximate posterior distribution parameterized with vector Φ, and p(z[s) is the true posterior distribution. The derivation of Equation is provided in Appendix B Although it. is often

---

Published in Transactions on Machine Learning Research (09/2023)
assumed in prior studies (Fujimoto et al. 2019 that z is statistically independent of 8, that is, p(z[s) = p(z); in our framework, p(z[s) should represent the behavior of the gating policy, =0(=|s). In our framework, the gating policy (=)s) determines the latent variable as = arg max (s, u( z')) However, the gating policy not explicitly modeled in our framework because it would increase computational complexity. To approximale the galing policy represented by the argmna function over the Q-function, we used the softmax distribution, which often used to approximate the argmax function, given by
exp (Qw uo uo 2)))
p(2(8) Σ Σxez exp (Q 8, ( z)))
(9)
Since we employ double-clipped Q-learning as in Fujimoto et al. 2018). we compute
Qu (s., (o)s)))
min Qw, (o) 1=12
(10)
in our implementation. The second term in Equation is approximated as the mean squared error, similar to that in the standard implementation of VAE. Based on Equation 7 and Equation function for training the mixture policy as follows:
S we obtain the objective
N (0, Σ (8i ai)o a;; 0,
(11)
This objective can be regarded as the weighted maximum likelihood (Kober & Peters 2011 of a mixture policy. Our objective function can be viewed as reconstructing the state-action pairs with adaptive weights, similar to that in Peters & Schaal (2007): ; Nair et al. (020)) Therefore, the policy samples actions within the support and does not evaluate out-of-distribution actions. The primary difference between the proposed and existing methods (Peters & Schaal, 2007; Nair et al., 2020) that: the use of a mixture of policies conditioned on discrete latent variables in our approach can be regarded dividing the state-action space. For example, in AWAC (Nair et al.) 2020), unimodal policy
used to reconstruct all of the "good" actions in the given dataset. However, in the context of offline RL, the given dataset may contain samples collected by diverse behaviors and enforcing the policy to cover all modes in the dataset can degrade the resulting performance. In our approach, policy Te(a) s, 2) is encouraged to mimic the state-action pairs that are assigned to the same values of z without mimicking the actions that are assigned to different values of 2.
A Approximation gap When training a stochastic policy, the first term in Equation 7 can be directly maximized because it is trivial to compute the expected log-likelihood Ellog [{]ss)] However, when a policy is given by mixture of deterministic policies, it is not trivial. For this reason,
used the variational lower bound in Equation g In addition, Ellog T [als, 2)) is replaced with MSE as in VAE. As described in Cremer 22018) the use of the objective in Equuaiin instead of log (als) leads to the approximation gap, EE log T 8) Levae (s,
et
al.
s, a) as in VAE. Although addressing the approximation gap using techniques investigated in Cremer et al. (2018 may improve the performance of DMPO, such investigation left for future work.
4.2 Mutual-information-baa
regularization
To improve the performance of the mixture of deterministic policies, we propose a regularization technique for a mixture policy based on the mutual information (MI) between & and the state action pair (s, a), which we denote by T(2; s, a). As shown in Barber & Agakov (2003), the variational lower bound of I(2; 8, a) is given follows:
= H(z) H(2(s, a) = B( [ogg(2((s H(())
(((a,))~((..) (DRI. (p(2|s, a) ()]ls((2|s, a))] B(e.a.))vp (2)ss H(2)
2 (s,,,,)>> [ + H(z).
(12)
where (2]2)8 a) is auxiliary distribution to approximate the posterior distribution p(z)s, a).
5

---

Published in Transactions on Machine Learning Research (09/2023)
Thus, the final objective function is follows:
C(0,0, 4)) (0, ()
Ex Bz)p(s) log (w) (8 z)).
(13)
MI-based regularization using the second term in Equation 13 encourages the diversity of the behaviors encoded in the sub-policy x(als, z). In Section 7 performance of the proposed method.
empirically show that this regularization improves the
To implement MI-based regularization, we introduced a network to represent gu(z|s, a) in addition to a network that represents the posterior distribution 4 (, While maximizing the objective CMr. in Equa- tion both the actor 2))s) and posterior distribution (0(2|s, a) are updated; however, the auxiliary distribution (z|s, a) is frozen. While maximizing Di Ez=p(z) log Iv (z)s; (s, 2)), both actor po 2) and auxiliary distribution Pe (z|s, a) are updated, but the posterior distribution (z(z s, a)is frozen. To max- imize log 9up (2)8;, Ho (8;, 2)), the latent variable is sampled from the prior distribution, that is, the uniform distribution in this case, and the maximization of loggv Iv (z) 8,, (8( =)) ) is approximated by minimizing the cross entropy loss between 2 and &, where the output of the network that represents ge (z(ss,
5 Training the critic for mixture of deterministic policies
To derive the objective function for training the critic for a mixture of deterministic policies using the gating policy in Equation 3] consider the following operator:
T2Q₂ r(8, a) + >Bar max Q2 pus', u(((,/1))
(14)
We refer to operator Te as the latent-mar- Q operator. Following the method in Ghasemipour et al. (002) we prove the following theorems.
Theorem 5.1. In the tabular setting, Tz is contraction operator in the Co norm. Hence, with repeated applications of Tx, any inátial Q function converges a unique fixed point.
The proof of Theorem 5.1 provided in Appendix
Theorem 5.2. Let Q= denote the unique fixed point achieved in Theorem 5.1 and ah denote the policy that chooses the latent variable as arg max Q(s, (us,)²)) and outputs the action given by uls, z) in determinislic manner. Then Q= the Q-value function corresponding to mz-
Proof. (Theorem
5.2 Rearranging Equation
14 with s' = arg max Qz "ulss,((s' ,((s,22)),, we obtain
T=Q= = r(s, a) + Ea (Qz(s',
(15)
Because Q= the unique fixed point of Tz, we have our result.
These theorems reveal that the latent-max-Q operator, Tz, retains the contraction and fixed-point existence properties. Based on these results, we estimate the Q-function by applying the latent-max-Q operator. In our implementation, we employed double-clipped Q-learning (Fujimoto et al. 2018 Thus, given dataset D, the critic is trained by minimizing
((w;)
Σ (())E)
[ (s., 8i
(16)
for j = 1,2, where target value is computed as
y: = ri + 7 max min Qw (6/s no (66, (66,z2)) z²)).
(17)
2'E J=1,2

---

Published in Transactions on Machine Learning Research (09/2023)
Algorithm Deterministic mixture policy optimization (DMPO)
Initialize the actor ,40, critic Qw, for j = 1, and the posterior (=(=)s,a)
for t= 1 to T do
Sample a minibatch {(8), a, s r:) from D
for cach element (s; 02 s/. rz) do
Compule the larget value y: + min awo (5% uo (s,, end for Update the critic by minimizing Ei=1 11 Qw, (8;; ai)ll for 1,2 if t mod dinterval = 0 then
M
Update the actor and the posterior by maximizing Equation 11
(optionally) Update the actor by maximizing E Bz=p(z) log Iep (e ss, uo z))
end if end for
6 Practical implementation
The proposed DMPO algorithm is summarized Algorithm Similar to that in TD3 (Fujimoto et al. the actor is updated once after dinterval updates of the critics. In our implementation, The discrete latent variable is represented by one-hot vector, and we used the Gumbel-Softmax trick to sample the discrete latent variable in a differentiable manner (Jang et al. 2017 Maddison et al. 2017). Herein, following Jang et al. (2017); Maddison et al. (2017). we assume that z is categorical variable with class probabilities (1,, ..2,..., 0g and categorical samples are encoded as k-dimensional one-hot vectors lying on the corners the (k 1)-dimensional simplex, A k-1,
2018
set dinterval
In the Gumbel-Softtmx trick, sample vectors
A k-1 are generated as follows:
exp ((log ai + Ga)/A)
(18)
2 exp ex (loga + Gz)/A)
where Gi is sampled from the Gumbel distribution as G; Gumbel 1), and À is the temperature. As the temperature À approaches O, the distribution of & smoothly approaches the categorical distribution p(z). As prior work on the VAE with the Gumbel-Softmax trick (Dupont 2018). we set = 0.67 in our implementation of DMPO.
There are several promising ways for learning discrete latent variable (van den Oord al. et al. 2019), and investigating the best way of learning the discrete latent variable is left for future work.
2017 Razavi
Additionally, we employed the state normalization method used in TD3+BC (Fujimoto & Gu 2021 During preliminary experiments, we found that when f* a) exp (bAT (8, a)) in Equation I1 scaling factor b has non-trivial effects on performance and the best value of b differs for each task. To avoid changing the scaling parameter for each task, we used the normalization of the advantage function as
a (A* a) (a))EDD) A (8, a)) (a) A à) ()) A (8.a)
fd (s) exp
(19)
where Dbatch is a mini-batch sampled from the given dataset D and a is a constant. We set = 10 for mujoco tasks and a = 5.0 for antmaze tasks in our experiments. For other hyperparameter details, please refer the Appendix
7 Experiments
We investigated the effect of policy structure on the resulting performance and training errors of critics. In the first experiment, we performed a comparative assessment of TD3+BC (Fujimoto & Gu 2021), AWAC (Nair
et al. 2020) and DMPO on toy problem where the distribution of samples in given dataset. is multimodal.

---

Published in Transactions on Machine Learning Research (09/2023)
2
(a) Task sctting and data samples.
(b) Result of TD3 BC.
(c) Result AWAC.
1
0-
-1
2
(d) Result LP-AWAC.
(e) Result of DMPO.
(f) Visualization the latent variable DMPO.
Figure 2: Performance on simple task with multimodal data distribution.
Table 1: Algorithm setup in the experiment.
TD3+BC
AWAC
LP-AWAC
DMPO double-clipped
double-clipped
double-clipped
double-clipped
Critic training
Q-learning
Q-learning
Q-lerning
Q-learning
deterministic
monolithic & deterministic
monolithic stochastic
on continuous
mixture & de- terministic
Policy type
latent action space
Regularization
BC term normalized
none normalized
none
none normalized
State
normalized
Advantage normalization
yes
Further, we conducted a quantitative comparison between the proposed and bascline methods with D4RL benchmark lasks (Fu et al. 2020 In the following section, we refer to the proposed method based on the objective in Equation I1 as DMPO, and variant of the proposed method with MI-based regularization in Equation 13 as infoDMPO. In both, the toy problem and the D4RL tasks, we used the author-prote prooiddd implementations of TD3+BC, and our implementations of DMPO and AWAC are based the author- provided implementation of TD3 Our implementation is available at https: httgggtttuuuu.eee cotuu...omTaakaaaa..
DMPO,
7.1 Multimodal data distribution on toy task
To show the effect multimodal data distribution a given dataset, we evaluated the performance of TD3+BC, AWAC, and DMPO on a toy task, as shown in Figure 2 We also evaluated the variant of AWAC that employs the policy structure usod in LAPO Chen al. (2022). which we refer to as LP-AWAC. In LP-AWAC, the continuous latent representations of state action pairs learned using conditional VAE with advantage weighting, and deterministic policy that outputs actions in the learned latent space is trained using DDPG. We found that the authors' implementation of LAPQ includes techniques to improve
http:/hhhhh.cccc/ppcceee/e////// of
8

---

Published in Transactions on Machine Learning Research (09/2023)
performance, such action normalization and clipping of the target value for the state-value function. While LP-AWAC employs the policy structure proposed by Chen et al. (2022), the implementation of LP-AWAC is largely modified from that of the authors' implementation of LAPO to minimize the difference among our implementation of AWAC, miXAWAC, LP-AWAC, and DMPO. LP-AWAC can be considored as a bascline method that incorporates a continuous latent variable in its policy structure. The implementation details of LP-AWAC is described in Appendix F The differences between the compared methods are summarized in Table D In our implementation of AWAC, LP-AWAC and DMPO, we used state normalization and double- clipped Q-learning in TD3+BC and the normalization of the advantage function described in Section G The difference among AWAC, LP-AWAC and DMPO indicates the effect of the policy representation.
In this toy task, the agent is represented as point mass, the state is the position of the point mass in two-dimensional space, and the action is the small displacement of the point mass. There are two goals in this task, which are indicated by red circles Figure 2 The blue circle denotes the starting position in Figure D and there are three obstacles, which are indicated by solid black circles. In this task, the reward is sparse: when the agent reaches one of the goals, it receives reward of 1.0, and the episode ends. If the agent makes contact with one of the obstacles, the agent receives reward of -1.0, and the episode ends. In the given dataset, trajectories for the two goals are provided, and there is no information on which goal the agent is heading to.
The scores are summarized in Table 2 Among the evalu-
Table 2: Performance on the toy task.
ated methods, only DMPO successfully solved this task. The policy obtained by TD3+BC did not reach its goal in stable manner, as shown in Figure bb) Similarly, as shown in Figure ( the policy learned by AWAC often slows down around point (0,0) and fails to reach the goal. This behavior implies that AWAC attempts to average over multiple modes of the distribution. In contrast, the policy learned by DMPO successfully reaches one of the goals. Because the main difference between AWAC and DMPO is the policy architccture, the result shows that the unimodal policy distribu- tion fails to deal with the multimoodal data distribution, whereas the mixture policy employed in DMPO successfully dealt with it. Similarly, the performance of LP-AWAC is significantly better than TD3+BC and AWAC, demonstrating the benefit of the policy structure based on the latent action space. On the other hand, the performance of DMPO was better than that of LP-AWAC, indicating the advantage of using the discrete latent variable in offline RL. The activation of the sub-policies is visualized in Figure 2(e). The color indicates the value of the discrete latent variable given by the gating policy, z" = arg max; Qw p1(8, z)). Figure dd) shows that different sub-policies are activated for different regions, thereby indicating that DMPO appropriately divides the state-action space.
TD3+BC
AWAC
LP-AWAC DMPO
-0.2+ 0.4 0.33+ 0.44
0.74:066
1.0+0.00
7.2 Effect of policy structure
We investigated the effect of policy structure by comparing the proposed method with existing methods that incorporate the importance weight based on the advantage function. We used AWAC as baseline methods. To investigate the difference between the mixture of the stochastic policies and the mixture of the deterministic policies, we evaluated a variant of AWAC with Gaussian mixture policies, which we refer to as miXAWAC. For mixAWAC, the Gumbel-Softmax trick was used to sample the discrete latent variable. All bascline methods used double-clipped Q-learning for the critic in this experiment.
The implementations of AWAC and DMPO were identical to those used in the previous experiment. In our evaluation, |Z| 8 was used for DMPO and infoDMPO. Appendix(pprrsennt the effect of the dimensionality of the discrete latent variables. In this study, we evaluated the baseline methods with mujoco-v2 and antmaze- v0 tasks.
7.2.1 Performance score on D4RL
A comparison between AWAC, mixAWAC, LW-A and DMPO is presented in Table 3 These methods incorporate importance weights based on the advantage function with different policy structures. Therefore, the differences between these methods indicate the effect of policy structure. In our experiments, we did not observe significant differences in the performance of AWAC and miXA This result indicates that
9

---

Published in Transactions on Machine Learning Research (09/2023)
Table 3: Comparison with methods incorporating advantage-weightiinn using D4RL-v2 datasets. Average
normalized scores over the last 10 test episodes and five seeds are shown. The boldface text indicates the best performance. HC HalfCheetah, HP Hopper, WK
Walker2d.
AWAC
mixAWAC
LP-AWAC
DMPO
Med. -R Med. Expert
HC
94.8=0.2
94.0+0.55
98.7
97.0==.0
НP
109.8=2.9
111.8=0.8
104.345.5
93.6215.1
WK
111.0=0.2
110.540.33
110.740.1
111.4=0.3
HC
92.710.88
92.1100.6
92.3.10.4
91.113.4
HP
98.6+10.7
102.0=17.5
90.5=21.6
78.4+19.0
WK
109.2=0.3
109.1#0.3
109.1+0.4
109.9=0.4
HC
40.9-0.6
41.510.4
39.8.0.3
45.2=0.8
HP
38.2=9.4
41.2+:.77
46.1
89.2=8.1
WK
65.0+15.7
67.748.88
50.245.5
8..1
HC
44.3-0.2
45.1=20.3
44.0:0.4
47.5=0.4
Med.
HP
57.513.0
57.2113.9
5281338
6.5
WK
81.0.2.55
4.8
77.412.7
49.4
HC
3.2=1.3
2.2+00.0
4.142.3
15.8=1.6
Rand.
HP
7.3-1.99
8,210.22
8.4.0.66
12.0
WK
3.1+1.0
4.9-1.1
4.0:1.22
2.542.6
umazz
49.8.6.2
57.4.6.2
56.6.14.1
83.6
Antmaze
umaze-d.
53.8113.0
46.816.9
66.615.55
43.217.8
med.-p.
0.0:0.0
0.0:1..0
0.0:0..0
77.0=5.1
med.-d.
0.0
0.0:10.0
0.0+0.0
56.8127.2
large-p. 0.0
0.0:10.0
0.0:0.00
1.0
large-d. 0.0
0.0.10.0
0.0.10.0
4.8
the use of Gaussian mixture policies does not lead to performance improvement. However, the performance of DMPO matched or exceeded that of AWAC, except for the Hopper-expert and hopper-medium-expee expert tasks. This result also confirms that the use of a mixture of deterministic policies is beneficial for these tasks, although the benefits would be task-dependent.
The difference between mixAWAC and DMPO implies difference between Gaussian mixture policy and a mixture of deterministic policy. In Gaussian mixture policy, there is possibility that one of the Gaussian components covers a large action space and interpolates the separate modes of action. If this happens, out- of-distribution ion actions will be generated by the learned policy. However, in a mixture of the deterministic policy, there is no such possibility that one of the components covers large action space.
In addition, DMPO outperformed LP-AWAC on mujoco-v2 and antmazo-v0 tasks. As the difference between DMPO and LP-AWAC indicates the difference between the discrete and continuous latent representations in our framework, this result also indicates that the use of a discrete latent variable is beneficial for offline RL tasks. A comparison with additional bascline methods is provided in Appendix E
7.2.2 Critic loss function
To investigate the effect of the policy structure the critic loss function, compared the value of the critic loss function among AWAC, mixAWAC, LP-AWAC, and DMPO. The normalized scores and value of the critic loss function during training are depicted in Figure B The value of the critic loss given by Equation 16 is plotted for every 5,000 updates. Previous studies have indicated that the critic loss value can accumulate over iterations (Brandfonbrener et al.) 2021). Figure 3 shows the accumulation of the critic loss AWAC on mujoco-v2 tasks. The difference between AWAC and mixAWAC indicates that using a Gaussian mixture policy often roduces the accumulation of the critic loss. The critic loss of mixAWAC is lower than that of AWAC in aallchetta--mediu--eelllll result shows that the use of multimodal policy reduce the accumulation of the critic loss in offline RL. In addition, the critic loss of DMPO is even lower than that of mixAWAC, and the result demonstrates that using a mixture of deterministic policies can further reduce the critic loss than using a Gaussian mixture policy. These results indicate that using a mixture of deterministic policies can reduce the generation of out-of-distribution actions, which is essential for offline RL.
and
(asks. This
Regarding LP- AWAC, the critic loss value increased rapidly at the beginning of the training. Although the critic loss value often decreases at. the end of the APPA training, the critic loss value still higher than
10

---

Published in Transactions on Machine Learning Research (09/202))
halfcheotah- haltchectah-mediimmrree v2
halfchectah Soatrnettah-mmlum--- expert-v2
walker2d modduum-repla--v
walker2d meddummmmmmmmmmmmmmmmm expert-v2
Nem Nomattee Sc28
1.0
Nomaltzed
0.4
AIVAC
0.75
Noralzed Sleed
1.0
0.2
LPAAWCC
0.50
0.5
0.25
0.0
0.0
0.00
0.50
1,00
0.50
0,75
0.25
100
0.50
1.00
Time steps (106)
Time stops (186)
Time stops (1e6)
Time steps (206)
halfcheetah-mmed umm-rpla.--2
haffcheeth-mmedlu-eeeppewvw
Malleer2b-eddum-eelayyy
malkkerddmeddu--epprrre
400
Chtcc (055
400
300
CAti loss
75
Crtcc loss
50
20
0.25
0.50
0,75
1.00
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.50
Time steps (1e6)
Time steps (1e6)
Time steps (1a6)
Time staps 1266)
Figure 3: Critic loss and normalized score during the training.
250
200
150
100
50
-40
-22
(a) AWAC
(b) miXAWAC
(c) LP-AWAC
(d) DMPO
Figure 4: Histogram of the Bellman errors after 20k steps on the satehbeett--mdiium-eee task.
that of DMPO. The surge the critic loss value indicates the generation out-of-distribution actions during training in LP-AWAC. Importantly, in DMPO, the value of the critic loss is clearly lower, and the performance of the policy is better than that of LP-AWAC. This result indicates that the of discrete latent variable can be more effective than using continuous latent variable these tasks. In Brandfonbrener al. (2021). it was shown that the accumulation of critic loss values can be reduced by introducing regularization. Our results indicate that the use mixture policy can also mitigate the accumulation of critic loss in offline RL, worth
which suggests the importance incorporating inductive bias in the policy structure. However, noting that the reduction in the critic loss given by Equation [6 does not necessarily improve the policy performance. In setthheettl-mdiimm---ee -epplluumeeppr-v-2, although the critic loss was significantly lower in DMPO than in AWAC, there no significant difference in performance between DMPO and AWAC. Recently, Fujimoto et al. indicated that lower value of the critic loss given by Equation [6 does not necessarily mean better performance, and the observation in Fujimoto et al.] (002)) aligns with our experiments. The metric to measure the accuracy of the value estimation is still an open problem in RL.
As another qualitative result, Figure shows the histograms of the Bellman error after training with 20 thousand steps. The Bellman error mixAWAC is distributed more widely than that in AWAC, indicating that the use of the Gaussian mixture policy can increase the variance during the training of the critic. In contrast, the distribution of the Bellman error in DMPO is more narrow than those of AWAC, mixAWAC. and LP-AWAC, indicating that the use of the mixture of deterministic policies may lead to reducing the variance during the critic training. This variance reduction could also be considered a reason why DMPO outperformed baseline methods.
7.3
Comparison with prevalent baselines
We compared the performance of the proposed method with that prevalent baselines. As baseline methods, we used TD3+BC, CQL (Kumar et al. 2020), IQL (Kostrikov et al. 2022), LAPO, and Diffusion et al. 2023 CQL incorporates conservative critic update and the entropy regularization. In the experiments reported in this section, used the authors' implementation of LAPO. Diffusion QL is recently proposed by
11

---

Published in Transactions on Machine Learning Research (09/2023)
Table 4: Results on mujoco tasks using D4RL-v2 datasets and AntMaze tasks. Average normalized scores
over the last 10 test episodes and five sceds are shown. HC = HalfChoetah, HP Hopper, WK Walker2d.
"Diff. QL" represents Diffusion QL proposed in Wang et al. (2023).
TD3+BC
CQL
IQL
LAPO
Diff. QL
DMPO
infoDMPO
(re-run)
(re-run)
(re-rum)
(re-run)
(re-run)
(ours)
(ours)
Expert
HC
96.3+0.9
22.0#9.6
96.1=1.5
95.440.3
86.3=15.9
97.011.0
95.6+2.0
HP
109.9+2.5
105.8+3.8
98.4+13.1
110.9+2.3
84.3=24.2
93.6=15.1
107.3
WK
110.21104
108.910.4
112.6.0.3
111.5:10.2
109.010.6
111.4-0.3
112.110.4
HC
89.4+7.2
38.448.4
90.7+t.33
94.3+1.1
83.8+15.3
91.1+3.4
91.4+2.5
Med.-
HP
95.519.4
88.4
73.9133.6
110.5+1.2
88.1
18.4=19.0
94,5
WK
110.210..3
109.211.9
111.4:1.1
111.010.2
110.1=0.6
109.9=0.4
110.1
HC
44.7
46.9+0.3
43.6+1.4
41.91100
45.6+0.6
45.2=0.8
46.7=0.6
Med.-
R
HP
13.8
95.5+1.7
90.6+14.3
59.7+14.2
56.1=24.0
89.2-8.1
98.5+2.0
WK
64.5.17.0
77.513.1
82.213.6
50.3118.6
84.1117.0
82.113.8
86.713.2
HC
48.2+0.33
48.2+0.4
48.2#0.2
45.7+0.3
46.7+0.7
47.5=0.4
48.6=0.4
Med.
HP
61.01:..2
77.414.0
61.243.5
56.215.1
57.1+114
71,21 6.5
86.417.6
WK
84.711.3
81.5+2.5
82.916.0
80.5+1.
62.1120.6
49.4
8..0
HC
11.520.6
24.1#1.
12.644.6
27.111.0
17.5+0.2
15.8=1.6
16.311.2
Rand.
HP
8.7+0.3
2.2+19
7.44033
15.218.6
0.8+0.5
12.0=10.0
20.419.8
WK
1.41199
4.3.17.9
5.511..6
2.2.115
6.213.4
2.51 2.6
2.312.0
Total
1010.0
930.3
1017.3
1012.4
942.1
1026.5
1102.1
Wang et al. (022)) and employs a diffusion model as a policy. We used the author implementation of Diffusion QL, and the results of Diffusion QL are based on the oflline model selection reported in Waan el al. (2023) IQL employs expectile regression for learning the critic to address the issue of generating out-of- actions during training. Because the aim of our study is to investigate the policy structure, the approach of IQL, which address the critic learning, is orthogonal to ours. IQL is the state-of-the-ar method for antmaze task on D4RL, which involves dealing with long horizons and requires "stitching" together sub-trajectories in a given dataset (Fu et al. 2020). In the implementation of IQL, several techniques, such as scheduling of the learning rate, were used to improve the performance. To compete with IQL on antmaze task, we also used techniques used in Chen al. (2022). Therefore, the implementations of DMPO and infoDMPO for antmaze tasks are slightly different from those for other tasks. In our preliminary experiment, we evaluated IQL using the techniques proposed in Chen et al. (2022), and observed that the original implementation of IQL showed better performance. Therefore, we used with the original implementation of IQL for comparison. In this experiment, we used the mujoco-v2, antmaze-vO, and adroit tasks on D4RL.
and i The boldface text indicates the best performance. In mujoco-v2 tasks, the performance of DMPO comparable/superte to that of the state- of-the-art methods. In addition, infoDMPO, which employs MI-based regularization, outperformed DMPO various tasks, and infoDMPO showed the best performance for 10 tasks among 15 mujoco-v2 tasks. This result. shows that encouraging the diversity of sub-policies using the proposed MI-based regularization is effective for DMPO.
A comparison of TD3+BC, CQL, and IQL is presented in Tables
The advantages of DMPO and infoDMPO over TD3+ BC and CQL are apparent for antmaze tasks. TD3+BC and CQL, did not work satisfactorily on antmaze tasks, indicating that techniques used these algorithms are not effective for such tasks. The performance of DMPO(ant ver.) and infoDMPO(ant ver.) on antmaze tasks is comparable to that of IQL and Diffusion QL, which are the state-of-the-ar method for these tasks. We observed similar results for adroit tasks. DMPO and infoDMPO clearly outperformed TD3+BC and CQL, and the performance of DMPO and infoDMPO was comparable to IQL. Considering that infoDMPO outperformed IQL on mujoco-v2 task, overall performance of infoDMPO is better than that of IQL. This result reveals that the use of a mixture of deterministic policies can result. in a significant. performance improvement in offline RL.
We also provide the result regarding the computational cost of infoDMPO in Table 7 We used a worksta- tion with GPU RTX A6000 and CPU Core 10-000E for this evaluation. The results indicate that the
12

---

Published in Transactions on Machine Learning Research (09/2023)
Table 5: Results on Aate tasks. Average normalized scores over the last 100 test episodes and five seeds
are shown.
TD3+BC
CQL
IQL
LAPO
Diff. QL
DMPO
infoDMPO (ant ver.)
(ant ver.)
(re-run)
(re-run)
(re-rum)
(re-run)
(re-run)
(ours)
(ours)
umaze
92.8.12.7
73.0.4.99
87.4
97.2.12.7
80.4135.3
92.8
89.4.15.1
Antmaze
umaze-d.
45.01:22.2
43.814.4
64.615.6
57.4111.7
8.0-21.9
32.6125.6
34.8118.00
med.-p.
0.0.10.0
9.046.4
74.618.1
73.8+4.88
40.5
63.0
62.6416.8
med.-d.
0.0+0..0
3.8+4.2
73.8+7.11
81.0+3.66
12.4+33.0
75.0-8.5
82.8+4.44
large-p. 0.0+:0.0
0.0+0.00
39.0+7..2
27.6+13.3
44.4+48.55
42.24-3.0
47.4+14.5
large-d. 0.0:0.0
0.0:0.44
48.019.0
26.2
48.6=48.8
56.614.5
38.0:t4.8
Table 6: Results on adroit tasks using the average normalized scores over the last 10 test episodes and five seeds.
TD8+BC
CQL(p)
IQL. (re-run)
LAPO
Diff. QL (re-run)
DMPO (ours)
infoDMPO (ours)
(re-run)
(re-run)
(re-run)
pen
0.8+8.0
98.3+81.8
88.8=21.2
78.9+14.1
42.1+57.5
86.1 8.8
94.8+16.5
Human
hammer
0.9+0.8
-7.1+0.1
1.0+0.22
1.140.4
0.3:+0.2
1.2+0.22
2.4
door
-0.3+0.0
-3.347.8
2.4:2.1
3.2+1.6
-0.420.00
1.3115
4.2+3.1
relocate
-0.3+0.0
0.3+2.4
0.0+0.00
0.0+0.0
-0.0:2.00
0.0 0.1
0.1+0.0
pen 0.547.0
-1.71:1.
39.2115.4
25,6112.2
19.01-2.2
36,0117.7
46.4116.7
Cloned
hammer
0.2:0.0
-7.020.1
0.940.4
0.7+0.4
0.2+0.0
0.820.6
1.2+0.33
door
-0.3:1.00
-9.400.0
0.6:
0.7:0.9
-0.310.00
0.0:0.00
0.810.8
relocate
-0.310.0
-2.110.0
-0.210.0
-0.210.0
-0.210.1
-0.210.0
-0.210.0
computational cost of Diffusion QL, approximately three times higher than that of infoDMPO. Therefore, the computational cost is also an advantage of infoDMPO over Diffusion QL.
As qualitative evaluation, we investigated the activation of sub-policies in DMPO. Activation of sub-policies in DMPO on the pen-human-v0 task is depicted in Figure In Figure 5 B(a,, the top row depicts the state at the 20th, 40th, 60th, and 80th time steps, and graphs in the middle row of the figure show the action-value of each sub-policy at each state, Qu (4(8 Figures 5 (a) and (b) show the results for different episodes. A previous study (Smith et al., 2018)) reported that in the option-critic framework Bacon et al. (2017) only few options are activated and that the remaining options do not learn meaningful behaviors. In contrast, the results in Figure 5 show that the value of each of the sub-policies (w, (s, ((s, z)) changes over time, and various sub-policies are activated during execution. This result implies the following: meaningful sub-policies are learned in DMPO and different behaviors are adaptively used to perform complicated manipulation task.
8 Limitation of the proposed method
In this work, we proposed a method based on a mixture of deterministic policies, which implicitly divides the state-action space by learning discrete latent variables. While the experimental results demonstrate that our method DMPO can avoids the accumulation of the critic loss, there are problems which cannot be addressed by our approach. For example, if the dataset contains only covers subset. of actions, there is the potential to overestimate the value of actions that not contained in the dataset. In such cases, dividing the state action space is not sufficient to avoid generating OOD actions, and it will be necessary to regularization to avoid the overestination of Q-values such as CQL Kumar et al. 2020
In addition, while DMPO demonstrated the performance comparable to Diffusion QL on mujoco tasks in D4RL, a inffusion-model--eee based policy is evidently more expressive than a mixture of deterministic policies. policy
When the data distribution in given dataset highly complex in offinne RL, diffusion-mool should demonstrate its advantage over mixture of deterministic policies.
13

---

Published in Transactions on Machine Learning Research (09/2023)
Table 7: Wall clock time for training and inference.
Time for training with million sleps 170 [in)]
Action inference time
inFoDMPO Diffusion QL
1.2-1.3 [ms]
600 [min]]
4.0-4.3 [ms]]
20
40
60
80
100
7.
7, 3
7 20
40
60
80
100
z==6
Z =
z=
Figure 5: Visualization of sub-policy activation in the pen-human-v0 task. The top depicts the state at the 20th, 40th, 60th, and 80th time steps; the graphs in the middle row depicts the action-values of each sub-policy at each state.
9 Conclusion
We presented DMPO, an algorithm for training mixture of deterministic policies in offline RL. This algorithm be interpreted as approach that divides the state-action space by learning the discrete latent variable and the corrosponding sub-policies in each region. In this study, we empirically investigated the effect of policy structure in oflline RL. The experimental results reveal that the use of a mixture of deterministic policies can mitigate the issue of critic error accumulation offline RL. In addition, the results indicate that the use mixture of deterministic policies significantly improves the performance of an offline RL algorithm. We believe that our study contributes advancing techniques leverage policy structure in offline RI..
14

---

Published in Transactions on Machine Learning Research (09/2023)
References
Riad Akrour, Davide Tateo, and Jan Peters. Contimuous action reinforcement learning from mixture of interpretable experts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021. doi: 10.1109/TPAMM22221.1000
Gaon An, Seungvong Moon, Jang- Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. In Advances in Newral Information Processing Systems (NeurIPS), 2021.
Pierre-Lue Bacon, Jean Harb, and Doina Precup. Option-critic architecture. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAT), 2017.
David Barber and Felix Agakov. The im algorithm A variational approach to information maximization. In Advances in Neural Information Processing Systems (NewrIPS), 2003.
David Brandfonbrener, William F. Whitney, Rajesh Ranganath, and Joan Bruna. Offline RL without off- policy evaluation. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Newral Information Processing Systems (NeurIPS), 2021a.
Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Jianhao Wang, Yuan Gao, Wenzhe Li, Bin Liang, Chelsea Finn, and Chongjie Zhang. LAPO: Latent-variable advantage-weightee policy optimization for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS). 2022.
Xiong- Hui Chen, Yang Yu, Qingyang Li, Fan-Ming Luo, Zhiwei Qin, Wenjie Shang, and Jieping Ye. Offine model-based adaptable policy learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021b.
Tom Minka Chris J. Maddison, Daniel Tarlow. A sampling, In Advances in Neural Information Processing Systems (NeurIPS), 2014.
Chris Cremer, Xuechen Li, and David Duvenaud. Inference suboptimality in variational autoencoders. In Proceedings of the International Conference on Machine Learning (ICML), 2018.
Christian Daniel, Gerhard Neumann, Oliver Kroemer, and Jan Peters. Hicrarchical relative entropy policy search. Journal of Machine Learning Reseurch, 17(93):1 50, 2016.
P. Dayan and Hinton. Using expectation-maaxiiiztioo for reinforcement learning. Newral Computation, 9:271 278, 1997.
Emilien Dupont. Learning disentangled joint continuous and discrete representations. In Advances in Newral Information Processing Systems (NeurIPS), 2018
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data- driven reinforcement learning. arXie, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement lcarning. Aduances in Neural Information Processing Systems (NewrIPS), 2021.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the International Conference on Machine Learning (ICML), pp. 1587-1596. 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the International Conference Machine Learning (ICMT.), pp. 2019.
15

---

Published in Transactions on Machine Learning Research (09/2023)
Scott Fujimoto, David Meger, Doina Precup, Ofrr Nachum, and Shixiang Shane Gu. Why should I trust you, Bellman? the Bellman error is a poor replacement for value error. In Proceedings of the International Conference on Machine Learning (ICML), 2022.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. EMaQ: Expected-max Q- learning operator for simple yet effective offlinc and online rl. In Proceedings of the International Conference on Machine Learning (ICML), volume 139, pp. 3682 3691. PMLR, 18 24 Jul 2021.
Wonjoon Goo and Scott Nickum. You only evaluate once: a simple bascline algorithm for offiine rl. In Proceedings of the Conference on Robot Learning (CoRL), 2021.
Caglar Gulcehre, Sergio Gomez, Jakub Sygnowski, Ziyu Wang, Tom Le Paine, Konrad Zolna, Razvan Pas- canu, Yutian Chen, and Matt Hoffman. Addressing extrapolation error in deepoffline reinforcement learn- ing. In Offline Reinforcement Learning Workshop at Neural Information Processing Systems (NeurIPS), 2020.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with Gumbel-sofftmxx In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
Diederik P Kingma and Max Welling. Anto-encoding variational Bayes. In Proceedings of the International Conference on Learning Representations (ICLR), 2014.
J. Kober and J. Peters. Policy search for motor primitives in robotics. Machine Learning, 84:171-203, 2011.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofr Nachum. Offline reinforcement learning with Fisher divergence critic regularization. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Πya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce- ment learning. In Advances in Neural Information Processing Systems (NewrIPS), 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement Learning, pp. 45 73, 2012.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arxiv, 2020.
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
Volodymyr Mnih, Adria Puigdomenoch Badia, Mchdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Pro- ceedings of the International Conference Machine Learning (ICML), 2016.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. AWAC: Accelerating online reinforcement learning with offline datasots. arXin, Xii:20006.09359, 2020.
Gerhard Neumann and Jan Peters. Fitted q-iteration by advantage weighted regression. In Advonces in Newral Information Processing Systems (NeurIPS), 2008.
Takayuki Osa, Voot Tangkaratt, and Masashi Sugiyama. Hierarchical reinforcement learning via advantage- weighted information maximization. In Proceedings of the International Conference Learning Repre- sentations (ICLR), 2019.
Jan Peters and Stefan Schaal. Reinforeement learning by reward- weighted regression for operational space control. In Proceedings of the International Conference Machine Learning (TCMT), 2007.
16

---

Published in Transactions on Machine Learning Research (09/2023)
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information. Processing Systems (NeurIPS), 2019.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Represenlations (ICLR), 2016.
Matthew Smith, Herke Hoof, and Joelle Pineau. An inference-based policy gradient method for learning options. In Proceedings of the International Conference on Machine Learning (ICML), 2018.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep condi- tional generative models. In Advances in Newral Information Processing Systems (NeurIPS). 2015.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. edition, 2018.
The MTT Press, Second
E. Todorov, T. Erez, and Y. Tassa. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RS International Conference on Intelligent Robols and Systems, pp. 5026 5026-5033, 2012.
Aaron den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. Tn Advances in Newral Information Processing Systems, 2017.
Qing Wang, Jiechao Xiong, Lei Han, peng sun, Han Liu, and Tong Zhang. Exponentially weighted imitation learning for batched historical data. In Aduances in Neural Informalion Processing Systems (NeurIPS), 2018.
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou Diffusion policics as an expressive policy class for oflline reinforcement learning, In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
Markus Wulfmeier, Abbas Abdolmaleki, Roland Hafner, Jost Tobias Springenberg, Michael Neunert, Tim Hertweck, Thomas Lampe, Noah Siegel, Nicolas Heess, and Martin Riedmiller. Compositional transfer in hierarchical reinforcement learning. In Proceedings of Robotics: Science and Systems (R:SS), 2020.
Markus Wulfmeier, Dushyant Rao, Roland Hafner, Thomas Lampe, Abbas Abdolmaleki, Tim Hertweck, Michael Neunert, Dhruva Tirunala, Noah Siegel, Nicolas Heess, and Martin Riedmiller. Data-efficient hindsight off- off-policy option learning. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Shangtong Zhang and Shimon Whiteson. DAC: The double actor-critic architecture for learning options. In Advances in Newral Information Processing Systems (NeurIPS), 2019.
Wenxuan Zhou, Sujay Bajracharya, and David Held. PLAS: Latent action space for offline reinforcement learning. In Proceedings of the Conference on Robot Learning (CoRL), volume 155, pp. 1719 1735, 2020.
A Applicability of the gating policy
In the proposed algorithm, we employ a gating policy that determines the value of the latent variable as Tollows:
= arg max Qw (8, (s x')),
(20)
where (s(s, z²) represents the deterministic sub-policy, and Qw, a) is the approximated Q-function. While
this gating policy appears specific to the case where Q" a) maximized, applicable to other objective
function such A (s, a), exp(Q* a)); and exp( A (s, a)). The advantage function is defined A (s, a)
17

---

Published in Transactions on Machine Learning Research (09/2023)
("(ssa) - (s). Because the state value function VT(s) is independent of the action, can obtain the
following equation:
arg max (s) arg max (Q" (s, (*))
(21)
arg max A (s, a).
(22)
a
Thus, we can rewrite the gating policy as
= arg max Qw (3, po z²))
(23)
arg max A no (8, 2')).
(24)
Similarly, exponential function exp(-) a monotonically increasing function. Thus, the extrema of Q" @s(ss a)
is also the extrema of exp(Q" a)). Consequently, we can also rewrite the gating policy
z arg max Qu ( z'))
(25)
arg max exp (Q (s, =o(s)22)))
(26)
arg max A u z²))
(27)
arg max exp (Aw (s, po 2')))
(28)
Because we used this gating policy, it is deterministic in our implementation.
B Derivation of the variational lower bound
We employed the variational lower bound in Equation to derive the objective function for the proposed method. Here, provide a detailed derivation, which was omitted in the main manuscript. We denote the true distribution induced by the policy (a(a)s) as p(-), and the distribution that approximates the true distribution denoted as q(-). The KL divergence between g(x:) and p(æ) is defined as
DRr. ([(x))))(())) = / ((x)) log g(x) dz. p(x))
(29)
Based on the above notation, the log- likeliieehood log Te (alss))) can be written as follows:
log (a l8:)
(s(=|s;;a;) log (ag))(())))
(30)
(0(=)e; a=) (:gg logp(z/a:) - log p(z)a; a:)
(31)
/ (x(s)ss log g(=))1, as)
dz
p(= 81.
I q(=|,, a.) (())_,, dz p(2)a,)
/ o(*]a, a.) log 2)dz
(32)
DKr. (u(=)s) a:) (p(=|s,. a.)) - (at./a(=/ss...... (es..t((ass,)))llaee)e))
+ (a) (a.g,)
(33)
In the first line, we consider marginalization over z. As log TT Taass) 9) is independent of the latent variable z, the equality in the first line holds. Because DKL (g(2ss, ())lp(())s)))))))))))))))) a)) > 0, can obtain variant of the variational lower bound of the conditional log-likelihood:
log no (a.(s)) 2 -DKL. (gs a: a.)l(((sss)) Eang(sle, [og (a 81, =))
(34)
18

---

Published in Transactions on Machine Learning Research (09/2023)
Table 8: Effect of the dimensionality of the discrete latent variable. WK-walker2d.
infoDMPO
infoDMPO
infoDMPO
infoDMPO
|2|
[2] 8
[2| 16
pen-human-vo
76.7 18.9
94.8 + 16.5
75.0 17.5
86.7 + 12.4
WK-cxpert-v2
99.7117.9
112.110.4
108.816.8
106.4110.2
WK-med -expertt--
89.1+:25.7
110.110.7
96.0117.0
109.911..6
-replaa--
81.6:4.55
86.713.2
85.413.77
86.313.11
WK-
81.81:2.5
85.0:1..8
69.9-28.3
84.311.0
C Proof of contraction of the latent-max-Q operator
We consider operator Te, which given by
T.Q(s, a) = Es + max Q(s', (u(ss
+
(35)
To prove the contraction of Te, we use the infinity norm given by
11Q Q21la
max sES.aEA
(Qi(s. a) Q2(s.a)).
(36)
where Qi and Q2 are different estimates of the Q-function. We consider the infinity norm of the difference between the two estimates, Q1 and Q2, after applying operator Tz:
||T=Q1
T2Q2lloo
(37)
((8,a) + +-y (s's (s'.
Eg (((s,a) a) + y Q2lsi ((()
(38)
/
max Qi (e', m/s a(d''22)) 2/)3
- /B
max Q2(e' (((s',z)))
(39)
Ey
max Q1 p(s' ((d',='))
Ey
max Q2(5' m1(' 22'))
(40)
=
Ey
max Qi(s', (aaiss''''ss'_ ((d', max (a(s'.
(41)
5 Bar 11Q: Q21lae|
(42)
$ |Q Q21lxo
(43)
The above relationship shows the contraction of operator Tz.
D Effect of dimensionality of the discrete latent variable
In our evaluation, we first examined the effect of the dimensionality of the discrete latent variable. The results are presented in Table g As shown, infoDMPO with |Z| demonstrated the best performance, while the performance with |2| 16 and |Z| 32 comparable. These results show that the policy performance is not very sensitive to the dimensionality of the latent variable. However, the performance with |Z| 4 is relatively weak, thereby indicating that the policy may not be sufficiently expressive when the dimensionality of the latent variable is significantly small. Because / = 8 consistently provided satisfactory performance, |Z| = 8 was used in the subsequent evaluations.
E
Comparison with additional baselines
We provide comparison with additional baselines for the mujoco-v2 tasks in D4RL in Table g We present the results of MAPLE, which is recent model-basee offline algorithm that uses latent representations (Chen et al. ) In addition, provide the results of decision transformer (Chen et al. 2021a), represen- tative transformer- based method. Although these methods are well-known and ttatc-of-the-art, we focused on model-free and non-transformerr-bee adapted the results reported in the original paper. DMPO and infoDMPO provide a consistently better or comparable performance to these bascline methods, although our implementation of DMPO and infoDMPO does not employ techniques such as ensemble of critics. This result indicates significant effect of the policy structure in offline RI..
methods in the main manuscript. For each baseline method, we
19

---

Published in Transactions on Machine Learning Research (09/2023)
Table 9: Results on mujoco tasks using D4RL-v2 datasets. Average normalized scores over the last test episodes and five seeds are shown. HC HalfCheetah, HP Hopper, WK Walker2d. The gray text indicates the performance lower than that of DMPO/infODMPO. The bold text indicates the best performance.
MAPLE
Decision
DMPO
infoDMPO
Transformer
(paper)
(paper)
(ours)
(ours)
Replay Expert
HC
+ 6.5
86.8
91.11 3.4
91.44 2.5
Med.- Med.
HP
42.5
107.6 +
78.44
94.3
WK
+
108.1 = 0.2
109.9 0.4
110.1 0.7
HC
59.0
1 0.6
36.6 1 0.8
45.2.1 0.8
46.7.1 0.6
HP
87.5 1 10.8
82.7 1 7.0
89:2.1 8.1
98.5.1 2.0
WK
76.7 + 3.8
66.6 3.0
82.1 3.8
86.7t 3.2
HC
50.4 + 1.9
42.6 + 0.1
47.54 0.4
48.61 0.4
Med.
НP
21.1
+ 1.2
67.6 1.0
71.24 6.5
86.41 7.6
WK
56.3 + 10.6
74.0 + 1.4
79.4 4.7
85.0% 0.8
F Hyperparameters and implementation details
Computational resource and license The experiments were run with Amazon Web Service and work- stations with NVIDIA RTX 3090 GPUs and Intel Core 19-10080XE CPUs at 3.0 GHz. We used the physics simulator, MuJoCo Todorov al. 2012 under an institutional license, and later we switched to the Apache license.
Software
The software versions used in the experiments are listed below:
Python 3.8
Pytorch 1.10.0
Gym 0.21.0
MuJoCo 2.1.0
mujoco-py 2.1.2.14
We used the author-pre author-provided implementations for TD3+BC and CQI DMPO, AWAC, mixAWAC, and IQL were implemented based on the the author-provided implementation of TD3. For IQL, we used the hyperparameters provided in Kostrikk al. (2022). To minimize the difference between DMPO and AWAC, used delayed update of the policy in both DMPO and AWAC. For simplicity, we did not use regular- ization technique for the actor such the dropout layer used in (Kostrikov et al. 2022). although the use of such techniques should further improve the performance. In our implementation of DMPO, the value of a part of the input to the actor network. Thus, different behaviors corresponding to different values of z are represented by the same actor network. The network architecture illustrated Figure
Computation of the advantage function In DMPO, policy is deterministic because both the gating policy <22(s) and sub-policy 7 (a)s, z) are deterministie. Thus, the state-value function is given by
(T(s) max Q" ((),
(44)
Therefore, the advantage function given by
( a) = (s, (=.. - = Q (s, max (T (s, 2)).
(45)
In the policy update, we use the target actor in the second term in Equation 45 Thus, in our implementation, the advantage function is approximated as
A(s, a; w, 0') (s, a: w) max Q(s, uo (o 2) w)
(46)
https://github. com/young geng/CU CQL
20

---

Published in Transactions on Machine Learning Research (09/2023)
(w a)
8
a
L(O.
a
2
qo (z|8.a) a)
p
(a)
Computation for maximizing Lxa in Equation
11
8
8
p(z)
μe (a)s,))
Iup ( 8,
(b) Computation for maximizing Σ i= Bamp(a) log (e (o =)).
Figure Connection between (0(z|1, a); (o( and go a) during training.
Target smoothing in DMPO Tn DMPO, policy is given by mixture of deterministic sub-policies, where a sub-policy is selected in deterministic manner, similar to that in Equation 3 Thus, the mixture policy in this framework is deterministic. As reported in Fujimoto & Gu (2021), the use of deterministic policy may lead to overfitling of the critic to narrow peaks. Because our policy is deterministic, we also employed a technique called target policy smoothing used TD3. Thus, the target value in Equation 17 is modified as follows:
= ri + max min Quy (8', (s', (6), + ectip)s
(47)
where Eelip is given by
Eclip min min(max(e,-c)) c), where E~N(0,0),
(48)
and constant e defines the range of the noise.
Techniques for Antmaze tasks In LAPO Chen et al. (2022). several techniques to stabilize the training of the value functions are used. Suppose the state-value function is approximated with Vu. (s) parameterized with a vector Wo and the Q-function is approximated with two models, which are represented by Qu (8, a) for 1,2. The stato-value function is updated by minimizing
(.w = Σ
lb.. Vw, (s))1
(49)
(eD
where the target value ū is the clipped target value computed as
-: max (min (yis Umax) Umin)
(50)
and yi is computed
3: min Qw, (8i, a:) + (1 -c) max Qw, (s;, a.)
(51)
j=1,2
j=1.2
21

---

Published in Transactions on Machine Learning Research (09/2023)
and is constant, and we used 0.7 as in Umin and Umax are computed as
Chen al. 2022
The minimum and maximum target value
Umin
min F Y rED
(52)
Umax
max Ví- MeD
(53)
The Q-function is updated by minimizing the following objective function:
(((w)
Σ
||: + YVo ( (5)) Qm a;)ll
(54)
For the antmaze tasks, we also used same techniques in DMPO.
Implementation of miXAWAC The difference between mixAWAC and AWAC is policy representation. For mixAWAC, we used Gaussian mixture policy. The discrete latent variable is sampled from a categorical distribution, and the corresponding Gaussian component policy is used to sample the action. As DMPO, the latent variable is represented as a one-hot vector, and the neural network that represents the Gaussian components takes the state and the one-hot vector as its input. The key part of the implementation is how to sample from categorical distribution in differentiable manner. We used the Gumbel-max trick for this purpose (Chris J. Maddison 2014 Jang et al. 2017 Maddison et al. 2017). The Gumbel-max trick is often used to learn discrete latent variable in VAE Kingma & Welling 2014).
In our implementation, the activation function of the last layer of the gating policy is the softmax function. The discrete latent variable sampled using on the Gumbel-max trick based on the output of the gating policy.
As in our implementation of AWAC, mixAWAC, and DMPO, the double- clipped Q-learning employed in LP-AWAC. In additioned to the Q-function, the state value function V.) is trained by minimizing the mean squared error:
Implementation of LP-AWAC
CLP-AWAC(W)- = Σ
Vu((i) - min Qw, (8,, ai) 1=1,2
(55)
(e:.0:)E
In LP-AWAC, the conditional VAE is trained using advantage weighting. Demoting the approximated pos- terior and likelihood by go(z|s, a) and pp(als,z), respectively, the encoder and decoder trained by maximizing the following objective function:
Lovae(D, 4b) =
E
W(s,, (-Dki(go(2)s,, (.)(s((2)).) + Ezog(z)enaa) (ails,, (56)
(D
where ((;;, a;) is the weight for advantage weighting. In our experiments, we used the normalized advantage weighting in Equation 19 Then, the deterministic latent actor (e is trained to output the latent variable z by maximizing the expected Q-value:
Llatent-nctor (0)
Σ Quy (8i, (o((:))) (e:
(s.,
(57)
where (s, a) is the decoder. The objective function for learning the continuous latent variable in LP- AWAC is very similar to that of DMPO in Equation [] for learning the discrete latent variable. When considering the deterministic latent actor (s(ss) in LW-AWAC as the gating policy that approximately solves arg max, ((s, 2), LP-AWAC can be considered as the variant of DMPO using the continuous latent variable. Thus, the difference between DMPO and AW-A indicates the difference of the discrete and continuous latent variable in framework.
22

---

Published in Transactions on Machine Learning Research (09/2023)
Table 10: Hyperparameters of DMPO infoDMPO.
Hyperparameter
Value
Optimizer
Adam
Critic learning rate
3e-4
(mujoco-v2, adroit)
/ 2e-4 (Antmaze)
Hyperparametens
Actor learning rale
3e-4
(mujoco-v2, adroit)
2e-4 (Antmaze)
Posterior learning rale
3e-4 (mujoco-v2, adroit)
2e-4 (Antmaze)
Mini-batch size
256
Discount factor
0.99
Target update rate
5e-3
Policy noise
0.2
Policy noise clipping
(-0.5, 0.5)
Policy update frequency
2
Critic hidden dim
256
Critic hidden layers
2 (mujoco-v2, adroit) / (Antmaze)
Architecture
Critic activation function
ReLU
Actor hidden dim
256
Actor hidden layers
2 (mujoco-v2, adroit) / (Antmaze)
Actor activation function
ReLU
Posterior hidden dim Posterior hidden layers
256
(mujoco-v2, adroit) / (Antmaze)
Pusterior aclivation function
ReLU
DMPO
Score scaling a
5.0 (human, Antmaze)
10.0 (mujoco-v2)
learning rate of the posterior for infomax
3e-6 (Adroit)
5e-7 (mujoco-v2)
infoDMPO
5e-7 (Antmaze)
Score scaling a
(Antmaze, -experr)
10.0 (others)
Number of updates In the pen-human-v0, hammer-humann-v0, door-human-v0, and relocate-human- vO tasks, the number of samples contained in the dataset is significantly smaller than that for the other datasets. While the datasets for mujoco tasks contained approximately million samples, the numbers of samples in the adroit-human tasks were as follows: pen-human-v0: 4,950 samples,
11,264 samples, door-luraan-00: 6,703 samples, and relocale-human-v0:: 9,906 samples. Thus, the pen- human-v0,
door-human- and relocate-human-v0) tasks, we updated the policy 10,000 times, whereas for the other tasks, updated the policy 1 million times. The aforementioned number of policy updates was applied to all methods.
Hyperparameters Tables 10 14 provide the hyperparameters used in the experiments. Regarding À in infoDMPO, the first and second terms in Equation (13) are maximized separately. Thus, we implicitly set the value of À by setting the different learning rates for the first and second terms in Equation (13). The learning rate for the first term in Equation (13) was fixed to 3e-4. We tested a set the learning rate {le-7, 5, 1c-6, 3--6} for the second term in Equation (13), and reported the best results the paper.
23

---

Published in Transactions on Machine Learning Research (09/2023)
Table 11: Hyperparameters of AWAC.
Hyperparameter Optimizer
Value
Adam
Hyperparaneters
Crilic learning rate
3e-4
Actor learming rale
3e-4
Mini-batch size
1024
Discount factor
0.99
Target update rate
5e-3
Policy update frequency
2
Score scaling
10.0
Critic hidden dim
256
Architecture
Critic hidden layers
Critic activation function
ReLU
Actor hidden dim
256
Actor hidden layers
Actor activation function
ReLU
Table 12: Hypeepaaraeeters of TD3+BC. The default hyperparameters in the TD3-BC GitHlu used.
Hyperparameter
Value
Optimizer
Adam
Hyperparanetos
Critic learning rate
3e-4
Actor learning rate
3e-4
Mini-batch size
256
Discount factor
0.99
Target update rate
5e-3
Policy noise
0.2
Policy noise clipping
(-0.5, 0.5)
Policy update frequency
2
2.5
Critic hidden dim
256
Architecture
Critic hidden layers
2
Critic activation function
ReLU
Actor hidden dim
256
Actor hidden layers
2
Actor activation function
ReLU
24

---

Published in Transactions on Machine Learning Research (09/2023)
Table 13: Hyperparameters of CQL. The default hyperparameters in the CQL GitHub are used.
Hyperparameter
Value
Optimizer
Adam
Critic learning rate
3e-4
Actor learning rate
3e-5
Hyperparanetess
Mini-batch size
256
Discount factor
0.99
Target update rate
5e-3
Target entropy
-1.Action Dim
Entropy in Q target Lagrange
False
False
Pre-training sleps
40e3
Num sampled actions (during eval)
10
Num sampled actions (logsumexp)
10
Critic hidden dim
256
Architecture
Critic hidden layers
3
Critic activation function
ReLU
Actor hidden dim
256
Actor hidden layers
3
Actor activation function
ReLU
Table 14: Hyperparameters of IQL. The default hyperparameters in the IQL in the paper Kostrikov et al.
2022 are used.
Hyperparameter
Value
Optimizer
Adam
Critic learning rate
3e-1
IQL lyperparameters
Actor learning rate
3e-4
Mini-batch size
256
Discount factor
0.99
Target update rate Expectile
5e-3
0.7 (mujoco-v2)
0.7 (adroit)
0.9 (antmaze)
Advantage scale
3.0 (mujoco-v2)
0.5 (adroit)
10.0 (antmaze)
Actor learning rate scheduling
cosine
Critic hidden dim
256
Architecture
Critic hidden layers
Critic activation function
ReLU
Actor hidden dim
256
Actor hidden layers
Actor activation function
ReLU
25


Reviews:
[{"review_id": "MVg4rpg8ht", "reviewer": "Reviewer_DypM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- This paper is overall well-written and easy to follow. \n\n- It is true that Gaussian policy would lead to extrapolation between modes, hence using a mixture of policies could be a solution.\n\n- Using the hierarchical design to avoid evaluation of OOD actions is novel to the best of my knowledge.\n\n- Empirical results appeared to be strong although some improvements could still be made.\n\n**Weaknesses**\n\n- The motivating hypothesis might not be necessarily correct. In the first paragraph of section 1, the authors stated: `` Our hypothesis is that the evaluation of the out-of-distribution actions can be avoided by dividing the state-action space.\" The evaluation of OOD actions is not necessarily solved by separating offline action modes. Taking a dummy MDP example, suppose $\\mathcal{S} = \\{s\\}$ and $\\mathcal{A} = \\{a_1, a_2\\}$ and both actions leads to a self-loop to $s$, and rewards corresponding to the actions are $r_1=-1, r_2=-2$ respectively. Suppose the $Q$-function is initialized to be $Q(s, a_1) = Q(s, a_2) = 0$. Considering an offline dataset contains only $a_1$, training an vanilla RL agent (without further regularization)  with this dataset will still lead to evaluations of $a_2$ even though there is only one mode in the dataset, because of the over-estimation of $a_2$. \n\n- Although I found the motivation hypothesis might not be fully correct, using latent policy to divide the state-action space should be able to avoid evaluation of OOD actions, because the latent policy avoids generating OOD actions, as observed in LAPO [1]. However, the OOD action issue might be mainly addressed by using latent policy rather than dividing the state-action space. The claims could be made more precises as I believe dividing state-action space without using latent policy won't be sufficient.\n\n- In particular, LAPO has already provided a promising solution to avoid evaluations of OOD actions via latent policy. According to Figure 2 in [1], using latent policy alone could learn multimodal behaviors and avoid extrapolation between modes. \nAlthough the authors provided comparisons between using discrete latent variables and continuous ones in Section 7, it is somewhat difficult to understand whether the difference is caused by the choice of latent variable or practical implementation.\nIt would be more compelling if the authors could also give some toy examples to demonstrate the advantage of discrete latent variables.\n\n\n- The comparison versus LAPO- (LAPO minus) is not quite convincing. I understand that techniques such as action normalization and target clipping could significantly change the training dynamics and final performance. However, removing components of LAPO may not result in a fair comparison as DMPO also introduces techniques such as advantage normalization in Eq (18). \n\n\n**Questions**\n\n- ``While maximizing the objective $ L_\\mathrm{ML} $ in Equation 11, both the actor $ µ_\\theta (s, z)$ and auxiliary distribution $g_\\psi (z|s, a)$ are updated.'' $ L_\\mathrm{ML} $ appears to be an function of $\\theta$ and $\\phi$, I wonder how is $g_\\psi$ updated when maximizing $L_\\mathrm{ML} $. \n\n[1] Chen, Xi, et al. \"Latent-variable advantage-weighted policy optimization for offline rl.\" arXiv preprint arXiv:2203.08949 (2022).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In the offline reinforcement learning (RL) setting, it has been observed that querying the value of out-of-distribution (OOD) actions may cause an accumulation of bootstrapping errors. To address this issue, existing works may penalize deviation from behavior policy or underestimate OOD actions. \n\nThis work addresses the aforementioned issue from the perspective of the parameterization of policies. Prior works often use (squashed) Gaussian policies, which admit a single-mode behavior, while the data are usually multi-modal. Using a Gaussian policy to model multi-modal data might lead to interpolation between modes, causing evaluations of OOD actions, hence the accumulation of bootstrapping error.\n\nFrom this aspect, the authors propose to divide the state-action space by learning discrete latent variables that activate (deterministic) sub-policies, to avoid interpolation between modes. And using this hierarchical policy structure to address the evaluation of OOD actions is novel to the best of my knowledge.\n\nThe authors then provide empirical evaluation in D4RL locomotion and AntMaze environments along with a toy multi-modal environment, comparing DMPO to several offline RL algorithms.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- If the authors agree with the weaknesses point 1 and 2, the authors may need carefully change their claims to make them more precise.\n\n- It is preferable to see further analysis of why discrete latent variables are better choices than continuous ones, as I found the current empirical comparison versus LAPO- could be made more convincing.\n\nMinor points:\n\n- ``In addition, DMPO outperformed LAPO-, which is the state-of-the-art method in offline RL.\" It is arguable that LAPO- can be still considered SOTA when several components of LAPO have been removed. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wvJvN61fOJ", "reviewer": "Reviewer_LayE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n- The motivation of this work is clear. I appreciate the expression in Figure 1.\n- The organization of method presentation is good, including both the derivation, theory and practical implementation.\n- The experiments are multifaceted.\n\n&nbsp;\n\n### Weaknesses:\n\n- Although the motivation of this paper is clear and makes sense to me, it is not novel since a prior work [1] studies the same problem. Therefore, the work[1] is closely related to this paper, while it is not included in this paper for discussion or empirical comparison.\n- My major concern lies at the introduction of the policy optimization objective in Section 4.1.\n    - I feel a gap between the derivation based on a stochastic policy and the deterministic policy used in DMPO. I found this is implicitly done by making use of MSE implementation of the log-likelihood maximization.\n    - In addition, I cannot understand the rationale of choosing Eq.9 as the prior distribution. I think more explanation may help.\n\n&nbsp;\n\n### Minors:\n- In Eq.12, the sampling distributions for the expectation notations are inconsistent.\n\n---\n### Reference:\n\n[1] Wang et al., Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning. ICLR 2023", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new offline RL algorithm called, Deterministic Mixture Policy Optimization (DMPO), to overcome the issue of most existing methods that fit a unimodal distribution (typically Gaussian distribution) when the offline data distribution is potentially multimodal.\n\nThe key idea of DMPO is to leverage a mixture policy built on a gating policy and multiple sub-policies. DMPO learns discrete latent variables to separate the state-action space based on the offline dataset via a typical conditional VAE manner. So that the gating policy selects a discrete variable $z$ under a specific state and the multiple sub-policies are represented by a $z$-conditioned deterministic policy.\n\nThe policy optimization objective is derived from a variational lower bound of typical policy optimization objective. In addition, a Mutual-Information (MI) objective is introduced to encourage the non-overlapness of multiple sub-policies. The theorems on contraction and convergence are provided under classic conditions.\n\nDMPO is evaluated in a toy example to show the effectiveness in dealing with synthesized multimodal data, while also compared against common baselines in D4RL, Antmaze and Adroit. Other analysis and visualization are provided.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Can the authors provide a pros-cons comparison between DMPO and Diffusion-QL [1], from the methodology perspective? I suggest the authors to include Diffusion-QL in the empirical comparison as well.\n2. Can the authors explain the gap existing in deriving the policy optimization objective with a stochastic policy while using a deterministic policy in DMPO?\n3. Can the authors explain the rationale of choosing Eq.9 as the prior?\n4. How $z$ are represented in practice since it should be taken as input in $u(s,z)$, one-hot representation or some learned representation by techniques like VQ-VAE?\n5. For infoDMPO, how is the hyperparameter $\\lambda$ selected?\n6. I recommend the authors to include the concrete implementation of mixAWAC in the appendix.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FvE9geLDyL", "reviewer": "Reviewer_PB5R", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* The paper is well written.\n* It investigates an interesting direction for improving offline RL.\n* The results are promising.\n\n**Weaknesses**\n* Whether the assumption that serves as motivation for the approach is really confirmed remains unclear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper deals with offline RL. \nIt is hypothesized that the use of a multitude of deterministic policies can help to reduce the extrapolation error of the value function.\nAn algorithm in two variants is presented that learns a gating agent and a multitude of deterministic policies.\nBoth variants are tested on a number of deterministic benchmarks and show very good, partly superior results compared to established methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "From my point of view, no major changes are necessary.\n\nA few small remarks:\n\n* Regarding \"a one-step offline RL approach\" it should be considered whether [1] should not be cited as well, in which as far as I know this idea was presented for the first time.\\\n[1] C. Gulcehre et al., Addressing Extrapolation Error in Deep Offline Reinforcement Learning, 2021\n* equation 1 -> Equation 1\n* 3.0GHZ -> 3.0 GHz\n\n\nIn the References\n* variationalapproach ->  variational approach\n* Offline model-base adaptable policy learning. -> Offline model-based adaptable policy learning.\n* In some places, there are unintentional, misspelled lowercase letters: Why should i trust you, bellman? the bellman error is a poor replacement for value error, gumbel, fisher\n", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors identified an issue in the major existing offline RL methods, which is the multimodality in policy. To overcome the problem, the authors introduced the deterministic mixture model for policy parametrization, which will be learned by a variational lower bound of the policy value, with a mutual-information regularization. 

The authors justified the performance of the proposed method on the existing benchmarks, demonstrating their benefits. After the rebuttal discussion period, the authors added more comparison on LAPO and diffusion-QL, which are the major competitors and should be added originally. 

In sum, all the reviewers acknowledge that although the idea to use more flexible policy parametrization is already proposed for offline RL, this paper is still solid. Considering the novelty is not the major evaluation criterion, the paper should be accepted. 

There are several minors need to be addressed:

1, The gradient approximation for discrete variable in the loss function is not specified. This part is very important for reproducibility. Please add the details of the implementations about gradient calculation. 

2, The loss function is confusing in Eq 6. I guess it should be 
$\int d(s)\beta(s, a)\log (\frac{\pi(a|s)}{\beta(a|s)}) f(s, a)dsda$ instead of $\int d(s)\beta(s, a)\log (\frac{\pi(a|s)}{\beta(a|s)} f(s, a) )dsda$? Please make this point clear.
Decision: Accept (poster)