On the Importance of the Policy Structure in Offline Reinforcement Learning
Authors: Takayuki Osa
Akinobu Hayashi
Pranav Deo
Naoki Morihira
Takahide Yoshiike
Venue: Submitted to ICLR 2023 (2023)
Keywords: offline reinforcement learning
discrete latent representations

Abstract: Offline reinforcement learning (RL) has attracted a great deal of attention recently as an approach to utilizing past experience to learn a policy. Recent studies have reported the challenges of offline RL, such as estimating the values of actions that are out of the data distribution. To mitigate the issues of offline RL, we propose an algorithm that leverages a mixture of deterministic policies. With our framework, the state-action space is divided by learning discrete latent variables, and sub-policies corresponding to each region are trained. The proposed algorithm, which we call Value-Weighted Variational Auto-Encoder (V2AE), is derived by considering the variational lower bound of the offline RL objective function. The aim of this work is to shed lights on the importance on the policy structure in offline RL. We show empirically that the use of the proposed mixture policy can reduce the accumulation of the approximation error in offline RL, which was reported in previous studies. Experimental results also indicate that introducing the policy structure improves the performance on tasks with D4RL benchmarking datasets.

Under review as a conference paper at ICLR 2023
ON THE IMPORTANCE OF THE POLICY STRUCTURE IN OFFLINE REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Offline reinforcement learning (RL) has attracted great deal of attention recently as an approach to utilizing past experience to learn policy. Recent studies have reported the challenges of offline RL, such as estimating the values of actions that are out of the data distribution. To mitigate the issues of offline RL, we propose an algorithm that leverages a mixture of deterministic policies. With our frame- work, the state-action space is divided by learning discrete latent variables, and sub-policies corresponding to each region are trained. The proposed algorithm, which we call Value- Weighted Variational Auto Encoder (V2AE), is derived by considering the variational lower bound of the offline RL objective function. The aim of this work is to shed light the importance of the policy structure in offline RL. We show empirically that the use of the proposed mixture policy can reduce the accumulation of the critic loss in offline RL, which was reported in previous studies. Experimental results also indicate that introducing the policy structure improves the performance on tasks with D4RL benchmarking datasets.
INTRODUCTION
Reinforcement learning (RL) (Sutton & Barto 2018) has had remarkable success in variety of applications. Many its successes have been achieved in online learning settings where the RL agent interacts with the environment during the learning process. However, such interactions
often time consuming and computationally expensive. The desirability of reducing the number of interactions in RL has motivated an active interest in offline RL (Levine al. (020), also known as batch RL (Lange et al. 2012). In offline RL, the goal is to learn the optimal policy from prepared dataset collected through an arbitrary and unknown process. Prior work on offline RL has focused on how to avoid estimating the Q-values of action that are out of the data distribution (Fujimoto et al.) 2019 Fujimoto& & Gu (02T). While previous studies often address this issue in terms of the regu- larization of critics (Kumar et al. 2020 An et al. 2021 Kostrikov et al. 2021 2022), we propose to mitigate the issue from the perspective of the policy structure. Our hypothesis is that evaluation of the out-of-distribution actions can be avoided by dividing the state-action space, which is potentially achieved by learning discrete latent variables of the state-action space. When the data distribution is multimodal, shown in Figure [ ()), fitting a policy modeled with unimodal distribution such as a Gaussian distribution may lead to interpolation between separate modes, which will result in the value estimation of actions that are out of the data distribution (Figure [[)).. To avoid this, we employ a mixture of deterministic policies (Figure [[c).. We divide the state-action space and learn sub-policies for each region. Ideally, this approach will enable us to avoid interpolating separate modes of the data distribution.
In this study, we propose to train a mixture policy by learning discrete latent representations, which can be interpreted as dividing the state-action space and learning sub-policies that correspond to each region. We derive the proposed algorithm by considering the variational lower bound of the offline RL objective function. We refer to the proposed algorithm as Value- Weighted Variational Auto-Encoder (V2AE). The main contribution of this study is an offline RL algorithm that trains mixture policy by learning discrete latent variables. We also propose regularization technique for a mixture policy based on the mutual information. We empirically show that the proposed reg- ularization technique improves the performance of the proposed algorithm. A previous study in (Brandfonbrener et al. 202I) reports the accumulation of the critic loss values during the training phase, which was considered the result of generating out-o---ist actions. We show empiri-

---

Under review as a conference paper at ICLR 2023
Sumples with high Q-values
Unimodal distribution finted to samples
Fil deterministic policy each region
Region there is datepoint
(a) Samples in state-action space.
(b) Result fitting uni- modal distribution.
(c) Proposed approach.
Figure 1: Schematic illustration of the proposed approach. (a) In offline RL, the distribution of sam- ples often multimodal; (b) Fitting unimodal distribution to such samples can lead to estimating the action out of the data distribution; (c) In the proposed approach, the latent discrete variable of the state-action space is learned, and a deterministic policy is learned for each region.
cally that the use of the proposed mixture policy can reduce the accumulation of the approximation error in offline RL. In experiments with benchmark tasks in D4RL (Fu et al.) (020), the proposed algorithms proved be competitive with the popular offline RL methods. While the experimen- tal result shows the promising performance we aim to shed light on the importance of the policy structure as inductive bias in offline RL, rather than claim the state-of-the thetttrrrttttt performance.
RELATED WORK
Recent studies have shown that regularization is the crucial component for offline RL. (Fujimoto et al.
2019 Kumar et al.] 2020 Levine et al., 2020: Kostrikov et al. (221]. For example, Kostrikov
itt al. 2021 I) proposed a regularization based on Fischer divergence, and Fujimoto & Gu (2021)
showed that simply adding behavior cloning term to the objective function in TD3 can achieve state-of-the-ar performance on D4RL benchmark tasks (Fu et al. 2020). Other research has inves- tigated the structure of the critic, proposing the use of ensemble of crities (An et al.) (2021) or offering a one-step offline RL approach (Brandfonbrener et al.) [022] Goo & Niekum| (2021) Previ- ous studies (Fujimoto et al. 2019 Fujimoto & Gul Gu[[20T]] have indicated that the source of the value approximation eтror is "extrapolation error" that occurs when the value of state-action pairs that are not contained in a given dataset estimated. Our hypothesis that such "extrapolation error" can be mitigated by dividing the state-action space, which is potentially achieved by learning discrete latent variables. We investigate the effect of incorporating the policy structure as inductive bias in offline RL, which has not been fully investigated.
Learning the discrete latent variable in the context of RL is closely related a mixture policy. where a policy is represented of the sub-policies activated for given state, and the module that determines which sub-policy to use is often called the gating policy (Daniel et al. 2016)) Because of the two-layered structure, a mixture policy is also called a hierarchical policy (Daniel et al. 2016). Although we do not con- sider temporal abstraction in this study, we note that well-known hierarchical RL framework with temporal abstraction is the option critic (Bacon et al. (2117). Since we consider policies without tem- poral abstraction, use the term "mixture policy," following the terminology in Wulffeiirr et al. (20TT) Previous studies have demonstrated the advantages of mixture policies in online RL. (Osa al. 2019 Zhang & Whiteson 2019 Wulfmeier al. 2020 202T Akrour et al. (02TT) In these existing methods, sub-policies are often trained to cover separate modes of the Q-function. which is similar our idea. While existing methods have leveraged the latent variable in offline RL (Zhou et al. 2020 Chen et al. 2021b) 2022), the latent variable is continuous in these methods. As indi- cated by studies on latent representations (Kingma & Welling 2014, Dupont] 2018 [Bronn et al., 2020), we think that the use of the discrete latent variable should be investigated in offline RL.
combination of a finite number of sub-policies In a mixture policy, one
PROBLEM FORMULATION
Reinforcement Learning Consider reinforcement learning problem under Markov decision process (MDP) defined by a tuple A. r, d). where S is the state space, A is the action space, (8 8t, ar) is the transition probability density, r(s, a) is the reward function, 7 is the discount.
2

---

Under review as a conference paper at ICLR 2023
factor, and d(so) is the probability density of the initial state. A policy (als)) : A R defined as the conditional probability density over actions given states. The goal of RL is to identify policy that maximizes the expected return E[ mr) where the return is the sum of discounted rewards over time given by R₂ EX " ax). The Q-function, Q" a), is defined as the expected return when starting from state 8 and taking action a, then following policy π under given MDP (Sutton& & Barto 2018).
In offline RI., it is assumed that the learning agent is given fixed dataset, D = {(*,, Os, r;) r consists of states, actions, and rewards collected by an unknown behavior policy. The goal of offline RL is to obtain policy that maximizes the expected return using D, without online interactions with an environment during the learning process.
Ji= that
Objective function We formulate the problem of offline RL as follows. Given a dataset D {(8;, 0; r=) obtained through the interactions between a behavior policy B(a|s) and the en- vironment, our goal to obtain a policy π that maximizes the expected return. In the process of training a policy in offline RL, the expected return is evaluated with respect to the states stored in the given dataset. Thus, the objective function given by
(TT BanD [f (s.
where f" is a function that quantifies the performance of policy т. In RI., there are several choices for f indicated in [Schulman al. (2016). TD3 employed the action-value function
as f 88, a) (s, a), and A2C employed the odvantage-function as (s (s a) (Mnih
al. 2016).
Other previous studies employ shaping with the exponential function as (s
exp
Q" [s. a (Peters & Schaal 2007 f"((sa) a) exp (A" a)) (Nuumann & Peters 2008
Wang et al. 2018). Without loss of generality, we assume that the objective function is given by
equation
We derive the proposed algorithm by considering the lower bound of the objective
function of offline RL in equation
Mixture policy In this study, we consider a mixture of policy given by
ग (als))
Σ zE2
Taate (z(8) suuh (a 8, 2),
where is discrete latent variable, guuee (z|s) )) is the gating policy that determines the value of the latent variable, and Tanb (a) 0, 2) is the sub-policy that determines the action for given 8 and z. We assume that sub-policy suu (a|s,, =) is deterministic; the sub-policy determines the action for given and in deterministic manner as = ne (s, =), where ( is parameterized with a vector 0. Additionally, we assume that the gating policy Tgate (2 determines the latent variable as
arg max Que (8, (s, z²)),
where Qw (s, a) is the estimated Q-function parameterized with a vector w. This gating policy is applicable to the objective functions such (*(ss exp (s, as))), fr a) A a), and (s.ss a) = cxp (A (s. a)). Please refer to Appendix Affor details.
4 TRAINING A MIXTURE POLICY BY MAXIMIZING THE VARIATIONAL LOWER BOUND
We consider a training procedure based on policy iteration (Sutton & Barto 2018), where the critic and the policy are iteratively improved. In this section, we describe the policy update procedure in the proposed method.
To derive the update rule for the policy parameter O, first consider the lower bound of the objec- tive function log J(x)) in equation [ We assume that f (8, a) in equation [ approximated with f a), which is parameterized with vector w. In manner similar to Dayan & Hinton 1197);; Kober & Peters 201 when f (s, a) for any and a, we can determine the lower bound of
3

---

Under review as a conference paper at ICLR 2023
log (x) using Jensen's inequality as follows:
log ((T) log / d (s)To(a(s) (s, addsda
(4)
= =log d {
Ta(a)s)
lit (s, a)dsda 2 / (s) log B(als)
To(a)s) fo (s, a)dsda (5)
B(als)
[(a,a)D log (a(ss) a)] B(,,))D log f(als)
(6)
where B(a]s) is the behavior policy, which is used for collecting the given dataset, and d (s) is the stationary distribution over the state induced by executing the behavior policy 3(a s). The second term in equation [ is independent of the policy parameter O. Thus, we maximize the
lower bound of (xx by maximizing Ei=1 log (a)aa)ss) (st, (8), a;). When employ (= a) exp (AT a)) and the policy is Gaussian, the resulting algorithm is equivalent to AWAC (Nair al., 2020). To employ mixture policy with a discrete latent variable, we further analyze the objective function in equation O As in Kingma & Welling (2014); et al. (2015), we can obtain a variant of the variational lower bound of the conditional log-likelihoon
log To (a, (s)) 2 - DKL (go(= a.) (1(((8))) + (Es-p(sle...)) [log (o( [a,]8;,2) 2)]
fevae(8; 0;; (),
where (=(2], a) is the approximate posterior distribution parameterized with a vector Φ. and (z]s) is the true posterior distribution. The derivation of equation 7 is provided in Appendix B
Although is often assumed in prior studies (Fujimoto et al. (019) that # is statistically independent
of s, i.e. p(z (s) = p(z). in our framework p(z[s) should represent the behavior of the gating policy
(z(2s)) Recognizing the challenge of representing exactly the gating policy (=(2s)) in equation we approximate it with the softmax distribution given by
p<2|5) = exp (Q u z)))
(8)
ΣxEz exp (Qw (s., (s 2)))
Since we employ double-clipped Q-learning as in Fujimoto et al. (018),, we compute Qu (s. (s z)) min Qwe ( z)) in our implementation. The second term in equa- tion is approximated as the mean squared error, as in the standard implementation of VAE. Based on equation and equation
7
we obtain the objective function for training mixture policy as NV
(0,
1d (8; Rovue a4; 0.
This objective can be regarded as the weighted maximum likelihood Kober & Peters 2011 for a mixture policy. Our objective function can be viewed as reconstructing the state-action action pairs with adaptive weights, as in |Ptters & Schaal (2007); Nair et al. (2020). Therefore, the policy samples actions within the support and does not evaluate out-of-distribution actions. The primary difference between the proposed method and the cxisting methods (Peters & Schaal 2007: Nair et al. 2020) is that the use of mixture of policies conditioned on discrete latent variables in ap- proach can be regarded as dividing the state-action space. For example, in AWAC (Nair et al. 2020), unimodal policy is used to reconstruct all of the "good" actions in the given dataset. However, in the context of offline RL, the given dataset may contain samples collected by diverse behaviors, and enforcing the policy cover all the modes in the dataset can degrade the resulting performance. In our approach, the policy To (a) 8, 2) is encouraged to mimic the state-action pairs which are assigned to the same values of z, without mimicking the actions which are assigned the different values of z.
In addition, we also propose regularization technique for a mixture policy based on the mutual in- formation (MI) between z and the state action pair (s, a), which we denote by / (2; 8, a). As shown by Barber & Agakov (2003), the variational lower bound of I(2; a) is given by Ellog Pup (z|s,, (s a)), where [ ss, a) is an auxiliary distribution to approximate the posterior distribution p(z]s, a). Thus, the final objective function is given by
N
L( Φb 1b) = LMi (0, Φ)) + Σ
E Ex-n(e) log 9u (2)s;, (o z)).
(10)
The MI-based regularization using the second term in equation 10 encourages the diversity of the behaviors encoded in the sub-policy (als, We empirically show that this regularization improves the performance of the proposed method in Section
4

---

Under review as a conference paper at ICLR 2023
Algorithm Value- Weighte Variational Auto-Encoder (V2AE)
Initialize the actor μas critic Qw, for j = 1, 2, and the posterior 4ø 8, 0)
for = 1 to I do Sample a minibatch {s; a, s/ 1 from D
for each element (8,, ags s',, rs) do
Compute the value the latent variable for s': x' = arg maxe Qu (s', Har 81))
Compute the target value: y/ + y minj-1,2 Qu, (8, po (s', , end for
Update the critic by minimizing E if' mod disterval- = 0 then
M Qw, (s )= for 1,2
Update the actor and the posterior by maximizing equation
(optionally) Update the actor by maximizing Ei=l if end if end for
Bz= log Ie 90 (z|s;, [o(s;)
TRAINING THE CRITIC FOR A MIXTURE OF DETERMINISTIC POLICIES
To derive the objective function for training the critic for the mixture of deterministic policies using the gating policy in equation 3 we consider the following operator:
T²Q= r( +Ba max Q2 ul(' ,(ss,=)))
(11)
We refer the operator T- the latent-max-Q operator. Following Ghasemipour al. (202 prove the following theorems.
Theorem 5.1. In the tabular setting, Tx is a contraction operator in the Loo norm. Hence, with repeated applications the Tz, any initial Q function converges unique fixed point.
The proof of Theorem i. provided in the Appendix
Theorem 5.2. Let Q= denote the unique fixed point achieved in Theorem 5.1 and let 7z denote the policy that chooses the latent variable given by p(s, 2) in a deterministic manner. Then Qx is the Q-value function corresponding to n
arg max (s,, z²)) and outputs the action
Proof. (Theorem 5.2 Rearranging equation 11 with x' = arg max Q2(s', (0 p(s', ( x²)), we obtain
T=Q= = r(s, E., Barnt, [Q=(s', a')]
(12)
Since by definition Q= the unique fixed point of Tz, have result.
These theorems show that the latent-max-Q operator T- retains the contraction and fixed-point ex- istence properties. Based on these results, we estimate the Q-function by applying the latent-mat max max-Q Q employed double-clipped Q-learning (Fujimoto et al.) 2018).
operator. In our implementation, Thus, given a dataset DD the critic is trained by minimizing
((w,)=
Σ
Q (s,, as),a.) yill"
(13)
(
for j 1,2, where the target value y; is computed
yi = + 7 max min Qwy (s', no (1 (s(,,²)). z()). ==1,2
(14)
PRACTICAL IMPLEMENTATION
The proposed Value- Weighted Variational gorithm I As in TD3 (Fujimoto et al.) 2018), the actor is updated once every after dinerval updates of the cnitics. In our implementation, we set dintrvall
(V2AE) algorithm summarized in Al-
2. For modeling the discrete latent variable,
the Gumbel-softmax trick (Jang et al. 2017 Maddison al. 2017). We also the state
5

---

Under review as a conference paper at ICLR 2023
(a) Task setting and (b) Behavior ob- (c) Bchavior ob- (d) Bchavior ob- (c) Visualization data samples.
tained by TD3+BC. tained by AWAC. tained by V2AE.
the latent variable V2AE.
Figure 2: Performance on simple task with multimodal data distribution.
Table Algorithm setup the experiment.
TD3+BC
AWAC
V2AE double-clipped Q-learning mixture & deterministic
Critic training
double-clipped Q-learning monolithic & deterministic
double-clipped Q-learning monolithic & stochastic
Policy type Regularization State
BC term normalized
normalized normalization used in TD3+BC (Fujimoto & Gu 2021 In preliminary experiments, found that when f (s, a) exp (b.A* (s, a)) in equation 9 the scaling factor b has non-trivee non-trivial effects on per- formance and the best value of is different for each task. To avoid changing the scaling parameter for each task, we used the normalization of the advantage function as
normalized
(A* (8, a) (a A (8,
(s max Pra A a. min(s.a)eD... A (s,
(15)
where Doatch is a mini-batch sampled from the given dataset D and is a constant; we set 10 for Mujoco tasks in our experiments. For other hyperparameter details, please refer to the Appendix L
EXPERIMENTS
We investigated the effect of the policy structure the resulting performance and the training error of the critics. In the first experiment, we performed a comparative assessment of TD3+BC (Fujimoto) & Gu 2021), AWAC (Nair et al. 2020) and V2AE on a toy problem, where the distribution of sam- ples in given dataset is multimodal. We also conducted a quantitative comparison between the pro- posed methods and baseline methods with D4RL benchmark tasks (Fu et al. (2020). In the following section, we refer to the proposed method based on the objective in equation 9 as V2AE and a variant of the proposed method with the MI-based regularization in equation T0 as i2fO. toy problem and the D4RL tasks, we used the author-provin provided implementations of TD3+BC, and our implementation of V2AE and AWAC are based on the author-implemennttion TD3+BC.
both the
7.1 MULTIMODAL DATA DISTRIBUTION ON TOY TASK
To show the effect of the multimodal data distribution in given dataset, evaluated the perfor- mance of TD3+BC, AWAC, and V2AE on a toy task shown in Figure The differences between the compared methods are summarized in Table In our implementation of AWAC and V2AE, we used the state normalization and double-clipped Q-learning as in TD3+BC and the normalization of the advantage function described in Section The difference between AWAC and V2AE indicates the effect of the policy representation. In this toy task, the agent is represented as a point mass, the state the position of the point mass wwo-dimensional space, and the action is the small displacement of the point mass. There are two goals in this task, which are indicated by red circles in Figure 2 A blue circle indicates the start position Figure and there are three obstacles, which are indicated by black solid circles. In this task, the reward is sparse: When the agent reaches one of the goals, the agent receives the reward I.0, and the episode ends. the agent makes a contact with one of the obstacles, the agent receives the reward -1.0, and the episode ends. In the given dataset, trajectories to the two goals are provided, and there is no information on which goal the agent is heading.
The score is summarized in Table Among the evaluated methods, only V2AE successfully solved the task. The policy obtained by TD3+BC did not reach the goal in
Table 2: Performance the toy task.
TD3+BC
AWAC
V2AE
0.2 0.4
0.33 0.44
1.0-0.0

---

Under review as a conference paper at ICLR 2023
Table 3: Results on mujoco tasks using D4RL-v2 datasets and AntMaze tasks. Average normalized scores over the last 10 test episodes and five seeds are shown. HC = HalfCheetah, HP Hopper,
WK Walker2d.
TD3+BC
CQL
easyBCQ (re-run)
AWAC
V2AE (ours)
info
(re-run)
(re-run)
(ours)
(ours)
Med. -R Med. -.d.-E Expert
HC
96.3+0.9
22.0:9.6
95.2 1.8
94.82 0.2
97.0= 1.0
95.64 2.0
HP
109.9:12..5
105.8:13..8
70.4:1 8.0
109.811 2.9
93.6.1 15.1
107.5: 2.9
WK
110.2:10.44
108.9:10.4
103.6:1: 23.0
111.0: 0.2
111.4: 0.3
112.1: 0.4
HC
89.4
38,418,4
92.2:10.9
92.7= 0.8
91.1
91.41
95.5+19.44
88.4+15599
14.6:t 3.4
98.6+ 10.7
78.4+ 19.0
94.5++1499
WK
110.2+0..
109.2+1.99
111.9 0.6
109.2 0.3
109.9+ 0.4
110.1 0.7
HC
44.7
06.9
41.31 0.8
40.9 0.6
45.2 0.8
46.7 0.6
HP
73.8+18.9
95.5
86.41 18.9
38.21
89.2 8.1
98.5
WK
64.5+17.0
77.5+
65.1 18.5
65.0+ 15.7
82.1+ 3.8
86.7
HC
48.2110.3
48.210.4
42.7 1.8
44.31
47.5 0.4
48.6 0.4
Med.
HP
61.011:.2
77.41:.00
79.1
57.5 3.0
71.2 6.5
86.4
WK
84.7++1.3
81.5+225
83.2+ 0.8
81.0 2.5
79.4 4.7
85.0 0.8
HC
11.5:10.6
24.1
9.1 0.6
3.2 1.3
15.811 1.6
16.31 1.2
Rand.
HP
8.710.3
2.2
7.6
7.3 0.9
12.0 10.0
20.4 9.8
WK
1.4+199
4.3
6.0 1.3
3.1 1.0
2.5 2.6
2.3+ 2.0
Total
1010.0
930.3
908.5
956.5
1026.5
1102.1
umaze
92.8+ 2.7
73.0#4.9
0.0:50.0
49.82 6.2
83.6
88.4 + 3.6
Antmaze
umaze-d.
45.0+ 22.2
43.8+4.44
9.6+ 12.2
53.8+ 13.0
43.2+t 7.8
52.8++7.9
med.-p.
0.0+ 0.0
9.0++6.4
0.0 0.0
0.0+
77.0+ 5.1
48.6-+25.3
med.-d.
0.0 0.0
3.8+4..2
0.0+000
0.0+
56.8+ 27.2
59.2++29..4
large-p.
0,0
0.0:0.00
0.0
0.0
1.0.1 1.3
5.2 1 8.4
large-d.
0.0:1 0.0
0.010..4
0.0 0.0
0.0 0.0
4.8.1: 9.6
6.6 1 5.2
Total
137.8
129.6
9.6
103.6
266.4
260.8
stable manner as shown in Figure 2 b). Similarly, as shown in Figure c).), the policy learned by AWAC often slows down around the point (0,0) and fails to reach the goal. This behavior implies that AWAC attempted to average over multiple modes of the distribution. By contrast, the policy learned by V2AE successfully reaches of the goals. As the main difference between AWAC and V2AE is the policy architecture, this result shows that the unimodal policy distribution fails to deal with the mutimodal data distribution, while mixture policy employed in V2AE successfully dealt with it. The activation of the sub-policies is visualized in Figure -e. The color indicates the value of the discrete latent variable given by the gating policy arg max, Qwo (8, (1(8, 2)). Figure d)) shows that different sub-policies are activated for different regions, which indicates that V2AE appropriately divided the trate-action space.
7.2 D4RL BENCHMARK TASKS
We evaluated the performance of the proposed method on the benchmarking tasks in D4RL. As base- line methods, we used TD3+BC, CQL (Kumar et al.] 2020), AWAC, and easyBCQ (Brandfonbbrner et al. 2021). EasyBCQ is one-step RL version of Batch Constraint Q-learning (Fujimoto al.) 2019). All the baseline methods use double-clipped Q-lcarning for the critic in this experiment. For easyBCQ the state normalization and double-clipped Q-learning
used in implementation.
The implementation of AWAC and V2AE was identical to those in the previous experiment, and the difference between AWAC and V2AE indicates the effect of the policy representation. In evaluation, |Z| = 8 was used for V2AE and info The effect of the dimensionality of the discrete latent variable is shown in Appendix D In this study, we evaluated the baseline methods with antmaze-vd) Kitchen, Adroit, and mujoco-v2 tasks. For completeness, we provide the results with the mujoco-vO datasets in Appendix
Performance scores Comparisons with the baseline methods are shown in Tables and 4 The bold text indicates the best performance; the underlined text indicates the tasks for which V2AE outperformed AWAC. In mujoco-v2 tasks, we can that the performance of V2AE is compa- rable/superior to the sfate-of-the-re art methods on mujoco-v2 tasks. Regarding the comparison be- tween V2AE and AWAC, the performance of V2AE matched exceeded that of AWAC except the Hopper-expert and Hopper-ne Hopper-medium medium eexiitt-expertt tasks. This result also confirms that the use a mixture policy beneficial for these tasks, although the benefits of using mixture policy should

---

Under review as a conference paper at ICLR 2023
Table 4: Results Kitchen and Adroit tasks using the average normalized scores over the last 10 test episodes and five seeds.
TD3+BC
CQL(p)
casyBCQ (re-run)
AWAC
V2AE (ours)
info
(re-rum)
(paper)
(ours)
(ours)
complete
3.5 1.5:77 7.0
31.3
1
48.0 14.4
59.0 12.5
49.5 16.2
Kitchen
mixed
0.0 0.0
52.4
10.0 5.7
55.0 6.9
51.0= 9.8
39.5#
10.3
partial
0.0 0.0
50.1
12,0 + 11.4
38.0 8.3
38.5 1:60
31.0 4.6
Total
3.5 7.0
133.8
39.0 + 26.0
141.0 + 29.6
148.5 + 28.3
120+ 31.1
pen
0.8 1 8.0
55.8
68.7 18.7.88.4
65.311 20.4
86.11 8.8
94.8 16.5
Human
hammer
1.9 0.8
2.1
2.3 1 2.4
1.7.1 0.5
1.2.1 0.2
2.4.1:
door
-0.
9.1
3.4 4.1
6.0 4.2
1.3 1.5
4.2: 3.1
relocate
0.0
0.0
0.2
0,0+0,0
0.0 0.1
0.1 0.0
Total
1.
67.0
74.6
15.2
73.0 + 25.1
88.6 = 10.6
97.3 = 20.5
pen
7.0
40.3
59.9 -29.7
17.74 14.9
36.0 17.7
46,41 16.7
Cloned
hammer
0,2
5.7
0.0
0.4.1
0.81 0.6
1.2.1..3
door
0.0
3.5
0.0
1.0 1.2
0.0
0.8 0.8
relocate
-0.3 0.0
-0.1
-0.3 0.0
-0.2 0.0
-0.2 0.0
-022 = 0.0
Total
0.1
42.4
60.0
29.7
18.9 + 16.1
36.1 = 18.3 48.2 = 15.8
be task-dependent. Remarkably, our implementation of AWAC showed significantly better perfor- mance than that reported by Nair et al. (2020) because employed the double-clipped Q-learning and state normalization. We provide the comparison between the original results of AWAC and ours in Appendix [ In addition, i2AE, which employs the MI-based regularization, outperformed V2AE on various tasks, and V2AB showed the best performance in terms of the sum of the over- all scores on mujoco-v2. This result shows that encouraging the diversity of the sub-policies using the proposed MI-based regularization is effective for V2AE. The advantage of V2AE and info V2AE over baseline methods is apparent for Antmaze tasks, which involve dealing with long horizons and require "stitching" together sub-trajectories in a given dataset (Fu et al. 2020p. TD3+BC, CQL, easyBCQ, and AWAC did not work well on Antmaze tasks, and this result indicates that techniques used in these algorithm are not effective for such tasks. As the difference between V2AE and AWAC indicates the effect of different policy representations, the results indicate that the use of the mixture policy improves the performance to deal with tasks that require stitching together sub-trajectories in given dataset. A comparison with additional baseline methods is provided in Appendix The effect of the scaling parameter in V2AE and info V2AE is reported in Appendix H
Critic loss function To investigate the effect of the policy structure on the critic loss function, we compare the value of the critic loss function between AWAC and V2AE. In addition, to see the differences between the discrete and continuous latent variables, we also evaluated variant of V2AE with the continuous latent variables, which we refer as VVEA.. The normalized scores and the value of the critic loss function during training are shown in Figure 3 The value of the critic loss given by equation 13 is plotted for every 5,000 updates. The shaded area that indicates the error bar of the critic loss is removed for V22AE on walker244mediim-eeppa loss of cV2AE exploded. Previous studies indicated that the value of the critic loss can accumulate over iterations (Brandionbrener et al. 2021). p. Figure 3 shows the accumulation of the critic loss in AWAC on the mujoco-v2 tasks. Importantly, in V2AE, the value of the critic loss is clearly lower, and the performance of the policy is better than that in AWAC. Branddfobener al. (2021) showed that the accumulation of the critic loss can be reduced by introducing regularization. Since the difference between AWAC and V2AE is the policy representation, our results indicate that the use of the mixture policy can also mitigate the accumulation of the critic loss in offline RL. This result suggests the importance of incorporating inductive bias in the policy structure. However, it is worth noting that the reduction of the critic loss given by equation [3 does not necessarily lead to the improved performance of the policy. In malfchhetta-meddiune expeem-exxpe2.. although the critic loss was significantly lower in V2AE than that in AWAC, there was no significant difference in performance between V2AE and AWAC. Recently, Fujimoto et al.] (2022) indicated that a lower value of the critic loss given by equation 13 does not necessarily lead to better performance, and the observation in Fujimoto et al. (2022) fits with what we observed in our experiments. Regarding the use of the continuous latent variable, while eV2AE often achieves lower values of the critic loss than AWAC, the critic loss of c V22E occasionally explodes. A possible explanation for these results may be that learning the continuous latent variable provides flexibility to policy representation but separate
because the critic
8

---

Under review as a conference paper at ICLR 2023
balfhhetah-mediineeeprec
hogper-medin--ellllyyy.
Walker dww.diumm--eeom pert-a22
1,00
Kome
9.4
9.75
Normi Dommttaaatii
0.2
AMAC
9.50
ARAC
AMACC COEAE V2AE
AWACC CY2A2
<YEAE
0.25
CUEAE VIAT
VOAC
0.20
0.25
3:75
6.5
1.00
0.25
0:76
5,50
3.00
Time (lei)
Time sieps Ilest
(2ee))
halfcheetah--ree edium expprttr22
hoppe--medimm-epply--
walker 26 eelum-expett-222
603
203
0.25
0.50
1.00
0.25
0.50
0:75
3,50
0.75
1.02
0.25
3.50
0.75
Time stopa (3:6)
Time steps (146)
Time stapa (240)
Time abaza (2ed)
Figure 3: Critic loss and normalized score during the training.
gee
20
1 40
80
100
2=5
z=0
z=6
2=
Figure 4: Visualization of the activation of sub-policies in the pen-human-v0 task. The top row shows the state at the 20th, 40th, 60th, and 80th time steps; the graphs in the middle row show the action-values of each of the sub-policies at each state.
modes of the objective function are still interpolated in the learned latent space, which results in generating aut-o-ddistriutton samples. The performance of cV2AE on mujoco-v2 tasks is reported in Appendix]] [ and more detailed results on the critic loss are provided in Appendix
Qualitative evaluation of the learned latent variable The activation of sub-policies in V2AE on the cen-human-vo) task is shown in Figure 4 In Figure 4 the top row shows the state at the 20th, 40th, 60th, and 80th time steps; the graphs in the middle row of the figure show the action- values of each of the sub-policies at each state, Q p1ss 2)). previous study on the option framework (Smith et al. (218)) reported that in the existing method only a few options are activated and that the remainder of the options do not learn meaningful behaviors. In contrast, the results in Figure show that the value of each of the sub-policies Qw 14(8 =)) changes over time, and various sub-policies are activated during the execution. A figure showing the activation sub-
policies in different episodes is provided in Appendix K
CONCLUSION
We have proposed an algorithm, which we call Value- weighted Variational Auto-Encoder (V2AE), for training mixture policy in offline RI. The V2AE algorithm can be interpreted an approach that divides the state-action space by learning the discrete latent variable and learns the correspond- ing sub-policies in each region. Our experimental results show that the use of the mixture policy can mitigate the issue of critic error accumulation in offline RL. In addition, the experimental results also indicate that the of the mixture policy significantly improves the performance of offline RL algorithm. We believe that our work represents an important step toward leveraging the policy structure in offline RI.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of the results, we will provide the codes by posting a comment directed the reviewers and area chairs and putting link an anonymous repository after the discussion forums are opened. We also summarized how to implement the proposed method and its variants in Section 6 More detailed descriptions of the implementation can be found in Appendix L open- open-sourced benchmark tasks in D4RL to ensure reproducibility.
We used
REFERENCES
Riad Akrour, Davide Tateo, and Jan Peters. Continuous action reinforcement learning from a mixture of interpretable experts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021. doi: 10.109/TAAM122211100011
Gaon An, Seungyong Moon, Jang-Hyum Kim, and Hyun Oh Song. Uncertainty- offline rein- forcement learning with diversified q-ensemble. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Pierre-Lue Bacon, Jean Harb, and Doina Precup. Option-critic architecture. In Proceedings of the AAAI Conference Artificial Intelligence (AAAI), 2017.
David Barber and Felix Agakov. The im algorithm variationalapproachh information maxi- mization. In Advances in Neural Information Processing Systems (NeurIPS), 2003.
David Brandfonbrener, William F. Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Lili Chen, Kevin La, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems (NeurIPS), 2021a.
Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Jianhao Wang, Yuan Gao, Wenzhe Li, Bin Liang, Chelsea Finn, and Chongjie Zhang. Lapo: Latent-variable advantage-weightee policyoptimization for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Xiong-Huu Hui Chen, Yang Yu, Qingyang Li, Fan-Ming Luo, Zhiwei Qin, Wenjie Shang, and Jieping Ye. Offline model-base adaptable policy learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021b.
Christian Daniel, Gerhard Neumann, Oliver Kroemer, and Jan Peters. Hierarchical relative entropy policy search. Journal of Machine Learning Research, 17(93):1-50, 2016.
Dayan and Hinton. Using pnpecttaioonmmaxiizzziino for reinforcement learning. Neural Com- putation, 2278,277, 1997.
Emilien Dupont. Learning disentangled joint continuous and discrete representations. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 2021.
10

---

Under review as a conference paper at ICLR 2023
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation emror in actor-critic methods. In Proceedings of the International Conference on Machine Learning (ICML), pp. 1587-1596, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the International Conference on Machine Learning (ICML), pD- 2052-2062, 2019.
Scott Fujimoto, David Meger, Doina Precup, Ofir Nachum, and Shixiang Shane Gu. Why should i trust you. bellman? the bellman error a poor replacement for value error. In Proceedings of the International Conference on Machine Learning (ICML). 2022.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected- max q-learning operator for simple yet effective offline and online rl. In Proceedings of the International Conference Machine Learning (ICML), volume 139, pp. 3682-3691. PMLR, 18-24 Jul 2021.
Wonjoon Goo and Scott Nickum. You only evaluate once:
simple baseline algorithm for offline rl.
In Proceedings of the Conference on Robot Learning (CoRL), 2021.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. In Proceedings of the International Conference on Learning Representations (ICLR), 2014.
J. Kober and J. Peters. Policy search for motor primitives robotics. Machine Learning, 84: 171-203, 2011.
Iyy Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In Proceedings of the International Conference Machine Learning (ICML), 2021.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine Offline reinforcement learning with implicit q- learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Sascha Lange. Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforce- ment Learning, pp. 45-73, 2012.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv, 2020.
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relax- ation of discrete random variables. In Proceedings of the International Conference on Learning Representations 2017,,
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the International Conference Machine Learning (ICML), 2016.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. AWAC: Accelerating online reinforcement learning with offline datasets. arXiv, arXiv:2006.09359, 2020.
Gerhard Neumann and Jan Peters. Fitted q-iteration by advantage weighted regression. In Advances in Neural Information Processing Systems (NeurIPS), 2008.
Takayuki Osa, Voot Tangkaratt, and Masashi Sugiyama. Hierarchical reinforcement learning via advantage-weigee information maximization. In Proceedings of the International Conference Learning Representations (ICLR), 2019.

---

Under review as a conference paper at ICLR 2023
Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted space control. In Proceedings of the International Conference on Machine Learning (ICML), 2007.
weighted regression for operational
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
Matthew Smith, Herke Hoof, and Joelle Pineau. An inference-based policy gradient method for learning options. In Proceedings of the International Conference Machine Learning (ICML), 2018.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems (NeurIPS), 2015.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, Second edition, 2018.
E. Todorov, T. Frez, and Tassa. Mujoco: physics engine for model-based control. 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5026-5033 2012.
Qing Wang, Jiechao Xiong, Lei Han, peng sun, Han Liu, and Tong Zhang. Exponentially weighted imitation learning for batched historical data. In Advances in Neural Information Processing Systems (NeurIPS). 2018
Markus Wulfmeier, Abbas Abdolmaleki, Roland Hafner, Jost Tobias Springenberg, Michael Ne- unert, Tim Hertweck, Thomas Lampe. Noah Siegel, Nicolas Heess, and Martin Riedmiller. Com- positional transfer hierarchical reinforcement learning. In Proceedings of Robotics: Science and Systems (R:SS), 2020.
Markus Wulfmeier, Dushyant Rao, Roland Hafner, Thomas Lampe, Abbas Abdolmaleki, Tim Her- tweck, Michael Neunert, Dhruva Tirumala, Noah Siegel, Nicolas Heess, and Martin Riedmiller. Data-efficient hindsight off-policy option learning. In Proceedings of the International Confer- ence on Machine Learning (ICML), 2021.
Shangtong Zhang and Shimon Whiteson. Dac: The double actor-critic architecture for learning options. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforce- ment learning. In Proceedings of the Conference on Robot Learning (CoRL), volume 155, pp. 1719-1735, 2020.
12

---

Under review as a conference paper at ICLR 2023
A APPLICABILITY OF THE GATING POLICY
In the proposed algorithm, we employ the gating policy that determines the value of the latent variable as follows:
arg max Qu Ho z²)),
(16)
where ( z') represents the deterministic sub-policy and Qw a) is the approximated Q- function. While this gating policy looks specific to the case where Q" a) is maximized, this gating policy is applicable to other objective function such A" a), (xp(Q=(s,a)) a)), and exp(A"(< a)). The advantage function is defined as A* a) = Q* a) - V*(s). As the
state value function (s) is independent from the action, we can obtain the following equation:
arg max Q' (s, a) = arg max (Q" (s, a) - (s))
(17)
a arg max A a).
(18)
Thus, we can rewrite the gating policy as
arg max Q 8, po p z'))
(19)
arg max A (s, ( z)))
(20)
x'
Similarly, the exponential function exp(-) is the monotonically increasing function. Thus, the ex- trema of (* a) is also the extrema of (epp("(ss. a)). Thus, we can also rewrite the gating policy as
x = arg max Qw uo(s z²))
(21)
arg eaxexp (Q (,, uo (8, z'))
(22)
arg max Aw (8, Ho z'))
(23)
arg exx (s (s, u )))
(24)
As the latent variable is discrete, can analytically compute arg max= (o, As used this gating policy, the gating policy is deterministic in our implementation.
B DERIVATION OF THE VARIATIONAL LOWER BOUND
We employed the variational lower bound in equation to derive the objective function for the proposed method. We provide the detailed derivation, which was omitted in the main manuscript due to the page limitation. We denote by p(-) the true distribution induced by the policy тH (als). and the distribution that approximates the true distribution is denoted by q()). The KL divergence between q(x) and p(x) is defined as
DKL (g(x))lp(x) = f g(xx) log dz.
g(0)
(25)
p(ø)
Based on the above notation, the likelioiddoood log To (a,a)s))) can be written as follows:
log no ( (a. =
(0(2)), a.) log To (a/ls,)ds
(26)
(0(2)s a.) (log T (s, x) - logp(z|ss) logp(> s,, a;))
(27)
= / 10 (0 a.) log g(2 a:) dz / a(2))), a:) log g(2 8i, ai)
dz
(28)
p(z]s;, a.)
p(z(s:)
+ / <<]|s, a.) log no
(29)
DKL (a(2) si a:))pp(((sss......)))) a:)) DKL ())ls(((sss)
+ E [x=g(=(+, [log no ( (8 =)
(30)
13

---

Under review as a conference paper at ICLR 2023
Table 5: Effect of the dimensionality of the discrete latent variable Adroit tasks.
info
info V2AE
info
infoV2AE
|Z|
|Z]
|2| =
= 32
pen human--n---
75.7 = 18,9
94.8 + 16.5
75.0 = 17.5
86.7 = 12,4
fitchen--comllete
30.0 + 13.5
35.5 + 29.7
49.5 = 16.2
42.5 + 20.8
walker22-exxertt-22
99.7111.99
112.110..4
108.8+6.8
106,4110.2
89.1122.7
110.1:10.7
96.01117.0
109.9110.6
walker2d-men medpiid-mmdduu--ppaa--a-a
81.6=44.5
86.7+3.2
85.443.7
86.3+3.11
liikerrddmedium---.
81.8+2.5
85.0
69.9+28.3
84.3++1.0
In the first line, we consider the marginalization over z. As log (a)s)) independent from the latent variable 2, the equality in the first line holds. As DKL (g(2|s, a) [[ a)) > 0, we can obtain a variant of the variational lower bound of the conditional log-likelihood:
log (a;a))s;)
-KL DKI(gs<z|s; (alss,;;))))) (:)ss((z)ss) Exwa(= a.,a)) [log (a
(31)
C PROOF OF THE CONTRACTION OF THE LATENT-MAX- OPERATOR
We consider the operator Tz, which is given by
a=Q(s, = Es (r(s.a) + max ((s's((s'' aa(s',z') x²)1]
(32)
To prove the contraction of Tz we use the infinity norm given by (Q, a) Q2(8.
|| - Q21lae
max =ES,aEA
(33)
where Q1 and Q2 are different estimates of the Q-function. We consider the infinity norm of the difference between these two estimates, Q1 and Q2, after applying the operator Tz:
|TQQ] T=Q21lac
(34)
Es (((s))) a) max Q1(s', p(s', ,'))
Ee [rss. a) + max Q2(s', (a(ss,u(s',,
(35)
Ex
max Q p(s'. x²)) - yE max Q2(s', uss', z'))
(36)
Es
max Q1 uls', x1)] - Eg max Q2(s', u(s' x'))
(((s'='))]
(37)
y
E. max Qi (s', muss, max Q2(s' u(s' z1))
(38)
S y | [|Q1 Q21lael
(39)
$ |1Q Q21loo
(40)
The above relationship shows the contraction of the operator Tz.
D EFFECT OF THE DIMENSIONALITY OF THE DISCRETE LATENT VARIABLE
In our evaluation, we first evaluated the effect of the dimensionality of the discrete latent variable. The results are shown in Table 5 As |Z| 8 consistently showed satisfactory performance, |2| was used in the subsequent evaluations.
As shown, i2fo with |Z| 8 demonstrated the best performance, while the performance with |Z| 16 and |Z| = 32 is comparable. These results show that the performance of the policy not sensitive to the dimensionality of the latent varialbe. However, the performance with |Z| = 4 is relatively weak, and it indicates that the policy may not expressive enough when the dimensionality of the latent varialbe is too small.
14

---

Under review as a conference paper at ICLR 2023
Table 6: Results Mujoco tasks using D4RL.-vO datasets. The average normalized scores after million time steps over the last 10 test episodes and five seeds are shown. HC HalfCheetah, HP = Hopper, WK Walker2d, Med.-E EE Mediaum-Expeer Expeer, Med.-R Medium- Reppium-Repla,, Med.-Medium, and Rand.=Random. The bold text indicates the best performance; the underlined text indicates the tasks for which V2AE outperformed AWAC. | for V2AE.
TD3+BC (re-run)
CQL
easyBCQ
AWAC
V2AE
(re-run)
(ours)
(ours)
(ours)
Med. Rod.-R/o Med. E Expert
HC
104,9 = 2.9
9.99:18.9
101.8 0.8
107.31 0.3
103.9 1.9
HP
112.1 + 0.3
112.4:10.4
108.4 1.8
112.0:: 0.2
112.2:: 0.2
WK
105.8 +
87.8 31.1
83.1 + 19.3
104.4+
3.2
105.5 3.7
HC
102.3 + 3.8
23.8 + 12.4
41.2:t 1.8
101.7 4.9
83,9 20.4
HP
111.9 0.1
112.3 + 1.0
103.7 + 13.5
111.9+ 0.1
112.2+ 0.1
WK
96.7
21.7
100.6 11.2
93.4 + 10.2
101.7+
103.3+ 2.5
HC
0.3
0.4
45.2+ 1.4
0.3
42.6 0.4
HP
33.5 +
29.8
32,2 + 2.0
27.7+ 1.3
32.0+ 4.8
WK
14.1
4.07
16.8 + 1.6
2.0
17.3+ 2.4
0.4
40.7:10.6
43.5 0.6
4.4
41.9
Rand. Med.
HP
99.8 1 0.3
47.01:26.0
37,4 10.0
3.3
57.9.1 25.6
WK
81.4
69.3+24.1
7.7
53.8+
62.0+ 13.4
HC
13.0
28.51 1.7
0.4
0.5
17.1 1.2
11.0
10.7.0.11
11.6 0.2
10.6.1..1
11.5 0.5
WK
1.3
1,5+3.2
5.2
3.2
2.0 2.5
E PERFORMANCE ON D4RL-vO DATASETS
As reported in (Fujimoto & Gu 2021); there is non-negligible difference when using the D4RL- v2 and D4RL-v0 datasets. Having presented our results using D4RL-v2 datasets in the main manuscript, here show the results using D4RL-vO datasets in Table 6 For CQL, show the results obtained by re-running the codes given in the website") We performed the experiments with implementations of easyBCQ and AWAC. The results were mixed, and it is difficult to identify the best algorithm for the MuloCo tasks with the D4RL-vO datasets. However, we can see that the performance of V2AE comparable the state-of of-ee---theerrr methods on the MuJoCo tasks with the D4RL-v0 datasets. Especially, compared to the baseline methods, the performance of V2AE con- sistently matched exceeded the performance of the other methods datasets containing expert demonstrations, e.g., *-expert and *-medium-expprt
F PERFORMANCE OF AWAC
We used two techniques in implementation of AWAC, which used in the original pa- per of AWAC (Nair et al. 2020): 1) state normalization proposed in (Fujimoto & Gu 2021) and 2) normalization of the advantage function in equation T3 We provide the comparison between the performance of our implementation of AWAC and the performance reported in the original pa- per (Nair al., (2020). Our experiments show that our implementation significantly outperformed the results reported in the original paper of AWAC.
G COMPARISON WITH ADDITIONAL BASELINES
We We provide comparison with additional baselines on mujoco-v2 tasks in D4RL in Table show the results of MAPLE, which is recent model-based offline algorithm using latent represen- tations (Chen et al. 2021b). In addition, we show the results of Decision Transformer (Chen al.) 2021a), as a representative of transformer-based methods. We also included Implicit Q-learning (IQL), which employs expectile regression for learning the Q-function/ Kostrikov al. al, (2022). While these methods
and tate-of-the-art, omit them in the main manuscript due to the page limitation, and we focused on methods that use double-clipped Q-learning for the critic in the
https::/gitthb..co avi ral kumak kumar2977/Q01
15

---

Under review as a conference paper at ICLR 2023
Table 7: Comparison of AWAC implementations Mujoco tasks using D4RL.vO datasets. The average normalized scores after million time steps over the last 10 test episodes and five seeds are shown. The bold text indicates the tasks for which the performance of our implemnetation AWAC exceeded the that of AWAC reported in the original paper (Nair et al. 2020 HC HalfCheetah, HP Hopper, WK Walker2d.
AWAC (paper)
AWAC
(ours)
Med. R--RRMM Med. E Expert
HC
78.5
107.3t 0.3
HP
85.2
112.0:: 0.2
WK
57.0
104,4+
3.2
HC
36.8
101.71 4.9
HP
80.9
111.9 0.1
WK
42.7
101.7+ 7.3
HC
40.7 0.3
HP
27.7 1.3
WK
13.9 2.0
37.4
40.0 0.4
Med.
HP
72.0
30.2.1 0.3
WK
30.1
53.8+ 9.9
HC
2.2
2.5.1 0.3
Rand.
9.6
10.6 0.1
WK
5.1
3.2.L 1.7
Table Results on MuJoCo tasks using D4RL-v2 datasets. Average normalized scores over the last 10 test episodes and five seeds shown. HC HalfCheetah, HP Hopper, WK Walker2d. The gray text indicates the performance lower than that of V2AE/infoV2AE. The bold text indicates the best performance.
MAPLE
Decision Transformer (paper)
IQL
V2AE
infoV2AE
(paper)
(paper)
(ours)
(ours)
Med. R Med. E
HC
63.5 + 6.5
86.8 + 1.3
86.7
91.1+ 3.4
91.44 2.5
HP
42.5 + 4.1
107.6 + 1.8
91.5
78.4: 19.0
94.5
WK
73.8
8.0
108.1 0.2
109.6
109.9
110.1
HC
59.0
+ 0.6
36.6 + 0.8
44.2
45.2+ 0.8
46.7+ 0.6
HP
87.5
1: 10.8
82.7 7.0
94.7
8..2
98.51 2.0
WK
76.7
3.8
6.00
73.9
8.81
86.7.1 3.2
HC
50.4
1.9
4..6
47.4
4.4
48.6+ 0.4
Med.
HP
21.1 1.2
67.6
66.3
71.2+ 6.5
86.4+ 7.6
WK
56.3
10.6
74.0 L 1.4
78.3
79.4.14.7
85.0.L 0.8
main manuscript. For each baseline methods, we adapted the results reported in the original paper. V2AE and info V2AE show performance consistently better than or comparable to these baseline methods, although our implementation of V2AE and infoV2AE do not employ techniques such as ensemble of critics. This result indicates the significant effect on the policy structure in offline RL.
G.1 COMPARISON WITH AWAC USING GAUSSIAN MIXTURE POLICY
We also provide the comparison with AWAC with a Gaussian mixture policy in Table Overall, the use of the Gaussian mixture policy does improve the performance of AWAC. recent study by Chen et al. 2022) indicates that the use of the Gaussian mixture policy does not improve the performance of AWAC. When Gaussian mixture policy employed, a Gaussian component that covers large part of the state space often appears, and it governs the resulting performance. This behavior is also often observed in the context of the option critic for online RL (see (Smihh et al. 2018 ). If that happens, cannot exploit the discrete latent variable.
By contrast, we employed a mixture deterministic policies, not a mixture the Gaussian policies. Unlike Gaussian policy, deterministic policy be seen distribution given by the dirac-delta function, which is the limit of the Gaussian as the standard deviation goes zero. When we use the mixture of deterministic policies, will not have component that covers the large state space.
16

---

Under review as a conference paper at ICLR 2023
Table 9:
Comparison with AWAC with a Gaussian mixture policy (mixAWAC) using D4RL-v2 datasets. Average normalized scores over the last 10 test episodes and five seeds are shown. The bold text indicates the best performance.
miXAWAC
AWAC
V2AE
info
Med. -RR Med. E Expert
HalfCheetah
94.0+0.5
94.8+ 0.2
97.0
1.0
95.6+ 2.0
Hopper
111880.8
109.8=29
93.61 15.1
107.51 2.9
Walker2d
110.5-0.3
111.0 0.2
111.41 0.3
112.11 0.4
HalfCheetah
92.1+0.6
92.7+ 0.8
91.1 3.4
91.4 2.5
Hopper
102.0
98.6.1 10.7
19.0
94.5114.9
Walker2d
109.1-0.3
109.2 0.3
109.91 0.4
110.11 0.7
HalfCheetah
41.5+0.4
40,9
45.2
46.7+ 0.6
Hopper
41.2+4.77
38.2+
89.2 8.1
98.5+ 2.0
Walker2d
67.74188
65.0.1 15.7
82.11
86.71 3.2
HalfCheetah
45.1+0..3
44.3 0.2
47.5
48.6
Med.
Hopper
57.2+3.9
57.5# 3.0
71.2+
86.4 7.6
Walker2d
78.7 4: 4.8
81.0+ 2.5
79.4+
85.0 0.8
HaltCheetah
2.2+0.0
3.2
15.8 1.6
16.3 1.2
Rand.
Hopper
8,240.22
0.9
12.0 10.0
20.4 9.8
Walker2d
4.911.1
3.1
2.6
2.3 2.0
umaze-v0
$7.416.2
49.8.16.2
83.6.14.5
8..4
Antmaze
umaze-diverse-vo
46.8116.9
53.8.113.0
7.8
52.8.1.99
-aaiium-play-vO
0.0 0.0
0.0+0.0
77.0+..11
48. 28.6:225.3
medium-divees divers--vv
0.0 0.0
0.0+0.00
56.8+:7.22
59.2++29.4
large-play-v0
0.0 0.0
0.0 0.0
1.011.
5.2 8.4
large-diverse-v0
0.0+ 0.0
0,0 0.0
4.8.19.6
6.6 5.2
The use of the mixture of deterministic policies allows us distribute each component separately, and we can exploit the benefit the discrete latent variable. In addition, we also leverage the vari- ational lower bound incorporate the prior distribution of the latent variable, which enables to obtain meaningful latent representations. These are the reasons why our methods clearly outper- formed AWAC, while simply using the Gaussian mixture policy does not improve the performance of AWAC.
G.2 COMPARISON WITH LAPO
Recently, Chen et al. 2022 proposed an algorithm, Latent-variable Advintage-weightee Policy Optimization (LAPO), which leverage the continuous latent space for policy learning. As the approach is related to our method V2AE, we provide the detailed comparison in this section. We found that the authors' implementation of LAPO in LAPO-offlienRe includes techniques to improve the performance, such as action normalization and clipping of the target value for the state-value function. Such techniques compatible with V2AE and our baseline methods, but they are not incorporated our experiments. For this reason, first evaluated LAPO without these techniques, which we refer LAPO-
com/ httm//cchenni//
The results are reported in Table 10 Overall, V2AE and info V2AE outperformed LAPO on mujoco- v2 tasks and Antmaze- v0 tasks. While our method V2AE leverages the discrete latent space, LAPO uses the continuous latent space. In this sense, LAPO is closer to cV2AE, which is a variant of V2AE that uses the continuous latent space. In the main manuscript, we showed that the critic loss value often explodes during the training of cV2AE. Similarly, the critic log value increases rapidly at the beginning of the training in LAPO, as shown in Figure 5 While the critic log value often decreases at the end of the training of LAPO, the critic loss value is still higher than that of V2AE. As the performance of the policy better in V2AE these tasks, the surge of the critic loss value indicates the generation of the out-of-distribution actions during the training in LAPO. These results support our observations in the main manuscript and indicate that the use the discrete latent variable is effective to reduce the generation of the out-of-distribution actions.
G.3 COMPARISON WITH IQL AND LAPO ON ANTMAZE
As the Antmaze tasks are considered as challenging tasks on offline RL, we provide detailed results on the Antmaze tasks in this section. As found that few techniques used in LAPO essential
17

---

Under review as a conference paper at ICLR 2023
Table 10: Comparison with AWAC with a Gaussian mixture policy (mixAWAC) using D4RL-v2
datasets. Average normalized scores over the last 10 test episodes and five seeds are shown. The
bold text indicates the best performance.
LAPO-
V2AE
info
Med. R Med. E Expert
HalfCheetah
94.9-FO.2
97.0+
1.0
95.6+ 2.0
Hopper
111.4:0.6
93.6:t 15.1
107.51 2.9
Walker2d
111.110.2
111.41 0.3
112.11
HalfCheetah
94.2+00.4
91.1
91.4 2.5
Hopper
110.911.0
78.41 19.0
94.5
Walker2d
110.4.0.1
109.91 0.4
110.11 0.7
HalfChectah
40.7
4..2
46.7=
Hopper
52.6+11.3
8.12+
98.5 2.0
Walker2d
60.1111.7
8..1
86.71
HalfCheetah
45.5#0.2
0.
48.6 0.4
Med.
Hopper
51,8#3.6
6.5
86.4 7.6
Walker2d
77.743.55
7.74+:..7
85.0 0.8
HalfCheetah
2.3+0.3
15.8 1.6
16.3= 1.2
Rand.
Hopper
21.8111.3
12.0 10.0
20.4= 9.8
Walker2d
5.6+3..1
2.5 2.6
2.3-
umaze-v0
76.41:..
83.6:t..5
88,4=3.66
Antmaze
mmaze-diverse-vv
56.618..6
43.2 7.8
52.8=7.9
medium-play
play-v0
0.8+1.6
77.0=5.11
48.6=25.3
mcdium-diveers-vv)
0.0+0.00
56.8++77.2
59.2-2994
large-play-v0
0.0+00.0
1.0+1.
5.2+ 8.4
large-diverse-v0
0.0+0.0
4.8+9.66
6.6 5.2
wwt----mmmummmmee replay
helicheedah-medlumme expeete-ettttn
exppert22
expprrt-2222
9.4
2.0
3.2
0.30
0.25
0.20
Time steps (19))
Time Stass (109)
Time staps 1199
Time stops llefi
bellchetth-mediummrsayyy
halftheetah medium expertvv2
walser2d were//dddeddiummmm expert-v2
walser2d reedomm---
603
Cost
603
Crisc
200
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
0.25
0.50
0.25
1.09
9.25
0.50
0.75
1.02
stape (16))
stapo (Le))
stops (et)
Figure 5: Critic loss and normalized score during training on mujoco-v2 locomotion tasks.
to reproduce the results in Chen al. 2022) we also evaluated variants V2AE, which incorporate such techniques. In methods named like lapo"," the techniques used in LAPO et al. (2022
are incorporated. We summarized the techniquesdiiferrnces as below:
computation of the target value Quaget 0.7 min Q1.Q2) 0.3 max (Q1, Q2)
target value clipping Utarget = min(max/ Utarget Umin) Umax)
reward scaling r + r x 100
network architecture: three hidden layers with 256 units, learning rate: 2c-4
For target value clipping, Umin and Umax are computed as Umin minp r T and Umax maxp r T-y, respectively. In LAPO- IQI,, V2AE, info we used the reward r +- r following the protocol used in the original IQL paper. While action normalization is used in LAPO, we found that the action normalization does not improve the performance of IQL and V2AE. Therefore, action normalization is used only in LAPO in our results. the table below, LAPO- shows LAPO without these techniques, although the network architecture is the same the original LAPO. "IQL (re- run)" indicates the results of re-running IQL in our code base, which fairly reproduced the results reported in the original IQL, paper. "V2AE" shows the result reported in main manuscript.
18

---

Under review as a conference paper at ICLR 2023
Table 11: Comparison with IQL on Antmaze tasks. Average normalized scores over the last 100 test episodes and five seeds are shown. The bold text indicates the best performance.
IQL (re-run)
lapo
V2AE
V2oE.lapo
infoV2AE
InfoV2AE.Iapo
umaze-v0
87.4:14.5
87.2.12.88
83.6 14.5
92.842.1
88.4113.6
89.4:1:.1
umaze-diverse-v0
64.6:15.6
54.0+11..4
43.2 7.8
32.64125.8
52.8:7.99
34.8:18..0
medium-play-v0
74.64:3.1
64.0.19.5
77.0+
63.0+:1.0
48.6+125.3
62.6:16.8
cediumm-diverse- vÖ
73.8=7.1
54.818.6
56.8+ 27.2
75.048.5
59.2:129.4
82.8+4.4
large-play-vo large-diven lagge-diverse-OO
39,0#7.2
20.4111.0
1.0
42.2+23.0
5,2+8.4
47.4
48.019.0
17.0
4.8 9.6
56.614.5
6.625.2
38.0
Table 12: Comparison with LAPO Antmaze tasks. Average normalized scores over the last 100 test episodes and five seeds are shown. The bold text indicates the best performance.
LAPO-
LAPO (re-run)
V2AR
V2AR lapo
infoV2AR
infoV2AE lapo
umazc-v0
76.4+3.3
97.8+1.3
83.6 +4.5
92.8+2.1
88.4+3.6
89.4+5.1
umaze-diverse-v)
56.6+
52.4+:0.55
43.2+ 7.8
32.6+25.8
52.8+7.9
34.8+18.0
medium-play-vO
0.8.1.66
64.2.1:..3
77.01 5.1
63.0113.0
48.6.125.3
62.6.6.8
mediun-diverse-v0
0.0.1..0
69.4112.8
56.811 27.2
75.018.5
59.2.129.4
82.8.1..4
large-play-v0
0.010.00
12,4116.9
1.0.L 1.3
42.2113.00
5,218.4
17.4
large-diverse-v0
0.0.1..0
18.2.11..1
4.8.L 9.6
56.614.55
6.6.5.2
38.0.14.8
we reported the results across five different seeds with 100 test episodes after million updates. Regarding LAPO, we could not reproduce the results reported in the paper, which may be due to the difference of the experiment procedure. In the original paper (Chen et al. 2022]. the methods are evaluated across three random seeds with 10 test episodes. When using the techniques used in LAPO, the overall performance of V2AE and info was better than that of LAPO. While LAPO nicely showed how to leverage the latent space for offline RL, LAPO does not utilize the discrete latent space. The difference in performance between LAPO and our methods shows the benefit of leveraging the discrete latent variable.
In Tables and 12
Regarding IQL, could fairly reproduce the performance reported in Kostrikov et al. 2022 In- terestingly, the techniques used in LAPO did not improve the performance of IQL. When using the techniques used in LAPO, the overall performance of V2AE and V2Ao was better compa- rable to IQL. This result shows that incorporating the policy structure can achieve the performance achieved by using the sate-of-the-art algorithm for the critic.
H EFFECT OF THE SCALING PARAMETER
we evaluated the performance of V2AE and info with a 5.0 and a 10.0. The results are shown in Table [3 Except hopper-medium- expert-v2, results with a = 10.0 are better than those with a 5.0. In the main manuscript, we report the result of a 5.0 for V2AE on hopper-medium--ee ex a 10.0 for the rest of the mujoco-v2 tasks.
To investigate the effect of the value of a in equation 15|
I PERFORMANCE OF CV2AE
We provide the performance of cV2AE, which variant of V2AE with the continuous latent variable, on mujoco-v2 tasks in Table 14 cV2AE outperformed AWAC in terms of the sum of the score across the tasks. However, overall performance is lower than V2AE and infoV2AE. In cV2AE, policy is more flexible and expressive than Gaussian policy, which resulted in performance better than AWAC. Meanwhile, the continuous latent space still interpolates the separate modes of the multimodal distribution in cV2AE, and it will result in generating the disti--distiibttion actions in offline RL. We think that is the reason why V2AE with the discrete latent variable outperforms e VVAA with the continuous latent variable.
19

---

Under review as a conference paper at ICLR 2023
Table 13: Results on mujoco tasks using D4RL-v2 datasets with different values of a in equation 15
Average normalized scores over the last 10 test episodes and five seeds are shown. HC = HalfChee- tah, HP = Hopper, WK = Walker2d.
V2AE
V2AE
info
info
Qr = 5.0
a = 10.0
0: 5.0
10.0
Med. R Med. -E Expert
HC
95
97.01 1,0
95.8
95.61 2.0
99.6.1..5
93.61:
105.9112.11
107.54 2.9
WK
111.3:1011
111.410.
106.1:11.00
112.14 0.4
HC
92,111.8
9,411
92.1
91.41
HP
74.6.19..3
78.41 19.0
94.511149
61.71 29.2
WK
109.310.55
109.94 0.4
109.110.5
110.14 0.7
HC
44,9+00.2
45.2+ 0.8
44.7=0.55
46.7+ 0.6
HP
83.5117.77
89.21 8.1
83.2.11.22
98.51 2.0
WK
78.618.9
3.8
77.843..5
86.7+
HC
46.4 0.4
0..5
46,520.3
48.6 0.4
Med.
HP
67.0+4..0
71.24 6.5
66.24:5.5
86.4- 7.6
WK
81.244299
79.4+ 4.7
81.644200
85.0 0.8
14.842.1
15.8 1.6
12.2111
16.3 1.2
Rand.
12.249.5
12.0+ 10.0
15.74:0.00
20.4 9.8
WK
1.9+2.2
2.5+ 2.6
2.3+1.8
2.3+ 2.0
Total
1013.3
1026.5
1033.7
1069.3
Table 14: Results mujoco tasks using D4RL-v2 datasets and AntMaze tasks. Average normalized scores over the last 10 test episodes and five seeds are shown. HC = HalfCheetah, HP = Hopper,
WK Walker2d.
AWAC
cV2AE
V2AE
info
Med. RR Med. B Expert
HC
94.8+ 0.2
95.8++0.8
97.0+ 1.0
95.6+ 2.0
HP
109.8 2.9
109.2::2.4
93.61 15.1
107.5: 2.9
WK
111.0
102.1
111.41
112.11 0.4
HC
92.7+ 0.8
92.4
91.1+
91.44 2.5
HP
98.6+ 10.7
59.8+-22.8
78.4+19.00
94.5+11.9
WK
109.2.1 0.3
73.4145.6
109.91
110.1 0.7
HC
40.9
44.3+0.3
4..2
46.74
HP
38.2+ 9.4
98.9+22.3
8.12+
98.5+ 2.0
WK
65.0.1 15.7
87.6.1..6
3.8
86.7.1 3.2
HC
44.34 0.2
47.2+:0.1
47.5 0.4
48.6 0.4
Med.
HP
57.54
75.0+5.8
71.24 6.5
86.4+ 7.6
WK
81.0 2.5
83.8.11.2
79.4 4.7
85.0.L 0.8
HC
3.2 1.3
20.6=0.6
15.8 1.6
16.3 1.2
Rand.
HP
7.3 0.9
16.4+8.9
12.0# 10.0
20.4 9.8
WK
3.1 1.0
-0.2+0.1
2.5 2.6
2.3+ 2.0
Total
956.5
1009.0
1026.5
1102.1
J THE CRITIC LOSS AND NORMALIZED SCORE IN MuJoCo TASKS
We show the critic loss and the normalized scores for MuJoCo tasks in Figures 6 The critic loss V2AE is comparable or lower than that of AWAC in MuJoCo tasks. However, there are exceptions, such as hopper-expert-v2 and hopper-medium-expee expeettv2, where the critic loss surges V2AE but not in AWAC. Thus, it is necessary to be aware that the use of the mixture policy may compound the accumulation of the critic error in some rare cases, while the use of the mixture policy often mitigates the accumulation the continuous latent variable, while cV2AE often achieves the lower values of the critic loss than AWAC, the critic loss of cV2AE occasionally explodes. possible explanation for these results may be that learning the continuous latent variable can provide flexibility to policy representation but separate modes of the objective function is still interpolated in the learned latent space, which results in generating distr--distributiin samples.
the critic error for many tasks. Regarding the use of
20

---

Under review as a conference paper at ICLR 2023
halfc www.eee.................
1,50
9.75
0.2
AMt-C c92AI
ARAC
9,50
AMSC
AWSC
CSCAE
VIAE
VO4E
9.00
0.25
3:75
0.53
1.30
0.5
0.5
1.00
Time
Time (ted]
walfcheetah-medimm-2
Polf walltheetaa-mmeddi---eeeaaaa
halft:/veettt eclum--nppertt0
na talKheetall-expittt
Cit:
0.25
0.50
0.25
1.00
0.25
0.50
0.75
3.35
0.50
1.00
0.50
0.75
Time Stops
linesaage (1:6)
Time staps (100)
Tinc Store (109)
Figure 6: Critic loss and normalized score during training on HalfCheetah tasks.
hopper-madlum-.
hoppar-madiun-rullay.
hopper-medie www.e---eddimmmmme 2idieeeeeeeert2
hopper-exowrt.v2
1.5
3.0
0.75
0.50
0.25
2.75
1.30
0.52
1.50
0.25
5:75
3.50
0.50
3.50
Tim Steps (199)
Time Steps 11e9)
Time Staps 11et
Time Stops l1es)
hopger 1uoger-meddum-22
hopper- hopper-medlum ro
1/tpper mmtpper-medium expert-v2
1e14
hopper-expert-22
CIR MYC
CIR MIS
CIP
0.2
0.4
0.6
1.0
0.25
0,50
0.75
1.00
0:23
0.50
0.75
1.00
0.2
0.4
0.6
0.5
3.0
Figure 7: Critic loss and normalized score during training on hopper tasks.
K ADDITIONAL RESULTS FOR THE ACTIVATION OF THE LATENT-CONDITIONED POLICY
Figure 9 shows the activation of sub-policies in V2AE on the pen-human-v0 task in three episodes. Here, used the same policy trained with 10,000 updates. As shown, the target orientation of the object is different in each episode, and different sub-policies are activated to achieve the given goals. The sub-policy corresponding to z (yellow) often activated during 20-40 steps. We
walkker2dmeddun-oooo splaay-v
walker d-wedium--eeeee.o
walter2dexxert.2
Nommattoe word unee 0.5
Narmarin score
0.5
Normatte
0.0
1
-05
-05
Time shepp (7ed)
Time eteps (1e6)
Time stepe (1+6)
me stops (1ef)
We liler 20-mmmmd
wellker2--mediuu--eeppy-
walkeer2d-xppettu22
100
30
0.25
053
5.75
3.00
0.25
8.50
0.75
1.00
0.25
8.5
0.75
1.03
0.25
0.50
0.75
1.55
Tim J005s (LO5)
Time stps (1u0)
Timestoss lice1
Time
Figure 8: Critic loss and normalized score during training on walker2d tasks.
21

---

Under review as a conference paper at ICLR 2023
think that the sub-policy corresponding to is effective for changing the orientation of the pen in this task. On the contrary, the sub-policies corresponding to 2 = 0, 1, 2 are not activated in many episodes. We think that these sub-policies correspond to the state-action pairs with low Q-values. In Offline RI., a given dataset often contains the mixed quality of the behavior, and is not effective reproduce all the behavior. Rather, the behaviors correspond the low scores should be ignored. We think that the proposed algorithm can implicitly achieve this by dividing the state-action space and investigate the quality of the sub-policies corresponding to each region. This qualitative result supports the claim that the different behaviors are encoded in each of the sub-policies.
visualization of the state-action pairs and the corresponding value of the latent variable on the kitchen- bichen-complete-ee task is provided in Figure [0] We also show the histogram of the activated sub-policies in Figure and the activation of sub-policies in a successful episode on the kitchen- complete-v0 task in Figure 12
L HYPERPARAMETERS AND IMPLEMENTATION DETAILS
Computational resource and license The experiments were run with Amazon Web Service and workstations with NVIDIA RTX 3090 GPU and Intel Core 19-10980XE CPU at 3.0GHz. We used the physics simulator MuloCo (Todorov et al. 2012) under the institutional license, and later we switched to the Apache license.
Software
The software versions used in the experiments are listed below:
Python 3.8
Pytorch 1.10.0
Gym 0.21.0
MuJoCo 2.1.0
mujoco-py 2.1.2.14
We used the author implementations for TD3+BC CQL, and EDAC For mujoco-v2 and Antmaze tasks, we used the updated CQL implementation For mujoco-vO, Kitchen and Adroit tasks, we used the original CQL implementation"" For a fair comparison with V2AE, we implemented easy- BCQ and AWAC ourselves. For rasyBCQ we used the hyperparameters used in the author- prooided implementation"" We implemented V2AE and AWAC based on the the author-providee implemen- tation of TD3. In our implementation of easyBCQ and AWAC, double-clipped Q-learning was employed. To minimize the difference between V2AE and AWAC, of the policy in both V2AE and AWAC. For simplicity, we did not use a regularization technique for the actor such the dropout layer used in (Kostrikov et al. 2022), although the of such techniques should further improve performance. In our implementation part of the input the actor network. Thus, the different behaviors corresponding the different values of = are represented by the same actor network.
also used delayed update
V2AE, the value of is
Computation of the advantage function In V2AE, policy is deterministic, both the gating policy π (=](s) and sub-policy π7 (a)s, z) are deterministic. Thus, the state-value function is given by
max (s,
(41)
Therefore, the advantage function is given by A² QT (s, VT (" a) max (T, (4(s, 2)).
(42)
In the update of the policy, used the target actor in the second term in equation 42 implementation, the advantage function is approximated with
Thus, in
((s, a; w. 0¹) (s,
(s, a; w) max Q(8 po (o 2); w)
(43)
2
httte:///gihabb..mmm/ffjjj///// BC
22

---

Under review as a conference paper at ICLR 2023
7 20
60
80
100
7.z4
2=55
2=3
2=3
20
80
100
7==3
2=4
/ 20
40
60
80
100
z=5
z=
z=6
z=5
Figure 9: Visualization of activation of sub- sub-policies in the pen- pen-human-vel task. The top row shows the state at the 20th, 40th, 60th, and 80th time steps; the graphs in the middle row show the action- values of each of the sub-policies each state.
Target smoothing in V2AE In V2AE, policy is given by mixture of deterministic sub-policies,
where a sub-policy is selected in a deterministic manner, as in equation 3 Thus, the mixture policy in our framework is deterministic. As reported by Fujimoto & Gul (2021), the use of a deterministic policy may lead to an overfitting the critic to the narrow peaks. Since our policy is deterministic, also employed technique called target policy smoothing used in TD3. Thus, the target value in
23

---

Under review as a conference paper at ICLR 2023
100
iataniverelle
(a) Latent variable sampled from lo [=|2,,00) a).
(b) Latent variable given by arg max Qu (8, (A z)).
Figure 11: Histogram of the activated latent variables on kitchen-complete-vO in 10 episodes.
Figure 10: Visualization of state-action pairs in the kitchen- complete-vO task D4RL. The color indicates the values of the latent variable.
cor
40
100'
160
180
2=4
2=2
z=3
Figure 12: Visualization of the activation of sub-policies in successful episode in the kitchen- complete-v0 task. The top row shows the state the 40th, 100th, 160th, and 180th time steps; the graphs in the middle row show the action-values of each of the sub-policies each state.
equation 14 is modified as
V/ = + max min Qui (s', (or 2²) ccui))
(44)
z'e2j jm1,2
where Elip is given by
Clpp = min(max(e, c) where c~ N(0,
(45)
and the constant defines the range of the noise.
regularization To implement the MI-based regularization, we intro- duced another network to represent Iyp (2 a) in addition to a network that represent the posterior distribution (z(2|s. a). When maximizing the objective LML. in equation both the actor no (s,s, and the auxiliary distribution Ow (2|s,, a) are updated, but the posterior distribution 4s (2(2s, a) is frozen. When maximizing ΣN Ez<p(z) log gw (x|s;, pees =)), both the actor (e 2) and the posterior distribution (2) a) are updated, but the auxiliary distribution Pu (2|ss, a) is frozen. For maximizing log ( pe (st, z)), the latent variable sampled from the prior distribution, i.e. the uniform distribution in this case, and maximization of log du (z|81, so (8; =)) is approximated as minimizing the squared difference || --=2pp 2 2|2 where & is the output of the network that represents Do (2|s, a).
Mutual-infie cattiaa--iooommtin-nmee
24

---

Under review as a conference paper at ICLR 2023
Qu(8.a
So(=|a,a
Hola a,
(a) Computation for maximizing Cxn. in equation 9
Po e)
(p(2)s,
(b) Computation for maximizing E Ex-n(() (zgow(2)ss, (s. 2)).
Figure 13: Connection between (0(2, a), (s 2), and Pop (2) a) during the training.
Implementation of a variant of V2AE with the continuous latent variable We refer to a vari- ant of V2AE with the continuous latent variable cV2AE in this paper. The differences from our main algorithm V2AE, which employs the discrete latent variable, are the reparametrization of sampling the latent variable, the KL divergence term in the loss function, and how to compute zx = arg max, Qw (s, ((s, z)). We assume that the prior distribution p(z) and posterior distribu- tion p(z]p( a) is Gaussian, and we used the reparameterization as in the standard VAE
M4 + E
(46)
where μ4 and σ. are the mean and the standard deviation of the posterior distribution p(z|s, a). As the latent variable is continuous, we cannot compute the softmax distribution in equation g given by
exp (Qw uo (s),)) z)))
p(2[8) 8)
(47)
Σzez exp (Qw (o 2)))
Thus, assumed that p(z|s) the Gaussian distribution in previous studies (Fujimoto et al. 2019), and the KL divergence term in the objective function in equation 7 is computed as in the standard VAE (Kingma & Welling 2014). When the latent variable continuous, we cannot explicitly compute z" = arg max, Qw (s),, Thus, we approximated it as follows: we gen- erate N samples 2, N (0 1) for O, .o..... N and determine 2' arg max Qw (s. plss, 2;))- In our implementation, the continuous latent variable z is two-dimensional, and N = 100. When the dimensionality of z is higher, the approximation of arg max, Qw, (1s z)) gets more coarse. We found that the performance is significantly lower when the dimensionality of z is more than two.
Number of updates In siichen-complete-vee pen-human-vo, hammer-human- door-human-v0, and Shocctt--human.. the number of samples contained in the dataset is significantly smaller than those for other datasets. While the datasets for MuJoCo tasks contained approximately 1 million samples, the numbers of samples in the Adroit-human tasks and the sttchen-complete-v0 were as follows: kiichen-complete--k: 3679 samples; pen-human-vO: 4950 samples; hammer-human-v0: 11264 samples; door-human-v0:: 6703 samples; and selocate-human-vee 9906 samples. Thus, in and relocate-human-
kitchen-complete--k) pen-human-vol vO, hammer-humman-v0 we updated the policy 10,000 times, while for the other tasks we updated the policy I million times. The Moove-mentioned number of policy updates was applied to all methods except easyBCQ which a one-step offline RL algorithm. For easyBCQ, the behavior policy was trained 500,000 times and the crities were trained 2 million times, except for h-uman--0 tasks in Adroit tasks, where the behavior policy was trained 50,000 times, and the critics were trained 200,000 times.
Hyperparameters
Tables 15 19 provide the hyperparameters used in the experiments.
25

---

Under review as a conference paper at ICLR 2023
Table 15: Hyperparameters of V2AE&infoV2AE.
Hyperparameter
Value
Optimizer
Adam
Critic learning rate
3c-4
Actor learning rate
3e-4
Hyperparameters
Posterior learning rate
3e-4
Mini-batch
256
Discount factor
0.99
Target update rate
5e-3
Policy noise
0.2
Policy noise clipping
(-0.5, 0.5)
Policy update frequency
Critic hidden dim
256
Critic hidden layers
Critic activation function
ReI.U
Actor hidden dim
256
Architecture
Actor hidden layers
Actor activation function
ReLU
Posterior hidden dim
256
Posterior hidden layers
Postcrior activation function
ReLU
V2AE
Score scaling
5.0 (kitchen, human, Antmaze)
10.0 (mujoco-v2. mujoco-vl)
learning rate of the posterior for infomax
3e-6 (Kitchen, Adroit)
Se-7 (mujoco-v2)
infoV2AE
5e-7 (Antmaze)
Score scaling
5.0 (Antmaze, hoopeer-mediiumeespee)))
10.0 (others)
Table 16: Hyperparameters of AWAC.
Hyperparameter
Value
Optimizer
Adam
Critic learning rate
3e-4
Actor learning rate
3e-4
Mini-bater size
1024
Hyperparameters
Discount factor
0.99
Target update rate
5c-3
Policy update frequency
2
Score scaling
10.0
Critic hidden dim
256
Critic hidden layers
2
Critic activation function
ReLU
Actor hidden
256
Architecture
Actor hidden layers
2
Actor activation function
ReLU
26

---

Under review as a conference paper at ICLR 2023
Table 17: Hyperparameters of TD3+BC. The default hyperparameters in the TD3+BC GitHub are used.
Hyperparameter
Value
Optimizer
Adam
Critic learning rate
3e-4
Actor learning rate
3e-4
Mini-hatch size
256
TD3 Hyperparameters
Discount factor
0.99
Target update
5c-3
Policy noise
Policy noise clipping
(-0.5, 0.5)
Policy update frequency
Critic hidden dim
256
Critic hidden layers
Critic activation function
ReLU
Architecture
Actor hidden dim
256
Actor hidden layers
Actor activation function
ReLU
TD3+BC Hyperparameterss
a
2.5
Table 18: Hyperparameters CQL. The default hyperparameters the CQL GitHub are used.
Hyperparameter Optimizer
Value Adam
Critic learning rate
3e-4
Actor learning
3e-5
Mini-batch size
256
SAC Hyperparameterrs
Discount factor
0.99
Target update rate
5e-3
Target entropy Entropy in Q target
-1-Action Dim
False
Critic hidden dim
256
Critic hidden layers
3
Critic activation function
ReLU
Architecture
Actor hidden dim
256
Actor hidden layers
3
Actor activation function
ReLU
Lagrange
False
10
CQL Hyperparameters
Pre-training steps
40e3
Num sampled actions (during eval)
10
Num sampled actions (logsumexp)
10
Table 19: Hyperparameters easyBCQ. The default hyperparameters in the easyBCQ the author- provided codes in GitHub used.
Hyperparameter
Value
Optimizer
Adam
Critic learning rate
le-4
Behavior policy learning rate
le-4
Mini-batch size
512
casyBCQ Hyperparameters
Discount factor
0.99
Target update rate
5c-3
number of action samples
100
Critic hidden dim
1024
Critic hidden layers
Critic activation function
ReLU
Architecture
Behavior policy hidden dim
1024
Behavior policy hidden layers
2
Behavior policy activation function
ReLU
27


Reviews:
[{"review_id": "rejtuqcbPdQ", "reviewer": "Reviewer_7tms", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel algorithm for offline RL that works by dividing the state-action space into discrete latent regions, learning a deterministic policy for each, and acting with a mixture of these policies. The authors demonstrate the effectiveness of their method on a series of experiments from the D4RL benchmarking datsets and argue that it helps avoid issues with out-of-distribution actions.", "summary_of_the_review": "This paper presents an interesting and novel algorithm for dealing with out-of-distribution actions. The writing, motivation, and algorithmic description are clear.\n\nMy main concern with this paper is in the evaulation of the method, sensitivity analysis of one of its core components, and comparison to baselines (see above for details).\n\nFor these reasons I am leaning towards an accept.", "strength_and_weaknesses": "# Strengths\nThe paper is well-written and well motivated, and the algorithm is novel (as far as I can tell).\n\nI also wanted to highlight the comparison between the original results of AWAC and the authors in Appendix E (although more to say about this point below).\n\n# Weaknesses\nThe main weaknesses of this paper are in evaluation.\n\n## Choice of $|\\mathcal{Z}|$\nThe first point is in the choice of the cardinality of $|\\mathcal{Z}|$, which is crucial to the algorithm. I feel that the importance of this choice is not highlighted enough (the first time I saw it discussed was in section 7.2 in page 7). In Algorithm 1, for instance, it is never mentioned that $|\\mathcal{Z}|$ needs to be selected.\n\nI appreciate that the effect of the dimensionality choice was evaluated to some extent in section 7.2, but I think this is an analysis that needs to be performed more extensively, as it is one of the core components of the proposed method. Some questions that were left unanswered for me:\n* How does one pick $|\\mathcal{Z}|$?\n* How does one know if it'd be better to pick cV2AE instead (where $|\\mathcal{Z}|$ doesn't have to be picked)?\n* How sensitive is the algorithm to $|\\mathcal{Z}|$ in general?\n\n## Implementation details\nIn section 7.2 the authors say \"For TD3+BC and CQL, we used the author-provided implementations.\" This makes for a comparison I do not trust. Implementation matters and there are often subtle design choices that have a big impact on performance. For proper \"apples-to-apples\" comparison, you should really be using the same codebase, otherwise the results are suspect. Furthermore, it seems you used your own implementation of TD3+BC for the toy experiments... Why did you not use the same one for the D4RL benchmark tasks?\n\nIndeed, right below table 4 the authors say: \"Remarkably, our implementation of AWAC showed significantly better performance than that reported by Nair et al. (2020).\" This is precisely why it's not good to compare across implementations.\n\n## Statistical significance\nIn the tables presented there are bolded numbers even when improvements are not statistically significant: you should only be bolding numbers when there are no overlapping CIs, which is not the case for most of e.g. Table 3 and Table 5.\n\n\n# Questions / suggestions\n1. In equation (2), it seems $\\mathcal{Z}$ has to have a pre-specified size. How is that chosen?\n1. In equation (5), to the right of the inequality I think it should be $d^{\\beta}(\\mathbf{s})$\n1. In equations (7) and (10) the $\\mathbf{z}$ is not qualified. Are you missing a sum or an integral over $\\mathcal{Z}$?\n1. At the end of equation (10) shouldn't it be $\\mathbf{s_i,a_i}$?\n1. In Algorithm 1 you should have $\\lbrace\\ldots \\rbrace$ surrounding $(s, a, s', r)$ to indicate its a set of transitions. You should also have a for loop or something like that saying \"For each element $(s, a, s', r)$ in the mini-batch...\n1. In the second expectation in equation (12), shouldn't it be $a\\sim\\pi_{z'}$?\n1. In the paragraph above equation (10) it should say \"In addition, we also propose a regularization\" (not an)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_g_6wAyUtW", "reviewer": "Reviewer_NNyS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a more expressive policy for offline reinforcement learning. In particular, the paper proposes to use a mixture of deterministic policies.", "summary_of_the_review": "The paper lacks novelty and omits comparisons to the current SOTA. For these reasons, I believe that they paper falls below the acceptance threshold.", "strength_and_weaknesses": "# Strength\n* More expressive policies are an interesting direction of offline reinforcement learning research.\n\n# Weaknesses\n* The approach lacks novelty. A VAE-style policy learning scheme has already been explored in [1]. While [2] proposes more expressive autoregressive policies. \n* The paper lacks comparisons to the current SOTA results. In particular, [1] and [3] achieve significantly stronger results on the antmaze dataset, which are considered to be more challenging for offline reinforcement learning [4].\n* The advantage scaling term (15) relies on the minimum and maximum over minibatches that might be sensitive to outliers.\n\n[1] Latent-Variable Advantage-Weighted Policy Optimization for Offline RL\nXi Chen, Ali Ghadirzadeh, Tianhe Yu, Yuan Gao, Jianhao Wang, Wenzhe Li, Bin Liang, Chelsea Finn, Chongjie Zhang\n\n[2] EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL\nSeyed Kamyar Seyed Ghasemipour, Dale Schuurmans, Shixiang Shane Gu\n\n[3] Offline Reinforcement Learning with Implicit Q-Learning\nIlya Kostrikov, Ashvin Nair, Sergey Levine\n\n[4] RvS: What is Essential for Offline RL via Supervised Learning?\nScott Emmons, Benjamin Eysenbach, Ilya Kostrikov, Sergey Levine", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BviCT76gDo", "reviewer": "Reviewer_RQRe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies an offline RL algorithm by utilizing the special policy structure of the behavior policy (which generates the offline data). The paper proposes to define the policy being learned by using a mixture of sub-policies, which is in the form of pi(s, a) = sum_z p(z|s) pi_sub (a | s, z), where z is the introduced latent variables. The sub policy pi_sub is deterministic and can be written as mu(s, z).  The basic idea behind this form is that the mixture policy is less likely to force the policy to cover all modes in the offline data; covering all modes using a single-mode policy is potentially harmful as it may provide support to many OOD actions, which results in overestimation. The paper then proposes to optimize the policy pi by maximizing the variational lower bound. Experiments are presented on commonly seen testing domains in offline RL literature. ", "summary_of_the_review": "Due to the concerns about the motivation, insight of the proposed method, and empirical results, I tend to reject the paper for now. ", "strength_and_weaknesses": "Strength: \n\n1. It seems novel to utilize policy structure to improve offline RL algorithms. \n\n2. The paper presents its idea reasonably clearly. \n\nWeaknesses. \n\nI will keep my comments short. I have several critical concerns below. \n\nMotivation. The motivation for using mixture distribution is not persuasive. The paper currently says the offline data may be multimodal, and a unimodal distribution policy can result in OOD actions, as shown in Fig 1. But the mixture distribution could also assign higher density to somewhere between two modes (i.e., Fig 1(c), somewhere between two regions could also have high density). I don’t see a clear benefit of using a mixture distribution.  \n\nLack of insight into the proposed approach. The proposed method has two key components: one is the use of a mixture distribution policy; another one is to use variational lower bound to learn the policy parameters. I expect strong empirical results to show the necessity and effectiveness of the two components. Particularly to show at least the following: \n\n1. The benefit of the proposed algorithm is not due to a more complicated NN. There is a reason to believe that a very large/complicated NN could be beneficial: it is less likely to touch another action’s density when updating one. So probably, the OOD action’s density can be kept low in the learning process. Why not use a mixture of Gaussian as a policy parameterization (with a relatively large NN) and plug it into an offline RL algorithm as a baseline? \n\n2. Why use variational lower bound?  \n\nTechnical clarification. Eq (3), why such a definition of z? Does it mean pi_gate is also deterministic? \n\nEmpirical results: \nWhy is there no implicit Q learning? IQL empirical works really well and should be included. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "81S4hhQh-Kb", "reviewer": "Reviewer_chjv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work aims to draw attention to the issue that there may be multiple “modalities” to the behavior data collected for offline RL and it is hypothesized that it is harmful to force a single policy to learn from the data from all those different behavior policies. \n\nThe paper proposed to use V2AE (a VAE inspired method) to separate out the different modalities in the behavior data and use one policy for each of those modalities. They propose a gating policy that selects the latent modality z with the maximum Q value, which in practice is approximated with a softmax distribution inthe variational lower bound. In addition, the authors found that by adding a MI regularization term that encourages diversity of the latent modality per (s,a) pair, they got improved performance. \n\nIn terms of the experiment results, the authors found that their proposal does well on a toy problem where there are two potential goals that the agents can head to. Existing methods exhibits hesitation around forks in the path towards those two goals, whereas the V2AE method solves it perfectly. More encouragingly, such gain is still exists for AntMaze, where the problem setup can be seen as a more complicated version of the toy problem. \n", "summary_of_the_review": "I recommend a score of 6 because this paper pointed out a unique perspective to the failure modes of offline RL methods and proposed a preliminary approach that combines various other areas of ML to solve it. It is valuable due to the novel perspective and to the sound motivation as demonstrated in the toy problem. Whether there are easier solutions to the current approach is still yet to be seen, and the clarity can be improved. That is why I did not give a higher score. ", "strength_and_weaknesses": "\nThe paper pointed out an often ignored factor of offline RL, which is that the dataset may come from multiple behavior policies and forcing one policy to adapt to all of them at once may make learning harder. The proposed solution is mathematically grounded and the motivation is clearly shown in the toy problem. The authors also illustrated the connection between policy structure and the critic loss. All of those points above make this paper a strong candidate. \n\nThere are also some weaknesses of the paper. One is that it is unclear what exactly caused fitting the multimodal distribution hard and whether it is a fundamental problem or something that will automatically go away with more advanced network architecture with stronger regularization, without explicitly modeling the latent z. Another area that can be improved is the clarity. (see the next section)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject