Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling
Authors: Huayu Chen
Cheng Lu
Chengyang Ying
Hang Su
Jun Zhu
Venue: ICLR 2023 poster (2023)
Keywords: offline reinforcement learning
generative models
diffusion models
behavior modeling

Abstract: In offline reinforcement learning, weighted regression is a common method to ensure the learned policy stays close to the behavior policy and to prevent selecting out-of-sample actions. In this work, we show that due to the limited distributional expressivity of policy models, previous methods might still select unseen actions during training, which deviates from their initial motivation. To address this problem, we adopt a generative approach by decoupling the learned policy into two parts: an expressive generative behavior model and an action evaluation model. The key insight is that such decoupling avoids learning an explicitly parameterized policy model with a closed-form expression. Directly learning the behavior policy allows us to leverage existing advances in generative modeling, such as diffusion-based methods, to model diverse behaviors. As for action evaluation, we combine our method with an in-sample planning technique to further avoid selecting out-of-sample actions and increase computational efficiency. Experimental results on D4RL datasets show that our proposed method achieves competitive or superior performance compared with state-of-the-art offline RL methods, especially in complex tasks such as AntMaze. We also empirically demonstrate that our method can successfully learn from a heterogeneous dataset containing multiple distinctive but similarly successful strategies, whereas previous unimodal policies fail.

Published as a conference paper at ICLR 2023
OFFLINE REINFORCEMENT LEARNING VIA HIGH- FIDELITY GENERATIVE BEHAVIOR MODELING
Huayu Chen', Cheng Lu', Chengyang Ying', Hang Su : Jun Zhu 1:2+ 'Department of Computer Science & Technology, Institute for AI. BNRist Center, Sanghua-Boschh Joint ML Center, THBI Lab, Tsinghua University Paazhou Lab, Guangzhou, 510330, China
(lucheng. wiaggeyyyynny11//mm/l....
(suhangss,
ABSTRACT
In offine reinforcement learning, weighted regression is common method ensure the learned policy stays close to the behavior policy and to prevent selecting out-of-sample actions. In this work, we show that due to the limited distributional expressivity of policy models, previous methods might still select unseen actions during training, which deviates from their initial motivation. To address this problem, we adopt generative approach by decoupling the learned policy into two parts: an expressive generative behavior model and an action evaluation model. The key insight is that such decoupling avoids learning an explicitly parameterized policy model with a closed-form expression. Directly learning the behavior policy allows to leverage existing advances in generative modeling, such diffusion- based methods, to model diverse behaviors. As for action evaluation, we combine method with an in-sample planning technique further avoid selecting out- of-sample actions and increase computational efficiency. Experimental results on D4RL datasets show that our proposed method achieves competitive or superior performance compared with state-of-the-ar offline RL methods, especially in complex tasks such AntMaze. We also empirically demonstrate that our method can successfully learn from a heterogeneous dataset containing multiple distinctive but similarly successful strategies, whereas previous unimodal policies fail. The wtwp:h/wwccoo//cc///////S//S
source code is provided at
INTRODUCTION
Offline reinforcement learning seeks to solve decision-maker makinn problems without interacting with the environment. This is compelling because online data collection can be dangerous expensive in many realistic tasks. However, relying entirely on static dataset imposes new challenges. One is that policy evaluation is hard because the mismatch between the behavior and the learned policy usually introduces extrapolation error (Fujimoto et al. 2019). In most offline tasks, it is difficult or even impossible for the collected transitions to cover the whole state-action space. When evaluating the current policy via dynamic programming, leveraging actions that are not presented in the dataset (out-of-sample) may lead to highly unreliable results, and thus performance degrade. Consequently, in offline RL it is critical to stay close to the behavior policy during training.
Recent advances in model-free offline methods mainly include lines of work. The first is the adaptation of existing off off-po algorithms. These methods usually include value pessimism about unseen actions or regulations of feasible action space (Fujimoto al. 2019] Kumar et al. 2019 2020). The other line of work (Peng et al. 2019: Wang et al.) 2020, Nair et al. (0200 is derived from constrained policy search and mainly trains parameterized policy via weighted regression. Evaluations of every state-action pair in the dataset are used as regression weights.
The main motivation behind weighted policy regression is that it helps prevent querying sample actions (Narr et al. 2020 Kostrikov et al. 2022). However, we find that this argument is untenable in
'H. Su and J. Zhu corresponding authors.

---

Published as a conference paper at ICLR 2023
certain settings. Our key observation is that policy models in existing weighted policy regression methods are usually unimodal Gaussian models and thus lack distributional expressivity, while the real world collected behaviors can be highly diverse. This distributional discrepancy might eventually lead to selecting unseen actions. For instance, given a bimodal target distribution, fitting with a unimodal distribution unavoidably results in covering the low-density area between two peaks. In we empirically show that lack of policy expressivity may lead to performance degrade.
Section 3.1
Ideally, this problem could be solved by switching to a more expressive distribution class. However, it is nontrivial in practice since weighted regression requires exact and derivable density calculation, which places restrictions on distribution classes that we can choose from. Especially, we may not know what the behavior or optimal policy looks like in advance.
To overcome the limited expressivity problem, we propose to decouple the learned policy into two parts: an expressive generative behavior model and an action evaluation model. Such decoupling avoids explicitly learning a policy model whose target distribution is difficult to sample from, whereas learning behavior model is much easier because sampling from the behavior policy is straightforward given the offline dataset collected by itself. Access to data samples from the target distribution is critical because it allows us to leverage existing advances in generative methods to model diverse behaviors. To sample from the learned policy, we use importance sampling to select actions from candidates proposed by the behavior model with the importance weights computed by the action evaluation model, which we refer to as Selecting from Behavior Candidates (SFBC).
approach introduces new challenges because it requires modeling behaviors with high fidelity, which directly determines the feasible action space. A prior work (Ghasemipour al. 2021) finds that typically-used VAEs do not align well with the behavior dataset, and that introducing building-in good inductive biases in the behavior model improves the algorithm performance. Instead, we propose to learn from diverse behaviors using a much more expressive generative modeling method, namely diffusion probabilistic models (Ho) al. (220),, which have recently achieved great success in modeling diverse image distributions, outperforming other existing generative models (Dhariwal & Nichol 2021). We also propose planning-band based operator for Q-learning. which performs implicit planning strictly within dataset trajectories based on the current policy, and is provably convergent. The planning scheme greatly reduces bootstrapping steps required for dynamic programming and thus can help to further reduce extrapolation error and increase computational efficiency.
However, the selecting- delecting-from-eeel
The main contributions of this paper are threefold: 1. We address the problem of limited policy expressivity in conventional methods by decoupling policy learning into behavior learning and action evaluation, which allows the policy to inherit distributional expressivity from a diffusion- -based behavior model. 2. The learned policy is further combined with an implicit in-sample planning technique to suppress extrapolation error and assist dynamic programming over long horizons. 3. Extensive experiments demonstrate that our method achieves competitive or superior performance compared with state-of-the-art offline RL methods, especially in sparse-reward tasks such as AntMaze.
BACKGROUND
2.1 CONSTRAINED POLICY SEARCH OFFLINE RL
Consider Markov Decision Process (MDP), described by a tuple (S, A, P,r,7). S denotes the state space and AA is the action space. P(s'|s, a) and r(s, a) respectively represent the transition and re- ward functions, and 7 E (0, 1] is the discount factor. Our goal is to maximize the expected discounted
return ( Eg~p- (a) (- [r(s, a)] of policy ग, where (s))) ==)== > > Px (8n 8) is the
>>>
discounted state visitation frequencies induced by the policy π (Sutton & Barto 1998).
According to the policy gradient theorem (Sutton et al. 1999), given parameterized policy me. and the policy's state-action function Q"T, the gradient of / (T)) can be derived as:
VoJ(TO) = 1 p p=((s) / Vomo(a)s) x) (s, a)da ds.
When online data collection from policy т is not possible, is difficult to estimate Pa (s) )) in Equation] I and thus the expected value of the Q-function TT( == Is Pr (x() SA ma(a)s)("
("ss))"((ss (s, a). Given static
2

---

Published as a conference paper at ICLR 2023
dataset DD consisting of multiple trajectories (S.,, Cn. collected by behavior policy ((aas),
previous off policy methods (Silver et al. 2014 Lillicrap et al. 2016) estimate nT with a surrogate objective ((())) by replacing P=(8) with (s( In offline settings, due to the importance of sticking with the behavior policy, prior works (Peng et al. 2019 Nair et al., 2020) explicitly constrain the
learned policy be similar μr, while maximizing the expected value of the Q-functions:
J (pu(s))
T m(als) a) da ds _ Or (u(()DD (T(-|8)]lp((1s)) ds.
arg maxx
(2)
The first term in Equation 2 corresponds to the surrogate objective (T)) where Q (8. a) is a learned Q-function of the current policy The second term is regularization term to constrain the learned policy within support of the dataset DR with a being the coefficient.
2.2 POLICY IMPROVEMENT VIA WEIGHTED REGRESSION
The optimal policy π" for Equation 2020 by use of Lagrange multiplier:
2 can be derived (Peters et 2010 Peng et al. 2019 Nair et
1 p(aas) exp (aQa(s,a))
(a|s)
(())
where (()) is the partition function. Equation forms policy improvement step.
Directly sampling from n² requires explicitly modeling behavior μa, which itself is challenging in
continuous action-space domains since μ can be very diverse. Prior methods (Peng et al. 2019 Wang
ct al. 2020 Chen et al. 2020) bypass this issue by projecting I onto a parameterized policy ne:
arg min
Eg-DA [DKL (Tr (-Is))[mma
1
arg max 0
((
log (a)s)) exp (aQa(s,
(())
Such method is usually referred weighted regression, with exp (aQa(s, a)) being the regres- sion weights.
Although weighted regression avoids the need model the behavior policy explicitly, it requires calculating the exact density function (aass)) as in Equation 4 This constrains the policy zo to distribution classes that have tractable expression for the density function. We find this in practice limits the model expressivity and could be suboptimal in some cases (Section 3.1
2.3 DIFFUSION PROBABILISTIC MODEL
al., 2020: Song et al. 2021b) are generative models by first defining forward process to gradually add noise to an unknown data distribution po( and then learning to reverse it. The forward process ( is defined by a stochastic differential equation (SDE) dæ, = f t) + g(t)dur, where w, standard Brownian motion and ft),, g(t) are hand-crafted functions (Song et al. (021bb such that the transition distribution Pro (x,) pro(#,,) (oo) = N(x, ja o for some au, ar > and pT(T)) N(xr10,, I). To reverse the forward process, diffusion models define scored-basee based model 88 and optimize the parameter 0 by:
Diffusion models (Sohl-Dickstein et al. 2015
arg min
Ex, [lla,so(x1,,) + cl/]]
(5)
where ~ L(O, xo po(2)). e N(O, I). x. = a,20 + OtE.
Sampling by diffusion models can be alternatively viewed as discretizing the diffusion ODEs Song
ct al. 20211), which are generally faster than discretizing the diffusion SDEs (Song et al. 202 la Lu et al. 2022). Specifically, the sampling procedure needs to first sample pure Gaussian øy (O, and then solve the following ODE from time T to time 0 by numerical ODE solvers:
dæ;- = [( t) - 9 (1)) 1)
1
(6)
Then the final solution Øp time 0 is the sample from the diffusion models.
3

---

Published as a conference paper at ICLR 2023
3 METHOD
We propose a for----oooo-e-oo---eeeeee ity problem in offline RL. Below we first motivate method by highlighting the importance of distributionally expressive policy in learning from diverse behaviors. Then we derive high-level solution to this problem from generative modeling perspective.
(SfBC) approach to address the limited expressiv-
3.1
LEARNING FROM DIVERSE BEHAVIORS
In this section, we show that the weighted regression broadly used in previous works might limit the distributional expressivity of the policy and lead to performance degrade. As described in Section 2.2 conventional policy regression methods project the optimal policy л" in Equation 3 onto a parameterized policy set. In continuous action-space domains, the projected policy usually limited to narrow range of unimodal distributions (e.g., squashed Gaussian), whereas the behavior policy could be highly diverse (e.g.. multimodal). Lack of expressivity directly prevents the RL agent from exactly mimicking a diverse behavior policy. This could eventually lead to sampling undesirable sut-of-sample actions during policy evaluation and thus large extrapolation error. Even if Q-values can be accurately estimated, an inappropriate unimodal assumption about the optimal policy might still prevent extracting a policy that has multiple similarly rewarding but distinctive strategies.
We design a simple task named Bidirec- tional Car to better explain this point. Con- sider environment where a car placed in the middle of two endpoints can go ei- ther side to gain the final reward. If RI. agent finds turning left and right simi- larly rewarding, by incorrectly assuming unimodal distribution the behavior pol- icy, ends up staying put instead of taking either one of the optimal actions (Figure Ip. As result, unimodal policies fail to completely solve this task or loss diversity whereas a more distributionally expressive policy easily succeeds.
108
(1:))
Success (right)
polley Unimodal policy smc palley (wars)
Socoes rate
52.0
167
5B5C BAIL COL
DT
IQL AWR
Figure 1: Illustration of the Bidirectional-CC task and comparison between SIBC and unimodal policies. See Section 6.2 for experimental details.
We therefore deduce that distributional expressivity a necessity to enable diverse behavior learning. To better model the complex behavior policy, need more powerful generative modeling for the policy distribution, instead of the simple and unimodal Gaussians.
3.2 SELECTING FROM BEHAVIOR CANDIDATES
In this section, provide generative view of how to model potentially diverse policy. Specifically, in order to model π with powerful generative models, essentially we need to perform maximum likelihood estimation for the model policy ne. which is equivalent minimizing KL divergence between the optimal and model policy:
arg max Ba (-1s) [log (a]s)) < arg min
EANDE [DEL (n* (-1s)][=(((1ss)
(7)
8
#
However, drawing samples directly from x² is difficult, so previous methods (Peng et al. 2019 Nair
al. 2020. Wang et al. 2020) rely on the weighted regression as described in Equation
The main reason that limits the expressivity of Ty is the need of calculating exact and derivable density function Te (a)s ss) in policy regression, which places restrictions on distribution classes that we can choose from. Also, we might not know what the behavior or optimal policy looks like previously.
Our solution based on a key observation that directly parameterizing the policy æ is not necessary. To better model diverse policy, we propose to decouple the learning of into two parts. Specifically, we leverage Equation to form a policy improvement step:
(8) One insight of the equation above is that minimizing KL divergence between μ and μá is much easier compared with directly learning no because sampling from μ is straightforward given DP. This
Talss) (a(a)s) exp (aQs(s, a)).
4

---

Published as a conference paper at ICLR 2023
allows to us to leverage most existing advances in generative modeling (Section 4.1 Qo(s,a) could be learned using the existing Q-learning framework (Section 4.2)
The inverse temperature parameter in Equation g serves trade-off between conservative and greedy improvement. We can see that when a + 0, the learned policy falls back to the behavior policy, and when a +oo the learned policy becomes greedy policy.
To sample actions from æ, we use an importance sampling technique. Specifically, for any state s, first draw M action samples from a learned behavior policy (-(((s) candidates. Then we evaluate these action candidates with a learned critic Qo. Finally, an action is resampled from M candidates with exp (aQa(* 8, a)) being the sampling weights. We summarize this procedure selecting from behavior candidates (SIBC), which could be understood as an analogue to rejection sampling.
Although generative modeling of the behavior policy has been explored by several works (Fujimoto iet al. 2019: Kumar et al. 2019), it was mostly used to form an explicit distributional constraint for the policy model πo. In contrast, we show directly leveraging the learned behavior model to generate actions is not only feasible but beneficial on the premise that achieved. We give practical implementation in the next section.
behavior modeling can be
PRACTICAL IMPLEMENTATION
In this section, we derive a practical implementation SfBC, which includes diffusion- brasssdio---aedd behavior modeling and planning-based Q-learning. An algorithm overview is given in Appendix A
4.1 DIFFUSION- CIFFUSIO-BBASD BEHAVIOR MODELING
is critical that the learned behavior model is high fidelity because generating any out-of-sample actions would result in unwanted extrapolation error, while failing cover all in-sample actions would restrict feasible action space for the policy. This requirement brings severe challenges to existing behavior modeling methods, which mainly include using Gaussians or VAEs. Gaussian models suffer from limited expressivity as we have discussed in Section 3.1 VAEs, on the other hand, need introduce variational posterior distribution to optimize the model distribution, which has
trade-off' between the expressivity and the tractability (Kingma et al., 2016 Lucas et al. 2019). This still limits the expressivity of the model distribution. An empirical study given in Section 6.3
To address this problem, we propose to learn from diverse behaviors using diffusion models (Ho et al. 2020), which have recently achieved great success in modeling diverse image distributions (Ramesh et al. 2022 Saharia et al. 2022). outperforming other generative models (Dhariwal & Nichol 2021].
Specifically, we follow Song al. (2021b) and learn a state-conditioned diffusion model so to predict the time-dependent noise added to the action a sampled from the behavior policy (u(|ss):
0 = arg min
E (ss.a)-Dr... ((
+ o,e, s, t) + =112],
(9)
where E ~ No I). t ~ u(0, T). a and o are determined by the forward diffusion process. Intuitively se is trained to denoise a, a,a a,E into the unperturbed action such that ar (( I) can be transformed into a ~ (-((ss) by solving an inverse ODE defined by sø (Equation 6).
4.2 Q-LEARNING VIA SAMPLE PLANNING
Generally, Q-learning can be achieved via the Bellman expectation operator:
a) = r(s, a) + = Q(sl))))(ssss.. .a).
(10)
However, T is based on one-step bootstrapping, which has two drawbacks: First, this can be computationally inefficient due to its dependence on many steps of extrapolation. This drawback exacerbated in diffusion settings since drawing actions from policy in Equation [0 is also time-consuming because of many iterations of Langevin-type sampling. Second, estimation errors may accumulate over long horizons. To address these problems, we take inspiration from episodic learning methods (Blundell et al. 2016 Ma et al.
2022) and propose planning-based operator T
T ((, == max (TH)*T"Q(s,0) a)}
(11)
n20
5

---

Published as a conference paper at ICLR 2023
E
Q-targets (k=B)
Q-targets (K=1)
Q-targets (K-2)
Q-targets (k::3)
Figure 2: Visualizations of the implicitly planned Q-targets R sampled from the dataset of an AntMaze task four consecutive value iterations. The red pentagram stands for the reward signal. Implicit planning helps to iteratively stitch together successful subtrajectories.
where j is the behavior policy. 7 combines the strengths of both the n-step operator (7*)", which enjoys fast contraction property, and the operator T which has more desirable fixed point. We prove in Appendix C that Thr is also convergent, and its fixed point is bounded between Q" and Q"
Practically, given a dataset D = {(3n: Ony Tn)) collected by behavior μ4, with n being the timestep in trajectory. We can rewrite Equation 11 in recursive manner to calculate the Q-learning targets:
((k) = rn + max(R (( v(lk-1),
(12)
where
V (k-1) = (. Qp(sn: a),
(13)
and Φ = arg min Be [10 an) 111
(14)
Above & E f1.2.... is the iteration number. We define R as the vanilla return of trajectories. Equation offers an implicit planning scheme within dataset trajectories that mainly helps avoid bootstrapping over unseen actions and to accelerate convergence. Equation] 13 enables the generaliza- tion of actions in similar states across different trajectories (stitching together subtrajectories). Note that we have omitted writing the iteration superscript of π and μ for simplicity. During training, alternate between calculating new Q-targets Rn and fitting the action evaluation model Qo-
Although the operator T is inspired by the multi-step estimation operator Trem proposed by Ma et al. 2022). They have notable differences in theoretical properties. First, Tvem can only apply to deterministic environments, while method also applies to stochastic settings. Second, unlike Tvem. T does not share the same fixed point with T We compare two methods in detail in Appendix G
5 RELATED WORK
Reducing extrapolation error in offfine RL. Offline RL typically requires careful trade-offs between maximizing expected returns and staying close the behavior policy. Once the learned policy deviates from the behavior policy, extrapolation error will be introduced in dynamic programming, leading to performance degrade (Fujimoto al. 2019). Several works propose address this issue by introducing either policy regularization on the distributional discrepancy with the behavior policy (Fujimoto et al. 2019 Kumar al. 2019 Wu et al. 2019 Fujimoto & Gu 202T), value pessimism about unseen actions (Kumar et al. 2020; Kostrikov et al. [021]]. Another line of research directly extracts policy from the dataset through weighted regression, hoping avoid selecting unseen actions (Peng et al. 2019; Nair et al.) 2020 Wang et al. (020). However, some recent works observe that trade-off techniques described above are not sufficient to reduce extrapolation error, and propose to learn Q-functions through expectile regression without ever querying policy-generated actions (Kostrikov et al., 20222 Ma et al. (2022). Unlike them, We find that limited policy expressivity the main reason that introduces extrapolation error in previous weighted regression methods, and use expressive policy model to help reduce extrapolation error.
Dynamic programming over long horizons. Simply extracting policies from behavior Q-functions can yield good performance in many D4RL tasks because it avoids dynamic programming and therefore the accompanied extrapolation error (Peng al. 2019 Chen et al. 2020 Brandfonbrener

---

Published as a conference paper at ICLR 2023
Dataset
Environment
SIBC (Ours)
IQL
VEM
AWR
BAIL
BCQ
CQL
DT
Ditfuser
Mediam- Expert
HallCheetah
92.6 1: 0.5
86.7
32.7
72.2
64.7
62.4
86.8
79.8
Medium- Expert
Hopper
108.6 2.1
91.5
27.1
106.2
100.9
98.7
107.6
107.2
Medium-Exper
Walker
109.8 0.2
109.6
53.8
107.2 2
57.5
111.0
108. 1
108. 4
Medium
HalfCheetah
2.2
4
37.4
30.0
40.7
44.4
42.6
Medium
Hopper
57.1+1.11
66.3
56.6
35.9
62.2
54..5
58.0
67.6 6
08.5
Medium
Walker
77.9 2.5
78.3
74.0
17.4
73.4
53.1
79.2
74.0
79.7
Medium- Replay
HallCheetah Hopper
37.1 1.7
44.2
40.3
40.3
38.2
46.2
36.6
Medium- Replly
86.2+
94.7
28.4
94.77
33.1
48.6
82.7
96.8
Medium-Rephyg
Walker
65.1
73.9
15.3
58.8
15.0
26.7
61.2
Average (Locomotion)
75.6
76.9
34.3
71.6
51.9
63.9
74.7
Default Diverse
AntMaze-umaze
92.0
87.5
87.5
36.0
85.0
78.9
74.0
59.2
AniMase urnazee
85.3 1. 3.6
62.2
78.0
70.3
76.7
55.0
84.0
53.0
Play
AntMaze-medium
2.6
71.2
78.0
0.0
15.0
0.0
61.2
0.0
Diverse
AntMane-medium
8.1
70.0
77.0
0.0
23.3
0.0
53.7
0.0
Play
AniMaze-large
59.3 14.3
39.6
57.0
0.0
0.0
6.7
15.8
0.0
Diverse
AntMaze-larg
45.5
6.6
47.5
58.0
0.0
8.3
2.2
14.9
0.0
Average (AntMaze)
74.2
63.0
72.6
21.0
46.7
23.8
50.6
18.7
Average (Maze2d)
74.0
50.0
10.8
9.1
7.7
Average (FrankaKitchenn)
57.1
53.3
8.7
11.7
48.2
Both-side
Bidirectional-CC.
100.0 0.0
15.7
0.0
0.0
52.0
88.0
42.3
33.3
Single side
Bidirectional Car
100.0 1. 0.0
100.0
100.0
96.3
100.0
100.0
100.0
100.0
Table I: Evaluation numbers of SIBC. Scores are normalized according to Fu et al. 2020), Numbers
within 5 percent of the maximum in every individual task are highlighted in boldface. Experiment and evaluation details are provided in Appendix B
We report scores with 15 diffusion steps.
et al. 2021
However, Kostrikov et al. (022) shows this method performs poorly in tasks that
require stitching together successful subtrajectories (e.g., Maze-like environments). Such tasks also challenging for methods based on one-step bootstrapping because they might require hundreds of steps to reach the reward signal, with the reward discounted and estimation error accumulated along the way. Episodic memory-basee methods address this problem by storing labeled experience in the dataset, and plans strictly within the trajectory to update evaluations of every decision (Blundell et al., 2016, Hu et al. 2021] Ma et al.) 2022). The in-sample planning scheme allows dynamic programming over long horizons to suppress the accumulation of extrapolation error, which inspires our method.
Generative models for behavior modeling. Cloning diverse behaviors in continuous action space requires powerful generative models. In offline RL, several works (Fujimoto et al. 2019 Kumar et al. 2019; Wu et al., 2019; Zhou al. 2021 Chen et al. 2022) have tried using generative models such as Gaussians or VAES to model the behavior policy. However, the learned behavior model only serves explicit distributional constraint for another policy during training. In broader RL research, generative adversarial networks (Goodfellow et al.] 2020)), masked autoencoders (Germain et al.) 2015), normalizing flows (Dinh al. (0166), and energy-based models (Du & Mordatch 2019) have also been used for behavior modeling (Ho & Ermon 2016 Ghasemipour et al.) 2021 Singh et al. 2020 Liu et al.) 2020). Recently, diffusion models (Ho et | 2020) have achieved great success in generating diverse and high-fidelity image samples (Dhariwal & Nichol 2022). However, exploration of its application in behavior modeling still limited. Janner al. (2022) proposes solve offline tasks by iteratively denoising trajectories, while our method uses diffusion models for single-step decision-making Concurrently with our work, Wang al. (2022) also studies applying diffusion models to offline RL to improve policy expressivity. However, they use diffusion modeling as an implicit regularization during training of the desired policy instead of an explicit policy prior.
EXPERIMENTS
6.1 EVALUATIONS ON D4RL BENCHMARKS
In Table I compare the performance of SÉBC to multiple offline RL methods in several D4RL (Ful cet al., 2020) tasks. MuJoCo locomot in is a classic benchmark where policy-generated datasets only cover narrow part of the state-action space, avoiding querying out-of-sample actions is
7

---

Published as a conference paper at ICLR 2023
Posttion
Poditin
Position
Spood
Groud Truth
(eBr(vers)
BAIL
Action
Spaad
Visualizations of actions taken by different RL agents the Bidirectional-Co task. The ground truth corresponds agent which always takes the best actions, which is either 1.0 or -1.0. White space indicates suboptimal decisions. Green bounding boxes indicate possible initial states.
Figure 3:
critical (Fujimoto et al. 2019; Kumar et al. 2020). The Medium dataset of this benchmark is generated by single agent, while the Medium- Expert and the Replay dataset are generated by a mixture of policies. MntMaze is about an ant robot navigating itself in maze, which requires both low-level robot control and high-level navigation. Since the datasets consist of undirected trajectories, solving AntMaze typically requires the algorithm to have strong "stitching" ability (Fu al. 2020). Different environments contain mazes of different sizes, reflecting different complexity. Maze2d is very similar to AntMaze except that it's about a ball navigating in maze instead of an robot. FrankaKit chen are robot-arm manipulation tasks. We only focus on the analysis of MuloCo locomotion and AntMaze tasks due to the page limit. Our choices of referenced baselines are detailed Appendix E
Overall, StBC outperforms most existing methods by large margins in complex tasks with sparse rewards such as AntMaze. We notice that VEM also achieves good results in AntMaze tasks and both methods share an implicit in-sample planning scheme, indicating that episodic planning is effective improving algorithms' stitching ability and thus beneficial in Maze-like environments. locomotion tasks, SiBC provides highly competitive results compared with sate-of-the-au algorithms. can be clearly shown that performance gain large datasets generated by a mixture of distinctive policies (Medium-Expel Expert) and is relatively small in datasets that are highly uniform (Medium). This reasonable because SIBC is motivated to better model diverse behaviors.
easier
6.2 LEARNING FROM DIVERSE BEHAVIORS
In this section, analyze the benefit of modeling behavior policy using highly expressive generative models. Although SfBC outperforms baselines in many D4RL tasks. The improvement mainly incremental, but decisive. We attribute this the lack of multiple optimal solutions in existing benchmarks. To better demonstrate the necessity of introducing an expressive generative model, we design simple task where heterogeneous dataset collected in an environment that allows two distinctive optimal policies.
Bidirectional-Co task. As depicted in Figure in the middle of two endpoints. The car chooses an action in the range [-1,1] at each step, representing throttle, to influence the direction and speed of the car. The speed of the car will monotonically increase based on the absolute value of throttle. The direction of the car is determined by the sign of the current throttle. Equal reward will be given on the arrival of either endpoint within the rated time. It can be inferred with ease that, in any state, the optimal decision should be either I or or- which is not a unimodal distribution. The collected dataset also contains highly diverse behaviors, with an approximately equal number of trajectories ending at both endpoints. For the comparative study, collect another dataset called "Single-Siae Sidd"" where the only difference from the original one that remove all trajectories ending at the left endpoint from the dataset.
consider an environment where a car is placed
8

---

Published as a conference paper at ICLR 2023
Antmaae
Normalized Score
K-1
K-2
50
K=5
U-diverse
M-play
M-diverse
L-diverse
L-play
MujoCo Locomotion
Normalized Score
100
K=2
K-
50
K- 4
K=5
hopper MR walker MR cheetah MR walker ME cheetah ME hopper-ME ME hopper M
walker
cheetah
Figure 4: Ablation studies of the value iteration number K in MuJoCo Locomotion and Antmaze 1 represents algorithms that use vanilla returns R as Q-learning targets without the implicit planning technique. All results averaged over 4 independent random seeds.
domains. =
We test our method against several baselines, with the results given in Table Among all referenced methods, StBC is the only one that can always arrive at either endpoint within rated time in the Bidirectional-Can environment, whereas most methods successfully solve the "Single-Side" task. To gain some insight into why this happens, we illustrate the decisions made by an SfBC agent and other RL agents in the 2-dimensional state space. As is shown in Figure 3] the SiBC agent selects actions of high absolute values at nearly all states, while other unimodal actors fail to pick either one of the optimal actions when presented with two distinctive high-rewarding options. Therefore, conclude that an expressive policy is necessary for performing diverse behavior learning.
6.3 ABLATION STUDIES
Diffusion vs. other generative models. Our first ablation study aims to evaluate variants of SfBC which are respectively based on diffusion models (Ho et al. 2020). Gaussian probabilistic models, and latent-based models (VAEs, Kingma & Welling (2014)) 4)). The three variants use exactly the same training framework with the only difference being the behavior modeling method. As is shown in Table 4 in Appendix D the diffusion- based policy outperforms the other two variants by a clear margin most experiments, especially in tasks with heterogeneous datasets (e.g., Medium-Expeer)) indicating that diffusion models are for "high-fidelity" behavior modeling.
Implicit in-sample planning. To study the importance of implicit in-sample planning the performance SiBC, we first visualize the estimated state values learned at different iterations of Q-learning in an AntMaze environment (Figure 2) We to iteratively stitch together successful subtrajectories and provides optimistic action evaluations. Then we aim study how the value iteration number K affects the performance of the algorithm in various environments. As shown in Figure 4 we compare the performance of K in the range (1,2, 2,3, 4, 5) and find that implicit planning is beneficial complex tasks like AatMaze-Medinng and AntMaze-I Large. However, it is less important in HtJoooo-locomoiin tasks. This finding is consistent with prior work (Brandfonbrener et 2021
that implicit planning helps
CONCLUSION
In this work, we address the problem of limited policy expressivity in previous weighted regression methods by decoupling the policy model into a behavior model and an action evaluation model. Such decoupling allows us to use a highly expressive diffusion model for high-fidelity behavior modeling, which is further combined with a planning-basee operator to reduce extrapolation error. Our method enables learning from a heterogeneous dataset in a continuous action space while avoiding selecting out-of-sample actions. Experimental results the D4RL benchmark show that our approach outperforms state-of-the-ar art algorithms in most tasks. With this work, we hope to draw attention to the application of high-capacity generative models in offline RL.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY
To ensure that our work is reproducible, we submit the source code supplementary material. We also provide the pseudo-code of our algorithm in Appendix and implementation details of our algorithm in Appendix B
ACKNOWLEDGEMENT
We thank Shiyu Huang, Yichi Zhou, and Hao Hu for discussing. This work was supported by the National Key Research and Development Program of China (2020AAA0106000, 2020AAA0106302 202 202IYFB2701000, NSFC Projects (Nos. 62061136001, 62076147, U19B2034, UI81 1461, U19A2081, 61972224), BNRist (BNR2022RC0006), Tsinghua Institute for Guo Qiang, and the High Performance Computing Center, Tsinghua University.
REFERENCES
Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z. Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis. Model-free episodic control. arXiv preprint arXiv:1606.04480 04460, 2016.
David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Advances in Neural Information Processing Systems, 34:4933-4946, 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems, 2021.
Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Jianhao Wang. Yuan Gao, Wenzhe Li, Liang Bin, Chelsea Finn, and Chongjie Zhang. Lapo: Latent-variable advantage-weightte policy optimization for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2022.
Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. BAIL: Best- action imitation learning for batch deep reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020.
Alfredo Clemente, Humberto N Castejón, and Arjun Chandra. Efficient parallel methods for deep reinforcement learning. arXiv preprint 17882,100.00486,, 2017.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, 2021.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605. 160..0803,, 2016.
John R Dormand and Peter J Prince. A family of embedded runge-kutta formulae. Journal of computational and applied mathematics, 6(1):19-26, 1980.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. Advances in Neural Information Processing Systems, 32, 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprini arXiv:2004.07219 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2052-2062. PMLR, 09-15 Jun 2019.
10

---

Published as a conference paper at ICLR 2023
Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation. In International conference on machine learning, pp. 081-889. PMLR, 2015.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pp. 3682-3691 PMLR, 2021.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 1)119-144, 2020.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems, volume 29, pp. 4565 4565-4573, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
Hao Hu, Jianing Ye, Zhizhou Ren, Guangxiang Zhu, and Chongjie Zhang. Generalizable episodic memory for deep reinforcement learning. arXiv preprint S.IV:22033.06669 2021.
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference Machine Learning, 2022.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016.
Iyy Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp- 5774-5783. PMLR, 2021.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-learnin. In International Conference on Learning Representattions 2022.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off q-lerrning via bootstrapping error reduction. CoRR, abs/1906.00999, 2019. URL httphtttpp///xxx.orrr////// 9906.00949
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04779 2020.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. ICLR, 2016.
Minghuan Liu, Tairan He, Minkai Xu, and Weinan Zhang. Energy- imitation learning. arXiv preprint ar XxXx:20044009395, 33, 2020.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00927 2022.
James Lucas, George Tucker, Roger Grosse, and Mohammad Norouzi. Understanding posterior collapse in generative latent variable models, 2019. URL https::/openrevis net/
Forum? Beerumiddr1xaVinn
Xiaoteng Ma, Yigin Yang. Hao Hu, Jun Yang, Chongjie Zhang, Qianchuan Zhao, Bin Liang, and Qihan Liu. Offline reinforcement learning with value- based episodic memory. In International Conference Learning Representations 2022.
11

---

Published as a conference paper at ICLR 2023
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09335, 2020.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighhted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 2019.
Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In 7wenty-Fourth
AAAI Conference on Artificial Intelligence, 2010.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint (rXiv:2204.06125. 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorcalistic text-to- -mnage diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin A. Riedmiller. Deterministic policy gradient algorithms. In Proceedings of the 31th International Conference on Machine Learning, ICMI. 2014, Beijing, China, 21-26 June 2014, pp. 387-395, 2014.
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning, arXiv preprint arXiv:2011. 2020..
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibriu thermodynamics. In International Conference on Machine Learning, pp. 2256 2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b. UtLpttps: /ttttppes///pppneeeleeeeee hww./eee//eeeeeee forum?
Add-PXTIG22RRE
Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998. ISBN 0262193981.
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinfrrcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint M.XXi:2208.06/99 2022.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 7768-7778. Curran Associates, Inc., 2020.
Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Dubureq, Minghao Zhang, Yi Su, Hang Su, and Jun Zhu. Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research, 23(267): 1-6, 2022. URL httt:://jm http://mmmm..ro..ppppae/ wttprr.rprpppppp/////3.
21 21-1127.0 html
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1161, 2019.
Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforce- ment learning. In Conference on Robot Learning, pp. 1719-1735. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2023
A ALGORITHM OVERVIEW
Algorithm Selecting from Behavior Candidates (Training)
Initialize the score-based model 8ø, the action evaluation model Q
Calculate vanilla discounted returns R(0) for every state-action pair in dataset Di Training the behavior model for each gradient stop do
Sample data points (s, a) from D" B Gaussian noises from ( I) and B time t from (0, T)
Perturb a according to a, == a.a + d,6
Update 0 < Vo E1l of s, L) + <l12]
end for
Training action evaluation model iteratively
for iteration k = to K do
Initialize training parameters & of the action evaluation model Qa
for each gradient step do
Sample B data points (8, a, 1() from D
Update & end for
V Ellla(s, [[(-1113
// Update the Q-training targets as in Algarithm Planning(D", pes Qo)
R(k)
end for
Algorithm 2 Implicit In sammle Planning
Input a behavior dataset DD (sequentially ordered), learned behavior policy pa. a critic model Q // Evaluate every state in dataset according Equation 73 with M Monte Carlo samples (parallelized) for each minibatch {s.,) splitted from D do
Sample M actions a from uo{- 8n), and calculate Q-values RA: Qalsn, a
Calculate state value V, = Em exp (a Rm
Em exp a R.
end for
Performing implicit hn-sample planning recursively
for timestep || to 0 do
Rn Tn y max Rn+1; Vn+1) 1) if n. is not the last episode step, else rn end for
Output the new Q-training targets {Rn)
*
action
similar states
reward
morisus
actiooss
demotising
diffission
V-value
Datasct trajectories
(b) Generation of action candidates
(e) Action selection
(d) Implicit in-sample planning
Figure An algorithm overview of SíBC.
13

---

Published as a conference paper at ICLR 2023
B EXPERIMENTAL DETAILS
B. IMPLEMENTATION DETAILS OF SFBC
Network Architecture. SfBC includes conditional acored-based model which estimates the score function of the behavior action distribution, and an action evaluation model which outputs the Q- values given state-action pairs. The architecture the behavior model resembles U-Nets, but with spatial convolutions changed to simple dense connections, inspired by Janner et al. (2022) For the action evaluation model, we use a 2-layer MLP with 256 hidden units and SiLU activation functions. The same network architecture is applied across all tasks except for AntMaze-Large, where we add an extra layer of 512 hidden units for the action evaluation model.
rack (P 100-
Hentral 02e 2000
Hemaal Black ctt ct0-4122
Bodter Beet (otee Ei
Resiine Book
Dues
Ceecotmaakon
ARE
Figure 6: The network architecture of the behavior model.
Behavior training. all experiments, we use the Adam optimizer and batch size of 4096. The conditional ccored-based model is trained for 500 data epochs with learning rate of 1e-4. For the data perturbation method, we use Variance Preserving (VP) SDE as introduced [Snn al.) (2021b), where we have dæ - (t) + /B(tjdw, such that we have f(x, = -= and (() Btt) in Equation 6. and also:
æo. pro(xx(<0o) = (1, of = ((x,le== p(s)de (e))dexp, [1 (1-e- -- (())de]I).
(15)
Following the default settings in Song et al. (2021b), we set B(t) = (8max Bmin) t + 8min. with Amin being 0.1 and Smax being 20.
Action evaluation via in-sample planning. The action evaluation model is trained for 100 data epochs with learning MuJoCo tasks, K = 4 for Antmaze-umaze tasks, and K = 5 for other Antmaze tasks. In each iteration, new Q-targets will be recalculated according Equation [2] and Equation [3 based on
of 1e-3 for each value iteration. We K == value iterations for all
the latest policy. We use Monte Carlo methods and importance sampling to estimate V, (kk--) in Equation 13
((k (k--1) =Eawi( ( a)
exp (aQo(sn a))
-arpeo(lee.)
(s(
2(8n)
exp (aQo (s,, a))
Q (Sn,
(16)
En exp (aQ s (m,, a))
with the inverse temperature set to 20 and the Monte Carlo sample number set to 16 all tasks. Note that the beginning of each value iteration, normalize Q-targets stored in the dataset and reinitialize the training parameters of the action evaluation model. Different from most prior works
(Fujimoto al. 2019 Kumar et al. 2020 Ma al. 2022 Kostrikov et al. 2022), we do not use either ensembled networks or target networks to stabilize Q-learning.
Diffusion sampling. To draw action samples from the behavior model,
3rd-order specialized diffusion ODE solver proposed by Lu et al. (2022) to solve the inverse ODE problem in Equation 6 We use diffusion step of D 15 for all reported results in Table which is significantly less than
14

---

Published as a conference paper at ICLR 2023
the typical 35-50 diffusion steps required if using ordinary RK45 ODE solver (Dormand & Prince 1980). We also compare the performance and runtime of diffusion steps in the range {5, 10, 15, 25 with the results reported in T Generally, find that 10-25 diffusion steps perform similarly well in Muloco Locomotion tasks and 15-25 diffusion steps perform similarly well Antmaze tasks.
all Antmaze and MuloCo experiments over 4 trials (different random seeds) and other experiments over 3 trials. For each trial, performance is averaged on another 100 test seeds for Antmaze tasks and 20 test seeds for other tasks at regular intervals (5 data epochs). During algorithm evaluation, we select actions in deterministic way. Specifically, the action with the highest Q-value within M behavior candidates will be selected for environment inference during evaluation. In MuJoCo Locomotion tasks, we average four actions with the highest Q-values among all candidates and find this technique helps to stabilize performance. We set the candidate number M to 32 in all experiments.
Evaluation. Following the evaluation metric proposed by Fu al. (2020),
Runtime. We test the runtime of our algorithm on RTX 2080Ti GPU. For algorithm training, the runtime cost of training the behavior model is 10.5 hours for 600 epochs, and the runtime cost of training the action evaluation model is about minutes IM data points considered), For a concrete example, it roughly takes 155 minutes to train the action evaluation model (K=5) and 10.5 hours to train the behavior model for the weaffcheetah-meding task.
each value iteration, (usually 2-5 iterations,
As for the evaluation runtime, theoretically, SIBC requires at least D times of network inference time compared with non-diffusion methods (D = 1), D being the diffusion steps. To accelerate algorithm evaluation, we implement a parallel evaluation scheme similar to rofCleentee et al., (2017), 7); Weng et al. (2022) that could allow evaluating the algorithm under multiple test seeds at the same time, allowing us to significantly reduce the evaluation runtime by utilizing the parallel computing power of GPUs (Figure
Diffusion Steps
5 steps
10 steps
15 steps
25 steps
Performance (cccmmotion)
2.3
72.9
75.6
74.4
Performance (Antmaze)
5.5
65.7
74.2
73.0
Runtime episode, # cnvs=1)
22.3 s
38.0
50.0 s
93.0
Runtime episode, # cnvs=20)
3.2
5.0
Table 2: Ablation studies of the diffusion steps. The runtime is reported for the "halfcheetah-mediine task on RTX 2080Ti GPU. episode stands for 1000 environment steps.
10
Runtime (4) IK env steps
10'
10"
parallel
parallel
20 parallel envs
100 parallel envs
5
15
20
25
30
Diffasion steps
Figure 7: Evaluation runtime of SfBC.
15

---

Published as a conference paper at ICLR 2023
B.2 IMPLEMENTATION DETAILS FOR ABLATION STUDIES
SBBC + VAEs/Gaussiass. For the VAE-based behavior model, we use exactly the same network architecture and training loss as Fujimoto et al. (019) and train the behavior model for 300k iterations at learning rate of 3e-4. For the couussian-based policy model, we follow Nair et al. (2020) and use 4-layer MLP with 256 hidden units and ReLU activation functions. The sampled action from the parameterized Gaussian distribution is squashed to range [-1, 1] by a Tanh activation function. The Gaussian behavior model is trained by directly maximizing the log-likelihood of the dataset distribution for 300k iterations at learning rate of 3e-4. Other experiment settings are consistent with the diffusion-basee method.
SfBC Planning. Removing the planning- -basid procedure from StBC is equivalent to performing SfBC for only one iteration, which only learns behavior Q-function purely from vanilla returns. Other than this, we use the same network architecture and training paradigm as SIBC.
B.3 SOURCES OF REFERENCED BASELINE NUMBERS
For IQL (Kostrikov et al. 2022), D4RL performance numbers are reported in its original pa- per, except for Maze2d tasks. which we reference Janner et al. (2022). The performance in the Bidirectional-Ca task is based on PyTorch reimplementation of the algorithm (https) 7/github. com/ gwt.ggtthhssssooo TOL-PyTorch, We use the same hyperparameters as in the origi- nal paper for MuloCo Locomotion tasks.
For VEM (Ma al.] 2022) and Diffuser (Janner al. 2022), all D4RL performance numbers come from their respective papers. Performance numbers of VEM in the Bidirectional-Car task are based on a slightly modified version the algorithm's 's official codebase (ittps:///itthub walgginnang///iomm
Since the performance of VEM is very sensitive to hyperparameter in their algorithm. We evaluate T E {0.1, ..., 0.9} and report the best-performing choice.
For AWR (Peng al. 2019), BCQ (Fujimoto al. 2019) and CQL (Kumar et al. 2020), all their D4RL performance numbers come from Fu et al. (2020). Their performance numbers in the Bidirectional-CS. task based three independent codebases: com/Farama wwwoooudattt...mm Bttp://4RR-EEvaautionnn for AWR, https: qithub..hu.. com/ sfujim/ BBQ for BCQ and https: htt.://aa...aoaaaaooo//////aaaaa
com/ wwwoongggnngggooooo CCL for CQL. We mostly
use the default hyperparameters in their respective codebases.
For BAIL (Chen et al. 2020). all reported performance numbers come from experiments based on a slightly modified version of its official codebase (https://gital t.ub...u.............. com /aomm/aaaavvk/// L.IIL. Note that BAIL proposes technique to replace oracle returns with augmented returns in MuJoCo Locomotion tasks, whereas we omit using this technique because it cannot be easily applied to other offline tasks. Other than this, we use default settings in the original codebase.
For DT (Chnn et al., 2021), D4RL performance numbers are reported in DT's paper, except for AntMaze tasks, which we reference Kostrikov al. (2022). The performance numbers in the Bidirectional-Car task are based on the algorithm's official codebase /tttps:///iitaii com/ kzl/deci iicccission stnrriisiiintrtrneeeee former). Locomotion tasks.
We use the same hyperparameters as they did for MuloCo
16

---

Published as a conference paper at ICLR 2023
C THEORETICAL ANALYSIS
In this section, we provide some theoretical analysis of our planning- basee operator:
T a) == max[ (T*)""'' a)}.
(17)
n20
First, provide the following proposition to discuss the contraction property of T and the bound of its fixed point.
Proposition 1. We have the following properties Ti"
T) owns monotonicity, i.e., for vQ1 S Q22, we have TxQ: S ST100..
2) TT is at least a n-contraction.
Assume the fixed point of T is Q, then have (=, a) s (s) Q* a) holds for Vs, a.
here Q", Q" are the fixed points T and T respectively.
Proof.
1) For vQ1 & Q2, Vs, a, Vn E N. we have
(T") (7))))
(by monotonicity of T and (Tt)")
(18)
Thus we have max (T")" s = =20
()))(s,a)
(19)
n20 220
2) For vQ1, Q2 Vs a, we have
(1, Tp Q2(22 = max Q(x((
max mox((
n20
220
n20
(20)
max n+1 1
n20
= 11Q - Q211x0
Consequently, T, is at least 7-contraction.
3) For Vs, a, we first prove (* a) a) Q(s,
a a). For Vm E N, we have
(s. < Q= (by monotonicity of TTr) (T.))
(21)
Thus (= v lim (Tp (" (s, ((s.a).
m m-
Now we prove that ((s, < (s a). We have
T=Q" R(s, a) + 7B, Exchn
(22)
R(ss, a) + EB, max Q'(''''''''''')')'')' a') = (s)
Similarly, we have ("Q*(s, < as)
Then for Vn, E N.
(s, 2 (s 2 (s,) 2 (TT)
(by monotonicity of T*)
2 (s..a.
(by monotonicity of (Tu)*)
Thus ('
~ (s,)
(23)
N TT
(by monotonicity of T,²)
2 ... (T..a
Thus
Q"
lim
(T) a) = a).
000
17

---

Published as a conference paper at ICLR 2023
Moreover, similar to the analysis in Ma et al. (2022), provide the following proposition to show that at the beginning the training when the current Q function estimates Q(s, a) is significantly pessimistic, our 7p provides relatively optimistic update and can contract the estimation eπror more quickly.
Proposition In practice, we consider QT, a) = mmxxxxxxss ((T)) a)}. Then we have:
IT, Q - S 5 ( Q|Q - Qn. bl + 11Q - Q01loo
Vs,a,
(24)
here n" n*(s, = arg max ( (T")"""'(s, }}} and Qn- is the fixed point of (TH) (M.
Proof. We can use the triangle inequality prove this result
Tp - (s)
Q(s. (* a)
< <| T' (ea) Qn (s, + (7
(s, (s
=I (QQ (Tu)r
(ea) as) + 12n (s, a) (s,
<n ( Q1Q 11 11Qu- 201
(25)
When is significantly lower than Qn., Q', 11..- - lo is often conspicuously lower than ||Q Qn- and (s, a) relatively large (this often happens the beginning of the training since the initial Q estimates are often near zero and thus pessimistic). At this time, based on this proposition, our operator T, could contract the estimation error with rate of around y" (s.a) which could significantly reduce extrapolation iterations required.
18

---

Published as a conference paper at ICLR 2023
D MISSING PERFORMANCE NUMBERS
Dataset
Environment
SFBC
IQL
VEM
AWR
BAIL
BCQ
CQL
DT
Diffuser
Sparse
Maze2d-umaze
73.9 6.6
47.4
1.0
12.8
5.7
113.9
Sparse
Maze2d-medium
73.8 2.9
34.9
7.6
5.0
121.5
Sparse
Maze2d-large
74.4 1.7
58.6
23.7
12.5
123.0
Average (Maze2d)
74.0
50.0
10.8
9.1
7.7
119.5
Complete
FrankaKilchen
0.6
62.5
Purtial
FrankaKitchen
4.19
46.3
15.4
18.9
49.8
Mixed
FrankaKitchen
45.4 + 1.6
51.0
10.6
8.1
51.0
Average (FrankaKitchen)
53.3
Table Additional performance numbers of SfBC in Maze2d and FrankaKitchen tasks. We report the mean and standard deviation over three seeds for SfBC. Scores normalized according to Fu et al. (020)
Dataset
Environment
SIBC
SIBC + Gaussian
SIBC VAE
SIBC Planning
Medium-Exie -xxeert Medium- Medium-Expert
HalfCheetah
92.6 0.5
79.4 = 1.4 107.8 +
85.2
91.4 0.6
Hopper
108.6
92.0 7.3
109.0
Medium Expertt
Walker
109.8
71.5 =
109.3
109.4 0.9
Medium
HalfCheetah
45.9 + 2.2
42.0
43.4 + 0.1
42.4 :: 0.2
Medium
Hopper
57.1 1: 4.1
58.1=11 1.5 1.5
65 65.6% + 3.3
60.1 + 4.2
Medium
Walker
77.9 : 2.5
82.4
79.1 + 2.5
80.3 = 0.9
Medium- Replay
HalfCheetah
37.1 1.7
36.2 =
42.4 +
37.5 +
Medium-Ree Replay
Hopper
86.2 9.1
67.8
58.6 4.8
58.6 + 1.3
Medium-RRe Replay
Walker
65. + 5.6
65.8
62.2 + 4.3
62.6 + 2.2
Average
75.6
67.9
70.9
72.3
Default Diverse
AntMaze-umaze
92.0 + 2.1
93.3 + 2.4
91.6 + 2.4
96.7 + 4.7
AntMaze-umaze
85.3 + 3.6
88.3 1 2.4
78.3 + 4.7
80.0 = 10.8
Play
AntMaze-medium
81.3 2.6
80.0
68.3 +
35.0
Diverse
AntMaze-medium
82.0 - 3.1
85.0
65.0 7.1
33.3
Play
AntMaze-large
59.3 = 14.3
43.3 = 7.1
35.0 + 8.2
8.3 8.5
Diverse
AntMaze-large
45.5 + 6.6
26.7 = 8.5
20.0 + 0.0
6.7 = 4.7
Average
74.2
69.4
59.7
43.3
Table 4: Ablations of generative modeling methods and the implicit planning method. We report the mean and standard deviation over four seeds for the main experiment and three seeds for other experiments. Scores are normalized according to Fu et al. 2020).
E CHOICES OF REFERENCED BASELINES
Referenced baselines methods of SfBC can be roughly divided into four categories: 1. Policy regression methods that require dynamic programming such as IQL (Kostrikov et 2022) and VEM (Ma et al. (022). 2. Policy regression methods that use vanilla returns as regression weights such as AWR (Peng et al. 2019) and BAIL (Chen et al. 2020). 3. Adaptations of existing off-policy algorithms with policy regularization such as BCQ (Fujimoto et al. 2019) and CQL (Kumar et al. 2020). 4. Sequence modeling methods such as DT (Chen et al. 2021) and Diffuser (Janner et al. 2022). Here we further highlight several methods which bear some resemblance to our approach: Both IQL and SIBC aim to entirely avoid selecting out-of-sample actions, except that IQL uses weighted regression while SfBC does not; VEM also
implicit in in-sample planning scheme similar to ours; BCQ also uses a generative model (VAE) for behavior modeling, but only to assist the learning of another policy model; Diffuser, like SfBC, is also diffusion- based algorithm, but uses approximated guided sampling at trajectory level instead of importance sampling at step level.
19

---

Published as a conference paper at ICLR 2023
F TRAINING CURVES
haltcheeda--nddiueeeppett
hopper-medium-ee moc.muue-------2222222
wwlker2dd-edlum-ppprr....
100
15 steps
stops
80
stops
25 stope
60
stops
40
steps
10 steps
10 stops
steps
40
20-
50
100
230
50
100
150
50
100
150
halfchoctah medium-v2
happer-men seceer-mddiimi
walker22 im edium v2
70
40
30
steps
50
stups
60-
25 stops
25 stops
25 steps
10 sleps
10 steps
40-
10 slaps
30
10
50
100
1.50
50
100
1.50
50
100
15.
halfchentah wwc.iaaaammmimm---eem replay-v2
mepper-medium-repllay
walkerr2d medium-mmplay--2
40
80-
80-
60
15 steps
60
20
stops
40
10
5 staps
40-
stops
15
20-
25 stops
10 stops
10 sleps
stops
100
2ermaae--
antmaze-um weteee--aaae---d-ee--.--
antmaaz--mellun--iieeee
100
80
80-
60
15 steps
60
steps steps
15 stops
steps
staps
20
100
200
300
100
200
300
150
200
300
400
centmazee-argeedvveeee diverse-22
antm aze-largeeeereeeooooooo 2aae--arre--pay--222
60
25 steps
10 staps
15 steps
6.-
25 steps
steps
steps
60
15 steps
40
40
stepp
steps
steps
20
20.
Figure 8: Training curves of SIBC for MuJoCo and Antmaze tasks with different diffusion steps. We report the mean and standard deviation over four seeds for all experiments. Note that the parameters of the critic model are initialized at the beginning of each value iteration (every 100 data epochs).
20

---

Published as a conference paper at ICLR 2023
G CONNECTIONS TO PRIOR WORKS
In this section, we discuss in more detail the connections between SfBC and two prior works, namely VEM (Ma et al. 2022) and EMAQ (Ghasemipour al., 202I).
G.1 VEM
Our in-sample planning-based Q-operator T bears some similarity to the multi-step estimation operator Trem proposed by Ma et al. (2022). A simplified version of Tvem is defined as:
Trem ((s)
max (T
(26)
a20
which is is built on TH an expectile-based V-learning operator proposed by VEM:
if r(8.a) + VV() 2 V(s) Vs)) = Ba) { a) [ (1-r)]r(s, + VV(') if ((s.a)++ V(s))
(27)
here T E 10,1) is a hyperparameter that helps interpolate the Bellman expectation operator T (T 0.5) and the Bellman optimality operator T (T - 1.0). V--) an arbitrary scalar function. T. has some nice properties such as monotonicity (T, (s)) > Tt T V(s) always holds for any V given T2 > 71). With these properties, Ma al. (002 derives that Trem and T share the same fixed point.
However, VEM cannot be applied to stochastic environments because Equation 27 requires comparing (ss) and rss, a) + +VV (V("). While Vss) is scalar given 8, r(s, a)) + >V(s' is a random variable since s' P(-| 8, a). To fix this problem, VEM simply assumes that the environment deterministic, namely r(s, a) and P(-|s, a) are all Dirac
Compared with VEM, our in-sample planning- Q-operator Thr is not dependent on the expectile- based V-operator T,. but uses an hypothetically improved policy π > μ for optimistic planning:
(=, a) == (TT)) a)}.
(28)
which does not require the environment to be deterministic. A disadvantage of using Th to replace TI is that we no longer have the monotonicity property (c.g.. T=Q(s, a) >> T"Q(s, a) always holds for any Q). However, we can still derive that the fixed point of Tp is bounded between Q and Q" (See Appendix C for detailed results and proofs).
G.2 EMAQ
The high-level idea of the selecting- www--eaavi-o--------aaeaaa the Expected-Mr Max Q-Learning (EMaQ) algorithm proposed by (Ghasemipour et al.) 2021). EMaQ is built upon BCQ (Fujimoto al. 2019), which computes the training target in Q-Learning by:
approach bears resemblance
TicoQ(s a) == r(s,a) + max
{Q(s', a' (s'(''a"))
(29)
a'>pol--as")
where (s a) is an explicitly constrained perturbation network that helps relax the constraint of behavior policy p. The core motivation for EMaQ is to remove the perturbation model (s a), by taking max over N Q-function evaluations:
((,aQQ(s, a) == r(s, a) + +
Q(s', a()). d(Eaa()
max
(30)
For EMAQ, serves as hyperparameter to interpolate TH and T When = 1, TEMaQ becomes TP. When oc, TENaq approaches T because {a{}* nearly covers the whole action space.
In contrast, for SiBC, the hyperparameter N is the number of Monte Carlo samples used to estimate the training Q-targets:
T T"Q(s,a) a) =r(s, a) + + ~ ((( a')
=r(s, a) + + exp (aQ(s' a') ((x')
Q(8'
exp (aQ(('s a'))
~r(s, a) + Σ
Σ exp exp (aQ(s', a')) Q(s' d')
N
21


Reviews:
[{"review_id": "UvfNDm8X92B", "reviewer": "Reviewer_eCnE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to learn the offline RL policy by a coupling of pure behavior cloning and importance sampling. The behavior cloning is modeled by a denoising diffusion probabilistic model, and an implicit in-sample Q-learning is applied to estimate the importance sampling weight. Experiments are done in the D4RL benchmark datasets to demonstrate the effectiveness of the method. ", "summary_of_the_review": "This paper proposes to separate the policy learning into two parts: 1) pure behavior cloning via diffusion models; 2) importance sampling via in-sample Q-learning. The idea of using diffusion models as an expressive policy is appealing. However, I do have concerns about the novelty and computational complexity. \n\nAs I mentioned in the weaknesses, most of the components from the paper are proposed by prior works. Moreover, by looking at the Algorithm 1, performing implicit in-sample planning has high computational complexity. There are two for loops, one for K and the other for N, which is dataset size and usually huge. Diffusion models are notoriously slow in sampling, while the training algorithm needs samples from diffusion models within the K$\\times$N loop, which has too high complexity.\nI also have some detailed questions as follows: \n1. What is the number of timesteps for the diffusion model used in the paper? I guess an ablation study on the number timesteps is needed to show the effect of it.  \n2. Better provide a training curve in the appendix to show the training stability. \n3. For the reported values in Table 1, what are these values? Are them the statistics based on the last points after the training or the highest points during the training? \n5. The policy network is much larger than the current policy networks, which makes the computational cost expensive and comparison not very fair. \n6. Expect to an ablation study on the K, since it is a key parameter in Algorithm 1. \n7. Minor: typo in Equation (12), I think it is $R^{(k-1)}_{n+1}$.\n", "strength_and_weaknesses": "Strengths: \n1. The performance of SfBC evaluated on D4RL is comparable to the current SOTA methods. \n2. The authors demonstrate the importance of using expressive policy class through toy examples. \n3. The paper is easy to follow and well written. \n\nWeaknesses:\n1. This paper lacks novelty. Policy improvement via weighted regression has been well studied in many papers, such as IQL and AWAC paper. The proposed Q-learning via in-sample planning is motivated from Ma et al. 2022. Applying diffusion models in offline RL is new while using diffusion models for behavior cloning seems an easy plug-in.  \n2. The paper misses some details about implementation. \n3. Computational complexity is a concern and not discussed. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NmXBGzlo_u", "reviewer": "Reviewer_dfM5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a method for offline reinforcement learning that involves fitting a more expressive density model and an episodic planning technique. The method demonstrates decent results on D4RL, a standard benchmark for offline reinforcement learning.", "summary_of_the_review": "In overall it's a simple a well-performing method. However, it has limited novelty and the writing needs to be improved before the paper can be accepted.", "strength_and_weaknesses": "# Strength \n* The approach is simple and produces strong results on the standard benchmark.\n\n# Weaknesses \n* Right now, the main weakness is a lack of clarity. It is not clear how the method is implemented, and the paper can be significantly improved by expanding the implementation details.\n* The approach has a limited novelty. Expressive policies have been previously studied in EMAQ [1]. Moreover, the value computation technique is similar to a soft version of EMAQ.\n* The related work section misses some important prior work in particular [1], which also introduces a similar approach.\n* At the end of section 4.2, it's mentioned that the n-step operator proposed in this work works in the stochastic setting. However, this statement is not completely clear. For example, does it mean that it's unbiased? \n\n\n[1] EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL\nSeyed Kamyar Seyed Ghasemipour, Dale Schuurmans, Shixiang Shane Gu", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9iC0kJFFIM", "reviewer": "Reviewer_3dPg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work uses a diffusion model as an expressive generative model for behavior cloning. They then turn this into a offline RL algorithm with model improvement by learning a Q-function and performing a form of importance sampling to rejection sample action samples with high value by sampling from the behavior clone model and resampling based on Q. They test this on some standard offline RL benchmarks.\n", "summary_of_the_review": "A well-present piece of work applying diffusion modeling to offline RL. It would be strengthened by making the source code available to ensure it can be reproduced and built on [I note this concern is fixable during the rebuttal period].\n\n[Updated]\n\nThe authors have uploaded source code, cited additional work and made other improvements. I have increase my rating.", "strength_and_weaknesses": "Strengths:\n1. The paper is well written and explains the method.\n\n2. The method is well motivated.\n\n3. The empirical results show good.\n\n4. The topic is off significant interest.\n\nWeaknesses:\n1. I have concerns about the reproducibility, I believe the norm (particularly for this topic) is to include the source code in the submission rather than a promise of future code. Generally, particularly in RL, people have struggled to exactly reproduce results just from pseudocode.\n\nFor example, I did not see in Appendix C the number of diffusion steps used (quite an important detail). It would also be helpful to discuss the runtime costs of sampling from diffusion models.\n\nIdeally the results would be over more than 3 seeds, but this is not vital.\n\nOther:\n\n[1] Is a closely related work that uses diffusion models for offline RL that should be mentioned in related work (although the method is not identical). It also makes similar claims about the importance of expressive, multi-modal policies.\n\n[1] Wang, Zhendong, Jonathan J. Hunt, and Mingyuan Zhou. \"Diffusion policies as an expressive policy class for offline reinforcement learning.\" arXiv preprint arXiv:2208.06193 (2022).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster