Is Conditional Generative Modeling all you need for Decision Making?
Authors: Anurag Ajay
Yilun Du
Abhi Gupta
Joshua B. Tenenbaum
Tommi S. Jaakkola
Pulkit Agrawal
Venue: ICLR 2023 notable top 5% (2023)
Keywords: Offline Reinforcement Learning
Conditional Generative Modeling
Sequential Decision Making
Diffusion Models

Abstract: Recent improvements in conditional generative modeling have made it possible to generate high-quality images from language descriptions alone. We investigate whether these methods can directly address the problem of sequential decision-making. We view decision-making not through the lens of reinforcement learning (RL), but rather through conditional generative modeling. To our surprise, we find that our formulation leads to policies that can outperform existing offline RL approaches across standard benchmarks. By modeling a policy as a return-conditional generative model, we avoid the need for dynamic programming and subsequently eliminate many of the complexities that come with traditional offline RL. We further demonstrate the advantages of modeling policies as conditional generative models by considering two other conditioning variables: constraints and skills. Conditioning on a single constraint or skill during training leads to behaviors at test-time that can satisfy several constraints together or demonstrate a composition of skills. Our results illustrate that conditional generative modeling is a powerful tool for decision-making. 

IS CONDITIONAL GENERATIVE MODELING ALL YOU NEED FOR DECISION-MAKING?
Anurag Ajay 139, Yilun Du *$11, Abhi Guptatiy, Joshua Tenenbaum?, Tommi Jaakkola Pulkit Agrawal Improbable AI Lab?
Operations Research Center+
Computer Science and Artificial Intelligence Lab Massachusetts Institute of Technology
ABSTRACT
Recent improvements in conditional generative modeling have made it possible generate high-quality images from language descriptions alone. We investigate whether these methods can directly address the problem of sequential decision- making. We view decision-make making not through the lens of reinforcement learning (RL), but rather through conditional generative modeling. To our surprise, we find that our formulation leads to policies that can outperform existing offline RL approaches across standard benchmarks. By modeling policy conditional diffusion model, we illustrate how we may circumvent the need for dynamic programming and subsequently eliminate many of the complexities that come with traditional offline RL. We further demonstrate the advantages of modeling policies conditional diffusion models by considering two other conditioning variables: constraints and skills. Conditioning on a single constraint skill during training leads to behaviors at test-time that can satisfy several constraints together or demonstrate composition of skills. Our results illustrate that conditional generative modeling is powerful tool for decision-making.
return-
INTRODUCTION
Over the last few years, conditional generative modeling has yielded impressive results in a range of domains, including high-resolution image generation from text descriptions (DALL-E, ImageGen) (Ramesh et al., 2022; Saharia et al., 2022). language generation (GPT) (Brown et al., 2020), and step-by-step solutions math problems (Minerva) (Lewkowyez et al., 2022). The success of generative models in countless domains motivates us to apply them to decision-making Conveniently, there exists wide body of research recovering high-performing policies from data logged by already operational systems (Kostrikov et al., 2022; Kumar et al., 2020; Walke et al., 2022). This is particularly useful in real-world settings where interacting with the environment is not always possible, and exploratory decisions can have fatal consequences (Dulac-Arold et al., 2021). 1). With access to such offline datasets, the problem of decision- maasing reduces to learning probabilistic model of trajectories, setting where generative models have already found success.
In offline sreisionnmaking we aim to recover optimal peward-maxmiine together sub-optimal reward-labeled trajectories in the training dataset. Prior works (Kumar et al., 2020; Kostrikov et al., 2022; Wu et al., 2019; Kostrikov et al., 2021; Dadashi et al., 2021; Ajay et al., 2020; Ghosh et al., 2022) have tackled this problem with reinforcement learning (RL) that uses dynamic programming for trajectory stitching. To enable dynamic programming. these works learn value function that estimates the discounted sum of rewards from given state. However, value function estimation is prone to instabilities due to function approximation, off-policy learning, and bootstrapping together, together known as the deadly triad (Sutton & Barto, 2018). Furthermore, stabilize value estimation in offline regime, these works rely on heuristics to keep the policy within the dataset distribution. These challenges make it difficult to scale existing offline RL algorithms.
trajectories by stitching
denotes equal contribution. Correspondence enjeygnit yi Tondu tnrit.eer edo,, abh grri eedo

---

condraaitte
Difinar
unjecturiee
storrnted inaectociia
Figure 1: Decision Making using Conditional Generative Modeling, Framing decision making as a conditional generative modeling problem allows to maximize rewards, satisfy constraints and compose skills.
In this paper, we ask if we can perform dynamic programming stitch together sub-optimal trajectories to obtain an optimal trajectory without relying on value estimation. Since conditional diffusion generative models can generate novel data points by composing training data (Saharia et al., 2022), we leverage it for trajectory stitching in offline decision-making. Given dataset of
trajectories, we adapt diffusion models (Sohl-Dickstein et al., 2015) to learn return-conditional trajectory model. During inference, we use classifier-free guidance with low- temperature sampling, which we hypothesize to implicitly perform dynamics programming, to capture the best behaviors in the dataset and glean return maximizing trajectories (see Appendix A). Our straightforward conditional generative modeling formulation outperforms existing approaches on standard D4RL tasks (Fu et al., 2020).
Viewing offline Decision-making through the lens of conditional generative modeling allows going beyond conditioning returns (Figure 1). Consider example (detailed in Appendix A) where
robot with linear dynamics navigates an environment containing two concentric circles (Figure 2). We are given dataset of state-action trajectories of the robot, each satisfying one of two constraints: (i) the final position of the robot is within the larger circle, and (ii) the final position of the robot is outside the smaller circle. With conditional diffusion modeling, we can use the datasets to learn a constraint- conditioned model that can generate trajectories satisfying any set of constraints. During inference, the learned trajectory model can merge constraints from the dataset and generate trajectories that satisfy the combined constraint, Figure shows that the constraint-conditioned model can generate trajectories such that the final position of the robot lies between the concentric circles.
Figure 2: Illustrative example. We visualize the 2d robot navigation environment and the constraints satisfied by the trajectories in the dataset derived from the enviromment. We show the ability of the conditional diffusion model generate trajectories that satisfy the combined constraints.
Here, we demonstrate the benefits of modeling policies as conditional generative models. First, conditioning constraints allows policies not only generate behaviors satisfying individual constraints but also generate novel behaviors by fiexibly combining constraints at test time. Further, conditioning on skills allows policies to not only imitate individual skills but also generate novel behaviors by composing those skills. We instantiate this idea with a state-sequence based diffusion probabilistic model (Ho et al., 2020) called Decision Diffuser, visualized Figure In summary, our contributions include (i) illustrating conditional generative modeling as an effective tool in offline decision making, (ii) using classifier-free guidance with low-temperature sampling, instead of dynamic programming, to get eeturn-maximiz trajectories and, (iii) leveraging the framework of conditional generative modeling to combine constraints and compose skills during inference flexibly.
BACKGROUND
2.1
REINFORCEMENT LEARNING
We formulate the sequential decision-making problem discounted Markov Decision Process (MDP) defined by the tuple (po, S, A, R, 7), where Po the initial state distribution, S and are state and action spaces, T S A S is the transition function, R. S A x S - R gives the reward at any transition and E [0, 1) is discount factor. The agent acts with a stochastic policy π transitions or trajectory 7 (5k, ak Tkk k200 with probability (T) and return R(T) Σς>0 *rr.. The standard objective in RL is to find return ettun-maximizing policy n² arg max, E.-pp, [R(7)].
S + A generating sequence of flate-action-rewe
2

---

Temporal Difference Learning TD methods (Fujimoto et al., 2018; Lillicrap et al., 2015) estimate Q* a) == starting in state and taking action a, with parameterized Q-function. This requires minimizing the following TD loss:
= 8, ag = al. the return achieved under the optimal policy x when
(TD(O) = E ((...,,))) + max 20(s',a')) Qo(s,)))
(1)
a'EAA
Continuous action spaces further require learning a parametric policy Ti (a|s) that plays the role of the maximizing action in equation 1. This results in policy objective that must be maximized:
[6) = E.ED, [Q(s a)]
(2)
Here, the dataset of transitions D evolves as the agent interacts with the environment and both Qo and &g are trained together. These methods make use of function approximation, off off-policy learning, and bootstrapping, leading to several instabilities in practice (Sutton, 1988; Van Hasselt et al., 20.
Offline RL requires finding a return-maximizing policy from a fixed dataset of transitions collected by an unknown behavior policy μ (Levine et al., 2020). Using TD-Jearning naively causes the state visitation distribution (s) move away from the distribution of the dataset d( In turn, the policy T begins to take actions that are substantially different from those already seen in the data. Offline RL algorithms resolve this distribution-shi shift by imposing a constraint of the form a [|d"), where D is some divergence metric, directly in the TD-learing procedure. The constrained optimization problem now demands additional implementation heuristics to achieve any reasonable performance (Kumar et al., 2021). The Decision Diffuser, in comparison, doesn" 't have any of these disadvantages. It does not require estimating any kind of Q-function, thereby sidestepping TD methods altogether. It also does not face the risk of distribution-she generative models are trained with maximum-likellhooe estimation.
2.2 DIFFUSION PROBABILISTIC MODELS
Diffusion models (Sohl- DichllDickttenn et al., 2015; Ho et al., 2020) are specific type of generative model that Iearn the data distribution ((x) from a dataset D == {xx Josi<M. They have been used most notably for synthesizing high-quality images from text descriptions (Saharia et al., 2022; Nichol et al., 2021). Here, the data- generating procedure is modelled with a predefined forward noising process ((<x+1/<<)) = (# /ou Eg. (1 I) and a trainable reverse process Pa Pa [xx) = (( 1 |i ( k), Ex), where N(u, Σ) denotes a Gaussian distribution with mean f and variance E, og E R determines the variance schedule, æo == x is a sample, æ1, 02, Ix are the latents, and IK ~ (O, I) for carefully chosen ag and long enough K. Starting with Gaussian noise, samples are then iteratively generated through series of "denoising" steps.
Although a tractable variational lower- bound on log pa can be optimized to train diffusion models, Ho et al. (2020) propose simplified surrogate loss:
Ldnnoise (0) [,=[[1K K) N,0.0, llle eo (x2, k)
The predicted noise eg (a k). parameterized with a deep neural network, estimates the noise ~ (0, I) added to the dataset sample To to produce noisy zg. This is equivalent to predicting the mean of po (=(=-11/2>x) since (2 k) can be calculated as a function of co (Z k) (Ho et al., 2020).
Guided Diffusion Modelling the conditional data distribution ((aly) makes it possible to generate samples with attributes of the label y. The equivalence between diffusion models and score- matching (Song et al., 2021), which shows (0 k)) x Va,, a, log p(æx), leads to two kinds of methods for conditioning: classifier-guided (Nichol & Dhariwal, 2021) and classifier-free (Ho & Salimans, 2022). The former requires training an additional classifier Pa(1/ on noisy data so that samples may be generated at test-time with the perturbed noise < k)) - w wvT ag æv log p(y (.)), where w is referred to as the guidance scale. The latter does not separately train a classifier but modifies the original training setup to learn both conditional (0 y, k) and an unconditional co (o k) model for the noise. The unconditional noise is represented, in practice, as the conditional noise ca (x Ø, k) where dummy value @ takes the place of y. The perturbed noise eo (xx, k) + w(ee k) co(2x, k:)) is used to later generate samples.
GENERATIVE MODELING WITH THE DECISION DIFFUSER
is useful to solve RL from offline data, both without relying on TD-Leaning and without risking distribution-shift. To this end, we formulate sequential decision-making the standard problem of
3

---

resarns
constraints
skilll
+-
state dicfaation cloetffer- gailánoe ne-tterperattrm cxmpiin
Rgua
%
Pg-3
1112
Laversso dynsamics
Figure 3: Planning with Decision Diffuser. Given the current state 8s and conditioning, Decision Diffuser uses classifier-fret free guidance with low-temperature sampling generate sequence of future states. It then uses inverse dynamics extract and execute the action o, that leads the immediate future state $14-1-
conditional generative modeling:
max E plo (((()/(((((())))))))))))))
Our goal to estimate the conditional data distribution with ps so we can later generate portions of a trajectory (D(r) from information y(r) about it. Examples of y could include the return under the trajectory, the constraints satisfied by the trajectory, or the skill demonstrated in the trajectory. We construct our generative model according to the conditional diffusion process:
1 (T))
pa p (1) ((T)
(5)
As usual, q represents the forward noising process while pa the reverse denoising process. In the following, we discuss how we may use diffusion for decision making. First, we discuss the modeling choices for diffusion in Section 3.1. Next, we discuss how we may utilize classifier-fres guidance to capture the best aspects of trajectories in Section 3.2. We then discuss the different behaviors that may be implemented with conditional diffusion models in Section 3.3. Finally, we discuss practical training details of our approach in Section 3.4.
3.1 DIFFUSING OVER STATES
In images, the diffusion process is applied across all pixel values in image. Naïvely, it would therefore be natural to apply a similar process to model the state and actions of trajectory. However, in the reinforcement learning setting, directly modeling actions using diffusion process has several practical issues. First, while states are typically continuous in nature in RL, actions are more varied, and are often discrete in nature. Furthermore, sequences over actions, which are often represented as joint torques, tend to be more high-frequency and less smooth, making them much harder to predict and model (Tedrake, 2022). Due to these practical issues, we choose to diffuse only over states, as defined below:
(T)(T) = (8t, St+1, 8t-H111
Here, & denotes the timestep in the forward process and denotes the time at which state was visited in trajectory т. Moving forward, we will view 2 (7) noisy sequence of states from trajectory of length H. We represent æ (T) as two-twwoeeeeeeee wwo-dimensional array with one column for each timestep of the sequence.
Acting with Inverse-Dynamics. Sampling states from a diffusion model is not enough for defining a controller. A policy can, however, be inferred from estimating the action a that led the state 8, to 8141 for any timestep t in (T). Given two consecutive states, we generate an action according to the inverse dynamics model (Agrawal et al., 2016; Pathak al., 2018):
0₂ == fo(st, 8t-1)
Note that the same offline data used to train the reverse process Po can also be used to learn fo- We illustrate in Table 2 how the design choice of directly diffusing state distributions, with inverse dynamics model to predict action, significantly improves performance over diffusing across both states and actions jointly. Furthermore, we empirically compare and analyze when use inverse dynamics and when to diffuse over actions in Appendix F.
4

---

3.2 PLANNING WITH CLASSIFIER FREE GUIDANCE
Given a diffusion model representing the different trajectories in a dataset, we next discuss how we may utilize the diffusion model for planning. To use the model for planning, it is necessary to additionally condition the diffusion process on characteristics (()). One approach could be to train a classifier (p( ( (=x to predict y()) from noisy trajectories 2y (T). In the case that ()() represents the return under a trajectory, this would require estimating a Q-function, which requires a separate, complex dynamic programming procedure.
One approach to avoid dynamic programming is to directly train a conditional diffusion model conditioned the returns yrr) in the offline dataset. However, as dataset consists of a set of sub-optimal trajectories, the conditional diffusion model will be polluted by such sub-optimal behaviors. To circumvent this issue, we utilize classifier-free guidance (Ho & Salimans, 2022) with low-temperature sampling, to extract high-likelihood trajectories in the dataset. We find that such trajectories correspond to the best set of behaviors in the dataset. For a detailed discussion comparing Q-function guidance and classifier-fr free guidance, please refer to Appendix K. Formally, to implement classifier free guidance, (T) is sampled by starting with Gaussian noise xx (T) and refining a& (T) into 1 (r) at each intermediate timestep with the perturbed noise:
é == eo (T), k) + ((eo(xx (T),, (0(
(8)
where the scalar w applied to (co (a (7), ((), k:) ca ( Ø,, A:) seeks to augment and extract the best portions of trajectories in the dataset that exhibit (T). With these ingredients, sampling from the Decision Diffuser becomes similar to planning in RL. First, we observe state in the environment. Next, we sample states later into the horizon with our diffusion process conditioned on y and history of last C' states observed. Finally, we identify the action that should be taken to reach the most immediate predicted state with our inverse dynamics model. This procedure repeats in a standard accding-horizon control loop described in Algorithm I and visualized in Figure 3.
3.3 CONDITIONING BEYOND RETURNS
So far have not explicitly defined the conditioning variable y()). Though have mentioned that it can be the return under a trajectory. we may also consider guiding our diffusion process towards sequences of states that satisfy relevant constraints demonstrate specific behavior.
Maximizing Returns To generate trajectories that maximize return, we condition the noise model on the return of trajectory so eo (T)) ( k) = e ()))((T); R(T), k). These returns normalized to keep R(T) E [0, 1]. Sampling a high return trajectory amounts to conditioning on R(T) = 1. Note that we do not make use of any Q-values, which would then require dynamic programming.
Satisfying Constraints Trajectories may satisfy a variety of constraints, each represented by the set C, such reaching specific goal, visiting states in a particular order, avoiding parts of the state space. To generate trajectories satisfying a given constraint C, we condition the noise model on a one-hot encoding so that (g(xx (T), ((T), k) = (xx (T), ( E C,), k). Although we train with an offline dataset in which trajectories satisfy only one of the available constraints, at inference we can satisfy several constraints together.
Composing Skills A skill i can be specified from a set of demonstrations B, To generate trajectories that demonstrate given skill, we condition the noise model on one-hot encoding so that co (T))(T,, y()), k) == co ()(2*(T), 1(T r E B,), k). Although we train with individual skills, we may further compose these skills together during inference.
Assuming we have learned the data distributions (< (T))( (f)).....,(()))))))) (T)) for different conditioning variables, can sample from the composed data distribution (T), (r) using the perturbed noise (Liu al., 2022):
n 8 (xx + (7)))) (
(T)
This property assumes that (g) (T) are conditionally independent given the state trajectory (T)..) However, we empirically observe that this assumption doesn't have to be strictly satisfied as long as the composition of conditioning variables is feasible. For more detailed discussion, please refer to Appendix D. We use this property to compose more than one constraint or skill together at test-time. We also show how Decision Diffuser can avoid particular constraint or skill (NOT) in Appendix J.
5

---

Algorithm Conditional Planning with the Decision Diffuser
1: Input: Noise model ég, inverse dynamics fa, guidance scale history length C, condition
2: Initialize h QuereeLLength = C). 1 4-
/I Maintain history of length
3: while not done do
4: Observe state (s..nrstr((): Initialize mnc()) N((0,a))
5: for B = K... do
6:
ma(r]] lengthh(h))
Constrain plan consistent with history
7:
E es (2 (7).8) (m)())) v. ca(z)(r).8))
#/ Classifier-free guidance
8:
(pk 1. Ek 1) Denoise((v(r)..)
9:
~ (HH-H,,,
10:
end for
Extract (st, 8641) from no(r)
12:
Execute a, Stall
end while
15
DARL Losanoies
DARL Kitdhen Koka Bock Stocking
Figure 4: Results Overview. Decision Diffuser performs better than both learning (CQL) and Behavorial Cloning (BC) across D4RL locomotion tasks, D4RL Kitchen tasks and Kuka Block Stacking tasks (single constraint) using only conditional generative modeling objective. For performance metric, we use normalized average returns (Fu al., 2020) for D4RL tasks (Locomotion and Kitchen) and success rate for Block Stacking.
3.4 TRAINING THE DECISION DIFFUSER
The Decision Diffuser, our conditional generative model for decision-making, is trained in supervised manner. Given a dataset the constraint that it satisfies, or the skill that it demonstrates, we simultaneously train the reverse diffusion process pe, parameterized through the noise model Eq, and the inverse dynamics model fo with the following loss:
trajectories, each labeled with the return it achieves,
④) (. colll--c(aaa(( (11)))(())+300 x)] (".., lalallllffe fo s')
For each trajectory т, we first sample noise e ~ NN,0 and a timestep 11(1,..... K}. Then, we construct noisy array of states (7) and finally predict the noise éo co (xxx(())) (r)), k). Note that with probability we ignore the conditioning information and the inverse dynamics is trained with individual transitions rather than trajectories.
Architecture We parameterize Eg with temporal U-Net architecture, a neural network consisting of repeated convolutional residual blocks (Janner et al., 2022). This effectively treats sequence of states (T) as an image where the height represents the dimension of single state and the width denotes the length of the trajectory. We encode the conditioning information (()) as either scalar or one-hot vector and project it into a latent variable E R with a multi-layen perceptron (MLP). When ( = 0, we zero out the entries of We also parameterize the inverse dynamics So with an MLP. For implementation details, please refer to the Appendix B.
Low-temperature Sampling In the denoising step of Algorithm 1, we compute μk-1 and ΣŞ-1 from a noisy sequence of states and predicted noise. We find that sampling æ;-1 N N(u 11) where the variance scaled by a E [0, leads to better quality sequences (corresponding to sampling lower temperature samples). For a proper ablation study, please refer to Appendix C.
EXPERIMENTS
In this section, we explore the efficacy of the Decision Diffuser on a variety of decision-making tasks (performance illustrated in Figure 4). In particular, evaluate (1) the ability recover effective RL policies from offline data, (2) the ability to generate behavior that satisfies multiple sets of constraints, (3) the ability compose multiple different skills together. In addition, we empirically justify use of classifier-fire fresiffer-freeeeeee guidance, low-temperature sampling (Appendix C), and inverse dynamics (Appendix F) and test the robustness of Decision Diffuser stochastic dynamics (Appendix G).
4.1 OFFLINE REINFORCEMENT LEARNING
Setup We first test whether the Decision Diffuser can generate return- maximmiixiiiiinggg trajectories. To test this, train state diffusion process and inverse dynamics model publicly available

---

D4RL datasets (Fu et al., 2020). We compare with existing offline RL methods, including model- free algorithms like CQL (Kumar et al., 2020) and IQL (Kostrikov et al., 2022), and model-based algorithms such trajectory transformer (TT, Janner al. (2021)) and MoReL (Kidambi et al., 2020). We also compare with sequence-models like the Decision Transformer (DT) (Chen et al. (2021) and diffusion models like Diffuser (Janner et al., 2022).
Dataset
Environment
BC
CQL
IQL
DT
TT
MOReL
Diffuser
DD
Med-Fxpen Expert
HaliChectah
55.2
91.6
86.7
86.8
95
53.3
79.8
90.6 +1.3
Med-Expert
Hopper
B2.5
105.4
91.5
107.6
110.0
108.7
107.2
111.8 11.8
Med-Expert
Walker2d
107.5
108.8
109.6
108.1
101.9
95.6
108.4
108.8 #1.7
Mediam
HalfChectah
42.6
44.0
47.4
42.6
46.9
44.2
49.1 +1.0
Medium
Hopper
52.9
58.5
66.3
67.6
61.1
95.4
58.5
79.3 1:36
Medium
Walker2d
75.3
74.0
79.7
82.5 #:1.4
Med-Replay
HalfChectah
36.6
45.5
44.2
36.6
41.9
40.2
42.2
39.3 1:4.1
Mod-Replay
Hopper
18.1
95
94.7
82.7
91.5
93.6
96.8
100 1.0.7
Med-Replay
Walker2d
26.0
77.2
66.6
82.6
61.2
75 =4.3
Average
51.9
77.6
77
74.7
78.9
75.3
81.8
Mixed
Kitchen Kitchen
51.5
52.4
51
65 #2.88
Partial
38
50.1
46.3
57 ::2.5
Average
44.8
51.2
48.7
61
Table Offline Reinforcement Learning Performance. We show that Decision Diffuser (DD) either matches or outperforms current offline RL approaches on D4RL tasks in terms of normalized average returns (Fu et al., 2020). We report the mean and the standard error over 5 random seeds.
Results Across different offline RL tasks, we find that the Decision Diffuser is either competitive or outperforms many offline RL baselines (Table 1). It also outperforms Diffuser and sequence modeling approaches, such as Decision Transformer and Trajectory Transformer. The difference between Decision Diffuser and other methods becomes even more significant on harder D4RL Kitchen tasks which require long-term credit assignment.
To convey the importance of classifier-free guidance, we also compare with the baseline CondDiffuser, which diffuses over both state and action sequences as in Diffuser without classifier- guidance. In Table 2, observe that condDiffuser improves over Diffuser in 2 out of 3 environments. Decision Diffuser further improves over CondDiffuser, performing better across all 3 environments. We conclude that learning the inverse dynamics is good alternative diffusing over actions. We further empirically analyze when to use inverse dynamics and when to diffuse over actions in Appendix F. We also compare against CondMLPDiffuser, a policy where the current action is denoised according to a diffusion process conditioned on both the state and return. We see that ConcMLPDiffuser performs the worst amongst diffusion models. Till now, we mainly tested on offline RL tasks that have deterministic (or near deterministic) environment dynamics. Hence, we test the robustness of Decision Diffuser to stochastic dynamics and compare to Diffuser and CQL as we vary the stochasticity in environment dynamics, in Appendix G. Finally, we analyze the runtime characteristies of Decision Diffuser in Appendix E.
4.2 CONSTRAINT SATISFACTION
Setup We next evaluate how well we can generate trajectories that satisfy a set of constraints using the Kuka Block Stacking environment (Janner et al., 2022) visualized in Figure 5. In this domain, there are four blocks which can be stacked as a single tower or rearranged into several towers. A constraint like BlockHe git > Bl BlockHe (j) requires that block be placed above block j. We train the Decision Diffuser from 10, 000 expert demonstrations each satisfying one of these constraints. We randomize the positions of these blocks and consider two tasks inference: sampling trajectories that satisfy a single constraint seen before in the dataset or satisfy group of constraints for which demonstrations were never provided. In the latter, we ask the Decision Diffuser to generate > BlockHeight (k) for three of the four
trajectories so Block (()h()) > BlockHeight blocks i, j, k. For more details, please refer to Appendix H.
Results In both the stacking and rearrangement settings, Decision Diffuser satisfies single constraints with greater success rate than Diffuser (Table 3). We also compare with BCQ (Fujimoto et al., 2019) and CQL (Kumar et al., 2020), but they consistently fail to stack rearrange the blocks leading to 0.0 success rate. Unlike these basclines, our method can just as effectively satisfy several constraints together according to Equation 9. For visualization of these generated trajectories, please see the website https://anuragajay wtthub.io/deciiion........
7

---

Hopper-> Diffaser CondDittueer CondML Piiifusee Decisiion Diffuser
Med-Expert
107.6
111.3
105.6
111.8 11.5
Medium
68.5
06.3
54.1
79.3 13.
Roplay
76.5
Table 2: Ablations. Using classifier-free guidance with Diffuser, resulting in CondDiffuser, improves performance (out of 3) environments. Additionally, using inverse dynamics for action prediction Decision Diffuser improves performance in all environments. given the current state and the target return, doesn't It perform as well.
that diffuses over current action
Enviromment
Diffuser
DD
Single Constraint Stacking
45.6 #3.1
58.0 #3.1
Single Constraint Rearrangement
58.9 #3.4
62.7 =3.1
Single Constraint Average
$2.3
60.4
Multiple Constraints Stacking
60.3 1:311
Multiple Constraints Rearrangement
67.2 =331
Multiple Constraints Average
63.8
Block Stacking through Constraint Minimization. Decision Difliser (DD) improves over Diffuser terms of success of generating trajectories satisfying set of block stacking constraints. It can also flexibly combine multiple constraints during test time. report the mean success rate and the standard ervor over 5 random seeds.
Table 3:
Figure Stacking task.
Kuka Block
4.3 SKILL COMPOSITION
Setup Finally, we look at how to compose different skills together. We consider the Unitree-go- running environment (Margolis & Agrawal, 2022), where a quadruped robot can be found running with various gaits, like bounding, pacing. and trotting. We explore if it is possible to generate trajectories that transition between these gaits after only training on individual gaits. For each gait, we collect a dataset of 2500 demonstrations on which we train Decision Diffuser.
Results During testing, we use the noise model of our reverse diffusion process according to equation 9 to sample trajectories of the quadruped robot with entirely new running behavior. Figure 6 shows a trajectory that begins with bounding but ends with pacing. Appendix provides additional visualizations of rumning gaits being composed together. Although it visually appears that trajectories generated with the Decision Diffuser contain more than one gait, we would like to quantify exactly how well different gaits can be composed. To this end, train classifier to predict at every time-step or frame in trajectory the running gait of the quadruped (i.e. bound, pace, or trott). We reuse the demonstrations collected for training the Decision Diffuser also train this classifier, where our inputs are defined as robot joint states over a fixed period of time (i.e. state sub-sequences of length 10) and the label is the gait demonstrated in this sequence. The complete details of our gait classification procedure can be found in Appendix I.
Only Bound
Only Pooe
Bound Pare
Gait Pribabiitt
Condition
Trott
Pace
Bound
Only Bound
0.8
1.0
98.2
Only Pace
1.4
97.7
0.9
Bound + Pace
1.4
38.5
60.1
Bound
Figure 7: Classifying Running Gaits. A classifier predicts the running gait of the quadruped at every timestep. On traeccoories generated by conditioning single skill, like only bounding or pacing, the classifier predicts the respective gait with largest probability. When conditioned on both skills, some timesteps are classified as bounding while others as pacing.
We use our running gait classifier in two ways: to evaluate how the behavior of the quadruped changes over the course of single, generated trajectory and to measure how often each gait emerges over several generated trajectories. Diffuser conditioned either on the bounding gait, the pacing gait, both. For every trajectory, we separately plot the classification probability of each gait over the length of the sequence. As shown in the plots of Figure 7, the classifier predicts bound and pace respectively to be the most likely running gait trajectories sampled with this condition. When the trajectory is generated by conditioning on both gaits, the classifier transitions between predicting one gait with largest probability to the
the former, we first sample three trajectories from the Decision
8

---

Beazd 7ace
?
Figure 6: Composing Movement Skills. Decision Diffuser can imitate individual running gaits using expert demonstrations and compose multiple different skills together during test time. The results are best illustrated by videos viewable at https:/anuragajjyy setgsjjjyyitthh.iooo wwtta//ioociiiiiniiiiiiiiii
other. In fact, there are several instances where the behavior of the quadruped switches between bounding and pacing according the classifier. This is consistent with the visualizations reported in Figure 6. In the table depicted in Figure/ 7, we consider 1000 trajectories generated with the Decision Diffuser when conditioned on one or both of the gaits as listed. We record the fraction of time that the quadruped's 's running gait was classified either trott, pace, or bound. It turns out that the classifier identifies the behavior as bounding for 38.5% of the time and as pacing for the other 60.1% when trajectories are sampled by composing both gaits. This corroborates the fact that the Decision Diffuser can indeed compose running behaviors despite only being trained on individual gaits.
RELATED WORK
Diffusion Models Diffusion Models is proficient in learning generative models of image and text data (Saharia et al., 2022; Nichol et al., 2021; Nichol & Dhariwal, 2021). It formulates the data sampling process as an iterative denoising procedure (Sohl-Dickstein et al., 2015; Ho et al., 2020). be alternatively interpreted as parameterizing the gradients of the data distribution (Song et al., 2021) optimizing the score matching objective (Hyvärinen, 2005) and thus as a Energy-Based Model (Du & Mordatch, 2019; Nijkamp et al., 2019; Grathwohl al., 2020). To generate data samples (eg: images) conditioned on some additional information (eg:text), prior works (Nichol & Dhariwal, 2021) have learned a classifier to facilitate the conditional sampling. More recent works (Ho & Salimans, 2022) have argued to leverage gradients of an implicit classifier, formed by the difference in score functions of a conditional and an unconditional model, to facilitate conditional sampling. The resulting classifier-free guidance has been shown to generate better conditional samples than classifier-base based guidance. Recent works have also used diffusion models to imitate human behavior (Pearce et al., 2023) and to parameterize policy in offline RL (Wang et al., 2022). Janner et al. (2022) generate trajectories consisting of states and actions with an unconditional diffusion model, therefore requiring trained reward function noisy state-action pairs. At inference, the estimated reward function guides the reverse diffusion process towards samples of high-return trajectories. In contrast, we do not train reward functions or diffusion processes separately, but rather model the trajectories in our dataset with a single, conditional generative model. This ensures that the sampling procedure of the learned diffusion process is the same at inference as is during training.
The denoising procedure
Reward Conditioned Policies Prior works (Kumar et al., 2019; Schmidhuber, 2019; Emmons et al., 2021: Chen et al., 2021) have studied learning of reward conditioned policies via reward conditioned behavioral cloning. Chen et al. (2021) used a transformer (Vaswani et al., 2017) to model the reward conditioned policies and obtained a performance competitive with offline RL approaches. Emmons et al. (2021) ) obtained similar performance as Chen et al. (2021) without using transformer policy but relied on careful capacity tuning of MLP policy. In contrast, Decision Diffuser can also model constraints skills and their resulting compositions.
6 DISCUSSION
We propose Decision Diffuser, a conditional generative model for sequential decision making. It frames offline sequential decision making as conditional generative modeling and sidesteps the need of reinforcement learning, thereby making the decision making pipeline simpler. By sampling for high returns, it is able to capture the best behaviors in the dataset and outperforms existing offline RL approaches on standard D4RL benchmarks. In addition to returns, it can also be conditioned on constraints or skills and can generate novel behaviors by flexibly combining constraints or composing skills during test time. In this work, we focused on offline sequential decision making, thus circumventing the need for exploration. Using ideas from Zheng et al. (2022), future works could look into online fine-tuning of Decision Diffuser by leveraging entropy of the state-sequence model for exploration. While our work focused on state based environments, it can be extended to image based environments by performing the diffusion in latent space, rather than observation space, as done in Rombach et al. (2022). For a detailed discussion on limitations of Decision Diffuser, please refer to Appendix I..

---

ACKNOWLEDGEMENTS
The authors would like thank Ofrr Nachum, Anthony Simeonov and Richard Li for their helpful feedback on an earlier draft of the work; Jay Whang and Ge Yang for discussions on classifier- free guidance; Gabe Margolis for helping with unitree experiments; Micheal Janner for providing visualization code for Kuka block stacking; and the members of Improbable Al Lab for discussions and helpful feedback. We thank MIT Supercloud and the Lincoln Laboratory Supercomputing Center for providing compute resources. This research was supported by an NSF graduate fellowship, DARPA Machine Common Sense grant, a MURI grant, an MIT-IBM grant, and ARO INF-211-- 0097.
This research was also partly sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19 2-1000. The views and conclusions contained this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Govermment purposes, notwithstanding any copyright notation herein.
AUTHOR CONTRIBUTIONS
Anurag Ajay conceived the framework of viewing Decision-making conditional diffusion generative modeling. implemented the Decision Diffuser algorithm, ran experiments on Offline RL and Skill Composition, and helped in paper writing.
Yilun Du helped in conceiving the framework of viewing docision-mmaiinn as conditional diffusion generative modeling, ran experiments on Constraint Satisfaction, helped paper writing and advised Anurag.
Abhi Gupta helped in running experiments on Offline RL and Skill Composition, participated in research discussions, and played the leading role in paper writing and making figures.
Joshua Tenenbaum participated in research discussions.
Tommi Jaakkola participated in research discussions and suggested the experiment of classifying running gaits.
Pulkit Agrawal was involved in research discussions, suggested experiments related to dynamic programming, provided feedback on writing, positioning of the work, and overall advising.
REFERENCES
Pulkit Agrawal, Ashvin V Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. Advances in neural information processing systems, 29, 2016.
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010. 2220000.11111.. 136/ 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Newral Information Processing Systems, 2021.
10

---

Robert Dadashi, Shideh Rezacifar, Nino Vieillard, Léonard Hussenot, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning with pseudometric learning, arXiv preprint arXiv:2103.01948, 2021.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy based models. In Advances in Neural Information Processing Systems, 2019.
Gabriel Dulac-Amold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 1100))2419-2468, 2021.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:2112.00751, 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, 2019.
Dibya Ghosh, Anurag Ajay, Pulkit Agrawal, and Sergey Levine. Offline rl policies should be trained to adaptive. In International Conference on Machine Learning, pp. 7513-7530. PMLR, 2022.
Will Grathwohl, Kuan-Chieh Wang. Joern-Henrik Jacobsen, David Duvenaud, and Richard Zemel. Learning the stein discrepancy for training and evaluating energy-based models without sampling. In International Conference on Machine Learning, 2020.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Picter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. Advances in neural information processing systems, 31, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, 2018
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12596 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching, Journal of Machine Learning Research, 2005.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems, 2021.
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, 2022.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model- based offline reinforcement learning. In Advunces in Neural Information Processing Systems, 2020.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696-21707, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Ilya Kostrikov, Jonathan Tompson, Rob Fergus, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. arXiv preprint arXiv:2 103. 103.08050 08050, 2021.
11

---

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Qeearning. In International Conference on Learning Representations, 2022.
Aviral Kumar, Xue Bin Peng, and Sergey Levine. Reward-conditinedd policies. arXiv preprint arXiv:1912.13465, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. A workflow for offline model-free robotic reinforcement learning. arXiv preprint arXiv:2 10813, 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint orXiv:2005.01643, 2020.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206. 14858, 2022.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 2015.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. arXiv preprint urXiv:2206.01714 2022.
Calvin Luo. Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208. 11970, 2022.
Gabriel Margolis and Pulkit Agrawal. Walk these ways: Gait-conditioned policies yield diversified quadrupedal agility. In Conference on Robot Learning, 2022.
Diganta Misra. Mish: A self regularized non-monotonic neural activation function. In British Machine Vision Conference, 2019.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew,
Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 /2. 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning. 2021.
Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non- persistent short-run MCMC toward energy-based model. In Advances in Neural Information Processing Systems, 2019.
Pedro A Ortega, Markus Kunesch, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Joel Veness, Jonas Buchli, Jonas Degrave, Bilal Piot, Julien Perolat, al. Shaking the foundations: delusions in sequence models for interaction and control. arXiv preprint arXiv:2110.. 10819, 2021.
Keiran Paster, Sheila MclIraith, and Jimmy Ba. You can't count on luck: Why decision transformers fail in stochastic environments. arXiv preprint arXiv:2205.1590 2022.
Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A Efros, and Trevor Darrell. Zero-shot visual imitation. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 2050-2053, 2018
Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al. Imitating human behaviour with diffusion models. arXiv preprint 2rXiv:2301.10677 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.066 06/25, 2022.
12

---

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-imee to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022.
Juergen Schmidhuber.
Reinforcement learning upside down: Don't predict rewards-juue map them to 2019.
actions. arXiv preprini
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In International Conference Machine Learning, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference Learning Representations, 2021.
Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 1988.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Russ Tedrake. Underactuated Robotics. 2022. URL /tRL:tttpp/undeeactttteee..
edu.
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint 1 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
Adam R Villaflor, Zhe Huang, Swapnil Pande. John M Dolan, and Jeff Schneider. Addressing optimism bias in sequence modeling for reinforcement learning. In International Conference on Machine Learning, pp. 22270-22283, PMLR, 2022.
Homer Walke, Jonathan Yang, Albert Yu, Aviral Kumar, Jedrzej Orbik, Avi Singh, and Sergey Levine. Don't start from scratch: Leveraging prior data to automate robotic reinforcement learning. arXiv preprint arXiv:2207.04703, 2022.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as expressive policy class for offline reinforcement learning, arXiv preprint arXiv:2208.0619 2022.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1rXiv:1911.1166/ 2019.
Yuxin Wu and Kaiming He. Group normalization. In European Conference on Computer Vision, 2018.
Mengjiao Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum. Dichotomy of control: Separating what you can control from what you cannot. arXiv preprint arXiv:2210. 13,335, 2022.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 1t..::1710.094000 2017.
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. arXiv preprint arXiv:2202.05607, 2022.
13

---

Appendix
In this appendix, we discuss details of the illustrative examples in Section A. Next, discuss hyperparameters and architectural details in Section B. We analyze the importance of low temperature sampling in Section C, further explain composition of conditioning variable in Section D, discuss the run-time characteristics of decision diffuser in Section E, discuss when to use inverse dynamics in Section F and analyze robustness of Decision Diffuser stochastic dynamies Section G. Finally, we provide details of the Kuka Block Stacking environment in Section H and the Unitree environment in Section I.
A ILLUSTRATIVE EXAMPLES
A.1 IMPLICIT DYNAMIC PROGRAMMING
Training dataset
Generation
Figure Al: Illustrative example. We demonstrate the ability of Decision Diffuser to stitch together sub- optimal trajectories in training dataset to obtain (near) optimal trajectories, therehy implicitly performing dynamic programming in Maze2D-open environment from Fu al. (2020).
We empirically demonstrate the ability of Decision Diffuser to perform implicit dynamic programming in Maze2D-open environment from Fu et al. (2020). The task in Maze2D-open environment is to reach point C and the reward is negative distance from point C. The training dataset consists of 500 trajectories from point A to point and 500 trajectories from point B to point C. The maximum trajectory length is 50. During test time, the agent starts from point A and needs reach point C as quickly as possible. As shown in Figure A1, Decision Diffuser can stitch trajectories in training dataset form trajectories that goes from point A to point in (near) straight lines.
A.2 CONSTRAINT COMBINATION
Setup In linear system robot navigation, Decision Diffuser is trained on 1000 expert trajectories either satisfying the constraint ||sT S R (R = 1) the constraint || 2
0.7). Here, 8T = [ yr] represents the final robot state in a trajectory, specifying its final 2d position. The maximum trajectory length 50. During test time. Decision Diffuser is asked to generate trajectories satisfying || < R and ||sT|| 2 r to test its ability to satisfy single constraints. Furthermore, Decision Diffuser is also asked to generate trajectories satisfying r S |T S to test its ability to satisfy combined constraints.
Results Figure shows that Decision Diffuser learns to generate trajectories perfectly (i.e. with 100% success rate) satisfying single constraints in linear system robot navigation. Furthermore, it learns to generate trajectories satisfying the composed constraint in linear system robot navigation with 91. 91..% accuracy where the standard error is calculated over 5 random seeds.
B HYPERPARAMETER AND ARCHITECTURAL DETAILS
In this section, we describe various architectural and hyperparameter details:
We represent the noise model with a temporal U-Net (Janner et al., 2022), consisting of a U-Net structure with 6 repeated residual blocks. Each block consisted of two temporal convolutions, each followed by group norm (Wu & He, 2018), and final Mish nonlinearity (Misra, 2019). Timestep and condition embeddings, both 128-dimensional vectors, are produced by separate 2-layered MLP (with 256 hidden units and Mish nonlinearity) and concatenated together before getting added to the activations of the first temporal convolution within each block. We borrow the code for temporal U-Net from
14

---

We represent the inverse dynamics fo with 2-layered MLP with 512 hidden units and ReLU activations.
We represent the gait classifier with 3 3-layered MLP with 1024 hidden units and ReLU activations.
We train Eg and fo using the Adam optimizer (Kingma & Ba, 2015) with a learning rate of 2e 4 and batch size of 32 for 2e6 train steps.
We train the gait classifier using the Adam optimizer with a learning rate of and batch size of 64 for 1e6 train steps.
We choose the probability of removing the conditioning information to be 0.25.
We use 100 diffusion steps.
We use a planning horizon of 100 in all the D4RL locomotion tasks, 56 in D4RL kitchen tasks, 128 in Kuka block stacking, 56 in umitree-o--runnig tasks, 50 in the illustrative example and 60 in Block push tasks.
We use guidance scale {1.2, 1.4, 1.6, 1.8) but the exact choice varies by task.
We choose = 0.5 for low temperature sampling.
We choose context length C 20.
C IMPORTANCE OF LOW TEMPERATURE SAMPLING
In Algorithm 1, compute 14p-1 and - from noisy sequence of states and predicted noise. We find that sampling I ~ N a. (where a E [0, 1)) with a reduced variance produces high-likelihood state sequences. We refer to this low-temperature sampling. To empirically show its importance, we compare performances of Decision Diffuser with different values of a (Table Al). We show that low temperature sampling (a 0.5) gives the best average returns. However, reducing the a to 0 eliminates the entropy in sampling and leads to lower returns. On the other hand, a 1.0 leads a higher variance in terms of returns of the trajectories.
Decision Diffuser
Hopper-Medium-
104.3 0.7
a = 0.5
111.8 11.6
1.0
107.1 1: 3.5
Table Al: Low-temperature sampling (a 0.5) allows us get high return trajectories consistently.
While = 1.0 leads to higher variance in returns of the trajectories, 0.0 eliminates entropy in the sampling and leads to lower returns.
D COMPOSING CONDITIONING VARIABLES
In this section, we detail how Decision Diffuser trained with different conditioning variables (r))) composes these conditioning variables together. It learns the denoising model co(*x(T), (T)), k) for given conditioning variable y()).. From the derivations outlined prior works (Luo, 2022; Song al., 2021), we know that Væz(7) log (()((T)) y (7()) xx -co(zx(T), ()),).)) Therefore, each conditional trajectory distribution (Q(xx(7)))) (T) can be modelled with single denoising model eo that conditions the respective variable 3 (T).
In order to compose n different conditioning variables (i.e. skills or constraints), we would like model ( (T) (7)) ). We assume that (7) conditionally independent given
15

---

xx((7 (T). Thus, can factorize follows:
(T) (())))))))))))))))(())) ((
(Bayes Rule)
g( (T))
logg(>> (r) (og
(log ((x))(())))) ((o)((()))) log (xgg(2x())) ()
logg(()
(r)(r) log logg((>(7))
V ( log (og (og
co( k) = cα (T),
(to( (,(()(())))))))))))ccc(()(()))))
(o,
Using the above equations, we can sample from using the perturbed noise:
with classifier free guidance
é =
((eo(a (2 (x(x)))))))))
(r), + Σ (co( cr(
-o(((())) 0,):)
We use the perturbed noise to compose skills combine constraints at test time. This derivation was borrowed from Liu et (2022) and is presented here for completeness.
While the composition of conditioning variables {T'( requires them to be conditionally independent given state trajectory (T()), we empirically observe that this condition doesn't 't have to be strictly satisfied. However, we require composition of conditioning variables to be feasible (i.e. (r( that satisfies all the conditioning variables). When the composition infeasible, Decision Diffuser produces trajectories with incoherent behavior, expected. This is best illustrated by videos viewable at https:/aaurrggajaygiilll itthubio/decisiin -ifffrciiionndffuser....
Requirements on the dataset First, the dataset should have diverse set of demonstrations that shows different ways of satisfying each conditioning variable y'(r). This would allow Decision Diffuser to learn diverse ways of satisfying each conditioning variable y'(r). Since we use inverse dynamics to extract actions from the predicted state trajectory (T))), we assume that the state trajectory æo(r) resulting from the composition of different conditioning variables contains consecutive state pairs (St, St+1) that come from the same distribution that generated the demonstration dataset. Otherwise, inverse dynamics can give erroneous predictions.
E RUNTIME CHARACTERISTIC OF DECISION DIFFUSER
We analyze the runtime characteristics of Decision Diffuser in this section. After training the Decision Diffuser on trajectories from the D4RL Hopper-Mediun-EExpe environment according to Algorithm 1. Every action taken in the environment requires running 100 reverse diffusion steps generate state sequence taking on average 1.26s in wall-clock time. We can improve the run-time of planning by warm-starting the state diffusion as suggested in Janner et al. (2022). Here, we start with a generated state sequence (from the previous environment step), run forward diffusion for fixed number of steps, and finally run the same number of reverse diffusion steps from the partially noised state sequence to generate another state sequence. Warm-starting in this way allows us decrease the number of denoising steps to 40 (0.48s average) without any loss in performance, to 20 (0.21s on average) with minimal loss in performance, and to with less than 20% loss in performance (0.06s average). We demonstrate the trade-off between performance, measured by normalized average return achieved the environment, and planning time, measured in wall-clock time after warm-starting the reverse diffusion process, in Figure A2.
dataset, we plan in the corresponding
16

---

Performance vs, Run-time
Normalized Return
100
90
10
10
Average Planning Time (seconds)
Figure A2: Performance vs planning time. We visualize the trade-off between performance, measured by normalized average return achieved in the environment, planning time, measured in wall-clock time after warm-starting the reverse diffusion process.
Environment
BC
CondDiffuser
Decision Diffuser
Position Control
57.3 #1.2
87.3 #3.1
87.8 =2.8
Torque Control
55.2 #1.5
71.8 #3.4
84.7 22.2
C
Table A2: Block pushing with different controls. Decision Diffuser and CondDiffuser perform similarly when the agent uses position control. However, when the agent uses torque control, CondDiffuser performs worse than Decision Diffuser given it's harder diffuse action trajectories. We the success rate of the red cube reaching the green circle as the performance metrie. report mean success rate and the standard error over 5 random seeds.
non-smooth
Figure A3: Block push environment.
F WHEN TO USE INVERSE DYNAMICS?
In this section, we try to analyze further when using inverse dynamics better than diffusing over actions. Table 2 showed that Decision Diffuser outperformed CondDiffuser on 3 hopper environment, thereby suggesting that inverse dynamics is better alternative to diffusing over actions. Our intuition that sequences
actions, represented joint torques in environments, tend to be more high-frequency and less smooth, thus making it harder for the diffusion model to predict (Kingma al., 2021). We now try verify this intuition empirically.
Setup We choose Block Push environment adapted from Gupta et al. (2018) where the goal is to push the red cube the green circle. When the red cube reaches the green circle, the agent gets reward of +1. The state space is 10-dimensional consisting of joint angles (3) and velocities (3) of the gripper, COM of the gripper (2) and position of the red cube (2). The green circle's position is fixed and at an initial distance of 0.5 from COM of the gripper. The red cube (of size 0.03) is initially distance of 0.1 from COM of the gripper and an angle O sampled from L1( T/4, 4, π /4) the start of every episode. The task horizon is 60 timesteps.
There are 2 control types: (i) torque control, where the agent needs specify joint torques (3 dimensional) and (ii) position control where the agent needs to specify the position change of COM of the gripper and the angular change in gripper's orientation (Ax, Ay, AΦ) (3 dimensional). While action trajectories from position control are smooth, the action trajectories from torque control have higher frequency components.
Offline dataset collection To collect the offline data, we use Soft Actor-Critic (SAC) (Haarnoja et al., 2018) first to train an expert policy for million environment steps. We then use million environment transitions the end of the training and random action trajectories collected at the beginning of the training. We collect 2 datasets, for each control type.
offline dataset, which contains expert trajectories collected towards
17

---

Results Table A2 shows that Decision Diffuser and CondDiffuser perform similarly when the agent uses position control. This because action trajectories resulting from position control are smoother and hence easier to model with diffusion. However, when the agent uses torque control, CondDiffuser performs worse than Decision Diffuser, given the action trajectories have higher frequency components and hence are harder to model with diffusion.
G ROBUSTNESS TO STOCHASTIC DYNAMICS
p
BC
Decision Diffuser
Diffuser
CQL
0.00
15.2
84.7:2.2
12.4+1.4
73,242.3
0.05
49.
77.3+3..
63. 63.242.9
61.8:3.7
0.10
25.843.8
53.2:14.11
52,31:466
51.2
0.15
15.114.3
41.3::4.9
41.61:.11
42.2
Table A3: Robustness to stochastic dynamics. Decision Diffuser's performance suffers when stochasticity is introduced in dynamics function. While it still outperforms Diffuser and CQL when p 0.05, its performance becomes similar that Diffuser and CQL for higher p values. We the the green circle as performance metric. report the mean success rate the standard error over random seeds.
of the red cuhe reaching
We empirically analyze robustness of Decision Diffuser stochasticity in dynamics function.
Setup We use Block Push environment, described in Appendix F, with torque control. However, inject stochasticity into the environment dynamics. For every environment step, either sample a random action from 4d[-1, -1, 1], [1, 1, 1l) with probability p or execute the action given by the policy with probability (1 -1--7 p). We use p E {0, 0.05,.0,00 0.1, 0.15) our experiments.
Offline dataset collection We collect separate offline datasets for different block push environments, each characterized by different value of p. Each offline dataset consists of million environment transitions collected using the method described in Appendix F.
Results Table A3 characterizes how the performance of BC, Decision Diffuser, Diffuser, and CQL changes with increasing stochasticity in the environment dynamics. We observe that the Decision Diffuser outperforms Diffuser and CQL for = 0.05, however all methods including the Decision Diffuser settle to similar performance for larger values of p.
Several works (Paster et al., 2022; Yang et al., 2022) have shown that the performance of return- conditioned policies suffers as the stochasticity in environment dynamics increases. This is because the return-conditione policies aren't ' able to distinguish between high returns from good actions and high returns from environment stochasticity. Hence, these stuum-conditioned policies can learn sub-optimal actions that got associated with high-return trajectories in the dataset due to environment stochasticity. Given Decision diffuser return conditioning generate actions in offline RI, its performance also suffers when stochasticity in environment dynamics increases.
Some recent works (Yang et al., 2022: Villaflor al., 2022) address the above issue by learning latent model for future states and then conditioning the policy on predicted latent future states rather than returns. Conditioning Decision Diffuser on future state information, rather than returns, would make it more robust to stochastic dynamics and could be an interesting avenue for future works.
H KUKA BLOCK STACKING
In the Kuka blocking stacking environment, the underlying goal is to stack set of blocks top of each other. Models have trained on a set of demonstration data, where a set 4 blocks are sequentially stacked top of each other to form block tower.
We construct state-space plans of length 128. Following (Janner et al., 2022), we utilize close-loop controller generate actions for each state in state-space plan (controlling the 7 degrees of freedom in joints). The total maximum trajectory length plan in Kuka block stacking is 384, We detail differences between the two consider conditional stacking environments below:
18

---

Stacking In the stacking environment, at test time wish to again construct tower of four blocks.
Rearrangement In the rearrangement environment, at test time wish to stack blocks in configuration where a set of blocks are above a second set. This set of stack-place relations may not precisely correspond to single block tower (can instead construct two block towers), making this environment an out-of-distribution challenge.
In addition to Diffuser (Janner et al., 2022). we used goal-conditioned variants of CQL (Kumar et al., 2020) and BCQ (Fujimoto et al., 2019) baselines for the block stacking and rearrangement with single constraint. However, they get a success rate of 0.0.
I UNITREE GO RUNNING
We consider Unitree-go-He go-running environment (Margolis & Agrawal, 2022) where a quadruped robot runs in 3 different gaits: bounding, pacing, and trotting. The state space is 56 dimensional, the action space is 12 dimensional, and the maximum trajectory length is 250.
As described in Section 4.3, we train Decision Diffuser on expert trajectories demonstrating individual gaits. During testing, we compose the noise model of our reverse diffusion process according to equation This allows us to sample trajectories of the quadruped robot with entirely new running behavior. Figures A4.A5,A6 shows the ability of Decision Diffuser to imitate bounding, trotting and pacing and their combinations.
1.1 QUANTITATIVE VERIFICATION OF COMPOSITION
We now try to quantitatively verify whether the trajectories resulting from composition of gaits does indeed contain only those gaits.
Setup We learn gait classifier that takes in sebusece of states (of length 10) and predicts the gait-ID. It is represented by 3-layered MLP with 1024 hidden units and ReLU activations that concatenates the sub-sequence of states (of length 10) into single vector of dimension 560 before taking in as an input. We train the gait classifier on the demonstration dataset. To ensure that the learned classifier use MixUp-style (Zhang et al., 2017) data augmentation during training. We create a synthetic sub- sequence of length 10 by concatenating two sampled sub-sequence (from the demonstration dataset) of length 1, and 15 (where 4 + Ij 10) from gaits with ID and j and give it label N. one-hot(i)
predict gait-ID trajectories generated by the composition of skills,
1 ho)- During training, we sample sub-sequence from the demonstration dataset with 4,47
70% probability and a sythenthic sub-sequence with 30% probability. We train the classifier for 2e6 train steps with learning rate of 2e and batch size of 64.
Figures A4.A5.A6 show that the classifier's prediction is consistent with the visualized composed trajectories. Furthermore, we use Decision diffuser to act in the environment and generate 1000 trott trajectories, 1000 pace trajectories, 1000 bound trajectories, and 1000 composed trajectories for each possible pair of individual gaits. We then evaluate the learned gait classifier on these trajectories and compute the percentage of timesteps a particular gait has the highest probability. From Figures A4,A5,A6, we can see that if trajectories are generated by the composition of two gaits, then those two gaits will have the two highest probabilities across different timesteps in those trajectories.
Results
1.2 A SIMPLE BASELINE FOR COMPOSITION
Let one-hot(i) and oot-ho represent two different gaits that can be generated using noise models (0(xx(7), one-hot(i), (i,,) k) and eø E)(xx(T), one-hot(j),, respectively. To compose these gaits, we compose the above-mentioned noise models using equation 9. As an alternative, model eo (o(xx(r)) (T), one mot(hot(i) + one-hot(j), k) can lead to composed gaits. However, we observe that ea (T), one-hot(i) + noethhotj) k)) catastrophically fail to generate any gait (see videos at https://anuragajjy github.tihoooo wedddcciioondiffusseee one-hot(i) + one-hot(j) was never seen by the noise model eg during training.
see if the noise
This happens because the condition variable
19

---

Trott
Pace
Trott
Pace
Only Trott
Only Pare
Trett Pace
Cast Probakalze
Condition
Trott
Pace
Bound
Only Trott
97.2
1.7
1.1
Only Pace
0.8
98.1
1.1
Trott + Pace
55.6
43.6
0.8
Text
Boand
Figure A4: Composing Trott and Pace. Decision Diffuser can imitate individual running gaits using expert demonstrations and compose multiple different skills together during test time. The results are hest illustrated by videos viewable at https://anuragajay..a http:jaayggiuubo/dcciiiiiiooo -diffudiiiiio.nfffsrr....
Trott
Bound
Trott
Bound
Only Trott
Only Bound
Trott Bound
Gnit Paohabiity
Condition
Trott
Pace
Bound
Only Trott
96.4
2.2
1.4
Only Bound
1.6
0.6
97.8
Trott + Bound
51.3
0.9
47.8
Figure A5: Composing Trott and Bound. Decision Diffuser can imitate individual running gaits using expert demonstrations and compose multiple different skills together during test time. The results are best illustrated by videos viewable at https://anuragajjy withry.gtttbbbhoooo iotdeiiioo/ diffuser/.
J NOT COMPOSITIONS WITH DECISION DIFFUSER
Decision diffuser also support "NOT" composition. Suppose wanted to sample from be the set of all conditioning variables. Then, following ([ y (T)). Let {y (
derivations from Liu et al. (2022) and using 8 = we can sample from ((<x(r))NO) y (T)) using the perturbed noise:
ē= co (T) w Σ (co(F (T), (T),) co(2x(7),
<(co(=x(7), (T), ( (T), 0,k)))
We demonstrate the ability of Decision Diffuser support "NOT" composition by using it to satisfy constraint of type (lockHeight(i) > BlockHeight(() AND (NOT BlockHeight(() > BlockHeiight(i) in Kuka block stacking task. visualized in videos at https://anuragajaaygiteee wwtphhbbooodcciiiooooooooo explicit density estimate for each skill, it can't natively support OR composition.
As the Decision Diffuser does not provide
20

---

Bound
Pace
Bound
Pace
Only Bound
Only Pare
Bound Pare
Cast Probakalze
Condition
Trott
Pace
Bound
Only Bound
0.8
1.0
98.2
-20
Only Pace
1.4
97.7
0.9
Bound + Pace
1.4
38.5
60.1
Text
Boand
Figure A6: Composing Bound and Pace. Decision Diffuser can imitate individual running gaits using expert demonstrations and compose multiple different skills together during test time. The results are hest illustrated by videos viewahle at http:/aurrragajgggguubbiibooooo -iifffiiiccissoooiiissrr.r. diffuser//
K COMPARING Q-FUNCTION GUIDED DIFFUSION AND CLASSIFIER-FRES GUIDED DIFFUSION
Classifier-fiee free guided diffusion and Q-value guided diffusion are theoretically equivalent. However, as noted several works (Nichol et al., 2021; Ho Salimans, 2022; Saharia et al., 2022), classifier-free guidance performs better than classifier guidance (i.e. Q function guidance in our case) in practice. This is due to following reasons:
Classifier Classifier-guided guided diffusion models learns an unconditional diffusion model along with classifier (Q-function in our case) and uses gradients from the classifier to perform conditional sampling. However, the unconditional diffusion model doesn't 't need to focus on conditional modeling during training and only cares about conditional generation during testing after it has been trained. In contrast, classifier-free guidance relies on conditional diffusion model to estimate gradients of the implicit classifier. Since the conditional diffusion model, learned when using classifier-free guidance, focuses on conditional modeling during train time, it performs better conditional generation during test time.
Q function trained distribution actions given any state. This problem has been extensively studied in offline RL literature (Kumar al., 2020; Fujimoto et al., 2019; Levine al., 2020). In online RL, this issue is automatically corrected when the policy acts in the environment, thinking an action be good but then receives low reward for it. In offline RL, this issue can't be corrected easily; hence, the learned Q-function can often guide the diffusion model towards ouist--diitribution actions that might be sub-optimal. In contrast, classifier-free guidance circumvents the issue of learning a Q-function and directly conditions the diffusion model on returns. Hence, classifier-fe free guidance doesn't 't suffer due to errors learned Q-functions and hence performs better than Q-function guided diffusion.
offline dataset erroneously predict high Q values for out-of-
L LIMITATIONS OF DECISION DIFFUSER
We summarize the limitations of Decision Diffuser:
No partial observability Decision Diffuser works with fully observable MDPs. Naive extensions to partially observed MDPs (POMDPs) may cause sell-delussons (Ortega et al., 2021) in Decision Diffuser. Hence, extending Decision Diffuser to POMDPs could be an exciting avenue for future work.
21

---

Inability to explore the environment and update itself in online setting In this work, focused on offline sequential decision making, thus circumventing the need for exploration. Using ideas from Zheng et al. (2022), future works could look into online fine-tuning of Decision Diffuser by leveraging entropy of the state-sequènce model for exploration.
Experiments on only state-based environments While our work focused on state based environments, it can be extended to image based environments by performing the diffusion in latent space, rather than observation space, as done in Rombach et al. (2022).
Only AND and NOT compositions are supported Since Decision Diffuser does not provide an explicit density estimate for each condition variable, it can't natively support OR composition.
Performance degradation in environments with stochastic dynamies environments with highly stochastic dynamics, Decision Diffuser loses its advantage and performs similarly Diffuser and CQL. To tackle environments with stochastic dynamics, recent works (Yang et al., 2022; Villaflor et al., 2022) propose learning latent model for future states and then conditioning the policy on predicted latent future states rather than returns. Conditioning Decision Diffuser future state information, rather than returns, would make it more robust to stochastic dynamics and could be an interesting avenue for future works.
Performance in limited data regime Since diffusion models are prone to overfitting in case of limited data, Decision Diffuser is also prone to overfitting in limited data regime.
22


Reviews:
[{"review_id": "vHNcS_jb1F", "reviewer": "Reviewer_87NB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents Decision Diffuser, a conditional generative model for sequential decision making. It frames offline sequential decision making as conditional generative modeling by considering  two other variables: constraints and skills. Conditioning on a single constraint or skill during training leads to behaviors at test-time that can satisfy several constraints together or demonstrates a composition of skills. Experiments are conducted on a couple of different decision making tasks.", "summary_of_the_review": "This paper is well motivated and easy to follow. Inspired from diffusion models in vision domain, this paper formulate the decision making process as a condition generation problem and which can naturally achieved by leveraging diffusion models. I like the way that this paper very clearly introducing the technical background and formulate the problem. From the experimental results, the proposed Decision Diffuser is promising on a couple of evaluating tasks.\nMy main concern is the limited technical novelty. The key contribution of this work is to formulate the decision making problems as a conditional generation problem. Based on this, this paper train a diffusion model on offline datasets. However, it is more like an application of diffusion models on decision making tasks. I don't see obvious novelty from either model design and/or training objectives.\nThe evaluations are weak. It only evaluate the DD variants on a few tasks. No comparisons of DD with the other state-of-the-art approaches are shown. Also, no ablation and discussion are shown. It is not convincing to me without such extensive study.\nIn addition, it did not submit the source code, so the reproducibility is hard to be validated. ", "strength_and_weaknesses": "Pros:\nThis paper is well motivated and easy to follow. Inspired from diffusion models in vision domain, this paper formulate the decision making process as a condition generation problem and which can naturally achieved by leveraging diffusion models. I like the way that this paper very clearly introducing the technical background and formulate the problem. From the experimental results, the proposed Decision Diffuser is promising on a couple of evaluating tasks.\n\nCos:\nMy main concern is the limited technical novelty. The key contribution of this work is to formulate the decision making problems as a conditional generation problem. Based on this, this paper train a diffusion model on offline datasets. However, it is more like an application of diffusion models on decision making tasks. I don't see obvious novelty from either model design and/or training objectives.\nThe evaluations are weak. It only evaluate the DD variants on a few tasks. No comparisons of DD with the other state-of-the-art approaches are shown. Also, no ablation and discussion are shown. It is not convincing to me without such extensive study.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hyN4gooGEz", "reviewer": "Reviewer_pyhf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Decision Diffuser, a diffusion-based model for sequential decision making where only the states of a trajectory are modeled and an inverse dynamics model is used to predict actions. Classifier free guidance is used to bring in conditional information, in the form of maximizing returns, satisfying constraints or composing together skills. This is used with low-temperature sampling at inference time yielding improved results. The proposed method is evaluated to generate trajectories for return maximization with offline reinforcement learning datasets. Kuka Block stacking and Unitree-go-running environments are used to assess the performance of generated trajectories in constraint satisfaction and skill composition tasks, respectively.\n", "summary_of_the_review": "The paper explores diffusion models for planning, which is a timely contribution since diffusion models are beating state of the art in various problems. It offers incremental advancements over the state of the art in this domain however the proposed advancements are shown to yield superior performance on various tasks. I think this is a good paper and I recommend acceptance.\n", "strength_and_weaknesses": "Strengths:\n- The paper is well written. Diffusion modeling for planning is an exciting and timely contribution given that diffusion models are beating state of the art in various other tasks in synthesis.\n- The proposed technique is shown to tackle compositionality of skills and constraints at inference time, which is interesting.\n\nWeaknesses:\n- One limitation of diffusion models is that the generations are slow. It would be great if authors can comment on the runtime characteristics of the proposed model.\n- It would be good to discuss limitations of the model. Some questions I have: \n  * How are the out-of-distribution generalization characteristics of the model? \n  * Can the model capture variable length trajectories? \n  * I am intrigued by the skill composition idea. Currently only “AND” style composition is demonstrated. What other compositions does this method can support?\n  * How does the method perform with limited data?\n\nOther comments:\n- The loss function used to train the model combines losses which are used to train the diffusion model and inverse dynamics models equally. What implications does equal weighting have?\n- The paper relies on diffusion models and class conditioning capabilities of such models (i.e. classifier free guidance) but I found it odd that the modeling choice is not mentioned in the abstract. \n- What is the performance metric used in Figure 4? Similarly, Table 3 is lacking an explanation of the metric used. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SshJMBuVSRt", "reviewer": "Reviewer_9svn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "**Update after rebuttal** I am very happy to see a very extensive and detailed authors' rebuttal with a number of additional evaluations and results. Most of my questions have been answered and my criticism has been addressed to a sufficient degree. Though there could be some detailed follow-up discussions, I think overall the paper is now ready for acceptance - I am on the fence between a 6 and an 8, but am very slightly leaning towards the latter, so I will raise my score from 5 to 8.\n\nThe paper explores the use of a diffusion model to generate reward-conditioned state-trajectories to reach goals and satisfy constraints in MDPs given a large set of state-action-reward trajectories (solving decision-making via conditional generative modeling). In contrast to previous work on synthesizing high-expected-return policies from expert data, this paper uses a classifier-free guided diffusion model. Additionally, the paper proposes to diffuse over state-trajectories only, rather than state-reward trajectories; which is empirically shown to lead better policies but requires learning an inverse dynamics model. The method is compared against a number of state-of-the-art competitors on two standard benchmarks (D4RL tasks, and Kuka Block stacking tasks). Finally, the paper shows how conditional generation of trajectories can not only be used for conditioning on high-rewards but also other subsets of training trajectories which either satisfy certain constraints or belong to a certain type of skill (such as a particular gait for running).", "summary_of_the_review": "The paper makes some incremental, but sensible improvements to previously proposed work - and the empirical results justify the modifications. While the main ingredients that went into the improvements have been reported before, the particular combination is novel. To me personally, the paper could be more impactful and of higher quality by justifying the introduced changes beyond mere overall performance on some benchmarks and discussing advantages and disadvantages in detail (including some insight into when and why the introduced modifications are beneficial compared to the previous method). Another novelty introduced is the possibility to combine constraints/skills - this is an interesting idea, but currently it is explored rather superficially, a deeper exploration (as suggested earlier in my review) would make the work stronger. Overall I am quite on the fence for this paper - while there is a clear and sensible main idea and empirical results are in favor of the idea, the paper also has a bit of a flavor of “we tried something and it worked, but we didn’t really look into it any further”. I think that’s potentially a missed chance, since the paper could be quite strong with a bit more work and attempting to answer some of the harder questions. I am therefore currently voting for a borderline rejection, and would not be upset if the paper got accepted - but to me personally the paper is currently on  the level of a very well written and very strong workshop contribution; promising main results but a bit of a lack in depth and clear and well-understood main findings. I am happy to reconsider my final verdict in light of the other reviews and author discussion.", "strength_and_weaknesses": "**Main contributions, Impact**\n1) Use of classifier-free guided diffusion. Classifier-guided diffusion requires a Q-value estimation procedure such that Q-values can be used to guide diffusion (which has been explored in Janner et al. 2022). The current work avoids this via classifier-free guidance which requires low-temperature sampling in datasets where demonstrations are of mixed quality. Empirically, this leads to improvements over previous methods - with a grain of salt it seems to be easier to pick out the high performing trajectories than learning good Q-value estimates. Impact: low to medium - it remains unclear under what exact conditions (both theoretically and empirically) classifier-free guidance / low-temperature sampling outperforms Q-value guidance and why.\n\n2) Diffusion of state-trajectories only, rather than state-action trajectories (which is compared to in the paper and shown to perform worse). This comes at the cost of requiring a good inverse dynamics model. Current impact: low - similar to 1) the empirical results are in favor of the version proposed in the paper but it remains unclear what the practical cost of requiring the inverse dynamics model is; in particular how robust the method is against errors and imperfections in the inverse dynamics estimate (is performance very brittle or quite robust). Under what conditions is it better to rely on an approximate inverse dynamics model, and when is it better to diffuse state-action trajectories?\n\n3) The paper shows how to incorporate combinations of constraints and skills: by identifying subsets of training trajectories corresponding to an individual constraint/skill and augmenting them with a label that can later be used to condition on. The approach is fairly straightforward, but interesting. Impact: low - the approach is currently only minimally explored in the paper and further conceptual/theoretical characterization of the kinds of constraints and the corresponding requirements for the dataset would be needed for higher impact.\n\n**Strengths**\n *  Very well written paper\n *  Empirical results show benefits of proposed method compared to high-quality SOTA methods on standard benchmarks\n * Incorporating constraints and combining skills via constraints is interesting.\n\n**Weaknesses**\n *  Often the paper’s main aim seems to be to reach the minimal result sufficient for publication, which is mainly to achieve good benchmark results. While this is a valid strategy to avoid risks, it also creates a sense of lack of ambition and depth. I personally think there are some very interesting ideas presented in the paper but they often seems to be addressed in a minimal sense. Beyond the empirical comparisons, a strong characterization of the advantages and disadvantages of these ideas is missing; and I would be very excited to see such a discussion and analysis (both empirically and theoretically). I think it would help raise the significance of each of the main contributions mentioned above.\n * To make the point above more concrete: \n   * It remains unclear when precisely and why classifier-free guidance and state-prediction only is preferable - when and why is it beneficial to avoid estimating Q-values at the cost of requiring an inverse dynamics model (how robust is each approach to errors in the approximate Q-values/dynamics, what are the conditions w.r.t. the dataset that make low-temperature sampling a good choice, etc).\n   * It remains unclear what exact conditions are needed for constraints/skills to be “combinable” (see comment under improvements).\n * The current experiments on combining skills are a bit hard to interpret - while they lead to somewhat visually different gaits it is unclear whether these should be considered successful combinations of skills. What’s missing is a clearly stated and (quantitatively) measurable goal for combining skills (which is well defined in the case of combining constraints).\n\n**Improvements**\n1) Naively it seems that the approach introduced allows to combine individual constraints / skills, as long as there is a separate subset of training trajectories for each constraint/skill and the combination at test time is a conjunction of constraints such that the intersection of the corresponding sets of trajectories constitutes a set of valid solutions (which corresponds to the situation in Fig 1). Is this a hard condition? Can something theoretical be said about the allowed combinations of individual constraints (conjunction, disjunction, exclusive disjunction)? What happens theoretically and empirically if satisfying a combination of constraints requires novel trajectories that have no support in the training data (i.e. such that simply “filtering” out the right training trajectories via conditioning and some mild generalization is not sufficient)?\n\n2) Combining skills needs some quantitative goal/metric, and an empirical evaluation. The videos are interesting and promising, but it is unclear how to interpret the results, let alone how other methods could be compared in terms of performance.\n\n3) Perturbation analysis comparing the Diffuser and Decision Diffuser in terms of robustness to noise or errors in the Q-value estimates/inverse-dynamics model respectively. While theoretical results would be nice, empirical results are probably easier to obtain and might already be insightful. Related: how well do both methods handle quite noisy dynamics? Since the latter two requires running additional experiments, potentially even designing toy environments, I do not have a hard expectation to see these results after the rebuttal; but they would certainly make the paper stronger.\n\n4) Discussion of limitations should be expanded\n * What are restrictions/requirements on the dataset, particularly w.r.t. allowing for combinable constraints/skills?\n * The current method is limited to fully observable MDPs, partial observability and latent variables might pose a hard to overcome problem, with self-delusions [1] potentially being a fundamental obstacle).\n * It seems unlikely that the method can be used to improve significantly beyond the performance of demonstrations (unlike RL, at least in theory). \n\n**Minor comments**\n\nA) What is the relation between constraint conditioning as presented in the paper and more general task conditioning (via natural language prompts) as in e.g. DeepMind’s Gato?\n\nB) Sec. 3 “It is useful to solve RL from offline data, both without relying on TD-learning and without risking distribution-shift.” - what are the conditions/requirements for the latter to be possible (at least informally)? In order to not *risk distribution shift* strong conditions on training-data coverage seem required.\n\nC) What is the fundamental difference between constraints and skills - are they more than simply two different names to refer to subsets of trajectories in the training data?\n\nD) “In contrast to these works, in addition to modeling returns, Decision Diffuser can also model constraints or skills and generate novel behaviors by flexibly combining multiple constraints or skills during test time.” - while this has not been explored in previous publications, is there a fundamental problem that prevents previous methods from using the same approach as used in this paper (i.e. augmenting data with a one-hot indicator for constraints/skills, and conditioning on that during generation)?\n\nE) P4, typo: “choices fof diffusion”\n\n[1] Shaking the foundations: delusions in sequence models for interaction and control, Ortega et al. 2021\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "S4sAxaygbI", "reviewer": "Reviewer_WRRm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents the decision diffuser for decision-making. The core idea is to model the decision making as a conditional generative model, which is different from the complex reinforcement learning-based works. Specifically, the method models a policy as a conditional generative model where the diffusion process is applied over states and gets return-maximizing trajectories. The experimental results show the performance on par with recent offline RL-based methods. ", "summary_of_the_review": "From my perspective, the paper is of high quality. It presents a well-motivated story. \n- Good motivation for modeling decision-making as a generative model. \n- More importantly, the method shows the ability to compose skills, which impresses me a lot.\n- The experimental results are also impressive. Though the conditional generative model is simple, it shows the performance on par with other complex reinforcement learning. \n\nAlthough I'm not very clear about some details of the diffusion process in the composition of skills, I tend to accept this paper given the impressive results. ", "strength_and_weaknesses": "## Strength\n- Modeling decision-making as the conditional generative model is novel and also interesting to the community.\n- Via conditional generative process, the method shows the ability to compose skills in order to maximize the rewards. This is impressive. \n- The paper in general is of high quality in terms of supportive experiments, and motivation. \n\n## Weakness\nI'm not an expert in decision-making/Reinforcement learning. So I'm not entirely sure if I follow the problem definition. To be specific, what's the design choice that leads to the ability to compose skills? And do other reinforcement learning-based method have the same ability? It would be better to make it simpler. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%