Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient
Authors: Zechu Li
Rickmer Krohn
Tao Chen
Anurag Ajay
Pulkit Agrawal
Georgia Chalvatzaki
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Diffusion Model
Multimodal Learning
Unsupervised Skill Discovery
Primary area: reinforcement_learning
TL;DR: Deep Diffusion Policy Gradient is a novel RL algorithm that successfully trains diffusion policies online, discovering and maintaining multimodal behaviors in complex environments.

Abstract: Deep reinforcement learning (RL) algorithms typically parameterize the policy as a deep network that outputs either a deterministic action or a stochastic one modeled as a Gaussian distribution, hence restricting learning to a single behavioral mode. Meanwhile, diffusion models emerged as a powerful framework for multimodal learning. However, the use of diffusion policies in online RL is hindered by the intractability of policy likelihood approximation, as well as the greedy objective of RL methods that can easily skew the policy to a single mode. This paper presents Deep Diffusion Policy Gradient (DDiffPG), a novel actor-critic algorithm that learns from scratch multimodal policies parameterized as diffusion models while discovering and maintaining versatile behaviors. DDiffPG explores and discovers multiple modes through off-the-shelf unsupervised clustering combined with novelty-based intrinsic motivation. DDiffPG forms a multimodal training batch and utilizes mode-specific Q-learning to mitigate the inherent greediness of the RL objective, ensuring the improvement of the diffusion policy across all modes. Our approach further allows the policy to be conditioned on mode-specific embeddings to explicitly control the learned modes. Empirical studies validate DDiffPG's capability to master multimodal behaviors in complex, high-dimensional continuous control tasks with sparse rewards, also showcasing proof-of-concept dynamic online replanning when navigating mazes with unseen obstacles. Our project page is available at https://supersglzc.github.io/projects/ddiffpg/.

Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient
Zechu Li
Rickmer Krohn 4.3 Tao Chen?
Anurag Ajay²
Pulkit Agrawal Georgia Chalvatzaki!
Technical University of Darmstadt Massachusetts Institute of Technology Hessian.AT
{zechu. ri.mirr krohn tu-darmstadt..
(taochen, aajay, aumii edu {georgia, chalvatzaki wtt-ddrmstadtt...ommmma
Abstract
Deep reinforcement learning (RL) algorithms typically parameterize the policy as a deep network that outputs either deterministic action or a stochastic one modeled as a Gaussian distribution, hence restricting learning to a single behavioral mode, Meanwhile, diffusion models emerged as powerful framework for multimodal learning. However, the use of diffusion policies in online RL hindered by the intractability of policy likelihood approximation, as well as the greedy objective of RL methods that can easily skew the policy to a single mode. This paper presents Deep Diffusion Policy Gradient (DDiftPG), novel actor-critic algorithm that learns from scratch multimodal policies parameterized as diffusion models while discovering and maintaining versatile behaviors. DDiffPG explores and discov- ers multiple modes through off-the-shelf unsupervised clustering combined with novelty- baaed intrinsic motivation. DDiffPG forms multimodal training batch and utilizes mode-specific Q-learning to mitigate the inherent greediness of the RI. objective, ensuring the improvement of the diffusion policy across all modes. Our approach further allows the policy to be conditioned on mode-specific embeddings explicitly control the learned modes. Empirical studies validate DDifiPG's capability to master multimodal behaviors in complex, high-dimensional continu- ous control tasks with sparse rewards, also showcasing proof-of-concept dynamic online replanning when navigating mazes with unseen obstacles. Our project page is available btttps://superggeee wiwporrgzz..gituub.b.... http://oocoo/aaaffffff.
Introduction
Reinforcement learning (RL) for continuous control has experienced significant advancements during the last decade, reshaping its applicability to domains like game playing 665 27 robotics 26 33
and autonomous driving 72. However, most RL algorithms choose parameterize policies as deep neural networks with deterministic outputs [47] (1 or Gaussian distributions 62] 25), limiting
learning single behavior mode. Moreover, the standard ssploration-exploiattii schemes can easily make a policy greedy towards one mode, in which the algorithm keeps exploiting to maximize its objective. The aforementioned issues hinder the possibility of training agents that successfully solve a task while showcasing versatility of behaviors-a property intuitive to intelligent systems like humans, who exhibit resourcefulness for completing task, even amidst unprecedented events.
Learning a multimodal policy has several practical applications. First, such a policy that learns many solutions to task is useful when acting in non-stationary environments. Imagine that routine path from the office to home is unexpectedly blocked; one needs to choose an alternate route. A policy that encompasses multiple solutions can better navigate such changing conditions, offering flexibility
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

OHER
O
We design (Top) four AntMaze tasks; AntMaze-vi, AntMaze-v2, AntMaze-v3, AnnMaze-v4, and (Below) four robotic tasks Reach, Peg-in-hole, Draver-close, and Cabinet-open that have high degree of multimodality.
Figure 1:
for replanning serving as prior for hierarchical RL. Second, while searching for diverse solutions, multimodal policy continues exploring even after finding viable solution, thereby facilitating agents escape local minima. Additionally, multimodal policies hold great promise for continual learning scenarios; they can effectively parameterize complex action distributions when new skills or solutions are introduced, potentially mitigating the issue of catastrophic forgetting [59]
Recently, Diffusion Models 30
68
a novel class of generative models acclaimed for impressive
image generation results 38
60) have been proposed a powerful parameterization for policy
learning. They have been extensively applied in the areas of learning from demonstrations [15]
58). offline RL 773 71I. and learning for trajectory optimization (34 46 One advantage of training diffusion policies offline is its ability to model multimodal datasets, which usually comprise trajectories stemming from suboptimal policies or various human demonstrations. However, only few studies explored these models for online RL, which focus on the formulation of the training objectives for policy optimization via diffusion and showease improved sample efficiency [55] [7] Notably, none of them studied the inherent multimodality within diffusion policies, nor have they explicitly tackled the challenge of exploration for discovering and learning multiple behavioral modes online.
In this paper, we introduce Deep Diffiusion Policy Gradient (DDifiPG). a novel actor-critic algorithm for training multimodal policies parameterized as diffusion models from scratch. Specifically, we aim learn a policy that capable of employing various strategies to accomplish task. Unlike existing multimodal methods that condition on latent variables, DDifIP emphasizes the explicit discovery, preservation, and improvement of behavioral modes, and for that, we decouple exploration and exploitation. For exploration, we apply novelty- -based intrinsic motivation and utilize an unsu- pervised hierarchical clustering approach to discover modes, being priori agnostic to the possible number of modes. For exploitation, we introduce mode-specific Q-functions to ensure independent improvements across modes and construct multimodal data batches preserve the multimodal action distribution. We empirically evaluate our method on high-dimensional and continuous control tasks with sparse rewards, comparing to SoTA basclines, verifying DDifiPG's capbility to master multimodal behaviors. Additionally, demonstrate the usability of our multimodal policies in an online replanning application in non-stationary mazes with unseen obstacles.
Our paper makes four contributions. First, we introduce diffusion policy gradient, novel way to train diffusion models following the RL objective. Second, we present DDiffPG, new actor-critic algorithm for training diffusion policies from scratch while discovering and preserving multimodal behaviors. Third, we achieve explicit mode control by policy conditioning on a mode-specific latent embedding during training, shown to be beneficial for online replanning. Finally, we design series of new challenging robot navigation and manipulation tasks with high degree of multimodality, serving as a testbed for multimodal policy learning, as shown in Fig.

---

2
Related Work
Policy learning via reinforcement learning for continuous control has exploded thanks to improved algorithms learning complex skills with online RL [25 5 while significant effort has been made to
provide improved methods for offline RL to take advantage of demonstrations and fixed datasets (40] 44). However, the policy parameterization in both settings considers policies that can only model single behavioral mode, e.g., by having a deterministic output [66] [3] or by learning a Gaussian distribution. Recently, the use of transformer models enabled learning for control through offline
data as sequence modelling [14 57 B1,, which, through language conditioning 77] 23], can handle multi-goal behavior generation; though, it is unclear if these models can explicitly exhibit behavior diversity per goal. Similarly, goal- conl-conditioned RL methods condition policies various goals, but each goal-conditioned policy suffers from the single-mode modeling, prevalent in typical deep RI.
algorithms 2 52 T0 53]
Diffusion policy as universal skill representation Diffusion models [30] have recently emerged as a promising parameterization for robust and multimodal robot learning. Early works used diffusion models for trajectory optimization 34
69 46). and as policy for behavioral cloning [15. Diffusion
policy has soon shown its power as universal skill representation (76) that allows learning complex
landscapes of multimodal behaviors [I 28), goal-conditioned [58, 61 or conguuge--onditionne
ones [3). Further, the expressivity of diffusion models has proven beneficial for offline RL [73
7T. Nevertheless, the approaches for online RL with diffusion policy optimization are scarce (55) 774
17,. while these works do not study multimodality preservation within diffusion models, nor have they explicitly tackled the challenge of learning multimodal policies online. In this work, we present a new framework that takes advantage of diffusion policy as universal skill representation model and introduces an algorithm that allows multimodal policy learning from scratch.
Unsupervised skill discovery Multimodal behavior learning has been addressed through skill discovery approaches [9] 42 32 54 36), either from offline data [64] or through unsupervised
RL 19 4. I which usually exploit variational inference for mode/skill discovery 43 with policy conditioning on latent vectors, or by forming rewards or goals to be achieved by different lowelevel
policies 39 49. Hierarchical learning methods, c.g., relying on options learning 3 T usually
depend on state-specific mode discovery that conditions low-level policy or triggers different
skills 45 35 but the policy usually reaches each goal greedily without exhibiting versatile behaviors.
3 Diffusion Policy Gradient
It is not straightforward to apply training approaches of popular deep RI. methods to train a diffusion policy online. First, it is practically intractable to approximate the policy likelihood (37]- Second, directly backprogating Q-values to the diffusion policy, like Q-learning methods 225 47 is not viable-the Markov chain may lead vanishing gradients (55).
To solve these issues, we first present diffusion policy gradient, a novel approach to training diffusion policies with the RL objective that is also suitable for learning multimodal behaviors. Similarly to
DPG [66], we compute V. (aQ(s, a) given a state s and action a. However, we choose not to directly use the action gradient to optimize the policy, this leads to vanishing gradients and instability. We rather obtain a target action a forget via a target +- a + v/ V (s, a), 1) being a suitable learning rate.
We can, then, train the diffusion policy based on the transitions in the datasets D using behavioral cloning (BC) objective
C())
E t~ (- torger D. ((..))
||e (atarged ,s,
(I)
where t ~ [1, T] refer to the diffusion step, and is the diffusion noise see Appx. A for brief introduction to Diffusion Models.
Implementation For every data collection, we store the transition (s, a, a torget, s') in the buffer D, where a farget is initialized as a. For every policy update, we sample the state-action pair (s, target) obtain a new target via gradient ascent, update the policy with equation and replace the new a target in the buffer. Thus, our policy update does not change the training objective of the diffusion model and avoids the aanishing-gradien problem. Note that the action larget that is used to update the policy is not inferred from the current policy, which provides an opportunity to control the behaviors of the policy in an off- oof-policy fashion and lay the foundation for learning multimodal behaviors later.
3

---

1
Data collection
Mode discovery
Trajectories
Hieracchcca clustering
DTW distance
Distance mat.
Brrennoki .00.0
a
FiF2
Multimodal training
3
Learning with mode-specific Qs
Clustered trajectories
mode
M,
Q
Bellman update
(arl)
M
mode embeed
With BC loss
lu
[a')
mode embed
M, U M
exxplon
Figure 2: Overview DDiffPG: (I) the agent interacts with the environment and collects a set trajectories [T.). (2) Given a set of goal-reached trajectories, a DTW distance matrix is computed and used for hierarchical clustering discover modes. (3) Each mode is associated with set of trajectories, which is used exclusively to train mode-specife Q-functions and an exploration-speccfic Qusplore.
(4) A multimodal batch is constructed by concatenating (s, surget) pairs sampled from every mode and used for the diffusion policy update.
Interpretation The conventional diffusion model is trained using dataset with supervised la- diffusion policy predicts actions given states with the dataset providing both the state and the corresponding ground-truth action as the target (label). In contrast, our online setting does not provide a predefined ground-trufe action but requires the discovery of good actions. To adapt without altering the supervised training framework of conventional diffusion models, we generate förget and consider it as the target. The action a target, derived through gradient ascent, represents an improved choice based on the current Q-function. During training, we additionally store gorget the replay buffer and continuously update it based on its preceding values, ensuring a continuity of learning. Intuitively, we are chasing the target by replacing it with the newly computed targgt in this online RL setting rather than learning a static target as in the offline setting.
bels [30]. In offline decision-makee
Relation to related works Our formulation is most closely related to DIPO [74] but has two key differences. First, while DIPO also performs gradient ascent on the action, replaces farget with the original from the buffer rather than retaining an additional a eerget Therefore, the transition no longer aligns with the current MDP dynamics and reward function due to the replacement of the original a. Given that DIPO is an off-policy algorithm, the reuse of these replaced transitions for training the Q-function could be problematic, as the agent training values of actions that have not been actually played out in the environment and their true outcome (reward and next state) are unknown. Second, DIPO uses different batches for Q-learning and policy updates, which does not guarantee coherent updates. Q-score matching (QSM) [55] also computes V. VaQ(s, a) and matches vector fields of V. log π (als) and VaQ(s,a) a). Therefore, QSM is optimizing on the score level while ours focuses on the action level.
4 Learning Multimodal Behaviors from Scratch
We consider the problem of learning multimodal behaviors with online continuous RL, i.c., in the absence of initial demonstrations. In this section, we introduce Deep Diffiasion Policy Gradient (DDiffPG), which builds off of three main ideas. First, we want to explicitly discover behavior modes and master them. Different modes should act differently certain states and then diverge, therefore being distinguishable from trajectories/seqqeeness of states. Second, we must prevent mode collapse once modes have been discovered, a common issue where the RL policy favors the mode with higher Q-values due to its inherent greediness. Finally, we expect the diffusion policy to capture and control the multimodal actions during evaluation. Fig. provides an overview of the proposed method, and the pseudocode is available in Alg. I

---

4.1 Unsupervised Mode Discovery
Novelty-baser Exploration In multimodal learning, it necessary to explore diverse behaviors (i.e., modes). Effective exploration is important especially when considering challenging high-dimension continuous control tasks with sparse rewards (cf. Fig. D. We adopt simple yet effective approach, namely using the difference between states' novelty intrinsic motivation [75] to prompt exploration beyond already visited state-space regions. Following (8), we define the following intrinsic reward (sl a, s') max max(novelty(s')
novelty(s), 0). where novelty is parameterized as Random Network Distillation (RND) [8] and a is a scaling factor. Note that while this intrinsic reward has been effectively verified for discrete state spaces, here we extend this exploration method to continuous domains, demonstrating its effectiveness for training diffusion policies.
Hierarchical Trajectory Clustering Our method explicitly identifies modes and masters them, unlike existing condittoondi approaches [32. Given a collection of goal reachee trajectories, each consisting of a sequence of state-action pairs, we categorize them into clusters and consider each a behavior mode. practice, we use an unsupervised hierarchical clustering approach [5T] as shown in Fig. 2), in the beginning, each trajectory is considered single cluster; then clusters within a small distance are progressively merged, continuing until a singular, unified cluster is formed.
Differently from clustering approaches like K- means (88] that require a predefined number of clus- ters (modes), we determine clusters using distance threshold. Fortunately, given that the distance be- tween different modes is usually large. hierarchical clustering is not as hyperparameter-sennse means. We show the clustering performance in Fig. 3 and the robust clustering threshold Tab. [2] For distance metric, we utilize Dynamic Time Warping
Start
Goal
as K-
Figure 3: Hierarchical clustering on AntMaze-v3 (Left) and AntMaze-v4 (Right). Each color repre- sents a mode.
(DTW) [50] with task-prior information, e.g., robot positions 32 An advantage its applicability variant-length trajectories, which removes the burden of padding or stitching trajectories. Note that our method is agnostic to the particular clustering approach used, and it can be adapted to different (learn- we provide an alternative clustering approach by
ing) approaches 63 [6]. For example, in Fig. (a)
training a VQ-VAE 770 to learn trajectory representations and using the codevectors for clustering, which learns meaningful representations suitable for our approach.
4.2 Mode Learning with Mode-specific Q-functions
To master multi-modes and improve them all together, we train a different Q-function per mode and construct multimodal data batches for policy learning.
Learning mode-specific Q-functions In RL, the objective to maximize expected return can skew the policy, leading to single-mode collapse. Let us consider the AutMaze-vl in Fig. [ there are two viable paths to reach the goal. If the goal position is reached via the top path, the success bonus will propagate through the TD updates, meaning that the Q-values for the top path will increase and guide the policy to the top. Even though a simple diffusion RL 774 73) method might initially explore both sides altogether, it will eventually end with unimodal behavior determined by which side was explored first (cf. Fig. IP & Fig. 12p). To address this issue, we propose training speeesp Q-function per discovered mode, allowing for parallel policy improvements across all modes. As shown in Fig. 3), trajectories that, for example, are categorized into two modes Mj and M2, will have two f-fuction Q1 and Q2. respectively. One needs to notice that such mode-specific Q-functions may capture suboptimal modes, which achieve the goal but require more steps hence yielding lower discounted returns. However, these suboptimal solutions may prove valuable in practical scenarios where the optimal action is infeasible, c.g., the routine path problem discussed in Sec. and Sec. 5.5
We additionally train a O-function dedicated to exploration, which can be considered as an exploratory mode. This Q-function is trained exclusively with the intrinsic rewards and transitions from all trajectories, e.g., M1 U M22 in F regardless of which behavioral mode they represent. Such decoupling of exploration-exploitatti
on the Q-function level ensures that we continue exploring even after certain modes are well-learned, since our intrinsic reward only considers state novelty.
5

---

Constructing a multimodal batch The diffusion model's multimodality stems from the underlying multimodal distribution of the data, An intuitive strategy to obtain multimodality is to construct multimodal training batch and feed it to the policy - each batch contains data from different modes. While the mode clustering is on the goal-reached trajectory level, it is essential to include data from unsuccessful trajectories as well. Practically, this is achieved by computing the distance between an unsuccessful trajectory and N goal-reached trajectories randomly sampled from each cluster. The cluster with the smallest average distance is then designated as the final cluster for the unsuccessful trajectory (lines 8-14 in Alg.
Re-clustering We perform re-clustering over trajectories at every F iterations (see Tab. E). During this process, for each newly formed cluster, we assess its overlap with clusters identified in the previous clustering iteration. The new cluster then inherits Q-functions and a terget from the preceding cluster that has the most overlap with. This ensures continuity of learning and adaptation across successive re-clustering stages. The pseudocode is in Alg. 2 and implementation details are in Appx C
4.3 Mode Control via Latent Embeddings
Controlling the learned multimodal policy exhibit specific behaviors rather than random generation is beneficial, especially in non-stationary test environments where certain modes may become nonviable. To achieve this, propose conditioning the diffusion policy mode-specific embeddings during training. As shown in Fig. 4). our method generates unique latent embedding for each mode, which then incorporated into the state information. Throughout the training process, we selectively include or mask (zero-out) these embeddings with a probability of p. By providing specific embeddings, we can, therefore, explicitly control the execution of desired modes or, alternatively, mask them to enable random mode selection. This technique affords several benefits:
non-stationary environments, planning approaches can empower the agent navigate around undesirable modes through controlled mode selection, improving their success rate. We demonstrate an application for online replanning Section 5.3
Our method learns multiple modes, some of which may be suboptimal, e.g., longer paths in the navigation problem. Given our knowledge of each mode's ' trajectories, we can estimate the expected return of each mode and select the one with the highest return to optimize performance.
Since the exploratory mode has its unique embedding, we control the ipploration-exploittaii tradeoff during training by adjusting the proportion of exploratory mode used in action generation at the data collection phase. Note that we exclude the exploratory mode to eliminate noisy exploratory behaviors at test time, leading to an increased success rate.
5 Experiments
In this section, we present comprehensive evaluation of our method against SoTA baselines. First, we verify that DDiftPG can learn multimodal behaviors and discuss the performance compared to baselines. We then highlight the advantages of learning a multimodal policy in encouraging explo- ration and overcoming local minima. Finally, we provide ablations on important hyperparameters and showcase practical application of replanning with such multimodal policy. We run each experiment with five random seeds and plot their mean and standard error.
5.1 Setup
Tasks We evaluate our method on four AntMaze tasks 20 and four robotic control tasks 22 as shown in Fig. [ Note that all tasks (I) are high- high-dimensional and continuous control tasks, c.g., in all AntMaze versions, the objective is to control the leg joints of an ant to reach the goal position; (2) contain multiple possible solutions, either with multiple goals or multiple ways that solve the task, e.g., in the Reach task, the robot arm can bypass the obstacle from four different directions; (3) are trained with sparse rewards, alleviating the need for engineering and reward shaping. The environment description is in Appx.
Baselines We consider the following baselines: (1) DIPO 74 which we have adapted to include the additional target action in replay buffer to ensure consistency in MDP dynamics and the reward function, as detailed in Section 3 (2) Diffusion-QL. [73 and (3) Consistency-AC [17], which use
diffusion model and consistency model for policy parameterization; (4) Reparameterized Policy

---

Artkaze
AntMaaze-
Antiaaze
0.0
0.8
0.8
0.8
0.6
0.6
0.4
0.5M
-M
1.5M
2M
2.5M
BM
0.5M
3M
1.5M
2M
2.5M
3M
0.5M 1M
15M 2M
2.5M Stap
3M
114
2M
3M
4M
SM
Stap
Stap
5tep
-DDIMPG- DEI---IPODDRRE MIftO-Ditusio--2 -L -Consistenyy-AC- SAC-RPG
Reaching
Drower
Cabinet open
0.6
0.6
0.0
220.4
0.4
0.4
1/-
0.4
0
0.5M
1M
1.5M
2M
2.5M
3M
0.5M
3M
1.2M
2M
0.5M
1M Step
1.5M
2M
0.5M
174
1.5M
2M
Step
Step
Slep
Figure Performance of DDifPP and baseline methods in the four AntMaze and robotic manipulation environments.
Gradient (RPG) 32 which uses model-based approach with multimodal policy parameterization; (5) TD3 22: (6) SAC 253. For fair comparisons, we use double Q-learning [29] and distributional RL [ for all baselines. Additionally, all baselines except RPG use the same intrinsic rewards as ours, while RPG has a similar built-in RND-based intrinsic reward as introduced their paper. Hyperparameters are available in Tab. E
5.2 DDifIG Masters Multimodal Behaviors
We investigate whether DDifiPG can learn multimodal behaviors from scratch. We first perform observational evaluations and count the number of different modes over 20 episodes. As shown in Tab. andd Tab. 4 DDifiPG demonstrates consistent exploration and acquisition of multiple behaviors. For instance, AntMaze-v3,, multiple paths exist within the maze, but not all are the same length; DDifIPG is capable of learning and freely executing all these paths, including the suboptimal one note that we call suboptimal a path with lesser discounted return than the shortest optimal one. However, assuming goal-reaching success indicator all paths are successful. Nonetheless, the suboptimal issue can be mitigated given our ability to control the agent's behavior through the mode embeddings, we discuss in Section 5.5] In Cabinet-open, the agent can move the arm to either layer and subsequently pull the door open. This contrasts with other methods, which fail to exhibit multimodal behavior. We observe that even policies parameterized as diffusion-bart -baed models, namely DIPO, Diffusion-QI.. and Consistency-AC, can quickly collapse to a single mode and thereafter follow the greedy solution. This verifies the significance of proposed method in capturing multimodality.
In Fig. we see that DDifiPG has comparable performance to the baselines on all eight tasks while acquiring multimodal behaviors. In the AntMaze tasks, the sample efficiency of DDiffPG, DIPO, TD3 and SAC are similar: in AntMaze-vl and AntMaze-v3, TD3 and DDiffPG surpass the performance of others; in AntMaze-v2 and AntMaze-v4, TD3 and DIPO are the most sample-efficient In the manipulation tasks, similar pattern emerges: DDiffPG leads Peg-in-hole, DIPO excels in Reach, and TD3 leads both Drawer-close and Cabinet-open. DDiffPG generally demonstrates lower sample efficiency than baselines in tasks that pose significant exploration challenges this is expected since our method strives to discover multiple solutions. For example, in AnzMaze-v2, the route to the top-left goal is more extended, and in Reach, the robotic dynamics make it difficult to explore the bottom paths. For simple exploration tasks, DDiftPG can achieve similar or even superior performance, DDifIPG simultaneously explores the environment from multiple directions and the design of mode-specific Q-function effectively narrows the scope, facilitating faster convergence.
Diffusion- QL and Consistency-Ae tend to lag behind as shown in Fig. Both methods optimize the diffusion policy by backpropagating directly through the diffusion model, and we observed that their actor gradient may remain zero throughout the training in some seeds, resulting in high variance (shadow area). contrast, our diffusion policy gradient approach, which turns training objective into minimizing the MSE loss w.r.t. the action target, demonstrates significantly greater stability. For
7

---

(a) DDiFPG
(b) DIPO
(c) Diflusion-QL.
(d) SAC
(c) RPG
Figure 5: Exploration maps of DDiffPG and baselines AnrMaze-v3.
BA
BA
DDMPG
DIP
-Diflssiongg Consillecyy
TD3
SAC -PG
(a) DDiRPG
DIPO
DiffusionQQL
SAC
State Coverage
Figure 6: (a)-(d) Q-value maps of baselines and DDifPG and (Right) the state coverage in AntMaze-v3.
RPG, its performance illustrates that policy learning remains challenging despite demonstrating good exploration. One potential reason is that VAEs condition the policy on latent variable, which offers pathway multimodality but sometimes leads to non-existing modes. This consideration led us to adopt a more straightforward yet effective clustering approach for explicit mode discovery.
5.3 Seeking Multimodality Encourages Exploration and Overcomes Local Minima
Observation 1. DDiffPG encourages exploration.
We demonstrate the potential of DDiffPG in exploration using the exploration density maps and state coverage rates in the AntMaze environments. We discretize the maze and track the cell visitation. To avoid the dominance of high-density areas such the starting positions, threshold of 100, which means the cell has been visited at least 100 times. As shown in Fig. 5 in selected results for AmMazzevv3, DDifPG explores multiple paths the separate goal positions, contrasting sharply with baselines that typically discover only a single path. For state coverage in Fig. [6] measure the binary coverage of each cell and find that DDiffPG achieves much higher coverage rate than basclines except RPG, verifying that the continuous exploration capability of DDifIPG helps exploration. While RPG achieves good exploration, fails to solve the task as shown in Fig. The maps for all AntMaze tasks and baselines are available in Appx. F
max density
Observation 2. DDiffPG
overcome local minima.
We showcase that DDiffPG effectively overcomes local minima when learning multimodal policy. The key intuition is that unlike other methods that explore the first solution and collapse into it, DDiftPG continuously explores and seeks different solutions, enabling to escape suboptimal local minima. As illustrated in Fig we present two tasks, each posing distinct local minima challenges.
In AntMaze-vl, an ant must circumvent central obstacle to reach its goal, with two possible routes: over the top or beneath the bottom. The optimal path depends on the ant's randomized starting position since there only one shortest path. As shown in Fig. [T] DDifIPG discovers both routes and learns to select the shortest one based on the starting location, while baselines struggle to adjust their paths adaptively. In AntMaze-12, the faces two goals: the top-left goal offers a higher reward, while the right-hand goal is easier to reach. Baseline models often get trapped going for the easier, lower-reward goal. However, we plot Q-value density maps in Fig. 13 and we find that DDiffPG locates both goals and learns go either one of them, effectively overcoming the local minima and reaching higher cumulative return. On the other hand, RPG also explores the top-left goal and overcomes the local minima. However, cannot consistently solve the task.

---

5,4 Ablation Studies
We investigate the impact of the number of diffusion steps, batch size, action gradient learning rate, and number of Updates-To- To-Dat (UTD) ratio. These hyperparameters are of particular interest given the diffusion policy and our learning procedure. For batch sizes in Fig. Fig-Daa), we observe that larger batches enhance sample efficiency. Due the mixed multimodal batch, larger batch-size helps smooth the learning. In Fig. bb). we find that the number of diffusion steps appears to have minimal impact on performance, with 5 steps being sufficient to acquire and maintain multimodal behaviors, aligning with the findings of other diffusion policy learning methods 77
1024
2048
-0.03
4096
50
-0.05
2..
8192
1100
0.2
0.5M
3M
3.5M
2M
2,5M
3M
0.3M
3M
1.5M Stop
2V1
2.90
3M
3M
1.5M Step
2.5M BM
0.5M
1,50
2M
2.5
Stap
S0ep
(a)
(b)
(c)
(d)
Figure 7: Abaltion studies for key parameters of DDifPPG: (a) batch size, (b) diffusion steps, (c) update-to-date (UTD) ratio, and (d) action gradient learning rate.
In Fig. 7c., the UTD ratio indicates that more frequent updates can accelerate learning; however, very large number would lead to increased computational demands and wall-clock time. In terms of the action-gradient learning rate (d). while higher rate initially aids learning, may introduce increased variance due to the larger step sizes in updating target actions. An adaptive approach to the learning rate might be beneficial. Finally, our computational time analysis in reveals that our DDiffPG method is approximately five times slower than both TD3 and SAC, primarily due to the overhead associated with updating target actions. However, we trust that continuous developments on diffusion models will allow much faster training and inference in the future [8]
5.5 Online Replanning with a Multimodal Policy
We present practical application of trained multimodal policy in online replanning, particularly in nonstationary environments. We replicate the routine path problem described in Sec. [ in our maze experiments by introducing random obstacles that obstruct certain paths. As discussed in Sec. our policy can selectively execute different modes by conditioning the mode's 's embedding. To capitalize on this feature, we developed a proof-of-concept planner, which iteratively tests different modes until successful route is found. Specifically, if the ant remains stationary at location for 10 consecutive steps, the planner initiates an alternative mode. As shown in Tab. 3 while baseline methods tend to fail, becoming stuck in front of newly introduced obstacles until the end of an episode, our moof-of-concept planner demonstrates significantly higher success rate. This demonstrates the adaptability and efficacy of our multimodal policy in performing in nonstationary environments.
Conclusions, Limitations & Future Work
In this work, we presented a novel algorithm, DDiffPG, for learning multimodal behaviors from scratch. We parameterized the policy as diffusion model and proposed diffusion policy gradient to enable training diffusion models with RI. objectives in online settings. Unlike existing methods that rely on latent conditioning to retain multimodality, we emphasized explicitly discovering, preserving. and improving multimodal behaviors. First, we employed novelty-based intrinsic motivation to explore different modes and used an unsupervised hierarchical clustering approach over trajectories to identify them. Nevertheless, the RL objective can skew the policy toward a single mode. We addressed the issue by introducing modee-ppeiife Q-functions to optimize each mode, providing the diffusion policy with multimodal batch to train on. We further achieved explicit mode control by conditioning the policy on a mode-specific latent embedding, which was shown to useful for online replanning. Our evaluation demonstrated the algorithm's 's effectiveness in learning multimodal behaviors complex control scenarios, such as AntMazes and robotie tasks, and showcased the potential of the multimodal policy to encourage exploration and overcome local minima.

---

Limitations However, our approach also has some limitations. Clustering knowledge prerequisite: Hierarchical clustering depends on a distance matrix calculated from trajectories. In practice, we compute it over the ant's s 2D positions or the robot's end-effector (EE) 3D positions, rather than the entire state space. This poses a problem in scenarios like locomotion control, where different gaits may exhibit similar position changes. A possible solution is to encode trajectories into latent embeddings for clustering. Exploration challenges: while our current intrinsic motivation approach is effective, its performance might drop in larger spaces, as our method demands uncovering many as possible viable solutions. This can be mitigated through different information-bant motivation, or by adding bound the information contained withing explored modes. Increased computation time: the diffusion process in diffusion models lengthens training and inference wall-clock times compared to simpler MLP models, limiting, for now, the real-time application of diffusion policies in domains such as robotic control.
Future directions Overall, we believe our work paves the way for training multimodal diffusion policies and offers several promising research avenues. Online replanning with long-horizon planners: while our paper demonstrates online replanning using multimodal policy with brute-force method, exploring planners that can efficiently orchestrate these modes for complex, long-horizon tasks presents an intriguing opportunity. Offline-to-online learning: diffusion policy has been widely used in offline settings. However, the offline dataset may contain suboptimal trajectories. Our method is suitable for offline-to-onling fine-tuning, allowing the agent to further refine policies without killing the previously learned modes. Open-ended learning on large-scale environments: viewing our method as a form of unsupervised skill discovery opens up possibilities for enabling the acquisition of diverse and useful skills beyond mere goal achievement.
7
Acknowledgements
This research work has received funding from the German Research Foundation (DFG) Emmy Noether Programme (CH 2676/1-1), the Daimler- Benz Foundation, and was co-financed by the EU's Horizon Europe project ARISE (Grant no.: 101 101135959).
References
[1] Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint arXiv: 10299, 2018.
[2] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong. Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[3] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
[4] Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. ArXiv, abs/2206.1 11795, 2022.
[5] Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, TB Dhruva, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. In International Conference on Learning Representations, 2018.
[6] Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforce- ment learning. In International Conference on Machine Learning, 2017.
[7] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.0681/ 2022.
[8] Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation. ArXiv, 111111110119.. 2018.
[9] Victor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning, pages 1317-1327. PMLR, 2020.
10

---

[10] Elliot Chane-Sane Cordelia Schmid, and Ivan Laptev. Goal Goal-conditioned reinforcement learning with imagined subgoals. In International Conference on Machine Learning, pages 1430-1440. PMLR, 2021.
[11] Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209. 2022.
[12] Li Chen, Peng Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li. End-to-end autonomous driving: Challenges and frontiers. ArXiv, abs/2306. 16927, 2023.
[13] Lili Chen, Shikhar Bahl, and Deepak Pathak. Playfusion: Skill acquisition via diffusion from language-annotated play. ArXiv, abs/2312.00544 2023.
[14] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 34:15084-15097, 2021.
[15] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[16) Shripad Vilasrao Deshmukh, Arpan Dasgupta, Balaji Krishnamurthy, Nan Jiang, Chirag Agar- wal, Georgios Theocharous, and Jayakumar Subramanian. Explaining rl decisions with trajecto- rics. In The Twelfth International Conference Learning Representations, 2023.
[17] Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning. In The Twelfth International Conference on Learning Representations, 2023.
[18] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. arXiv preprint arXiv:2403.03220 033200 2024.
[19] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. In International Conference on Learning Representations, 2018.
[20] Justin Fu, Aviral Kumar, Ofir Nachum, G. Tucker, and Sergey Levine. D4rl: Datasets for deep
data-driven reinforcement learning. ArXiv, abs/2004.07219, 2020.
[21] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[22] Quentin Gallouédec, Nicolas Cazin, Emmanuel Dellandréa, and Liming Chen, cunda-gymi Open-Source Goal-Conditioneel Environments for Robotic Learning. 4th Robot Learning Workshop: Self-Supervised and Lifelong Learning at NeurIPS, 2021.
[23] Huy Ha, Pete Florence, and Shuran Song. Scaling up and distilling down: Language-guided robot skill acquisition. In Conference on Robot Learning, pages 3766-3777. PMLR, 2023.
[24] Huy Ha, Peter R. Florence, and Shuran Song. Scaling up and distilling down: Language-guidedt robot skill acquisition. ArXiv, abs/2307.145535 20233
[25] Tuomas Haarojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[26] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint
2018.
[27] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. ArXiv, abs/2010.02193, 2020.
[28] Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit glaarnin an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10s3 10573, 2023.
[29] H. V. Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double g-leamnig. In AAAI Conference on Artificial Intelligence, 2015.
11

---

[30] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-68511 2020.
[31] Shengchao Hu, Li Shen, Ya Zhang, Yixin Chen, and Dacheng Tao. On transforming reinforce- ment learning with transformers: The development trajectory. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[32] Zhiao Huang, Litian Liang, Z. Ling, Xuanlin Li, Chuang Gan, and Hao Su. Reparameterized policy learning for multimodal trajectory optimization. In International Conference on Machine Learning, 2023.
[33] Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine. How train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotics Research, 40(4-5):698-7211 2021.
[34] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, pages 9902-9915. PMLR, 2022.
[35] Snehal Jauhri, Jan Peters, and Georgia Chalvatzaki, Robot learning of mobile manipulation with reachability hehavior priors. JEEE Robotics and Automation Letters, 7(3):8399-84006 8406, 2022.
[36] Plerre-Alexandre Kamienny, Jean Tarbouriech, Alessandro Lazaric, and Ludovic Denoyer. Direct then diffuse: Incremental unsupervised skill discovery for state covering and goal reaching. ArXiv, 0bbs/2110.1447,, 2021.
[37] Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[38] Ayush Karn, Shubham Kumar, Sonu K. Kushwaha, and Rahul Katarya. Image synthesis using gans and diffusion models. 2023 IEEE International Conference on Contemporary Computing and Communications (InC4), 1:1-6, 2023.
[39] Seongun Kim. Kyowoon Lee, and Jaesik Choi. Variational curriculum reinforcement learning for unsupervised discovery of skills. International Conference on Machine Learning, 2023.
[40] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 119- 1191, 2020.
[41] Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto. and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[42] Sang-Hyun Lee and Seung- Seo. Unsupervised skill discovery for learning shared structures across changing environments. In International Conference on Machine Learning, pages 19185- 19199. PMLR, 2023.
[43] Sang- Syug-Hyun Lee and Seung- Woo Seo. Unsupervised skill discovery for learning shared structures across changing environments. International Conference on Machine Learning. 2023.
[44] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.06644 2020.
[45] Chengshu Li, Fei Xia, Roberto Martin-Martin, and Silvio Savarese. Hrl4in: Hierarchical reinforcement learning for interactive navigation with mobile manipulators. In Conference on Robot Learning, pages 603-616. PMLR, 2020.
[46]
Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. In International Conference on Machine Learning, 2023.
[47] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 1
2015.
[48] MacQueen. Some methods for classification and analysis of multivariate observations. Mathematics, 1967.
[49] Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. Dis- covering and achieving goals via world models. ArXiv, abs/2110.0055 2021.
12

---

[50] Meinard Müller. Dynamic time warping. Information retrieval for music and motion, pages 69-84, 2007.
[51] Fionn Murtagh and Pedro Contreras. Algorithms for hierarchical clustering: an overview. Wiley Intendisciplinary Reviews: Data Mining and Knowledge Discovery, 2(1):86-97) 97, 2012.
[52] Soroush Nasiriany, Vitchyr H. Pong, Ashvin Nair, Alexander Khazatsky, Glen Berseth, and Sergey Levine. Disco rl: iestrbuttion-condiionn reinforcement learning for general-purpos policies. 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6635-6641, 2021.
[53] Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Hiql: Offline goal- conditioned rl with latent states as actions. In Thirty-seventhh Conference on Neural Information Processing Systems, 2023.
[54] Seohong Park, Kimin Lee, Youngwoon Lee, and P. Abbeel. Controllability-awaa unsupervised skill discovery. In International Conference on Machine Learning, 2023.
[55] Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning diffusion model policy from rewards via q-score matching. ArXiv, abs/2312.1 2023.
[56] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming.
John Wiley & Sons, 2014.
[57] Scott Reed, Konrad ZoIna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel marth-maron, Mai Giménez. Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al.
A generalist agent. Transactions on Machine Learning Research, 2022.
[58] Moritz Reuss, Maximilian Xiling Li, Xiaogang Jia, and Rudolf Lioutikov. Goal-conditioned imitation learning using score-based diffusion policies. ArXiv, abs/2304.02532, 2023.
[59] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Greg Wayne. Experi- ence replay for continual learning. In Neural Information Processing Systems, 2018.
[60] Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674-10685, 2021.
[61] Vaibhav Saxena, Yotto Koga, and Danfei Xu. Constrained-context conditional diffusion models for imitation learning. ArXiv, abs/231 1.01419, 2023.
[62] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[63] Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State en- tropy maximization with random encoders for efficient exploration. In International Conference on Machine Learning, pages 9443-9454 PMLR, 2021
[64] Tanmay Shankar and Abhinav Gupta. Learning robot skills with temporal variational inference. In International Conference on Machine Learning, pages 8624-8633 PMLR, 2020.
[65] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap. Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484-488 2016.
[66] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pages 387-395. Pmir, 2014.
[67] Jascha Sohh-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
[68] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[69] Julen Urain, Niklas Funk, Jan Peters, and Georgia Chalvatzaki. Se (f)-diffussonnfield:: Learning smooth cost functions for joint grasp and motion optimization through diffusion. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5923-5930. IEEE, 2023.
13

---

[70] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[71] Siddarth Venkatraman, Shivesh Khaitan, Ravi Tej Akella, John Dolan, Jeff Schneider, and Glen Berseth. Reasoning with latent diffusion in offline reinforcement learning. ArXiv, abs/2309.06599, 2023.
[72] Sen Wang, Daoyuan Jia, and Xinshuo Weng. Deep reinforcement learning for autonomous driving. ArXiv,
2018.
[73] Zhendong Wang, Jonathan J. Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. ArXiv, hiss/208.061995 2022.
[74] Long Yang. Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning. ArXiv, abs/2305.13122, 2023.
[75] Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E. Gonzalez, and Yuandong Tian. Noveld: A simple yet effective exploration criterion. In Neural Information Processing Systems, 2021.
[76] Zhengbang Zhu, Hanye Zhao, Haoran He, Yichao Zhong, Shenyu Zhang, Yong Yu, and Weinan Zhang. Diffusion models for reinforcement learning: A survey. ArXiv, abs/231 abs/331111 11.1111,,,2. 2023.
14

---

A Preliminaries
is a class of generative models that deploys stochastic denoising process for learning to generate samples from probability distribution p(x) by mapping Gaussian noise to the target distribution through an iterative process, assuming (x)) == Po )dx,,TT where Xo. ..... Xy latent variables of the same dimensionality the data Xo ~ p(xn). A diffusion model approximates the posterior g ((x:r|/xo) xx) through a forward diffu- sion process, i.e., fixed Markov chain, which adds gradually Gaussian noise the data Xo g(xp) according to a variance schedule Bi,..., ßT, defined as ((XIT xo) = II g(x+ (x-1)), with ((x,[x,-1) 1+1) = [xx vI ,, 8,T). Diffusion models learn to sample from target dis- tribution p(xr) by sampling noise from a Gaussian p(xr) 0, and iteratively denoising the noise to generate in-distribution samples, through a reverse diffissin process pé (x:- (xxx_1 de- fined as p(xo.r) := p(xx) IT==1 Po (x),))x),, with po(Xx-1(Xx) == (x po (x,, t). (x,,, t).
Diffusion Model Diffusion models 30
67
68
The reverse diffusion process optimized by minimizing a surrogate loss-function [30] L(0)
cx(x,, t)ll- After training, we sample the diffusion model by XI ~ p(Xx) and run the reversed diffusion chain to go from = to to =
Markov Decision Process A Markov Decision Process (MDP) is the M M A,, [56]. where is the state space, A is the action space, R. S x A
+ IR is the reward function, P x A S is the transition kernel, and y [0. 1) is the discount factor. We define policy t E II x A + R as the probability distribution of the event of executing an action a state A policy ı induces value function corresponding the expected cumulative discounted reward collected by the agent when executing action a in state 8, and following policy π thereafter: (*. a) _ E transition. Solving expected discounted return. In this paper, we are particularly interested in tasks where there are multiple goals or there is only one goal but with multiple solutions to it.
= r=
a; = a, T]. where r;-1 is the reward obtained after the i-th MDP means finding the optimal policy x², i.e., the one maximizing the
B Pseudoalgorithm
Algorithm Deep Diffusion Policy Gradient
Input: initial policy parameters 0, initial Q-function parameters Φ, replay buffer D, diffusion iteration N
2: for each iteration do
3: for I,... .T' do 4: Observe state 8; and sample action af ~ To(ar)s)) via reverse diffusion process
5: Execute action a; af where N
6: Initialize a, target a+ Tt,, 86+1) in D
target 7: Store (St, Ct, a,
8: end for
9:
for y 1, G do
10:
Sample M batch B, = {(8, a, a lorget sr,'))} from replay buffer D, where M is the number of modes discovered
11:
Get Qø, i i= 1, 2,... M from Algo. 2 for each mode do
13:
Update Qe, with Bellman Equation on B, for 1, K do
15:
Compute target action a target by one step of gradient ascent following farget a target + V a target)
16:
end for
17: Replace target in D
18: end for
Concatenate ((s, a (.r and update diffusion policy my following I
20: end for
21:
end for
15

---

Algorithm 2 Mode Discovery via Clustering
1: Input: goal-reached trajectories {r²}, unsuccessful trajectories {r"), previous cluster Cotd
2: Compute distance matrix of {r*} with DTW metric [50]
3: Obtain cluster C' via hierarchical clustering with matrix D
4: for in do
5: Find the cluster Cold with the largest overlap between and Cold
6: Assign Qo and target from Cold to c
7: end for
8: for 7" in { do
9: for in C' do
Sample N trajectories from cluster e 11: Compute average distance between 7 and {TA)
10:
12: end for
13: Find the cluster with the smallest average distance
14: Add 7 to c
end for
C Implementation details
Our approach includes piih--perfommance implementation of the proposed algorithm. First, each trajectory is assigned a unique identifier (ID), where clusters group many such IDs representing distinet modes. All trajectory elements, including states, actions, target actions, and rewards, tagged with this ID. This allows storing all trajectories in one replay buffer, enabling efficient batch sampling. Second, computing distances between trajectories can be computationally demanding. To address this, we implement a hashmap for storing these distances, keyed by the trajectory IDs. This strategy ensures that distance computation between any two specific trajectories performed only once.
D Environment details
The AntMaze environments are implemented based on D4RL benchmark 20). The AntMaze is navigation task, in which the agent controls the movement of complex 8-DOF "Ant" quadruped robot. The objective to reach goal positions represented by the red ball(s). The robotic manipulation environments with Franka are based (22). The agent controls 7-DOF Franka arm in joint space for different manipulation tasks.
For the above environments, we designed to contain multiple possible solutions to show the multimodality, either with multiple goals or multiple ways that solve the task. Note that the goal position is static and invisible to the agent, meaning that the agent has to explore the goal first. We use sparse reward 0-1 reward for all environments, which is activated upon reaching the goal. We describe each environment and its multimodal solutions as follows:
AntMaze-vl: it contains one goal in the maze. The ant is expected to bypass central obstacle reach the goal position, with two possible routes, either over the top or beneath the bottom of the obstacle. The optimal path varies depending on the ant's randomized starting position, as there is only one shortest path. The episode length is 500.
AntMaze-v2: it contains two goals in the maze, with the top-left goal offering higher reward than the right-hand goal. Due the higher reward, the optimal path is to reach the top-left goal, however, the ant may get trapped the right goal as much easier to explore. The episode length is 500.
H
16

---

AntMaze-v3: it contains two goals in the maze, each accessible via multiple routes. For the goal the top-left, three routes are viable: (1) go upwards to the end and then turn left; (2) move diagonally towards the top-left until encountering the left border, then head upwards, and (3) move towards the bottom-left, circumvent the left obstacle, and then go upwards. The distances of the first two paths are comparable, whereas the third is much longer. For the goal in the right-bottom, there are two comparable routes: (1) moving right and then downwards or (2) going downwards and then right. Possible solutions shown in the visualization of clusetering performance in Fig. The episode length is 700.
AntMaze-v4. 44: it contains two goals in the maze, each accessible through two routes. For the top goal. the agent can bypass the obstacle by going up. then has the option to either go up down to reach it. The routes to the bottom goal is symmetrical. Possible solutions are shown in Fig. B The episode length is 700.
Reach: the agent controls the Franka arm reach the red ball, navigat- ing around a fixed cross-shaped obstacle that lies in the path. Despite the presence of single goal position, the agent can bypass the obsta- cle in four distinct ways, offering multiple solutions to the task. The episode length is 100.
Peg-in-hole: the agent controls the Franka arm perform peg- insertion task. With two holes available on the desk, the agent can successfully complete the task by inserting the peg into either hole. The episode length is 100.
Drawer-close: the agent controls the Franka arm to close drawers. There are four drawers on the desk, and the agent can close either drawer to finish the task. The episode length is 100.
Cabinet-open: the agent controls the Franka arm to open cabinet. The cabinet has two layers therefore the agent can move the arm to either layer and subsequently pull the door open finish the task. The episode length is 100.
E Hyperparameters
Here, we list the hyperparameters used for all basclines and tasks.
17

---

Table 1: Hyperparameter setup for all tasks. For RPG, the default hyperparameteess for sparse reward.
Hyperparameter
Num. Environments
256
Critic Learning Rate
5 10 10-4
Actor Learning Rate
3 10
Action Learning Rate
3 10-2 (DDiffPG/DIPO)
Alpha (ox) Learning Rate
5 10-3 (SAC)
__min (distributional RL)
0
V V_max (distributional RL)
5
Num. Atoms (distributional RL)
51
Optimizer
Adam
Target Update Rate (T)
10-2
Batch Size
4,096
UTD ratio
Discount Factor (r)
0.99
Gradient Clipping
1.0
Replay Buffer Size
2, 000 trajectories ss 1 10t (DDifIPG)
1 10t (baselines)
Reclustering Frequency
100 (DDifiPG)
Mode Embedding Dim.
5 (DDiffPG)
Table 2: Clustering threshold, The default threshold is set to 0.7 max(2]:; 2]) corresponding with MAT- LAB(TM) behavior, where Z is the linkage matrix.
Clustering threshold
AntMaze-vl AntMaze-v2
50
70
AntMaze-v3
70
AntMaze-v4
50
Reach Peg nng-in-hole
default
default
Drawer-close
default
Cabinet-open
default
18

---

Additional experimental results.
Table 3: Number of modes discovered, success rate (S.R.), ). and episode length (E.L.) for AntMazes and the with randomly initialized obstacles, averaged over random seeds per case.
DDiftPG
RPG
TD3
SAC
DIPO
Diff- F
Con-AC
#modes
1
0
AntMaze-vl
S.R.
1.0
0.2
1.0
0.98
0.98
0.63
0.0
E.L.
75.7
450.1
59.2
89.1
75.3
241.7
500
#modes
1.5
AntMaze-v2
S.R.
1.0
0.5
0.75
0.53
0.75
0.59
0.0
E.L.
66.8
259.4
35.6
222.3
34.6
225.7
500
#modes
4.3
1
AntMaze-v3
S.R.
0,98
0.1
1.0
0.8
1.0
0.77
0.25
E.L.
142.5
642.1
83.4
207.8
99.3
226.6
545.1
#modes
3.8
0
0
AniMaze-v4
S.R.
1.0
0.0
1.0
0.75
E.L.
151.1
700
88.1
92.8
86.9
249.5
700
#modes
N.A.
N.A.
N.A.
N.A.
N.A.
N.A.
N.A.
Randomized
S.R.
1.0
0.0
0.5
0.45
0.5
0.45
0.1
E.L.
162.3
700
310.2
342.1
293.5
421.4
582.3
Table Number modes, success rate (S.R.), and episode length (E.L.) for robotic tasks, averaged over 20 random seeds.
DDiffPG
TD3
SAC
DIPO
Diff-QL.
Con-AC
#modes
2.8
Reach
S.R.
1.0
1.0
0.95
1.0
0.75
0.5
E.L.
23.8
18.0
20.5
18.6
40.5
60.5
#modes
2
Peg-in-hole
S.R.
1.0
1.0
1.0
1.0
1.0
0.2
E.L.
5.9
4.7
4.7
5.1
4.92
80.91
#modes
3.5
Drawer-close
S.R.
1.0
1.0
1.0
1.0
0.8
0.2
E.I.
23.6
22.0
24.7
22.8
34.5
80.74
#modes
2
1
1
Cabinet-open
S.R.
1.0
0.98
1.0
1.0
0.75
0.5
E.L. 21.1
14.3
24.3
19.6
42.1
59.5
We provide an evaluation of computational time compared with baselines. We use NVIDIA GeForce RTX 4090 for all experiments. However, given the intense research landscape in diffusion models, we hope to make the training and inference more time-efficient our future work.
For data collection, DDifPG needs more wall-clock time than others, which is due to the trajectory processing, clustering, etc. We note that DIPO has similar wall-clock time with the time of TD3 and SAC, implying that the impact of inference speed of diffusion model is not significant.
For policy updates, DDifiPG and DDifiPG (v) require less computational time. This is because TD3 and SAC need estimate the Q-value during policy updates, while DDiffPG and DIPO only need to minimize the MSE loss.
For critic update, DDifIPG requires more wall-clock time due multiple Q-functions.
For target action update, only DDiffPG and DIPO needs to compute the target action. DDiffPG requires more wall-clock time because has multimodal batches and needs compute target action for each sub-batch.
19

---

Dats collection
Pokky uptins
Criti updare
action updare
2000
1800
1000
DDMPP
1D3
SAC
Algortthms
Figure Comparison on wall-clock time.
We implement an alternative clustering approach on the nigh-dimensional state space by training vector-quantized variational autoencoder VAE) learn trajectory representations. We then performed clustering using the codevectors. The visualized cluster performance and projected embedding space verify that VQ-VAE learns meaningful representations and can be used in approach, as an alternative clustering approach. We would like to point that any unsupervised clustering method be coupled with our approach.
BA
((a)) Clustering performance of VQ-VAE.
((b)) Projected embedding space.
Figure 9: Comparison of VQ- VAE clustering and projected embedding space.
TD3-explore
(b) Consisenny-AC-explre
TD3-Q
Figure 10: The rest of exploration maps and density maps in Antmaze-v3.
20

---

DDiAPG
Diffusion-QL
SAC
(c) TD3
Figure 11: (a)-(d Q-value maps of baselines and DDiffPG in Antmaze-vl.
DISIIGG
(b) DIPO
(c) Diffision-QU.
(d) SAC
(6) RPG
( TD3
(g) Consistency-AC
Figure Exploration maps DDifIPG and baselines in AntMaze-vl.
HHHH
(n) DIVIIN
(b) DIPO
(c) DiclissoonQU
(d) SAC
(6) TD3
Figure 13: (a)-(d) Q-value maps of baselines and DDifIPG in Antmaze-12.
H HHHHE
DDiAPG
DIPO
(c) Diffision-QL
SAC
RPG
H
Consisteney-AC
Figure 14: Exploration maps of DDiftPG and basclines in AntMaze-v2.
21

---

DDiANO
DIPO
Diffusion-OU
(6) SAC
TD)
Figure 13: (d)-(d) Q-value maps of baselines and DDitiPG in Antmaze-v4.
(4) DIING
(b) DIPO
(c) Diftision-QU.
(d)
(c) RPG
i TD3
(g) Consistency-AC
Figure Exploration maps DDifIPG and baselines in AntMaze-v4.
-DDiffPG
-DIPO
0.0
-Diffusion-QL
-Consistency-AC
-TD3
SAC
-RPG
02
0.5M
3M
3.50
2:51
0.5M
1M
1.5M
2M
2.5M
34
Step
[e) Antimau--1
(6) Arrmuzoo22
(e) Animazzsd
Figure State coverage in Antmaze-vl, Antmaze-v2. Antrrze-- and Antmaze-v4
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly stated the motivation and explained the contributions made in the paper.
2. Limitations
Question:
Does the paper discussed the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The authors discuss the limitations of the work in the conclusion section.
22

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: does not include theoretical results.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The authors provided pseudocode of the algorithm B implementation de-
tails C and hyper- pateeeparameerss// H used in experiments. The authors provided environment
descriptions D
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code is open-sourced.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The author provided hyperparameters in Section
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We ran each experiment with five random seeds and plotted their mean and standard error. For each datapoint, we evaluated on 20 episodes.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
We evaluated the computational time Section and stated that we used NVIDIA GeForce RTX 4090 for all experiments.
Justification:
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conforms the NeurIPS Code of Ethics.
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
23

---

Justification: This paper is within the area of decision- decision-making, which calls for learning multimodal behaviors from scratch by using diffusion model. Due the wide application areas of decision-making. this work may have a broad practical impact ranging from autonomous systems to cybersecurity. However, the current version of the work only progress in the field of Reinforcement Learning, and there are no immediate impacts on society, etc.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper has no such risks.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The authors cited the existing assets.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The authors designed a series of multimodal tasks, accompanied by detailed description in Section D
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human
Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
24


Reviews:
[{"review_id": "Wmj98qI544", "reviewer": "Reviewer_U1rA", "summary": "This paper aims to solve online RL problems with diffusion policy. It includes 1. a diffusion policy optimization method for diffusion online training. 2. A combination of intrinsic rewards motivated skill discovery method and model-seeking Q-learning to facilitate exploration and prevent mode-collapse behavior. 3. Several self-designed environments where there might  be multiple optimal solutions and thus require expressive exploration policy.", "questions": "None", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper shows diffusion policy has a big potential in online RL because it enables multimodal exploration.\n2. The self-designed environments are a good contribution to the research field by showcasing the necessity of diffusion exploration.\n3. Performance clearly surpasses several baselines.\n\nfollow up:\n\nThe experiments basically support comments in the paper.  The paper sets out to handle the single-mode exploration problem in online RL, and the self-designed environments, unlike most previous classics, allow diverse optimal behaviors and can benefit from multimodal exploration. The experiments show that the proposed method outperforms several classic baselines including some diffusion-based methods.", "weaknesses": "1. The proposed diffusion training objective seems handcrafted and requires a lot of tunning. This may limit the algorithms' further application.\n2. Besides the diffusion optimization methods. Other proposed techniques are more like a good combination of previous work. This indicates limited theoretical novelty.\n3. Code is not provided. For this style of paper, I think code quality is essential, and a mere promise to release the code is not convincing.\n\nfollow up:\n1. The ablation studies are not strong enough to prove the improved performance number actually comes from multimodal exploration. I cannot be certain which part of the method works from the experiments.  More visualization/empirical results/analyses should be given.\n2. The formatting of the table/figure can be greatly improved. For instance, the title of Figure 4 is wrong/incomplete. Table 3/4 is referenced as the main results in the paper but only put in the appendix. \n3. The diffusion optimization results also lack very strong novelty. The loss function is basically a supervised learning loss adapted for online RL, without strong convergence or policy improvement guarantee. Still, the diffusion+online RL theories are a known unsettled and hard problem, so this kind of exploration is fine and meaningful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and insightful comments. We address the reviewer's concerns and questions as follows.\n\n> The proposed diffusion training objective ... further application.\n\nWe proposed diffusion policy gradient, a method that combines RL and behavioral cloning (BC) for stable online diffusion policy updates. Inspired by methods like the Deterministic Policy Gradient (DPG) [1], the policy is optimized to follow the action-gradient $\\nabla_{a} Q(s, a)$.\n\nDirectly backpropagating the action gradient through the diffusion model for policy optimization is known to be difficult and may cause the gradient vanishing problem and instability [2, 3]. This issue was evident in the high variance of the baseline Diffusion-QL in Fig. 4. We have also included additional plots to visualize this issue in the attached PDF. To mitigate this issue, we imitate the $a^{target}$ obtained via action gradient ascent. This way, we obtain an action that follows the updated Q and serves as the target for our BC objective. The solution we provided is general and requires no tuning. This can also be seen in the codebase we provided to the Meta-reviewer. \n\nWe validated our method on 8 challenging robotic tasks with high-dimensional and continuous action spaces, requiring no handcrafting of the policy update objectives. The proposed objective only introduces an additional hyper-parameter, which is the step size $\\eta$ of the action gradient. In Sec. 5.4, we provided a hyper-parameter analysis on the step size and found that a step size of 0.03 can generally work well. As a result, we used this value for all tasks without any further tuning, proving the generality of the proposed method.\n\n> Besides the diffusion ... limited theoretical novelty.\n\nWhile we acknowledge that some concepts used in our paper have been explored in prior works, **the combination of these techniques in an online setting to learn multimodal behaviors using diffusion policy has not been studied before**. For instance, DIPO [4] and QSM [2] focused only on the diffusion policy update objective. They do not study how to take advantage of the diffusion model to represent multimodal action distributions, nor do they consider the additional exploration challenges for discovering and learning multiple behavioral modes online. Our experiments show that naively learning a diffusion policy does not yield multimodal policies and might even be an \"overkill\" if multimodality is not a concern. Additionally, we highlighted the issue of the greedy RL objective in learning multimodal behaviors, which was not examined in prior works.\n\n> Code is not provided. ... the code is not convincing.\n\nWe agree with the reviewer on the importance of open-source code for the community. We have sent the code link of an anonymous repo to the AC, and kindly suggest reaching out to them for access to review it.\n\n> The ablation studies ... should be given.\n\nFirst, we would like to emphasize that our main contribution is learning diffusion policies that capture multimodal behavioral modes. Tab. 3 and 4 show that our approach uniquely masters diverse behaviors. Despite exploring a larger space, Fig. 4 demonstrates that DDiffPG achieves comparable sample efficiency to baselines across all 8 tasks.\n\nSecond, we included DIPO as a baseline, which also serves as an ablation of our framework. As discussed in Sec. 3, Lines 130-137, we modified DIPO for consistency in MDP dynamics and reward function. The DIPO lacks clustering, mode-specific Q-functions, and multimodal batch. However, DIPO fails to explore and learn multimodal behaviors, proving the necessity of our design choices. \n\nThird, we appreciate the reviewer's constructive suggestions and provide additional ablations on mode-specific Qs and the multimodal batch. The attached PDF shows that a single Q-function cannot address the greedy RL objective, guiding the policy toward the first explored solution. Moreover, in the ablation of multimodal batch, we find that while the diffusion policy learns multiple solutions, the modes are imbalanced due to the lack of enforced diversity in the batch, causing minority modes to appear less frequently. Thus, both design choices are crucial for learning multimodal behaviors.\n\n> The formatting ... in the appendix.\n\nWe thank the reviewer for pointing out these inconsistencies. We will update the caption of Figure 4 to \"Performance of DDiffPG and baseline methods in the four AntMaze and robotic manipulation environments.\" Due to space limitations, we have placed Tables 3 and 4 in the appendix. Since the camera-ready version allows for one additional page, we will include these tables in the main body upon acceptance.\n\n> The diffusion ... fine and meaningful.\n\nIn our methodology and concurrent works [2, 4], policy update for a diffusion process cannot follow the usual deterministic policy gradient derivation. The stochastic nature of diffusion models poses significant challenges in deriving formal theoretical guarantees of policy improvement. However, intuitively, our approach updates target actions under the guidance of the Q-function. As the Q-function improves, these target actions are refined accordingly, which in turn trains the diffusion policy on these improved targets. While a formal proof remains challenging, this intuitive process reflects an alignment between the policy and the evolving Q-function, suggesting effective policy improvement in practice.\n\n[1] Silver, David, et al. \"Deterministic policy gradient algorithms.\" ICML, 2014.\n\n[2] Psenka, Michael, et al. \"Learning a diffusion model policy from rewards via q-score matching.\" ICML, 2024.\n\n[3] Wallace, Bram, et al. \"Diffusion model alignment using direct preference optimization.\" CVPR, 2024.\n\n[4] Yang, Long, et al. \"Policy representation via diffusion probability model for reinforcement learning.\" arXiv, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hbzpzeWplQ", "reviewer": "Reviewer_HLNL", "summary": "This paper addresses the challenges associated with employing diffusion policy in online reinforcement learning (RL), particularly the intractability of policy likelihood approximation and the bias towards a single mode. The author introduces the Deep Diffusion Policy Gradient (DDiffPG) method, which decouples exploration from exploitation. For exploration, novelty-based intrinsic motivation and hierarchical clustering are utilized to identify modes, while for exploitation, the author describes the mode-specific Q-function and a multimodal data batch. Empirical evaluations demonstrate that DDiffPG effectively masters multimodal behaviors.", "questions": "+ In Lines 263-267, the author explains that DDiffPG could learn suboptimal paths. Is this statement intended to justify the suboptimal performance compared to TD3? The author suggests that this suboptimal issue can be mitigated by the mode embeddings. It would be more effective to present the best performance and use the suboptimal trajectories as ablations, specifically when blocking the optimal path, to highlight the significance of multiple trajectories.\n+ Why does directly using the action gradient to optimize the policy lead to vanishing gradients and instability? Is this due to the large denoising steps? Including corresponding ablation studies would provide a better illustration.\n+ Unlike the offline setting where trajectories are stable, the replay buffer with the updated Q function results in changed pairs of $(s, a^{target})$. Does training the diffusion model with a supervised framework on continually changing pairs lead to instability in learning? (Lines 126-129)\n+ Why does DIPO, which uses the original $a$ from the buffer, not know the true outcome? I understand that the replay buffer contains the past trajectories $(s,a,r,s')$ (Lines 134-137).\n+ Are the mode-specific Q-functions also applicable to other standard policies?", "limitations": "--", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The application of diffusion policy for multiple modes in an online setting is promising and addresses a previously unexplored area in the literature.\n+ The introduction of a diffusion-based policy gradient method is novel and represents a significant contribution to the field.\n+ The work is well-motivated, and the visualization of multimodal behaviors using antmaze examples effectively enhances understanding and illustrates the practical utility of the approach.", "weaknesses": "+ Several claims require additional support. For instance, the author asserts that standard exploration-exploitation strategies may easily converge towards a single mode (Lines 25-27) without providing theoretical or experimental evidence. Similar issues are present in Lines 35-36 and Lines 52-53. These statements are crucial for constructing the paper's motivation and thus require more substantial support to enhance their reliability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and insightful comments. We want to address the reviewer's concerns and questions as follows.\n> Several claims require additional support. ... enhance their reliability.\n\nWe thank the reviewer for the constructive comment. First, given the RL objective of maximizing the expected return, policies in goal-oriented tasks will converge to a solution once it is discovered. Tab. 3 and 4 experimentally show that traditional RL baselines fail to learn multimodal behaviors even when multiple solutions exist.\n\nSecond, DDiffPG maintains an exploratory mode throughout training, allowing it to continuously discover versatile behaviors. Fig. 6 demonstrates that DDiffPG achieves the highest state coverage, supporting our claim. Additionally, Fig. 12 illustrates a local minima scenario where the ant faces two goals. Baseline models often get trapped pursuing the easier, lower-reward goal, whereas DDiffPG continues to explore and eventually learns to reach the higher-reward goal, helping to escape the local minimum.\n\nThird, DDiffPG achieves explicit mode discovery via hierarchical clustering with the DTW metric, preserves the multimodal action distribution by constructing multimodal training batches for policy updates, and improves the modes collectively with mode-specific Q-functions.\n\n> In Lines 263-267, ... significance of multiple trajectories.\n\nWe agree with the reviewer that presenting the best performance is beneficial. DDiffPG achieves the same success rate as TD3 in Fig. 4 but usually requires more steps to reach the goal in Tab. 3 and 4. In response, we provide the episode lengths that execute only the mode with the shortest path below. We will include these results in Tab. 3 and 4 in the revised version. Note that further training of DDiffPG would lead to further optimizing all paths. We report here the numbers obtained by the policies reported in the paper. \n\n||DDiffPG|TD3|\n|-|-|-|\n|Maze-v1|65.3|59.2|\n|Maze-v2|40.9|35.6|\n|Maze-v3|79.8|83.4|\n|Maze-v4|103.5|88.1|\n|Reach|19.1|18.0|\n|Peg-insertion|5.0|4.7|\n|Drawer-close|20.5|22.0|\n|Cabinet-open|16.8|14.3|\n\n\n> Why does directly ... better illustration.\n\nWe thank the reviewer for the insightful suggestions. Directly backpropagating the action gradient through the diffusion model for policy optimization is challenging and can cause instability due to the Markov chain in the diffusion process and its stochastic nature [1, 2, 3]. This issue is evident in the high variance of the baseline Diffusion-QL in Fig. 4. Additionally, we have included plots in the attached PDF to measure the gradient in Diffusion-QL, confirming the vanishing gradient issue.\n\n[1] Psenka, Michael, et al. \"Learning a diffusion model policy from rewards via q-score matching.\" ICML, 2024.\n\n[2] Wallace, Bram, et al. \"Diffusion model alignment using direct preference optimization.\" CVPR, 2024.\n\n[3] Pascanu, Razvan, et al. \"On the difficulty of training recurrent neural networks.\" ICML, 2013.\n\n\n> Unlike the offline setting ... (Lines 126-129)\n\nIn the proposed diffusion policy gradient, we can tune the step size $\\eta$ of the action gradient for stability. In Section 5.4, we provided a hyper-parameter analysis on the step size and found that a step size of 0.03 can generally work well. However, an aggressive value, e.g., 0.05, could speedup the learning at the early stage but increases the variance. Consequently, we used a step size of 0.03 for all tasks without any further tuning.\n\n> Why does DIPO, ... (Lines 134-137).\n\nAccording to Algorithm 2 in [4], DIPO stores transition $(s_t, a_t, s_{t+1}, r(s_{t+1}|s_t, a_t))$ into the replay buffer, performs gradient ascent on $a_t$, and replaces $a_t$ in the original buffer. Therefore the transition $(s_t, a_t, s_{t+1}, r(s_{t+1}|s_t, a_t))$ no longer aligns with the current MDP dynamics and reward function, due to the replacement of $a_t$. Given that DIPO is an off-policy algorithm, the reuse of these replaced transitions for training the Q-function could be problematic, as the agent is training values of actions that have not been actually played out in the environment, and their true outcome (reward and next state) are unknown.\n\n[4] Yang, Long, et al. \"Policy representation via diffusion probability model for reinforcement learning.\" arXiv, 2023.\n\n> Are the mode-specific ... standard policies?\n\nWe agree with the reviewer that the mode-specific Q-functions may be applicable to other candidate model parameterizations. The most straightforward example is to learn a separate unimodal actor for each Q-function. However, we advocate for the benefits of learning a unified model:\n1. Learning a single model allows to share information (e.g., representations) across modes, which is particularly beneficial for future research, e.g., on image-based-observation tasks [5]. Our multimodal policy learning also draws an analogy to multitask RL, in which the objective is to solve multiple tasks with a single policy rather than separate policies per task, benefiting from knowledge sharing [6]. \n2. Learning separate unimodal policies would significantly increase the computational time and agent interactions with the environment. With separate policies, we need to iteratively update them; however, only one backpropagation is needed for a single multimodal policy.\n3. When a new mode is discovered, our diffusion model continues learning, adding this mode in its landscape, without forgetting previous knowledge. However, if we have separate policies, we have to initialize a new policy and learn from scrach, otherwise additional techniques have to be introduced to transfer knowledge from previous training. \n\n[5] Kalashnikov, Dmitry et al. “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation.” CoRL, 2018.\n\n[6] Hendawy, Ahmed et al. “Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts.” ICLR, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wHQlxr08RP", "reviewer": "Reviewer_TVD7", "summary": "This paper introduces DDiffPG for online reinforcement learning with multi-modal behaviour discovery. DDiffPG consists of two parts: 1) a new policy improvement method to stabilise the diffusion policy by cloning a target action; 2) a mode discovery mechanism to train mode-specific and intrinsic Q functions. In their experiments, the authors have shown that DDiffPG can achieve comparable performance with the baselines while producing multi-modal behaviours, which provides a series of benefits like avoiding mode collapse.", "questions": "- In section 4.2, line 183, I’m not fully convinced that the RL objective can skew the policy towards a single mode. Suppose we have a Q function that nicely captures two modes. During policy improvement, let’s say we sampled a batch of trajectories that equally captures both modes, and we perform policy improvement by $\\max \\mathbb{E}\\left[Q(s, a)\\right]$. Given that our Q function already nicely captures both modes, why does such an objective cause mode collapse? Could you provide more explanations? Considering the success of DDiffPG on capturing the multi-modality in the policy space, is this really because of the way you perform policy improvement in Eqn. 1, or is it because the DDiffPG used multiple Q functions for separate modes, which just better fits the multi-modal distribution?\n\n- Regarding the use of mode-specific Q functions, it is a bit unclear to me how to stabilise the training. One issue is that during online exploration, the dataset is continuously being updated and modes are being updated. In this case, how do we fix the correspondence between the Q functions being learned and the mode? Besides, according to line 167, DDiffPG requires no predefined number of clusters, and the number of modes could be dynamic. However, we have to initialise a fixed number of Q functions. This seems a bit contradictory to me. How to define the number of Q functions during training?\n\n- It seems to me the exploration is only guaranteed by the training of $Q_\\mathrm{explore}$ using RND. However, how do we balance the exploration and exploitation during RL?", "limitations": "- In line 175, the definition of DTW requires task privileged information, e.g., object positions. This is a major issue and unrealistic. In more realistic scenarios, people normally have no access to such information, and as a result, this clustering approach is not applicable.\n- The paper only conducted experiments on 8 simple state-based environments. It is unclear how the approach could generalise to more realistic environments and tasks with visual observations. Although at the current stage, I understand that the current results reasonably demonstrate the capability of the proposed method, more realistic tasks will better support many claims made by the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper has introduced an interesting idea. To the best of my knowledge, this is the first work that allows diffusion policy to learn multi-modal behaviours during online RL. According to the experiments, the proposed method has produced a reasonable performance with nice multi-modal behaviours. Besides, the paper has provided nice visualisations and discussions to help understand the proposed approach.", "weaknesses": "There are several main weaknesses of the paper.\n\n- The paper is hard to follow and the presentation has a certain room for improvement.\n\n- In section 3, the formal theoretical derivation of the newly introduced policy improvement objective is missing. Although it shows that this method worked empirically, it remains unclear how the resulting policy theoretically maximises the expected return in general.\n\n- I feel the paper is a bit over-claiming for certain aspects. In section 5.3, the authors claimed that DDiffPG can *overcome* local minimum issues and encourage exploration. However, the exploration comes from the use of RND when learning $Q_\\mathrm{explore}$, rather than the architecture itself. In addition, it is a very strong claim that DDiffPG **overcomes** the local minimum issues. The experiments are conducted on only 8 state-based tasks, from my point of view, which is insufficient to support such a general claim. I understand that by capturing multi-modal distributions, DDiffPG allows better generalisation, but I would suggest the authors moderate the claims a bit.\n\nMinor issues:\n- In line 157, is this a typo? In $r^\\mathrm{intr}(s, a, s’) = \\max(\\mathrm{novelty}(s’) - \\alpha \\mathrm{novelty}(s’), 0)$, should this be $r^\\mathrm{intr}(s, a, s’) = \\max(\\mathrm{novelty}(s’) - \\alpha \\mathrm{novelty}(s), 0)$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. Due to the rebuttal limit and the number of questions, our responses are concise. We are happy to provide more detailed answers during the discussion period.\n\n> The paper is hard to follow ... for improvement.\n\nBased on the reviewer's feedback, we will:\n1. Moderate our claim about overcoming local minima and clarify DDiffPG's continuous exploration ability in Sec. 5.3.\n2. Fix the typo in the intrinsic reward definition.\n3. State clearly scenarios where an imperfect Q-function can lead to mode collapse.\n4. Highlight the reclustering process in Sec. 4.2 and the exploration-exploitation balance in Sec. 4.3.\n5. Clarify the information used in DTW computation, such as robot positions.\n\n> In section 3, ... in general.\n\nIn our method and concurrent works (Yang et al. 2023, Psenka et al. 2024), policy update for a diffusion process cannot follow the usual deterministic policy gradient derivation. The stochastic nature of diffusion models poses significant challenges in deriving formal theoretical guarantees of policy improvement. However, our approach updates target actions under the guidance of the Q-function. As the Q-function improves, these target actions are refined accordingly, which in turn trains the diffusion policy on these improved targets. While formal proof remains challenging, this intuitive process reflects an alignment between the policy and the evolving Q-function, suggesting effective policy improvement in practice.\n\n> I feel the paper ... moderate the claims a bit.\n\nWe apologize for any ambiguity in our claim. **We mean that the continuous exploration capability of DDiffPG helps increase state coverage and overcome local minima**. This capability comes from the exploratory mode, not from RND alone, as the **same intrinsic reward** was used **for all baselines**. DDiffPG maintains an exploratory mode throughout training, allowing it to discover versatile behaviors continuously. Unlike baselines that quickly converge to the first explored solution, DDiffPG keeps exploring, which is crucial for learning multimodal behaviors online. Fig. 6 and 12 support our claim, showing that DDiffPG achieves the highest state coverage, while overcoming the lower-reward goal, reaching the top-left goal with the higher reward, helping to escape the local minimum.\n\n> In line 157, is this a typo? ...\n\nYes, the intrinsic reward is defined as the novelty difference between $s'$ and $s$, which encourages to visit the boundary of explored region.\n\n> In section 4.2, ... distribution?\n\nWe agree that a perfect Q-function nicely capturing both modes could prevent mode collapse. However, in practice, achieving a perfect Q-function is unlikely since both the policy and Q-function are learned from scratch. This often leads to:\n\n1. the policy being guided towards the first explored goal.\n2. the policy converging to the solution with higher Q-values due to the discount factor, even if two modes are initially captured.\n\nDDiffPG's design mitigates mode collapse through mode-specific Q-functions, each focused on a single mode, thereby isolating their Q-values. By sampling actions from the replay buffer rather than relying solely on policy-inferred actions, we ensure balanced improvement across all modes. However, **policy improvement alone is not sufficient for multimodal behavior learning**, as confirmed by the results of DIPO (Tab. 3 and 4).\n\n> Regarding the use of mode-specific ... during training?\n  \nYes, we re-cluster trajectories every $F$ iterations, updating modes continuously. After re-clustering, new clusters inherit Q-functions and $a^\\text{target}$ from the most overlapping cluster in the previous iteration. If the number of clusters increases, we initialize new Q-functions. This ensures a one-to-one correspondence between modes and Q-functions without needing to predefine a fixed number of Q-functions. This process is explained in Sec. 4.2 (lines 210-214), with further details in Alg. 2 and App. C. The implementation can also be found in our provided code to the Meta-reviewer under `ddiffpg/utils/Q_scheduler.py`.\n\n> It seems ... during RL?\n\nWe balance exploration-exploitation by adjusting the proportion of the exploratory mode during data collection. As explained in Sec. 4.3 (lines 230-233), we condition the diffusion policy on mode-specific embeddings, including an exploratory mode trained by $Q_\\text{explore}$. We set the proportion for exploration to $p = \\max(0.3, 1/\\text{modes})$, ensuring at least 30% of actions are exploratory. Implementation details can be found in `ddiffpg/algo/ddiffpg.py` (lines 104-129).\n\n> In line 175, ... not applicable.\n\nWe apologise for any confusion. **We only use the robot position for clustering, i.e., the Ant's position and the robot end-effector's position, which are accessible in real-world scenarios**. We do not use any information about objects or goals in the environment. In the PDF, we provide an alternative clustering approach by training a VQ-VAE to learn trajectory representations and using the codevectors for clustering, which learns meaningful representations suitable for our approach.\n\n> The paper only ... by the paper.\n\nWe want to emphasize that the 8 robot locomotion and manipulation tasks are nontrivial. They are all **high-dimensional and continuous control** tasks with **sparse rewards**. In the AntMaze tasks, the agent must solve the Ant-locomotion task through joint control, a notoriously difficult continuous control problem, while also learning to steer locomotion gaits to find goals in complex mazes. All tasks are **goal-agnostic**. The agent must explore to find the goal, making it challenging to learn multimodal behaviors due to the vast exploration space. As the first work exploring online multimodal diffusion policies, we see our current tasks as a strong starting point and plan to extend our approach to long-horizon and vision-based tasks with greater multimodality in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for their feedback and constructive suggestions on our manuscript. We are glad that the reviewers find our paper to be:
* introducing a novel and interesting idea (Reviewer HLNL, Reviewer TVD7)
* having informative and good-quality visualizations and experiments (Reviewer U1rA, Reviewer HLNL, Reviewer TVD7)
* showcasing superior performance againt baselines (Reviewer U1rA, Reviewer TVD7)
* having practical impacts on the community (Reviewer U1rA, Reviewer HLNL)

In response to the reviewers concerns we add the following responses

### Code Release

As Reviewer U1rA suggested, we sent an anonymous repo to the meta-reviewer to be distributed internally.

### Additional experiments in attached PDF
1. As Reviewer U1rA suggested, we conducted additional ablations on mode-specific Q-functions and multimodal training batch. Based on the state visitation that demonstrates the behavior of the policy during training and the behaviors in evaluation, we found that a single Q-function cannot address the issue of greedy RL objective. For the ablation of the multimodal batch, while the diffusion policy learns multiple solutions, the modes are imbalanced due to the lack of enforced diversity in the batch, causing minority modes to appear less frequently. Thus, both design choices are crucial for learning multimodal behaviors.
2. As Reviewer HLNL suggested, we measure the actor gradient norm between our approach and Diffusion-QL to verify the gradient vanishing problem. As shown in the figure, the actor gradient of Diffusion-QL is very small and particularly, the high variance in the results reported in Fig. 4 indicates that the gradient of certain seeds remains zero throughout the training.
3. As Reviewer TVD7 suggested, we implement an alternative clustering approach on the high-dimensional state space by training a vector-quantized variational autoencoder (VQ-VAE) to learn trajectory representations. We then performed clustering using the codevectors. The visualized cluster performance and projected embedding space verify that VQ-VAE learns meaningful representations and can be used in our approach, as an alternative clustering approach. We would like to point that any unsupervised clustering method can be coupled with our approach.

The figures and results are included in the attached PDF. Next, we will address each reviewer’s concerns individually.

Meta-review: This work proposes a novel combination of DDPG and diffusion. The work overcomes some of the challenges in training diffusion policies in online learning settings. The reviewers agree that the contribution is novel and an interesting way to create a policy that can better explore multi-modal behaviors. However, the authors should be careful with wording and overclaiming with respect to the limitations of prior methods that may or may not be able to overcome multi-modal exploration.
Decision: Accept (poster)