Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularization
Authors: Ziqi Wang
Jiashun Liu
Ling Pan
Venue: NeurIPS 2025 poster ()
Keywords: reinforcement learning
multimodal policies
diversity
Primary area: reinforcement_learning
TL;DR: We introduce an RL algorithm leveraging reparameterization and distance-based diversity regularization to train intractable multimodal policies for diversity-critical tasks.

Abstract: Traditional continuous deep reinforcement learning (RL) algorithms employ deterministic or unimodal Gaussian actors, which cannot express complex multimodal decision distributions. This limitation can hinder their performance in diversity-critical scenarios. There have been some attempts to design online multimodal RL algorithms based on diffusion or amortized actors. However, these actors are intractable, making existing methods struggle with balancing performance, decision diversity, and efficiency simultaneously. To overcome this challenge, we first reformulate existing intractable multimodal actors within a unified framework, and prove that they can be directly optimized by policy gradient via reparameterization. Then, we propose a distance-based diversity regularization that does not explicitly require decision probabilities. We identify two diversity-critical domains, namely multi-goal achieving and generative RL, to demonstrate the advantages of multimodal policies and our method, particularly in terms of few-shot robustness. In conventional MuJoCo benchmarks, our algorithm also shows competitive performance. Moreover, our experiments highlight that the amortized actor is a promising policy model class with strong multimodal expressivity and high performance.  Our code is available at https://github.com/PneuC/DrAC

Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularization
Ziqi Wang
Jiashun Liu
Ling Pan"
Hong Kong University of Science and Technology
Abstract
Traditional continuous deep reinforcement learning (RL. algorithms employ deter- ministic or unimodal Gaussian actors, which cannot express complex multimodal decision distributions. This limitation can hinder their performance in diversity- critical scenarios. There have been some attempts to design online multimodal RL algorithms based on diffusion or amortized actors. However, these actors are intractable, making existing methods struggle with balancing performance, decision diversity, and efficiency simultaneously. To overcome this challenge, we first reformulate existing intractable multimodal actors within unified framework, and prove that they can be directly optimized by policy gradient via reparameteri- zation. Then, propose a distance-based diversity regularization that does not explicitly require decision probabilities. We identify two diversity-critica domains, namely multi-goal achieving and generative RL, to demonstrate the advantages of multimodal policies and our method, particularly in terms of few-shot robustness. In conventional MuJoCo benchmarks, our algorithm also shows competitive perfor- mance. Moreover, our experiments highlight that the amortized actor is promising policy model class with strong multimodal expressivity and high performance. Our code is available at https:: // /ggthtbt.b..../. com PneuC/DrAC
Introduction
Despite the remarkable progress in reinforcement learning (RL) for
Training ewircnment with multiple guals
continuous control and decision-making tasks 23
55
43
learning
multimodal policies remains challenging 220
33
48
However,
most statc-of-the-art RL algorithms predominantly l employ determin- which limits their ability to capture the complex, multimodal decision distribu- tions. However, real-world applications can have multiple goals, making multimodality and decision diversity essential [13] 28 41, where unimodal policies can be brittle to perturbations in the environ- ment. For example, given navigation task in maze with multiple valid paths, unimodal policies typically converge a single shortest route, creating critical vulnerability when this route becomes ob- structed during deployment, such policies fail catastrophically [26). In contrast, multimodal policy with high decision diversity learn varied navigation strategies, allowing to adapt when faced with unexpected obstacles. In zero-sum games, strategic diversity that requires multimodal policies is also critical for agents to main- tain robustness against adaptive opponents [3I]. Additionally, RL
istic or unimodal Gaussian policies 22
39
14
Test 1, goal univallable
Test secuatio 2, obstacks blocked purtal paths
Figure 1: Motivational exam- ples. A multimodal policy with maximized diversity can achieve multiple goals and en- able robustness against envi- ronmental changes.
Coorespondence to: Ling Pan (lingpan @ ust.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

has been applied to generative tasks in recent years M. where multimodal policies can potentially improve the diversity of generated objects with less harm to quality [50].
The main challenge of learning expressive multimodal policies lies in their intractability. Existing online RL algorithms that train multimodal policies are typically based on the maximum-entropy RL framework [14]. However, entropy regularization encourages policies to be more uniform, but do not specifically prefer multimodality. Meawhile, computing entropy requires an analytical expression of decision probability, which is not directly applicable to intractable actors. Some prior works attempt to mitigate this problem with a workaround that employs tractable but less expressive multimodal actors [32 2 (5], sacrificing expressivity. Another line of research leverages Stein variational gradient descent (SVGD) as gradient estimator [13] or as a sampler that directly samples actions from the value distribution learned by the critic [33 However, we empirically observe that it either underperforms traditional algorithms or incurs significant computation costs. Besides, there have been some online RI. algorithms training diffusion actors 53 36
48), but to our best knowledge, only the diffusion actor-critic with entropy regulator (DACER) 48 takes decision diversity into account among these algorithms. DACER controls diversity by scaling an additional unimodal Gaussian noise instead of optimizing diversity through gradients w.r.t. the actor's parameters.
In this paper, we propose a novel challenge. First, this paper shows that existing multimodal actors can be viewed as combination of a parameterized mapping function with fixed latent random distribution, which we call stochastic- mapping actors. We then highlight that the policy gradient of any stochastic-mape mapiing actor can be estimated via the reparameterization trick, without accessing the gradient of decision probabilities. This provides a general methodology to optimize intractable we propose a distance- based diversity regularization instead of entropy, which does not explicitly require the decision probability. Combining with the policy regularization theorems [48], we develop a novelty actor-critic algorithm named Eiversity-regularinn Actor Critic (DrAC). Main contributions of this paper are summarized as follows
RL framework to address the aforementioned
mappang actors. Secondly,
1. We formulate a class of actors named stochastic-mapping actors, bridge policy gradient and intractable stochastic actors via reparameterization trick. This formulation serves as a useful tool for understanding and designing multimodal RL models and algorithms.
2. We propose learning framework with a distance-based diversity regularization, which simultaneously optimizes expected return and decision diversity without requiring explicit access to decision probabilities.
3. We identify two domains where multimodality and diversity are critical, and conduct comprehensive experiments, including performance in standard MuJoCo benchmarks. In addition to justifying the advantages of multimodal policies and our algorithm, we also present new insights on multimodality and diversity.
2 Related Works
Policy Diversity in RL Decision diversity (or randomness) is an essential ingredient of online which not only enforces exploration but also reduces variance .
deep RL algorithms 29 Some research has shown that decision diversity can improve sample efficiency or final performance
regarding expected return [6
34
19
54 6). Furthermore, diversity can serve in pretraining.
Eysenbach et al. proposed that training policies with only a diversity objective can discover useful skills and serve as an effective pretraining method (9]. Ying et al. pretrain diffusion actors with intrinsic reward to discover diverse behaviors and fine-tune diffusion actors to fast generalize to downstream tasks [56]. Besides, Zhang et al. propose an algorithm to find varied policies for the same task [57]. Kumar et al. train multiple policies with diverse behaviors, and highlight that diversity is the key to ensuring few-shot robustness in -iittri-ittiittiinn scenarios 26
Another category of research on policy diversity in RL considers diversity as an independent objective distinguished from reward. An emerging topic called quality-diversity RL aims learning a set of policies that cover a task-specified behavior space as completely as possible, and maximizing the RL is also applied in generative tasks, including code generation, music generation, artificial intelligence for science, and game content generation [.. Some of these generative tasks treat diversity as an essential need besides quality. For example, Popova et al. formulate drug design as a Markov decision process (MDP) rewarded by a predictive
quality of each policy at the same time 17 51 52
2

---

model to generate diverse new chemical structures [35]. Wang et al. leverage diversity-driven ensemble RL generate diverse game levels with high quality [50].
Learning Multimodal Policies An early attempt in online multimodal RL is soft Q-learning (SQL) [3]. It proposes the maximum-entropy RL framework with an amortized actor. As the actor loss in maximum-entropy RL is KL-divergence, SQL leverages SVGD to estimate the gradient of the actor loss. Stein soft actor-critic (S? AC) [3] views the entropy reguuap-eee critic as an energy that directly samples actions based on it via SVGD. To improve inference speed, SCAC proposes to train an additional amortized actor that mimics the SVGD sampler. However, our experiments show that SQL performs poorly in many benchmarks, including MuJoCo, while training S AC is much more memory-hungry and slower. Huang et al. propose a model-based RL method learning through evidence lower bound train multimodal actors [20). To our best knowledge, fast and effective algorithm to train an amortized actor was absent prior to our work.
Another line of multimodal RL builds upon diffusion actors. There has been lot of work investigating
offline RL. based on diffusion actor [58 49 24 3. but online methods have been a few. Specifically,
Yang et al. [53] apply approximated policy improvement
generate target actions and train the diffusion actor to match the target actions. Psenka et al. [36 train the diffusion actor by matching the ccore-based structure with the action gradient of the Q-function. Ren et al. model the diffusion process as low-level MDP and train the diffusion actor through a two-stage policy gradient [37]. Consistency policy methods have been demonstrated feasible train diffusion actors online 77.
Wang et al. propose DACER, an online RI. algorithm with entropy regulator to train diffusion actor 48). taking decision diversity into consideration by automatically tuning the scale of an additional action noise. In addition, Ying et al. pre-train a Gaussian actor with intrinsic reward [56] to collect trajectories, and then fine-tune a diffusion actor in downstream tasks. Li et al. cluster the trajectories explored by learning with intrinsic rewards to discover modes, then utilize behavior cloning to train diffusion actor (28] Jain et al. propose an energy-based -based training method for diffusion actors [2]
Ishfaq et al. employ diffusion model to generate synthetic training samples 211.
There has also been some work leveraging tractable multimodal policy models 38 50
Mazoure
al. leverage normalizing flow to improve exploration (32). Wang et al. represent decision distribution by Gaussian mixture model to generate diverse game levels [50]. However, these models may be weaker in expressivity compared with intractable models.
3 Preliminaries
3.1 Deep Reinforcement Learning
Deep reinforcement learning 41 aims optimizing policy powered by deep learning to solve an MDP. An MDP involves state space S, an action space A, a reward function r(.:)) S x A + R, and a transition dynamics modeling the transition probability from one state to another given an action. Specifically, this paper focuses on the continuous action space. At each time step t. the policy observes a state S₁ E S to determine a decision distribution π (-I(St) and draw an action A, E AA from (-]SS), and then receives a reward signal R2 r(Sr At). The objective of RL is optimizing ı to maximize the expected return J(xx) = F D 7 RR). A state-action value function conditioned by policy и is defined as Q* (s((s. a) E,, [> k= RR+ SS = s, A₁ = aj evaluate state-action pair. Prevalent continuous deep RL algorithms [29] 39 [1 primarily follow the
actor-critic architecture. In this framework, an actor Ty models the policy, while critic Qo learns an approximation of the Q-function conditioned by The actor is typically optimized by policy
gradient [40 based on Q-value approximated by the critic.
3.2 Policy Regularization
Maximum entropy RL 59
4 45) is a classic framework to optimize both expected return and
diversity. Haarnoja al. introduce deep learning framework for intropy-regularizer RL (14), which modifies the standard RL objective by adding an entropy regularization ((T) = E, EEep (R, + (HH()()[S)).. where H(-) indicates the entropy of given distribution. The state-action value
function is defined Qun(s, (s) E[R, Di=1 x (Be+k + (H)(s(( A, = a). The
30
3

---

policy is optimized by applying soft policy iteration (SPI) operator as follows
exp Q 2on Aan
(s,
Nnw f argmin DKL
[[-|s)
(1)
where ZT (s) is scalar to normalize the distribution, which is intractable but negligible in practice.
Wang al. extend the entropy-regularian
RL framework general form [50]. Given arbitrary
regularization function (-s{(-))), and a regularized learning objective J(xz) = B 1 (R, ag((T (-(((())))))) (-1),))), the regularized Q-value is defined in the following form
000
Oig (s. Ex Re Ex_1 7 (Re+k ag(=(-1S(++))) S s, =
(2)
With the regularized Q-value defined, the policy can be optimized by regularized policy iteration with a policy improvement operator expressed as
Tnew (-1s) +- argmax [<ve(T(-|s)) B (s, a)]]
for all S. Furthermore, the policy gradient of the regularized learning objective is
(Z)(m)= = Eggd [Vee(Te(-1s)) Q (s, da
(4)
where d' is the discounted state distribution given n
Method
We propose
actor-critic (DrAC), which learn intractable multimodal policies through reparameterization and diversity regularization. Fig. illlsstrtes the key ingredients of DrAC. A detailed pseudo-code is provided in Appendix A.
Amortized actor
Diffusion actor ma]lal
Qia learned by the critic
"3
fols.-)
Backpnpager gradeuus
Mo.
Ty-a momn
Laver disteatce
Gradiem of maximizing
Gandiem moximiziing diversity
(b) We train diversity- regularized critic and directly back- propagate the gradient of and diversity the actor's weights, as VoQ(s, fo(s, =)) estimator policy gra- dient via reparameterization trick.
(a) We unify amortized actors and difffusion actors within stochastic-men mapping formulation me (fo. Pa) } (See Scction
Figure 2: Key ingredients of our proposed method.
4.1 Unifying Intractable Multimodal Policies and Policy Gradient
We define class of stochastic-mapping actors as mo = (Jo,P2), which combination of parameterized function fo(:..) : x 2 + A, and fixed latent distribution P= over latent space Z. The action is drawn by sampling an independent random variable from P= first, and then feeding state and into fe. i.e., ma (-1s) +- fo(s, z ps. We reformulate existing multimodal actors, namely amortized actors and diffission actors, under this definition as follows.
Amortized actors This type of actors 13 employ a neural network (NN) ge- which takes the state and the latent variable z as input, and directly output the action a. We follow the most straightforward mechanism in SQL, concatenate and z directly feed into Formally, this is
f.Amorr (s, 2) (o $ z).
(5)
This definition follows
13 and 33 is narrower than the concept of amortized inference.

---

Diffusion actors This type of actors [3 37 48 build upon the diffusion models [8] which are powerful generative models. We adopt the implementation from DACER [48]. The policy is parameterized by an NN ca(t)'s-), and can be formulated by = Po) as follows
f M (s, 2) = 20:
Bt VI
It
s0(8, + +v V
(6)
at
2T.
In Eq. 6 a and 3t are scalars calculated by some specific schedule mechanisms to control the diffusion process, 20, 21,". 2r are i.i.d. noise vectors drawn from ( 1) and we have treated 2 = ... 2r). : the number of diffusion steps. Note that the t's here are not the time steps of MDP but the time steps of the diffusion process.
These actors are powerful, but their decision probabilities Ta(als) does not have a closed-form expression, making policy gradient not directly applicable. However, as they follow the stochastic mapping formulation, is feasible to calculate policy gradient via reparameterizzation trick (PGRT) mentioned in [27] by the equation below
(mo) = Bend VoQ(ss.lsss..))Vaff...
(7)
PGRT demonstrates that we can directly backpropagate the Q-function's gradient to fe to train stochastic-mane mappsngg actors, where the Q-function can be approximated by differentiable critics. We noticed that DACER [88] backpropagates the gradient of the Q-function to train diffusion actors, but the paper of DACER did not mention the relation to policy gradient and did not disclose that this
method can be generalized to other actors.
4.2 Actor-Critic Learning with Diversity Regularization
As pursue decision diversity, effective diversity regularization is desired. As discussed earlier, the traditional entropy regularization do not prefer multimodality and is not applicable intractable actors. To overcome this challenge, we propose distance-based regularization to encourage multimodal diversity.
A straightforward based diversity metric is the mean pairwise distance, formulated as D (s) = (1)) where -). is a distance metric. However, this metric may overestimate the diversity when the data distribution forms some tiny clusters far away from each other (See Fig. 33 We empirically found that this issue led to underwhelming performance. Therefore, we consider the geometric mean
Avg.
[0(2 y)],
X 3.663 3.131
Y 8.758
2,248
GM(()(2), ..),... 5(2n, vn))) (T- 5(2,, 9:)) 1/x of pairwise distances instead. The geometric mean is sensitive small values as multiplication is taken instead of summa- tion, which mitigates the overestimation issue (See Fig. (3) Furthermore, reshape diversity on the log scale make casier to balance reward and diversity. We propose a diversity regularization follows
Y. misture of Nijs nan and NCC-6, n2..
Figure 3: Average (Avg.) and geo- metric mean (GM) of pairwise dis- tances for two synthesized data dis- tributions. Average pairwise dis- tance overestimates the diversity of distribution Y.
D (s) = (1) 6(x, v)].
(8)
Specifically, distance is adopted as the distance metric. This diversity regularization is added to the original RL objective with coefficient a, resulting an objective to maximize as
J= Er En
(R, aDD((.))
We estimate the diversity of the actor at a certain state by sampling n pairs of i.i.d. latent vectors (2 2²).... (> 2!), and compute the average log- dis-aistance over samples. This is denoted 2))). Though this estimation requires multiple samples, time-efficient such process can be easily parallelized on GPUs.
as ()(s)= E log ((fo(s =(")) 2f), fo
5

---

The practical DrAC algorithms follow the common actor-critic architecture. The actor is a stochastic- mapping actor, and the critics follow the learning tricks in soft actor-critic (SAC) [14] Double crities Q(-:00)) Q( 1t @2) and double target critics ((::;:)) é1). -i d2) are employed. By substituting Do into Eq. (2 and unrolling it into a bootstrap learning target, the critic loss is derived as
c = Esansn MSE(Q(s, a: 6.), (V((''))) + aDo(s')) ; {1,2]
(10)
where D represents the off- ool-ppolcy replay buffer, (is:: (1.2) Q a'; Φ:) is an approxima- tion of the value of the next state s' based the target critics and a' is sample drawn from ma(+1s"). The target critics are updated by the target smoothing strategy [1
Regarding actor learning, combine PGRT (See Eq. theorem (See Eq. 44 derive a loss function as follows
7 with the regularized policy gradient
Le = - E.. [Q( fa(s, Ø) + aDo(s)),
(11)
where Q(s, fo(s, 2); Φ) = (1.2. Q(s, fo 2); Øz). following the technique used in SAC. This loss function can also be considered an approximation of Eq. 3
4.3 Automatic Coefficient Adjustment based on Target-Diversity
Multimodal actors are expressive and flexible. However, given the diversity regularization with a fixed coefficient, such properties can also lead to instability in critic learning, which may harm the training. To mitigate this issue and also make hyperparameter tuning easier, we borrow the automatic coefficient adjustment technique from SAC [TS]. Specifically, we parameterize with a learnable scalar w such that a = exp(w) if w < and a = w + 1 otherwise. Then a is treated as learnable and updated by descending the gradient of dual-optimization loss defined as
La EgwD (as D)},
(12)
where is target diversity value. Minimizing this loss will make Do(s) match with D adaptively, As Do(s) is defined on the log-scale, determining its range can be trivial. We introduce temperature hyperparameter 3 such that D / /-Al) ease the triviality. The scale of is comparable to pairwise distances after normalization by the action space dimensionality |.A].
Experiments
Domains
Previous works on multimodal RL 13
33 48
only verify the multimodal ability with
toy 2D multi-goal navigation task and mainly focus on the improvements in MuJoCo. To demonstrate the advantages of DrAC and multimodal policies, we include two diversity-critice benchmarks in multi-gon goal achieving and generative RL domains. We also test DrAC and representative multimodal RL algorithms in standard MuJoCo benchmarks compare the general performance.
Baselines By installing DrAC with an amortized actor and a diffusion actor separately, we obtain two versions of DrAC, namely DrAmort and DrDiffus. We include SQL [13], DACER (48, S AC 33 as multimodal RL baselines, and SAC [15] as a unimodal baseline. Table summarizes the techniques of these algorithms. same as the default setting of SAC to enable fair comparison 14). Appendix A.2 details the
keep NN architectures and common hyperparameters all the
hyperparameters. All trainings are conducted with five seeds.
5.1 Multi-goal Achieving
In real-world scenarios, there could be multiple goals to achieve. We build multi-goal version of the PointMaze environment in D4RL [10] conduct a case study of multi-goal achieving. This environment requires the agent to leam to steer ball from a fixed original point to navigate to one of the goals within a maze. The maze contains multiple goals. The reward sparse, only reaching goal returns 100 reward and ends the episode with a success flag. Besides the success rate, the number of reachable goals is also evaluated. We designed three maze maps with increasing difficulty levels, as illustrated in Fig. 9 We notice that the work of (28) tested more complex multi-goal achieving

---

Table 1: summary of compared algorithms. Bold texts indicate our contributions. The "Temperature"
column indicates the technique of adjusting the emphasis on diversity used by each algorithm.
Actor Model
Actor Learning
Critic Learning
Diversification
Temperature
DrAmort
Amortized
Regularized PGRT
Conventional
Max-Diversity
Target-Matchin
SQL [33
Amortized
SPI via SVGD Regularized PGRT
Conventional
Max fax-Ennropy
Constant
DrDiffus
Diffusion
Conventional
Max-Diversity
Parget-Matching
DACER 48
Diffusion
PGRT SPI via SVGD SPI
Distributional 8
Noise Scaling
Banget-Macciig
ACAC 33
Energy-based
Conventional
Max-Entropy
Constant
SAC
14
Gaussian
Conventional
Max-Entropy
Target- uaget-Matching
benchmarks. However, that work autonomously collects and clusters trajectories first, then clones behaviors using these clusters, while our work focuses on basic RL algorithms. To ensure a fair and meaningful comparison, we conducted grid search for each algorithm before the formal evaluation. The highest temperature that ensures each algorithm reaches the optimal success rate is picked.
The trajectories in the simple maze of learned policies visualized in Fig- It is clear that DrAmort learns the most diverse and uniform trajectories. Despite using the same diffusion actor, DrDiffus learn to reach different goals, DACER does not. Fig. shows that DrAmort and SQL consistently learns to reach the most goals in all mazes, and DrAmort learns faster than SQL. In terms of success rate, DrAmort, DrDiffus, and DACER learn fast and stably, while SQL, SAC, and S AC exhibit decrease in success rate in the early stage and reach optimal success rate more slowly. Given the results, we consider that amortized actors possess superior multimodal expressivity. DACER struggles with learning multimodal behaviors in PointMaze. This is most likely due to DACER only controlling the diversity via noise scaling. Meanwhile, SQL shows better multimodality than DrDiffus. The underwhelming multimodality of diffusion actors may be owing to a small number of diffusion steps of 20 we used as suggested in [48]. However, despite using a small number of diffusion steps, the inference and training speed is much slower than amortized actors. Therefore, we advocate that the amortized actor is a promising actor class for representing multimodal policies. Trajectories in medium and hard mazes are presented in Appendix B.2
Dikraat (twsttiee Acted
50. (hnotiers Acterl
Droites www.uiiiiiioooooooo Actort
DACER (DBhainnkkli)
SAC (Daziiaa Artert
S/AC Sinegy faaed Acted
Figure 4: Evaluation trajectories of all tested algorithms the simple maze.
Strgle
Medium
Hydd
Single
Medium
Hordl
Tabinng depl
(a) Success rate
(b) Number of reachable goals
Mediu
Simple
Mediu
020 0.0
210
8.3
050
6.5
2.5
3.1
(c) Robustness against removal
(d) Robustness against obstacle
Figure 5: Learning curves in multi-goal PointMaze. Solid lines represent mean performance, and shaded regions indicate standard deviation. All curves are smoothed by the exponential moving average with coefficient of 0.5.

---

Inspired by (26], we investigate whether diversity empowers few-shot robustness in out-of-distribun scenarios. We consider two types of perturbations: 1) Removal: half of the goals are removed during the test; 2) Obstacle: some obstacles that do not exist in the training phase, appear in the test phase and block the shortest paths to the goals. The first case requires agents to reach all goals with uniform frequency. The second case requires the agents to navigate to the goal with diverse trajectories, so that the they can bypass the obstacles by chance. Similar to [26), we test the five-episode success rate of all algorithms in each scenario. That is, the probability that the agent can successfully reach any goal at least once within five independent trials. The results are shown in Fig. 5c and Fig. 5d
DrAmort shows the best robustness in all mazes, while SQL outperforms others among the rest algorithms. According to Fig. DrAmort exhibits the best trajectory diversity, so has more chances to bypass the obstacles. Meanwhile, DrAmort reaches goals more uniformly, so higher robustmess against removal is guaranteed. The results confirm that multimodality and diversity empower few- scenarios. DrDiffus outperforms DACER regarding robustness
shot robustness in against removal. Regarding SAC, although a high temperature is set, it struggles in learning to reach multiple goals and generalizing to out-of distriibittiitoinnn test cases. This phenomenon confirms the fundamental limitation of unimodal actors.
5.2
Generative RL
There has been some application of RL in generative tasks, e.g., game content generation 25 50 and program generation [30. We take game level generation benchmark in (50) as a case study because policy multimodality is important to balance quality and diversity in this domain, according (50)
In this game level generation benchmark, the agent needs to observe a sliding window on an existing game level and propose new level segments to be concatenated with the existing level. The level segment is represented by a decoder, which is the generator of pretrained generative adversarial networks 2 477. That means the agent will output continuous latent vector at each step, and the decoder will decode the action into a piece of level. There are two expert knowledge- based formulations of the reward function in this benchmark, which aim evaluate the quality of generated levels under some game design principles. The reward also includes a penalty for vnsolvable level segments to guide agents to generate solvable levels. Solvable not is checked by rules in our experiments. Different reward functions result in different styles of generated levels, namely MarioPuzzle and MultiFacet. Fig. shows example levels of the styles generated by DrAmort. Besides episodic return, the diversity of generated levels is also desired in this domain.
Figure 6: Example MarioPuzzle style (left) and MultiFacet style (right) levels generated by DrAmort.
We train algorithms with the same temperature settings as multi-goal PointMaze. The learning curves are shown in Fig. 7a For both styles, DrAmort shows the best episodic returns, namely the best qualities. This is possibly attributed to the superior expressivity of the amortized actor. As the distribution of high-quality content is most likely multimodal, only highly expressive actors can represent high-performance policies under high temperatures. Though SQL also applies amortized actor, it performs badly. As SQL directly uses SVGD as the gradient estimator for approximating SPI, is possible that reparameterization trick a better gradient estimator than SVGD for RL. S AC, which applies SVGD to directly sample actions, performs the worst on all tasks. On the other hand, DrDiffus and DACER do not outperform SAC.
We also evaluate the diversity of generated levels by computing the average pairwise Hamming distance following [50], and then visualize the performance in the 2D quality-diversity objective space of all trained policies in Fig. ?bb Each marker in the figure represents a policy trained by one of the algorithms with an independent seed. Under MarioPuzzle style, DrAmort generally dominates other algorithms, i.e., outperforms other algorithms in terms of both return and diversity. Under the MultiFacet style, DrAmort dominates other algorithms except for SQL. Though SQL shows the best diversity, its episodic return is poor. Under the other style, SQL does not demonstrate better diversity, and the episodic return is still poor. DrDiffus, DACER, and SAC are non-dominated with each other. But diffusion actor is much slower than Gaussian and amortized actors. Therefore, diffusion
8

---

Mar oPuzzle
Nultil
MarioPuzzle
MultiFacet
1800
1050
Eatoodic Roturn
20
750
30-
800
-20
DiAmort
50L
800
<.BAnart
-54C
450
x50c
5/AC
x.50
50L
300
300
16
Baaning xtepss
Tharting aleps
Esisodic Return
(a) Learning curves regarding average episodic return, (b) Locations of all trained policies in the episodic return-diversity performance space.
Le.. average quality of generated levels.
Figure 7: Evaluation results in the game content generation benchmark.
actors might not be a good choice for this task. We present 1-SNE embedding visualizations [46 of generated levels in Appendix [.5 to demonstrate their diversity more intuitively.
5.3 Conventional Benchmarks
To evaluate the general performance of DrAC in conventional tasks, test DrAC in MuJoCo-v4 and compare it with baselines. We do not include S²AC as it is too computationally intensive and uses an unusual implicit actor, making it less meaningful to compare standard performance. Learning curves in the six MuJoCo locomotion tasks are presented in Fig.
DrAmort exhibits competitive perfor- mance. achieves the best perfor- mance in three out of the six tasks and surpasses SQL in a total of five, demonstrating that DrAC is strong method for training amortized ac- tors. Overall, the performance of DrDiffus comparable to DACER, though DACER leverages more ad- vanced critic learning technique [8] Therefore, DrAC can serve high- performance base algorithm for learn- ing multimodal policies, particularly for amortized actors. Mcanwhile, it is surprising that DACER and DrDiffus do not outperform SAC, though they apply more complex diffusion actors. We notice that the paper of DACER [48 uses deeper networks, more advanced activations, and smaller learning rates than the default settings of [14]. It is possible that diffusion actors need more powerful networks and more elaborate hyperparameter tuning to fulfill their potentials.
Ard
HallChwatahh
Hooper
SA
N
0.0
Traking
160
Humonold
Switmmrr
Walkerzd
Po Si. Pag
Eg etr
2/
0.00 0.25
0.50
0.75
1.00
6.25
0.75
1.00
0.50
0
2.00
Talning Stepe
1mc
Figure 8: Learning curves in MuJoCo.
6 Conclusion
We propose reigaasiy-eegullrizeddd Actor Critic (DrAC), novel and versatile RL algorithm designed to train intractable multimodal policies. DrAC addresses the intractability challenge in training multimodal policies by leveraging the reparameterization trick and distance-based diversity reg- ularization. Meanwhile, DrAC is compatible with all policy models that can be formulated as stochastic-mapping actor defined as Ty = {fo. P=} such that (-(-))) a + fo 2), P=- Experiments show DrAC can train amortized actors to achieve superior performance in multi- goal achieving and generative RL benchmark, which also verifies the significance of multimodal policies in these domains. Notably, using an amortized actor, DrAC demonstrates strong few-shot robustness 26 in out-of-distribution test scenarios within the multi-goal PointMaze environment. Nevertheless, this paper only investigates a basic implementation of DrAC, leaving ample room for further improve- ments. Future work could investigate scheduling the temperature, designing better stochastic-mapping actors, exploiting other distance and diversity metrics, and integrating more learning tricks, etc.

---

Our empirical results also indicate that the amortized actor possesses strong multimodal expressivity and performs well in all domains involved in this paper. On the other hand, the diffusion actor does not show significant advantages. It is possible that diffusion actors need more careful tuning of hyperparameters and better NN architecture design. However, the training and inference speed of amortized actors is comparable to traditional actors and significantly faster than diffusion actors. Therefore, we posit that the amortized actor represents a promising actor class for multimodal RL.
Acknowledgments and Disclosure of Funding
This work is supported by the National Natural Science Foundation of China 62406266.
References
[1] Yuanjiang Cao, Quan Z. Sheng, Julian McAuley, and Lina Yao. Reinforcement learning for generative Al: A survey, arXiv preprint arXiv:2308. /4328, 2023.
[2] Edoardo Cetin and Oya Celiktutan. Policy gradient with serial markov chain reasoning. Advances in Neural Information Processing Systems, 35:8824-8839, 2022.
[3] Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu. Score regularized policy optimization through diffusion behavior. In International Conference on Learning Representations, 2024.
[4] Huayu Chen, Kaiwen Zheng. Hang Su, and Jun Zhu. Aligning diffusion behaviors with Q-functions for efficient continuous control. In A. Globerson, L. Mackey, Belgrave, Fan, U. Paquet, Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 119949-119975. Curran Associates, Inc., 2024.
[5] Yuhui Chen, Haoran Li, and Dongbin Zhao. Boosting continuous control with consistency policy. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pages 335-344, 2024.
[6] Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. Improving exploration evolution strategies for deep reinforcement learning via a population novelty- seeking agents. Advances neural information processing systems, 31, 2018.
[7] Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement Icarning. In The Tweifth International Conference on Learning Representations, 2024.
[8] Jingliang Duan, Wenxuan Wang. Liming Xiao, Jiaxin Gao, Shengbo Eben Li. Chang Liu, Ya-Qin Zhang Bo Cheng. and Keqiang Li. Distributional soft actor-critic with three refinements. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[9] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. International Conference on Learning Representations, 2019.
[10] Justin Fu, Aviral Kumar, Ofr Nachum, George Tucker, and Sergey Levine. D4RL:: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[11] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error actor-critic methods. In International Conference on Machine Learning, pages 1587-1596. PMLR, 2018.
[12] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems, 27, 2014.
[13] Tuomas Haarnoja. Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep easegy-based policies. International Conference on Machine Learning, pages 1352-1361. PMLR, 2017.
[14] Tuomas Haamoja, Aurick Zhou, Pieter Abbecl, and Sergey Levine. Soft actor-critie: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning, pages 1861-1870. PMLR, 2018
[15] Tuomas Haarnoja, Aurick Zhou. Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta. Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05900, 2018
10

---

[16] Seungyul Han and Youngchul Sung. Diversity actor-critic: Sample-aware entropy regularization for sample-efficient exploration. In International Conference on Machine Learning, pages 4018-4029. PMLR, 2021.
[17] Shashank Hegde, Sumeet Batra, KR Zentner, and Gaurav Sukhatme. Generating behaviorally diverse policies with latent diffusion models. Advances in Neural Information Processing Systems, 36:7541-7554, 2023.
[18] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, Advances in Neural Information Processing Systems, 33:6840-6851, 2020. 2020.
[19] Zhang- Wei Hong. Tzu- Yun Shann, Shih- Shin-Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun- Yi Lee. Diversity-driven exploration strategy for deep reinforcement learning, Advances in Neural Information Processing Systems, 31, 2018.
[20] Zhiao Huang, Litian Liang. Zhan Ling, Xuanlin Li, Chuang Gan, and Hao Su. Reparameterized policy learning for multimodal trajectory optimization. In International Conference on Machine Learning, pages 11957-13973 PMLR, 2023.
[21] Haque Ishfaq, Guangyuan Wang, Sami Nur Islam, and Doina Precup. Langevin soft actor-critic: Effi- cient exploration through uncertainty-driven critic learning. In International Conference on Learning Representations, 2025.
[22] Vineet Jain, Tara Akhound-Sadeen Sadegh, and Siamak Ravanbakhsh. Sampling from energy-based policies using diffusion. In Reinforcement Learning Conference, 2025.
[23] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figumov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidck, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583 589, 2021.
[24] Bingyi Kang, Xiao Ma. Chao Du. Tianyu Pang. and Shuicheng Yan. Efficient diffission policies for offline reinforcement learning. Advances in Neural Information Processing Systems,
2023.
[25] Ahmed Khalifa, Philip Bontrager, Sam Earle, and Julian Togelius. PCGRL: Procedural content generation reinforcement learning. Proceedings of the AAAI Conference on Artificial Inteliience and Interactive Digital Entertainment, volume 16, pages 95-101, 2020.
[26] Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured MaxEnt RL. Advances in Neural Information Processing Systems, 33:8198-824 2020.
[27] Qingfeng Lan, Samuele Tosatto, Homayoon Farrahi, and Rupam Mahmood. Model-free policy learning with reward gradients. In International Conference on Artificial Intelligence and Statistics, pages 4217- 4234. PMLR, 2022.
[28] Steven Li, Rickmer Krohn, Tao Chen, Anurag Ajay, Pulkit Agrawal, and Georgia Chalvatzaki. Learning multimodal behaviors from scratch with diffusion policy gradient. Advances in Neural Information Processing Systems, 37:38456 38479, 2024.
[29] Timothy Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver. and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
[30] Guan-Ting Liu, En-Pei Hu, Pu-Jen Cheng, Hung-Yi Yi Lee, and Shao-Hua Sun. Hierarchical programmatic reinforcement learning via learning to compose programs. In International Conference on Machine Learning, pages
PMLR, 2023.
[31] Xiangyu Liu, Hangtian Jia, Ying Wen, Yujing Hu, Yingfeng Chen, Changjic Fan, Zhipeng Hu, and Yaodong Yang. Towards unifying behavioral and response diversity for open-ended learning in zero-sum games. Advances in Neural Information Processing Systems, 34:941-952, 2021.
[32] Bogdan Mazoure, Thang Doan, Audrey Durand, Joelle Pineau, and R Devon Hjelm. Leveraging exploration in off-policy algorithms via normalizing flows. In Conference on Robot Learning, pages 430-444, PMLR, 2020.
[33] Safa Messaoud, Billel Mokeddem, Zhenghai Xue, Linsey Pang. Bo An, Haipeng Chen, and Sanjay Chawla. S AC: Energy- based reinforcement learning with Stein soft actor critic. In International Conference on Learning Representations, 2024.

---

[34] Jack Parker-Holder. Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective diversity in population based reinforcement learning. Advances in Neural Information Procesxing Systems, 33: 18050-18062, 2020.
[35] Mariya Popova, Olexandr Isayev, and Alexander Tropsha. Deep reinforcement learning for de drug design. Science advances, 4(7)xeaap7885, 2018.
[36] Michael Psenka, Alejandro Escontrela, Picter Abbeel, and Yi Ma. Learning diftussin model policy from rewards via Q-score matching. In International Conference on Machine Learning, pages 41163 41182. PMLR, 2024.
[37] Allen Z Ren, Justin Lidard, Lars Lien Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. In International Conference on Learning Representations, 2024.
[38] Ren, Yewen Li, Zihan Ding, Wei Pan, and Hao Dong. Probabilistic mixture-of-experts for efficient deep reinforcement learning. arXiv preprint 1oXiv:2/04.09122 2021.
[39] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17Xiv:1777006477 2017.
[40] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deter- ministic policy gradient algorithms. In International Conference on Machine Learning, pages 387-395. PMLR, 2014.
[41] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[42] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In S. Solla, T. Leen, and K. Müller, editors, Advances in Neural Information Processing Systems, volume 12, pages
MIT Press, 1999.
[43] Chen Tang, Ben Abbatematieo. Jiaheng Hu, Rohan Chandra. Roberto Martin, and Peter Stone. Deep reinforcement learning for robotics: A survey of real-world successes. Annual Review of Control, Robotics, and Autonomous Systems, 8(Volume 8, 2025):153- 2025. ISSN 2573-5144.
[44] Emanuel Todorov. General duality between optimal control and estimation. In 2008 47th IEEE conference on decision and control, pages 4286-4292. IEEE, 2008.
[45] Mare Toussaint. Robot trajectory optimization using approximate inferenee. In Proceedings of the 26th annual international conference on machine learning, pages 1049-100 1056, 2009.
[46] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(11), 2008
[47] Vanessa Volz, Jacob Schrum, Jialin Liu, Simon M Lucas, Adam Smith, and Sebastian Risi. Evolving Mario levels in the latent space of deep convolutional generative adversarial network. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 221-228, 2018
[48] Yinuo Wang. Likun Wang, Yuxuan Jiang, Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang, Liming Xiao, liang Wu, Jingliang Duan, et al. Diffusion actor-critic with entropy regulator. Advances Neural Information Processing Systems, 37:54183-5420 2024.
[49] Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou Diffusion policies as an expressive policy class for offline reinforcement learning. In International Conference on Learning Representations, 2023.
[50] Ziqi Wang, Chengpeng Hu, Jialin Liu, and Xin Yao. Negatively correlated ensemble reinforcement learning for online diverse game level generation. In International Conference on Learning Representations, 2024.
[51] Shuang Wu, Jian Yao, Haobo Fu, Ye Tian, Chao Qian. Yaodong Yang. Qiang Fu, and Yang Wei. Quality- similar diversity via population based reinforcement learning. In International Conference on Learning Representations, 2023.
[52] Ke Xue, Ren-Jian Wang. Pengyi Li, Dong Li, Jianye Hao, and Chao Qian. Sample-efficient quality-diversie by cooperative coevolution. In International Conference Learning Representations, 2024.
[53] Long Yang. Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang. Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning. arXiv preprint 233. 2023.
12

---

[54] Zhengyu Yang, Kan Ren. Xufang Luo, Minghuan Liu, Weiging Liu, Jiang Bian, Weinan Zhang, and Dongsheng Li. Towards applicable reinforcement learning: Improving the generalization and sample efficiency policy ensemble. International Joint Conference on Artificial Intelligence, 2022.
[55] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Im- proved data-augmented reinforcement learning. International Conference on Learning Representations. 2022.
[56] Chengyang Ying. Huayu Chen, Xinning Zhou, Zhongkai Hao, Hang Su, and Zhu. Exploratory diffusion policy for unsupervised reinforcement learning. arXiv preprint arXiv:2502.0722 2025.
[57] Yunbo Zhang, Wenhao Yu, Greg Turk. Learning novel policies tasks. International Conference on Machine Learning, pages 7483-7492. PMLR, 2019.
[58] Zhengbang Zhu, Hanye Zhao, Haoran He, Yichao Zhong, Shenyu Zhang, Haoquan Guo, Tingting Chen, and Weinan Zhang. Diffusion models for reinforcement learning: A survey, arXiv preprint arXiv:231.002233 2023.
[59] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1444 1438, 2008
[60] Eckart Zitzler and Lothar Thiele. Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach. IEEE transactions on Evolutionary Computation, 3(4):257-271, 2002.
13

---

A Technical Details
A.1 Pseudo code of DrAC
d DrAC.
Algorithm 1 Diversity-regulaaizz Actor-Critic (DrAC)
Require:
Actor weights 0; Critic weights & {61, Φ2); Target critic weights a = f0i, 22) Let gradient_step__e for : > T do
Sample an action a <(-|s)
Execute and observe s' d
r, r,s', Execute a in the environment
Store as, d) in the buffer
while gradient_sten stepsstepooo cerp_cuuttto < + d1*reply_raiii do
Sample batch (s, a, rrr,,, s',
from the replay memory
Sample 2; (2f ".,,.. ... (2n, = from P=
Let Ds('') - Ei log fa((s
Compute Q-target y +- r r
(Q -1-
2); 0) aD)(s))
Update Φ1 and de with MSE(Q(s, (x). [1,2}
Sample 2; (2f "...... (2n , = from p:
Let (s) 4 Ei= logó( (o(,,
Update 0 with La = Q(, (s) -
Update with La a(Do(s) - D)
Update à by δ pō (1 p)
Let graddiet_steep_oom count & seadient_step_comee +
end while end for
A.2 Experiment Settings
The customized maze maps used our multi-goal PointMaze environment are shown Fig.
Simple Maze
Medium Maze
Hard Maze
Figure 9: Maze maps the Multi-goal PointMaze environment. The red circle indicates the start point, green cells indicate goals, gray cells are walls, and light gray cells are obstacles that only appear in the robustness evaluation stage.
The hyperparameters are listed in Table The temperature hyperparameters listed in Table are used in the multi- goal PointMaze environment and the game content generation environment. While for MuloCo, lower temperature for DrAC. For baseline algorithms, the default temperatures MuloCo provided in their original papers. Temperature hyperparameters all algorithms MuJoCo are listed Table 3
14

---

Table Hyperparameters
Hyperparameter
DrAmort
DrDiffus
DACER
SQL
S²AC
SAC
Replay buffer size
1000000
MLP hidden layers in actors critics
(256, 256)
Hidden layer activation all networks Optimizer
ReL.U
Adam
Learning rate of actors & crities Discount rate (r)
3e -4
0.99
Target smoothing coefficient (p)
0.005
Replay ratio (gradient step)
Mini-batch size
256
Diversity temperature (3)
0.8
N/A
Target entropy (Pl)
N/A
0.5|
N/A
0.5|
Learning rate for auto-adjustment
5e
3c
N/A
3e
Entropy coefficient (a)
N/A
0.3
0.7
N/A
Number of divensity estimation pairs (n)
8
N/A
Number of SVGD particles (k)
N/A
10
32
N/A
Policy delay (actor update period)
N/A, equivalent to 1
N/A, equivalent to 1
Table 3: Temperature settings for MuJoCo
DrAmort
DrDiffus
DACER
SQL
S²AC
SAC
0.5
0.2
-0.9
0.2 for Ant, the rest
|A]
Regarding compute resources, all of our experiments conducted with Linux server with 8 NVIDIA RTX 3090 GPUs and Intel Xeon Platinum 8375C CPU. All trainings done with GPU.
A.3 Relation to Amortized Inference and Traditional Gaussian Actor
The formulation of stochastic mapping actors is closed to the definition of amortized inference. Due to that the term "amortized actor" has been used in [13 and adopted in [33] mapping actors".
formulation "stochastic
The formulation of stochastic mapping actors is also compatible with traditional Gaussian actor, by treating f(, 2) = no (s) + 00z ~ N NO,,7 T). Our method is also directly applicable to Gaussian actor.
15

---

B
Additional Results
B.1
Computational costs
We record the average time per 1000 training steps in Ant-v4 environment. The test conducted Linux server with NVIDIA RTX GPUs and an Intel Xeon Platinum 8375C CPU. For each algorithm, we run three trials on GPU cards 0, and 2, respectively, and then average the training time per 1000 steps. All algorithms are implemented with Py Torch 2.5.1, and the CUDA version is 12.5.
Table 4: Computational costs of all compared algorithms
SAC
SQL
DrAmort
DACER
DrDiffus
S2AC
Training time per 1000 steps
22.7s
18.9s
21.0s
76.9s
84.8s
220.5s
Peak GPU memory usage
352M
472M
392M
364M
608M
3462M
According the table, we found that DrAmort is even faster than SAC. The reason is that SAC computes and paok-propagates the log-prohability at certain actions, which requires more sequential tensor operations. Though DrAmort requires samples, the computation parallelized and takes fewer sequential tensor operations. As result, DrAmort uses fewer GPU clock ticks and slightly faster than SAC, given sufficiendly small compared to the GPU's parallel computation capability. For DrDiffus, it takes about 10% more time for training compared to DACER, but its diversity in multi-goal PointMaze is better.
neural network
For large-scale tasks that require larger networks. such visual control, can inject the latent variable at the last few layers. For example, if CNN is used, the latent variables can be injected after the convolutional layers. this case, the computation time and memory usage will not significantly increase.
B.2 Trajectories in Medium and Hard Multi-goal PointMaze
Figs.
10 and
demonstrate the trajectories in medium and hard mazes, respectively.
5/c Ortegy lacedl
Figure 10: Evaluation trajectories of all tested algorithms the medium maze.
Figure 11: Evaluation trajectories of all tested algorithms in the hard maze.
Consistently, DrAmort exhibits the best diversity.
16

---

B.3 Sensitivity of Temperature Hyperparameter in Multi-goal PointMaze
We display the learinng curves all evaluated algorithms with varied temperatures. For DrAmort and DrDiffus, we show sensitivity study with five different temperatures. For baseline algorithms, we present three temperatures to explain our choice of their temperatures in our comparison study. Specifically, for all baseline algorithms, using higher temperature than our choice leads to suboptimal success rate in at least one maze; while using lower one leads lower diversity.
Learning curves of DrAmort with varied temperatures Simple
Medium
Hard
Total reward
0.8
Total reward
0.8
Total reward
0.8
0.6
#-0.6
8-0.8
0.6
P+0.6
8-0.8
0.6
8-0.6
8-03
3-0.7
8-0.9
0-0.7
8-0.9
B-0.7
8-0.9
0.4
0.2
0.2
0.2
0.0
0.5
1.0
2
3
Training steps
le5
Training steps
le5
Training steps
le5
Simple
Medium
Hard
8
Reachable modes
Reachable modes
Reachable modes
6-
8-0.8
8-0.6
$-0.8
2-
2
8-0
8-0.9
4
0-0.7
8-0.9
2
8-08,
8-0.7
8-0.9
0
0.0
1.0
Training steps
1e5
Training steps
le5
Training steps
1e5
Figure 12: Learning curves of DrAmort in multi-goal PointMaze with varied temperatures.
Learning curves of DrDiffus with varied temperatures
Simple
Medium
Hard
1.0
1.0
1.0
Total reward
Total T reward
Total Totalrewan reward
0.6
0.6
0.6
0.4
0.4
0.4
4-066
8-0.8
f-0.6
840.8
8-0.
A-88
B-0.7
$+0.9
8-0.7
$40.9
B-0.7
8-0.9
0.0
0.0
0.0
0.0
1.0
Training steps
1e5
Training steps
1e5
Training steps
le5
Simple
Medium
Hard
Reachable modes
Reachable modes
Reachable modes
6=0.6
0=0
8.0.
8=0.9
8-0.6
8+0.8
B-0.6
8-0.8
5-0.7
8-0.9
B-0.7
6+0.9
0.0
0.5
1.0
2
3
Training steps
le5
Training steps
le5
Training steps
le5
Figure 13: Learning curves of DrDiffus in multi molti-goal PointMaze with varied temperatures.
17

---

Learning curves of SQL with varied temperatures Simple
Medium
Hard
1.0
reward
0.8
Total reward
0.8
Tota reward
0.8
0.4
0.4
0.4
A-
0.2
0-
a=0.4
0.2
0-02
a=0.4
0.2
0=0.
a=0.8
2-0.3
2-0.3
0-0.3
0.0
0.5
1.0
3
Training steps
1e5
Training steps
1e5
Training steps
1e5
Simple
Medium
Hard
Reachable modes
0.3
0-0.2
2-0.4
Reachable modes
Reachable modes
6
d-0.2
000.3
a-0.4
2
4-
2
2-0.3
0
0.0
0.5
1.0
Training steps
1e5
Training steps
1e5
Training steps
1e5
Figure 14: Learning curves of SQL in multi-goal PointMaze with varied temperatures.
Learning curves of DACER with varied temperatures
Simple
Medium
Hard
1.0
1.0
1.0
Total Total reward
Total T reward
Total reward
0.6
0.6
0.6
0.4
0.4
Total
0.4
R-0.8
1
Km0.8
4-1-2
4-08
1-1
1-1.0
1=100
1=10
0.0
0.0
0.0
0.0
1.0
Training steps
1e5
Training steps
1e5
Training steps
le5
Simple
Medium
Hard
Reachable modes
N-0.8
1.10
1-
Reachable modes
8-0.8
1=100
1-1.2
Reachable modes
2=0.8
1=10
1-122
0.0
0.5
1.0
2
3
Training steps
les
Training steps
le5
Training steps
le5
Figure 15: Learning curves of DACER in multi-goal PointMaze with varied temperatures.
18

---

Learning curves of SAC with varied temperatures Simple
Medium
Hard
reward
0.8
reward
0.8
Tota reward
0.8
N=100
0.4
0.4
0.4
0.2
0.2
N=0.8
H=1.2
0.2
K=0.8
Nw1.2
R=10
1=10
0.0
0.5
1.0
3
Training steps
1e5
Training steps
1e5
Training steps
1e5
Simple
Medium
Hard
Reachable modes
r=0.8
M=12
Reachable modes
0-0.8
i=12
Reachable modes
1=12
R-1.0
R-1.0
6
R=1.0
2
4-
2
0
0.0
0.5
1.0
Training steps
1e5
Training steps
1e5
Training steps
1e5
Figure 16: Learning curves of SAC in multi-goal PointMaze with varied temperatures.
Learning curves A2AC with varied temperatures Simple
Medium
Hard
1.0
1.0
1.0
Total reward
Total reward
0.8
Tota reward
0.8
0.6
0.6
0.6
0.4
0.2
0
d=0.8
0.2
2=0:6
a=0.
0.2
Q=0:6
0=0::
0-0.7
0-0.7
0-0.7
0.0
0.5
1.0
2
3
Training steps
1e5
Training steps
1e5
Training steps
1e5
Simple
Medium
Hard
Reachable modes
0=0
2-0.7
0
Reachable modes
0=0
g=0.7
a=0.8
Reachable modes
6
5-0.6
0=7.7
0=0.8
2-
0.0
0.5
1.0
2
3
Training steps
le5
Training steps
le5
Training steps
le5
Figure 17: Learning curves of S2 AC in multi-goal PointMaze with varied temperatures.
19

---

B.4 Sensitivity of Temperature in Game Content Generation
We train each algorithm with three groups of temperatures, namely high, mid, low, investigate the sensitivity of temperature. The high temperature group uses the same temperature with the multi-goal PointMaze. The low temperature group are to IIA for SAC and DACER and near zero DrAC, SQL and SCAC. The mid temperature group use temperatures averaged between low and high groups. Fig. [8 shows the algorithm performance across temperatures.
Algorithm Performance Comparison Across Temperatures
MarioPuzzle- Quality
MarioPuzzie Diversity
1930
Qualky
30
Diverstyy
#acs
avea
exent
end
ax
Awdd
# Boks
Baga
10
Quality
Diversiy
10
1030
Quality
Diversitty
12
16
22p
2-12
paga
8-05
6x2
2081
2-07
P.05
4-8-
Baga
Heg0
ax2
a-02
R-
de
He
DrAmort
50L
PAC
Figure 18: Sensitivity of temperature across tasks and metrics. The ticks of r-axis details the temperature.
Generally, the quality decreases along with raising temperature while diversity increases along with raising temperature. To provide overall evaluation. treat all policies trained by each algorithm across temperatures and seeds a population, and compute the hypervolume (HV) metric for these populations (60) The reference point is set according to the lowest quality and diversity performance across all algorithms, temperatures and seeds.
Hypervolume (HV) Comparison Algorithms Tasks
MarioPuzzle
Multifacet
60000
48420.6
40000
36524
8000
78554.6
Voluc
6594.11
30000
6000
5594.2
5572.3
5240.9
16437.1
11810.6
16000
7000
Didnont
brbifoo
DACER
94C
Dutmort
Denifor
DMCER
ypc
5d
gtac
Figure 19: HV comparison policy populations obtained by training each algorithm through three different temperatures.
The results show DrAmort produces the best population in terms of the HV metric, DrDiffuus outperform baselines in MultiFacet and shows competitive performance in MarioPuzzle.
20

---

B.5 (-SNE visualization of Generated Game Levels
To compare the diversity of levels generated by each algorithm more intuitively, embed levels generated by each algorithm with t-SNE (6). and plot the emheddings in Fig. and Fig. 21
MarioPuzzle Seed 1
Seed 0
Seed 2
Seed 3
Seed 4
Legends
DrAmort
SQL
DrDiffus
DACER
SAC
Figure 20: t-SNE embeddings levels generated by each algorithm, under MarioPuzzle style.
MultiFacet Seed 1
Seed 0
Seed 2
Seed 3
Seed 4
Legends
DrAmort
SQL
DrDiffus
DACER
SAC
Figure 21: (-SNE embeddings of levels generated by each algorithm, under MultiFacet style.
Observing the embeddings, differences in terms of diversity are not significant. Different algorithms generally form different clusters. This may indicate that there is still ample room to improve the diversity with little harm quality. Future work may investigate how unleash the expressivity of multimodal policies more completely. Besides, average pairwise Hamming distance vanilla method measure diversity of game content, more in-depth quantitative analysis may be considered in future work.
21

---

B.6 Sensitivity of Temperature Hyperparameter in MuJoCo
We demonstrate the sensitivity of temperature hyperparameter of DrAmort and DrDiffus as follows.
DrAmort HalfCheetah-v4
Ant-v4
Hopper-v4
10000
7500
3000
5000
2000
2000
2500
1000
8=0.4
8=0.4
0=0.3
A
B-0.3
$=0.5
8=0.3
8=0.5
0.5
0.0
0.5
0.0
Training steps
1e6
Training steps
1e6
Training steps
1e6
Humanoid-v4
Swimmer-v4
Walker2d-v4
6000
4000
25
2000
2000
R=02
$-0.4
F-0.2
A-0.4
P=02
B=0.4
8-0.3
8-0.5
8-0.3
8-0.5
0-
8-0.5
-2
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Training steps
1e6
Training steps
1e6
1e6
Figure 22: Learning curves of DrAmort with varied temperatures in MuJoCo
DrDiffus HalfCheetah-v4
Ant-v4
Hopper-v4
6000
3000
3000
4000
2000
2000
2000
1000
1000
$=02
$=0.4
8=02
$=0.4
B=0.2
0=0.4
9=0.3
$-0.5
0-
9=0.3
0=0.5
0=0.3
8-0.5
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Training steps
1e6
Training steps
1e6
Training steps
1e6
Humanoid-v4
Swimmer-v4
Walker2d-v4
6000
4000
100
4000
3000
2000
50
2000
hawp
1000
B-0-2
8-0.4
5-02
8-0.4
8=02
8=0.4
0-0.3
8-0.5
8-0.3
8-0.5
8=0,3
8=0.5
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Training steps
le6
Training steps
1e6
1e6
Figure 23: Learning curves of DrDiffuus with varied temperatures in MuJoCo
DrAmort robust to the temperature, overall 3 0.5 deliveries superior performance. DrDiffus suffers performance drop in Humanoid with 3 0.2. More investigation into this phenomenon is an important future work.
22

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately describe the proposed algorithm (policy gradient
via reparameleriizaiin distance-based diversity regularization) and experimental scope (multi-god) achieving, generative RL. MuloCo). The abstract and the introduction align with the technical content in the paper (Section Appendix A.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals motivation long it is clear that these goals not attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The conclusion (Section involves a discussion of limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings. model well- sell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice
what implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. general, empirical results often depend on implicit assumptions, which should he articulated.
The authors should reflect on the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with datasct size.
applicable, the authors should discuss possible limitations of their approach to address problems privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
23

---

All the theorems, formulas, and proofs in the paper should be numbered and cross tross-reference..
All assumptions should he clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Benchmarks and tested algorithms are described Section S Experiment settings, including evaluation criteria, are described within Section Hyperparameters are detailed in Appendix A Appendix A.I provides detailed pseudo-code of the proposed algorithm. Our code will be released after acceptance.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments. No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can accomplished various ways. example, contribution novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results. access to a hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g.. with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source source models, it may be that access to the model is limited in some way (e.g.. registered users). but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data code, with sufficient instructions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [Yes]
Justification: We will submit our code supplemental matorials, and we will release our code with instructions after acceptance. Our code contains training and evaluation. Customized RL environments are also included with the necessary assets. We do not use training data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines http:::///iiss.ccpupppee muiider//ooeeSubmiisssooooooood for more details.
24

---

While we encourage the release of code and data, we understand that this might not be possible. "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g.. for a new open-soure benchmark).
The instructions should contain the exact command and environment needed the results. See the NeurlPS code and data submission guidelines (https: //tp:////ppspaaepe ccppbiiic/
to reproduce
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release amonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental sefting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Experiment settings and evaluation criteria are introduced within the Section Hy- perparameters, including network architecture and the type of optimizer, are detailed in Appendix
A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: We repeat all training with five seeds, and all results are reported with mean and standard deviation across the five seeds. All algorithms used the set of seeds. All evaluations for all checkpoints are repeated least 20 times to average.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance lests, at least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
25

---

Answer: [Yes]
Justification:
We describes our compute resources Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code Ethics http:p///eaarrps...c///aaalllaaa
Answer: [Yes]
Justification: The rescarch involves algorithmic development and ovaluation on simulation benchmarks and source game benchmarks. does not involve human subjects obviously ethically sensitive applications, and we expect that it conforms to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from Code of Ethics.
The authors should make sure preserve anonymity (e.g-, if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
This research focuses on basic deep RL algorithm development, and all experiments are conducted in simulated imaginary environments or digital games without any sensitive subjects. We expect there is no socictal impact.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, is legitimate point out that an improvement the quality generative models could used to generate deepfakes for disinformation. On other hand, it is not needed to point out that a generic algorithm for optimizing neural networks
could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms monitor- ing misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
26

---

Answer: [NA]
Justification: This work focuses on algorithm development experiments are conducted with simulated environments and open-source games, so we expect the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example hy requiring that users adhere to usage guidelines restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are creators or original owners assets (e.g., code, data, models), used the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our experiments dependent on open-source assets in D4RL 10 and game content generation [50]. which are credited by citations in Section
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should he provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for dattasets. Their licensing guide help determine the license dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it changed) should be provided.
this information not available online, the authors encouraged reach out the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This work does not involve new assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/mooel as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). can either create an amonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve any crowdsoureing experiment nor research with human subjects.
Guidelines:
27

---

The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should he included in the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, other labor should be paid at least. minimum wage in country data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This work does not involve any crowdsourcing experiment nor research with human subjects.
Guidelines:
The answer NA that the paper does not involve crowdsourcing research with human subjects.
Depending on the country which research is conducted. IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
16. Declaration LLM usage
Does the paper describe the usage of LLMs if it important, original. non-standard component
Question:
core methods this research? Note that if the LLM used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the rescarch, declaration is not required.
Answer: [NA]
Justification: This work does not involve LLMss as any important, original, or non-standard compo- nents.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs any important, original, non-standard components.
Please refer LLM policy hhttps://neurr: htt.a///.......eeeee/////////// http:///pp///////o../../o/////// should or should be described.
for what
28


Reviews:
[{"review_id": "782immDzJl", "reviewer": "Reviewer_T7pd", "summary": "This paper proposes a class of diversity-regularized actor-critic (DrAC) algorithms for learning multimodal policies in RL with continuous actions. DrAC assumes a reparameterizable policy class (including amortized and diffusion actors/policies) and uses the reparameterization trick for calculating the policy gradient. The diversity of actions given a state is promoted by a geometric-based distance loss for a pair of generated actions. The policy optimization loss and this distance loss are balanced by a coefficient that is tuned in a similar way to SAC’s temperature. Two instances of DrAC algorithms based on amortized and diffusion actors, respectively, are empirically evaluated against some common baselines. In multi-goal environments and generative tasks, the amortized variant is shown to be generally better, achieving both high return and more diverse behaviors, while the diffusion variant is shown to be comparable to a diffusion baseline. In a conventional benchmark (MuJoCo), both variants and other amortized/diffusion-based baselines do not outperform SAC and are sometimes worse.", "questions": "Questions that may impact the evaluation:\n1. For the multi-goal achieving experiment, could you elaborate on the process of tuning and selecting the temperature hyperparameter? What hyperparameters are tuned for each method?\n2. How is $\\beta$ tuned in the MuJoCo experiment?\n3. Are DrAmort and DrDiffus sensitive to $\\beta$ in different settings, respectively?\n\nQuestions or suggestions that do not impact the rating:\n1. What is used as the ground truth density for Figure 3?\n2. If I understand correctly, the actor in S$^2$AC should also be considered an amortized actor. However, I’m not sure if the current definition is broad enough to include this instance. Could you shed some light on this? If possible, extending the current definition to be more general would be helpful.\n3. In Line 132, the introduction of the entropy-regularized RL framework is attributed to Haarnoja et al. (2018). However, at least Toussaint (2009) introduced such a framework under the name of probabilistic inference. The discussion around Line 132 should be updated to assign credit properly.\n\nToussaint, M. (2009, June). Robot trajectory optimization using approximate inference. In Proceedings of the 26th annual international conference on machine learning (pp. 1049-1056).", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The strengths of the paper include the following:\n1. The paper provides some valuable insights about the utilities of more expressive policies in RL with continuous actions. On the one hand, unlike recent works that mostly focus on a simple 2D multi-goal toy problem [11, 28, 41], the paper demonstrates the utility of multimodal policies in two settings that are understudied in relevant work. On the other hand, it is surprising that, despite consistent claims of improvement for more expressive policies, they do not outperform the standard Gaussian policy in MuJoCo under the same neural network (NN) structure and hyperparameters. Both sets of results are interesting to many RL researchers who are curious about more expressive policies and call for further investigation on when they are useful.\n2. The paper proposes a novel loss to encourage diversity for multimodal policies and demonstrates its effectiveness. The automatic coefficient adjustment also reduces the need to tune the relevant hyperparameters.\n3. The paper is well-written, clear, and very easy to follow. \n\nThe paper also has some weaknesses:\n1. The current results are suggestive and not thorough.\n    - In the MuJoCo experiments, multimodal policies, being evaluated under the same neural network structure and hyperparameters as standard Gaussian policies, didn't outperform them. It's unclear whether this is due to insufficient hyperparameter tuning or if multimodal policies simply aren't necessary in these environments.\n    - The process of tuning the temperature hyperparameters in the multi-goal experiment is not elaborated, potentially hindering the validity and rigorousness of the results.\n2. There is no analysis on the sensitivity of the temperature hyperparameter $\\beta$. There appear to be two values used in different settings (0.8 and 0.15), and little is known about their sensitivity. Further analysis on this could improve the understanding and applicability of the work.\n3. One of the claimed contributions, specifically the derivation of policy gradient for reparameterizable policies, appears to be highly incremental to prior work (for example, Theorem 3 in Lan et al., 2022) and does not introduce substantial novelty beyond that. A more thorough review and discussion of existing work is necessary to accurately establish its place within the field. Considering other contributions, this is minor but still a weak point of the paper. \n\nLan, Q., Tosatto, S., Farrahi, H., & Mahmood, R. (2022). Model-free Policy Learning with Reward Gradients. In International Conference on Artificial Intelligence and Statistics (pp. 4217-4234). PMLR.", "was_revised": true, "final_justification": "My main concerns around hyperparameter tuning and sensitivity to temperature are resolved. I believe with the new results and analysis promised in the rebuttal, the paper has become stronger.\n\nI’ve also reviewed other reviews and the corresponding responses. While I agree that insufficient explanation of why amortized actors are better than diffusion actors is a weakness of the paper, I think the observation itself is already interesting. \n\nOverall, I believe that this paper provides interesting results and contributions to the community, and I support the acceptance of the paper.", "rebuttal": "We thank the reviewer for the insightful comments and suggestions. We run additional experiments to address reviewer’s questions and improve the thoroughness of our work. Due to the character limit, we only present most relevant and informative results using tables here. All presented results are averaged over 5 training seeds.\n\n**Regarding reviewer's questions:**\n\n**Q1**: We conducted a grid search for each algorithm before the formal evaluation. Based on the grid search, we picked the highest temperature that ensures each algorithm reaches an optimal success rate, to enable a fair and meaningful comparison. We have revised the paper to clarify this.\n\n**Q2**: As SAC is a standard baseline in MuJoCo, we compute the average log-distance (our diversity regularization metric) of the trained SAC agents with default target entropy $-|\\mathcal{A}|$ in each MuJoCo task. Then, as we set target diversity for DrAC as $\\hat D = \\log(\\beta \\sqrt{|\\mathcal{A}|})$, so we have $\\beta \\approx \\exp(D(\\pi))/\\sqrt{|\\mathcal{A}|}$, where $D(\\pi)$ is the expected pairwse log-distance.  So we computed the corresponding $\\beta$ value in each task as follows\n\n| Ant | HalfCheetah | Hopper | Humanoid | Swimmer | Walker2d |\n| --- | --- | --- | --- | --- | --- |\n| 0.1237 | 0.2133 | 0.1465 | 0.3555 | 0.1143 | 0.1784 |\n\nExcept for Humanoid, all the results are around 0.15, so we chose $\\beta=0.15$ to make the target diversity comparable with the default target entropy of SAC. We will also add an evaluation of the sensitivity to temperature in MuJoCo during the revision.\n\n**Q3**: We conduct additional experiments to investigate this question. Based finished experiments, we present some results below to respond to this question. We will add comprehensive results and analysis in our paper.\n\n**Multi-goal Maze:**\n\nWe run DrAC with different values of $\\beta$ in the our multi-goal maze environments, the final performances in the Hard maze are presented as follows. \n\n**DrAmort** \n\n| beta | Success Rate | Reachable Modes | Robustness-Removel | Robustness-Obstacle |\n| --- | --- | --- | --- | --- |\n| 0.6 | 1.00 | 1.0 | 0.50 | 0.04 |\n| 0.7 | 1.00 | 1.4 | 0.56 | 0.01 |\n| 0.8 | 1.00 | 4.6 | 0.91 | 0.62 |\n| 0.9 | 0.41 | 7.4 | 0.60 | 0.24 |\n| 1.0 | 0.36  | 7.4 | 0.55 | 0.23 |\n\n**DrDiffus**\n\n| $\\beta$ | Success Rate | Reachable Modes | Robustness-Removel | Robustness-Obstacle |\n| --- | --- | --- | --- | --- |\n| 0.6 | 1.00 | 1.6 | 0.58 | 0.02 |\n| 0.7 | 1.00 | 2.0 | 0.55 | 0.03 |\n| 0.8 | 1.00 | 2.4 | 0.65 | 0.21 |\n| 0.9 | 0.90 | 6.8 | 0.83 | 0.65 |\n| 1.0 | 0.45 | 7.2 | 0.57 | 0.25 |\n\nAccording to the results, success rate is not sensitive to $\\beta$ within a wide range  (about <0.8 for DrAmort and <0.9 for DrDiffus). diversity-related performance is sensitive to $\\beta$. Higher $\\beta$ values enable DrAC to reach more modes and enhance few-shot robustness. However, overly high temperatures reduce the success rate, which also harms robustness metrics. We will discuss the sensitivity of beta comprehensively in our revised version.\n\n**Game Content Generation:**\n\nWe have run DrAC with three temperatures. The results for MarioPuzzle are displayed as follows\n\n**DrAmort**\n\n| $\\beta$ | Content Quality | Content Diversity |\n| --- | --- | --- |\n| 0.2 | 30.9 | 631.3 |\n| 0.5 | 29.9 | 1137.3 |\n| 0.8 | 32.8 | 1384.9 |\n\n**DrDiffus**\n\n| $\\beta$ | Content Quality | Content Diversity |\n| --- | --- | --- |\n| 0.2 | 35.3 | 744.3 |\n| 0.5 | 23.1 | 1061.3 |\n| 0.8 | 21.1 | 1009.3 |\n\nRegarding DrAmort, the diversity of generated levels increase along with beta raises. The content quality appears not sensitive to $\\beta$. \n\nRegarding DrDiffus, when raising $\\beta$ from 0.2 to 0.5, quality decreases and  diversity increase. When raising $\\beta$ from  0.5 to 0.8, the performance does not change much. \n\nThere are two aspects may be related to this experiment phenomenon: (1) higher temperature also improves exploration which could be helpful to quality (2) policy diversity is correlated but not identical to the diversity of generated levels. More fine-grained ablation study will be conducted and discussed in our revision. Future work in generative RL for game content generation includes multi-objecive RL, where our method could serve as a promising base algorithm.\n\n**MuJoCo**\n\nWe are running additional experiments to analyze the sensitivity of DrAC in MuJoCo. Such experiments will be added in our revised version to improve thoroughness. We have finished partial experiments now. The performances of DrAmort with different values of $\\beta$, in the first two MuJoCo environments, are presented as follows\n\n| $\\beta$ | Ant | HalfCheetah |\n| --- | --- | --- |\n| 0.15 | 2513.8 | 5487.0 |\n| 0.25 | 3327.9 | 5640.9 |\n| 0.35 | 3440.2 | 6558.2 |\n| SAC | 3422.7 | 5515.7 |\n\nWe also include the performance of SAC for reference. According to the table, raising $\\beta$ value (as we used $\\beta=0.15$ in the paper) improves the performance in Ant and HalfCheetah. Notably, the improvement in Ant from $\\beta=0.15$ to $\\beta=0.35$ is >36%, and with $\\beta=0.35$, DrAmort outperforms SAC by >18% in HalfCheeta. More elaborate self-adaptation/annealing techniques could be researched in future work.\n\n**Q4**: The blue one ($X$) is a diagnoal Gaussian distribution $\\mathcal{N}(\\boldsymbol{0}, I)$, while the red one is a Gaussian mixture model with uniform weights and two components  $\\{\\mathcal{N}([6, 6], 0.2I),\\mathcal{N}([-6, -6], 0.2I)\\}$. We have modified the figure to describe the two ground truth distributions in our paper.\n\n**Q5**: The actor in SAC outputs the mean and per-dimension scale of a diagonal Gaussian distribution and applies $\\tanh$ to bound the output action. It is closed to the concept of amortized inference. In the paper of SQL [1] and S2AC [2], the term “amortized actor” is used to refer to GAN-like actors, which take an additional random variable as input and output the action. So we follows there naming, specifically refer to SQL-style actor as “amortized actor”, which may be narrower than the concept of amortized inference. For SAC’s actor, we refer to it as “Gaussian actor”. \n\nWe define “stochastic-mapping actors” as the framework to unify amortized actors and diffusion actors. This framework is also compatible with SAC-like actor, by treating $f_\\theta(s,z) = \\tanh(\\mu_\\theta(s) + \\sigma_\\theta(s)z), z \\sim \\mathcal{N}(\\boldsymbol{0}, I)$, where $\\mu_\\theta(s)$ and $\\sigma_\\theta(s)$ is the mean vector and scale vector outputs by the neural network parameterized with $\\theta$. That also means it will be straight-forward to apply our algorithm to train Gaussian actors. We did not test this as our main focus is training intractable multimodal actors. We will discussed this compatibility with Gaussian actor in our revision.\n\n[1] Haarnoja, Tuomas, et al. \"Reinforcement learning with deep energy-based policies.\" *International conference on machine learning*. PMLR, 2017.\n\n[2] Messaoud, Safa, et al. \"S$^2$AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic.\" *The Twelfth International Conference on Learning Representations*.\n\n**Q6**:  Thank for the careful review. We have updated Section 3.2 to acknowledge foundational works properly.\n\n**Regarding specific weaknesses:**\n\n>In the MuJoCo experiments ... It's unclear whether this is due to insufficient hyperparameter tuning or if multimodal policies simply aren't necessary in these environment\n\nWe are addressing this by additional experiments (See response to question 2  and 3).\n\n>The process of tuning the temperature hyperparameters in the multi-goal experiment is not elaborated, potentially hindering the validity and rigorousness of the results.\n\nWe address this in the response to question 1.\n\n>One of the claimed contributions, ... to accurately establish its place within the field.\n\nWe thank the reviewer for pointing out this related work, as we were not aware of it before. (Lan et al 2022) has discussed the reparameterized policy gradient as an additional discovery, where the conclusion equation is equivalent to our Equation (7) in Proposition 1. We have revised Section 4.1 to cite Lan et al 2022 and re-clarify the contributions. We also revised our contribution statement 1 in the introduction.\n\nAfter reviewing related works more comprehensively, we would like to re-clarify our contribution as follows \n\n1. Amortized actors are lightweight actors with strong multimodal expressiveness. However, they are underexplored in the field of multimodal RL. Previous practices in training amortized actors are based on SVGD [1,2]. The applicability of the reparameterized policy gradient for training amortized actors has not been formally discussed. We highlight that the reparameterized policy gradient is effective for training amortized actors.\n2. Though there have been some diffusion policy-based RL algorithms back-propagating the gradient of the Q-function [3,4] to update the diffusion policy, the theoretical perspective from reparameterized policy gradient has not been formally discussed. \n3. We formulate diffusion actors and amortized actors into a unified framework to show the versatility of reparameterized policy gradient for training intractable multimodal actors. This formulation serve as a useful tool for understanding and devising multimodal RL models and algorithms.\n\n[3] Wang, Yinuo, et al. \"Diffusion actor-critic with entropy regulator.\" *Advances in Neural Information Processing Systems* 37 (2024): 54183-54204.\n\n[4] Ding, Zihan, and Chi Jin. \"Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning.\" *The Twelfth International Conference on Learning Representations*."}, {"review_id": "KV2hHSEECx", "reviewer": "Reviewer_C7Lq", "summary": "This paper introduces PGRep, a reparameterized policy gradient method that works with a wide range of policies, including amortized and diffusion actors. It avoids using log-probabilities by adding a simple distance-based diversity regularizer, making it easier to train complex policies. The ideas build on existing methods like SAC and SQL, but the paper puts them together into a clear and useful framework. The experiments cover navigation, level generation, and continuous control tasks, showing strong results across settings.", "questions": "- For Figure 7(b), would you be able to provide **quality-diversity curves** for each algorithm by sweeping the diversity-regularisation coefficient (e.g., $\\alpha$ or $\\beta$) and plotting the resulting final return versus diversity?  This would reveal (i) how each method moves along the return–diversity frontier, and (ii) whether DrAC’s Pareto front consistently dominates those of the baselines.\n- Related to the previous point, how sensitive is DrAC’s performance to the choice of $\\alpha$/$\\beta$? \n- Figure 8 reports only episodic return on MuJoCo tasks.  To support the claim that DrAC delivers *both* quality and diversity, could you include an explicit diversity measure or visualization in this setting as well (e.g., action entropy, pairwise action distance, trajectory-level variation, or heat map)?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strength**\n- The paper provides a clear, general gradient formula that works for any stochastic-mapping actor, such as amortized or diffusion policies. This makes the method broadly applicable.\n- The method works well on a range of tasks, including navigation, level generation, and continuous control. It often outperforms strong baselines, showing that the approach is robust.\n\n**Weaknesses**\n- The reparameterization trick is standard in SAC and SQL-style methods. While some diffusion-based approaches use similar ideas [1] for trajectory optimization, typical diffusion policies for imitation learning do not involve policy gradients or reparameterization.\n- The benchmark for diversity measure is narrow. It only uses the reachable goal count as the diversity metric in evaluation. However, I think some diversity measures in conventional benchmarks are also important and insightful (see questions below).\n\n[1] Huang, Zhiao, et al. \"Reparameterized policy learning for multimodal trajectory optimization.\" International Conference on Machine Learning. PMLR, 2023.\n\n**Typos**\n- **Line 77** There is an extra period “.” at the end of the sentence—please remove it.\n- **Line 142 (Eq. 3)** The first term inside the argmax should read\n  $\n  \\alpha\\,\\varrho\\bigl(\\pi(\\cdot|s)\\bigr)\n  $\n  instead of\n  $\n  \\alpha\\,\\varrho\\bigl(\\pi(\\cdot|S_{t+k})\\bigr).\n  $\n- **Line 197** Replace “i.d.d.” with the standard abbreviation “i.i.d.”", "was_revised": true, "final_justification": "After reading the rebuttal, I still believe this is a technically solid paper and lean toward accepting it. Therefore, I've adjusted my confidence score accordingly.", "rebuttal": "We thank the reviewer for the insightful comments and suggestions. We run additional experiments to address reviewer’s questions and improve the thoroughness of our work. Due to the character limit, we only present most relevant and informative results using tables here. All presented results are averaged over 5 training seeds.\n\n**Regarding reviewer's questions:**:\n\n**Q1**: We are running all algorithms with different temperatures in the game content generation environment. We will add these figures in our revision. We have finished partial experiments, in which we train all algorithms with three temperatures (five seeds each) and test the quality and diversity of content generated by the trained policies. To provide informative response, we compute hyper volume (HV) metric [a] to evaluate the policy population obtained by each algorithm. Specifically, we treat the 15 policies learned by each algorithm with 3 temperatures and 5 seeds each as one population.We take the global lowest quality and global lowest diversity among all policies learned by all algorithm as the reference point, then compute the HV metric for each population. The result is displayed as follows. (Training of S$^2$AC has not finished as it is too computational intensive)\n\n**MarioPuzzle**\n\n| DrAmort | DrDiffus | SAC | DACER | SQL |\n| --- | --- | --- | --- | --- |\n| 48470.6 | 35318.0 | 36824.3 | 26880.4 | 17101.0 |\n\nAccording to the table, our DrAmort demonstrates highest HV value. The HV values of DrDiffus and SAC are closed. \n\n**MultiFacet**\n\n| DrAmort | DrDiffus | SAC | DACER | SQL |\n| --- | --- | --- | --- | --- |\n| 7559.7 | 6402.5 | 4157.8 | 3957.7 | 3603.0 |\n\nAccording to the table, our DrAmort and DrDiffus achieve the top 2.\n\nComprehensive experiment results will be discussed in our revised version, where we will also show visualization of Pareto fronts. Future work in generative RL for game content generation includes multi-objecive RL, where our method could serve as a promising base algorithm.\n\n[a] Zitzler, Eckart, and Lothar Thiele. \"Multiobjective evolutionary algorithms: a comparative case study and the strength Pareto approach.\" *IEEE transactions on Evolutionary Computation* 3.4 (2002): 257-271.\n\n**Q2**: We conduct additional experiments to investigate this question. Based finished experiments, we present some results below to respond to this question. We will add comprehensive results and analysis in our paper.\n\n**Multi-goal Maze**\n\nWe run DrAC with different values of $\\beta$ in the our multipoint-goal maze environments, the final performances in the Hard maze are presented as follows. \n\n**DrAmort** \n\n| beta | Success Rate | Reachable Modes | Robustness-Removel | Robustness-Obstacle |\n| --- | --- | --- | --- | --- |\n| 0.6 | 1.00 | 1.0 | 0.50 | 0.04 |\n| 0.7 | 1.00 | 1.4 | 0.56 | 0.01 |\n| 0.8 | 1.00 | 4.6 | 0.91 | 0.62 |\n| 0.9 | 0.41 | 7.4 | 0.60 | 0.24 |\n| 1.0 | 0.36 | 7.4 | 0.55 | 0.23 |\n\n**DrDiffus**\n\n| beta | Success Rate | Reachable Modes | Robustness-Removel | Robustness-Obstacle |\n| --- | --- | --- | --- | --- |\n| 0.6 | 1.00 | 1.6 | 0.58 | 0.02 |\n| 0.7 | 1.00 | 2.0 | 0.55 | 0.03 |\n| 0.8 | 1.00 | 2.4 | 0.65 | 0.21 |\n| 0.9 | 0.90 | 6.8 | 0.83 | 0.65 |\n| 1.0 | 0.45 | 7.2 | 0.57 | 0.25 |\n\nAccording to the results, success rate is not sensitive to beta within a wide range  (about <0.8 for DrAmort and <0.9 for DrDiffus). diversity-related performance is sensitive to beta. Higher $\\beta$ values enable DrAC to reach more modes and enhance few-shot robustness. However, overly high temperatures reduce the success rate, which also harms robustness metrics. We will discuss the sensitivity of beta comprehensively in our revised version.\n\n**Game content generation**: We finished evaluation with three $\\beta$ values for MarioPuzzle style. Results are presented below\n\n**DrAmort**\n| beta | Content Quality | Content Diversity |\n| --- | --- | --- |\n| 0.2 | 30.9 | 631.3 |\n| 0.5 | 29.9 | 1137.3 |\n| 0.8 | 32.8 | 1384.9 |\n\n**DrDiffus**\n| beta | Content Quality | Content Diversity |\n| --- | --- | --- |\n| 0.2 | 35.3 | 744.3 |\n| 0.5 | 23.1 | 1061.3 |\n| 0.8 | 21.1 | 1009.3 |\n\nThe quality of DrAmort is not sensitive to $\\beta$, while diversity increases by raising $\\beta$. For DrDiffus, quality increases and diversity decreases from $\\beta=0.2$ to $\\beta=0.5$, but $\\beta=0.8$ does not show significant difference with $\\beta=0.5$\n\nThere are two aspects may be related to the experiment phenomenon: (1) higher temperature also improves exploration which could be helpful to quality (2) policy diversity is correlated but not identical to the diversity of generated levels. More fine-grained ablation study will be conducted and discussed in our revision. Future work in generative RL for game content generation includes multi-objecive RL, where our method could serve as a promising base algorithm.\n\n**MuJoCo**\n\nWe are running additional experiments to analyze the sensitivity of DrAC in MuJoCo. Such experiments will be added to our revised version. We have finished partial experiments now. The performances of DrAmort with different values of $\\beta$, in the first two MuJoCo environments, are presented as follows\n\n| beta | Ant | HalfCheetah |\n| --- | --- | --- |\n| 0.15 | 2513.8 | 5487.0 |\n| 0.25 | 3327.9 | 5640.9 |\n| 0.35 | 3440.2 | 6558.2 |\n| SAC | 3422.7 | 5515.7 |\n\nPerformance of SAC is included as a baseline. According to the table, $\\beta$ values that are higher than our original setting ($\\beta=0.15$) improve the performance in Ant and HalfCheetah. Notably, the improvement in Ant from $\\beta=0.15$ to $\\beta=0.35$ is >36%, and with $\\beta=0.35$, DrAmort outperforms SAC by >18% in HalfCheetah. More elaborate self-adaptation/annealing techniques could be researched in future work.\n\n**Q3**: \nWe train DrAmort and DrDiffus with a higher $\\beta$ in Ant, and records the trajectories by evaluating the final policies. We compute average distance among those trajectories as a diversity measure, as presented below.  \n\n| $\\beta$ | DrAmort | DrDiffus |\n| --- | --- | --- |\n| 0.15 | 3736.6 | 7962.8 |\n| 0.35 | 5463.3 | 8676.6 |\n\nAccording to the table, higher $\\beta$ make trajectory diversity increase, though the only objective in MuJoCo is moving forward faster. We will visualize the trajectories in our revision.\n\n**Regarding specific weakness:**\n> The reparameterization trick is standard in SAC and SQL-style methods. While some diffusion-based approaches use similar ideas [1] for trajectory optimization, typical diffusion policies for imitation learning do not involve policy gradients or reparameterization.\n> \n> [1] Huang, Zhiao, et al. \"Reparameterized policy learning for multimodal trajectory optimization.\" International Conference on Machine Learning. PMLR, 2023.\n\nWe  have revised our paper to discuss [1]. We would like to clarify the novelty and significance here as follows:\n\n1. SAC applies reparameterization trick but only for tractable (squashed) Gaussian actors. SQL employs Stein variational gradient descent to back-propagate the estimated gradient of the KL-divergence objective in entropy-regularized RL. This is different from our gradient estimator formulated in Equation (7). To our best knowledge, no work has trained amortized actors with the reparameterized policy gradient as formulated in our Equation (7) before. Our work highlights the overlooked effectiveness of training amortized actors through the reparameterized policy gradient. We also demonstrates that amortized actor is highly potential for learning multimodal policy, which is underexplored before.\n2. Both [1] and our work adopt the reparameterization trick, however, [1] applies the reparameterization trick to estimate the gradient of an evidence lower bound, while our work applies the reparameterization trick to estimate the standard policy gradient. Our work  primarily differ from [1] in two aspects. (1) The algorithm proposed in [1] is a model-based algorithm that additionally learns a latent dynamics model, while our work focuses on the model-free setting; (2) Paper [1] reformulates the learning objective into an evidence lower bound of optimality, while our learning objective is expected diversity regularized return.\n3. Recently, some online RL algorithms for diffusion policies have been proposed, in which [b,c] also backpropagates the gradient of $Q$. These methods show promising performances. However, the theoretical interpretation of this operation from the perspective of reparameterized policy gradient is overlooked. We highlights that backpropagating $Q$ is an estimator of policy gradient with reparameterization trick, which is applicable to all stochastic-mapping actors including diffusion actors and amortized actors. This formulation serves as a useful tool for understanding and designing online model-free multimodal RL algorithms. \n\nWe have revised our paper to discuss related works more comprehensively and improve the clarity of our contributions \n\n[b] Wang, Yinuo, et al. \"Diffusion actor-critic with entropy regulator.\" *Advances in Neural Information Processing Systems* 37 (2024): 54183-54204.\n\n[c] Ding, Zihan, and Chi Jin. \"Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning.\" *The Twelfth International Conference on Learning Representations*.\n\n**For typos:**  We thank for reviewer's time and rigorous review. We will carefully revise them and further proofread our paper rigorously."}, {"review_id": "yfHaI8zh2i", "reviewer": "Reviewer_yEqZ", "summary": "This work proposes an actor-critic RL algorithm, Diversity-regularized actor-critic (DrAC), to train multimodal policies. The authors motivate their approach by noting that while previous works have employed generative models such as diffusion models as policies, they are limited by the fact that computing policy likelihoods for such models is intractable. To solve this issue, DrAC considers a parameterization that maps current states and a latent noise vector to the action, which also includes existing works on diffusion models. The authors propose training these actors through PGRep, formalized in Proposition 1. Experiments are performed in a modified PointMaze environment to study multimodal behaviours, game content generation to study the effectiveness of DrAC for generative tasks, and MuJoCo benchmark for continuous control tasks.", "questions": "- It is somewhat surprising that DrAmort works better than DrDiffus, since one would expect diffusion models to be more powerful samplers compared to a simple neural network conditioned on a noisy latent. The authors offer one possible explanation, that the number of denoising steps is too low. Could the authors try with more diffusion steps to get a sense of how well DrDiffus (and the other diffusion methods) can perform if we disregard the extra compute?\n- The parameter $\\beta$ seems like a crucial component of DrAC since it controls the trade-off between diversity and optimizing the Q-function. Could the authors discuss how sensitive DrAC is to $\\beta$ and how to tune this parameter in practice? Including an ablation study would also strengthen the empirical contribution.\n- Another parameter that warrants a discussion is the number of diversity estimation pairs. How does increasing the number of pairs affect the performance of DrAC?\n- From Table 2, the authors use the target entropy for SAC and DACER to be $0.5|\\mathcal{A}|$, whereas the standard practice is to use $|\\mathcal{A}|$. Could the authors explain this particular choice?\n- Also from Table 2, the authors use a fixed entropy coefficient for DACER in the multi-goal and game generation experiments, whereas the paper uses automatic temperature tuning like SAC. Could the authors explain this particular choice?\n- The authors mention (line 318) that DACER uses a more advanced critic learning technique. Could the authors elaborate on this? From their paper, it looks like they use standard double Q-learning.\n- Line 220: the authors state that determining the range of $D_\\theta$ is trivial, and they introduce $\\beta$ to “ease the triviality”. Could the authors explain what they mean here?\n\n**Minor errors and suggestions:**\n\n- Line 132 should also include the citation for Ziebart et al. [X] when discussing maximum entropy RL.\n- Line 134: the definition of $Q_\\text{soft}$ should include entropy for all future states $\\mathcal{H}(\\cdot | s_{t+k})$\n- Line 160: in Equation 6, the model $\\epsilon_\\theta$ does not predict the mean, it predicts the noise at each step.\n- Line 189: “The geometric mean is sensitive to small values as production is taken instead of summation” is grammatically incorrect - “production” is not a valid word to refer to the product of several terms.", "limitations": "The authors discuss avenues for future work, including improved architectures, different diversity metrics, and temperature schedules. However, there is no discussion on the limitations of their approach. Some points to include here could be the strong full-rank assumption in Proposition 1, the fact that the diversity surrogate may not align with the true entropy of the policy, and the dependence of DrAC on the parameter $\\beta$.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- This work unifies amortized actors and diffusion actors under a single framework, which could be useful for studying such parameterizations.\n- The paper contains diverse experiments, demonstrating that DrAC elicits multimodal behaviours, generates diverse latent vectors for a generative model, and performs competitively with other methods on continuous control tasks.\n- The authors have taken care to use the same network architecture across all methods for comparison. They also tune SAC properly on the MuJoCo benchmark, matching my personal observation that it is indeed a very strong baseline and most methods do not actually outperform SAC on these tasks.\n- The authors have provided a detailed hyperparameter table for DrAC and baselines in the appendix, as well as provided the code in the supplementary material. While I did not study the code, the details in the appendix are quite thorough, and the authors’ efforts in promoting reproducibility are appreciated.\n\n**Weaknesses**\n\n- Proposition 1 asserts that any actor expressible as a deterministic mapping $a = f_\\theta(s,z)$, where $z$ is sampled from some noise distribution, can be trained with the gradient $\\nabla_a Q \\nabla_\\theta f_\\theta$. This seems to be a restatement of the deterministic policy gradient theorem [1], except that the policy takes an additional input. As noted by the authors, existing methods like DACER (and also Diffusion QL [2] in the offline setting) use DPG to train actors. Because this practice of taking gradients through the Q-function is well understood and widely used, could the authors clarify what conceptual or algorithmic advance is provided by PGRep?\n- The assumption in Proposition 1 that $\\nabla_z f_\\theta(s, z)$ is full rank for all $s \\in \\mathcal{S}$ is a very strong assumption. It is well known that the gradients of a neural network develop a low-rank structure [3], which means the conditions for Proposition 1 will almost never be satisfied in practice.\n- The paper adopts a very specific operational definition of diversity and makes two claims that merit reconsideration.\n    - Lines 46–50 and 106–108 state that DACER is *unique* in accounting for decision diversity because it injects Gaussian noise into the sampled action. This overlooks the fact that any actor that samples from a multimodal policy (e.g., diffusion actors trained to sample approximately from the Boltzmann distribution of $Q$ [4,5,6,7]) inherently produces action diversity without additional noise, and DACER’s unimodal Gaussian perturbation does not necessarily translate into richer behavioural diversity. The current wording, therefore, understates prior work that achieves diversity implicitly.\n    - Diversity is measured via the geometric mean of pairwise action distances, which favours a single, broadly dispersed cloud over well-separated clusters. In many RL settings, separated clusters better capture multimodal behaviour, with each cluster corresponding to a distinct strategy, whereas a diffuse unimodal cloud may just reflect stochastic deviation around one mode. The paper would benefit from explaining why a “balanced spread” is preferable to explicit mode coverage. Since this regularizer is intended as a proxy for entropy, an empirical check would help: e.g., compute the actual entropy of the two distributions in Figure 3 - does distribution X indeed have higher entropy than distribution Y? Demonstrating that the geometric mean aligns more closely with entropy than alternative measures would strengthen the motivation for this choice.\n- Some closely related works [6,7] also learn a policy that samples from the Boltzmann distribution of the Q-function, and are not discussed. In particular, DQS [6] also exhibits multimodal behavior in a 2D navigation setting with sparse rewards, very similar to the experiment in Section 5.1\n\n*[1] Silver, David, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. \"Deterministic policy gradient algorithms.\" In International conference on machine learning, pp. 387-395. Pmlr, 2014.*\n\n*[2] Wang, Zhendong, Jonathan J. Hunt, and Mingyuan Zhou. \"Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning.\" In The Eleventh International Conference on Learning Representations, 2023.*\n\n*[3] Baker, Bradley T., Barak A. Pearlmutter, Robyn Miller, Vince D. Calhoun, and Sergey M. Plis. \"Low-rank learning by design: the role of network architecture and activation linearity in gradient rank collapse.\" arXiv preprint arXiv:2402.06751 (2024).*\n\n*[4] Psenka, Michael, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. \"Learning a diffusion model policy from rewards via q-score matching.\" arXiv preprint arXiv:2312.11752 (2023).*\n\n*[5] Yang, Long, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. \"Policy representation via diffusion probability model for reinforcement learning.\" arXiv preprint arXiv:2305.13122 (2023).*\n\n*[6] Jain, Vineet, Tara Akhound-Sadegh, and Siamak Ravanbakhsh. \"Sampling from Energy-based Policies using Diffusion.\" arXiv preprint arXiv:2410.01312 (2024).*\n\n*[7] Ishfaq, Haque, Guangyuan Wang, Sami Nur Islam, and Doina Precup. \"Langevin Soft Actor Critic: Efficient Exploration through Uncertainty-Driven Critic Learning.\" arXiv preprint arXiv:2501.17827 (2025).*", "was_revised": true, "final_justification": "During the rebuttal period, the authors provided ablation studies on the two main hyperparameters and an additional comparison between the diffusion actor and the amortized actor. These results improved the empirical contribution of the work.\n\nThe main concerns were (1) regarding the definition of diversity adopted in the paper, and whether it is a useful proxy for entropy (as was motivated in the paper), and (2) Proposition 1 being a restatement of the deterministic policy gradient theorem. \n\nDuring the discussions, the authors indicated that the goal was *not* to suggest a proxy for the entropy, and provided some empirical evidence that increasing the diversity using their method leads to an increase in entropy. The authors also assured that Proposition 1 will be rewritten as an equation since it is a known result. In light of these changes, I update my rating to weak accept.", "rebuttal": "We thank the reviewer for the insightful comments and suggestions.  We run additional experiments to address reviewer’s questions and improve the thoroughness of our work. Due to the character limit, we only present most relevant and informative results using tables here. All presented results are averaged over 5 training seeds.\n\n**Regarding reviewer's questions:**\n\n**Q1:** Results in the Medium maze are as follows\n\n**DrDiffus**\n| Diffusion steps | Success Rate (SR) | Reachable Modes (RM) | Robustness-Removal (RR) | Robustness-Obstacle (RO) |\n| --- | --- | --- | --- | --- |\n| 10 | 1.00 | 1.2 | 0.59 | 0.13 |\n| 20 | 1.00 | 1.6 | 0.65 | 0.10 |\n| 30 | 1.00 | 1.4 | 0.69 | 0.14 |\n| 40 | 1.00 | 2.6 | 0.81 | 0.11 |\n\n**DACER**\n| Diffusion steps | SR | RM | RR | RO |\n| --- | --- | --- | --- | --- |\n| 10 | 1.00 | 1.0 | 0.50  | 0.10 |\n| 20 | 1.00 | 1.0 | 0.50 | 0.09 |\n| 30 | 1.00 | 1.0 | 0.50  | 0.12 |\n| 40 | 1.00 | 1.0 | 0.50  | 0.12 |\n\nThe results demonstrate that the diversity-related performance of DrDiffus is improved by raising the number of diffusion steps. While for DACER, the diversity-related performance is not improved. This is probably due to that DACER controls diversity by scaling an additional Gaussian noise instead of optimizing diversity through gradients w.r.t. the actor's parameters.\n\n**Q2:** We conduct additional experiments for this question, existing results includes: \n\n**Multi-goal Maze:**\n\n**DrAmort**\n| beta | SR | RM | RR | RO |\n| --- | --- | --- | --- | --- |\n| 0.6 | 1.00 | 1.0 | 0.50 | 0.04 |\n| 0.7 | 1.00 | 1.4 | 0.56 | 0.01 |\n| 0.8 | 1.00 | 4.6 | 0.91 | 0.62 |\n| 0.9 | 0.41 | 7.4 | 0.60 | 0.24 |\n| 1.0 | 0.36  | 7.4 | 0.55 | 0.23 |\n\n**DrDiffus**\n| beta | SR | RM | RR | RO |\n| --- | --- | --- | --- | --- |\n| 0.6 | 1.00 | 1.6 | 0.58 | 0.02 |\n| 0.7 | 1.00 | 2.0 | 0.55 | 0.03 |\n| 0.8 | 1.00 | 2.4 | 0.65 | 0.21 |\n| 0.9 | 0.90 | 6.8 | 0.83 | 0.65 |\n| 1.0 | 0.45 | 7.2 | 0.57 | 0.25 |\n\n**Game Content Generation:**\n\n**DrAmort**\n| beta | Content Quality | Content Diversity |\n| --- | --- | --- |\n| 0.2 | 30.9 | 631.3 |\n| 0.5 | 29.9 | 1137.3 |\n| 0.8 | 32.8 | 1384.9 |\n\n**DrDiffus**\n| beta | Content Quality | Content Diversity |\n| --- | --- | --- |\n| 0.2 | 35.3 | 744.3 |\n| 0.5 | 23.1 | 1061.3 |\n| 0.8 | 21.1 | 1009.3 |\n\n**MuJoCo**\n\n| beta | Ant | HalfCheetah |\n| --- | --- | --- |\n| 0.15 | 2513.8 | 5487.0 |\n| 0.25 | 3327.9 | 5640.9 |\n| 0.35 | 3440.2 | 6558.2 |\n| SAC | 3422.7 | 5515.7 |\n\nWe will add comprehensive results and analysis to study this question in our paper.\n\nRegarding how to tune the temperature hyperparameter, generally when we need better diversity or exploration we shall raise $\\beta$, and when we need more exploitation we shall reduce $\\beta$. Our definition $\\beta$ is normalized so that the scale of $\\beta$ is comparable to the per-dimension pairwise distance between actions (See the response to Q7).  So one can roughly decide the range of $\\beta$ and may be able to set $\\beta$ according to the expected pairwise distance of actions. We will detail this in our paper.\n\n**Q3:**  In Medium Maze:\n\n**DrAmort**\n| Number of pairs | SR | RM | RR | RO |\n| --- | --- | --- | --- | --- |\n| 4 | 98.9% | 3.88 | 98.8% | 51.7% |\n| 8 | 99.5% | 3.20 | 97.9% | 49.6% |\n| 12 | 99.2% | 3.76 | 97.7% | 49.8% |\n\n**DrDiffus**\n| Number of pairs | SR | RM | RR | RO |\n| --- | --- | --- | --- | --- |\n| 4 | 84.3% | 1.52 | 64.1% | 14.4% |\n| 8 | 100.0% | 1.64 | 62.8% | 10.6% |\n| 12 | 100.0% | 1.52 | 64.9% | 13.1% |\n\nThe success rate of DrDiffus with $n=4$ is lower than that with larger values of $n$. Expect for this, the performances of DrDiffus and DrAmort are not sensitive to $n$. To provides some intuition for tuning $n$, it is most likely that raising $n$ will reduce the variance in estimating diversity, but usually it not needs to be very large.\n\n**Q4:**\nFor multi-goal PointMaze, we conducted a grid search for each algorithm before the formal evaluation. Based on the grid search, we picked the highest temperature that ensures each algorithm reaches an optimal success rate, to enable a fair and meaningful comparison. $0.5|\\mathcal{A}|$ is the highest temperature value we found that still ensure them reach the optimal success rate, so we picked this value. \n\nFor game content generation, we just follows the temperature settings in multi-goal PointMaze. \n\nFor the MuJoCo experiements, we kept the default target entropy value as described in their papers. Our Table 3 lists the temperature/target entropy/entropy coefficients for all algorithms in MuJoCo.\n\n**Q5:**\nThis is a typo and we are very sorry for that. We used temperature tuning technique like SAC following the paper of DACER. DACER does not use fixed $\\alpha$ so there should be N/A in the corresponding cell in Table 2, and $\\alpha=0.6$ is for S2AC. We sincerely thank the reviewer for the careful review and have revised this error.\n\n**Q6:**\nUnder equation (11) of the DACER paper (the NeurIPS 2024 camera-ready version), it is stated “we employ the tricks in DSAC [11, 10] to mitigate the problem of Q-value overestimation.” And we carefully checked their official code and found that they used such a technique. That technique was introduced in [a], which is a distributional critic. We have revised our paper to make it more clear.\n\n[a] Duan, Jingliang, et al. \"Distributional soft actor-critic with three refinements.\" *IEEE Transactions on Pattern Analysis and Machine Intelligence* (2025).\n\n**Q7:**\nThe auto-tuning technique operates on $\\alpha$ to match the actual pairwise distance-based diversity meaure $D(\\pi)$ with a target $\\hat D$, i.e., $D(\\pi)=\\mathbb{E}[\\log\\delta(a^x, a^y)] \\approx \\hat D$. The diversity regularization $\\mathbb{E}[\\log\\delta(a^x, a^y)]$ takes logarithms and it is not so straightforward to determine the proper range of it. By introducing $\\beta$, we have $\\mathbb{E}[\\log\\delta(a^x, a^y)] \\approx \\log(\\beta|\\sqrt{\\mathcal{A}}|) \\Rightarrow \\mathbb{E}[\\delta(a^x,a^y)]/\\sqrt{|\\mathcal{A}|} \\approx \\beta$, so the proper range is easier to understand and one may determine $\\beta$ according to the promising value of distances between actions.\n\n**Regarding specific weaknesses:**\n> Proposition 1 asserts that ... could the authors clarify what conceptual or algorithmic advance is provided by PGRep?\n\n1. Amortized actors are lightweight policy models with strong multimodal expressiveness. However, algorithm for training amortized actors is underexplored in the field of multimodal RL, and to our best knowledge, no work has trained amortized actors reparameterized policy gradient. We highlight that the reparameterized policy gradient is effective for training amortized actors.\n2. Though there have been some diffusion policy-based RL algorithms back-propagating the gradient of the Q-function to update the diffusion policy, the theoretical interpretation from the perspective of reparameterized policy gradient is overlooked. \n3. We introduce a unified formulation for a range of intractable actors and highlight that PGRep is applicable to all these actors. This formulation serves as a useful tool for understanding and designing multimodal RL models and algorithms.\n\n> The assumption in Proposition 1 that is full rank ... almost never be satisfied in practice.\n\nThanks for raising this point. We acknowledge that this type of assumption, which serves as a sufficient condition to establish the result, is strong. Yet, we find that it is not necessary for the practical algorithm to work well, which is also a common theory-practice gap in deep RL literature, where theoretical guarantees often rely on stronger assumptions while practical algorithms succeed despite some violations [b]. We would like to emphasize that this provides insights for the development of our algorithm in Section 4.2 even when the assumption is not strictly satisfied, where our extensive experiments show that DrAC works effectively across diverse environments and backbone components. This consistent performance gains suggest that the core mechanism in DrAC is robust to potential violations of theoretical assumptions. \n\nIn the updated version, we will discuss the implications for Proposition 1 and explore connections to rank-deficient approximations that may explain why the method works despite assumption violations, and also further contextualize this assumption within the broader RL literature, which is a very interesting future direction for our work to explore smoother assumptions (e.g., regularity conditions for the gradients). Despite this theoretical point, our work remains significant, which aims to develop a unified framework for multimodal actors with a novel distance-based regularization approach that achieves consistent performance gains in diversity-critical domains with improved generalization ability.\n\n[b] Tang, Y., & Agrawal, S. (2018). Boosting trust region policy optimization by normalizing flows policy. *arXiv preprint arXiv:1809.10326*.\n\n> Diversity is measured via the geometric mean of pairwise action distances, ... would strengthen the motivation for this choice.\n\nIn Figure 2, the ground truth of $X$ is a Gaussian distribution with $\\sigma=1$, while the ground truth of $Y$ is a mixture-of-Gaussian with uniform weights and two Gaussian components with $\\sigma_1=\\sigma_2=0.2$. We calculated their entropy: $\\mathcal{H}(X)=2.838$,  and $\\mathcal{H}(Y)\\approx 0.3121$. This demonstrates that geometric mean aligns more closely with entropy compared with the vanilla average pairwise distance measure. We have updated Figure 3 to include such information.\n\nWe agree that separated clusters better capture multimodal behaviour, while there still exists a trade-off, since overly small clusters might harm local exploration. The regularizer presented in Equation (8) is our best practice so far.\n\n**For other concerns about missed references, understatement for prior works, minor errors:** We thank for reviewer's time and rigorous review and will carefully revise them in our paper."}, {"review_id": "qWAE1v67Qs", "reviewer": "Reviewer_76hu", "summary": "The paper proposes a policy gradient method for multimodal actors, and distance-based diversity regularization to boost exploration for diverse policies. Experiments are mainly conducted on two diversity-critical benchmarks, to show the improved diversity of the policies and learning performances. Experiments on conventional benchmark MuJoCo show the worse performances of the proposed methods against unimodal baseline SAC.", "questions": "In Fig. 5 (a), why are there a drop in performances in other baselines for simple and medium environments?\n\nHow much additional computation is introduced by distance-based diversity measure during training?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe paper proposes a new sampling-based diversity measure for regularization purpose.\n\nThe two diversity-critical benchmarks are more proper to evaluate the multimodal policy representation.\n\nThe experiment analysis is quite comprehensive.\n\nWeakness:\n\nFig. 2 is hard to interpret, I cannot understand how Fig. 2(a) unifies the two types of actors.\n\nThe unification of amortized and diffusion actors with reparameterized policy gradient seems trivial.\n\nThe paper mentions that the sampling-based diversity measure can be parallelized over GPUs for generating multiple samples, but this inevitably introduces additional computational costs during training.\n\nOn traditional MoJoCo benchmarks, the proposed methods perform worse than unimodal policy SAC.\n\nMissing references and baselines. The multimodal policy representation in RL has been studied for a long time, and I think there are quite a lot of papers missing as references and baselines from this paper. Discussion and comparisons against these baselines would be necessary to prove the advantages of the proposed methods.\n\nFor mixture-of-Gaussian:\n\n[1] Peng, X. B., Chang, M., Zhang, G., Abbeel, P., and Levine, S. MCP: learning composable hierarchical control with multiplicative compositional policies.\n\n[2] Ren, Jie, et al. \"Probabilistic mixture-of-experts for efficient deep reinforcement learning.\n\nFor gradient optimization on diffusion policy:\n\n[3] Ren, Allen Z., et al. \"Diffusion policy policy optimization.\n\n[4] Chen, Yuhui, Haoran Li, and Dongbin Zhao. \"Boosting continuous control with consistency policy.\"\n\n[5] Ding, Zihan, and Chi Jin. \"Consistency models as a rich and efficient policy class for reinforcement learning.\"", "was_revised": true, "final_justification": "Thanks for the rebuttal. Given the unaddressed concerns regarding why amortized actor is better than diffusion actor, the lack of baseline like DPPO method, and unclear whether the diffusion gradients of policy value and entropy term are carefully handled, I keep my current score.", "rebuttal": "We thank the reviewer for the insightful comments and suggestions. We run additional experiments to address reviewer’s questions and improve the thoroughness of our work. Due to the character limit, we only present most relevant and informative results using tables here. All presented results are averaged over 5 training seeds.\n\n**Regarding reviewer's questions:**\n\n**Q1:** \nThose baselines suffering from a performance drop are entropy-regularized algorithms. Based on the periodic visualization of trajectories in the mazes, we observed that the performance drop of those baselines is accompanied by exploration behaviours. At that stage, these algorithms appear to “wobble” among different goals. Possibly the gradient of entropy bonus pushes the actor to other directions and makes the policy explore in the maze and fails to reach goals. DACER handles entropy regularization by auto-adjusting the scale of an auxiliary noise instead of back-propagating the gradient of entropy regularization. Moreover, the computational cost of adjusting the scale is high, making such adjustment usually perform at a very low frequency (by default, DACER updates the scale every 10000 exploration steps). This possibly be the reason why DACER does not suffer from a performance drop. A more in-depth investigation shall be future work.\n\n**Q2:** \nLet $n$ ($n=8$ in our experiments) denote the number of diversity estimation pairs, we summarize the number of forward inferences and back-propagation per transition in one single update as follows.\n\n|  | Actor forward | Actor backward | Critic forward | Critic backward |\n| --- | --- | --- | --- | --- |\n| w/o diversity regularization | 2 | 1 | 3 | 2 |\n| w/ diversity regularization | 4n+2 | 2n+1 | 3 | 2 |\n\nThe diversity estimation introduces $4n$ additional forward inference and $2n$ additional back-propagation, without additional forward and backward of the critic. As the neural networks used for continuous control are typically small (e.g., SAC uses MLPs with two 256-dimensional hidden layers by default), we empirically found DrAmort and DrDiffusion do not induce significantly more training times and memory usage, as displayed in the following table.\n\n>This table shows average time cost per 1000 training steps in Ant-v4 environment. The test is conducted on a Linux server with 8 NVIDIA RTX 3090 GPUs and an Intel Xeon Platinum 8375C CPU. For each algorithm, we run three trials on GPU cards 0, 1 and 2, respectively, and then average the training time per 1000 steps. All algorithms are implemented with PyTorch 2.5.1, and the CUDA version is 12.5. \n|  | SAC | SQL | DrAmort | DACER | DrDiffus | S2AC |\n| --- | --- | --- | --- | --- | --- | --- |\n| Time per 100 training steps | 22.7s | 18.9s | 21.0s | 76.9s | 84.8s | 220.5s |\n| Peak GPU memory usage | 352M | 472M | 392M | 364M | 608M | 3462M |\n\nAccording to the table, we found that DrAmort is even faster than SAC. The reason is that SAC requires computing and back-propagating the log-probability at certain actions, which requires more sequential tensor operations. Though DrAmort requires more FLOPs, the computation is parallelized and takes fewer sequential tensor operations. As a result, DrAmort uses even fewer GPU clock ticks and is slightly faster than SAC, given that the neural network is sufficiently small compared to the GPU’s parallel computation capability. For DrDiffus, it takes about 10% more time for training compared to DACER, but its number of reachable goals in multi-goal PointMaze is much higher.\n\nFor large-scale tasks that require larger networks, such as visual control, we can inject the latent variable at the last few layers. For example, if a CNN is used, the latent variables can be injected after the convolutional layers. In this case, the computation time and memory usage will not significantly increase.\n\n**Regarding specific weakness:**\n\n> The unification of amortized and diffusion actors with reparameterized policy gradient seems trivial.\n\nWe would like to justify our contribution regarding this point as follows:\n1. Previous practices in training amortized actors are based on SVGD [a,b]. The effectiveness of the reparameterized policy gradient for training amortized actors is underexplored. We highlight that the reparameterized policy gradient is effective for training amortized actors.\n2. Previous works did not formally discuss the theoretical connections between backpropagating $Q$ and policy gradient. \n3. By formulating diffusion actors and amortized actors into a unified framework, we demonstrate the versatility of reparameterized policy gradient for training intractable multimodal actors. This formulation is general and extendable. It serve as a useful tool for understanding and devising multimodal RL models and algorithms. \n\nWe have revised section 4.1 and the contribution statement 1 in the introduction to make the contribution of this part clearer and accurate.\n\n[a] Haarnoja, Tuomas, et al. \"Reinforcement learning with deep energy-based policies.\" *International conference on machine learning*. PMLR, 2017.\n\n[b] Messaoud, Safa, et al. \"S$^2$AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic.\" *The Twelfth International Conference on Learning Representations*.\n\n> The paper mentions that the sampling-based diversity measure can be parallelized over GPUs for generating multiple samples, but this inevitably introduces additional computational costs during training.\n\nThough the estimation of our diversity measure inevitably introduces additional computations during training. The actual training time is similar to the baselines as those operations are fully parallelized. For large-scale tasks, we can inject the latent variable at the last few layers to avoid much additional computation (See the response to Q2).\n\n> On traditional MoJoCo benchmarks, the proposed methods perform worse than unimodal policy SAC.\n\nOur core contribution is training mutlimodal actors with optimized diversity, which is verified with the other two domains. The main purpose of our experiment in MuJoCo benchmarks is to show our algorithm achieves competitive performance against SAC in conventional tasks. It is not so quite surprising that DrAC does not outperform SAC, as the only goal in MuJoCo is to move faster, making multimodality less important. In addition, while amortized actors are fast and possess strong multimodal expressiveness, there is a lack of an efficient and effective algorithm to train them prior to our work. SQL performs badly on MuJoCo and S2AC costs much more time and memory usage. Our algorithm fills this research gap.\n\nMeanwhile, as suggested by other reviewers, we are conducting comprehensive ablation study, including in MuJoCo benchmarks. We have finished the training of DrAmort with other $\\beta$ values in Ant and HalfCheetah. The results suggest that the reason that our algorithm does not outperforms SAC is probably insufficient parameter tuning\n\n| $\\beta$ | Ant | HalfCheetah |\n| --- | --- | --- |\n| 0.15 | 2513.8 | 5487.0 |\n| 0.25 | 3327.9 | 5640.9 |\n| 0.35 | 3440.2 | 6558.2 |\n| SAC | 3422.7 | 5515.7 |\n\n> Missing references and baselines. The multimodal policy representation in RL has been studied for a long time, and I think there are quite a lot of papers missing as references and baselines from this paper. Discussion and comparisons against these baselines would be necessary to prove the advantages of the proposed methods.\n\nWe have revised our paper to discuss related works more comprehensively. \n\nTo address reviewer’s concern, we have tested PMOE [2] and CPQL [4] in our multi-goal environment. For CPQL, we run the official code with default settings as it does not provide a hyperparameter to trade off between return and diversity. For PMOE, we perform grid search as we done for other algorithms, and picked the highest target entropy that ensures it achieves the optimal success rate. The results in the Medium maze are presented as follows.\n\n|  | Success Rate | Reachable Modes | Robustness-Removal | Robustness-Obstacle |\n| --- | --- | --- | --- | --- |\n| DrAmort | 1.00 | 3.2 | 0.98 | 0.48 |\n| DrDiffus | 1.00 | 1.6 | 0.65 | 0.10 |\n| PMOE | 1.00 | 2.0 | 0.82 | 0.15 |\n| CPQL | 1.00 | 1.0 | 0.50 | 0.00 |\n\nAccording to the results, CPQL does not exhibit diversity. PMOE demonstrates diversity but our DrAmort outperforms PMOE. Moreover, PMOE trains tractable mixture-of-Gaussian policies that only represent a fixed number of modes, while our algorithm is able to train intractable policies that can express arbitrary number of modes."}]

Author rebuttal:

Meta-review: This paper introduces Diversity-Regularized Actor-Critic (DrAC), a method for learning multimodal policies in continuous control settings. DrAC enforces diversity using a distance-based regularization loss and is implemented with two types of actors, i.e., amortized (DrAmort) and diffusion-based (DrDiffus). While DrAC's performance is on par with SAC on standard MuJoCo benchmarks, it demonstrates clear benefits in multi-goal and generative tasks where policy diversity is essential.

Reviewers found the paper to be a valuable exploration of expressive policies in understudied but important domains. During the rebuttal, the authors provided additional experiments that successfully addressed most initial concerns regarding hyperparameter sensitivity and the comparison between the DrAmort and DrDiffus variants.

A concern shared across multiple reviewers remains regarding the observation that the amortized actor outperforms the diffusion actor. While the authors correctly state there is no prior evidence to suggest the contrary, the paper would be strengthened by providing more intuition or analysis to explain this performance difference. Nevertheless, the overall contribution is solid and the paper is recommended for acceptance.
Decision: Accept (poster)