Comparing Deterministic and Soft Policy Gradients for Optimizing Gaussian Mixture Actors
Authors: Sheelabhadra Dey
Guni Sharon
Venue: Accepted by TMLR ()

Abstract: Gaussian Mixture Models (GMMs) have been recently proposed for approximating actors in actor-critic reinforcement learning algorithms. Such GMM-based actors are commonly optimized using stochastic policy gradients along with an entropy maximization objective. In contrast to previous work, we define and study deterministic policy gradients for optimizing GMM-based actors. Similar to stochastic gradient approaches, our proposed method, denoted $\textit{Gaussian Mixture Deterministic Policy Gradient}$ (Gamid-PG), encourages policy entropy maximization. To this end, we define the GMM entropy gradient using $\textit{Variational Approximation}$ of the $KL$-divergence between the GMM's constituting Gaussians.  We compare Gamid-PG with common stochastic policy gradient methods on benchmark dense-reward MuJoCo tasks and sparse-reward Fetch tasks. We observe that Gamid-PG outperforms stochastic gradient-based methods in 3/6 MuJoCo tasks while performing similarly on the remaining 3 tasks. In the Fetch tasks, Gamid-PG outperforms single-actor deterministic gradient-based methods while performing worse than stochastic policy gradient methods. Consequently, we conclude that GMMs optimized using deterministic policy gradients (1) should be favorably considered over stochastic gradients in dense-reward continuous control tasks, and (2) improve upon single-actor deterministic gradients.

Published in Transactions on Machine Learning Research (12/2024)
Comparing Deterministic and Soft Policy Gradients for Optimizing Gaussian Mixture Actors
Sheelabhadra Dey
sheelabhadno@gmail.coo
Deportment of Computer Science and Engineering Teras ABM University
Guni Sharon
gumii Gtama edu
Deportment of Computer Science and Engineering
Texas ABM University
Reviewed on OpenReview: https: // openreview net/ per/ffrum? id= 9S9PPuB0Dt
Abstract
Gaussian Mixture Models (GMMs) have been recently proposed for approximating actors in actor-critic reinforcement learning algorithms. Such GMM-based actors are commonly optimized using stochastic policy gradients along with an entropy maximization objective. In contrast to previous work, we define and study deterministie policy gradients for optimiz- ing GMM-based actors. Similar to stochastic gradient approaches, proposed method, denoted Gaussian Minture Deterministic Policy Gradient (Gamid-PG), encourages policy entropy maximization. To this end, we define the GMM entropy gradient using Varia- lional Approcimation of the K KL-divergence between the GMM's component Gaussians. We compare Gamid-PG with common stochastic policy gradient methods on benchmark dense- reward MuJoCo tasks and sparso-reward Fetch tasks. We observe that Gamid-PG outper- forms stochastic gradient-based methods in 3/6 MuJoCo tasks while performing similarly on the remaining 3 tasks. Tn the Fetch tasks, Gamid-PG outperforms single-actor determinis- tic gradient-based methods while performing worse than stochastic policy gradient methods. Consequently, we conclude that GMMs optimized using deterministic policy gradients (1) should be favorably considered over stochastic gradients in dense-reward continuous control tasks, and (2) improve upon single-actor deterministic gradients.
1 Introduction
This study presents comparison between deterministic and stochastic policy gradients for optimizing policic represented Gaussian Mixture Models (GMMs) in model-free deep reinforcement learning (RL). Model-
free RL was successfully demonstrated in various control domains. Examples include video games Mnih
et al.
Wurman et al. 2022). robotic control Ellguea-Aguinac et al. 2023 Dey et al. 2021 2024 Kober
ct al.
2013). traffic applications (Ault Sharon 2021 Sharon 2021), and medical procedures [hoo et al.
2021
Coronato et. al. 2020). Specifically, actor-critic methods (Liliierap et al. 2015 Fujimoto et. al. 2018
Haarnoja et al. 2018 Schulman et al., (017) have proven be effective for RL in continuous control domains.
These methods were shown be most effective when coupled with high-capacity function approximators such neural networks for approximating both the actor and the critic.
Prior work on actor approximation can be broadly divided into two categories; (1) deterministic Siiver et al. 2014), where the policy approximator maps states to actions, and (2) soft (Haarnoja et al. 2018). where the policy approximator maps states to a distribution defined over the action space. Each these two classes has known benefits and limitations. In particular, deterministic actors are unable to represent complex policies that capture several modes of optimal behavior while soft actors commonly assume a specific parametric distribution (often oversimplified) for which closed-form gradient, with respect to the action distribution

---

Published in Transactions on Machine Learning Research (12/2024)
parameters (e.g.. mean and variance of a Gaussian), is well-defined. A line of publications (Ren et al. 2021 Peng et al. 2020; Akrour et al. 2021 Baram et al. (021) attempted to address this gap by defining a closed- form gradient for general distributions within soft-actor optimization algorithms. These methods attempt to fit the soft actor's action distribution to the critic's action-value distribution. Empirically (on benchmark MuJuCo domains), such approaches present no, or marginal, benefits over a single-Gaussin variant (Baram
ct al. 2021).
To overcome the limitations associated with single Gaussian actors, Baram et al. 2022 proposed using Gaussian Mixture Model (GMM) the policy approximator in maximum entropy (MaxEnt) framework. GMMs are known to be universal approximators of densities (Alspach & Sorenson 1972 and result. in a general soft actor. The proposed approach was implemented and evaluated within the Soft. Actor-Critic (SAC) (Haarnoja et al. 2018) algorithm. The authors did not find clear empirical evidence supporting the use of GMMs over a single Gaussian. They attribute this finding to two reasons: "First, benchmarked tasks are unimodal in nature, so a unimodal policy should do. Second, the mixture policy collapses to mean policy in the presence of equal mixing weights." In contrast to previous work, we propose training the GMM- based soft actor using deep deterministic policy gradient (DDPG) approach (Lillicrap et al. 2015; Fujimoto ct al. 2018 Unlike soft actor-critic methods, DDPG docs not attempt to capture an underlying action-value distribution but attempts to converge on a (deterministic) local optimal action. When coupled with a GMM, such deterministic policy gradient algorithms converge on a set of local optima under common convergence conditions. However, as previously reported (Baram et al. 2021), GMM-based actors might suffer from situations where "the mixture policy collapses to mean policy". Addressing this issue, we demonstrate how maximum entropy RL principles (Aghasadeghi & Bretl, 2011: Toussaint, 2009 Rawlik et al. 2012 Haarnoja ct al. 2018 Fox ct al. 2016) can be incorporated into the deterministic actor's gradicnt using Variational Approzimation (Hershey & Olsen 2007) of the KI-divergence between the GMM actor and a single Gaussian. We term our proposed approach Gaussian Mixture Deterministic Policy Gradient (Gamid-PG).
We conclude our study with four sets of experiments: (1) we present illustrative results on a simplified domain to showcase Gamid-PG's capability to capture multimodal behavior; (2) we conduct an empirical investigation to determine the conditions under which Gamid-PG outperforms stochastic policy gradients, and vice versa; (3) we undertake comparative evaluation by benchmarking Gamid-PG against established deterministic and stochastic madient-bbssed methods in challenging continuous control environments with high-dimensional action spaces; and (4) we test the sensitivity of Gamid's d's hyperparameters on its performance on a representative domain.
The reported results suggest that for dense-reward control tasks with continuous action spaces, the determin- istic gradient approach (Gamid-PG) can be more effective than stochastic gradient approaches to optimize GMMs and should be considered by researchers and practitioners. While, for sparse-reward tasks, stochastic gradient approaches are observed to be more effective than deterministic gradient approaches. Nevertheless, Gamid-PG is shown to improve upon single-actor deterministic gradient approaches across all the benchmark domains.
2 Preliminaries
2.1 Reinforcement learning (RL)
In RL. a policy (actuation function) is optimized over an underlying Markov decision process (MDP) which is a tuple (S, A, P. R. 7, To}. is the state space; A is the action space; P(s, a,,,a,s)) s') is the transition probability of the form P: : 8 x A S + [0, 1], representing the probability of transitioning from state to state s' after taking action a; R(s,a) a) is the reward function of the form R : S x A - E, representing the immodiate utility gained from being in state and taking action a; y is the discount factor, representing the factor of lost utility in future rewards; finally, lu is a distribution over the initial state.
An RL agent is assumed to follow an internal policy, n, which maps states to actions. & is commonly defined either as deterministic, i.c., π S + A, or as soft (stochastic), i.c., mapping states to a distribution over actions. The agent's chosen action (at) at the current state (s:) affects the environment such that new state emerges ($e+1) well as some reward (re) representing the immediate utility gained from
2

---

Published in Transactions on Machine Learning Research (12/2024)
performing action de at state St, given by R(st, at). is used to denote finite horizon trajectory of the form (8o, do, ro, $1,-.., -1,
The expected relurn or expected sum of discounted rewards for a given policy is denoted by (TT) = le 2Tr. In RL the observed return is used to tune a policy such that (TT) is maximized. The policy arg max, [xx)] is the optimal policy and is denoted by π -
Common RL frameworks include value-based, policy-gradient, and actor-critic approaches. A value- based approach attempts to learn the expected future utility from states (state value) or from action-state pairs (action value or q-value). The policy returns actions that maximize the expected utility (T(s) arg ( a))). A prominent example of value-based approach is the model-free deep Q-learming algorithm (DQN) (Mnih et al., 2015). In the policy-gradient approach (Williams) 1992) a policy is defined through parameterized differential equation, where the policy parameters are iteratively updated, following the policy gradient, towards favorable outcomes (as experienced through the reward function). Using state or action value approximations for defining favorable outcomes for policy-gradient updates is usually referred to as an actor-critic approach. A prominent example of a statc-of-tho-art actor-critic approach is deep- caraniiiisii--ooiiegggaiieeee (DDPG) (Lillicrap el. al. 2015 Fujimoto et al. 2018
2.2 Distributional RL
In order to capture the intrinsic uncertainty of MDPs, a line of publications proposed to extend value- based approaches to estimate the distribution over returns. These include SPI-DQN (Luo et al. 2021 bistibuutioua--polleg-raaddeei
(Singh et al., 2022 Song Zhao 2020), and disttubbutee--dissriuutooo-----d (D4PG) (Barth-Maron et al., [0118]. These approaches still converge on a deterministie policy and, thus, might fail capture diverse modes of optimal behavior, i.e., while the critic (Q-network) can capture multiple modes of optimality in the Q-value space, the (unimodal) actor is limited in its ability to do the same.
2.3 Soft policy density approximation
Another line of work suggests training a policy as a distribution over actions, i.e., a soft policy. The fit- ted distribution is commonly parametric to allow closed-form gradient computation. For example, training mean and variance parameters of a Gaussian. Common parametric policy distributions used in the liter- ature include Gaussian (Schulman al., 2015 2017), Beta (Chou et al.) (017)) and Delta (Silver et al., 2014 Lillicap et al., 2015 Fujimoto et al. (018)) While such parametric distributions are casy train (having closed-form gradients), they provide a limited representation power. Addressing this issue, Tessler (Koenker & Hallock, (Ostrovski et al. 2018) that can represent arbitrarily densitics. However, the reported results for GAC are not better (asymptotic performance and sample cffi- ciency) than those reported for Soft Aclor-Criläa (SAC) (Haarnoja et al. 2018 which uses a single Gaussian as a soft actor policy, Another approach for capturing complex distributional properties, such as skewness, kurtosis, multimodality, and covariance structure is the Semi-Implicit Actor (SIA) (Yue al.) 2020). This approach adopts a semi-implicit hierarchical construction (Yin & Zhou (yet not general) parametric distributions.
et al. 2019) proposed Generative Actor-Critic (GAC). It applies
2001 over huutoregressiee-Ippiiee
Chuantile-Netwone
2018 for fitting highly expressive
2.4 Gaussian Mixture Model (GMM)
GMMs are probability density functions where the marginal densities of x E Rd under f are f(x) = EveN P,N(z; μ4, E,), for a mixture of N Gaussians where N(x; p4, E,) is the marginal density of x for a single Gaussian, i E N. p: non-negative weighting coefficients with Σ., Pn = 1. GMMs are universal ap- proximators of densities, i.e., GMM with sufficient components can approximate any other density function to arbitrary precision (Alspach & Sorenson 1972..
GMMs in RL. GMMs were previously proposed as function approximators within RL frameworks. Agos- tini & Celaya| (2010) proposed using GMM as Q-function approximators. While such an approach was shown to naturally conform to distributional RL, its overall performance (asymptotic return and sample
In some tasks vrgmax, is not unique. In such cases, x* may refer to any optimal policy.
3

---

Published in Transactions on Machine Learning Research (12/2024)
efficiency) is outperformed by deop.neurallnnetookk-esss Q-approximators (Mnih et al. 2015 Another line of work (Nematollahi et al. 2022 suggested training GMM controller using samples obtained from a SAC agent. However, the GMM controller was not integrated into the SAC algorithm (e.g., as the actor or critic), but was trained using supervised learning independently from the SAC agent. Another work (Peng et al. 2020) proposed to run multiple RL agents simullaneously with a shared reply buffer while actively encouraging policy diversity between the agents. While cach agent's i policy can be represented with a single Gaussian, the poliey combination was not optimized as a single GMM model. Later, [Baram et al. (2021)
proposed the use of GMMs as policy approximators in maximum entropy (MaxEnt) frameworks. The pro- posed approach was implemented and evaluated within the Soft Actor-Critic (SAC) (Haarnoja et al. 2018) algorithm. The authors did not find clear evidence supporting the use of GMMss over a single Gaussian. They attribute this finding to two reasons: "First, benchmarked tasks are unimodal in nature, so a unimodal policy should do. Second, that the mixture policy collapses to a mean policy in the presence of equal mixing weights." In contrast their claims, our experimental results (Presentod in Soction 4 suggest that training GMM-based actor is advantages. In contrast. to Baram et al. (2021), Ren et al. (2021) reported posi- tive results (marginally outperforming SAC and PPO with a unimodal policy) when training parameterized mixing weights. In that work, the mixing weights are trained using a routing function as part of a Mixture- of of-Experts RL model (Jacobs et al. 1991: Peng et al.) 2019 Neumann et al. 2009 Akrour et al. 2021). Building on the partial successes of GMM-based actors in soft-actor algorithms,, we extend this approach (training GMM-baser actors) to a deterministic policy gradient algorithm.
2.5 Deep Deterministic Policy Gradient (DDPG)
DDPG (Lillicrap et al. 2015) is a benchmark continuous control RL algorithm that trains both a critic, as a differentiable Q-function approximator, Q S x A + R, and a deterministic differentiable actor, л : S - A.
DDPG Training. Given randomly sampled batch of transitions, cach of type < 8, s, >, (1) the critic is trained to minimize the L2 TD-error (Doya, 1995), i.e., minimize:(Q(s.a) (r (+())= and (2) the actor, n, is trained to maximize ((s π (s)) while assuming the Q parameters constant.
DDPG Exploration. An advantage of off-policy algorithms such as DDPG is that they can treat the problem of exploration independently from the trained policy. As such, DDPG performs random exploration by sampling a noise value from an Ornstein- Uhlenbeck process (Uhlenbeck & Ornstein 1930 to generate temporally correlated noise. The authors do not mention a particular reason for this choice (applying correlated noise). Moreover, the authors mention that other random noise processes can be used. In our experiments with DDPG, we found that comparable results are achieved when the noise is sampled using a (simpler) non-correlated Gaussian with mean zero and variance, Σ I.
3 Gaussian Mixture Deterministic Policy Gradient
We propose a variant of the DDPG approach termed Gaussian Mixture Deterministic Policy Gradient (GAMID-PG), or Gamid for short, where we (1) define the actor as a mixture of N Gaussians (instead of a single Gaussian in DDPG), (2) define the actor's policy through GMM sampling, and (3) include a GMM diversífication objective as part of the actor's gradient. The main motivation for the proposed al- gorithm is to enable deterministic policy gradients for general densities. This is justified by (1) the fact that specific types of density functions e.g., a single Gaussian, are sometimes incapable of converging to optimal policy (Tessler et al.) 2019). and (2) benefits reported for training such general-density actors in -oft-actor-critic optimization (with non-deterministie policy gradients) (Ren et al. 2021). Similar to DDPG, Gamid assumes MDPs with a continuous action space as the underlying environment. Our Gamid approach is detailed in Algorithm [ available in the form of pseudocode. The hyper(meta) (npte((mtta--arrmette parameters include (1) the number of Gaussians for the GMM policy, N, (2) a shared variance for all Gaussians as a scaled identity matrix, E, (3) a policy divergence temperature (possibly docaying), T, and, (4) the target network update rate, ax
4

---

Published in Transactions on Machine Learning Research (12/2024)
Algorithm 1: Gaussian Mixture Deterministic Policy Gradient (Gamid)
hyperparameters: (1) number of Gaussians, N; (2) shared variance, Σ x T; (3) policy divergence temperature, т (possibly decaying); (4) target update rate,
init
(1) policy parameters (for N Gaussians), 0 U 0[j]; (2) Q-function parameters, Φ; (3) empty replay buffer, D; (4) initial state, 8 ~ To
Set target parameters main parameters, Otarg 8, Vtarg +;
while not converyed do
Sample a Gaussian (out of N), + SampleGaussian (N, where P USeN is distribution over N outcomes (e.g,, uniform i.e., Vi, p: - 1/M)
Sample an action, < Clip (u(s; 0[ni)) + e, &Low; (High); where ~ _..,..... OJT, E): Execute a in the environment, observe next state s', reward and done signal d indicating whether is terminal:
Store (s, a, s', in the reply buffer, D;
Advance the environment, 5-;;; if d is TRUE, then reset the environment, lo; Randomly sample batch of transitions, B {(s, a, d)) from D; Compute Q targets:
10
srr,''d)
- -r 7(1 d) max [Q (s',
Update Q-lunction with one-step gradient descent using
Σ
for i E [O,..., .V 1] do Update mean for Gaussian i, i.., p(s; with one-step gradient ascent while considering other Gaussians (j # i) constant, using
11
Σ Qo(s, (o((()():)()))+++++ (Tk (o))
[(
where
(KL(i](NN log exp(-Ila(s;0F)))) (a(s; ([)))1
end
13
Update target parameters, Otarg altarg + a}0, and drang addarg + - a)d;
end
At this point, one might wonder "are GMMs still considered universal density approximators when using single shared variance?". The answer 'Yes!. Calcaterra (2008) showed that linear combinations of Gaussians with a single variance are, indcod, universal density approximator. However, considering a single variance commonly requires combining more Gaussian in order reach similar approximation accu- racy levels. Nonetheless, we observed (empirically) that optimal performance for Gamid is achieved when the number of Gaussians is fairly low (2-5). Morcover, setting constant variance allows for practical approximation of an entropy gradient term, as described later in Section 3.2
Given the shared variance, the GMM policy defined solely by N means. These means are approximated per Gaussian, E [o...., N 1), with differential function approximator, #4 S x 0 R, where 0] are the approximator's tunable parameters. f U, 0[]] is initialized randomly. Similarly, the Q-function approximator's parameters, d, also initialized randomly.

---

Published in Transactions on Machine Learning Research (12/2024)
3.1 GMM policy sampling
Sampling an action for a given state, 8, from the N Gaussians GMM is performed in two stages. First,
(Line 3), we sample one Gaussian, n, from a distribution defined by the GMM weighting coefficients, P. Next, (Line Dh we sample an action from n. P can be set in many ways, e.g., as a uniform distribution, Vi. p: = 1/N. We observed (empirically) that setting the coefficients using an e-greedy approach (Mnih) et al. 2015) is beneficial. That is, set p. = - c + e/N for arg max; Q(s, ([])) and Vj + i, P3 = c</N. This approach, however, introduces an extra hyperparameter, E, which requires meta tuning. We also experimented with existing approaches for tuning P to be proportional to the Q-values Ren et al. (2021). However, such an approach was not observed to perform better compared to the simpler e-greedy approach.
3.2
GMM training
Known techniques for training GMM approximators from samples are mostly applicable for supervised learning, i.c., when the target GMM distribution is known (Figucirodo et al. 1999) or can be sampled (Arenz) et al. 2020). Since, in our case, x² is not known a priori and is not available for sampling, we apply a cetterninistiic-olie---eaee
approach for training the GMM (as the actor in Gamid). The proposed gradient is defined with respect to one Gaussian (i E N 1]) and one state, s. Similar to the original DDPG algorithm, since we consider Σ, to be constant, the gradient defined only with respect to the mcan, m(3;0 ([)). It is based on two (possibly conflicting) optimization objectives.
1. Maximize the expected return for policy that follows Gaussian i, ie., max [Q(s; |ax 0[i)).
2. Maximize the cross-entropy between Gaussian i and the GMM excluding Gaussian i, i.c., maxogalf (Max u(s; O[i)), Σ) log (MMy, ((;
Optimization objective #1 follows the original DDPG actor training procedure. Optimization objective #2 is inspired by MaxEnt frameworks, which complement the standard maximum reward objective (Objective
#1) with an entropy maximization term (Aghasadegli & Breul 2011 Toussaint 2009; Rawlik et al. 2012
Haarnoja et al. 2018 Fox et al. 2016). These two optimization objectives are presented in Line 11 of
Algorithm I where policy is updated in the direction that increases the approximated Q value AND increases the Kullback-Leibler (KL) Divergence [Kullback] 1997) from GMM distribution which includes all the other Gaussians (other than i). At this point, the reader might wonder "Why maximize the KT- divergence and not the cross entropy as stated above?". There are two reasons for this choice: (1) in Gamid, KL-divergence and cross entropy are equivalent with respect to the resulting gradients; (2) it allows utilization of srate-of-thh-ar KL-divergence approximation techniques for GMMs (Hershey & Olsen 2007)
From cross-entropy to KL-divergence. Cross-entropy (H) is similar to KL-Divergence (DKL) with the addition of an entropy term. More specifically, the cross-entropy of a distribution f relative to distribution g is defined as H (f.g) DKL(fllg) + (f)).
Proposition 1. In Gamid, cross-entropy and KT--dinergence result in the same gradients with respect to single Gaussian, i, and the GMM excluding i, denoted GMMNNe
Proof. The Shannon entropy of a single Gaussian, f= Ni,, E), is H()) = 0.5 det Huber et al. 2008 (note that x refers to the mathomatical constant and not a policy here). Since Hff) is not a function of p, we get (J) = 0. As a result, for any other densily y (e.g., a GMM), we have V (f.9)) = V DKL(f Il g). Note that (f.l is not needed for Gamid because it assumes a constant Σ.
Approximating DKL- For two distributions f and g, DKL(fllg) returns the expected log probability ratio between the two distributions, g(z)
f(x) when x is sampled from f. Formally, (fl(f) == Ezz log g(#)
f(x)
So fx) log f(x) dr. For k-dimensional Gaussians f and the KL-divergence has a closed form expression, s(z)
1 [Σ E, (flg) = 2
+ Tr 'E - k + (u4 (4g) " (ng - m95
log
(1)
6

---

Published in Transactions on Machine Learning Research (12/2024)
However, closed-form expression is known for two GMMs. As result, might wonder "how can compute DEL. in Gamid (Line 11
Assume that f and are GMMs with the following marginal densities: EpaN par Ca)
f(x) [8)=-Doek
s(2) PaN( Eb)
A commonly used (closed form) approximation Dxl(fll)) for such cases is the Variational Approzima- tion (Hershey & Olsen 2007) which is defined as follows
Dear (fllg) = Σ Pa log Ev Par Par exp(-Dri,
(2)
E, P6P6eepp exp(- DKL. (fall9l9g)))
In Gamid, f represents a single Gaussian (out of N). As such, we get simplified Dear expression, specifically: Dvar (fllg) = log Σ pr exp(-Dxit [xxlxp(-Dx(((lllo)))
(3)
In Gamid all Gaussians comprising the GMM have a shared E. When sctting E = I, i.c., the identity matrix, Equation is simplified to ||l poll?, and consequently, Equation can be further simplified to
Dvar (fllg) - log Σ pb exp(- 11py - mbll²)
(4)
As we seek to diversify the GMM policy overall composing Gaussians, we consider uniform weights (vb, Ps = 1/N) when setting Dear in Line 11 of Gamid. As this is a constant scalar value, is simply omitted and can be viewed as component of the temperature scalar (r).
Finally, it is important to note, that Gamid is compatible with various DDPG variants. As such, when secking state-of-the-art. performance, one should implement it within the most effective DDPG variant. Indeed, in our experimental section, we report results for a Gamid implementation extending the Twin Delayed DDPG (TD3) variant (Fujimoto et al. 2018).
3.3 Convergence of Gamid
When setting the number of Gaussian to one (N 1), Gamid is effectively the same as DDPG and shares similar convergence guarantees. However, when considering a mixture of N > Gaussians, the objective function per Gaussian differs from that of DDPG following the MaxEnt sub-objective. As such, DDPG and Gamid might converge on a different local optimumm. This issue can easily be addressed by decaying the policy divergence temperature (r). Nonetheless, matching the convergence guarantees of DDPG provides minor value such guarantees provided for the general case. Specifically, Lillicrap et al. (2015) state, "As with Q learning, introducing non-linear function approximators means that convergence is longer guaranteed. However, such approximators appear essential in order to learn and generalize on large state spaces".
4 Experiments
The goals of the reported experimental study are fourfold: (G1) to illustrate the benefits of GMM-basod actor over a single Gaussian policy in the context of Gamid when allempting to capture multiple modes of optimality, (G2) to compare deterministic policy gradients versus stochastic policy gradients for optimizing GMM-basod actor, (G3) to compare Gamid against contemporary RL algorithms for continuous control tasks, and (G4) to demonstrate the performance sensitivity for Gamid's hyperparameters. Full descriptions for all the domains (state and action space, reward function) are provided in Appendix A
7

---

Published in Transactions on Machine Learning Research (12/2024)
SAC
SAC
SAC
SAC
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
TD3/DDPG
TD3/DDPG
TD3/DDPG
TD3/DDPG
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
-1
Gamid
Gamid
Gamid
Gamid
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.0
0.0
0.0
SACM
SACM
SACM
SACM
1.0
1.0
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
-1
(a) Step## 0
(b) Step# 2000
(c) Step## 4000
(d) Step## 9000
Figure 1: Policy distribution at different training steps in the 1-D continuous bandit problem with the reward function shown green. x-axis represents the action space and y-axis the scaled PDF (probability density function). Unlike single Gaussian policy-based methods such as SAC and TD3/DDPG, and GMM policy-based SACM (shaded in blue), Gamid (shaded in red) converges on the optimal action.
4.1 (G1) Capturing Multiple Modes of Optimality
To illustrate the benefits of Gamid over a single Gaussian policy, as commonly used in SAC and DDPG/TTD in terms of escaping the local optima and finding the optimal solution, we consider toy continuous bandit problem with 1-D action space. We adopt this problem from Huang et al. (2023). It has multi- modal (2 modes) deterministic reward function defined over the action space in bandit setting. The exact experimental settings follow those presented in Huang et al. (023 For completeness, these settings are provided in Appendix A
Figure D showcases the performance of Gamid against a single Gaussian policy with a parameterized mean (as in both SAC and DDPG) and standard deviation (as SAC). The Gaussians for both SAC and DDPG are initialized with means around (Figure Iap. We observe that both the single Gaussian actor variants move toward the suboptimal action (Figures lb and Ich and converge on it by the end of the training phase (Figure ld Similarly, Gamid finds the suboptimal action during the initial learning stages but, in contrast to the single Gaussian variants, it gradually spreads out and converges on the optimal action by the end of the training phase. These results suggest that a GMM-based actor can be helpful in escaping local (sub)-optimum as opposed to a single Gaussian actor. However, when examining the performance SACM (with 5 Gaussians), we notice that also converges on the suboptimal action, despite training a GMM actor. While SACM uses a GMM- actor, similar Gamid, trains it using stochastic gradients, in contrast to Gamid. These results suggest that when training GMM actor, deterministic gradients be more
8

---

Published in Transactions on Machine Learning Research (12/2024)
effective in escaping local optimum when compared to stochastic gradients. The inability of a stochastic gradient approach (SAC, SACM) to capture the optimal mode is identified in prior work [2] from which we adopted the 1-D continuous bandit environment. They provide the following explanation: "The Gaussian policy, initialized at O with a large standard deviation, can cover the whole solution space. However, the gradient with respect to j is positive, which means the action probability density will be pushed towards the right, as the expected return on the right side larger than the left side, although the left side contains higher extreme value. As result, the policy will move right and get stuck at the local optimum with a low chance of jumping out". We agree with the intuition provided in this explanation which highlights the limitations of stochastic gradient-based actor optimization in escaping local optimum.
4.1.1 GMM Actor Divergence
We analyze the diversity between the Gaussians in Gamid captured by the approximated K L-vivergeece,, Specifically, utilize Equation 4 for approximating the of K KL-divergence
all Gaussians with respect to the GMM. That is, we define DDvrr = Σi= D \ i)) (see delinition from Line 11 Algorithm D. Figure 2 contains plots of ED var at each training step on three representative MuJoCo tasks (Todorov et al.) 2012), 'Hopper-v3', 'HallCheetab-v3, and 'Walker2d-v3' with increasing order of action dimensionalityy. We provide results for two extreme т values (0 and 10) to highlight their effect DDvar- We expect DDerr to be relatively higher for higher values of 7 since higher value of maximizes Drar (Line 11 in Algorithm D. The DD curves for all the tasks in Figure 2
follow the expected trend as the curve corresponding to т = 10 consistently higher than the one corresponding to T 0. For comparison, we also include curves for the tuned 5 values (T = opt.) as listed in Table which we observe to be between the curves corresponding to
and 10. These curves are closer T= since the tuned values are closer to zero at (0.01 0.3). We further visualize the evolution of the GMM Gaussian component means in Gamid during different steps of the training stage. See Appendix D.1 for full details and results.
18
30
22
16
20
25
14
12
t opt.
20
16
t
15
14
12
T 10
10
8-
5
2
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6 Step
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Step
Step
ie
(a) H3pper-v3
(b)
HalfCheetah-v3
(c) Walker2d-v3
Figure 2: Sum of the Dear over N Gaussian components (EDoar) at each training step for 3 different settings of on 3 representative MuJoCo tasks. Curves have been smoothed (100,000 steps moving window) for visual clarity.
4.2 (G2) Deterministic vs Stochastic Policy Gradients
To analyze when deterministic policy gradients can be more effective than stochastic policy we consider 2 domains. The first domain is 2-D maze navigation grid-world environment, denoted 'MazeGrid', with sparse reward function adopted from Huang et all (2023). The agent starts at the center of the grid and its objective is to reach the optimal goal. The other domain is the "HalfChettah-v3' environment from MuJoCo Todorov et al. 2012). Full details regarding the domains are provided in Appendix
we observe that SACM outperforms Gamid in terms sample efficiency by more than 1 standard deviation in 'MazeGrid'. Prior work has shown single Caussian-based actors using Maxent-basod policy optimization to be more effective in challenging exploration (sparse reward) tasks as compared
In Figure/ 3
deterministic policies (Dawood et al. 2023 Singh et al. 2019) which might explain these trends.
9

---

Published in Transactions on Machine Learning Research (12/2024)
Gamid
SACM
SAC
1.9
+2
1.4
1.1
0.8
+0.2
0.6
0.3
0.0
-0.3
0.0
0.2
1.5 2.0
Steps (1e4)
Figure 3: Training curves on MazeGrid. Solid curves present the average over five runs while the shaded region represents two standard deviations.
On the other hand, in Figure 4b we observe that Gamid outperforms SACM in terms of sample efficiency in 'HalfCheetab-v3: Compared stochastic policies, deterministic policies have been shown be more effective in tasks requiring precise control (as is common in robotics domains) (Montenegro et al. 2024). We speculate that Gamid builds these properties and uses the diversified actors to speed up exploration during the initial stages of learning. This suggests that a GMM optimized using deterministic policy gradients can be effective in dense reward robotics tasks when compared to using stochastic gradients.
4.3 (G3) Comparative Evaluation Benchmark Tasks
For the comparative evaluation, we compare Gamid against common RL algorithms for continuous control tasks using benchmark MuJoCo (Todorov et al.) 2012) and Fetch (Plappert et al. 2018) domains. The MuJoCo tasks utilize dense reward functions to learn locomotion tasks. The Fetch domains consist of a 7-DoF robotic arm fitted with a gripper relying on sparse reward functions to learn to solve goal-reaching tasks.
We consider the following baseline RL algorithms for continuous control tasks:
1. Soft Actor-Critic (SAC) (Harrnjj et al. 2018 (Off-Policy) MaxEnt actor-critic algorithm where the policy is trained to maximize weighted combination of expected return and policy entropy.
Soft Actor-Critic Mixture (SACM) (Baram et al. 2021) (Off-Policy) soft-actor optimization approach that uses GMM as the policy approximator. This approach was reported to not improve performance over single Gaussian policy (SAC).
Probabilistic Mixture-of-Experte SAC (PMOE) (Ren et al. 2021 (Off-Policy) - a soft-actor optimization approach that uses a mixture critics. This approach was shown to outperform other MOE approaches, specifically: MOE with gating operation (Jacobs al. 1991). Double Option Actor-Critic (DAC) option framework (Zhang & Whiteson, 2019), the Multiplicative Compositional Policies (MCP) Peng al., 2019). and PMOE with Chumbel-Softmas (Maddison et al. 2016
Proximal-Poliee--ppiiiiaaaa (PPO) Schulman et al. 2017) (On-Policy) - trust region
policy optimization variant (Schulman et al. 2015) using clipped gradients to restrict the policy change between policy updates.
5. Twin Delayed DDPG (TD3) Fujimoto et al.
(Off-Poicy) extension of the original DDPG algorithm which introduces three enhancements, namely, (1) Clipped Double-Q Learning, (2) Delayed Policy Updates, and (3) Target Policy Smoothing.
2018
For Baselines 1, 4, and 5 we used the implementations provided in Stable-haseliness (Raffin et al. 2021 For Baseline 2, since there was no official implementation provided by the authors, we implemented it following the pseudocode provided in the paper. For Baseline used the implementation provided
10

---

Published in Transactions on Machine Learning Research (12/2024)
Gamid
PMOE-SAC
SACM
SAC
PPO
350
3500
300
Average Return
10000
Average Return
3000
250
8000
2500
Average
200
2000
150
1500
4000
100
1000
2000
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
1.0
0.0
0.2
0.4
0.8
1.0
Steps (1e6)
Steps (1e6)
Steps (1e6)
(a) Swimmer-v's
(b) HalfCheetah--?
(c) Hopper-v3
6000
5000
7000
5000
6000
Return
Average Return
Average Return
4000
5000
3000
3000
4000
2000
2000
3000
1000
2000
1000
1000
0
Steps (1e6)
Steps (1e6)
Steps (1e6)
(d) Ant-v3
(e)
Walker2d-v3
(f) Humanoid-v3
1.0
Average Return
0.8
Average Return
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.6 Steps (1e6)
0.4
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
(1e6)
Steps (1e6)
(g) RetchPush--1
(h) FetchSlide-v1
(i) FetchPiikAnddPaocxx
Figure 4: Training curves on continuous control benchmarks. Solid curves present the mean over five runs while the shaded region represents the tolerance interval with a = 0.05 and 3 = 0.7 Pattteronn et al., 2023 Curves have been smoothed (100 steps moving window) for visual clarity. Gamid (Blue curve) consistently performs on par or better compared to existing baseline methods.
by the authors. The hyperparameter values for each algorithm were set the recommended values. For completeness, these values are provided in Appendix B The codebase for these experiments is available at https://github. Sttn/ii-Starr-aagaaaiii
None of the baselines results in meaningful learning in the Fetch tasks due to sparse rewards. Consequently, we follow prior work (Tbar et al. 2021 Raffin et al. 2021 Bajaj et al. 2023) and combine them with Hindsight Experience Replay (HER) (Andrychowicz et al. 2017). Note that PPO and PMOE are not straightforward to combine with HER and are thus omitted for these
tasks.
Post-training performance. Table I presents the post-training performance of Gamid and the baseline algorithms at the end of the training stage (2M training steps for 'Humanoid-v3' and 1M for the rest) on the MuJoCo and Fetch tasks. Results reported in the table and trends in Figure 5 suggest that Gamid performs on par with the baseline methods on tasks with lower degrees of freedom, as in 'Swimmer-v3" (Figure da), and 'Hopper-v3' (Figure 4cb. outperforms them on tasks with higher degrees of freedom We hypothesize these trends stem from increased modes of optimality present high-
(Figures
2No prior work combined PPO PMOE with HER best of knowledge.
11

---

Published in Transactions on Machine Learning Research (12/2024)
Table 1: Mean performance and the 1-standard deviation on continuous control benchmarks. The best- performing RL algorithms have been highlighted in bold.
TD3
SAC
SACM
PMOE
PPO
Gamid
MuJoCo (u3)
Swimmer
263 =
139
318 +
43
336
1.4
50 +
3.82
318
+
22.1
344 + 4.1
HalfCheetah
9,578
648
10,427 = 206
9,599
357
7,934 = 1,072
5,251 +
907
11,063
+ 300
Hopper
3,392
125
3,027 1 664
2,848
677
2900
1 461
918
1 156
3,301
1 209
Ant
4,264
1,671
3,613 + 1,365
4,159
512
1,874 1 491
2,204
+ 781
4,998
1 828
Walker2d
4,153
429
4237 + 429
4187
245
4,046
+ 530
3,023
1,543
4,835
1 187
Humanoid
5,727
5,949 +
5,498
5,039
+ 394
812 +
136
6,141
t
Fetch (vI)
Push
0.34
0.43
0.99
1 0.02
0.99
0.02
0.95 + 0.05
Slide
0.13
0.18
0.76
+ 0.26
0.75
0.27
0.45 + 0.31
PickAndPlace
0.03
0.04
0.99 + 0.01
0.99 0.01
0.66 + 0.24
dimensional control tasks and Gamid's superior ability to capture such complex behavior patterns. Results in all the Fetch tasks (Figures 4g 4h and 4i show that Gamid consistently outperforms TD3 in terms of post-training performance. However, SAC and SACM outperform both Gamid and TD3. These results align with Figure 3 where we observe that stochastic gradient-based approaches are relatively more effective in sparse reward tasks than deterministic gradient-based approaches.
Sample efficiency. We observe that Gamid has better sample efficiency than the baselines in "HalsCheetah-v3' (Figure 4b) and 'Walker2d-v3' (Figure 4ee. In the rest of the domains, we do not see any specific trend for Gamid as compared to the baselines. In all Fetch domains, Gamid has better sample efficiency than TD3 but has worse sample efficiency than SAC and SACM.
Performance consistency. In terms of the post-training performance, we observe that Gamid consistently performs at least as well as TD3 on both dense-reward (MuJoCo) and sparse-reward (Fetch) tasks.
terms of the average performance, Gamid out-erforrs TD3 on 8/9 lasks (see Table 1 We also report an independent two-sample f-test (Cressie & Whitford 1986) with the p-value significance level set 0.05 comparing Gamid and TD3. The results indicate that the advantage of Gamid over TD3 is statistically significant in 4/9 tasks ("HallCheetah-vv3, 'Walker2d-v3' 'FetchPush-v11,, and TeeechPickAnnPPaaceVV)) For the rest of the domains, the performance difference is not statistically significant. These results suggest that, while training with deterministic gradients, utilizing GMM-based actor (as in Gamid) is consistently advantageous with respect to returns when compared to utilizing a Linle-Gauusie actor (as in TD3).
4.4 (G4) Sensitivity Analysis of Gamid
We examine the sensitivity of Gamid's performance with respect to its hyperparameters. The reported results exclude the hyperparameters that are shared with the original DDPG algorithm since ablation study for those was presented in previous publications (Lillicrpp et al. 2015 Fujimoto al, 2018). The results are reported for a single domain ("Walkerdd") where Gamid performs significantly better than ex- isting approaches. Nonetheless, for completeness, ablation results for the other domains are reported in Appendix D
Number of Gaussians, N. We start by examining the impact of varying the number of Gaussians used by Gamid (the N hyperparameterr.. Figure 5a presents learning curves for five N values: (original TD3), 2.....5. The other hyperparameters were kept constant with values as reported in the comparative study, We observe that increasing the number of Gaussians up post-training performance when compared to a single Gaussian. N provides the best exploration balance while the marginal benefit. from adding more Gaussians diminishes and stagnates at about four. This is
improves both the sample efficiency and
12

---

Published in Transactions on Machine Learning Research (12/2024)
5000
5000
5000
Average Return
Average Retur
Average Return
4000
4000
4000
3000
N=5
3000
3000
N-4
T=0,01
greedy (5=0)
2000
N-3
2000
T=0.1
2000
low (e=1/10M)
1000
N=2
1000
T=0.5
1000
med. e (5=1/2N)
N=1
T=0.1 - 0.01
high E (6=1/N)
0
0.1 0.2 0.3
0.4 0.5
0.6
0.7 0.8 0.9
0.1 0.2 0.3 0.4 0.5
0.6
0.7 0.8 0.9
0.1
0.2 0.3 0.4 0.5 0.6
0.7 0.8
Steps (1e6)
Steps (1e6)
Steps (1e6)
(a) Number of Gaussians (N)
(b) Policy divergence temperature (7)
(c) GMM weights
Figure 5: Ablation curves for the 'Walker2D' domain. These results suggest that (1) adding more Gaussians to the GMM-based actor is beneficial, (2) setting a docaying policy divergence paramcter beneficial, and (3) using an e-greedy approach with low (yet not zero) e value beneficial.
reasonable result as, with sufficient Gaussians, the GMM becomes expressive enough to represent any target policy, so the addition more Gaussians is not helpful.
Policy divergence temperature, т. Next, we examine the impact of varying the policy divergence temperature hyperparameter (T). Figure 5b presents learning curves for three static 7 values (0.01, 0.1, 0.5) and a decaying т version (linear decay from 0.1 to 0.01 in 30% of the total training steps). We observe that decaying value performs best. This outcome (favoring entropy lemperature decay) in line with similar results reported by Schulman
(2017) and HHarnoja et al. (2018).
GMM weights, 6. Finally, we examine the impact of varying the GMM weights setting from Line 3 (denoted P). Figure 5c presents learning curves for four P e-greedy assignments: e e (greedy), e (10N) (low epsilon). c= 1/ (2N) (high epsilon), = 1/N (uniform). The results suggest that low (yet not zero) epsilon performs best. This result in line with e-greedy trends reported in prior work (Mnih al. 2015
5
Discussion
While prior works have compared deterministic and stochastic policy gradicnts they did so for single Gaussian actors. These studies found that stochastic gradients outperform deterministic gradients on common bench- mark domains (Haarnoja et al. 2018). Our experimental study suggests that this trend (stochastic gradients outperform deterministic gradients) does not necessarily apply GMM-based actors. We observe that in 5 out of 6 denso-reward MuJoCo domains, optimizing such actors with deterministic gradients (Gamid) performed better compared to stochastic gradients (SACM). Gamid also leads to improvements over (single Gaussian) SAC in 3/6 MuJoCo tasks while consistently performing similar or better than TD3 in all the MuJoCo tasks. In the 3 sparse-reward Fetch tasks, we observe that stochastic gradient-based actors (SAC, SACM) outperform deterministic gradient- gradient-based actors (GMM). In stochastic gradient- based algorithms, the entropy of the policy, derived from a learned standard deviation as opposed to fixed standard deviation deterministic gradient-based algorithms, generally high during the initial learning stages in sparse-reward tasks. During this stage, the agent receives close to non-zero rewards that result in high policy stan- dard deviation and hence close-to-random exploration which is key in such scenarios. We speculate that such a property makes SAC and SACM more effective as compared to Gamid and TD3 in the Fetch tasks. Nonetheless, Gamid outperforms TD3 in all the Fetch (asks, suggesting that using stochastic gradients to train a GMM actor can be more effective than doing the same over a single Gaussian actor. We observe that GMM-based actors optimized using deterministic gradients as presented in Gamid do not adversely affect the post-training performance of TD3 in the MuJoCo tasks. These results stand in contrast with findings reported in Baram al. (2021) that did not find any significant advantage for using GMM-based actors.
13

---

Published in Transactions on Machine Learning Research (12/2024)
It should be noted that Gamid adds additional hyperparameters top of DDPG/TD3 which may raise concerns about its practicality in real-world settings, given the need for required hyperparameter tuning. We emphasize that Gamid, in its current form, is an effort to spark interest within the scientific community on the effectivencss of deterministic policy gradients when coupled with GMMs. Future work will focus on developing practical variants of Gamid by lixing or automating hyperparameter values.
6 Summary
In this paper, we presented a comparison between stochastic and deterministic policy gradients to optimize Gaussian mixture model (GMM)-based policies. We introduced novel approach, denoted Gamid, for training GMM using deterministic gradients. Similar to the maximization of entropy in stochastic actors, Gamid incorporates a diversification objective that encourages the actors to spread out aiding the exploration capabilities of the policy. Empirical studies on benchmark MuJuCo tasks show that, in terms of sample efficiency and post-training performance, Gamid improves over the single Gaussian stochastic variant. (SAC) in 3/6 domains. It consistently performs on par or better than the single Gaussian deterministic variant (TD3) in all the MuJoCo and Fetch tasks. These results suggest that deterministie gradient approaches can be more effective for training GMM actors compared to stochastic gradient approaches for dense-reward control tasks. Empirical results on sparse-reward Fetch tasks show that stochastic gradient approaches are more effective than Gamid. Nevertheless, Gamid improves over the single Gaussian deterministic variant in 3/3 Fetch domains. Consequently, we hope this work will seed research on training soft. actors using deterministic gradient approaches.
References
Hervé Abdi and Lynne J Williams. Principal component analysis. Wiley interdisciplinory reviews: compu- tational statistics, 2(4):433 459, 2010.
Navid Aghasadeghi and Timothy Bretl. Maximum entropy inverse reinforcement learning in continuous state spaces with path integrals. In 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1561 1566. IEEE, 2011.
Alejandro Agostini and Enric Celaya. Reinforcement learning with a gaussian mixture model. In The 2010 International Joint Conference on Neural Networks (IJCNN) pp. 1-8. IEEE, 2010.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next- generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD interna- tional conference on knowledge discovery & data mining, pp. 2623 2631, 2019.
Riad Akrour, Davide Tatco, and Jan Peters. Continuous action reinforcement lcarning from a mixture of interpretable experts. IEEE Transactions on Pallern Analysis and Machine Intellignce, 44(10):6795- 6806, 2021.
D. Alspach and H. Sorenson. Nonlincar bayesian estimation using gaussian sum approximations. Transactions on Aulomatic Control, 17(4):439 448, 1972. doi: 10.1109 1AA.1099TAC..172.100000.
JEEE
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advences in neural information processing systems, 30, 2017.
Oleg Arenz, Mingjun Zhong, and Gerhard Neurnann. Trust-region variational inference with gaussian mixture models. The Journal of Machine Learning Research, 21(1):6534 6593, 2020.
James Ault and Guni Sharon. Reinforcement learning benchmarks for traffic signal control. In Proceedings of the 35th Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks, December 2021.
14

---

Published in Transactions on Machine Learning Research (12/2024)
Vaibhav Bajaj, Guni Sharon, and Peter Stone. Task phasing: automated curriculum learning from demon-
strations. pp. 542-550 2023. doi: 10.1609 20.1609/icaps.voe vocapp.vv3i..27771 URL
18811. 27235
Nir Baram, Guy Tennenholtz, and Shie Mannor. Maximum entropy reinforcement learning with mixture policies. arXic preprint arXiv:2103.101: 2021.
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint ar arx:::8804008111 2018.
Craig Calcaterra. Linear combinations of gaussians with a single variance are dense in 12. In Proceedings of the World Congress on Engineering, volume 2, 2008.
Po-Wei Chou, Daniel Maturana, and Sebastian Scherer. Improving stochastic policy gradients in continuous control with deep reinforcement learning using the beta distribution. In International conference on machine learning, pp. 834-843. PMLR, 2017.
Antonio Coronato, Muddasar Naeem, Giuseppe De Pietro, and Giovanni Paragliola. Reinforcement learning for intelligent healthcare applications: A survey. Artificial Intelligence in Medicine, 109:101964, 2020.
NAC Cressie and HJ Whitford. How to use the two sample t-test. Biometrieal Journal, 28(())131 148, 1986.
Murad Dawood, Nils Dengler, Jorge de Heuvel, and Maren Bennewitz. Handling sparse rewards reinforce- ment learning using model predictive control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 879-885. IEEE, 2023.
Sheelabhadra Dey, Sumedh Pendurkar, Guni Sharon, and Josiah P. Hanna. A joint imitaation-reinfooremmmntt learning framework for reduced baseline regret. In 2021 IEEE/ RRSJ International Conference on Intelligent Robots and Systems (TROS), pp. 3485-30 3491. TEEE Pross, 2021. doi: 10.1109/TROSSI
URL https: //doi. org 10. 1109 IROS51168. .936299
Shoelabhadra Dey, James Ault, and Guni Sharon. Continual optimistic initialization for value-based re- inforcement learning. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiogent Systems, pp. 453-462 2024.
Kenji Doya. Temporal difference learning in continuous time and space. Advances in neural information processing systems, 8, 1995.
fnigo Elguea-Aguinaco, Antonio Serrano-Muñoz, Dimitrios Chrysostomou, Thai Inciarte-Hidalgo, Simon Bøgh, and Nestor Arana--Arexllleiii A review on reinforcement learning for contact- contcct-rich robotic manip- ulation tasks. Robotics and Computer-Integruteed Manufacturing, 81:102517, 2023.
Málrio AT Figueiredo, Jose MN Leitao, and Anil K Jain. On fitting mixture models. In Energy Minimization Methods in Compuler Vision and Paltern Recognition: Second International Workshop, EMMCVPR '99 York, UK, July 26 26-29, 1999 Proceedings 2, pp. 54-69. Springer, 1999.
Roy Fox, Ari Pakman, and Naftali Tishby. Taming the noise in reinforcement learning via soft updates.
Conference on Uncertainty in Artificial Intelligence (UAI), 2016.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
John R. Hershey and Peder A. Olsen. Approximating the kullback leibler divergence between gaussian mixture models. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP '07, volume pp. TV-317- 11---V--220 320, 2007. doi: 10.109//CASSPP::0:::600
15

---

Published in Transactions on Machine Learning Research (12/2024)
Zhiao Huang, Litian Liang, Zhan Ling, Xuanlin Li, Chuang Gan, and Hao Su. Reparameterized policy learning for multimodal trajectory optimization. In International Conference on Machine Learning, pp- 13957 13975. PMLR, 2023.
Marco F Huber, Tim Bailey, Hugh Durrant-Whyte, Whyte, and Uwe D Hanebeck. On entropy approximation for gaussian mixture random vectors. In 2008 IEEE International Conference Multisensor Fusion and Integration for Intelligenl Systems, pp. 181-188 188. IEEE, 2008.
Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine. How to train your robot with deep reinforcement. learning: lessons we have learned. The International Journal of Robolics Research, 40(4-5):698 7218 2021.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The Interna- tional Journal of Robotics Research, 32(11):12288 1274, 2013.
Roger Koenker and Kevin F Hallock. Quantile regression. Journal of economic perspectives, 15(4):143-156, 2001.
Solomon Kullback. Information theory and statistics. Courier Corporation, 1997.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Hoess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.0200 2015.
Yudong Luo, Guiliang Liu, Haonan Duan, Oliver Schulte, and Pascal Poupart. Distributional reinforcement learning with monotonic splines. In International Conference on Learning Representations, 2021.
Chris J Maddison, Andriy Muih, and Yee Whye Teh. The conerete distribution: A continuous relaxation of discrete random variables. arXiv preprint 1r1:0:1611.00712, 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529 533, 2015.
Alessandro Montenegro, Marco Mussi, Alberto Maria Metelli, and Matteo Papini. Learning optimal deter- ministic policies with stochastic policy gradients. arXie preprint arXiv:2405.02233 2024.
Iman Nematollahi, Erick Rosete-Beas, Adrian Röfer, Tim Welschehold, Abhinav Valada, and Wolfram Bur- gard. Robot skill adaptation via soft actor-critic gaussian mixture models. In 2022 International Confer-
ence on Robotics and Automation (ICRA), pp. 8651-8657. IEEE, 2022.
Gerhard Neumann, Wolfgang Maass, and Jan Peters. Learning complex motions by sequencing simpler motion templates. In Proceedings of the 26th Anmual International Conference on Machine Learning, pp. 753-760, 2009.
Georg Ostrovski, Will Dabney, and Rémi Munos. Autoregressive quantile networks for generative modeling. In International Conference on Machine Learning, pp. 3936-3945. PMLR, 2018.
Andrew Patterson, Samuel Neumann, Martha White, and Adam White. Empirical design in reinforcement lcarning. arXiv preprint arXiv:2304.01315, 2023.
Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. MCP: Learning composable hierarchical control with multiplicative compositional policies. Advances in Newral Information Processing Systems, 32, 2019.
Zhenghao Peng, Hao Sun, and Bolei Zhou. Non-local policy optimization via riversity collabo- rative exploration. ar Xiu preprint ar t=Xiv:2006.0774 2020.
16

---

Published in Transactions on Machine Learning Research (12/2024)
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojeiech Zaremba. Multi- goal reinforcement learning: Challenging robotics environments and request for research, 2018.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stablo-basclines3: Reliable reinforcement learning implementations. The Journal of Machine Learning Research, 22(1):12346 12355, 2021.
Konrad Rawlik, Mare Toussaint, and Sethu Vijayakumar. On stochastic optimal control and reinforcement learning by approximate inference. Proceedings of Robolics: Science and Systems VIII, 2012.
Jie Ren, Yewen Li, Zihan Ding, Wei Pan, and Hao Dong. Probabilistic mixture-of-experls for efficient deep reinforcement learning, 2021.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint Xrv: 707777700000 707.06347, 2017.
Guni Sharon. Alleviating road traffic congestion with artificial intelligence. In Zhi-Hua Zhou (od.), Proceed- ings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 4965 4969. In- ternational Joint Conferences on Artificial Intelligence Organization, 8 2021. doi: 10.24963 10.24963/jjcai (jccai2021/7000 URL https:tt/ot....oooo org/ 2002222932/i// ijcai. 2021/704 Early Career.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deter- ministic policy gradient algorithms. In International conference on machine learning, pp. 387 395, Pmlr, 2014.
Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic rein- forcement learning without reward engineering. arXiv preprint Xiv: 1904009407784 2019.
Rahul Singh, Keuntack Lee, and Yongxin Chen. bamele-based distributional policy gradient. In Learning for Dynamics and Control Conference, pp. 676-688. PMIR, 2022.
Jun Song and Chaoyue Zhao. Optimistic distributionally robust policy optimization. ar preprinl arXiv:2006.0768 07815, 2020.
Chen Tessler, Guy Tennenholtz, and Shie Mannor. Distributional policy optimization: An alternative ap- proach for continuous control. Advances in Neural Information Processing Systems, 32, 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012. doi: 10.1109/IR00 100.0///R0...02...62.. 2012.6386109.
Mare Toussaint. Robot trajectory optimization using approximate inference. In Proceedings of the 26th annuol international conference machine learning, pp.
2009.
George E Uhlenbeck and Leonard S Ornstein. On the theory the brownian motion. Physical review, 36 (5): 1930.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, ((3-4):229-4 226-2 1992.
Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, (22(789))2223222)) 2022.
Mingzhang Yin and Mingyuan Zhou. Semi-implicit variational inference. In International Conference on PMI.R 2018.
Machine Learning, pp.
17

---

Published in Transactions on Machine Learning Research (12/2024)
Yuguang Yue, Zhendong Wang. and Mingyuan Zhou. Implicit distributional reinforcement learning. Advances in Neural Information Processing Systems, 33:7135 7147, 2020.
Shangtong Zhang and Shimon Whiteson. Dac: The double actor-critic architecture for learning options. Advances in Neural Information Processing Systems, 32, 2019.
S Kevin Zhou, Hoang Ngan Le, Khoa Luu, Hien V Nguyen, and Nicholas Ayache. Deep reinforcement learning in medical imaging: literature review. Medical image analysis, 73:102193, 2021.
A Domain descriptions
1-D Continuous Bandit: A bandit problem with a continuous 1-dimensional action space as shown in Figure It has a 1-D action space and a non-convex reward landscape with modes optimality; a global and a local.
bandit reward
1.0
0.5
0.0
0
Figure 6: 1-D Continuous Bandit environment
MazeGrid: A 2D maze navigation lask as shown in Figure The maze consists of 5x5 grids. Each of them is connected with neighbors with a narrow passage. The agent starts in the center grid and can move in four directions. The action space is its position change in two directions (Ax, Ay).
+0.2
Figure MazeGrid environment
Mu.JoCo: A comprehensive domain description for all the MuJoCo (v3) tasks along with installation and exact instructions can be found through: https: http//yymasiiu farama. Table 2 provides a summary of the state (observation) and action spaces dimensionalltyy along with the guiding reward function definition per task.
B Hyperparameters
We adaptod the 1-D continuous bandit. environment introduced in prior work (Huang et al. 2023) and used the recommended hyperparameters for REINFORCE for setting up lightweight version of SAC (no critic and replay buffer). We also similar lightweight. version of DDPG/TD3 (no critic and
18

---

Swim. Published in Transactions on Machine Learning Research (12/2024)
Table 2: (a) Action space ofgrees--offreedonn (Action dim.), (b) Observation space dimensionality (Obs. dim.), (c) reward function (Reward), per domain.
Domain
Action dim.
Obs. dim.
Reward
furward_reward clrl clrl_cost
HalfCh.
17
[orward reward ctrl cost
Hopper.
healthy reward + forward Burward_reward cirl_cost
Ant.
healthy_rewardd reward forward reward ctrl cost contact _ost
Walker.
17
healthy_rewaar + forward_ rewarrd ctrl _crl_ccceee _ost
Human.
healthy reward forward Corward_reward_ ctrl cost contact cost moalthy_rewardd
Push.
25
sparse; block final target position reached = 0/not reached = -1
Slide.
sparse; puck final target position reached 0/not reached -1
Pick.
25
sparse; block final target position reached = 0/not reached = -1
replay buffer). For the noise in DDPG, we used a Gaussian distribution with a mean of and a fixed standard deviation of 0.1, the recommended setting for MuJoCo tasks RL BaselinesS Zoo library (https: //github.coomDL.R http://D..-R//t1tbaaaeeeeeeeeeee
Tables 8 present the default hyperparameters values used in our experiments the MuJoCo and Fetch tasks. To allow for easy reproduction of the results, the presented hyperparam names follow the variable names in the Stable-baselines3 (v1.6.2) (Raffin et al. 2021) open-source codcbase. We used the tuned hyperparameter values for SAC, TD3, and PPO that have been provided in RL Baselines3 Zoo which is built on top of Stable-baseee baselnes3,, For SACM and PMOE we used hyperparameters recommended in (Baram) ct al. ) and (Ren et al., 2021) respectively for the MuJoCo tasks. For the Fetch tasks, we tunod the hyperparameters of SACM using Optuna (Akiba al. 2019).
Table Shared TD3 hyperparameters (MuJoCo)
Hyperparom.
Swimmer-u8
HalfCheetah-us
Hopper-u8
Ant-u3
Walker2d-u3
Humanoid-03
n. timesteps
1000000
1000000
1000000
1000000
1000000
2000000
learning_ rate
0.001
0.001
0.0003
0.001
0.001
0.0003
learning_stann starts
10000
10000
10000
10000
10000
10000
batch size gamma
100
100
256
100
100
256
0.9999
0.99
0.99
0.99
0.99
0.99
train freq
(1, episode)
1
(1, episode)
(1, episode)
1
gradient_sstep
1
noise nopse_type noise_std
'normal'
'normal'
'normal'
'normal'
'normal'
"normal"
0.1
0.1
0.1
0.1
0.1
0.1
net arcrc activation fn
[256, 256]
[256, 256]
[256, 256]
2256, 256]
[256, 256]
[256, 256]
Rn.R
nn. ReLU
Rn.ReLU
Rn.R
ReLU
nn.ReLU
C Compute
distributed between 2 machines; (1) machine with 64 32-core AMD Ryzen Threadripper PRO 5975WX CPUs, each clocked at 4.3 GHz with 250 GB RAM with 2 NVIDIA GeForce RTX 3090 24 GB GPUs (2) machine with 16 8-core Intel(R) Core(TM) i7-9800X CPUs, each clocked at 3.8 GHz with 16 GB RAM and an NVIDIA GeForce RTX 20280 Ti 12 GB GPU.
All reported experiments
Additional experiments
D.1 Gaussian Spread
Figures and 9 (Appendix D.1 show the spread of the GMM Gaussian means at different steps in the training stage. Since the action dimensionality in all of the MuJoCo environments is greater than two,
19

---

Published in Transactions on Machine Learning Research (12/2024)
Table Shared TD3 hyperparameters (Fetch)
Hyperparom.
PetchPush-vl
FetchSlide-vi
FetchPPikAAnPPace-e
nI timesteps
1000000
1000000
1000000
learning rate
0.001
0.001
0.001
learning starts starts
1000
1000
1000
batch size
2048
2048
1024
gamma
0.95
0.95
0.95
noise noiee_type noise_ss std
'normal'
'normal'
'normal'
0.1
0.1
0.1
net arch
5122 512, 512]
512, 512, 512]
[512, 512, 512]
activation fn replay buffer class
ReLL
ReLR
ReLU
HerReplayBuufe
MeeReelayyBuffer
HerReplay Buffer
replay buffer kwargs 'dict(online. sampling-True, goal selection strubieyuutttteee n sampled ggal=o)aalll
Table 5: Gamid-spocific hyperparameters
Hyperparam.
Swim.
HalfCh.
Hopper.
Ant.
Walker.
Human.
Push.
Slide.
Pick.
n. actors (N)
3
3
lemperature initial (r)
0.1
0.1
0.1
0.1
0.1
0.3
0.3
0.3
0.3
lemperature final (r)
0.01
0.1
0.1
0.1
0.1
temperature fraction
1.0
1.0
1.0
1.0
1.0
0.3
1.0
1.0
1.0
epsilon initial (e)
0.3
0.2
0.1
0.1
0.3
0.3
0.1
0.1
epsilon final (c)
0.03
0.02
0.1
0.1
0.03
0.1
0.01
0.01
0.01
epsilon fraction
0.5
0.5
1.0
1.0
1.0
0.3
1.0
1.0
1.0
perform principal component analysis (PCA) (Abdi & Williams 2010) the Gaussian means and plot the top two PCA components. Comparing the top and middle rows of Figures corresponding to 7 0 and = 10 in 'Hopper-v3' respectively, we obscrve that the Gaussian means have higher spread for 7 - 10. A similar observation can also be made when comparing the top and middle rows of Figures
9 corresponding to 10 in "HalfCheetah-v3 respectively. The trend in bottom rows Figures S and corresponding r = opt. less clear which we speculate could be attributed noisy weight updates duc mini-batch gradient descent during training.
7 = 0 and
D.2 Sensitivity analysis for MuJoCo tasks
Following Haarnoja et al. (2018), we include the sensitivity analysis for all the MuJoCo tasks (beyond the representative, single task, results presented in the main text).
Figures 10 15 present training curves similar to those presented in the ablation study from the main text albeit all the MuJoCo tasks. These results generally support the conclusions provided in the main text (1) adding more Gaussians to the GMM-based actor has positive impact up to certain threshold, beyond which, performance stagnates or even deteriorates, (2) setting decaying policy divergence parameter is beneficial, and (3) using an e-greedy approach with a low (yet not zero) e value is beneficial.
20

---

Published in Transactions on Machine Learning Research (12/2024)
Table 6: SAC hyperparameters (MuJoCo)
Hyperparam.
Swimmer--u
HolfChectah--8
Hopper-u3
Ant-v3
Walker2d-u8
Humanoid-e3
timesteps
1000000
1000000
1000000
1000000
1000000
2000000
learning rate
0.0003
0.0003
0.0003
0.0003
0.0003
0.0003
learning starts
10000
10000
10000
10000
10000
10000
gradient_stedd steps garmma
1
0.9999
0.99
0.99
0.99
0.99
0.99
tau
0.05
0.05
0.05
0.05
0.05
0.05
net arch activation fn
256, 256]
[256, 256]
[256, 256] nn. nn.ReLU
[256, 256 nn. R.
[256, 256] nn. Re.U
[256, 256
nn.ReLU
nn. ReLU
nn. ReLU
Table 7: SACM hyperparameters (MuJoCo)
Hyperporam.
Swimmer-n3
HalfChectah-o3
Hopper-u3
Ant-u3
Walker2d-u8
Humanoid-u3
n_ components
3
3
3
3
learning _rate
0.0006
0.0003
0.0003
0.0003
0.0003
0.0003
learning starts
10000
10000
10000
10000
10000
10000
gradient
_reps steps
-1
tau
0.05
0.05
0.05
0.05
0.05
0.05
net arch activation fn
256)
[256, 256]
[256, 256]
[256, 256]
[256, 256]
[256, 256]
mn. ReLU
ReLU
nn.ReLU
UeLU
nn.ReLU
un.ReLU
Table SAC and SACM hyperparameters (Fetch)
Hyperparam.
FetchPush-ol
FetchStide-ol
PetcchPikknndPlucc--11
n timesteps rate _aae
1000000
1000000
1000000
learning_
0.001
0.001
0.001
learning_sst _tarrss starts batch size gamma ent ent_ net arch activation fn components (SACM) replay_buffer buffer class
1000
1000
1000
2048
2048
1024
0.95
0.95
0.95
'auto'
'auto'
'auto'
512, 512, 512]
512, 512, 512
512, 512, 512)
mn. ReLU
mn. ReLU
Rn.R
3
3
HerReplay
HerReplay Buffer
HerReplay Buffer
replay_boe buffer kwargs
'dict(online Stwpaiiinn_smmplngggeeee goal selection Inlbcttoo_stratteyy-ffee "frrateyy='futrre" n. sassaammed goal=4) _g
21

---

Published in Transactions on Machine Learning Research (12/2024)
Table PPO hyperparameters (MuJoCo)
Hyperparum.
Swimmer-u3
HalfCheelah-o3
Hopper-o3
Ant-e3
2dlker2d-03
Humanoid-o3
normalize
True
True
True
True
True
True
gamma
0.9999
0.98
0.999
0.99
0.99
0.95
steps
1024
512
512
2048
512
512
batch size
256
32
64
32
256
learning rate
0.0006
9.8e
3e -
4
0.0001
5.05e
5
3.56e
ent__oet
4.02e4
1e 4e-4
0.0022
5.8e
0.0024
clip range
0.9999
0.1
0.2
0.2
0.2
0.3
n epoohhs
20
20
5
10
20
gae lambda
0.98
0.92
0.99
0.95
0.95
0.9
max max_graad
_rad__d____ grad_norm norm
0.8
0.8
0.7
0.5
1
vf coef
4.02c
0.58
0.83
0.5
0.87
0.43
net net_aacch arch
[256, 256]
[256, 256]
[256, 256
[256, 256]
[256, 256]
[256, 256]
activation fn
Rn.ReLU
Rn.ReL.U
nn.ReL.U
nn.ReL.U
nn.ReL.U
nn.ReL.
log std init
0.0
-2
-2
0.0
0.0
-2
ortho init
True
False
False
True
True
False
22

---

Published in Transactions on Machine Learning Research (12/2024)
P03
10
203
#03
opt.
R00
7CA2
PC4.
(a) Slep# 250,000
(b) Step# 500,000
(c) Step# 750,000
(d) Step# 1,000,000
Figure 8: Principal component analysis (PCA) visualization of Gamid Gaussian means in 'Hopper-v3' with == 0 (top row), 10 (middle row) and 7 set to the tuned value (bottom row). The legend represents the ID of each Gaussian.
23

---

Published in Transactions on Machine Learning Research (12/2024)
-2
KOA
PCL
opt.
(a) Step# 250,000
(b) Step# 500,000
(c) Step# 750,000
(d) Step## 1,000,000
Figure 9: Principal component analysis (PCA) visualization of Gamid Gaussian means in "HallCheetah-v3'
with- 7 = 0 (top row), r = (middle row) and set the tuned value (bottom row). The legend represents the ID of each Gaussian.
400
350
300
300
300
Average Iverage Return
250
250
250
200
(verage
200
Iverage
N=4
150
T=0.01
150
greedy (ε=0)
150
100
N=3
100
T=0.1
100
low (s=1/10N)
N-2
0.5
med. (E=1/2N)
50
50
50
N=1
T=0.1
high (E=1/N)
0.1 0.2 0.3 0.4 0.5 0.6
0.7 0.8 0.9
0.1 0.2
0.3 0.4 0.5 0.6
0.7
0.8 0.9
0.1
0.2
0.3 0.4 0.5 0.6 0.7 0.8 0.9
Steps (1e6)
Steps (1e6)
Steps (1e6)
(a) Number of Ganssians (N)
(b) Policy divergence temperature (7)
(c) GMM weights
Figure 10: Ablation curves for the 'Swimmer-v33 domain
24

---

Published in Transactions on Machine Learning Research (12/2024)
11000
11000
10000
10000
Average Retur
Average Return
10000
9000
9000
Average
8000
6000
8000
7000
N=4
0.01
greedy (s=0)
4000
7000
6000
N=3
(c=1/10M)
5000
N=2
2000
6000
med. (E=1/2N)
N-1
T-0.1
5000
high (c=1/N)
4000
0.1 0.2 0.3 0.4 0.5 0.6
0.7 0.8 0.9
0.1 0.2 0.3 0.4 0.5 0.6
0.7 0.8 0.9
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Steps (1e6)
Steps (1e6)
Steps (1e6)
(a) Number of Ganssians (N)
(b) Policy divergence temperature (7)
(c) GMM weights
Figure 11:
Ablation curves for the 'HalfChoctah-v3' domain
3500
3500
Average Retum
3000
Average Return
3000
2500
2500
2500
Average
2000
2000
2000
N-4
T=0,01
greedy (s=0)
1500
1500
1500
N=3
T=0.1
(t=1/10M)
1000
N=2
1000
1000
med. (s=1/2N)
500
N=1
500
T=0:1-0.01
high (6=1/N)
0.1 0.2 0.3 0.4 0.5 0.6
0.7 0.8 0.9
0.1 0.2 0.3 0.4 0.5 0.6
0.7 0.8 0.9
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Steps (1e6)
Steps (1e6)
Steps (1e6)
(a) Number of Gaussians (N)
(b) Policy divergence lemperature (7)
(c) GMM weights
Figure 12:
Ablation curves for the 'Hopper-v3' domain
6000
6000
6000
Average Return
5000
Average Return
5000
Average Return
5000
3000
N=4
3000
0.001
3000
greedy (8=0)
2000
N=3
2000
2000
(s=1/10M)
N=2
med. (s=1/2N)
1000
N=1
1000
T-0.1
0.01
1000
high (8=1/N)
0.1 0.2 0.3 0.4 0.5 0.6
0.7 0.8 0.9
0.1 0.2 0.3 0.4 0.5 0.6
0.7 0.8 0.9
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Steps (1e6)
Steps (1e6)
Steps (1e6)
(a) Number of Gaussians (N)
(b) Policy divergence lemperature (7)
(c) GMM weights
Figure 13: Ablation curves for the 'Ant-v3' domain
5000
5000
5000
Average Retur
Average Return
Average Return
4000
4000
4000
3000
N=5
3000
3000
N=4
T=0.01
greedy (s=0)
2000
N-3
2000
T-0.1
2000
(8=1/10N)
N=2
med. (c=1/2N)
1000
1000
1000
N=1
high (E=1/N)
0.1 0.2 0.3
0.4 0.5 0.6
0.7 0.8 0.9
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Steps (1e6)
Steps (1e6)
Steps (1e6)
(a) Number of Gaussians (N)
(b) Policy divergence temperature (7)
(c) GMM weights
Figure 14: Ablation curves for the 'Walker2d-v3" domain
25

---

Published in Transactions on Machine Learning Research (12/2024)
6000
6000
5000
Average Return
5000
Average Return
5000
4000
4000
N=4
3000
T=0.01
3000
greedy (s=0)
N=3
T=0.1
low (5- 1/10M)
2000
N=2
2000
T=0.5
2000
med. & (s=1/2N)
1000
N=1
1000
T=0.1
0.01
1000
high (c=1/N)
1.4
0.4
1.8
Steps (1e6)
Steps (1e6)
Steps (1e6)
(a) Number of Gaussians (N)
(b) Policy divergence lemperature (7)
(c) GMM weights
Figure 15: Ablation curves for the 'Humanoid-v3' domain
26


Reviews:
[{"review_id": "4nNbkcWDmy", "reviewer": "Reviewer_byV9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "SAC with a GMM slightly outperforms a baseline SAC method when trained according to this paper, suggesting that the KL divergence term or exploration strategy are the reason prior works did not see much benefit from using a GMM. The DDPG variant, Gamid, has more mixed results, although this is roughly in line with what I've seen in prior RL algorithms papers.\n\nThere are an extensive set of ablations inspecting how the number of Gaussians, temperature, etc. affect the method, indicating benefit a small number of Gaussians > 1 and a small but non-zero epsilon greedy exploration.\n\nOverall, although I am worried about the extra hyperparams introduced by this method, I do think it suggests some practical implementation choices for how to more effectively use GMMs within RL.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies using a Gaussian Mixture Model to represent the actor in an actor-critic setup. Prior work has found mixed results from using a GMM.\n\nThis work reexamines using GMMs, within a DDPG variant. Instead of using a single Gaussian, the actor is a mixture of Gaussians. Traditional DDPG uses a deterministic actor and samples exploration noise from a Gaussian centered around the deterministic actor. In this work the sampling is from the learnt GMM. Finally, there is an added term to encourage diversification of the GMM. To me this is the main change compared to prior work, or compared to just using a GMM within SAC.\n\nWhen computing the Q-target, the next Q-value is taken by maximizing over the $N$ Gaussian parameters. When updating the actor, each Gaussian's mean is updated separately, with an extra diversity term $D_{KL}(i || N \\setminus i)$, indicating the KL between the $i$th Gaussian and the mean of every other Gaussian in the mixture. There isn't a closed form for this, but there is a closed form approximation that is straightforward to compute, especially when all Gaussians share the same variance.\n\nFor exploration, the action is sampled according to epsilon-greedy: the maximizing Gaussian in the mixture is selected $1-\\epsilon$ of the time, and the Gaussian is picked randomly otherwise.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Would appreciate a bit more analysis on what mixtures have been learned in the MuJoCo experiments. Most of the plots are for expected return, and although that is important, it would be good to see if the diversity KL term is successfully pushing the Gaussians to be diverse.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xf3xUavlk1", "reviewer": "Reviewer_tB8P", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strenghts \n+ The paper is clear in the presentation of the proposed algorithm;\n+ The authors introduce well the idea of using distributional policies in MaxEnt Actor-Critics and use that to motivate their approach.\n\n# Weaknesses\n+ It's not as clear why would using a distributional parametrization of the policy of a Deterministic Actor-Critic would bring benefits, especially vs using such parametrization for the Critic (as is done in D4PG);\n+ While the algorithm introduces new hyperparameters that are tuned the comparisons against other algorithms use just standard hyperparameters with no extra tuning, giving an unfair advantage to Gamid-PG in the comparisons;\n+ The number of runs of each algorithm is small (n = 5), which can hamper the validity of the results when just mean and standard deviation are reported.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a method for training a DDPG-based Actor-Critic agent where the policy is parameterized using a Gaussian Mixture Model. Furthermore the agents trained using this algorithm are compared against TD3, which is equivalent to the presented method when the mixture has a single Gaussian, and SAC and its own distributional variants, which are seen as a different family of off-policy Actor-Critic methods which are based on Max Entropy Reinforcement Learning. The authors claim that by adding GMMs to a DDPG formulation they can get benefits from MaxEnt Actor-Critcs and from Deterministic Actor-Critics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Clarification Changes\n+ It would be great if the authors could add a couple of sentences comparing with D4PG, especifically on the merits of using distributional RL on the Critic vs the Policy. There's no need to run experiments for the comparison, especially given the many additional differences between Gamid-PG and D4PG. The comparison is necessary for approval;\n+ In section 4.1 the authors state \"We attribute this ability to the entropy maximization term in Gamid, which encourages the GMM components to spread out\" why wouldn't this also apply to SAC and lead it to also achieve optimal performance? Also adding SACM to this comparison could potentially help shed a light on the difference of policies parameterized by a Gaussian vs a GMM. The explanation would be necessary for approval, the extra comparison is encouraged;\n+ In Figures 3g-i PMOE and PPO don't seem to be benchmarked in the Fetch tasks, while they were benchmarked in the Mujoco tasks, is there any reason for that? An explanation is necessary for approval, but additional experiments are not.\n\n# Validity Changes\n + The number of runs is small as mentioned previously, and we don't know whether the performance of agents in the tasks is normally distributed, as such reporting mean and standard deviation (by plotting mean perfomance and using a 2-std shaded region) may be misleading, and so we request the authors to instead plot the mean agent and tolerance intervals as described in [1]. This change is necessary for approaval;\n+ For the toy bandit environment in Figure 1 we request the authors tune the hyperparameters for TD3/DDPG and SAC as they were tuned for Gamid-PG, and that the exploratory phase described in Section 7 of [1] be used for SAC and TD3/DDPG, as it seems to be fundamental to get good performance from the algorithms. This change is also necessary for approval.\n\n[1]: Empirical Design in Reinforcement Learning, Patterson et al, https://arxiv.org/abs/2304.01315", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "K0ew2y763g", "reviewer": "Reviewer_SD3W", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n* The paper provides a comprehensive and clear introduction and related works to demonstrate the motivation. The whole paper is well written in general. \n\n* The authors provide comprehensive experimental details and results, enhancing the study's reproducibility and reliability. \n\nWeakness: \n* Although the authors claim the method relies on dense and accurate reward environment, the experiments are mainly on relatively simple tasks in MuJoCo. Can author add more experiments on more complex environment, e.g. Antmaze or beyond MuJoCo? Since in real life it is hard to have accurate stepwise reward, it would be great if the method can be proved working on more diverse tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "* The paper proposes Gaussian Mixture Deterministic Policy Gradient (Gamid-PG), which applies deterministic policy gradients to Gaussian Mixture Models (GMMs) in reinforcement learning. This approach contrasts with the commonly used stochastic gradient-based methods for optimizing GMMs. \n\n* The paper conducts an extensive comparison between deterministic and stochastic policy gradients for GMMs across multiple benchmark tasks, demonstrating that deterministic policy gradients can outperform stochastic ones in dense-reward tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* I approciate the authors by providing figures to show the training procedure on MuJoCo. However, since there are multiple methods to be compared, it would be better if the author also provide some tables to report the numbers of each task. In this way, we can clearly see the performance of the proposed method and baselines. \n\n* As I mentioned in the weakness, I'm wondering if the authors can provide more experiments on more environments.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All of the reviewers believe that the the contribution is novel and the claims have been supported, and after a few rounds of discussion, eventually proposed a mix of Accept / Learning Accept. 

Specifically, the main changes were:
  * Reviewer tB8P asked for more fair hyperparameter tuning over baselines, and more rigorous plotting. There was also confusion on why distributional RL would help D4PG, but this was cleared by the authors in the paper.
  * Reviewer SD3W asked for experiments on high-exploration environments (e.g. Ant-Maze).  Authors responded that GMM's don't completely solve the exploration problem.

Judging by the potential impact however, this paper currently does not warrant a featured certification, as the consensus has been lukewarm.
Decision: Accept (poster)