Probabilistic Mixture-of-Experts for Efficient Deep Reinforcement Learning
Authors: Jie Ren
Yewen Li
Zihan Ding
Wei Pan
Hao Dong
Venue:  (2021)
Keywords: Deep Reinforcement Learning
Sample Efficiency
Gaussian Mixture Models
Mixture-of-Experts

Abstract: Deep reinforcement learning (DRL) has successfully solved various problems recently, typically with a unimodal policy representation. However, grasping the decomposable and hierarchical structures within a complex task can be essential for further improving its learning efficiency and performance, which may lead to a multimodal policy or a mixture-of-experts (MOE). To our best knowledge, present DRL algorithms for general utility do not deploy MOE methods as policy function approximators due to the lack of differentiability, or without explicit probabilistic representation. In this work, we propose a differentiable probabilistic mixture-of-experts (PMOE) embedded in the end-to-end training scheme for generic off-policy and on-policy algorithms using stochastic policies, e.g., Soft Actor-Critic (SAC) and Proximal Policy Optimisation (PPO). Experimental results testify the advantageous performance of our method over unimodal polices and three different MOE methods, as well as a method of option frameworks, based on two types of DRL algorithms. We also demonstrate the distinguishable primitives learned with PMOE in different environments.

Under review as a conference paper at ICLR 2021
PROBABILISTIC MIXTURE-OF-EXPERTS FOR EFFICIENT DEEP REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Deep reinforcement learning (DRL) has successfully solved various problems recently, typically with unimodal policy representation. However, grasping the decomposable and hierarchical structures within a complex task can be essential for further improving learning efficiency and performance, which may lead to a multimodal policy mixture-of-expert (MOE). To our best knowledge, present DRL algorithms for general utility do not deploy MOE methods as policy function approximators due to the lack of differentiability, representation. In this work, we propose a differentiable probabilistic mixture- of-experts (PMOE) embedded in the end-to-end training scheme for generic off- policy and on-policy algorithms using stochastic policies, e.g., Soft Actor-Critic (SAC) and Proximal Policy Optimisation (PPO). Experimental results testify the ad advantageous performance of our method over unimodal polices and three different MOE methods, as well as a method of option frameworks, based on two types of DRL algorithms. We also demonstrate the distinguishable primitives learned with PMOE in different environments.
without explicit probabilistic
INTRODUCTION
The mixture- wfaeee-of--eeeeeee off-experss method (MOE) (Jacobs et al. 1991a) is testified to be capable of improving the generalisation ability of reinforcement learning (RL) agents (Hausknecht & Stone 20166a] Peng et al., et al. Among these methods, the Gaussian Mixture Models (GMM) are promising model multimodal policy in RL (Peng et al. 2019 Akrour et al. 2020), in which distinguishable experts or so-called primitives are learned. The distinguishable experts can propose several solutions for a task and have larger range of exploration space, which can potentially lead better task performance and sample efficiency compared to its unimodal counterpart (Bishop) 2007). The multimodal policy can be learned by various methods, such as two-stage training approach (Peng et al. 2019), specific clustering method (Akrour et al.) [2020), or especially parameterised actions design (Hausknecht & Stone 2016b). However, these methods are limited, neither applicable to complicated scenarios such high-dimensional continuous control tasks nor the training algorithms are too complex to deal with general utility. To the best of our knowledge, the present DRL. algorithms for general utility do not deploy MOE to model the multimodal policy mainly due to the lack of differentiability, or without explicit probabilistic representation. Therefore, in the policy gradient- based algorithms (Sutton al. 1999a), the gradient of the performance concerning the policy parameters is undifferentiated. The undifferentiability problem also remains to learn a deep neural network policy thus making the combinations of MOE and DRL not trivial
In this paper, we propose a probabilistic framework to tackle the undifferentiated problem by holding the mixture distribution assumption. We will still use the GMM to model the multimodal policies. Once the undifferentiated problem is solved, our training methods can be combined with the policy gradient algorithms by simply setting the number of experts (mixtures) greater than one. Hereafter, the contribution can be summarised as follows:
We analyse the undifferentiabiiliyy problem of approximating policy the GMM in DRL and its associated drawbacks.
We propose an end-to-end training method to obtain the primitives with probability in a frequentist manner solve the undifferentiabiiity problem.

---

Under review as a conference paper at ICLR 2021
Our experiments show the proposed method can achieve better task performance and sample efficiency by exploring larger behaviours space, especially in complicated continuous control tasks, compared with unimodal RL algorithms and three different MOE methods or option frameworks.
2 RELATED WORK
There are two main related hierarchical policy structures. The feudal schema (Dayan & Hinton 1992) has two types of agents: managers and workers. The managers first make high-level decisions, then the workers make low-level actions according these high-level decisions. The options framework (Sutton et al. 1999b) has an upper-level agent (policy-over-options. which decides whether the lower level agent (sub-policy) should start or terminate. In the early years, it's the subject of research to discover temporal abstractions autonomously often in discrete actions and the state space (McGovern & Barto 2001| Menache et al., 2002, (Simsek & Barto [0088] Siiver & Ciosek 2012). Recently, (Mankowitz et al. 2016) proposes method that assumes the initiation sets and termination functions have particular structures. (Kulkarni et al. 2016) uses internal and extrinsic (Bacon al. 2017) trains sub-policies and policy-over-option in end-to-end fusion with deep termination function. (Vezhnevets et al.) 2017) generalises the feudal schema into continuous action space and uses embedding operation to solve the indifferentiable problem. (Peng et al., (016)) introduces mixture of actor-critic experts approaches learn terrain-adaptic dynamic locomotion skills. (Peng et al. (219)) changes the mixture-of-expert distribution addition expression into the multiplication expression.
Hierarchical Policies
rewards to learn sub- policiilice and volicy-over-optione
coxture-offExpertte and Ensemble Methods To speed up the learning and improve the generali- sation ability on different scenarios, Jacobs al. (1991a) proposed to use several different expert networks instead of a single one. To partition the data space and assign different kernels for different spaces, Lima al. (2007) y Yao et al. (2009) combines MOE with SVM. To break the dependency among training outputs and speed the convergence, Gaussian process (GP) is generalised similarly to MOE [Trspp] 2000 Yuan & Neubauer 2008; Luo & Sun 2017). MOE can be also combined with
RL (Doya al. 2002 Neumann et al.j Peng et al. 2016 Hausknecht & Stone 201 6a; Peng et al.,
2019), in which the policies are modelled as probabilistic mixture models and each expert aim to learn distinguishable policies.
Polecy-based RL Policy-based RL aims to find the optimal policy to maximise the expected return through gradient updates. Among various algorithms, Actor-critic is often employed (Barto et al. 1983 Sutton & Barto 1998). Off-policy algorithms (O (O'Donoghue et al. 2016 [illicrap a 2016 Gu et al. 2017 | Tuomas et al. 2018) are more sample efficient than on-policy ones (Peters Schaal 2008 Schulman al. [0177] Mnih al. 2016 Gruslys et al. 2017). However, the learned policies are still unimodal.
METHOD
3.1 NOTATION
The model-free RL problem be formulated by Markov Decision Process (MDP), denoted tuple (S, A, P.r), r), where S and A are continuous state and action space, respectively. The agent observes state 8, S and takes action a, AA time step The environment emits reward [r (rmin: "mminFmman Fmax] and transitions to a new state 84+1 according to the transition probabilities P: x S x A + [0, xc). deep reinforcement learning algorithms, we always use the Q-value function (s at) to describe the expected return after taking an action a, in the state 8t. The Q-value can be iteratively computed by applying the Bellman backup given by:
r S x
Q(s;, 01) A E [r(se, az) + (((
(1)
Our goal is to maximise the expected return:
(a arg max E. Ba Bax (a))))))
(2)
2

---

Under review as a conference paper at ICLR 2021
where è denotes the parameters of the policy network л. With Q-vauue network (critic) Qo param- eterised by Φ. Stochastic gradient descent (SGD) based approaches are usually used to update the policy network:
+ = θ +
Veloanat(ale)) at)]-
(3)
3.2 PROBABILISTIC MIXTURE- OF EXPPRTS (PMOE)
The proposed PMOE method decomposes stochastic policy и as a mixture of low-level policies while retaining the probabilistic properties of the stochastic policy as a probability distribution, with the following formula:
K
K Σ
[[.w) (aolse)
Σ wo (s:) (acls)) s.t.
1, wg. > 0,
(4)
where each nw, denotes the action distribution within each low-level policy, i.e. a primitivee and K denotes the number of primitives. We, is the weight that specifies the probability of the activating primitive TH; which is called the routing function. O; and th; parameters of wa, and Tye respectively. After the policy decomposition with PMOE method, we can rewrite the update rule in Eq. ass
(Q 0 = 0 + Po (0 at),
(5)
< = & V E (o (=)] QQ at)].
In practice, we usually apply a Gaussian distribution for either a unimodal policy or the low-level policies here in PMOE. making the overall stochastic policy with PMOE to be GMM. However, sampling from the distributions of primitives usually embeds sampling process from a categorical distribution w, which makes the differential calculation of policy gradients commonly applied in DRL hard to achieve. We provide a theoretically guaranteed solution for approximating the gradients in the sampling process of PMOE and successfully optimising the PMOE policy model within DRL, which will be described in details.
3.3 LEARNING THE ROUTING
To optimise the routing function w. which involves a sampling process from categorical distribution, we propose a frequency loss function to approximate the gradients, which is theoretically proved to approximate w as the probability of the corresponding primitive being the optimal one w.r.t. the Q-value function.
Specifically, given a state 8t, we sample one action at from each primitive Tyk to get total of K actions {aş i 2, .K, and compute K Q-value estimations (Qo a 1, 2, ... K}
for each of the actions. Then select "optimal" primitive index j arg max, Qo(st, at). Then we encode a one-hot code vector
= [Vi, U2 ... ,U]] with:
1. if = arg arg Qs (52,
(6)
0, otherwise.
Here define frequency loss function as:
Lireg = (v-
W) [we Wos. ... woxl-
(7)
We use the proposed frequency loss Lfreg smooth and differentiable function to update the routing function parameters 0, which is guaranteed to approximate wa, as the probability of the i-th primitive being the optimal primitive for current state. Detailed proof is provided in Appendix B
3.4 LEARNING THE PRIMITIVE
To update the eh; within each primitive, we provide two approaches of optimising the primitives: back- propagation-all and back- pooo---orpaaion-mmaee manners.
For the back- pacc-proppgaiion-al approach, we update all the primitive:
K
hpa = - Σ pri
Qo(si, a(), of nV. (a.ls)).
(8)
3

---

Under review as a conference paper at ICLR 2021
For the back propagatioonnaax approach, use the highest Q-value estimation as the primitive loss:
Eyr max (Qo(se, dii = 1,2, .. ,K),, oj To. (aclst)-
(9)
With either approach, we have the stochastic policy gradients as following:
Ver pri = V 4t Bug ( de)l
(10)
-Bay, Q a₂ V logTT, TH (a.ls;)]
Ideally, both approaches are feasible for learning PMOE model. However, practice, we find that the back-propagations approach will tend to learn primitives that are close to each other, while the procck-ropggaiin--ee max approach is capable of keeping primitives distinguishable. The phenomenon is demonstrated in our experimental analysis. Therefore, we adopt the back-prepaggaionnnma approach as the default setting of PMOE model without additional clarification.
3.5 LEARNING THE CRITIC
Similar to standard off-policy RL algorithms, our Q-value network is also trained to minimise the Bellman residual:
Ceritic (eee.aa)==llQQ((e ae) - [rx + y max Q4(
(11)
ax+1
where Ø is the parameters of the target network.
The learning component can be easily embedded into the popular actor-critic algorithms, such soft actor-critic (SAC) (Tuomas et al., 2018), one of the state-of-the-art off-policy RL algorithms. In SAC, (st, af) is substituted with Q (s,, at) + a.H,, where a is temperature and H. log Ty. (ag|s;) the entropy which are the same in SAC. The algorithm is summarised in Algorithm I When K = 1, our algorithm simply reduced to the standard SAC.
4 EXPERIMENTS
In our experiments, we will answer the following questions concerning the advantages of PMOE method and the training settings for achieving good performance:
Q1: Is our proposed method more efficient and stable than other model-free RL algorithms?
Q2: What is the benefit of using GMM-basee policy approximation unimodal Gaussian policy?
Q3:
How does the number of primitives affect the performance?
To answer Q1, we conduct thorough comparison
set of challenging continuous control tasks in
OpenAl Gym MuJoCo environments (Brockman et al. 2016) with other basclines, including a MOE
method with gating operation (Jacobs et al. 1991b), Double Actor-Critic (DAC) Zhang & Whiteson
2019) option framework, and the Multiplicative Compositional Policies (MCP) (Peng et al. (219)).
Well-known sample efficient algorithms involving Soft Actor- Critic (SAC) (Tuomas et al. 2018) and
Proximal Policy Optimisation (PPO) (Schulman et al. 2017) are adopted as basic DRL algorithms in our experiments, where different multimodal policy approximation methods are built on top. This verifies the generality of PMOE for different DRL algorithms. To answer Q2, a deeper investigation of PMOE is conducted to find out the additional effects caused by deploying mixture models rather than single model in policy learning. We start with a simple self-defined target-reaching task to show our method can indeed learn various optimal solutions with distinguishable primitives, which are further demonstrated on complicated tasks in MuJoCo. Additionally, the exploration behaviors are also compared between PMOE and other baselines to explain the advantageous learning efficiency of PMOE. To know how the number of primitives affects the performance (Q3), we test PMOE with different different values of K on the HumanoidStandupp- environment. We provide intuitive experiences for determining the number of primitives when applying PMOE. We also demonstrate the advantages of proposed PMOE its exploration behaviours and the distinctiveness of different primitives learned with PMOE against the baselines.
4

---

Under review as a conference paper at ICLR 2021
Algorithm 1: POME Training Algorithm.
Input: Policy network { with parameters {0, v.}. Critic network Qo with parameters 6.
Initialise target policy network and critic network parameters: Ô t 0, + v; 0 + Φ.
Initialise an empty replay buffer: D c 4
while not converge do
for each environment step do Sample action from policy: a2 [T (aclse).
Interact with the environment: 81+1 = p(8 (
Store data in replay buffer: D = D {st, ae, Stt1, r ($t, a.).
for each update step do
Sample from replay buffer: {st, at s141. ~ D.
for each policy update step do
Sample K actions from primitives: {at ~ Tu. (a 1,2, K}. 1Q a)); = 1,2, ... K}.
Compute target Q-value estimations Q
compute primitive loss Eprs according
Compute mixing coefficients w(s)) and one-hot code vector according to Eq. 6
then compute freq-low loss freg according to Eq.
Update policy network: = 8 - V VoL
V Cpri-
for each critic update step do
Sample action from mixture policy: a++1 T1 (ac-1 (1t+1))
Compute next time step target Q-value estimation: Q5($1+1, a++1).
Compute Q-value estimation Qo as) and compute critic loss Ceritic according to
Eq. 11
Update critic network: o Φ V Voferitie.
Update target network: 0 == =0 70 + (1 - r)0, sb TV + (1 T)O, 0 0 = =0 + +(1 (1- T)0.
Output: 0, d.
4.1
COMPARATIVE EVALUATION
The evaluation on the average returns with SAC-based and PPO-based algorithms are shown in Fig. [ and Fig. 2 respectively. In addition, we compute the AUC (the area under the curves) values for those comparisons in Appendix our PMOE for policy approximation are compared against the MCP (Peng et al. 2019) and gating operation methods (Jacobs et al.) 1991b) in terms of the average returns across six typical MuJoCo tasks. In Fig. 2 comparison with similar settings while basing on different DRI. algorithm is conducted to show the generality of PMOE method. Specifically, we compare the proposed PMOE for policy approximation on PPO with DAC and PPO-based MCP methods. From the comparisons, we found that in relatively simple environments such as Ant-v2 and Walker2d-v2, large K might not be necessary. While complicated environments like Humanoid-v2 and HumanoidStandup. larger K is preferred. This is possible because the optimal solutions might be unique in simple environments thus making multimodal representation unnecessary. Nevertheless, our method still more sample efficient, showing the capability of degradation of models in our method, where the redundant primitives automatically re-fit onto other necessary primitives and diminish the effective number of K. PMOE is testified to provide considerable improvement for general DRL algorithms with stochastic policies on a variety tasks. Training details are provided in Appendix
Specifically, in Fig. SAC-based algorithms with either unimodal policy
4.2 EXPERIMENT ANALYSIS
We provide in-depth analysis of the proposed PMOE to analysis the differences between PMOB method and other methods in RL process (Q2), and estimate the effects of the number of primitives (Q3). We use a self-defined target-reaching environment to analyse our method. In this environment, the agent starts from fixed initial position, then acts to reach a random target position within certain range and avoids the obstacles on the path. Details about this environment is provided in Appendix C
5

---

Under review as a conference paper at ICLR 2021
4000
6000
6000
Average Return
Average Retur
Average Return
5000
3000
4000
4000
2000
3000
2000
1000
2000
1000
%
8.0
1.0
Steps
Million Steps
Million Steps
(a) Ant-v2
(b) Hopper-v2
(c) Walker2d-v2
5000
160000
12500
Average Return
4000
Average Return
Average Return
10000
140000
3000
2000
120000
5000
MCP-SAC
SAC
1000
2500
Gating
100000
PMOE-SAC
8.0
0.2
0.4
0.6
0.8
1.0
0.00
0.25
0.50
0.75
1.00
8.0
0.2
0.4
0.6
0.8
1.0
Million Steps
Million Steps
Million Steps
(d) Humanoid-v2
(c) HumanoidStandup-v2
(f) HalfChectah-v2
Figure I: Training curves on MuJoCo benchmarks with SAC-based algorithms. We set PMOE with
K 4 in all the experiments except HalfCheetah-v2 with K
and HumanoidStandupp12 with
K 10.
3000
4000
Average Return
Average Return
Average Return
3000
5000
1000
1000
10000
8.0
0.2
0.4
0.6
0.8
1.0
2
Million Steps
Million
Steps
Million Steps
(a) Ant-v2
(b) Hopper-v2
(c) Walker2d-v2
1500
90000
4000
Average Return
1250
Averadi Retura
80000
Mwernge Return
1000
70000
750
60000
Average
2000
500
50000
DAC
1000
MCP-PPO
250
40000
PMOE-PPO
8.0
0.2
1.0
3000g
8.0
Steps
Steps
Steps
(d) Humanoid-v2
(c) HumanoidStandup--2
(f) HalfCheetah-v2
Figure 2: Training curves on MuJoCo benchmark with PPO-based algorithms. We set a larger number,
for the high-dimensional environments Humanoid-v2 and FlumunoddSundup-v2 and K = 4 for other environments The results show that our PMOE has comparable performance with the baseline on low-dimension environments, but significantly better than the baseline on high-dimensional environments.
Distinguishable Primitives. Fig. demonstrates the distinguishable primitives (Peng al. 2019) learned with PMOE on the target-reaching environment, for providing a simple and intuitive under- standing. After the training stage, sample 10 trajectories for each method and visualise them in

---

Under review as a conference paper at ICLR 2021
(a)
(c)
(d)
Figure 3: Trajectories of the agents with our method and the baselines in the target-reaching en- vironment. We fix the reset locations of target, obstacles and agent. (a), (6), (c) and (d) visualize the 10 trajectories collected with methods involving: original SAC, gating operation with SAC, back-pne teek---ropggtion--a all PMOE (discussed in Sec. (3.4) and back boock-ropagation-mae PMOE, respectively. (e) shows the trajectories collected with two individual primitives with our approach.
Class
Class 1
Class
Primitive 0 Primitiva Primitve 2 Primitive 3
Class
(a) Gating operation.
(b) States clusters.
(c) Our approach.
Figure 4: Visualisation of distinguishable primitives learned with PMOE using t-SNE plot Hopper- v2 environment. The states are first clustered as (b). Then actions within the same state cluster plotted with t-SNE in (a) and (c) for the gating method and our approach, respectively. Our method clearly demonstrates more distinguishable primitives for the policy.
Fig. 3 As we can see in Fig. 3 PMOE trained in back Breck--ropagatin--eee max manner generates two distinguishable trajectories for different primitives.
In Fig. we further demonstrate that PMOE can learn distinguishable primitives on more complex environments with the t-SNE (van der Maaten & Hinton 2008) method. We sample 10K states 10K} } from 10 trajectories and use t-SNE to perform dimensionality reduction on states and visualise the results Fig. 4(b) We randomly choose state cluster and sample actions corresponding to the states in that cluster. Then we use t-SNE to perform dimensionality reduction on those actions. The reason for taking state clustering process before the action clustering to reduce the variances of data in state-action spaces, so that we can better visualise the action primitives for a specific cluster of states. The visualisation of action clustering with our approach and the gating operation are displayed in Fig. 4(a) and Fig. [[c) More t-SNE visualisations for other MuloCo environments can be found in Appendix Our proposed PMOE method is testified to have stronger
capability in distinguishing different primitives during policy learning process. The t-SNE parameters are the same in all the experiments: number of components-2, perplexity=300, learning rate=150, number of iterations=7000.
Exploration Behaviours. Fig. 5 demonstrates the exploration trajectories the target-reaching environment, although all trajectories start from the same initial positions, our methods demonstrate larger exploration ranges compared against other baseline methods, which also yields a higher visiting frequency to the target region (in green) and therefore accelerates the learning process. To some extent, this comparison can explain the improvement of using PMOE as the policy representation a general unimodal policy. We find that by leveraging the mixture models, the agents gain
7

---

Under review as a conference paper at ICLR 2021
effective information more quickly via different exploration behaviours, which cover larger range of exploration space at the initial stages of learning and therefore ensure a higher ratio of target reaching.
100
00
Success Ra5e
0.4
55
Steps
(a) Ours
(b) Success Rate in Learning
(C) SAC
Figure 5: Visualisation of exploration trajectories in the initial training stage for the target-reaching environment. The initial 10K steps (the grey region on the learning curves in (b) of exploration trajectories plotted in (a) and (c) for our PMOE method (red) and SAC (bluc), respectively. The green rectangle is the target region.
Number of Primitives. We investigate the effects caused by different numbers of prim- itives in GMM, as shown in Fig. 6 This experiment conducted on relatively com- plex environment HumanoidStandupp-2 that has observations with dimension of 376 and actions with dimension of 17, therefore various skills could possibly lead to the goal of the task. The number of primitives K is se- lected from {2, 4, 8, 10, 14, 16). The results show that K 10 seems to perform the best, and K = performs the worst among all settings, showing that increasing the num- ber of primitives can improve the learning efficiency in some situations. Our intuitive choice is that the number of primitives needs be increased when the dimension of action is large.
170000
150000
130000
K-4
120000
K=10
110000
Kw14
K=16
100000
0.0
0.2
0.4
0.6
0.0
1.0
Million Steps
Figure
Comparison different numbers of primitives K in terms of average returns
HumanoidStandup-22 environment. For each case, conduct runs and take the means. The perfor- mance increases when increase from 2 to 10, but decreases if K keep increasing from 10 16.
CONCLUSION
To cope with the problems of low learning efficiency and multimodal solutions in high- dimensional continuous control tasks when applying DRL, this paper proposes the differentiable PMOE method that enables an end-to-end train- ing scheme for generic RI. algorithms with stochastic policies. Our proposed method is compatible with seiiey-gradiet-baarr
algorithms, like SAC and PPO. Experiments show performance improve- ment across various tasks achieved by applying our PMOE method for policy approximation, as well as displaying distinguishable primitives for multiple solutions.
REFERENCES
Riad Akrour, Davide Tateo, and Jan Peters. Reinforcement learning from a mixture of interpretable experts. CoRR, abs/2006.05911, 2020.
Pierre-Lue Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In AAAI, pp. 1736-1734, 2017.
8

---

Under review as a conference paper at ICLR 2021
Andrew G. Barto, Richard S. Sutton, and Charles W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Trans. Syst. Man Cybern., 13(5):834-84) 1983.
Christopher M. Bishop.
Pattern recognition and machine learning, 5th Edition. Information science and statistics. Springer, 2007.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. CoRR, abs/1 015000 2016.
Peter Dayan and Geoffrey Hinton. Feudal reinforcement learning. In NewrlPS, pp. 271-278, 1992.
Kenji Doya, Kazuyuki Samejima, Ken-ichi Katagiri, and Mitsuo Kawato. Multiple model-based reinforcement learning. Neural Comput., 14(6):1347-136) 2002.
Audrunas Gruslys, Mohammad Gheshlaghi Azar, Marc G. Bellemare, and Rémi Munos. The reactor:
A sample-efficint actor-critic architecture. CoRR, abs/1 1e//1104.046666 704.04451,,20 2017.
Shixiang Gu, Timothy P. Lillicrap. Zoubin Ghahramani, Richard E. Turner, and Sergey Levine. Q-prop: Sample-effccient policy gradient with an off-polio critic. In ICLR, 2017.
Matthew Hausknecht and Peter Stone. Deep reinforcement learning in parameterized action space. In ICLR, 2016a.
Matthew J. Hausknecht and Peter Stone. Deep reinforcement learning in parameterized action space. In ICLR, 2016b.
Robert Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural Comput., 3(1):79-87, 1991a.
Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Comput., 3(1):79-87, 1991b.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Tejas D. Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In NeurIPS, pp. 3675-3683, 2016.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR. 2016.
Clodoaldo Ap. M. Lima, André I.. V. Coelho, and Fernando J. Von Zuben. Hybridizing mixtures of experts with support vector machines: Investigation into nonlinear dynamic systems identification. Inf. Sci., 177010):049-2074, 2007.
Chen Luo and Shiliang Sun. Variational mixtures of gaussian processes for classification. In IJCAI, pp. 4603-4609, 2017.
Daniel J. Mankowitz, Timothy A. Mann, and Shie Mannor. Adaptive skills adaptive partitions (ASAP). In NeurlPS, pp. 1588-1596, 2016.
Amy McGovern and Andrew G. Barto. Automatic discovery of subgoals in reinforcement learning using diverse density. In ICML, pp. 361-3681 2001.
Ishai Menache, Shie Mannor, and Nahum Shimkin. Q-cut dynamic discovery of sub-goals in reinforcement learning. In ECML, volume 2430, pp. 295-306, 2002.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. ICML, volume 48, pp. 1928-1937, 2016.
Gerhard Neumann, Wolfgang Maass, and Jan Peters. Learning complex motions by sequencing simpler motion templates. In ICML., volume 382, pp. 753-760.

---

Under review as a conference paper at ICLR 2021
Brendan O' D'Dongguue Rémi Munos, Koray Kavukcuoglu, and Volodymyr Mnih. PGQ: combining policy gradient and q-learning. CoRR, abs/1611.01622 2016.
Xue Bin Peng, Glen Berseth, and Michiel van de Panne. Terrain-adaptive locomotion skills using deep reinforcement learning. ACM Trans. Graph., 35:4):8111-81:12, 2016.
Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. MCP: learning composable hierarchical control with multiplicative compositional policies. In NeurlPS, pp. 3681-3692, 2019.
Jan Peters and Stefan Schaal. Reinforcement learning of motor skills with policy gradients. Neural Networks, 21()):82-6677 2008.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, 0es//777.00347, 2017.
David Silver and Kamil Ciosek. Compositional planning using optimal option models. In ICML, 2012.
Özgür Simsek and Andrew G. Barto. Skill characterization based on betweenness. In NeurlPS, pp. 1497-1504, 2008.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning introduction. Adaptive computation and machine learning, MIT Press, 1998.
Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NeurlPS, pp. 1057-1063, 1999a.
Richard S. Sutton, Doina Precup, and Satinder P. Singh. Between mdps and semi-mdps: A framework for temporal abstraction reinforcement learning. Artif. Intell., 112(1-2):181-211, 1999b.
Volker Tresp. Mixtures of gaussian processes. In NeurIPS, pp. 654-660, 2000.
Haarnoja Tuomas, Zhou Aurick, Abbeel Pieter, and Levine Sergey. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. volume 80, pp. 1861-1870. PMLR, 10-15 Jul 2018.
Laurens van der Maaten and Geoffrey Hinton. Viualizing data using t-sne. JMLR, 9:2579-2605, 11 2008.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In ICMI., volume 70, pp. 3540-3549, 2017.
Bangpeng Yao, Dirk B. Walther, Diane M. Beck, and Fei-Fei Li. Hierarchical mixture of classification experts uncovers interactions between brain regions. NeurlPS, pp. 2178-2186, 2009.
Chao Yuan and Claus Neubauer. Variational mixture of gaussian process experts. NeurIPS, pp. 1897-1904, 2008.
Shangtong Zhang and Shimon Whiteson. DAC: the double actor-critic architecture for learning options. In NeurlPS, pp. 2010-2020, 2019.
10

---

Under review as a conference paper at ICLR 2021
A PROBABILISTIC FORMULATION OF PMOE AND GATING OPERATION
In this section, we show detailed comparison of probabilistic formulation for GMM (as Eq. 12 and [3) used in our PMOE method) and the gating operation method (Eq. (44 to (16 ) in term of their PDFs. The gating operation degenerates the multimodal action to a unimodal distribution, which is different from our PMOE method.
For GMM, suppose primitive (+) is Gaussian distribution ((alu(s), (+)). drawing a sample
from the mixture model can be seen as the following operation:
K
a a~ (als) = Σ
wi w;(s)T:((as))
(12)
1=1
where the PDF is:
(s) exp( (a pT(s)) (x)
p(a) = (s)
(13)
For gating operation, the outputs of the weight operation are the weights of each action from different primitives. With those weights, the gating operation uses the weighted action the final output action according to Peng et al. (2019):
w. as) s.t. a, ~ = (((lu)))()))
=
(14)
As primitive is a Gaussian distribution, Eq.
14 becomes:
K
K
a ~ ((a
w.
())
Σ (x)))
(15)
1=1
where the PDF is:
(a E K (s))
p(a) 2 2t Σ (s)
exp(
(16)
2 DΣ (s))(((())
The above PDF shows that the gating operation could degenerate the Gaussian mixture model into the univariate Gaussian distribution. Other methods (Jacobs et al. 1991a Peng et al. 2016 Vezhnevets
ct al. 2017) also have the similar formulation.
B PROOF OF FREQUENCY Loss
In this section, prove that the proposed frequency loss in our PMOE method produces consistent estimation of the gradients for optimizing GMMs. The mixing coefficient, also called the weight w, stands for the probability of choosing the optimal primitive.
Given the state st, the frequency arg max; (s a²) j being sampled is defined as fz. When we randomly choose N (close to infinity) samples with state s, from the replay buffer, the number of samples satisfying arg max, Q0(se, at) = j should be N fj while the number of those with arg max; Qo (St, at) * N(1 - fz). Suppose that we have loss function for wa, (6r) as:
L (ex) = [1 wa, (8t)) + N(1 f;){0 - wa, (s:))
[1 (1-00) wa, (se)) (1 f/3 (s:)]
(17)
NNI;, - 2f, (st) + w2 (st) + w3, (st) - f5 (si)
Niw (se) 2f, we, (st) fzl.
Then the gradient for B is:
V (wo, (o.) = Vo,NWu (iwaa(())) (st) - 2f (st) + fyl
= -N(V,, w6, (st) - Vo, wa, (st) 2N (st) f,) (8t);
(18)
Therefore, optimising Eq. 17 is the same as minimising the distance between wo, (s,) and fz. with the optimal situation as wo, (8) ʃz when letting the last formula of Eq. 18 be zero.
11

---

Under review as a conference paper at ICLR 2021
C DETAILS OF TARGET-REACHIN ENVIRONMENT
The visualisation of the target-reaching environment is shown in Fig the blue circle is the agent, gray circles are obstacles and the circle in red the target. The agent state represented by an action vector = [Ug; (y) and velocity vector = [Mg. vyy] The playground of the environment is continuous and limited in [-5, 5] in both x-axis and y-axis. The agent speed is limited into [-2, 2]. the blue coloured agent is placed at position [ag. Magl 1-4.5, -4.5] and the red coloured target is randomly placed at position [Ttg. Jtyl. where Tty. ~ (1(,,)) 3) and denotes uniform distribution.
There are gray coloured obstacles with each position Le Vohale where Tahss Vore ~ N (0,,3)) and A denotes Gaussian distribution. The observation is composed of [[xrg - Bag: Vtg -Yogl: {
Tag: Vos. Vagl: 1, M J. a, u/. The input action is the continuous acceleration a which the range of 2].
Figure 7: Visualisation of the target-reaching environment
The immediate reward function for each time step defined as:
100, the agent reaches the target;
10, if the agent collides with edges obstacles;
(19)
|o||a l2, otherwise.
D TRAINING DETAILS
For PMOE-SAC policy network, units and ReLU activation in each layer. For primitive network тg. we use a two single-layer FC network, which outputs μ and for the Gaussian distribution. Both the output layers for j and have the same number of units, which is K dim((A) with K as the number of primitives and dim((A) as the dimension of action space, e.g., 17 for Humanoid-v2. For the routing function network wa. use a single FC layer with K hidden units and the softmax activation function.. In critic network we use three-layer FC network with 256, 256 and hidden units in each layer and ReLU activation for the first two layers. Other hyperparameters for training are showed in Tab. 1(a). For PMOE-PPO we use two-layer FC network to extract the features of state observations. The FC layers have 64 and 64 hidden units with ReLU activation. The policy network has a single layer with the Tanh activation function. The routing function network has single FC layer with K units and the softmax activation function. The critic contains one layer only. Other training hyperparameters are showed in Tab. 1(b). We use the same hyperparameters all the experiments without any fine-tuning. For MCP-SAC, we the same network structure as MCP-PPO, other training hyperparemeters the same as shown in Tab. (a)) For other basclines, we use original hyperparameters mentioned in their paper. The full algorithm is summarised in Algorithm [
two-layer fully-connected (FC) network with 256 hidden
12

---

Under review as a conference paper at ICLR 2021
(a) Hyperparameters for PMOE-SAC
Parameter
Value
optimiser
Adam
Kingma & Ba 2015
learning rate
10-3
discount (7)
0.99
replay buffer size
106
alpha
0.2
batch size
100
polyak (r)
0.995
episode length
10
target update interval
(b) Hyperparameters PMOF-PPO
Parameter
Value
optimiser
Adam
Kingma & Ba 2015
learning rate
10
discount (r)
0.99
alpha
0.2
batch size
64
polyak (r)
0.95
episode length
2 10
gradient clip
0.2
optimisation epochs
20
Table 1: Hyperparameters
PROBABILITY VISUALISATION
We visualise the probabilities of each primitive over the time steps in the MuloCo HalfCheetah-v2 environment, As shown in Fig. S we found that the probabilities are changed periodically. We also visualise the actions at the selected time steps in one period. As shown in Fig. 9 the primitives are distinguishable enough to develop distinct specialisations.
1.0
Primitive Primitive 1
0.6
0.4
0.2
0.0
10
15
20
25
steps
Figure 8: Visualisation of the probabilities of each primitive over the time steps in the MuloCo HalfCheetah-v2 environment. The y-axis shows the probabilities of different primitives.
13

---

Under review as a conference paper at ICLR 2021
(a) time step
(h) time step 3
(c) time step
(d) time step
(c) time step 7
1.00
Primitive Primitive
0.75
0.50
0.25
0.00
0
2
4
8
10
Time steps
Figure 9: Visualisation of the actions at the selected 5 time steps in one period. The y-axis shows the probabilities of different primitives. This result shows that the primitives develop distinet specialisations, with the primitive 0 becomes the most active when the front leg touches the ground, while the primitive becomes the most active when the leg leaves the ground.
14

---

Under review as a conference paper at ICLR 2021
F T-SNE VISUALISATION
Ant-v2
HumanoldStandup-y
Humanoid-v2
HallChectah-ve
Class Class 1 Class2
Walker2d-v2
Prinitive Primitivé Prinitive Primitive
(a) Gating Operation
(b) State Observations
(c) Ours
Figure 10: We plot the t-SNE visualisation for other 5 MuloCo environments: An/-v2, HumanoidStandup-22. Humanoid-v2. HalfCheetal--2 and Walker2d-v2. Parameters and other details are the same as the setting mentioned in Sec 4.2)
G COMPARISON OF AUC
We compute the AUC (the under the learning curve) make the figure more readable.
15

---

Under review as a conference paper at ICLR 2021
For SAC-based experiments, assume the AUC of SAC is 1, and AUC values for all methods
shown in 2
Env
Walker
HalfCheetah
Humanoid
HumanoidStandup
Ant
Hopper
SAC
100%
100%
100%
100%
100%
100%
MCP-SAC
104.8%
103.2%
91.2%
98.2%
96.4%
100.5%
Gating
97.6%
95.1% %
84.4%
92.3%
89.3%
109.9%
PMOE-SAC (ours)
105.5%
99.6%
94.5%
113.3%
113.4%
115.4%
Table 2: Comperation of the AUC for the SAC-based methods.
For PPO-based experiments, we assume the AUC of DAC is 1, and AUC values for all methods are shown in Table
Env DAC
Walker
HalfCheetah
Humanoid
HumanoidStandup
Ant
Hopper
100%
100%
100%
100%
100%
100%
MCP-PPO
75.8%
158.6% 171.3%
159.5% 162.2%
116.2%
83.8%
93.5%
PMOE-SAC(ours)
101.1%
154.3%
93.3%
123.0%
Table 3: Comparison of the AUC for the PPO-based methods.
For the number of K experiments, we assume the AUC for PMOE-SAC with K = 2 is 1, so all the methods AUC values shown Table
Number of K
HumanoidStanduu
100%
4
107.1%
8
106.5%
10
111.7%
14
108.0%
107.0%
Table 4: Comparison of the AUC as function of K for PMOE-SAC algorithm HumamoidStandup- v2 environment.
H COMPARISON WITH OTHER OPTIMIZATION METHODS
As shown in Fig. 11 we compare our methods with the Gumbel-softmax and REINFORCE in one low dimensional MuJoCo task Hopper-v2 and one high dimensional MuloCo task HumanoidStandupp-22 For all the methods, we randomly choose 5 seeds to plot the learning curves.
I COMPARISON WITH PPO
As shown in Fig. 12 we compared with the PPO in the MuJoCo task HalfCheetah-v2. For all the methods, we randomly choose seeds to plot the learning curves.
J ROBUSTNESS EVALUATION
To evaluate the robustness our approach, we develop an experiment on the Hopper-v2 environment. We add random noise (0,00)) the state observation and use the noised state observation s + as the input of the policy. Our approach has a better performance in the noised input observation situation, which is shown in Table
16

---

Under review as a conference paper at ICLR 2021
160000
3500
Average Return
Average Return
3000
140000
2500
120000
GUMBEL INGE-SAC
GUMBEL
MCP-SAC
2000
MCP-SAC
REINFORCE
REINFORCE
100000
Gut-ng
SAC
SAC
Caling
PNCEE SAC
1508.0
0.2
0.6
1.0
0.00
0.25
0.50
0.75
1.00
Million Steps
Million Steps
(a) Hopper-v2
(b) HumanoidStandup-v2
Figure 11: Comparison with Gumbel-softmax and REINFORCE in the MuJoCo tasks Hopper-v2 and THumanoidSsandap--2. We found our method PMOE-SAC has a better performance.
4000
Average Return
3000
2000
DAC
MCP-PPO
1000
PMOE-PPO
PPO
8.0
0.2
0.4
0.6
0.8
1.0
Million Steps
Figure 12: Comparison with PPO in the MuJoCo task HalfCheetah-v2. We found our method PMOE-PPO has a better performance.
K COMPARISON WITH DIFFERENT PRIMITIVE NUMBERS AND DIFFERENT ENTROPY REGULARISATION
To analyse the relationship different K and different entropy regularisation(alpha)) we compared settings in the MuJoCo task HumanoidStandup-22 where K is the number of primitives and alpha is the entropy regularisation. For each setting, we randomly choose seeds to plot the learning curves in Fig. 13
Method
0
0.05
0.1
SAC
3387,4 = 2,0
1994.9 718.6
1006.2 389.6
gating
3444.8 3.1
2606.9 + 864.4
1626.0 + 771.9
MCP
3524.8 + 114.6
1610.2 = 357.0
1008.4 + 333.4
Ours
3632.2 1 4.0
3460.4 1 456.7
1730.0 703.1
Table 5: We test approach in the Hopper-v2 environment, each column stands for the average return with different variance the noise distribution, the average return of each methods is averaged over 100 rounds.
17

---

Under review as a conference paper at ICLR 2021
180000
160000
Average Return
140000
120000
K=4,alpha=0.2
K=10,alpha=0.2
K=4,alpha=1
100000
K=4,alpha=100
SAC,alpha
SAC,alpha=1
SAC,alpha=10
80000
0.0
0.2
0.4
0.6
0.8
1.0
Million Steps
Comparison with different K and different amounts of entropy regularisation. Our approach can be considered as kind of entropy regularisation method and the number of primitives positively correlated with the entropy of the policy. The larger number of primitives with smaller entropy has similar performance to the smaller number of primitives with larger entropy.
Figure 13:
18


Reviews:
[{"review_id": "y_tvFsu4CFj", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThe paper focuses on the policy architecture of deep reinforcement learning algorithms. Specifically, the authors apply the probabilistic mixture-of-experts (PMOE) model in the policy of a reinforcement learning agent, where each primitive is a unimodal Gaussian distribution and the gating model is a simple state-conditioned categorical distribution. The authors derive the corresponding policy gradient objective for the PMOE policy.\n\nThe authors apply the PMOE policy on top of SAC and PPO, and perform experiments on the continuous locomotion tasks in the MuJoCo environments. The results indicate that in some tasks, the PMOE policy outperforms the naive policy baseline. The paper also includes ablation studies and visualizations to demonstrate the diversity of the learned primitives of the PMOE policy.\n\n\n### Comments\nThe paper is well written and the idea proposed in this paper is really easy to follow. The authors also include a wide suite of experiments with both on-policy and off-policy RL algorithms to demonstrate the performance of the proposed method, and various ablation studies and visualizations to demonstrate the behavior of PMOE policy. Despite these advantages, I cannot recommend acceptance of this paper due to the lack of novelty and significant performance improvement.\n\nFirst of all, as the experiment results in this paper suggest, the performance gain of the PMOE policy is marginal and highly task-specific. Only in one of the 6 tasks the PMOE policy exhibits significant benefit over baselines. Therefore, from the scope of experiments in this paper, it is hard to conclude that PMOE policy really has meaningful advantages over a naive policy parameterization.\n\nMoreover, as described in the paper, the PMOE model is a fairly well-studied model, and it seems like the only contribution in this paper is the application of it in the reinforcement learning setting. I’m not convinced that such a straightforward application has enough contribution, especially given the fact that the performance improvement is not significant.\n\nTherefore, due to the lack of novelty and significant performance improvement, I cannot recommend acceptance of this paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "pA0Ld67ElCU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies the problem of differentiating through the policy return when the policy is a Gaussian mixture model. The main contribution of the paper is a heuristic approach for computing this gradient. Having defined the policy update, the authors integrate it to two RL algorithms: PPO and SAC. The experiments show that the algorithms with GMMs behave roughly the same as with a single Gaussian save for one (SAC) or two (PPO) environments. On the other side the authors show that their algorithm can learn multiple solutions on a reaching task and that exploration is better behaved on this task too.\n\nThe main problem I had with the paper is the lack of formalism in tackling such a fundamental problem. Given a parameterized GMM model, the contribution of the paper is to study the gradient w.r.t. the GMM’s parameters of an expectation, E[f(X)], where X is sampled from this GMM distribution. It seems like a fundamental problem and authors should discuss more thoroughly existing approaches to computing this gradient. For instance one could simply use the score ratio trick (i.e. REINFORCE) or use importance sampling as in PPO or perhaps a reparameterization trick (with Gumbel-softmax?) if applicable.  At least the two first one are perfectly valid alternatives and should be discussed and ideally compared to the current solution. \n\nIn addition it is hard to understand if the proposed solution is an approximation or a heuristic with no guarantees. The authors propose a proof in the appendix but there is no formal statement of what the proof is supposed to show. Is it that the solution is a consistent estimator of the gradient? \n\nIn Appendix A, I do not understand what the authors are saying. Somehow they go from a GMM (Eq. 12) to a MoE (Eq. 14-16), i.e. from summing densities to summing random variables. Since these two are very different, what ends up being used in the paper?  \n\nI am also confused by the role of the GMM compared to a single Gaussian policy. On one side, the GMM appears to find multiple solutions to an RL problem (Fig. 3), on the other side  the GMM appears to decompose a single solution into sub-policies that are active in different regions of the state-space (Fig. 9). But aren’t these two at odds with each other?\n\nRegarding the exploration advantage, is it known why a GMM should initially better explore than a unimodal Gaussian? Is it task specific or was it also observed on the Mujoco tasks?\n\nWhy was there no PPO baseline in Fig. 2? Please also clarify the number of seeds for experiments in Fig. 1 and 2. and discuss statistical significance, since the plots are all very close to each other.\n\nFor the Hopper cluster visualization, is the proposed algorithm able to learn different solutions? If that is the case, I think providing a video, or a sequence of images of two different solutions would be more impactful than Fig. 4.\n\nOverall, I think the paper needs significant changes to properly discuss the technical problem it tackles, discuss existing methods, and describe more formally the proposed solution including the theoretical statements provided in App. B. For the experiments, demonstrating that the algorithm is able to simultaneously find multiple solutions is interesting but should be extended to at least one of the highest dimensional locomotion tasks of the paper. For instance showing sampled trajectories for the Hopper as a replacement to Fig. 4 could be very encouraging. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "XJlSSILJ9Mh", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I would like to thank the authors of \"Probabilistic Mixture-of-Experts for Efficient Deep Reinforcement Learning \" for their valuable submission.\n\nSummary of the paper\n-\nThe paper proposes an end-to-end method to train probabilistic mixture-of-experts policies in RL agents. They show that the approach can be applied in the context of popular on-policy and off-policy algorithms, and that it compares favourably (performance and sampling efficiency) to using the same algorithms to train the corresponding unimodal policies. Furthermore they perform an empirically analysis of the individual resulting components and of the impact of these on exploration,\n \nAssessment\n-\n\n-- The positives --\n\nThe proposed approach is sound and seems to work well in practice.\nThe empirical evaluation is extensive - the fact that the paper evaluates the proposed approach in combination with different baseline algorithms makes the findings more robust. The analysis is overall quite insightful, especially in terms of understanding the diversity of individual components and the impact of backprop-max vs backprop-all.\n \n-- The concerns --\n\nThe paper notes that the mixture of experts seems especially beneficial in high dimensional problems (such as continuous control). This seems an important claim, but it is not clearly backed up. It would be useful to make this statement more quantitative by plotting the improvement in performance over the baseline as a function of the number of dimensions.\n\nThe paper notes that the mixture of experts might help by improving exploration. It would be therefore interesting to include a parameter study showing the performance of the baseline algorithm for different amounts of entropy regularisation. This would help to compare the proposed approach to a simpler way of tuning exploration, assess whether a mixture of experts delivers further benefits on top of this (e.g. by providing “deeper” exploration), and allow the reader to compare the sensitivity to the parameter K of the proposed approach to the sensitivity of the baseline to the weight of the entropy regularisation.\n \nSuggestions\n-- \n\n* Figure 6 could be made more readable by plotting a parameter study instead of a bunch of learning curves, e.g. plot AUC as a function of K.\n* It would be helpful for the authors to better discuss the relation, similarity and difference between the proposed approach and popular HRL approaches, in order to better assess the novelty of the method, and to ensure that it is placed in the appropriate context.\n\nFinally, the paper could use one more pass general pass to ensure the writing is fully correct and make it as readable as possible. Please also fix the following typos:\n- or without explicit probabilistic representation → the sentence doesn’t connect to the previous one\n- Is our method outperform? → does our method outperform?", "extra_scores": {}, "extra_text": {}}, {"review_id": "GtnQ1o6Mx4c", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose to use mixture models of policies and present good experimental results. The approach is quite sound, and the experimental results looks solid.\n\nHowever, there are some concerns that make this paper cannot be fully appreciated:\n1. The first and second listed contributions are about \"the undifferentiability problem\", and the third is a verification of the propose algorithm.\n    However, a more fundamental question would be the following ones:\n    A). Why the author would like to propose mixture models and hope the community to adopt this framework, to further enhance the DRL performance, right?    I agree the mixture models have great potential, but could you please justify your topic/target issue first, and then present your solution?   I would say, if the authors naturally expect the reviewer or readers to naturally believe in what you did, this is not right. Please show your intellectual contributions explicitly, and use them to convince readers.\n         Now, from a reader's perspective, mixture model is well-established, and this work extends it a little.  I agree mixture models are promising, but I am not convinced by this work.  How would the authors respond?\n         Since, it is some kind of extension, the novelty cannot be fully appreciated.\n    B). In terms of algorithm, is there some innovative contribution?  It is hard to tell, I think the authors would also agree that such steps are straightforward.\n         Therefore, will the experiments solely enough to justify an acceptance?\n    C). Regarding your experiments, besides the three questions raised by the authors (which are reasonable).\n          I would ask some questions beyond, would be mixture model be more powerful or practically important to address the robustness, stability issues that faced by current DRL algorithms?  (Q1 mentioned stability, while the figures do not fully address it, right?)\n          Also, as for mixture models, would an adaptive algorithms be more appropriate solution?   Since mixture model allows more freedom to balance exploration and exploitation, and an adaptive scheme would allocate exploration budget to more informative policies or state regions?\n          Therefore, the current experiments do not fully justify the value of this paper.  I have concerns.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies mixture of expert policies for reinforcement learning agents, focusing on the problem of policy gradient estimation. The paper proposes a new way to compute the gradient, apply it to two reinforcement learning algorithms, PPO and SAC, and demonstrate it in continuous MuJoCo environments, showing results that are comparable to or slightly exceeds unimodal policies. The main issue raised by multiple reviewers is novelty. Mixture of expert models have been widely studied in the context of reinforcement learning, and while the paper proposes a new method for the gradient computation, a more suitable format, as pointed out by Reviewer 2, could be to ground the paper around the proposed gradient estimator, and compare, both analytically and empirically, it to existing alternatives. Therefore, I recommend rejecting this submission.
Decision: Reject