Specializing Versatile Skill Libraries using Local Mixture of Experts
Authors: Onur Celik
Dongzhuoran Zhou
Ge Li
Philipp Becker
Gerhard Neumann
Venue: CoRL2021 Poster (2021)
Keywords: Episodic Policy Search
Versatile Skill Learning
Hierarchical RL
Curriculum Learning

Abstract: A long-cherished vision in robotics is to equip robots with skills that match the versatility and precision of humans.
For example, when playing table tennis, a robot should be capable of returning the ball in various ways while precisely placing it at the desired location. 
A common approach to model such versatile behavior is to use a Mixture of Experts (MoE) model, where each expert is a contextual motion primitive.
However, learning such MoEs is challenging as most objectives force the model to cover the entire context space, which prevents specialization of the primitives resulting in rather low-quality components. 
Starting from maximum entropy reinforcement learning (RL), we decompose the objective into optimizing an individual lower bound per mixture component.
Further, we introduce a curriculum by allowing the components to focus
on a local context region, enabling the model to learn highly accurate skill representations.
To this end, we use local context distributions that are adapted jointly with the expert primitives. Our lower bound advocates an iterative addition of new components, where new components will concentrate on local context regions not covered by the current MoE.
This local and incremental learning results in a modular MoE model of high accuracy and versatility, where both properties can be scaled by adding more components on the fly. 
We demonstrate this by an extensive ablation and on two challenging simulated robot skill learning tasks. We compare our achieved performance to LaDiPS and HiREPS, a known hierarchical policy search method for learning diverse skills. 

Specializing Versatile Skill Libraries using Local Mixture of Experts
Onur Celik', Dongzhuoran Zhou', Ge Li", Philipp Becker', and Gerhard Neumann'
Autonomous Learning Robots, KIT, Germany
{celik, ge.li, philipp becker, gerhard. neumann @kit. edu
Abstract: A long-cherished vision in robotics is to equip robots with skills that match the versatility and precision of humans. For example, when playing table tennis, a robot should be capable of returning the ball in various ways while pre- cisely placing at the desired location. A common approach model such versa- tile behavior is to use Mixture of Experts (MoE) model, where each expert is a contextual motion primitive. However, learning such MoEs is challenging as most objectives force the model to cover the entire context space, which prevents spe- cialization of the primitives resulting rather low-quality components. Starting from maximum entropy reinforcement learning (RL), we decompose the objec- tive into optimizing an individual lower bound per mixture component. Further, we introduce curriculum by allowing the components to focus a local con- text region, enabling the model to learn highly accurate skill representations. To this end, we use local context distributions that are adapted jointly with the expert primitives. Our lower bound advocates an iterative addition of new components, where components will concentrate local context regions not covered by the current MoË. This local and incremental learning results in a modular MoE model of high accuracy and versatility, where both properties can be scaled by adding more components on the fly. We demonstrate this by an extensive abla- tion and two challenging simulated robot skill learning tasks. We compare our achieved performance to LaDiPS and HiREPS, a known hierarchical policy search method for learning diverse skills.
Keywords: Curriculum, Versatile Skill Learning, Episodic Hierarchical RL
Introduction
Human motor skills are precise and versatile, which allows us to perform motor tasks in different ways while achieving consistent movement quality. For example, when playing table tennis, we can hit the ball various ways while still targeting a specific landing point of the ball on the oppo- nent's side of the table. Another example is grasping an object which lies behind an obstacle. We can grasp even small objects precisely with different grasp types while avoiding the obstacle. Such versatile skills are crucial if we want to employ robots in unstructured and dynamically changing environments. Such skills, often represented as movement primitives, were already successfully learned for challenging robot learning tasks, e.g., the ball-in-a-cup [1, 2] task, by variety of pol- icy search algorithms [3]. Yet, most of these algorithms cannot find multiple, versatile, and precise solutions to the multi-modal solution space, as they usually assume Gaussian policy [4, 2, 3]
In this paper, we model versatile behavior with contextual skill libraries of motion primitives [6, 7], formalized by Mixtures of Experts (MoEs). Here the context defines task properties, e.g., reaching different goal positions or different friction parameters of an object to manipulate [5]. Our goal to learn versatile skills, i... different movement styles to solve given context. Given context, the MoE first selects a component, i.c., a motion primitive, to exccute. Subsequently, the compo- nent adjusts the primitive's 's parameters and a controller executes the primitive. Such models are
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

already commonly used [8. 9]. Yet, the quality and versatility of the learned skill libraries remain limited using existing algorithms. Most algorithms are based on pppectatio-maximmzaae
(EM) [10] techniques [8], and suffer from local optima and mode averaging [10] which prevents the single components from specializing in local context region. Moreover, existing algorithms [8, 9] do not explicitly optimize the versatility of the library. Hence, they often yield degenerated libraries with only a single movement style. We propose a new objective for learning contextual, precise, and versatile MoE models based context distribution, which provides a curriculum for each component of the MoE model. We use a variational lower bound [11] to decompose the objective into individual updates for the components and their related local context distributions, allowing the components to specialize in local regions of the context space and preventing mode averaging. Due to the curriculum, the MoE does not have to cover the whole context space during training, which prevents the averaging effects that negatively affect most other approaches. Yet, not covering the whole context space leads to poor performance for some contexts, which is also undesirable. Thus, we introduce a heuristic-free mechanism to add new components during training until the whole context space is covered. Hence our algorithm pro- vides a modular approach that learns highly precise and versatile skills that cover the whole context space.
maximum entropy formulation. We also introduce a learnable
We evaluate our approach in a simulated beer pong and a table tennis environment. Both envi- ronments allow different motion styles to solve the tasks, which are discovered by our algorithm. Moreover, we present ablation studies showing the importance of the single elements and hyperpa- rameters of our algorithm.
2 Related Work
Contextual Episodic Policy Search. Episodic policy search [3] aims at maximizing the expected return by optimizing the parameters 0 of controller, c.g., a motion primitive [12, 6]. Most ap- proaches use stochastic search distribution π (0) over the parameter space and aim to optimize the expected return under this search distribution [3], i.c.,
max (e) [R(O)], (())
where R(O)= = E, Fr is the summed reward over whole episode obtained when executing con- troller parameter O. As it is common in the literature, we will denote (0) as our policy even though it only indirectly chooses the control actions of the agent by selecting the controller parameters 0. Different optimization schemes such as policy gradients Sehnke et al. [13], natural gradients [14], stochastic search strategies [15, 16] trust-region optimization techniques [8, 17] have been used. Researchers extended these approaches to the contextual setting [18, 4], where the search distribu- tion (0]c) now depends context vector which describes the task, e.g., a goal location to reach. The contextual objective is typically formalized as
max Ep(c) [E.(a
<(0cc
where p(c) is the given distribution over context vector and the rewards now also depends on the context. Klink et al. [2] introduce a curriculum into contextual policy search. By having an adaptable distribution over the contexts, they allow the agent to concentrate on easy-to- so-vve contexts first and then generalize to the whole context space.
Versatile Skill Learning. The Hierarchical Relative Entropy Policy Search (HiREPS) algorithm [8] extends the classical Relative Entropy Policy Search (REPS) approach [19] to MoEs, which allows learning versatile skills in contextual episodic policy search setting. In similar approach, Layered Direct Policy Search (LaDiPS) [9] also uses MoE policies, but builds on Model-Based Relative Entropy Policy Search (MORE) [17] instead of REPS. Both HiREPS and LaDiPS address the same problems as our approach, yet there are also considerable differences. First, HiREPS jointly optimizes the whole mixture model and introduces an additional constraint, which bounds the entropy loss of the responsibilities in each iteration. This constraint is crucial for obtaining versatile and well-performing solutions. LaDiPS uses separate updates for the different parts of the mixture but also relies on additional constraints, where the entropy of the gating is lower bounded with constant value. In contrast, for our approach, the objective and separate updates of the individual mixture parts follow naturally from the maximum entropy formulation. Second, neither HiREPS

---

nor LaDiPS uses a curriculum for training. Thus, in both approaches, the MoE always has to cover the whole context space and, hence, the components cannot specialize.
Variational Inference. Our algorithm is also related to several recent advances in variational in- ference [11, 20, 21J. It is well known that maximum entropy RL is equivalent to inference in an appropriate probabilistic model [22]. Similar to previous works [23, 24], we exploit this relation and draw inspiration from recent research into variational inference and density estimation for Gaussian mixture models and MoEs [11, 20, 21J. We reformulate the lower bound objectives introduced in those approaches for our maximum entropy RL setting and extend them with a curriculum for the mixture components.
Related Step-Based Approaches. In the step-based setting, the policy does not learn function from contexts to parameters of an episodic controller but directly maps from system states to actions and the policy updates are performed with the information from each time-step. Practitioners often use deep neural networks to parameterize step-based policies, giving rise to the field of deep RL. In this context, versatile policy learning is also very active research area [25, 26, 27, 28]. These approaches use a similar MoE model where the mixture component is only chosen at the begin- ning of an episode. Yet, the component is chosen randomly without conditioning on context or state variable. Moreover, these approaches reformulate a mutual information based objective into a maximum entropy objective while we develop more direct maximum entropy maximization.
Curriculum Learning. Researchers also worked on introducing curricula into deep RL. In a first approach Ghosh al. [29] proposed partitioning the initial state distribution using clustering. They then learn individual policies for each partition while keeping the partitioning fixed. Strictly, this is not a curriculum as the partitioning is not adjusted, yet it still allows specialization of the individual policies in different regions of the state space. To automatically generate and adapt curriculum for deep RL approaches, Klink et al. [30] extended their approach from the episodic setting [2] to the step-based setting. Yet, neither of these approaches addresses versatility. While we follow an episode-based approach, both methodologies have their benefits and limitations [3] which are, however, not the focus of this paper. We offer further discussion and quantitative comparison to a step-based approach for a common benchmark task in the appendix, therefore.
Options. A related hierarchical approach is the options framework [31, 33). The options frame- work extends the standard MDP to semi-MDP include temporal abstraction of low-level con- trol policies. Given a termination condition, the executed low-level policy can be terminated and another can be turned on. Our policy structure be seen simplification of the options frame- work where the option is only selected at the beginning of each episode. Yet, the options framework does not explicitly address learning versatile skills.
3 Specializing Versatile Mixture of Expert Models
To allow versatile solutions, we employ a Mixture of Experts (MoE) model as policy representation which given as <0|cc) E. न (olc)=(() o). where (olc) is the gating distribution, assign- ing a probability to component o given the context с and ( o) is the component distribution for component o, which adapts the motion primitive's 's parameters 0 to the given context c. In this section, we derive a lower bound to optimize each component and its corresponding context dis- tribution independently.It order to implement curriculum over the context c, we also introduce a leamed context distribution TCc)= = E. (CC)o)T(O), which is also a mixture model specified by the component-ve context distribution riclo) and the component weights (o). By applying Bayes' Rule to replace the gating distribution π (olc), we can now rewrite the general mixture of experts model as
T (0(c) = Σ T(ClO)T(O) T(c) (0)c,)))
(1)
This policy definition allows each component o to adjust its curriculum by explicitly optimizing for TC(o)) (Section 3.3) and thus concentrating on local region in the context space. We model the components T0lcc, o) a linear conditional Gaussian distribution and the component-wise state dis- tribution π (c () as a Gaussian. The prior weights π (o) define a categorical distribution. Throughout the next sections, we denote every probability distribution which is adaptable through the optimiza- tion process with æ as it is part of our policy π T00(c, cc. Furthermore we show the full derivations for the next sections in the Appendix A.
3

---

3.1 Maximum Entropy Skill Learning with Curriculum
We consider a maximum entropy objective [22] for episodic policy search, i.e.
max Epte) [Bxe) [R(c, 0)] + aH c)]]
=((c)
where p(c) is the task specific context distribution, R(c, O) is the reward function, H(x(()c)) = - [o x ( c) log (0)c)dd the entropy and (ffc) is our MoE model. The reward maximization enforces preciseness while the entropy bonus enforces versatility. However, the standard maximum entropy objective does not allow each mixture component to create its own curriculum, since an optimization over the component-wise context distributions <<clo) is not given. Inspired by the work from Klink et al. [2], we extend and modify the objective to
max En(e) [En(ole) [R(c, 0)] + aH (+ c]]] - 3KL (=(() || p(c)). x(()o),=(e)
(3)
where a and B are scaling parameters, (O) - x
(Olc) log ( (0lc)d0 is the entropy and m(e) de denotes the KL.-divergence. Note the difference in the op-
fc ((c) log plc) timization variables compared Eq. (2). The Kullback-Leible (KL) term ensures that the context distribution π (c) is close to the task specific context distribution p(c) while (c)) can choose to have low probability in regions of the context space where the MoE model is performing poorly. Note that this objective is similar to the negative I-projection of the joint distribution <(c, O) used in vari- ational inference. Here, we also exploit the properties of the I-projection for learning the context distribution - the I-projection is mode seeking instead of mode-averaging and therefore allows for specialization on a local context area. Yet, the given objective difficult to optimize for mixture models as the sum over the mixture components is appearing inside the log terms of the entropy and the KL. However, similar to [11], we can replace T (f(cc) in Obj. (3) with our mixture model and apply Bayes theorem to arrive at
KL (=(c) p(c))
augmented reward for component o
augmented reward for contexi distributions
max E ((oo, Bx(0lec,)) R(c, 0) + a log Tolc, 0) 3 llog(c)) + (B ax) log π (Clc) =(e,0)
(4)
+oE [[[T(Occ, + (ET( [H[[] + [H
The exact derivations are given in the Appendix A. Note that Eq. (4) is equivalent Eq. (3), yet, instead of the entropy for the whole mixture model, it now contains entropy terms for each hierarchy layer of the MoE model, i.c., H (m]]c, H [c(]]] and H (o)]. which are much simpler to compute. We also introduced log responsibilities (o,c, 0) = (0 c, ()lc)) and (o]c) = (Clo)) (o) (c),, occurring in the augmented rewards. They return high negative reward for component o, if the context-parameter pair (c, 0) or the context sample is already covered by another component, pushing the component to uncovered regions of the parameter space or context space respectively. Yet, the regions for the components will still overlap due the entropy bonuses for π (0 c, o) and (clo). Without this reward augmentation, each component could be optimized completely independently in Eq. (4) using a max-entropy objective. However, in this case, all components would concentrate on learning the best solution irrespective of whether this solution has already been covered by another component. Yet, the log responsibilities still hinder us from optimizing each component x (f)c, o) and its corresponding context distribution <C]o) independently, since the sum over from the mixture models Tcc) л (0|c) respectively appears in the log term. In the following sections we show, how we can overcome this limitation by introducing lower bound inspired by variational inference [1 1]. As we consider each possible context as equally important, p(c) is assumed uniformly distributed in a given interval in the following and thus, can be neglected in the objective.
3.2 Lower-Bound Decomposition for Component Distributions
In order to maximize the objective in Eq. (4) for each component л (0fcc o) individually, we can first extract the terms which only depend on π (6 ccc o) for a specific o as
max E Ex (x(c))),,(0))..)) R 0) + log (olc, 0)] + a(Er(clo) [0|c,,
(5)
(0|c.o)
The responsibilities are still hindering us to optimize each component =(0lc,o) independently. However, similar to [11], we can obtain tight lower-bound by introducing a variational distribution

---

(olc, 0) and replacing the responsibilities in Eq. (5) with 7 ( 0). This variational distribution is fixed during the optimization and can be computed according to the last policy model allowing us to update each component independently. It is easy to show that after the update of the variational distribution #(olc, 0) = (olc, f), the introduced lower bound is tight. Please refer to the ap- pendix. The resulting lower bound is a standard maximum entropy RL objective with an additional reward augmentation of a log a(o|c, 0) and thus can be optimized with any suitable Policy Search algorithm. Here we use maximum-entropy-adee
version of contextual MORE [18].
3.3
Lower-Bound Decomposition for Context Distributions and Prior Weights
To update T ((c(o) for a specific o, we extract the relevant terms from Obj. (4)
max E (cloo) [Le(o, c) + (3 a:) log A [o/cc]] + ,H (I<e))))
(6)
*(e|o)
where c)) = (o)c,o))) [R(c, 0) + log (ole, 0)] + [I(0]c, o)] corresponds the expected augmented maximum entropy objective of component (0]cc, o) in context and ((occ) Tot((o/c) is a second variational distribution, which we introduced to be able to optimize for each x (c|o) individually. Similarly to the previous section, the objective given in Eq. (6) is tight lower bound to the original objective where the responsibilities π (olc) are used instead of ((occ). We approximate the integral over 0 with single sample, as typically only have a single parameter sample per context available. Yet, as we still have the outer expectation Ex(ejo) in Eq. (6) which we approximate by multiple context samples, the whole Monte- Carlo estimation of the expectations is still unbiased and with low variance. After the optimization step (Eq. 6), we obtain the optimal solution T (clo) and tighten the bound by setting (o/cc) I (olc) and (olc,, 0) = x- (olc, 0). Again, this lower bound corresponds to an augmented maximum entropy RL objective and thus we can be updated it with any suitable policy search method. Like [11], we use an adjusted version of MORE [17].
Finally, we can formulate the objective for updating the component weights (()), which resembles a lower bound of the original Objective (4) and corresponds to the highest hierarchy in our update scheme. The objective is given as
Σ (() [E>(c))) [L(O, + (3 a) log [(lc)]] + (H + SH (T(O)),
max x()
which is a maximum entropy RI. objective for categorical distributions. Here, we use REPS [19].
To summarize, we split the initial objective in Eq. (4) into different hierarchies, allowing us to opti- mize the different terms in our mixture model individually, Starting by first updating the components T c, o) using the maximization problem in Eq. (5), can optimize for the component wise con- text distributions π (clo) with Obj. (6) after tightening the bound. The components π ( c, o) adjust the movement primitive parameters A given context c, while the component-win context distri- butions π (c)o) ensure that the components only see context samples from a local context region. Finally, we update the weight distribution =(0) using Eq. (7).
Algorithmic Details and Addition of Components. We initialize our algorithm with only one component and incrementally add components, and their corresponding context distributions, ran- domly. We fix all components except for the newly added one and optimize it for K iterations to let it discover new solutions in yet undiscovered context regions. For updating T (0)c,, o) (Obj. (5)) we sample from the local context distribution π (clo). By also updating π (cco)) according to Obj. (6), the components can adjust their curriculum and search for their favored context regions. After K iterations we add a new component and repeat the procedure. Due to the augmented rewards, the new component will focus on undiscovered solutions and the local context distribution will cover uncovered areas of the context space. Note that such simple adding procedure is only possible due the mode-seeking properties of the I-projection, over multiple modes but can specialize on local context region. We also fix the weights (T() to be uniformly distributed among all components during learning, since otherwise components which are already fully trained might dominate the optimization. By allowing to fine-tune all components every H iterations, the previously added components can adjust to the newly added ones. After finishing adding components, we update the weights (o) at the end of our optimization procedure. The variables K and the number of components added in total are desk-dependent. We describe the algorithm in more detail in the Appendix B.
the components do not need to average
5

---

(a) aug. reward, 10
(b) aug. reward, 0.5
(c) aug. reward
(i) no aug. reward the responsibilities the augmented rewards and comparison to HiREPS. A snapshot of the learned policies of the planar reaching task for a 2-dim context space (il- lustrated in (c) and (f)) by considering log (olc, 0). log (olc) ((a)+()+(C)) and neglecting the aux- iliary distributions ((d)+(f). In ((a)+()+(()) we can learn more diverse solutions (a) + (b) and cover the whole 2d-context space (c) with different entropy bonuses, whereas the solutions in ((d)+(I) show nearly the same, partially invalid solutions by going through the red rectangles (obstacles) (d) and are not able to cover the whole context space (f) leading to poor generalization performance. The solutions by HiREPS (e) indicate less versatility compared to the solutions (a) + (b). Note that each color indicates a different component and the red dots indicate the 6 chosen context vectors used for sampling.
(d) no aug. reward
(e) HiREPS
Figure Importance
4 Experiments
We start by investigating the importance of the different terms of the objectives derived Section 3 and subsequently evaluate the versatility and precision of the learned skills on simulated robot beer pong and table tennis experiments. Appendix we report all hyperparameters and provide comparison and analysis to step-based policy search strategy (PPO [34]) in different scenarios.
4.1 Ablation Studies
We investigate the importance of the augmented rewards on a 10-link planar reaching task with two- dimensional context space. For this purpose, we update the component distributions π T( c, o) and the corresponding context distributions (Cclo) by optimizing the objectives in (5, 6) with i) consid- ering the responsibilities (as given by our algorithm) and ii) by setting log (olc, 0) and log (olc) to zero. Furthermore, we compare the solutions found by method the solutions found by the SOTA method HiREPS.
Versatility is Induced by the Augmented Rewards. In the 10-link planar reaching task (see Figure 1a) the robot has to reach the red dots with its end-effector in 2-dim context-space, while avoiding the rectangle-shape shapee obstacles. By adding 60 components, we have trained both versions (i and fi) over 15 seeds/trials and have chosen the best performing parameter constellations (Fig. la) for D, Fig. 1d) for ii)). We then picked the first model and sampled for each of the shown six contexts vectors (red dots) 100 samples and plotted the corresponding mean each sampled component. The plots for i) (Fig. la) + Fig. 1b)) show versatile several modes to the same context and precise small distance of end-effector (green dot) to goal (red dot), while avoiding obstacles- solutions while fully covering the context space (Fig. 1c). A trend of covering more modes with higher a can also be seen in (Fig. 1b)). For the case without the responsibilities ii) (Fig. Id)), the solutions are not precise and invalid -reaching through the obstacles-. As Fig. If)) shows, the context distributions focus on easy-to- to-yvlvee context regions (top and bottom part) and do not cover the full context space, leading to extrapolated solutions from components that are not trained for these contexts (red dots).
We pick the best performing model among 15 seeds/trials after hyperpa- rameter optimization. In Fig. le) the mean solutions of the sampled components -sampled in the same way as before can seen. HiREPS shows less diverse and qualitative solutions, where only 3 of the 60 components were chosen in total.
Comparison HiREPS.

---

Mian Rewards
Moan Rowals mls
2
HIREPS
-13
Ppicatic Surephec
Nonther Spledd: Surples
(a) Rewards- BP
(b) Rewards- TT
(c) Context- Context-Space BP
(d) Context- Santext-Space TT
13
42
(e) Successes BP Ours
(1) Successes BP HiREPS
(g) Ball Trajectories BP
(h) Environments
Figure 2: Beer Pong (BP) and Table Tennis (TT) Experiments. In the BP experiment -left of (h)- the robot has to throw the red ball into a cup the table. In the TT experiment -right of (h) the robot hast to return the ball to the opponent's 's table side. The learned mixture of expert (MoE) models show high quality of the skills, reflected in the high reward values (a + b) and dense coverage of the context spaces, i... the 2-dim. position of the cup (c) for BP and the 4-dim desired outgoing landing and incoming landing ball positions (d) for TT. For BP we also compare the success rates (92% in average) of our approach (c) and HiREPS (75% in average) (f) throughout the context space, where consider trial successful if the ball goes into the cup. For BP versatile skills induce versatile ball trajectories for given context (g). Versatile strikes for the TT experiment are shown in Fig. 3)
4.2 Simulated Robotic Experiments
We test and compare algorithm simulated Beer Pong and Table Tennis environments in Mu- JoCo [35], where we have chosen HiREPS [8] and LaDiPS [9] as baselines. Both methods are suitable baselines since they contextual policy search algorithms and consider optimizing a mix- ture model. Throughout our experiments we choose probabilistic movement primitives (ProMP) [6] as policy representation, where depending on the weights O desired trajectories are generated and subsequently tracked with a PD-controller. In our experiments, given context c, the components (0 c, o) adjust the weight vector and the length of the trajectory, which are summarized in the vec- tor O. We consider non- markovian rewards. in which the reward depends on the history of state and actions. This type of reward function is not applicable to common step-based RL methods which build on markovian properties. As for analyzing the results of the experiments, we focus on the questions how does our algorithm perform compared to SOTA basclines, ii) are we able to cover the whole context space, and iii) are we able to learn versatile skills?
The goal of the Barret WAM robot is to throw the red ball into the cup the table. The 2-dim contexts resemble the position of the cup on the table. We incrementally add 70 com- ponents in our method, while HiREPS and LaDiPS directly start with 70 components. We have run all algorithms over 20 seeds/trials and report the performance in Fig. 2a)², where we plot the mean reward with two times the standard error. While HiREPS already converges after around half a million samples, our approach can quickly outperform it. Since LaDiPS uses intra-option learn- ing, it outperforms our method in terms of sample efficiency. However, with the increasing number of components, we achieve a higher end-reward indicating that we can learn more qualitative so- lutions. Our algorithm allows to cover the whole context space with the learned component-wise distributions (see Fig. 2c)) resulting in high end-reward. This performance is reflected in Fig. 2e), where we have divided the context space into a fine grid and sampled 100 times for each context a component from which we have executed the mean. We repeated that for all 20 different models and plot the mean success rate of throwing the ball into the cup. We did the same procedure for
Beer Pong.
'To reflect the model's true performance, the lastly added component was excluded from testing, since it is not fully trained yet and would not be chosen by the model if (T()) would not be uniform-distribeene distribution.
7

---

Figure 3: Versatile Strikes for the Table Tennis (TT) Experiment illustrated for fixed context. The robot can hit the (yellow) ball in various ways, also indicated through the different colors the racket sides. Note that the red and yellow dots the table markers for the serving and desired landing position respectively.
HiREPS in Fig. 2f). We can observe that HiREPS has much darker rectangles, showing that the success rate in these context regions is low. Although versatility is encouraged in the joint space of the robot, different joint trajectories often yield different ball trajectories. Given one context, in Fig. 2g) we show the z-coordinates of the ball trajectories over time resulting from sampling 20 times from the MoE model. Each component leads to a different number of "ball-jumps". In Appendix C.2.1 we qualitatively show that we can learn more versatile solutions than LaDiPS and report that can achieve much higher expected mixture entropy (Fig. 4), which indicates that solutions are versatile.
Table Tennis. The task of the robot is return different incoming balls to desired targets on the opponent's table side different ways. We consider a four dimensional context space, including ball's initial serve position and ball's target landing position (right and left half of table Fig. 2d)). For both parts of the context, we fix the position and vary the x and y coordinates. We incrementally add 50 components in method, while HiREPS and LaDiPS directly start with 50 components. We have run all algorithms over seeds/trials and report the performance Fig. 2b)', , in which the mean reward with times the standard error is plotted. The component- wise context distributions are spread among the context space (Fig. 2d)) and allow each component to locally specialize on context region. This high coverage of the context space allows for high reward achievement, outperforming HiREPS and LaDiPS. In Fig. 3), we show three different striking skills sampled from our trained MoE model, for fixed context, i.e., fixed serving and desired ball position. The first two skills use the green side of the racket to hit the ball (forehand), while the third skill uses the red side of the racket (backhand) to hit the ball. In contrast the second strike, the first one performs smash-like strike and ends it with the red side of the racket pointing to the camera.
Conclusion
We proposed a new objective for learning contextual and versatile Mixtures Experts (MoE) mod- els. We based our objective on maximum entropy formulation to increase the versatility of the solutions and introduced a curriculum to allow the components to specialize. Our formulation also allows for easy online adaptation of the model complexity during training. We conducted ab- lation to show the importance of the individual parts our objective. Further, we showed that method learns precise and versatile solutions and outperforms the baseline sophisticated simulated robotic tasks. This work aims at presenting mathematically well-founded method and demonstrates its general feasibility on various challenging tasks. Currently, the major drawback of our approach is sample efficiency. as we do not share experience between the components. We intend address this issue in future work, e.g., by intra-option learning. Another direction for fu- ture research extending the approach to more complex models, such as non-linear mixtures of experts. We expect to need fewer components cover the whole context space with more complex component model representations.

---

Acknowledgments
Calculations for this research were conducted on the high performance computer of the state of Baden-Württemben Württemberg
References
[1] Kober and J. Peters. Policy search for motor primitives in robotics. In Learning Motor Skills, pages 83-117. Springer, 2014.
[2] P. Klink, H. Abdulsamad, B. Belousov, and J. Peters. Self-paced contextual reinforcement learning. In Conference on Robot Learning, pages 513-529. PMLR, 2020.
[3] M. P. Deisenroth, G. Neumann, J. Peters, et al. A survey on policy search for robotics. Foun- dations and trends in Robotics, 2(1-2))388 403, 2013.
[4] A. Abdolmaleki, D. Simöes, N. Lau, L. P. Reis, and G. Neumann. Contextual direct policy search. Journal of Intelligent & Robotic Systems, 96(2):441-157, 2019.
[5] A. Kupesik, M. Deisenroth, J. Peters, and G. Neumann. Data-efficient generalization of robot skills with contextual policy search. In Proceedings of the AAAI conference on artificial intel- ligence, volume 27, 2013.
[6] A. Paraschos, C. Daniel, J. R. Peters, and G. Neumann. Probabilistic movement primitives. Advances in neural information processing systems, 26:2616-2624 2013.
[7] A. J. Ijspeert, J. Nakanishi, and S. Schaal. Learning attractor landscapes for learning motor primitives. Technical report, 2002.
[8] C. Daniel, G. Neumann, and Peters. Hierarchical relative entropy policy search. In Artificial Intelligence and Statistics, pages 273-281. PMLR, 2012.
[9] F. End, R. Akrour, J. Peters, and G. Neumann. Layered direct policy search for learning hierarchical skills. pages 6442-644) 05 2017. doi: 1A0.000000/CCRA222777997766
[10] C. M. Bishop. Pattern recognition and machine learning. springer, 2006.
[11] O. Arenz, G. Neumann, and M. Zhong. Efficient gradient-free variational inference using policy search. In International conference on machine learning, pages 234-243. PMLR, 2018.
[12] Schaal, J. Peters, Nakanishi, and A. Ijspeert. Learning movement primitives. In Robotics research. the eleventh international symposium, pages 561-572. Springer, 2005.
[13] F. Sehnke, C. Osendorfer, T. Rückstiess, A. Graves, J. Peters, and Schmidhuber. Parameter- exploring policy gradients. Neural Networks, 21(4):551-559, May 2010. 1o1:10.1016/..nemmett 0009.12.0044
[14] D. Wierstra, T. Schaul, T. Glasmachers, Y. Sun, J. Peters, and Schmidhuber. Natural evolu- tion strategies. The Journal of Machine Learning Research, 15(1):949-995 -800, 2014.
[15] N. Hansen and A. Ostermeier. Completely derandomized self-adaptation in evolution strate- gies. Evolutionary computation, (12):159-195, 2001.
[16] S. Mannor, Y. Rubinstein, and Y. Gat. The cross entropy method for fast policy search. In Proceedings of the 20th International Conference Machine Learning (ICML-03), pages 512-519, 2003.
[17] A. Abdolmaleki.
R. Lioutikov. N. Lua, L. Paulo Reis, J. Peters, and G. Neumann. Model- based relative entropy stochastic search. In Proceedings of Advances in Neural Information Processing Systems (NIPS), pages 153-154, 2015.
[18] V. Tangkaratt, H. van Hoof, S. Parisi, G. Neumann, J. Peters, and M. Sugiyama. Policy search with high-dimensional context variables. Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.

---

[19] J. Peters, K. Mulling, and Y. Altun. Relative entropy policy search. In Proceedings tfe AAAI Conference on Artificial Intelligence, volume 24, 2010.
[20] O. Arenz, M. Zhong, and G. Neumann. Trust-region variational inference with gaussian mix- ture models. Journal of Machine Learning Research, 21(163):1-60, 2020.
[21] P. Becker, O. Arenz, and G. Neumann. Expected information maximization: Using the projection for mixture density estimation. In International Conference on Learning Represen- tations, 2020. URL /tttps://openreviees www./eet/ooooee Fowooommmiiiiir Bid-BBgLLLFDDS..
[22] S. Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprini arXiv: /80500099 2018.
[23] G. Neumann et al. Variational inference for policy search in changing situations. In Proceed- ings of the 28th International Conference on Machine Learning, ICML 2011, pages 817-824, 2011.
[24] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off- maximum tropy deep reinforcement learning with a stochastic actor. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference ceedings of Machine Learning Research, pages 1861-1870. PMLR, 10-15 Jul 2018. URL http: http:///proceedngss.. mlr wwwppprssssviseee http///////aaaj/eeeaaee ja8bb html.
Machine Learning, volume 80 of Pro-
[25] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. CoRR, abs/1802.06670, 2018. URL http://arxiv, org/abs/
[26] S. Kumar, A. Kumar, S. Levine, and C. Finn. One solution is not all you need: Few-shot extrapolation via structured maxent rl. arXiv preprint arXiv:2010.14484, 2020.
[27] T. Osa, V. Tangkaratt, and M. Sugiyama. Discovering diverse solutions in deep reinforcement learning, arXiv preprint arXiv:2 00.22103.00 103.07084, 2021.
(28) V. Campos, A. Trott, C. Xiong, R. Socher, X. Giró-i Nieto, and J. Torres Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning, pages 1317-1327. PMLR, 2020.
[29] D. Ghosh, A. Singh, A. Rajeswaran, V. Kumnr, and S. Levine. divide-and-conque rein- forcement learning. In International Conference on Learning Representations, 2018. URL net. htt///orummm/ -wwerweebb----------..--
[30] P. Klink, C. D' Eramo, J. R. Peters, and J. Pajarinen. Self-paced deep reinforcement learn- ing. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Ad- vances in Neural Information Processing Systems, volume 33, pages 9216-9227, Curran Associates, Inc., 2020. URL https://proceedingg wew.ess.eerrs... wctpp/pee////////////.. .dff.
[31] R. S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for tem- poral abstraction in reinforcement learning. Artificial intelligence, 111111):1 1999.
[32] P.-L. Bacon, J. Harb, and D. Precup. The option-critic architecture. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[33] M. Riemer, M. Liu, and G. Tesauro. Learning abstract options. arXiv: /583, 2018.
arXiv preprint
[34] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017.
[35] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026- 5033, 2012.
[36] G. Brockman, V. Cheung, I.. Pettersson, Schneider, Schulman, J. Tang, and Zaremba. Openai gym, 2016.
10

---

[37] A. Raffin, A. Hill, M. Ernestus, A. Gleave, A. Kanervisto, and N. Dormann. Stable baselines https: wesee-baaeeeeeeee ines3, 2019.


Reviews:
[{"review_id": "g22CXzgMuVQ", "reviewer": "Reviewer_Kbdi", "summary": "The paper goes after two issues\n1) How to learn to handle context (parameter) changes in the learning tasks, such as a variety of goals or friction values.\n2) How to learn to execute different strategies to do the same thing (use a forehand or a backhand swing).\nThe suggests a mixture of experts approach. A more useful terminology emphasizes that policies are composed of classifiers that\nselect \"simple\" policies depending on the context and desired strategy. I would suggest the term \"classifier-based policies\" or\n\"selection-based policies\". Mixture of experts often don't have a classifier and use weighted blends of the experts' outputs, which\nis a very different thing.\n\nKey ideas include:\n1) Develop a curriculum to support learning the classifier and the component policies. This helps avoid interference during learning, as\nwell as getting good coverage.\n2) Control the scope of the component policies by optimizing the worst case for each component.\n3) Optimize the classifier and the component policies simultaneously (vs. in some EM-like alternation).\n4) Iteratively add components to improve coverage.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nI am very happy to see a paper on learning multiple \"strategies\" to do the same thing, as I think this is a very important issue\nfor skill learning.\n\nWeaknesses:\n\nIt is easy to do this kind of stuff in simulation. Not so easy on real robots. Not having an implementation of something on a real robot\nis a big weakness.\n\nThe paper promises \"two challenging simulated robot skill learning tasks\". Implementations of both of these tasks have been done\non real robots\nin the past. Ball throwing is not that hard (see 1988 paper\nhttps://www.researchgate.net/publication/37596892_Model-Based_Robot_Learning). Robot ping pong (table tennis) is challenging,\nbut perhaps the major part of the challenge is perception rather than action. Lots of people have implemented robot ping\npong players. Search for \"robot ping pong table tennis players\" on Youtube. I would back off \"challenging\" unless I was doing some\ntask that really hasn't been done many times before on a real robot. Or demonstrate real robot performance on a larger range of tasks\nwith appropriate generalization across tasks: Simultaneously learn: tennis, squash, badminton, paddle ball, racquetball, table tennis, soft tennis, platform tennis, beach tennis, smash ball, hand tennis, basque pelota, ...\n\nDid the paper show suitable generalization across contexts? Was interference avoided? How was this evaluated?\nAt the end it says \"Currently, the major drawback of\nour approach is sample efficiency, as we do not share experience between the components.\"\nWhat about between contexts when the same option is selected? When a different option is selected?\n\nThe motivation and notation for Equation 1 is not clear.\nGiven c, the option o is selected by the agent. This function is known, so why not use it directly instead of using Bayes rule to replace it.\npi(theta|c) = sum_o pi(theta|c,o) p(o|c)\nBayes rule applies to probabilities such as prob(o|c)prob(c) = prob(c|o)prob(o).\nTo me, the policy pi(o|c) is clearly prob(o|c).\nThe mention of Bayes' Rule in the derivation of equation 1 implies\npi(c) is not meaningful, prob(c) is, and has nothing to do with the agent. So how is pi(c) not equal to prob(c)?\npi(c|o) and pi(o) are not meaningful either, but prob(c|o) and prob(o) are and can be computed from pi(o|c) = prob(o|c), which is under\nthe control of the agent.\nClearly I am confused.\n\nI don't understand why an entropy bonus is useful in the case where c has nearly irrelevant components. Won't this favor adding  unnecessary theta variance or irrelevant component policies? A useful regularization in reinforcement learning is to penalize complexity.\nCan you explain why the entropy bonus is useful, and what practical effect it has.\n\nI don't see why pi(c|o) or pi(o) (really the probabilities, that are only under indirect control by selecting pi(o|c)) are useful.\nThe difference between pi(c) and probability(c) is not clear. I shouldn't have to read Kink et al. [2] to understand it.\npi(c) is presumably the probability of c, and is not under control of the agent, so the notation should be p(c) or prob(c).\nWhat is the meaning of pi(c|o)? The option selected has no effect on the actual context. Perhaps you should introduce a context\nestimate hat(c) = sum_o pi(hat(c)|o)pi(o)\n\nThe distinction between trial-based and time-step-based learning algorithms was not clear. Perhaps \"step-based\" is too abbreviated in\n\"Related Step-Based Approaches. In the step-based setting, the policy does not learn a function\nfrom contexts to parameters of an episodic controller but directly maps from system states to actions.\"\nI might try to make the distinction clearer by emphasizing the difference between training policies\na) by adjusting a function from a context or goal to a small number of policy parameters at the end of trials (trial-based? episodic?)\nvs.\nb) by adjusting a function from states to actions at each time step of execution (time-step-based?)\n\nFor parametric component policies, I don't think there is a fundamental distinction between episodic and time-step-based approaches, especially\nwhen there are no episodes (life-long learning).\n\nIt might be useful to make a connection to \"options\" as described by Sutton.\n\nGiven the tasks chosen are deterministic (at least when done with real robots),\nit may also be useful to consider the deterministic model-based case, where selector() and pi_option() are deterministic.\npi(action|state) = Sum( selector(state)*pi_option(state) )\nIf the selector is one for one option at a time, this selects an option.\nAnything else blends options. Time-step-based policies (options) can\nbe learned in the same way as episodic policies. This leads to much\nsimpler gradient learning rules than the stochastic case.\nwww.cs.cmu.edu/~cga/papers/acc2012-tr.pdf\n\nHow good are the lower bounds, and how much does it matter?\n\nThe paper promises \"unprecedented accuracy and versatility,\" How is this demonstrated? Too much hype?\n\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "How to represent multiple strategies to do the same task (same initial conditions and goal) is an important question.\nIf multiple strategies were demonstrated on a real robot, that would be very interesting.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "See the issues discussed above.\n\nClearly explain (so someone could actually write code) a simple example like the multi-joint reaching.\n\nImplement behaviors on a real robot.\n\nIn the wonderful world of open review, all the submitted papers are\navailable to the authors. There are at least 3 \"generate and test\"\npapers:\n190 Specializing Versatile Skill Libraries using Local Mixture of Experts\n98 Data-efficient learning of object-centric grasp preferences\n89 Hierarchical Policies for Cluttered-Scene Grasping with Latent Plans\nYou should take advantage of this opportunity to put your work into\na larger context (and take a look at the reviews these papers got.)\n"}}, {"review_id": "TNWchdXC_Mh", "reviewer": "Reviewer_vGwW", "summary": "This paper proposes a novel objective function for learning a Mixture-of-Experts (MoE) model to solve robot motion tasks. The objective function uses maximum entropy to encourage diversity in a collection of learned motion primitives, which are called \"components.\" An interesting aspect of this paper, in comparison to existing work, is that the motion primitives are added and learned sequentially, with the objective function encouraging each one to differentiate from the others, thereby covering the entire space of \"contexts\" (i.e., different variations of the same task). The method outperforms the existing HiREPS method in simulation on difficult tasks, justifying the underlying math that went into creating the objective function.\n\nUpdate 1 Sept: I am more confident in my \"accept\" recommendation now. I'll leave it as a \"weak accept\" because, per the authors' comments, it seems I'm not super knowledgeable about this area. But, I think the general consensus on this paper is an \"accept\" with the understanding that it's interesting and good, but maybe not mind-blowing or completely revolutionary (but still good!).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS\n- I love the idea of local context distributions that adapt and grow; this feels a lot like human learning, and it makes sense that the approach would be very versatile.\n- I really appreciate the clear statement of strengths/weaknesses of the related methods\n- It is clear the authors put effort into stepping the reader through their thought process\n- The comparison to HiREPS seems appropriate and well-justified\n\nWEAKNESSES\n- The word \"versatility\" in the title is never formally defined, so the assessment of versatility seems entirely qualitative, but it would ideally be quantitative\n- I feel that the word \"context\" is a bit misleading when considering what is meant by a context in this paper; here, contexts are limited to a continuum of task properties (e.g., goal position or friction parameters). I would generally imagine contexts to be a much higher-level abstraction, such as \"kitchen\" versus \"living room.\" But, perhaps this is just a part of the MoE vocabulary with which I'm not familiar?\n- Some justification for claims in the paper are in the appendices, but I think it would be better if they could be brought into the paper; I've made note of them below", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am putting \"weak accept\" because I am not super familiar with mixture-of-experts or skill library learning. I think this paper is pretty good, and just needs some clarifications to be acceptable. But, if it's already been done or other more knowledgeable reviewers have serious problems with it, then I understand rejecting it.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "Abstract\n- It would be helpful to note any comparison methods in the abstract.\n\nIntroduction / Related Work\n- The claim that \"most of these algorithms... usually assume a Gaussian policy\" needs some citations to back it up; I think [2] in the paper is the relevant one?\n- For the proposed method, depending on the dimension of the context space, it would take an exponential number of local regions to effectively cover it. How can we avoid this blowup? I guess removing suboptimal solutions helps out.\n- I would appreciate a sentence that clarifies the \"ad hoc constraint\" for HiREPS and LaDiPS (i.e., just add more details explaining the constraint)\n\nSpecializing Versatile Mixture of Expert Models\n- The terms R and H in the maximum entropy aren't defined (one can figure out that R is reward and H is entropy, but it would be good to state it explicitly)\n- It would be helpful to number all equations for the sake of reviewers :)\n- On line 149, the text refers to \"log terms of the entropy and the KL\" but these log terms haven't appeared explicitly yet (i.e., the entropy and KL terms should be expanded before (3) so it's clearer for the reader)\n- Why is it reasonable to approximate the integral over $\\theta$ with a single sample? I feel like this statement needs further justification\n- How does one know when a component is sufficiently trained before adding a new one? How does one know that enough components have been added? While these questions may be answered in Appx. B, it would be helpful to at least briefly answer them in the main text.\n\nExperiments\n- In Fig. 1, I presume that the dotted lines are the 10-link robot. What are the red boxes? I guess obstacles? (these questions are answered in the main text around line 228, but should be explained in the figure caption as well)\n- I'm confused by how the beer pong task shows versatility; I think it's best shown by the way that the robot bounces the ball multiple times. But it feels like true versatility would be for a single MoE model that learns _both_ beer pong and table tennis, or something really wild like that. I realize that I'm asking for a lot, but I think that the paper should have a much more precise statement about what \"versatility\" means (i.e., it learns many variations of a single task, as opposed to learning multiple tasks, and variations, with a single model)\n- To be clear, I do think the table tennis task shows versatility :)"}}, {"review_id": "ON0cdyo55F", "reviewer": "Reviewer_xPhY", "summary": "Learn a set of motion primitives that can achieve various parameterized behaviors. Then learn an expert agent which chooses the parameters of the behavior from the task definition. The expert policy is optimized for selection of motion primitive parameters using maximum entropy reinforcement learning, which regularizes with the entropy of the policy given context. An additional regularization on the distribution of the context vector pushes the KL distributional distance towards uniform. This learning can be broken into three distributions, the distributions over the component weights based on context and primitives, based on context, and the raw distribution. These can be used by Bayes identities to determine the primitive and expert policies.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nProposes an effective derivation of an entropy based distribution of behavior\n\nA strong collection of experiments that visualize an otherwise complex description\n\nThe experiments are interesting and provide encouraging results on robotics tasks. A physical robot would have been nice but not necessary\n\nWeaknesses:\n\nWhile assuming that p(c) is uniform simplifies the derivation, it is not clear whether this assumption has detrimental effects on the final policies\n\nThe clarity of the derivation between equations 2 and 3 is somewhat confusing. The identity only becomes clear when looking at the appendix, where the order of terms and the values is made clear. However, even after the derivation is identified, it is hard to follow the flow of the equation because there is little intuitive description of the six terms in the equation. The work provides little to no intuitive meaning for terms, as the subsequent sections mostly focus on describing the auxiliary distributions used to optimize them.\n\nThis work introduces a large number of terms, including describing mixture of experts, component weights, context vectors and primitive parameters. However, it is not clear what the intuitive interpretation of these terms is, as they are only described vaguely in the introduction. Even in the experiments, it is still not abundantly clear what the intuitive meaning of these terms are without familiarity with the methods used (ProMP). Considering that the work handles a good amount of derivation, it would be useful to have a more in-depth description of the intuitive meanings, especially since final terms are not particularly clear.\n\nSome discussion in related work of mutual information based skill learning methods, such as Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills (Campos et. al.), or other work in the hierarchical reinforcement learning literature would be useful, since this is a work that uses a different jargon but aims to achieve very much the same set of hierarchical decision-making. \nOne minor limitation is that though the derivation is general, the results rely heavily on ProMPs for the primitives, which means that the executed behaviors tend to be particular smooth motions, rather than adaptable behaviors which might be possible with a less dynamically based policy class such as a neural policy.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Most of the score of weak accept is given because it was difficult to parse this work and capture the intuitive meanings. The method and derivation seem sound and the novelty is good, but again, because it is somewhat difficult to follow the paper, it is difficult to have high confidence in the result.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "I believe the work to be interesting and have good results. However, it is difficult to make that assessment with confidence because the paper itself is hard to parse. Some changes to the clarity of the derivation and the intuitive definition of terms, grounded in the experiments, would greatly improve the paper. "}}, {"review_id": "5gcTuf9ln3d", "reviewer": "Reviewer_as3F", "summary": "The paper proposed a learning algorithm for obtaining a mixture of control policies that solves tasks with contexts in different ways. The main contribution of the paper is an objective that allows different expert policies (components) to be learned independently, thus enabling incremental learning of different components. They demonstrate that the proposed method can achieve good coverage on the context space of the task and good performance of the tasks.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The idea of incremental learning of components that covers different parts of the context space is interesting.\n+ The experiment result shows interesting behaviors learned by the proposed method. The comparison to prior methods in general supports the idea in this work.\n\n- The writing is not very clear in general, making the paper not very easy to understand. More details in issues below.\n- It’s not clear if the proposed method can handle tasks with high-dimensional context space, or with varying difficulties for different contexts.\n- Though the paper provided comparison to the HiREPS algorithm and showed superior performance, it seems that LaDiPS [8] is a better baseline in this case given that it shares the same policy search algorithm as the presented work. It’ll better support the value of the proposed incremental training scheme.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper presents a concrete learning algorithm and demonstrates good performance on a suite of difficult problems. However, the exposition of the paper could be improved and more discussions/comparisons would make the paper even stronger.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- A notation issue: \\pi is used everywhere in the paper, what does it stand for in this case? Initially I interpreted it as the control policy, but later realized it is more like a probability distribution, which is a bit confusing.\n- The identities in line 151 are not very clear: should there be a summation over o in it? Otherwise what should o be given that it doesn’t appear on the left-hand side?\n- The result in the video shows that the method can learn to solve a particular task with different styles. Are those styles from different components, or do they emerge from the max-entropy objective under the same component?\n- Add discussion regarding if the proposed method can handle tasks with high-dimensional context space: are all the terms in the algorithm easily scalable to high-dimensional spaces? \n- Also, how would the method work if different contexts correspond to tasks with very different difficulties? Would policies attempting to solve those tasks get constantly pruned due to being suboptimal?\n- Ideally, add some comparisons to LaDiPS [8].\n"}}]

Author rebuttal:

Meta-review: Strengths:

- The high level objective of learning skills that encode different ways of accomplishing the same goals was viewed as important.

- The work appears to be technically sound, leveraging max ent policy search and regularization strategies from variational inference. The work extends existing policy search approaches to a multi-modal context in a technically sound way.

- The simulated evaluation of the method on Beer Pong and Ping Pong demonstrated that the method can work in simulation.

Weaknesses:

- Most reviewers had a hard time understanding the development in sect 3. In particular, the development from eq 2 to eq 3 was hard to understand. Some reviewers suggest adding intuitive descriptions of the various terms in eq 3. The use of pi(c) as a variational distribution in eq 2 was unclear (if that’s what it was). Generally, I think there is a lot of “inside baseball” drawn from the policy search community and the variational inference community that makes it hard for many people to parse what is going on here.

- There were questions about the significance of the experimental results relative to the general state of the art. The work would be strengthened by: 1) evaluating the method in simulation on many more tasks; and/or 2) performing experiments on real robots. There was also a question about whether HiREPS was the right baseline for this approach. Additional baselines would help.

- There were questions about sample efficiency resulting from the fact that experiences are not shared between components for the purposes of policy learning.

Post Rebuttal:

The authors have significantly improved the clarity of the described method. This was the primary criticism and multiple reviewers now feel positively after seeing this change.
Decision: Accept (Poster)