Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper goes after two issues
1) How to learn to handle context (parameter) changes in the learning tasks, such as a variety of goals or friction values.
2) How to learn to execute different strategies to do the same thing (use a forehand or a backhand swing).
The suggests a mixture of experts approach. A more useful terminology emphasizes that policies are composed of classifiers that
select "simple" policies depending on the context and desired strategy. I would suggest the term "classifier-based policies" or
"selection-based policies". Mixture of experts often don't have a classifier and use weighted blends of the experts' outputs, which
is a very different thing.

Key ideas include:
1) Develop a curriculum to support learning the classifier and the component policies. This helps avoid interference during learning, as
well as getting good coverage.
2) Control the scope of the component policies by optimizing the worst case for each component.
3) Optimize the classifier and the component policies simultaneously (vs. in some EM-like alternation).
4) Iteratively add components to improve coverage.
strengths_and_weaknesses: Strengths:

I am very happy to see a paper on learning multiple "strategies" to do the same thing, as I think this is a very important issue
for skill learning.

Weaknesses:

It is easy to do this kind of stuff in simulation. Not so easy on real robots. Not having an implementation of something on a real robot
is a big weakness.

The paper promises "two challenging simulated robot skill learning tasks". Implementations of both of these tasks have been done
on real robots
in the past. Ball throwing is not that hard (see 1988 paper
https://www.researchgate.net/publication/37596892_Model-Based_Robot_Learning). Robot ping pong (table tennis) is challenging,
but perhaps the major part of the challenge is perception rather than action. Lots of people have implemented robot ping
pong players. Search for "robot ping pong table tennis players" on Youtube. I would back off "challenging" unless I was doing some
task that really hasn't been done many times before on a real robot. Or demonstrate real robot performance on a larger range of tasks
with appropriate generalization across tasks: Simultaneously learn: tennis, squash, badminton, paddle ball, racquetball, table tennis, soft tennis, platform tennis, beach tennis, smash ball, hand tennis, basque pelota, ...

Did the paper show suitable generalization across contexts? Was interference avoided? How was this evaluated?
At the end it says "Currently, the major drawback of
our approach is sample efficiency, as we do not share experience between the components."
What about between contexts when the same option is selected? When a different option is selected?

The motivation and notation for Equation 1 is not clear.
Given c, the option o is selected by the agent. This function is known, so why not use it directly instead of using Bayes rule to replace it.
pi(theta|c) = sum_o pi(theta|c,o) p(o|c)
Bayes rule applies to probabilities such as prob(o|c)prob(c) = prob(c|o)prob(o).
To me, the policy pi(o|c) is clearly prob(o|c).
The mention of Bayes' Rule in the derivation of equation 1 implies
pi(c) is not meaningful, prob(c) is, and has nothing to do with the agent. So how is pi(c) not equal to prob(c)?
pi(c|o) and pi(o) are not meaningful either, but prob(c|o) and prob(o) are and can be computed from pi(o|c) = prob(o|c), which is under
the control of the agent.
Clearly I am confused.

I don't understand why an entropy bonus is useful in the case where c has nearly irrelevant components. Won't this favor adding  unnecessary theta variance or irrelevant component policies? A useful regularization in reinforcement learning is to penalize complexity.
Can you explain why the entropy bonus is useful, and what practical effect it has.

I don't see why pi(c|o) or pi(o) (really the probabilities, that are only under indirect control by selecting pi(o|c)) are useful.
The difference between pi(c) and probability(c) is not clear. I shouldn't have to read Kink et al. [2] to understand it.
pi(c) is presumably the probability of c, and is not under control of the agent, so the notation should be p(c) or prob(c).
What is the meaning of pi(c|o)? The option selected has no effect on the actual context. Perhaps you should introduce a context
estimate hat(c) = sum_o pi(hat(c)|o)pi(o)

The distinction between trial-based and time-step-based learning algorithms was not clear. Perhaps "step-based" is too abbreviated in
"Related Step-Based Approaches. In the step-based setting, the policy does not learn a function
from contexts to parameters of an episodic controller but directly maps from system states to actions."
I might try to make the distinction clearer by emphasizing the difference between training policies
a) by adjusting a function from a context or goal to a small number of policy parameters at the end of trials (trial-based? episodic?)
vs.
b) by adjusting a function from states to actions at each time step of execution (time-step-based?)

For parametric component policies, I don't think there is a fundamental distinction between episodic and time-step-based approaches, especially
when there are no episodes (life-long learning).

It might be useful to make a connection to "options" as described by Sutton.

Given the tasks chosen are deterministic (at least when done with real robots),
it may also be useful to consider the deterministic model-based case, where selector() and pi_option() are deterministic.
pi(action|state) = Sum( selector(state)*pi_option(state) )
If the selector is one for one option at a time, this selects an option.
Anything else blends options. Time-step-based policies (options) can
be learned in the same way as episodic policies. This leads to much
simpler gradient learning rules than the stochastic case.
www.cs.cmu.edu/~cga/papers/acc2012-tr.pdf

How good are the lower bounds, and how much does it matter?

The paper promises "unprecedented accuracy and versatility," How is this demonstrated? Too much hype?






summary_of_recommendation: How to represent multiple strategies to do the same task (same initial conditions and goal) is an important question.
If multiple strategies were demonstrated on a real robot, that would be very interesting.

issues: See the issues discussed above.

Clearly explain (so someone could actually write code) a simple example like the multi-joint reaching.

Implement behaviors on a real robot.

In the wonderful world of open review, all the submitted papers are
available to the authors. There are at least 3 "generate and test"
papers:
190 Specializing Versatile Skill Libraries using Local Mixture of Experts
98 Data-efficient learning of object-centric grasp preferences
89 Hierarchical Policies for Cluttered-Scene Grasping with Latent Plans
You should take advantage of this opportunity to put your work into
a larger context (and take a look at the reviews these papers got.)
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposed a learning algorithm for obtaining a mixture of control policies that solves tasks with contexts in different ways. The main contribution of the paper is an objective that allows different expert policies (components) to be learned independently, thus enabling incremental learning of different components. They demonstrate that the proposed method can achieve good coverage on the context space of the task and good performance of the tasks.

strengths_and_weaknesses: + The idea of incremental learning of components that covers different parts of the context space is interesting.
+ The experiment result shows interesting behaviors learned by the proposed method. The comparison to prior methods in general supports the idea in this work.

- The writing is not very clear in general, making the paper not very easy to understand. More details in issues below.
- It’s not clear if the proposed method can handle tasks with high-dimensional context space, or with varying difficulties for different contexts.
- Though the paper provided comparison to the HiREPS algorithm and showed superior performance, it seems that LaDiPS [8] is a better baseline in this case given that it shares the same policy search algorithm as the presented work. It’ll better support the value of the proposed incremental training scheme.

summary_of_recommendation: The paper presents a concrete learning algorithm and demonstrates good performance on a suite of difficult problems. However, the exposition of the paper could be improved and more discussions/comparisons would make the paper even stronger.

issues: - A notation issue: \pi is used everywhere in the paper, what does it stand for in this case? Initially I interpreted it as the control policy, but later realized it is more like a probability distribution, which is a bit confusing.
- The identities in line 151 are not very clear: should there be a summation over o in it? Otherwise what should o be given that it doesn’t appear on the left-hand side?
- The result in the video shows that the method can learn to solve a particular task with different styles. Are those styles from different components, or do they emerge from the max-entropy objective under the same component?
- Add discussion regarding if the proposed method can handle tasks with high-dimensional context space: are all the terms in the algorithm easily scalable to high-dimensional spaces? 
- Also, how would the method work if different contexts correspond to tasks with very different difficulties? Would policies attempting to solve those tasks get constantly pruned due to being suboptimal?
- Ideally, add some comparisons to LaDiPS [8].
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Learn a set of motion primitives that can achieve various parameterized behaviors. Then learn an expert agent which chooses the parameters of the behavior from the task definition. The expert policy is optimized for selection of motion primitive parameters using maximum entropy reinforcement learning, which regularizes with the entropy of the policy given context. An additional regularization on the distribution of the context vector pushes the KL distributional distance towards uniform. This learning can be broken into three distributions, the distributions over the component weights based on context and primitives, based on context, and the raw distribution. These can be used by Bayes identities to determine the primitive and expert policies.
strengths_and_weaknesses: Strength:

Proposes an effective derivation of an entropy based distribution of behavior

A strong collection of experiments that visualize an otherwise complex description

The experiments are interesting and provide encouraging results on robotics tasks. A physical robot would have been nice but not necessary

Weaknesses:

While assuming that p(c) is uniform simplifies the derivation, it is not clear whether this assumption has detrimental effects on the final policies

The clarity of the derivation between equations 2 and 3 is somewhat confusing. The identity only becomes clear when looking at the appendix, where the order of terms and the values is made clear. However, even after the derivation is identified, it is hard to follow the flow of the equation because there is little intuitive description of the six terms in the equation. The work provides little to no intuitive meaning for terms, as the subsequent sections mostly focus on describing the auxiliary distributions used to optimize them.

This work introduces a large number of terms, including describing mixture of experts, component weights, context vectors and primitive parameters. However, it is not clear what the intuitive interpretation of these terms is, as they are only described vaguely in the introduction. Even in the experiments, it is still not abundantly clear what the intuitive meaning of these terms are without familiarity with the methods used (ProMP). Considering that the work handles a good amount of derivation, it would be useful to have a more in-depth description of the intuitive meanings, especially since final terms are not particularly clear.

Some discussion in related work of mutual information based skill learning methods, such as Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills (Campos et. al.), or other work in the hierarchical reinforcement learning literature would be useful, since this is a work that uses a different jargon but aims to achieve very much the same set of hierarchical decision-making. 
One minor limitation is that though the derivation is general, the results rely heavily on ProMPs for the primitives, which means that the executed behaviors tend to be particular smooth motions, rather than adaptable behaviors which might be possible with a less dynamically based policy class such as a neural policy.

summary_of_recommendation: Most of the score of weak accept is given because it was difficult to parse this work and capture the intuitive meanings. The method and derivation seem sound and the novelty is good, but again, because it is somewhat difficult to follow the paper, it is difficult to have high confidence in the result.
issues: I believe the work to be interesting and have good results. However, it is difficult to make that assessment with confidence because the paper itself is hard to parse. Some changes to the clarity of the derivation and the intuitive definition of terms, grounded in the experiments, would greatly improve the paper.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a novel objective function for learning a Mixture-of-Experts (MoE) model to solve robot motion tasks. The objective function uses maximum entropy to encourage diversity in a collection of learned motion primitives, which are called "components." An interesting aspect of this paper, in comparison to existing work, is that the motion primitives are added and learned sequentially, with the objective function encouraging each one to differentiate from the others, thereby covering the entire space of "contexts" (i.e., different variations of the same task). The method outperforms the existing HiREPS method in simulation on difficult tasks, justifying the underlying math that went into creating the objective function.

Update 1 Sept: I am more confident in my "accept" recommendation now. I'll leave it as a "weak accept" because, per the authors' comments, it seems I'm not super knowledgeable about this area. But, I think the general consensus on this paper is an "accept" with the understanding that it's interesting and good, but maybe not mind-blowing or completely revolutionary (but still good!).
strengths_and_weaknesses: STRENGTHS
- I love the idea of local context distributions that adapt and grow; this feels a lot like human learning, and it makes sense that the approach would be very versatile.
- I really appreciate the clear statement of strengths/weaknesses of the related methods
- It is clear the authors put effort into stepping the reader through their thought process
- The comparison to HiREPS seems appropriate and well-justified

WEAKNESSES
- The word "versatility" in the title is never formally defined, so the assessment of versatility seems entirely qualitative, but it would ideally be quantitative
- I feel that the word "context" is a bit misleading when considering what is meant by a context in this paper; here, contexts are limited to a continuum of task properties (e.g., goal position or friction parameters). I would generally imagine contexts to be a much higher-level abstraction, such as "kitchen" versus "living room." But, perhaps this is just a part of the MoE vocabulary with which I'm not familiar?
- Some justification for claims in the paper are in the appendices, but I think it would be better if they could be brought into the paper; I've made note of them below
summary_of_recommendation: I am putting "weak accept" because I am not super familiar with mixture-of-experts or skill library learning. I think this paper is pretty good, and just needs some clarifications to be acceptable. But, if it's already been done or other more knowledgeable reviewers have serious problems with it, then I understand rejecting it.
issues: Abstract
- It would be helpful to note any comparison methods in the abstract.

Introduction / Related Work
- The claim that "most of these algorithms... usually assume a Gaussian policy" needs some citations to back it up; I think [2] in the paper is the relevant one?
- For the proposed method, depending on the dimension of the context space, it would take an exponential number of local regions to effectively cover it. How can we avoid this blowup? I guess removing suboptimal solutions helps out.
- I would appreciate a sentence that clarifies the "ad hoc constraint" for HiREPS and LaDiPS (i.e., just add more details explaining the constraint)

Specializing Versatile Mixture of Expert Models
- The terms R and H in the maximum entropy aren't defined (one can figure out that R is reward and H is entropy, but it would be good to state it explicitly)
- It would be helpful to number all equations for the sake of reviewers :)
- On line 149, the text refers to "log terms of the entropy and the KL" but these log terms haven't appeared explicitly yet (i.e., the entropy and KL terms should be expanded before (3) so it's clearer for the reader)
- Why is it reasonable to approximate the integral over $\theta$ with a single sample? I feel like this statement needs further justification
- How does one know when a component is sufficiently trained before adding a new one? How does one know that enough components have been added? While these questions may be answered in Appx. B, it would be helpful to at least briefly answer them in the main text.

Experiments
- In Fig. 1, I presume that the dotted lines are the 10-link robot. What are the red boxes? I guess obstacles? (these questions are answered in the main text around line 228, but should be explained in the figure caption as well)
- I'm confused by how the beer pong task shows versatility; I think it's best shown by the way that the robot bounces the ball multiple times. But it feels like true versatility would be for a single MoE model that learns _both_ beer pong and table tennis, or something really wild like that. I realize that I'm asking for a lot, but I think that the paper should have a much more precise statement about what "versatility" means (i.e., it learns many variations of a single task, as opposed to learning multiple tasks, and variations, with a single model)
- To be clear, I do think the table tennis task shows versatility :)
review_title: null

Meta-review: Strengths:

- The high level objective of learning skills that encode different ways of accomplishing the same goals was viewed as important.

- The work appears to be technically sound, leveraging max ent policy search and regularization strategies from variational inference. The work extends existing policy search approaches to a multi-modal context in a technically sound way.

- The simulated evaluation of the method on Beer Pong and Ping Pong demonstrated that the method can work in simulation.

Weaknesses:

- Most reviewers had a hard time understanding the development in sect 3. In particular, the development from eq 2 to eq 3 was hard to understand. Some reviewers suggest adding intuitive descriptions of the various terms in eq 3. The use of pi(c) as a variational distribution in eq 2 was unclear (if that’s what it was). Generally, I think there is a lot of “inside baseball” drawn from the policy search community and the variational inference community that makes it hard for many people to parse what is going on here.

- There were questions about the significance of the experimental results relative to the general state of the art. The work would be strengthened by: 1) evaluating the method in simulation on many more tasks; and/or 2) performing experiments on real robots. There was also a question about whether HiREPS was the right baseline for this approach. Additional baselines would help.

- There were questions about sample efficiency resulting from the fact that experiences are not shared between components for the purposes of policy learning.

Post Rebuttal:

The authors have significantly improved the clarity of the described method. This was the primary criticism and multiple reviewers now feel positively after seeing this change.
Decision: Accept (Poster)