Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The submission tackles an important problem of learning highly varied skills. The approach relies on dividing the task space into subareas (defined by task context vectors) over which individual policies are trained, but are still required to operate well on tasks outside their context.

The exposition is clear and the method is well-motivated. I see no issues with the mathematical correctness of the claims made in the paper. The experimental results show a convincing benefit over TRPO and Distral on a number of manipulation and locomotion tasks. I would like to have seen more discussion of the computational costs and scaling of the method over TRPO or Distral, as the pairwise KL divergence terms grow quadratically in the number of contexts. 

While the method is well-motivated, the division of tasks into subareas seems arbitrarily chosen. It would be very useful for readers to see performance of the algorithm under other task decompositions to alleviate the worries that the algorithm is not sensitive to the decomposition choice.

I would also like to see more discussion of curriculum learning, which also aims at tackling a similar problem of reducing complexity in early stages of training by choosing on simper tasks and progressing to more complex. Would such progressive tasks decompositions work better in your framework? Does your framework remove the need for curriculum learning?

Overall, I believe this is in interesting piece of work and I believe would be of interest to ICLR community.
review_title: Good submission, although could use more evaluation

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents a method for learning a global policy over multiple different MDPs (referred to as different "contexts", each MDP having the same dynamics and reward, but different initial state).  The basic idea is to learn a separate policy for each context, but regularized in a manner that keeps all of them relatively close to each other, and then learn a single centralized policy that merges the multiple policies via supervised learning.  The method is evaluated on several continuous state and action control tasks, and shows improvement over existing and similar approaches, notably the Distral algorithm.

I believe there are some interesting ideas presented in this paper, but in its current form I think that the delta over past work (particularly Distral) is ultimately too small to warrant publication at ICLR.  The authors should correct me if I'm wrong, but it seems as though the algorithm presented here is virtually identical to Distral except that:
1) The KL divergence term regularizes all policies together in a pairwise manner.
2) The distillation step happens episodically every R steps rather than in a pure SGD manner.
3) The authors possibly use a TRPO type objective for the standard policy gradient term, rather than REINFORCE-like approach as in Distral (this one point wasn't completely clear, as the authors mention that a "centralized DnC" is equivalent to Distral, so they may already be adapting it to the TRPO objective? some clarity on this point would be helpful).
Thus, despite better performance of the method over Distral, this doesn't necessarily seem like a substantially new algorithmic development.  And given how sensitive RL tasks are to hyperparameter selection, there needs to be some very substantial treatment of how the regularization parameters are chosen here (both for DnC and for the Distral and centralized DnC variants).  Otherwise, it honestly seems that the differences between the competing methods could be artifacts of the choice of regularization (the alpha parameter will affect just how tightly coupled the control policies actually are).

In addition to this point, the formulation of the problem setting in many cases was also somewhat unclear.  In particular, the notion of the contextual MDP is not very clear from the presentation.  The authors define a contextual MDP setting where in addition to the initial state there is an observed context to the MDP that can affect the initial state distribution (but not the transitions or reward).  It's entirely unclear to me why this additional formulation is needed, and ultimately just seems to confuse the nature of the tasks here which is much more clearly presented just as transfer learning between identical MDPs with different state distributions; and the terminology also conflicts with the (much more complex) setting of contextual decision processes (see: https://arxiv.org/abs/1610.09512).  It doesn't seem, for instance, that the final policy is context dependent (rather, it has to "infer" the context from whatever the initial state is, so effectively doesn't take the context into account at all).  Part of the reasoning seems to be to make the work seem more distinct from Distral than it really is, but I don't see why "transfer learning" and the presented contextual MDP are really all that different.

Finally, the experimental results need to be described in substantially more detail.  The choice of regularization parameters, the precise nature of the context in each setting, and the precise design of the experiments is all extremely opaque in the current presentation.  Since the methodology here is so similar to previous approaches, much more emphasis is required to better understand the (improved) empirical results in this eating.

In summary, while I do think the core ideas of this paper are interesting: whether it's better to regularize policies to a single central policy as in Distral or whether it's better to use joint regularization, whether we need two different timescales for distillation versus policy training, and what policy optimization method works best, as it is right now the algorithmic choices in the paper seem rather ad-hoc compared to Distral, and need substantially more empirical evidence.

Minor comments:
• There are several missing words/grammatical errors throughout the manuscript, e.g. on page 2 "gradient information can better estimated".
review_title: Interesting approach, but seems like a fairly incremental advance on previous work

final_score: rating: 7: Good paper, accept
review_content: This paper presents a reinforcement learning method for learning complex tasks by dividing the state space into slices, learning local policies within each slice, while ensuring that they don't deviate too far from each other, while simultaneously learning a central policy that works across the entire state space in the process. The most closely related works to this one are Guided Policy Search (GPS) and "Distral", and the authors compare and contrast their work with the prior work suitably.

The paper is written well, has good insights, is technically sound, and has all the relevant references. The authors show through several experiments that the divide and conquer (DnC) technique can solve more complex tasks than can be solved with conventional policy gradient methods (TRPO is used as the baseline). The paper and included experiments are a valuable contribution to the community interested in solving harder and harder tasks using reinforcement learning.

For completeness, it would be great to include one more algorithm in the evaluation: an ablation of DnC which does not involve a central policy at all. If the local policies are trained to convergence, (and the context omega is provided by an oracle), how well does this mixture of local policies perform? This result would be instructive to see for each of the tasks.

The partitioning of each task must currently be designed by hand. It would be interesting (in future work) to explore how the partitioning could perhaps be discovered automatically.
review_title: Good paper, pushing the limits of RL to harder tasks.

Meta-review: This paper proposes a specific architecture for training an ensemble of separate policies on a family of easier tasks with the goal of obtaining a single policy that can perform well on a harder task. There are significant similarities to the recently published Distral algorithm, but I am convinced that this work offers a meaningful contribution beyond that work. Moreover, the authors performed a thorough comparison between their method and Distral and found that DnC performs better.
Decision: Accept (Poster)