Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper looks at the problem of transferring a policy learned in a simulator to  a target real-world system.  The proposed approach considers using an ensemble of simulated source domains, along with adversarial training, to learn a robust policy that is able to generalize to several target domains.

Overall, the paper tackles an interesting problem, and provides a reasonable solution.  The notion of adversarial training used here does not seem the same as other recent literature (e.g. on GANs).  It would be useful to add more details on a few components, as discussed in the question/response round.  I also encourage including the results with alternative policy gradient subroutines, even if they don’t perform well (e.g. Reinforce), as well as results with and without the baseline on the value function. Such results are very useful to other researchers.

review_title: Review

final_score: rating: null
review_content: Can you add details about the baseline you use to stabilize the policy gradient estimate (bottom of p.3). Is this estimated from all trajectories, or model-specific?

Can you add code for the BatchPolOpt subroutine (Algorithm 1)? What are the pros/cons of using TRPO for this, compared to other methods?  How do the results change if you use a different policy optimization method, e.g Reinforcement w/baseline?

How many samples are used for the Fig.1 results?  How many samples are necessary to pick a good setting for epsilon?

I would expect to see comparison to the bayesian GP-TD approach.  What do these results show?  A comparison to Mordatch et al. would also be informative. Currently, there are no comparison to any other method.

What 

What is the relation between your approach and boosting?

review_title: Questions

final_score: rating: 7: Good paper, accept
review_content: Paper addresses systematic discrepancies between simulated and real-world policy control domains. Proposed method contains two ideas: 1) training on an ensemble of models in an adversarial fashion to learn policies that are robust to errors and 2) adaptation of the source domain ensemble using data from a (real-world) target domain. 

> Significance

Paper addresses and important and significant problem. The approach taken in addressing it is also interesting 

> Clarity

Paper is well written, but does require domain knowledge to understand. 

My main concerns were well addressed by the rebuttal and corresponding revisions to the paper. 
review_title: ICLR 2017 conference review

final_score: rating: null
review_content: 1. The EPOpt algorithm in some abstract sense reminds me of:

Wang, J. M., Fleet, D. J., Hertzmann, A. Optimizing Walking Controllers for Uncertain Inputs and Environments. ACM Transactions on Graphics 29, 4 (Proceedings of SIGGRAPH 2010). 

Can you comment on the relationship? 

2. It would be interesting to see how the illustrated performance (particularly in Figure 1, right) changes as a function of \epsilon, e.g., as epsilon is increased (or decreased) from 0.1 default. Has this been tried? Can the differences in performance be characterized?

 



review_title: Pre-review questions

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper explores ensemble optimisation in the context of policy-gradient training. Ensemble training has been a low-hanging fruit for many years in the this space and this paper finally touches on this interesting subject. The paper is well written and accessible. In particular the questions posed in section 4 are well posed and interesting.

That said the paper does have some very weak points, most obviously that all of its results are for a very particular choice of domain+parameters. I eagerly look forward to the journal version where these experiments are repeated for all sorts of source domain/target domain/parameter combinations.

<rant 
Finally a stylistic comment that the authors can feel free to ignore. I don't like the trend of every paper coming up with a new acronymy wEiRDLY cAsEd name. Especially here when the idea is so simple. Why not use words? English words from the dictionary. Instead of "EPOpt" and "EPOpt-e", you can write "ensemble training" and "robust ensemble training". Is that not clearer?
/>
review_title: Ensemble training and transfer, a good submission

Meta-review: The approach here looks at learning policies that are robust over a parameterized class of MDPs (in the sense the probability that the policy doesn't perform well is small over this class. The idea is fairly straightforward, but the algorithm seems novel and the results show that the approach does seem to provide substantial benefit. The reviewers were all in agreement that the paper is worth accepting.
 
 Pros:
 + Nice application of robust (really stochastic, since these are chance constraints) optimization to policy search
 + Compelling demonstration of the improved range of good performance over methods like vanilla TRPO
 
 Cons:
 - The question of how to parameterize a class of MDPs for real-world scenarios is still somewhat unclear
 - The description of the method as optimizing CVaR seems incorrect, since they appear to be using an actual chance constraint, whereas CVaR is essentially a convex relaxation ... this may be related to the work in (Tamar, 2015), but needs to be better explained if so.
Decision: Accept (Poster)