Title: Efficient iterative policy optimization
Authors: Nicolas Le Roux
Venue: ICLR 2017

Abstract: We tackle the issue of finding a good policy when the number of policy updates is limited. This is done by approximating the expected policy reward as a sequence of concave lower bounds which can be efficiently maximized, drastically reducing the number of policy updates required to achieve good performance. We also extend existing methods to negative rewards, enabling the use of control variates.


Review discussion:
It proposes an interesting modification to PoWER. Questions:
- Could you elaborate on the statement in second paragraph of section 2, where you state since the “event is highly unpredictable... carefully crafted Q-functions are unlikely to yield improvement”, and why Iterative PoWER has edge over those methods? “Carefully-crafted” seems inaccurate, given they use flexible neural network function approximators.
- Have you tried the method on more difficult continuous control tasks in Gym? The cartpole example optimizes only 4 parameters. It would be helpful to see results on generic neural network policies, and comparison with other state-of-the-art methods such as TRPO-GAE on the minimal number of policy updates required to solve tasks.
Hi and thank you for your comments.
Regarding the first question, a common example is maximizing the number of clicks. For each ad displayed, the reward is binary (either there is a click or there isn't) with an average value of around 0.01 (about 1% of the ads displayed are clicked in this example). We might try to predict if an ad will be clicked but, since it is very hard to capture what leads to a click, click prediction systems will almost all the time output values between 0.001 and 0.1 (let's say).
So, we can either use Q-learning which is likely to have high bias, or a technique such as doubly robust which will hardly reduce the variance (going from 1 to 0.9 only has little effect). In that setting, our best attempt at maximizing clicks is with direct policy optimization, even though we still need to gather many samples to have a reasonable variance.
I understand the confusion with "Carefully crafted". What I meant is that, even though the features might not be selected by hand such as when using a deep net, there is a usually a lot of care put in designing an optimizing the Q-function (choosing the initialization, the size of the layers, the connectivity and the optimizer parameters). Maybe "Carefully tuned" would be more appropriate?
Regarding the second (multiple) question:
- I did not try on more complex tasks on Gym as my goal was to demonstrate that iPower could also be applied in the longer horizon setting. Since my goal was to focus on the policy optimization problem, I did not want to try it on problems where the issue lies elsewhere, for instance in designing good exploration policy (e.g. the chicken problem).
However, I agree with you that 5 tunable parameters is small.
That is why I also tested it on a real dataset when there were 400 free parameters.
But, should there be another problem in Gym with more parameters not requiring carefully tuned exploration policies, I'd be happy to try iPower on them. I also plan to release the code once I added comments and traded efficiency for readability.
- There is a fundamental difference between what TRPO tries to achieve and what we try to achieve. TRPO helps avoid ending with a terrible policy, and does so by controlling the KL between the current policy and the new one. By design, it thus makes small moves in policy space between two rollouts. In our setting, on the other hand, we do not want to be conservative and we truly want the best estimated policy given that we have many samples at our disposal.
Thus, we absolutely do not want to bound the distance between the current policy and the new one (or only insofar as we do not want to be hurt by the variance too much so adding a regularizer might still be good, but not of the KL form).
However, if you wish me to compare iPower to TRPO on the Cartpole problem, I can try to do this before the end of the reviewing period (though NIPS will take its toll).
I must emphasize that, even though I am aware of many other successful policy optimization techniques, I do not know of many which do not require the careful setting of hyperparameters. That setting is not an issue in many cases, for instance when trying to solve robotics tasks, but it is when the policy optimization routine is part of a large production system which is run on severals tens of models per day.
I hope my answer clarifies your points.
The paper presents an interesting modification to PoWER algorithm that is well motivated. The main limitation of this paper is the lack of comparison with other methods and on richer problems. The experiments haven't given confidence to show its claimed benefits, generality and scalability over prior methods. Giving this confidence doesn’t necessarily require running your method on all large-scale domains or doing exhaustic hyper-parameter search, but for example it could go beyond current domains. Cartpole only optimizes 5 parameters. Ad targeting task lacks comparison with alternative methods. Since this method is built on PoWER and closely connected with RWR, it is likely there are limits to performance which may become apparent when the method is tried on other domains and with other benchmark methods, e.
g. even standard ones like importance sampling-based off-policy learning is known to suffer in high-dimensional or continuous action space; limits of RWR/PoWER-like methods based on their connection with entropy-regularized RL. https://arxiv. org/abs/1604. 06778 may be a valuable starting point for comparison, e. g. it compares RWR with policy gradient methods, and it has open-sourced codes.
Thank you for your review and comments. We acknowledge your comment about the lack of baselines and will add more experiments as soon as possible.
The paper considers the problem of reinforcement learning where the number of policy updates is required to be low. The problem is well motivated and the author provides an interesting modification to the PoWER algorithm, along with variational bounds on the value function (lemmas 3,4) which are interesting in themselves. They also provide numerical results on the cartpole problem and a problem in online advertising with real data.
Overall this is a strong, well-written paper. My main reservation is whether it is completely appropriate for ICLR, since the log-concavity assumption the model relies on appear to restrict to simpler models where representations will be not in fact be learned.
Other comments:
- There is a general lack of baselines in the numerical experiment section. I acknowledge this is somewhat of an unusual setting, but even a simple, well-justified baseline would have been welcome. Since cartpole is a relatively simple problem and the advertising dataset is presumably private, perhaps a way to generate a synthetic advertising dataset would have been interesting.
- I was confused by the control variates as constant scalars - are they meant to be constant baselines? And if so, they appear to be treated as hyperparameters -  why are they not learned or estimated?
- There is an interesting section on constrained optimization, but as it is, feels a bit disconnected from the rest of the paper. It appears applicable to the problem of online advertising, but is not mentioned in the corresponding experimental section.
Also might be worth adding a citation to the literature of constrained MDPs which develops similar lagrangian ideas.
Thank you for your review and comments.
The appropriateness for ICLR could indeed be debated.
However, bear in mind that deep policies can be decomposed in two parts: a nonlinear transformation of the state and a linear policy on that transformed state. This paper simplifies the optimization of the latter part, a strategy already used in supervised learning when people use the NLL rather than other, nonconvex losses. A subsection was added at the end of the paper to address this point.
Regarding the control variates, we learnt at each timestep the control variate leading to the minimum variance for the estimator, then used multiples of that "optimal" control variate.
Thus, cv=0.5, for instance, uses a control variate which is half of the variance-minimizing control variate, this value being modified during learning based on the runs observed. This was clarified in the last version of the paper.
The section on constrained optimization was moved to the end of the paper so as to not break the flow of the core idea.
Finally, the comment on the lack of baselines was noted. They are not in the current version of the paper but I will try to add more experiments as soon as possible.
This paper presents iterative PoWER, an off-policy variation on PoWER, a policy gradient algorithm in the reward-weighted family.
I'm not familiar enough with this type lower bound scheme to comment on it. It looks like the end result is less conservative step sizes in policy parameter space. All expectation-based algorithms (and their KL-regularized cousins a-la TRPO) take smallish steps, and this might be a sensible way to accelerate them.
The description of the experiments in Section VI is insufficient for reproducibility. Is "The cart moved right" supposed to be "a positive force is applied to the cart"? How is negative force applied? What is the representation of the state? What is the distribution of initial states? A linear policy is insufficient for swing up and balance of a cart-pole. Are you only doing balancing? What is the noise magnitude of the policy? How was it chosen? How long were the episodes?
The footnote at the bottom of page 8 threw me off. If you're using Newton's method, where is the discussion of gradients and Hessians? I thought the argmax_theta operator was a stand-in for an EM-style step, which I how I read Eq (8) in the Kober paper.
https://papers.nips.cc/paper/3545-policy-search-for-motor-primitives-in-robotics.pdf
I might be missing something basic here.
The control variates thing seems cool. I only read up on it now and I don't think I've seen it before in the RL literature. Seems like a powerful tool.
Section 6.2 has too much business jargon, I could barely read it.
Thank you for your review and comments.
First, you mention the lack of details on the cartpole experiment. Since the OpenAI Gym framework was used, all the details can be found on their website (https://gym.openai.com/). At each time step, the action consists in moving the cart to the right or to the left. The state is represented by the position and velocity of the cart as well as the angle and angular velocity of the pole. The pendulum always starts in the upright position and each episode is of length 400 (see https://gym.openai.com/envs/CartPole-v0 for more details). As the action space is discrete, there is no noise magnitude.
Second, about the optimization of the policy. Our optimization is a sequence of convex optimization problems. As the goal was to assess the influence of the length of the sequence on the quality of the policy, details about each optimization were voluntarily omitted. I am not sure as to which discussion on gradients and Hessians you allude to.
Finally, reviewers in a past submission asked for more details on the business case. It was rewritten to make it more readable to people reluctant to business jargon.

Meta-review:
role: PC
data: comment: The reviewers generally agreed that exploring policy search methods of this type is interesting, but the results presented in the paper are not at the standard required for publication. There are no comparisons of any sort, and the only task that is tested is trivially simple, so it's impossible to conclude anything about the effectiveness of the method. Despite the author promising to add additional experiments, nothing was added in the current draft. Besides this, reviewers raised concerns about the relevance of this approach to ICLR. The crucial point here is that it's unclear if the method will scale -- while there is nothing wrong in principle in proposing a general policy search method, there isn't really a compelling argument that can be made that it is suitable for learning representations if there is no plausible story for how it will scale to sufficiently complex policy classes that can actually learn representations.

Decision: Reject