Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: "whose goal is to learn something about a family of tasks"
"we train the optimizer to learn on randomly generated tasks"
"we explicitly aim to avoid capturing any regularities about the task" 
What is "randomly generated task"? a task where f(x) is random or a task sampled from some (as broad/narrow as you wish) family of tasks? 
The former does not make much sense to learn, the latter is what you and the others are doing. 
review_title: randomly generated task

final_score: rating: null
review_content: In each experiment you perform, you change the number of objective functions that the optimizer was trained on. Why? How sensitive is the optimizer to this choice?

Have you tried this method on a real dataset?
review_title: Number of objective functions

final_score: rating: null
review_content: 
This paper is a valuable addition to the recent learning to learn literature using neural networks.

(i) Could you please state what might be the advantages of using RL at the metal level instead of gradient descent?
(ii) If the meta-optimizer does RL, it could then be used to optimize over discrete spaces in addition to continuous spaces. I would love to hear your thoughts on this more general setup.

review_title: RL vs supervised training of optimizers and discrete/continuous optimization spaces

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
This papers adds to the literature on learning optimizers/algorithms that has gained popularity recently. The authors choose to use the framework of guided policy search at the meta-level to train the optimizers. They also opt to train on random objectives and assess transfer to a few simple tasks.

As pointed below, this is a useful addition.

However, the argument of using RL vs gradients at the meta-level that appears below is not clear or convincing. I urge the authors to run an experiment comparing the two approaches and to present comparative results. This is a very important question, and the scalability of this approach could very well hinge on this fact. Indeed, demonstrating both scaling to large domains and transfer to those domains is the key challenge in this domain.

In summary, the idea is a good one, but the experiments are weak.


review_title: Review

final_score: rating: 7: Good paper, accept
review_content: The current version of the paper is improved w.r.t. the original arXiv version from June. While the results are exactly the same, the text does not oversell them as much as before. You may also consider to avoid words like "mantra", etc. 
I believe that my criticism given in my comment from 3 Dec 2016 about "randomly generated task" is valid and you answer is not.
review_title: null

final_score: rating: null
review_content: 
This work appeared on arxiv just before we were able to release our learning to learn by gradient descent by gradient descent paper. 
In our case, we were motivated by neural art and a desire to replace the lBFGS optimizer with a neural Turing machine (both have the same equation forms - see appendix of our arxiv version). In the end, we settled on an LSTM optimizer that is also learned by SGD. 

This paper, on the other hand, uses guided policy search to choose the parameter updates (policy). Importantly, this paper also emphasizes the importance of transfer of optimizers to novel tasks.

This is undoubtedly a timely, good contribution to the learning to learn literature, which has gained great popularity in recent months.
review_title: A good contribution to learning to learn

final_score: rating: 7: Good paper, accept
review_content: This paper proposes an approach to learning a custom optimizer for a given class optimization problems. I think in the case of training machine learning algorithms, a class would represent a model like “logistic regression”. The authors cleverly cast this as a reinforcement learning problem and use guided policy search to train a neural network to map the current location and history onto a step direction/magnitude. Overall I think this is a great idea and a very nice contribution to the fast growing meta-learning literature. However, I think that there are some aspects that could be touched on to make this a stronger paper.

My first thought is that the authors claim to train the method to learn the regularities of an entire class of optimization problems, rather than learning to exploit regularities in a given set of tasks. The distinction here is not terribly clear to me. For example, in learning an optimizer for logistic regression, the authors seem to claim that learning on a randomly sampled set of logistic regression problems will allow the model to learn about logistic regression itself. I am not convinced of this, because there is bias in the randomly sampled data itself. From the paper in this case, “The instances are drawn randomly from two multivariate Gaussians with random means and covariances, with half drawn from each.” It seems the optimizer is then trained to optimize instances of logistic regression *given this specific family of training inputs* and not logistic regression problems in general. A simple experiment to prove the method works more generally would be to repeat the existing experiments, but where the test instances are drawn from a completely different distribution. It would be even more interesting to see how this changes as the test distribution deviates further from the training distribution.

Can the authors comment on the choice of architecture used here? Why one layer with 50 hidden units and softplus activations specifically? Why not e.g., 100 units, 2 layers and ReLUs?
Presumably this is to prevent overfitting, but given the limited capacity of the network, how do these results look when the dimensionality of the input space increases beyond 2 or 3?

I would love to see what kind of policy the network learns on e.g., a 2D function using a contour plot. What do the steps look like on a random problem instance when compared to other hand-engineered optimizers?

Overall I think this a really interesting paper with a great methodological contribution. My main concern is that the results may be oversold as the problems are still relatively simple and constrained. However, if the authors can demonstrate that this approach produces robust policies for a very general set of problems then that would be truly spectacular.

Minor notes below.

Section 3.1 should you be using \pi_T^* to denote the optimal policy? You use \pi_t^* and \pi^* currently.

Are the problems here considered noiseless? That is, is the state transition given an action deterministic? It would be very interesting to see this on noisy problems.

review_title: Very nice methodological contribution, though I'm not quite convinced it is as general as claimed

Meta-review: The authors propose an approach to learning optimization algorithms by framing the problem as a policy search task, then using the guided policy search algorithm. The method is a nice contribution to the "learning to learn" framework, and actually was developed simultaneously to a few papers that have since already been published. It's definitely a useful addition to this space.
 
 The biggest issue with this paper is that the results simply aren't that compelling. The methodology and proposed approach is nice, and the text is improved upon a previous version posted to Arxiv, but it seems that for most problems the results aren't that much better than some of the more common off-the-shelf optimization approaches that _don't_ require learning anything. Furthermore, in the one domain where the method does seem to (marginally) outperform the other methods, the neural net domain, it's unclear why we'd want to use the proposed approaches instead of SGD-based methods and their like (which of course everyone actually does for optimization).
 
 Pros:
 + Nice contribution to the learning to learn framework
 + Takes a different approach from past (concurrent) work, namely one based upon policy search
 
 Cons:
 - Experiments are not particularly compelling
 
 Overall, this work is a little borderline. Still, the PCs have determined that it was deserving of appearing at the conference. We hope the authors can strengthen the empirical validation for the camera ready version.
Decision: Accept (Poster)