Title: Learning to Optimize
Authors: Ke Li
Jitendra Malik
Venue: ICLR 2017
Keywords: Reinforcement Learning
Optimization

Abstract: Algorithm design is a laborious process and often requires many iterations of ideation and validation. In this paper, we explore automating algorithm design and present a method to learn an optimization algorithm. We approach this problem from a reinforcement learning perspective and represent any particular optimization algorithm as a policy. We learn an optimization algorithm using guided policy search and demonstrate that the resulting algorithm outperforms existing hand-engineered algorithms in terms of convergence speed and/or the final objective value. 

TL;DR: We explore learning an optimization algorithm automatically. 

Review discussion:
In each experiment you perform, you change the number of objective functions that the optimizer was trained on. Why? How sensitive is the optimizer to this choice?
Have you tried this method on a real dataset?
In general, generalization ability of the learned optimizer improves with the number of objective functions used for training. If the number of objective functions is too few, the learned optimizer tends to overfit to the objective functions in the training set and does not generalize to new objective functions. If the objective functions correspond to loss functions for training a model, this means that the learned optimizer only learns how to train the model on tasks seen during training and does not learn how to train the model in general.
The improvement in generalization ability gained by increasing the number of objective functions in the training set gradually diminishes, so we picked a training set size for each family of objective functions that resulted in a learned optimizer that generalizes. We did not do a grid search over the training set size, so it is possible that a slightly smaller training set would still allow the learned optimizer to generalize.
However, a much smaller training set than the ones we used results in an optimizer that fails to generalize. We found that using a bigger training set results in an optimizer that generalizes just as well, but not much better.
By “dataset”, I assume you mean a base-level dataset that defines a loss/objective function, rather than the meta-level dataset of loss/objective functions. It is conceptually somewhat tricky to use real datasets for training the optimizer, since a dataset corresponds to a *single* loss/objective function, and so we would need many diverse datasets to train an optimizer that can generalize. We are looking at collections of datasets that are suitable for this purpose and will explore this in future work.
"whose goal is to learn something about a family of tasks"
"we train the optimizer to learn on randomly generated tasks"
"we explicitly aim to avoid capturing any regularities about the task"
What is "randomly generated task"? a task where f(x) is random or a task sampled from some (as broad/narrow as you wish) family of tasks?
The former does not make much sense to learn, the latter is what you and the others are doing.
By "randomly generated task", we mean the objective function that takes the form of a loss function for training a model on a randomly generated task (so the objective function is random).
For example, if we want to train an optimizer for solving linear regression, we would train it on a dataset of regression problems on various randomly generated tasks. More formally, the dataset consists of the following objective functions:
{ f_{X,y}(v) = ||y – Xv||^2 : X, y drawn randomly }  (X is a matrix and y and v are vectors. X, y and v denote the data, the labels and the parameters respectively, so each objective function corresponds to one particular task/data-to-label mapping)
So, each f is randomly generated.
While it does not make sense to learn to predict the output of the f's given the input, it does make sense to learn how to optimize such functions. The goal is to learn an optimizer that is able to generalize to a new regression problem, f_{X',y'}(v), where X' and y' are possibly unrelated to the X's and y's seen during training.
This paper is a valuable addition to the recent learning to learn literature using neural networks.
(i) Could you please state what might be the advantages of using RL at the metal level instead of gradient descent?
(ii) If the meta-optimizer does RL, it could then be used to optimize over discrete spaces in addition to continuous spaces. I would love to hear your thoughts on this more general setup.
These are very good questions.
(i) In our view, one advantage to using RL as opposed to supervised learning is the ability of RL methods to learn policies that generalize better. RL methods are designed to overcome the problem of compounding errors, which is characterized by the following phenomenon: once a policy encounters gradients that are dissimilar from those seen during training, the step vector it chooses becomes slightly off, causing the next iterate to become more dissimilar from those encountered during training, resulting in more dissimilar gradients.
In other words, a policy trained in a supervised fashion often has difficulty in recovering from mistakes, which causes errors to compound over time and leads to divergence.
On the other hand, a policy trained using RL learns how to compensate for errors introduced from earlier iterations. In our experience, policies trained using RL tend to be more stable and generalize better to new objective functions, which may induce gradients that have different statistics from those seen during training.
(ii) This is an astute observation.
Indeed, because the proposed framework uses RL, it can be extended to discrete action spaces. This could potentially enable us to learn more general optimization algorithms whose update rules contain discrete components.
For example, we can learn a block coordinate descent algorithm, where the policy picks the coordinates to descend along, as well as the step that is used to update the coordinates.
This work appeared on arxiv just before we were able to release our learning to learn by gradient descent by gradient descent paper.
In our case, we were motivated by neural art and a desire to replace the lBFGS optimizer with a neural Turing machine (both have the same equation forms - see appendix of our arxiv version).
In the end, we settled on an LSTM optimizer that is also learned by SGD.
This paper, on the other hand, uses guided policy search to choose the parameter updates (policy).
Importantly, this paper also emphasizes the importance of transfer of optimizers to novel tasks.
This is undoubtedly a timely, good contribution to the learning to learn literature, which has gained great popularity in recent months.
just wanted to make a similar comment about the arXiv version
This paper proposes an approach to learning a custom optimizer for a given class optimization problems. I think in the case of training machine learning algorithms, a class would represent a model like “logistic regression”. The authors cleverly cast this as a reinforcement learning problem and use guided policy search to train a neural network to map the current location and history onto a step direction/magnitude.
Overall I think this is a great idea and a very nice contribution to the fast growing meta-learning literature.
However, I think that there are some aspects that could be touched on to make this a stronger paper.
My first thought is that the authors claim to train the method to learn the regularities of an entire class of optimization problems, rather than learning to exploit regularities in a given set of tasks. The distinction here is not terribly clear to me.
For example, in learning an optimizer for logistic regression, the authors seem to claim that learning on a randomly sampled set of logistic regression problems will allow the model to learn about logistic regression itself. I am not convinced of this, because there is bias in the randomly sampled data itself. From the paper in this case, “The instances are drawn randomly from two multivariate Gaussians with random means and covariances, with half drawn from each.” It seems the optimizer is then trained to optimize instances of logistic regression *given this specific family of training inputs* and not logistic regression problems in general. A simple experiment to prove the method works more generally would be to repeat the existing experiments, but where the test instances are drawn from a completely different distribution. It would be even more interesting to see how this changes as the test distribution deviates further from the training distribution.
Can the authors comment on the choice of architecture used here? Why one layer with 50 hidden units and softplus activations specifically? Why not e.g., 100 units, 2 layers and ReLUs?
Presumably this is to prevent overfitting, but given the limited capacity of the network, how do these results look when the dimensionality of the input space increases beyond 2 or 3?
I would love to see what kind of policy the network learns on e.g., a 2D function using a contour plot. What do the steps look like on a random problem instance when compared to other hand-engineered optimizers?
Overall I think this a really interesting paper with a great methodological contribution. My main concern is that the results may be oversold as the problems are still relatively simple and constrained.
However, if the authors can demonstrate that this approach produces robust policies for a very general set of problems then that would be truly spectacular.
Minor notes below.
Section 3.1 should you be using \pi_T^* to denote the optimal policy? You use \pi_t^* and \pi^* currently.
Are the problems here considered noiseless?
That is, is the state transition given an action deterministic? It would be very interesting to see this on noisy problems.
As per your suggestion, we tested the learned neural net optimizer on problems with different data distributions.
Specifically, we considered classification problems on datasets drawn from mixtures of different numbers of random Gaussians (so that as the number of mixture components increases, the data distributions at test time deviate more and more from those used for training). We found the learned optimizer to be fairly robust and appeared to work well for various numbers of mixture components we tested. We have updated the paper to include these results (see the appendix in the updated paper).
We tried increasing the number of layers and increasing the number of units in a layer. We found that the former resulted in overfitting and the latter resulted in similar performance. For higher dimensional problems, we found that a similar number of hidden units also suffices. We conjecture that the number of hidden units depends more on how complex the geometry of the objective function is rather than how high the dimensionality of the input is.
Indeed, if one were to consider the update rules of most existing optimization algorithms, the complexity of the update rule does not change with the dimensionality.
Thanks for the suggestion regarding contour plots. We have added them for random 2D logistic regression problems (see section 6.4 of the updated paper).
Good catch – \pi_t^* should have been \pi^*.
The state transition given an action is not deterministic because the next state contains the gradient and the objective value at the next iterate, which cannot be determined solely from the current state and action.
The current version of the paper is improved w.r.t. the original arXiv version from June.
While the results are exactly the same, the text does not oversell them as much as before. You may also consider to avoid words like "mantra", etc.
I believe that my criticism given in my comment from 3 Dec 2016 about "randomly generated task" is valid and you answer is not.
We are not sure what your criticism is exactly, but the section it pertains to is not the main point we wanted to make in our paper. Your point might be valid; hopefully we'll get a chance to discuss it further in person sometime.
This papers adds to the literature on learning optimizers/algorithms that has gained popularity recently. The authors choose to use the framework of guided policy search at the meta-level to train the optimizers. They also opt to train on random objectives and assess transfer to a few simple tasks.
As pointed below, this is a useful addition.
However, the argument of using RL vs gradients at the meta-level that appears below is not clear or convincing. I urge the authors to run an experiment comparing the two approaches and to present comparative results. This is a very important question, and the scalability of this approach could very well hinge on this fact.
Indeed, demonstrating both scaling to large domains and transfer to those domains is the key challenge in this domain.
In summary, the idea is a good one, but the experiments are weak.
We note that our comment on the possible advantages of RL over supervised learning was in response to your question asking us to speculate on this subject.
Accordingly, the comment conveys our opinion, which is based on the RL literature, e.g. [1-4], and was not part of the paper.
Of course, to settle this question definitively in this domain, empirical evidence may be needed, which would be an interesting topic for exploration in future work.
[1] Ross, Stéphane, and Drew Bagnell. "Efficient Reductions for Imitation Learning." AISTATS. 2010.
[2] Pomerleau, Dean A. "ALVINN: An Autonomous Land Vehicle in a Neural Network." NIPS. 1989.
[3] Levine, Sergey, and Vladlen Koltun. "Learning Complex Neural Network Policies with Trajectory Optimization." ICML. 2014.
[4] Venkatraman, Arun, Martial Hebert, and J. Andrew Bagnell. "Improving Multi-Step Prediction of Learned Time Series Models." AAAI. 2015.

Meta-review:
role: PC
data: comment: The authors propose an approach to learning optimization algorithms by framing the problem as a policy search task, then using the guided policy search algorithm. The method is a nice contribution to the "learning to learn" framework, and actually was developed simultaneously to a few papers that have since already been published. It's definitely a useful addition to this space.
 
 The biggest issue with this paper is that the results simply aren't that compelling. The methodology and proposed approach is nice, and the text is improved upon a previous version posted to Arxiv, but it seems that for most problems the results aren't that much better than some of the more common off-the-shelf optimization approaches that _don't_ require learning anything. Furthermore, in the one domain where the method does seem to (marginally) outperform the other methods, the neural net domain, it's unclear why we'd want to use the proposed approaches instead of SGD-based methods and their like (which of course everyone actually does for optimization).
 
 Pros:
 + Nice contribution to the learning to learn framework
 + Takes a different approach from past (concurrent) work, namely one based upon policy search
 
 Cons:
 - Experiments are not particularly compelling
 
 Overall, this work is a little borderline. Still, the PCs have determined that it was deserving of appearing at the conference. We hope the authors can strengthen the empirical validation for the camera ready version.
role: Author
data: value: comment: We have additional results in our new paper (https://arxiv.org/abs/1703.00441) that deals specifically with optimizing neural nets, which further validate the approach. 

Decision: Accept (Poster)