Reverse engineering learned optimizers reveals known and novel mechanisms
Authors: Niru Maheswaranathan
David Sussillo
Luke Metz
Ruoxi Sun
Jascha Sohl-Dickstein
Venue:  (2021)
Keywords: learned optimizers
optimization
recurrent neural networks
RNNs
interpretability

Abstract: Learned optimizers are algorithms that can themselves be trained to solve optimization problems. In contrast to baseline optimizers (such as momentum or Adam) that use simple update rules derived from intuitive principles, learned optimizers use flexible, high-dimensional, nonlinear parameterizations. Although this can lead to optimizers with better performance in certain settings, their inner workings remain a mystery. How is it that a learned optimizer is able to outperform a well tuned baseline? Has it learned a sophisticated method for combining existing optimization techniques, or is it implementing completely new behavior? In this work, we address these questions by visualizing and understanding learned optimizers. We study learned optimizers trained from scratch on three disparate tasks, and discovered that they have learned interpretable mechanisms, including: momentum, gradient clipping, schedules, and a new form of learning rate adaptation. Moreover, we show how the dynamics of trained learned optimizers enables these behaviors. Our results elucidate the previously murky understanding of what learned optimizers learn, and establishes tools for interpreting future learned optimizers.

Under review as a conference paper at ICLR 2021
REVERSE ENGINEERING LEARNED OPTIMIZERS REVEALS KNOWN AND NOVEL MECHANISMS
Anonymous authors
Paper under double-blind review
ABSTRACT
Learned optimizers algorithms that can themselves be trained to solve op- timization problems. In contrast to baseline optimizers (such as momentum or Adam) that use simple update rules derived from theoretical principles, learned optimizers use flexible, high-dimensional, nonlinear parameterizations. Although this can lead to better performance in certain settings, their inner workings remain a mystery. How is learned optimizer able to outperform a well tuned baseline? Has it learned sophisticated combination of existing optimization techniques, or is it implementing completely new behavior? In this work, we address these ques- tions by careful analysis and visualization of learned optimizers. We study learned optimizers trained from scratch on three disparate tasks, and discover that they have learned interpretable mechanisms, including: momentum, gradient clipping, learning rate schedules, and new form of learning rate adaptation. Moreover, we show how the dynamics of learned optimizers enables these behaviors. Our results help elucidate the previously murky understanding of how learned opti- mizers work, and establish tools for interpreting future learned optimizers.
INTRODUCTION
Optimization algorithms underlie nearly all of modern machine learning. A recent thread of research
is focused on learning optimization algorithms, by directly parameterizing and training an optimizer on a distribution of tasks. These so-called learned optimizers have been shown to outperform base-
line optimizers in restricted settings (Andrychowicz et al.) 2016 Wichrowska et al. 2017 Lv et al. 2017
Bello al. 2017 Li & Malik Metz et al. 2019, 2020).
Despite improvements in the design, training, and performance of learned optimizers, fundamental questions remain about their behavior. We understand remarkably little about how these systems work. Are learned optimizers simply learning clever combination of known techniques? Or do they learn fundamentally new behaviors that have not yet been proposed in the optimization literature? they did learn
optimization technique, how would we know?
Contrast this with existing "hand-designed" optimizers such as momentum (Polyak) 1964). AdaGrad (Duchi al. 2011) 1). RMSProp (Tieleman & Hinton, 2012), Adam (Kingma & Ba 2014). These algorithms are motivated and analyzed via intuitive mechanisms and theoretical principles (such as accumulating update velocity in momentum, or rescaling updates based on gradient magnitudes in RMSProp or Adam). This understanding of underlying mechanisms allows future studies to build on these techniques by highlighting flaws in their operation (Loshchilov & Hutter 2018), studying convergence (Reddi et al. 2019), and developing deeper knowledge about why key mechanisms work (Zhang et 2020). Without analogous understanding of the inner workings of a learned optimizers, it is incredibly difficult to analyze or synthesize their behavior.
In this work, we develop tools for isolating and elucidating mechanisms in nonlinear, high- dimensional learned optimization algorithms ($3). Using these methods we show how learned op- timizers utilize both known and novel techniques, across three disparate tasks. In particular, we demonstrate that learned optimizers learn momentum (14.1), gradient clipping (44.2). learning rate schedules (44.3), and a new type of learning rate adaptation ($4.4). Taken together, our work can be seen part of new approach scientifically interpret and understand learned algorithms. We provide code for training and analyzing learned optimizers, as well as the trained weights for the learned optimizers studied here, redacted URL.

---

Under review as a conference paper at ICLR 2021
2 BACKGROUND AND RELATED WORK
We are interested in optimization problems that minimize a loss function (f) over parameters (æ)). We focus on first-order optimizers, which at iteration k have access to the gradient o f(()) and produce an update Azy. These are component-wise optimizers that are applied to each parameter or component (x,) of the problem in parallel. Standard optimizers used in machine learning (e.g. momentum, Adam) are this category Going forward, we use x for the parameter to optimize, g for its gradient, k for the current iteration, and drop the parameter index (i) to reduce excess notation.
An optimizer has two parts: the optimizer state (h) that stores information about the current problem, and readout weights (w) that update parameters given the current state. The optimization algorithm is specified by the initial state, the state transition dynamics, and readout, defined as follows:
ht k-1 = F(h", 9(h",9")
(1)
x A-1 = x² - w h+1 A+1
(2)
where h is the optimizer state, F governs the optimizer state dynamics, and w are the readout weights. Learned optimizers are constructed by parameterizing the function F, and then learning those parameters along with the readout weights through meta-optimization (detailed in Appendix Hand-designed optimization algorithms, by distinction, specify these functions at the outset.
For example, in momentum, the state is scalar (known the velocity) that accumulates a weighted average of recent gradients. For momentum and other hand-designed optimizers, the state variables dindeieniional, and their dynamics high-dimensional state variables, and the potential for rich, nonlinear dynamics. As these systems learn complex behaviors, it has historically been difficult to extract simple, intuitive descriptions of the behavior of a learned optimizer.
straightforward. In contrast, learned optimizers have
Our work is heavily inspired by recent work using neural networks parameterize optimizers. Andrychowicc et al. (2016) originally showed promising results on this front, with additional studies improving robustness (Wichrowska al. 2017: Lv et al. 2017), meta-training (Metz et al. 2019). and generalization (Metz et al. (020)) of learned optimizers.
We also build on recent work reverse engineering dynamical systems. (ussiilo & Barak (2013) showed how linear approximations to nonlinear dynamical systems can yield insight into the algo- rithms used by these networks. More recently, these techniques have been applied to understand trained RNNs in variety of domains, from natural langauge processing (Maheswaranathan et al., 2019a Maheswaranathan & Sussillo) (2020) to neuroscience (Schaeffer et al. (2020). Additional work on treating RNNs as dynamical systems has led to insights into their computational capabili- ties (Jordan et al. 2019
Krishnamurthy al. 2020 Can al. 2020).
3 METHODS
3.1 TRAINING LEARNED OPTIMIZERS
We parametrize the learned optimizer with a recurrent neural network (RNN), similar to Andrychww
iez et al. (2016). Specifically, we gated recurrent unit (GRU) (Cho et al. 2014) with 256 units. The only input to the optimizer is the gradient. The RNN is trained by minimizing a meta-objective, which we define as the average training loss when optimizing a target problem. See Appendix C.2 for details about the optimizer architecture and meta-training procedures.
We trained these learned optimizers on each of three tasks. These tasks were selected because they are fast to train (particularly important for neta-optimization) and covered a range of loss surfaces (convex and non-convex, low- and high-dimensional)::
Convex, quadratic: The first task consists of random linear regression problems f(x) _ Al bll.. where A and b are randomly sampled. Much of our theoretical understanding of the behavior of optimization algorithms derived using quadratic functions. in part because they have a constant
Notable exceptions include quasi-Newton methods such as L-BFGS NNcedall & Wright 2006 or K- FAC Martenss & Grosse 2015).
2

---

Under review as a conference paper at ICLR 2021
(a)
Linear Regression
(b) Rosenbrock
(c)
Neural network training
1.0
3
10a
1.5
y-coordinate
o08 Bolg spales
Density
Feature
0.5
0.0
10
10a
-2
-1.5
2.5
Condition number (K)
X- x-coordinate
Feature
10a
100
10
101
Loss
Loss
Loss
10-3
10-1
10-1_
10-5
10-3
100
200
100
200
Iteration
Iteration
Iteration
RMSProp
Momentum
Adam
Learned Optimizer
Figure 1: Learned optimizers outperform well tuned baselines on three different tasks: (a) linear regression, (b) the Rosenbrock function, and (c) training a neural network on the two moons dataset. Upper row: Task schematics (described in $ Bottom row: Optimizer performance, shown as loss curves (mean + std. error over 128 random seeds) for different optimizers: momentum (orange), RMSProp (yellow), Adam (red) and learned optimizer (blue).
Hessian (AT A) over the entire parameter space. The choice of how to sample the problem data A and b will generate particular distribution of Hessians and condition numbers. The distribution of condition numbers for our task distribution shown in Figure Th.
Non-convex, low-dimensional: The second task is minimizing the Rosenbrock function (Rosen- brock 1960). a commonly used test function for optimization. It is a non-convex function which contains a curved valley and single global minimum. The function is defined over two parame- ters (a and y) as f(z, y) (1 x²² + 100(y - =²)². The distribution of problems for this task consists of the same loss function with different initializations sampled uniformly over grid, The rosenbrock loss surface is shown in Figure on a log scale to highlight the curved valley. The grid used sample initializations the same the grid shown in the figure; the x-coordinate sampled uniformly from (-2, 2) and the y-coordinate is sampled uniformly from (-1
Non-convex, high-dimensional: The third task involves training a neural network classify toy dataset, the two moons dataset (Figure [p) As the data are not linearly separable, nonlinear classifier is required to solve the task. The optimization problem to train the weights of a three hidden layer fully connected neural network, with 64 units per layer and tanh nonlinearities. The distribution of problems involves sampling the initial weights of the network.
On each task, additionally tuned three baseline optimizers (momentum, RMSProp, and Adam). We selected the hyperparameters for each problem out 2500 samples randomly drawn from a grid. Details about the exact grid ranges used for each task are in Appendix C33
Figure [ (bottom row) compares the performance of the learned optimizer (blue) to baseline opti- mizers (red, yellow, and orange), each of the three tasks described above. Across all three tasks, the learned optimizer outperforms the baseline optimizers on the meta-objective (Apperiii (Appendix Fig.
3.2 PARAMETER UPDATE FUNCTION VISUALIZATIONS
First, we introduce a visualization tool get handle on what an optimizer is doing. Any optimizer, at particular state, can be viewed as a scalar function that takes in a gradient (g) and returns a
2 As the meta-objective the average training loss during an optimization run, it naturally penalizes the training curve earlier in training (when loss values are large). This explains the discrepancy in the curves for linear regression (Fig. [. bottom) where momentum continues to decrease the loss late in training. Despite this, the learned optimizer has overall smaller meta-ohjective due having lower loss earlier itcrations.
3

---

Under review as a conference paper at ICLR 2021
(a) Gradient Descent
(b) Gradient Clipping
(c) Momentum
(d) RMSProp
0.02
Parameter
update (Ax) 0.00
-0.02
-1
Gradient
Figure 2: Visualizing optimizer behavior with update functions (see (3. for details) for different commonly used optimization techniques. (a) Gradient descent is a (stateless) linear function, whose slope is the learning rate. (b) Gradient clipping saturates the update, beyond threshold. (c) Mo- mentum introduces a vertical offset depending on the accumulated velocity (colors indicate different values of the accumulated momentum).
(d) RMSProp changes the slope (effective learning rate) of the update (colors denote changes in the state variable, the accumulated squared gradient).
change in the parameter (Az). We refer to this the optimizer update function. Mathematically, the update function is computed as the the state update projected onto the readout, Az w F(h, g), following equations and (2). In addition, the slope of this function with respect to the input be thought of as the effective learning rate particular state We will use
gradient 0Ar
8g
both the overall update function and the effective learning rate to understand optimizer behavior.
instructive visualize these update functions for commonly used optimizers (Figure 2}. For gradient descent, the update (Ar = -ag) is stateless and is always a fixed linear function whose slope is the learning rate, (Fig, 21. Gradient clipping is also stateless, but is saturating function of the gradient (Fig. 2b). For momentum, the update is Ar = - a(( + 3g), where v denotes the momentum state (velocity) and 8 the momentum hyperparameter. The velocity adds an offset to the update function (Fig. 2b. For adaptive optimizers such as RMSProp, the state variable changes the slope, or effective learning rate, within the linear region of the update function (Fig. 2d). As the optimizer picks up positive (or negative) momentum, the curve shifts downward (or upward), thus incorporating a bias to reduce (or increase) the parameter.
Now, what about learned optimizers, dimensional state variables? One advantage of update functions that, as scalar functions, they be easily visualized and compared to the known methods in Figure Whether not the under- lying hidden states are interpretable, for a given learned optimizer, remains to be seen.
optimizers with much more complicated high-
3.3 A DYNAMICAL SYSTEMS PERSPECTIVE
We study the behavior of optimizers by treating them dynamical systems. This perspective has yielded number of intuitive and theoretical insights (Su et al., 2014 Wilson et al. 2016 Shi al.) 2019). In order to understand the dynamics of learned optimizer, approximate the nonlinear dynamical system via linearized approximations (Strogatz 2018).
These linear approximations hold near fixed points of the dynamics. Fixed points points in the state space of the optimizer, where- as long as input gradients do not perturb it it-the system does not move. That is, approximate fixed point h" satisfies the following: h" F(h",g for particular input g"
We numerically find approximate fixed points (Sussillo & Barak 2013 Maheswaranathan al., 2019b), by solving an optimization problem where we find points (h) that minimize the following h1ll The solutions this problem (there may be many) the approximate fixed points of the system F' for given input, g". There may be different fixed points for different values of the input (g). First we will analyze fixed points when g" introduce additional behavior that occurs as varies (84.4).
loss: / [[
(:4.1) and then later
We compute this slope at O. We find that the update function is always linear in the middle with saturation the extremes, thus the slope
gond summary of the effective learning rate.
4

---

Under review as a conference paper at ICLR 2021
4 RESULTS
We discovered four mechanisms in the learned optimizers responsible for their superior perfor- mance. In the following sections, we go through each in detail, showing how it implemented. For the behaviors that are task dependent, we highlight how they vary across tasks.
4.1
MOMENTUM
Linear Regression
Rosenbrock
Neural network training
Total variance
PC #4
PC #2
0
max
O
max
0
maX-
Iteration
200
200
PC #2
PC #1
PC
10
0.02
0.015
Update (Ax)
0-
0.00
0.000
-10
-0.02
-0.015
-2
0
-10
-0.01
0.00
0.01
Gradient (g)
0.5
max
0.5
max
0.5
max
are Leannn
Learning Rate
Learning Rate
3(A)
0.0
0.0
0.0
-0.5
-0.5
-0.5
(A)
0
Figure 3: Momentum in learned optimizers. Each column shows the same phenomena, but for opti- mizers trained on different tasks. Top row: Projection of the optimizer state around a convergence point (black square). Inset: the total variance of the optimizer states over test problems goes to zero as the trajectories converge. Middle row: visualization of the update functions (3322 along the slow mode of the dynamics (colored lines correspond to arrows in (a)). Along this dimension, the effect on the system is to induce an offset in the update, just as in classical momentum (cf. Fig. ( Bot- tom row: Eigenvalues of the linearized optimizer dynamics at the convergence fixed point (black square in (a)) plotted in the complex plane. The eigenvalue magnitudes are momentum timescales, and the color indicates the corresponding learning rate. See 4.1 for details.
We discovered that learned optimizers implement momentum using approximate linear dynamics (Figure 3) First, we found that each optimizer converges to single global fixed point of the dynamics. We can see this as the total variance of hidden states across test problems goes zero as the optimizer is run (inset in Fig. 3). The top row of Fig. 3 is a projectior of the hidden state space, showing the convergence fixed point (black square). Around this fixed point, the dynamics are organized along line (gray circles). Shifting the hidden state along this line (indicated by colored arrows) induces a corresponding shift in the update function (middle row of Fig. 3, similar to what is observed in classical momentum (cf. Fig, 2
At a fixed point, we can lincarly approximate the nonlincar dynamics of the optimizer using the Jacobian of the state update. This Jacobian is matrix with N eigenvalues and eigenvectors. Writing
"We use principal components analysis (PCA) to project the nigh-dimensional hidden state into 2D. De- pending on the task, we found that different mechanisms would correspond to different principal components (hence the different numbers on the and y- axes of the top row of Fig.
5

---

Under review as a conference paper at ICLR 2021
the update in these coordinates allows to rewrite the learned optimizer as a momentum algorithm (see Appendix B). albeit with N timescales instead of just one. The magnitude of the eigenvalues exactly momentum timescales, each with corresponding learning rate. Note that this type of optimizer has been previously proposed as aggregated momentum by Lucas et al. (2018).
We find that learned optimizers use single mode to implement momentum. The bottom row of shows the eigenvalues (computed at the convergence fixed point) in the complex plane, colored by that mode's 's learning rate (see Appendix B for how these quantities are computed). This reveals a single dominant eigenmode (colored in purple), whose eigenvector corresponds to the momentum direction and whose eigenvalue is the corresponding momentum timescale.
While we analyze the best performing learned optimizers in the main text, we did find a learned optimizer on the linear regression task that had slightly worse performance but strongly resembled classical momentum; in fact, this optimizer recovered the optimal momentum parameters for the particular task distribution. We analyze this optimizer in Appendix Aaas it is instructive for under- standing the momentum mechanism.
4.2
GRADIENT CLIPPING
Linear Regression
Rosenbrock
Neural network training
20
0.1
0.05
Update (Ax)
0.0
0.00
-20
-0.1
-0.05
-5
0
5
-1000
0
1000
-0.1
0.0
0.1
Gradient (g)
Density (log scale)
0
5
-1000
1000
-0.05
0.00
0.05
Gradient (g)
Figure 4: Gradient clipping in learned optimizer. Top row: The update function computed the initial state saturates for large gradient magnitudes. The effect of this is similar to that of gradient clipping (ef. Fig. 2p). Bottom row: the empirical density of encountered gradients for each task (note the different ranges along the x-axes). Depending on the problem, the learned optimizer can tune its update function so that most gradients are in the linear portion of the function, and thus not use gradient clipping (seen in the linear regression task, left column) or can potentially use more of the saturating region (seen the Rosenbrock task, middle column).
In standard gradient descent, the parameter update is linear function of the gradient. Gradient clipping (Pascanu et al. 2013) instead modifies the update to be a saturating function (Fig.2b).
We find that learned optimizers also use saturating update functions the gradient magnitude in- creases, thus learning a soft form of gradient clipping (Figure (p. Although we show the saturation for particular optimize state (the initial state, top row of Fig. thresholds are consistent throughout the state space.
find that these saturating
The strength of the clipping effect depends on the training task. We can see this by comparing the update function to the distribution of gradients encountered for a given task (bottom row of Fig, 4 For some problems, such linear regression, the learned optimizer largely stays within the linear region of the update function (Fig. [ left column). For others, such as the Rosenbrock problem (Fig. 4 right column), the optimizer utilizes more of the saturating part of the update function.

---

Under review as a conference paper at ICLR 2021
Linear Regression
Rosenbrock
Neural network training
Component #2
0
0
0
-4
-4
4
-4
-4
4
Component #1
Effective learning rale
15
0.003
10
0.002
5
0.001
0.000
0
100
200
100
200
100
200
Iteration
Figure 5: Learning rate schedules mediated by autonomous dynamics. Top row: Low- Low-dimensional projection of the dynamics of the learned optimizer in response zero gradients (no input). These autonomous dynamics allow the system to learn a learning rate schedule (see (4.3). Bottom row: Effective learning (measured as the slope of the update function) function of iteration during the autonomous trajectories in the top row. We only observe a clear learning rate schedule in the linear regression task (left column), which includes both warm-up and decay. For context, dashed lines indicate the best (tuned) learning rate for momentum.
4.3 LEARNING RATE SCHEEDLES
Practitioners often tune a learning rate schedule, that is, a learning rate that varies per iteration. Originally motivated for use with stochastic gradients to guarantee convergence to fixed point (Robbins& Monro 195T), schedules are now used more broadly (Schaul al. 2013] Smith al.] 2017 Ge et al.) 2hoi
Choi et al. 2019). These schedules are often a decaying function of the iteration iaaiionn-eaninng the learning rate goes down as optimization progresses - although Goyal et al. 2017) use an additional (increasing) warm-up period, and even more exotic schedules have also
been proposed (Loshchilov Hutter 2016 Smith 2017 Li & Arora 2019).
We discovered that learned optimizers input driven- -aann-da dyyamics. By moving the initial state (which are trainable parameters) away from the convergence fixed point, then in the absence of input, autonomous dynamics will encode a particular trajectory as a function of the iteration as the system relaxes to the fixed point. Fur- thermore, this autonomous trajectory evolves in subspace orthogonal to the readout weights used to update the parameters. This ensures that the autonomous dynamics themselves do not induce changes in the parameters, but only change the effective learning rate.
implement schedule using autonomous that is, not
For the linear regression task, we found 2D subspace where the autonomous dynamics occur (Figure 5). driving the system from the initial state (black circle) to the final convergence point (black square). The shaded gray points in the top row of Fig. 5 are slow points of the dynamics (Sssillo & Barak (013), which shape the trajectory.
By computing the effective learning rate (slope of the update function) of the system along the autonomous trajectory, we can study the effect of these dynamics. We find that for the linear regres- sion task (left column of Fig. 5). the system has learned to initially increase the learning rate over
We found this subspace by looking for dimensions that maximized the variation of the autonomous trajec- tory; this subspace is different from the low-dimensional projection used in Figures and

---

Under review as a conference paper at ICLR 2021
the course of 25 iterations, followed by a roughly linear decay. We find that the learned optimizer trained on the other tasks does not learn to use learning rate schedule.
4.4
LEARNING RATE ADAPTATION
Linear Regression
Rosenbrock
Neural network training
10
10
10
PC #4
PC #2
PC
-10
-10
-10
-15
0
15
-15
0
15
-15
0
15
PC #2
PC #1
PC #1
20
0.02
0.01
Update (Ax)
0
0.00
0.00
-20
-0.02
-0.01
0
-10
0
10
-0.005
0.000
0.005
Gradient (g)
Effective leaming rate
20
0.0020
2.0
15
0.0015
1.5
10
0.0010
1.0
5
0.0005
0.5
0.0000
0.0
10-1
100
10
-0-3
100
103
10
10-1
Gradient magnitude (lgl)
Figure 6: Learning rate adaptation in learned optimizers. Top row: Approximate fixed points (col- ored circles) of the dynamics computed for different gradients reveal an S-curve structure. Middle row: Update functions (:3.2) computed different points along the S-curve (corresponding rows from the top row). The effect of moving towards the edge of the S-curve is to make the update function more shallow (thus have a smaller effective learning rate, cf. Fig. 21.. The effect similar along both arms; only one arm is shown for clarity. Bottom row: Summary plot showing the effec- tive learning rate along each arm of the S-curve, for negative (red) and positive (green) gradients. TTe overall effect is to reduce learning rates when the gradient magnitude is large.
The final mechanism we discovered is type of learning rate adaptation. The effect of this mecha- nism decrease the learning rate of the optimizer when large gradients are encountered, similar to adaptive learning rate methods such Adagrad RMSProp. However, the mechanism that enables this behavior in learned optimizers is novel.
To understand how momentum is implemented by learned optimizers, we studied the linear dynam- ics the optimizer near a fixed point (44.1)) That fixed point was found numerically (:3.3) by searching for points h* that satisfy h" 2s F(h", .g"). where hold the input (gradient) fixed at zero (g" = 0). To understand learning rate adaptation, we need to study the dynamics around fixed points with non-zero input. We find these fixed points by setting g" to fixed non-zero value.
8

---

Under review as a conference paper at ICLR 2021
We sweep the value of g" over the range of gradients encountered for particular task. For each value, we find a single corresponding fixed point. These fixed points are arranged in an S-curve, shown in the top to find that fixed point. One arm of this curve corresponds to negative gradients (red), while the other corresponds to positive gradients (green). The tails of the S-curve correspond the largest magnitude gradients encountered by the optimizer, and the central spine of the S-curve contains the final convergence poin
of Figure 6 The color of each point corresponds to the value of g" used
These fixed points are all attractors, meaning that if we held the gradient fixed at a particular value, the hidden state dynamics would converge that corresponding fixed point. In reality, the input (gradient) to the optimizer is constantly changing, but if large (positive or negative) gradient is seen for number of timesteps, the state will be attracted to the tails of the S-curve. As the gradient goes to zero, the system converges to the final convergence point in the central spine.
What the functional benefit of these additional dynamics? To understand this, we visualize the update function corresponding to different points along the S-curve (middle row of Fig. 6). The curves are shown for just one arm of the S-curve (green, corresponding to positive gradients) for visibility, but the effect is the symmetric across the other arm well. We see that as we move along the tail of the S-curve (corresponding to large gradients) the slope of the update function becomes more shallow, thus the effect is to decrease the effective learning rate.
The changing learning rate along both arms of the S-curve are shown in the bottom row of Fig, 6 for positive (green) and negative (red) gradients, plotted against the magnitude of the gradient log scale. This allows the system to increase its learning rate for smaller gradient magnitudes. For context, the best tuned learning rate for classical momentum for each task is shown as dashed line.
DISCUSSION
In this work, we trained learned optimizers on three different optimization tasks, and then studied their behavior. We discovered that learned optimizers learn plethora of intuitive mechanisms: momentum, gradient clipping, schedules, and learning rate adaptation. While the coarse behaviors qualitatively similar across different tasks, the mechanisms are tuned for particular tasks.
While we have isolated individual mechanisms, we still lack a holistic picture of how a learned op- timizer stitches these mechanisms together. One may be able to extract or distill compressed op- timizer from these mechanisms, perhaps using data-drived techniques (Brunton et al. 2016 Cham- pion et 2019) or symbolic regression (Cranmer al. 2020).
Finally, we are left wondering why the learned optimizers we have studied learned these particular mechanisms. Presumably, they have to do with properties of the tasks used train the optimizer. What are these quantitative properties (e.g. curvature, convexity, or something else)? Understanding these relationships would allow us to take learned optimizers trained one setting, and know when and how to apply them to new problems.
Previously, not much was known about how learned optimizers worked. The analysis presented here demonstrates that learned optimizers capable of learning number of interesting optimization phenomena. The methods we have developed (update functions and visualization of state dynam- ics) should be part of growing toolbox nonlinear dynamics of learned optimizers, and meta-learned algorithms more generally.
to extract insight from the nigh-dimensional
'The top of Figure uses the same projection the top row of Figure just zoomed

---

Under review as a conference paper at ICLR 2021
REFERENCES
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in neural information processing systems, pp. 3981-3989, 2016.
Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoc V Le. Neural optimizer search with rein- forcement learning. arXiv preprint arv: 170.077477 2017.
Steven L Brunton, Joshua L Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113(15))3932-3333 2016.
Tankut Can, Kamesh Krishnamurthy, and David J Schwab. Gating creates slow modes and controls phase-space complexity in grus and Istms. arXiv preprint arXiv:2002.00022, 2020.
Kathleen Champion, Bethany Lusch. J Nathan Kutz, and Steven L Brunton. Data-driven discovery of coordinates and governing equations. Proceedings of the National Academy Sciences, 116 (45):22445-22451 2019.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406. 2014.
Dami Choi, Christopher Shallue, Zachary Nado, Jaehoon Lee, Chris J Maddison, and George E Dahl. On empirical comparisons of optimizers for deep learning. arXiv preprint arXiv:1910.05446, 2019.
Miles Cranmer, Alvaro Sanchez-Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho. Discovering symbolic models from deep learning with inductive biases. arXiv preprint arXiv:2006. 1287, 2020.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Rong Ge, Sham M Kakade, Rahul Kidambi, and Praneeth Netrapalli. The step decay schedule: A near optimal, geometrically decaying learning rate procedure for least squares. In Advances in Neural Information Processing Systems, pp. 14977-14988, 2019.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arvi: 1tt:Xxi::10000000 026777267, 2017.
Ian Jordan, Piotr Aleksander Sokol, and Memming Park. Gated recurrent units viewed through the lens of continuous time dynamical systems. arXiv preprint arXiv: 1906. 900600005, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.
Kamesh Krishnamurthy, Tankut Can, and David J Schwab. Theory of gating in recurrent neural networks. arXiv preprint 14822,,,07. 2020.
Ke Li and Jitendra Malik. Learning to optimize. arXiv preprint arXiv: 1606.11886...... 2016.
Zhiyuan Li and Sanjeev Arora. An exponential learning rate schedule for deep learning. arXiv preprint arXiv:1910.07454, 2019.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. preprint N.XXiv:6608.09900 2016.
arXiv
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2018.
James Lucas, Shengyang Sun, Richard Zemel, and Roger Grosse. Aggregated momentum: Stability through passive damping. arXiv preprint arXiv: vrXiv:180400033 1804.00325 2018.
10

---

Under review as a conference paper at ICLR 2021
Kaifeng Lv, Shunhua Jiang, and Jian Li. Learning gradient descent: Better generalization and longer horizons. arXiv preprint Sediv:1703.036655 2017.
Niru Maheswaranathan and David Sussillo. How recurrent networks implement contextual process-
ing in sentiment analysis. arXiv preprint arXiv:2004.08013, 2020.
Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Re- verse engineering recurment networks for sentiment classification reveals line attractor dynamics. In Advances in Neural Information Processing Systems, pp. 15696-15705, 2019a.
Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Uni- versality and individuality in neural dynamics across large populations of recurrent networks. In Advances in neural information processing systems. pp. 15629-15641, 2019b.
James Martens and Roger Grosse. Optimizing neural networks with kronecker factore approximate curvature. In International conference on machine learning, pp. 2408-2411111 2008-2417, 2015.
Luke Metz, Niru Maheswaranathan, Jeremy Nixon, Daniel Freeman, and Jascha Understanding and correcting pathologies in the training of learned optimizers. In International Conference on Machine Learning, pp. 4556-4565, 2019.
Luke Metz, Niru Maheswaranathan, C. Daniel Freeman, Ben Poole, and Jascha Sohl-Dickstein. Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves, 2020.
Jorge Nocedal and Stephen Wright. Numerical optimization. Springer Science & Business Media, 2006.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference machine learning, pp. 1310-1318. 2013.
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. USSR
Computational Mathematics and Mathematical Physics, 4(5):1-17, 1964.
Sashank Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. arXiv
preprint arXiv: 190400979 2019.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400-407, 1951.
HoHo Rosenbrock. An automatic method for finding the greatest or least value of a function. The Computer Journal, 3(3):155-144 1960.
Rylan Schaeffer, Mikail Khona, Leenoy Meshulam, Ila Rani Fiete, al. Reverse-engineering recur- rent neural network solutions to a hierarchical inference task for mice. bioRxiv, 2020.
Tom Schaul, Sixin Zhang, and Yann LeCun. No more pesky learning rates. In International Conference on Machine Learning, pp. 343-351, 2013.
Bin Shi, Simon S Du, Weijie Su, and Michael Jordan. Acceleration via symplectic discretization of high-resolution differential equations. In Advances in Neural Information Processing Systems, pp. 5744-5752, 2019.
Leslie N Smith. Cyclical learning rates for training neural networks. 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 464-472. IEEE, 2017.
Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoe V Le. Don't decay the learning rate, increase the batch size. arXiv preprint arXiv:1711 1.0.11091,.049,,, 2017.
Steven H Strogatz. Nonlinear dynamics and chaos with student solutions manual: With applications to physics, biology, chemistry, and engincering. CRC press, 2018.
Weijie Su, Stephen Boyd, and Emmanuel Candes. A differential equation for modeling nesterov's accelerated gradient method: Theory and insights. In Advances in neural information processing systems, pp. 2510-2518, 2014.
11

---

Under review as a conference paper at ICLR 2021
David Sussillo and Omri Barak. Opening the black box: low-dimensional dynamics in high- dimensional recurrent neural networks. Neural computation, 25(3):626 649, 2013.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26- 31, 2012.
Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize. arXiv preprint arXiv: 703114 2017.
Ashia C Wilson, Benjamin Recht, and Michael I Jordan. Iyapunov analysis of momentum methods in optimization. arXiv preprint Xiv: 161 1.02635, 2016.
Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse. Understanding short-horizon bias in
stochastic meta-optimization. arXiv preprint arXiv:1803.0000
2018.
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping acceler- ates training: A theoretical justification for adaptivity. In International Conference on Learning Representations, 2020.
12

---

that this mode Under review as a conference paper at ICLR 2021
A A LEARNED OPTIMIZER THAT RECOVERS MOMENTUM
When training learned optimizers on the linear regression tasks, we noticed that we could train a learned optimizer that seemed strongly mimic momentum, both in terms of behavior and perfor- mance. With additional training, the learned optimizer would eventually start to outperform momen- tum (Figure Th.. We highlight this latter, better performing optimizer in the main text. However, is still instructive to go through the analysis for the learned optimizer that mimics momentum. This example in particular clearly demonstrates the connections between eigenvalues, momentum, and dynamics.
The learned optimizer that performs as well as momentum learns mimic linear dynamics (we also used a GRU for this optimizer). That is, the dynamics of the nonlinear optimizer could be very well approximated using& linearization computed at the convergence point. This linearization is shown in Figure We find single mode pops out of the bulk of eigenvalues (Fig. 7a). Additionally, if we plot these eigenvalue magnitudes, which are the momentum time scales, against the corresponding extracted learning rate of each mode, discussed below in Appendix B
also has a large learning rate compared to the bulk (top right blue circle in Fig. ( Moreover, the extracted momentum timescale and learning rate for this mode essentially exactly match the best tuned hyperparameters (gold star in Fig. (D) from tuning the momentum algorithm directly, which also be derived from theory.
Finally, if we extract and run just the dynamics along this particular mode, we see that it matches the behavior of the full, nonlinear optimizer almost exactly (Fig. 77p). This suggests that in this scenario, the learned optimizer has simply learned the single mechanism of momentum. Moreover, the learned optimizer has encoded the best hyperparameters for this particular task distribution in its dynamics. Our analysis shows how to separate the overall mechanism (linear dynamics along eigenmodes) from the particular hyperparameters of that mechanism (the specific learning rate and momentum timescale).
B LINEARIZED OPTIMIZERS AND AGGREGATED MOMENTUM
In this section, we elaborate on the connections between linearized optimizers and momentum with multiple timescales. We begin with our definition of an optimizer, equations (I) and (2) in the main text:
h
F(h ((h",g" ,g") zt w h k+1
k-
(a) Jacobian eigenvalues
(b) Extracted hyperparameters
(c) Extracted vs Full
10
0.8
10F
a(A)
Momentum
0.6
Loss
0.4
10:1-
0.2
-1
150
200
0.0
Iteration (k)
0
10-4
10-2
100
S((A)
Learning rate (a)
Figure 7: A learned optimizer that recovers momentum on the linear regression task. (a) Figenvalues of the Jacobian of the optimizer dynamics evaluated at the convergence fixed point. There is single eigenmode that has separated from the bulk. (b) Another way of visualizing eigenvalues by translating them into optimization parameters (learning rates and momentum timescales), described Appendix When we this this particular optimizer. we see that the slow eigenvalue (momentum timescale closest to one) also has a large learning rate. These specific hyperparameters match the best tuned momentum hyperparametrs for this task distribution (gold star). (c) When extract and just the dynamics along this single mode (orange dashed line), we see that this reduced optimizer matches the full, nonlinear optimizer (solid line) almost exactly.
13

---

Under review as a conference paper at ICLR 2021
where h is the optimizer state, 9 is the gradient, x is the parameter being optimized, and k is the current iteration. Note that since this is a component-win wise optimizer, is applied to each parameter (I,)) of the target problem in parallel; therefore we drop the index (i) to reduce notation.
Near a fixed point of the dynamics, we approximate the recurrent dynamics with a linear approxi- mation. The linearized state update can be expressed as:
0F F(h 9 2 h" + 0h (h* -h"= + 0g 9
OF
(3)
where ht is a fixed point of the dynamics, oh a vector that controls how the scalar gradient enters the system. Both of these latter two quantities evaluated at the fixed point, h- and g" O.
OF is a square matrix known as the Jacobian, and AF is
Bg
For a linear dynamical system, as we have now, the dynamics decouple along eigenmodes of the system. We can this by rewriting the state in terms of the left eigenvectors of the Jacobian matrix. Let v = U denote the transformed coordinates, in the left eigenvector basis U (the columns of U are left eigenvectors of the matrix df). In terms of these coordinates, have:
x+ 0² + B (o+ + +0") + ag",
(4)
where B is diagonal matrix containing the eigenvalues of the Jacobian, and is a vector obtained by projecting the vector that multiplies the input
8F 0g
from eqn. (3) onto the left eigenvector basis.
If we have an -idimensional state vector h, then eqn. defines N independent (decoupled) scalar equations that govern the evolution of the dynamics along each eigenvector: u
07 + B3 (o) + v²) + +ayg", where we use 3, to denote the jth eigenvalue and aj is the jth com- ponent of eqn. 44. Collecting constants yields the following simplified update:
of+1 Byuţ + 0gg + const.,
(5)
which is exactly equal to the momentum update (u = 3 + agg), up to a (fixed) additive con- stant. The main difference between momentum and the linearized momentum in eqn. (5) is that we now have N different momentum timescales. Again these timescales are exactly the eigenvalues of the Jacobian matrix from above. Moreover, learning rate associated with eigenmode j. as a... This particular optimizer (momentum with multi- ple timescales) has been proposed under the name aggregated momentum by Lucas al. (2018).
also have way of extracting the corresponding
Taking a step back, we have drawn connections between linearized approximation of a nonlinear optimizer, and form of momentum with multiple timescales. What this now allows to do is inter- pret the behavior of learned optimizers near fixed points through this new lens. In particular, we have a way of translating the parameters of a dynamical system (Jacobians, eigenvalues and eigenvectors) into more intuitive optimization parameters (learning rates and momentum timescales).
C SUPPLEMENTAL METHODS
C.I TASKS FOR TRAINING LEARNED OPTIMIZERS
An optimization problem is specified by both the loss function to minimize and the initial parame- ters. When training a learned optimizer (or tuning baseline optimizers), we sample this loss function and initial condition from a distribution that defines a task. Then, when evaluating an optimizer, we sample new optimization problems from this distribution to form test set.
The idea is that the learned optimizer will discover useful strategies for optimizing the particular task it was trained on. By studying the properties of optimizers trained across different tasks, we gain insight into how different types of tasks influence the learned algorithms that underlie the operation of the optimizer. This sheds insight the inductive bias of learned optimizers; i.e. want to know what properties of tasks affect the resulting learned optimizer and whether those strategies are useful across problem domains.
We train and analyzed learned optimizers on three distinct tasks. In order to train a learned opti- mizer, for each task, must repeatedly initialize and the corresponding optimization problem
14

---

Under review as a conference paper at ICLR 2021
Optimizer
Parameters at step k
v Vx(x)
Fø
Parameters at step k+1
Function to optimize
Learned optimizer parameterized by
Figure 8: Schematic of learned optimizer.
(resulting in thousands of optimization runs). Therefore we focused on simple tasks that could be optimized within a couple hundred iterations, but still covered different types of loss surfaces: con- vex and non-convex functions, over low- and high-dimensional parameter spaces. We also focused on deterministic functions (whose gradients are not stochastic), to reduce variability when training and analyzing optimizers.
C.2 TRAINING A LEARNED OPTIMIZER
We train learned optimizers that are parameterized by recurrent neural networks (RNNs). In all of the learned optimizers presented here, we use gated recurrent unit (GRU) (Cho al. 2014) to parameterize the optimizer. This means that the function F' in eqn. ( is the state update function of GRU, and the optimizer state the GRU state. In addition, for all of our experiments, we set the readout function r in eqn. (2) to be linear. The parameters of the learned optimizer are now the GRU parameters, and the weights of the linear readout. We meta-learn these parameters through neta-oppimization procedure, described below.
In order to apply learned optimizer, we sample an optimization problem from our task distribu- tion, and iteratively feed in the current gradient and update the problem parameters, schematized in Figure | This iterative application of an optimizer builds an unrolled computational graph, where the number of nodes in the graph is proportional the number of iterations of optimization (known as the length of the unroll). This is sometimes called the inner optimization loop, to contrast it with the outer loop that is used to update the optimizer parameters.
In order to train a learned optimizer, we first need to specify a target objective to minimize. In the average loss over the unrolled (inner) loop this meta-objective. In order
this work, meta-objective, we compute the gradient of the meta-objective with respect to the to minimize the optimizer weights. We do this by first running unrolled computational graph, and then using backpropagation through the unrolled graph in order to compute the meta-gradient.
This unrolled procedure is computationally expensive. In order get single meta-gradient, need to initialize, optimize, and then backpropagate back through an entire optimization problem. This is why we focus on small optimization problems, that are fast to train.
Another known difficulty with this kind of meta- meta- optimization arises from the unrolled inner loop. In order to train optimizers on longer unrolled problems, previous studies have truncated this inner computational graph, effectively only using pieces of it in order to compute meta-gradients. While this saves computation, is known that this induces bias in the resulting meta-gradient (Wu al,, 2018 Metz et al.) 2019).
To avoid this, we compute and backpropagate through fully unrolled inner computational graphs. This places a limit on the number of steps that we can then run the inner optimization for, in this work, we set this unroll length 200 for all three tasks. Backpropagation through single unrolled optimization
gives single (stochastic) meta-gradient, when meta-training, average these over a batch size of 32.
Now that we have procedure for computing meta-gradients, we can use these to iteratively update parameters of the learned optimizer (the outer loop, also known as metaaoptimizztiinn) We do this using Adam the meta-optimizer, with the default hyperparameters (except for the initial learning rate, which was tuned via random search). In addition, we use gradient clipping (with a clip value of five applied to each parameter independently and decay the learning rate exponentially (by
15

---

Under review as a conference paper at ICLR 2021
Linear Regression
Rosenbrock
Neural network training
1.5
Meta-objective
0.5
RMSProp
Momentum
Adam
Learned
RMSProp
Momentum
Adam
Learned
RMSProp
Momentum
Adam
Leamed
Figure 9: Performance summary. Each panel shows the meta-objective (average training loss) over 64 random test problems for baseline and learned optimizers. Error bars show standard error. The learned optimizer has the lowest (best) meta-objective each task.
factor of 0.8 every 500 steps) during meta-training. We added small & 6y-regularization penalty to parameters of the learned optimizer, with a penalty strength of 10-5, We trained each learned optimizer for total of 5000 steps.
For each task, we ended up with a single (best performing) learned optimizer architecture. These the optimizers that then analyzed, and form the basis of the results in the main text. The final meta-objective for each learned optimizer and best tuned baselines are compared below in Figure
C.3 HYPERPARAMETER SELECTION FOR BASELINE OPTIMIZERS
We tuned the hyperparameters of each baseline optimizer, separately for each task. For each com- bination of optimizer and task, we randomly sampled 2500 hyperparameter combinations from grid, and selected the best one using the same meta-objective that was used for training the learned optimizer. We ensured that the best parameters did not occur along the edge of any grid.
For momentum, we tuned the learning rate (a:) and momentum timescale (3). For RMSProp. we
tuned the learning rate (a) and learning rate adaptation parameter (r). For Adam, we tuned the learning rate (a), momentum (31), and learning rate adaptation (32) parameters. The result of these
hyperparameter runs are shown in Figures TO (linear regression), [1] (Rosenbrock), and 12 (two moons classification). In each of these figures, the color scale is the same lame--urrpe purple denotes the optimal hyperparameters.
16

---

Under review as a conference paper at ICLR 2021
Linear regression
Momentum
RMSProp
10"
10
Nomallzzatinn (1-v)
Momentum (1-B)
10-1
10"1
10-2
10-2
1.0
0.8
10
10:3
10
10
10
101
102
10
10
109
10
10a
0.6
Learning rate (a)
Learning rate (a)
0.4
Adam
Adam
100
100
0.2
Normalization (1-V)
Momentum (1-B)
0.0
10-1
10"1
10-2
10-2
10-3
10-3
10
10-1
10
101
10a
10-2
10
10a
Learning rate (a)
Learning rate (a)
Figure 10: Hyperparameter selection for linear regression.
Rosenbrock
Momentum
RMSProp
10
Normalization (1-v)
Momentum (1-B)
10-1_
10-1
10-2
10-2
1.0
0.8
10-5
10-8
10
10
10
10
10
10
10
10
0.6
Learning rate (a)
Learning rate (a)
0.4
Adam
Adam
10
10"
0.2
Normalization (1-B2)
Momentum (1-B,)
0.0
10-1
10-1
10
10
10-3
10-3
10-
10
2
10-1
10"
1
10-1
Learning rate (a)
Learning rate (a)
Figure 11: Hyperparameter selection for Rosenbrock.
17

---

Under review as a conference paper at ICLR 2021
Two Moons
Momentum
RMSProp
10a
100
Normalization (1-V)
Momontum (1-
10-1
10-1
10-2-
10-2
1.0
10-8
10-3
0.8
10
10-3
10
10
109
10-1
10-0
10
10
10"
Learning rate (a)
Learning rate (a)
0.6
Adam
Adam
0.4
109
109
Normalizatin (1-B2)
0.2
Momentum (1-B,)
10
10-1
0.0
10-2
10-2
10-6
10-8
10
10-a
10
10-
10"
10
10
10
10
10"
Learning rate (a)
Learning rate (a)
Figure 12: Hyperparameter selection for training neural network two moons data.
18


Reviews:
[{"review_id": "vLDZIxo1b3s", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe author proposed a set of tools to analyze the properties of the learned neural network-based optimizers.\n\nThis toolset consists of (i) update function visualizer (ii) linearization of the update equation around the fixed point. The author takes an RNN-based optimizer as an example and analyzes it in the empirical section. \n\nThe author investigates several properties including (1) momentum using linearization (2) gradient clipping using the function visualizer (3) learning rate schedule and (4) learning rate adaptation. It shows that the learned optimizer indeed possesses some useful properties. \n\n--------\nReview:\nClarity: In general, some parts of the paper is clearly written and motivated such as the problems of the current learned optimizer.  However, the paper quality can be improved if some sections can be more clearer. For example, in section 3.2, the author only mentioned the high-level idea of the proposed tool (linearization) without a detailed mathematical introduction. It would be much more helpful the author can give a brief introduction to the linearization in the Appendix, rather than describe it in words and postpone it in the momentum analysis section.\n\nNovelty: The tool seems to be novel but the idea of linearization is not new, which has been used in other analyses of the dynamical systems as mentioned by the author. As for the function visualizer, it seems to be a plot of the parameter update, which is standard.\n\nTechnical soundness:\nI have checked some of the details, it seems to be correct.\n\nSignificance:\nThe proposed tools may be useful for analyzing neural network-based optimizers, and help to diagnose their behaviors. The tools are easy to implement and can be applied to a broad class of optimizers.\n\nWeakness:\nAlthough the paper proposed an interesting way to visualize some of the properties of learned optimizers, I think the paper in the current stage is not enough for a full conference paper. Here are some of my concerns and questions for the author:\n1. The proposed tools are mainly used to visualize some of the properties of the learned optimizer. This may be helpful for visualizing some properties but it provides no analysis on why it has those behaviors and what advantages these properties when optimizing a function $f$. Or what properties of the function $f$ can induce such behavior of the learned optimizer.\n2. The author only demonstrates the usage of the tools in analyzing a single/isolated property. How these properties are combined is not analyzed. \n3. The author only analyzes one NN-based optimizer. are there any other forms of the leaned optimizers? What advantages of those optimizers compared to the one used in the paper? Can your analysis confirm those advantages and provide possible reasons? This would be stronger evidence to back up the validity of the proposed method compared to the current analysis without comparisons.\n4. As mentioned in the clarity, better use mathematical equations to explain certain terms rather than words. For example, in section 3.1, the slope w.r.t what, the gradient $g$?\n5. In section 3.2, I am a bit confused about the difference between the fixed point and convergence point? Any examples of scenarios that it is a fixed point but not a convergence point?\n6. In section 4.1 (momentum), for the first figure in the bottom row (Figure 3), is it the eigenvalue point in the regression task? What about the first one used in Figure 7 (Appendix B). Are they the same? \n7. In section 4.3, your analysis shows the optimizer has this autonomous behavior. What advantages does this behavior provide? The author also mentions that the parameters should not be updated during the autonomous behavior. How can we confirm this through the plot (Figure 5)?\n8. I am not fully sure what you mean in the second paragraph (section 4.4). Do you mean that at the convergence points, you manually give some gradient perturbations, and the fixed points are moved away from the convergence point to get the S-curved shape?", "extra_scores": {}, "extra_text": {}}, {"review_id": "uaIHU6hnu-L", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to shed the light on the work (mechanisms) of the learned optimizers. The main contributions of this paper are the following ones:\n 1. The authors found the connection of learned optimization patterns with classical optimization techniques: such as momentum/gradient clipping, etc.\n 2. The authors applied the methods from dynamical systems to the analysis of learned optimizers. In particular, the authors used linearization of dynamical systems in vicinity of fixed points to analyse the properties mentioned above, as well as the trajectory of hidden states of learned optimizers during optimization of the inner problem.\n\nOverall, the findings of this paper sound interesting but the paper requires further development. For example, the following directions can be considered:  theoretical analysis of the obtained results, practical applications of the results or investigation of resulting properties on a larger class of optimizers training methods.\n\nTo analyse different properties of the learned optimizers the authors considered a set of three simple tasks using an optimizer training method similar to [1] and tried to find the connection between the learned optimization pattern and classical optimization techniques or identify some new behaviour patterns. While the choice of such simple tasks is well-motivated by the fact that optimizers training is very computationally intensive the choice of training method from [1] leads to significant limitations of obtained results from my point of view. In particular, the main difference between [1] and newer methods is its generalization property. As authors from [1] noted the considered method has problems with generalization (for example, if the activation function of the inner task is changed, the learned optimizer is no longer able to generalize and train NN with the different activation function). Due to this it is difficult to say whether findings from this paper can be useful in the more interesting settings or with more advanced optimizer training methods.\n\nOverall, this paper is well-written, but there are some sections of the paper that are difficult to follow. For example, section 3.2. One of the main techniques used in the paper is linearization of dynamical systems around fixed points. I would recommend adding into the paper a brief description of the method with its limitations and all necessary definitions. In particular, I would provide the rigorous definitions of fixed point/stable fixed point/global fixed point of dynamical systems.\n\nSome other questions and issues:\n1. Appendix C.2 has incorrect reference to Figure 5.\n2. Figures 10/11/12. Adam optimizer has 3 parameters alpha, beta, gamma. The plots corresponding to Adam have 2 parameters each, what is the value of the third parameter?\n3. The authors of [2] noticed that it is beneficial to provide additional input features that are borrowed from classical optimization methods (e.g. momentums) besides gradients.  Could you please clarify how these observations align with the findings that learned optimizers are able to learn classical techniques?\n4. How will the method properties change if the hidden size of the learned optimizer will be decreased (e.g. from 256 to 40/20)?\n\n[1] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in neural information processing systems, pp. 3981–3989, 2016.\n\n[2] Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize. arXiv preprint arXiv:1703.04813, 2017.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "m1fRUrbNYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThis paper studies learned optimizers and tries to discover what they have learned. The authors argue that the dynamics of the learned optimizers are responsible for those behaviors. Results are mainly presented as visualization and with tools from reverse-engineering dynamical systems. \n\n# Strength\n\n- This seems to be the first paper on re-discovering what the neural optimizers have learned from dynamical system angle.\n- Overall, the paper is easy to follow.\n\n# Weakness\n\n- All three experiments are done on toy datasets. Although the authors argue in Section 4 that \"These tasks were selected because they are fast to train (particularly important for meta-optimization) and covered a range of loss surfaces (convex and non-convex, low- and high-dimensional)\", I completely do not agree. Even in the original neural optimizer paper by Andrychowicz et al. they considered CIFAR (subset). If we only run the optimizers for several hundreds of steps, I think it's totally affordable. It is not convincing that the observed behaviors will generalize to real-world problems.\n- In the title and abstract, the authors mention \"novel mechanisms\". But it is not clear to me which behaviors shown in Section 4 are considered novel? I guess the authors might refer to learning rate adaption, \"the effect is to decrease the learning rate of the optimizer when large gradients are encountered\". But in Section 4.4, paragraph 4, the authors say \".... similar to the changes observed as the RMSProp state varies,\" which might suggest this is not novel to human-designed optimizers with a memory mechanism. Even this is novel, this is only observed on three toy datasets.", "extra_scores": {}, "extra_text": {}}, {"review_id": "55KSiUaxBaM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Hello authors,\n\nThank you for your submission.  I very much enjoyed reading it.  I found the writing to be clear and only found one grammatical error (detailed below).  As with any black-box system like a learned optimizer, there is naturally a lot of interest in what, actually, the optimizer itself is learning, and why it has learned in the way it has.  In this effort, the authors perform interesting experiments with intriguing results for the first of those two questions.\n\nI found the experiments to be comprehensive and the figures to be adequately described.  It would be great if the authors could provide more details in the main paper on the precise process used to approximate the nonlinear dynamical system.  It could be easy for a reader to assume, without details, that a learned optimizer is \"definitely\" learning momentum, while missing the nuance of how those results were obtained.  (For the record I am not saying I disagree with the results, only that if more detail can be provided, it would be helpful.)\n\nThe experiments are only performed with one form of learned optimizer, using an RNN with 256 GRUs.  What happens if different learned optimizer architectures are used?  Is there reason to believe that they would exhibit similar behavior?  Have the authors perhaps done any experiments with other architectures to see if the results are replicated?  Adding some extra information about this question to Section 4 could improve the paper.\n\nOverall, I think this is a good paper and should definitely be accepted.\n\nSmall bits:\n\n - Can we really drop the subscript notation in (1)?  Couldn't F use other elements of g?  Or are we restricting the situation for this exploration?\n - You might consider switching Figure 2 and Figure 1; the background on learned optimizers meshes well with Figure 2 and so it may be more streamlined to introduce it there.  However, your call.\n - Page 4: \"The RNN is is trained\" -> \"The RNN is trained\"\n - It would be really helpful if Figure 3 was on the same page as Section 4.1.  This also applies to Figures 4 and 5.\n\n## Post-rebuttal comments\n\nThanks again to the authors for the submission.  My concerns are clarified and so I will leave my rating intact.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: A line of work since 2016 has investigated learning NN-based optimisers, which produce optimisation updates by processing loss/gradient info with neural networks. This paper tries to understand the learned dynamics of these NN-based optimisers by linear approximation to the learned non-linear dynamics. Visualisation of these approximations are shown on 3 optimisation problems: linear regression, Rosenbrock function, and a toy neural network classification problem, with the hope of covering different types of objective landscapes. 

Reviewers agreed that the paper studies an important research question, which would interest researchers working on meta-learning learning algorithms. However there are several major concerns raised by the reviewers: (1) the example optimisation problems are toyish, and (2) the paper does not explain very well the link between the visualised behaviour and the better optimisation results, i.e. it is unclear to the reviewers why the learned dynamics lead to better optimisation results. 

While I am not too concerned about issue (1), I think issue (2) is a significant one, flagging that the clarity of the paper needs to be improved. Ultimately, the paper is motivated by the question "How is a learned optimizer able to outperform a well tuned baseline?", so a reader would expect some clear explanation towards answering this question. Also some reviewers are concerned about the fact that only the RNN-based optimiser in Andrychowicz et al. (2016) is analysed; since there exists other forms of learned optimisers, focusing on studying only one type of them might lead to early conclusions that are not so accurate.
Decision: Reject