Reverse engineering learned optimizers reveals known and novel mechanisms
Authors: Niru Maheswaranathan
David Sussillo
Luke Metz
Ruoxi Sun
Jascha Sohl-Dickstein
Venue: NeurIPS 2021 Poster (2021)
Keywords: optimization
learned optimizers
reverse engineering
RNNs

Abstract: Learned optimizers are parametric algorithms that can themselves be trained to solve optimization problems. In contrast to baseline optimizers (such as momentum or Adam) that use simple update rules derived from theoretical principles, learned optimizers use flexible, high-dimensional, nonlinear parameterizations. Although this can lead to better performance, their inner workings remain a mystery. How is a given learned optimizer able to outperform a well tuned baseline? Has it learned a sophisticated combination of existing optimization techniques, or is it implementing completely new behavior? In this work, we address these questions by careful analysis and visualization of learned optimizers. We study learned optimizers trained from scratch on four disparate tasks, and discover that they have learned interpretable behavior, including: momentum, gradient clipping, learning rate schedules, and new forms of learning rate adaptation. Moreover, we show how dynamics and mechanisms inside of learned optimizers orchestrate these computations. Our results help elucidate the previously murky understanding of how learned optimizers work, and establish tools for interpreting future learned optimizers.

Reverse engineering learned optimizers reveals known and novel mechanisms
Niru Maheswaranathan' Google Research, Brain Team niruChey.o com
David Sussillo"
Google Research, Brain Team
Luke Metz
Ruoxi Sun
Jascha Sohl-Dickstein Google Research, Brain Team jaschasd@google. com
Google Research, Brain Team lmetz@google.com com
Google Research, Brain Team ruoxio@google.coo com
Abstract
Learned optimizers are parametric algorithms that can themselves be trained to solve optimization problems. contrast baseline optimizers (such as momentum or Adam) that use simple update rules derived from theoretical principles, learned optimizers use flexible, high- sigh-dimensional, nonlinear parameterizations. Although this can lead to better performance, their inner workings remain a mystery. How is a given learned optimizer able outperform well tuned baseline? Has it learned sophisticated combination of existing optimization techniques, or is it implementing completely new behavior? In this work, we address these questions by careful analysis and visualization of learned optimizers. We study learned optimizers trained from scratch on four disparate tasks, and discover that they have learned interpretable behavior, including: momentum, gradient clipping, learning rate schedules, and learning rate adaptation. Moreover, we show how dynamics and mechanisms inside of learned optimizers orchestrate these computations. Our sults help elucidate the previously murky understanding of how learned optimizers work, and establish tools for interpreting future learned optimizers.
Introduction
Optimization algorithms underlie nearly all of modern machine learning; thus advances in optimiza- tion have broad impact. Recent research uses meta-learning to leam new optimization algorithms, by directly parameterizing and training an optimizer a distribution of tasks. These so-called learned optimizers have been shown to outperform baseline optimizers in restricted settings 777.
Despite improvements in the design, training, and performance of learned optimizers, fundamental questions remain about their behavior. We understand remarkably little about how these optimizers work. Are learned optimizers simply learning clever combination of known techniques? Or do they learn fundamentally new behaviors that have not yet been proposed in the optimization literature? they did learn new optimization technique, how would know?
Contrast this with existing "hand-designed" optimizers such as momentum [ AdaGrad 9 RM- SProp [10) Adam [II. These algorithms are motivated and analyzed using intuitive mechanisms and theoretical principles (such as accumulating update velocity momentum, or rescaling updates based on gradient magnitudes in RMSProp Adam). This understanding of underlying mechanisms allows future studies to build on these techniques by highlighting flaws in their operation [12),
Work conducted while Google Research. Currently at Meta Reality Labs.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

studying convergence [3), and developing deeper knowledge about why key mechanisms work [4]. Without analogous understanding of the inner workings of learned optimizers, it is incredibly difficult to analyze or synthesize their behavior.
In this work, we develop tools for isolating and elucidating mechanisms in nonlinear, high- dimensional learned optimization algorithms ($4). Using these methods we show how learned optimizers utilize both known and novel techniques, across four disparate tasks. In particular, we demonstrate that learned optimizers learn momentum ($..1) gradient clipping ($5.2). learning rate schedules ($53)), and methods for learning rate adaptation ($5.4) $5.5). Taken together, our work can be seen as part of new approach to scientifically interpret and understand learned algorithms.
We provide code for training and analyzing learned optimizers, as well as the trained weights for the learned optimizers studied here, at at/https:/// pttttp::/bit..1//epppp. 1y/3eggNrI
2 Related Work
Our work is heavily inspired by recent work using neural networks to parameterize optimizers. Andrychowicz et al. [ originally showed promising results on this front, with additional studies
improving robustness
2 3
meta-training (6) and generalization 7 of learned optimizers.
Here, we study the behavior of optimizers by treating them as dynamical systems. This perspective has yielded number of intuitive and theoretical insights [15H88). We also build on recent work on reverse engineering recurrent neural networks (RNNs). Sussillo and Barak [9] showed how linear approximations of nonlinear RNNs can reveal the algorithms used by trained networks to solve simple tasks. These techniques have been applied to understand trained RNNs a variety of domains, from natural langauge processing (20 [1] to neuroscience [22]. Additional work on treating RNNs as
dynamical systems has led to insights into their computational capabilities (53
Methods
3.1 Preliminaries
We are interested in optimization problems that minimize a loss function (f) over parameters (æ). We focus first-order optimizers, which at iteration & have access to the gradient of = ()({)) and produce an update Ant. These are component-wise optimizers that are applied to each parameter (x;) of the problem in parallel. Standard optimizers used in machine learning (e.g. momentum, Adam) are in this category Going forward, we use for the parameter to optimize, 9 for its gradient, k for the current iteration, and drop the parameter index (i) to reduce excess notation.
One can think of an optimizer as being comprised of two parts: the optimizer state (h) that stores information about the current problem, and readout weights (w) that are used to update parameters given the current state. An optimization algorithm is specified by an initial state, state transition dynamics, and readout, defined as follows:
h
(((h,.9* xt
x k-1
w"' wThh
h R+1
(2)
where h is the optimizer state, F' governs the optimizer dynamics, and w are the readout weights.
Learned optimizers are constructed by parameterizing the function F, and then learning those parameters along with the readout weights through cettmoppimizttionn (detailed in App. D.2). Hand- designed optimization algorithms, by distinction, specify these functions at the outset.
For example, in momentum, the state variable is single number (known as the velocity). and is updated as linear combination of the previous state and the gradient (e.g. A 3hh* + + where ( is the momentum hyperparameter). For momentum and other hand-designed optimizers, the state variables are now-dimensional, and their dynamics are (largely) straightforward. In contrast, learned optimizers have high-dimensional state variables, and the potential for rich, nonlinear dynamics. As these systems learn complex behaviors, it has historically been difficult to extract simple, intuitive descriptions of the behavior of learned optimizer.
Notable exceptions include quasi-Newtor methods such L-BFGS 26 or K-FAC 27

---

RMSProp
Momenium
Adam
Learned Optimizer
(8)
Linear Regression
(b)
Rosenbrock
(c)
Two Moons
(d)
MNIST
10
109
Loss
10
Loss
Loss
Loss
109
10-9-
10'1 1-
10-4
10
10-1
0
100 Iteration
200
100 Iteration
200
100
200
200
Iteration
Iteration
3
1.5
Frequency
y-coordinate
CNN Architecture
Loss
Feature
Layer
Layer
Layer
10
102 10
-1.0
Condition number (k)
x-coordinate
-1.5
Feature
2.5
Layer
Figure I: Learned optimizers outperform well tuned baselines on four tasks: (a) linear regression, (b) the Rosenbrock function, (c) training a fully connected neural network on the two moons dataset, and (d) training convolutional neural network on the MNIST dataset. Top row: Optimizer performance. shown as loss curves (mean + std. error over 64 random seeds) for momentum (orange), RMSProp (yellow), Adam (red) and learned optimizer (bluc). Bottom row: Additional information pertaining to each task (described in 3.2)
3.2 Training learned optimizers
We parametrize the learned optimizer with recurrent neural network (RNN), similar Andrychowicz et al. []. The only input to the optimizer the gradient. The RNN is trained by minimizing meta-objective, which we define as the average training loss when optimizing target problem. See App. D.2 for details about the optimizer architecture and meta-training procedures. Below, we only analyze the final (best) trained optimizer, however we do analyze aspects of the meta-training dynamics in App B.2
We trained learned optimizers each of four tasks. These tasks were selected because they converge in relatively small number of iterations (particularly important for meta-optimization) and cover range of loss surfaces (convex and non-convex, low- and high-dimensional, deterministic and stochastic):
Linear regression: The first task consists of random linear regression problems f(x) - where A and are randomly sampled. Much of our theoretical understanding of the behavior of optimization algorithms is derived using quadratic loss surfaces such as these, in part because they have a constant Hessian (A" A) over the entire parameter space. The choice of how to sample the problem data A and b will generate particular distribution of Hessians and condition numbers. A histogram of condition numbers for our task distribution is shown in Figure l..
The second task is minimizing the Rosenbrock function (28.. a commonly used test function for optimization. It is non-convex function with eurved valley and single global minimum. The function is defined over two parameters as [xx y) = (1 x²2² + 100(y - x2)² (Figure The distribution of problems for this task consists of different initializations sampled uniformly over a grid. The grid used to sample initializations is the same as the grid shown in the figure; the and y- coordinates are sampled from the ranges (-2, 2) and (-1, 3), respectively.
Rosenbrock:
Two moons:
The third task involves training a fully connected neural network to classify a toy dataset, the two moons dataset (Figure T.). As the data are not linearly separable, nonlinear classifier is required. The optimization problem to train the weights of a three hidden layer fully connected neural network, with 64 units per layer and tanh nonlinearities (for a total of 8,577 parameters). The distribution of problems involves sampling the initial weights of the fully connected network.
MNIST: The fourth task is to train four layer convolutional network to classify digits from the MNIST dataset. We use minibatch size of 100 examples; thus the gradients fed to the optimizer stochastic, unlike the previous three problems. The network consists of three convolutional

---

layers each with 16 channels with a 3x3 kernel size and ReLU activations, followed by final (fully connected) dense layer, for total of 82,250 parameters (Figure
We additionally tuned three haseline optimizers (momentum, RMSProp. and Adam) individually for each task. We selected the hyperparameters for each problem of 2500 samples randomly drawn from a grid. Details about the exact grid ranges used for each task are in App. D.3
Figure [ row) compares the performance of the learned optimizer (blue) to baseline optimizers (red, yellow, and orange), on each the four tasks described above. Across all tasks, the learned optimizer outperforms the baseline optimizers the meta-objective (App. Fig. 17).
4 Tools for understanding optimizers
In order to analyze learned optimizers, we make extensive use of two methods. The first is a way to visualize what optimizer is doing a particular optimizer state. The second is way of making sense of how the optimizer state changes, that is, understanding the optimizer state dynamics. We describe these below, and then use them to analyze optimizer behavior and mechanisms in $
4.1 Update functions
(a) Gradient Descent
(b) Gradient Clipping
(c) Momentum
(d) RMSProp
0.02
Parameter update (Ax)
0.00
-0.02
-1
-1
-1
-1
Gradient (g)
Figure 2: Visualizing optimizer behavior with update functions (see $4.1 for details) for different commonly used optimization techniques. (a) Gradient descent is a (stateless) linear function, whose slope is the learning rate. (b) Gradient clipping saturates the update. beyond threshold. (c) Momentum introduces a vertical offset depending on the accumulated velocity (colors indicate different values of the accumulated momentum). (d) RMSProp changes the slope (effective learning rate) of the update (colors denote changes in the state variable, the accumulated squared gradient).
First, introduce a visualization tool get handle what optimizer is doing. Any optimizer, at a particular state, can be viewed as a scalar function that takes in gradient (g) and returns a change in the parameter (Az). We refer to this the optimizer update function.
Mathematically, the update function is computed as the state update projected onto the readout, Ar = w" F F(h.g) g). following equations (1) and ( In addition, the slope of this function with
respect to the input gradient eAr can be thought of as the effective learning rate at particular
optimizer state A steeper slope means that the parameter update is more sensitive to the gradient (as expected for a higher learning rate), and shallower slope means that parameter updates are smaller for the same gradient magnitude (i.e. lower learning rate).
As the optimizer state varies, the update function and effective learning rate can change. This provides a mechanism for learned optimizers to implement different types of behavior: through optimizer state dynamics that induce particular types of changes in the update function.
AAs the meta-ohjective the average training loss during an optimization run, it naturally penalizes the training curve earlier in training (when loss values are large). This explains the discrepancy in the curves for linear regression (Fig. TT. top) where momentum continues to decrease the loss late in training. Despite this, the learned optimizer has overall smaller meta-objective due having lower loss earlier iterations.
'e compute this slope at O, in the middle of the update function. We find that the update function is always affine in the middle with saturation at the extremes, thus the slope at 0 is a natural way to summarize the effective learning rate.

---

It is instructive to first visualize update functions for commonly used optimizers (Figure 2 For gradient descent, the update (Ar -ag) stateless and is always a fixed linear function whose slope is the learning rate, (Fig. 24). Gradient clipping is also stateless, but is saturating function of the gradient (Fig.2) P)). For momentum, the update is Az 3e - og. where denotes the momentum state (velocity) and / is the momentum timescale. The velocity adds an offset to the update function (Fig. 2)) ). As the optimizer picks up positive (or negative) momentum, the curve shifts downward (or upward), thus incorporating a bias to reduce (or increase) the parameter. For adaptive optimizers such as RMSProp, the state variable changes the slope, or effective learning rate, within the linear region of the update function (Fig. D
Now, what about learned optimizers, or optimizers with much more complicated or high-dimensional state variables? One advantage of update functions is that, as scalar functions, they can be easily visualized and compared to the known methods Figure Whether or not the underlying hidden states are interpretable, for given learned optimizer, remains to be seen.
4.2 A dynamical systems perspective
In order understand the state dynamies of learned optimizer, we approximate the nonlinear dynamical system (eq. (I) via linearized approximations [29]. These linear approximations hold near fixed points the dynamics. Fixed points are points in the state space of the optimizer, where - as long as input gradients do not perturb it it- it-thh the system does not move. That is, an approximate fixed point ht satisfies the following: h" = F(h".g' g6"), for a particular input g"
We numerically find approximate fixed points 119 30 by solving an optimization problem where we find points (h) that minimize the following loss: Fl gr) 1l The solutions to this problem (there may be many) are approximate fixed points of the system F for given input, g" In general, there may be different fixed points for different values of the input (g). First we will analyze fixed points when g = 0 ($5.1). and then later discuss additional behavior that occurs as g" varies ($5.4).
One can think of the structure of fixed points as shaping dynamical skeleton that governs the optimizer behavior. As we will see, for a well trained optimizer, the dynamics around fixed points enable interesting and useful computations.
5 Mechanisms of learned optimizers
We selected and analyzed the best learned optimizer on each of the four tasks in $3.2 Across these, we discovered a number of mechanisms responsible for their superior performance: momentum, gradient clipping, learning rate schedules, and new types of learning rate adaptation In the following sections, we go through each mechanism in detail, showing how it is implemented.
In general, we found similar mechanisms across learned optimizers trained all four tasks. Thus, for brevity, we only show the results in the main text from one optimizer for each mechanism. For any mechanisms that were found be task dependent, point this out in the relevant section. Results for all tasks are presented in App. A
5.1 Momentum
We discovered that learned optimizers implement classical momentum, and do using dynamics that are well described by a linear approximation.
To see how, first consider the dynamics of learned optimizer near fixed point. Here, we can linearly approximate the state dynamics (cq. [ using the Jacobian of the optimizer update 29 [9
The linearized state update is given by
OF (h* h²) + OF 0h
F(h g h
0g g²
where h* is a fixed point of the dynamics, ah the Jacobian matrix, and OF is vector that controls 8g how the scalar gradient enters the system. Both of these latter two quantities are evaluated the fixed point, h" and g" O.
5

---

(a)
Dynamics converge fixed point (FP)
(b)
Vertical shifts in update function
(c)
Eigenvalues Jacobian computed at convergence point
0.02
0.5
max
Update (Ax)
Learning Rate
PC #2
Total variance
0
33
max
0.00
0.0
200 teration
-0.02
-0.5
PC #1
Gradient (g)
R(A)
Figure 3: Momentum in learned optimizers. Plots are for the optimizer trained the Rosenbrock task (we see similar behavior for optimizers trained on the other tasks, see App. A.1). (a): Projection of the optimizer state near convergence point (black square). Inset: the total variance of the optimizer states over test problems goes to zero as the trajectories converge. (b): visualization of the update functions ($4.1) along the slow mode of the dynamics (colored lines correspond arrows in (a)). Along this dimension, the effect on the system to induce an offset in the update, just as in classical momentum (cf. Fig. 2c). (c): Eigenvalues of the linearized optimizer dynamics the fixed point (black square in (a)) plotted in the complex plane. The eigenvalue magnitudes are momentum timescales, and the color indicates the corresponding learning rate. See $5.1 for details.
This Jacobian is a matrix with N eigenvalues and eigenvectors, where N is the dimensionality of the optimizer state (for the RNN architectures that we use, N is the number of RNN units). For linear dynamical system, as we have now, the dynamics decouple along the N eigenmodes of the system.
Writing the update along these coordinates (let's denote them 24, for j .... .N) allows to rewrite the learned optimizer as a momentum algorithm (see App. C for details) with N timescales:
uf+1 = Byu + agg + const.,
where the magnitude of the eigenvalues of the Jacobian exactly momentum timescales (Bz), each with a corresponding learning rate (a,). Incidentally, momentum with multiple timescales has been previously studied and called aggregated momentum by Lucas et al. (31.
To summarize, near a fixed point, a (nonlinear) learned optimizer is approximately linear, and the eigenvalues of the its Jacobian can be interpreted momentum timescales.
We then looked to see if (and when) learned optimizers operated near fixed points. We found that across all tasks, optimizers converged to fixed points; often to a single fixed point. F shows this for learned optimizer trained on the Rosenbrock task. Fig. 3 a 2D projection of the hidden state (using principal components analysis"). and shows the single fixed point for this optimizer (black square). All optimizer state trajectories converge to this fixed point, this can be seen as the total variance of optimizer states across test examples goes zero (Fig. 3a. inset).
Around this fixed point, the dynamics are organized along line (gray circles). Shifting the hidden state along this line (indicated by colored arrows) induces a corresponding shift in the update function (Fig.) BD), similar to what observed in classical momentum (cf. Fig. 2
This learned optimizer uses a single eigenmode to implement momentum. Fig. 3
shows the
eigenvalues of the Jacobian (computed at the convergence fixed point) in the complex plane, colored by that mode's 's learning rate (see App. C for how these quantities are computed). This reveals single dominant eigenmode (colored in purple). whose eigenvector corresponds to the momentum direction (gray points in Fig. 3l) and whose eigenvalue is the corresponding momentum timescale.
For some learned optimizers, this momentum eigenvalue exactly matched the best tuned momentum hyperparameter; these cases the optimizer's performance matched that of momentum as well. We analyze one such optimizer in App. B as it is instructive for understanding the momentum mechanism.

---

(a)
Update function saturates
(b)
Gradient histogram
for large gradients
0.1
Update (Ax)
Density (llg scale)
0.0
-0.1
-1000
1000
-1000
1000
Gradient (g)
Gradient (g)
Figure 4: Gradient clipping in learned optimizer trained on the Rosenbrock task (results for additional tasks are in App. A.2). (a): The update function computed at the initial state saturates for large gradient magnitudes. The effect of this is similar to that of gradient clipping (cf. Fig. pp). (b): The empirical density of encountered gradients for this task. This shows that while most of the gradients occur in the linear regime, small but non-negligible fraction quite large and will saturate the update function.
5.2 Gradient clipping
In standard gradient descent, the parameter update is linear function of the gradient. Gradient clipping [32] instead modifies the update to be a saturating function (Fig. [p)). This prevents large gradients from inducing large parameter changes, which is useful for optimization problems with non-smooth gradients [14].
We find that learned optimizers also use saturating update functions as the gradient magnitude increases, thus learning a soft form gradient clipping. We show this for the learned optimizer trained on the Rosenbrock problem in Figurelpp. Although Fig. 4 shows the saturation for a particular optimizer state (the initial state in this case), we find that these saturating thresholds are consistent throughout the optimizer state space.
The strength of the clipping effect depends on the training task. We can see this by comparing the update function for a given optimizer to the distribution of gradients encountered for that task (Fig. P the higher the probability of encountering gradient that is in the saturating regime of the update function, the more clipping is used.
For some problems, such linear regression, the learned optimizer largely stays within the línear region of the update function (App. (..)). For others, such as the Rosenbrock problem presented in Fig. 4 the optimizer utilizes more of the saturating part of the update function.
5.3 Learning rate schedules
Practitioners often tune learning rate schedules along with other optimization hyperparameters. Originally motivated to guarantee convergence in stochastic optimization (33] schedules are now used more broadly (74.77] These schedules are typically a decaying function of the iteration meaning the learning rate goes down as optimization progresses although Goyal al. [3 use an additional increasing warm-up period, and even more exotic schedules have been proposed [39
We discovered that learned optimizers input driven- - dynamics. If the initial optimizer state is away from fixed points of the state dynamics, then even in the absence of input, autonomous dynamics will encode particular trajectory the system relaxes to a fixed point. This trajectory can then be exploited by the learned optimizer to induce changes in optimization parameters, such as the effective learning rate.
implement schedule using autonomous that is, not
We run PCA across a large set of optimizer states visited during test examples to visualize optimizer state trajectories.

---

(a)
(b)
Learning rate along autonomous trajectory
Autonomous dynamics
4
Effective learning rate
15
Component #2
10
0
5
-4
0
4
100
200
Component #1
Iteration
Figure 5: Learning rate schedules mediated by autonomous dynamics, shown for the linear regression task (additional tasks are in App. A.3). (a): cow-dimensional projection of the dynamics of the optimizer in response autonomous dynamics allow the system to learn a learning rate schedule (see $5.3). (b): Effective learning rate computed at the autonomous state trajectories in (a). Dashed line indicates the best. (tuned) learning rate for momentum on this task.
input (blue line) around approximate fixed points (gray circles). These
Indeed, find that for two of the tasks (linear regression and MNIST classification) learned optimizers learn an autonomous trajectory that modifies the learning rate, independent of being driven by any actual gradients.
This trajectory is shown for the linear regression task in Fig. 5p. starting from the initial state (black circle) and converging to global fixed point (black square). Along this trajectory, we compute update functions and find that their slope changes; this is summarized in Fig. [5p as the effective learning rate changing over time. Results from the other tasks are presented in App. A.3
5.4 Learning rate adaptation
The next mechanism we discovered is type of learning rate adaptation. The effect of this mechanism to decrease the learning rate of the optimizer when large gradients are encountered. The effect is qualitatively similar to adaptive learning rate methods such as AdaGrad or R.SPPrp, but it is implemented in a new way in learned optimizers.
To understand how momentum is implemented by learned optimizers, we studied the linear dynamics
of the optimizer near fixed point ($5.1) That fixed point found numerically (.4.2) by searching for points h" that satisfy h" F(h gr) where we hold the input (gradient) fixed at zero (g* = 0). To understand learning non-zero input. We find these fixed points by setting g to a fixed non-zero value.
adaptation, need to study the dynamics around fixed points with
We sweep the value of over the range of gradients encountered for particular task. For each value, we find single corresponding fixed point. These fixed points are arranged in an S-curve, shown in Figure Gh. The color of each point corresponds the value of g" used find that fixed point. One arm of this curve corresponds to negative gradients (red), while the other corresponds to positive gradients (green). The tails of the S-curve correspond the largest magnitude gradients encountered by the optimizer, and the central spine of the S-curve contains the final convergence poin
These fixed points are all attractors, meaning that if we held the gradient fixed particular value, the hidden state dynamics would converge to that corresponding fixed point. reality, the input (gradient) to the optimizer constantly changing, but if large (positive negative) gradient is seen for a number of timesteps, the state will be attracted to the tails the S-curve. As the gradient goes to zero, the system converges to the final convergence point in the central spine of Fig. 6a.
Note that this autonomous trajectory evolves in subspace orthogonal to the readout weights used to update the parameters. This ensures that autonomous dynamics themselves do not induce changes the parameters, only change effective learning rate. Fig. 6u uses the same projection in Fig. it is just zoomed out (note the different ranges).

---

(a)
Attractor fixed points vary function of input
(b)
Update function
(c) Effective learning rate decreases with increasing gradient magnitude
at different fixed points
10
0.02
Effective learning rate
2.0x10
Update (Ax)
1.5x10
PC #2
00
0.00
1.0x10
0.5x10
-10
-0.02
-15
15
-10
0
10
0.0
Gradient (g)
10
100
10a
PC #1
Gradient magnitude (Igl)
Figure 6: Learning rate adaptation in learned optimizers, shown for the Rosenbrock task (results similar for other tasks, see App. A.4). (a) Approximate fixed points (colored circles) the optimizer state dynamics computed for different gradients reveals an S-curve structure. Large positive (negative) gradients push the optimizer state to the dark green (red) tails of the S-curve. (b) Update functions ($4.1) computed different points along the S-curve, corresponding of moving towards the tail of the S-curve is to make the update function more shallow (and thus have smaller learning rate, ef. Fig. 2 The effect similar along both arms; only
the arrows in (a). The effect
is shown for clarity. (c) Summary plot showing the effective learning rate along each arm of the S-curve, for negative (red) and positive (green) gradients.
What the functional benefit of these additional dynamics? To understand this, we visualize the update function corresponding to different points along the S-curve (Fig.Gh) The curves are shown for just one arm of the S-curve (green, corresponding to positive gradients) for visibility, but the effect is the symmetric across the other arm as well. We see that we move along the tail of the S-curve (corresponding to large gradients) the slope of the update function becomes more shallow, thus the effect is to decrease the effective learning rate.
The learning rate along both arms of the S-curve are summarized in Fig. o for positive (green) and negative (red) gradients, plotted against the magnitude of the gradient on log scale. This mechanism allows the system to increase its learning rate for smaller gradient magnitudes. For context, the best tuned learning rate for classical momentum is shown dashed line.
5.5 Tuning per layer and parameter type
Average optimizer
Trajectories different (b)
(c) Different gradient statistics per parameter type
state trajectories
learning rate schedules
Effective learning rate
Weights
Weights Blases
10P
Layer
Layer
PC #3
Layer
Layer
Blases
Layer
10
2
PC 82
10*
10
10P
#1
100
200
Gradient magnitude (lgi)
Iteration
Figure 7: Parameter type-specific tuning found in learned optimizer trained MNIST classi- fication problem. (a) Trajectories of the optimizer, averaged across parameters within a particular layer and of specific type (either weight bias), exhibit different trajectories through optimizer state space. (b) Effective learning rate (computed as the slope of the update function) along each of the trajectories from panel (a). (c) Histograms of gradient magnitudes separated by layer and parameter type. The learned optimizer uses these differences in gradient statistics like these to induce the different trajectores from panels (a) and (b).

---

The final behavior we identified only exists in the learned optimizer trained on the MNIST CNN classification problem. a way for the learned optimizer to tune optimization properties (such as the effective learning rate) across different layers and across parameter types (either weight or bias) in the CNN being trained. We can see this most easily by taking all of the optimizer state trajectories for particular layer or parameter type, and averaging them. These average trajectories are shown in Figure a projected onto the top three PCA components. The initial state is in the bottom right, and trajectories up and over before converging over to the left side of the figure.
What is the functional benefit of separating out trajectories according to parameter type? To investigate this, we computed the update function at each point along the trajectories in Figure 7p. We found that the effective learning rate varied across them (Fig. (P)). In particular, the bias parameters all have a much lower learning rate than the weight parameters. Within a parameter type, later layers have larger learning rates than earlier layers.
clue for how this happens can be found by looking at the gradient magnitudes across layers and parameter types at initialization (Fig. (+). We see that the bias parameters (in orange) all have much larger gradient magnitudes on average compared to the weight parameters (in blue) in the later layers. This is plausible hypothesis for the signal that the network uses to separate trajectories in Fig. 7..
We want to emphasize that these are correlative, not causal, findings. That is, while the overall effect appears if the network is separating out state trajectories based on layer or parameter type, it is possible that the network is really attempting to separate trajectories based on some additional factor that happens to be correlated with depth and parameter type in this CNN.
6 Discussion
In this work, we trained learned optimizers on four different optimization tasks, and analyzed their behavior. We discovered that learned optimizers learn plethora of intuitive mechanisms: momentum, gradient clipping, schedules, forms of learning rate adaptation. While the coarse behaviors are qualitatively similar across different tasks, the mechanisms are tuned for particular tasks.
While we have isolated specific mechanisms, we still lack holistic picture of how these are stitched together. One may be able to extract distill compressed optimizer from these mechanisms, perhaps using data-driven techniques 42 or symbolic regression 44]
The methods developed in this paper also pave the way for studies of when and how learned optimizers generalize. By mapping different mechanisms to the underlying task used to train an optimizer, we can identify how quantitative properties of loss surfaces (c.g. curvature, convexity, etc.) give rise to particular mechanisms in learned optimizers. Understanding these relationships would allow us to take learned optimizers trained in one setting, and know when and how to apply them to new problems.
Previously, not much was known about how learned optimizers worked. The analysis presented here demonstrates that learned optimizers are capable of learning a number of interesting optimization phenomena. The methods we have developed (visualizing update functions and linearizing state dynamics) should be part of growing toolbox we can use to extract insight from the high-dimensional nonlinear dynamics of learned optimizers, and meta-learned algorithms more generally.
Acknowledgments
The authors would like to thank C. Daniel Freeman and Ben Poole for helpful discussions and for comments on the manuscript.
Funding transparency
The authors were employed by Google, Inc, while this research was being conducted.
10

---

References
[1] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, pages 3981-3989, 2016.
[2] Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize. arXiv preprint IrXivv1703.044:
2017.
[3] Kaifeng Lv, Shunhua Jiang, and Jian Li. Learning gradient descent: Better generalization and longer horizons. arXiv preprint arXiv:1703. 1ee.111033003333 2017.
[4] Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoe V Le. Neural optimizer search with reinforcement learning. arXiv preprint 1101:::170007044 2017.
[5] Ke Li and Jitendra Malik. Learning optimize. arXiv preprint
2016.
[6] Luke Metz, Niru Maheswaranathan, Jeremy Nixon, Daniel Freeman, and Jascha Sohl-Dickstein. Understanding and correcting pathologies in the training of learned optimizers. In International Conference on Machine Learning, pages 4556-4565, 2019.
[7] Luke Metz, Niru Maheswaranathan, C. Daniel Freeman, Ben Poole, and Jascha Sohl-Dickstein. Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves, 2020.
[8] Boris T Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 4(5):1-17, 1964.
[9] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
[10] Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-mmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2): 26-31, 2012.
[11] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.
[12] Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2018
[13] Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. arXiv preprint 1tti:::900.092377 2019.
[14] Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In International Conference on Learning Representations, 2020. / httpr//rww....ttoooooom/m//////a
[15] Weijie Su, Stephen Boyd, and Emmanuel Candes. A differential equation for modeling nes- terov's accelerated gradient method: Theory and insights. Advances in neural information processing systems, pages 2510-2518, 2014.
[16] Ashia C Wilson, Benjamin Recht, and Michael Jordan. A Iyapunov analysis of momentum methods in optimization. arXiv preprint arXiv: www.:111..0000 102635, 2016.
[17] Gabriel Goh. Why momentum really works. Distill, 2(4):e6, 2017.
[18) Bin Shi, Simon $ Du, Weijie Su, and Michael Jordan. Acceleration via symplectic discretiza- tion of high-resolution differential equations. Advances Neural Information Processing Systems, pages 5744-5752, 2019.
[19] David Sussillo and Omri Barak. Opening the black box: low-dimensional dynamics in high- dimensional recurrent neural networks. Neural computation, 25(3):626-66 649, 2013.
11

---

[20] Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Re- verse engineering recurrent networks for sentiment classification reveals line attractor dynamics. In Advances in Neural Information Processing Systems, pages 15696-15777 15705, 2019.
[21] Niru Maheswaranathan and David Sussillo. How recurrent networks implement contextual processing in sentiment analysis. arXiv preprint arXiv:2004.08013, 2020.
[22] Rylan Schaeffer, Mikail Khona, Leenoy Meshulam, Il Rani Ficte, et al. Reverse-engineering recurrent neural network solutions to hierarchical inference task for mice. bioRxiv, 2020.
[23] Ian D Jordan, Piotr Aleksander Sokol, and II Memming Park. Gated recurrent units viewed through the lens of continuous time dynamical systems. arXiv preprint 1xXiv:190.00105, 2019.
[24] Kamesh Krishnamurthy, Tankut Can, and David J Schwab. Theory of gating in recurrent neural networks. arXiv preprint arXiv:2007. 2et.:20077.44.,,., 14823, 2020.
[25] Tankut Can, Kamesh Krishnamurthy, and David J Schwab. Gating creates slow modes and controls phase-space complexity in grus and Istms. arXiv preprint arXiv:2002.00025, 2020.
[26] Jorge Nocedal and Stephen Wright. Numerical optimization. Springer Science & Business Media, 2006.
[27] James Martens and Roger Grosse. Optimizing neural networks with fronecker-factored approx- imate curvature. In International conference on machine learning, pages 2408-2417, 2015.
[28]
HoHo Rosenbrock. An automatic method for finding the greatest or least value of a function. The Computer Journal, 3(3):175 184, 1960.
[29] Steven H Strogatz. Nonlinear dynamics and chaos with student solutions manual: With applications to physics, biology, chemistry, and engineering. CRC press, 2018
[30] Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Universality and individuality in neural dynamics across large populations of recurrent networks. In Advances neural information processing systems, pages 15629-15641, 2019.
[31] James Lucas, Shengyang Sun, Richard Zemel, and Roger Grosse Aggregated momentum: Stability through passive damping. arXiv preprint arXiv: 1XXiv:1804.00355 2018.
[32] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310-1318, 2013.
[33] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400-407, 1951.
[34] Tom Schaul, Sixin Zhang, and Yann LeCun. No more pesky learning rates. In International Conference on Machine Learning, pages 343-351, 2013.
[35] Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V Le. Don't 't decay the learning rate, increase the batch size. arXiv preprint arXiv: 1 1.00889, 2017.
[36] Rong Ge, Sham M Kakade, Rahul Kidambi, and Praneeth Netrapalli. The step decay schedule: A near optimal, geometrically decaying learning rate procedure for least squares. In Advances in Neural Information Processing Systems. pages 14977-14988, 2019.
[37] Dami Choi, Christopher J Shallue, Zachary Nado, Jaehoon Lee, Chris J Maddison, and George E Dahl. On empirical comparisons of optimizers for deep learning. arXiv preprint arXiv:1910.05446, 2019.
[38] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangging Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv:1706.02677, 2017.
[39] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:160s 03983, 2016.
12

---

[40] Leslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 464 IEEE, 2017.
[41] Zhiyuan Li and Sanjeev Arora. An exponential learning rate schedule for deep learning, arXiv preprint 1r11v:1910.07454, 2019.
[42] Steven L Brunton, Joshua L Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, (13(15):3932-399) 2016.
[43] Kathleen Champion, Bethany Lusch, J Nathan Kutz, and Steven L Brunton. Data-driven discovery of coordinates and governing equations. Proceedings of the National Academy of Sciences, 116445):2444-22251 2019.
[44] Miles Cranmer, Alvaro Sanzhez-Gonvalez,, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho. Discovering symbolic models from deep learning with inductive biases. arXiv preprint arXiv:2006.11287, 2020.
[45] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using encoder- decoder for statistical machine translation. arXiv preprint 14X1v:1406.1074 2014.
[46] Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse. Understanding short-horizon bias in stochastic meta-optimization. arXiv preprint
2018.
[47] Charles Harris, K Jarrod Millman, Stéfan van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al. Array programming with numpy. Nature, 585(7825):357-333 2020.
[48] Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. Scipy 1.0: fundamental algorithms scientific computing in python. Nature methods, 17(3):261-772. 2020.
[49] John D Hunter. Matplotlib: A 2d graphies environment. Computing in science & engineering. 93):90-995 2007.
[50] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, and Skye Wanderman-Minn Jax: composable transformations of python+ numpy programs. URL http://github. com/google/jax, 2018.
13


Reviews:
[{"review_id": "pYsMWMRHz9z", "reviewer": "Reviewer_WPbX", "summary": "In this work, the author got trained learned optimizers on four different optimization tasks, and analyzed their behavior. They discovered that learned optimizers learn a plethora of intuitive mechanisms: momentum, gradient clipping, schedules, forms of learning rate adaptation. While the coarse behaviors are qualitatively similar across different tasks, the mechanisms are tuned for particular tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nTo the best of my knowledge, this kind of work which try to figure out the working mechanism of learned optimizer is rare. Very few people try to interpret it but they want to put forward a powerful optimizer. Thus, I think this work is innovative.\n\nQuality:\nI have seen that you do experiments on different suitations. This is very good and I believe that your analysis is convincing. \n\nClarity:\nThe description and visualizition part is good.\n\nSignificance:\nI admit that your work is innovative and give new idea to the following researchers. However, after visualizition of the learned optimizer. You give very limited suggestions about how to benefit from it. Such as how to design a better optimizer? How to select the design between different tasks and so on.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This work give visualizition of the learned optimizer. But they give very limited suggestions about how to benefit from it. Such as how to design a better optimizer? How to select the design between different tasks and so on.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "0.5 hours"}, "extra_text": {}}, {"review_id": "WUY8_MGVT3O", "reviewer": "Reviewer_QfSG", "summary": "This submission focuses on analyzing the behaviors of learned optimizers. The goal is to elucidate properties and patterns learned by such learned optimizers, in order to better analyze, understand, and lay groundwork to potentially improve the behavior of learned and/or manually-designed optimizers. A novel reverse engineering approach is taken and identifies learned interpretable and fundamental behavior such as momentum, gradient clipping, learning rate schedules, and learning rate adaptations.\n\nThe submission trained learned optimizers on each of four representative tasks: Linear Regression, Rosenbrock, Two Moons, MNIST . These tasks were selected because they are fast to train, cover a range of loss surfaces (convex and non-convex, low- and high-dimensional, deterministic, and stochastic). Additionally, three baseline optimizers (momentum, RMSProp, and Adam) were tuned individually for each task. The learned optimizers outperformed the baseline optimizers on the meta-objective.\n\nThe submission uses two primary methods to analyze optimizer behavior and mechanisms: 1) visualize optimizer update function at a particular optimizer state. 2) analyze optimizer dynamics around fixed points.  The study was able to show that learned optimizers implement classical momentum and a soft form of gradient clipping. The paper visualizes and provides a careful explanation for the best learned optimizer for Rosenbrock (others in appendix) for both. The report also analyzes learned behavior for learning rate schedules and learning rate adaptation. The study isolated the behavior of tuning per layer and parameter type in the CNN for the MNIST problem.\n\nThe submission includes a detailed Appendix that illustrates the same behaviors for all the other learned optimizers not included in the main paper, along with other materials such as the meta-learning approach.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The topic of the submission, learned optimizers, is an important and timely topic. Overall, the submission is well-written, takes a systematic approach to analyzing the behaviors of learned optimizers, and leverages visualizations and detailed explanations to provide evidence of the learned behaviors.  The findings are useful in validating that behaviors of learned optimizers align with manually-designed optimizers, and equally important, valuable as a foundation for further analysis and potential advancement of learned optimizers.\n\nThe cases for momentum, gradient clipping, and learning rate adaptation were especially clearly laid out and visualized. \n\nThe learning rate schedule discussion (Section 5.3) illustrated that a learned optimizer adjusted the learning rate over iterations. The report claims the learned optimizer is implementing a “schedule” using autonomous (not input driven) dynamics. However, it is unclear from this analysis that the learned optimizer is actually implementing a “schedule” or has only learned to adapt the learning rate. Additional analysis and evidence to support the assertion that a schedule was learned would strengthen this section.\n\nThe learning rate adaptation discussion (Section 5.4), explained well the effects of the learning rate adaptation by the learned optimizer. The illustrations breaking down the behaviors for the positive and negative gradients were well explained and very helpful for illustrating adaptation taking place by the optimizer.  However, line 236 refers to a “new mechanism” implemented by the learned optimizer. Given that manually-designed/baseline optimizers also leverage adaptive learning rates, it was unclear why this is being characterized as a “new mechanism.” \n\nThe additional materials and discussion in the Appendix materially added to the findings, and it would have been helpful to have had some of the information from the Appendix (e.g., potential functional benefits of some of the learned behaviors) earlier, in the main text). \n\nIt would also be interesting to understand which of the identified behaviors the learned optimizers that performed suboptimally did not exhibit and whether some connection could be made to these behaviors and the performance of the optimizer.\n\nNit: 193 “(gray line in Fig. 3b)” Perhaps it was an issue with the rendering, but I could not see a gray line in 3b (?)\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No potential negative societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "OmCedmA23et", "reviewer": "Reviewer_sWNV", "summary": "The paper deals with the analysis of the learned optimizers (LO). In particular, it tries to isolate their properties and distinctions with respect to the existing baseline optimizers.\n\nIt is true that a nascent field of learned optimizers, while enjoying good performance, currently lacks in depth analysis of why the results work. This paper doesn't answer all the questions, but it attempts to bridge the gap between a mysterious performance of LO and known well analyzed optimization techniques like momentum, gradient clipping, LR schedules and adaptation.\n\nThe paper uses 4 different tasks for the analysis (Linear regression, Rosenbrock, Two moons and MNIST) and uses a simple RNN optimizer for a LO. \n\nThe presented empirical result is analyzed from a theoretical perspective of a general LO-state update function. The theory has a very mild assumptions on the LO for analysis (essentially only that they have to converge to the fixed point). \n\nThe paper is mostly well written, however, some parts are hard to read, misleading or not explained well (see bellow).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First of all, I think that these kind of analysis paper are very valuable and I encourage the authors to design a toolbox that can be applied to a given LO in order to analyze and understand its performance.\n\nMy main concern is the generalization and applicability of the analysis of the paper. From what I understand the paper interlaces and switches between two modes:\n- theoretical analysis of a general LO of the form from eqn. (1-2). This was used e.g. to derive an expression for Momentum in section 5.1. \n- empirical analysis of 4 tasks with limited theoretical understanding (e.g. section 5.2, 5.3 and partially 5.4).\n\nThe theoretical analysis has a very mild assumptions on the LO form and only requires that the optimizer state converges to a fixed point. The empirical analysis focuses on the RNN LO, but I'm curious what example of a general convergent update rule *are not* covered by this analysis? Is RNN just use as a general example with any other LO being applicable as well? \n\nIt is a little bit confusing how much we can really generalize beyond 4 tasks and which conclusions to make out of this analysis? The authors showed that a given optimizer exhibit certain types of behavior that we know from known optimizers, but _is this surprising and unexpected_? What can we learn from this analysis, beyond just \"it is a first step and we hope the results would be find useful down the line\"? It would be great if the authors add a comparative table in the main text (or appendix) with a summary of what they learned from each of 4 tasks. Together with a loss decrease curves from fig 1 top these would be a good succinct outcome. \n\nMost of the other comments below are just clarifying the confusions when I read the paper. While this is not crucial, the paper would benefit greatly from a careful proofreading. \n\nComments:\n- I would be a little careful with the notation, e.g. l.61 defines $w$ as readout weights. I don't see why they are called like this? What exactly do they represent? They are not defined in Andrychowicz et al nor anywhere else I looked. Are $w$ are being used at all in the paper?\n- l.69: state variable is a scalar, later: state var are low-dimensional.\n- l.72: \"dynamics are straightforward\". Arguably so, e.g. Nesterov accelerated gradient.\n- a lot of important information is hidden in the Appendix. It is quite hard to read the paper as is. without looking back and forth to the Appendix.\n- l84: \"fast to train\". Not precise enough. Fast with respect to what? Low number of iterations? Fast gradient computation? All of the above?\n- l.86: maybe instead of random, create 2-3 group with low/normal/hard Hessian conditioning? I don't think that fig 1(a) would be too packed if one adds few more curves.\n- Fig 1: maybe use log-x and show the dynamic till convergence?\n- Fig 1: different shares of red/orange are very hard to distinguish. Consider using different color palette.\n- which tasks were used to train the meta-optimizer? Same as the training? The paper mention this at l.164, but it is not clear.\n- l.178: misleading: learned optimizer is not *approximately linear*, the authors *linearly approximate it* using the Jacobian. These two statements mean very different things.\n- Momentum is defined differently at l.135 than at line 623 and uses different lingo (velocity and mom. hyperparameter vs timescale and learning rate).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not include any section to address societal impact of their work. Since this is mostly an analysis of _existing_ algorithms, I think it is ok. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 "}, "extra_text": {}}, {"review_id": "8aVjG_BwmKZ", "reviewer": "Reviewer_Phij", "summary": "The authors devise tools for investigating the behaviour of learned, gradient-based, optimization algorithms in order to understand better how they achieve their empirically validated high performance. \n\nTo this end, they visualize the update function, i.e., the change in parameters in dependence of the gradient input, which allows to identify mechanisms employed also by hand-designed optimizers, such as gradient clipping, momentum, and learning rate adaptation. The authors demonstrate that this visualization is particularly insightful when analyzing the update rule using the linearized optimizer state dynamics around fixed points. Furthermore, the authors identify that learned optimizers encode learning rate schedules through autonomous state trajectories. \n\nUsing their tools, the authors analyze one existing type of learned optimizer (the RNN-based approach proposed by Andrychowicz et al., \"Learning to Learn by Gradient Descent by Gradient Descent\") on four relatively simple problems (linear regression, Rosenbrock, two moons, MNIST). They find that (depending on the problem) the learned optimizer can exhibit well-known mechanisms like gradient clipping, momentum, learning rate adaptation, and learning rate schedules. Furthermore, they provide clues that the optimizer can learn parameter-type (bias/weights) and layer-specific learning-rate schedules in neural-network based problems.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The authors propose to combine visualization tools and methods from dynamical system analysis to gain useful insights into the behaviour of learned, gradient based optimizers. While neither their analyzation methods nor the analyzed optimization algorithm are novel, the application of those methods to better understand the learned optimizers is new and the way the authors combine said methods is clever and insightful. \n\nQuality: The proposed analyzation approach uses well-understood methods from dynamical system analysis to visualize the behaviour of learned optimizers. The authors apply these methods in a technically sound manner and on a range of simple problems. As the approach is intuitive and concise, it yields a range of useful insights into the behaviour of learned optimizers and unveils both parallels to known optimization mechanisms as well as novel behaviours. The authors admit that their work does not yet yield a \"holistic picture\" of learned optimizer behaviour. I agree with that, i.p., because the authors analyze only one particular type of learned optimizer. The submission could be strengthened by including different learned optimizer architectures.\n\nClarity: The submission is very well written and the visualizations are of high quality. In my opinion, the exposition could be strengthened by considering the following points:\n\n- Sec. 5.1:\n    - Maybe try to merge some of the information/equations of App. C into the main text.\n    - Provide some more background on why the \"learned optimizer uses a single eigenmode to implement momentum\" (l. 189).\n- Sec. 5.3:\n    - Provide some more background on the connection between autonomous trajectories and the implementation of learning rate schedules.\n- Sec. 3.2:\n    - How do you generate a distribution of problems for MNIST? Do you also sample the initial weights of the CNN?\n\nMinor points (not influencing my score):\n\n- l. 36: three disparate tasks → four disparate tasks\n- Eq. (2) does not contain the readout function $r$ referred to in l. 658.\n\nSignificance: The authors compile a toolbox of analyzation methods for learned optimizers. While the used methods are not new, the authors apply them in an intuitve and clever way. Similarly, while the experiments do not yet yield a holistic picture (i.p. because the authors apply their tools to only one type learned optimizer), they still provide some interesting and important insights into the behaviour of the RNN-based optimizer. Therefore, I think the submission in its current form is significant, i.p., because future work will benefit from it. I therefore recommend acceptance.\n\n---------------------------------------------------\nAfter reading the other reviews, I am even more convinced that the paper is interesting and valuable and the proposed methodology is sound. In addition, the authors will improve on the clarity of their manuscript. Thus, I raise my score to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No potential negative societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was very actively discussed amongst the reviewers.

Points in favor were summarized by reviewer QfSG as follows:

1. The analytical tools presented in the paper are a strong foundation for characterizing learned optimizers. Such tools are lacking in the field. In addition to the tools themselves (which are useful for other L2O researchers), the systematic approach taken to the experiments and explanations could also be helpful for others seeking to contribute such tools (for and beyond L2O methods).

2. The methodology, experiments, and conclusions taken from the experiments are well-grounded and solid, especially if one takes into consideration the significant additional depth provided in the appendix.

3. The results show that learned optimizers match or mimic techniques in state-of-the-art optimizers, despite being trained on diverse types of optimization problems - which gives merit to both L2O approaches and to the tools presented. That is, even if it is not surprising, given how learned optimizers are derived, that learned optimizers learn similar techniques as known optimizers -- that finding (learning to similar methods) brings merit to L2O approaches as finding 'trusted' approaches.

4. The authors demonstrates applicability beyond just the 4 tasks, and responded to the concerns.

Reviewer sWNV replied to these 4 points as follows:

1. I agree that these tools are lacking and having a paper like this (and especially a toolbox) would lie a good foundation onto a better understanding of the learned optimizers. This is undeniable contribution and I'm happy that this paper tackles it.

2. The techniques provided in the paper are good, however, there are still some questions remain. For example, the momentum analysis (sec 5.1 and App C) relies on the linear approximation of the recurrent dynamics. It is not clear to me that this approximation is useful and actually holds in practice. It would be useful to actually run the momentum optimizer with the parameters recovered by this analysis and see if it is close to the training dynamics of the LO. Of course, this also would be not accurate, since the LO also do much more than just momentum...

3. I think that it is a bit of an overstatement to say that LO match or mimic known optimizers, they specifically being tested to match or mimic the specific property of a given optimizer. I would love to see an example when this does not happen and the LO solely memorizes the task without exhibiting the properties of the known optimizers. 

4. My main criticism is that the from the title, introduction and the discussion it seems like the analysis holds and applies to many Learned Optimizers, whereas in reality it is only 1 that is being analyzed. This by itself is not a problem and doesn't deny the technical contribution, but it is misleading to the reader (especially the cursory reader that relies of Abstract + Intro for the quick understanding of the paper). I would really encourage the authors to pivot the narrative of the paper and instead focus on the techniques and methods of general LO analysis instead of claims about Learned Optimizers in general. To this extend, I would like to re-iterate that the release of the general and easy-to-use toolbox that can generate an extensive report given a used provided LO and a task (ideally a benchmark of many tasks) should be an essential part of this paper.

After this internal discussion, the reviewers offered to change the narrative and title of the paper to alleviate the concerns in 4. above.

Overall, the paper presents a solid contribution to the field of learned optimizers, and I recommend acceptance. I encourage the authors to act on the suggestions by reviewer sWNV, especially adapting narrative + title.
Decision: Accept (Poster)