Optimizer Amalgamation
Authors: Tianshu Huang
Tianlong Chen
Sijia Liu
Shiyu Chang
Lisa Amini
Zhangyang Wang
Venue: ICLR 2022 Poster (2022)
Keywords: Learning to Optimize
Knowledge Amalgamation
Stability-Aware Training

Abstract: Selecting an appropriate optimizer for a given problem is of major interest for researchers and practitioners. Many analytical optimizers have been proposed using a variety of theoretical and empirical approaches; however, none can offer a universal advantage over other competitive optimizers. We are thus motivated to study a new problem named Optimizer Amalgamation: how can we best combine a pool of "teacher" optimizers into a single "student" optimizer that can have stronger problem-specific performance? In this paper, we draw inspiration from the field of "learning to optimize" to use a learnable amalgamation target. First, we define three differentiable amalgamation mechanisms to amalgamate a pool of analytical optimizers by gradient descent. Then, in order to reduce variance of the amalgamation process, we also explore methods to stabilize the amalgamation process by perturbing the amalgamation target. Finally, we present experiments showing the superiority of our amalgamated optimizer compared to its amalgamated components and learning to optimize baselines, and the efficacy of our variance reducing perturbations.


Published as a conference paper at ICLR 2022
OPTIMIZER AMALGAMATION
Tianshu Huang!.? 1.2 Tianlong Chen', Sijia Liu", Shiyu Chang', Lisa Amini', Zhangyang Wang' 'University of Texas at Austin, "Carnegie Mellon University, "Michigan State University. *University of California, Santa Barbara, 'MITIBM Watson AI Lab, IBM Research t tianshuêcmu. edu, {tianlong. chen, at atlasaang] eet.ueeeeesuuuu liusi ji50i151mm 115mmu.....mmmm edu, chang87 edu, lisa ami www.iiiii.mmmmmm com
ABSTRACT
Selecting an appropriate optimizer for given problem is of major interest for researchers and practitioners. Many analytical optimizers have been proposed using a variety of theoretical and empirical approaches; however, none can offer a universal advantage over other competitive optimizers. We are thus motivated to study a new problem named Optimizer Amalgamation: how can we best combine pool of "teacher" optimizers into a single "student" optimizer that can have stronger problem-specific performance? In this paper, we draw inspiration from the field of "learning optimize" to a learnable amalgamation target. First, we define three differentiable amalgamation mechanisms to amalgamate a pool of analytical optimizers by gradient descent. Then, in order to reduce variance of the amalgamation process, we also explore methods to stabilize the amalgamation process by perturbing the amalgamation target. Finally, we present experiments showing the superiority of our amalgamated optimizer compared to its amalgamated components and learning to optimize baselines, and the efficacy of our variance reducing perturbations. Our code and pre-trained models are publicly available at attp://gitann
-roo- Opt imi zer Ama gamatittionnn
INTRODUCTION
Gradient-based optimization is ubiquitous in machine learning; accordingly, a cottage industry of pradient-based optimizer design has emerged (Schmidt al., 2020). These optimizers generally propose algorithms that aim to make the "best" parameter update for computed gradient (Kingma & Ba, 2017; Liu al., 2020), with some also modifying the location where the parameters are computed (Zhang et al., 2019b). However, each gradient-based optimizer claim specific problems where they hold performance advantages, none can claim to be universally superior. Due to the "No Free Lunch" theorem for optimization (Wolpert & Macready, 1997). no optimizer can provide better performance on a class of problems without somehow integrating problem-specifi knowledge from that class.
Furthermore, problems such as training neural networks are not homogeneous. In the spatial dimension, different layers or even parameters can have different behavior (Chen et al., 2020b). Also, evidenced by the popularity of learning rate schedules, neural network optimization also behaves very differently in the temporal dimension as well (Golatkar et al., 2019). This implies that no optimizer can provide the best performance for all parameters single problem or best performance over the entire optimization process.
In order build stronger optimizer, we propose the new problem of optimizer amalgamation: how can we best combine pool of multiple "teacher" optimizers, each of which might be good in certain cases, into single stronger "student" optimizer that integrates their strengths and offsets their weaknesses? Specifically, we wish for our combined optimizer to be adaptive both per- per-parameter and per-iteration, and exploit goblem--peciffe knowledge to improve performance on a class of problems.
To "amalgamate" an optimizer from a pool of optimizers, we draw inspiration from recent work in Learning to Optimize (Chen al., 2021a) which provides a natural way to parameterize and train optimization update rules. In Learning to Optimize, optimizers are treated as policies to be learned from data. These "learned" optimizers are typically parameterized by a recurrent neural network
Work done while the author was at the University of Texas Austin.

---

Published as a conference paper at ICLR 2022
(Andrychowicz et al., 2016; Lv et al., 2017); then, the optimizer is meta-traine trained to minimize the loss of training problems, or "optimizees", by gradient descent using truncated back propppppaatttinn through time. Yet to best knowledge, no existing work has leveraged those learnable parameterizations to amalgamate and combine analytical optimizers.
For our proposed formulation of optimizer amalgamation, we treat the learned optimizer as the amal- gamation target. Then, we define amalgamation losses which can be used to combine feedback from multiple analytical optimizers into a single amalgamated optimizer, and present several amalgamation schemes. Finally, we explore smoothing methods that can be used during the amalgamation process to reduce the variance of the amalgamated optimizers. Our contributions are outlined below:
We formulate the new problem of optimizer amalgamation, which we define finding way to best amalgamate a pool of multiple analytical optimizers to produce a single stronger optimizer. We present three schemes of optimizer amalgamation: additive amalgamation, min-max amalgamation, and imitation of a trained choice.
We observe instability during the amalgamation process which leads to amalgamated opti- mizers having varied performance across multiple replicates. To mitigate this problem, we explore ways to reduce amalgamation variance by improving smoothness of the parameter space. We propose smoothing both by random noise or adversarial noise.
We present experiments showing extensive and consistent results that validate the effective- ness of our proposal. Specifically, we find that more advanced amalgamation techniques and weight space training noise lead better average case performance and reduced variance. We also show that our amalgamation method performs significantly better than previous methods on all problems, with few exceptions.
2 RELATED WORKS
Knowledge Distillation and Amalgamation The prototype of knowledge distillation was first introduced by (Bucilua et al., 2006), which used it for model compression in order train neural networks ("students") to imitate the output of more complex models ("teachers"). Knowledge distillation was later formalized by (Hinton et al., 2015), who added a temperature parameter to soften the teacher predictions and found significant performance gains as result.
The success of knowledge distillation spurred significant efforts to explain its effectiveness. Notably, Chen et al. (2020e); Yuan et al. (2020) discovered that trained distillation teachers could be replaced by hand-crafted distributions. (Yuan et al., 2020) provided further theoretical and empirical explanation for this behavior by explicitly connecting Knowledge distillation to label smoothing, and (Ma et al.; creditedd Chen et al., 202 1b) further credited the benefits of knowledge distillation to the improved smoothness of loss surfaces, which has been demonstrated help adversarial training Cohen al. (2019); Lecuyer et al. (2019) and the training of sparse neural networks Ma et al..
The potential of knowledge distillation to improve the training of neural networks also spurred diverse works extending knowledge distillation. For example, (Romero et al., 2015; Wang et al., 2018; Shen et al., 2018; 2019b; Ye al., 2020b) propose using intermediate feature representations as distillation targets instead of just network outputs, and (Tarvainen & Valpola, 2017; Yang et al., 2018; Zhang et al., 2019a) unify student and teacher network training to reduce computational costs. Knowledge distillation has also been extended to distilling multiple teachers, which is termed Knowledge Amalgamation (Shen et al., 2019a; Luo et al., 2019; Ye et al., 2019; 2020a).
Although using output logits from pre-trained networks has been extensively explored in knowledge distillation, we study a new direction of research distilling optimization knowledge from sophisticated analytical optimizers to produce stronger "learned" optimizers, hence the name "optimizer amalga- mation". Not only this is a new topic never studied by existing knowledge distillation literature, but also it needs to distill longitudinal output dynamics- not one final output from multiple teachers.
Learning to optimize Learning to Optimize is branch of meta learning which proposes to replace hand-crafted analytical optimizers with learned optimizers trained by solving optimization problems, or optimizees. The concept was first introduced by (Andrychowicz et al., 2016), who used a Long Short- Memory (LSTM) based model order to parameterize gradient- based optimizers. This model took the loss gradient as its input and output learned update rule which was then trained by
2

---

mechanism for Published as a conference paper at ICLR 2022
gradient descent using truncated backpropagation through time. (Andrychowicz et al., 2016) also established a coordinate-wir wise design pattern, where the same LSTM weights are applied to each parameter of the optimizee in order to facilitate generalization to models with different architectures.
Building on this architecture, Wichrowska et al. (2017) and Lv et al. (2017) proposed improvements such hierarchical architectures connecting parameter RNNs together and augmenting the gradient with additional inputs. Many methods have also been proposed to improve the training of learned optimizers such as random scaling and convex augmentation (Lv et al., 2017), curriculum learning and imitation learning (Chen et al., 2020a), and Jacobian regularization (Li et al., 2020). Notably, Chen et al. (2020a) also proposed method of imitation learning. which can be viewed as a way of distilling single analytical optimizer into learned parameterization.
Learning to Optimize has been extended to variety of other problems such as graph convolutional networks (You et al., 2020), domain generalization (Chen et al., 2020b), noisy label training (Chen et al., 2020c), adversarial training (Jiang et al., 201 8; Xiong Hsieh, 2020), and mixmax optimiza- tion (Shen et al., 2021). Moving away from gradient-based optimization, black-box optimization has also been explored (Chen et al., 2017: Cao et al., 2019). For comprehensive survey with benchmarks, readers may refer to Chen et al. (2021a).
Perturbations and Robustness The optimization process is naturally subject to many possible sources of noise, such the stochastic gradient noise Devolder et al. (2011); Gorbunov al. (2020); Simsekli et al. (2019) which is often highly non- man-Gassiinn and heavy-tail in practice; the random initialization and (often non-optimal) hyperparameter configuration; the different local minimum reached each time in non-convex optimization Jain & Kar (2017); and the limited numerical precision in implementations De Sa et al. (2017). The seen and unseen optimizees also constitute domain shifts in our case. In order for consistent and reliable amalgamation process, the training needs to incorporate resistance to certain perturbations of the optimization process.
We draw inspiration from deep learning defense against various random malicious perturbations. For example, stability training Zheng et al. (2016) stabilizes deep networks against small input distortions by regularizing the feature divergence caused by adding random Gaussian noises to the inputs. Adversarial robustness measures the ability of a neural network to defend against malicious perturbations of its inputs (Szegedy et al., 2013; Goodfellow et al., 2014). For that purpose, random smoothening (Lecuyer et al., 2019; Cohen et al., 2019) and adversarial training (Madry et al., 2017) have been found to increase model robustness with regard to random corruptions or worst-case perturbations; as well as against testing-time domain shifts Ganin et al. (2016). Recent work (He et al., 2019; Wu al., 2020) extends input perturbations weight perturbations that explicitly regularize the flatness of the weight loss landscape, forming both inputs and weights.
Other Approaches The problem of how to better train machine learning models has many diverse approaches outside the Learning to Optimize paradigm that we draw from, and forms the broader AutoML problem Hutter et al. (2018) together with model selection algorithms. Our approach falls under meta-learning, which also includes learned initialization approaches such as MAML (Finn al., 2017) and Reptile (Nichol et al., 2018). Other optimizer selection and tuning methods include hypergradient descent (Baydin et al., 2017) and bayesian hyperparameter optimization (Snock et al., 2012). Similar to our knowledge amalgamation approach, algorithm portfolio methods (where many algorithms are available, and some subset is selected) have also been applied to several problem domains such as Linear Programming (Leyton-Brown et al., 2003) and SAT solvers (Xu et al., 2008).
3
OPTIMIZER AMALGAMATION
3.1 MOTIVATION
Optimizer selection and hyperparameter optimization is a difficult task even for experts. With a vast number of optimizers choose from with varying performance dependent the specific problem and data (Schmidt et al., 2020), most practitioners choose a reasonable default optimizer such as SGD or Adam and tune the learning rate to be "good enough" following some rule of thumb.
As a consequence of the No Free Lunch theorem (Wolpert & Macready, 1997), the best optimizer to use for each problem, weight tensor within each problem, or each parameter may be different.
3

---

Published as a conference paper at ICLR 2022
In practice, different layers within given neural network can benefit from differently tuned hyper- parameters, for example by meta-tuning learning rates by layer (Chen et al., 2020b).
Accordingly, we wish to train optimizer which is sufficiently versatile and adaptive at different stages of training and even to each parameter individually. Many methods have been proposed to parameterize optimizers in learnable forms including coordinate-wic LSTMs Andrychowicz et al. (2016); Lv et al. (2017), recurrent neural networks with hierarchical architectures Wichrowska et al. (2017); Metz al. (2019), and symbolically in terms of predefined blocks Bello al. (2017). Due to its high expressiveness and relative ease of training, we will use the workhorse of LSTM-based RNNProp architecture described by Lv et al. (2017) as our amalgamation target; more details about this architecture can be found in Appendix C.I.
3.2 THE BASIC DISTILLATION MECHIANISM
Knowledge distillation can be viewed as regularizing the training loss with a distillation loss that measures the distance between teacher and student predictions (Hinton et al., 2015). In order to distill a pool of teacher optimizers T = Ti, T2.................. Th into our target policy P truncated backpropagation (Appendix A: Algorithm 1), we start by defining training loss and amalgamation loss.
Meta Loss In the context of training optimizers, the training loss is described by the meta loss, which is function of the optimizee problem loss each step (Andrychowicz et al., 2016). Suppose we are training a policy d' with parameters & on a problem M X' + E whose output is loss for each point in data domain X. During each iteration during truncated backpropagation through time, is used to compute parameter updates for M to obtain trajectory of optimizee parameters 01, 02..... ON where for the ith data batch æ; and parameters 0; at step i.e. 0.+1 = 0, - o. 0,)).
For some weighting function f1, f2...... ..2....fo. fN. the meta loss is me xx, 0;: 6) = Σ i=1 f.(M(z, 0()): specifically, will use the scaled log meta loss fi(m) llg(mm) log( 0o)), which can be interpreted as the "mean log improvement."
Distillation Loss The distillation loss in knowledge distillation measures the distance between teacher predictions and student predictions. In training optimizers, this corresponds to the distance between the optimization trajectories generated by the teacher and student. Suppose we have optimizee parameter trajectories 0, = (o. ,o(T)) generated by the teacher and student, respectively. Then, our distillation loss L for teacher T is given by the lg log-loss:
Cr σ) = N Σ log ( OT))
1
While knowledge distillation generally refers to imitating model and imitation learning imitating a policy, the optimizer in our case can be regarded as both a model and a policy. As such, our loss function is similar the imitation loss mechanism used by Chen et al. (2020a), which can be thought of as a special case of optimizer amalgamation where only single teacher is used.
3.3 AMALGAMATION OF MULTIPLE TEACHER OPTIMIZERS: THREE SCHEMES
Now, what there are multiple teachers that we wish to amalgamate into a single policy? How to best combine different knowledge sources is a non-trivial question. We propose three mechanisms:
(1) Mean Amalgamation: adding distillation loss terms for each of the optimizers with constant equal weights.
(2) Min-max Amalgamation:
using min-max approach to combine loss terms for each of the optimizers, i.e., "the winner (worst) takes all".
(3) Optimal Choice Amalgamation: First training an intermediate policy to choose the best optimizer to apply at each step, then distilling from that "choice optimizer".
Mean Amalgamation In order to amalgamate our pool of teachers T Ti.... .T. we generate
[T + 1 trajectories 0, = ( (((T))
((T))) and add distillation losses for each teacher:
[
Ereaa (x; 0;:0) Cime (x; 0!²)
log |o!-)
0. 0,2000
(2)
4

---

Published as a conference paper at ICLR 2022
If we view knowledge distillation regularizer which provides soft targets during training, mean amalgamation the logical extension of this by simply adding multiple regularizers to training.
An interesting observation is: when multiple teachers diverge, mean amalgamation loss tends to encourage the optimizer to choose one of the teachers to follow, potentially discarding the influence of all other teachers. This may occur if one teacher is moving faster than another in the optimizee space, or if the teachers diverge in the direction of two different minima. As this choice is local minimum with respect the mean log amalgamation loss, the optimizer may "stick" to that teacher, even if it is not the best choice.
Min-Max Amalgamation In order to address this stickiness, we propose a second method: min- max amalgamation, where, distillation losses are instead combined by taking the maximum distillation loss among all terms at each time step:
N
main-max (x; 0): Lmera ( +
max log |0!-) -0 TET
(3)
N
This results in a v-shaped loss landscape which encourages the amalgamation target to be between the trajectories generated by the teacher pool and prevents the optimizer from "sticking" to one of the teachers.
One weakness shared by both mean and min-max amalgamation is memory usage. Both require complete training trajectories for each teacher in the pool to be stored in memory, resulting in memory usage proportional the number of teachers, which limits the number of teachers that we could amalgamate from in one pool.
Min-max amalgamation also does not fully solve the problem of diverging teachers. While min- max amalgamation does ensure that no teacher is ignored, it pushes the amalgamation target to the midpoint between the optimizee weights of the two teachers, which does not necessarily correspond to good optimizee loss. In fact, when teachers diverge into multiple local minima, any solution which considers all teachers must necessarily push the learned optimizer against the gradient, while any solution which allows the learned optimizer to pick one side must discard a number of teachers.
Optimal Choice Amalgamation To fully unlock the power of knowledge amalgamation, we propose solve the teacher divergence problem by first training an intermediate amalgamation target. By using only one teacher for a final distillation step, we remove the possibility of multiple teachers diverging while also allowing us to use more teachers without memory penalty.
For optimizer pool T. we define an choice optimizer C' which produces choices C1 C2. CN of which optimizer in the pool to apply each time step, producing updates 0.C) 0( Te (g)).
The objective of the choice optimizer is to minimize the meta loss Lueu (C: x:) with respect to these choices CJNN- We parameterize the choice function C as a small two-layer LSTM, and train by gradient descent. The LSTM takes the outputs of each optimizer in the pool, the layer type, and time step as inputs; more details are provided in Appendix C.I. To make it easier to train C by truncated poopapropag through time, we relax the choices C1.VV to instead be soft choices c, E ||| a 2 0, ||d:|l1 = 1, resulting in the policy (0 = 0(0) - >) as a teacher to produce our final loss:
f e.
(g.). Now, we use
1
n
log 10!00 0.
Lehoies -ymera (cb; xx) + a
(4)
4 STABILITY-AWARE OPTIMIZER AMALGAMATION
4.1 MOTIVATION
Modern optimization, even analytical, is subject to various forms of noise. For example, stochastic first-order method are accompanied with gradient noise (Devolder et al., 2011; Gorbunov al., 2020; Simsekli et al., 2019) which often highly non-Gaussian and heavy-tail in practice. Any non-convex optimization could reach different local minimum when solving multiple times (Jain &
5

---

Published as a conference paper at ICLR 2022
Kar, 2017). When training deep neural networks, thousands millions of optimization steps typically run, and the final outcome can be impacted by the random initialization. (often non-optimal) hyperparameter configuration, and even hardware precision (De Sa et al., 2017). Hence, it is highly desirable for optimizers to be stable: across different problem instances, between multiple training runs for the same problem, and throughout each training run (Lv al., 2017).
Meta-training optimizers tends to be unstable. During the amalgamation process, we encounter significant variance where identically trained replicates achieve varying performance on our evaluation problems; this mirrors problems with meta-stability encountered by Metz et al. (2019). While amalgamation variance can be mitigated in small-scale experiments by amalgamating many times and using the best one, that variance represents significant obstacle to large-scale training (i.e. on many and larger problems) and deployment of amalgamated optimizers. Thus, besides the aforementioned optimization stability issues, we also need to consider mela-stability, denoting the relative performance of optimizers across meta-training replicates.
In order to provide additional stability to the amalgamation process, we turn to adding noise during training, which is known to improve smoothness (Chen Hsieh, 2020; Lecuyer et al., 2019; Cohen et al., 2019) and in turn improve stability (Miyato et al., 2018). Note that one can inject either random noise or adversarial perturbations onto either the input or the weight of the learnable optimizer. While perturbing inputs is more common, recent work (Wu et al., 2020) identified that a flatter weight loss landscape (loss change with respect to weight) leads to smaller robust generalization gap in adversarial training, thanks its more "global" worst-case view.
We also discover in our experiments that perturbing inputs would make the meta-training hard to converge, presumably because the inputs to optimizers (gradients, etc.) already contain large amounts of batch noise and do not tolerate further corruption. We hence focus on perturbing optimizer weights for smoothness and stability.
4.2 WEIGITT SPACE PERTURBATION FOR SMOOTINEES
Weight space smoothing produces noised estimate of the loss by adding noise to the optimizer parameters Φb By replacing the loss L(db, æ)) with a noisy loss f = L(Ø, æ), we encourage the optimizer to be robust to perturbations of its weights, increasing the meta-stability. We explore two mechanisms to increase weight space smoothness during training, by adding (1) random perturbation to the weights as a gradient estimator, and (2) an adversarial perturbation in the form of projected gradient descent attack (PGD).
Though new to our application, these two mechanisms have been adopted for other problems where smoothness is important such neural architecture search (Chen & Hsich, 2020) and adversarial robustness (Lecuyer et al., 2019; Cohen et al., 2019).
Random Gaussian Perturbation In the first type of noise, we add gaussian noise with variance x² to each parameter of the optimizer at each iteration, @ = e + (V o2T)
Since optimizer weights tend to vary largely in magnitude especially between different weight tensors, we modify this gaussian noise be adaptive to the magnitude of the l2 norm of each weight tensor [(w), For tensor size (o(w)), the added noise is given by
((()) ((()) +N (0.02 (()[)(())) 12 [ob(w) 1
(5)
Projected Gradient Descent For the second type of noise, we use adversarial noise obtained by projected gradient descent (Appendix A, Algorithm 2). For A adversarial steps, the noised parameters are given by ă = o + vh where y/u 0, and M4+1
t/ + 17 clip, (V, L) for optimizer loss L.
As with random Gaussian perturbations, we also modify the adversarial perturbation to be adaptive with magnitude proportional to the l/ norm of each weight tensor Φb. Here, the adversarial attack step for weight tensor is instead given by
E||0|12 V
(n(n C
(10) i+1 w
(w)
(6)
||
Cl/2

---

Published as a conference paper at ICLR 2022
5 EXPERIMENTS
Optimizee Details All optimizers were amalgamated using a 2-layer convolutional neural network (CNN) on the MNIST (LeCun & Cortes, 2010) dataset (shortened as "Train") using batch size of 128. During evaluation, we test the generalization of the amalgamated optimizer to other problems: O Different Datasets: FMNIST (Xiao et al., 2017) and SVHN (Netzer et al., 2011) 1). We also run experiments on CIFAR (Krizhevsky et al., 2009); since the Train network is too small to obtain reasonable performance on CIFAR, substitute it for the Wider architecture and 28-layer ResNet (He et al., 2015) labelled "CIFAR" and "ReNNet" respectively. @ Different Architectures: a 2-layer MLP (MLP), a CNN with twice the number of units in each layer (Wider), and a deeper CNN (Deeper) with 5 convolutional layers. 2 Training settings: training with a smaller batch size of 32 (Small Batch). We also try new setting of training with differential privacy (Abadi et al., 2016) (MNIST-DP). Appendix B provides full architecture and training specifications.
Optimizer Pool We use two different optimizer pools in our experiment: "small," which consists of Adam and RMSProp, and "large." which also contains SGD, Momentum, AddSign, and PowerSign. Each optimizer has learning rate tuned by grid search over grid of {5 x 10 x 10-$,22 x 10 3 ....1): 1}. The selection criteria is the best validation loss after epochs for the Train network on MNIST, which matches the meta-training settings of the amalgamated optimizer. Appendix C.2 describes the optimizers used and other hyperparameters.
Baselines First, we compare amalgamated optimizer against analytical optimizer teachers which are combined into a "oracle optimizer," which is the optimizer in our pool of teachers with the best validation loss. We also compare against the optimal choice optimizer used in amalgamation, which functions like per-iteration trained approximation of the oracle optimizer. Then, we evaluate previous learned optimizer methods: the original "Learning to Learn by Gradient Descent by Gradient Descent" optimizer Andrychowicz et al. (2016) 6) which we refer to as "Original", RNNProp (Lv et al., 2017), a hierarchical architecture presented by "Learned Optimizers that Scale and Generalize" (Wichrowska et al., 2017) which we refer to as "Scale", and the best setup from Chen et al. (2020a) which shorten as "Stronger Baselines."
Training and Evaluation Details The RNNProp amalgamation target was trained using truncated backpropagation though time with a constant truncation length of 100 steps and total unroll of up to 1000 steps and meta-optimized by Adam with a learning rate of 10 10-3 For our training process, we also apply random scaling (Lv et al., 2017) and curriculum learning (Chen et al., 2020a); more details about amalgamation training are provided in Appendix C.3. Amalgamation takes up to 6.35 hours for optimal choice amalgamation using the large pool and up 10.53 hours when using adversarial perturbations; a full report of training times is provided in Appendix C.4.
For each optimizer amalgamation configuration tested, we independently trained 8 replicate optimiz- ers. Then, each replicate was evaluated 10 times on each evaluation problem, and trained to a depth of 25 epochs each time. Finally, we measure the stability of amalgamated optimizers by defining three notions of stability for meta-trained optimizers: Optimization stability: the stability of the optimizee during the optimization process. Viewing stability of the validation loss proxy for model stability with respect to the true data distribution, we measure the epoch-to-epoch variance of the validation loss after subtracting a smoothed validation loss curve (using a Gaussian filter). e Evaluation stability: the variance of optimizer performance across multiple evaluations. We find that the evaluation stability roughly the same for all optimizers (Appendix 1) Meta-stability: the stability of the amalgamation process, i.e. the variance of amalgamation replicates after correcting for evaluation variance. Meta-stability and evaluation stability are jointly estimated using a linear mixed effects model. The stability is reported as standard deviation. More details are in Appendix D.
5.1 OPTIMIZER AMALGAMATION
Amalgamation Methods Figure/ compares the mean performance of the three amalgamation methods with the small pool and Choice amalgamation with the large pool. Mean and min-max amalgamation were not performed on the large pool due to memory constraints. The amalgamated optimizers using optimal choice amalgamation perform better than Mean and Min-Max amalgamation. The size of the optimizer pool does not appear to have a significant effect Optimal Choice amalgamation, with small pool and large pool amalgamated optimizers obtaining similar results.
7

---

Published as a conference paper at ICLR 2022
(aarqe
T Choice 15mall!
T Kean
Minetax
Beg
0.0
-21
Tron
MLP
Wider
Deeper
RINIST
SWiN
CIFAR-10
Reshit
Small Batch
KNST
Figure 1: Amalgamated optimizer performance measured by the best log validation
and log training
loss (lower is better) after 25 epochs; 95% confidence intervals are shown, and are estimated by linear mixed effects model (Appendix D). In order to use a common y-axis, the validation loss is measured relative to the
mean validation loss of the optimizer amalgamated from the large pool using optimal Choice amalgamation.
Araigamated
Streeger Baoell
I RNWFrop
I
Scale
044
Original
I
-0.2 Trin
MLP
Witer
Deeper
FMNIST
SVHe
CIFAR.10
Resket
Small Batch
MNST
Figure 2: Comparison with other leamed optimizers; for each problem. 95% confidence intervals of the mean are computed using a linear mixed effects model (Appendix D). Error bars are normalized by subtracting mean log validation loss of amalgamated optimizer to use same Y-axis. Uncropped and accuracy versions can be found in Appendix E.3. The amal maalgamatdd optimizer performs better than other learned optimizers on all problems, and is significantly better except in some problems when compared to the Stronger Baselines trained RNNProp (Chen al., 2020a).
Deeper
-2.5
Choice
-32
-35
-32
-20
16
15
25
13
23
23
25 Tachit
28
26
25
10
15
26
Stral Batch
-03
-2.0
-05
3.50
-34
02
-02
30
15
13
15
20
23
28 25 20
38 22
26
25
10
12
28
Figure 3: Comparison between the best Amalgamated Optimizer (blue), the optimal Choice optimizer used to train it (orange), and the oracle optimizer (grange); the shaded area shows -2 standard deviations from the mean. optimizee; full definitions can be found in Appendix B. An version of
The title of each plot corresponds this plot showing validation accuracy can be found in Appendix E.3. The amalgamated optimizer performs similarly or better than the choice optimizer and Oracle analytical optimizer on problems spanning a variety of training settings, architectures, and datasets.
Previous Learned Optimizers Figure 2 compares the amalgamated optimizer against the baselines from Learn- ing to Optimize. Optimizer amalgamation performs signif- icantly better than all previous methods on all problems, with few exceptions (where it performs better but not sig- nificantly better).
500
Acdtign Powerigh
3:30
3.00
0:00
Analytical Optimizers In Figure 3, we compare the best replicate amalgamated from the large pool using Choice amalgamation with the "oracle optimizer" de- scribed above. The amalgamated optimizer achieves sim- ilar or better validation losses than the best analytical opti- mizers, indicating that our amalgamated optimizer indeed captures the "best" loss coss-minimizatinn characteristics of each optimizer.
107
1.06
2.45
Figure 4: Relationship hetween optimiza- tion stability and performance as measured by validation loss on the Train optimizee; smaller stability and validation loss are better. Error bars show 95% confidence intervals; large pool
The amalgamated optimizer also benefits from excellent optimization stability, meeting or exceeding the optimiza- tion stability of the best analytical optimizers in the large
analytical optimizers the optimizer amalgamated from the large pool using Optimal Choice are shown.
8

---

Published as a conference paper at ICLR 2022
0.20
I Choice (aange
I
Adem
0.15
RNSProp
Operitzatoon
0.10
Homentum
Addiiign
0.05
PowerSign
0.00
Train
MLF
Wider
Desper
FMMST
SWHN
CFAR-10
Resket
Small Batch
KNGTT
Figure 5: Optimization stability (lower is more stable) optimizer amalgamated by Optimal Choice from the large pool compared to optimization stability of the optimizers in that pool; 95% confidence intervals are shown. A larger version of this figure showing training loss and validation loss as well is provided in Appendix E.3.
pool (Figure 5). Comparing analytical optimizers, we observe a general inverse relationship between optimization performance and optimization stability: in order to achieve better optimization, an optimizer typically sacrifices some optimization stability in order to move faster through the optimizee weight space. By integrating problem- problem-specific knowledge, the amalgamated optimizer is able to combine the best optimization performance and optimization stability characteristics (Figure 4).
5.2
STABILITY-AWAI AWARE OPTIMIZER AMALGAMATION
Input Perturbation While we also tested perturbing the inputs of the optimizer during amalgama- tion, we were unable improve stability. These experiments are included in Appendix E.4.
Random Perturbation Min-max amalgamation was trained on the small optimizer pool with random perturbation relative magnitudes of ε (5 x 10 10 2 10 10-3.5 10-3 10 10-2).
E = 10
was also tested, but all replicates tested diverged and are not reported here.
Comparing perturbed amalgamation against the non- perturbed baseline (5 = 0), we observe that perturbations increase meta eeta-stability up about 10-3 (Figure 6).
Advemsraa
0.8
Rondom
For larger perturbation magnitudes, meta-stability begins to decrease as the perturbation magnitude overwhelms the weight "signal." eventually causing the training process to completely collapse for larger perturbation values. While the stability with random perturbation than 10-3, this is likely due to random chance, since we small sample size of 8 replicates.
0.6
10-2 is better
0.1
1x10 2x10-15110 1x10-> 2010 solo 1xl0 0.3
Magnitat
Adversarial Perturbation Since adversarial perturba- tion more computationally expensive than random per- turbations, min-max amalgamation was tested coarser grid of relative magnitudes to adversarial attack depth of step. These results are also reported in Figure 6, with = 10 omitted since all replicates diverged during training.
Figure 6: Amalgamation meta-stability for varying magnitudes of random and adver- sarial perturbations (lower better). Meta- stability is measured by the variance across replicates of the training loss after 25 epochs the Train convolutional network, adjusted for the variance of evaluation.
(10-1 10-$, 10-2), and
From our results, we observe that adversarial perturbations are about as effective as random pertur- bations. We also observe that the maximum perturbation magnitude that the amalgamation process can tolerate is much smaller for adversarial perturbations compared to random perturbations, likely because adversarial perturbations much "stronger." Due to the significantly larger training of adversarial perturbations, we recommend random perturbations for future work.
Application to Other Methods Random and Adversarial perturbations can be applied to any gradient-based optimizer meta-training method, including all of our baselines. An experiment applying Gaussian perturbations to the RNNProp baseline can be found Appendix E.5.
CONCLUSION
We define the problem of optimizer amalgamation, which we hope can inspire better and faster optimizers for researchers and practitioners. In this paper, we provide procedure for optimizer amalgamation, including differentiable optimizer amalgamation mechanisms and amalgamation stability techniques. Then, evaluate our problem different datasets, architectures, and training settings to benchmark the strengths and weaknesses of our amalgamated optimizer. In the future, we hope to bring improve the generalizability of amalgamated optimizers even more distant problems.

---

Published as a conference paper at ICLR 2022
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Oct 2016. doi: 10.1145/297674.29788181 URL http: doi www.dddi..o...ooo.oooo wtt.o//g///...........
2978318.
Marein Andrychowiez, Misha Denil, Sergio Gomez, Matthew Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in neural information processing systems, 2016.
Atilim Gunes Baydin, Robert Comish, David Martinez Rubio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. arXiv preprint arXiv:1703.0478. 2017.
Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoe V. Le. Neural optimizer search with reinforce- ment learning, 2017.
Cristian Bucilua, Rich Caruana, and Alexandru Niculescu-Mizii. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '06, pp. 535-541, New York. NY, USA. 2006. Association for Computing Machinery. ISBN 1595933395, doi: 10.1145/115040..1 15544001.1..0.44... 150464. 1150402. 1150464.
URL https: org/o..00.....oooooooooooo 1145/
Yue Cao, Tianlong Chen, Zhangyang Wang, and Yang Shen. Learning to optimize in swarms, In Advances in Neural Information Processing Systems, pp. 15018-15028, 2019.
Tianlong Chen, Weiyi Zhang, Jingyang Zhou, Shiyu Chang, Sijia Liu, Lisa Amini, and Zhangyang Wang. Training stronger baselines for learning to optimize. arXiv preprint arXiv:2010.094: 10.0909, 2020a.
Tianlong Chen, Xiaohan Chen, Wuyang Chen, Howard Heaton, Jialin Liu, Zhangyang Wang, and Wotao Yin. Learning to optimize: A primer and a benchmark, 2021a.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Robust overfitting may be mitigated by properly learned smoothening. In International Conference on Learning Representations, 2021b.
Wuyang Chen, Zhiding Yu, Zhangyang Wang, and Animashree Anandkumar. Automated synthetic- to-real generalization. In International Conference on Machine Learning, pp. 7746-1756. PMLR, 2020b.
Xiangning Chen and Cho-Jui Hsieh. Stabilizing differentiable architecture search via perturbation- based regularization. CoRR, abs/2002.05283, 2020. URL 05283.
http://awwiv..../aa/////a//o/a/t wot.//rgaaaaao ab/gg//bb./....../////// 2002.
Xuxi Chen, Wuyang Chen, Tianlong Chen, Ye Yuan, Chen Gong, Kewei Chen, and Zhangyang Wang. Self-pu: Self boosted and calibrated positive-unlabeled training. In International Conference on Machine Learning, pp. 1510-1519. PMLR, 2020c.
Yutian Chen, Matthew W Hoffmman Sergio Gómez Colmenarejo, Misha Denil, Timothy P Lillicrap. Matt Botvinick, and Nando De Freitas. Learning to learn without gradient descent by gradient descent. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp- 748-756. JMLR. org, 2017.
Jeremy M Cohen, Elan Rosenfeld, and J. Zico Kolter. Certified adversarial robustness via randomized smoothing, 2019.
Christopher De Sa, Matthew Feldman, Christopher Ré, and Kunle Olukotun. Understanding and optimizing asynchronous low-precision stochastic gradient descent. In Proceedings of the 44th Annual International Symposium on Computer Architecture, pp. 561-574, 2017.
Olivier Devolder et al. Stochastic first order methods smooth convex optimization. Technical report, CORE, 2011.
10

---

Published as a conference paper at ICLR 2022
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- leatalg for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp.
1126-1135. PMLR, 06-11 Aug 2017. URL https: ptttps://prooee prrroceeiing.... ml www.prrsss/vee vv00/ finn17 1an. html.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research, 17(1):2096-200 2016.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little near convergence. arXiv preprint arXiv: 13277, 2019.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. 2014.
Eduard Gorbunov, Marina Danilova, and Alexander Gasnikov. Stochastic optimization with heavy- tailed noise via accelerated gradient clipping. arXiv preprint arXiv:20055.0788. 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR,
2015. URL http:/// aaaaiaiivvvv.//..//aa /abs///////// 111./.111100 03385.
Zhezhi He, Adnan Siraj Rakin, and Deliang Fan. Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 588 188-597,7 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253: 2015.
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren (eds.). Automated Machine Learning: Methods, Systems, Challenges. Springer, 2018. In press, available at http://uttollorg/hoook
Prateek Jain and Purushottam kar. Non-convex optimization for machine learning. arXiv preprint arXiv: 2017.
Haoming Jiang. Zhehui Chen, Yuyang Shi, Bo Dai, and Tuo Zhao. Learning to defense by learning to attack. arXiv preprint arXiv: 1.01213, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http: httt:://yana /yann.
lecun. com/ exdb haadb/mnii mmm/s//.../////////m//mmm/..mmmm
Mathias Lecuyer, Vaggelis Atlidakis. Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy, 2019.
Kevin Leyton-Breet Brown, Eugene Nudelman, Galen Andrew, Jim McFadden, Yoav Shoham, al. A portfolio approach to algorithm selection. In IJCAI, volume 3, pp. 1542-1543, 2003.
Chaojian Li, Tianlong Chen, Haoran You, Zhangyang Wang, and Yingyan Lin. Halo: Hardware- learning to optimize. In European Conference on Computer Vision pp. 500-518. Springer, 2020,
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond, 2020.
Sihui Luo, Xinchao Wang. Gongfan Fang, Yao Hu, Dapeng Tao, and Mingli Song. Knowl- edge amalgamation from heterogencous networks by common feature learning. arXiv preprint arXiv:1906.1055 2019.
Kaifeng Lv, Shunhua Jiang, and Jian Li. Learning gradient descent: Better generalization and longer horizons. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2247-2255. JMLR. org, 2017.

---

Published as a conference paper at ICLR 2022
Haoyu Ma, Tianlong Chen, Ting-Kuei Hu, Chenyu You, Xiaohui Xie, and Zhangyang Wang. Good students play big lottery better. arXiv preprint arXiv:2101.03255.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706. 06080, 2017.
Luke Metz, Niru Maheswaranathan, Jeremy Nixon, Daniel Freeman, and Jascha Sohl-Dickstein. Understanding and correcting pathologies in the training of learned optimizers. In International Conference on Machine Learning, pp. 4566-4565. PMLR, 2019.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 11(8):197911993, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.0223 02099, 2018
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets, 2015.
Robin M. Schmidt, Frank Schneider, and Philipp Hennig. Descending through crowded valley- benchmarking deep learning optimizers. CoRR, abs/2007.01547, 2020. URL https: //axxiv.//aaxv...
axg//bs/2001 2007 01547.
Chengchao Shen, Xinchao Wang. Jie Song. Li Sun, and Mingli Song. Amalgamating knowledge towards comprehensive classification, 2018.
Chengchao Shen, Xinchao Wang, Jie Song. Li Sun. and Mingli Song. Amalgamating knowl- edge towards comprehensive classification. Proceedings of the AAAI Conference on Artifi- cial Intelligence, 33(01))3068-3000 3075, Jul. 2019a. doi: https: jojs aaai. org/ index phtpppppp//a//pppppppppppppppp/p /A/.AAAAA//aatat http:////aaatiee//eeee////////// 4.65.
URL
Chengchao Shen, Mengqi Xue, Xinchao Wang, Jie Song. Li Sun, and Mingli Song. Customizing stu- dent networks from heterogeneous teachers via adaptive knowledge amalgamation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019b.
Jiayi Shen, Xiaohan Chen, Howard Heaton, Tianlong Chen, Jialin Liu, Wotao Yin, and Zhangyang Wang. Learning a minimax optimizer: pilot study. In International Conference on Learning Representations, 2021. URL https::/p//o //wp/r/eeeeeeeeeee//e/eeee new.eiiwwwweeeeeeeeeeeee...e.eee forum www.:ommmddnnIiIIoooo....
Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis stochastic gradient noise in deep neural networks. In International Conference Machine Learning, pp. 5827-5837. PMLR, 2019.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties neural networks. arXiv preprint arXiv:1312.6199 2013.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consittnccy targets improve remi-supervised deep learning results. arXiv preprint arXiv: 2017.
Hui Wang, Hanbin Zhao, Xi Li, and Xu Tan. Progressive blockwise knowledge distillation for neural network acceleration. In Proceedings of the Twenry-Sevene International Joins Conference on Artificial Intelligence, IJCAI-18, pp. 2769-2775. International Joint Conferences Artificial Intelligence Organization, 2018. doi: 10.24963/jcca..01 2018/////...21//84444 10. 22.224463382 jjai 2018//844
URL //oii /rrg/
12

---

Published as a conference paper at ICLR 2022
Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo. Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize. In Proceedings of the 34th International Conference Machine Learning. 2017.
D.H. Wolpert and W.G. Maeready. No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1(1):67-82, 1997. doi: 10.1 00.1109/4335588888
Dongxian Wu, Shu-Tar Tho Xia, and Yisen Wang. Adversarial weight perturbation helps robust general- ization. Advances in Neural Information Processing Systems, 33, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. 08 2017.
Yuanhao Xiong and Cho-Jui Hsieh. Improved adversarial training via learned optimizer, 2020.
Lin Xu, Frank Hutter, Holger H Hoos, and Kevin Leyton- Brown. Satzilla: portfolio-basee based algorithm selection for sat. Journal of artificial intelligence research, 32:565-606 2008.
Chenglin Yang, Lingxi Xie, Chi Su, and Alan L. Yuille. Snapshot distillation: Teacher-student optimization in one generation, 2018.
Jingwen Ye, Yixin Ji, Xinchao Wang, Kairi Ou, Dapeng Tao, and Mingli Song. Student becoming the master: Knowledge amalgamation for joint scene parsing, depth estimation, and more. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2829-2838, 2019.
Jingwen Ye, Yixin Ji, Xinchao Wang, Xin Gao, and Mingli Song. Data-free knowledge amalgamation via group-stack dual-gan. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12516-12525, 2020a.
Jingwen Ye, Yixin Ji, Xinchao Wang. Xin Gao, and Mingli Song. Data-free knowledge amalgamation via group-stack dual-gan, 2020b.
Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. L2-gen: Layer-wise and learned efficient training of graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2127-2135, 2020.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation label smoothing regularization. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 3903-3911, 2020.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your teacher: Improve the performance of convolutional neural networks via self distillation, 2019a.
Michael R. Zhang, James Lucas, Geoffrey E. Hinton, and Jimmy Ba. Lookahead optimizer: k steps forward, 1 step back. CoRR, abs//197,00860, 2019b. URL 08610.
Stephan Zheng, Yang Song, Thomas Leung, and lan Goodfellow. Improving the robustness of deep neural networks via stability training. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 4480-4488, 2016.
13

---

Published as a conference paper at ICLR 2022
A ALGORITHMS
In this section, we provide a detailed description of the key algorithms used in our paper.
Truncated Back-propagatton: Algorithm shows truncated back-propagation applied to optimizer amalgamation. For an unrolling length N, .V data points (batches, in the case of mini-batch SGD) are sampled, which are split into N/t truncations of length t. Note that this requires N to be divisible by t; in our implementation, we require and N/t to be specified as integers. For each truncation, the optimizee and teachers are trained for t iterations, and meta-grann vetaaggradiens are computed over that truncation and applied.
Adversarial Weight Perturbation: Algorithm 2 shows adversarial perturbations applied to opti- mizer amalgamation. For each adversarial attack step, eeta-gradients are taken with respect to the parameters, and are normalized for each tensor with respect to its tensor norm before being applied as an adversarial perturbation.
Algorithm 1: Distillation by Truncated Back-propagaticn
Algorithm 2: Adversarial Weight Perturba-
tion for Truncated Back paok-propagation
Inputs
Amalgamation loss La Policy P with parameters & Teacher policies = Ti...... T(r) Optimizee M, X, 0g Unrolling, truncation lengths N,
Inputs Truncated back-propagation
parameters La P. T. M, X.
No.Nt
Adversarial attack steps A
Outputs Updated policy parameters Φ
Outputs : Updated policy parameters é
Sample N data points and initialize optimizee parameters for
Sample N data points 21, 21,... &N from X.
( (T
(TTrr) 9
N/t do 4/u 5 for = 1, 2. A do Compute trajectories ((1-1)4ad) for
for i 1.2....N//
N/t do
for j = 1,.. do
(i -1)
P and T
Update optimizee for P:
c. C
0 0 P MM )
F)(i-1)tadd) 0 (G- + via)
Update optimizees for each teacher: for lo
for each weight tensor w do
1.... ( do
(())
( +
n+1 (Tk) Tk V.Mann ((T
l1a
(1) t v + o||0|1277
end end
end
end
Compute distillation loss:
C.
La
L; + La ())t::)) ((6-1)0:01 ②) Update using VL,
1) O (i- & + via) Update & using V
end
end
B OPTIMIZEE DETAILS
Table shows a summary training problems used. While all training performed on a 2-layer CNN MNIST, we evaluated our optimizer on 4 different datasets described in (B.1) and 5 different architectures (described in B.2). We also experiment with different training settings, which are described in B.3.
B. DATASETS
All datasets used classification datasets, with cross entropy used the training loss. The MNIST dataset (LeCun & Cortes, 2010) used during training; the other datasets are, from most to least similar, are:
14

---

Published as a conference paper at ICLR 2022
Table Summary Optimizee Specifications. Dataset, architecture, and training setting specifica- tions are given in sections 1. B.2, and B.3 respectively.
Optimizee Name
Dataset
Architecture
Parameters
Other Settings
Train
MNIST
2-layer CNN
18122
MLP
MNIST
2-layer MLP
15910
Wider
MNIST
2-layer CNN, 2x width
61834
Deeper
MNIST
6-layer CNN
72042
FMNIST
FMNIST
2- 2-aayer CNN 2-Yeyer CNN
18122
SVHN
SVHN
21290
CIFAR
CIFAR-I0
2-layer CNN, 2x width
68170
ResNet
CIFAR-I0
28-layer ResNet
372330
Small Batch MNIST-DP
MNIST
2-yer CNN
18122
Batch size
MNIST
2-1ayer CNN
18122
Differentially Private
FMNIST:
Fashion MNIST (Xiao et al., 2017). FMNIST is also drop-in replacement for MNIST with classes and 28x28 grayscale images. Unlike MNIST or KMNIST, it features images of clothing instead of handwritten characters.
SVHN: Street View House Numbers, cropped (Netzer et al., 2011). While SVHN also has 10 classes of numerical digits, the images are 32x32 RGB, and have significantly more noise than MNIST including "distraction digits" on each side.
CIFAR-10 (Krizhevsky et al., 2009): the least similar dataset. While CIFAR- 10 still has 10 classes and 32x32 RGB images, it has much higher noise and within-class diversity.
Sample images from these datasets shown in Figure 7. All datasets accessed using Tensor- Flow Datasets and have CC-BY 4.0 license.
3
4
0
4
2
7
(a) MNIST
(b) FMNIST (c) SVHN
(d) CIPAR-10
Figure 7: Dataset sample images.
B.2 ARCHITECTURES
The Train convolutional network (Table 2a) has one convolution layer with 16 3x3 filters and one convolution layer with 32 5x5 filters. Each convolution layer uses ReLU activation, has stride Ixl, and is followed by max pooling layer with size 2x2. Finally, fully connected softmax layer is used at the output.
The four architectures evaluated are:
1. MLP: a 2-layer MLP with 20 hidden units and sigmoid activation
2. Wider: modified version of Train with double width each layer (Table 2b)
3. Deeper: a deeper network with 5 convolutional layers instead of 2 (Table 2c), again using
ReLU activation and 1x1 stride
4. ResNet: a 28-lr ResNet (He et al., 2015) (Table 2d)
15

---

Published as a conference paper at ICLR 2022
(a) Train
(b) Wider
Layer
Size
Units
Layer
Size
Units
Conv
3x3
16
Conv
3x3
32
Max Pool
2x2
Max Pool
2x2
Conv
5x5
32
Conv
5x5
64
Max Pool
2x2
Max Pool
2x2
Dense
10
Dense
10
(d) ResNet
(c) Deeper
Block
Size
Units
Conv
3x3
16
Layer
Size
Units
Conv
3x3
64
Conv
3x3
16
Residual (4x)
3x3
64
Conv
3x3
32
Max Pool
2x2
Conv
3x3
32
Cony
3x3
128
Max Pool
2x2
Residual (4x)
3x3
128
Conv
3x3
64
Max Pool
2x2
Conv
3x3
64
Cony
3x3
256
Max Pool
2x2
Residual (4x)
3x3
256
Dense
10
Average Pool
Global
Dense
10
Table 2: Convolutional Optimizee Architectures, Note that for the 28-layer ResNet, each residual block consists of layers, adding up 28 convolutional layers total.
B.3 OPTIMIZEE TRAINING
During training, batch size of 128 is used except for the Small Batch evaluation, which has a batch size of 32. During training and evaluation, datasets are reshuffled each iteration.
To match the warmup process used in meta- meta-training, warmup is also applied during evaluation. The SGD learning rate is fixed 0.01, which is very conservative learning rate which does not optimize quickly, but is largely guaranteed to avoid divergent behavior.
For differentially private training, we implement differentially private SGD (Abadi al., 2016). In differentially private SGD, gradients are first clipped to a fixed l2 norm on per-sample basis; then, gaussian noise with standard deviation ae where experiments,
is added to the aggregated batch gradients. In clipping norm 1.0 and noise ratio are used as training sets order to simulate transfer from a non non- ooivariv dataset (MNIST) used for meta-training private dataset (KMNIST).
1.1. Both MNIST and KMNIST
C AMALGAMATION DETAILS
C.1 OPTIMIZER ARCHITECTURES
In this section, we provide the exact architecture specifications and hyperparameters of our amal- gamated optimizer along with other training details and training time. Our implementation is open source, and can be found here: http: com/ /wt./VIIAAGG-o-/ ielppiimiiiiii 1gga
C.1.1 RNNPROP ARCHITECTURE
For our amalgamation target, we use RNNProp architecture described by Lv et al. (2017). For each parameter on each time step, this architecture takes as inputs RMSProp update g/ Vū and Adam update fi/ / Vē using momentum (mn decay parameter 31 0.9 and variance (6) decay parameter 32 = 0.999, matching the values used for our analytical optimizers. These values pass through a 2-layer LSTM with tanh activation, sigmoid recurrent activation, and 20 units per layer. The output of this 2-layer LSTM is passed through final fully connected layer with tanh activation to produce a scalar final update for each parameter.
16

---

Published as a conference paper at ICLR 2022
1.1.2 CHOICE NETWORK ARCHITECTURE
Our Choice network for Optimal Choice Amalgamation modified RNNProp architecture. The update steps for each analytical optimizer are given as inputs to the same 2-layer LSTM used in RNNProp. Additionally, the current time step and tensor number of dimensions are provided, with the number of dimensions being encoded as one-hot vector.
Then, instead of directly using the output of fully connected layer as the update. LSTM output passes through fully connected layer with one output per optimizer in the pool. This fully connected layer has softmax activation, and is used as weights to combine the analytical optimizer updates.
C.2 OPTIMIZER POOT.
We consider six optimizers as teachers in this paper: Adam, RMSProp. SGD, Momentum, AddSign, and PowerSign. These optimizers are summarized in table
Joining the popular hand-crafted optimizers Adam, RM- SProp. SGD, and Momentum, AddSign and PowerSign two optimizers discovered by neural optimizer search (Bello et al., 2017). These two optimizers share the de- sign principle that update steps should be larger when the momentum and gradient are in agreement:
Table 3: Optimizer pool update rules; all updates include an additional learning rate hyperparameter.
Optimizer
Update Rule
SGD
g m
Momentum
AddSign x g(1 + sign(m) (igg(())))g)(()) PowerSign exp((sggn( sgggn))))
RMSProp
g/ V
Adam
m/ VV
AddSign
g + sign (w (m)))ng())) exxp(sign (m (s))
Here, represents the gradient, mn an exponential mov- ing average of the gradient. In order to AddSign and Powersign as teachers for gradient-basee based distillation, we modify them to be differentiable by replacing the sign function with a scaled tanh with magnitudes normalized by Vb:
PowerSign
sign m (gg tanh(m// tanh (vū)
(8)
By dividing by Vo, we provide a consistent magnitude to the tanh function so that sign agreement mechanism is not affected by overall gradient magnitudes.
For all optimizers, the momentum decay parameter is set to 81 = 0.9, the variance decay parameter set to Bg 0.999, and the learning rate multiplier is found by grid search the Train optimizee over grid of {5 x 10-4, x 10 10-3,2 3,2 10 ....1}.
C.3 ADDITIONAL TRAINING DETAILS
During amalgamation, we apply a number of techniques from previous Learning to Optimize literature in order boost training:
Curriculum Learning: We apply curriculum learning (Chen et al., 2020a) to progressively increase the unrolling steps across a maximum of stages with length 100, 200, 500, and 1000. During curriculum learning, checkpoints
saved and validated every 40 "meta-epochs," which refers to single optimizee trajectory trained with truncated back poo--propagttion.
Random Scaling: We apply random scaling (Lv et al., 2017) to reduce overfitting to the gradient magnitudes of the training problem. This random scaling is only applied the amalgamation target; amalgamation teachers receive "clean" (unscaled) gradients.
Warmup: Instead of initializing each training optimizee with random weights, we first apply 100 steps of SGD optimization networks. A SGD learning rate of 0.01 is used during this period, and was chosen to be very conservative all optimizees tested.
"warmup" to avoid the turbulent initial phase of optimizing neural
These techniques also applied all of our baselines, except for Random Scaling only applied to baselines using the RNNProp architecture since we find that it harms the performance other optimizer architectures.
17

---

Published as a conference paper at ICLR 2022
C.4 TRAINING COST
Table 4 provides a summary of the training costs for each amalgamation method and baseline are provided. For optimal choice amalgamation, this includes both training the optimal choice optimizer and amalgamation training. All values are reported as the mean across replicates.
Table 4: Amalgamation and baseline training times.
Method
Time (hours)
Method
Training Time (hours)
Mean
3.86
Random
5.27
Min-max
3.85
Adversarial
10.53
Choice (small)
5.28
RNNProp
2.39
Choice (large) 6.35
Stronger Baselines
3.56
All experiments were single nodes with 4x Nvidia 1080ti GPUs, providing us with meta- batch size of simultaneous optimizations. In order to replicate our results, GPUs with at least 11GB of memory required, though less memory can be used if the truncation length for truncated back- prokkppppagttinnn is reduced.
D STABILITY DEFINITIONS
In this section, we provide the mathematical definition and measurement details of evaluation stability, and optimization stability.
meta sttba-staiility,
D.1 META-STABILITY AND EVALUATION STARILITY
In order to quantify meta-stability and evaluation stability, we first summarize the performance of each evaluation using the best validation loss obtained and the training loss of the last epoch. Then, we model the best validation loss (val) and final training loss y (train) for replicate and evaluation with the linear mixed effect model kj
YJ = μl ag + Eaj,
where i is the true mean, a, are IID random variables representing the meta-stability of the amal- gamated optimizer, and Ei are IID random variables representing the evaluation stability of each replicate. The meta-stability and evaluation stability are then quantified by standard deviations and a..
D.2 OPTIMIZATION STABILITY
To measure optimization stability, we model the validation loss Ltg(() evaluation j as
epoch / for replicate and
= (t)
(10)
for smooth function (ty(t) which represents the behavior of the evaluation and random variable mij
(t) which captures the optimization stability; we assume that Mij IID with respect to
In order to estimate on, we first estimate (ty(t) by applying a Gaussian filter with standard deviation (epochs) and filter edge mode "nearest," and a(()) is calculated accordingly. Finally, o (19) is treated as a summary statistic for each evaluation, and the mixed effect model described previously
(Equation 9) is fit to obtain final confidence interval for the mean optimization stability.
18

---

Published as a conference paper at ICLR 2022
E ADDITIONAL RESULTS
E EVALUATION STABILITY
Table 5 summarizes the evaluation stability of analytical and amalgamated optimizers. All optimizers obtain similar evaluation stability, except for cases where an optimizer cannot reliably train the optimizee at all such as Momentum, AddSign, and PowerSign on the Deeper CNN. In these cases, the optimizer consistently learns a constant or random classifier, which results in very low variance and high "stability."
Table 5: Evaluation stability of analytical and amalgamated optimizers: all optimizers amalga- mated from the small pool, except for Optimal Choice Amalgamation on the large pool, which is abbreviated "large". A dash indicates cetimizer-probbennt pairs where optimization diverged.
Best Log Validation Loss
Problem
Adam
RMSSPrp
SGD
Monsentum
AddSign
PowerSign
Mean
Min-Max
Choice
Large
Triin
0.068
0.046
0.048
0.068
0.068
0.049
0.064
0.072
0.064
0.058
MI.P
0.044
0.046
0.027
0.045
0.022
0.039
0.052
0.048
0.046
0.042
Wider
0.000
0.123
0.046
0.056
0.072
0.039
0.071
0.064
0.063
0.035
Doeper
0.084
0,075
0.047
1.971
1.679
0.085
0,062
0,087
FMNIST
0.011
0.017
0.015
0.000
0.019
0.017
0.023
0.020
0.018
0.021
0.099
0.095
CIFAR-10
0.049
0.025
0.030
0.000
0.031
0.032
0. 0.040
0.044
0.031
0.021
ResNet
0.042
0.054
0.072
0.052
0.044
0.040
Small Batch
0.047
0.119
0.079
0.068
0.056
0.106
0.085
0.078
0.0G6
0.064
MP
0.065
0.055
0.154
0.151
0.289
0.229
0.075
0.076
0.070
0.089
E.2 LARGER EVALUATIONS
In order explore the limits of optimizer, evaluated the amalgamated optimizer with 52-layer ResNet (763882 parameters -40x the train network size), and the same 52-layer ResNet on CIFAR- 100 instead of CIFAR-10. These results compared to CIFAR-10 and CIFAR-10 on a 28-layer ResNet using Adam as a single baseline (Figure 8).
2-layer network
While our amalgamated optimizer has significant performance advantages on the shallow CIFAR-10 network in our original evaluations and achieves performance parity in the 28-layer ResNet, the amalgamated optimizer can no longer perform as well the oracle once we reach 52 layers and change to CIFAR-100.
Layer
Restit
oyer
lyer CFAN100
2.6
1:33
0.50
22-
-025
-32
0030
0
25
25
Spach
120ch
Epach
60cch
Figure 8: Amalgamated optimizer ablations on problems of increasing size relative the training problem.
E.3
ADDITIONAL PLOTS
In this section, we include plots providing alternate versions of Figures 2, 3, and in the main text which had some outliers cropped out in order to improve readability.
19

---

Published as a conference paper at ICLR 2022
2.0
Analgamated
Strongor Bardinn RAWPProp
3.5
Relettee Best Log
Scali
1.0
Orgiiaa
9.5
I
0.0
Trai
HLP
Wildr
Decper
PHNSS
SVAN
CIFAR
Reshet
Small Batch
NNIST DP
vatcaston Accuracy
0.00
H
-015
Deeper
SVIIN
KNIST-DP
Figure 9: Uncropped version of Figure 2. Poor performance of "Scale" and "Original" on small batch and differentially private training differences between the best performers (Amalgamated and Stronger Baselines) unreadable. version showing best validation accuracy also included (higher better); the accuracy results largely preserve relative differences between methods, and lead to the same conclusion.
2.0
1005
2.5
2.0
Train
MLP
Wider
Deeper
FRNSST
SWHNN
DFF DFAA-100
Reshat
Small Batch
MNIST-DP
Best Train L.00
T
Train
MPP
Wider
Deeper
FHNIST
SVHN
OFFR-10
Reshet
Small Batch
MNIST.D
Optmitstton
0.2
Train
MUP
Wider
Deeper
FMMIST
SVHN
OFAB-10
Buchal
Small Butch
ANIST-DP
Figure 10: Uncropped version Figure including similar plots for Training and Validation loss.
20

---

Published as a conference paper at ICLR 2022
Trah
HIP
dir
Denper
FHVVST
0.35
1390
1.00
1.140
0.92
Are ERTETT
1.71
60
10 D1
Small
0.91
2.33
Co50
2.83
0.555
64
0.42
0.83
25
10 35
10
15
Figure 11: An accuracy version of Figure 3 comparing the best Amalgamated Optimizer (blue) and the Oracle Optimizer (orange); the shaded area shows #2 standard deviations from the mean. The title of each plot corresponds to optimizee; full definitions similarly or better than the Oracle analytical optimizer on problems spanning a variety of training settings, architectures, datasets, and has the largest advantage on more difficult problems such as CIFAR-10, ResNet, and MNIST-DP.
be found in Appendix B. The amalgamated optimizer performs
E.4 INPUT PERTURBATION
When applying input perturbations, we perturb the inputs to the optimizer, or the optimizee gradients, instead of the optimizer weights:
01t1 = 0,
+ +N(,0
(11)
We tested magnitudes 10-1 and 10
smaller experiment size replicates using Choice amal- gamation the small pool bascline; these results are given in Table Many experiment variants remain relating input noise such as adding noise proportional to parameter norm or gradient norm or trying smaller mag- of future nitudes of noise, and this may be potential study. However, we believe that input noise is generally helpful optimizer amalgamation, and did not study it further.
Table 6: Meta-Stability with varying
magnitudes of Input Perturbation
Magnitude
Meta-stability
0.104
10 0.485
a = 10-1
1.637
E.5 BASELINES WITH RANDOM PERTURBATION
Our perturbation methods can be applied to any gradient-based optimizer meta-training method, including all of our baselines. To demonstrate this application, trained 8 RNNProp replicates with Gaussian perturbations with magnitude 1 x 10-4; all other settings were identical to the RNNProp baseline. With perturbations, the RNNProp baseline is significantly improved, though not enough to match the performance of our amalgamation method.
0.3
I Amalgamated
RNNProp Saussisr
I
2.0
MLP
Deeper
FMNIST
SWiN
Buditll
Figure 12: Comparison of replicates amalgamated from the Large pool using Choice amalgamation with RNNProp baseline with Gaussian perturbations, with magnitude 1 10 With perturbations, the RNNProp baseline significantly improved, though enough match the performance of our amalgamation method.
21


Reviews:
[{"review_id": "s0Y8Bzhws6-", "reviewer": "Reviewer_kuqL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- This paper is mostly clear and organized, except for some points mentioned in Weaknesses below.\n- The paper made an interesting attempt to distill the knowledge from analytical optimizers with three amalgamation methods and conducted comprehensive experiments to show the effectiveness of the amalgamated optimizer.\n\n\nWeaknesses:\n- There are some places that the paper did not describe the content clearly, especially for the definition of stability. When to use $Y_{ij}^\\text{val}$ and $Y_{ij}^\\text{train}$ in Eq. (9) was not discussed and thus it is confusing to me how to derive the meta-stability and evaluation stability.\n- In terms of both the validation loss and the validation accuracy, the amalgamated optimizer did not have obvious advantages over the oracle optimizer, as shown in Figure 4 and Figure 9, which makes the method not very practical.\n- Although the authors claimed that they tried to distill the knowledge from traditional optimizers, it is also related to imitation learning. However, there was no discussion about the different between knowledge distillation and imitation learning when trying to learn the optimization pattern from analytical optimizers. Some imitation baselines might be incorporated into the meta-loss as well.\n- I am wondering whether first initializing the learned optimizer via the proposed amalgamation methods or imitation learning methods and then training it with the meta loss will have better performance. The reason is that randomly initialized optimizer is hard to train at the beginning and might disturb the whole training trajectory.\n- It seems that adversarial perturbation performed similarly as random perturbation. Since adversarial perturbation requires more computing resources, then why not just use random perturbation? Are there any other advantages of adversarial perturbation?\n- I am also interested in the performance of the choice optimizer C. Maybe the authors can expand on it, such as comparing C with the analytical optimizers it learns from. \n- I don't quite understand the choice of l2-log loss. Is there any related paper using this type of distillation loss?  Maybe the authors can try the gradient matching loss in [1], which is shown to be effective for dataset distillation and works with a similar number of parameters.\n- The time and memory cost for training the amalgamated optimizer is not reported, which is important for the practicality of the method. \n\n[1] Zhao, B., Mopuri, K. R., & Bilen, H. (2020). Dataset condensation with gradient matching. https://openreview.net/forum?id=mSAKhLYLSsl\n", "summary_of_the_paper": "This paper proposed a new problem called Optimizer Amalgamation and made an attempt to obtain a more powerful learned optimizer from several analytical optimizers. More concretely, three amalgamation losses are designed to train the amalgamated optimizer. In the meanwhile, two types of noise, random gaussian perturbation and projected gradient descent are incorporated into the training objective to increase the stability of the optimizer. The evaluation part compared the amalgamated optimizer with its original teachers, i.e., those analytical optimizers, and some L2O baselines on image classification tasks.", "summary_of_the_review": "Generally, I think the paper is marginally below the acceptance threshold. Although optimzer amalgamation is interesting, it seems expensive to train such an optimizer based on my experience and the authors did not report any time/memory complexity. Besides, the performance is not improved obviously compared with the best analytical optimizer, and the paper lacks the detailed discussion about imitation learning.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KSVd64kaj7", "reviewer": "Reviewer_ZiUc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths \n* The problem is well motivated as a solution to choosing from the large pool of possible optimizers. It naturally follows from prior work on knowledge distillation and amalgamation to instead distill from multiple optimizers. \n* Empirically, optimizer amalgamation performs better (and sometimes significantly so) than baselines from learning to optimize. It also performs favorably to the best choice of analytic optimizer.\n* Paper is clearly written and easy to follow.\n\nWeaknesses\n* It is a bit unclear how well an amalgamated optimizer can generalize to various settings -- do we need to amalgamate for each specific problem?\n* I have some concerns with the experiments discussed in the questions below.\n* Overall, the experiments are conducted in relatively small settings. It would be useful to know whether an optimizer learned with amalgamation could be used in a large experiments. Such experiments do not need to be extensive, but would improve the significance of the paper.\n\nQuestions\n1. It would be interesting to explore further the choice of optimizer pool and its effects on performance. This can be done by choosing subsets of the six optimizers you consider. SGD and Adam would be especially interesting given their popularity.\n2. In the case of mean amalgamation, do you observe the phenomena of the optimizer sticking to one of the teachers?\n3. Can the weight space perturbations be applied to the baselines to improve their stability?\n\nMinor\n* reference under knowledge distillation is undefined", "summary_of_the_paper": "This paper discusses the problem of selecting a neural network optimizer from a pool of possible optimizers. They propose three variants of a meta-algorithm which combines optimizers from the pool, whereby a differentiable meta-loss is defined on the training loss achieved by selection protocol. They show that their algorithm, equipped with weight space training noise leads to better performance on a variety of problems.", "summary_of_the_review": "I think this paper is marginally above the acceptance threshold. The algorithm proposed is well motivated for a specific task, and the authors conduct experiments with various ablations to their methods. I have some concerns listed in the questions, but overall believe this to be a good first step in tackling the problem.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8JZNHz5saUD", "reviewer": "Reviewer_h9GD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n=\n\n1. The paper is generally well-written and easy to follow. \n\n2. The proposed optimizer amalgamation method is interesting and effective. \n\n3. The experiment is comprehensive and supports the main claim of the paper. \n\n\n\nWeakness\n=\nSelecting an appropriate optimizer for a given problem is not a new research topic, and many papers exist in the areas of algorithm selection, algorithm portfolio, and meta-learning. I think one weakness of the paper is that it lacks the discussion about the related work, and how the paper can be better positioned in the literature. \n", "summary_of_the_paper": "This paper presents a new optimizer amalgamation method to combine a pool of optimizers into one in order to achieve stronger problem-specific performance. Three differentiable amalgamation mechanisms are designed and stabilization methods are explored. The proposed method is empirically shown to be effective when compared to a large number of baselines. \n\n", "summary_of_the_review": "Although the research topic is old, the proposed optimizer amalgamation method looks interesting and the experimental results verified the effectiveness of the proposed method. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "64HOm4Iq-c1", "reviewer": "Reviewer_Zbiy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1. The idea of distilling from multiple \"teacher\" optimizer is novel although distilling from one \"teacher\" optimizer has been proposed before as imitation learning in L2O. \n\n2. The three mechanisms proposed to distill from several \"teachers\" are interesting, especially the optimal choice amalgamation, which could shed light some further improvements for L2O.\n\n3. The use of random and adversarial pertubation on weights to improve robustness is a nice technique to help with the stability that is a big issue in L2O. \n\nWeakness:\n\n1. The comparison of the amalgamated optimizer with the analytical optimizers are interesting, but the advantage of the analytical optimizers is that they tend to work reasonably well for a wide range of problem or model sizes. Since the amalgamated optimizer is still learned on specific settings, I wonder how far it could generalize and when would such generalization break? For example, if the model size or the training steps is 10X or 100X larger, would the learned optimizer still perform well? These could shed some light on the limitation of current approaches and inspire future works. \n\n2. The \"oracle optimizer\" is using the best validation loss among the analytical optimizer's in the pool, but a stronger oracle might be the optimal choice optimizer that uses the trained LSTM to pick which optimizer to use at each step. It would help to add this stronger baseline into the analysis and comparison in the experiments since it can help understand whether the bottleneck is in the optimal choice LSTM or the distillation process. \n\n3. It might help to add more details about the Optimal Choice Amalgamation to help with reproducibility. For example, \"the LSTM takes the outputs of each optimizer in the pool, the layer type, and time step as inputs\", I wish there are more descriptions (maybe in the supplementary material) about the details such as how the inputs are encoded and how often are the LSTM updated to help with reproducibility. It would also be very helpful if the code can open sourced. \n\n4. It would be helpful to add some experiments to justify that it is necessary to distill from multiple optimizers instead of just one since the \"small\" setting just uses two optimizers anyway and the gain of the proposed approach from baseline might come more from the better stability in training due to perturbation. \n", "summary_of_the_paper": "This work presents an approach to distill several \"teacher\" optimizer into a \"student\" optimizer through learning to optimize (L2O). They compared three different approaches for doing such distillation, proposed to use random and adversarial perturbation to help with stability, and compared with the analytical optimizers that it distills from and the L2O baselines and showed superior performance.\n", "summary_of_the_review": "This work presents a novel approach called optimizer amalgamation that distills multiple \"teacher optimizer\" into a learned \"student\" optimizer. They compared three different approaches and proposed to use perturbation to improve stability in L2O. The result is promising compared to previous baselines, but it would be helpful if more analyses (see weakness section) can be done to understand the importance and necessity of different components and more details of the proposed approach can be released to help with reproducibility. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a meta-learning approach to "amalgamate" optimizers. The reviewers all found the idea interesting and unanimously found it to be acceptable for publication. In particular, I appreciate that the authors expanded their results to include more larger problems. One of the outstanding questions that would be interesting to address in future work is the use of tuned learning rate schedules.
Decision: Accept (Poster)