LEARNING  TO LEARN  WITH  BETTER  CONVERGENCE
Authors: Patrick H. Chen
Sashank Reddi
Sanjiv Kumar
Cho-Jui Hsieh
Venue:  (2020)

Abstract: We consider the learning to learn problem, where the goal is to leverage deeplearning  models  to  automatically  learn  (iterative)  optimization  algorithms  for training machine learning models. A natural way to tackle this problem is to replace the human-designed optimizer by an LSTM network and train the parameters on some simple optimization problems (Andrychowicz et al., 2016).  Despite their success compared to traditional optimizers such as SGD on a short horizon, theselearnt (meta-) optimizers suffer from two key deficiencies: they fail to converge(or can even diverge) on a longer horizon (e.g., 10000 steps). They also often fail to generalize to new tasks. To address the convergence problem, we rethink the architecture design of the meta-optimizer and develop an embarrassingly simple,yet powerful form of meta-optimizers—a coordinate-wise RNN model. We provide insights into the problems with the previous designs of each component and re-design our SimpleOptimizer to resolve those issues. Furthermore, we propose anew mechanism to allow information sharing between coordinates which enables the meta-optimizer to exploit second-order information with negligible overhead.With these designs, our proposed SimpleOptimizer outperforms previous meta-optimizers and can successfully converge to optimal solutions in the long run.Furthermore, our empirical results show that these benefits can be obtained with much smaller models compared to the previous ones.

Under review as a conference paper at ICLR 2020
LEARNING TO LEARN WITH BETTER CONVERGENCE
Anonymous authors Paper under double-blind review
ABSTRACT
We consider the learning to learn problem, where the goal is leverage deep learning models to automatically learn (iterative) optimization algorithms for training machine learning models. A natural way to tackle this problem is to replace the human- deenanddesigned optimizer by an LSTM network and train the parameters on some simple optimization problems (Andrychowiez et al., 2016). Despite their success compared to traditional optimizers such as SGD on a short horizon, these learnt (meta-) optimizers suffer from two key deficiencies: they fail to converge (or can even diverge) to generalize to new tasks. To address the convergence problem, we rethink the architecture design of the meta-optimizer and develop embarrassingly simple, yet powerful form of meta-optimizers- coordinate- wise RNN model. We provide insights into the problems with the previous designs of each component and design our SimpleOptimizer to resolve those issues. Furthermore, we propose a mechanism to allow information sharing between coordinates which enables the meta-optimizer to exploit second-order information with negligible overhead. With these designs, our proposed SimpleOptimizer outperforms previous meta- optimizers and can successfully converge to optimal solutions in the long run. Furthermore, our empirical results show that these benefits can be obtained with much smaller models compared to the previous ones.
longer horizon (e.g.. 10000 steps). They also often fail
INTRODUCTION
Optimization is an important problem for almost all tasks in machine learning. For optimizing non-convex problems that arise in machine learning, such as minimizing training loss of neural network, SGD is still the de-facto algorithm. However, success of SGD often hinges on careful selection of learning rate and its decay schedule. But finding such schedule for a new problem is laborious, cone-conu and computationally expensive. To resolve this issue, adaptive methods such as AdaGrad (Duchi et al.) 201T) and ADAM (Kingma & Ba 2014) were proposed. These methods adaptively re-scale the step-size and typically do not require such a learning rate schedule. However, they still require tuning of initial learning rate, and more importantly, can have worse generalization compared to SGD (Wilson et al. 2017 Berrada et al. 2018).
Recently, learning to learn paradigm (I.2L.) proposed tackle this problem. This approach attempts to eliminate the need for human desinnde optimization rules favor of a meta-learner capable of learning good optimization update strategy from experience (Lv al., 2017). Here, experience refers to other related optimization tasks. Recurrent neural network (RNN) is the common choice for the meta-optimizer since it can capture long term dependencies, which is of paramount importance in optimization. Recent results show that well-trained, LSTM-based meta-optimizers often outperform hand-tune approaches on supervised tasks (Andrychowicz 2016 Wichrowskal
et al. 2017).
Despite the promising initial results in this field, previous work on meta-optimizers often struggle to generalize to new problems. More importantly, these meta-optimizers usually cease to make progress (and can sometimes even diverge) after running for a large number of steps. To handle this issue, existing methods either add many pre-processing rules inspired by traditional optimizer (Lv et al., 2017) or employ very complicated architectures (Wichrowska et al. 2017). often leading to reproducibility issues. Furthermore, none of these methods clearly resolve the issue of generalization in a longer run. To make matters worse, the complex architectural design the existing meta- optimizers often hinders attempts understand the issue from technical angle.

---

Under review as a conference paper at ICLR 2020
In the light of these problems, rethink the architectural design of the meta-optimizer and de- sign simple yet clean form of meta-optimizers- coordinate- wise RNN model with embedding sharing. We then carefully analyze the problems with the previous design and fix those problems by re-designing its structure and loss, thereby, obtaining a new meta-optimizer with much better generalization properties. More specifically, our contributions are as follows:
We provide careful analysis of vanilla RNN-based meta- optimizer, and identify subtle issues related to the bias term in the meta-optimizer. We empirically show that by removing the bias term in RNN, long horizon convergence issues can be addressed.
Structure of each RNN:
Loss function:
We show the loss function used previous L2L frameworks tend to favor a myopia metaa-optimizer, one that quickly decreases the objective in short term but has slow long-term convergence. Inspired by the convergence rate of SGD, we propose a new loss based weighted difference, which helps the meta-optimizer explore better strategy in the later stage of optimization.
Information sharing betwen RNNs: Motivated by second-order optimization methods, we propose novel embedding sharing technique enable information exchanging between the RNN of different coordinates. This new technique successfully improves the convergence speed of meta-optimizers by sharing information across coordinates.
Fine-tuning scheme: We propose novel strategy to fine-tune the meta-optimizer. In this strategy, when given a new task, the learnt meta-optimizer can be quickly fine-tuned on a small subset of data. Experimental results show that this scheme works well in practice. This will enable the meta-optimizer to seamlessly generalize to a wider set of optimization problems, ones which it did not encounter during training.
With these design changes, proposed SimpleOptimizer converges much faster than the previous meta-optimizers and can successfully converge to optimal solutions in the long run. Furthermore, these benefits can be reaped at no additional cost of model complexity; in fact, the number of parameters in our model is smaller than the previous works. For instance, our optimizer (in Section 4) only has 132 parameters, while the previous meta-optimizers have at least 5000 parameters, with per-iteration speed 10 times slower than our method. This property is of great importance when meta-optimizer is applied large models.
RELATED WORK
The area of meta-learning has garnered significant interest recently (Brazdil et al. 2008), and includes
topics such as perammeeeet tuning (Golovin et al. [017]] Thornoon et | 2011)) and architecture
search (Zoph & Le [2066] Liu et al. 2018). We mainly focus on learning a meta optimize to optimize loss functions for training machine learning models. In this section, we briefly review current training and meta-learning algorithms that are most relevant to our work.
daptivv Gradient Methods Stochastic gradient descent (SGD) has been commonly used in training neural networks. At each step, SGD picks a batch of training samples, computes gradient of the batch and then conducts model updates. Several techniques have been proposed to improve SGD and eliminate the need to hand-tur its learning rate. This includes momentum updates (Sutskever cct al. 2013), Adagrad (Duchi et al.) 20IT), which uses an adaptive learning rate for each dimension, and ADAM (Kingma& & Ba 2014) which combines adaptive rules and momentum updates. There several recent discussions about Adam's convergence behavior and improved version of Adam (Reddi) al. 2018). Also, few recent works show that SGD with a carefully chosen learning rate can be competitive or even outperform Adam (e.g. Wilson et al. (017)).
Learning step size Some of meta-learning works aim to learn the learning rate schedule directly. In these works (Baydin et al. 2017 Wu et al. 2018 Li & Malik 2016), gradient information is used as input to meta-optimizer, but update direction is still aligned with gradient direction. In general, not all of these methods employ RNNs learning problem is framed as an MDP problem, which can then be naturally tied to reinforcement learning. While this line of research falls under the umbrella of meta-learning, our paper focuses
optimizt For example, in Li & Malik (2016), the
2

---

Under review as a conference paper at ICLR 2020
finding a meta-optimizer generating both learning rate as well as direction updates. Thus, do not include this line of research in our empirical comparisons.
Learning to learn (L2L) Instead of using some human- defined update rules optimizers, natural question to ask is whether it's possible to automatically learn the update rule. The seminal work by Cotter & Conwell (1990) argued that RNN can be used model optimization algorithms, and some earlier attempts have been made in Younger et al. (2001) | to use gradient descent to learn RNN optimizer on convex problems. Recently, due the difficulty of optimizing deep networks, the problem of learning meta-optimizers has become an important topic. (Andrychowicz et al. (2016) proposed to use L.TTM to model the update rule of a meta-optimizer, and showed that the learnt rules on simple problems can generalize to more complex problems. However, it is also reported in the paper that the learnt optimizer cannot generalize to long-term training-the algorithm cannot converge stationary points when running for longer horizon. To resolve this issue, Lv al. (2017) proposed several modifications, including randomly scaling the parameters of the model and pre-processing the gradients to mimic ADAM, and Linear Unit) (Clevert et al, 2015). Wichrowska et al. (0177) proposed multi-layer hierarchical LSTMs as the meta-optimizer, extending the input dimension by adding momentum, dynamic scaling and attention mechanism.
architecture based on ELU (Exponential
Unfortunately, even with the sophisticated design of meta-optimizer, the generalization issues (to more steps or various models/datseets) are not fully resolved. Lv et al. (2017) fails to generalize to longer than 10K steps and Wichrowska et al. (2017) fails generalize to deeper model such as GoogleNet. Although pre pro-pecesssing (e.g.. like equation equation 3) is claimed to be important in all the previous works. there is no clear reason given for that. Furthermore, the architecture in Wichrowska et al. (2017) is complicated and hard to validate. Most importantly, there is still very little understanding about the root cause of the unstable convergence behavior.
PROPOSED META OPTIMIZER
In this section, we first define the notation used in the paper and introduce the overall architecture of our meta-optimizer. Optimizee f is the task or loss function we want to minimize. At each time step, a random batch of training data is sampled and the optimizee model f will use the data to generate gradient, which is then fed as an input to meta-optimizze m. While the Optimizer m could be any model, in this paper, we restrict our attention to the case where m is a recurrent neural network (RNN). To make the optimizer generalizable to problems with various dimensions, the whole procedure is done in a coordinate-wise manner. Each coordinate of ff will maintain separate hidden state and share the same RNN parameters. Thus, the update rule of optimizing f can be written
oc+1 = 0₂ + (gt),
where O: denotes the parameters of optimizee model f at time step t, and gr is the gradient of based on the sampled batch of training data at step t. Training of RNN requires us to select a truncated Backpropagation Through Time (BPTT) steps. Within time horizon T, each hidden state of RNN will relate to future hidden state generation and it allows the model to capture longer dependencies in the optimizee task. The objective function of training the meta-optimizer for some horizon T steps can be written as:
L(m)= = 1=1
w. (0.))
(2)
Here wy's 's are arbitrary weights associated with each time-step. Our overall framework illustrated in Figure [ We defer the discussion on "extract information share" in Figure [ which is new structure for information sharing between coordinates, to Section 3.3.
3.1 STRUCTURE OF EACH RECURRENT UNIT
Previous design In the first L2L paper (Andrychowicz al. 2016), the authors proposed to LSTM for each recurrent unit. To stabilize the meta-optimizer's convergence, they further proposed pre-process gradient by the following function before feeding into LSTM:
log lgl sgg(g))
if [gl 2 p
f(g) 1 (-1,eP
(3)
else,
3

---

Under review as a conference paper at ICLR 2020
36.1
30,
AM
Update Model
Output Projection
()oes[(+1) Loss(t)
000
Concalonation
II Meta-Losg
Lossit-1)
(ss(()
Extract Share
8, Optinizee Model
Informsiion
DD
Cacrdinata-wise
hiddan state
RNN
RNN
Shared RNN
Grad eni impul
v9,
V
Figure 1: Illustration of the proposed SimpleOptimizer architecture.
where p is parameter controlling how small gradients disregarded. All followup works build upon similar LSTM preprocessing structure.
Our design Instead of simply using LSTM, take step back and study characteristics of the recurrent cell that leads to better convergence. To this end, is instructive to take the meta-optimizer m be vanilla RNN with only 2 hidden states to demonstrate the subtle problem with the bias term. In this setting, the update rule of equation becomes:
0 = 0, + tanh(w wig: gr + w2h, + b)
(4)
h++1 tanh(wig: + w2h4 b),
(5)
where wi, We are the RNN weights and is the bias term. As we optimize function f by changing 8, we will gradually get smaller loss and, thus, smaller gradients. When @ approaches a stationary point of the function f. in order converge, the update needs to be small enough to stay close to the stationary point. However, from the update rule as equation 4 Fand equation we can clearly that even when Ør is small, unless b equals to 0, the update rule will still produce an update with magnitude proportional to lan In the extreme case when 0 is the optimal point and when the previous hidden state is almost 0, this update rule will still pushes the parameter away from the optimal point. Thus, the bias term is problematic as optimization proceeds close to the optimal solution.
The same observation can be made even with the pre-processing methods equation For these methods, depending on the gradient magnitude, we could further write the update as:
0++1 + tanh(w11 log gell
wissign(ge) + w2he + b),
or Oc+1 0, + tanh(-lwn + w1ze + waht + b).
Notice the constant terms 1 or -1 in the equations. This, again, will be problematic when the corresponding parameters Wi1 and W12 are not O. Clearly, prepprccssiig involving the separation of the magnitude and the direction of gradients does not help to address the problem. This is an issue in practice as well. For instance, in one of our experiments, we notice that the maximal bias term of LSTM in Lv et al. (2017) is 0.41, and maximal bias term of LSTM in Andrychowicz et al. (2016) is 0.31; this, consequently, leads to bad performance. Meta-optimizers fail to learn bias= 0 since they are only trained in limited horizon and it is beneficial to "not" have bias term to be 0 to speed up initial stage optimization.
Following the analysis, conclude that the bias term in the meta-optimizer cause convergence issues. we remove the bias term in the RNN model, we hypothesize that we do not require any pre- prepprocessing to ensure that the model trains well and behaves properly when applied longer horizon. We empirically validate this statement in the later part of the paper (see Section
3.2 WEIGHTED DIFFERENCE OF OBJECTIVE FUNCTIONS
Another problem in previous leaming-to-learn algorithms is the design of loss function.
4

---

Under review as a conference paper at ICLR 2020
Previous loss function For the previous meta-optimizer's loss function, weights in equation are either set to or 0. For example, in Andrychowicz et al. (2016), wi is set to for all steps ((0)) f()))(
and this corresponds to requiring meta-optimizer to generate better loss for all T steps. In Lv et al. 2017), authors propose w, be for all except wy be for the final step. Thus, the loss only depends on f(Or), and when running for iT iterations the loss will be f(rr f(02)) +..+)++++--+----...... f(Brr).
We first discuss why the previous loss function leads to myopia meta-optimizer that has slower convergence in the long term. Andrychowicz et al. (20T6) the weights are uniform (w, = 1 for all t). Consider the following two sequence of loss (f(,,) under 6 BPTT steps:
loss of sequence :100, 20, 5, 1.25, 0.625
loss of sequence 2 :100, 2.5, 2.25, 2.125, 2.3125
in the uniform weighting scheme, the loss of sequence 1 is larger than sequence 2, so meta-optimizer that produces sequence is preferred. However, in terms of optimizing function for a longer run, clearly sequence is better choice. In Lv et al. 2017), the loss will be ((2r))++.... [(0r)) which will still face the same problem a sequence that reduces objective function faster in the beginning is preferred over sequence that decreases slower but could generalize to more steps.
Our proposed loss function Intuitively, we want to have higher weights for larger / to avoid meta-optimizer shortsightedly focusing initial stage of learning, but how up the weights? We propose new weighted loss motivated by the convergence properties of SGD. For a strongly convex problem, is known that SGD has an O(1/T) convergence rate, which means
f(0.) f(0") C/t,
where C' is a constant. However, in practice we do not know f(0*), and to avoid that we consider
f(0(-1) f(0.) = f(0))))))))))))))) C/(t f(00) C/t= -( -
This means that the difference between consecutive losses will decrease quadratically. To make all the updates equally important in the final loss function of meta-optimizer, should compensate the smaller difference of losses by giving a larger weight (c.g., Ott)) to the objective function reduction the 1-th step. The rate be changed due to different convergence properties of the meta-optimizer, but should always be an increasing sequence to make the weighted difference of losses become roughtly euqal.
Therefore, we use the following objective function to train meta-optimizer:
T
[(m) = Σ w. (((()-)))-(()))
(6)
where wi is an increasing function (0(?)) of i, which means that more weightage is given to the later stage of optimization. Without this weighting function, the meta-optimizer will tend to focus only the initial stage of optimization.
3.3 EMBEDDING SHARING BETWEEN COORDINATES
Previous design. A meta-optimizer uses a coordinate wise recurrent structure to adapt to problems with any dimension. Thus, for a d-dimensional problem, most of previous works consider meta- optimizers with d independent chains, and there is no information exchange between coordinates of the update. Notice that some prior works have already designed certain information exchange mechanism, but it's either in another line of research (Li & Malik 2016) too complicated to validate (Wichrowska et al. 2017). In our design, we will demonstrate a simple yet effective mechanism could be achieved.
Our design. In order to make a meta-optimizer learn to exploit second order information, it is necessary design mechanism to enable information sharing between those d RNN chains. We propose a novel embedding sharing mechanism to achieve this goal. Let hy..... ..... ha be the hidden states of those d RNN chains,
aggregator gather information from all these hidden states
5

---

Under review as a conference paper at ICLR 2020
into embedding vector, and then concatenate this new hidden state with the original hidden state of each RNN chain. This can be formulated as
hg t [a he, 3 Aggregate( ((h+)
Vt = l,..,dd
where a, B E R are two learnable variables shared by all the coordinates (in order to generalize to arbitrary dimension), and Aggregate is the aggregation function that gathers an arbitrary number of embeddings into a fixed sized vector. In general, any aggregation function can be applied. In our experiments we set it to be simple mean, which consistently yields better results than running d independent RNN chains. Each he will then be used to compute the update vector and meanwhile passed to the next iteration, so the global information can affect all the RNN chains. Furthermore, this aggregation mechanism only has two parameters a, 8 and is thus, easy to train. We show in the experiments that this simple information sharing mechanism leads to further improvement, enabling the meta-optimizer to outperform standard optimizers like SGD and ADAM.
3.4 FINE-TUNING
The current L.2L. frameworks train the meta-optimizer on one or set of "basic problems" and hope the optimizer can generalize to different types of neural networks Wichrowska et al. (2017). Empricially, it is observed that the learned optimizer does not work well on all types of networks. In addition, as neural network designs include more and more diverse types of architecture, it's difficult to define what subset of problems enough to cover all different training dynamics in real world problems.
To solve this problem we propose another scheme when the meta-optimizer is applied to new problems. We introduce a fast fine-tuning training step in order to let meta-optimizer slightly adjust the parameters. Each time when a new optimizee model or dataset comes, we firstly sub-sample 5% of the training data and then fine-tune the meta-optimizer on it. Since the sub-samples dataset is smaller than the full training set, the training speed would be much faster than the exploring on full dataset. At each iteration, meta-optimizer keeps taking input batches and update the optimizee for certain fixed steps until the average training loss is less than 1/2 of initial sampled training loss. We repeat this procedure for times and stop the warm-up training. This fined-tuned meta-optimizer is then applied to train the optimizee on the full dataset. We empirically show that this procedure does not make the meta-optimizer overfit to the sub-sampled small dataset and leads to better training loss on the full dataset. Note that fine-tuning introduces only ignorable overhead Besally--uuualy less than 1000 batches are used for fine-tuning and this enables much better generalization to 10K or more steps. Furthermore, empirically we found the meta-optimizer is not sensitive to the sub-sample rate for fine-tuning, On all the problems we tested, any numter between 3% and 5% leads to good results.
4 EXPERIMENTS
In all the experiments, we firstly train meta-optimizers on 2 layer MLP on MNIST dataset. The first layer maps the input into 500 dimensions and the second layer further projects it into logits of 10 classes. ReLu is used as the activation function in the neural network. We follow the literature to call this model base MIP. To demonstrate the effectiveness of the proposed method, we just use vanilla RNN with 10 hidden dimensions as the meta-optimizer. An additional linear layer of size 20x1 (10 for hidden state and 10 for aggregate vector) is applied in order to perform coordinate-wis. update. Most importantly, we disable bias term of both RNN and linear output. There is no further pre-processing, so the coordinate-we optimizer with Ir 1e-3 to train meta-optimizers. We provide code submission to implementation of Baseline MLP task and complete code will be released on submission to ensure the reproducibility.
gradient is directly fed into the vanilla RNN. We use ADAM
We compare the proposed method (SimpleOptimizer) to two baseline methods. The first method is the primitive meta-optimizer introduced in Andrychowiez et al. 2016). It contains layers of L.TTMs and the hidden dimension is set to 20. We call this model DMOptimizer. The other meta- meta-optimizer is RNNPROP Lv et al. (2017). adds ADAM-like pre-processing units and adopts cempled-scaling of optimizee functions in every training step. In Table we list the model sizes of the three meta- optimizers. Clearly our model is much smaller and simpler than the previous approaches, yet, as we shall see it is able to generalize better than the previous ones. Furthermore, due to smaller size, our model enjoys much faster (more than 10 times) computation time, which means limited overhead added over gradient computation for each step.

---

Under review as a conference paper at ICLR 2020
Table 1: Number of parameters and inference speed of three meta-optimiizrs.
SimpleOprimizer
DMOptimizer
RNNProp
# parameters
132
5301
6801
Time per step (ms)
21.88
37.02
Learned Optimzier Base MLP steos
Learned Opt on Base MLP 10000 steps
Simpleopbmbber
SimpleOptizzer
20
DW0pilnn
DMCptimizer
RNN/PROP
RMMPROP
<00 locs
4.5
0
25
50
75
100
125
150
175
200
2020
4503
6020
8020
15020
of Batches
(a) Update 200 batches.
(b) Update 10000 batches
Figure 2: Comparison to baseline models. Results show that baseline models still have convergence
issue over 10k steps. In contrast, SimpleOptimizer resolves the issue without complicated design.
In addition, also compare meta-optimizers with standard optimizer such as SGD and adaptive optimizer such as ADAM. When evaluating these methods for basic setup, we choose a fixed learning rate by searching for the best performing the best performance. When evaluating under the fine-tuning setup, we have both meta-optimizer and SGD/ADAM to train/search the best learning rate in the small sub-sampled dataset. The best setup found will then used in the final evaluation over whole dataset.
from the set {1, 0.1, 0.01, 0.001} and report
Comparison to Baseline Methods We first compare our SimpleOptimizer with the baselines. Following the same setting as the previous work Andrychowicz et al. (2016), train all meta- optimizers on base MLP with 20 BPTT steps and train on model for only 200 steps in total. Trained meta-optimizers evaluated the same setup except they applied in up 10,000 steps. The dataset used MNIST. Figure 2(a) we can observe that RNNPROP indeed improves over the vanilla L2L model, and leads a decent decrease loss over initial 200 steps; however, as shown in figure 2(b when applying it with significantly many steps (over 200 steps), it does not converge. On the other hand, results show that our approach leads very good convergence. The model keeps decreasing toward zero even after 10,000 batches of data.
Ablation Analysis of Proposed Method We conduct ablation analysis on the proposed SimpleOp- timizer to the effect of each proposed idea. Fach time, remove one proposed feature from the model and compare the performance. In Figure 3(a) we train an optimizer by adding back the bias term. We can observe that this model cannot approach better solution. Previous works seek to address this by having more complicated models (e.g. LSTMs) whereas we point out that the root cause of the problem the presence of the bias term. In Figure (b) we compare SimpleOptimizer versus the same architecture without aggregated information. We can observe that sharing information between coordinates leads to faster convergence. Also, by comparing SimpleOptimizer with and without weighted loss, we conclude that weighted loss is important for convergence.
Generalization to Convolutional Neural Network (CNN) In this experiment, we check whether the meta-optimizer trained on the base MIP could generalize its capability to another architecture. In particular, we want to see if our meta-optimizer (trained on base MLP) could perform well on a convolutional neural network, more specifically, LeNet. As shown in Figure (aa the proposed method can still optimize CNN very well, and it even outperforms well-tuned SGD. This could imply that given the same dataset, the meta-optimizer can be transferred to different structures relatively easily. We again observe that previous methods will eventually diverge while training over 10,000 steps. In particular, RNNPROP achieves significant progress in the early stage of the optimization, but the earlier pre-processing techniques do not help the meta-optimizer stay close

---

Under review as a conference paper at ICLR 2020
Comparison of Proposed Methods on Base NLP with MMST
Importance of Weighting and information sharing Base WLP wit MNIST
25
Simpletyommooweeiiee Simple@ypimiue nnightng Single@ptiiie--iieeee--- L2L. Biss
@mpploppimbeeeeee weigheigttn
Simpeeppimias Mareeigghing
Smmpeeptmmiee thhste--nly
lote
15
0.08
100
0.05
004
150
200
400
302
2000
2000
6020
ED0C
10200
Batches
Batches
(a) Importance of removing bias term.
(b) Importance of weighted-loss and information sharing.
Figure 3: Ablation amalysis of proposed SimpleOptimizer. (a) Adding back the bias term, we observe that meta-optimizer cannot converge. (b) With weighted difference of loss the training objective, meta-optinizer further improves. Adding aggregate information sharing also helps in convergence.
Generaliaation Lekket on MNIST dataset
General zation to LeVet on CIFAR-10
SimpleDyriiiier
Simp ecptiinisee fine tuned
SG0
Simp @Oplinizer
RANPROP
200
S0D,r-1e-
loee
1.5
DMoptimieer
ADAM.r=1e2
SED,
AD4N,r-Je3
1.6
05
1.0
2000
2020
5000
3502
10000
0
1000
2002
3100
4000
5000
6000
7000
* Baiches
# Batches
(a)
(b)
Figure 4: Generalization to different models and datasets. (a) Results of generalization to LeNet on MNIST dataset. (b) Results of generalization to LeNet on CIFAR-10 dataset.
the optimal solution. In the subsequent experiments, we focus on comparisons with standard gradient methods instead of meta-learning baselines.
Generalization to CIFAR-10 dataset Now we generalize the base meta meta-optimizer to a new model structure and dataset at the same time. Results are shown in the Figure 4(b) Noticeably, SimpleOp- timizer could achieve comparable results to not well-tuned traditional optimizer even without any tuning. This shows the good generalization capability our method. When the fast fine-tuning procedure is applied, SimpleOptimizer converges faster than well-tuned SGD ADAM.
Generalization Deeper MNST
Generalization to GoogleNie on CIF4R-10
Simp fine
1.2
0.30
Simp eOstimier
10
S60, r=le1
ADAMi==1e3
=223
8
0.25
02
1020
1200
3400
1603
1602
7002
7200
7400
920
2003
4103
6300
5002
10000
Batches
Batches
(a)
(b)
Figure 5: Generalization to deep models. (a) Experimental results of generalization to Deeper MLP on MNIST dataset. (b) Experimental results of generalization GoogleNet VI CIFAR- dataset.
8

---

Under review as a conference paper at ICLR 2020
Generalization to Deeper Models In previous experiments, all models tested are relatively small. This is important to demonstrate the generalization capability of SimpleOptimizer on larger models. In this experiment, aim to test the performance of SimpleOptimizer on larger models.
Firstly, we evaluate on layers MLP with 16 times more parameters. Results are shown in the Figure [ Surprisingly, SimpleOptimizer without any fine-tuning outperforms well-tuned traditional optimizers. This result implies that the model structure similar (in this case it exactly the same except larger), learned optimizer can be directly applied without any fine-tuning.
Secondly, we apply the learned SimpleOptimizer to GoogleNet inception VI, which is 250 times larger than base MIP. Results shown in Figure It again shows that SimpleOptimizer is able to optimize the new model, and after the fast fine-tuning procedure, it can converge faster. However, SimpleOptimizer only outperforms ADAM but not well-tuned SGD.
These results basically validate the generalization capability of our proposed SimpleOptimizer. In particular, SimpleOpimizer does not diverge when the training steps increased to 10k. In most cases we observe that SimpleOptimizer might not achieve the best performance in the early stage of the optimization, but eventually the loss drops faster than the traditional optimizers, demonstrating the non-myopic nature of our meta-optimizer. Also, fast fine-tuning works in all the cases to further improve the performance of the meta-optimizer. These experimental results validate the contributions of this work.
5 CONCLUSION
In this paper, we propose a simple yet effective meta-optimizer which can generalize to different structures and outperform well-tuned traditional optimizer. We analyze why the structure and loss used in previous meta-optimizers cannot lead to convergence in a long run, and propose simple fix for each of the components. Furthermore, propose novel information sharing scheme further improve the convergence.
REFERENCES
Marcin Andrychowiez, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, pp. 3981-3989, 2016.
Atilim Gunes Baydin, Robert Cornish, David Martinez Rubio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. arXiv preprint arXiv: 1703.0472, 2017.
Leonard Berrada, Andrew Zisserman, and M Pawan Kumar. Deep frank-wolfe for neural network optimization. arXiv preprint arXiv: 1.05599/ 2018.
Pavel Brazdil, Christophe Giraud Carrier, Carlos Soares, and Ricardo Vilalta. Metalearning: Applica- fions to data mining. Springer Science & Business Media, 2008.
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint urXiv:151 (00:::511.02200 1.0728,, 2015.
Neil E Cotter and Peter R Conwell. Fiigh- networks can learn. In 1990 IJCNN International Joint Conference on Neural Networks, pp. 553-559. IEEE, 1990.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12000):2121-2155: 201 I.
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D Sculley. Google vizier: service for black-box optimization. In Proceedings of the 23nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1487-1495. ACM, 2017.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Ke Li and Jitendra Malik. Learning optimize. arXiv preprint arXiv: 2016.

---

Under review as a conference paper at ICLR 2020
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search, arXiv preprint arXiv:1806.09000 09055, 2018.
Kaifeng Lv, Shunhua Jiang, and Jian Li, Learning gradient descent: Better generalization and longer horizons. arXiiv preprint 1XX3::1703.033333 2017.
Sashank Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. 2018.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance initialization and momentum in deep learning. In International conference on machine learning, pp. 1139-1147, 2013.
Chris Thornton, Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Auto-weka: Combined selection and hyperparameter optimization of classification algorithms. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 847-855. ACM, 2013.
Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize. arXiv preprint arXiv: 7003 2017.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stem, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems, pp. 4 4158-415, 2017.
Xiaoxia Wu, Rachel Ward, and Léon Bottou. Wngrad: Learn the learning rate in gradient descent.
arXiv preprint 1803vv1803.008000 2018.
Steven Younger, Sepp Hochreiter, and Peter R Conwell. Meta-learning with backpropagation. In Neural Networks, 2001. Proceedings. IJCNN'01. International Joint Conference on, volume 3. IEEE, 2001.
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. arXiv preprint 2016.
10


Reviews:
[{"review_id": "SJefgrxCKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors build on the 'learning to learn' work, that aims to leverage deep learning models with optimization algorithms, most commonly with recurrent networks.  The goal is to utilize meta-learners that can adapt the optimization update strategy based on data/experience.  The authors aim to tackle problems that often arise in such meta-optimization schemes, such as covergence and generalization problems.  The paper is overall well-written, and several experiments are presented\n\nBuilding on previous work, the authors propose some variations to the meta-learning schemes and architecture.  I comment on these below.\n\n- The authors remove the bias term based on the intuition that for the meta-learners, removing the bias can lead to better convergence.  This is supported by a set of experiments in Fig 3a (but with only one learning rate?).  This experiment shows an extraordinary difference in accuracy between including and not including the RNN bias.  I think, because this difference is substantial (along with papers that do use the bias in meta-learners showing good results overall), more evidence/ablations are required.  \n\nCan we can be sure that if the bias is non-zero (as in other reported works), we should expect a worse performance?.  Clearly this change seems to speed up the process, and in the example explained in the paper it makes sense, but could there be examples where lack of bias might lead to worse results?\n\n- The authors claim that one of the problems of learning-to-learn algorithms is the late-stage optimization strategy.  While previous works introduce a weighting for the meta-optimizer's loss function (usually binary), the authors extend this to continuous monotonically increasing functions in order to weight the late stage steps of the algorithm more.  The authors propose compensating small loss difference happening during late optimization with a larger weight.   This naturally brings the question of which function should be used, as showing that any function will do seems difficult in practice. Have the authors tried any functions that failed or led to overfitting in comparison to other works?  Could one argue that if the loss difference is not large then perhaps the network can be more prone to overfitting at least in some cases?\n\nFinally, the authors introduce embedding sharing between coordinates (aggregating hidden states), and also propose fine-tuning (sampling 5% of training data 5 times for warm-up).  Sharing hidden state information is expected to be useful (and has been often employed in literature), similarly to fine-tuning.  Also, 5% of dataset and 5 times seems to be an arbitrary choice - probably more related to the experimental section rather than the method itself.  \n\na question on fine-tuning:  if fine-tuning was used for the proposed method, have the compared methods also been pre-trained to provide fair comparisons?  This is particularly relevant to fig 4B where only the fine-tuned method slightly overperforms SGD.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyesVOJatr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper presents several improvements over the existing learning to learn models including Andrychowicz et al. (2016) and Lv et al. (2017). Specifically, this paper analyzes the issues in the original learning to learn paradigm (L2L), including instability during training and bias term issues in the RNN. It proposes a new loss based on weighted difference for improving the meta-optimizer in the later stage of optimization. It also proposes information sharing between RNNs for each coordinate. Finally it presents how to fine-tune the meta-optimizer on new task. \n\n\nPros:\nReasonable technical improvements to fix some issues of the learning to learn framework\n(1) reduce the number of parameters in the meta-optimizer significantly\n(2) improve the stability of the meta-optimizer.\n(3) improve the generalization to new tasks and datasets.\n\n\nCons\n1. The novelty is not good enough and the method does not seem to be solid enough. Except for the technique in the *structure of each recurrent unit\" section, other techniques are tricks that are hard to tell why they could work. That said, I think the experiments should verify each of the proposed components, and see their roles in the proposed method.\n\nAlso, it is claimed in the Abstract and the paper that the proposed method *successfully converge to optimal solutions\"... I think this claim is quite unprofessional. How do you know it converges the optima?\n\n\n2. With respect to the experiments:\n\nThe experiments are only done on the MNIST and CIFAR10 datasets, which are small-scale datasets. Current meta learning model has achieved advancement on more challenging datasets, e.g., Mini-Imagenet and CIFAR1000.\n\nFor the experiment section of comparison to baseline methods. It would be fair to compare all the methods with removing bias or not removing bias setting. As the author mentions that \"the maximal bias term of LSTM in Lv et al. (2017) is 0.41, and maximal bias term of LSTM in Andrychowicz et al. (2016) is 0.31; this, consequently, leads to bad performance\". It would be interesting to know that if the bias term of the two baseline model are removed, how is the performance difference compared to the method proposed by the authors? \n\nHow does the number of parameters of the meta optimizer scales with the problem size? That is, how does the number of parameters in the meta-optimizer grow with increasing the number of image classes in the problem at hand, e.g., 20 classes, 50 classes, etc?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Byg2F33lcH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work suggests a host of  improvements and simplifications to the meta-learning approach of Andrychowicz et. al.  The authors have carefully analyzed weaknesses in previous work and I think their experiments do suggest that they have improved on them.  However, I would still recommend rejecting, as imo\n1: the  absolute state of these approaches seems unpromising, and \n2: the authors do not do a good job contextualizing how well the learned optimization performs compared to more standard methods.\n Each step used to train the meta-optimizer could be used instead to pick hyperparameters (choice of which \"classical\" optimizer, and its hyper-parameters, like lr etc.).  However, there is not really any discussion about the trade-offs between the time it takes to train the meta-optimizer and the results that would be obtained by hyper-parameter search.   Indeed, even with the tiny hyper-parameter search spaces the authors use, in most of their figures, one of ADAM or plain SGD does comparably or better than their method. \n\nIt is not clear if the figures represent train or test loss; the authors should report both.  Furthermore, imo, almost the figures are cut off too soon- the models all still seem to be learning.    What are the error rates at the cutoffs on test and train?\n\nFinally, I wonder at the choice of test problems.  Why not pick something where one naturally will need to run an optimization many times (e.g. style transfer) rather than a toy problem like mnist or cifar?  Note that there are already other approaches (not learning based on learning gradient descent) that have been successful there. \n  \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an improved (over Andrychowicz et al) meta-optimizer that tries to to learn better strategies for training deep machine learning models. The paper was reviewed by three experts, two of whom recommend Weak Reject and one who recommends Reject. The reviewers identify a number of significant concerns, including degree of novelty and contribution, connections to previous work, completeness of experiments, and comparisons to baselines. In light of these reviews and since the authors have unfortunately not provided a response to them, we cannot recommend accepting the paper.
Decision: Reject