LEARNING  TO LEARN  WITH  BETTER  CONVERGENCE

1 INTRODUCTION . Optimization is an important problem for almost all tasks in machine learning . For optimizing non-convex problems that arise in machine learning , such as minimizing training loss of a neural network , SGD is still the de-facto algorithm . However , success of SGD often hinges on careful selection of learning rate and its decay schedule . But finding such a schedule for a new problem is laborious , time-consuming and computationally expensive . To resolve this issue , adaptive methods such as AdaGrad ( Duchi et al. , 2011 ) and ADAM ( Kingma & Ba , 2014 ) were proposed . These methods adaptively re-scale the step-size and typically do not require such a learning rate schedule . However , they still require tuning of initial learning rate , and more importantly , can have worse generalization compared to SGD ( Wilson et al. , 2017 ; Berrada et al. , 2018 ) . Recently , learning to learn paradigm ( L2L ) was proposed to tackle this problem . This approach attempts to eliminate the need for human-designed optimization rules in favor of a meta-learner capable of learning a good optimization update strategy from experience ( Lv et al. , 2017 ) . Here , experience refers to other related optimization tasks . Recurrent neural network ( RNN ) is the common choice for the meta-optimizer since it can capture long term dependencies , which is of paramount importance in optimization . Recent results show that a well-trained , LSTM-based meta-optimizers often outperform hand-tuned approaches on supervised tasks ( Andrychowicz et al. , 2016 ; Wichrowska et al. , 2017 ) . Despite the promising initial results in this field , previous work on meta-optimizers often struggle to generalize to new problems . More importantly , these meta-optimizers usually cease to make progress ( and can sometimes even diverge ) after running for a large number of steps . To handle this issue , existing methods either add many pre-processing rules inspired by a traditional optimizer ( Lv et al. , 2017 ) or employ very complicated architectures ( Wichrowska et al. , 2017 ) , often leading to reproducibility issues . Furthermore , none of these methods clearly resolve the issue of generalization in a longer run . To make matters worse , the complex architectural design of the existing metaoptimizers often hinders attempts to understand the issue from technical angle . In the light of these problems , we rethink the architectural design of the meta-optimizer and design a simple yet clean form of meta-optimizers—a coordinate-wise RNN model with embedding sharing . We then carefully analyze the problems with the previous design and fix those problems by re-designing its structure and loss , thereby , obtaining a new meta-optimizer with much better generalization properties . More specifically , our contributions are as follows : • Structure of each RNN : We provide a careful analysis of vanilla RNN-based metaoptimizer , and identify subtle issues related to the bias term in the meta-optimizer . We empirically show that by removing the bias term in RNN , long horizon convergence issues can be addressed . • Loss function : We show the loss function used in previous L2L frameworks tend to favor a myopia meta-optimizer , one that quickly decreases the objective in a short term but has slow long-term convergence . Inspired by the convergence rate of SGD , we propose a new loss based on weighted difference , which helps the meta-optimizer to explore a better strategy in the later stage of optimization . • Information sharing betwen RNNs : Motivated by second-order optimization methods , we propose a novel embedding sharing technique to enable information exchanging between the RNN of different coordinates . This new technique successfully improves the convergence speed of meta-optimizers by sharing information across coordinates . • Fine-tuning scheme : We propose a novel strategy to fine-tune the meta-optimizer . In this strategy , when given a new task , the learnt meta-optimizer can be quickly fine-tuned on a small subset of data . Experimental results show that this scheme works well in practice . This will enable the meta-optimizer to seamlessly generalize to a wider set of optimization problems , ones which it did not encounter during training . With these design changes , our proposed SimpleOptimizer converges much faster than the previous meta-optimizers and can successfully converge to optimal solutions in the long run . Furthermore , these benefits can be reaped at no additional cost of model complexity ; in fact , the number of parameters in our model is smaller than the previous works . For instance , our optimizer ( in Section 4 ) only has 132 parameters , while the previous meta-optimizers have at least 5000 parameters , with per-iteration speed 10 times slower than our method . This property is of great importance when meta-optimizer is applied to large models . 2 RELATED WORK . The area of meta-learning has garnered significant interest recently ( Brazdil et al. , 2008 ) , and includes topics such as hyper-parameter tuning ( Golovin et al. , 2017 ; Thornton et al. , 2013 ) and architecture search ( Zoph & Le , 2016 ; Liu et al. , 2018 ) . We mainly focus on learning a meta-optimizer to optimize loss functions for training machine learning models . In this section , we briefly review current training and meta-learning algorithms that are most relevant to our work . Adaptive Gradient Methods Stochastic gradient descent ( SGD ) has been commonly used in training neural networks . At each step , SGD picks a batch of training samples , computes gradient of the batch and then conducts model updates . Several techniques have been proposed to improve SGD and eliminate the need to hand-tune its learning rate . This includes momentum updates ( Sutskever et al. , 2013 ) , Adagrad ( Duchi et al. , 2011 ) , which uses an adaptive learning rate for each dimension , and ADAM ( Kingma & Ba , 2014 ) which combines adaptive rules and momentum updates . There are several recent discussions about Adam ’ s convergence behavior and improved version of Adam ( Reddi et al. , 2018 ) . Also , a few recent works show that SGD with a carefully chosen learning rate can be competitive or even outperform Adam ( e.g . Wilson et al . ( 2017 ) ) . Learning step size Some of meta-learning works aim to learn the learning rate schedule directly . In these works ( Baydin et al. , 2017 ; Wu et al. , 2018 ; Li & Malik , 2016 ) , gradient information is used as input to meta-optimizer , but update direction is still aligned with gradient direction . In general , not all of these methods employ RNNs as a meta-optimizer . For example , in Li & Malik ( 2016 ) , the learning problem is framed as an MDP problem , which can then be naturally tied to reinforcement learning . While this line of research falls under the umbrella of meta-learning , our paper focuses on finding a meta-optimizer generating both learning rate as well as direction updates . Thus , we do not include this line of research in our empirical comparisons . Learning to learn ( L2L ) Instead of using some human-defined update rules as optimizers , a natural question to ask is whether it ’ s possible to automatically learn the update rule . The seminal work by Cotter & Conwell ( 1990 ) argued that RNN can be used to model optimization algorithms , and some earlier attempts have been made in Younger et al . ( 2001 ) to use gradient descent to learn a RNN optimizer on convex problems . Recently , due to the difficulty of optimizing deep networks , the problem of learning meta-optimizers has become an important topic . Andrychowicz et al . ( 2016 ) proposed to use LSTM to model the update rule of a meta-optimizer , and showed that the learnt rules on simple problems can generalize to more complex problems . However , it is also reported in the paper that the learnt optimizer can not generalize to long-term training—the algorithm can not converge to stationary points when running for a longer horizon . To resolve this issue , Lv et al . ( 2017 ) proposed several modifications , including randomly scaling the parameters of the model and pre-processing the gradients to mimic ADAM , and a new architecture based on ELU ( Exponential Linear Unit ) ( Clevert et al. , 2015 ) . Wichrowska et al . ( 2017 ) proposed multi-layer hierarchical LSTMs as the meta-optimizer , extending the input dimension by adding momentum , dynamic scaling and attention mechanism . Unfortunately , even with the sophisticated design of meta-optimizer , the generalization issues ( to more steps or various models/datasets ) are not fully resolved . Lv et al . ( 2017 ) fails to generalize to longer than 10K steps and Wichrowska et al . ( 2017 ) fails to generalize to deeper model such as GoogleNet . Although pre-processing ( e.g. , like equation equation 3 ) is claimed to be important in all the previous works , there is no clear reason given for that . Furthermore , the architecture in Wichrowska et al . ( 2017 ) is complicated and hard to validate . Most importantly , there is still very little understanding about the root cause of the unstable convergence behavior . 3 PROPOSED META OPTIMIZER . In this section , we first define the notation used in the paper and introduce the overall architecture of our meta-optimizer . Optimizee f is the task or loss function we want to minimize . At each time step , a random batch of training data is sampled and the optimizee model f will use the data to generate a gradient , which is then fed as an input to a meta-optimizer m. While the Optimizer m could be any model , in this paper , we restrict our attention to the case wherem is a recurrent neural network ( RNN ) . To make the optimizer generalizable to problems with various dimensions , the whole procedure is done in a coordinate-wise manner . Each coordinate of f will maintain a separate hidden state and share the same RNN parameters . Thus , the update rule of optimizing f can be written as : θt+1 = θt +m ( gt ) , ( 1 ) where θt denotes the parameters of optimizee model f at time step t , and gt is the gradient of f based on the sampled batch of training data at step t. Training of RNN requires us to select a truncated Backpropagation Through Time ( BPTT ) steps . Within a time horizon T , each hidden state of RNN will relate to future hidden state generation and it allows the model to capture longer dependencies in the optimizee task . The objective function of training the meta-optimizer for some horizon T steps can be written as : L ( m ) = ∑T t=1 wtf ( θt ) . ( 2 ) Here wt ’ s are arbitrary weights associated with each time-step . Our overall framework is illustrated in Figure 1 . We defer the discussion on “ extract information share ” in Figure 1 , which is a new structure for information sharing between coordinates , to Section 3.3 . 3.1 STRUCTURE OF EACH RECURRENT UNIT . Previous design In the first L2L paper ( Andrychowicz et al. , 2016 ) , the authors proposed to use LSTM for each recurrent unit . To stabilize the meta-optimizer ’ s convergence , they further proposed to pre-process gradient by the following function before feeding into LSTM : f ( g ) = { ( log |g|p , sgn ( g ) ) if |g| ≥ e −p ( −1 , epg ) else , ( 3 ) where p is a parameter controlling how small gradients are disregarded . All followup works build upon a similar LSTM preprocessing structure . Our design Instead of simply using a LSTM , we take a step back and study characteristics of the recurrent cell that leads to better convergence . To this end , it is instructive to take the meta-optimizer m to be vanilla RNN with only 2 hidden states to demonstrate the subtle problem with the bias term . In this setting , the update rule of equation 1 becomes : θt+1 = θt + tanh ( w1gt + w2ht + b ) ( 4 ) ht+1 = tanh ( w1gt + w2ht + b ) , ( 5 ) where w1 , w2 are the RNN weights and b is the bias term . As we optimize function f by changing θ , we will gradually get smaller loss and , thus , smaller gradients . When θ approaches a stationary point of the function f , in order to converge , the update needs to be small enough to stay close to the stationary point . However , from the update rule as in equation 4 and equation 5 , we can clearly see that even when gt is small , unless b equals to 0 , the update rule will still produce an update with magnitude proportional to tanh ( b ) . In the extreme case when θ is the optimal point and when the previous hidden state is almost 0 , this update rule will still pushes the parameter away from the optimal point . Thus , the bias term b is problematic as optimization proceeds close to the optimal solution . The same observation can be made even with the pre-processing methods equation 3 . For these methods , depending on the gradient magnitude , we could further write the update as : θt+1 = θt + tanh ( w11 log |gt| p + w12sign ( gt ) + w2ht + b ) , or θt+1 = θt + tanh ( −1w11 + w12epg + w2ht + b ) . Notice the constant terms 1 or -1 in the equations . This , again , will be problematic when the corresponding parameters w11 and w12 are not 0 . Clearly , pre-processing involving the separation of the magnitude and the direction of gradients does not help to address the problem . This is an issue in practice as well . For instance , in one of our experiments , we notice that the maximal bias term of LSTM in Lv et al . ( 2017 ) is 0.41 , and maximal bias term of LSTM in Andrychowicz et al . ( 2016 ) is 0.31 ; this , consequently , leads to bad performance . Meta-optimizers fail to learn bias= 0 since they are only trained in a limited horizon and it is beneficial to “ not ” have bias term to be 0 to speed up initial stage of optimization . Following the analysis , we conclude that the bias term in the meta-optimizer can cause convergence issues . If we remove the bias term in the RNN model , we hypothesize that we do not require any pre-processing to ensure that the model trains well and behaves properly when applied on a longer horizon . We empirically validate this statement in the later part of the paper ( see Section 4 ) .

Summary: This paper presents several improvements over the existing learning to learn models including Andrychowicz et al. (2016) and Lv et al. (2017). Specifically, this paper analyzes the issues in the original learning to learn paradigm (L2L), including instability during training and bias term issues in the RNN. It proposes a new loss based on weighted difference for improving the meta-optimizer in the later stage of optimization. It also proposes information sharing between RNNs for each coordinate. Finally it presents how to fine-tune the meta-optimizer on new task.