Improved Training Speed, Accuracy, and Data Utilization via Loss Function Optimization

1 INTRODUCTION . Much of the power of modern neural networks originates from their complexity , i.e. , number of parameters , hyperparameters , and topology . This complexity is often beyond human ability to optimize , and automated methods are needed . An entire field of metalearning has emerged recently to address this issue , based on various methods such as gradient descent , simulated annealing , reinforcement learning , Bayesian optimization , and evolutionary computation ( EC ) ( Elsken et al. , 2018 ) . While a wide repertoire of work now exists for optimizing many aspects of neural networks , the dynamics of training are still usually set manually without concrete , scientific methods . Training schedules , loss functions , and learning rates all affect the training and final functionality of a neural network . Perhaps they could also be optimized through metalearning ? The goal of this paper is to verify this hypothesis , focusing on optimization of loss functions . A general framework for loss function metalearning , covering both novel loss function discovery and optimization , is developed and evaluated experimentally . This framework , Genetic Loss-function Optimization ( GLO ) , leverages Genetic Programming to build loss functions represented as trees , and subsequently a Covariance-Matrix Adaptation Evolution Strategy ( CMA-ES ) to optimize their coefficients . EC methods were chosen because EC is arguably the most versatile of the metalearning approaches . EC , being a type of population-based search method , allows for extensive exploration , which often results in creative , novel solutions ( Lehman et al. , 2018 ) . EC has been successful in hyperparameter optimization and architecture design in particular ( Miikkulainen et al. , 2019 ; Stanley et al. , 2019 ; Real et al. , 2019 ; Loshchilov & Hutter , 2016 ) . It has also been used to discover mathematical formulas to explain experimental data ( Schmidt & Lipson , 2009 ) . It is , therefore , likely to find creative solutions in the loss-function optimization domain as well . Indeed , on the MNIST image classification benchmark , GLO discovered a surprising new loss function , named Baikal for its shape . This function performs very well , presumably by establishing an implicit regularization effect . Baikal outperforms the standard cross-entropy loss in terms of training speed , final accuracy , and data requirements . Furthermore , Baikal was found to transfer to a more complicated classification task , CIFAR-10 , while carrying over its benefits . At first glance , Baikal behaves rather unintuitively ; loss does not decrease monotonically as a network ’ s predictions become more correct . Upon further analysis , Baikal was found to perform implicit regularization , which caused this effect . Specifically , by preventing the network from being too confident in its predictions , training was able to produce a more robust model . This finding was surprising and encouraging , since it means that GLO is able to discover loss functions that train networks that are more generalizable and overfit less . The next section reviews related work in metalearning and EC , to help motivate the need for GLO . Following this review , GLO is described in detail , along with the domains upon which it has been evaluated . The subsequent sections present the experimental results , including an analysis of the loss functions that GLO discovers . 2 RELATED WORK . In addition to hyperparameter optimization and neural architecture search , new opportunities for metalearning have recently emerged . In particular , learning rate scheduling and adaptation can have a significant impact on a model ’ s performance . Learning rate schedules determine how the learning rate changes as training progresses . This functionality tends to be encapsulated away in practice by different gradient-descent optimizers , such as AdaGrad ( Duchi et al. , 2011 ) and Adam ( Kingma & Ba , 2014 ) . While the general consensus has been that monotonically decreasing learning rates yield good results , new ideas , such as cyclical learning rates ( Smith , 2017 ) , have shown promise in learning better models in fewer epochs . Metalearning methods have also been recently developed for data augmentation , such as AutoAugment ( Cubuk et al. , 2018 ) , a reinforcement learning based approach to find new data augmentation policies . In reinforcement learning tasks , EC has proven a successful approach . For instance , in evolving policy gradients ( Houthooft et al. , 2018 ) , the policy loss is not represented symbolically , but rather as a neural network that convolves over a temporal sequence of context vectors . In reward function search ( Niekum et al. , 2010 ) , the task is framed as a genetic programming problem , leveraging PushGP ( Spector et al. , 2001 ) . In terms of loss functions , a generalization of the L2 loss was proposed with an adaptive loss parameter ( Barron , 2017 ) . This loss function is shown to be effective in domains with multivariate output spaces , where robustness might vary across between dimensions . Specifically , the authors found improvements in Variational Autoencoder ( VAE ) models , unsupervised monocular depth estimation , geometric registration , and clustering . Additionally , work has found promise in moving beyond the standard cross-entropy loss for classification ( Janocha & Czarnecki , 2017 ) . L1 and L2 losses were found to have useful probabilistic properties . The authors found certain loss functions to be more resilient to noise than the crossentropy loss . Notably , no existing work in the metalearning literature automatically optimizes loss functions for neural networks . As shown in this paper , evolutionary computation can be used in this role to improve neural network performance , gain a better understanding of the processes behind learning , and help reach the ultimate goal of fully automated learning . 3 THE GLO APPROACH . The task of finding and optimizing loss functions can be framed as a functional regression problem . GLO accomplishes this through the following high-level steps ( shown in Figure 1 ) : ( 1 ) loss function discovery : using approaches from genetic programming , a genetic algorithm builds new candidate loss functions , and ( 2 ) coefficient optimization : to further optimize a specific loss function , a covariance-matrix adaptation evolutionary strategy ( CMA-ES ) is leveraged to optimize coefficients . 3.1 LOSS FUNCTION DISCOVERY . GLO uses a population-based search approach , inspired by genetic programming , to discover new optimized loss function candidates . Under this framework , loss functions are represented as trees within a genetic algorithm . Trees are a logical choice to represent functions due to their hierarchical nature . The loss function search space is defined by the following tree nodes : Unary Operators : log ( ◦ ) , ◦2 , √ ◦ Binary Operators : + , ∗ , − , ÷ Leaf Nodes : x , y , 1 , −1 , where x represents a true label , and y represents a predicted label . The search space is further refined by automatically assigning a fitness of 0 to trees that do not contain both at least one x and one y . Generally , a loss function ’ s fitness within the genetic algorithm is the validation performance of a network trained with that loss function . To expedite the discovery process , and encourage the invention of loss functions that make learning faster , training does not proceed to convergence . Unstable training sessions that result in NaN values are assigned a fitness of 0 . Fitness values are cached to avoid needing to retrain the same network twice . These cached values are each associated with a canonicalized version of their corresponding tree , resulting in fewer required evaluations . The initial population is composed of randomly generated trees with a maximum depth of 2 . Recursively starting from the root , nodes are randomly chosen from the allowable operator and leaf nodes using a weighting ( where log ( ◦ ) , x , y are three times as likely and √ ◦ is two times as likely as + , ∗ , − , ÷ , 1 , −1 ) . This weighting can impart a bias and prevent , for example , the integer 1 from occurring too frequently . The genetic algorithm has a population size of 80 , incorporates elitism with six elites per generation , and uses roulette sampling . Recombination is accomplished by randomly splicing two trees together . For a given pair of parent trees , a random element is chosen in each as a crossover point . The two subtrees , whose roots are the two crossover points , are then swapped with each other . Figure 1 presents an example of this method of recombination . Both resultant trees become part of the next generation . Recombination occurs with a probability of 80 % . To introduce variation into the population , the genetic algorithm has the following mutations , applied in a bottom-up fashion : • Integer scalar nodes are incremented or decremented with a 5 % probability . • Nodes are replaced with a weighted-random node with the same number of children with a 5 % probability . • Nodes ( and their children ) are deleted and replaced with a weighted-random leaf node with a 5 % ∗ 50 % = 2.5 % probability . • Leaf nodes are deleted and replaced with a weighted-random element ( and weightedrandom leaf children if necessary ) with a 5 % ∗ 50 % = 2.5 % probability . Combined , the iterative sampling , recombination , and mutation of trees within the population leads to the discovery of new loss functions which maximize fitness . 3.2 COEFFICIENT OPTIMIZATION . Loss functions found by the above genetic algorithm can all be thought of having unit coefficients for each node in the tree . This set of coefficients can be represented as a vector with dimensionality equal to the number of nodes in a loss function ’ s tree . The number of coefficients can be reduced by pruning away coefficients that can be absorbed by others ( e.g. , 3 ( 5x + 2y ) = 15x + 6y ) . The coefficient vector is optimized independently and iteratively using a covariance-matrix adaptation evolutionary strategy ( CMA-ES ) ( Hansen & Ostermeier , 1996 ) . The specific variant of CMA-ES that GLO uses is ( µ/µ , λ ) -CMA-ES ( Hansen & Ostermeier , 2001 ) , which incorporates weighted rank-µ updates ( Hansen & Kern , 2004 ) to reduce the number of objective function evaluations that are needed . The implementation of GLO presented in this paper uses an initial step size σ = 1.5 . As in the discovery phase , the objective function is the network ’ s performance on a validation dataset after a shortened training period . 4 EXPERIMENTAL EVALUATION . This section provides an experimental evaluation of GLO , on the MNIST and CIFAR-10 image classification tasks . Baikal , a GLO loss function found on MNIST , is presented and evaluated in terms of its resulting testing accuracy , training speed , training data requirements , and transferability to CIFAR-10 . Implementation details are presented in the appendix in Section A.1 . 4.1 TARGET TASKS . Experiments on GLO are performed using two popular image classification datasets , MNIST Handwritten Digits ( LeCun et al. , 1998 ) and CIFAR-10 ( Krizhevsky & Hinton , 2009 ) . Both datasets , with MNIST in particular , are well understood , and relatively quick to train . The choice of these datasets allowed rapid iteration in the development of GLO and allowed time for more thorough experimentation . The selected model architectures are simple , since achieving state-of-the-art accuracy on MNIST and CIFAR-10 is not the focus of this paper , rather the improvements brought about by using a GLO loss function are . More information on the datasets , along with their corresponding architectures and experimental setup is provided in the appendix , under Section A.2 . Both of these tasks , being classification problems , are traditionally framed with the standard crossentropy loss ( sometimes referred to as the log loss ) : LLog = − 1n ∑n−1 i=0 xi log ( yi ) , where x is sampled from the true distribution , y is from the predicted distribution , and n is the number of classes . The cross-entropy loss is used as a baseline in this paper ’ s experiments .

Summary: This paper proposes a very interesting idea of loss function optimization. At first sight, loss function is the goal of optimization and can not be optimized directly. However, the true goal of optimization is the final accuracy (for classification). So lots of loss functions can be designed and combined to form a large search space. In this paper, the authors adopt genetic programming to design loss functions hierarchically.  And experiments show that GLO (Genetic Loss-function Optimization) based loss function can achieve better results than cross entropy.