Optimizing Loss Functions Through Multivariate Taylor Polynomial Parameterization

1 INTRODUCTION . As deep learning systems have become more complex , their architectures and hyperparameters have become increasingly difficult and time-consuming to optimize by hand . In fact , many good designs may be overlooked by humans with prior biases . Therefore , automating this process , known as metalearning , has become an essential part of the modern machine learning toolbox . Metalearning aims to solve this problem through a variety of approaches , including optimizing different aspects of the architecture from hyperparameters to topologies , and by using different methods from Bayesian optimization to evolutionary computation ( Schmidhuber , 1987 ; Elsken et al. , 2019 ; Miikkulainen et al. , 2019 ; Lemke et al. , 2015 ) . Recently , loss-function discovery and optimization has emerged as a new type of metalearning . Focusing on neural network ’ s root training goal it aims to discover better ways to define what is being optimized . However , loss functions can be challenging to optimize because they have a discrete nested structure as well as continuous coefficients . The first system to do so , Genetic Loss Optimization ( GLO ; Gonzalez & Miikkulainen , 2020 ) tackled this problem by discovering and optimizing loss functions in two separate steps : ( 1 ) representing the structure as trees , and evolving them with Genetic Programming ( GP ; Banzhaf et al. , 1998 ) ; and ( 2 ) optimizing the coefficients using Covariance-Matrix Adaptation Evolutionary Strategy ( CMA-ES ; Hansen & Ostermeier , 1996 ) . While the approach was successful , such separate processes make it challenging to find a mutually optimal structure and coefficients . Furthermore , small changes in the tree-based search space do not always result in small changes in the phenotype , and can easily make a function invalid , making the search process ineffective . In an ideal case , loss functions would be mapped into fixed-length vectors in a Hilbert space . This mapping should be smooth , well-behaved , well-defined , incorporate both a function ’ s structure and coefficients , and should by its very nature exclude large classes of infeasible loss functions . This paper introduces such an approach : Multivariate Taylor expansion-based genetic loss-function optimization ( TaylorGLO ) . With a novel parameterization for loss functions , the key pieces of information that affect a loss function ’ s behavior are compactly represented in a vector . Such vectors are then optimized for a specific task using CMA-ES . Special techniques can be developed to narrow down the search space and speed up evolution . Loss functions discovered by TaylorGLO outperform the standard cross-entropy loss ( or log loss ) on the MNIST , CIFAR-10 , CIFAR-100 , and SVHN datasets with several different network architectures . They also outperform the Baikal loss , discovered by the original GLO technique , and do it with significantly fewer function evaluations . The reason for the improved performance is that evolved functions discourage overfitting to the class labels , thereby resulting in automatic regularization . These improvements are particularly pronounced with reduced datasets where such regularization matters the most . TaylorGLO thus further establishes loss-function optimization as a promising new direction for metalearning . 2 RELATED WORK . Applying deep neural networks to new tasks often involves significant manual tuning of the network design . The field of metalearning has recently emerged to tackle this issue algorithmically ( Schmidhuber , 1987 ; Lemke et al. , 2015 ; Elsken et al. , 2019 ; Miikkulainen et al. , 2019 ) . While much of the work has focused on hyperparameter optimization and architecture search , recently other aspects , such activation functions and learning algorithms , have been found useful targets for optimization ( Bingham et al. , 2020 ; Real et al. , 2020 ) . Since loss functions are at the core of machine learning , it is compelling to apply metalearning to their design as well . Deep neural networks are trained iteratively , by updating model parameters ( i.e. , weights and biases ) using gradients propagated backward through the network ( Rumelhart et al. , 1985 ) . The process starts from an error given by a loss function , which represents the primary training objective of the network . In many tasks , such as classification and language modeling , the cross-entropy loss ( also known as the log loss ) has been used almost exclusively . While in some approaches a regularization term ( e.g . L2 weight regularization ; Tikhonov , 1963 ) is added to the the loss function definition , the core component is still the cross-entropy loss . This loss function is motivated by information theory : It aims to minimize the number of bits needed to identify a message from the true distribution , using a code from the predicted distribution . In other types of tasks that do not fit neatly into a single-label classification framework different loss functions have been used successfully ( Gonzalez et al. , 2019 ; Gao & Grauman , 2019 ; Kingma & Welling , 2014 ; Zhou et al. , 2016 ; Dong et al. , 2017 ) . Indeed , different functions have different properties ; for instance the Huber Loss ( Huber , 1964 ) is more resilient to outliers than other loss functions . Still , most of the time one of the standard loss functions is used without a justification ; therefore , there is an opportunity to improve through metalearning . Genetic Loss Optimization ( GLO ; Gonzalez & Miikkulainen , 2020 ) provided an initial approach into metalearning of loss functions . As described above , GLO is based on tree-based representations with coefficients . Such representations have been dominant in genetic programming because they are flexible and can be applied to a variety of function evolution domains . GLO was able to discover Baikal , a new loss function that outperformed the cross-entropy loss in image classification tasks . However , because the structure and coefficients are optimized separately in GLO , it can not easily optimize their interactions . Many of the functions created through tree-based search are not useful because they have discontinuities , and mutations can have disproportionate effects on the functions . GLO ’ s search is thus inefficient , requiring large populations that are evolved for many generations . Thus , GLO does not scale to the large models and datasets that are typical in modern deep learning . The technique presented in this paper , TaylorGLO , aims to solve these problems through a novel loss function parameterization based on multivariate Taylor expansions . Furthermore , since such representations are continuous , the approach can take advantage of CMA-ES ( Hansen & Ostermeier , 1996 ) as the search method , resulting in faster search . 3 LOSS FUNCTIONS AS MULTIVARIATE TAYLOR EXPANSIONS . Taylor expansions ( Taylor , 1715 ) are a well-known function approximator that can represent differentiable functions within the neighborhood of a point using a polynomial series . Below , the common univariate Taylor expansion formulation is presented , followed by a natural extension to arbitrarily-multivariate functions . Given a Ckmax smooth ( i.e. , first through kmax derivatives are continuous ) , real-valued function , f ( x ) : R→ R , a kth-order Taylor approximation at point a ∈ R , f̂k ( x , a ) , where 0 ≤ k ≤ kmax , can be constructed as f̂k ( x , a ) = k∑ n=0 1 n ! f ( n ) ( a ) ( x− a ) n. ( 1 ) Conventional , univariate Taylor expansions have a natural extension to arbitrarily high-dimensional inputs of f . Given a Ckmax+1 smooth , real-valued function , f ( x ) : Rn → R , a kth-order Taylor approximation at point a ∈ Rn , f̂k ( x , a ) , where 0 ≤ k ≤ kmax , can be constructed . The stricter smoothness constraint compared to the univariate case allows for the application of Schwarz ’ s theorem on equality of mixed partials , obviating the need to take the order of partial differentiation into account . Let us define an nth-degree multi-index , α = ( α1 , α2 , . . . , αn ) , where αi ∈ N0 , |α| = ∑n i=1 αi , α ! = ∏n i=1 αi ! . x α = ∏n i=1 x αi i , and x ∈ Rn . Multivariate partial derivatives can be concisely written using a multi-index ∂αf = ∂α11 ∂ α2 2 · · · ∂αnn f = ∂|α| ∂xα11 ∂x α2 2 · · · ∂x αn n . ( 2 ) Thus , discounting the remainder term , the multivariate Taylor expansion for f ( x ) at a is f̂k ( x , a ) = ∑ ∀α , |α|≤k 1 α ! ∂αf ( a ) ( x− a ) α . ( 3 ) The unique partial derivatives in f̂k and a are parameters for a kth order Taylor expansion . Thus , a kth order Taylor expansion of a function in n variables requires n parameters to define the center , a , and one parameter for each unique multi-index α , where |α| ≤ k. That is : # parameters ( n , k ) = n+ ( n+k k ) = n+ ( n+k ) ! n ! k ! . The multivariate Taylor expansion can be leveraged for a novel loss-function parameterization . Let an n-class classification loss function be defined as LLog = − 1n ∑n i=1 f ( xi , yi ) . The function f ( xi , yi ) can be replaced by its kth-order , bivariate Taylor expansion , f̂k ( x , y , ax , ay ) . More sophisticated loss functions can be supported by having more input variables beyond xi and yi , such as a time variable or unscaled logits . This approach can be useful , for example , to evolve loss functions that change as training progresses . For example , a loss function in x and y has the following third-order parameterization with parameters θ ( where a = 〈θ0 , θ1〉 ) : L ( x , y ) = − 1 n n∑ i=1 [ θ2 + θ3 ( yi − θ1 ) + 12θ4 ( yi − θ1 ) 2 + 16θ5 ( yi − θ1 ) 3 + θ6 ( xi − θ0 ) +θ7 ( xi − θ0 ) ( yi − θ1 ) + 12θ8 ( xi − θ0 ) ( yi − θ1 ) 2 + 12θ9 ( xi − θ0 ) 2 + 12θ10 ( xi − θ0 ) 2 ( yi − θ1 ) + 16θ11 ( xi − θ0 ) 3 ] ( 4 ) Notably , the reciprocal-factorial coefficients can be integrated to be a part of the parameter set by direct multiplication if desired . As will be shown in this paper , the technique makes it possible to train neural networks that are more accurate and learn faster than those with tree-based loss function representations . Representing loss functions in this manner confers several useful properties : • It guarantees smooth functions ; • Functions do not have poles ( i.e. , discontinuities going to infinity or negative infinity ) within their relevant domain ; • They can be implemented purely as compositions of addition and multiplication operations ; • They can be trivially differentiated ; • Nearby points in the search space yield similar results ( i.e. , the search space is locally smooth ) , making the fitness landscape easier to search ; • Valid loss functions can be found in fewer generations and with higher frequency ; • Loss function discovery is consistent and not dependent on a specific initial population ; and • The search space has a tunable complexity parameter ( i.e. , the order of the expansion ) . These properties are not necessarily held by alternative function approximators . For instance : Fourier series are well suited for approximating periodic functions ( Fourier , 1829 ) . Consequently , they are not as well suited for loss functions , whose local behavior within a narrow domain is important . Being a composition of waves , Fourier series tend to have many critical points within the domain of interest . Gradients fluctuate around such points , making gradient descent infeasible . Additionally , close approximations require a large number of terms , which in itself can be injurious , causing large , high-frequency fluctuations known as “ ringing ” , due to Gibb ’ s phenomenon ( Wilbraham , 1848 ) . Padé approximants can be more accurate approximations than Taylor expansions ; indeed , Taylor expansions are a special case of Padé approximants where M = 0 ( Graves-Morris , 1979 ) . However , unfortunately Padé approximants can model functions with one or more poles , which valid loss functions typically should not have . These problems still exist , and are exacerbated , for Chisholm approximants ( a bivariate extension ; Chisholm , 1973 ) and Canterbury approximants ( a multivariate generalization ; Graves-Morris & Roberts , 1975 ) . Laurent polynomials can represent functions with discontinuities , the simplest being x−1 . While Laurent polynomials provide a generalization of Taylor expansions into negative exponents , the extension is not useful because it results in the same issues as Padé approximants . Polyharmonic splines can represent continuous functions within a finite domain , however , the number of parameters is prohibitive in multivariate cases . The multivariate Taylor expansion is therefore a better choice than the alternatives . It makes it possible to optimize loss functions efficiently in TaylorGLO , as will be described next . 4 THE TAYLORGLO METHOD Candidate Evaluation0 0 00 0 0 [ ] 0 0 Build TaylorGLO Loss FunctionCMA-ES Mean Vector Covariance Matrix Sampler Partial Model Training ( Few Epochs ) ℒ = − 1 n n ∑ i=1 f ( xi , yi ) 1.1 0.8 1.41.2 1 1.2 [ ] 1.4 0.8 Build TaylorGLO Loss Function Initial Solution Mean Vector Best Solution Validation Set Evaluation Figure 1 : The TaylorGLO method . Starting with a population of initially unbiased loss functions , CMA-ES optimizes their Taylor expansion parameters in order to maximize validation accuracy after partial training . The candidate with the highest accuracy is chosen as the final , best solution . TaylorGLO ( Figure 1 ) aims to find the optimal parameters for a loss function represented as a multivariate Taylor expansion . The parameters for a Taylor approximation ( i.e. , the center point and partial derivatives ) are referred to as θf̂ : θf̂ ∈ Θ , Θ = R # parameters . TaylorGLO strives to find the vector θ∗ f̂ that parameter- izes the optimal loss function for a task . Because the values are continuous , as opposed to discrete graphs of the original GLO , it is possible to use continuous optimization methods . In particular , Covariance Matrix Adaptation Evolutionary Strategy ( CMA-ES Hansen & Ostermeier , 1996 ) is a popular population-based , black-box optimization technique for rugged , continuous spaces . CMA-ES functions by maintaining a covariance matrix around a mean point that represents a distribution of solutions . At each generation , CMA-ES adapts the distribution to better fit evaluated objective values from sampled individuals . In this manner , the area in the search space that is being sampled at each step grows , shrinks , and moves dynamically as needed to maximize sampled candidates ’ fitnesses . TaylorGLO uses the ( µ/µ , λ ) variant of CMA-ES ( Hansen & Ostermeier , 2001 ) , which incorporates weighted rank-µ updates ( Hansen & Kern , 2004 ) to reduce the number of objective function evaluations needed . In order to find θ∗ f̂ , at each generation CMA-ES samples points in Θ . Their fitness is determined by training a model with the corresponding loss function and evaluating the model on a validation dataset . Fitness evaluations may be distributed across multiple machines in parallel and retried a limited number of times upon failure . An initial vector of θf̂ = 0 is chosen as a starting point in the search space to avoid bias . Fully training a model can be prohibitively expensive in many problems . However , performance near the beginning of training is usually correlated with performance at the end of training , and therefore it is enough to train the models only partially to identify the most promising candidates . This type of approximate evaluation is common in metalearning ( Grefenstette & Fitzpatrick , 1985 ; Jin , 2011 ) . An additional positive effect is that evaluation then favors loss functions that learn more quickly . For a loss function to be useful , it must have a derivative that depends on the prediction . Therefore , internal terms that do not contribute to ∂∂yLf ( x , y ) can be trimmed away . This step implies that any term t within f ( xi , yi ) with ∂∂yi t = 0 can be replaced with 0 . For example , this refinement simplifies Equation 4 , providing a reduction in the number of parameters from twelve to eight : L ( x , y ) = − 1 n n∑ i=1 [ θ2 ( yi − θ1 ) + 12θ3 ( yi − θ1 ) 2 + 16θ4 ( yi − θ1 ) 3 + θ5 ( xi − θ0 ) ( yi − θ1 ) + 12θ6 ( xi − θ0 ) ( yi − θ1 ) 2 + 12θ7 ( xi − θ0 ) 2 ( yi − θ1 ) ] . ( 5 )

Summary: This paper proposed a method, called TaylorGLO, to learn the loss functions, for training deep neural network, by meta-learning. Specifically, the authors proposed to parameterize the loss function with multivariate Taylor polynomial, and then learn the parameters in the polynomial using evolutionary algorithm within the meta-learning framework. The experiments showed improved performance of the TaylorGLO over cross-entropy baseline on several datasets and with different network architectures.