Effective Regularization Through Loss-Function Metalearning

Loss-function metalearning can be used to discover novel , customized loss functions for deep neural networks , resulting in improved performance , faster training , and improved data utilization . A likely explanation is that such functions discourage overfitting , leading to effective regularization . This paper demonstrates theoretically that this is indeed the case for the TaylorGLO method : Decomposition of learning rules makes it possible to characterize the training dynamics and show that the loss functions evolved by TaylorGLO balance the pull to zero error , and a push away from it to avoid overfitting . This observation leads to an invariant that can be utilized to make the metalearning process more efficient in practice , and result in networks that are robust against adversarial attacks . Loss-function optimization can thus be seen as a well-founded new aspect of metalearning in neural networks . 1 INTRODUCTION . Regularization is a key concept in deep learning : it guides learning towards configurations that are likely to perform robustly on unseen data . Different regularization approaches originate from intuitive understanding of the learning process and have been shown to be effective empirically . However , the understanding of the underlying mechanisms , the different types of regularization , and their interactions , is limited . Recently , loss function optimization has emerged as a new area of metalearning , and shown great potential in training better models . Experiments suggest that metalearned loss functions serve as regularizers in a surprising but transparent way : they prevent the network from learning too confident predictions ( e.g . Baikal loss ; Gonzalez & Miikkulainen , 2020a ) . While it may be too early to develop a comprehensive theory of regularization , given the relatively nascent state of this area , it may be possible to make progress in understanding regularization of this specific type . That is the goal of this paper . Since metalearned loss functions are customized to a given architecture-task pair , there needs to be a shared framework under which loss functions can be analyzed and compared . The TaylorGLO ( Gonzalez & Miikkulainen , 2020b ) technique for loss function metalearning lends itself well to such analysis : It represents loss functions as multivariate Taylor polynomials , and leverages evolution to optimize a fixed number of parameters in this representation . In this framework , the SGD learning rule is decomposed to coefficient expressions that can be defined for a wide range of loss functions . These expressions provide an intuitive understanding of the training dynamics in specific contexts . Using this framework , mean squared error ( MSE ) , cross-entropy , Baikal , and TaylorGLO loss functions are analyzed at the null epoch , when network weights are similarly distributed ( Appendix C ) , and in a zero training error regime , where the training samples ’ labels have been perfectly memorized . For any intermediate point in the training process , the strength of the zero training error regime as an attractor is analyzed and a constraint on this property is derived on TaylorGLO parameters by characterizing how the output distribution ’ s entropy changes . In a concrete TaylorGLO loss function that has been metalearned , these attraction dynamics are calculated for individual samples at every epoch in a real training run , and contrasted with those for the cross-entropy loss . This comparison provides clarity on how TaylorGLO avoids becoming overly confident in its predictions . Further , the analysis shows ( in Appendix D.2 ) how label smoothing ( Szegedy et al. , 2016 ) , a traditional type of regularization , can be implicitly encoded by TaylorGLO loss functions : Any representable loss function has label-smoothed variants that are also representable by the parameterization . From these analyses , practical opportunities arise . First , at the null epoch , where the desired behavior can be characterized clearly , an invariant can be derived on a TaylorGLO loss function ’ s parameters that must hold true for networks to be trainable . This constraint is then applied within the TaylorGLO algorithm to guide the search process towards good loss functions more efficiently . Second , lossfunction-based regularization results in robustness that should e.g . make them more resilient to adversarial attacks . This property is demonstrated experimentally by incorporating adversarial robustness as an objective within the TaylorGLO search process . Thus , loss-function metalearning can be seen as a well-founded and practical approach to effective regularization in deep learning . 2 BACKGROUND . Regularization traditionally refers to methods for encouraging smoother mappings by adding a regularizing term to the objective function , i.e. , to the loss function in neural networks . It can be defined more broadly , however , e.g . as “ any modification we make to a learning algorithm that is intended to reduce its generalization error but not its training error ” ( Goodfellow et al. , 2015 ) . To that end , many regularization techniques have been developed that aim to improve the training process in neural networks . These techniques can be architectural in nature , such as Dropout ( Srivastava et al. , 2014 ) and Batch Normalization ( Ioffe & Szegedy , 2015 ) , or they can alter some aspect of the training process , such as label smoothing ( Szegedy et al. , 2016 ) or the minimization of a weight norm ( Hanson & Pratt , 1989 ) . These techniques are briefly reviewed in this section , providing context for loss-function metalearning . 2.1 IMPLICIT BIASES IN OPTIMIZERS . It may seem surprising that overparameterized neural networks are able to generalize at all , given that they have the capacity to memorize a training set perfectly , and in fact sometimes do ( i.e. , zero training error is reached ) . Different optimizers have different implicit biases that determine which solutions are ultimately found . These biases are helpful in providing implicit regularization to the optimization process ( Neyshabur et al. , 2015 ) . Such implicit regularization is the result of a network norm—a measure of complexity—that is minimized as optimization progresses . This is why models continue to improve even after training set has been memorized ( i.e. , the training error global optima is reached ) ( Neyshabur et al. , 2017 ) . For example , the process of stochastic gradient descent ( SGD ) itself has been found to provide regularization implicitly when learning on data with noisy labels ( Blanc et al. , 2020 ) . In overparameterized networks , adaptive optimizers find very different solutions than basic SGD . These solutions tend to have worse generalization properties , even though they tend to have lower training errors ( Wilson et al. , 2017 ) . 2.2 REGULARIZATION APPROACHES . While optimizers may minimize a network norm implicitly , regularization approaches supplement this process and make it explicit . For example , a common way to restrict the parameter norm explicitly is through weight decay . This approach discourages network complexity by placing a cost on weights ( Hanson & Pratt , 1989 ) . Generalization and regularization are often characterized at the end of training , i.e . as a behavior that results from the optimization process . Various findings have influenced work in regularization . For example , flat landscapes have better generalization properties ( Keskar et al. , 2017 ; Li et al. , 2018 ; Chaudhari et al. , 2019 ) . In overparameterized cases , the solutions at the center of these landscapes may have zero training error ( i.e. , perfect memorization ) , and under certain conditions , zero training error empirically leads to lower generalization error ( Belkin et al. , 2019 ; Nakkiran et al. , 2019 ) . However , when a training loss of zero is reached , generalization suffers ( Ishida et al. , 2020 ) . This behavior can be thought of as overtraining , and techniques have been developed to reduce it at the end of the training process , such as early stopping ( Morgan & Bourlard , 1990 ) and flooding ( Ishida et al. , 2020 ) . Both flooding and early stopping assume that overfitting happens at the end of training , which is not always true ( Golatkar et al. , 2019 ) . In fact , the order in which easy-to-generalize and hard-togeneralize concepts are learned is important for the network ’ s ultimate generalization . For instance , larger learning rates early in the training process often lead to better generalization in the final model ( Li et al. , 2019 ) . Similarly , low-error solutions found by SGD in a relatively quick manner—such as through high learning rates—often have good generalization properties ( Yao et al. , 2007 ) . Other techniques tackle overfitting by making it more difficult . Dropout ( Srivastava et al. , 2014 ) makes some connections disappear . Cutout ( DeVries & Taylor , 2017 ) , Mixup ( Zhang et al. , 2018 ) , and their composition , CutMix ( Yun et al. , 2019 ) , augment training data with a broader variation of examples . Notably , regularization is not a one-dimensional continuum . Different techniques regularize in different ways that often interact . For example , flooding invalidates performance gains from early stopping ( Ishida et al. , 2020 ) . However , ultimately all regularization techniques alter the gradients that result from the training loss . This observation suggests loss-function optimization might be an effective way to regularize the training process . 2.3 LOSS-FUNCTION METALEARNING . The idea of metalearning loss-functions has a deep history with many recent developments that have shown promise in practical settings . Prior work in reinforcement learning showed that metalearning various types of objectives is useful . For instance , in evolving policy gradients ( Houthooft et al. , 2018 ) , the policy loss is not represented symbolically , but rather as a neural network that convolves over a temporal sequence of context vectors . In reward function search ( Niekum et al. , 2010 ) , the task is framed as a genetic programming problem , leveraging PushGP ( Spector et al. , 2001 ) . Various actor-critic reinforcement learning approaches have tackled learning a meta-critic neural network that can generate losses ( Sung et al. , 2017 ; Zhou et al. , 2020 ) . Metalearned critic network techniques have also been applied outside of reinforcement learning to train better few-shot classifiers ( Antoniou & Storkey , 2019 ) . In unsupervised representation learning , weight update rules for semi-supervised learning have themselves been metalearned successfully ( Metz et al. , 2018 ) . The update rules were constrained to fit a biological neuron model and transferred well between tasks . Concrete loss-function metalearning for deep networks was first introduced by Gonzalez & Miikkulainen ( 2020a ) as an automatic way to find customized loss functions that aim to optimize a performance metric for a model . The technique , a genetic programming approach , named GLO , discovered one particular loss function , Baikal , that improves classification accuracy , training speed , and data utilization . Baikal appeared to achieve these properties through a form of regularization that ensured the model would not become overly confident in its predictions . That is , instead of monotonically decreasing the loss when the output gets closer to the correct value , Baikal loss increases rapidly when the output is almost correct , thus discouraging extreme accuracy . This paper shows how training dynamics are specifically impacted in this manner when training with the Baikal loss . Overall , GLO demonstrated that while learned loss functions ’ generalization effects transfer across datasets and models to some extent , they are most powerful when they are customized to individual tasks and architectures . Different loss functions can take advantage of the different characteristics of each such setting . Other techniques have advanced this new field further , for example by metalearning state-dependent loss functions for inverse dynamics models ( Morse et al. , 2020 ) , and using a trained network that is itself a metalearned loss function ( Bechtle et al. , 2019 ) . One particular technique , TaylorGLO ( Gonzalez & Miikkulainen , 2020b ) , lends itself well to analyzing what makes loss-function metalearning effective . TaylorGLO represents loss functions as parameterizations of multivariate Taylor polynomials . This parameterization allows it to scale to models with millions of trainable parameters , including a variety of deep learning architectures in image classification tasks . TaylorGLO loss functions have a tunable complexity based on the order of the polynomial ; third-order functions were identified to work best in practical settings . The third-order TaylorGLO loss function parameterization provides a fixed representation that can be used to analyze a large family of loss functions through a unified methodology . In prior work , TaylorGLO loss functions were shown to improve generalization empirically ( Gonzalez & Miikkulainen , 2020b ) . This paper complements that work , aiming to explain why that is the case by analyzing the dynamics of training theoretically .

Summary: This paper analyzes a learned loss function called TaylorGLO based on third-order Taylor expansion and its regularization properties. This approach is novel and interesting in that the loss function is also learned on data. The analysis of the TaylorGLO loss and another learned loss function Baikal loss near zero error reveals interesting properties of preventing overconfident predictions.