A Gradient-based Kernel Approach for Efficient Network Architecture Search

1 INTRODUCTION . Understanding and improving the optimization of deep networks has been an active field of artificial intelligence . One of the mysteries in deep learning is why extremely deep neural networks are hard to train . Currently , the vanishing and exploding gradient problem is widely believed to be the main answer ( Bengio et al. , 1994 ; Hochreiter & Schmidhuber , 1997 ; Pascanu et al. , 2013 ) . Gradients exponentially decrease ( or increase ) from the top layer to the bottom layer in a multilayer network . By extension , the current vanishing and exploding gradient problem also refers to extremely small ( or large ) gradient values . Following this explanation , several widely-used training techniques have been proposed to assist optimization . These studies can be roughly classified into four categories : initialization-based approaches ( Glorot & Bengio , 2010 ; He et al. , 2015 ; Zhang et al. , 2019 ) , activation-based approaches ( Hendrycks & Gimpel , 2016 ; Klambauer et al. , 2017 ) , normalization based approaches ( Ioffe & Szegedy , 2015 ; Salimans & Kingma , 2016 ; Ulyanov et al. , 2016 ; Lei Ba et al. , 2016 ; Nguyen & Salazar , 2019 ) , and skip-connection-based approaches ( He et al. , 2016 ; Huang et al. , 2017a ) . These techniques provide good alternatives to stabilize gradients and bring promising performance improvements . Despite much evidence showing the importance of steady gradient values , we are still curious about whether there are unexplored but important factors . To answer this question , we conduct extensive experiments and find some surprising phenomena . First , the accuracy curves of some models converge well with the remaining gradient vanishing and exploding problem . Second , we find several cases where models with similar gradient values show different convergence performance . These results indicate that gradient values may not be vital as much as we expect , and some hidden factors play a significant role in optimization . To answer this question , we explore other gradient features , such as covariance matrix , correlations , variance , and so on . These features are used to evaluate gradients from different perspectives . The covariance matrix evaluates the similarity between any two parameters , which is widely used in optimization studies ( Zhao & Zhang , 2015 ; Faghri et al. , 2020 ) . Gradient correlation is defined to evaluate the similarity between any two examples . Among them , experiments show that gradient correlation is an important factor . Compared to widely-explored 1We will release the code on publication . gradient values , gradient correlations are rarely studied . As gradient values affect the step size in optimization , vanishing gradients prevent models from changing weight values . In the worst case , this could completely stop training . Gradient correlations evaluate the randomness of gradient directions in Euclidean space . Lower gradient correlations indicate more “ random ” gradient directions and more “ conflicting ” parameter updates . For a better understanding , we visualize absolute gradient values and gradient correlations ( See Section 4 for more details ) . It illustrates that models with either small values or small correlations show worse convergence and generalization performance , indicating the importance of these two factors . Following the new finding , we take a further step to explore gradient correlations and gradient values in network architecture search . We first formulate two factors into a unified gradient-based kernel , which is defined as the average of a gradient dot-product matrix . Motivated by the observation that architectures with larger kernels at initialization tend to have better average results , we develop a lightweight network architecture search approach , called GT-NAS , which evaluates architectures according to the gradient-based kernel at initialization . Unlike other NAS approaches that evaluate architectures via full training , which takes massive computation resources , GT-NAS only relies on a few examples for kernel calculation . We first select top-k architectures with largest kernels as candidates , which are then trained to select the best one based on validation performance . In practice , k is usually set to be a very small value . Experiments show that GT-NAS achieves competitive results with extremely fast search on NAS-Bench-201 ( Ying et al. , 2019 ) , a NAS benchmark dataset . The low search cost allows us to apply NAS on diverse tasks . Specifically , the structure searched by the proposed policy outperforms the naive baseline without NAS on 4 datasets covering image classification and text classification . The main contributions are summarized as follows : • We propose that both gradient correlations and gradient values matter in optimization , which gives a new insight to understand and develop optimization techniques . • Following the new finding , we propose a gradient-based kernel approach for NAS , which is able to search for well-performing architectures efficiently . • Experiments show that GT-NAS achieves competitive results with orders of magnitude faster than the naive baseline . 2 RELATED WORK . Understanding and improving optimization Understanding and improving the optimization of deep networks has long been a hot research topic . Bengio et al . ( 1994 ) find the vanishing and exploding gradient problem in neural network training . To address this problem , a lot of promising approaches have been proposed in recent years , which can be classified into four research lines . The first research line focuses on initialization ( Sutskever et al. , 2013 ; Mishkin & Matas , 2016 ; Hanin & Rolnick , 2018 ) . Glorot & Bengio ( 2010 ) propose to control the variance of parameters via appropriate initialization . Following this work , several widely-used initialization approaches have been proposed , including Kaiming initialization ( He et al. , 2015 ) , Xaiver initinalization ( Glorot & Bengio , 2010 ) , and Fixup initialization ( Zhang et al. , 2019 ) . The second research line focuses on normalization ( Ioffe & Szegedy , 2015 ; Lei Ba et al. , 2016 ; Ulyanov et al. , 2016 ; Wu & He , 2018 ; Nguyen & Salazar , 2019 ) . These approaches aim to avoid the vanishing gradient by controlling the distribution of intermediate layers . The third is mainly based on activation functions to make the derivatives of activation less saturated to avoid the vanishing problem , including GELU activation ( Hendrycks & Gimpel , 2016 ) , SELU activation ( Klambauer et al. , 2017 ) , and so on . The motivation behind these approaches is to avoid unsteady derivatives of the activation with respect to the inputs . The fourth focuses on gradient clipping ( Pascanu et al. , 2013 ) . Gradient-based Kernel Our work is also related to gradient-based kernels ( Advani & Saxe , 2017 ; Jacot et al. , 2018 ) . NTK ( Jacot et al. , 2018 ) is a popular gradient-based kernel , defined as the Gram matrix of gradients . It is proposed to analyze model ’ s convergence and generalization . Following these studies , many researchers are devoted to understand current networks from the perspective of NTK ( Lee et al. , 2019 ; Hastie et al. , 2019 ; Allen-Zhu et al. , 2019 ; Arora et al. , 2019 ) . Du et al . ( 2019b ) use the Gram matrix of gradients to prove that for an m hidden node shallow neural network with ReLU activation , as long as m is large enough , randomly initialized gradient descent converges to a globally optimal solution at a linear convergence rate for the quadratic loss function . Following this work , Du et al . ( 2019a ) further expand this finding and prove that gradient descent achieves zero training loss in polynomial time for a deep over-parameterized neural network with residual connections . Network architecture search Neural architecture search aims to replace expert-designed networks with learned architectures ( Chang et al. , 2019 ; Li et al. , 2019 ; Zhou et al. , 2020 ; He et al. , 2020a ; Fang et al. , 2020 ; He et al. , 2020b ; You et al. , 2020 ; Alves & de Oliveira , 2020 ) . The first NAS research line mainly focuses on random search . The key idea is to randomly evaluate various architectures and select the best one based on their validation performance . However , despite promising results , many of them require thousands of GPU days to achieve desired results . To address this problem , Ying et al . ( 2019 ) propose a reinforcement learning based search policy which introduces an architecture generator with validation accuracy as reward . Another research line is based on evolution approaches . Real et al . ( 2019 ) propose a two-stage search policy . The first stage selects several well-performing parent architectures . The second stage applies mutation on these parent architectures to select the best one . Following this work , So et al . ( 2019 ) apply the evolution search on Transformer networks and achieve new state-of-the-art results on machine translation and language modeling tasks . Although these approaches can reduce exploration costs , the dependence on validation accuracy still leads to huge computation costs . To get rid of the dependence on validation accuracy , several studies ( Jiang et al. , 2019 ; Liu et al. , 2019 ; Dong & Yang , 2019 ; Zela et al. , 2020 ; Chu et al. , 2020 ) re-formulate the task in a differentiable manner and allow efficient search using gradient descent . Unlike these studies , we propose a lightweight NAS approach , which largely reduces evaluation cost . Furthermore , the new approach does not rely on additional search models , thus also saves memory usage . 3 NOTATIONS : GRADIENT VALUE AND GRADIENT CORRELATION . Assume g ( l ) i = { g ( l ) i,1 , · · · , g ( l ) i , r } is the gradient vector of example i with respect to all parameters in the l-th layer . r is the total number of parameters . We combine all gradient vectors together to construct a gradient matrix G ( l ) ∈ Rd×r where d refers to the number of training examples . We define a matrix P ( l ) to store the dot-product of gradients . The dot-product between two gradient vectors is : P ( l ) i , j = 〈G ( l ) i , G ( l ) j〉 . ( 1 ) We divide P ( l ) by the norm of gradient vectors to get a gradient correlation matrix C ( l ) : C ( l ) i , j = P ( l ) i , j ||G ( l ) i|| · ||G ( l ) j || . ( 2 ) Then , we use the average value of C ( l ) as the final correlation score : R = E ( C ( l ) ) . ( 3 ) Here we also evaluate the absolute gradient value : V = E ( |G ( l ) | ) . ( 4 ) Since it requires too much time to calculate R and V for plenty of training examples , we randomly sample µ examples to estimate R and V in implementation for fast calculation . The dot-product between vectors with too many elements will cause the overflow problem . We also randomly sample θ parameters when calculating R. To avoid the sampling bias that might affect our findings , we test different µ ∈ [ 50 , 100 , 200 , 300 , 400 , 500 ] and θ ∈ [ 50 , 100 , 200 , 300 , 400 , 500 ] , and observe the same phenomena . For simplicity , we show our findings in the next section with µ = 50 and θ = 50 .

Summary: This paper pointed out that both gradient correlations and gradient values have strong impacts on model training, which is an insightful finding. Based on this finding, they explore a simple yet effective network architecture search (NAS) approach. The new approach replaces the expensive “train-then-test” evaluation paradigm with a new lightweight function according to the gradient-based kernel at initialization. Experiments show the high speed of the approach.