A Gradient-based Kernel Approach for Efficient Network Architecture Search

1 INTRODUCTION . Understanding and improving the optimization of deep networks has been an active field of artificial intelligence . One of the mysteries in deep learning is why extremely deep neural networks are hard to train . Currently , the vanishing and exploding gradient problem is widely believed to be the main answer ( Bengio et al. , 1994 ; Hochreiter & Schmidhuber , 1997 ; Pascanu et al. , 2013 ) . Gradients exponentially decrease ( or increase ) from the top layer to the bottom layer in a multilayer network . By extension , the current vanishing and exploding gradient problem also refers to extremely small ( or large ) gradient values . Following this explanation , several widely-used training techniques have been proposed to assist optimization . These studies can be roughly classified into four categories : initialization-based approaches ( Glorot & Bengio , 2010 ; He et al. , 2015 ; Zhang et al. , 2019 ) , activation-based approaches ( Hendrycks & Gimpel , 2016 ; Klambauer et al. , 2017 ) , normalization based approaches ( Ioffe & Szegedy , 2015 ; Salimans & Kingma , 2016 ; Ulyanov et al. , 2016 ; Lei Ba et al. , 2016 ; Nguyen & Salazar , 2019 ) , and skip-connection-based approaches ( He et al. , 2016 ; Huang et al. , 2017a ) . These techniques provide good alternatives to stabilize gradients and bring promising performance improvements . Despite much evidence showing the importance of steady gradient values , we are still curious about whether there are unexplored but important factors . To answer this question , we conduct extensive experiments and find some surprising phenomena . First , the accuracy curves of some models converge well with the remaining gradient vanishing and exploding problem . Second , we find several cases where models with similar gradient values show different convergence performance . These results indicate that gradient values may not be vital as much as we expect , and some hidden factors play a significant role in optimization . To answer this question , we explore other gradient features , such as covariance matrix , correlations , variance , and so on . These features are used to evaluate gradients from different perspectives . The covariance matrix evaluates the similarity between any two parameters , which is widely used in optimization studies ( Zhao & Zhang , 2015 ; Faghri et al. , 2020 ) . Gradient correlation is defined to evaluate the similarity between any two examples . Among them , experiments show that gradient correlation is an important factor . Compared to widely-explored 1We will release the code on publication . gradient values , gradient correlations are rarely studied . As gradient values affect the step size in optimization , vanishing gradients prevent models from changing weight values . In the worst case , this could completely stop training . Gradient correlations evaluate the randomness of gradient directions in Euclidean space . Lower gradient correlations indicate more “ random ” gradient directions and more “ conflicting ” parameter updates . For a better understanding , we visualize absolute gradient values and gradient correlations ( See Section 4 for more details ) . It illustrates that models with either small values or small correlations show worse convergence and generalization performance , indicating the importance of these two factors . Following the new finding , we take a further step to explore gradient correlations and gradient values in network architecture search . We first formulate two factors into a unified gradient-based kernel , which is defined as the average of a gradient dot-product matrix . Motivated by the observation that architectures with larger kernels at initialization tend to have better average results , we develop a lightweight network architecture search approach , called GT-NAS , which evaluates architectures according to the gradient-based kernel at initialization . Unlike other NAS approaches that evaluate architectures via full training , which takes massive computation resources , GT-NAS only relies on a few examples for kernel calculation . We first select top-k architectures with largest kernels as candidates , which are then trained to select the best one based on validation performance . In practice , k is usually set to be a very small value . Experiments show that GT-NAS achieves competitive results with extremely fast search on NAS-Bench-201 ( Ying et al. , 2019 ) , a NAS benchmark dataset . The low search cost allows us to apply NAS on diverse tasks . Specifically , the structure searched by the proposed policy outperforms the naive baseline without NAS on 4 datasets covering image classification and text classification . The main contributions are summarized as follows : • We propose that both gradient correlations and gradient values matter in optimization , which gives a new insight to understand and develop optimization techniques . • Following the new finding , we propose a gradient-based kernel approach for NAS , which is able to search for well-performing architectures efficiently . • Experiments show that GT-NAS achieves competitive results with orders of magnitude faster than the naive baseline . 2 RELATED WORK . Understanding and improving optimization Understanding and improving the optimization of deep networks has long been a hot research topic . Bengio et al . ( 1994 ) find the vanishing and exploding gradient problem in neural network training . To address this problem , a lot of promising approaches have been proposed in recent years , which can be classified into four research lines . The first research line focuses on initialization ( Sutskever et al. , 2013 ; Mishkin & Matas , 2016 ; Hanin & Rolnick , 2018 ) . Glorot & Bengio ( 2010 ) propose to control the variance of parameters via appropriate initialization . Following this work , several widely-used initialization approaches have been proposed , including Kaiming initialization ( He et al. , 2015 ) , Xaiver initinalization ( Glorot & Bengio , 2010 ) , and Fixup initialization ( Zhang et al. , 2019 ) . The second research line focuses on normalization ( Ioffe & Szegedy , 2015 ; Lei Ba et al. , 2016 ; Ulyanov et al. , 2016 ; Wu & He , 2018 ; Nguyen & Salazar , 2019 ) . These approaches aim to avoid the vanishing gradient by controlling the distribution of intermediate layers . The third is mainly based on activation functions to make the derivatives of activation less saturated to avoid the vanishing problem , including GELU activation ( Hendrycks & Gimpel , 2016 ) , SELU activation ( Klambauer et al. , 2017 ) , and so on . The motivation behind these approaches is to avoid unsteady derivatives of the activation with respect to the inputs . The fourth focuses on gradient clipping ( Pascanu et al. , 2013 ) . Gradient-based Kernel Our work is also related to gradient-based kernels ( Advani & Saxe , 2017 ; Jacot et al. , 2018 ) . NTK ( Jacot et al. , 2018 ) is a popular gradient-based kernel , defined as the Gram matrix of gradients . It is proposed to analyze model ’ s convergence and generalization . Following these studies , many researchers are devoted to understand current networks from the perspective of NTK ( Lee et al. , 2019 ; Hastie et al. , 2019 ; Allen-Zhu et al. , 2019 ; Arora et al. , 2019 ) . Du et al . ( 2019b ) use the Gram matrix of gradients to prove that for an m hidden node shallow neural network with ReLU activation , as long as m is large enough , randomly initialized gradient descent converges to a globally optimal solution at a linear convergence rate for the quadratic loss function . Following this work , Du et al . ( 2019a ) further expand this finding and prove that gradient descent achieves zero training loss in polynomial time for a deep over-parameterized neural network with residual connections . Network architecture search Neural architecture search aims to replace expert-designed networks with learned architectures ( Chang et al. , 2019 ; Li et al. , 2019 ; Zhou et al. , 2020 ; He et al. , 2020a ; Fang et al. , 2020 ; He et al. , 2020b ; You et al. , 2020 ; Alves & de Oliveira , 2020 ) . The first NAS research line mainly focuses on random search . The key idea is to randomly evaluate various architectures and select the best one based on their validation performance . However , despite promising results , many of them require thousands of GPU days to achieve desired results . To address this problem , Ying et al . ( 2019 ) propose a reinforcement learning based search policy which introduces an architecture generator with validation accuracy as reward . Another research line is based on evolution approaches . Real et al . ( 2019 ) propose a two-stage search policy . The first stage selects several well-performing parent architectures . The second stage applies mutation on these parent architectures to select the best one . Following this work , So et al . ( 2019 ) apply the evolution search on Transformer networks and achieve new state-of-the-art results on machine translation and language modeling tasks . Although these approaches can reduce exploration costs , the dependence on validation accuracy still leads to huge computation costs . To get rid of the dependence on validation accuracy , several studies ( Jiang et al. , 2019 ; Liu et al. , 2019 ; Dong & Yang , 2019 ; Zela et al. , 2020 ; Chu et al. , 2020 ) re-formulate the task in a differentiable manner and allow efficient search using gradient descent . Unlike these studies , we propose a lightweight NAS approach , which largely reduces evaluation cost . Furthermore , the new approach does not rely on additional search models , thus also saves memory usage . 3 NOTATIONS : GRADIENT VALUE AND GRADIENT CORRELATION . Assume g ( l ) i = { g ( l ) i,1 , · · · , g ( l ) i , r } is the gradient vector of example i with respect to all parameters in the l-th layer . r is the total number of parameters . We combine all gradient vectors together to construct a gradient matrix G ( l ) ∈ Rd×r where d refers to the number of training examples . We define a matrix P ( l ) to store the dot-product of gradients . The dot-product between two gradient vectors is : P ( l ) i , j = 〈G ( l ) i , G ( l ) j〉 . ( 1 ) We divide P ( l ) by the norm of gradient vectors to get a gradient correlation matrix C ( l ) : C ( l ) i , j = P ( l ) i , j ||G ( l ) i|| · ||G ( l ) j || . ( 2 ) Then , we use the average value of C ( l ) as the final correlation score : R = E ( C ( l ) ) . ( 3 ) Here we also evaluate the absolute gradient value : V = E ( |G ( l ) | ) . ( 4 ) Since it requires too much time to calculate R and V for plenty of training examples , we randomly sample µ examples to estimate R and V in implementation for fast calculation . The dot-product between vectors with too many elements will cause the overflow problem . We also randomly sample θ parameters when calculating R. To avoid the sampling bias that might affect our findings , we test different µ ∈ [ 50 , 100 , 200 , 300 , 400 , 500 ] and θ ∈ [ 50 , 100 , 200 , 300 , 400 , 500 ] , and observe the same phenomena . For simplicity , we show our findings in the next section with µ = 50 and θ = 50 .

Summary: This paper defines a new scoring function for efficiently selecting between different network architectures, without requiring the expensive training of the models.  This scoring function is based on the expected dot-products between the stochastic gradient vectors corresponding to different pairs of training examples.  The paper shows that this method can much more quickly select between different architectures than previous network architecture search (NAS) algorithms, while attaining relatively similar performance.