GradSign: Model Performance Inference with Theoretical Insights

A key challenge in neural architecture search ( NAS ) is quickly inferring the predictive performance of a broad spectrum of networks to discover statistically accurate and computationally efficient ones . We refer to this task as model performance inference ( MPI ) . The current practice for efficient MPI is gradient-based methods that leverage the gradients of a network at initialization to infer its performance . However , existing gradient-based methods rely only on heuristic metrics and lack the necessary theoretical foundations to consolidate their designs . We propose GradSign , an accurate , simple , and flexible metric for model performance inference with theoretical insights . The key idea behind GradSign is a quantity Ψ to analyze the optimization landscape of different networks at the granularity of individual training samples . Theoretically , we show that both the network ’ s training and true population losses are proportionally upper bounded by Ψ under reasonable assumptions . In addition , we design GradSign , an accurate and simple approximation of Ψ using the gradients of a network evaluated at a random initialization state . Evaluation on seven NAS benchmarks across three training datasets shows that GradSign generalizes well to real-world networks and consistently outperforms state-of-the-art gradient-based methods for MPI evaluated by Spearman ’ s ρ and Kendall ’ s Tau . Additionally , we integrate GradSign into four existing NAS algorithms and show that the GradSign-assisted NAS algorithms outperform their vanilla counterparts by improving the accuracies of best-discovered networks by up to 0.3 % , 1.1 % , and 1.0 % on three real-world tasks . 1 INTRODUCTION . As deep learning methods evolve , neural architectures have gotten progressively larger and more sophisticated ( He et al. , 2015 ; Ioffe & Szegedy , 2015 ; Krizhevsky et al. , 2017 ; Devlin et al. , 2019 ; Rumelhart et al. , 1986 ; Srivastava et al. , 2014 ; Kingma & Ba , 2017 ) , making it increasingly challenging to design model architectures that can achieve state-of-the-art predictive performance manually . To alleviate this challenge , recent work has proposed several approaches to automatically discover statistically accurate and computationally efficient neural architectures . The most common approach is neural architecture search ( NAS ) , which explores a comprehensive search space of potential network architectures based on compositions of basic network modules to discover architectures outperforming human-designed counterparts ( Liu et al. , 2018a ; Zoph & Le , 2016 ; Pham et al. , 2018 ) . A key challenge in existing automated approaches is quickly assessing the predictive performance of a diverse set of candidate architectures to discover performant ones . We refer to this task as model performance inference ( MPI ) . A straightforward approach for MPI is directly training candidate architectures on a dataset until convergence and recording the achieved training losses and validation accuracies ( Frankle & Carbin , 2018 ; Chen et al. , 2020a ; Liu et al. , 2018a ; Zoph & Le , 2016 ) . Though accurate , this approach is computationally prohibitive and can not scale to large networks or datasets . The current practice to efficient MPI is gradient-based methods that leverage the gradient information of a network at initialization to infer its predictive performance ( Lee et al. , 2018 ; Wang et al. , 2020 ; Tanaka et al. , 2020 ) . Compared to directly measuring the accuracy of candidate networks on a training dataset , gradient-based methods are computationally more efficient since they only require evaluating a mini-batch of gradients at initialization . However , existing gradient-based methods rely only on heuristic metrics and lack the necessary theoretical insights to consolidate their designs . In this paper , we propose GradSign , a simple yet accurate metric for MPI with theoretical foundations . GradSign is inspired by analyzing the optimization landscapes of individual training samples . GradSign takes as inputs a mini-batch of sample-wise gradients evaluated at a random initialization point and outputs the statistical evidence of a network that highly correlates to its well-trained predictive performance measured by accuracy on the entire dataset . Prior theoretical results ( Allen-Zhu et al. , 2019 ) show that for a sufficiently large neighborhood of a randomly initialized point , the optimization landscape is nearly convex and semi-smooth . To realize its potential for MPI , we generalize these results to sample-wise optimization landscapes and propose a quantity Ψ to measure the density of sample-wise local optima in the convex areas around a random initialization point . Additionally , we prove that both the training and population losses ( i.e. , generalization error ) of a network are proportionally upper bounded by Ψ2 under reasonable assumptions . Based on our theoretical results , we design GradSign , an accurate and simple approximation of Ψ , and empirically show that GradSign generalizes to realistic setups that may violate our assumptions . In addition , our method uses only the sample-wise gradient information at a random initialization point to provide statistical evidence to infer Ψ , which makes it efficient to compute and easy to implement . Extensive evaluation on seven NAS benchmarks ( i.e. , NAS-Bench-101 , NAS-Bench-201 , and five design spaces of NDS ) across three datasets ( i.e. , CIFAR-10 , CIFAR-100 , and ImageNet16-120 ) shows that GradSign consistently outperforms existing SOTA gradient-based methods in all circumstances . Furthermore , we integrate GradSign into existing NAS algorithms and show that the GradSign-assisted variants of these NAS algorithms lead to more accurate architectures . Contributions . This paper makes the following contributions : • We provide a new perspective to view the overall optimization landscape of a network as a combination of sample-wise optimization landscapes . Based on this insight , we introduce a new quantity Ψ that provides an upper bound on both a network ’ s training and population losses under reasonable assumptions . • To infer Ψ , we propose GradSign , an accurate and simple estimation of Ψ. GradSign enables fast and efficient MPI using only the gradients of a network at initialization . • We empirically show that our method could generalize reasonably well for modern network architectures by outperforming other SOTA gradient-based MPI methods . We additionally show that with the help of our method , SOTA NAS algorithms can be further improved . 2 RELATED WORK . Table 1 summarizes existing approaches to inferring the statistical performance of neural architectures . Sample-based methods assess the performance of a neural architecture by training it on a dataset . Though accurate , sample-based methods require a surrogate training procedure to evaluate each architecture . EconNAS ( Zhou et al. , 2020 ) mitigates the cost of training candidate architectures by reducing the number of training epochs , input dataset sizes , resolution of input images , and model sizes . Theory-based methods leverage recent advances in deep learning theory , such as Neural Tangent Kernel ( Jacot et al. , 2018 ) and Linear Region Analysis ( Serra et al. , 2018 ) , to assess a model ’ s accuracy ( Chen et al. , 2020a ; Mellor et al. , 2021 ; Park et al. , 2020 ) . In particular , NNGP ( Park et al. , 2020 ) infers a model ’ s performance by fitting its kernel regression parameters on a training dataset and evaluating its accuracy on a validation set , which alleviates the burden of training . As another example , Chen et al . ( 2020a ) utilizes the kernel condition number proposed in Xiao et al . ( 2020 ) , which can be theoretically proved to correlate to training convergence rate and generalization performance . However , this theoretical evidence is only guaranteed for extremely wide networks while having a specialized initialization mode . While the linear region analysis used in Mellor et al . ( 2021 ) , Lin et al . ( 2021 ) and Chen et al . ( 2020a ) is easy to implement , such technique is only applicable to networks with ReLU activations ( Agarap , 2018 ) . Learning-based methods train a separate network ( e.g. , graph neural networks ) to predict a network ’ s accuracy ( Liu et al. , 2018a ; Luo et al. , 2020 ; Dai et al. , 2019 ; Wen et al. , 2020 ; Chen et al. , 2020b ; Siems et al. , 2020 ) . Though these learned models can achieve high accuracies on a specific task , this approach requires constructing a training dataset with sampled architectures for each downstream task . As a result , existing learning-based methods are generally task-specific and computationally expensive . Gradient-based methods infer the statistical performance of a network by leveraging the gradient information at initialization , which can be easily obtained using an automated differentiation tool in today ’ s ML frameworks , such as PyTorch ( Paszke et al. , 2017 ) and TensorFlow ( Abadi et al. , 2016 ) . The weight-wise salience score computed by several pruning at initialization ( Lee et al. , 2018 ; Wang et al. , 2020 ; Tanaka et al. , 2020 ) methods can easily be adapted to MPI settings by summing scores up . Though lack of theoretical foundations , such migrations have been empirically proven to be effective as baselines in recent works ( Abdelfattah et al. , 2021a ; Mellor et al. , 2021 ; Lin et al. , 2021 ) . An alternative stream of work ( Turner et al. , 2019 ; 2021 ; Theis et al. , 2018 ) uses approximated secondorder gradients , known as empirical Fisher Information Matrix ( FIM ) , at a random initialization point to infer the performance of a network . Empirical FIM ( Martens , 2014 ) is a valid approximation of a model ’ s predictive performance only if the model ’ s parameters are a Maximum Likelihood Estimation ( MLE ) . However , this assumption is invalid at a random initialization point , making FIM-based algorithms inapplicable . The key difference between GradSign and existing gradient-based methods is that our method is based on a fine-grained analysis of sample-wise optimization landscapes rather than heuristic insights . In addition , GradSign also provides the first attempt for MPI by leveraging the optimization landscape properties contained in sample-wise gradient information , while prior gradient-based methods only focus on gradients evaluated in a full batch fashion . 2.1 NEURAL ARCHITECTURE SEARCH . Recent works ( He et al. , 2021 ; Cai et al. , 2019 ; 2018 ; Tan & Le , 2019 ; Howard et al. , 2019 ) have proposed several algorithms to explore a NAS search space and discover highly accurate networks . RS ( Bergstra & Bengio , 2012 ) is one of the baseline algorithms that generates and evaluates architectures randomly in the search space . REINFORCE ( Williams , 1992 ) moves a step forward by reframing NAS as a reinforcement learning task where accuracy is the reward and architecture generation is the policy action . Given limited computational resources , BOHB ( Falkner et al. , 2018 ) uses Bayesian Optimization ( BO ) to propose candidates while uses HyperBand ( HB ) ( Li et al. , 2017 ) for searching resource allocation . REA ( Real et al. , 2019 ) uses a simple yet effective evolutionary searching strategy that achieves state-of-the-art performance . GradSign is complementary to and can be combined with existing NAS algorithms . We integrate GradSign into the NAS algorithms mentioned above and show that GradSign can improve the search procedure of NAS algorithms on various real-world tasks . 2.2 OPTIMIZATION LANDSCAPE ANALYSIS . Inspired by the fact that over-parameterized networks always find a remarkable fit for a training dataset ( Zhang et al. , 2016 ) , optimization landscape analysis has been one of the main focuses in deep learning theory ( Brutzkus & Globerson , 2017 ; Du et al. , 2018 ; Ge et al. , 2017 ; Li & Yuan , 2017 ; Soltanolkotabi , 2017 ; Allen-Zhu et al. , 2019 ) . Even though existing theoretical results for optimization landscape analysis rely on strict assumptions on the landscape ’ s smoothness , convexity , and initialization point , we can leverage theoretical insights to guide the design of GradSign . In θ * θ * 1 l ( fθ ( x1 ) , y1 ) θ * 2 l ( fθ ( x2 ) , y2 ) |θ * 1 − θ * 2 | θ l ( a ) Optimization landscape with sparser sample-wise local optima corresponding to worse J ( θ∗ ) . θ l θ * θ * 1 l ( fθ ( x1 ) , y1 ) θ * 2 l ( fθ ( x2 ) , y2 ) |θ * 1 − θ * 2 | ( b ) Optimization landscape with denser sample-wise local optima corresponding to better J ( θ∗ ) . Figure 1 : Illustration of our theoretical insight that denser sample-wise local optima indicate lower training losses . As the distances ( |θ∗1 − θ∗2 | , shown in red ) between the local optima across samples reduce , there is a higher probability that the gradients of different samples have the same sign at a random initialization point , shown as the green areas . addition , SGD-based optimizers trained from randomly initialized states hardly encounter nonsmoothness or non-convexity in practice for varieties of architectures ( Goodfellow et al. , 2014 ) . Furthermore , Allen-Zhu et al . ( 2019 ) provides theoretical evidence that for a sufficiently large neighborhood of a randomly initialized point , the optimization landscape is nearly convex and semismooth . Different from existing optimization landscape analyses depending on objectives evaluated across a mini-batch of training samples , we propose a new perspective that decomposes a mini-batch objective into the aggregation of sample-wise optimization landscapes . To the best of our knowledge , our work is the first attempt to inferring models ’ performance by leveraging sample-wise optimization landscapes .

Summary: This work discusses the problem of neural architecture search. While existing gradient methods are based on heuristics, this work proposes a metric called Gradsign for model performance inference, which provides some theoretical guarantees and performs well in practise.   Authors compare gradsign to a number of state of the art methods using 3 benchmarks involving CIFAR10, CIFAR100, and ImageNet16-120 datasets. Gradsign shows better performance compared to these methods.