GradSign: Model Performance Inference with Theoretical Insights

A key challenge in neural architecture search ( NAS ) is quickly inferring the predictive performance of a broad spectrum of networks to discover statistically accurate and computationally efficient ones . We refer to this task as model performance inference ( MPI ) . The current practice for efficient MPI is gradient-based methods that leverage the gradients of a network at initialization to infer its performance . However , existing gradient-based methods rely only on heuristic metrics and lack the necessary theoretical foundations to consolidate their designs . We propose GradSign , an accurate , simple , and flexible metric for model performance inference with theoretical insights . The key idea behind GradSign is a quantity Ψ to analyze the optimization landscape of different networks at the granularity of individual training samples . Theoretically , we show that both the network ’ s training and true population losses are proportionally upper bounded by Ψ under reasonable assumptions . In addition , we design GradSign , an accurate and simple approximation of Ψ using the gradients of a network evaluated at a random initialization state . Evaluation on seven NAS benchmarks across three training datasets shows that GradSign generalizes well to real-world networks and consistently outperforms state-of-the-art gradient-based methods for MPI evaluated by Spearman ’ s ρ and Kendall ’ s Tau . Additionally , we integrate GradSign into four existing NAS algorithms and show that the GradSign-assisted NAS algorithms outperform their vanilla counterparts by improving the accuracies of best-discovered networks by up to 0.3 % , 1.1 % , and 1.0 % on three real-world tasks . 1 INTRODUCTION . As deep learning methods evolve , neural architectures have gotten progressively larger and more sophisticated ( He et al. , 2015 ; Ioffe & Szegedy , 2015 ; Krizhevsky et al. , 2017 ; Devlin et al. , 2019 ; Rumelhart et al. , 1986 ; Srivastava et al. , 2014 ; Kingma & Ba , 2017 ) , making it increasingly challenging to design model architectures that can achieve state-of-the-art predictive performance manually . To alleviate this challenge , recent work has proposed several approaches to automatically discover statistically accurate and computationally efficient neural architectures . The most common approach is neural architecture search ( NAS ) , which explores a comprehensive search space of potential network architectures based on compositions of basic network modules to discover architectures outperforming human-designed counterparts ( Liu et al. , 2018a ; Zoph & Le , 2016 ; Pham et al. , 2018 ) . A key challenge in existing automated approaches is quickly assessing the predictive performance of a diverse set of candidate architectures to discover performant ones . We refer to this task as model performance inference ( MPI ) . A straightforward approach for MPI is directly training candidate architectures on a dataset until convergence and recording the achieved training losses and validation accuracies ( Frankle & Carbin , 2018 ; Chen et al. , 2020a ; Liu et al. , 2018a ; Zoph & Le , 2016 ) . Though accurate , this approach is computationally prohibitive and can not scale to large networks or datasets . The current practice to efficient MPI is gradient-based methods that leverage the gradient information of a network at initialization to infer its predictive performance ( Lee et al. , 2018 ; Wang et al. , 2020 ; Tanaka et al. , 2020 ) . Compared to directly measuring the accuracy of candidate networks on a training dataset , gradient-based methods are computationally more efficient since they only require evaluating a mini-batch of gradients at initialization . However , existing gradient-based methods rely only on heuristic metrics and lack the necessary theoretical insights to consolidate their designs . In this paper , we propose GradSign , a simple yet accurate metric for MPI with theoretical foundations . GradSign is inspired by analyzing the optimization landscapes of individual training samples . GradSign takes as inputs a mini-batch of sample-wise gradients evaluated at a random initialization point and outputs the statistical evidence of a network that highly correlates to its well-trained predictive performance measured by accuracy on the entire dataset . Prior theoretical results ( Allen-Zhu et al. , 2019 ) show that for a sufficiently large neighborhood of a randomly initialized point , the optimization landscape is nearly convex and semi-smooth . To realize its potential for MPI , we generalize these results to sample-wise optimization landscapes and propose a quantity Ψ to measure the density of sample-wise local optima in the convex areas around a random initialization point . Additionally , we prove that both the training and population losses ( i.e. , generalization error ) of a network are proportionally upper bounded by Ψ2 under reasonable assumptions . Based on our theoretical results , we design GradSign , an accurate and simple approximation of Ψ , and empirically show that GradSign generalizes to realistic setups that may violate our assumptions . In addition , our method uses only the sample-wise gradient information at a random initialization point to provide statistical evidence to infer Ψ , which makes it efficient to compute and easy to implement . Extensive evaluation on seven NAS benchmarks ( i.e. , NAS-Bench-101 , NAS-Bench-201 , and five design spaces of NDS ) across three datasets ( i.e. , CIFAR-10 , CIFAR-100 , and ImageNet16-120 ) shows that GradSign consistently outperforms existing SOTA gradient-based methods in all circumstances . Furthermore , we integrate GradSign into existing NAS algorithms and show that the GradSign-assisted variants of these NAS algorithms lead to more accurate architectures . Contributions . This paper makes the following contributions : • We provide a new perspective to view the overall optimization landscape of a network as a combination of sample-wise optimization landscapes . Based on this insight , we introduce a new quantity Ψ that provides an upper bound on both a network ’ s training and population losses under reasonable assumptions . • To infer Ψ , we propose GradSign , an accurate and simple estimation of Ψ. GradSign enables fast and efficient MPI using only the gradients of a network at initialization . • We empirically show that our method could generalize reasonably well for modern network architectures by outperforming other SOTA gradient-based MPI methods . We additionally show that with the help of our method , SOTA NAS algorithms can be further improved . 2 RELATED WORK . Table 1 summarizes existing approaches to inferring the statistical performance of neural architectures . Sample-based methods assess the performance of a neural architecture by training it on a dataset . Though accurate , sample-based methods require a surrogate training procedure to evaluate each architecture . EconNAS ( Zhou et al. , 2020 ) mitigates the cost of training candidate architectures by reducing the number of training epochs , input dataset sizes , resolution of input images , and model sizes . Theory-based methods leverage recent advances in deep learning theory , such as Neural Tangent Kernel ( Jacot et al. , 2018 ) and Linear Region Analysis ( Serra et al. , 2018 ) , to assess a model ’ s accuracy ( Chen et al. , 2020a ; Mellor et al. , 2021 ; Park et al. , 2020 ) . In particular , NNGP ( Park et al. , 2020 ) infers a model ’ s performance by fitting its kernel regression parameters on a training dataset and evaluating its accuracy on a validation set , which alleviates the burden of training . As another example , Chen et al . ( 2020a ) utilizes the kernel condition number proposed in Xiao et al . ( 2020 ) , which can be theoretically proved to correlate to training convergence rate and generalization performance . However , this theoretical evidence is only guaranteed for extremely wide networks while having a specialized initialization mode . While the linear region analysis used in Mellor et al . ( 2021 ) , Lin et al . ( 2021 ) and Chen et al . ( 2020a ) is easy to implement , such technique is only applicable to networks with ReLU activations ( Agarap , 2018 ) . Learning-based methods train a separate network ( e.g. , graph neural networks ) to predict a network ’ s accuracy ( Liu et al. , 2018a ; Luo et al. , 2020 ; Dai et al. , 2019 ; Wen et al. , 2020 ; Chen et al. , 2020b ; Siems et al. , 2020 ) . Though these learned models can achieve high accuracies on a specific task , this approach requires constructing a training dataset with sampled architectures for each downstream task . As a result , existing learning-based methods are generally task-specific and computationally expensive . Gradient-based methods infer the statistical performance of a network by leveraging the gradient information at initialization , which can be easily obtained using an automated differentiation tool in today ’ s ML frameworks , such as PyTorch ( Paszke et al. , 2017 ) and TensorFlow ( Abadi et al. , 2016 ) . The weight-wise salience score computed by several pruning at initialization ( Lee et al. , 2018 ; Wang et al. , 2020 ; Tanaka et al. , 2020 ) methods can easily be adapted to MPI settings by summing scores up . Though lack of theoretical foundations , such migrations have been empirically proven to be effective as baselines in recent works ( Abdelfattah et al. , 2021a ; Mellor et al. , 2021 ; Lin et al. , 2021 ) . An alternative stream of work ( Turner et al. , 2019 ; 2021 ; Theis et al. , 2018 ) uses approximated secondorder gradients , known as empirical Fisher Information Matrix ( FIM ) , at a random initialization point to infer the performance of a network . Empirical FIM ( Martens , 2014 ) is a valid approximation of a model ’ s predictive performance only if the model ’ s parameters are a Maximum Likelihood Estimation ( MLE ) . However , this assumption is invalid at a random initialization point , making FIM-based algorithms inapplicable . The key difference between GradSign and existing gradient-based methods is that our method is based on a fine-grained analysis of sample-wise optimization landscapes rather than heuristic insights . In addition , GradSign also provides the first attempt for MPI by leveraging the optimization landscape properties contained in sample-wise gradient information , while prior gradient-based methods only focus on gradients evaluated in a full batch fashion . 2.1 NEURAL ARCHITECTURE SEARCH . Recent works ( He et al. , 2021 ; Cai et al. , 2019 ; 2018 ; Tan & Le , 2019 ; Howard et al. , 2019 ) have proposed several algorithms to explore a NAS search space and discover highly accurate networks . RS ( Bergstra & Bengio , 2012 ) is one of the baseline algorithms that generates and evaluates architectures randomly in the search space . REINFORCE ( Williams , 1992 ) moves a step forward by reframing NAS as a reinforcement learning task where accuracy is the reward and architecture generation is the policy action . Given limited computational resources , BOHB ( Falkner et al. , 2018 ) uses Bayesian Optimization ( BO ) to propose candidates while uses HyperBand ( HB ) ( Li et al. , 2017 ) for searching resource allocation . REA ( Real et al. , 2019 ) uses a simple yet effective evolutionary searching strategy that achieves state-of-the-art performance . GradSign is complementary to and can be combined with existing NAS algorithms . We integrate GradSign into the NAS algorithms mentioned above and show that GradSign can improve the search procedure of NAS algorithms on various real-world tasks . 2.2 OPTIMIZATION LANDSCAPE ANALYSIS . Inspired by the fact that over-parameterized networks always find a remarkable fit for a training dataset ( Zhang et al. , 2016 ) , optimization landscape analysis has been one of the main focuses in deep learning theory ( Brutzkus & Globerson , 2017 ; Du et al. , 2018 ; Ge et al. , 2017 ; Li & Yuan , 2017 ; Soltanolkotabi , 2017 ; Allen-Zhu et al. , 2019 ) . Even though existing theoretical results for optimization landscape analysis rely on strict assumptions on the landscape ’ s smoothness , convexity , and initialization point , we can leverage theoretical insights to guide the design of GradSign . In θ * θ * 1 l ( fθ ( x1 ) , y1 ) θ * 2 l ( fθ ( x2 ) , y2 ) |θ * 1 − θ * 2 | θ l ( a ) Optimization landscape with sparser sample-wise local optima corresponding to worse J ( θ∗ ) . θ l θ * θ * 1 l ( fθ ( x1 ) , y1 ) θ * 2 l ( fθ ( x2 ) , y2 ) |θ * 1 − θ * 2 | ( b ) Optimization landscape with denser sample-wise local optima corresponding to better J ( θ∗ ) . Figure 1 : Illustration of our theoretical insight that denser sample-wise local optima indicate lower training losses . As the distances ( |θ∗1 − θ∗2 | , shown in red ) between the local optima across samples reduce , there is a higher probability that the gradients of different samples have the same sign at a random initialization point , shown as the green areas . addition , SGD-based optimizers trained from randomly initialized states hardly encounter nonsmoothness or non-convexity in practice for varieties of architectures ( Goodfellow et al. , 2014 ) . Furthermore , Allen-Zhu et al . ( 2019 ) provides theoretical evidence that for a sufficiently large neighborhood of a randomly initialized point , the optimization landscape is nearly convex and semismooth . Different from existing optimization landscape analyses depending on objectives evaluated across a mini-batch of training samples , we propose a new perspective that decomposes a mini-batch objective into the aggregation of sample-wise optimization landscapes . To the best of our knowledge , our work is the first attempt to inferring models ’ performance by leveraging sample-wise optimization landscapes .

Summary: Model performance inference is a key challenge in neural architecture search. This paper introduces GradSign, an accurate, simple, and flexible metric for model performance inference. GradSign approximately analyzes the optimization landscape of different networks at the granularity of individual training samples using the gradients evaluated at a random initialization state. Experimental results show that GradSign can generalize well to real-world networks and outperform state-of-the-art gradient-based methods for model performance inference.