Neural Capacitance: A New Perspective of Neural Network Selection via Edge Dynamics

Efficient model selection for identifying a suitable pre-trained neural network to a downstream task is a fundamental yet challenging task in deep learning . Current practice requires expensive computational costs in model training for performance prediction . In this paper , we propose a novel framework for neural network selection by analyzing the governing dynamics over synaptic connections ( edges ) during training . Our framework is built on the fact that back-propagation during neural network training is equivalent to the dynamical evolution of synaptic connections . Therefore , a converged neural network is associated with an equilibrium state of a networked system composed of those edges . To this end , we construct a network mapping φ , converting a neural network GA to a directed line graph GB that is defined on those edges in GA. Next , we derive a neural capacitance metric βeff as a predictive measure universally capturing the generalization capability of GA on the downstream task using only a handful of early training results . We carried out extensive experiments using 17 popular pre-trained ImageNet models and five benchmark datasets , including CIFAR10 , CIFAR100 , SVHN , Fashion MNIST and Birds , to evaluate the fine-tuning performance of our framework . Our neural capacitance metric is shown to be a powerful indicator for model selection based only on early training results and is more efficient than state-of-the-art methods . 1 INTRODUCTION . Leveraging a pre-trained neural network ( i.e. , a source model ) and fine-tuning it to solve a target task is a common and effective practice in deep learning , such as transfer learning . Transfer learning has been widely used to solve complex tasks in text and vision domains . In vision , models trained on ImageNet are leveraged to solve diverse tasks such as image classification and object detection . In text , language models that are trained on a large amount of public data comprising of books , Wikipedia etc are employed to solve tasks such as classification and language generation . Although such technique can achieve good performance on a target task , a fundamental yet challenging problem is how to select a suitable pre-trained model from a pool of candidates in an efficient manner . The naive solution of training each candidate fully with the target data can find the best pre-trained model but is infeasible due to considerable consumption on time and computation resources . This challenge motivates the need for an efficient predictive measure to capture the performance of a pre-trained model on the target task based only on early training results ( e.g. , predicting final model performance based on the statistics obtained from first few training epochs ) . In order to implement an efficient neural network ( NN ) model selection , this paper proposes a novel framework to forecast the predictive ability of a model with its cumulative information in the early phase of NN training , as practised in learning curve prediction ( Domhan et al. , 2015 ; Chandrashekaran & Lane , 2017 ; Baker et al. , 2017 ; Wistuba & Pedapati , 2020 ) . Most prior work on learning curve prediction aims to capture the trajectory of learning curves with a regression function of models ’ validation accuracy . Some of the previous algorithms developed in this field require training data from additional learning curves to train the predictors ( Chandrashekaran & Lane , 2017 ; Baker et al. , 2017 ; Wistuba & Pedapati , 2020 ) . On the other hand , our model does not require any such data . It solely relies on the NN architecture . Ranking models according to their final accuracy after fine-tuning is a lot more challenging as the learning curves are very similar to each other . The entire NN training process involves iterative updates of the weights of synaptic connections , according to one particular optimization algorithm , e.g. , gradient descent or stochastic gradient descent ( SGD ) ( Bottou , 2012 ; LeCun et al. , 2015 ) . In essence , many factors contribute to impact how weights are updated , including the training data , the neural architecture , the loss function , and the optimization algorithm . Moreover , weights evolving during NN training in many aspects can be viewed as a discrete dynamical system . The perspective of viewing NN training as a dynamical system has been studied by the community ( Mei et al. , 2018 ; Chang et al. , 2018 ; Banburski et al. , 2019 ; Dogra , 2020 ; Tano et al. , 2020 ; Dogra & Redman , 2020 ; Feng & Tu , 2021 ) , and many attempted to make some theoretical explanation of the convergence rate and generalization error bounds . In this paper , we will provide the first attempt in exploring its power in neural model selection . One limitation of current approaches is that they concentrated on the macroscopic and collective behavior of the system , but lacks a dedicated examination of the individual interactions between the trainable weights or synaptic connections , which are crucial in understanding of the dependency of these weights , and how they co-evolve during training . To fill the gap , we study the system from a microscopic perspective , build edge dynamics of synaptic connections from SGD in terms of differential equations , from which we build an associated network as well . The edge dynamics induced from SGD is nonlinear and highly coupling . It will be very challenging to solve , considering millions of weights in many convolutional neural networks ( CNNs ) , e.g. , 16M weights in MobileNet ( Howard et al. , 2017 ) and 528M in VGG16 ( Simonyan & Zisserman , 2014 ) . Gao et al . ( 2016 ) proposed a universal topological metric for the associated network to decouple the system . The metric will be used for model selection in our approach , and it is shown to be powerful in search of the best predictive model . We illustrate our proposed framework in Fig.1 . The main contributions of our framework can be summarized as follows : • View NN training as a dynamical system over synaptic connections , and first time investigate the interactions of synaptic connections in a microscopic perspective . • Propose neural capacitance metric βeff for neural network model selection . • Empirical results of 17 pre-trained models on five benchmark datasets show that our βeff based approach outperforms current learning curve prediction approaches . • For rank prediction according to the performance of pre-trained models , our approach improves by 9.1/38.3/12.4/65.3/40.1 % on CIFAR10/CIFAR100/SVHN/Fashion MNIST/Birds over the best baseline with observations from learning curves of length only 5 epochs . 2 RELATED WORK . Learning Curve Prediction . Chandrashekaran & Lane ( 2017 ) treated the current learning curve ( LC ) as an affine transformation of previous LCs . They built an ensemble of transformations employing previous LCs and the first few epochs of the current LC to predict the final accuracy of the current LC . Baker et al . ( 2017 ) proposed an SVM based LC predictor using features extracted from previous LCs , including the architecture information such as number of layers , parameters , and training technique such as learning rate and learning rate decay . A separate SVM is used to predict the accuracy of an LC at a particular epoch . Domhan et al . ( 2015 ) trained an ensemble of parametric functions that observe the first few epochs of an LC and extrapolate it . Klein et al . ( 2017a ) devised a Bayesian NN to model the functions that Domhan formulated to capture the structure of the LCs more effectively . Wistuba & Pedapati ( 2020 ) developed a transfer learning based predictor that was trained on LCs generated from other datasets . It is a NN based predictor that leverages architecture and dataset embeddings to capture the similarities between the architectures of various models and also the other datasets that it was trained on . Dynamical System View of NNs . There are many efforts to study the dynamics of NN training . Some prior work on SGD dynamics for NNs generally have a pre-assumption of the input distribution or how the labels are generated . They obtained global convergence for shallow NNs ( Tian , 2017 ; Banburski et al. , 2019 ) . System identification itself is a complicated task ( Haykin , 2010 ; Lillicrap et al. , 2020 ) . In studying the generalisation phenomenon of deep NNs , Goldt et al . ( 2019 ) formulated SGD with a set of differential equations . But , it is limited to over-parameterised two-layer NNs under the teacher-student framework . The teacher network determines how the labels are generated . Also , some interesting phenomena ( Frankle et al. , 2020 ) are observed during the early phase of NN training , such as trainable sparse sub-networks emerge ( Frankle et al. , 2019 ) , gradient descent moves into a small subspace ( Gur-Ari et al. , 2018 ) , and there exists a critical effective connection between layers ( Achille et al. , 2019 ) . Bhardwaj et al . ( 2021 ) built a nice connection between architectures ( with concatenation-type skip connections ) and the performance , and proposed a new topological metric to identify NNs with similar accuracy . Many of these studies are built on dynamical system and network science . It will be a promising direction to study deep learning mechanism . 3 PRELIMINARIES . Dynamical System of a Network . Many real complex systems , e.g. , plant-pollinator interactions ( Waser & Ollerton , 2006 ) and the spread of COVID-19 ( Thurner et al. , 2020 ) , can be described with networks ( Mitchell , 2006 ; Barabási & Pósfai , 2016 ) . Let G = ( V , E ) be a network with node set V and edge set E. Assuming n = |V | , the interactions between nodes can be formulated as a set of differential equations ẋi = f ( xi ) + ∑ j∈V Pijg ( xi , xj ) , ∀i ∈ V , ( 1 ) where xi is the state of node i . In real systems , it could be the abundance of a plant in ecological network , the infection rate of a person in epidemic network , or the expression level of a gene in regulatory network . The term P is the adjacency matrix of G , where the entry Pij indicates the interaction strength between nodes i and j . The functions f ( · ) and g ( · , · ) capture the internal and external impacts on node i , respectively . Usually , they are nonlinear . Let x = ( x1 , x2 , . . . , xn ) . For a small network , given an initial state , one can run a forward simulation for an equilibrium state x∗ , such that ẋ∗i = f ( x ∗ i ) + ∑ j∈V Pijg ( x ∗ i , x ∗ j ) = 0 . However , when the size of the system goes up to millions or even billions , it will pose a big challenge to solve the coupled differential equations . The problem can be efficiently addressed by employing a mean-field technique ( Gao et al. , 2016 ) , where a linear operator LP ( · ) is introduced to decouple the system . In specific , the operator depends on the adjacency matrix P and is defined as LP ( z ) = 1TPz 1TP1 , ( 2 ) where z ∈ Rn . Let δin = P1 be nodes ’ in-degrees and δout = 1TP be nodes ’ out-degrees . For a weighted G , the degrees are weighted as well . Applying LP ( · ) to δin , it gives βeff = LP ( δin ) = 1TPδin 1T δin = δToutδin 1T δin , ( 3 ) which proves to be a powerful metric to measure the resilience of networks , and has been applied to make reliable inferences from incomplete networks ( Jiang et al. , 2020b ; a ) . We use it to measure the predictive ability of a NN ( see Section 4.3 ) , whose training in essence is a dynamical system . For an overview of the related technique , the readers are referred to Appendix H. NN Training is a Dynamical System . Conventionally , training a NN is a nonlinear optimization problem . Because of the hierarchical structure of NNs , the training procedure is implemented by two alternate procedures : forward-propagation ( FP ) and back-propagation ( BP ) , as described in Fig.1 ( a ) . During FP , data goes through the input layer , hidden layers , up to the output layer , which produces the predictions of the input data . The differences between the outputs and the labels of the input data are used to define an objective function C , a.k.a training error function . BP proceeds to minimize C , in a reverse way as did in FP , by propagating the error from the output layer down to the input layer . The trainable weights of synaptic connections are updated accordingly . Let GA be a NN , w be the flattened weight vector for GA , and z be the set of activation values . As a whole , the training of GA can be described with two coupled dynamics : A on GA , and B on GB , where nodes in GA are neurons , and nodes in GB are the synaptic connections . The coupling relation arises from the strong inter-dependency between z andw : the states z ( activation values or activation gradients ) of GA are the parameters of B , and the states w of GB are the trainable parameters of GA . If we put the whole training process in the context of networked systems , A denotes a node dynamics because the states of nodes evolve during FP , and B expresses an edge dynamics because of the updates of edge weights during BP ( Mei et al. , 2018 ; Poggio et al. , 2020a ; b ) . Mathematically , we formulate the node and edge dynamics based on the gradients of C : ( A ) dz/dt ≈ hA ( z , t ; w ) = −∇zC ( z ( t ) ) , ( 4 ) ( B ) dw/dt ≈ hB ( w , t ; z ) = −∇wC ( w ( t ) ) , ( 5 ) where t denotes the training step . Let a ( ` ) i be the pre-activation of node i on layer ` , and σ ` ( · ) be the activation function of layer ` . Usually , the output activation function is a softmax . The hierarchical structure of GA exerts some constraints over z for neighboring layers , i.e. , z ( ` ) i = σ ` ( a ( ` ) i ) , 1 ≤ i ≤ n ` , ∀1 ≤ ` < L and z ( L ) k = exp { a ( L ) k } / ∑ j exp { a ( L ) j } , 1 ≤ k ≤ nL , where n ` is the total number of neurons on layer ` , and GA has L+ 1 layers . It also presents a dependency between z and w. For example , when GA is an MLP without bias , a ( ` ) i = w ( ` ) T i z ( ` −1 ) , which builds an interconnection from GA to GB . It is obvious , givenw , the activation z satisfying all these constraints , is also a fixed point of A . Meanwhile , an equilibrium state of B provides a set of optimal weights for GA .

Summary: This paper proposes a framework to select the neural networks for downstream tasks. To identify the better generalization model, the authors propose a new metric (Neural Capacitance, NCP) to predict precise learning curves. And the authors provide the theoretical explanations for NCP.  Then the authors have verified the advantages of the proposed method when being applied to different datasets (CIFAR10, CIFAR100, SVHN, Fashion MNIST, Birds) with 17 CNN models.