Neural Capacitance: A New Perspective of Neural Network Selection via Edge Dynamics

Efficient model selection for identifying a suitable pre-trained neural network to a downstream task is a fundamental yet challenging task in deep learning . Current practice requires expensive computational costs in model training for performance prediction . In this paper , we propose a novel framework for neural network selection by analyzing the governing dynamics over synaptic connections ( edges ) during training . Our framework is built on the fact that back-propagation during neural network training is equivalent to the dynamical evolution of synaptic connections . Therefore , a converged neural network is associated with an equilibrium state of a networked system composed of those edges . To this end , we construct a network mapping φ , converting a neural network GA to a directed line graph GB that is defined on those edges in GA. Next , we derive a neural capacitance metric βeff as a predictive measure universally capturing the generalization capability of GA on the downstream task using only a handful of early training results . We carried out extensive experiments using 17 popular pre-trained ImageNet models and five benchmark datasets , including CIFAR10 , CIFAR100 , SVHN , Fashion MNIST and Birds , to evaluate the fine-tuning performance of our framework . Our neural capacitance metric is shown to be a powerful indicator for model selection based only on early training results and is more efficient than state-of-the-art methods . 1 INTRODUCTION . Leveraging a pre-trained neural network ( i.e. , a source model ) and fine-tuning it to solve a target task is a common and effective practice in deep learning , such as transfer learning . Transfer learning has been widely used to solve complex tasks in text and vision domains . In vision , models trained on ImageNet are leveraged to solve diverse tasks such as image classification and object detection . In text , language models that are trained on a large amount of public data comprising of books , Wikipedia etc are employed to solve tasks such as classification and language generation . Although such technique can achieve good performance on a target task , a fundamental yet challenging problem is how to select a suitable pre-trained model from a pool of candidates in an efficient manner . The naive solution of training each candidate fully with the target data can find the best pre-trained model but is infeasible due to considerable consumption on time and computation resources . This challenge motivates the need for an efficient predictive measure to capture the performance of a pre-trained model on the target task based only on early training results ( e.g. , predicting final model performance based on the statistics obtained from first few training epochs ) . In order to implement an efficient neural network ( NN ) model selection , this paper proposes a novel framework to forecast the predictive ability of a model with its cumulative information in the early phase of NN training , as practised in learning curve prediction ( Domhan et al. , 2015 ; Chandrashekaran & Lane , 2017 ; Baker et al. , 2017 ; Wistuba & Pedapati , 2020 ) . Most prior work on learning curve prediction aims to capture the trajectory of learning curves with a regression function of models ’ validation accuracy . Some of the previous algorithms developed in this field require training data from additional learning curves to train the predictors ( Chandrashekaran & Lane , 2017 ; Baker et al. , 2017 ; Wistuba & Pedapati , 2020 ) . On the other hand , our model does not require any such data . It solely relies on the NN architecture . Ranking models according to their final accuracy after fine-tuning is a lot more challenging as the learning curves are very similar to each other . The entire NN training process involves iterative updates of the weights of synaptic connections , according to one particular optimization algorithm , e.g. , gradient descent or stochastic gradient descent ( SGD ) ( Bottou , 2012 ; LeCun et al. , 2015 ) . In essence , many factors contribute to impact how weights are updated , including the training data , the neural architecture , the loss function , and the optimization algorithm . Moreover , weights evolving during NN training in many aspects can be viewed as a discrete dynamical system . The perspective of viewing NN training as a dynamical system has been studied by the community ( Mei et al. , 2018 ; Chang et al. , 2018 ; Banburski et al. , 2019 ; Dogra , 2020 ; Tano et al. , 2020 ; Dogra & Redman , 2020 ; Feng & Tu , 2021 ) , and many attempted to make some theoretical explanation of the convergence rate and generalization error bounds . In this paper , we will provide the first attempt in exploring its power in neural model selection . One limitation of current approaches is that they concentrated on the macroscopic and collective behavior of the system , but lacks a dedicated examination of the individual interactions between the trainable weights or synaptic connections , which are crucial in understanding of the dependency of these weights , and how they co-evolve during training . To fill the gap , we study the system from a microscopic perspective , build edge dynamics of synaptic connections from SGD in terms of differential equations , from which we build an associated network as well . The edge dynamics induced from SGD is nonlinear and highly coupling . It will be very challenging to solve , considering millions of weights in many convolutional neural networks ( CNNs ) , e.g. , 16M weights in MobileNet ( Howard et al. , 2017 ) and 528M in VGG16 ( Simonyan & Zisserman , 2014 ) . Gao et al . ( 2016 ) proposed a universal topological metric for the associated network to decouple the system . The metric will be used for model selection in our approach , and it is shown to be powerful in search of the best predictive model . We illustrate our proposed framework in Fig.1 . The main contributions of our framework can be summarized as follows : • View NN training as a dynamical system over synaptic connections , and first time investigate the interactions of synaptic connections in a microscopic perspective . • Propose neural capacitance metric βeff for neural network model selection . • Empirical results of 17 pre-trained models on five benchmark datasets show that our βeff based approach outperforms current learning curve prediction approaches . • For rank prediction according to the performance of pre-trained models , our approach improves by 9.1/38.3/12.4/65.3/40.1 % on CIFAR10/CIFAR100/SVHN/Fashion MNIST/Birds over the best baseline with observations from learning curves of length only 5 epochs . 2 RELATED WORK . Learning Curve Prediction . Chandrashekaran & Lane ( 2017 ) treated the current learning curve ( LC ) as an affine transformation of previous LCs . They built an ensemble of transformations employing previous LCs and the first few epochs of the current LC to predict the final accuracy of the current LC . Baker et al . ( 2017 ) proposed an SVM based LC predictor using features extracted from previous LCs , including the architecture information such as number of layers , parameters , and training technique such as learning rate and learning rate decay . A separate SVM is used to predict the accuracy of an LC at a particular epoch . Domhan et al . ( 2015 ) trained an ensemble of parametric functions that observe the first few epochs of an LC and extrapolate it . Klein et al . ( 2017a ) devised a Bayesian NN to model the functions that Domhan formulated to capture the structure of the LCs more effectively . Wistuba & Pedapati ( 2020 ) developed a transfer learning based predictor that was trained on LCs generated from other datasets . It is a NN based predictor that leverages architecture and dataset embeddings to capture the similarities between the architectures of various models and also the other datasets that it was trained on . Dynamical System View of NNs . There are many efforts to study the dynamics of NN training . Some prior work on SGD dynamics for NNs generally have a pre-assumption of the input distribution or how the labels are generated . They obtained global convergence for shallow NNs ( Tian , 2017 ; Banburski et al. , 2019 ) . System identification itself is a complicated task ( Haykin , 2010 ; Lillicrap et al. , 2020 ) . In studying the generalisation phenomenon of deep NNs , Goldt et al . ( 2019 ) formulated SGD with a set of differential equations . But , it is limited to over-parameterised two-layer NNs under the teacher-student framework . The teacher network determines how the labels are generated . Also , some interesting phenomena ( Frankle et al. , 2020 ) are observed during the early phase of NN training , such as trainable sparse sub-networks emerge ( Frankle et al. , 2019 ) , gradient descent moves into a small subspace ( Gur-Ari et al. , 2018 ) , and there exists a critical effective connection between layers ( Achille et al. , 2019 ) . Bhardwaj et al . ( 2021 ) built a nice connection between architectures ( with concatenation-type skip connections ) and the performance , and proposed a new topological metric to identify NNs with similar accuracy . Many of these studies are built on dynamical system and network science . It will be a promising direction to study deep learning mechanism . 3 PRELIMINARIES . Dynamical System of a Network . Many real complex systems , e.g. , plant-pollinator interactions ( Waser & Ollerton , 2006 ) and the spread of COVID-19 ( Thurner et al. , 2020 ) , can be described with networks ( Mitchell , 2006 ; Barabási & Pósfai , 2016 ) . Let G = ( V , E ) be a network with node set V and edge set E. Assuming n = |V | , the interactions between nodes can be formulated as a set of differential equations ẋi = f ( xi ) + ∑ j∈V Pijg ( xi , xj ) , ∀i ∈ V , ( 1 ) where xi is the state of node i . In real systems , it could be the abundance of a plant in ecological network , the infection rate of a person in epidemic network , or the expression level of a gene in regulatory network . The term P is the adjacency matrix of G , where the entry Pij indicates the interaction strength between nodes i and j . The functions f ( · ) and g ( · , · ) capture the internal and external impacts on node i , respectively . Usually , they are nonlinear . Let x = ( x1 , x2 , . . . , xn ) . For a small network , given an initial state , one can run a forward simulation for an equilibrium state x∗ , such that ẋ∗i = f ( x ∗ i ) + ∑ j∈V Pijg ( x ∗ i , x ∗ j ) = 0 . However , when the size of the system goes up to millions or even billions , it will pose a big challenge to solve the coupled differential equations . The problem can be efficiently addressed by employing a mean-field technique ( Gao et al. , 2016 ) , where a linear operator LP ( · ) is introduced to decouple the system . In specific , the operator depends on the adjacency matrix P and is defined as LP ( z ) = 1TPz 1TP1 , ( 2 ) where z ∈ Rn . Let δin = P1 be nodes ’ in-degrees and δout = 1TP be nodes ’ out-degrees . For a weighted G , the degrees are weighted as well . Applying LP ( · ) to δin , it gives βeff = LP ( δin ) = 1TPδin 1T δin = δToutδin 1T δin , ( 3 ) which proves to be a powerful metric to measure the resilience of networks , and has been applied to make reliable inferences from incomplete networks ( Jiang et al. , 2020b ; a ) . We use it to measure the predictive ability of a NN ( see Section 4.3 ) , whose training in essence is a dynamical system . For an overview of the related technique , the readers are referred to Appendix H. NN Training is a Dynamical System . Conventionally , training a NN is a nonlinear optimization problem . Because of the hierarchical structure of NNs , the training procedure is implemented by two alternate procedures : forward-propagation ( FP ) and back-propagation ( BP ) , as described in Fig.1 ( a ) . During FP , data goes through the input layer , hidden layers , up to the output layer , which produces the predictions of the input data . The differences between the outputs and the labels of the input data are used to define an objective function C , a.k.a training error function . BP proceeds to minimize C , in a reverse way as did in FP , by propagating the error from the output layer down to the input layer . The trainable weights of synaptic connections are updated accordingly . Let GA be a NN , w be the flattened weight vector for GA , and z be the set of activation values . As a whole , the training of GA can be described with two coupled dynamics : A on GA , and B on GB , where nodes in GA are neurons , and nodes in GB are the synaptic connections . The coupling relation arises from the strong inter-dependency between z andw : the states z ( activation values or activation gradients ) of GA are the parameters of B , and the states w of GB are the trainable parameters of GA . If we put the whole training process in the context of networked systems , A denotes a node dynamics because the states of nodes evolve during FP , and B expresses an edge dynamics because of the updates of edge weights during BP ( Mei et al. , 2018 ; Poggio et al. , 2020a ; b ) . Mathematically , we formulate the node and edge dynamics based on the gradients of C : ( A ) dz/dt ≈ hA ( z , t ; w ) = −∇zC ( z ( t ) ) , ( 4 ) ( B ) dw/dt ≈ hB ( w , t ; z ) = −∇wC ( w ( t ) ) , ( 5 ) where t denotes the training step . Let a ( ` ) i be the pre-activation of node i on layer ` , and σ ` ( · ) be the activation function of layer ` . Usually , the output activation function is a softmax . The hierarchical structure of GA exerts some constraints over z for neighboring layers , i.e. , z ( ` ) i = σ ` ( a ( ` ) i ) , 1 ≤ i ≤ n ` , ∀1 ≤ ` < L and z ( L ) k = exp { a ( L ) k } / ∑ j exp { a ( L ) j } , 1 ≤ k ≤ nL , where n ` is the total number of neurons on layer ` , and GA has L+ 1 layers . It also presents a dependency between z and w. For example , when GA is an MLP without bias , a ( ` ) i = w ( ` ) T i z ( ` −1 ) , which builds an interconnection from GA to GB . It is obvious , givenw , the activation z satisfying all these constraints , is also a fixed point of A . Meanwhile , an equilibrium state of B provides a set of optimal weights for GA .

Summary: Accurately predicting the performance at early stage is important for efficient model selection without incurring too much computation. The paper proposes a neural capacitance metric as a predictive measure to capture the performance of a model on the downstream task using only a handful of early training results. The metric is derived from a line graph mapped from a neural network by modeling the dynamical system of the network.