Neural Capacitance: A New Perspective of Neural Network Selection via Edge Dynamics
Authors: Chunheng Jiang
Tejaswini Pedapati
Pin-Yu Chen
Yizhou Sun
Jianxi Gao
Venue: ICLR 2022 Submitted (2022)
Keywords: neural network selection
transfer learning
dynamical system
edge dynamics
network science

Abstract: Efficient model selection for identifying a suitable pre-trained neural network to a downstream task is a fundamental yet challenging task in deep learning. Current practice requires expensive computational costs in model training for performance prediction. In this paper, we propose a novel framework for neural network selection by analyzing the governing dynamics over synaptic connections (edges) during training. Our framework is built on the fact that back-propagation during neural network training is equivalent to the dynamical evolution of synaptic connections. Therefore, a converged neural network is associated with an equilibrium state of a networked system composed of those edges. To this end, we construct a network mapping $\phi$, converting a neural network $G_A$ to a directed line graph $G_B$ that is defined on those edges in $G_A$. Next, we derive a \textit{neural capacitance} metric $\beta_{\rm eff}$ as a predictive measure universally capturing the generalization capability of $G_A$ on the downstream task using only a handful of early training results. We carried out extensive experiments using 17 popular pre-trained ImageNet models and five benchmark datasets, including CIFAR10, CIFAR100, SVHN, Fashion MNIST and Birds, to evaluate the fine-tuning performance of our framework. Our neural capacitance metric is shown to be a powerful indicator for model selection based only on early training results and is more efficient than state-of-the-art methods.

Under review as a conference paper at ICLR 2022
NEURAL CAPACITANCE: A NEW PERSPECTIVE OF NEURAL NETWORK SELECTION VIA EDGE DYNAMICS
Anonymous authors Paper under double-blind review
ABSTRACT
Efficient model selection for identifying a suitable pre-trained neural network downstream task is fundamental yet challenging task in deep learning. Current practice requires expensive computational costs in model training for performance prediction. In this paper, we propose a novel framework for neural network selec- tion by analyzing the governing dynamics over synaptic connections (edges) during training. Our framework is built on the fact that back- backkpropagation during neural network training is equivalent the dynamical evolution of synaptic connections. Therefore, a converged neural network is associated with an equilibrium state of a networked system composed of those edges. To this end, we construct a network mapping s converting neural network G to directed line graph G' that is defined on those edges in A.. Next, we derive a neural capacitance metric Sett as predictive measure universally capturing the generalization capability of G on the downstream task using only a handful of early training results. We carried out extensive experiments using 17 popular traitrenned ImageNet models and five benchmark datasets, including CIFAR1O, CIFAR 100, SVHN, Fashion MNIST and Birds, evaluate the fine-tuning performance of our framework. Our neural capacitance metric is shown to be a powerful indicator for model selection based only on early training results and is more efficient than state-of-the-an methods.
INTRODUCTION
Leveraging a pre-trained neural network (i.e., a source model) and fine-tuning to solve a target task is a common and effective practice in deep learning, such as transfer learning. Transfer learning has been widely used solve complex tasks in text and vision domains. In vision, models trained on ImageNet are leveraged to solve diverse tasks such as image classification and object detection. In text, language models that are trained on a large amount of public data comprising of books, Wikipedia etc are employed to solve tasks such classification and language generation. Although such technique can achieve good performance on a target task, a fundamental yet challenging problem is how to select suitable pre-trained model from a pool of candidates in an efficient manner. The naive solution of training each candidate fully with the target data can find the best pre-trained model but is infeasible due to considerable consumption on time and computation resources. This challenge motivates the need for an efficient predictive measure to capture the performance of a pre-trained model on the target task based only on early training results (e.g., predicting final model performance based on the statistics obtained from first few training epochs).
In order to implement an efficient neural network (NN) model selection, this paper proposes a novel framework to forecast the predictive ability of a model with its cumulative information in the early phase of NN training, as practised learning curve prediction (Domhan et al., 2015) Chandrashekaran & Lane 2017 Baker et al.] 2017:: Wistuba & Pedapati| (020). Most prior work on learning curve prediction aims to capture the trajectory of learning curves with a regression function of models' validation accuracy. Some of the previous algorithms developed in this field require training data from additional learning curves to train the predictors (Chandrashe
& Lane 2017] Baker et al. 2017: Wistuba & Pedapati 2020). On the other hand, our model does not require any such data. solely relies on the NN architcture. Ranking models according to their final accuracy after fine-tuning is lot more challenging as the learning curves very similar each other.

---

Under review as a conference paper at ICLR 2022
The entire NN training process involves iterative updates of the weights of synaptic connections, according to one particular optimization algorithm, e.g., gradient descent or stochastic gradient descent (SGD) (Bottou 2012 LeCun et al. 2015]. In essence, many factors contribute to impact how weights are updated, including the training data, the neural architecture, the loss function, and the optimization algorithm. Moreover, weights evolving during NN training in many aspects can be viewed as a discrete dynamical system. The perspective of viewing NN training as dynamical system has been studied by the community (Mei al. 2018 Chang et al. 2018 Banburski et al. 2019 Dogra 2020 Tano et al. [0200] Dogra & Redman (020)Feng & Tu (021), and many attempted to make some theoretical explanation of the convergence rate and generalization error bounds. In this paper, we will provide the first attempt in exploring its power in neural model selection.
One limitation of current approaches is that they concentrated on the macroscopic and collective behavior of the system, but lacks a dedicated examination of the individual interactions between the trainable weights or synaptic connections, which are crucial in understanding of the dependency of these weights, and how they co-evolve during training. To fill the gap, we study the system from a microscopic perspective, build edge dynamics of synaptic connections from SGD in terms of differential equations, from which we build associated network well. The edge dynamics induced from SGD nonlinear and highly coupling. It will be very challenging to solve, considering millions of weights in many convolutional neural networks (CNNs), e.g.. 16M weights in MobileNet (Howard eet al. (017)) and 528M in VGG16 (Simonyan & Zisserman 2014). Gao et al.] (2016) proposed a universal topological metric for the associated network to decouple the system. The metric will be used for model selection in our approach, and it is shown to be powerful in search of the best predictive model. We illustrate our proposed framework in Fig
(a) Deep Neural Network Gj
Line Graph Gp
Inputs
Hiskkns
Ourputs
c= G. Cp
Network Mapping
G Neural Capacitance Probe (NCP)
Weight Interactions A
5x
4x
7x
= f[w)+
Pu
SPs, wg)
Synaptic Connections
Output
(c)
Randomly Inittalizing Freezing Layers
Train Pat
Pur PPPPI
New 1 ageess
Output Laysr
Epach
Layer
Layer
Validdstion Accuracy
Predicted
Fine-tuning Layers
Observed
Layer
Layer
Trausfer
Learning
Epach
Source
Turgei Data
Learning Curve Prediction
Figure I: Illustration of our framework. (a) An example multilayer perceptron (MLP) G is mapped to a directed line graph GB, which governed by an edge dynamics B. Each node (dichromatic square) of GB is associated with synaptic connection linking two neurons (in different colors) from different layers of G (b) A diagram of transfer learning from the source domain (left stack) to target domain (right stack), The pre-trained model is modified by adding additional layers, i.e. installing a neural capacitance probe (NCP) unit, on top of the bottom layers. The NCP is frozen with of randomly initialized weights, and only the bottom layers fine-tuned. (c) Observed partial learning curves (green line segments) of validation accuracy over the carly-stage training epochs and the corresponding neural capacitance metric Bett during fine-tuning. The predicted final accuracy Bes - 0 (red dot) is used to select the best one from a set models. The metric Best relies on G 's weighted adjacency matrix P. which itself is derived from the reformulation of the training dynamics. To predict the performance, lightweight Bett of the NCP is used instead of the heavyweight one over the entire network on the right stack of (b).
2

---

Under review as a conference paper at ICLR 2022
The main contributions of our framework can be summarized follows:
View NN training as a dynamical system over synaptic connections, and first time investigate the interactions of synaptic connections in a microscopic perspective.
Propose neural capacitance metric Seir for neural network model selection.
Empirical results of 17 pre-trained models on five benchmark datasets show that our Boet based approach outperforms current learning curve prediction approaches.
For rank prediction according to the performance of pre-trained models, our approach im- proves by 11/38..3/224/65... I % on CIFARTO/CIFRS WOO/SHn//ashion MNIST/Birds over the best baseline with observations from learning curves of length only 5 epochs.
RELATED WORK
Learning Curve Prediction. Chandrashckaran & Lane 2017) treated the current learning curve (LC) as an affine transformation of previous LCs. They built an ensemble of transformations employing previous LCs and the first few epochs of the current LC to predict the final accuracy of the current LC. Baker et al. (2017) proposed an SVM based LCC predictor using features extracted from previous L.Cs, including architecture information such as number layers, parameters, and training technique such learning rate and learning rate decay. A separate SVM is used to predict the accuracy of an LC at particular epoch. Domhan et al. (2015) trained an ensemble of parametric functions that observe the first few epochs of an LC and extrapolate it. Klein et al. (2017a) devised Bayesian NN to model the functions that Domhan formulated to capture the structure of the LCs more effectively. Wistuba & Pedapati| (2020) developed a transfer learning based predictor that trained LCs generated from other datasets. It is a NN based predictor that leverages architecture and dataset embeddings capture the similarities between the architectures of various models and also the other datasets that it was trained on.
Dynamical System View of NNs. There are many efforts study the dynamics of NN training. Some prior work on SGD dynamics for NNs generally have a pre-assumption of the input distribution or how the labels are generated. They obtained global convergence for shallow NNs (Tin 2017 Banburski et al. 2019). System identification itself is complicated task (Haykin) 2010 Lillirilr
it al.] 2020). In studying the generalisation phenomenon of deep NNs, Goldt al. (2019) formulated SGD with set of differential equations. But, it is limited to over-paramettrrsse two-layer NNs under the teacher-student framework. The teacher network determines how the labels are generated. Also, some interesting phenomena (Frankle et al.) (020) observed during the early phase of NN training, such as trainable sparse sub-networks emerge (Frankle et al., 2019), gradient descent moves into a small subspace (Gur-Ari et al. 2018), and there exists critical effective connection between layers (Achille et al.) 2019). Bhardwaj et al.] (201)) built a nice connection between architectures (with concatenation-Vye type skip connections) and the performance, and proposed new topological metric to identify NNs with similar accuracy. Many of these studies are built on dynamical system and network science. will be promising direction study deep learning mechanism.
PRELIMINARIES
Dynamical System of Network. Many real complex systems, e.g., plant-pollinator interac- tions WWaser& & Ollerton! 2006) and the spread of COVID-19 (Thurner al. 2020). be described with networks (Mitchell 2006 Barabási & Pósfai 2016). Let G = (V, E) be a network with node set V and edge set E. Assuming = [V], the interactions between nodes can be formulated set of differential equations
$ = f(x)) + Σ Pug((,,
JEV
where is the state of node i. In real systems, it could be the abundance of plant in ecological network, the infection rate of a person in epidemic network, or the expression level of a gene in regulatory network. The term P is the adjacency matrix of G, where the entry P., indicates the interaction strength between nodes and j. The functions f(.) and g(- capture the internal and external impacts on node respectively. Usually, they are nonlinear.
3

---

Under review as a conference paper at ICLR 2022
Let a (21,22.... 2n). For small network, given an initial state, can run a forward simulation for an equilibrium state x", such that á; = f(()) + Ejev ( 25) = O. However, when the size of the system goes up to millions or even billions, it will pose a big challenge to solve the coupled differential equations. The problem can be efficiently addressed by employing mean-field technique (Gao et al., 2016), where a linear operator (- is introduced to decouple the system. In specific, the operator depends on the adjacency matrix P and is defined
1 Pz
Cr(z)
P1
where = E R². Let P1 be nodes' in-degrees and dout P be nodes' out-degsees. For weighted G, the degrees are weighted as well. Applying Lp(-) to ōin. it gives
Bott = LP(din) = P5
5uu din
8
8
which proves be a powerful metric to measure the resilience of networks, and has been applied to make reliable inferences from incomplete networks (Jiang et al. 20200b a). We use it to measure the predictive ability of a NN (see Section 4.3), whose training in essence a dynamical system. For an overview of the related technique, the readers are referred to Appendix
NN Training is a Dynamical System. Conventionally, training a NN is a nonlinear optimization problem. Because of the hierarchical structure of NNs, the training procedure is implemented by two alternate procedures: forward-pre pawaaddproppgation (FP) and paok-propagation (BP), as described in Fig I a). During FP, data goes through the input layer, hidden layers, up to the output layer, which produces the predictions of the input data. The differences between the outputs and the labels of the input data used to define objective function C, a.k.a training error function. BP proceeds to minimize C, in reverse way as did in FP, by propagating the error from the output layer down to the input layer. The trainable weights of synaptic connections are updated accordingly.
Let GA be a NN, w be the flattened weight vector for GA. and x be the set of activation values. As a whole, the training of G can be described with two coupled dynamics: A G., and B on Gn. where nodes in G are neurons, and nodes in GB are the synaptic connections. The coupling relation arises from the strong inter-dependency between & and w: the states z (activation values or activation gradients) of G are the parameters of B, and the states w of GB are the trainable parameters of GA. we put the whole training process in the context of networked systems, denotes node dynamics because the states of nodes evolve during FP, and B expresses an edge dynamics because of the updates of edge weights during BP (Mei et al. 2018 Poggio et al.) 2020a b). Mathematically, we formulate the node and edge dynamics based on the gradients of C:
(A) dz / dt zo ha(z, t; w) = - V=C(x(t)). (B) dw dt z hn(w, 2; (w))
(4)
(5)
where denotes the training step. Let a, be the pre-activation of node on layer E, and (r(+) be the activation function of layer f. Usually, the output activation function is softmax. The hierarchical structure of G exerts some constraints over z for neighboring layers, i.c., x!") = (a ()), 1 < is ne, V1 < é < T. and =
(f.) exp (1.) ( E, exp(a,") (1) ), s k nı, where n, is the total number of neurons layer f, and G has L ++1 layers. It also presents dependency between and For example, when GA is an MI.P without bias, a(5) un; !!( 2(1-1), which builds an interconnection from G to GB. It is obvious, given w, the activation z satisfying all these constraints, is also fixed point of A. Meanwhile, an equilibrium state of B provides a set of optimal weights for GA.
4 OUR FRAMEWORK
The metric Berr is universal metric to characterize diffeernt types of networks, including biological neural networks (hhu et al. 2021) (Section 3 Because of the generality of Betf, we analyze how it looks on artificial neural networks which are designed to mimic the biological counterparts for general intelligence. Therefore, we set up analogue system for the trainable weights. To the end, we build a line graph for the trainable weights (Section 4.1, and reformulate the training dynamics in the same form of the general dynamies (Eq. [ (Section 4.2 The reformulated dynamics reveals
4

---

Under review as a conference paper at ICLR 2022
simple yet powerful property regarding Ber (Section 4.3 which is utilized to predict the final accuracy of GA with a few observations during the early phase of the training (Section For a detailed description of the core idea of our framework, Appendix
4.1 LINE GRAPH GB
We build a mapping scheme Φ GA H GB, from an NN GA to an associated graph GB. The topology of the synaptic connections (edges) is established as well-defined line graph proposed by Nepusz& & Vicsek (2012), and nodes of GB are the synaptic connections of GA. More precisely, each node in Gg is associated with trainable parameter in GA. For MIP, each synaptic connection is assigned a trainable weight, the edge set of G is also the set of synaptic connections of G For a CNN, this one-to-one mapping from neurons on layer to layer + is replaced by one-to-more mapping because of weight-sharing. e.g., parameter in a convolutional filter is repeatedly used in FP and associated with multiple pairs of neurons from the two neighboring layers. Since the error gradients flow in reversed direction, we reverse the corresponding links of the proposed line graph GB. In specific, given any pair of nodes GB. if they share an associated intersection neuron in FP propagation routes, a link with a reversed direction will be created for them. In [ig[aa) we demonstrate how the mapping is performed on an example MLP. We have the topology of GB B in place, but the weights of links in Ga are not yet specified. To make up this missing components, we reveal the interactions synaptic connections from SGD, quantify the interaction strengths and then define the weights of links in GB accordingly. Related technical details are disclosed in next section.
4.2 EDGE DYNAMICS B
In SGD, each time a small batch of samples are chosen to update w. i.e., ur +- - a V VwC, where O is the learning rate. When desired conditions are met, training is terminated.
We denote the activation gradients as g( [ac/0z] of activation function a for layer é as o¹ (a) understand how the weights W(() affect each other, explicitly expand 8(1);
aC 0c/ 100 E R" and the derivatives
(e) ) o'((nn) 17 E R 1 S E L. To
(t)) w(++1)T (Wil+2))) (..((WL- (W/L)T (2(L) - y)) o ooo)) o ) 2 d(t1),
where @ is the Hadamard product. We find that parameters (()) are associated with all accessible parameters on downstream layers, and such recursive relation defines high-order hyper-network in- teraction (Casadiego et al. 2017) between any W(() and the other parameters. The Hadamard product a aoy has an equivalent matrix multiplication form, i.e. æ@y = A(y)a,, where A (y) is a diagonal ma- trix consisting of the entries of y on the diagonal. Therefore, we have (+) = W(++1)T A(o (8 and (()) (e+ 1)T 1 (o'44))W(++22) ((0'42) ( 1y7 A ((()) (z) y). For ReLU (r))), of binary depending on the sign of the input pre-activation values (()) of layer l. If a < 0. then (a = O, blocking a BP propagation route of the prediction deviations ((4)
and giving rise vanishing gradients.
Our purpose is to build direct interactions between synaptic connections. It can be done by identifying which units provide direct physical interactions to given unit and appear on the right hand side of its differential equation B in Eq. 4 and how much such interactions come into play. There are multiple routes to build up direct interaction between any pair of network weights from different layers, as presented by the product terms in 8(0) However, the coupled interaction makes it an impossible task, which is well known as credit assignment problem (Whittington & Bogacz 2019 Lillicrap et al. 2020). We propose a remedy. The impacts of all the other units on W()) is approximated by direct, local impacts from (f+1)) and the others' contribution as a whole is implicitly encoded in the activation gradient g(++1),
Moreover, we have the weight gradient (see Appendix A for detailed derivation)
V W A(0 -aor==>((())) ((+1)) Caat//eiia/loooaaaa ((
(6)
In some literature ( is defined gradients with respect a! which does affect our analysis.
5

---

Under review as a conference paper at ICLR 2022
which shows the dependency of WX() on W(++1) and itself can be viewed as explicit description of the dynamical system B in Eq. Put it in terms of a differential equation, we have
aw /dt - A(o)) (( 1 20 F(W(+)1))
Because of the mutual dependency of the weights and the activation values, it is hard to make an exact decomposition of the impacts of different parameters W()), However, in the gradient V wre,, W(++1) presents explicit term and contributes the direct impact W(O), To capture such direct impact and derive the adjacency matrix P for GB, we apply Taylor expansion on V weo and have
= = /////////wwwww/ (ow((t))
(8)
which defines the interaction strength between each pair of weights from layer ++ to layer f.
See Appendix B for detailed derivation of P on MLP, and Appendix C on general NNs. Let a flattened vector of all trainable weights of G Given a pair of weights w; and wy,, one from layer f1. another from layer f2. If f2 = l + 1, the entry P. is defined according to Eq. otherwise Piş- = 0. Considering the scale of trainable parameters in G P is very sparse.
w (Wi, W2,
Let W (B=1)* be the equilibrium states (Appendix C. the training dynamics Eq. is reformulated into the form of Eq. and gives the edge dynamics B for Gg:
wb = f(ws) + > Pugg(us, wy):
(9)
3
with f(w:)) = F(wF)) and g( w5) = wy W The value of weights at equilibrium state {w²}) is unknown, but a constant and does not affect the computing of Bett.
4.3 NEURAL CAPACITANCE
we have the weighted adjacency matrix P of GB in place. Now we can quantify the total impact that a trainable parameter (or synaptic connection) receives from itself and the others, which corresponds to the weighted in-degrees 8n P1. Applying Lp(+) (see Eq. D to din we get "counterpart" metric 3ett (P((in) to measure the predictive ability of neural network GA. as the
According to Eq. g
resilience metric (see Eq. 3 does general network G (see Dynamical System of Network in
Section 3
If GA is an MLP, we can explicitly write the entries of P. hence Sell explicitly
26=2 1 (1 (1-2)) x (2-1)- O 1 1 (1))
o' (T @ 0o'41l
Bell
(10)
E 1 31 [ [TT x 17 o o'l
For details of how derive Pnd Bett of an MIP, Appendix B Moreover, we prove in Theorem
1 below that as GA converges, V/ vanishes, and Serl approaches zero (see Appendix D
Theorem 1. Let ReLU be the activation function G When G. converges, then Bel O.
For MLP GA. it is possible to derive an analytical form of Bea- However, becomes extremely complicated for a deep NN with multiple convolutional layers. To realize 3aft for deep NNs in any form, we take advantage of the automatic differentiation implemented in TensorFlow Considering the number of parameters, it is still computationally expen- sive, and prohibitive to calculate Seit for the entire G. Because of this, we seek to derive a surrogate from a partial of G As shown in Section 4.4 we insert a neural capacitance probe (NCP) unit, i.e., putting additional lay- on top of the beheaded G (excluding the original output layer), and estimate the predictive ability of the entire GA using Bett of the NCP unit. Therefore, in the context of model selection from pool of pre-trained models, if no confusion arises, we call Belt neural capacitance.
Algorithm 1 Implement NCP and Compute Belr
Input: A pre-trained model F, = (J!!) (.. with bottom layers F(2) and output layer F: a target dataset D,, the maximum number of epochs T
1: Remove F!!) from F. and add on top of Fz() an NCP unit 2/ with multiple layers (Fig 1b)
2: Initialize with random weights and freeze '4
3: Train F, for epochs of
(F _u4 by fine-tuning F
D,
4: Obtain P from 2/ according to Eq.
5: Compute Ber with P according to Eq or Eq. 10
phtps//wwww.enssrfoww..gg

---

Under review as a conference paper at ICLR 2022
4.4 MODEL SELECTION WITH Belf
Here show novel application of our proposed neural capacitance Bell to model selection. In specific, we transfer the pre-trained models by (i) removing the output layer, (ii) adding some layers on top of the remaining layers (Fig.D). [D), and fine-tune them using small learning rate. As shown in Algorithm[] | the newly added layers 2/ on top of the bottom layers of F. are used as an NCP unit. The specifies of the NCP unit are detailed in Section 5 The NCP does not involve in fine-tuning, and is merely used to calculate Betf, then to estimate the performance of G A over the target domain D,.
According to Theorem [ when the model converges, Bell + O. In an indirect way, the predictive ability of the model can be determined by the relation between the training Ser and the validation accuracy I. Since both Berl and I are available during fine-tuning, we collect set of data points of these two in the early phase as the observations, and fit regularized linear model / = (B 0) with Bayesian ridge regression (Tipping 200T), where A are the associated coefficients (see Appendix E for technical details). The estimated predictor / = h( 0²) makes prediction of the final accuracy of models by setting 9etf = 0, i.c., 1 = h0; 0*), see an example in row 3 of Fig 2 For full training of the best model, one can either retain or remove the NCP and fine-tune the selected model.
EXPERIMENTS AND RESULTS
Pre-trained models and datasets. We evaluate 17 pre-trained ImageNet models implemented in Keras including AlexNet, VGGs (VGG16/19), ResNets RRsNNet50/50V2/1011100 ///////V2220000000//////22222 DenseNets (DenseNea1211/1199201 MobileNets (MobileNet and MobileNerV2), Inceptions (In- ception InceptionResNett22) and Xception, to measure the performance of our approach. Four benchmark datasets CIFARIO, CIFARIO0, SVHN, Fashion MNIST of size 32 x 32 x 3, and one Kaggle challenge dataset Birds" of size 224 x 224 3 are used, and their original train/test splits are adopted. In addition, 15K original training samples are set aside as validation set for each dataset.
V2/1
Experimental setup. To get a well-defined Beit. GA requires least three hidden layers (see Appendix x C Also, batch normalization (loffe & Szegedy 2013) is usually beneficial because it can stabilize the training by adjusting the magnitude of activations and gradients. To this end, on top of each praitrd model, we put a NCP unit composed of (1) a dense layer of size 256, (2) dense layer of size 128, each of which follows (3) a batch normalization and is followed by (4) a dropout layer with a dropout probability of 0.4. Before fine-tuning, we initialize the NCP unit using Kaiming Normal initialization (He et al. 2015).
We set batch size of 64 and learning rate of 0.001, fine-tune each pre-trained model for T = 50 epochs, and repeated it for 20 times. As shown in Fig the pre-trained models are converged after the fine-tuning on CIFARI0 10. For each model, we collect the validation accuracy (blue stars in row 1) and Bei on the training set (green squares in row 2) during the early stage of the fine-tuning as the observations (c.g.. green squares in row 3 marked by the green box for 5 epochs). then use these observations to predict the test accuracy unseen before the fine-tuning terminates. For better illustration, learning curves are visualized on a log-scale.
Evaluation. We apply the Bayesian ridge regression on the observations to capture the relation between Sell and the validation accuracy, and to estimate a learning curve predictor I = h(3eer: 0*). The performance of the model is revealed as /* h(3art 0²) with 8ür = 0. As shown in row 3 of Fig 2 the blue lines are estimated h(:: 0), the true test accuracy at T and the predicted accuracy are marked as red triangles and blue stars, respectively. Both the estimates and predictions are accurate.
We aim to select the best one from a pool of candidates. A relative rank of these candidates matters more than their exact values of predicted accuracy. To evaluate and compare different approaches, we choose Spearman's 's rank correlation coefficient p as the metric, and calculate p over the true test accuracy at epoch T and the predicted accuracy I* of all pre- tre-trained models. In Fig report the true and predicted accuracy for each model on CIFAR1O, as well as the overall ranking performance measured by p. It indicates that our B-based model ranking is reliable with p > 0.9. For the results on all five datasets, see Appendix Fig F4
"httss//kensssollplaapiiiiiiiii
7

---

Under review as a conference paper at ICLR 2022
MobileNet
ResNet
DenseNet
Inception
VGG
1.00
Aucancy
0.95
0.95
0.97
0.95
0.75
Vali
0.93
0.90-
0.90
0.90
0.5
2
10P
10
0.00
10
10'
0
10P
10'
100
10 Epoch t
10P
101
Epoch b
Epoch
Epoch
Epoch c
0.96
Vali Aecuney
0.94
0.94
0.92
0.8
Pred. Acc.
0.92
0.92
0.94
0.90
0.6
Obsik Po.
2.
BIC-32
0.90-
to 3.
BIC-33
0.90
1o- 3.
BIC-32
0.92
2,
BIC-3.3
0.88
6-
4,
BIC-4,5
10
10
10
10
10
100
10
10
10
10
10
10
10
10
Train [o]
Train
Train 18
Tmin [8]
Tmin [8]
Figure 2: Learning curves of five representative pre-trained models w.r.t accuracy (row 1) and Peft (row 2). A regularized linear model lu(:; 0) (blue curve in row 3) is estimated with Bayesian ridge regression using a few observations of Berr on training set and validation accuracy I during early fine-tuning. The starting epoch lo of observations affects the fit of h, and is automatically determined according to BIC, and the true test accuracy at epoch 50 is predicted with I" = h(0; )*). (a)
(b)
(c)
Pre-tranied
1.0
True Aoe.
0.96
DemeNel
Spearman Corr.
Starting Epoch to
Spcurman Cosr.
Size of Train Set
10K
Incoption
15K
0.94
MochileNer
0.8
5
0.8
25K
ResNet
P 0.92
VGG
30K
0.92
Spcarman Corr.
Xccption
0.6
35K
0.92
0.94 0.96
10
15
0.6
10
Predicted Avc.
Length of Learinng Curve
Length of Learning Curve
Figure 3: (a) Our Beer based prediction of the validation accuracy versus the true test accuracy at
epoch 50 of seven representative pre-trained models. Each shape is associated with one type of pre-trained models. Distinct models of the same type are marked in different colors. Because the accuracy of AlexNet is much lower than others, we exclude it for better visualization. Its predicted accuracy 0.871, 1, and the true test accuracy is 0.868. If it is included, p = 0.93 > 0.92. (b) Impacts of the starting epoch to of the observations and (c) the number of training samples on the ranking performance of our Pear based approach.
The estimation quality of h determines how well the relation between and Berr is captured. Besides the regression method, the starting epoch to of the observations also plays role in the estimation. As shown in Fig 3 b), 3b), we evaluate the impact of lo on p of our approach. goes as expected, when the length of learning curves is fixed, higher to usually produces a better p. Since our ultimate goal is to predict with the early observations, to should also be constrained to a small value. To make the comparisons fair, we view to as a hyper-parameter, and select it according to the Bayesian information criterion (BIC) (Friedman et al. 200I), as shown in row 3 of Fig
Impact of size of training set. CIFARIO has 50K original training and 10K testing samples. Generally, the 50K samples are further split into 35K for training and 15K for validation. In studying the dynamics of the NN training, it is essential to understand how varying the training size influences the effecttieness of our approach. We select the first (10,15 1,,20,,,,,2,,00
of the original 50K samples as the training set of reduced size, and the last 10K samples the validation set fine-tune the pre-trained models for 50 epochs. As shown in Fig ( c). we can use a training set of size as small as 25K to achieve similar performance to that uses all 35K training samples. It has an important implication for efficient NN training. because the size of required training set can be greatly reduced (around 30% in our experiment) while maintaining similar model ranking performance. To be noted that the true test accuracy used in computing p is the same test accuracy for the model trained from 35K training samples and it's shared by all the five cases 110,15,20,2 20,25,00 25 in our analysis.
Ours versus baselines. We select BGRN Baker et al. 2017) and CL (Chandrashekaran & Lane 2017) as the baselines, as well as two heuristic rules of using the last seen value (LSV) (Klein et al. 20170 7b) or the best seen value (BSV) of learning curve for extrapolation.
8

---

Under review as a conference paper at ICLR 2022
Table I: A comparison between our Bett based approach and the baselines in model ranking. The no- tation LLC represents the length of the learning curve, and Impry represents the relative improvement of our approach to the best baseline. Due to the failure of the supporting package of LC, there is a missing p at LLC of 10, which does not affect our conclusions.
Dataset
CIFARIO
CIFAR 100
SVHN
Fashion MNIST
Birds
LLC
10
10
5
10
10
|
10
Ours
0.93
0.98
0.77
0.80
0.84
0.88
0.95
0.89
0.74
0.79
BSV
0.86
0.89
0.55
0.80
0.74
0.78
0.53
0.60
0.52
0.61
LSV
0.85
0.87
0.55
0.80
0.73
0.70
0.49
0.45
0.48
0.45
BGRN
0.74
0.78
0.45
0.60
0.63
0.65
0.57
0.59
0.53
0.52
LC
0.85
0.85
0.50
0.58
0.44
0.10
0.55
0.61
0.50
Impry (%)
9.1
10.2
38.3
-0.9
12.4
13.3
65.3
49.2
40.1
30.6
We compare the performance of ours with the baselines. As shown in Table and Appendix Fig F.5 using a few of observations, e.g., only 5 epochs, our approach achieve 2/8..///12.4/45.3/// relative improvements over the best baseline on CIFARIO/CIFARIOOS L00/S 10OVsi//sionooooooaaiioooooo MNIST/Birds.
Running time analysis. Our approach is efficient, especially for large and deep NNs. Different from the training task that involves a full FP and BP, i.c. Traain
TFp + TBP, computing Sett only g on the NCP unit, Tp... TNCCP-
requires to compute the adjacency matrix P according to Eq. Although the computation is complicated, the NCP is lightweight.
The computing cost per epoch is comparable to the training time per epoch (see Appendix Fig G Let T
x Terain If > i.e., Te. is higher than 7 Traann vice versa. Considering the required epochs, our approach needs k observations, and takes Tours epochs, it takes Tull- K x Tuainn If T'ull Tours. our Setf based prediction is cheaper than "just training longer". indicates that K x Trrain K X Tost = (K - c x k) x Tirain O, saving us K x k more training epochs.
x TBa To obtain the ground-truth final accuracy by running K
We perform a rumning time analysis of the two tasks with 4x NVIDIA Tesla V100 SXM2 32GB, and visualize the related times in Appendix Fig(G. On average Tg /Trrrin = 1.3, computing Bal takes 1.3 times of the training per epoch. But the efforts are paying off, as we can predict the final accuracy by observing only A 10 of K 100 full training epochs, Tours is only 13% of Trull
When the observations are used for learning curve prediction, the heuristics LSV and BSV directly take one observation (last best) the predicted value, so they are mostly computationally cheap but have suboptimal model ranking performances. Relatively, BGRN and CL are more time-consuming because both require train predictor with a set of full learning curves from other models. Our approach also estimates a predictor, but does not need any external learning curves, Here we assume that each model is observed for only 5 epochs, and conduct a running time analysis of these approaches over learning curve prediction, including estimating a predictor. As shown in Appendix Table G.2, our approach applies Bayesian ridge regression to efficiently estimate the predictor I = [[otr;; 0), taking comparable time as BGRN, significantly less than CL, but performs best in model ranking. In contrast, the most expensive CL, does not perform well, sometimes even worst.
6 CONCLUSION AND DISCUSSION
We present a new perspective of NN model selection by directly exploring the dynamical evolution of synaptic connections during NN training. Our framework reformulates the SGD based NN training dynamics as an edge dynamics B to capture the mutual interaction and dependency of synaptic connections. Accordingly. a networked system is built by converting an NN G line graph GB with the governing dynamics B, which induces definition of the link weights in GB- Moreover, topological property of GB named neural capacitance 9ee is developed and shown to an effective metric in predicting the ranking of set of pre- pre-trained models based early training results.
There are several important directions that we intend to explore in the future, including (i) simplify the adjacency matrix P to capture the dependency and mutual interaction between synaptic connections, e.g., approximate gradients using local information (Jaderberg et al.) (017)) (ii) extend the proposed framework to NAS benchmarks Ying al.) 2019 Dong & Yang 2020; Dong et al. 2021 Zela et al.,
2020; Li et al. (02T) to select the best subnetwork, and (iii) design an efficient algorithm to directly optimize NN architectures based on Bett.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using performance prediction. arXiv preprint arXiv:/70S. 1705. 2017.
Andrzej Banburski, Qianli Liao, Brando Miranda, Lorenzo Rosasco, Fernanda De La Torre, Jack Hidary, and Tomaso Poggio. Theory III: Dynamics and generalization in deep networks. arXiv preprint orXiv:1903. 1903,099,9 2019.
Albert-Láázıó Barabási and Márton Pósfai. Network Science. Cambridge University Press, 2016.
Kartikeya Bhardwaj, Guihong Li, and Radu Marculescu. How does topology influence gradient propagation and model performance of deep networks with densenet-type skip connections? In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 13498-15 13507, 2021.
Léon Bottou. Stochastic gradient descent tricks. In Neural networks: Tricks of the Trade, pp. 421-436.
Springer, 2012.
Jose Casadiego, Mor Nitzan, Sarah Hallerberg, and Marc Timme. Model-free inference of direct network interactions from nonlinear collective dynamics. Nature Communications, 8(1):1-10, 2017.
Akshay Chandrashekaran and R Lane. Speeding up syper-parametee optimization by extrapolation of learning curves using previous builds. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 477-492. Springer, 2017.
Bo Chang, Minmin Chen, Eldad Haber, and H Chi. AntisymmetricRNN: A dynamical system view on recurrent neural networks. In International Conference on Learning Representations, 2018.
Akshunna S Dogra. Dynamical systems and neural networks. arXiv preprint arXiv:2004. 1826, 2020.
Akshunna S. Dogra and William Redman. Optimizing neural networks via Koopman operator theory. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 2087-2097. Curran Associates, Inc., 2020.
Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. 'weent-fourth International Joint Conference Artificial Intelligence, 2015.
Xuanyi Dong and Yi Yang. NAS-Bench-201, Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.003:: 000000000000....... 2020.
Xuanyi Dong, Lu Liu, Katarzyna Musial, and Bogdan Gabrys. NATS-Bench: Benchmarking nas algorithmss for architecture topology and size. IEEE transactions pattern analysis and machine intelligence, 2021.
Yu Feng and Yuhai Tu. Phases of learning dynamics in artificial neural networks: in the absence or presence of mislabeled data. Machine Learning: Science and Technology, 2021.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint vrXiv:1903.016111 2019.
Jonathan Frankle, David Schwab, and Ari S Morcos. The early phase of neural network training. arXiv preprint arXiv:2002.10363 2020.
Jerome Friedman, Trevor Hastie, Robert Tibshirani, et al. The elements of statistical learning, volume 1. Springer series in statisties New York, 2001.
10

---

Under review as a conference paper at ICLR 2022
Jianxi Gao, Baruch Barzel, and Alszró-László Barabási. Universal resilience patterns in complex networks. Nature, 530(7590):007-312 2016.
Sebastian Goldt, Madhu Advani, Andrew M Saxe, Florent Krzakala, and Lenka Zdeborová. Dynamies of stochastic gradient descent for two-layer neural networks in the teacher-student setup. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer. Gradient descent happens in tiny subspace. arXiv preprint arXiv: 18/2.04444 2018.
Simon Haykin. Neural Networks and Learning Machines. Pearson Education India, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance ImageNet classification. In Proceedings the IEEE International Conference on Computer Vision, pp. 1026-1034, 2015.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 2017.
Sergey loffe and Christian Szegedy. Batch Normalization:
Accelerating deep network training by reducing internal covariate shift. International Conference Machine Learning, pp. 448-456. PMLR, 2015.
Max Jaderberg. Wojeiech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In International Conference on Machine Learning, pp. 1627-1635. PMLR, 2017.
Chunheng Jiang, Jianxi Gao, and Malik Magdon-Ismail. Inferring degrees from incomplete networks and nonlinear dynamics. In Proceedings of the Twenty-Ninte International Joint Conference on Artificial Intelligence, IJCAI-20, pp. 3307-3313, International Joint Conferences on Artificial Intelligence Organization, 2020a.
Chunheng Jiang, Jianxi Gao, and Malik Magdon-Ismail. True nonlinear dynamics from incomplete networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 131-138, 2020b.
Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter. Fast Bayesian optimization of machine learning hyperparameters large datasets. In Artificial Intelligence and Statistics, pp. 528-536. PMLR, 2017a.
Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, and Frank Hutter. Learning curve prediction with Bayesian neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017b.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436 444. 2015.
Chaojian Li, Zhongzhi Yu, Yonggan Fu, Yongan Zhang, Yang Zhao, Haoran You, Qixuan Yu, Yue Wang, and Yingyan Lin. HW-NAS-Bemn Hardware-awarr neural architecture search benchmark. arXiv preprint arXiv:2 10.221130/00000 10584, 2021.
Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin Akerman, and Geoffrey Hinton. Backprop- agation and the brain. Nature Reviews Neuroscience, pp. 1-12, 2020.
David JC MacKay. Bayesian interpolation. Neural Computation, 4(3):415 447, 1992.
Song Mei, Andrea Montanari, and Phan-Minn Minn-Minh Nguyen. A mean field view of the landscape of two- layer neural networks. Proceedings of the National Academy of Sciences, L15(33):E7665-E76) 2018.
11

---

Under review as a conference paper at ICLR 2022
Melanie Mitchell. Complex systems: Network thinking. Artificial Intelligence, 2006.
Tamás Nepusz and Tamás Vicsek. Controlling edge dynamics in complex networks. Nature Physics, 8(7):568-573, 2012.
Tomaso Poggio, Andrzej Banburski, and Qianli Liao. Theoretical issues in deep networks. Proceed- ings of the National Academy of Sciences, (1144)):3003-330445 2020a.
Tomaso Poggio, Qianli Liao, and Andrzej Banburski. Complexity control by gradient descent in deep networks. Nature Communications, 11(1):1-5, 2020b.
Pin Shu, Hong Zhu, Wen Jin, Jie Zhou, Shanbao Tong, and Junfeng Sun. The resilience and vulnerability of human brain networks across the lifespan. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 29:1756-1780 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Mauricio E Tano, Gavin D Portwood, and Jean Ragusa. Accelerating training artificial neural networks with dynamic mode decomposition. arXiv preprint arXiv:2006. /4371, 2020.
Stefan Thumer, Peter Klimek, and Rudolf Hanel. A network-based explanation of why most covid-19 infection curves are linear. Proceedings of the National Academy of Sciences, 117(37):22684- 22689, 2020.
Yuandong Tian. An analytical formula of population gradient for two-layered ReLU network and its applications in convergence and critical point analysis. In International Conference on Machine Learning, pp. 3404-3413, PMLR, 2017.
Michael Tipping. Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning Research, (Juu)):211 2211)224,144441 244, 2001.
Nickolas M Waser and Jeff Ollerton. Plant-pollinator interactions: from specialization generaliza- tion. University of Chicago Press, 2006.
James CR Whittington and Rafal Bogacz. Theories of error back-propagation in the brain. Trends in Cognitive Sciences, 23(3):235-250, 2019.
Martin Wistuba and Tejaswini Pedapati. Learning to rank learning curves. International Conference Machine Learning, pp. 10303-10311 PMLR, 2020.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS- Bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning, pp. 7105-7114, PMLR, 2019.
Arber Zela, Julien Siems, and Frank Hutter. NAS-Bench-1Shotl Benchmarking and dissecting one-shot neural architecture search, arXiv preprint arXiv:2001. 10422, 2020.
12

---

Under review as a conference paper at ICLR 2022
A ERROR GRADIENTS
Let GA be an MIP. To understand the learning mechanism, we take a sample a with label y. and go through the entire training procedure, including forward pass FP and a backward pass BP. To be convenient, we rewrite 20) I as the inputs, let 2 and z 22(-1) be the activations of the first and the last hidden layers, respectively, and let x(4) be the outputs. The MLP is a parameterized model O = 2²)) = (x) with w = (W(1), w ..... W(L)), where ((() is the weight matrix of synaptic connections from layer 8 to layer f, and 1 I l S L. Suppose there are ng neurons on layer& E, W(() has the size Rig X 1- The inputs æ are fed into MLP, after forward pass from layer I down to layer L and layer L. Each neuron receives a cumulative imput signal from the previous layer, and sends an activated signal to downstream layer. Let de be the activation function of layer l, dz
(2) = w (e)T_ (t-1)) be the pre-activation value of neuron on layer E, we have 2q = (a ()) (2) being the ith row of W( 1 S S ng. The output activation function σr is generally with wi
softmax function, i.e. 2 (T) = expla. ( D3 exp(a) (T)) ). and z(L) is a probability distribution over n classes, i.e. 1 12) 1.
With the predictions zL4 and the ground truth y. we can calculate the prediction error C(2(L), y). which is often cross entropy loss, i.e. C(=(4), yy - - Es yi log = To minimize C, Bs applied, and the weights w are updated backward, from the output layer up to the first hidden layer.
Now we derive the gradients for a close examination. First, we get the derivatives of C w.r.t (L) and W(L) Because = exp(a; (L) 3/E, E, exp(a) (T)) we get the gradient of the output 2 w.r.t wi (L)
(L) (I),
0z(()) /Ow!!
2 (6ki 24 )= (1-1)
where dRs = 1 if k- = i, otherwise dRs =
On layer L, we get the derivatives of C w.r.t (1) and W(:;
ac 02(4)
y, [1)]
Br [2) Σ Oc
OC
0z (L)
[L] (z)
- y1) 1)2
(11)
dz
Ow!y
1j
Then, examine layer L - 1. The activation function ØL-1 associates to pair of neurons u
(L-1)
on layer I. - 1 and u (2-2) on layer L. - with unique connection weight wiy (L-1) Since 2;
expla (1) }/ E, expla, (L)) and 2! 2/4-1) = o ((-1)) we get 0z( 10z, (1-11-1) = 2k (wik
(L) (L)
E, 2 jj = 2 (w w (LyT 2(1)), and
(1)) (L)
0z!4 /0w!
=/= (L-1) (L-2) (1-2)) d' (1(4-1))
The derivatives of C are
ac
Σn ac 0z
Ex yk (w (w!(
(L)T =(1.) -w -
w (1())(((1)) y).
(L)T
az! d=!!-11 ac
0z (4) dz( (1-1
de 0200 (1-1)
wai (z(11) - -y) -/)2(1-2) (44-)'0/_ 2100
(L)T
(L-2)
(a! (a(4-11)
Ow dw.--
0z (1-1) ow! (1-1)
On layer where 1 S S 5 L 2, we get Ozk
(f+1) 10: wki
dehi (a (a(tt))) and ac (e+1)
(1+1)
8C dz!?]
= Ex Dz
ac
(2 (el))
Wii of (a (e+1)), (a (0),
0z
15
02
ac
= ac 82
(t-1)
B2 Bu
02 2j
according to the relations
2,
(f+1) = o (a:
(++1) and 2; = or(a (8)). (1)
Let ((() = |ac/ /0=(") ..... 00/020010 E Rm 0f 10/(aa (0) ), ... of Halll)) E 1S
The gradients can be written in a dense form:
(z(L) 2) (L-1)T E RELXHL-1
waz
8 E-
W(L) (L) v) ER (8(1--1) 3 _6¹_ - E R
W
(12)
8(0)
((1+1)((81+11) @ 2'uui) Rm
V WCC
(810 oz) Rmexne-1
13

---

Under review as a conference paper at ICLR 2022
B WEIGHTED DEGREES OF GB
We examine three hidden layers {e - 1, 1 of GA and three neurons on these layers {ī, i, k}. (f+1) connects the neuron & layer e + 1 to the neuron i layer E, wi be the synaptic connection weight between the neuron on layer e and the neuron i on layer e - and w/fm" connects the neuron on layer & - 1 and the neuron on layer & 2-2. 2.
(f)
Let uki
Now we have close look at 0c/0ul According the chain rule, we have
ac ac Oz. (0) (e-1) )((())(((a{)) =2 = Σ b yu
k deti
ju
(e+1)
auly (2)
dz, (e)
Ouly (t)
The gradient term & ((++)) aC / (6+1) is highly coupled function of all accessible synaptic connection weights of w on the forward propagation route from 2 (P) to the output neurons. with no synaptic
To ease the analysis, we simplify it with numerical value synthetic connection weight included. Therefore, the summation term can be viewed as a simple linear
function of all synaptic connection weights Why (f+1) associated with neuron on layer é, and the associated coefficient is p(wki (f+1) (()) = ((--1) oo(aa ((e+")) (' (
which defines the to wif on GB. Similarly, we have the edge weight from wij to w/(e"" i.e., plu (f) Wjm " (L-2) 10-3)''-1(0 (a) yo! da Therefore, we are able to calculate the in- degree and out-degree of wig
edge weights from Wki (E-1) (1) ..ig
(f) which are defined as the sum of the weights of all in-bound connections to Wi and the sum of the weights of all out-bound connections from wiy i.e.
om (w =2- -2)--1 (e-1)
o'(a;:)) (((aa((())))
Σ
b da ((+))))
((+1)
(13)
Sont (wl) (0)
Σ 1K-2 (e- 1) 1o (((
Joe
(14)
m
There are several exceptions, including the first hidden (l = 1), the last hidden (E = L 1) and the output (L L) layers. For the output layer, we have
oc au!!
Σ Oc 02k (L) (L)
= f1- (2) ys).
(15)
Oz
dw!y
Because dL is softmax, no explicit relation regarding w (T) can be built. It implies that no well-defined in-bound connections to wif i.e., dIn (m((w!f)) = But, we can build the connections from wifh to
w ((--)) is easy derive
OC
ac 0zft--1
(I-
o (a (a (1-11)
Σ
(2k
(1)
ys
yx) (L)
(16)
11-1] Du
Oz (1-11 Ow (L-1)
k)
(L-1) we get pf Mukk Wij
(L) (b-1))
(L-2)
02 (a (() yk); from
From the perspective of Wij the perspective of ! we have p( (L) (1-1))
211 (L-2)
0 (
(L-
(x. -y:). So we have
Sour(w (w (L))
= =1Em (1-2) of (a
(L (1-1)
(2) (L) -V)),
2m
8in (w (1-1))
= =zg
(1--2)
o (a
11-1))
(2 (L) - y))) = 0,
dont (103)
11-1))
[Imm 2m
(1-3)
"o (a)
2) 0o (011 (1(1-))))
(1-1)
The softmax (-) makes the output values sum up to one, i.e., Σ Ub = 1, and 8in (w)) (1 0. Now, we examine the first hidden layer. Similar to the output layer, there no well-defined out-bound connections for wiy Sout (w:g) 0. Setting f = 1 in Eq. 13 we can get the in-degree of w!)
Jin (w! (3) = 2500 (a Σ
6 og (a (2)
14

---

Under review as a conference paper at ICLR 2022
Based on definition of the weights of GB. when the number of layers is small, is trivial that Bel = 0. To get a non-trivial Sett. we identify the minimum number of hidden layers in GA. First, examine G with one hidden layer, i.c. L = 2, whose degrees are
din (w (2) = onut ( (1) = din (w:gg 0, Sout (wij ( [m 2m ]o M=
(2) vı)
Since the degrees up to zero, Bett O, regardless of how many hidden neurons in GA.
GA only has two hidden layers, i.e. L = 3, the in-degrees are
8in (w (1)) 2j
(0) (a (1) [E k= o/ (0)
8in w (2) = ( (3) = O.
The out-degrees are summarized as follows:
Sout (W) 0,
dout (2)
[mm=
of [a (1) 02(0, (2)) (2) (2 y1). (3)
2m
Sout (Wig
(3)
E -m=1 2m 02
(1)
The total degree may be non-zero, but Bei O alway holds.
Therefore, the minimum number of hidden layers required for a well-defined 3er is three, i.e., L 4. We summarize the in-degrees
Sin (m (w)
()) o9
(1)
Ex 8(2) o5
8in (w
2j
oi (1)) ][ ox
(-11) o'(aa
g(tt)) oft
oa
(at
(1 <
dma (wig (I-1))
=0,
(I))
8mm (wis"
= O.
and the dut-dessees
Sout (w
(1)
0,
Sout (w ij
(2))
Em 2lth 2) of (a
(1-2)
(f-1) o((a)) v1 < L-11
(L-1))
(L-3) o 20 1
16! o ((((1))).
Sout (wij
Em am
Saut (wrg)
(L))
Imm am
(L-2)
(L) -w).
It is easy to derive
8 dout
Ens Erelel-1 8m (m(
(1lg Dout (me (6)
(1)
= Ens n. Excrel11 [E
(E-2)
8-1 (aj (o((aa)))))))))))) (9))61) [E of
g((t2)
oe (a ).
Eleret [1" 2(t-21] 1 (1-1) ol o' x 1'1000 o off 1 (8 d'41l-
Now, we move forward to compute the total degree
1 Tm
Ey 1 (a (1( [E, of Ei Ei (((a(9)) En of o' (0 (o(+t)))]
s((+1))
2
= 1J 1 x x ooll + Encre
11 x [1 x T ge O o'
= Ei ((-)) x [1 of x g16+1) 1
The definitions of in-degree and out-degree ensure that 1Th 1ouuu must hold. Let's prove it:
1 I dou..
= Eig [Eiess En, m am of (a) (a (1-1)
(6-2)
5 (a (0) + Em zm _(L-2) ( d' (() - y;)]
= - (( x [ [ x 1T ( @ o
= 1 1'5m
With the fact that 0²2² = o for ReLUU, according to Fq. we have
Eh T 2 (f- 2)
0. 1 17
o off
x 1 ((++1) e c
Best =
2) x [To (T) o']
15

---

Under review as a conference paper at ICLR 2022
C DERIVATION OF ADIACENCY MATRIX P OF GB
The right hand side (RHS) of Eq. is a function of W(++1), and denoted as F(W(+t)).
Here we derive the strength of the impact from W(++1) and other weights W(- = (W(0), W(.)..., W(O,, w(++2)
W(L) for building the edge dynamics. Let W
(W(++1), W(-0) and F(W) = dw /dt. We denote V)(()) as the current states of W(-0)),
W+(++1) as equilibrium point, and W = (W+(t+1), W(--)), According to the Taylor expansion, we linearize F at W* and have
dw F(W") |
BF(W ((t=")))))))))))))) (w(++1) W*(+)) aw
aF(W (W(-+) W(-D). awt-n
The last term on the RHS can be cancelled out when the realizations of W(-O) of ( i.. W(-O), The gradient is simplified as
take the current states
dw F(W") + (P(W-(+1),,WC-
OW((+1)
The term (F(W /0w PC(WV(4+)) / aw((+1) effectively captures the interaction strengths between W(O and w(++1), because measures how much F affected by unit perturbation on W(++1), Usually, w=((+1) are not available before updat- ing ((++)) following the update of W(!,, system can be viewed as realization of the general Eq. [ with linear f(W(0) = F(W*) and
use the current states of W(++) instead. The
g(W(), W((+))- = W-((+1) W(++1,, Now, we can immediately have the adjacency matrix P of G3 with p(l++)) (CC(W(())) /ar-)))/w(waw A1 visES
D PROOF OF THEOREM
The second order gradient p(I,-1) (PC//w(00awettt is proposed to measure the interaction strength between W(O and W(+1), V1 I l S L. Considering an MLP, and assume that each activation function de is ReLU for l L, when GA converges,
vanishes, i.e., v v
( (f-1)T = 0 (Eq. 12 in Appendix B It indicates that (8(()
)zz) (f-1) 0, i.e., either (f-1) 0, v(i,j). According Eq. 10 the numerator involves the product of
( o 0 or 2j
terms (00) and z (L-1)), which are zeros
7
so Beet = 0.
E BAYESIAN RIDGE REGRESSION
Ridge regression introduces an 6z-regularization to linear regression, and solves the problem
argmin(y X0) x0) 0|
(17)
where E Raxd, y E R", E Rd is the associated set of coefficients, the hyper- peper-parameter controls the impact of the penalty term [0) 2.
Bayesian ridge regression introduces uninformative priors over the Syper-paramettee of the model, and estimates a probabilistic model of the problem in Eq. 17 Usually, the ordinary least squares method posits the conditional distribution of y to be a Gaussian, i.e., p(y X 0) = A A'yy) X0, a Ia). where > 0) is syyper-prammter to be tuned, and La is d'x d identity matrix. Moreover, if we assume a spherical Gaussian prior 0, i.e., p(0)= = =0 La), where T > 0 another hyper-parameter to be estimated from the data at hand. According to Bayes' theorem, p(0) X, y) p(0)p(y)XX, 0), the estimates of the model are made by maximizing the posterior distribution p(O X. 3). i.c.,
arg max log p(0 y) arg max / (y) + log (0 - L2), 0
which is maximum-a taniimum-a-posterioni posteriori (MAP) estimation of the ridge regression when À = o2 2 All and are estimated jointly during the fit of the model, and a TVX.
A small constant is added the denominator of 8Br avoid division by zero
16

---

Under review as a conference paper at ICLR 2022
To estimate I = h(Ser: 0), scikit-leary" which is built on the algorithm deseribed in Tipping 2001) updating the regularization parameters À and T according to MacKay (1992).
F RANKING PERFORMANCE ON ALL FIVE DATASETS
CIFARIO
CIFARI00
SVIN
Fashion MNIST
Binds
Thus Pest Aee
0.96
088
1.00
DezseNet
0.96
0.94
Iaeopiie
0.94
0.96
MakdiNe
0.95
Reshit
93 Spearnan Carr.
0,70 Speaniin Corr.
0.84 Speamtnn Clrr.
0.95 Spearman Corr.
0,74 Speaman Clorr
VDG
0.92
0.92
Xesyfon
0.92
6.94
0.96
0.7 0.
0.95
0.6
0.35
0.94
0.92
0,96
1.00
Prodicted Apc.
Predicted Ave.
Prodicted Ace.
Predicted Ace.
Predicted Acc.
Figure F4:: Predictions of the validation accuracy of pre-trained models all five datasets based on Bett v.s. true test accuracy of these models after fine-tuning for 7 50 epochs. The Spearman's 's ranking correlation p used quantify the performance in model selection. Each shape associated with one type of pre-trained models. Distinct models of the same type are marked in different colors. To be noted, each includes AlexNet in computing ps.
CIFAR10
CIFAR100
SVHN
Fashion MNIST
Birds
Sparrnnn Catr. e
1.00-
0.75
Oues
0.50
BSV
0.25
10
15
10
15
10
15
10
15
5
10
15
Length Leumnnng Curve
Figure F.5: A comparison between our Bott based approach and the baselines in model ranking.
G RUNNING TIME ANALYSIS
500
Figure G.6: Training time per epoch versus computing time for Bet per epoch all 17 pre-trained models and five datasets discussed the main text. Each data point is associated with one pre-trained model over one dataset. The relative of our approach in comput- ing Bell with respect to training more epochs can be measured by average, it is ē Rs 1.3 (slope of the pink line).
CIFAR 10
3m
300
CIFAR 100
SVHN
Fashion MNIST
100
1.332
Birds
100
300
500
Tg... /Turain- On
Trran
Table G.2: Running time (in seconds) in learning curve prediction, includ- ing the predictor estima- tion (if necessary, e.g., Ours, CL and BGRN).
Dataset
Ours
BGRN
LSV
BSV
CL
CIFAR 10 CIFAR 100 SVHN
0.491
0.610
0.059
0.049
3966. 128
0.414
0.628
0.051
0.045
5256.478
0.506
0.607
0.074
0.044
4690.507
Fashion MNIST
0.493
0.625
0.057
0.046
4552.194
Birds
0.460
0.636
0.071
0.044
4734.992
inar_mode!
Tayesiankiddge.he html
17

---

Under review as a conference paper at ICLR 2022
H MEAN-FIELD APPROACH
We summarize the main idea of the mean-field approach developed by Cioo et al. (2016), and show how Eq.
3 is obtained (Jiang et al.) 2020a
Based on the notations described in Section 3 we consider a vertex and the interaction term
E, P.gg(T;;;;) 25) in Eq. | where Poj is the influence j has on i. Similarly, influences / with
weight PJ.. We define the in-degree 6thh = Ej Pi and the out-degree or = Σ, Pyt- The interaction term can be rewritten as
P 6." E, Pygg(1,,z))
(18)
Σx
Here the in-degsees 8tn captures the idiosyncratic part, and the average g(::-) captures the network effect. The mean-field approximation to replace local averaging with global averaging, which approximates the network impact on a vertex nearly homogeneous. Specifically, we can get
E, Pig((nn))))) PiR
Eig Pug((an 1"Pg(z,,
(19)
Eie Pik
1TP1
where the vector g x)) has the jth component g( zz). A linear operator
TTP gout z P1 z gout 1
(r(2)
(20)
defined for a weighted average of the entries z. The mean-field approximation gives
c = f(x,) + 8thh Cp(o(z;,)))
(21)
In the first order linear approximation, we can take the L--average inside g. The average of external interactions is approximately the interaction with its average, i.e. Lp(g(Z,, æ)) z g(z,, (p())) and
i f(x)) 6t g(xx;
(22)
where Cp( is global state. Let Vay - (ar) Applying Lp to both sides of Eq. 22 gives
[x]f( Lp(8 xm
(23)
According the extensive discussion and tests in (Gao et al. 2016), the in-degrees gin and the
interaction with the external Tay are roughly uncorrelated, so the pvaeragee of the product is roughly the product of --avvergess. Therefore, Lp(5'n g(x, x L (5 pgg((,, x.v)) Using the first order linear approximation, we take the Ppaaverege inside f and
L =[(C(()) + CC
(24)
Therefore, we have
fav = [Fav) + Beag (Favi Fav), where Best (p(iin) is the resilience metric, and its steady-state is the effective network impact Teft. satisfying det = (xxtt) ) + ( kott) 0.
I CORE PROCEDURE
Our framework is built on several different techniques and the related contents are dispersed in different sections. Here we briefly summarize the core idea of this paper and show how these sections are organized, see Fig. I.7 for a flowchart of our core procedure.
dynamical system, and directly model the evolving of the trainable weights the SGD based training as set of differential equations (Section 3 characterized by general dynamics in Eq. Usually, it is convenient to study the dynamics of agents (trainable weights in our case) on a regular network, where each node represents an agent in the dynamical system and the interactions of agents in network science and dynamical systems, e.g. the universal metric 3.l developed by Gao et al. (2016) to quantify and categorize various types of networks, including biological neural networks
We view the NN training
governed by Eq. [ Many powerful techniques have been developed
18

---

Under review as a conference paper at ICLR 2022
(Shu al. 2.21). Because of the generality of the metric, analyze how it looks artificial neural networks which are designed to mimic the biological counterpart for general intelligence. Therefore, an analogue system of the trainable weights under the context of the general dynamics is set up in framework. To the end, we build a line graph for the trainable weights (Fig. T and Section 4.1 and "rewrite" (Section 4.2 and Appendix C the training dynamics in the form of Eq. [ which inc ludes
self-driving force f(-). an external driving force :) and an adjacency matrix P (Eqs.
The reformulated training dynamics yields simple yet powerful property. It is proved that as the neural network converges, Seif approaches zero (Theorem I in Section 4.3 also one of our primary contribu-
NN Training as A Dynamical System
Network
Dynamies
Neural Network
Training Dynamics Eqs. 6 &
tions). As shown in Fig. (c) and Section 1 we
exploit the property to predict the final accuracy of neural network model with a few observations during the early phase of the training, and apply it to select the pre-trained models (Algorithm T Section 4.4). Generally speaking, the metric Belr should be calcu- lated for the entire neural network. However, because many oate-off-the-art neural network models have large-scale trainable weights. If all layers are consid- ered, will be prohibitive to compute the associated Sef.. We make a compromise, and estimate Berr of the entire network using Ber of the NCP unit (i.e., partial part of the entire network, see the second to the last sentence of Section 4.3). It's confirmed from our empirical experiments (Section 5) that the simplified, lightweight version of Betr still effective in predicting the final accuracy of the entire network.
Section
Section 4.2
Line Graph
Edge Dynamies Eqs. &
G₁
Mean-Fiell based Approach
PTP
Pen
relies on the adjacency matrix P (Eq. 8)
Sections 4.3 & 4.4
Learning Curve Prediction & Model Selection
Option
X
Option 2
Full G2 Gfull
glull
NCP G2
goOP
Figure 1.7: A flowchart of the core procedure of our framework.
The metric Berr developed by Gao et al. (2016) is universal to characterize different types of networks. Although our framework utilizes the metric, our application to artificial neural network training dynamics and the related theoretical results as specified by Theorem D are novel. Specifically, is applied to study the NN training (Section and predict the final accuracy of an NN with a few observations during the early phase of the traming (Fig. (b)..[p)) But 3.н relies on the adjacency matrix P of GA (Eq. [3) To derive P, resort reformulation (Section 4.2 and Appendix C of the training dynamics in the same form of the general dynamics (Fq. [ One issue in calculating 3er is the complexity if the entire G is considered. As resolution, we propose use the lightweight 3ell of the NCP unit- partial of G to predict the performance of the entire network (Sections 4.3 &
19


Reviews:
[{"review_id": "_ILC2wE4c6", "reviewer": "Reviewer_acZh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n1. Application of dynamic systems theory to NNs.\n\nWhile the theory described in the paper is not novel, the application of it to the task of predicting model performance is novel as far as I can tell.\n\n**Weaknesses:**\n\n1. Clarity\n\nThe paper uses multiple prior ideas from dynamical systems and machine learning literature but many of these choices are not well motivated or explained. This makes it difficult for a broad ML audience to understand the paper and its significance.\n- The paper draws heavily from theory described in Gao et al 2016 which proposes a method to convert the reslience function on a multi-dimensional system to a single dimensional function x_eff where the critical points B_eff are a one-dimensional value. Gao et al 2016 describes this approach for a non-neural network system and it is unclear why the assumptions introduced in this paper apply to neural networks.\n\nThe simplifying assumption used in the paper is that the dynamics of a node in a complex multi-dimensional system can be characterized by the average nearest-neighbor activity. However, in a neural network, perturbing a single weight doesn't only affect the gradients of the immediate neighbors of the node. Thus, it is unclear why this approximation is appropriate for neural networks. This is briefly touched upon in section 4.2 where it is stated that \"the others’ contribution as a whole is implicitly encoded in the activation gradient\" however the connection to the original theory described ini Gao et al 2016 and why this is an appropriate simplification needs more explanation.  \n\n- Next, the motivation for the NCP is not discussed in the paper. By initializing it randomly and freezing it, a random amount of fixed noise is introduced into the process and it is unclear what the purpose of this is. Additionally, is this randomly initialized each time for all the experiments? And if so, are the same experiments repeated with different initializations to control for randomness?\n\n- What is the motivation of using the second order gradient of C to quantify the interaction strength?\n\n2. Correctness\n- In Appendix B, section D \"PROOF OF THEOREM 1\", it is claimed that the gradient vanishes when one of the terms of the product is 0. Then, it is stated that when either goes to 0, the numerator in beta_eff goes to 0. However, it appears that if the gradient (left) term goes to 0, the denominator also goes to 0 since the term is also present in the denominator product. This will result in beta_eff being undefined. Please clarify whether this is accurate.\n\n3. Details\n- In the last sentence of section 4.2, the self-dynamics part, f(w_i) is defined as F(w_i*), however, I couldn't find where this was defined. What does this correspond to in the context of neural networks? Additionally, why is g(w_i, w_j) = w_j - w_j*?\n\n4. Figures\n- The figures are small and difficult to read. A better presentation of the results such as a table could be added to the main paper while the figures are either added to the appendix or enlarged.", "summary_of_the_paper": "The paper formulates neural network training as a dynamical system and uses existing theory to formulate a metric (beta_eff) to predict how a pre-trained model will perform on a downstream task without requiring to train the model to convergence.", "summary_of_the_review": "The paper introduced a novel formulation of neural networks based on prior dynamical systems theory for an important machine learning problem. However, there are some parts of the paper that require further explanation and in its current state, I think it is fairly difficult for a broad ML audience to understand the paper and it's significance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Y8ITgIKVEe0", "reviewer": "Reviewer_A1LA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is novel. Experiments show that the proposed neural capacitance more accurately predicts the accuracy of the model. Ablation study have been performed to study the effect of parameters such as starting epochs, size of training set, etc.\n\nOverall, I think the paper is potentially a good paper, though I'm not an expert in this topic and does not fully understand the mathematical derivation in Section 4.2 and 4.3. My major concern is\n\n- I'm not sure if I fully understand NCP units. Why the weights are randomly initialized and freezed during finetuning? And in Figure 1(b) does the output layer of right stack do the same thing as output layer of left stack for classfication tasks? I do not find the relationship between the new layers and what is described in section 4.2 and 4.3. Why two dense layers are chosen to be the architecture of NCP units? I think this part is missing from the paper and should be elaborated to build the connection between the previous section.\n", "summary_of_the_paper": "Accurately predicting the performance at early stage is important for efficient model selection without incurring too much computation. The paper proposes a neural capacitance metric as a predictive measure to capture the performance of a model on the downstream task using only a handful of early training results. The metric is derived from a line graph mapped from a neural network by modeling the dynamical system of the network.", "summary_of_the_review": "Overall, I think the paper proposes an interesting idea to model the training NN as a dynamical system and is potentially a good paper, but the some part of the method needs more elaboration.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Mxb6rrcjy5l", "reviewer": "Reviewer_EL9b", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am not an expert on this research topic. maybe I cannot give a precise review.\n\nStrengths:\n\nThe whole paper is written in high-quality and a clear manner.\n\nTo validate the outstanding performance of the proposed method, the authors conduct experiments on many popular CNN models.\n\nWeakness:\n\nI am not 100% sure about the performance, because the author reports the accuracy with figures rather than tables.\n\nThe authors conduct the experiments on well-known models, what about applying this framework to select the subnetwork on many NAS benchmarks e.g., [1].\n\n\n[1]. Hw-nas-bench: Hardware-aware neural architecture search benchmark \n", "summary_of_the_paper": "This paper proposes a framework to select the neural networks for downstream tasks. To identify the better generalization model, the authors propose a new metric (Neural Capacitance, NCP) to predict precise learning curves. And the authors provide the theoretical explanations for NCP.\n\nThen the authors have verified the advantages of the proposed method when being applied to different datasets (CIFAR10, CIFAR100, SVHN, Fashion MNIST, Birds) with 17 CNN models.", "summary_of_the_review": "NA", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for inferring which of a set of pretrained neural networks, once fine-tuned on a transfer task, will generalize the best. This is accomplished by deriving a quantity based on a mean-field approximation of a dynamical system defined on the adjacency matrix of the weights of a neural network, known as the "neural capacitance". The model selection procedure involves attaching a fixed, randomly initialized network onto the outputs of the pretrained network and fine-tuning for a small number of iterations, and computing the metric; the fixed network is called the "neural capacitance probe" (NCP).

Reviews, though low confidence, awarded borderline scores, and a central concern was clarity and motivation, in particular the role of the NCP. acZh, the highest confidence and most verbose reviewer, echoed these concerns along with specific criticisms, for example about the heavy reliance on Gao et al (2016) without elaboration. The authors have responded in considerable depth but unfortunately the reviewer has not acknowledged these responses. On the NCP, the authors note that this is an approximation to the ideal metric that they have empirically validated.

Reading the updated draft, I find myself still concurring with reviewer acZh in large degree. The draft has improved with the noted additions, such as Appendix G devoted to an explanation of Gao et al (2016), but the presentation is still quite challenging to follow. I am left with fundamental questions about the soundness of the approximation being made, its wider applicability, and the many arbitrary decisions regarding the architecture of the NCP that appear out of nowhere. How sensitive is the procedure to these choices? Did the authors tune these architectural hyperparameters? Using what data? The table of results does not include units, and for a paper proposing a general purpose metric I'd ideally want to see a a robust rationale for hyperparameter selection of method-specific hyperparameters as well as a rigorous statistical treatment of the method's performance. Since it involves an approximation, a comparison to the "ideal" or "exact" procedure on a toy problem where the latter is feasible would strengthen the paper considerably. I do appreciate the breadth of architectures and datasets examined, but I believe the central focus of the paper should be explaining the mathematical motivation (perhaps at a higher level and deferring more detail to the appendix), why precisely it makes sense in the context of neural networks (also raised by acZh, with an answer provided that I believe partially addresses this) and justifying the concrete, approximate instantiation of the method involving the NCP and the hyperparameter selection and evaluation protocol that led you to the particular NCP employed.

At a higher level, this is a very mathematically dense paper that relies considerably on concepts outside of what might be considered typical expertise in the ICLR community, reflected in the confidence scores of the reviewers. While I feel that the issues described above already preclude acceptance at this time, I believe it may be difficult to do the proposed method justice in the short conference paper format, and would suggest to the authors to consider a journal submission instead, where a didactic presentation can be given the full attention it deserves without the difficulty created by length constraints.

Finally, I'd like to apologize to the authors for the non-responsiveness of the Area Chair. The original Area Chair was not able to complete their duty and I have been belatedly assigned this paper to evaluate it, and it is clear that not as much discussion took place as would have been ideal.
Decision: Reject