Convolutional Neural Network Dynamics: A Graph Perspective

1 INTRODUCTION . Neural networks ( NNs ) have driven advancements in many domains , including computer vision and image processing ( Hu et al. , 2018 ) , natural language processing ( Sutskever et al. , 2014 ; Bahdanau et al. , 2015 ) , and bioinformatics ( Cao et al. , 2020 ; Li et al. , 2019 ) . As task complexity increases , networks grow deeper and larger , consequently requiring more computational resources and training data , as well as sacrificing interpretability for improved task performance . Some works have focused on understanding and interpreting deep NNs ( Raghu et al. , 2017 ; Chakraborty & et al. , 2017 ; Ioffe & Szegedy , 2015 ; He et al. , 2016 ) . One approach towards this goal involves representing the NN as its underlying graph structure , and studying selected graph properties , such as clustering coefficient , path length ( You et al. , 2020 ) , modularity ( Filan et al. , 2021 ) , persistence ( Rieck et al. , 2019 ) . For example , You et al . ( 2020 ) represent NNs as relational graphs capturing the message passing process , and investigate the correlation between the predictive performance of NNs and architectural changes . However , the studies of NN structures as graphs are limited , and the structural changes of the underlying graph during the training process have been largely overlooked in the literature . To fill this gap , in this work , we take the graph perspective and aim to predict the performance of an NN by capturing early NN dynamics during the training phase . Successful performance prediction based on only a few epochs could be used for early stopping ( Yu & Zhu , 2020 ) , and thus , more efficient NN training . To solve the performance prediction problem , we propose a multi-step framework , depicted in Fig . 1 . Specifically , we propose to represent the underlying graph structure of an NN as a time-evolving k-partite graph , where each part corresponds to a different NN layer , and each graph snapshot in the evolving graph maps to an NN instance at a specific epoch . We build on existing graph representations of fully-connected and convolutional layers , both of which are key components of popular NN architectures , and introduce a new , compact , efficient-to-compute ( “ rolled ” ) graph representation for convolutional layers . Then , we extract well-known node features ( weighted degree and eigenvector centrality ) from the time-evolving graph and construct temporal signatures by computing summary statistics on the node feature distributions . Finally , we cast NN performance prediction as a classification task and a regression task , each of which operates on the temporal structural signatures of the NN . Our main contributions are : • New graph-based NN representation : We introduce a new graph representation for convolutional layers that is more compact and efficient-to-compute than the existing unrolled representation ( Rieck et al. , 2019 ) , while not sacrificing accuracy in the NN performance prediction task . • Framework for NN performance prediction : We propose a simple , multi-step graph-based framework to solve the NN performance prediction problem by capturing the early NN dynamics during the training phase . • Extensive empirical analysis : Using well-known image classification datasets ( ImageNet and CIFAR-10 ) , and a variety of NN architectures ( AlexNet , VGG , LeNet , and ResNet ) , we show that our framework can effectively predict the performance of NNs by observing only a few epochs of training , well before their corresponding early stopping epochs . For instance , using our framework to capture the changes in the graph structure in only 5 training epochs of the ResNet architecture results in classification accuracy of over 90 % . 2 PRELIMINARIES . We first present the key concepts that our work builds upon . Table 1 gives the major symbols and their descriptions . An NN model is a collection of connected units ( neurons ) that are organized in layers , and is defined by a set of parameters that adjust during the training process . We refer to the training process of a single architecture along with its respective hyperparameters as an ‘ instance ’ . We focus on two types of layers that are the key components of many powerful NN models such as LeNet ( Lecun et al. , 1998 ) , VGG ( Simonyan & Zisserman , 2015 ) and ResNet ( He et al. , 2016 ) : fully connected layers ( fc ) and convolutional ( conv ) layers . 2.1 GRAPHS : TERMINOLOGY AND NOTATION . Let G = ( V , E ) be an undirected , weighted graph with node set V , edge set E , and weighted adjacency matrix W ∈ R|V|×|V| . The neighbors of node v are defined asNG ( v ) = { u : ( u , v ) ∈ E } ; i.e. , the set of all nodes that connect directly to v. Graph G is k-partite if its nodeset V can be partitioned into k independent sets : V = ⋃k i=1 Vi and Vi ∩ Vj = ∅ , i 6= j . A time-evolving graph is a series of static graph snapshots over time : G = { G1 , G2 , ... , GT } . The graphs in the series may have different nodesets and edgesets . Here we focus on two of the most commonly-used node features in graph mining and network science , degree and eigenvector centrality , which capture different types of node importance or influence . We discuss the interpretation of these features in the context of neural network dynamics in § 3.2 . • Degree : Degree is the simplest and most efficient to compute node feature , and captures the connectivity of a node . The weighted degree of node v is defined as the sum of weights of the edges that are incident to i : dwi = ∑ jWij . Tracking changes in the degree of a node over time is the most direct way of capturing its structural evolution . • Eigenvector centrality : This centrality is a sophisticated extension of degree centrality , related to Google ’ s PageRank , which indicates the influence of a node in G ( Bonacich , 1972 ) . If a node is connected to several nodes with high eigenvector centrality , then that node will have high centrality . The eigenvector centrality of node i is defined as the ith element in the principal eigenvector v of W ( i.e. , the eigenvector that corresponds to the largest eigenvalue λ ) : Wv = λmaxv . 2.2 FULLY-CONNECTED LAYERS : GRAPH REPRESENTATION . Let x ∈ RDin be the input vector , W ∈ RDin×Dout the learnable weight matrix , b ∈ RDout the bias vector and z a non-linear activation functions . Then , the output of this layer is given as y = z ( xW+b ) , where Din and Dout correspond to the dimension of input and output layers . Fully-connected layers are straightforward to represent with a weighted , undirected graph ( Filan et al. , 2021 ) . Each neuron , including those in the input and output layers , corresponds to a node . Two neurons are connected via an edge if they appear in consecutive layers . More formally , let nodeset Vi be the set of all neurons at layer i , and nodeset Vj the set of all neurons at layer j ( which follows layer i ) . The edges that connect all nodes across the two nodesets are defined by the learnable weight matrix W ∈ R|Vi|×|Vj | . 2.3 CONVOLUTIONAL LAYERS : UNROLLED GRAPH REPRESENTATION Let X ∈ Rc×h×w be the input and K ∈ Rf×c×hker×wker the convolutional kernel , where c is the number of channels , and hker , wker are the height and width of the kernel respectively . After performing the convolution , we have Y ∈ RF×hout×wout , where hout = h−hker+1 and wout = w−wker+1 . Typical multilayer CNNs consist of convolutional layers followed by fully-connected layers . The representation of fully connected layers is straightforward but can not be used to model convolutional layers . Rieck et al . ( 2019 ) proposed to `` unroll '' the convolution to convert conv layers into graphs . In Sec . 3 , we introduce a new , compact , rolled representation that is both efficient to compute and effective at predicting the NN performance , as we show empirically in Sec . 4 . For a convolutional layer , we first unroll the convolutional operation and then represent the graph as in the case of a fully-connected layer . In this representation , the nodes and edges of the graph are defined through the convolutional operation as matrix multiplication ( Gebhart et al. , 2019 ) . Specifically , for an input image X and kernel Ki ∈ Rfi×ci×hi×wi , each node in layer l is defined to be the output of the mapped feature of that input for each filter fi . Edges connect each of these nodes to the corresponding nodes of the output neuron in the next layer . These edges are weighted by the activation value of that neuron ( i.e. , the input image for a specific stride is multiplied by the filter value at that location in the image ) . For an input image filter size ( ci × hi × wi ) that results in feature-map ( output ) of size 1 × oh × ow , the number of nodes of output layer of the graph representation is a function of oh × ow , while the number of nodes in the conv operation layer of graph is a function of ci × hi × wi . As an example , Fig . 2 depicts the unrolled graph representation of a toy convolutional layer . It shows one conv operation on a small 2-dimensional input image ( 4× 4 ) and filter ( 2× 2 ) . The nodes and edges of the represented graph are shown based on one stride ( of size 2 ) of convolution operation . 3 NEURAL NETWORK PERFORMANCE PREDICTION : A TEMPORAL GRAPH-BASED APPROACH . In this section , we first formally introduce the problem that we seek to solve . Then , we present a new , compact , and efficient-to-compute graph representation for convolutional layers ( § 3.1 ) , and describe our proposed temporal graph-based framework that captures the NN dynamics during training ( § 3.2 ) . Problem 1 ( NN Performance Prediction ) Let N = { Ntr1 , Ntr2 , . . . , Ntrn } be a training set of n NNs trained for T epochs and A = { α1 , α2 , . . . , αn } their corresponding downstream task accuracies ( e.g. , for image classification ) . We seek to predict the accuracy αtst of a new instance Ntst trained for a very small number of t T epochs by using t epochs for the trained NNs in N . Our proposed solution takes the graph perspective , and its first step is to represent each NN as a temporal graph . In addition to the graph representations that we presented in Sec . 2 , we introduce a new , efficient representation for convolutional layers , which we describe next . 3.1 ROLLED GRAPH REPRESENTATION FOR CONVOLUTIONAL LAYERS . We call our proposed graph representation for conv layers “ rolled , ” since it avoids unrolling the convolutional operations introduced in ( Rieck et al. , 2019 ) . Overview & Motivation . To preserve the semantic meaning of conv layers , we represent each filter as a node , and link filters in consecutive layers via weighted edges ( we define the weights below ) , as shown in Fig . 3 . The motivation behind this approach is that for larger networks , there is an explosion of nodes by unrolling the convolutions ( unrolled representation explained in § 2.3 ) , and the integrity of a unit ( a single filter or kernel ) becomes untenable . Also , by mapping the nodes to specific entities of NNs , our proposed graph model is more interpretable than the unrolled model , and thus it is easier to interpret the outputs of downstream graph analysis on our graph representation ( e.g. , computing node features , tracking the evolution of the graph ) . Detailed Description . Formally , let tensor Ki be a kernel in layer i with fi filters , each with ci channels and dimensions hi×wi . We use bracket notation to index into the kernel : for example , Ki [ l , : , : , : ] indexes the lth filter of kernel Ki . We create fi nodes representing each filter { v ( i ) 1 , v ( i ) 2 , ... v ( i ) fi } , with features defined as the corresponding biases in that layer . Let Kj be the next convolutional layer defined analogously . While edge weights between neurons in FC layers are defined in the standard way , we define the edge weights between conv layers as the norm over each kernel ’ s channels . The edge between node v ( i ) k representing the k th filter in layer i ( i.e. , Ki [ k , : , : , : ] ) and node/filter v ( j ) l in layer j ( i.e . Kj [ l , : , : , : ] ) has weight wv ( i ) k , v ( j ) l = norm ( Kj [ l , k , : , : ] ) , which is the norm of the kth channel of the lthfilter in the jth layer . Though the proposed representations enable alternative edge weight configurations , we focus on the norm of filters as other studies , including those on pruning NNs ( Li et al. , 2016 ) , have demonstrated that this metric has strong correlation with filter importance . In the case of two conv layers , the resultant graph is an attributed bipartite graph with fi+ fj nodes and fi× fj edges , where node attributes include flattened weight vectors or filter maps . Other information such as average gradients or the bias vector can also be used for node features .

Summary: The purpose of this research is to attempt to forecast a neural network's performance from early NN dynamics. The dynamics of the neural network are stored in a time-evolving graph. The summary of two node properties, the weighted degree, and the eigenvector centrality is extracted and utilized to train classification and regression algorithms. This methodology, when combined with the paper's suggested rolled graph representation for convolutional layers, results in faster prediction than the classic early stopping method.  The paper's primary contribution is that the suggested rolled graph model for convolutional layers improves time and space efficiency in comparison to the existing unrolled representation.