Convolutional Neural Network Dynamics: A Graph Perspective

1 INTRODUCTION . Neural networks ( NNs ) have driven advancements in many domains , including computer vision and image processing ( Hu et al. , 2018 ) , natural language processing ( Sutskever et al. , 2014 ; Bahdanau et al. , 2015 ) , and bioinformatics ( Cao et al. , 2020 ; Li et al. , 2019 ) . As task complexity increases , networks grow deeper and larger , consequently requiring more computational resources and training data , as well as sacrificing interpretability for improved task performance . Some works have focused on understanding and interpreting deep NNs ( Raghu et al. , 2017 ; Chakraborty & et al. , 2017 ; Ioffe & Szegedy , 2015 ; He et al. , 2016 ) . One approach towards this goal involves representing the NN as its underlying graph structure , and studying selected graph properties , such as clustering coefficient , path length ( You et al. , 2020 ) , modularity ( Filan et al. , 2021 ) , persistence ( Rieck et al. , 2019 ) . For example , You et al . ( 2020 ) represent NNs as relational graphs capturing the message passing process , and investigate the correlation between the predictive performance of NNs and architectural changes . However , the studies of NN structures as graphs are limited , and the structural changes of the underlying graph during the training process have been largely overlooked in the literature . To fill this gap , in this work , we take the graph perspective and aim to predict the performance of an NN by capturing early NN dynamics during the training phase . Successful performance prediction based on only a few epochs could be used for early stopping ( Yu & Zhu , 2020 ) , and thus , more efficient NN training . To solve the performance prediction problem , we propose a multi-step framework , depicted in Fig . 1 . Specifically , we propose to represent the underlying graph structure of an NN as a time-evolving k-partite graph , where each part corresponds to a different NN layer , and each graph snapshot in the evolving graph maps to an NN instance at a specific epoch . We build on existing graph representations of fully-connected and convolutional layers , both of which are key components of popular NN architectures , and introduce a new , compact , efficient-to-compute ( “ rolled ” ) graph representation for convolutional layers . Then , we extract well-known node features ( weighted degree and eigenvector centrality ) from the time-evolving graph and construct temporal signatures by computing summary statistics on the node feature distributions . Finally , we cast NN performance prediction as a classification task and a regression task , each of which operates on the temporal structural signatures of the NN . Our main contributions are : • New graph-based NN representation : We introduce a new graph representation for convolutional layers that is more compact and efficient-to-compute than the existing unrolled representation ( Rieck et al. , 2019 ) , while not sacrificing accuracy in the NN performance prediction task . • Framework for NN performance prediction : We propose a simple , multi-step graph-based framework to solve the NN performance prediction problem by capturing the early NN dynamics during the training phase . • Extensive empirical analysis : Using well-known image classification datasets ( ImageNet and CIFAR-10 ) , and a variety of NN architectures ( AlexNet , VGG , LeNet , and ResNet ) , we show that our framework can effectively predict the performance of NNs by observing only a few epochs of training , well before their corresponding early stopping epochs . For instance , using our framework to capture the changes in the graph structure in only 5 training epochs of the ResNet architecture results in classification accuracy of over 90 % . 2 PRELIMINARIES . We first present the key concepts that our work builds upon . Table 1 gives the major symbols and their descriptions . An NN model is a collection of connected units ( neurons ) that are organized in layers , and is defined by a set of parameters that adjust during the training process . We refer to the training process of a single architecture along with its respective hyperparameters as an ‘ instance ’ . We focus on two types of layers that are the key components of many powerful NN models such as LeNet ( Lecun et al. , 1998 ) , VGG ( Simonyan & Zisserman , 2015 ) and ResNet ( He et al. , 2016 ) : fully connected layers ( fc ) and convolutional ( conv ) layers . 2.1 GRAPHS : TERMINOLOGY AND NOTATION . Let G = ( V , E ) be an undirected , weighted graph with node set V , edge set E , and weighted adjacency matrix W ∈ R|V|×|V| . The neighbors of node v are defined asNG ( v ) = { u : ( u , v ) ∈ E } ; i.e. , the set of all nodes that connect directly to v. Graph G is k-partite if its nodeset V can be partitioned into k independent sets : V = ⋃k i=1 Vi and Vi ∩ Vj = ∅ , i 6= j . A time-evolving graph is a series of static graph snapshots over time : G = { G1 , G2 , ... , GT } . The graphs in the series may have different nodesets and edgesets . Here we focus on two of the most commonly-used node features in graph mining and network science , degree and eigenvector centrality , which capture different types of node importance or influence . We discuss the interpretation of these features in the context of neural network dynamics in § 3.2 . • Degree : Degree is the simplest and most efficient to compute node feature , and captures the connectivity of a node . The weighted degree of node v is defined as the sum of weights of the edges that are incident to i : dwi = ∑ jWij . Tracking changes in the degree of a node over time is the most direct way of capturing its structural evolution . • Eigenvector centrality : This centrality is a sophisticated extension of degree centrality , related to Google ’ s PageRank , which indicates the influence of a node in G ( Bonacich , 1972 ) . If a node is connected to several nodes with high eigenvector centrality , then that node will have high centrality . The eigenvector centrality of node i is defined as the ith element in the principal eigenvector v of W ( i.e. , the eigenvector that corresponds to the largest eigenvalue λ ) : Wv = λmaxv . 2.2 FULLY-CONNECTED LAYERS : GRAPH REPRESENTATION . Let x ∈ RDin be the input vector , W ∈ RDin×Dout the learnable weight matrix , b ∈ RDout the bias vector and z a non-linear activation functions . Then , the output of this layer is given as y = z ( xW+b ) , where Din and Dout correspond to the dimension of input and output layers . Fully-connected layers are straightforward to represent with a weighted , undirected graph ( Filan et al. , 2021 ) . Each neuron , including those in the input and output layers , corresponds to a node . Two neurons are connected via an edge if they appear in consecutive layers . More formally , let nodeset Vi be the set of all neurons at layer i , and nodeset Vj the set of all neurons at layer j ( which follows layer i ) . The edges that connect all nodes across the two nodesets are defined by the learnable weight matrix W ∈ R|Vi|×|Vj | . 2.3 CONVOLUTIONAL LAYERS : UNROLLED GRAPH REPRESENTATION Let X ∈ Rc×h×w be the input and K ∈ Rf×c×hker×wker the convolutional kernel , where c is the number of channels , and hker , wker are the height and width of the kernel respectively . After performing the convolution , we have Y ∈ RF×hout×wout , where hout = h−hker+1 and wout = w−wker+1 . Typical multilayer CNNs consist of convolutional layers followed by fully-connected layers . The representation of fully connected layers is straightforward but can not be used to model convolutional layers . Rieck et al . ( 2019 ) proposed to `` unroll '' the convolution to convert conv layers into graphs . In Sec . 3 , we introduce a new , compact , rolled representation that is both efficient to compute and effective at predicting the NN performance , as we show empirically in Sec . 4 . For a convolutional layer , we first unroll the convolutional operation and then represent the graph as in the case of a fully-connected layer . In this representation , the nodes and edges of the graph are defined through the convolutional operation as matrix multiplication ( Gebhart et al. , 2019 ) . Specifically , for an input image X and kernel Ki ∈ Rfi×ci×hi×wi , each node in layer l is defined to be the output of the mapped feature of that input for each filter fi . Edges connect each of these nodes to the corresponding nodes of the output neuron in the next layer . These edges are weighted by the activation value of that neuron ( i.e. , the input image for a specific stride is multiplied by the filter value at that location in the image ) . For an input image filter size ( ci × hi × wi ) that results in feature-map ( output ) of size 1 × oh × ow , the number of nodes of output layer of the graph representation is a function of oh × ow , while the number of nodes in the conv operation layer of graph is a function of ci × hi × wi . As an example , Fig . 2 depicts the unrolled graph representation of a toy convolutional layer . It shows one conv operation on a small 2-dimensional input image ( 4× 4 ) and filter ( 2× 2 ) . The nodes and edges of the represented graph are shown based on one stride ( of size 2 ) of convolution operation . 3 NEURAL NETWORK PERFORMANCE PREDICTION : A TEMPORAL GRAPH-BASED APPROACH . In this section , we first formally introduce the problem that we seek to solve . Then , we present a new , compact , and efficient-to-compute graph representation for convolutional layers ( § 3.1 ) , and describe our proposed temporal graph-based framework that captures the NN dynamics during training ( § 3.2 ) . Problem 1 ( NN Performance Prediction ) Let N = { Ntr1 , Ntr2 , . . . , Ntrn } be a training set of n NNs trained for T epochs and A = { α1 , α2 , . . . , αn } their corresponding downstream task accuracies ( e.g. , for image classification ) . We seek to predict the accuracy αtst of a new instance Ntst trained for a very small number of t T epochs by using t epochs for the trained NNs in N . Our proposed solution takes the graph perspective , and its first step is to represent each NN as a temporal graph . In addition to the graph representations that we presented in Sec . 2 , we introduce a new , efficient representation for convolutional layers , which we describe next . 3.1 ROLLED GRAPH REPRESENTATION FOR CONVOLUTIONAL LAYERS . We call our proposed graph representation for conv layers “ rolled , ” since it avoids unrolling the convolutional operations introduced in ( Rieck et al. , 2019 ) . Overview & Motivation . To preserve the semantic meaning of conv layers , we represent each filter as a node , and link filters in consecutive layers via weighted edges ( we define the weights below ) , as shown in Fig . 3 . The motivation behind this approach is that for larger networks , there is an explosion of nodes by unrolling the convolutions ( unrolled representation explained in § 2.3 ) , and the integrity of a unit ( a single filter or kernel ) becomes untenable . Also , by mapping the nodes to specific entities of NNs , our proposed graph model is more interpretable than the unrolled model , and thus it is easier to interpret the outputs of downstream graph analysis on our graph representation ( e.g. , computing node features , tracking the evolution of the graph ) . Detailed Description . Formally , let tensor Ki be a kernel in layer i with fi filters , each with ci channels and dimensions hi×wi . We use bracket notation to index into the kernel : for example , Ki [ l , : , : , : ] indexes the lth filter of kernel Ki . We create fi nodes representing each filter { v ( i ) 1 , v ( i ) 2 , ... v ( i ) fi } , with features defined as the corresponding biases in that layer . Let Kj be the next convolutional layer defined analogously . While edge weights between neurons in FC layers are defined in the standard way , we define the edge weights between conv layers as the norm over each kernel ’ s channels . The edge between node v ( i ) k representing the k th filter in layer i ( i.e. , Ki [ k , : , : , : ] ) and node/filter v ( j ) l in layer j ( i.e . Kj [ l , : , : , : ] ) has weight wv ( i ) k , v ( j ) l = norm ( Kj [ l , k , : , : ] ) , which is the norm of the kth channel of the lthfilter in the jth layer . Though the proposed representations enable alternative edge weight configurations , we focus on the norm of filters as other studies , including those on pruning NNs ( Li et al. , 2016 ) , have demonstrated that this metric has strong correlation with filter importance . In the case of two conv layers , the resultant graph is an attributed bipartite graph with fi+ fj nodes and fi× fj edges , where node attributes include flattened weight vectors or filter maps . Other information such as average gradients or the bias vector can also be used for node features .

Summary: The paper proposes a graph-based approach to assess the performance of deep neural networks in vision tasks. The approach is essentially based on representing the first few epochs of SGD-based training of the neural architecture seen as a graph (where nodes stand for neurons and edges stand for connections among neurons), followed by feature extraction, signature construction and finally a classifier (to predict the high or low level of final accuracy) or a regress or (to predict the exact finally accuracy score) layer. The approach is assessed on several experimental settings, though limited to the case of static networks (e.g., without recurrent/recursive connections). The results seem encouraging and interesting