Dimensional Reweighting Graph Convolution Networks

1 INTRODUCTION . Deep neural networks ( DNNs ) have been widely applied in various fields , including computer vision ( He et al. , 2016 ; Hu et al. , 2018 ) , natural language processing ( Devlin et al. , 2019 ) , and speech recognition ( Abdel-Hamid et al. , 2014 ) , among many others . Graph neural networks ( GNNs ) is proposed for learning node presentations of networked data ( Scarselli et al. , 2009 ) , and later be extended to graph convolutional network ( GCN ) that achieves better performance by capturing topological information of linked graphs ( Kipf & Welling , 2017 ) . Since then , GCNs begin to attract board interests . Starting from GraphSAGE ( Hamilton et al. , 2017 ) defining the convolutional neural network based graph learning framework as sampling and aggregation , many follow-up efforts attempt to enhance the sampling or aggregation process via various techniques , such as attention mechanism ( Veličković et al. , 2018 ) , mix-hop connection ( Abu-El-Haija et al. , 2019 ) and adaptive sampling ( Huang et al. , 2018 ) . In this paper , we study the node representations in GCNs from the perspective of covariance between dimensions . Suprisingly , applying a dimensional reweighting process to the node representations may be very useful for the improvement of GCNs . As an instance , under our proposed reweighting scheme , the input covariance between dimensions can be reduced by 68 % on the Reddit dataset , which is extremely useful since we also find that the number of misclassified cases reduced by 40 % , compared with the previous SOTA method . We propose Dimensional reweighting Graph Convolutional Networks ( DrGCNs ) , in which the input of each layer of the GCN is reweighted by global node representation information . Our discovery is that the experimental performance of GCNs can be greatly improved under this simple reweighting scheme . On the other hand , with the help of mean field theory ( Kadanoff , 2009 ; Yang et al. , 2019 ) , this reweighting scheme is also proved to improve the stability of fully connected networks , provding insight to GCNs . To deepen the understanding to which extent the proposed reweighting scheme can help GCNs , we develop a new measure to quantify its effectiveness under different contexts ( GCN variants and datasets ) . Experimental results verify our theoretical findings ideally that we can achieve predictable improvements on public datasets adopted in the literature over the state-of-the-art GCNs . While studying on these well-known benchmarks , we notice that two of them ( Cora , Citeseer ) suffer from duplicates and feature-label information leaks . We fix these problems and offer refined datasets for fair comparisons . To further validate the effectiveness , we deploy the proposed DrGCNs on A * 1 company ’ s recommendation system and clearly demonstrate performance improvements via offline evaluations . 2 DRGCNS : DIMENSIONAL REWEIGHTING GRAPH CONVOLUTIONAL NETWORKS . 2.1 PRELIMINARIES . Notations . We focus on undirected graphs G = ( V , E , X ) , where V = { vi } represents the node set , E = { ( vi , vj ) } indicates the edge set , and X stands for the node features . For a specific GCN layer , we use Rin = ( rin1 , ... , r in n ) to denote the input node representations and R out = ( rout1 , ... , r out n ) to symbolize the output representations.2 For the whole layer-stacked GCN structure , we use H0 to denote the input node representation of the first layer , and Hl ( l > 0 ) to signify the output node representation of the lth layer , which is also the output representation of the ( l − 1 ) th layer . Let A be the adjacency matrix with aij = 1 when ( vi , vj ) ∈ E and aij = 0 otherwise . Graph Convolutional Networks ( GCNs ) . Given the input node set V , the adjacency matrix A , and the input representations Rin , a GCN layer uses such information to generate output representations Rout : Rout = σ ( aggregator ( Rin , A ) ) , ( 1 ) where σ is the activation function . Although there exist non-linear aggregators like the LSTM aggregator ( Hamilton et al. , 2017 ) , in most GCN variants the aggregator is a linear function which can be viewed as a weighted sum of node representations among the neighborhood ( Kipf & Welling , 2017 ; Huang et al. , 2018 ) , followed by a matrix multiplication on a refined adjacency matrix Ã , with a bias added . The procedure can be formulated as follows : Rout = σ ( WRinÃ+ b ) , ( 2 ) where W is the projection matrix and b denotes the bias vector . Development on GCNs mainly lies in different ways to generate Ã . GCN proposed some variants including simply taking Ã = AD−1 , which is uniform average among neighbors with D being the diagonal matrix of the degrees , or weighted by degree of each node Ã = D− 1 2AD− 1 2 , or including self-loops Ã = I+D− 1 2AD− 1 2 . Other methods include attention ( Veličković et al. , 2018 ) , or gated attention ( Zhang et al. , 2018 ) , or even neural architecture search methods ( Gao et al. , 2019 ) to generate Ã . To improve scalability , some GCN variants contain a sampling procedure , which samples a subset of the neighborhood for aggregation ( Chen et al. , 2018 ; Huang et al. , 2018 ) . We can set all unsampled edges to 0 in Ã in sampling-based GCNs , in this case Ã even has some randomness . 2.2 MODEL FORMULATION FOR DRGCNS . Given input node representations of a GCN layer Rin , the proposed DrGCN tries to learn a dimensional reweighting vector s = ( s1 , ... , sd ) , where si is an adaptive scalar for each dimension i . This reweighting vector s then helps reweighting each dimension of the node representation rinv to r re v , v ∈ V , where rrev = rinv ◦ s. Here we use ◦ to denote component-wise multiplication , i.e. , rrev , j = sjr in v , j , ∀1 ≤ j ≤ d , ∀v ∈ V. ( 3 ) We define S as the diagonal matrix with diagonal entries corresponding to the components of s. Then a DrGCN layer can be formulated as : Rout = σ ( WSRinÃ+ b ) . ( 4 ) Inspired by SENet ( Hu et al. , 2018 ) , we formulate the learning of the shared dimensional reweighting vector s in two stages . First we generate a global representation rin , whose value is the expectation of rinv on the whole graph . Then we feed r in into a two-layer neural network structure to generate 1To preserve anonymity we use A * for the company and dataset name . 2For the convenience of our analysis , we use columns of R , H , X instead of rows to represent node repre- sentations . s of the same dimension size . Equation ( 5 ) denotes the procedure to generate s given node weight { wv|v ∈ V , ∑ v∈V wv = 1 } and node representations { rinv |v ∈ V } : rin =E [ rinv |v ∈ V ] = ∑ v∈V wvr in v , g = σg ( Wgr in + bg ) , s = σs ( Wsg + bs ) , ( 5 ) where g is output of the first layer ; Wg , bg , Ws , and bs are parameters to be learnt . Figure 1 summarizes the dimensional reweighting block ( Dr Block ) in DrGCNs . Combining With Existing GCN variants . The proposed Dr Block can be implemented as an independent functional process and easily combined with GCNs . As shown in equation ( 4 ) , Dr Block only applies on Rin and does not involve in the calculation of Ã , W and b . Hence , the proposed Dr Block can easily be combined with existing sampling or aggregation methods without causing any contradictions . In § 4 , we will experimentally test the combination of our Dr Block with different types of sample-and-aggregation GCN methods . Suppose that the input features are X , DrGCNs can be viewed as follows : Hl = σl ( W lSlHl−1Ãl + bl ) , ∀1 ≤ l ≤ k , ( 6 ) where H0 = X and Hk being the output representation for a k-layer DrGCN : Complexity of Dr Block . Consider a GCN layer with a input and b output channels , n nodes and e edges in a sampled batch , the complexity of a GCN layer isO ( abn+be ) . The proposed Dr block has a complexity of O ( ag ) , where g is the dimension of g. In most cases , we have g < b and n > > 1 , so we could have O ( ag ) = o ( abn + be ) , which indicates that Dr block introduces negligible extra computational cost . 3 THEORETICAL ANALYSES . In this section , we connect our study to mean field theory ( Yang et al. , 2019 ) . We theoretically prove that the proposed Dr Block is capable of reducing the learning variance brought by perturbations on the input , making the update more stable in the long run . To deepen the understanding , we further develop a measure to quantify the stability gained by Dr block . 3.1 MEAN FIELD APPROXIMATION . ( Lee et al. , 2018 ; Yang et al. , 2019 ) employ mean field approximation to analyze fully-connected networks . Following their ideas , we provide theoretical analyses for Dr Blocks on fully-connected networks . GCNs are different from fully-connected networks only in Ã , and degrade to fullyconnected networks when Ã = I , our idea is to provide insight to GCNs from the analysis of fully-connected networks . We assume the average of the data is 0 in the following discussions as transformation does not affect the covariance structure . For simplicity , we only consider neural networks with constant width and assume all layers use the same activation function φ . We follow the pre-activation recurrence relation hli = W lσl ( S l ◦ hl−1i ) +bl in ( Yang et al. , 2019 ) to facilitate the problem . When S being a diagonal matrix with positive entries , and φ is ReLU activation , φ ( SH ) = Sφ ( H ) holds for all H. We can take the pre-activation step as the post-activation step of the previous layer and generalize our analysis to post-activation . So the recursive relation is : Hl = Wlφ ( SlHl−1 ) + bl . ( 7 ) We apply the mean field approximation to replace the input data by a normal random variable with the same mean and variance and define an estimator Vφ to characterize the fluctuation of the random variable φ ( SlH ) . Define : Vφ ( Sl , Cl−1 ) , E [ φ ( SlH ) φ ( SlH ) T ] . ( 8 ) In this equation H ∼ N ( 0 , Cl−1 ) , φ is the ReLU activation , and Cl represents the covariance matrix of Hl , i.e . Cl = Cov ( Hl ) . Note that Vφ does not completely coincide with variance , but can reflect the covariance structure . We call it covariance matrix for convenience . With mean field approximation , the covariance matrix can be updated by ( note that bl is independent with H ) : Cl = WlVφ ( S l , Cl−1 ) ( Wl ) T + Cov ( bl ) . ( 9 ) We assume this dynamical system has a BSB1 ( Block Symmetry Breaking 1 ( Yang et al. , 2019 ) ) fixed point ( where Cl = Cl−1 ) , i.e . a solution having the form C∗ = q∗ ( ( 1− c∗ ) I+ c∗~1~1T ) of the following equation with respect to C : C = WlVφ ( S l , C ) ( Wl ) T + Cov ( bl ) . ( 10 ) Next we make a reduction of Vφ so that only the second slot matters . Take Slh as a whole , it follows N ( 0 , SlCl−1Sl ) distribution . Note that S is diagonal so S = ST . Thus equation ( 9 ) can be written as : Cl = Vφ ( I , S lCl−1Sl ) + Cov ( bl ) . ( 11 ) The derivative of Cl measures the fluctuation of our updating algorithm when input perturbations exist , hence it characterizes the sensitivity of the algorithm to the data structures and its robustness . We will turn to show that Dr can relieve this sensitivity . We fix the point Cl where we are taking derivative at . For most common types of activation functions , this recursive map has a fixed point , at which this linearization is most useful . Recall that such a derivative will be a linear map from symmetric matrices to symmetric matrices . Define : Jφ ( C1 ) , dVφ ( I , C ) dC |Cl ( C1 ) . ( 12 ) Here C1 could be intuitively understood as the increment near Cl . We denote by Hd the space of symmetric matrices of size d× d. Using these notations , we prove that : Theorem 1 . There exist diagonal matrices Sl , constants 0 < γl < 1 such that , ||Jφ ( SlCSl ) ||F ≤ γl||Jφ ( C ) ||F ||S l||F d for any fixed general C. By general , we mean there exists a Haar measure on the collection of symmetric matrices HB with respect to which the statement fails has measure zero 3 . Detailed proofs and explanations are included in the Appendix G , H . For symmetric matrices , with λi denoting eigenvalues of A , we have : ||A||2F = d∑ i=1 λ2i . ( 13 ) This norm measures the overall magnitude of eigenvalues of the operator . This result demonstrates that our method brings variance reduction and improves the stability of the updating algorithms . To summarize , for any input data , there exists a vector s that improves the stability of the updating algorithms .

Summary: This paper proposes a method called "Dimensional Reweighting" for graph convolutional networks. The method involves a reparametrization of GCNs (by adding an extra reweighting block in each layer), which the authors show theoretically can reduce variance. The authors supplement this with extensive empirical experiments showing slightly improved performance by adding their method to existing methods.