Title: GRESNET: GRAPH RESIDUAL NETWORK FOR REVIV-ING DEEP GNNS FROM SUSPENDED ANIMATION

Abstract: The existing graph neural networks (GNNs) based on the spectral graph convolutional operator have been criticized for its performance degradation, which is especially common for the models with deep architectures. In this paper, we further identify the suspended animation problem with the existing GNNs. Such a problem happens when the model depth reaches the suspended animation limit, and the model will not respond to the training data any more and become not learnable. Analysis about the causes of the suspended animation problem with existing GNNs will be provided in this paper, whereas several other peripheral factors that will impact the problem will be reported as well. To resolve the problem, we introduce the GRESNET (Graph Residual Network) framework in this paper, which creates extensively connected highways to involve nodes' raw features or intermediate representations throughout the graph for all the model layers. Different from the other learning settings, the extensive connections in the graph data will render the existing simple residual learning methods fail to work. We prove the effectiveness of the introduced new graph residual terms from the norm preservation perspective, which will help avoid dramatic changes to the node's representations between sequential layers. Detailed studies about the GRESNET framework for many existing GNNs, including GCN, GAT and LOOPYNET, will be reported in the paper with extensive empirical experiments on real-world benchmark datasets.

1. INTRODUCTION:

Graph neural networks (GNN), e.g., graph convolutional network (GCN) Kipf & Welling (2016) and graph attention network (GAT) Veličković et al. (2018) , based on the approximated spectral graph convolutional operator Hammond et al. (2011) , can learn the representations of the graph data effectively. Meanwhile, such GNNs have also received lots of criticism, since as these GNNs' architectures go deep, the models' performance will get degraded, which is similar to observations on other deep models (e.g., convolutional neural network) as reported in He et al. (2015) . Meanwhile, different from the existing deep models, when the GNN model depth reaches a certain limit (e.g., depth ≥ 5 for GCN with the bias term disabled or depth ≥ 8 for GCN with the bias term enabled on the Cora dataset), the model will not respond to the training data any more and become not learnable. Formally, we name such an observation as the GNNs' suspended animation problem, whereas the corresponding model depth is named as the suspended animation limit of GNNs. Here, we need to add a remark: to simplify the presentations in this paper, we will first take vanilla GCN as the base model example to illustrate our discoveries and proposed solutions in the method sections. Meanwhile, empirical tests on several other existing GNNs, e.g., GAT Veličković et al. (2018) and LOOPYNET Zhang et al. (2018) , will also be studied in the experiment section of this paper.

As illustrated in Figure 1 , we provide the learning performance of the GCN model on the Cora dataset, where the learning settings (including train/validation/test sets partition, algorithm implementation and fine-tuned hyper-parameters) are identical to those introduced in Kipf & Welling (2016) . The GCN model with the bias term disable of seven different depths, i.e., GCN(1-layer)-GCN(7-layer), are compared. Here, the layer number denotes the sum of hidden and output layers, which is also equal to the number of spectral graph convolutional layers involved in the model. For instance, besides the input layer, GCN(7-layer) has 6 hidden layer and 1 output layer, both of which involve the spectral graph convolutional operations. According to the plots, and have comparable performance, which both outperform GCN(1-layer). Meanwhile, as the model depth increases from 3 to 7, its learning performance on both the training set and the testing set degrades greatly. It is easy to identify that such degradation is not caused by overfitting the training data. What's more, much more surprisingly, as the model depth goes deeper to 5 or more, it will suffer from the suspended animation problem and does not respond to the training data anymore. (Similar phenomena can be observed for GCN (bias enabled) and GAT as illustrated by Figures 9 and 10 in the appendix of this paper, whose suspended animation limits are 8 and 5, respectively. Meanwhile, on LOOPYNET, we didn't observe such a problem as shown in Figure 11 in the appendix, and we will state the reasons in Section 6 in detail.)

In this paper, we will investigate the causes of the GNNs' suspended animation problem, and analyze if such a problem also exists in all other GNN models or not. GNNs are very different from the traditional deep learning models, since the extensive connections among the nodes render their learning process no longer independent but strongly correlated. Therefore, the existing solutions proposed to resolve such problems, e.g., residual learning methods used in ResNet for CNN He et al. (2015) , cannot work well for GNNs actually. In this paper, several different novel graph residual terms will be studied for GNNs specially. Equipped with the new graph residual terms, we will further introduce a new graph neural network architecture, namely graph residual neural network (GRESNET), to resolve the observed problem. Instead of merely stacking the spectral graph convolution layers on each other, the extensively connected high-ways created in GRESNET allow the raw features or intermediate representations of the nodes to be fed into each layer of the model. We will study the effectiveness of the GRESNET architecture and those different graph residuals for several existing vanilla GNNs. In addition, theoretic analyses on GRESNET will be provided in this paper as well to demonstrate its effectiveness from the norm-preservation perspective.

The remaining parts of this paper are organized as follows. In Section 2, we will introduce the related work of this paper. The suspended animation problem with the spectral graph convolutional operator will be discussed in Section 3, and the suspended animation limit will be analyzed in Section 4. Graph residual learning will be introduced in Section 5, whose effectiveness will be tested in Section 6. Finally, we will conclude this paper in Section 7.

2. RELATED WORK:

Graph Neural Network: Graph neural networks Monti et al. (2017) ; Atwood & Towsley (2016) ; Masci et al. (2015) ; Kipf & Welling (2016) ; Battaglia et al. (2018) ; Bai et al. (2018) ; Scarselli et al. (2009) ; Zhou et al. (2018) ; Niepert et al. (2016) have become a popular research topic in recent years. Traditional deep models cannot be directly applied to graph data due to the graph interconnected structures. Many efforts have been devoted to extend deep neural networks on graphs for representation learning. GCN proposed in Kipf & Welling (2016) feeds the generalized spectral features into the convolutional layer for representation learning. Similar to GCN, deep loopy graph neural network Zhang (2018) proposes to update the node states in a synchronous manner, and it introduces a spanning tree based learning algorithm for training the model. LOOPYNET accepts nodes' raw features into each layer of the model, and it can effectively fight against the suspended animation problem according to the studied in this paper. GAT Veličković et al. (2018) leverages masked self-attentional layers to address the shortcomings of GCN. In this year, we have also witnessed some preliminary works on heterogeneous graph neural networks Wang et al. (2019) ; Liu et al. (2018) . Similar to GCN, GEM Liu et al. (2018) utilizes one single layer of attention to capture the impacts of both neighbors and network heterogeneity, which cannot work well on real-world complex networks. Based on GAT, HAN Wang et al. (2019) learns the attention coefficients between the neighbors based on a set of manually crafted meta paths Sun et al. (2011) , which may require heavy human involvements. DIFNN Zhang et al. (2018) introduce a diffusive neural network for the graph structured data specifically, which doesn't suffer from the oversmoothing problem due to the involvement of the neural gates and residual inputs for all the layers. Due to the limited space, we can only name a few number of the representative graph neural network here. The readers are also suggested to refer to page1 , which provides a summary of the latest graph neural network research papers with code on the node classification problem.

Residual Network: Residual learning Srivastava et al. (2015) ; He et al. (2015) ; Bae et al. (2016) ; Han et al. (2016) ; Gomez et al. (2017) ; Tai et al. (2017) ; Yu et al. (2017) ; Ahn et al. (2018) ; Li et al. (2018a) ; Behrmann et al. (2019) has been utilized to improve the learning performance, especially for the neural network models with very deep architectures. To ease gradient-based training of very deep networks, Srivastava et al. (2015) introduces the highway network to allow unimpeded information flow across several layers. Innovated by the high-way structure, He et al. (2015) introduces the residual network to simplify highway network by removing the fusion gates. After that, residual learning has been widely adopted for deep model training and optimization. Bae et al. (2016) introduces residual learning for image restoration via persistent homology-guided manifold simplification; Tai et al. (2017) ; Li et al. (2018a) introduces a recursive residual network for image resolution adjustment. Some improvement of residual network has also been proposed in recent years. A reversible residual network is introduced in Gomez et al. (2017) , where each layer's activations can be reconstructed exactly from the next layer's; Han et al. (2016) improves the conventional model shape with a pyramidal residual network instead; Yu et al. (2017) introduce the dilated residual network to increases the resolution of output feature maps without reducing the receptive field of individual neurons; and Ahn et al. (2018) studies the cascading residual network as an accurate and lightweight deep network for image super-resolution. Readers can also refer to He (2016) for a detailed tutorial on residual learning and applications in neural network studies.

3. SUSPENDED ANIMATION PROBLEM WITH GCN MODEL:

In this part, we will provide an analysis about the suspended animation problem of the spectral graph convolutional operator used in GCN to interpret the causes of the observations illustrated in Figure 1 . In addition, given an input network data, we will provide the theoretic bound of the suspended animation limit for the GCN model.

3.1. VANILLA GRAPH CONVOLUTIONAL NETWORK REVISIT:

To make this paper self-contained, we will provide a brief revisit of the vanilla GCN model in this part. Formally, given an input network G = (V, E), its network structure information can be denoted as an adjacency matrix A = {0, 1} n×n (where |V| = n). GCN defines the normalized adjacency matrix of the input network as Â = D-1 2 Ã D-1 2 , where Ã = A + I n×n and D is the diagonal matrix of Ã with entry D(i, i) = j Ã(i, j). Given all the nodes in V together with their raw feature inputs X ∈ R n×dx (d x denotes the node raw feature length), GCN defines the spectral graph convolutional operator to learn the nodes' representations as follows:

EQUATION

where W ∈ R dx×d h is the variable involved in the operator.

Furthermore, GCN can involve a deep architecture by stacking multiple spectral graph convolutional layers on each other, which will be able to learn very complex high-level representations of the nodes. Here, let's assume the model depth to be K (i.e., the number of hidden layers and output layer), and the corresponding node representation updating equations can be denoted as: -1) .

       H (0) = X, H (k) = ReLU ÂH (k-1) W (k-1) , ∀k ∈ {1, 2, • • • , K -1}, Ŷ = softmax ÂH (K-1) W (K

(2)

3.2. SUSPENDED ANIMATION PROBLEM WITH GCN:

By investigating the spectral graph convolutional operator defined above, we observe that it actually involves two sequential steps:

H (k) = ReLU ÂH (k-1) W (k-1) ⇔ MC Layer: T (k) = ÂH (k-1)

FC Layer:

EQUATION

where the first term on the right-hand-side defines a 1-step Markov chain (MC or a random walk) based on the graph and the second term is a fully-connected (FC) layer parameterized by variable W (k-1) . Similar observations have been reported in Li et al. (2018b) as well, but it interprets the spectral graph convolutional operator in a different way as the Laplacian smoothing operator used in mesh smoothing in graphics instead.

Therefore, stacking multiple spectral graph convolutional layers on top of each other is equivalent to the stacking of multiple 1-step Markov chain layers and fully-connected layers in a crosswise way.

Considering that the variables W (k-1) for the vector dimension adjustment are shared among all the nodes, given two nodes with identical representations, the fully-connected layers (parameterized by W (k-1) ) will still generate identical representations as well. In other words, the fully-connected layers with shared variables for all the nodes will not have significant impacts on the convergence of Markov chain layers actually. Therefore, in the following analysis, we will simplify the model structure by assuming the mapping defined by fully-connected layers to be the identity mapping.

We will investigate the Markov chain layers closely by picking them out of the model to compose the Markov chain of multiple steps:

EQUATION

Meanwhile, the Markov chain layers may converge with k layers iff T (k) = T (k-1) , i.e., the representations before and after the updating are identical (or very close), which is highly dependent on the input network structure, i.e., matrix Â, actually. DEFINITION 1. (Irreducible and Aperiodic Network): Given an input network G = (V, E), G is irreducible iff for any two nodes v i , v j ∈ V, node v i is accessible to v j . Meanwhile, G is aperiodic iff G is not bipartite. LEMMA 1. Given an unweighted input graph G, which is irreducible, finite and aperiodic, if its corresponding matrix is asymmetric, starting from any initial distribution vector x ∈ R n×1 (x ≥ 0 and x 1 = 1), the Markov chain operating on the graph has one unique stationary distribution vector π * such that lim t→∞ Ât x = π * , where π * (i) = d(vi) 2|E| . Meanwhile, if matrix Â is symmetric, the stationary distribution vector π * will be a uniform distribution over the nodes, i.e., π * (i) = 1 n .

Based on the above Lemma 1, we can derive similar results for the multiple Markov chain layers in the GCN model based on the nodes' feature inputs, which will reduce the learned nodes' representations to the stationary representation matrix. THEOREM 1. Given a input network G = (V, E), which is unweighted, irreducible, finite and aperiodic, if there exist enough nested Markov chain layers in the GCN model, it will reduce the nodes' representations from the column-normalized feature matrix X ∈ R n×dx to the stationary representation

Π * = [π * , π * , • • • , π * ] ∈ R n×dx . Furthermore, if G is undirected, then the stationary representation will become Π * = 1 n • 1 n×dx .

Theorem 1 illustrates the causes of the GNNs' suspended animation problem. Proofs of Lemma 1 and Theorem 1 are provided in the appendix attached to this paper at the end.

4. SUSPENDED ANIMATION LIMIT ANALYSIS:

Here, we will study the suspended animation limit of the GCN model based on its spectral convolutional operator analysis, especially the Markov chain layers.

4.1. SUSPENDED ANIMATION LIMIT BASED INPUT NETWORK STRUCTURE:

Formally, we define the suspended animation limit of GCN as follows:

DEFINITION 2. (Suspended Animation Limit): The suspended animation limit of GCN on network G is defined as the smallest model depth τ such that for any nodes' column-normalized featured matrix input X in the network G the following inequality holds:

GCN(X; τ ) -Π * 1 ≤ . (5)

For representation convenience, we can also denote the suspended animation limit of GCN defined on network G as ζ(G) (or ζ for simplicity if there is no ambiguity problems).

Based on the above definition, for GCN with identity FC mappings, there exists a tight bound of the suspended animation limit for the input network.

THEOREM 2. Let 1 ≥ λ 1 ≥ λ 2 ≥ • • • ≥ λ n be

the eigen-values of matrix Â defined based on network G, then the corresponding suspended animation limit of the GCN model on G is bounded

EQUATION

)

In the case that the network G is a d-regular, then the suspended animation limit of the GCN model on G can be simplified as

EQUATION

)

The suspended animation limit bound derived in the above theorem generally indicates that the network structure G determines the maximum allows depth of GCN. Among all the eigen-values of Â defined on network G, λ 2 measures how far G is from being disconnected; and λ n measures how far G is from being bipartite. In the case that G is reducible (i.e., λ 2 = 1) or bipartite (i.e., λ n = -1), we have ζ → ∞ and the model will not suffer from the suspended animation problem.

In the appendix of Kipf & Welling (2016) , the authors introduce a naive-residual based variant of GCN, and the sepctral graph convolutional operator is changed as follows (the activation function is also changed to sigmoid function instead):

EQUATION

For the identity fully-connected layer mapping, the above equation can be reduced to the following lazy Markov chain based layers

EQUATION

Such a residual term will not help resolve the problem, and it will still suffer from the suspended animation problem with the following suspended animation limit bound:

COROLLARY 1. Let 1 ≥ λ 1 ≥ λ 2 ≥ • • • ≥ λ n be

the eigen-values of matrix Â defined based on network G, then the corresponding suspended animation limit of the GCN model (with lazy Markov chain based layers) on G is bounded

EQUATION

)

Proofs to Theorem 2 and Corollary 1 will be provided in the appendix as well.

4.2 OTHER PRACTICAL FACTORS IMPACTING THE SUSPENDED ANIMATION LIMIT According to our preliminary experimental studies, GCN is quite a sensitive model. Besides the impact of input network explicit structures (e.g., network size n, directed vs undirected links, and network eigenvalues) as indicated in the bound equation in Theorem 2, many other factors can also influence the performance of GCN model a lot, which are summarized as follows:

• Network Degree Distribution: According to Theorem 1, if the input network G is directed and unweighted, the Markov chain layers at convergence will project the features to Π * = 

ual R H (k-1) , X; G = H (k-1)

Node residual terms are assumed to be independent and determined by the current state only. graph-naive residual R H (k-1) , X; G = ÂH (k-1) Node residual terms are correlated based on network structure, and can be determined by the current state. raw residual R H (k-1) , X; G = X Node residual terms are assumed to be independent and determined by the raw input features only. graph-raw residual R H (k-1) , X; G = ÂX Node residual terms are correlated based on network structure, and are determined by the raw input features.

[π * , π * , • • • , π * ],

where π * (i) is determined by the degree of node v i in the network. For any two nodes v i , v j ∈ V, the differences of their learned representation can be denoted as

EQUATION

According to Newman (2003) , the node degree distributions in most complex networks follow the power-law Faloutsos et al. (1999) , i.e., majority of the nodes are of very small degrees. Therefore, for massive nodes in the input network with the same (or close) degrees, the differences between their learned representations will become not distinguishable. • Raw Feature Coding: Besides the input network, the raw feature coding can also affect the learning performance greatly. Here, we can take the GCN with one single Markov chain layer and one identity mapping layer. For any two nodes v i , v j ∈ V with raw feature vectors X(i, :) and X(j, :), we can denote the differences between their learned representations as follows:

EQUATION

Different from Equation (11), analysis in the above equation is not based on the stationary representation of the nodes, and it is based on the MC-layer representation as indicated in Equation (3) and Equation (4). For the one-hot feature coding used in the source code of GCN Kipf & Welling (2016) and other GNNs, matrix X can be also very sparse as well. Meanwhile, vector Â(i, :) -Â(j, :) is also a sparse vector, which renders the right-handside term to be a very small value.

• Training Set Size: Actually, the nodes have identical representation and the same labels will not degrade the learning performance of the model. However, if such node instances actually belong to different classes, it will become a great challenge for both the training and test stages of the GCN model. • Gradient Vanishing/Exploding: Similar to the existing deep models, deep GNNs will also suffer from the gradient vanishing/exploding problems Pascanu et al. (2012) , which will also greatly affect the learning performance of the models.

Although these factors mentioned above are not involved in the suspended animation limit bound representation, but they do have great impacts on the GCN model in practical applications. In the following section, we will introduce graph residual network GRESNET, which can be useful for resolving such a problem for GCN.

5. GRAPH RESIDUAL NETWORK:

Different from the residual learning in other areas, e.g., computer vision He et al. (2015) , where the objective data instances are independent with each other, in the inter-connected network learning setting, the residual learning of the nodes in the network are extensively connected instead. It renders the existing residual learning strategy less effective for improving the performance of GCN.

5.1. GRAPH RESIDUAL LEARNING:

Residual learning initially introduced in He et al. (2015) divides the objective mapping into two parts: the inputs and the residual function. For instance, let H(x) be the objective mapping which projects input x to the desired domain. The ResNet introduced in He et al. (2015) divides H(x) as F (x) + R(x) (where R(x) = x is used in He et al. (2015) ). This reformulation is motivated by the counterintuitive phenomena about the degradation problem observed on the deep CNN. Different SGC G < l a t e x i t s h a 1 _ b a s e 6 4 = " /

V t + 0 R d S S 5 A / s b V i R u a b h 3 H o i B U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k W W m I U J I E L 2 V v m Y M P e 3 m V 3 z 4 R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S A T X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q 6 z h V D F s s F r H q B F S j 4 B J b h h u B n U Q h j Q K B j 8

H 4 e u Y / P q H S P J Y P Z p K g H 9 G h 5 C F n 1 F j p v n p T 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E U r D B N W 6 6 7 m J 8 T O q D G c C p 6 V e q j G h b E y H 2 L V U 0 g i 1 n 8 1 P n Z I z q w x I G C t b 0 p C 5 + n s i o 5 H W k y i w n R E 1 I 7 3 s z c T / v G 5 q w k s / 4 z J J D U q 2 W B S m g p i Y z P 4 m A 6 6 Q G T G x h D L F 7 a 2 E j a i i z N h 0 S j Y E b / n l V d K u 1 z y 3 5 t 3 V K 4 2 r P I 4 i n M A p n I M H F 9 C A W 2 h C C x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g B S 3 Y 0 l < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " /

V t + 0 R d S S 5 A / s b V i R u a b h 3 H o i B U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k W W m I U J I E L 2 V v m Y M P e 3 m V 3 z 4 R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S A T X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q 6 z h V D F s s F r H q B F S j 4 B J b h h u B n U Q h j Q K B j 8

H 4 e u Y / P q H S P J Y P Z p K g H 9 G h 5 C F n 1 F j p v n p T 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E U r D B N W 6 6 7 m J 8 T O q D G c C p 6 V e q j G h b E y H 2 L V U 0 g i 1 n 8 1 P n Z I z q w x I G C t b 0 p C 5 + n s i o 5 H W k y i w n R E 1 I 7 3 s z c T / v G 5 q w k s / 4 z J J D U q 2 W B S m g p i Y z P 4 m A 6 6 Q G T G x h D L F 7 a 2 E j a i i z N h 0 S j Y E b / n l V d K u 1 z y 3 5 t 3 V K 4 2 r P I 4 i n M A p n I M H F 9 C A W 2 h C C x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g B S 3 Y 0 l < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " /

V t + 0 R d S S 5 A / s b V i R u a b h 3 H o i B U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k W W m I U J I E L 2 V v m Y M P e 3 m V 3 z 4 R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S A T X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q 6 z h V D F s s F r H q B F S j 4 B J b h h u B n U Q h j Q K B j 8

H 4 e u Y / P q H S P J Y P Z p K g H 9 G h 5 C F n 1 F j p v n p T 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E U r D B N W 6 6 7 m J 8 T O q D G c C p 6 V e q j G h b E y H 2 L V U 0 g i 1 n 8 1 P n Z I z q w x I G C t b 0 p C 5 + n s i o 5 H W k y i w n R E 1 I 7 3 s z c T / v G 5 q w k s / 4 z J J D U q 2 W B S m g p i Y z P 4 m A 6 6 Q G T G x h D L F 7 a 2 E j a i i z N h 0 S j Y E b / n l V d K u 1 z y 3 5 t 3 V K 4 2 r P I 4 i n M A p n I M H F 9 C A W 2 h C C x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g B S 3 Y 0 l < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " /

V t + 0 R d S S 5 A / s b V i R u a b h 3 H o i B U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k W W m I U J I E L 2 V v m Y M P e 3 m V 3 z 4 R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S A T X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q 6 z h V D F s s F r H q B F S j 4 B J b h h u B n U Q h j Q K B j 8

H 4 e u Y / P q H S P J Y P Z p K g H 9 G h 5 C F n 1 F j p v n p T 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E U r D B N W 6 6 7 m J 8 T O q D G c C p 6 V e q j G h b E y H 2 L V U 0 g i 1 n 8 1 P n Z I z q w x I G C t b 0 p C 5 + n s i o 5 H W k y i w n R E 1 I 7 3 s z c T / v G 5 q w k s / 4 z J J D U q 2 W B S m g p i Y z P 4

m A 6 6 Q G T G x h D L F 7 a 2 E j a i i z N h 0 S j Y E b / n l V d K u 1 z y 3 5 t 3 V K 4 2 r P I 4 i n M A p n I M H F 9 C A W 2 h C C x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g B S 3 Y 0 l < / l a t e x i t > X < l a t e x i t s h a 1 _ b a s e 6 4 = " M c N d i W G T h Q s 7 u 8 o H n k B R i f D 5 r u w = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y C u 4 K k k 3 u i y 6 c V n B P q A p Z T K d t E M n k z B z I 5 T Q 3 3 D j Q h G 3 / o w 7 / 8 Z J m 4 W 2 H h g 4 n H M v 9 8 w J E i k M u u 6 3 s 7 G 5 t b 2 z W 9 o r 7 x 8 c H h 1 X T k 4 7 J k 4 1 4 2 0 W y 1 j 3 A m q 4 F I q 3 U a D k v U R z G g W S d 4 P p X e 5 3 n 7 g 2 I l a P O E v 4 I K J j J U L B K F r J r / k R x U k Q Z r 1 5 b V i p u n V 3 A b J O v I J U o U B r W P n y R z F L I 6 6 Q S W p M 3 3 M T H G R U o 2 C S z 8 t + a n h C 2 Z S O e d 9 S R S N u B t k i 8 5 x c W m V E w l j b p 5 A s 1 N 8 b G Y 2 M m U W B n c w j m l U v F / / z + i m G N 4 N M q C R F r t j y U J h K g j H J C y A j o T l D O b O E M i 1 s V s I m V F O G t q a y L c F b / f I 6 6 T T q n l v 3 H h r V 5 m 1 R R w n O 4 Q K u w I N r a M I 9 t K A N D B J 4 h l d 4 c 1 L n x X l 3 P p a j G 0 6 x c w Z / 4 H z + A I j A k V Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M c N d i W G T h Q s 7 u 8 o H n k B R i f D 5 r u w = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y C u 4 K k k 3 u i y 6 c V n B P q A p Z T K d t E M n k z B z I 5 T Q 3 3 D j Q h G 3 / o w 7 / 8 Z J m 4 W 2 H h g 4 n H M v 9 8 w J E i k M u u 6 3 s 7 G 5 t b 2 z W 9 o r 7 x 8 c H h 1 X T k 4 7 J k 4 1 4 2 0 W y 1 j 3 A m q 4 F I q 3 U a D k v U R z G g W S d 4 P p X e 5 3 n 7 g 2 I l a P O E v 4 I K J j J U L B K F r J r / k R x U k Q Z r 1 5 b V i p u n V 3 A b J O v I J U o U B r W P n y R z F L I 6 6 Q S W p M 3 3 M T H G R U o 2 C S z 8 t + a n h C 2 Z S O e d 9 S R S N u B t k i 8 5 x c W m V E w l j b p 5 A s 1 N 8 b G Y 2 M m U W B n c w j m l U v F / / z + i m G N 4 N M q C R F r t j y U J h K g j H J C y A j o T l D O b O E M i 1 s V s I m V F O G t q a y L c F b / f I 6 6 T T q n l v 3 H h r V 5 m 1 R R w n O 4 Q K u w I N r a M I 9 t K A N D B J 4 h l d 4 c 1 L n x X l 3 P p a j G 0 6 x c w Z / 4 H z + A I j A k V Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M c N d i W G T h Q s 7 u 8 o H n k B R i f D 5 r u w = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y C u 4 K k k 3 u i y 6 c V n B P q A p Z T K d t E M n k z B z I 5 T Q 3 3 D j Q h G 3 / o w 7 / 8 Z J m 4 W 2 H h g 4 n H M v 9 8 w J E i k M u u 6 3 s 7 G 5 t b 2 z W 9 o r 7 x 8 c H h 1 X T k 4 7 J k 4 1 4 2 0 W y 1 j 3 A m q 4 F I q 3 U a D k v U R z G g W S d 4 P p X e 5 3 n 7 g 2 I l a P O E v 4 I K J j J U L B K F r J r / k R x U k Q Z r 1 5 b V i p u n V 3 A b J O v I J U o U B r W P n y R z F L I 6 6 Q S W p M 3 3 M T H G R U o 2 C S z 8 t + a n h C 2 Z S O e d 9 S R S N u B t k i 8 5 x c W m V E w l j b p 5 A s 1 N 8 b G Y 2 M m U W B n c w j m l U v F / / z + i m G N 4 N M q C R F r t j y U J h K g j H J C y A j o T l D O b O E M i 1 s V s I m V F O G t q a y L c F b / f I 6 6 T T q n l v 3 H h r V 5 m 1 R R w n O 4 Q K u w I N r a M I 9 t K A N D B J 4 h l d 4 c 1 L n x X l 3 P p a j G 0 6 x c w Z / 4 H z + A I j A k V Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M c N d i W G T h Q s 7 u 8 o H n k B R i f D 5 r u w = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y C u 4 K k k 3 u i y 6 c V n B P q A p Z T K d t E M n k z B z I 5 T Q 3 3 D j Q h G 3 / o w 7 / 8 Z J m 4 W 2 H h g 4 n H M v 9 8 w J E i k M u u 6 3 s 7 G 5 t b 2 z W 9 o r 7 x 8 c H h 1 X T k 4 7 J k 4 1 4 2 0 W y 1 j 3 A m q 4 F I q 3 U a D k v U R z G g W S d 4 P p X e 5 3 n 7 g 2 I l a P O E v 4 I K J j J U L B K F r J r / k R x U k Q Z r 1 5 b V i p u n V 3 A b J O v I J U o U B r W P n y R z F L I 6 6 Q S W p M 3 3 M T H G R U o 2 C S z 8 t + a n h C 2 Z S O e d 9 S R S N u B t k i 8 5 x c W m V E w l j b p 5 A s 1 N 8 b G Y 2 M m U W B n c w j m l U v F / / z + i m G N 4 N M q C R F r t j y U J h K g j H J C y A j o T l D O b O E M i 1 s V s I m V F O G t q a y L c F b / f I 6 6 T T q n l v 3 H h r V 5 m 1 R R w n O 4 Q K u w I N r a M I 9 t K A N D B J 4 h l d 4 c 1 L n x X l 3 P p a j G 0 6 x c w Z / 4 H z + A I j A k V Y = < / l a t e x i t > ReLU softmax SGC SGC G < l a t e x i t s h a 1 _ b a s e 6 4 = " / V t + 0 R d S S 5 A / s b V i R u a b h 3 H o i B U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k W W m I U J I E L 2 V v m Y M P e 3 m V 3 z 4 R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S A T X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q 6 z h V D F s s F r H q B F S j 4 B J b h h u B n U Q h j Q K B j 8 H 4 e u Y / P q H S P J Y P Z p K g H 9 G h 5 C F n 1 F j p v n p T 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E U r D B N W 6 6 7 m J 8 T O q D G c C p 6 V e q j G h b E y H 2 L V U 0 g i 1 n 8 1 P n Z I z q w x I G C t b 0 p C 5 + n s i o 5 H W k y i w n R E 1 I 7 3 s z c T / v G 5 q w k s / 4 z J J D U q 2 W B S m g p i Y z P 4 m A 6 6 Q G T G x h D L F 7 a 2 E j a i i z N h 0 S j Y E b / n l V d K u 1 z y 3 5 t 3 V K 4 2 r P I 4 i n M A p n I M H F 9 C A W 2 h C C x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g B S 3 Y 0 l < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " / V t + 0 R d S S 5 A / s b V i R u a b h 3 H o i B U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k W W m I U J I E L 2 V v m Y M P e 3 m V 3 z 4 R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S A T X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q 6 z h V D F s s F r H q B F S j 4 B J b h h u B n U Q h j Q K B j 8 H 4 e u Y / P q H S P J Y P Z p K g H 9 G h 5 C F n 1 F j p v n p T 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E U r D B N W 6 6 7 m J 8 T O q D G c C p 6 V e q j G h b E y H 2 L V U 0 g i 1 n 8 1 P n Z I z q w x I G C t b 0 p C 5 + n s i o 5 H W k y i w n R E 1 I 7 3 s z c T / v G 5 q w k s / 4 z J J D U q 2 W B S m g p i Y z P 4 m A 6 6 Q G T G x h D L F 7 a 2 E j a i i z N h 0 S j Y E b / n l V d K u 1 z y 3 5 t 3 V K 4 2 r P I 4 i n M A p n I M H F 9 C A W 2 h C C x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g B S 3 Y 0 l < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " / V t + 0 R d S S 5 A / s b V i R u a b h 3 H o i B U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k W W m I U J I E L 2 V v m Y M P e 3 m V 3 z 4 R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S A T X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q 6 z h V D F s s F r H q B F S j 4 B J b h h u B n U Q h j Q K B j 8 H 4 e u Y / P q H S P J Y P Z p K g H 9 G h 5 C F n 1 F j p v n p T 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E U r D B N W 6 6 7 m J 8 T O q D G c C p 6 V e q j G h b E y H 2 L V U 0 g i 1 n 8 1 P n Z I z q w x I G C t b 0 p C 5 + n s i o 5 H W k y i w n R E 1 I 7 3 s z c T / v G 5 q w k s / 4 z J J D U q 2 W B S m g p i Y z P 4 m A 6 6 Q G T G x h D L F 7 a 2 E j a i i z N h 0 S j Y E b / n l V d K u 1 z y 3 5 t 3 V K 4 2 r P I 4 i n M A p n I M H F 9 C A W 2 h C C x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g B S 3 Y 0 l < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " / V t + 0 R d S S 5 A / s b V i R u a b h 3 H o i B U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k W W m I U J I E L 2 V v m Y M P e 3 m V 3 z 4 R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S A T X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q 6 z h V D F s s F r H q B F S j 4 B J b h h u B n U Q h j Q K B j 8 H 4 e u Y / P q H S P J Y P Z p K g H 9 G h 5 C F n 1 F j p v n p T 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E U r D B N W 6 6 7 m J 8 T O q D G c C p 6 V e q j G h b E y H 2 L V U 0 g i 1 n 8 1 P n Z I z q w x I G C t b 0 p C 5 + n s i o 5 H W k y i w n R E 1 I 7 3 s z c T / v G 5 q w k s / 4 z J J D U q 2 W B S m g p i Y z P 4 m A 6 6 Q G T G x h D L F 7 a 2 E j a i i z N h 0 S j Y E b / n l V d K u 1 z y 3 5 t 3 V K 4 2 r P I 4 i n M A p n I M H F 9 C A W 2 h C C x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g B S 3 Y 0 l < / l a t e x i t > X < l a t e x i t s h a 1 _ b a s e 6 4 = " M c N d i W G T h Q s 7 u 8 o H n k B R i f D 5 r u w = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y C u 4 K k k 3 u i y 6 c V n B P q A p Z T K d t E M n k z B z I 5 T Q 3 3 D j Q h G 3 / o w 7 / 8 Z J m 4 W 2 H h g 4 n H M v 9 8 w J E i k M u u 6 3 s 7 G 5 t b 2 z W 9 o r 7 x 8 c H h 1 X T k 4 7 J k 4 1 4 2 0 W y 1 j 3 A m q 4 F I q 3 U a D k v U R z G g W S d 4 P p X e 5 3 n 7 g 2 I l a P O E v 4 I K J j J U L B K F r J r / k R x U k Q Z r 1 5 b V i p u n V 3 A b J O v I J U o U B r W P n y R z F L I 6 6 Q S W p M 3 3 M T H G R U o 2 C S z 8 t + a n h C 2 Z S O e d 9 S R S N u B t k i 8 5 x c W m V E w l j b p 5 A s 1 N 8 b G Y 2 M m U W B n c w j m l U v F / / z + i m G N 4 N M q C R F r t j y U J h K g j H J C y A j o T l D O b O E M i 1 s V s I m V F O G t q a y L c F b / f I 6 6 T T q n l v 3 H h r V 5 m 1 R R w n O 4 Q K u w I N r a M I 9 t K A N D B J 4 h l d 4 c 1 L n x X l 3 P p a j G 0 6 x c w Z / 4 H z + A I j A k V Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M c N d i W G T h Q s 7 u 8 o H n k B R i f D 5 r u w = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y C u 4 K k k 3 u i y 6 c V n B P q A p Z T K d t E M n k z B z I 5 T Q 3 3 D j Q h G 3 / o w 7 / 8 Z J m 4 W 2 H h g 4 n H M v 9 8 w J E i k M u u 6 3 s 7 G 5 t b 2 z W 9 o r 7 x 8 c H h 1 X T k 4 7 J k 4 1 4 2 0 W y 1 j 3 A m q 4 F I q 3 U a D k v U R z G g W S d 4 P p X e 5 3 n 7 g 2 I l a P O E v 4 I K J j J U L B K F r J r / k R x U k Q Z r 1 5 b V i p u n V 3 A b J O v I J U o U B r W P n y R z F L I 6 6 Q S W p M 3 3 M T H G R U o 2 C S z 8 t + a n h C 2 Z S O e d 9 S R S N u B t k i 8 5 x c W m V E w l j b p 5 A s 1 N 8 b G Y 2 M m U W B n c w j m l U v F / / z + i m G N 4 N M q C R F r t j y U J h K g j H J C y A j o T l D O b O E M i 1 s V s I m V F O G t q a y L c F b / f I 6 6 T T q n l v 3 H h r V 5 m 1 R R w n O 4 Q K u w I N r a M I 9 t K A N D B J 4 h l d 4 c 1 L n x X l 3 P p a j G 0 6 x c w Z / 4 H z + A I j A k V Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M c N d i W G T h Q s 7 u 8 o H n k B R i f D 5 r u w = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y C u 4 K k k 3 u i y 6 c V n B P q A p Z T K d t E M n k z B z I 5 T Q 3 3 D j Q h G 3 / o w 7 / 8 Z J m 4 W 2 H h g 4 n H M v 9 8 w J E i k M u u 6 3 s 7 G 5 t b 2 z W 9 o r 7 x 8 c H h 1 X T k 4 7 J k 4 1 4 2 0 W y 1 j 3 A m q 4 F I q 3 U a D k v U R z G g W S d 4 P p X e 5 3 n 7 g 2 I l a P O E v 4 I K J j J U L B K F r J r / k R x U k Q Z r 1 5 b V i p u n V 3 A b J O v I J U o U B r W P n y R z F L I 6 6 Q S W p M 3 3 M T H G R U o 2 C S z 8 t + a n h C 2 Z S O e d 9 S R S N u B t k i 8 5 x c W m V E w l j b p 5 A s 1 N 8 b G Y 2 M m U W B n c w j m l U v F / / z + i m G N 4 N M q C R F r t j y U J h K g j H J C y A j o T l D O b O E M i 1 s V s I m V F O G t q a y L c F b / f I 6 6 T T q n l v 3 H h r V 5 m 1 R R w n O 4 Q K u w I N r a M I 9 t K A N D B J 4 h l d 4 c 1 L n x X l 3 P p a j G 0 6 x c w Z / 4 H z + A I j A k V Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M c N d i W G T h Q s 7 u 8 o H n k B R i f D 5 r u w = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y C u 4 K k k 3 u i y 6 c V n B P q A p Z T K d t E M n k z B z I 5 T Q 3 3 D j Q h G 3 / o w 7 / 8 Z J m 4 W 2 H h g 4 n H M v 9 8 w J E i k M u u 6 3 s 7 G 5 t b 2 z W 9 o r 7 x 8 c H h 1 X T k 4 7 J k 4 1 4 2 0 W y 1 j 3 A m q 4 F I q 3 U a D k v U R z G g W S d 4 P p X e 5 3 n 7 g 2 I l a P O E v 4 I K J j J U L B K F r J r / k R x U k Q Z r 1 5 b V i p u n V 3 A b J O v I J U o U B r W P n y R z F L I 6 6 Q S W p M 3 3 M T H G R U o 2 C S z 8 t + a n h C 2 Z S O e d 9 S R S N u B t k i 8 5 x c W m V E w l j b p 5 A s 1 N 8 b G Y 2 M m U W B n c w j m l U v F / / z + i m G N 4 N M q C R F r t j y U J h K g j H J C y A j o T l D O b O E M i 1 s V s I m V F O G t q a y L c F b / f I 6 6 T T q n l v 3 H h r V 5 m 1 R R w n O 4 Q K u w I N r a M I 9 t K A N D B J 4 h l d 4 c 1 L n x X l 3 P p a j G 0 6 x c w Z / 4 H z + A I j A k V Y = < / l a t e x i t > softmax SGC SGC SGC SGC SGC SGC GCN GResNet (naive) SGC G < l a t e x i t s h a 1 _ b a s e 6 4 = " / V t + 0 R d S S 5 A / s b V i R u a b h 3 H o i B U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k W W m I U J I E L 2 V v m Y M P e 3 m V 3 z 4 R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S A T X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q 6 z h V D F s s F r H q B F S j 4 B J b h h u B n U Q h j Q K B j 8 H 4 e u Y / P q H S P J Y P Z p K g H 9 G h 5 C F n 1 F j p v n p T 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E U r D B N W 6 6 7 m J 8 T O q D G c C p 6 V e q j G h b E y H 2 L V U 0 g i 1 n 8 1 P n Z I z q w x I G C t b 0 p C 5 + n s i o 5 H W k y i w n R E 1 I 7 3 s z c T / v G 5 q w k s / 4 z J J D U q 2 W B S m g p i Y z P 4 m A 6 6 Q G T G x h D L F 7 a 2 E j a i i z N h 0 S j Y E b / n l V d K u 1 z y 3 5 t 3 V K 4 2 r P I 4 i n M A p n I M H F 9 C A W 2 h C C x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g B S 3 Y 0 l < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " / V t + 0 R d S S 5 A / s b V i R u a b h 3 H o i B U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k W W m I U J I E L 2 V v m Y M P e 3 m V 3 z 4 R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S A T X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q 6 z h V D F s s F r H q B F S j 4 B J b h h u B n U Q h j Q K B j 8 H 4 e u Y / P q H S P J Y P Z p K g H 9 G h 5 C F n 1 F j p v n p T 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E U r D B N W 6 6 7 m J 8 T O q D G c C p 6 V e q j G h b E y H 2 L V U 0 g i 1 n 8 1 P n Z I z q w x I G C t b 0 p C 5 + n s i o 5 H W k y i w n R E 1 I 7 3 s z c T / v G 5 q w k s / 4 z J J D U q 2 W B S m g p i Y z P 4 m A 6 6 Q G T G x h D L F 7 a 2 E j a i i z N h 0 S j Y E b / n l V d K u 1 z y 3 5 t 3 V K 4 2 r P I 4 i n M A p n I M H F 9 C A W 2 h C C x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g B S 3 Y 0 l < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " / V t + 0 R d S S 5 A / s b V i R u a b h 3 H o i B U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k W W m I U J I E L 2 V v m Y M P e 3 m V 3 z 4 R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S A T X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q 6 z h V D F s s F r H q B F S j 4 B J b h h u B n U Q h j Q K B j 8 H 4 e u Y / P q H S P J Y P Z p K g H 9 G h 5 C F n 1 F j p v n p T 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E U r D B N W 6 6 7 m J 8 T O q D G c C p 6 V e q j G h b E y H 2 L V U 0 g i 1 n 8 1 P n Z I z q w x I G C t b 0 p C 5 + n s i o 5 H W k y i w n R E 1 I 7 3 s z c T / v G 5 q w k s / 4 z J J D U q 2 W B S m g p i Y z P 4 m A 6 6 Q G T G x h D L F 7 a 2 E j a i i z N h 0 S j Y E b / n l V d K u 1 z y 3 5 t 3 V K 4 2 r P I 4 i n M A p n I M H F 9 C A W 2 h C C x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g B S 3 Y 0 l < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " / V t + 0 R d S S 5 A / s b V i R u a b h 3 H o i B U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k W W m I U J I E L 2 V v m Y M P e 3 m V 3 z 4 R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S A T X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q 6 z h V D F s s F r H q B F S j 4 B J b h h u B n U Q h j Q K B j 8 H 4 e u Y / P q H S P J Y P Z p K g H 9 G h 5 C F n 1 F j p v n p T 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E U r D B N W 6 6 7 m J 8 T O q D G c C p 6 V e q j G h b E y H 2 L V U 0 g i 1 n 8 1 P n Z I z q w x I G C t b 0 p C 5 + n s i o 5 H W k y i w n R E 1 I 7 3 s z c T / v G 5 q w k s / 4 z J J D U q 2 W B S m g p i Y z P 4 m A 6 6 Q G T G x h D L F 7 a 2 E j a i i z N h 0 S j Y E b / n l V d K u 1 z y 3 5 t 3 V K 4 2 r P I 4 i n M A p n I M H F 9 C A W 2 h C C x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g B S 3 Y 0 l < / l a t e x i t > X < l a t e x i t s h a 1 _ b a s e 6 4 = " M c N d i W G T h Q s 7 u 8 o H n k B R i f D 5 r u w = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y C u 4 K k k 3 u i y 6 c V n B P q A p Z T K d t E M n k z B z I 5 T Q 3 3 D j Q h G 3 / o w 7 / 8 Z J m 4 W 2 H h g 4 n H M v 9 8 w J E i k M u u 6 3 s 7 G 5 t b 2 z W 9 o r 7 x 8 c H h 1 X T k 4 7 J k 4 1 4 2 0 W y 1 j 3 A m q 4 F I q 3 U a D k v U R z G g W S d 4 P p X e 5 3 n 7 g 2 I l a P O E v 4 I K J j J U L B K F r J r / k R x U k Q Z r 1 5 b V i p u n V 3 A b J O v I J U o U B r W P n y R z F L I 6 6 Q S W p M 3 3 M T H G R U o 2 C S z 8 t + a n h C 2 Z S O e d 9 S R S N u B t k i 8 5 x c W m V E w l j b p 5 A s 1 N 8 b G Y 2 M m U W B n c w j m l U v F / / z + i m G N 4 N M q C R F r t j y U J h K g j H J C y A j o T l D O b O E M i 1 s V s I m V F O G t q a y L c F b / f I 6 6 T T q n l v 3 H h r V 5 m 1 R R w n O 4 Q K u w I N r a M I 9 t K A N D B J 4 h l d 4 c 1 L n x X l 3 P p a j G 0 6 x c w Z / 4 H z + A I j A k V Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M c N d i W G T h Q s 7 u 8 o H n k B R i f D 5 r u w = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y C u 4 K k k 3 u i y 6 c V n B P q A p Z T K d t E M n k z B z I 5 T Q 3 3 D j Q h G 3 / o w 7 / 8 Z J m 4 W 2 H h g 4 n H M v 9 8 w J E i k M u u 6 3 s 7 G 5 t b 2 z W 9 o r 7 x 8 c H h 1 X T k 4 7 J k 4 1 4 2 0 W y 1 j 3 A m q 4 F I q 3 U a D k v U R z G g W S d 4 P p X e 5 3 n 7 g 2 I l a P O E v 4 I K J j J U L B K F r J r / k R x U k Q Z r 1 5 b V i p u n V 3 A b J O v I J U o U B r W P n y R z F L I 6 6 Q S W p M 3 3 M T H G R U o 2 C S z 8 t + a n h C 2 Z S O e d 9 S R S N u B t k i 8 5 x c W m V E w l j b p 5 A s 1 N 8 b G Y 2 M m U W B n c w j m l U v F / / z + i m G N 4 N M q C R F r t j y U J h K g j H J C y A j o T l D O b O E M i 1 s V s I m V F O G t q a y L c F b / f I 6 6 T T q n l v 3 H h r V 5 m 1 R R w n O 4 Q K u w I N r a M I 9 t K A N D B J 4 h l d 4 c 1 L n x X l 3 P p a j G 0 6 x c w Z / 4 H z + A I j A k V Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M c N d i W G T h Q s 7 u 8 o H n k B R i f D 5 r u w = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y C u 4 K k k 3 u i y 6 c V n B P q A p Z T K d t E M n k z B z I 5 T Q 3 3 D j Q h G 3 / o w 7 / 8 Z J m 4 W 2 H h g 4 n H M v 9 8 w J E i k M u u 6 3 s 7 G 5 t b 2 z W 9 o r 7 x 8 c H h 1 X T k 4 7 J k 4 1 4 2 0 W y 1 j 3 A m q 4 F I q 3 U a D k v U R z G g W S d 4 P p X e 5 3 n 7 g 2 I l a P O E v 4 I K J j J U L B K F r J r / k R x U k Q Z r 1 5 b V i p u n V 3 A b J O v I J U o U B r W P n y R z F L I 6 6 Q S W p M 3 3 M T H G R U o 2 C S z 8 t + a n h C 2 Z S O e d 9 S R S N u B t k i 8 5 x c W m V E w l j b p 5 A s 1 N 8 b G Y 2 M m U W B n c w j m l U v F / / z + i m G N 4 N M q C R F r t j y U J h K g j H J C y A j o T l D O b O E M i 1 s V s I m V F O G t q a y L c F b / f I 6 6 T T q n l v 3 H h r V 5 m 1 R R w n O 4 Q K u w I N r a M I 9 t K A N D B J 4 h l d 4 c 1 L n x X l 3 P p a j G 0 6 x c w Z / 4 H z + A I j A k V Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M c N d i W G T h Q s 7 u 8 o H n k B R i f D 5 r u w = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y C u 4 K k k 3 u i y 6 c V n B P q A p Z T K d t E M n k z B z I 5 T Q 3 3 D j Q h G 3 / o w 7 / 8 Z J m 4 W 2 H h g 4 n H M v 9 8 w J E i k M u u 6 3 s 7 G 5 t b 2 z W 9 o r 7 x 8 c H h 1 X T k 4 7 J k 4 1 4 2 0 W y 1 j 3 A m q 4 F I q 3 U a D k v U R z G g W S d 4 P p X e 5 3 n 7 g 2 I l a P O E v 4 I K J j J U L B K F r J r / k R x U k Q Z r 1 5 b V i p u n V 3 A b J O v I J U o U B r W P n y R z F L I 6 6 Q S W p M 3 3 M T H G R U o 2 C S z 8 t + a n h C 2 Z S O e d 9 S R S N u B t k i 8 5 x c W m V E w l j b p 5 A s 1 N 8 b G Y 2 M m U W B n c w j m l U v F / / z + i m G N 4 N M q C R F r t j y U J h K g j H J C y A j o T l D O b O E M i 1 s V s I m V F O G t q a y L c F b / f I 6 6 T T q n l v 3 H h r V 5 m 1 R R w n O 4 Q K u w I N r a M I 9 t K A N D B J 4 h l d 4 c 1 L n x X l 3 P p a j G 0 6 x c w Z / 4 H z + A I j A k V Y = < / l a t e x i t > SGC SGC SGC SGC SGC SGC GResNet (graph naive) SGC G < l a t e x i t s h a 1 _ b a s e 6 4 = " / V t + 0 R d S S 5 A / s b V i R u a b h 3 H o i B U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k W W m I U J I E L 2 V v m Y M P e 3 m V 3 z 4 R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S A T X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q 6 z h V D F s s F r H q B F S j 4 B J b h h u B n U Q h j Q K B j 8

H 4 e u Y / P q H S P J Y P Z p K g H 9 G h 5 C F n 1 F j p v n p T 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E U r D B N W 6 6 7 m J 8 T O q D G c C p 6 V e q j G h b E y H 2 L V U 0 g i 1 n 8 1 P n Z I z q w x I G C t b 0 p C 5 + n s i o 5 H W k y i w n R E 1 I 7 3 s z c T / v G 5 q w k s / 4 z J J D U q 2 W B S m g p i Y z P 4

m A 6 6 Q G T G x h D L F 7 a 2 E j a i i z N h 0 S j Y E b / n l V d K u 1 z y 3 5 t 3 V K 4 2 r P I 4 i n M A p n I M H F 9 C A W 2 h C C x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g B S 3 Y 0 l < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " / V t + 0 R d S S 5 A / s b V i R u a b h 3 H o i B U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k W W m I U J I E L 2 V v m Y M P e 3 m V 3 z 4 R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S A T X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q 6 z h V D F s s F r H q B F S j 4 B J b h h u B n U Q h j Q K B j 8

H 4 e u Y / P q H S P J Y P Z p K g H 9 G h 5 C F n 1 F j p v n p T 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E U r D B N W 6 6 7 m J 8 T O q D G c C p 6 V e q j G h b E y H 2 L V U 0 g i 1 n 8 1 P n Z I z q w x I G C t b 0 p C 5 + n s i o 5 H W k y i w n R E 1 I 7 3 s z c T / v G 5 q w k s / 4 z J J D U q 2 W B S m g p i Y z P 4

m A 6 6 Q G T G x h D L F 7 a 2 E j a i i z N h 0 S j Y E b / n l V d K u 1 z y 3 5 t 3 V K 4 2 r P I 4 i n M A p n I M H F 9 C A W 2 h C C x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g B S 3 Y 0 l < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " / V t + 0 R d S S 5 A / s b V i R u a b h 3 H o i B U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k W W m I U J I E L 2 V v m Y M P e 3 m V 3 z 4 R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S A T X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q 6 z h V D F s s F r H q B F S j 4 B J b h h u B n U Q h j Q K B j 8

H 4 e u Y / P q H S P J Y P Z p K g H 9 G h 5 C F n 1 F j p v n p T 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E U r D B N W 6 6 7 m J 8 T O q D G c C p 6 V e q j G h b E y H 2 L V U 0 g i 1 n 8 1 P n Z I z q w x I G C t b 0 p C 5 + n s i o 5 H W k y i w n R E 1 I 7 3 s z c T / v G 5 q w k s / 4 z J J D U q 2 W B S m g p i Y z P 4

m A 6 6 Q G T G x h D L F 7 a 2 E j a i i z N h 0 S j Y E b / n l V d K u 1 z y 3 5 t 3 V K 4 2 r P I 4 i n M A p n I M H F 9 C A W 2 h C C x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g B S 3 Y 0 l < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " / V t + 0 R d S S 5 A / s b V i R u a b h 3 H o i B U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k W W m I U J I E L 2 V v m Y M P e 3 m V 3 z 4 R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S A T X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q 6 z h V D F s s F r H q B F S j 4 B J b h h u B n U Q h j Q K B j 8

H 4 e u Y / P q H S P J Y P Z p K g H 9 G h 5 C F n 1 F j p v n p T 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E U r D B N W 6 6 7 m J 8 T O q D G c C p 6 V e q j G h b E y H 2 L V U 0 g i 1 n 8 1 P n Z I z q w x I G C t b 0 p C 5 + n s i o 5 H W k y i w n R E 1 I 7 3 s z c T / v G 5 q w k s / 4 z J J D U q 2 W B S m g p i Y z P 4

m A 6 6 Q G T G x h D L F 7 a 2 E j a i i z N h 0 S j Y E b / n l V d K u 1 z y 3 5 t 3 V K 4 2 r P I 4 i n M A p n I M H F 9 C A W 2 h C C x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g B S 3 Y 0 l < / l a t e x i t > X < l a t e x i t s h a 1 _ b a s e 6 4 = " M c N d i W G T h Q s 7 u 8 o H n k B R i f D 5 r u w = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y C u 4 K k k 3 u i y 6 c V n B P q A p Z T K d t E M n k z B z I 5 T Q 3 3 D j Q h G 3 / o w 7 / 8 Z J m 4 W 2 H h g 4 n H M v 9 8 w J E i k M u u 6 3 s 7 G 5 t b 2 z W 9 o r 7 x 8 c H h 1 X T k 4 7 J k 4 1 4 2 0 W y 1 j 3 A m q 4 F I q 3 U a D k v U R z G g W S d 4 P p X e 5 3 n 7 g 2 I l a P O E v 4 I K J j J U L B K F r J r / k R x U k Q Z r 1 5 b V i p u n V 3 A b J O v I J U o U B r W P n y R z F L I 6 6 Q S W p M 3 3 M T H G R U o 2 C S z 8 t + a n h C 2 Z S O e d 9 S R S N u B t k i 8 5 x c W m V E w l j b p 5 A s 1 N 8 b G Y 2 M m U W B n c w j m l U v F / / z + i m G N 4 N M q C R F r t j y U J h K g j H J C y A j o T l D O b O E M i 1 s V s I m V F O G t q a y L c F b / f I 6 6 T T q n l v 3 H h r V 5 m 1 R R w n O 4 Q K u w I N r a M I 9 t K A N D B J 4 h l d 4 c 1 L n x X l 3 P p a j G 0 6 x c w Z / 4 H z + A I j A k V Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M c N d i W G T h Q s 7 u 8 o H n k B R i f D 5 r u w = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y C u 4 K k k 3 u i y 6 c V n B P q A p Z T K d t E M n k z B z I 5 T Q 3 3 D j Q h G 3 / o w 7 / 8 Z J m 4 W 2 H h g 4 n H M v 9 8 w J E i k M u u 6 3 s 7 G 5 t b 2 z W 9 o r 7 x 8 c H h 1 X T k 4 7 J k 4 1 4 2 0 W y 1 j 3 A m q 4 F I q 3 U a D k v U R z G g W S d 4 P p X e 5 3 n 7 g 2 I l a P O E v 4 I K J j J U L B K F r J r / k R x U k Q Z r 1 5 b V i p u n V 3 A b J O v I J U o U B r W P n y R z F L I 6 6 Q S W p M 3 3 M T H G R U o 2 C S z 8 t + a n h C 2 Z S O e d 9 S R S N u B t k i 8 5 x c W m V E w l j b p 5 A s 1 N 8 b G Y 2 M m U W B n c w j m l U v F / / z + i m G N 4 N M q C R F r t j y U J h K g j H J C y A j o T l D O b O E M i 1 s V s I m V F O G t q a y L c F b / f I 6 6 T T q n l v 3 H h r V 5 m 1 R R w n O 4 Q K u w I N r a M I 9 t K A N D B J 4 h l d 4 c 1 L n x X l 3 P p a j G 0 6 x c w Z / 4 H z + A I j A k V Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M c N d i W G T h Q s 7 u 8 o H n k B R i f D 5 r u w = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y C u 4 K k k 3 u i y 6 c V n B P q A p Z T K d t E M n k z B z I 5 T Q 3 3 D j Q h G 3 / o w 7 / 8 Z J m 4 W 2 H h g 4 n H M v 9 8 w J E i k M u u 6 3 s 7 G 5 t b 2 z W 9 o r 7 x 8 c H h 1 X T k 4 7 J k 4 1 4 2 0 W y 1 j 3 A m q 4 F I q 3 U a D k v U R z G g W S d 4 P p X e 5 3 n 7 g 2 I l a P O E v 4 I K J j J U L B K F r J r / k R x U k Q Z r 1 5 b V i p u n V 3 A b J O v I J U o U B r W P n y R z F L I 6 6 Q S W p M 3 3 M T H G R U o 2 C S z 8 t + a n h C 2 Z S O e d 9 S R S N u B t k i 8 5 x c W m V E w l j b p 5 A s 1 N 8 b G Y 2 M m U W B n c w j m l U v F / / z + i m G N 4 N M q C R F r t j y U J h K g j H J C y A j o T l D O b O E M i 1 s V s I m V F O G t q a y L c F b / f I 6 6 T T q n l v 3 H h r V 5 m 1 R R w n O 4 Q K u w I N r a M I 9 t K A N D B J 4 h l d 4 c 1 L n x X l 3 P p a j G 0 6 x c w Z / 4 H z + A I j A k V Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M c N d i W G T h Q s 7 u 8 o H n k B R i f D 5 r u w = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y C u 4 K k k 3 u i y 6 c V n B P q A p Z T K d t E M n k z B z I 5 T Q 3 3 D j Q h G 3 / o w 7 / 8 Z J m 4 W 2 H h g 4 n H M v 9 8 w J E i k M u u 6 3 s 7 G 5 t b 2 z W 9 o r 7 x 8 c H h 1 X T k 4 7 J k 4 1 4 2 0 W y 1 j 3 A m q 4 F I q 3 U a D k v U R z G g W S d 4 P p X e 5 3 n 7 g 2 I l a P O E v 4 I K J j J U L B K F r J r / k R x U k Q Z r 1 5 b V i p u n V 3 A b J O v I J U o U B r W P n y R z F L I 6 6 Q S W p M 3 3 M T H G R U o 2 C S z 8 t + a n h C 2 Z S O e d 9 S R S N u B t k i 8 5 x c W m V E w l j b p 5 A s 1 N 8 b G Y 2 M m U W B n c w j m l U v F / / z + i m G N 4 N M q C R F r t j y U J h K g j H J C y A j o T l D O b O E M i 1 s V s I m V F O G t q a y L c F b / f I 6 6 T T q n l v 3 H h r V 5 m 1 R R w n O 4 Q K u w I N r a M I 9 t K A N D B J 4 h l d 4 c 1 L n x X l 3 P p a j G 0 6 x c w Z / 4 H z + A I j A k V Y = < / l a t e x i t > SGC SGC SGC SGC SGC SGC GResNet (graph raw) Â Â Â Â Â Â Â Â Â Â Â Â Â Â ReLU ReLU … ReLU … ReLU … Ŷ < l a t e x i t s h a 1 _ b a s e 6 4 = " o z 9 b n o K A m X r k l l 8 d m 1 p i U 9 c H r k 8 = " > A A A B + 3 i c b V C 7 T s M w F H X K q 5 R X K C O L R Y v E V C V d Y K x g Y S w S f a C m q h z X a a 0 6 T m T f I K o o v 8 L C A E K s / A g b f 4 P T Z o C W I 1 k 6 O u d e 3 e P j x 4 J r c J x v q 7 S x u b W 9 U 9 6 t 7 O 0 f H B 7 Z x 9 W u j h J F W Y d G I l J 9 n 2 g m u G Q d 4 C B Y P 1 a M h L 5 g P X 9 2 k / u 9 R 6 Y 0 j + Q 9 z G M 2 D M l E 8 o B T A k Y a 2 d W 6 N y W Q e i G B q R + k D 1 l W H 9 k 1 p + E s g N e J W 5 A a K t A e 2 V / e O K J J y C R Q Q b Q e u E 4 M w 5 Q o 4 F S w r O I l m s W E z s i E D Q y V J G R 6 m C 6 y Z / j c K G M c R M o 8

C X i h / t 5 I S a j 1 P P T N Z J 5 R r 3 q 5 + J 8 3 S C C 4 G q Z c x g k w S Z e H g k R g i H B e B B 5 z x S i I u S G E K m 6 y Y j o l i l A w d V V M C e 7 q l 9 d J t 9 l w n Y Z 7 1 6 y 1 r o s 6 y u g U n a E L 5 K J L 1 E K 3 q I 0 6 i K I n 9 I x e 0 Z u V W S / W u / W x H C 1 Z x c 4 J + g P r 8 w f m o 5 R V < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o z 9

b n o K A m X r k l l 8 d m 1 p i U 9 c H r k 8 = " > A A A B + 3 i c b V C 7 T s M w F H X K q 5 R X K C O L R Y v E V C V d Y K x g Y S w S f a C m q h z X a a 0 6 T m T f I K o o v 8 L C A E K s / A g b f 4 P T Z o C W I 1 k 6 O u d e 3 e P j x 4 J r c J x v q 7 S x u b W 9 U 9 6 t 7 O 0 f H B 7 Z x 9 W u j h J F W Y d G I l J 9 n 2 g m u G Q d 4 C B Y P 1 a M h L 5 g P X 9 2 k / u 9 R 6 Y 0 j + Q 9 z G M 2 D M l E 8 o B T A k Y a 2 d W 6 N y W Q e i G B q R + k D 1 l W H 9 k 1 p + E s g N e J W 5 A a K t A e 2 V / e O K J J y C R Q Q b Q e u E 4 M w 5 Q o 4 F S w r O I l m s W E z s i E D Q y V J G R 6 m C 6 y Z / j c K G M c R M o 8

C X i h / t 5 I S a j 1 P P T N Z J 5 R r 3 q 5 + J 8 3 S C C 4 G q Z c x g k w S Z e H g k R g i H B e B B 5 z x S i I u S G E K m 6 y Y j o l i l A w d V V M C e 7 q l 9 d J t 9 l w n Y Z 7 1 6 y 1 r o s 6 y u g U n a E L 5 K J L 1 E K 3 q I 0 6 i K I n 9 I x e 0 Z u V W S / W u / W x H C 1 Z x c 4 J + g P r 8 w f m o 5 R V < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o z 9

b n o K A m X r k l l 8 d m 1 p i U 9 c H r k 8 = " > A A A B + 3 i c b V C 7 T s M w F H X K q 5 R X K C O L R Y v E V C V d Y K x g Y S w S f a C m q h z X a a 0 6 T m T f I K o o v 8 L C A E K s / A g b f 4 P T Z o C W I 1 k 6 O u d e 3 e P j x 4 J r c J x v q 7 S x u b W 9 U 9 6 t 7 O 0 f H B 7 Z x 9 W u j h J F W Y d G I l J 9 n 2 g m u G Q d 4 C B Y P 1 a M h L 5 g P X 9 2 k / u 9 R 6 Y 0 j + Q 9 z G M 2 D M l E 8 o B T A k Y a 2 d W 6 N y W Q e i G B q R + k D 1 l W H 9 k 1 p + E s g N e J W 5 A a K t A e 2 V / e O K J J y C R Q Q b Q e u E 4 M w 5 Q o 4 F S w r O I l m s W E z s i E D Q y V J G R 6 m C 6 y Z / j c K G M c R M o 8

C X i h / t 5 I S a j 1 P P T N Z J 5 R r 3 q 5 + J 8 3 S C C 4 G q Z c x g k w S Z e H g k R g i H B e B B 5 z x S i I u S G E K m 6 y Y j o l i l A w d V V M C e 7 q l 9 d J t 9 l w n Y Z 7 1 6 y 1 r o s 6 y u g U n a E L 5 K J L 1 E K 3 q I 0 6 i K I n 9 I x e 0 Z u V W S / W u / W x H C 1 Z x c 4 J + g P r 8 w f m o 5 R V < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o z 9

b n o K A m X r k l l 8 d m 1 p i U 9 c H r k 8 = " > A A A B + 3 i c b V C 7 T s M w F H X K q 5 R X K C O L R Y v E V C V d Y K x g Y S w S f a C m q h z X a a 0 6 T m T f I K o o v 8 L C A E K s / A g b f 4 P T Z o C W I 1 k 6 O u d e 3 e P j x 4 J r c J x v q 7 S x u b W 9 U 9 6 t 7 O 0 f H B 7 Z x 9 W u j h J F W Y d G I l J 9 n 2 g m u G Q d 4 C B Y P 1 a M h L 5 g P X 9 2 k / u 9 R 6 Y 0 j + Q 9 z G M 2 D M l E 8 o B T A k Y a 2 d W 6 N y W Q e i G B q R + k D 1 l W H 9 k 1 p + E s g N e J W 5 A a K t A e 2 V / e O K J J y C R Q Q b Q e u E 4 M w 5 Q o 4 F S w r O I l m s W E z s i E D Q y V J G R 6 m C 6 y Z / j c K G M c R M o 8

C X i h / t 5 I S a j 1 P P T N Z J 5 R r 3 q 5 + J 8 3 S C C 4 G q Z c x g k w S Z e H g k R g i H B e B B 5 z x S i I u S G E K m 6 y Y j o l i l A w d V V M C e 7 q l 9 d J t 9 l w n Y Z 7 1 6 y 1 r o s 6 y u g U n a E L 5 K J L 1 E K 3 q I 0 6 i K I n 9 I x e 0 Z u V W S / W u / W x H C 1 Z x c 4 J + g P r 8 w f m o 5 R V < / l a t e x i t > Ŷ < l a t e x i t s h a 1 _ b a s e 6 4 = " o z 9

b n o K A m X r k l l 8 d m 1 p i U 9 c H r k 8 = " > A A A B + 3 i c b V C 7 T s M w F H X K q 5 R X K C O L R Y v E V C V d Y K x g Y S w S f a C m q h z X a a 0 6 T m T f I K o o v 8 L C A E K s / A g b f 4 P T Z o C W I 1 k 6 O u d e 3 e P j x 4 J r c J x v q 7 S x u b W 9 U 9 6 t 7 O 0 f H B 7 Z x 9 W u j h J F W Y d G I l J 9 n 2 g m u G Q d 4 C B Y P 1 a M h L 5 g P X 9 2 k / u 9 R 6 Y 0 j + Q 9 z G M 2 D M l E 8 o B T A k Y a 2 d W 6 N y W Q e i G B q R + k D 1 l W H 9 k 1 p + E s g N e J W 5 A a K t A e 2 V / e O K J J y C R Q Q b Q e u E 4 M w 5 Q o 4 F S w r O I l m s W E z s i E D Q y V J G R 6 m C 6 y Z / j c K G M c R M o 8

C X i h / t 5 I S a j 1 P P T N Z J 5 R r 3 q 5 + J 8 3 S C C 4 G q Z c x g k w S Z e H g k R g i H B e B B 5 z x S i I u S G E K m 6 y Y j o l i l A w d V V M C e 7 q l 9 d J t 9 l w n Y Z 7 1 6 y 1 r o s 6 y u g U n a E L 5 K J L 1 E K 3 q I 0 6 i K I n 9 I x e 0 Z u V W S / W u / W x H C 1 Z x c 4 J + g P r 8 w f m o 5 R V < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o z 9

b n o K A m X r k l l 8 d m 1 p i U 9 c H r k 8 = " > A A A B + 3 i c b V C 7 T s M w F H X K q 5 R X K C O L R Y v E V C V d Y K x g Y S w S f a C m q h z X a a 0 6 T m T f I K o o v 8 L C A E K s / A g b f 4 P T Z o C W I 1 k 6 O u d e 3 e P j x 4 J r c J x v q 7 S x u b W 9 U 9 6 t 7 O 0 f H B 7 Z x 9 W u j h J F W Y d G I l J 9 n 2 g m u G Q d 4 C B Y P 1 a M h L 5 g P X 9 2 k / u 9 R 6 Y 0 j + Q 9 z G M 2 D M l E 8 o B T A k Y a 2 d W 6 N y W Q e i G B q R + k D 1 l W H 9 k 1 p + E s g N e J W 5 A a K t A e 2 V / e O K J J y C R Q Q b Q e u E 4 M w 5 Q o 4 F S w r O I l m s W E z s i E D Q y V J G R 6 m C 6 y Z / j c K G M c R M o 8

C X i h / t 5 I S a j 1 P P T N Z J 5 R r 3 q 5 + J 8 3 S C C 4 G q Z c x g k w S Z e H g k R g i H B e B B 5 z x S i I u S G E K m 6 y Y j o l i l A w d V V M C e 7 q l 9 d J t 9 l w n Y Z 7 1 6 y 1 r o s 6 y u g U n a E L 5 K J L 1 E K 3 q I 0 6 i K I n 9 I x e 0 Z u V W S / W u / W x H C 1 Z x c 4 J + g P r 8 w f m o 5 R V < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o z 9

b n o K A m X r k l l 8 d m 1 p i U 9 c H r k 8 = " > A A A B + 3 i c b V C 7 T s M w F H X K q 5 R X K C O L R Y v E V C V d Y K x g Y S w S f a C m q h z X a a 0 6 T m T f I K o o v 8 L C A E K s / A g b f 4 P T Z o C W I 1 k 6 O u d e 3 e P j x 4 J r c J x v q 7 S x u b W 9 U 9 6 t 7 O 0 f H B 7 Z x 9 W u j h J F W Y d G I l J 9 n 2 g m u G Q d 4 C B Y P 1 a M h L 5 g P X 9 2 k / u 9 R 6 Y 0 j + Q 9 z G M 2 D M l E 8 o B T A k Y a 2 d W 6 N y W Q e i G B q R + k D 1 l W H 9 k 1 p + E s g N e J W 5 A a K t A e 2 V / e O K J J y C R Q Q b Q e u E 4 M w 5 Q o 4 F S w r O I l m s W E z s i E D Q y V J G R 6 m C 6 y Z / j c K G M c R M o 8

C X i h / t 5 I S a j 1 P P T N Z J 5 R r 3 q 5 + J 8 3 S C C 4 G q Z c x g k w S Z e H g k R g i H B e B B 5 z x S i I u S G E K m 6 y Y j o l i l A w d V V M C e 7 q l 9 d J t 9 l w n Y Z 7 1 6 y 1 r o s 6 y u g U n a E L 5 K J L 1 E K 3 q I 0 6 i K I n 9 I x e 0 Z u V W S / W u / W x H C 1 Z x c 4 J + g P r 8 w f m o 5 R V < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o z 9

b n o K A m X r k l l 8 d m 1 p i U 9 c H r k 8 = " > A A A B + 3 i c b V C 7 T s M w F H X K q 5 R X K C O L R Y v E V C V d Y K x g Y S w S f a C m q h z X a a 0 6 T m T f I K o o v 8 L C A E K s / A g b f 4 P T Z o C W I 1 k 6 O u d e 3 e P j x 4 J r c J x v q 7 S x u b W 9 U 9 6 t 7 O 0 f H B 7 Z x 9 W u j h J F W Y d G I l J 9 n 2 g m u G Q d 4 C B Y P 1 a M h L 5 g P X 9 2 k / u 9 R 6 Y 0 j + Q 9 z G M 2 D M l E 8 o B T A k Y a 2 d W 6 N y W Q e i G B q R + k D 1 l W H 9 k 1 p + E s g N e J W 5 A a K t A e 2 V / e O K J J y C R Q Q b Q e u E 4 M w 5 Q o 4 F S w r O I l m s W E z s i E D Q y V J G R 6 m C 6 y Z / j c K G M c R M o 8

C X i h / t 5 I S a j 1 P P T N Z J 5 R r 3 q 5 + J 8 3 S C C 4 G q Z c x g k w S Z e H g k R g i H B e B B 5 z x S i I u S G E K m 6 y Y j o l i l A w d V V M C e 7 q l 9 d J t 9 l w n Y Z 7 1 6 y 1 r o s 6 y u g U n a E L 5 K J L 1 E K 3 q I 0 6 i K I n 9 I x e 0 Z u V W S / W u / W x H C 1 Z x c 4 J + g P r 8 w f m o 5 R V < / l a t e x i t > Ŷ < l a t e x i t s h a 1 _ b a s e 6 4 = " o z 9

b n o K A m X r k l l 8 d m 1 p i U 9 c H r k 8 = " > A A A B + 3 i c b V C 7 T s M w F H X K q 5 R X K C O L R Y v E V C V d Y K x g Y S w S f a C m q h z X a a 0 6 T m T f I K o o v 8 L C A E K s / A g b f 4 P T Z o C W I 1 k 6 O u d e 3 e P j x 4 J r c J x v q 7 S x u b W 9 U 9 6 t 7 O 0 f H B 7 Z x 9 W u j h J F W Y d G I l J 9 n 2 g m u G Q d 4 C B Y P 1 a M h L 5 g P X 9 2 k / u 9 R 6 Y 0 j + Q 9 z G M 2 D M l E 8 o B T A k Y a 2 d W 6 N y W Q e i G B q R + k D 1 l W H 9 k 1 p + E s g N e J W 5 A a K t A e 2 V / e O K J J y C R Q Q b Q e u E 4 M w 5 Q o 4 F S w r O I l m s W E z s i E D Q y V J G R 6 m C 6 y Z / j c K G M c R M o 8

C X i h / t 5 I S a j 1 P P T N Z J 5 R r 3 q 5 + J 8 3 S C C 4 G q Z c x g k w S Z e H g k R g i H B e B B 5 z x S i I u S G E K m 6 y Y j o l i l A w d V V M C e 7 q l 9 d J t 9 l w n Y Z 7 1 6 y 1 r o s 6 y u g U n a E L 5 K J L 1 E K 3 q I 0 6 i K I n 9 I x e 0 Z u V W S / W u / W x H C 1 Z x c 4 J + g P r 8 w f m o 5 R V < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o z 9

b n o K A m X r k l l 8 d m 1 p i U 9 c H r k 8 = " > A A A B + 3 i c b V C 7 T s M w F H X K q 5 R X K C O L R Y v E V C V d Y K x g Y S w S f a C m q h z X a a 0 6 T m T f I K o o v 8 L C A E K s / A g b f 4 P T Z o C W I 1 k 6 O u d e 3 e P j x 4 J r c J x v q 7 S x u b W 9 U 9 6 t 7 O 0 f H B 7 Z x 9 W u j h J F W Y d G I l J 9 n 2 g m u G Q d 4 C B Y P 1 a M h L 5 g P X 9 2 k / u 9 R 6 Y 0 j + Q 9 z G M 2 D M l E 8 o B T A k Y a 2 d W 6 N y W Q e i G B q R + k D 1 l W H 9 k 1 p + E s g N e J W 5 A a K t A e 2 V / e O K J J y C R Q Q b Q e u E 4 M w 5 Q o 4 F S w r O I l m s W E z s i E D Q y V J G R 6 m C 6 y Z / j c K G M c R M o 8

C X i h / t 5 I S a j 1 P P T N Z J 5 R r 3 q 5 + J 8 3 S C C 4 G q Z c x g k w S Z e H g k R g i H B e B B 5 z x S i I u S G E K m 6 y Y j o l i l A w d V V M C e 7 q l 9 d J t 9 l w n Y Z 7 1 6 y 1 r o s 6 y u g U n a E L 5 K J L 1 E K 3 q I 0 6 i K I n 9 I x e 0 Z u V W S / W u / W x H C 1 Z x c 4 J + g P r 8 w f m o 5 R V < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o z 9

b n o K A m X r k l l 8 d m 1 p i U 9 c H r k 8 = " > A A A B + 3 i c b V C 7 T s M w F H X K q 5 R X K C O L R Y v E V C V d Y K x g Y S w S f a C m q h z X a a 0 6 T m T f I K o o v 8 L C A E K s / A g b f 4 P T Z o C W I 1 k 6 O u d e 3 e P j x 4 J r c J x v q 7 S x u b W 9 U 9 6 t 7 O 0 f H B 7 Z x 9 W u j h J F W Y d G I l J 9 n 2 g m u G Q d 4 C B Y P 1 a M h L 5 g P X 9 2 k / u 9 R 6 Y 0 j + Q 9 z G M 2 D M l E 8 o B T A k Y a 2 d W 6 N y W Q e i G B q R + k D 1 l W H 9 k 1 p + E s g N e J W 5 A a K t A e 2 V / e O K J J y C R Q Q b Q e u E 4 M w 5 Q o 4 F S w r O I l m s W E z s i E D Q y V J G R 6 m C 6 y Z / j c K G M c R M o 8

C X i h / t 5 I S a j 1 P P T N Z J 5 R r 3 q 5 + J 8 3 S C C 4 G q Z c x g k w S Z e H g k R g i H B e B B 5 z x S i I u S G E K m 6 y Y j o l i l A w d V V M C e 7 q l 9 d J t 9 l w n Y Z 7 1 6 y 1 r o s 6 y u g U n a E L 5 K J L 1 E K 3 q I 0 6 i K I n 9 I x e 0 Z u V W S / W u / W x H C 1 Z x c 4 J + g P r 8 w f m o 5 R V < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o z 9

b n o K A m X r k l l 8 d m 1 p i U 9 c H r k 8 = " > A A A B + 3 i c b V C 7 T s M w F H X K q 5 R X K C O L R Y v E V C V d Y K x g Y S w S f a C m q h z X a a 0 6 T m T f I K o o v 8 L C A E K s / A g b f 4 P T Z o C W I 1 k 6 O u d e 3 e P j x 4 J r c J x v q 7 S x u b W 9 U 9 6 t 7 O 0 f H B 7 Z x 9 W u j h J F W Y d G I l J 9 n 2 g m u G Q d 4 C B Y P 1 a M h L 5 g P X 9 2 k / u 9 R 6 Y 0 j + Q 9 z G M 2 D M l E 8 o B T A k Y a 2 d W 6 N y W Q e i G B q R + k D 1 l W H 9 k 1 p + E s g N e J W 5 A a K t A e 2 V / e O K J J y C R Q Q b Q e u E 4 M w 5 Q o 4 F S w r O I l m s W E z s i E D Q y V J G R 6 m C 6 y Z / j c K G M c R M o 8 C X i h / t 5 I S a j 1 P P T N Z J 5 R r 3 q 5 + J 8 3 S C C 4 G q Z c x g k w S Z e H g k R g i H B e B B 5 z x S i I u S G E K m 6 y Y j o l i l A w d V V M C e 7 q l 9 d J t 9 l w n Y Z 7 1 6 y 1 r o s 6 y u g U n a E L 5 K J L 1 E K 3 q I 0 6 i K I n 9 I x e 0 Z u V W S / W u / W x H C 1 Z x c 4 J + g P r 8 w f m o 5 R V < / l a t e x i t > Ŷ < l a t e x i t s h a 1 _ b a s e 6 4 = " o z 9 b n o K A m X r k l l 8 d m 1 p i U 9 c H r k 8 = " > A A A B + 3 i c b V C 7 T s M w F H X K q 5 R X K C O L R Y v E V C V d Y K x g Y S w S f a C m q h z X a a 0 6 T m T f I K o o v 8 L C A E K s / A g b f 4 P T Z o C W I 1 k 6 O u d e 3 e P j x 4 J r c J x v q 7 S x u b W 9 U 9 6 t 7 O 0 f H B 7 Z x 9 W u j h J F W Y d G I l J 9 n 2 g m u G Q d 4 C B Y P 1 a M h L 5 g P X 9 2 k / u 9 R 6 Y 0 j + Q 9 z G M 2 D M l E 8 o B T A k Y a 2 d W 6 N y W Q e i G B q R + k D 1 l W H 9 k 1 p + E s g N e J W 5 A a K t A e 2 V / e O K J J y C R Q Q b Q e u E 4 M w 5 Q o 4 F S w r O I l m s W E z s i E D Q y V J G R 6 m C 6 y Z / j c K G M c R M o 8

C X i h / t 5 I S a j 1 P P T N Z J 5 R r 3 q 5 + J 8 3 S C C 4 G q Z c x g k w S Z e H g k R g i H B e B B 5 z x S i I u S G E K m 6 y Y j o l i l A w d V V M C e 7 q l 9 d J t 9 l w n Y Z 7 1 6 y 1 r o s 6 y u g U n a E L 5 K J L 1 E K 3 q I 0 6 i K I n 9 I x e 0 Z u V W S / W u / W x H C 1 Z x c 4 J + g P r 8 w f m o 5 R V < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o z 9

b n o K A m X r k l l 8 d m 1 p i U 9 c H r k 8 = " > A A A B + 3 i c b V C 7 T s M w F H X K q 5 R X K C O L R Y v E V C V d Y K x g Y S w S f a C m q h z X a a 0 6 T m T f I K o o v 8 L C A E K s / A g b f 4 P T Z o C W I 1 k 6 O u d e 3 e P j x 4 J r c J x v q 7 S x u b W 9 U 9 6 t 7 O 0 f H B 7 Z x 9 W u j h J F W Y d G I l J 9 n 2 g m u G Q d 4 C B Y P 1 a M h L 5 g P X 9 2 k / u 9 R 6 Y 0 j + Q 9 z G M 2 D M l E 8 o B T A k Y a 2 d W 6 N y W Q e i G B q R + k D 1 l W H 9 k 1 p + E s g N e J W 5 A a K t A e 2 V / e O K J J y C R Q Q b Q e u E 4 M w 5 Q o 4 F S w r O I l m s W E z s i E D Q y V J G R 6 m C 6 y Z / j c K G M c R M o 8

C X i h / t 5 I S a j 1 P P T N Z J 5 R r 3 q 5 + J 8 3 S C C 4 G q Z c x g k w S Z e H g k R g i H B e B B 5 z x S i I u S G E K m 6 y Y j o l i l A w d V V M C e 7 q l 9 d J t 9 l w n Y Z 7 1 6 y 1 r o s 6 y u g U n a E L 5 K J L 1 E K 3 q I 0 6 i K I n 9 I x e 0 Z u V W S / W u / W x H C 1 Z x c 4 J + g P r 8 w f m o 5 R V < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o z 9

b n o K A m X r k l l 8 d m 1 p i U 9 c H r k 8 = " > A A A B + 3 i c b V C 7 T s M w F H X K q 5 R X K C O L R Y v E V C V d Y K x g Y S w S f a C m q h z X a a 0 6 T m T f I K o o v 8 L C A E K s / A g b f 4 P T Z o C W I 1 k 6 O u d e 3 e P j x 4 J r c J x v q 7 S x u b W 9 U 9 6 t 7 O 0 f H B 7 Z x 9 W u j h J F W Y d G I l J 9 n 2 g m u G Q d 4 C B Y P 1 a M h L 5 g P X 9 2 k / u 9 R 6 Y 0 j + Q 9 z G M 2 D M l E 8 o B T A k Y a 2 d W 6 N y W Q e i G B q R + k D 1 l W H 9 k 1 p + E s g N e J W 5 A a K t A e 2 V / e O K J J y C R Q Q b Q e u E 4 M w 5 Q o 4 F S w r O I l m s W E z s i E D Q y V J G R 6 m C 6 y Z / j c K G M c R M o 8

C X i h / t 5 I S a j 1 P P T N Z J 5 R r 3 q 5 + J 8 3 S C C 4 G q Z c x g k w S Z e H g k R g i H B e B B 5 z x S i I u S G E K m 6 y Y j o l i l A w d V V M C e 7 q l 9 d J t 9 l w n Y Z 7 1 6 y 1 r o s 6 y u g U n a E L 5 K J L 1 E K 3 q I 0 6 i K I n 9 I x e 0 Z u V W S / W u / W x H C 1 Z x c 4 J + g P r 8 w f m o 5 R V < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o z 9

b n o K A m X r k l l 8 d m 1 p i U 9 c H r k 8 = " > A A A B + 3 i c b V C 7 T s M w F H X K q 5 R X K C O L R Y v E V C V d Y K x g Y S w S f a C m q h z X a a 0 6 T m T f I K o o v 8 L C A E K s / A g b f 4 P T Z o C W I 1 k 6 O u d e 3 e P j x 4 J r c J x v q 7 S x u b W 9 U 9 6 t 7 O 0 f H B 7 Z x 9 W u j h J F W Y d G I l J 9 n 2 g m u G Q d 4 C B Y P 1 a M h L 5 g P X 9 2 k / u 9 R 6 Y 0 j + Q 9 z G M 2 D M l E 8 o B T A k Y a 2 d W 6 N y W Q e i G B q R + k D 1 l W H 9 k 1 p + E s g N e J W 5 A a K t A e 2 V / e O K J J y C R Q Q b Q e u E 4 M w 5 Q o 4 F S w r O I l m s W E z s i E D Q y V J G R 6 m C 6 y Z / j c K G M c R M o 8 C X i h / t 5 I S a j 1 P P T N Z J 5 R r 3 q 5 + J 8 3 S C C 4 G q Z c x g k w S Z e H g k R g i H B e B B 5 z x S i I u S G E K m 6 y Y j o l i l A w d V V M C e 7 q l 9 d J t 9 l w n Y Z 7 1 6 y 1 r o s 6 y u g U n a E L 5 K J L 1 E K 3 q I 0 6 i K I n 9 I x e 0 Z u V W S / W u / W x H C 1 Z x c 4 J + g P r 8 w f m o 5 R V < / l a t e x i t > SGC G < l a t e x i t s h a 1 _ b a s e 6 4 = " / V t + 0 R d S S 5 A / s b V i R u a b h 3 H o i B U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k W W m I U J I E L 2 V v m Y M P e 3 m V 3 z 4 R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S A T X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q 6 z h V D F s s F r H q B F S j 4 B J b h h u B n U Q h j Q K B j 8

H 4 e u Y / P q H S P J Y P Z p K g H 9 G h 5 C F n 1 F j p v n p T 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E U r D B N W 6 6 7 m J 8 T O q D G c C p 6 V e q j G h b E y H 2 L V U 0 g i 1 n 8 1 P n Z I z q w x I G C t b 0 p C 5 + n s i o 5 H W k y i w n R E 1 I 7 3 s z c T / v G 5 q w k s / 4 z J J D U q 2 W B S m g p i Y z P 4

m A 6 6 Q G T G x h D L F 7 a 2 E j a i i z N h 0 S j Y E b / n l V d K u 1 z y 3 5 t 3 V K 4 2 r P I 4 i n M A p n I M H F 9 C A W 2 h C C x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g B S 3 Y 0 l < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " / V t + 0 R d S S 5 A / s b V i R u a b h 3 H o i B U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k W W m I U J I E L 2 V v m Y M P e 3 m V 3 z 4 R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S A T X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q 6 z h V D F s s F r H q B F S j 4 B J b h h u B n U Q h j Q K B j 8

H 4 e u Y / P q H S P J Y P Z p K g H 9 G h 5 C F n 1 F j p v n p T 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E U r D B N W 6 6 7 m J 8 T O q D G c C p 6 V e q j G h b E y H 2 L V U 0 g i 1 n 8 1 P n Z I z q w x I G C t b 0 p C 5 + n s i o 5 H W k y i w n R E 1 I 7 3 s z c T / v G 5 q w k s / 4 z J J D U q 2 W B S m g p i Y z P 4

m A 6 6 Q G T G x h D L F 7 a 2 E j a i i z N h 0 S j Y E b / n l V d K u 1 z y 3 5 t 3 V K 4 2 r P I 4 i n M A p n I M H F 9 C A W 2 h C C x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g B S 3 Y 0 l < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " / V t + 0 R d S S 5 A / s b V i R u a b h 3 H o i B U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k W W m I U J I E L 2 V v m Y M P e 3 m V 3 z 4 R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S A T X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q 6 z h V D F s s F r H q B F S j 4 B J b h h u B n U Q h j Q K B j 8 H 4 e u Y / P q H S P J Y P Z p K g H 9 G h 5 C F n 1 F j p v n p T 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E U r D B N W 6 6 7 m J 8 T O q D G c C p 6 V e q j G h b E y H 2 L V U 0 g i 1 n 8 1 P n Z I z q w x I G C t b 0 p C 5 + n s i o 5 H W k y i w n R E 1 I 7 3 s z c T / v G 5 q w k s / 4 z J J D U q 2 W B S m g p i Y z P 4 m A 6 6 Q G T G x h D L F 7 a 2 E j a i i z N h 0 S j Y E b / n l V d K u 1 z y 3 5 t 3 V K 4 2 r P I 4 i n M A p n I M H F 9 C A W 2 h C C x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g B S 3 Y 0 l < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " / V t + 0 R d S S 5 A / s b V i R u a b h 3 H o i B U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k W W m I U J I E L 2 V v m Y M P e 3 m V 3 z 4 R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S A T X x n W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q 6 z h V D F s s F r H q B F S j 4 B J b h h u B n U Q h j Q K B j 8 H 4 e u Y / P q H S P J Y P Z p K g H 9 G h 5 C F n 1 F j p v n p T 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E U r D B N W 6 6 7 m J 8 T O q D G c C p 6 V e q j G h b E y H 2 L V U 0 g i 1 n 8 1 P n Z I z q w x I G C t b 0 p C 5 + n s i o 5 H W k y i w n R E 1 I 7 3 s z c T / v G 5 q w k s / 4 z J J D U q 2 W B S m g p i Y z P 4 m A 6 6 Q G T G x h D L F 7 a 2 E j a i i z N h 0 S j Y E b / n l V d K u 1 z y 3 5 t 3 V K 4 2 r P I 4 i n M A p n I M H F 9 C A W 2 h C C x g M 4 R l e 4 c 0 R z o v z 7 n w s W g t O P n M M f + B 8 / g B S 3 Y 0 l < / l a t e x i t > X < l a t e x i t s h a 1 _ b a s e 6 4 = " M c N d i W G T h Q s 7 u 8 o H n k B R i f D 5 r u w = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y C u 4 K k k 3 u i y 6 c V n B P q A p Z T K d t E M n k z B z I 5 T Q 3 3 D j Q h G 3 / o w 7 / 8 Z J m 4 W 2 H h g 4 n H M v 9 8 w J E i k M u u 6 3 s 7 G 5 t b 2 z W 9 o r 7 x 8 c H h 1 X T k 4 7 J k 4 1 4 2 0 W y 1 j 3 A m q 4 F I q 3 U a D k v U R z G g W S d 4 P p X e 5 3 n 7 g 2 I l a P O E v 4 I K J j J U L B K F r J r / k R x U k Q Z r 1 5 b V i p u n V 3 A b J O v I J U o U B r W P n y R z F L I 6 6 Q S W p M 3 3 M T H G R U o 2 C S z 8 t + a n h C 2 Z S O e d 9 S R S N u B t k i 8 5 x c W m V E w l j b p 5 A s 1 N 8 b G Y 2 M m U W B n c w j m l U v F / / z + i m G N 4 N M q C R F r t j y U J h K g j H J C y A j o T l D O b O E M i 1 s V s I m V F O G t q a y L c F b / f I 6 6 T T q n l v 3 H h r V 5 m 1 R R w n O 4 Q K u w I N r a M I 9 t K A N D B J 4 h l d 4 c 1 L n x X l 3 P p a j G 0 6 x c w Z / 4 H z + A I j A k V Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M c N d i W G T h Q s 7 u 8 o H n k B R i f D 5 r u w = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y C u 4 K k k 3 u i y 6 c V n B P q A p Z T K d t E M n k z B z I 5 T Q 3 3 D j Q h G 3 / o w 7 / 8 Z J m 4 W 2 H h g 4 n H M v 9 8 w J E i k M u u 6 3 s 7 G 5 t b 2 z W 9 o r 7 x 8 c H h 1 X T k 4 7 J k 4 1 4 2 0 W y 1 j 3 A m q 4 F I q 3 U a D k v U R z G g W S d 4 P p X e 5 3 n 7 g 2 I l a P O E v 4 I K J j J U L B K F r J r / k R x U k Q Z r 1 5 b V i p u n V 3 A b J O v I J U o U B r W P n y R z F L I 6 6 Q S W p M 3 3 M T H G R U o 2 C S z 8 t + a n h C 2 Z S O e d 9 S R S N u B t k i 8 5 x c W m V E w l j b p 5 A s 1 N 8 b G Y 2 M m U W B n c w j m l U v F / / z + i m G N 4 N M q C R F r t j y U J h K g j H J C y A j o T l D O b O E M i 1 s V s I m V F O G t q a y L c F b / f I 6 6 T T q n l v 3 H h r V 5 m 1 R R w n O 4 Q K u w I N r a M I 9 t K A N D B J 4 h l d 4 c 1 L n x X l 3 P p a j G 0 6 x c w Z / 4 H z + A I j A k V Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M c N d i W G T h Q s 7 u 8 o H n k B R i f D 5 r u w = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y C u 4 K k k 3 u i y 6 c V n B P q A p Z T K d t E M n k z B z I 5 T Q 3 3 D j Q h G 3 / o w 7 / 8 Z J m 4 W 2 H h g 4 n H M v 9 8 w J E i k M u u 6 3 s 7 G 5 t b 2 z W 9 o r 7 x 8 c H h 1 X T k 4 7 J k 4 1 4 2 0 W y 1 j 3 A m q 4 F I q 3 U a D k v U R z G g W S d 4 P p X e 5 3 n 7 g 2 I l a P O E v 4 I K J j J U L B K F r J r / k R x U k Q Z r 1 5 b V i p u n V 3 A b J O v I J U o U B r W P n y R z F L I 6 6 Q S W p M 3 3 M T H G R U o 2 C S z 8 t + a n h C 2 Z S O e d 9 S R S N u B t k i 8 5 x c W m V E w l j b p 5 A s 1 N 8 b G Y 2 M m U W B n c w j m l U v F / / z + i m G N 4 N M q C R F r t j y U J h K g j H J C y A j o T l D O b O E M i 1 s V s I m V F O G t q a y L c F b / f I 6 6 T T q n l v 3 H h r V 5 m 1 R R w n O 4 Q K u w I N r a M I 9 t K A N D B J 4 h l d 4 c 1 L n x X l 3 P p a j G 0 6 x c w Z / 4 H z + A I j A k V Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M c N d i W G T h Q s 7 u 8 o H n k B R i f D 5 r u w = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y C u 4 K k k 3 u i y 6 c V n B P q A p Z T K d t E M n k z B z I 5 T Q 3 3 D j Q h G 3 / o w 7 / 8 Z J m 4 W 2 H h g 4 n H M v 9 8 w J E i k M u u 6 3 s 7 G 5 t b 2 z W 9 o r 7 x 8 c H h 1 X T k 4 7 J k 4 1 4 2 0 W y 1 j 3 A m q 4 F I q 3 U a D k v U R z G g W S d 4 P p X e 5 3 n 7 g 2 I l a P O E v 4 I K J j J U L B K F r J r / k R x U k Q Z r 1 5 b V i p u n V 3 A b J O v I J U o U B r W P n y R z F L I 6 6 Q S W p M 3 3 M T H G R U o 2 C S z 8 t + a n h C 2 Z S O e d 9 S R S N u B t k i 8 5 x c W m V E w l j b p 5 A s 1 N 8 b G Y 2 M m U W B n c w j m l U v F / / z + i m G N 4 N M q C R F r t j y U J h K g j H J C y A j o T l D O b O E M i 1 s V s I m V F O G t q a y L c F b / f I 6 6 T T q n l v 3 H h r V 5 m 1 R R w n O 4 Q K u w I N r a M I 9 t K A N D B J 4 h l d 4 c 1 L n x X l 3 P p a j G 0 6 x c w Z / 4 H z + A I j A k V Y = < / l a t e x i t > SGC SGC SGC SGC SGC SGC GResNet (raw) ReLU … Ŷ < l a t e x i t s h a 1 _ b a s e 6 4 = " o z 9 b n o K A m X r k l l 8 d m 1 p i U 9 c H r k 8 = " > A A A B + 3 i c b V C 7 T s M w F H X K q 5 R X K C O L R Y v E V C V d Y K x g Y S w S f a C m q h z X a a 0 6 T m T f I K o o v 8 L C A E K s / A g b f 4 P T Z o C W I 1 k 6 O u d e 3 e P j x 4 J r c J x v q 7 S x u b W 9 U 9 6 t 7 O 0 f H B 7 Z x 9 W u j h J F W Y d G I l J 9 n 2 g m u G Q d 4 C B Y P 1 a M h L 5 g P X 9 2 k / u 9 R 6 Y 0 j + Q 9 z G M 2 D M l E 8 o B T A k Y a 2 d W 6 N y W Q e i G B q R + k D 1 l W H 9 k 1 p + E s g N e J W 5 A a K t A e 2 V / e O K J J y C R Q Q b Q e u E 4 M w 5 Q o 4 F S w r O I l m s W E z s i E D Q y V J G R 6 m C 6 y Z / j c K G M c R M o 8 C X i h / t 5 I S a j 1 P P T N Z J 5 R r 3 q 5 + J 8 3 S C C 4 G q Z c x g k w S Z e H g k R g i H B e B B 5 z x S i I u S G E K m 6 y Y j o l i l A w d V V M C e 7 q l 9 d J t 9 l w n Y Z 7 1 6 y 1 r o s 6 y u g U n a E L 5 K J L 1 E K 3 q I 0 6 i K I n 9 I x e 0 Z u V W S / W u / W x H C 1 Z x c 4 J + g P r 8 w f m o 5 R V < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o z 9 b n o K A m X r k l l 8 d m 1 p i U 9 c H r k 8 = " > A A A B + 3 i c b V C 7 T s M w F H X K q 5 R X K C O L R Y v E V C V d Y K x g Y S w S f a C m q h z X a a 0 6 T m T f I K o o v 8 L C A E K s / A g b f 4 P T Z o C W I 1 k 6 O u d e 3 e P j x 4 J r c J x v q 7 S x u b W 9 U 9 6 t 7 O 0 f H B 7 Z x 9 W u j h J F W Y d G I l J 9 n 2 g m u G Q d 4 C B Y P 1 a M h L 5 g P X 9 2 k / u 9 R 6 Y 0 j + Q 9 z G M 2 D M l E 8 o B T A k Y a 2 d W 6 N y W Q e i G B q R + k D 1 l W H 9 k 1 p + E s g N e J W 5 A a K t A e 2 V / e O K J J y C R Q Q b Q e u E 4 M w 5 Q o 4 F S w r O I l m s W E z s i E D Q y V J G R 6 m C 6 y Z / j c K G M c R M o 8 C X i h / t 5 I S a j 1 P P T N Z J 5 R r 3 q 5 + J 8 3 S C C 4 G q Z c x g k w S Z e H g k R g i H B e B B 5 z x S i I u S G E K m 6 y Y j o l i l A w d V V M C e 7 q l 9 d J t 9 l w n Y Z 7 1 6 y 1 r o s 6 y u g U n a E L 5 K J L 1 E K 3 q I 0 6 i K I n 9 I x e 0 Z u V W S / W u / W x H C 1 Z x c 4 J + g P r 8 w f m o 5 R V < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o z 9 b n o K A m X r k l l 8 d m 1 p i U 9 c H r k 8 = " > A A A B + 3 i c b V C 7 T s M w F H X K q 5 R X K C O L R Y v E V C V d Y K x g Y S w S f a C m q h z X a a 0 6 T m T f I K o o v 8 L C A E K s / A g b f 4 P T Z o C W I 1 k 6 O u d e 3 e P j x 4 J r c J x v q 7 S x u b W 9 U 9 6 t 7 O 0 f H B 7 Z x 9 W u j h J F W Y d G I l J 9 n 2 g m u G Q d 4 C B Y P 1 a M h L 5 g P X 9 2 k / u 9 R 6 Y 0 j + Q 9 z G M 2 D M l E 8 o B T A k Y a 2 d W 6 N y W Q e i G B q R + k D 1 l W H 9 k 1 p + E s g N e J W 5 A a K t A e 2 V / e O K J J y C R Q Q b Q e u E 4 M w 5 Q o 4 F S w r O I l m s W E z s i E D Q y V J G R 6 m C 6 y Z / j c K G M c R M o 8 C X i h / t 5 I S a j 1 P P T N Z J 5 R r 3 q 5 + J 8 3 S C C 4 G q Z c x g k w S Z e H g k R g i H B e B B 5 z x S i I u S G E K m 6 y Y j o l i l A w d V V M C e 7 q l 9 d J t 9 l w n Y Z 7 1 6 y 1 r o s 6 y u g U n a E L 5 K J L 1 E K 3 q I 0 6 i K I n 9 I x e 0 Z u V W S / W u / W x H C 1 Z x c 4 J + g P r 8 w f m o 5 R V < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o z 9 b n o K A m X r k l l 8 d m 1 p i U 9 c H r k 8 = " > A A A B + 3 i c b V C 7 T s M w F H X K q 5 R X K C O L R Y v E V C V d Y K x g Y S w S f a C m q h z X a a 0 6 T m T f I K o o v 8 L C A E K s / A g b f 4 P T Z o C W I 1 k 6 O u d e 3 e P j x 4 J r c J x v q 7 S x u b W 9 U 9 6 t 7 O 0 f H B 7 Z x 9 W u j h J F W Y d G I l J 9 n 2 g m u G Q d 4 C B Y P 1 a M h L 5 g P X 9 2 k / u 9 R 6 Y 0 j + Q 9 z G M 2 D M l E 8 o B T A k Y a 2 d W 6 N y W Q e i G B q R + k D 1 l W H 9 k 1 p + E s g N e J W 5 A a K t A e 2 V / e O K J J y C R Q Q b Q e u E 4 M w 5 Q o 4 F S w r O I l m s W E z s i E D Q y V J G R 6 m C 6 y Z / j c K G M c R M o 8

C X i h / t 5 I S a j 1 P P T N Z J 5 R r 3 q 5 + J 8 3 S C C 4 G q Z c x g k w S Z e H g k R g i H B e B B 5 z x S i I u S G E K m 6 y Y j o l i l A w d V V M C e 7 q l 9 d J t 9 l w n Y Z 7 1 6 y 1 r o s 6 y u g U n a E L 5 K J L 1 E K 3 q I 0 6 i K I n 9 I x e 0 Z u V W S / W u / W x H C 1 Z x c 4 J + g P r 8 w f m o 5 R V < / l a t e x i t > from the learning settings of CNN, where the data instances are assumed to be independent, the nodes inside the input network studied in GCN are closely correlated. Viewed in such a perspective, new residual learning mechanism should be introduced for GCN specifically.

Here, we need to add a remark: For the presentation simplicity in this paper, given the objective H(x) = F (x) + R(x), we will misuse the terminologies here: we will name F (x) as the approximated mapping of H(x), and call R(x) as the graph residual term. Formally, by incorporating the residual learning mechanism into the GCN model, the node representation updating equations (i.e., Equation 2) can be rewritten as follows:

EQUATION

The graph residual term R H (k-1) , X; G , ∀k ∈ {1, 2, • • • , K} can be defined in different ways.

We have also examined to put R H (k-1) , X; G outside of the ReLU(•) function for the hidden layers (i.e., H (k) = ReLU ÂH (k-1) W (k-1) + R H (k-1) , X; G ), whose performance is not as good as what we show above. In the appendix of Kipf & Welling (2016) , by following the ResNet (CNN) He et al. (2015) , the graph residual term R H (k-1) , X; G is simply defined as H (k-1) , which is named as the naive residual term in this paper (Here, term "naive" has no disparaging meanings). However, according to the studies, such a simple and independent residual term for the nodes fail to capture information in the inter-connected graph learning settings.

5.2. GRESNET ARCHITECTURE:

In this paper, we introduce several other different representations of the graph residual term, which are summarized in Table 1 . If feature dimension adjustment is needed, e.g., for raw residual term, an extra variable matrix W adj can be added to redefine the terms (which are not shown in this paper). For the graph residual term representations in Table 1 , naive residual and raw residual are based on the assumption that node residuals are independent and determined by either the current state or the raw features. Meanwhile, the graph naive residual and graph raw residual assume the residual terms of different nodes are correlated instead, which can be computed with the current state or raw features. We also illustrate the architectures of vanilla 2-layer GCN and the 7-layer GRESNETs (taking GCN as the base model) with different graph residual terms in Figure 2 . We Vanilla GCN: For the vanilla GCN network used in Kipf & Welling (2016) , i.e., the left plot of Figure 2 , given the inputs G and X, it employs two SGC layers to project the input to the objective labels. For the intermediate representations, the hidden layer length is 16, and ReLU is used as the activation function, whereas softmax is used for output label normalization.

GRESNET Network: For the GRESNET network, i.e., the right four models in Figure 2 , they accept the identical inputs as vanilla GCN but will create graph residual terms to be added to the intermediate representations. Depending on the specific residual term representations adopted, the corresponding high-way connections can be different. For the hidden layers involved in the models, their length is also 16, and ReLU is used as the activation function for the intermediate layers.

By comparing the output Ŷ learned by the models against the ground-truth Y of the training instances, all the variables involved in the model, i.e., Θ, can be effectively learned with the backpropagation algorithm to minimize the loss functions (Y, Ŷ; Θ) (or (Θ) for simplicity). In the following part, we will demonstrate that for the GRESNET model added with graph residual terms. It can effectively avoid dramatic changes to the nodes' representations between sequential layers.

5.3. GRAPH RESIDUAL LEARNING EFFECTIVENESS ANALYSIS:

In this part, we will illustrate why the inclusion of the graph residual learning can be effective for learning deep graph neural networks. Here, we assume the ultimate model that we want to learn as H : X → Y, where X and Y denote the feature and label spaces, respectively. For analysis simplicity, we have some assumptions about the function H Zaeemzadeh et al. (2018) . ASSUMPTION 1. Function H is differentiable, invertible and satisfies the following conditions:

• ∀x, y, z ∈ X with bounded norm, ∃α > 0, (H (x) -

H (y))z ≤ α • x -y • z , • ∀x, y ∈ X with bounded norm, ∃β > 0, H -1 (x) -H -1 (y) ≤ β • x -y ,

• ∃x ∈ X with bounded norm such that Det(H (x)) > 0. In the above conditions, terms α and β are constants.

To model the function H, the GRESNET actually defines a sequence of K sequential mappings with these K layers:

EQUATION

where x (k-1) and x (k) denote the intermediate node representations serving as input and output of the k th layer. F (k-1) (•) and R (k-1) (•) denote the function approximation and residual term 

EQUATION

)

where δ ≤ c • log(2K) K and c = c 1 • max{α • β • (1 + β), β • (2 + α) + α} for some c 1 > 0.

Proof of Theorem 3 will be provided in the appendix. The above theorem indicates that in the learning process, the norm of loss function against the intermediate representations doesn't change significantly between sequential layers. In other words, GRESNET can maintain effective representations for the inputs and overcome the suspended animation problem. In addition, we observe that the bound of the gap term δ, i.e., c • log(2K) K , decreases as K increases (when K ≥ 2). Therefore, for deeper GRESNET, the model will lead to much tighter gradient norm changes, which is a desired property. In the following section, we will provide the experimental results of GRESNET compared against their vanilla models on several graph benchmark datasets.

6. EXPERIMENTS:

To demonstrate the effectiveness of GRESNET in improving the learning performance for graph neural networks with deep architectures, extensive experiments will be done on several graph benchmark datasets. Similar to the previous works on node classification Kipf & Welling (2016) ; Veličković et al. (2018) , the graph benchmark datasets used in the experiments include Cora, Citeseer and Pubmed from Sen et al. (2008) . For fair comparison, we follow exactly the same experimental settings as Kipf & Welling (2016) on these datasets.

In this paper, we aim at studying the suspended animation problem with the existing graph neural networks, e.g., GCN Kipf & Welling (2016) , GAT Veličković et al. (2018) and LOOPYNET Zhang (2018) , where LOOPYNET is not based on the spectral graph convolutional operator. We also aim to investigate the effectiveness of these proposed graph residual terms in improving their learning performance, especially for the models with deep architectures. In addition, to make the experiments self-contained, we also provide the latest performance of the other baseline methods on the same datasets in this paper, which include state-of-the-art graph neural networks, e.g., APPNP Klicpera et al. (2019) , GOCN Jiang et al. (2019) and GraphNAS Gao et al. (2019) , existing graph embedding models, like DeepWalk Perozzi et al. (2014) , Planetoid Yang et al. (2016) and MoNet Monti et al. (2016) and representation learning approaches, like ManiReg Belkin et al. (2006 ), SemiEmb Weston et al. (2008) , LP Zhu et al. (2003) and ICA Lu & Getoor (2003) . As the model depth increase, performance of LOOPYNET remains very close but converge much more slowly. By taking GCN as the base model, we also show the performance of GRESNET with different residual terms in Figure 3 . By comparing these plots with Figure 1 , both naive and graphnaive residual terms help stabilize the performance of deep GRESNET(GCN)s. Meanwhile, for the raw and graph-raw residual terms, their contributions are exceptional. With these two residual terms, deep GRESNET(GCN)s can achieve even better performance than the shallow vanilla GCNs.

Besides the results on the Cora dataset, in Table 2 , we illustrate the best observed performance by GRESNET with different residual terms based on GCN, GAT and LOOPYNET base models respectively on all the datasets. Both the best accuracy score and the achieved model depth are provided. According to the results, for the vanilla models, GCN, GAT and LOOPYNET can all obtain the best performance with shallow architectures. For instance on Cora, GCN(2-layer) obtains 0.815; GAT(2-layer) gets 0.830; and LOOPYNET(2-layer) achieves 0.826, respectively. Added with the residual terms, the performance of all these models will get improved. In addition, deep GRESNET(GCN), GRESNET(GAT) and GRESNET(LOOPYNET) will be able to achieve much better results than the shallow vanilla models, especially the ones with the graph-raw residual terms.

The best scores and the model depth for each base model on these datasets are also highlighted. The time costs of learning the GRESNET model is almost identical to the required time costs of learning the vanilla models with the same depth, which are not reported in this paper.

6.2. A COMPLETE COMPARISON WITH EXISTING NODE CLASSIFICATION METHODS:

Besides the comparison with GCN, GAT and LOOPYNET shown in Table 2 , to make the experimental studies more complete, we also compare GRESNET(GCN), GRESNET(GAT) and GRES-NET(LOOPYNET) with both the classic and the state-of-the-art models, whose results are provided in Table 3 . In the table, we didn't indicate the depth of the GRESNET models and results of GCN, GAT and LOOPYNET (shown in Table 2 already) are not included. According to the results, compared against these baseline methods, GRESNETs can also outperform them with great advantages. Without the complex model architecture extension or optimization techniques used by the latest methods APPNP Klicpera et al. (2019) , GOCN Jiang et al. (2019) and GraphNAS Gao et al. (2019) , adding the simple graph residual terms into the base models along can already improve the learning performance greatly.

7. CONCLUSION:

In this paper, we focus on studying the suspended animation problem with the existing graph neural network models, especially the spectral graph convolutional operator. We provide a theoretic analysis about the causes of the suspended animation problem and derive the bound for the maximum allowed graph neural network depth, i.e., the suspended animation limit. To resolve such a problem, we introduce a novel framework GRESNET, which works well for learning deep representations from the graph data. Assisted with these new graph residual terms, we demonstrate that GRESNET can effectively resolve the suspended animation problem with both theoretic analysis and empirical experiments on several benchmark node classification datasets.

8.2. EXPERIMENTAL RESULTS ON DEEPER MODELS:



8.2.1. DEEPER MODEL PERFORMANCE SUMMARY:

As suggested by the reviewers, we also add the studies of GCN, GRESNET(GCN, naive), GRES-NET(GCN, graph-naive), GRESNET(GCN, raw) and GRESNET(GCN, graph-raw) with deeper architectures (more than 7 layers) on the Cora dataset in the following table and plots

As illustrated in Table 7 , we show the experimental results on Cora obtained by the models with 2, 10, 20, 30, 40, 50 layers, respectively. According to the numbers, GCN cannot really work well for all the layers other than 2, and the performance of the naive and graph-naive residual terms is not good neither. However, the performance of the raw and graph-raw residual terms is exceptionally great. For instance, for the model with 40 layers, GRESNET(GCN, raw) and GRESNET(GCN, graph-raw) can still achieve 0.790 and 0.838 as the accuracy scores. In addition to Table 7 , we also illustrate the performance of the models in the learning process in Figures 4 5 6 7 8 , where both the training accuracy and testing accuracy on each epoch are provided. According to the results, GCN and GRESNET(GCN, graph-naive) will fail to work when the model architecture goes deeper. GRESNET(GCN, naive) can slightly revive the models for 10 layers, but will also fail to work for deeper architectures, e.g., 20, 30, 40 and 50 layers. However, GRES-NET(GCN, raw) and GRESNET(GCN, graph-raw) work very great for all these different depth in the plot, which also illustrate the effectiveness of the raw and graph-raw residual terms proposed in this paper. LEMMA 1. Given an irreducible, finite and aperiodic graph G, starting from any initial distribution vector x ∈ R n×1 (x ≥ 0 and x 1 = 1), the Markov chain operating on the graph has one unique stationary distribution vector π * such that lim t→∞ Ât x = π * , where π * (i) = d(vi) 2|E| . If matrix Â is symmetric (i.e., G is undirected), π * will be a uniform distribution over the nodes, i.e., π * (i) = 1 n .

Proof. The stationary distribution vector existence and uniqueness has been proved in Norris (1998) .

Here, we need to identify on vector π at convergence such that Âπ = π, i.e., π(i) = j Â(i, j)π(j).

(16)

According to the definition of Â, it is easy to have j w i,j d w (j) π(j) = π(i).

where w i,j denotes the initial connection weight between v i and v j . For the unweighted network, w i,j will be in {0, 1} indicating if v i and v j are connected or not. Notation d w (i) denotes the rough degree of node v i in the network subject to the weight w : E ∈ R, which sums the weight of edges connected to the nodes in the network. So, it is enough to have π(j) ∝ d w (j). More precisely, we can set

EQUATION

In this case, 

Meanwhile, for the symmetric and normalized adjacency matrix Â, we can prove the stationary distribution π * (i) = 1 n in a similar way, which concludes the proof. 

which together with Lemma 1 conclude the proof.

10.2.3. PROOF OF THEOREM 2:

Prior to introducing the proof of Theorem 2, we will introduce the following lemma first.

LEMMA 2. For any vector x ∈ R n , the following inequality holds: 

EQUATION

(|x i | p ) q p p q n i=1 1 q q-p 1-p q = n i=1 |x i | q p q (n) 1-p q .

(24) Therefore,

x p = n i=1 |x(i)| p 1 p ≤   n i=1 |x i | q p q (n) 1-p q   1 p = n i=1 |x i | q 1 q (n) 1 p -1 q = (n) 1 p -1 q x q .

(25) THEOREM 2. Let 1 ≥ λ 1 ≥ λ 2 ≥ • • • ≥ λ n be the eigen-values of matrix Â defined based on network G, then the corresponding suspended animation limit of the GCN model on G is tightly bounded

EQUATION

)

In the case that the network G is a d-regular, then the suspended animation limit of the GCN model on G can be simplified as

EQUATION

)

Proof. Instead of proving the above inequality directly, we propose to prove that ζ is suspended animation limit by the following inequality instead

EQUATION

which can derive the following inequality according to Lemma 2:

EQUATION

)

Let v 1 , v 2 , • • • , v n be the eigenvectors of Â and ∀x

EQUATION

where α i = x, v i .

Therefore, we have

EQUATION

Considering that λ ζ 1 = 1 and

EQUATION

and

EQUATION

where Therefore, we have 

EQUATION

where σ max (•) and σ min (•) denote the maximum and minimum singular values of the input matrix, respectively.

Proof. Due to the triangle inequality, the upper bound is easy to prove:

EQUATION

In the case that σ max (M) ≥ 1, the lower bound is trivial to prove since I + M is non-singular, we have σ min (I + M) > 0.

(42)

Meanwhile, in the case that σ max (M) < 1, it is easy to know that |λ max (M)| < 1, where λ max (•) denotes the latest eigenvalue of the input matrix.

EQUATION

which concludes the proof for the lower bound. THEOREM 3. Let H denote the objective function that we want to model, which satisfies Assumption 1, in learning the K-layer GRESNET model, we have the following inequality hold:

EQUATION

)

where δ ≤ c log(2K) K and c = c 1 max{αβ(1 + β), β(2 + α) + α} for some c 1 > 0.

Proof. We can represent the Jacobian matrix J of x (k) with x (k-1) . Therefore, we have

EQUATION

)

Matrix J can be rewritten as J = I + ∇F (k-1) (x (k-1) ), where ∇F (k-1) (x (k-1) ) = lim t→0 + F (k-1) (x (k-1) + tv) -F (k-1) (x (k-1) ) t

EQUATION

)

Meanwhile, it is easy to know that σ min (J)

EQUATION

)

Based on the above lemma, we have

EQUATION

where σ = σ max (∇F (k-1) (x (k-1) )).

Furthermore, we know that σ max (∇F (k-1) (x (k-1) ) = sup v ∇F (k-1) (x (k-1) )v 2 v 2 = lim t→0 + sup v F (k-1) (x (k-1) + tv) -F (k-1) (x (k-1)

EQUATION

where • L denotes the Lipschitz seminorm of the input function and it is defined as

EQUATION

)

Meanwhile, according to the Theorem 1 in Bartlett et al. (2018) (whose proof will not be introduced here), we know that

EQUATION

which concludes the proof. In the above equation, K denotes the layer depth of the model.

:

https://paperswithcode.com/task/node-classification

https://github.com/anonymous-sourcecode/GResNet

GPU Server: ASUS X99-E WS motherboard, Intel Core i7 CPU 6850K@3.6GHz (6 cores), 3 Nvidia GeForce GTX 1080 Ti GPU (11 GB buffer each), 128 GB DDR4 memory and 128 GB SSD swap. For the deep models which cannot fit in the GPU memory, we run them with CPU instead.

Figures and Tables:

Figure 1: The learning performance of GCN (bias disabled) with 1-layer, 2-layer, . . . , 7-layer on the Cora dataset. The x axis denotes the iterations over the whole training set. The y axis of the left plot denotes the training accuracy, and that of the right plot denotes the testing accuracy.

Figure 2: A comparison of vanilla GCN and GRESNET (GCN) with different graph residual terms. The vanilla GCN has two layers, and the GRESNET models have a deep architecture which involves seven layers of SGC operators. The intermediate ReLU activation functions used between sequential layers in GRESNET are omitted for simplicity. Notation Â denotes the normalized adjacency matrix of input network G, which indicates the correlated graph residual terms.

Figure 3: The learning performance of GRESNET with GCN as the base model and different graph residual terms on the Cora dataset: (a)-(b) GRESNET(GCN, naive); (c)-(d) GRESNET(GCN, graph-naive); (e)-(f) GRESNET(GCN, raw); (g)-(h) GRESNET(GCN, graph-raw). For plot (b), the curves corresponding to the 5-layer, 6-layer and 7-layer models are hidden by the legend. For plot (c) and (d), the curve of 7-layer model is hidden by the legend.need to add a remark here that similar solutions to the raw residual terms have been studied in the latest research papersWang et al. (2018);Wang & Gupta (2018) as well. However, in this paper, we aim to generalize such techniques as a unified framework, which can accept different types of residual terms not just the raw residual term.

(a) Training Accuracy (b) Testing Accuracy

Figure 4: Deeper GCN on Cora.

Figure 5: Naive Residual: deeper GRESNET(GCN, naive) on Cora.

Figure 6: Graph-Naive Residual: deeper GRESNET(GCN, graph-naive) on Cora.

Figure 7: Raw Residual: deeper GRESNET(GCN, raw) on Cora.

Figure 8: Graph-Raw Residual: deeper GRESNET(GCN, graph-raw) on Cora.

Figure 9: The learning performance of GCN (bias enabled) on the Cora dataset.

Figure 10: The learning performance of GAT on the Cora dataset

10.2.2 PROOF OF THEOREM 1THEOREM 1. Given a input network G = (V, E), which is unweighted, irreducible, finite and aperiodic, if there exist enough nested Markov chain layers in the GCN model, it will reduce the nodes' representations from the column-normalized feature matrix X ∈ R n×dx to the stationary representationΠ * = [π * , π * , • • • , π * ] ∈ R n×dx . Furthermore, if G is undirected, then the stationary representation will become Π * = 1 n • 1 n×dx .Proof. This theorem can be proved based on Lemma 1. For any initial state distribution vectorx ∈ R |V|×1 , for the Markov chain at convergence, we have lim t→∞ Ât x = π * . (20) We misuse the notation Â * = lim t→∞ Ât . In this case, Â * X = [ Â * X(:, 1), Â * X(:, 2), • • • , Â * X(:, d x )] = [π * , π * , • • • , π * ],

Proof. According to Hölder's inequalityHölder (1889), for ∀a, b ∈ R n×1 and r > 1, |a(i)| = |x i | p , |b(i)| = 1 and r = q p ,

whereλ max = max{|λ 2 |, |λ 3 |, • • • , |λ n |} = max{λ 2 , |λ n |}.Therefore, to ensuremax{λ 2 , |λ n |} ≤ Let 1 ≥ λ 1 ≥ λ 2 ≥ • • • ≥ λ n bethe eigen-values of matrix Â defined based on network G, then the corresponding suspended animation limit of the GCN model (with lazy Markov chain based layers) on G is tightly bounded For the lazy Markov chain layer, we have its updating equation as follows diag({d w (i)} vi∈V ))H (k-1) = ÃH(k-1) .(38)It is easy to show that Ã is positive definite and we have its eigen-valuesλ 1 ≥ λ 2 ≥ • • • ≥ λ n ≥ 0. Therefore, λ max = max{|λ 2 |, |λ 3 |, • • • , |λ n |} = max{λ 2 , |λ n |} = λ 2 ,(39)which together with Theorem 2 conclude the proof.10.3 PROOF OF THEOREM 3Prior to introducing the proof of Theorem 3, we first introduce the following lemma. LEMMA 3. For any non-singular matrix I + M, we have 1 -σ max (M) ≤ σ min (I + M) ≤ σ max (I + M) ≤ 1 + σ max (M),

A Summary of Graph Residual Terms and Physical Meanings.

Table content: <table><tr><td>Name</td><td>Residual Term</td><td>Description</td></tr><tr><td>naive resid-</td><td/><td/></tr></table>

Best performance (accuracy) and model depth summarization of GRESNET with different residual terms on the benchmark datasets (we take GCN, GAT and LOOPYNET as the base models). -1 th layer of the model. When training these K sequential mappings, we have the following theorem hold for the representation gradients in the learning process. THEOREM 3. Let H denote the objective function that we want to model, which satisfies Assumption 1, in learning the K-layer GRESNET model, we have the following inequality hold:

Table content: <table><tr><td/><td>Methods</td><td colspan="3">Datasets (Accuracy &amp; Model Depth)</td></tr><tr><td colspan="2">Base Models Residuals</td><td>Cora</td><td>Citeseer</td><td>Pubmed</td></tr><tr><td colspan="3">vanilla GCN (Kipf &amp; Welling (2016)) 0.815 2-layer</td><td>0.703 2-layer</td><td>0.790 2-layer</td></tr><tr><td/><td>naive</td><td>0.814 3-layer</td><td>0.710 3-layer</td><td>0.814 3-layer</td></tr><tr><td>GCN</td><td>graph-naive raw</td><td>0.833 2-layer 0.826 4-layer</td><td>0.715 3-layer 0.727 4-layer</td><td>0.811 2-layer 0.810 3-layer</td></tr><tr><td/><td>graph-raw</td><td>0.843 5-layer</td><td>0.722 4-layer</td><td>0.817 7-layer</td></tr><tr><td colspan="3">vanilla GAT (Veličković et al. (2018)) 0.830 2-layer</td><td>0.725 2-layer</td><td>0.790 2-layer</td></tr><tr><td/><td>naive</td><td>0.844 5-layer</td><td>0.735 5-layer</td><td>0.809 3-layer</td></tr><tr><td>GAT</td><td>graph-naive raw</td><td>0.855 3-layer 0.842 3-layer</td><td>0.732 4-layer 0.733 3-layer</td><td>0.815 5-layer 0.814 4-layer</td></tr><tr><td/><td>graph-raw</td><td>0.847 3-layer</td><td>0.729 5-layer</td><td>0.822 4-layer</td></tr><tr><td colspan="3">vanilla LOOPYNET (Zhang (2018)) 0.826 2-layer</td><td>0.716 2-layer</td><td>0.812 2-layer</td></tr><tr><td/><td>naive</td><td>0.833 2-layer</td><td>0.728 3-layer</td><td>0.830 4-layer</td></tr><tr><td>LOOPYNET</td><td>graph-naive raw</td><td>0.832 2-layer 0.836 2-layer</td><td>0.728 3-layer 0.730 5-layer</td><td>0.819 2-layer 0.828 4-layer</td></tr><tr><td/><td>graph-raw</td><td>0.839 4-layer</td><td>0.737 5-layer</td><td>0.814 4-layer</td></tr><tr><td>learned by the k</td><td/><td/><td/><td/></tr></table>

Learning result accuracy of node classification methods. In the table, '-' denotes the results of the methods on these datasets are not reported in the existing works. Performance of GCN, GAT and LOOPYNET shown in Table2are not provided here to avoid reporting duplicated results. Both the datasets and source code used in this paper can be accessed via link 2 . Detailed information about the server used to run the model can be found at the footnote 3 .6.1 EFFECTIVENESS OF THE GRAPH RESIDUAL TERMSIn addition to Figure1for GCN (bias disabled) on the Cora dataset, as shown in Figures9-11in the appendix, for the GCN (bias enabled) and GAT with deep architectures, we have observed similar suspended animation problems. Meanwhile, the performance of LOOPYNET is different. Since LOOPYNET is not based on the spectral graph convolution operator, which accepts nodes' raw features in all the layer (it is quite similar to the raw residual term introduced in this paper).

Table content: <table><tr><td>Methods</td><td colspan="3">Datasets (Accuracy) Cora Citeseer Pubmed</td></tr><tr><td>LP (Zhu et al. (2003))</td><td>0.680</td><td>0.453</td><td>0.630</td></tr><tr><td>ICA (Lu &amp; Getoor (2003))</td><td>0.751</td><td>0.691</td><td>0.739</td></tr><tr><td>ManiReg (Belkin et al. (2006))</td><td>0.595</td><td>0.601</td><td>0.707</td></tr><tr><td>SemiEmb (Weston et al. (2008))</td><td>0.590</td><td>0.596</td><td>0.711</td></tr><tr><td colspan="2">DeepWalk (Perozzi et al. (2014)) 0.672</td><td>0.432</td><td>0.653</td></tr><tr><td>Planetoid (Yang et al. (2016))</td><td>0.757</td><td>0.647</td><td>0.772</td></tr><tr><td>MoNet (Monti et al. (2016))</td><td>0.817</td><td>-</td><td>0.788</td></tr><tr><td>APPNP (Klicpera et al. (2019))</td><td>0.851</td><td>0.757</td><td>0.797</td></tr><tr><td>GOCN (Jiang et al. (2019))</td><td>0.848</td><td>0.718</td><td>0.797</td></tr><tr><td>GraphNAS (Gao et al. (2019))</td><td>-</td><td>0.731</td><td>0.769</td></tr><tr><td>GRESNET(GCN)</td><td>0.843</td><td>0.727</td><td>0.817</td></tr><tr><td>GRESNET(GAT)</td><td>0.855</td><td>0.735</td><td>0.822</td></tr><tr><td>GRESNET(LOOPYNET)</td><td>0.839</td><td>0.737</td><td>0.830</td></tr><tr><td>Reproducibility:</td><td/><td/><td/></tr></table>

Learning result by models with deeper architectures.

Table content: <table><tr><td>Methods</td><td/><td colspan="4">Model Depth (Accuracy)</td></tr><tr><td/><td>2</td><td>10</td><td>20</td><td>30</td><td>40</td><td>50</td></tr><tr><td>GCN</td><td colspan="6">0.815 0.174 0.072 0.072 0.072 0.072</td></tr><tr><td>GRESNET(GCN, naive)</td><td colspan="6">0.804 0.416 0.126 0.057 0.057 0.057</td></tr><tr><td colspan="7">GRESNET(GCN, graph-naive) 0.833 0.169 0.057 0.057 0.057 0.057</td></tr><tr><td>GRESNET(GCN, raw)</td><td colspan="6">0.815 0.814 0.802 0.812 0.798 0.800</td></tr><tr><td>GRESNET(GCN, graph-raw)</td><td colspan="6">0.813 0.823 0.823 0.818 0.838 0.799</td></tr><tr><td colspan="7">8.2.2 DEEPER MODEL LEARNING LOSS ON TRAINING AND TESTING SETS PER EPOCH</td></tr></table>

Peer review (iclr2020):
Summary: This paper studies the “suspended animation limit” of various GNNs – an important one for how to train a good Graph network. The authors provide sufficient analysis by simplify GNNs as a series of 1-step Markov chains (which is my concern as stated in the section on main issues), while pointing out the limitations quantitatively as in the Theorem 2. Under the assumption, the authors propose several new forms of ResNets for GCNs, which can successfully overcome the limitation. Overall, the motivation of this work is clear and meaninfgful. The proposed residual architecture is effective, and the presentation is clear and easy to understand. However, my main concerns are on the initial assumptions for analyzing the suspension of GNNs. See the following comments. This paper is generally well written and easy to understand. The organization of each part is well-balanced. Originality: To the best of my knowledge, numerous methods (i.e., targeting on applications) address this problem by augmenting A [1] or X [2] with similarity of feature representation learned from other sources. However, this paper specifically analyzes the problem in a principle way. I consider this work is generally novel. [1] X. Wang and A. Gupta. Videos as Space-Time Region Graphs. ECCV 2018. [2] N. Wang, Y. Zhang, Z. Li , Y. Fu, W, Liu, Y. Jiang. Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images. ECCV 2018. Significance: The significance lies mostly in motivation and the proposed GResNet. Motivation: This paper studies the phenomenal that GNNs tends to not respond to the input data when certain depth of a network is reached, which the authors called as suspended animation limit. Studying problem is fundamental and important, and also unique since different with CNNs where data are independent, the data instance within GNNs are highly correlated. GResNet: Given the differences, and within the Theorem 2 where the residual formulation of CNNs does not apply to GNNs, the paper also proposes several new formulations, i.e., in figure 2, where the suspension is avoidable and the performance under the same experimental settings is obviously boosted. Main issues: My major concern to this work lies in the assumption used throughout section 3 and 4. At the beginning of Section 3.2, the authors assume that W is identity. Since X is assumed to be column-wise normalized, the nonlinearity is removable. However, this is not true in real cases: W is actually learnable and not bounded. When W is learned to be negative, Relu layer is not removable, and the behavior of the network will be completely different with what the paper depicted. Indeed, GNNs contain stacked linear+nonlinear functions, which cannot be simplified as a linear Markov chain. It is analogy to CNNs, which is not possibly be simplified as a group of average poolings. Minor issues: 1. I agree that under the assumption, eq. (11) shows that the differences between the learned representations are not discriminative, however the claim “majority of the nodes are of very small degrees” is not justified and only apply to the internet topology in Faloutsos et al. (1999). 2. I feel the “Raw Feature Coding” and the “Network Degree Distribution” are sort of repetitive, and the eq. (11) is eq. (12) at the stationary point.