Local Augmentation for Graph Neural Networks
Authors: Songtao Liu
Hanze Dong
Lanqing Li
Tingyang Xu
Yu Rong
Peilin Zhao
Junzhou Huang
Dinghao Wu
Venue: ICLR 2022 Submitted (2022)
Keywords: Local Augmentation
Graph Neural Networks

Abstract: Data augmentation has been widely used in image data and linguistic data but remains under-explored on graph-structured data. Existing methods focus on augmenting the graph data from a global perspective and largely fall into two genres: structural manipulation and adversarial training with feature noise injection. However, the structural manipulation approach suffers information loss issues while the adversarial training approach may downgrade the feature quality by injecting noise. In this work, we introduce the local augmentation, which enhances node features by its local subgraph structures. Specifically, we model the data argumentation as a feature generation process. Given the central node's feature, our local augmentation approach learns the conditional distribution of its neighbors' features and generates the neighbors' optimal feature to boost the performance of downstream tasks. Based on the local augmentation, we further design a novel framework: LA-GNN, which can apply to any GNN models in a plug-and-play manner. Extensive experiments and analyses show that local augmentation consistently yields performance improvement for various GNN architectures across a diverse set of benchmarks.

Under review as a conference paper at ICLR 2022
LOCAL AUGMENTATION FOR GRAPH NEURAL NET- WORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Data augmentation has been widely used in image data and linguistic data but remains under-explored for Graph Neural Networks (GNNs). Existing methods focus on augmenting the graph data from global perspective and largely fall into two genres: structural manipulation and adversarial training with feature noise injection. However, recent graph data augmentation methods ignore the importance of local information for the GNNs message passing mechanism. In this work, we introduce the local augmentation, which enhances the locality node representations by their subgraph structures. Specifically, we model the data augmentation as a feature generation process. Given a node's features, our local augmentation approach learns the conditional distribution of its neighbors' features and generates more neighbors' features to boost the performance of downstream tasks. Based on the local augmentation, we further design a novel framework: LA-GNN, which can apply to any GNN models in a plug-and-play manner. Ex- tensive experiments and analyses show that local augmentation consistently yields performance improvement for various GNN architectures across a diverse set of benchmarks.
INTRODUCTION
Graph Neural Networks (GNNs) and their variants (Abu-El-Haija al. 2019 Kipf & Welling 2017
Veličković et al. 2018) have achieved state-of-the-art performance for many tasks on graphs such as recommendation system (Ying et al. (2018) and traffic prediction (Guo et al. 2019). However, most of the GNN models, such as GCN (Kipf & Welling (017)) and GAT (Velččković et al. 2018), learn the node representations by aggregating information over only the 2-hop neighborhood. Such shallow architectures limit their ability to extract information from higher-layer neighborhoods (Wang & Dem 202T). But deep GNNs are prone over-smoothing (Li et al.) (018), which suggests the node representations tend to converge to certain vector and thus become indistinguishable. One solution to address this problem to preserve the locality of node representations when increasing the number of layers. For example, JKNet (Xu et al. 2018)) 8) densely connects (Huang al. 2017) each hidden layer to the final layer. GCNII (Chen et (020)) employs an initial residual to construct a skip connection from the input layer. Besides, ZZeg al. (2021) pointed out that the key for GNN is to smooth the local neighborhood into informative representation, no matter how deep is. And they decouple the depth and scope of GNNs help capture local graph structure. Prior works have emphasized the importance of local information, but one property of the graph is that the number of nodes in the local neighborhood is far fewer than higher-order neighbors. And this property limits the expressive power of GNNs due to the limited neighbors in the local structure. A very intuitive idea is use data augmentation increase the number of nodes in the local substructure.
However, existing graph data augmentation methods ignore the importance of local information and only perturb the topology-level and feature-level from a global perspective, which can be divided into two categories: topology-level augmentation (Rong et al. 2020. Wang et al. 2020b Zhao cet al. 202[) and feature-level augmentation (Deng et al. 2019 Feng et al. 2019 Kong et al. 2020). Topology-level augmentation perturbs the adjacency matrix, yielding different graph structures. On the other hand, existing feature-leve augmentation mainly exploits perturbation of node attributes guided by adversarial training (Deng et al. 2019 Feng et al. 2019 Kong et al., 2020). These augmentation techniques have two drawbacks. I) Some of they employ full-batee batch training for augmentation, which is computationally expensive, and introduce some additional side effects such

---

Under review as a conference paper at ICLR 2022
The type of feature-level augmentation is roarse-grained which focuses global augmentation and overlooks the local information of the neighborhood. Moreover, to our best knowledge, none of the existing approaches combines both the feature representations and the graph topology, especially the local subgraph structures, for graph-level data augmentation.
over-smoothing. 2)
In this work, we propose a framework: Local Augmentation for Graph Neural Networks (LA-GNNs), to further enhance the locality of node representations based on both the topology-level and feature- level information in the substructure. 'hee term "local augmentation" refers the generation of neighborhood features via a generative model conditioned on local structures and node features. Specifically, our proposed framework learns the conditional distribution of the connected neighbors' representations given the representation of the central node, bearing some similarities with the Skip- gram (Mikolov et al., 2013) and Deepwalk Perozzi et al. (0114), with the difference that our method does not base on word or graph embedding.
The motivation behind this work concludes three-fold. 1) Existing feature-level augmentation works primarily pay attention global augmentation without considering the informative neighborhood. 2) The distributions of the representations of the neighbors are closely connected to the central node, making ample room for feature augmentation. 3) Preserving the locality of node representations is key to avoiding over-smoothing (Xu et al. 2018 Kliepera et al., 2019 Chen et al. (020). And there several benefits in applying local augmentation for the GNN training. First, local augmentation is essentially a data augmentation technique that can improve the generalization of the GNN models and prevent over-fitting Second, we can recover some missing contextual information of the local neighborhood in an attributed graph via the generative model (Jia & Benson (020). Third, our proposed framework is flexible and can be applied to various popular backbone networks such as GCN (Kipf & Welling 2017), GAT (Veličković et al., (018)) GCNII (Chen et al.) 2020), and GRAND (Feng al. 2020) to enhance their performance. Extensive experimental results demonstrate that our proposed framework could improve the performance of GNN variants on 7 benchmark datasets.
BACKGROUND
Notations. Let G (V. E) represent the graph, where V is the set of vertices {vi,... with V N and E the set of edges. The adjacency matrix is defined as A E (0, 13 M and Au 1 if and only if u,), E E. Let N₁, (uylAn 13 denotes the neighborhood of node v, and D denote the diagonal degree matrix, where D = Σ Aij. The feature matrix is denoted X E R.NF where each node is associated with F-dimensional feature vector X. Y E {0, 10,1) denote the one-hot label matrix, where Y, E {0,1)C is a one-hot vector and 2j=1 Y = for any U; V.
)
GNN. Graph Neural Network (GNN) is a type of neural network that directly operates on the graph structure, such GCN and GAT (Kipf & Welling 2017] Veličković al. 2018), that capture the dependence of graphs via message passing between the nodes of a graph as
(1) where f denotes the specific GNN layer for different models, ( are the hidden vectors of the L-th layer and (0) x. For example, [(A, H) = o (AWw) for GCN, where Á = = D is the degree matrix of Á, i.c., D = E, Aig, and A = A=A+1 11
O = f(A, H((,))))))
Topology-level Augmentation. Topology- levell augmentation usually perturbs A to generate dif- ferent graph structures, which can be formulated as A' F(A.X), where F(.) is structure perturbation function. For example, DropEdge (Rong et al. 2020) considers F(A, = A - A, which is independent of X, where Ag is sparse matrix consists of subset of the original edges E. GAUG-O (Zhao et al. 2021 leverages their proposed neural edge predictors to produce a different
structure A' where A/
I++ Pa) + 1 Psj aM,, + (1 A) M a (ZZ"). distribution, Gumbel 1) is a
= f (A, (A, X)), , 7 is the temperature of Gumbel random variate, and is hyperparameter mediating the influence of edge predictor on the original graph.
2

---

Under review as a conference paper at ICLR 2022
Feature-level Augmentation.
Table I: Comparison of existing graph data augmentation.
Besides, feature-level augmenta- tion function be defines
Graph Dala Augmentation
Considered
Type
Perturhed
X = H(A, X), where H(-) is
DropEdge
A
Sampling Reconstruction
A
feature perturbation function. FLAG (Kong et al., 2020) de- fines the perturbation function as H(A,
GAUG-O
A&X
FLAG
X
Noise Injection
X
G-GCN
A&X
Reconstruction
X
Augmentation
Generation
X + ö where perturbation is updated iteratively during the adversarial training phase. G-GCN (plain) Zhu
al. 2020) obtains the global attribute feature matrix X RRNxd through minimizing the
where U the set of all attributes, CA(e) is the exp(x V.)
objective Ivvv ILEEC(()) Enev exp( x! .v.)
sampled context attributes of v, and V E Rd, denotes the parameters. Obviously, the perturbation function of G-GCN has no close-form solution. In this work, we propose a novel feature-level augmentation method, named local augmentation. And the comparison of the details of various graph data augmentation techniques can be found in Table
LOCAL AUGMENTATION
In this section, describe details of the proposed method. The local augmentation framework consists of three modules: learning the conditional distribution via a generative model, the active learning trick, and the downstream GNN models, illustrated in Figure Note that the proposed algorithm enhances the locality of node representations through augmenting 1-hop neighbors in generative way. Specifically, we exploit a generative model to learn the conditional distribution of the connected neighbors' representations given the representation of node. We describe the details of learning the conditional distribution and the motivation for why local augmentation is able to improve the performance in probabilistic view in Sec. Sec.3.1 detail the architecture of downstream GNN models in Sec. 3.2 We finally elaborate the training procedure of both the generative model and the downstream GNN models with the active learning trick in Sec. 3.3
The initial neighborhood
Graph Neural
Local
Augmentation
Network
Generative model
The feature matrix
The initial neighborhood as input generative
The augmented neighborhood
obtained local augmentation
Figure 1: A schematic depiction of our local augmentation. The purple and yellow circles on the graph correspond the central node and augmented neighbors respectively. After augmenting the neighborhood, exploit the initial and the generated feature matrix as input for downstream GNNs.
3.1 LEARNING THE CONDITIONAL DISTRIBUTION
We start by reviewing the semi- supervise learning of GNNs probabilistic view. Most existing GNN models (Kipf & Welling 2017 Veličković et al. 2018) are viewed as classification function to predict the class labels of the graph nodes. In this work, we use GNN classification estimator PY A, X) (0 is the parameter) to model the conditional distribution of label Y with respect to the graph structure A and feature matrix X. Given training samples {A,X, X, Y). the parameter can be estimated using Maximum Likelihood Estimation (MLE), by optimizing the following likelihood function:
max II Po (Y,)A,X), KEK
(2)
3

---

Under review as a conference paper at ICLR 2022
where K is the set of node indices of the training dataset whose labels visible during the semi-supervine training. To further boost the performance of GNN, we introduce a new model PY X|A A X), where X generated features by feature-level augmentation. For this model, the MLE method needs to optimize a marginalized probability Pg over the generated feature matrix X:
II
Po (Y,, X|A.)
KEK X
For Bayesian tractability, we decompose Pg in Eq. as a product of two posterior probabilities:
(Yx = Po (Y A, X,
(4)
where P, X) and Qx A, X) denote the probabilistic distributions approximated by the downstream GNN and the (feature-level augmentation) generator respectively, parameterized by 0 and ø. There are two benefits in the decomposition above. First, it allows to decouple the training of the downstream predictor Po and the generator Qs. enabling the generator to easily generalize to other downstream tasks. Moreover, inspired by the successes of data augmentation via deep-learning-basee bband generative modeling (Antoniou et al. (017)). the representation power of Eq.(4) is superior than that of a single predictor Po (Y A, X) without data augmentation.
Consequently, once a generator Q+ is trained very well, our training procedure can optimize Po A, X) with samples X drawn from the fixed conditional distribution Qo- Now, we show how train the generator as follows.
Generator To learn a feature augmentation generator, naive solution is to learn one single distribution for all the neighbors using the MLE method, i.e., solving the following optimization problem
max Σ
log (X, X,) = max log JEN.
II Pe (X), JEN,
(5)
t
where IIEN, X. Then Py can be used to augment features for all the neighbors. However, this method ignores the differences between all the neighbors, which may induce severe noise.
To overcome the limitation, we assume that each neighbor satisfies a different conditional distribution. Specifically, there exists conditional distribution [(- 2,) with latent random variable 2j. such that we have X, ~ XXX;;, 2,) for X JIJ,e, Once we obtain p(- ((-)X,22 22) some way, we can generate augmented features X, and then we can train Po (Y A, X, X) instead of Po A, X) to improve the final performance of Pg. Below, we will present how to find p(-(X,,,,) 2,). which will produce the generator Qo.
To achieve our purpose, suitable method is the conditional variational auto-encoder (CVAB) KKiggma & Welling 2013 Sohn et al. 2013). which can help learn the distribution of the latent variable Zj. and the conditional distribution p(- XX 2.). So, CVAE model Q₂ (X A, X) is adopted as our generator, where ø = {1p, }. yp denotes the variational parameters and & represents the generative parameters. To derive the optimization problem for CVAE, log Pa (X) XX)) can be written with latent
variables as follows, following previous work (Pandey & Dukkipati 2017 Sohn et al. 2015):
logpy (X, X,) / (z) X.) log
Pio (X X,
dz + K X.)l, X.))
U (z),,, X,
(x)x X,) log (X,, X;)
dz,
Ap (z|X,, X,)
and the evidence lower bound (ELBO) can be written as:
2(X,. x: KL (lp / 90 (z(,, X.) Pio (X, X,, z)dz, (6)
where the encoder q (z)
X,)
NX(X,, Xz), 9X,,, X,)) and decoder pa (X,) X,, z)
(h(,X,,,,) z), c/). The encoder is a two-layer MLP. f and g share the first layer, and their second layers employ different parameters. The decoder is two-layer MLP. For simplicity and tractability,
the implemented generator Q (X] A, X) uses the same parameters across all nodes eq V.
4

---

Under review as a conference paper at ICLR 2022
Optimization of the MLE Now, present how to optimize the MLE Eq. (4) using the feature matrix produced from the generator. Once the augmented feature matrix can be sampled from the generator,
optimize the parameters of Eq.(4) in the following way. Firstly, the parameter Φ = { Φ can be optimized by maximizing the ELBO of the generator 6). i.e., we train the generator. Secondly, the parameter I is optimized by maximizing the MLL Eq. 4 with ø fixed, which the conditional distribution of Y given A, X, and X, i.e., train the downstream GNN model.
In this paper, the MLE is formulated by a downstream GNN model as follows:
Po Y A, -Z(0)A, 6), where x EvET Efer Yas In (softmax (GNN(A,,,X))11)
3.2 THE ARCHITECTURE LA-GNN
We discuss the details of downstream GNN models. And GCN, GAT, GCNII, and GRAND as the backbones and test them on semi-supervised node classification tasks. We name the modified GNN architecture LA-GNN, where LA means local augmentation.
LA-GCN 2-layer LA-GCN defined as follows:
( (A A (a (a (AxW(") (AX. I...) (... (AX, w
where X; (i 1,2, ,n) the augmented feature matrix produced by the generator, denotes an operator of column-wist -wisn concatenation, w!") (i = 2... ,,,...,7 denotes the parameters of the first LA-GCN layer, and ( denotes the parameters of the second LA-GCN layer.
LA-GCNII Since GCNII (Chen et al. 2020) applies fully-connected neural network on X to obtain a lower-dimensional initial representation H (U) before the forward propagation, we apply fully-connected neural network on X and X to obtain ()) for LA-GCNII as follows:
H (x
||o
X W (0) II-I) 1I.-.-10 Xx,, W n+1)
(9)
(()) is fed into the next forward propagation layer. Besides, we do not modify the architecture of GAT and GRAND, and just add our generated feature matrix to the input.
3.3 ACTIVE LEARNING
In this section, we introduce a trick for the overall training framework. After the training of the generator finishes, it contains an issue of using Q (X A, X) (4) for inference because Q may generate some samples from the side part of the distribution. This critical question makes the inferences inefficient. Inspired by Nielsen & Okoniewski (2019), we introduce active learning to capture the suitable generated feature matrix and the corresponding generator, which improves the inference efficiency and helps the optimization of the MLE. During active learning, the probability of each feature proportional its uncertainty evaluated by acquisition function. We adopt the Bayesian Active Learning by Disagreement (BALD) acquisition function (Houlsby et al., 2011 to sample the most important inferences with the approximation from the Monte Carlo (MC) dropout samples as
N P w..) N
N (Y.
UXX)
+ H
1
M
(10)
where is the number of MC samples and wn are the parameters of the network sampled for the n-th MC dropout sample. A high BLAD score indicates network with high uncertainty about the generated feature matrix. So it tends to be selected to improve the GNN model. Finally, the overall algorithm framework is summarized in Algorithm which shows the optimization of 4).
5

---

Under review as a conference paper at ICLR 2022
Algorithm 1 The framework to train the Generator Qo and the downstream GNN Po using the initial feature matrix X and the generated feature matrix X selected by the acquisition function
Input: Adjacency matrix A, feature matrix X
1: Initialize Un-inf, X. Qs. X,, and Q%
2: for = to the number of generator iterations do
3: Train the generator Qo using A and X
4: Generate feature matrix X using Qo
Compute (XX using Eq.400.
6:
if (TU())> then
7:
U = (X) if f > N Nwarmup then
8:
9:
Train GNN Pa using and X the number of continued GNN training iterations X'=X,
10:
4 =0=Qa
X=
Q==Qz Q
Train the downstream GNN P4 with the generated feature matrix X. and generator Qs
4 DISCUSSION
In this section, we discuss the motivation of this work and provide some analysis.
Connection EP-B and GraphSAGE We discuss how our proposed model distinguishes from the classical representation learning models on graphs. Previous methods such EP-B (García- Durán Niepert 2017) and GraphSAGE (Hamilton et al. 2017) rely on reconstruction loss function between the central node and its neighbors' embeddings. EP-B aims minimize the reconstruction error by optimizing the objective min Enev)( ( d(X,,, X.) d(X.. X..) where X., repre-
sents the target node; X. denotes the neighbor nodes; X, AGG(X, e N(v)) indicates the reconstruction from neighbors: and refers to the bias. Besides, GraphSAGE exploits the negative sampling to differentiate the representations of remote node-pairs. GraphS enforce nearby nodes to have similar representations and to enforce disparate nodes to be distinct by minimizing the objective min EEw>N(v) log (xx(x. - XEe., log ((o(-X," s., X.))) where X,, denotes target node; X,, represents the neighbor node; X., is disparate node; and (v(v) is the negative sampling. These approaches build upon the assumption that adjacent nodes share similar attributes. In contrast, our model does not rely on such assumption and instead generates the neighboring node features from the conditional distribution of central node representations. Given the target node, X,, our aim is to learn the conditional distribution the neighbor nodes, X., A comparison between the reconstruction-baser representation learning graphs and our proposed framework is illustrated in Figure 2 And our local augmentation method is the third paradigm to exploit neighbors generative way.
rectinstuce recenstuct
X.a
Xuz
Xa1
A
Xu2
Xui
simbarr
Infer
Infer
loss
dissimilar
X
X
X
Xua
(a) The Original Graph
EP-B
(c) GraphSAGE
(d) Local Augmentation
Figure 2: (a) The original graph. (b) EP-B exploits the neighbors to reconstruct the central node's 's embedding. (c) GraphSAGE encourages nearby nodes to have similar embeddings. (d) Given the representation of the central node, our aim is to infer the representations of the connected distribution of neighbors.
Local Augmentation vs. General Augmentation General image augmentation algorithms in- clude geometric transformations, feature space augmentation, adversarial training, and generative adversarial networks (Shorten & Khoshgoftaar, 2019). It is impossible to apply geometric transfor- mations directly to graph data augmentation since graphs sensitive to node permutation. General

---

Under review as a conference paper at ICLR 2022
adversarial training, feature space augmentation, and generative adversarial networks don't take the graph structure into account. Graphs consist of set of identities with certain pairs these identities connected by edges. We need to consider node features and the graph structure when designing the graph data augmentation framework. Our proposed method of local augmentation fully considers these two points. By extracting the neighbors' feature vectors, we have enough data points learn the distribution. There are two benefits to designing local augmentation. First, by taking the sub- graph structure and feature representation associated with this sub-graph structure as input for the generative model, we can learn the information of the sub-graph structure. Second, the number of data points learn the distribution depends on the node degree. This assures that we have enough data points compared with the general feature augmentation and we can learn a better distribution.
Complementing missing information Jia Benson (2000) points out that some attribute informa- tion might be missing subset of vertices. By learning the distribution of node representations from the observed data, we can utilize the produced node representations from the generative model to com- plement the information missing in the nodes' attributes, which boosts the robustness of downstream tasks. And we show that our model still works the scenario that nodes lose a certain percentage of attributes. In other words, we can exploit the well-learned distribution complement the contextual information of the local neighborhood to enhance the locality of the node representations.
5 EXPERIMENTS
In this section, we evaluate the perfor- mance of our proposed model on semi- supervised node classification tasks on a variety of public graph datasets and com- pare our model with the state-of-the-an graph neural networks. We also carry out additional experiments to showcase the ne- cessity our design and its robustness missing information.
Table 2: Classification results on fixed split (%)
Method
Cora
Citeseer
Pubmed
Chebyshev
Defferrard et al.
2016)
81.2
69.8
74.4
APPNP Klicpera el 2019
83.8
71.6
79.7
MixHop (bb 13
2019
81.9
71.4
80.8
Graph U-ner (Cia &
2019
84.4
73.2
79.6
GSNN-M [Wang
20208
83.9
72.2
S²GC Zhu Konius
2021
83,5
73.6
80.2
GCN (Kipf Welling
2017
70.3
G G-GCN (Zhu et al
2124
83.7
71.3
80.9
DropEdge-GC [Rone
2020
82.8
72.3
79.6
5.1 DATASETS
BAUG-O-GCN
Ziio et 1041
83,6
73.3
79.3
LA-GCN
84.1
72.5
81.3
We utilize seven public graph datasets (Cora, Citeseer, Pubmed, Squirrel, Ac- tor, Chamelcon, and Cornell) for semi- supervised node classification tasks. The details of these datasets can be found in the appendix.
GAT Veličković al.
2018
83.0
70.4
OOM
LA-GAI
72.3
GCNI (Chen al LA-GCNII
2020
85.2
73.1
80.0
73.7
GRAND (Feng al. LA-GRAND
2020
75.4
85.8
75.8
83,3
5.2 SEMI-SUPERVISED NODE CLASSIFICATION
Baselines and Experimental Setup. We apply the standard fixed splits YYngg et al. 2016) on three datasets Cora, Citeseer, and Pubmed, with 20 nodes per class for train- ing, 500 nodes for validation, and 1,000 nodes for testing. And we consider four backbones: GCN (Kipf & Welling 2017).
Table 3: Classification results on random split (%)
Method
Squirrel
Actor
Chameleon
Cornell
APPNP S'GC
21.6
32.1
21.3
27.8
30.2
57.2
22.5
Drop GCN
21.9
26.5
25.0
53.6
LA-GCN
23.2
27.0
28.9
56.1
GAT (Veličković et al. 2018), GCNII (Chen al. 2020), and GRAND (Feng al. 2020)
GAT
24.2
27.2
34.8
55.8
LA-GAT
28.2
to evaluate our proposed framework and com- pare our model against state-of-thed the-art mod- els including 1) backbone models: Cheby- shev (Defferrard et al. 2016). GCN, GAT,
GENI
25.3
31.9
30.2
57.3
LA-GCNIT
28.6
32.7
32.5
56.6
APPNP (Klicpera al. 2019), Graph U-net (Gao & Ji 2019). MixHop (Abu-El-Haija et al. 2019).
GCNII, GSNN-M (Wang et al. 2020a)), S²GC (Zhu & Koniusz 2021). and GRAND and 2) feature-
level and topology-level augmentation models: G-GNNS (Zhu et al. 2020), DropEdge (Rong et al.
7

---

Under review as a conference paper at ICLR 2022
2020) and GAUG-O (Zhao et al., 2021). For four datasets Squirrel, Actor, Chameleon, and Cornell, we take 10 random splits (Shchur et al, 2018) where 10%, 30%, and 60% of the date for training. validation, testing; measure the performance of GCN, GAT, GCNII, and corresponding modified models.
Results For three datasets Cora, Citeseer, and Pubmed, we report the mean classification accu- racy on the test nodes of all our models after 100 runs and report the values after running the ex- periments of their models with server under their setting hyperparameters in their original pa- pers. The results of the evaluation experiments are summarized in Tables 2 3 and in the ap- pendix, which demonstrate that the backbone models equipped with our method achieve the best performance across all the datasets except the Cornell dataset. More specifically, we can improve upon GCN by a margin of 2.5%, 2.2%, and 2.4% on Cora, Citeseer, and Pubmed respec- tively. Moreover, LA-GNN outperforms other backbone models including GAT and GCNII as well as data augmentation models (Zhu et al. 2020; Rong et al., 2020; ZZhao al. 2021) these citation network datasets. Besids, we also provide the analysis of the distribution of our generated feature matrix. And Figure 3 shows the distribution of the attributes of the original and inference neighbors, which can demonstrate our inference feature matrix follow the distribution of the initial feature matrix.
distribution altributes reighbore
Original Neighbors Inference Neighbars
0.10
hM
Damalty
0.08
0.04
0.02
0.00
10 15 20
Feature Attiibute Bin
Figure 3: The distribution of the attribute bin of the inference neighbors the distribution of the attribute bin of the original neighbors, with KL divergence 0.0026. The value of each feature bin is the sum of the attribute values of multiple di- mensions of the feature vector. We split the feature vector into multiple feature bins.
5.3 ABLATION STUDY
In this section, demonstrate the effectiveness of our proposed generative framework, we con- duct experiments that compare LA-GNN sev- eral of its ablated variants without generative modeling. The results are shown Table 4 "GCN + width" only increases the first network layer width for GCN and GCNII match LA- GNN without giving generated samples input. "+ concatenation" " only replaces the generated feature matrix of LA-GNN with the original fea- ture matrix of the central node. "+ plain neigh- borhood" replaces the generated feature matrix of LA-GNN with a neighborhood feature ma- trix where each row corresponds to the feature vector of the randomly sampled neighbor. The results show that the first variants provide no notable improvement for the backbone models, and the third variant even results in degradation. By eliminating the possibility that these confounding factors irrelevant our core approach may contribute the final performance, it's evident that the performance gain in Table 2 and are due to our proposed generative local augmentation framework.
Table 4: Effects of different components of our framework evaluated on the standard split of the Cora, Citeseer and Pubmed dataset.
Method
Cora
Citeseer
Pubmed
GCN GCNI
81.6
70.3
78.9
85.2
73.1
80.0
GCN width
82.0
71.4
79.5
GCN + concatenation
81.8
71.6
78.8
GCN plain neighborhood
80.9
68.8
75.0
GCNI width
85.1
73.1
80.2
GCNI concalenation
85.2
73.3
80.2
GCNI plain neighhorhood
83.3
71.9
78.1
LA-GCN
84.1
72.5
81.3
LA-GCNII
85.2
73.7
81.6
5.4 ROBUSTNESS TO MISSING INFORMATION
In this section, we conduct experiments to verify that our proposed framework can robustify down- stream tasks against missing information in the feature attributes. Specifically, mask certain percentage of the attributes of each feature vector and use the same pipeline to do augmentation for the masked feature matrix. As shown in Table we that as the mask ratio increases,
8

---

Under review as a conference paper at ICLR 2022
the gap of the performance between the GCN and LA-GCN enlarges in most cases in Cora and Citeseer, which corroborates our insight discussed in Section [4] Since there exists large redundancy in the features of the Pubmed dataset, the performance of GCN and LA-GCN decreases little the mask ratio increases and the gap of the performance does not enlarge. To conclude, our model can complement the contextual information of the local neighhorhood to enhance the locality of the node representations.
Table Summary of results on recovering study in terms of classification accuracy (%). 4 means a decrease compared with the accuracy if features are not masked.
Puhned
Misk Ratio
088
0.3
GCN (100))06)) (1.6)) Bod (115) (L5() (.02)) (2.3)) (7.2(131) (1.3 (UB4) (1054 (1.5(1144) 76.9
(3.5)) (1.1 (1.(1225) (1300) (.0.3) (108) 69.3 65.9 (16.6) (1.1) (10.6) (0.5(1100) (12.)
RELATED WORK
Graph Neural Networks In general, convolution in the graph domain involves non-spectral (spa- tial) and spectral approaches. Non-spectral methods generalize convolutions operating on spatially close neighbors to the graph domain, such as Duvenaud et (2015): Atwood & Towsley (2016); Niepert et al. (016); Monti et al. (2017). Spectral approaches define the convolution operations based on the spectral formulation, such as Bruna al. 20014); Defferrard et al. (2016); Kipt & Welling (2017). Recently, several methods Abuu-ElHaaja et al. Lia Liao et al. (2019) based on GCN have been proposed to obtain the higher-order filters. Besides, GAT (Veličković et al., 2018). Graph U-Nets (Gao & Ji. 2019) combine attention networks and pooling operation with GNN separately, which achieve state-of-the-art performance on node and link classification tasks. In this work, local augmentation can be applied on various backbone models to improve performance.
Graph Generative Models Generative models (Goodfellow et al. 2014 Kingma & Welling 2013) are powerful tools of learning data distribution through unsupervised learning, and they have achieved tremendous success in various applications. Recently, researchers have proposed several interesting generative models for graph data generation. Variational graph auto-encoder (VGAE) (Kipf & Welling 2016) makes use of latent variables and learns interpretable latent representations for undirected graphs. [Salha et al. (2019) replace the GCN encoder in VGAE with simple linear model and emphasize the effectiveness of simple node encoding scheme. Xu et al. (2019) propose a generative model framework to learn node representations, by sampling graph generation sequences constructed from observed graph data. ConDgen (Yang et al., 2019) exploits the GCN encoder to handle the inherent challenges of flexible context-structure conditioning and permutation-invaiice generation. Besides, some methods have been proposed to apply the graph generative models in various applications such as graph matching (Simonovsky & Komodakis 2018), molecule design (Liu) et al. 2018), retrosynthesis prediction (Shi al.) (020) and chemical design (Samanta et al. 201 8). Compared with these approaches mainly focusing on structure generation, our model takes full use of the power of the generative model for feature representation generation, which can serve as an enhanced technique for the downstream backbone models.
7 CONCLUSION
We propose local augmentation, a brand-new technique that exploits the generative model to learn the conditional distribution of the central node's neighbors' feature representations given its represen- tation. We can augment more 1 1-hop neighbors from a well-trained generative model to enhance the performance of backbone GNN models. Experiments show that our model can improve performance across various GNN architectures and benchmark datasets by enriching local information. Besides, our model achieves new state-of-the-art results on various semi- semi-supervised node classification tasks. One limitation of our proposed framework is that we do not exploit the 2-hop neighbors or use the random walk to find more related neighbors for the central node. And one future work is that can extract more 2/3-hop neighbors if the central node's degree is small and learn the conditional distribution for random sampling nodes if the graph is large.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving. Michael Isard, et al. Tensorflow: system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI) 16), pp. 265-283, 2016.
Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In international conference on machine learning. pp. 21 21-29. PMLR, 2019.
Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv preprini arXiv: 1711.3311 2017.
James Atwood and Don Towsley. Diffusiinn-convluutiona neural networks. In Advances in neural information processing systems, pp. 1993-2001, 2016.
Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. In International Conference on Learning Representations, 2014.
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International Conference on Machine Learning, pp. 1725-1735. PMLR, 2020.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems, 2016.
Zhijie Deng, Yinpeng Dong, and Jun Zhu, Batch virtual adversarial training for graph convolutional networks. arXiv preprint arXiv: / /902.09992 2019.
David Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparin Iparraguire., Rafael Gómez-Bore Bombarelli, Timo- thy Hirzel, Alán Aspuru-Guzik, and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. In Advances in Neural Information Processing Systems, 2015.
Fuli Feng, Xiangnan He, Jie Tang, and Tat-Seng Chua. Graph adversarial training: Dynamically regularizing based on graph structure. IEEE Transactions on Knowledge and Data Engineering, 2019.
Wenzheng Feng, Jie Zhang, Yuxiao Dong, Yu Han, Huanbo Luan, Qian Xu, Qiang Yang, Evgeny Kharlamov, and Jie Tang. Graph random neural network for semi-supervised learning on graphs. In 20, 2020.
Hongyang Gao and Shuiwang Ji. Graph u-nets. In international conference on machine learning, pp- 2083-2092. PMLR, 2019.
Alberto García-Durán and Mathias Niepert. Learning graph representations with embedding propaga- tion. In Advances in Neural Information Processing Systems, 2017.
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In Advances in Neural Information Processing Systems, 2014.
Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. Attention based spatial- temporal graph convolutional networks for traffic flow forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 922-929, 2019.
William L Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances Neural Information Processing Systems, 2017.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv: / 1/2.5745 2011.
10

---

Under review as a conference paper at ICLR 2022
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700 7700-4708, 2017.
Junteng Jia and Austion R Benson. Residual correlation in graph neural network regression. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 588 508,5998200 2020.
Diederik P Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational bayes.
arXiv preprint
Thomas N Kipf and Max Welling. Variational graph nuto-encoders. NIPS Workshop on Bayesian Deep Learning, 2016.
Thomas N. Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representation, 2017.
Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günneeann. Predict then propagate:
Graph neural networks meet personalized pagerank. In International Conference on Learning Representation. 2019.
Kezhi Kong, Guohao Li, Mucong Ding, Zuxuan Wu, Chen Zhu, Bernard Ghanem, Gavin Taylor, and Tom Goldstein. Flag: Adversarial data augmentation for graph neural networks. arXiv preprint arXiv:2010.09891. 2020.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
Renjie Liao, Zhizhen Zhao, Raquel Urtasun, and Richard Zemel. Lanczosnet: Multi-scale deep graph convolutional networks. In International Conference on Learning Representations, 2019.
Qi Liu, Miltiadis Allamanis, Marc Brockschmidt, and Alexander L Gaunt. Constrained graph variational autoencoders for molccule design. Advances in Neural Information Processing Systems, 2018.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed repre- sentations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, 2013.
Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. Geometric deep learning on graphs and manifolds using mixture model enns. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5115-5124, 2017.
Christopher Nielsen and Michal M Okoniewski. Gan data augmentation through active learning inspired sample acquisition. In CVPR Workshops, pp. 109-112, 2019.
Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. Learning convolutional neural networks for graphs. In International conference on machine learning, pp. 2014-2023. PMLR, 2016.
Gaurav Pandey and Ambedkar Dukkipati. Variational methods for conditional multimodal deep learning. In 2017 International Joint Conference on Neural Networks (IJCNN), pp. 308-315. IEEE, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances Neural Information Processing Systems, 2019.
Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gen: Geometric graph convolutional networks. In International Conference Learning Representations, 2020.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representa- tions. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 701-710, 2014.
11

---

Under review as a conference paper at ICLR 2022
Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convo- lutional networks on node classification. In International Conference on Learning Representation. 2020.
Benedek Rozemberezki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. Journal of Complex Networks, 9(2)cnab001, 2021.
Guillaume Salha, Romain Hennequin, and Michalis Vazirgiannis. Keep it simple: Graph autoen- coders without graph convolutional networks. Workshop on Graph Representation Learning, 33rd Conference on Neural Information Processing Systems (NeurIPS), 2019.
Bidisha Samanta, Abir De, Niloy Ganguly, and Manuel Komez-Rodriguue Designing random graph models using variational autoencoders with applications chemical design. arXiv preprint arXiv: 2018.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification network data. magazine, 29(3):93-93, 2008
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprini arXiv: 1.000668 2018.
Chence Shi, Minkai Xu, Hongyu Guo, Ming Zhang, and Jian Tang. A graph graphs framework for retrosynthesis prediction. International Conference on Machine Learning, pp. 8818-8827. PMLR, 2020.
Connor Shorten and Taghi M Khoshgoftaar. A survey image data augmentation for deep learning. Journal of Big Data, 6(1):1-48, 2019.
Martin Simonovsky and Nikos Komodakis. Graphvae: Towards generation of small graphs using variational autoencoders. In International Conference on Artificial Neural Networks, pp. 412-422. Springer, 2018.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances neural information processing systems, 28:3483-3491. 2015.
Jie Tang, Jimeng Sun, Chi Wang. and Zi Yang. Social influence analysis in large-scale networks. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 807-816, 2009.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations. 2018.
Haibo Wang, Chuan Zhou, Xin Chen, Jia Wu, Shirui Pan, and Jilong Wang. Graph stochastic neural networks for semi-supervised learning. In Advances in Neural Information Processing Systems, 2020a.
Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma. Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang Deep graph library: A graph-centric, perforppe package for graph neural networks. arXiv preprint 10X.v:1909.01315. 2019.
Yiwei Wang, Wei Wang, Yuxuan Liang, Yujun Cai, Juncheng Liu, and Bryan Hooi. Nodeaug: Semi-supervised node classification with data augmentation. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 207-217, 2020b.
Yu Wang and Tyler Derr. Tree decomposed graph neural network. In Proceedings of the 30th ACM International Conference on Information Knowledge Management, pp. 2040-2049, 2021.
Da Xu, Chuanwei Ruan, Kamiya Motwani, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Generative graph convolutional network for growing graphs. In ICASSP 2019-2019 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3167-3171. IEEE, 2019.
12

---

Under review as a conference paper at ICLR 2022
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International Conference Machine Learning, pp. 5453-5462. PMLR, 2018.
Carl Yang, Peiye Zhuang, Wenhan Shi, Alan Luu, and Pan Li. Conditional structure generation through graph variational generative adversarial nets. In NeurIPS, pp. 1338-1349, 2019.
Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. International Conference on Machine Learning, pp. 40-48, 2016.
Rex Ying, Ruining Hc, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp- 974 983, 2018.
Hanqing Zeng, Muhan Zhang, Yinglong Xia, Ajitesh Srivastava, Andrey Malevich, Rajgopal Kannan, Viktor Prasanna, Long Jin, and Ren Chen. Decoupling the depth and scope of graph neural networks. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
Tong Zhao, Yozen Liu, Leonardo Neves, Oliver Woodford, Meng Jiang, and Neil Shah. Data augmentation for graph neural networks. In The Thirty-Fifth AAAI Conference Artificial Intelligence, 2021.
Danhao Zhu, Xin- Yu Dai, and Jiajun Chen. Pre-train and learn: Preserve global information for graph neural networks. In Proceedings of the AAAI Conference Artificial Intelligence, 2020.
Hao Zhu and Piotr Koniusz. Simple spectral graph convolution. International Conference on Learning Representations, 2021.
13

---

Under review as a conference paper at ICLR 2022
A
PROOF OF EQ.
We give more details of the derivation of the generator ELBO as follows:
log Pe (x)
q: (z(x
Po X,(X,)
Pv (X,. X,)
10 (x| x,)
log
dz
P
/ 4o (x) X.)
Pv (X), X, X ( X1, z)
log
dz
Pv (X, (X:)p (X;, xi,%)
f n0 (x X log
Pe (X X3, 2.
1
(4,3) dz Pe (X,
Pyl (X;)
= f ! (2|3,, x,)
Py. Pa ),,)
log
Pe (z| X,) dz
= / 4o (z) X,)
Pv (X,, x2(X,) 9o J,, X;) dz
log
Pv (z( X,) 9G (z X;)
/ (z|x X;)
Pvi Pu (X,, 2) x
+ log 9 (z X X.) Pvo (2|X,, x.) dz
log
(p X,)
/ 4 (z|X,,, X.)
PN( x(X.))
log
dz + XX/gg///
24 (z) XX, X.)
2 / (x| X
Pv (X,, 2/ x({,))
log
dz
24 (z) X,, X;)
Pub (X,z x, x,,
LELBO-
16 (2) X.) log
1o (z) |, X,)
Ps (X X,, .z)
90 (z(, log
4u 2z) Xj, ),) (X:) dz
/ 90 (z)x) X.) log
XX, X; 2. paa (X,, 22)
( X,, j, X; k.)p (X,) Psb (X, X,, 2 p (z(X,)
[ 9 (z( log
dz
4u x,, X.)
/ (z), log
dz + / (p(z X.) (x Xx,, z)dz O:0 Xi, xx, X:)
Pv (x) X,)
(2]1 +/ (x,,,,,,, X,) log X z)dz
14

---

Under review as a conference paper at ICLR 2022
B REPRODUCIBILITY
B.1 DATASETS DETAILS
Cora, Citeseer, and Pubmed standard citation network benchmark datasets Sen (2008). In these datasets, nodes represent documents, and edges denote citations; node feature corresponds to elements of a bag-of-worde representation of a document, and node label corresponds one of the academic topics. Besides, we utilize four datasets used in Pei et al. (2020) for evaluation. Chameleon and squirrel are two page-page networks on specific topics Wikipedia Rozemberezki et al. (202T). In these datasets, nodes represent web pages, and edges denote mutual links between pages; node features correspond to several informative nouns the Wikipedia pages and labels correspond to the number of the average monthly traffic of the web page. WebKB from various universities. We use the one subdataset of it, Cornell. In this dataset, nodes represent web pages, and edges hyperlinks between them; node features correspond to the bag-of-wa -fords-words representation of web pages and labels correspond to five categories, student, project, course, staff, and faculty. Film dataset is the actor-only induced subgraph of the filn-iirectoracto--iiiie network (Tang et al. (209)). In this dataset, Nodes represent actors, and edges denote co-occurrence on the same Wikipedia page; node features correspond to some keywords in the Wikipedia pages and labels correspond to five categories in terms of words of actor's Wikipedia. All the dataset statistics are summarized in Table
webpage dataset collected
Table
Datasets statisties
Dataset
Cora
Cite.
Pubm.
Cham.
Squi.
Actor
Corn.
# Nodes
2708
3327
19717
2277
5201
7600
183
# Edges
5429
4732
44338
36101
217073
33544
295
# Features
1433
3703
500
2325
2089
931
1703
# Classes
7
6
3
5
5
5
5
B.2 IMPLEMENTATION DETAILS
We use Pytorch Paszke et al. 2019) to implement LA-GNNs. The codes of SºGC (Zhu & Koniusz
2021), LA-GCN, LA-GAT, LA- GCNII, LA-GRAND, and DropEdge-GCT, are implemented referring
to Pytorch implementation SC GCN Kipf Welling 2017), 7, GAT Veličković et al. 2018
G (Cenn et al. (2020) GRAND (eeng et al. 2020), and DropEdge-GCr GCn et al. 2020
Besides. we implement APPNP (Kliepera et al. 2019) with DGL (Wang et al. (2019) version of
APPNE The datasets Cora, Citeseer, Pubmed are downloaded from TensorFlow (Abadi al. 2016
implementation of GCN" and the datasets Chameleon, Squirrel, Actor, and Cornell are downloaded
from the implementation of Geom-GCNID Pei et al. 2020). All the experiments in this work conducted on a single NVIDIA Tesla V100 with 32GB memory size. The operating system behind the Docker where the experiments are running Red Hat 4.8.2-16 And the software that we use for experiments are Python 3.6.8, numpy 1.19.2, sklearn 0.0, scipy 1.5.4, networkx 2.5.1, torch 1.6.0, torchvision 0.7.0, CUDA 10.2.89, and CUDNN 8.0.2.
1/www/wkh
"htps://gihub.ccmm/// http://www...ocaaaaaaaooo//ssCCc
httpss//ggthbbboom///eee///////
15

---

Under review as a conference paper at ICLR 2022
B.3 HYPERPARAMETER DETAILS
LA-GNNs introduce an additional parameter, that is the hidden layer for generated feature matrix X before concatenation. The difference of architectures between GCN and LA-GCN be found in Figure and the LA-GCNII architecture can be found in Figure 5
X
GCN Layer
H
X
GCN Layer
GCN Layer
GCN Layer
X
GCN Layer
11
H(3)
H
GCN
LA-GCN
Figure GCN and LA-GCN architectures. The difference between GCN and LA-GCN architectures that the LA-GCN has an additional convolutional layer for X and it uses concatenation operation to mix the hidden representations.
X
H
H
H
Figure 5: LA-GCNII architecture. The difference between GCNII and LA-GCNII is that the LA- GCNII has an additional MLP layer for X and it uses concatenation operation to mix the hidden representations.
The difference of hyperparameters between the GCN and LA-GCN is only the hidden layer size before concatenation. For the LA-GCNII, LA-GAT, LA-GRAND, tune the hyperparameters in the same way as described in their original papers with validation set.
16


Reviews:
[{"review_id": "yqxjR8ct_ug", "reviewer": "Reviewer_cVVh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents an interesting method to enhance the graph representations with the proposed local augmentation techniques. The authors propose to leveraging a generator to vary the node features conditioned on the neighbor information in the subgraph.  The generated samples are further re-selected with an importance sampling technique to ensure the generated features are informative to enhance representation learning.\n\nFrom the side of technique in the paper, the proposed method well differs the current augmentation methods and shows the improvement on different backbones. The presentation of the technical part is generally well written and clear. However, I still have some small confusion about the notations in section 2.2. For example: Is $p_\\psi(z|X_i)$ a parameterized by $\\psi$ or not? In the text I found $p_\\psi(z|X_i) = N(0,I)$. \n\n\nThe experiments show the results of classification with different backbones on different datasets and we can see the improvement of local augmentations. Some expected ablations are not shown; thus I have some suggestions for this part to improve this paper:\n\n- I guess it is non-trivial to re-organize Table 3, as it would be easier to compare if the authors could put methods on the same backbone together (e.g. GCN, G-GCN, DE-GCN, GAUG-OGCN and LA-GCN).  \n\n- If it is possible, it will be better to show the results of GAT, GCNII and GRAND with the other augmentation methods. \n\n- I am curious how the proposed local augmentation performs when combined with other augmentation techniques. For example, will it further improve the GNNs by putting LA and DropEdge together? \n\n- How does the importance sampling module improve the generated feature quality? Probably it could be shown in Fig 3 to justify the importance of the proposed module. \n\n------- update after rebuttal ---------------\n\n\nAfter carefully reading all the reviews, the authors' responses, and corresponding revisions, I think the quality of this paper is better than the original submission. However, as their presentation is still not clear enough to me and some points raised by other reviewers, e.g. computation complexity vs performance improvement are not well explained, I am inclined to keep my current rating.", "summary_of_the_paper": "This paper proposes a local augmentation strategy to enhance the graph representation and benefits corresponding downstream tasks, where the augmentation is achieved with a generation conditioned on the local neighbor structure. The authors justify the proposed method with both quantitative and qualitative results on benchmark datasets.", "summary_of_the_review": "This paper presents an interesting local augmentation method in a way of generation to enhance the graph representations and show the results of classification with different backbones on different datasets and we can see the improvement of local augmentations, but can be further improved by providing more ablation studies. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VrUCkaAcLs7", "reviewer": "Reviewer_XsGb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe idea of using generative model to augment GNN training data is interesting.\n\nWeakness:\n1. The writing and organization of this paper need to be polished before submission. The presentation is chaotic. There are also grammatical errors and typos through out the paper. eg. 'data argumentation' in Abstract. The experiment numbers across the seven benchmark datasets are hard to read. It is unclear why they are not presented fully in a single table.\n2. The description of the proposed methods are unnecessarily notion heavy making it time consuming to decipher what the main ideas really were. Authors may consider elaborating the intuitions of the main ideas clearly before formulating them with notions.\n3. Given the complexity of implementing the proposed augmentation methods, the accuracy gain is marginal comparing to the baselines with / without the complex data augmentations. It is unclear if the proposed methods can be useful for real-world applications. \n", "summary_of_the_paper": "This paper presents a data augmentation methods for training graph neural networks in general. They firstly fit a generative model that learns the conditional probability of input node features. The training data are then augmented using the generative model using the proposed importance sampling method and used for training GNNs. The proposed methods were evaluated on 7 public graph benchmark datasets. Authors attempted to show that the proposed augmentation methods could improve the benchmark classification accuracy over the baseline methods.", "summary_of_the_review": "Though the idea of using generative model to augment GNN training data sounds interesting, I would not recommend to accept this paper given its current presentation quality. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NXTLYRDIWxO", "reviewer": "Reviewer_Dob2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Generally, the idea is clear and easy to follow. The proposed local augmentation is feature-level augmentation, which uses CVAE to generate the neighboring features based on the local structure and the center node's features. The augmented features can 1) improve the generalization and 2) recover the noisy/missing features. However, several issues require addressing as listed below.\n\n1. The unclear comparison with baselines. GraphVAE can actually generate the structure and node features by VAE (\\tilde{A} and \\tilde{F} in the paper). It is suggested to discuss the main difference. Moreover, in Section 2.2, it states that the feature-level augmentation \\bar{X} is from other papers [Kong et al., 2020]. It is unclear that this paper modifies [Kong et al., 2020] or directly applies it. Last, the idea of using Skip-gram in graph embedding has been developed before (e.g., Deepwalk). The difference should be highlighted and the papers should be cited.\n2. The theoretical contribution is minor. The optimization of VAE is mature, while there are also several papers using importance sampling for estimating marginal likelihood in VAEs.\n3. The improvement to GRAND is relatively small in Table 3, e.g., 0.47% on Cora dataset. Moreover, the experiments on the robustness to missing information should contain other baselines to demonstrate the superiority of the proposed approach.", "summary_of_the_paper": "This paper studies the problem of feature augmentation for training the graph neural networks. Specifically, given the central nodes features and local structure, the proposed approach estimates the distribution of the node features of neighbors. Experimental results show that the proposed approach can further improve the baseline backbones.", "summary_of_the_review": "The idea is interesting but the difference and theoretical contributions are unclear. It may be the presentation issue but seems to borrow several parts from other papers. Moreover, the improvement is minor (0.47% on Cora dataset).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BuTqPGUa9v5", "reviewer": "Reviewer_E5PW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- This paper solves an important problem, graph augmentation, which has a large impact on many graph-related tasks such as node classification.\n- Eq. (2) to (4) provide a good motivation for why augmentation is able to improve the performance of a classifier, in a probabilistic view.\n- The proposed approach improves various GNN models in benchmark datasets.\n\nWeaknesses\n- The node index i just appears at Eq. (5) without any background information. Does i iterate over all nodes or only the training nodes? Do we assume the independence between different i values? Clarification is needed.\n- The discussion before the “Optimization of the MLE” paragraph seems redundant. The local augmentation is not very different from previous approaches, since X_v works as the condition variable of the variational inference to infer the X_u1 and X_u2; X_v is expected to be similar to X_u1 and X_u2 for better estimation.\n- Eq. (6) is incomplete and needs much clarification. a) What are the structures of networks f and g? b) Do the authors use the reparameterization trick for generating Z? c) Why do the authors use the same f function for both the encoder and decoder functions? d) What is the meaning of running variational inference if all neighbors of each node have Z sampled from the same distribution N(0, I)?\n- The name of Section 2.3 should be active learning rather than importance sampling, since a) the goal is not to estimate an unknown probability distribution, and b) the augmentation is done for improving only the current status of the classifier; there is no guarantee that the quality of \\bar{X} improves globally.\n- The proposed LA-GNN structure contains much more parameters, which result in a higher risk of overfitting as a consequence. An additional study is needed for the number of parameters, running time, and the sensitivity to hyperparameters.\n- It is unclear what the authors want to say with Figure 3; what does it mean?\n- The experiment of Section 4.3 seems meaningless, since the approaches with partial ideas such as “+width” or “+concatenation” are not reasonable at all. For example, “+concatenation” just repeats the original features multiple times. Such results do not give any insights on the proposed approach.\n", "summary_of_the_paper": "Data augmentation remains under-explored on graph-structured data. In this work, the authors introduce local augmentation that enhances the feature of each node by its local subgraph structure. Specifically, they model the data augmentation as a feature generation process: given a node’s feature, they learn the conditional distribution of its neighbors’ features to make a generative augmentation model. Experiments show that the proposed local augmentation makes performance improvement on real-world data.", "summary_of_the_review": "This paper proposes a new graph augmentation method that enhances the performance of various GNN models by augmenting input features. However, the technical quality of this paper is insufficient, and the ideas, writings, and experiments are unclear and overwrapping. The idea of running variational inference for graph augmentation is intriguing, so I suggest the authors further develop their approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new algorithm for data augmentation in graph neural networks. The algorithm works by learning a conditional model of a node's neighbor features, and augment the neighborhood representation using the generative model.

 In response to the reviews, the authors provided long answers and clarified much of the text. Nonetheless, after the discussion, two main concerns remained. First, the presention still felt subpar, too notationally heavy for what was presented. Second, the gains with respect to the baselines were assessed as not sufficiently significant to justify the approach which is substantially more complex than a baseline such as GRAND.
Decision: Reject