GraphCGAN: Convolutional Graph Neural Network with Generative Adversarial Networks
Authors: Sheng Zhang
Rui Song
Wenbin Lu
Venue:  (2021)

Abstract: Graph convolutional networks (GCN) achieved superior performances in graph-based semi-supervised learning (SSL) tasks. 
Generative adversarial networks (GAN) also show the ability to increase the performance in SSL.
However, there is still no good way to combine the GAN and GCN in graph-based SSL tasks.
In this work, we present GraphCGAN, a novel framework to incorporate adversarial learning with convolution-based graph neural networks, to operate on graph-structured data. 
In GraphCGAN, we show that generator can generate topology structure and features of fake nodes jointly and boost the performance of convolution-based graph neural networks classifier.
In a number of experiments on benchmark datasets, we show that the proposed GraphCGAN outperforms the baseline methods by a significant margin. 

model the graph's Under review as a conference paper at ICLR 2021
GRAPHCGAN: CONVOLUTIONAL GRAPH NEURAL NETWORK WITH GENERATIVE ADVERSARIAL NET- WORKS
Anonymous authors Paper under double-bling review
ABSTRACT
Graph convolutional networks (GCN) achieved superior performances in graph- based supervised learning (SSL) tasks. Generative adversarial networks (GAN) also show the ability to increase the performance in SSL. However, there is still no good way to combine the GAN and GCN graph-based SSL tasks. this work, we present GraphCGAN, a novel framework to incorporate adversar- ial learning with convolution- based graph neural network, to operate on graph- structured data. In GraphCGAN, structure and attributes/features of fake nodes jointly and boost the performance of convolution- based graph neural network classifier. In number of experiments on benchmark datasets, we show that the proposed GraphCGAN outperforms the reference methods by significant margin.
show that generator can generate topology
INTRODUCTION
Graph-based semi-supervised learning (SSL) aims classify nodes in graph, where only small amounts of nodes are labeled due to the expensive and time-consuming label collection process. To solve such task, various graph neural networks (GNNs) have been proposed using the idea of convolutional neural networks (CNN) to implicitly propagate the information of labeled nodes to unlabeled nodes through the linkage between nodes (Kipf & Welling 2016) Veličković al., 2017]] Hamilton et al.) 2017). These convolution-based graph neural networks have achieved superior performance on multiple benchmark datasets graph-based SSL tasks (Wu et al. 2019).
Recently, generative adversarial networks (GANs) (Goodfellow et al.) 2014) have been shown power improving the performance of imsee SSL problems (Odenal 2016 Salimans et al. 2016 Li et al. 2019b). In semi-GAN (Salimans et al. 2016). authors converted the Mcllass classi- fication task into solving (M + cllclass problem where the synthetic (M + 1)th class is generated by the GAN's generator. Later Dai et al. provided a theoretical insight that the generated data are able to boost the performance of classifier under certain assumptions. Our work is motivated by the the semi-GAN.
GraphSGAN (Ding et al. 2018) first investigated the adversarial learning over graph, where the graph is embedding into an embedding space and synthetic data are generated in the correspond- ing space. The multi-layer perceptron (MLP) is trained as the classifier on the embedding vectors. However, to our knowledge, there is still no existed method combine the adversarial learning to convolution-be based GNNs on graph-based SSL task. In this work, we explore the potential of incor- porating the convolution-be based GNN and GAN. The challenges of constructing a general framework have three folds: first, the attributed graph data are non-Fuclidean whose distribution contains in- formation of graph topology structure as well as the attributes of nodes. Hence, it is not trivial to construct generator to model the distribution. Second, even the generator distribution, the generator should be trained properly to boost the performance of the classifier, A poor-quality generator would introduce noise to the existed graph and affect the classifier. Third, many variants of GCN have been proposed continuously. The framework should be built with flexi- bility to adapt different convolution-ben based GNNs.
We construct a novel approach called GraphCGAN to deal with above challenges. First, to model the distribution of graph, the generator is built sequentially from two sub-generators: models

---

Under review as a conference paper at ICLR 2021
the attribute information (node's attribute) and another models the graph topology structure (adjacency relation of node). Details can be found in Section 3.1 Second, in GraphCGAN, the generator is trained based on the feature matching technique (Salimans et al. 2016) which minimizes the distance between generated nodes and real nodes in the constructed feature space. This technique showed good performance in SSL tasks in practice. The details for construction of loss functions can be found in Section 3.3] For GCN, the attributes of nodes are aggregated convolutionally by multiple layers. The representation of the last layer is usually considered labels. For variants of GCN, the main differences exist in the strategy of layer aggregation (Hamilton jt al., 2017). our framework, we choose the second the last layer of convolution based GNN as the feature matching functions. Therefore, our framework is easily extended to variants of GCN. More discussions can found in Section 3.2
the prediction for the
PRELIMINARY
We first introduce the notation about graph. Let E) denote a graph. where V is the set of nodes with |V| n and E c V x V is a set of edges with | = m. The adjacency matrix A E V VI is defined as As = 1 if node v; and uy has edge, otherwise Asj = 0. Suppose each node Un has d-dimensional feature x; IRd and single value label semi-supervised learning setting, there is a disjoint partition for the nodes, V = V VM, such that, for U/ E VC, the corresponding label is known and for ūjj Vu the corresponding label is unknown. The distributions of node in labeled set V and unlabeled set V are denoted as pve and pyu, respectively. The semi-supe learning is to learn the label for unlabeled set {uyluz E VM) given adjacency matrix A, feature matrix X xileEV and labels for labeled sets {u/le E V-J.
{1.2. .22...... M). }. In the
2.1 CONVOLUTION BASED GRAPH NEURAL NETWORK CLASSIFIER
Based on the Laplacian smoothing, the convolution-based GNN models propagate the information of nodes features across the nodes' neighbors in each layer. Specifically, in GCN, the layer-wise propagation rule can be defined as follows:
(!+1)) oD- "H ( + bb)), (= 0, 0,1,,,,,, L-1
where ( and (() are layer-specific trainable weight matrix and bias, respectively. a(-) is an activation function. D is the diagonal degree matrix with D': = Σ; Ar.. Hence, D- represents normalization of adjacency matrix A. The initial layer H))) is the feature matrix X. The final layer (L) followed by soft max layer can be viewed the prediction of one-hot representation for the true label y.
Recently, many variants of the GCN layer-wise propagation rule had been proposed, including graph attention network, cluster GCN (Veličković et al.) 2017: Chiang et al. 2019), which achieved state- of-the-ar art performances in many benchmark datasets.
2.2 GENERATIVE ADVERSARIAL NETWORK BASED
LEARNING
In semi-GAN, the classifier C' and generator G play non-cooperative game, where classifier aims classify the unlabeled data well as distinguish the generated data from real data; generator attempts to match feature of real data and that of generated data. Therefore, the objective function for classifier can be divided into two parts (Salimans al. 2016). The first part the supervised loss function
Csup R. log Pe y S M)
which is the log probability of the node label given the real nodes. The second part is the unsuper- vised loss function
Lun-sup Euwpyu log Poly S M|p) Euwpya Poly M 1/b)
which is the sum of log probability of the first M classes for real nodes and the log probability of the (M + 1)th class for generated nodes V The classifier C can be trained by maximize the objective function
Lc Lxuu + Lun-sup-
2

---

Under review as a conference paper at ICLR 2021
For objective function of generator, Salimans et al. (2016) found minimizing feature matching loss in Equation 3 accieved superior performance in practice
LG = Burpiv (f(u)) (x (f(G
(3)
where the feature matching function f(.) maps the input into a feature space and z ~ Pz(z) is drawn from a given distribution like uniform distribution. Furthermore, |Dai et al. (2017) provided theoretical justification that complementary generator G was able boost the performance of classifier C in SSL task.
FRAMEWORK OF GRAPHCGAN
To combine the aforementioned Laplacian smoothing graph and semi-GAN on SSL together, develop GraphCGAN model, using generated nodes to boost the performance of convolution-based GNN models.
3.1 CONSTRUCTION OF GENERATOR FOR GRAPHCGAN
The generator G generates fake node vo by generating feature vector R' and adjacency relation ao E IR. jointly, where 0o.; = 1 if the fake node is connected to real node v; otherwise On,i = 0. Therefore, the distribution for generated node pG (vo) can be expressed by the joint distribution of the corresponding feature and adjacency relation PG (Xo, ag). From the conditional distribution formula, the joint distribution can be written as pa (Xo, ao) pG, (Xo)PG, (ao Xo). We use sub-generators G1 and G2 to generate fake feature Xy and a Xn.xo. respectively. In practice, an ag/Xo can be modeled by Gz(z; xb) G2 (z; G₁ (z)) where the adjacency relation an is constructed by sub-generator G₂ given the input of Xy- The distribution of generated node can be denoted by
PG(uu) (x0, ao) = Pa (x(ox)p(aa)x)) (Z)(())))G
p(G(z)).
(4)
B nodes (Vo.1, Up... B) are generated, the generated feature matrix is denoted as X, (x x x²nn)) ) and generated adjacency matrix has form Ag = (a al.2 ag,n)² Hence, the combined adjacency matrix can be denoted as
Á
A
A
E ()(++B)(x+B))
(5)
A
IB
The combined feature vector is
X
X
RR (n++BBxxd
(6)
Xo
The diagonal degree matrix D [)a++B)(n++)) can be denoted as
D.
0
where D, gaxx
0 D
with Da,ú = E, Aij + Es Ao,b and D3 E BxB with DA,M6 = Σ,
Ao,b +
3.2 ANALYSIS OF CLASSIFIER FOR GRAPHCGAN
In GraphCGAN, we adopt the convolution- based GNN, such as GCN, GraphSage (Hamilton et al., 2017) GAT (Veličković et al.) 2017), the the classifier. The classifier is applied the enlarged graph |X, A] to obtain the prediction ý of nodes V U VG.
3

---

Under review as a conference paper at ICLR 2021
Specially, considering the layer-wise propagation of GCN (Equation I GAN, the propogation rule can be denoted as
the classifier in GraphC-
H (++1) (D
ÁH
W(
b +
D
0
A
A
H.
W (1)
(b
0
D
Ao
IB
H
+
(b))
D AH,
D.,
A
(1)
W(
D Ao
(1)
+ +D +DJHH,
H
[ (1)
(7)
D AH W +b.h
o
DD H + +DB D +bg
(+1)
(+1)
where the first layer chosen as the enlarged feature matrix H) X. Weight matrix (( has the same in Equation 1 Bias vector (()) has dimension (n+ which is denoted as [b())T b b
(TTT
We denote bl" D A w'!) - b(4) to make the format clear. From Equation the layer propagation of real nodes (first rows) follows the same format as the GCN layer propagation in Equation [ As a special case, for the zero generator A, = or X = 0, the performance of classifier VU yd would be the same that of original classifier V.
For the last layer (L) E (g(++B)xA) adopt the strategy in Salimans al. (016 to obtain the (M + 1) class label y by
soft max (H
(xx+B)xx)
(8)
where || denotes concatenation and x E classifier in GraphCGAN follows the same format in Equation 2
(n+ is zero matrix. The loss function for
x\x1
3.3 LOss FUNCTIONS
Let us denote g(:; : (c) the map from feature vector and adjacency vector the space of second to the last layer in convolution based GNN with trainable parameter lc. Specially, in the case of GCN, for node v with feature vector x; and adjacency vector a,
g(x, ag Oc) (L-1)
where H(!-1) denotes the i-th row of (1-1) and fc (
According to Equation 4 the loss function of generator G be decomposed into two parts: the loss functions of sub-generators G₁ and G2 separately. To construct G,, the feature matching function f in Equation 3 should solely depend on feature vector. Therefore, we mask the adjacency matrix A identity matrix I RR (n+ (x in layer propagation. Formally, the feature matching loss function of G is constructed as
LG. = Ex, Lg Ex
(x) (g( 0; 0.0)))
where I; denote the i-th row of identity matrix I Raxx and 0 is the vector.
After Xo = G1 (z) is built, the feature matching loss function of G2 can be constructed similarly from
La. En, (g(x; a;; 0c)) - (g G2(2); 0c))
Therefore, loss function for G can be written as
LG La. + LG.-
(10)
Furthermore, when multiple fake nodes are generated, [Salimans et al. (2016) showed that adding pull-away item loss function increase the entropy of generator which led to better performance
4

---

Under review as a conference paper at ICLR 2021
in practice. The pull-away loss for sub sub-generators G1, G2 can be denoted as
0: 0c) ( [(G[
G BB 1)
|| lg 0, 0c) ((GG))))llGGGG))))))))))))))))))
374
and
1
B
g G2(Z,): [c) (eoo))gg((xx) Xoo. G2(z)),
CG
B(B B
(xo, i G2 (z.): 2c) (xo. G2 (2(z)),ee)
jtt
The loss function for G with pull-away item be written
La La + f cpt G22
(1I)
Besides, Dai al. (2077) constructed the complementary loss by
LG = Ex-pa, log(p(x))](((xX > e), fG. = Ea-pe, log(p(a)) (()()) > >)),
which could also increase performance. Therefore, the loss function for G with complementary loss can be written as
C LG LG, for
(12)
The procedure is formally presented in Algorithm 1
Algorithm 1: GraphCGAN Algorithm
Input: Adjacency matrix A, Node feature X, initialized fake nodes VG [Ao, Xo)- hyper-parameters including dimension of the noise vector dnoises the number of steps KD. and the size of fake nodes B and early stop error.
Output: Prediction Y
while not early stop do
Combine the fake nodes g to the graph and obtain Á and X from Equation and
Equation
Classifier:
iter 0
while itern < KD do
Use convolution-bant -baed GNN as the classifier C, and extract the map to the intermediate layer g(...) as Equation
Train C by minimizing Lo (Equation 2 combined graph, obtain predicted result Y;
iter iterp + 1;
Generator:
Generate random noise vector Z (0, E R Bxdanize:
10
Train generator [G1; ; G22] by minimizing Equation 10 or Equation or Equation
Obtain X G₁ (Z) and Ao Gz (Z; G₁ (Z)).
RELATED WORK
4.1 GRAPH-BASER SEMI-SUPERVISED LEARNING
The challenge for graph-based SSL is to leverage unlabeled data to improve performance in classi- fication. There are three categories of the Graph-based semi-supervised learning. The first one is the Laplacian regularization based methods (Xiaojin & Zoubin 2002; Lu & Getoor 2003 [elkin methods, including Deep (Perozzi al.]
al. 2006). The second type the embedding- 2014 SemiEmb (Weston et al.) 2012). and Planetoid (Yang et al., 2016). The third type is convo- lutional based graph neural networks such as GCN (Kipf & Welling (0166). GAT (Veličković et al.) 2017). ClusterGCN (Chiang et al.) 2019) and DeepGCN (Li et al. 2019a). Such methods address the semi-supervised learning in an end-to-end manner. Convolution based methods perform the graph convolution by taking the weighted average of node's 's neighborhood information. In many graph semi-supervised learning tasks, the convolution- based methods achieved the state-of-f of the-art performance (Wu et al. 2019)

---

Under review as a conference paper at ICLR 2021
4.2 GNN LEARNING WITH GAN
GAN is wildly used in obtaining generative graph models. GraphGAN (Wang et al. 2018) proposed framework for graph embedding task. Specifically, GraphGAN can generate the link relation for center node. However, GraphGAN cannot be applied to attributed graph.
MolGAN (De Cao & Kipf 2018) proposed a framework for generating the attributed graph of molecule by generating the adjacency matrix and feature matrix independently. After that, MolGAN used an the score for the generated molecule as reward function to choose the reasonable combina- tion of attributes and topology structure by an auxiliary reinforcement learning model. In compari- son, GraphCGAN can generate attributes and adjacency matrix of the attributed graph jointly, which capture the correlation between the attributes and topology relation.
DGI [Veličković et al. 2018) proposed a general approach for learning node representations within graph-structtred data in unsupervised manner. For the generator, In DGI, the fake nodes created from a pre-specified corruption function applied on the original nodes. In contrast, our GraphCGAN can generate the fake nodes from dynamic generator during the training GAN pro- cess. For the classifier, the DGI uses GCN only, however, our GraphCGAN is flexible and adaptive other convolution-based GNN models.
4.3
GAN WITH SEMI-SUPERVISED LEARNING
SGAN Odena 6 first introduced the adversarial learning the semi-supervised learning on image classification task. GAN-FM (Salimans et al., 2016) stabilized training process in SGAN by introducing matching and minibatch techniques. KumKr al.] (2017), authors discuss about the effects of adding fake samples and claimed that moderate take samples could improve the performance image classification task.
GraphSGAN Ding et al. (2018) proposed a framework for graph Laplacian regularization based clas- sifier with GAN to solve graph-based semi-supervised learning tasks. GraphSGAN, fake samples in the feature space of hidden layer are generated, hence it can not be applied to convolutional based classifiers. In constrast, our model generates fake nodes directly and is adaptive to convolutional based classifiers.
EXPERIMENTS
In this section, our primary goal is to show that the adversary learning can boost the performance of convolution-based GNNs in graph-based SSL under our framework. We evaluate GraphCGAN on established graph-based benchmark tasks against baseline convolution-based GNN models and some other related methods. We first introduce the dataset, experiment setup and results. Besides, we study the property of the generated nodes from our model during the training process. The ablation study is also provided in this section. The code GraphCGAN-ICI.R.. is provided the supplementary file.
5.1 DATASETS
Three standard citation network benchmark datasets Cora, Citeseer and Pubmed (Sen et al. 2008 are analyzed. We closely follow the setting in Kipf Welling (2016) and Veličković al. (0017) which allows for only 20 nodes class to be used for training. The predictive power of the trained models evaluated on 1000 test nodes, and 500 additional nodes are used for validation purposes.
5.2 EXPERIMENT SETUP AND RESULT
GNNs, GCN and GAT, are considered as classifiers in GraphCGAN. In order to show the generated nodes can help improve the performance of the meth- ods. We adopt the same model setting in the original papers (Kipf & Welling 2016 Veličković et al. the number of layers L is 2, the dimension of the hidden layer is 16, the dropout rate 0.5, activation function in the hidden layer is Relu. For GraphCGAN-GAT, the number of layers L is 2, the dimension of the hidden layer is 8, and number
Two widely used of convolution-bast
2017). Specially, for classifier in GraphCGAN-

---

Under review as a conference paper at ICLR 2021
Figure 1: Ablation study for size of fake nodes B. It can be shown that moderate size of fake nodes can boost the classifier in graph-based SSL.
GCN
GAT
0.82
0.04
0.82
0.80
0.76
Cloiseer pubmned
0.78
Closser
pubmed
0.74
0.72
0.70
0
10
12
2
10
12
log2 of size of take nodes (log(1B1)
log2 size of take nodes (lcg21B1)
Method
Cora
Citeseer
Pubmed
MLP
55.1%
46.5%
71.4%
GraphSGAN (Ding
83.0 + 1.3%
73.1 + 1.8%
77.2 + 2.6 % 76.8 = 0.6 %
DGI relivckavic S).
82.3 + 0.6%
71.8 0.7%
GCN (Kief Welling, 2017)
81.5%
70.3%
79.0 %
GraNhCGAN-GCN (ours)
82.4 + 0.6%
72.6 + 1.0%
79.9 + 1.0%
Gain for GCN
0.9
2.3 %
0.9 %
GAT (velivekovic al., GraphCGAN-GAT (ours)
83.0 + 0.7%
72.5 + 0.7%
79.0 + 0.3%
84.0 0.5%
73.2 + 1.0%
80.7 1.5%
Gain for GAT
1.0 %
0.7 %
1.7 %
Table 1: Summary of results in terms of classification accuracy under 100 repetitions. The best and the second best results are masked in bold font. The results show that GraphCGAN-GCN and GraphCGAN-GAT outperform GCN and GAT in significant margin, respectively.
of attention heads is 8, the dropout rate is 0.6, activation function in the hidden layer is Sigmoid. The hyper-parameter for the weight of L2 regularization is 5e-4. For the generator, we use the loss function in Equation 12 (Ablation study for loss function of generator can be found in Table 2 Ap- pendix A). Cora and Citeseer, we generate B 64 fake nodes. In Pubmed, the number fake nodes is B 256. The ablation study of size of fake nodes are provided in Figure
The results presented in Table I the best and the second best results are masked bold font. We particularly note that both GraphCGAN-GCN and GraphCGAN-CAA outperform GCN and GAT in significant margin, respectively. More specifically, we are able to improve upon GCN by a margin of 0.9%, 2.3% and 0.9% on Cora, Citescer and Pubmed, respectively. Besides, GraphCGAN- GAT can improve upon GAT by a margin 1.0%. 0.7% and 1.7%, suggesting that the adding fake nodes strategy in GraphCGAN model can boost the performance for reference convolution-banl GNN model. To be noticed that GraphCGAN can be easily extended to other convolution based GNN models.
5.3 VISUALIZATION OF GAN PROCESS
In this subsection, we investigate about the distribution of the generated nodes during the training process. We consider three datasets to illustrate generated nodes in different perspectives. For Karate club graph (Zachary 1977). it contains 34 nodes without features. The feature matrix X is set as identity matrix during the training process. Therefore, the plot (first row in Figure 2 of fake nodes shows the distribution of Gz(z; I). It can be found, after training, fake nodes mainly connect to the boundary n which is preferred as discussed GraphSGAN (Ding et al. 20188). MNIST datasets (LeCun al. 1998) contain the images of handwritten digit. We can consider it graph
Roundary nodes nodes connected different clusters

---

Under review as a conference paper at ICLR 2021
with image feature by constructing identity adjaceney matrix À = Therefore, the plot (second row in Figure 2) of fake feature shows the distribution of G1 (z) which has the shape around to digit eight. Last, generated B 256 nodes for Cora dataset which
plotted in two-dimension by "T-HHN (Van Der Maaten 2014) techniques on the feature space of g(; : 0c) shown in the third row in Figure 2 which be considered the distribution for Giz). We can find the generated nodes present as a complementary part for the existed nodes
Figure 2: Representation of the generated nodes during training process. Karat club: Real nodes are shown as colored dots representing different manually assigned groups. Three fake nodes (black) generated, for fake nodes, the initial generated adjacency vectors fake nodes mainly connect to the boundary nodes; MNIST: One fake image as node's feature are generated, after training, the fake image shows the shape around to digit eight; Cora: 256 fake nodes (black) with features are generated, the plots show the t-SNE embedding for the feature space of the graph, the generated nodes show complementary part for the existed nodes (colored dots).
set as 1, after training,
Data
Karate Club (Adj only)
MNIST (Feature only)
Cora (Adj+Feature)
Epoch
Epoch 5
Epoch 10
6 CONCLUSION
We propose GraphCGAN, novel framework improve the convolution- GNN using GAN. In GraphCGAN, we design generator generate attributed graph, which is able to generate adjacency matrix and feature jointly. We also provide new insight for the semi-supervised learning with convoluntional graph neural network under GAN structure. A flexible algorithm is proposed, which can be easily extended to other sophisticated architecture GraphC, such as GAAN (Zhang et al, 2018) and GIN (Xu et al. 2018).
One potential future direction is to extend the GraphCGAN other relevant tasks including commu- nity detection, co-embedding of attributed network (Meng et al. 2019) and even graph classification. Extending the model to incorporate edge features by generating the fake edge will allow us to tackle larger amount of problems. Finally, in GAN, the stability of training process can be studied.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. Manifold regularization: A geometric frame- work for learning from labeled and unlabeled examples. Journal of machine learning research, (Nov):2399- -434,, 2006.
Wei-Lin Chiang, Xuanqing I. Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh. Cluster-gen: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp- 257-266, 2019.
Zihang Dai, Zhilin Yang, Fan Yang, William W Cohen, and Russ Salakhutdinov. Good semi- supervised learning that requires bad gan. In Advances in neural information processing systems, pp. 6510-6520, 2017.
Nicola De Cao and Thomas Kipf. Molgan: An implicit generative model for small molecular graphs arXiv preprint arXiv: 1805. 1973, 2018.
Ming Ding, Jie Tang, and Jie Zhang. Semi-supervised learning on graphs with generative adversarial nets. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pp. 913-922. ACM, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs.
In Advances neural information processing systems, pp. 1024-1034, 2017.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: 1609.022090 2016.
Abhishek Kumar, Prasanna Sattigeri, and Tom Fletcher. Semi-supervised learning with gans: Mani- fold invariance with improved inference. In Advances in Neural Information Processing Systems, pp. 5534-5544, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. tradient-based learning applied to document recognition. Proceedings of the IEFF, 86(11):2278-25) 1998.
Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. Deepgens: Can gens go deep as cnns? In Proceedings of the IEEE International Conference on Computer Vision, pp. 9267-9276, 2019a.
Wenyuan Li, Zichen Wang, Jiayun Li, Jennifer Polson, William Speier, and Corey W Arnold. Semi- supervised learning based on generative adversarial network: a comparison between good gan and bad gan approach. In CVPR Workshops, 2019b.
Qing Lu and Lise Getoor. Link-based classification. Proceedings of the 20th International Conference Machine Learning (ICML-03), pp. 496-503, 2003.
Zaiqiao Meng, Shangsong Liang, Jinyuan Fang, and Teng Xiao. Semi-supervisedly co-embedding attributed networks. In Advances in Neural Information Processing Systems, pp. 6504-6513, 2019.
Augustus Odena. Semi-supervised learning with generative adversarial networks. arXiv preprint arXiv:1606.01583, 2016.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social repre- sentations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 701-710. ACM, 2014.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2242, 2016.

---

Under review as a conference paper at ICLR 2021
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi- -aad. Collective classification in network data. Al magazine, 29(3):93-99) 93, 2008.
Laurens Van Der Maaten. Accelerating t-sne using tree-based algorithms. The Journal of Machine Learning Research, 15()):32113325.. 2014.
Petar Veličkovié, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 1700:/001,0000,,, 2017.
Petar Veličković, William Fedus, William L Hamilton, Pictro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. arXiv preprint arXiv:1809.10344/ 2018.
Hongwei Wang, Jia Wang, Jialin Wang, Miao Zhao, Weinan Zhang, Fuzheng Zhang, Xing Xie, and Minyi Guo. Graphgan: Graph representation learning with generative adversarial nets. In Thirty-Second AAAI Conference on Artificial Intelligence. 2018.
Jason Weston, Frédéric Ratle, Hossein Mobahi, and Ronan Collobert. Deep learning via semi- supervised embedding. In Neural Networks: Tricks of the Trade, pp. 639-655. Springer, 2012.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu. A comprehensive survey graph neural networks. arXiv preprint arXiv: wrXv::9011.00566 2019.
Zhu Xiaojin and Ghahramani Zoubin. Learning from labeled and unlabeled data with label propa- gation. Tech. Rep., Technical Report CMU-CALD-02-107, Carnegie Mellon University, 2002.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint 1xX.v:881000082. 2018.
Zhilin Yang, William W Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. arXiv preprint arXiv:1603 100.0066.3.0881, 2016.
Wayne W Zachary. An information flow model for conflict and fission in small groups. Journal of anthropological research, pp. 452-473, 1977.
Jiani Zhang, Xingjian Shi, Junyuan Xie, Hao Ma, Irwin King, and Dit- Yan Yeung. Gaan:
Gated attention networks for learning on large and spatiotemporal graphs. arXiv preprint arXiv:1803.07294, 2018.
A ABLATION STUDY FOR LOSS FUNCTION OF GENERATOR
Method
Cora
Citeseer
Pubmed
GCN (Kip' & Welling,
81.5%
70.3%
79.0 %
GraphCGAN -CN (Co)
81.6 + 0.5%
71.6 0.6%
79.2 + 0.5%
CraphCGAN-GCI (C2)
82.3 = 0.7%
72.5 + 1.0%
80.0 + 1.1%
GraNhCGAN-GCN (Lz)
82.4 + 0.6%
72.6 + 1.0%
79.9 + 1.0%
GAT (velivokovic al., 2017)
83.0 = 0.7%
72.5 = 0.7%
79.0 1 0.3%
GraphCGAN-GAT (Lo)
82.9 + 0.4%
72.9 + 0.6%
78.9 + 0.5%
GraphCGAN-GAT (4%
84.1 + 0.6%
72.9 1.1%
80.0 1.4%
GraphCGAN-GAT (Lg)
84.0 + 0.5%
73.2 + 1.0%
80.7 + 1.5%
Table 2: Ablation study on loss function of generator.
10


Reviews:
[{"review_id": "qz85LtkiAmY", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper combines adversarial learning with graph neural networks to improve the performance of GNN for semi-supervised node classification. To generate fake nodes, the authors designed generator G1 to generate node attributes of fake nodes and G2 to generate the links of the fake nodes to existing nodes.\n\n+ Positive\n1. The idea of combining GNN with GAN for semi-supervised node classification makes sense\n2. The authors provides visualization of the generated fake nodes to help understand the proposed method.\n3. The proposed method is flexible, which can be used for various GNNs\n\n- Negative\n1. The novelty of the paper is limited. It is simple extension of GAN for semi-supervised learning to GNNs. The proposed method heavily relies on existing techniques with little novelty.\n2. It is unclear why the generated fake nodes can only link to existing nodes, not to fake nodes. Intuitively, the fake nots are more realistic if they can also link to each other. The authors may need to give some explanations on such design.\n3. Equations (9) and L_G1 are difficult to understand. For \\tilde{H}_i^{L-1}, it aggregates the information of the (L-1)-hop neighborhood of x_i.  It is unclear to me why \\tilde{H}_i^{L-1} can be written as g(x_i, a_i; \\theta_C), i.e., only relevant to x_i and a_i. In L_G1, why do we have I_i for x_i while 0 for G_1(z). Shouldn’t both be \\tilde{I}_i as we treat the adjacency matrix as I.\n\nQuestion for rebuttal:\nPlease answer 2 and 3 above.\n\nJustification for score：\nThe paper studies an important problem. However, it lacks novelty. Some part of the paper is unclear. I will change the rating if the authors can clarify the contribution and novelty, and address my questions.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "pQpGI3zXN3", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel framework to incorporate adversarial learning with convolution-based graph neural network, to operate on graph-structured data.\nThe proposed method is inspiring. However, some problems remain with the proposed technique.\n\n1.\tThis paper announces that it proposes a novel approach called GraphCGAN to deal with the three challenges of constructing a general framework. However, the motivation is not strong enough that the proposed approach looks like a patchwork of two models.\n\n2.\tMore experiments involving different scale networks are needed to prove the effectiveness of the proposed method. \n\nThis paper is globally well organized and clearly written. However, some important details are missing.\n\n1\tThe details about generator are unclear.\n2\tThe paper lacks of analysis on the experimental result.\n3\tThe details about networks are unclear.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "mb7vS5ZO8V0", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##### 1. Summary\nThe paper presents a method to combine graph convolutional neural networks (GCNs) with generative adversarial networks (GANs). The authors focus on the problem of semi-supervised learning on graphs and propose an end-to-end framework in which the generative model is followed by direct convolutions on the graph nodes. Experiments are conducted on standard benchmark datasets and the proposed method, GraphCGAN is compared against several state-of-the-art approaches.\n \n##### 2. Rationale for the score\nAs stated by the authors, the proposed method is an extension of GraphSGAN. In GraphSGAN a similar process of generating fake nodes is performed. As in GraphSGAN, the fake nodes in GraphCGAN are also linked to real nodes in the graph. The difference is that in GraphSGAN a graph Laplacian regularization is done while GraphCGAN applies a convolution on the graph. This could be considered a novel change if the authors were to show that the proposed approach outperforms GraphSGAN. Yet, if there exists an increase in classification performance, it is modest at best. The accuracy and margins of error in Table 1 do not show any actual increase in performance between GraphCGAN and GraphSGAN.\n\n##### 3. Positive aspects\n- The plots included in section 5.3 are interesting and match the behavior previously reported in GraphSGAN.\n- In the analysis, the authors show how their method compares to GCN and GAT when these two are included as classifiers in the end-to-end learning process of GraphCGAN. This is a good idea and the aim is to determine if there are advantages in using the generative model. Although the idea is good, there are issues with the reported values (see section 4 of this review).\n- The authors included the code associated with the method. This is always appreciated.\n\n##### 4. Negative aspects\n- The experiments are not comprehensive and it is not clear if the authors performed the experiments from scratch. For example, the performance values reported for GAT and GCN are the same as the ones listed in Table 3 of [R1].\n- The claim that GraphCGAN outperforms GCN and GAT is not supported by all the results in Table 1. For example, the performance on Pubmed is within the margin of error for GCN. Similarly for GAT, on all three datasets.\n- When comparing against state-of-the-art methods, the authors omitted the comparison against Graph U-Nets [R1]. This is, in my opinion, the main weakness of this manuscript. If we have a look at the results obtained by Graph U-Nets (reported in Table 3 of [R1]), GraphCGAN does not seem to outperform Graph U-Nets in any dataset, including margin of errors. \n- The authors refer to an ablation study but only a couple of parameters are analyzed, e.g., number of fake nodes in Figure 1 and loss function of the generator in Table 2. The many other parameters of the model are listed in section 5.2 without details about how they were derived.\n\n##### 5. Questions to be addressed during rebuttal period\n* How does GraphCGAN compare to Graph U-Nets on the Cora, Citeseer and Pubmed datasets?\n* The margin of errors are quite high, similarly to those reported for GraphSGAN. Can the authors elaborate on why this is the case?\n* Can the authors confirm on what set (hopefully a partition of the training set) were the plots in Figure 1 obtained? There is no margin of error associated to each point in the plots and it gives the impression of overfitting to the test set.\n\n##### 6. Additional references\n[R1] Gao, Hongyang, and Shuiwang Ji. \"Graph U-Nets.\" ICML 2019. PMLR 97:2083-2092", "extra_scores": {}, "extra_text": {}}, {"review_id": "MElkMy8Rznu", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros: \n1.\tThis paper proposes the first combination of GNN with GAN for semi-supervised learning.\n2.\tThe structure of the paper is clear and easy to follow.\nCons:\n1.\tSome sentences are hard to parse and many grammar errors.\n2.\tThe contribution and novelty are limited.\n\nThis paper deals with semi-supervised learning on graphs based on GANs by proposing a framework named GraphCGAN.  The proposed framework can be easily extended to include other GNN methods. However, the novelty of the proposed model is limited, and the motivation of this paper is not strong. Also, the only motivation stated by the authors is to improve the performance on semi-supervised learning, while the improvement of the performance in the experiments is limited (i.e. ~1% improvement). The followings are the details of comments regarding this paper from three aspects.\n\n1.\tMotivation and significance are not clear.\n(1)\tThe author(s) claim(s) that combining GCN with GANs could boost the performance of semi-supervised learning, which however is not solidly validated by the experiment results. \n(2)\tThe author claims that GANs have never been applied to the SSL task in graphs. However, the existing method GraphSGAN [1] has already done this and GraphSGAN shows better scalability than GCN. The only difference between the proposed GraphCGAN and GraphSGAN is the selection of classifiers (MLP vs. GCN). Thus, the significance of the proposed GraphCGAN is not clear compared to the existing GCN and GraphSGAN. \nAs a summary, considering the limited improvement on performance, I would like to see more other motivations why we need graph-convolution-based GANs for semi-supervised learning on graphs.\n\n       [1] Ding, M., Tang, J., & Zhang, J. (2018, October). Semi-supervised learning on graphs with generative adversarial nets. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (pp. 913-922).\n\n2.\tThere are some technical issues.\n(1)\tThe generator seems very weak. As stated in Section 3.1, the generator generates each node and edge only based on randomized latent representation “z”. If so, each new node (and edge) is generated independently without considering the dependency on the existing nodes and the dependency between the nodes and its adjacent relations.  \n(2)\tEquation 5 is confusing. As expressed in Equation 5, the adjacent matrix of all the newly generated nodes is defined as a unit diagonal matrix I_B. It is not clear why the adjacent relation between each pair of the newly generated nodes is set to zero. \n(3)\tThe author mentions “pull-away item” and “complementary loss” without giving any explanations on these two items in the loss function.  An ablation study is better to be conducted to validate the necessity of these two terms in loss function.\n\n3\tThe presentation of the paper has many grammar errors, typos, and hard-parsing sentences.\n(1)\tIt is hard to parse “…as the map from feature vector and adjacency vector to the space of second to the last layer in convolution-based GNN...”. It is not clear what “the space of second to the last layer in convolution-based GNN” refers to.\n(2)\tIt is hard to parse “….and extract the map to the intermediate layer g(:; :) ….” In Algorithm 1. The term “map” is confusing. \n(3)\tIt is not clear what \"i\" refers to in the equation of L_{G_2}. If it refers to the index of one node, then there may lost a “sum” symbol to sum up all the nodes into the loss.\n(4)\tThere are many grammar errors and typos, e.g., \n-----“existed methods”\n                 -----the font of word “softmax” is not unified, see Section 2.1 on Page 2.\n                 -----“…Later on, Dai et al. provided a theoretical….” lacks the hyper-reference.\n----- “: =”  in Equation 4.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method to combine graph convolutional neural networks (GCNs) with generative adversarial networks (GANs) for graph-based semi-supervised learning.

**Strengths:**
  * It is a reasonable attempt to combine GCN with GAN for semi-supervised node classification.
  * The proposed method is general in that it can work with different graph neural networks.

**Weaknesses:**
  * The novelty of this work is limited.
  * The proposed method, GraphCGAN, has no significant performance improvement over state-of-the-art methods.
  * The writing has much room to improve in terms of both clarity and the linguistic quality.

Since both the novelty and the significance of this paper in its current form are limited, it is premature for publication. There is consensus among all the reviewers that this paper is not up to the acceptance standard of ICLR.

Decision: Reject